ipkdbg: Generate gdb environments from opkg package archives

ipkdbg automates the process of generating gdb environments for interactive debugging and coredump analysis of split-debug binaries by exploiting bitbake's runtime package management support outside the context of the BMC.

Use of ipkdbg

Assuming the presence of the appropriate integration, using ipkdbg requires minimal fuss.

SYNOPSIS
    ipkdbg [-q] RELEASE FILE CORE [PACKAGE...]

It handles the following use-cases:

Core Dump Analysis

After extracting a core file from the BMC an interactive gdb session can be set up against it with:

$ ipkdbg 2.11.0 - my-application.core
...
(gdb)

Where:

1. ipkdbg is the script
2. 2.11.0 is the OpenBMC version tag that locates the appropriate opkg.conf
3. - is the FILE parameter. - specifies it must be extracted from the core
4. my-application.core is a core dump generated by /usr/bin/my-application

Note that for convenience ipkdbg automatically runs bt in the gdb session once the core has been loaded to provide a backtrace.

Note that we don't need to specify any packages to install into the rootfs if the image package database is available. ipkdbg will perform a reverse-lookup to map the absolute binary path extracted from the core to the package that installed it. From there opkg will resolve all the dependencies. Additionally, ipkdbg will identify the associated debug symbol and source packages and install them as well.

If an image package database is not available then ipkdbg will require that the necessary packages are listed on the command-line.

Scripted Backtrace Extraction

For use in scripting environments to automate backtrace extraction ipkdbg has the -q option to quit gdb immediately after the backtrace is printed. The invocation looks like:

ipkdbg -q 2.11.0 - my-application.core

Interactive debug

Interactive debugging is handled by attaching gdbserver on the BMC to the running instance of /usr/bin/my-application on the BMC. ipkdbg is then used to generate the gdb environment:

$ ipkdbg 2.11.0 /usr/bin/my-application -
...
(gdb)

Where:

1. ipkdbg is the script
2. 2.11.0 is the OpenBMC version tag that locates the appropriate opkg.conf
3. /usr/bin/my-application is the absolute path of the application to debug
4. - is the CORE parameter. - specifies that there is no core file

Note that we don't need to specify any packages to install into the rootfs if the image package database is available. ipkdbg will perform a reverse-lookup to map the absolute binary path provided on the command-line to the package that installed it. From there opkg will resolve all the dependencies. Additionally, ipkdbg will identify the associated debug symbol and source packages and install them as well.

If an image package database is not available then ipkdbg will require that the necessary packages are listed on the command-line.

Once the (gdb) prompt is reached the usual remote debugging command applies:

(gdb) target remote $BMC:1234

Whoops, I Forgot A Package

To save exiting the gdb session and re-invoking ipkdbg to add a package to the rootfs environment, ipkdbg installs an opkg helper into the PATH of the gdb process. With this, you can drop to a shell from gdb via the sh command, and then run opkg install ... to install the necessary packages. Exit the shell subprocess to return to your gdb prompt with the new package included in the gdb environment.

Theory of Maintenance

ipkdbg is intended to operate with minimal assumptions about its environment. The assumptions it does make are:

1. It's running in a POSIX environment with access to
   1. coreutils
   2. awk
   3. tar
   4. gzip
   5. wget
2. A multi-arch capable gdb is installed

However, ipkdbg also relies on opkg to get its job done. As opkg is itself a package manager, it tends not to be available as a package via distro package managers. A conflicting requirement is that we want ipkdbg to be able to run almost anywhere without fuss. To resolve these issues ipkdbg packages opkg binaries inside itself as a self-extracting shell-script. What you see in this directory is the input script (ipkdbg.in) and build infrastructure (Makefile) for generating an ipkdbg executable that packages one or more opkg binaries.

ipkdbg supports multiple distros and architectures by defining a directory scheme to contain opkg binaries. This hierarchy is archived and compressed using tar and gzip, and the archive embedded in the ipkdbg shell script by the Makefile. The opkg path for a given architecture, distro and release combination is defined by the following directory hierarchy in terms of os-release(5):

$(uname -m)/${ID}/${VERSION_ID}/opkg

This hierarchy must be placed under a bin/ directory alongside the Makefile.

To avoid licensing and distribution legal issues openbmc-tools does not provide pre-built opkg binaries. You must build your own and integrate them into the hierarchy under bin/ outlined above. To this end, the build-opkg script is provided. It probably won't work without tinkering for your target distro, but it might get you 90% of the way there.

Maintenance in Practice

Once you have built the required opkg binaries and integrated them into the hierarchy under bin/ alongside the Makefile, run make to generate the final ipkdbg script. The output script can then be freely copied between machines supported by the embedded opkg binaries. The build system will handle stripping the opkg binaries to ensure their size is reduced as much as practical prior to archiving and compression. It is worth committing the binaries under bin/ to ensure that the build for your output ipkdbg script is reproducible.

Theory of Integration

For ipkdbg to do its job via opkg it must be possible for it to acquire an opkg.conf that describes the location of the package archive for your target environment. The IPKDBG_CONF_* variables help describe a wget-compatible URL where this configuration lives. ipkdbg will bootstrap by fetching this opkg.conf and then passing it as a parameter to all invocations of opkg, where these invocations of opkg populate a rootfs used as your gdb debugging environment.

A consequence of this is you must also host ipk package archives whose URLs can be described in the acquired opkg.conf. You will need one complete package archive per BMC target environment (tag or release).

The packages can be extracted from the bitbake build tree by archiving tmp/deploy/ipk once bitbake obmc-phosphor-image && bitbake package-index has exited successfully.

Finally, ipkdbg works best when it has access to the image's installed package database. This can be captured from the build tree when the build is configured with INC_IPK_IMAGE_GEN = "1" by archiving the directory hierarchy under ./tmp/work/*/obmc-phosphor-image/1.0-r0/temp/saved with the following incantation:

tar -cJf opkg-database.tar.xz \
    -C ./tmp/work/*/obmc-phosphor-image/1.0-r0/temp/saved/target/ \
    info lists status

ipkdbg assumes that the appropriate opkg-database.tar.xz can be fetched using wget and that its URL can be generated in the same manner as that for opkg.conf.