tools: ipkdbg: Generate gdb environments from opkg package archives
ipkdbg serves interactive debugging and coredump analysis of split-debug
binaries by exploiting bitbake's runtime package management support
outside the context of the BMC.
To enable ipkdbg in your environment you will need to be familiar with
bitbake's support of [package feeds][package-feeds].
[package-feeds]: https://docs.yoctoproject.org/dev-manual/common-tasks.html?highlight=package+feed#using-runtime-package-management
ipkdbg MUST have access to an appopriate opkg.conf that identifies the
location of the ipk package archive from which the binary under
inspection was installed. ipkdbg supports fetching opkg.conf from a
well-known, remote location if required.
ipkdbg MUST have access to a gdb binary that supports multi-arch for
cross-architecture debugging.
It is RECOMMENDED that ipkdbg also has access to the opkg database used
for populating the rootfs of the BMC firmware image. This is used for
reverse-mapping of absolute binary paths to the package that installed
the binary. With this capability, it is no-longer necessary to list the
set of packages to include in the debug rootfs on the ipkdbg
command-line when processing a core dump, they are automatically
discovered through extracting the path of the failed binary from the
core file.
To make bitbake retain the opkg database for a given build, set
[`INC_IPK_IMAGE_GEN = "1"`][incremental-builds] in your bitbake
configuration, and capture
`./tmp/work/*/obmc-phosphor-image/1.0-r0/temp/saved` as a build artefact
using the following incantation:
$ tar -cJf opkg-database.tar.xz \
-C ./tmp/work/*/obmc-phosphor-image/1.0-r0/temp/saved/target/ \
info lists status
[incremental-builds]: https://git.openembedded.org/openembedded-core/commit/?id=adf587e55c0f9bc74f0bef415273c937401baebb
Finally, opkg binaries are not provided directly due to licensing and
distribution concerns. The binaries should be built and copied into a
bin/ directory alongside `ipkdbg.in` using the
${arch}/${release_id}/${release_version_id}/opkg scheme outlined in the
code:
```
ipkdbg_opkg_path() {
...
local arch=$(uname -m)
local release_id=$(. /etc/os-release; echo $ID)
local release_version_id=$(. /etc/os-release; echo $VERSION_ID)
local p=${root}/bin/${arch}/${release_id}/${release_version_id}/opkg
...
```
Once placed in bin/ the Makefile handles stripping and archiving them
for packaging into the final `ipkdbg` script.
A helper script for building opkg, `build-opkg`, is provided in place of
the binaries themselves.
Help output:
$ ./ipkdbg -h
NAME
ipkdbg - debug OpenBMC applications from an (internally) released firmware
SYNOPSIS
ipkdbg [-q] RELEASE FILE CORE [PACKAGE...]
DESCRIPTION
RELEASE is the firmware release whose packages to install
FILE is the absolute path to the binary of interest in the target environment
CORE is an optional core file generated by FILE. Pass '-' for no core file
PACKAGES will be used to populate a temporary rootfs for debugging FILE
OPTIONS
-h
Print this help.
-q
Quit gdb once done. Intended for use in a scripting environment in combination
with a core file, as the backtrace will be printed as an implicit first command.
ENVIRONMENT
There are several important environment variables controlling the behaviour of
the script:
IPKDBG_OPKG_CACHE
A package cache directory for opkg. Defaults to empty, disabling the cache.
IPKDBG_CONF_HOST
Hostname for access to opkg.conf over the web interface
Defaults to 'host.local'
IPKDBG_CONF_MNT
Mount-point for access to opkg.conf
Defaults to 'mountpoint'
IPKDBG_CONF_LOC
Geo-location for access to opkg.conf
Defaults to 'themoon'
IPKDBG_CONF_ROOT
Path to the directory containing build artifacts, for access to opkg.conf
Defaults to 'path'
IPKDBG_CONF_USER
Username for access to opkg.conf over the web interface
Defaults to $USER (andrew)
IPKDBG_GDB
The gdb(1) binary to invoke. Automatically detected if unset.
IPKDBG_WGET_OPTS
User options to pass to wget(1) when fetching opkg.conf. Defaults to
'--quiet'
IPKDBG_ZSTD
The zstd(1) binary to extract the compressed core dump. Automatically
detected if unset.
EXAMPLE
ipkdbg 1020.2206.20220208a \
/usr/bin/nvmesensor - \
dbus-sensors dbus-sensors-dbg
Signed-off-by: Andrew Jeffery <andrew@aj.id.au>
Change-Id: Ib5a7619d0c657754bc0fa2e04cd97e64e4b6da47
diff --git a/ipkdbg/ipkdbg.in b/ipkdbg/ipkdbg.in
new file mode 100644
index 0000000..ee5bb5c
--- /dev/null
+++ b/ipkdbg/ipkdbg.in
@@ -0,0 +1,430 @@
+#!/bin/sh
+
+set -eu
+
+: ${IPKDBG_OPKG_CACHE:=}
+: ${IPKDBG_CONF_HOST:=host.local}
+: ${IPKDBG_CONF_MNT:=mountpoint}
+: ${IPKDBG_CONF_LOC:=themoon}
+: ${IPKDBG_CONF_ROOT:=path}
+: ${IPKDBG_CONF_USER:=$USER}
+: ${IPKDBG_WGET_OPTS:="--quiet"}
+: ${IPKDBG_ZSTD:=zstd}
+
+ipkdbg_error() {
+ /bin/echo -e "$@" | fold >&2
+}
+
+ipkdbg_info() {
+ /bin/echo -e "$@" | fold
+}
+
+ipkdbg_help() {
+/bin/echo -e "\033[1mNAME\033[0m"
+/bin/echo -e "\tipkdbg - debug OpenBMC applications from an (internally) released firmware"
+/bin/echo -e
+/bin/echo -e "\033[1mSYNOPSIS\033[0m"
+/bin/echo -e "\tipkdbg [-q] RELEASE FILE CORE [PACKAGE...]"
+/bin/echo -e
+/bin/echo -e "\033[1mDESCRIPTION\033[0m"
+/bin/echo -e "\tRELEASE is the firmware release whose packages to install"
+/bin/echo -e "\tFILE is the absolute path to the binary of interest in the target environment"
+/bin/echo -e "\tCORE is an optional core file generated by FILE. Pass '-' for no core file"
+/bin/echo -e "\tPACKAGES will be used to populate a temporary rootfs for debugging FILE"
+/bin/echo -e
+/bin/echo -e "\033[1mOPTIONS\033[0m"
+/bin/echo -e "\t\033[1m-h\033[0m"
+/bin/echo -e "\tPrint this help."
+/bin/echo -e
+/bin/echo -e "\t\033[1m-q\033[0m"
+/bin/echo -e "\tQuit gdb once done. Intended for use in a scripting environment in combination"
+/bin/echo -e "\twith a core file, as the backtrace will be printed as an implicit first command."
+/bin/echo -e
+/bin/echo -e "\033[1mENVIRONMENT\033[0m"
+/bin/echo -e "\tThere are several important environment variables controlling the behaviour of"
+/bin/echo -e "\tthe script:"
+/bin/echo -e
+/bin/echo -e "\t\033[1mIPKDBG_OPKG_CACHE\033[0m"
+/bin/echo -e "\tA package cache directory for opkg. Defaults to empty, disabling the cache."
+/bin/echo -e
+/bin/echo -e "\t\033[1mIPKDBG_CONF_HOST\033[0m"
+/bin/echo -e "\tHostname for access to opkg.conf over the web interface"
+/bin/echo -e
+/bin/echo -e "\tDefaults to '${IPKDBG_CONF_HOST}'"
+/bin/echo -e
+/bin/echo -e "\t\033[1mIPKDBG_CONF_MNT\033[0m"
+/bin/echo -e "\tMount-point for access to opkg.conf"
+/bin/echo -e
+/bin/echo -e "\tDefaults to '${IPKDBG_CONF_MNT}'"
+/bin/echo -e
+/bin/echo -e "\t\033[1mIPKDBG_CONF_LOC\033[0m"
+/bin/echo -e "\tGeo-location for access to opkg.conf"
+/bin/echo -e
+/bin/echo -e "\tDefaults to '${IPKDBG_CONF_LOC}'"
+/bin/echo -e
+/bin/echo -e "\t\033[1mIPKDBG_CONF_ROOT\033[0m"
+/bin/echo -e "\tPath to the directory containing build artifacts, for access to opkg.conf"
+/bin/echo -e
+/bin/echo -e "\tDefaults to '${IPKDBG_CONF_ROOT}'"
+/bin/echo -e
+/bin/echo -e "\t\033[1mIPKDBG_CONF_USER\033[0m"
+/bin/echo -e "\tUsername for access to opkg.conf over the web interface"
+/bin/echo -e
+/bin/echo -e "\tDefaults to \$USER ($USER)"
+/bin/echo -e
+/bin/echo -e "\t\033[1mIPKDBG_GDB\033[0m"
+/bin/echo -e "\tThe gdb(1) binary to invoke. Automatically detected if unset."
+/bin/echo -e
+/bin/echo -e "\t\033[1mIPKDBG_WGET_OPTS\033[0m"
+/bin/echo -e "\tUser options to pass to wget(1) when fetching opkg.conf. Defaults to"
+/bin/echo -e "\t'$IPKDBG_WGET_OPTS'"
+/bin/echo -e
+/bin/echo -e "\t\033[1mIPKDBG_ZSTD\033[0m"
+/bin/echo -e "\tThe zstd(1) binary to extract the compressed core dump. Automatically"
+/bin/echo -e "\tdetected if unset."
+/bin/echo -e
+/bin/echo -e "\033[1mEXAMPLE\033[0m"
+/bin/echo -e "\tipkdbg 1020.2206.20220208a \\"
+/bin/echo -e "\t\t/usr/bin/nvmesensor - \\"
+/bin/echo -e "\t\tdbus-sensors dbus-sensors-dbg"
+}
+
+IPKDBG_OPT_QUIT=0
+
+while getopts hq f
+do
+ case $f in
+ q) IPKDBG_OPT_QUIT=1;;
+ h|\?) ipkdbg_help ; exit 1;;
+ esac
+done
+shift $(expr $OPTIND - 1)
+
+trap ipkdbg_help EXIT
+
+ipkdbg_core_extract()
+{
+ if [ "-" = "$1" ]
+ then
+ echo -
+ else
+ local src="$(realpath "$1")"
+ local dst="${src%.zst}"
+
+ command -v $IPKDBG_ZSTD > /dev/null
+ $IPKDBG_ZSTD --decompress --quiet --quiet --force -o "$dst" "$src" || true
+ echo "$dst"
+ fi
+}
+
+IPKDBG_BUILD=$1; shift
+IPKDBG_FILE=$1; shift
+IPKDBG_CORE=$(ipkdbg_core_extract "$1"); shift
+IPKDBG_PKGS=$@
+
+: ${IPKDBG_GDB:=}
+if [ -n "$IPKDBG_GDB" ]
+then
+ ipkdbg_info "Using provided gdb command '$IPKDBG_GDB'"
+else
+ os_id=$(. /etc/os-release; echo ${ID}-${VERSION_ID})
+ case $os_id in
+ rhel-8.6 | fedora*)
+ IPKDBG_GDB=gdb
+ if [ -z "$(command -v $IPKDBG_GDB)" ]
+ then
+ ipkdbg_error "Please install the gdb package:"
+ ipkdbg_error
+ ipkdbg_error "\tsudo dnf install gdb"
+ ipkdbg_error
+ exit 1
+ fi
+ ;;
+ rhel*)
+ IPKDBG_GDB=gdb-multiarch
+ if [ -z "$(command -v $IPKDBG_GDB)" ]
+ then
+ ipkdbg_error "Please install the gdb-multiarch package:"
+ ipkdbg_error
+ ipkdbg_error "\tsudo dnf install gdb-multiarch"
+ ipkdbg_error
+ exit 1
+ fi
+ ;;
+ ubuntu*)
+ IPKDBG_GDB=gdb-multiarch
+ if [ -z "$(command -v $IPKDBG_GDB)" ]
+ then
+ ipkdbg_error "Please Install the gdb-multiarch package"
+ ipkdbg_error
+ ipkdbg_error "\tsudo apt install gdb-multiarch"
+ ipkdbg_error
+ exit 1
+ fi
+ ;;
+ *)
+ ipkdbg_error "Unrecognised distribution $release_id. Please set IPKDBG_GDB or " \
+ "install an appropriate gdb binary to invoke"
+ exit 1
+ ;;
+ esac
+ ipkdbg_info "Using gdb command ${IPKDBG_GDB} ($(command -v $IPKDBG_GDB))"
+fi
+
+ipkdbg_archive_extract() {
+ local offset=$1
+ local work=$2
+ tail -n+$offset $0 | base64 --decode - | tar -xz -C $work
+}
+
+ipkdbg_opkg_path() {
+ local root=$1
+ local arch=$(uname -m)
+ local release_id=$(. /etc/os-release; echo $ID)
+ local release_version_id=$(. /etc/os-release; echo $VERSION_ID)
+ local p=${root}/bin/${arch}/${release_id}/${release_version_id}/opkg
+ if [ ! -x "$p" ]
+ then
+ ipkdbg_error "Unsupported environment:"
+ ipkdbg_error
+ ipkdbg_error "Architecture:\t$arch"
+ ipkdbg_error "Distribution ID:\t$release_id"
+ ipkdbg_error "Distribution Version:\t$release_version_id"
+ exit 1
+ fi
+ echo $p
+}
+
+if [ ! -f $0 ]
+then
+ ipkdbg_error "Please execute the script with a relative or absolute path"
+ exit 1
+fi
+
+IPKDBG_DATA=$(awk '/^__ARCHIVE_BEGIN__$/ { print NR + 1; exit 0 }' $0)
+IPKDBG_WORK=$(mktemp -t --directory ipkdbg.XXX)
+IPKDBG_BINS=${IPKDBG_WORK}/tools
+IPKDBG_ROOT=${IPKDBG_WORK}/root
+IPKDBG_CONF=${IPKDBG_WORK}/opkg.conf
+IPKDBG_DB=${IPKDBG_WORK}/database
+
+cleanup() {
+ rm -rf $IPKDBG_WORK
+}
+
+trap cleanup EXIT INT QUIT KILL
+
+mkdir $IPKDBG_BINS $IPKDBG_DB
+ipkdbg_archive_extract $IPKDBG_DATA $IPKDBG_BINS
+
+IPKDBG_OPKG_BIN=$(ipkdbg_opkg_path $IPKDBG_BINS)
+
+ipkdbg_build_gen_path() {
+ local build=$1
+ local component="$2"
+ echo /${IPKDBG_CONF_MNT}/${IPKDBG_CONF_LOC}/${IPKDBG_CONF_ROOT}/${build}/"$component"
+}
+
+ipkdbg_build_gen_url() {
+ local build=$1
+ local component="$2"
+ echo https://${IPKDBG_CONF_HOST}/${IPKDBG_CONF_MNT}/${IPKDBG_CONF_LOC}/${IPKDBG_CONF_ROOT}/${build}/${component}
+}
+
+ipkdbg_build_gen_cache() {
+ local build=$1
+ local component="$2"
+ echo "${HOME}/.cache/ipkdbg/builds/${build}/${component}"
+}
+
+ipkdbg_opkg_conf_gen_path() {
+ local build=$1
+ ipkdbg_build_gen_path $build bmc_ipk/opkg.conf
+}
+
+ipkdbg_opkg_fetch_path() {
+ local path=$1
+ local output=$2
+ cp "$path" "$output" > /dev/null 2>&1
+}
+
+ipkdbg_opkg_conf_gen_url() {
+ local build=$1
+ ipkdbg_build_gen_url $build bmc_ipk/opkg.conf
+}
+
+ipkdbg_opkg_fetch_url() {
+ local url=$1
+ local output=$2
+ # We don't want URL to wrap
+ ipkdbg_info "Authenticating as user $IPKDBG_CONF_USER"
+ if ! wget --http-user=$IPKDBG_CONF_USER \
+ --ask-password \
+ --output-document $output \
+ $IPKDBG_WGET_OPTS \
+ $url
+ then
+ ipkdbg_error "Failed to fetch resource"
+ exit 1
+ fi
+}
+
+ipkdbg_opkg_conf_gen_cache() {
+ local build=$1
+ ipkdbg_build_gen_cache $build opkg.conf
+}
+
+ipkdbg_opkg_conf_fetch_cache() {
+ local build=$1
+ local output=$2
+ local path="$(ipkdbg_opkg_conf_gen_cache $build)"
+ cp "$path" "$output" > /dev/null 2>&1
+}
+
+ipkdbg_opkg_conf_install() {
+ local build=$1
+ local output=$2
+ mkdir -p $(dirname $output)
+ if ! ipkdbg_opkg_conf_fetch_cache $build $output
+ then
+ local cache="$(ipkdbg_opkg_conf_gen_cache $build)"
+ mkdir -p $(dirname $cache)
+ url=
+ ipkdbg_opkg_fetch_path "$(ipkdbg_opkg_conf_gen_path $build)" $cache ||
+ (echo "Configuring opkg via $(ipkdbg_opkg_conf_gen_url $build)" &&
+ ipkdbg_opkg_fetch_url "$(ipkdbg_opkg_conf_gen_url $build)" $cache)
+ ipkdbg_opkg_conf_fetch_cache $build $output
+ fi
+}
+
+ipkdbg_opkg_db_gen_path() {
+ local build=$1
+ ipkdbg_build_gen_path $build bmc_ipk/opkg-database.tar.xz
+}
+
+ipkdbg_opkg_db_gen_url() {
+ local build=$1
+ ipkdbg_build_gen_url ${build} bmc_ipk/opkg-database.tar.xz
+}
+
+ipkdbg_opkg_db_gen_cache() {
+ local build=$1
+ ipkdbg_build_gen_cache $build opkg-database.tar.xz
+}
+
+ipkdbg_opkg_db_install() {
+ local build=$1
+ local root=$2
+ local state=${root}/var/lib/opkg
+ local cache="$(ipkdbg_opkg_db_gen_cache $build)"
+ mkdir -p $state
+ if ! [ -f $cache ]
+ then
+ mkdir -p $(dirname $cache)
+ ipkdbg_opkg_fetch_path "$(ipkdbg_opkg_db_gen_path $build)" $cache ||
+ ipkdbg_opkg_fetch_url "$(ipkdbg_opkg_db_gen_url $build)" $cache ||
+ rm -f $cache
+ fi
+ tar -xf $cache -C $state 2> /dev/null
+ mkdir -p ${root}/usr/local
+ ln -s ${root}/var ${root}/usr/local/var
+}
+
+ipkdbg_opkg() {
+ $IPKDBG_OPKG_BIN \
+ $([ -z "$IPKDBG_OPKG_CACHE" ] ||
+ echo --cache-dir $IPKDBG_OPKG_CACHE --host-cache-dir) \
+ -V1 -f $IPKDBG_CONF -o $IPKDBG_ROOT $@
+}
+
+ipkdbg_gdb_extract_bin() {
+ local core=$1
+ $IPKDBG_GDB --core $core -ex quit 2> /dev/null |
+ awk -F "[\`']" '/Core was generated by/ { print $2 }' |
+ awk -F " " '{ print $1 }' # Chop off the arguments, we only want the binary path
+}
+
+ipkdbg_opkg_find() {
+ ipkdbg_opkg find $@ | awk '{ print $1 }'
+}
+
+ipkdbg_opkg_find_extra() {
+ local pkg=$1
+
+ # Try appending -dbg and -src to the binary package name
+ extra_pkgs="$(ipkdbg_opkg_find ${pkg}-dbg) $(ipkdbg_opkg_find ${pkg}-src)"
+
+ # If that fails, we probably have a split binary package
+ if [ -z "$extra_pkgs" ]
+ then
+ # Strip the last component off as it's probably the split binary package name and
+ # try again
+ extra_pkgs="$(ipkdbg_opkg_find ${pkg%-*}-dbg) $(ipkdbg_opkg_find ${pkg%-*}-src)"
+ fi
+ echo $extra_pkgs
+}
+
+ipkdbg_opkg_conf_install $IPKDBG_BUILD $IPKDBG_CONF
+
+# Extract the binary path from the core
+if [ '-' = "$IPKDBG_FILE" -a '-' != "$IPKDBG_CORE" ]
+then
+ IPKDBG_FILE=$(ipkdbg_gdb_extract_bin $IPKDBG_CORE)
+fi
+
+# Update the package database before potentially looking up the debug packages
+ipkdbg_opkg update
+
+# Extract the package name for the binary
+if [ '-' != "$IPKDBG_CORE" ]
+then
+ if ipkdbg_opkg_db_install $IPKDBG_BUILD $IPKDBG_DB
+ then
+ # Look up the package for the binary
+ IPKDBG_CORE_PKG="$(IPKDBG_ROOT=$IPKDBG_DB ipkdbg_opkg search ${IPKDBG_DB}${IPKDBG_FILE} | awk '{ print $1 }')"
+ if [ -n "$IPKDBG_CORE_PKG" ]
+ then
+ # Look up the extra (debug, source) packages for the binary package
+ IPKDBG_PKGS="$IPKDBG_PKGS $IPKDBG_CORE_PKG"
+ IPKDBG_PKGS="$IPKDBG_PKGS $(ipkdbg_opkg_find_extra $IPKDBG_CORE_PKG)"
+ fi
+ fi
+
+ if [ -z "$IPKDBG_PKGS" ]
+ then
+ ipkdbg_error "Unable to determine package-set to install, please specify" \
+ "appropriate packages on the command line"
+ exit 1
+ fi
+fi
+
+# Force installation of gcc-runtime-dbg to give us debug symbols for libstdc++
+IPKDBG_PKGS="gcc-runtime-dbg $IPKDBG_PKGS"
+
+if [ -n "$IPKDBG_OPKG_CACHE" ]
+then
+ mkdir -p "$IPKDBG_OPKG_CACHE"
+ ipkdbg_opkg install --download-only $IPKDBG_PKGS
+fi
+
+ipkdbg_opkg install $IPKDBG_PKGS | grep -vF 'Warning when extracting archive entry'
+
+cat <<EOF > ${IPKDBG_BINS}/opkg
+#!/bin/sh
+exec $IPKDBG_OPKG_BIN -f $IPKDBG_CONF -o $IPKDBG_ROOT \$@
+EOF
+chmod +x ${IPKDBG_BINS}/opkg
+
+PATH=${IPKDBG_BINS}:${PATH} $IPKDBG_GDB -q \
+ -iex "set solib-absolute-prefix $IPKDBG_ROOT" \
+ -iex "add-auto-load-safe-path $IPKDBG_ROOT" \
+ -iex "set directories $IPKDBG_ROOT" \
+ -iex "cd $IPKDBG_ROOT" \
+ $([ '-' = "$IPKDBG_CORE" ] || echo -ex bt) \
+ $([ 0 -eq $IPKDBG_OPT_QUIT ] || echo -ex quit) \
+ ${IPKDBG_ROOT}${IPKDBG_FILE} \
+ $([ '-' = "$IPKDBG_CORE" ] || echo $IPKDBG_CORE)
+
+exit 0
+
+__ARCHIVE_BEGIN__