tools: ipkdbg: Generate gdb environments from opkg package archives

ipkdbg serves interactive debugging and coredump analysis of split-debug
binaries by exploiting bitbake's runtime package management support
outside the context of the BMC.

To enable ipkdbg in your environment you will need to be familiar with
bitbake's support of [package feeds][package-feeds].

[package-feeds]: https://docs.yoctoproject.org/dev-manual/common-tasks.html?highlight=package+feed#using-runtime-package-management

ipkdbg MUST have access to an appopriate opkg.conf that identifies the
location of the ipk package archive from which the binary under
inspection was installed. ipkdbg supports fetching opkg.conf from a
well-known, remote location if required.

ipkdbg MUST have access to a gdb binary that supports multi-arch for
cross-architecture debugging.

It is RECOMMENDED that ipkdbg also has access to the opkg database used
for populating the rootfs of the BMC firmware image. This is used for
reverse-mapping of absolute binary paths to the package that installed
the binary. With this capability, it is no-longer necessary to list the
set of packages to include in the debug rootfs on the ipkdbg
command-line when processing a core dump, they are automatically
discovered through extracting the path of the failed binary from the
core file.

To make bitbake retain the opkg database for a given build, set
[`INC_IPK_IMAGE_GEN = "1"`][incremental-builds] in your bitbake
configuration, and capture
`./tmp/work/*/obmc-phosphor-image/1.0-r0/temp/saved` as a build artefact
using the following incantation:

    $ tar -cJf opkg-database.tar.xz \
        -C ./tmp/work/*/obmc-phosphor-image/1.0-r0/temp/saved/target/ \
        info lists status

[incremental-builds]: https://git.openembedded.org/openembedded-core/commit/?id=adf587e55c0f9bc74f0bef415273c937401baebb

Finally, opkg binaries are not provided directly due to licensing and
distribution concerns. The binaries should be built and copied into a
bin/ directory alongside `ipkdbg.in` using the
${arch}/${release_id}/${release_version_id}/opkg scheme outlined in the
code:

```
ipkdbg_opkg_path() {
...
        local arch=$(uname -m)
        local release_id=$(. /etc/os-release; echo $ID)
        local release_version_id=$(. /etc/os-release; echo $VERSION_ID)
        local p=${root}/bin/${arch}/${release_id}/${release_version_id}/opkg
...
```

Once placed in bin/ the Makefile handles stripping and archiving them
for packaging into the final `ipkdbg` script.

A helper script for building opkg, `build-opkg`, is provided in place of
the binaries themselves.

Help output:

    $ ./ipkdbg -h
    NAME
            ipkdbg - debug OpenBMC applications from an (internally) released firmware

    SYNOPSIS
            ipkdbg [-q] RELEASE FILE CORE [PACKAGE...]

    DESCRIPTION
            RELEASE is the firmware release whose packages to install
            FILE is the absolute path to the binary of interest in the target environment
            CORE is an optional core file generated by FILE. Pass '-' for no core file
            PACKAGES will be used to populate a temporary rootfs for debugging FILE

    OPTIONS
            -h
            Print this help.

            -q
            Quit gdb once done. Intended for use in a scripting environment in combination
            with a core file, as the backtrace will be printed as an implicit first command.

    ENVIRONMENT
            There are several important environment variables controlling the behaviour of
            the script:

            IPKDBG_OPKG_CACHE
            A package cache directory for opkg. Defaults to empty, disabling the cache.

            IPKDBG_CONF_HOST
            Hostname for access to opkg.conf over the web interface

            Defaults to 'host.local'

            IPKDBG_CONF_MNT
            Mount-point for access to opkg.conf

            Defaults to 'mountpoint'

            IPKDBG_CONF_LOC
            Geo-location for access to opkg.conf

            Defaults to 'themoon'

            IPKDBG_CONF_ROOT
            Path to the directory containing build artifacts, for access to opkg.conf

            Defaults to 'path'

            IPKDBG_CONF_USER
            Username for access to opkg.conf over the web interface

            Defaults to $USER (andrew)

            IPKDBG_GDB
            The gdb(1) binary to invoke. Automatically detected if unset.

            IPKDBG_WGET_OPTS
            User options to pass to wget(1) when fetching opkg.conf. Defaults to
            '--quiet'

            IPKDBG_ZSTD
            The zstd(1) binary to extract the compressed core dump. Automatically
            detected if unset.

    EXAMPLE
            ipkdbg 1020.2206.20220208a \
                    /usr/bin/nvmesensor - \
                    dbus-sensors dbus-sensors-dbg

Signed-off-by: Andrew Jeffery <andrew@aj.id.au>
Change-Id: Ib5a7619d0c657754bc0fa2e04cd97e64e4b6da47
diff --git a/ipkdbg/Makefile b/ipkdbg/Makefile
new file mode 100644
index 0000000..9262f1f
--- /dev/null
+++ b/ipkdbg/Makefile
@@ -0,0 +1,44 @@
+all: ipkdbg
+
+ARCH_PPC64LE_BINS=
+ARCH_X86_64_BINS=
+
+SOURCE_DATE_EPOCH=$(shell git log --format='%at' -n1 -- `git ls-files bin`)
+
+STRIPPED_ARCH_PPC64LE_BINS=$(patsubst %,stripped/%,$(ARCH_PPC64LE_BINS))
+STRIPPED_ARCH_X86_64_BINS=$(patsubst %,stripped/%,$(ARCH_X86_64_BINS))
+
+STRIPPED_BINS=$(STRIPPED_ARCH_PPC64LE_BINS) \
+	      $(STRIPPED_ARCH_X86_64_BINS)
+
+$(STRIPPED_ARCH_PPC64LE_BINS): stripped/%: %
+	mkdir -p $(dir $@) && \
+	powerpc64le-linux-gnu-strip --strip-all -o $@ $<
+
+$(STRIPPED_ARCH_X86_64_BINS): stripped/%: %
+	mkdir -p $(dir $@) && \
+	x86_64-linux-gnu-strip --strip-all -o $@ $<
+
+# https://reproducible-builds.org/docs/archives/
+# https://www.gnu.org/software/tar/manual/html_section/create-options.html
+bin.tar.gz: $(STRIPPED_BINS)
+	tar \
+		--sort=name \
+		--clamp-mtime \
+		--mtime="@${SOURCE_DATE_EPOCH}" \
+		--owner=0 --group=0 --numeric-owner \
+		--pax-option=exthdr.name=%d/PaxHeaders/%f,delete=atime,delete=ctime \
+		-czf $@ \
+		-C stripped $(patsubst stripped/%,%,$^)
+
+bin.tar.gz.b64: bin.tar.gz
+	base64 $^ > $@
+
+ipkdbg: ipkdbg.in bin.tar.gz.b64
+	cat $^ > $@
+	chmod +x $@
+
+.PHONY: clean
+clean:
+	$(RM) -r stripped
+	$(RM) bin.tar.gz bin.tar.gz.b64 ipkdbg
diff --git a/ipkdbg/README.md b/ipkdbg/README.md
new file mode 100644
index 0000000..b52b00f
--- /dev/null
+++ b/ipkdbg/README.md
@@ -0,0 +1,194 @@
+## `ipkdbg`: Generate gdb environments from opkg package archives
+
+`ipkdbg` automates the process of generating `gdb` environments for interactive
+debugging and coredump analysis of split-debug binaries by exploiting bitbake's
+runtime package management support outside the context of the BMC.
+
+### Use of `ipkdbg`
+
+Assuming the presence of the [appropriate integration](#theory-of-integration),
+using `ipkdbg` requires minimal fuss.
+
+```
+SYNOPSIS
+	ipkdbg [-q] RELEASE FILE CORE [PACKAGE...]
+```
+
+It handles the following use-cases:
+
+#### Core Dump Analysis
+
+After extracting a core file from the BMC an interactive `gdb` session can be
+set up against it with:
+
+```
+$ ipkdbg 2.11.0 - my-application.core
+...
+(gdb)
+```
+
+Where:
+
+1. `ipkdbg` is the script
+2. `2.11.0` is the OpenBMC version tag that locates the appropriate `opkg.conf`
+3. `-` is the FILE parameter. `-` specifies it must be extracted from the core
+4. `my-application.core` is a core dump generated by `/usr/bin/my-application`
+
+Note that for convenience `ipkdbg` automatically runs `bt` in the `gdb` session
+once the core has been loaded to provide a backtrace.
+
+Note that we don't need to specify any packages to install into the rootfs *if*
+the image package database is available. `ipkdbg` will perform a reverse-lookup
+to map the absolute binary path extracted from the core to the package that
+installed it. From there `opkg` will resolve all the dependencies. Additionally,
+`ipkdbg` will identify the associated debug symbol and source packages and
+install them as well.
+
+If an image package database is not available then `ipkdbg` will require that
+the necessary packages are listed on the command-line.
+
+#### Scripted Backtrace Extraction
+
+For use in scripting environments to automate backtrace extraction `ipkdbg` has
+the `-q` option to quit `gdb` immediately after the backtrace is printed. The
+invocation looks like:
+
+```
+$ ipkdbg -q 2.11.0 - my-application.core
+```
+
+#### Interactive debug
+
+Interactive debugging is handled by attaching `gdbserver` on the BMC to the
+running instance of `/usr/bin/my-application` on the BMC. `ipkdbg` is then used
+to generate the `gdb` environment:
+
+```
+$ ipkdbg 2.11.0 /usr/bin/my-application -
+...
+(gdb)
+```
+
+Where:
+
+1. `ipkdbg` is the script
+2. `2.11.0` is the OpenBMC version tag that locates the appropriate `opkg.conf`
+3. `/usr/bin/my-application` is the absolute path of the application to debug
+4. `-` is the CORE parameter. `-` specifies that there is no core file
+
+Note that we don't need to specify any packages to install into the rootfs *if*
+the image package database is available. `ipkdbg` will perform a reverse-lookup
+to map the absolute binary path provided on the command-line to the package that
+installed it. From there `opkg` will resolve all the dependencies. Additionally,
+`ipkdbg` will identify the associated debug symbol and source packages and
+install them as well.
+
+If an image package database is not available then `ipkdbg` will require that
+the necessary packages are listed on the command-line.
+
+Once the `(gdb)` prompt is reached the usual remote debugging command applies:
+
+```
+(gdb) target remote $BMC:1234
+```
+
+#### Whoops, I Forgot A Package
+
+To save exiting the `gdb` session and re-invoking `ipkdbg` to add a package to
+the rootfs environment, `ipkdbg` installs an `opkg` helper into the PATH of the
+`gdb` process. With this, you can drop to a shell from `gdb` via the `sh`
+command, and then run `opkg install ...` to install the necessary packages. Exit
+the shell subprocess to return to your `gdb` prompt with the new package
+included in the `gdb` environment.
+
+### Theory of Maintenance
+
+`ipkdbg` is intended to operate with minimal assumptions about its environment.
+The assumptions it does make are:
+
+1. It's running in a POSIX environment with access to
+   1. `coreutils`
+   2. `awk`
+   3. `tar`
+   4. `gzip`
+   5. `wget`
+2. A multi-arch capable `gdb` is installed
+
+However, `ipkdbg` also relies on `opkg` to get its job done. As `opkg` is itself
+a package manager, it tends not to be available as a package via distro package
+managers. A conflicting requirement is that we want `ipkdbg` to be able to run
+almost anywhere without fuss. To resolve these issues `ipkdbg` packages `opkg`
+binaries inside itself as a self-extracting shell-script. What you see in this
+directory is the input script ([`ipkdbg.in`](ipkdbg.in)) and build
+infrastructure ([`Makefile`](Makefile)) for generating an `ipkdbg` executable
+that packages one or more `opkg` binaries.
+
+`ipkdbg` supports multiple distros and architectures by defining a directory
+scheme to contain `opkg` binaries. This hierarchy is archived and compressed
+using `tar` and `gzip`, and the archive embedded in the `ipkdbg` shell script by
+the [`Makefile`](Makefile). The `opkg` path for a given architecture, distro and
+release combination is defined by the following directory hierarchy in terms of
+[`os-release(5)`][os-release]:
+
+[os-release]: https://www.man7.org/linux/man-pages/man5/os-release.5.html#OPTIONS
+
+```
+$(uname -m)/${ID}/${VERSION_ID}/opkg
+```
+
+This hierarchy must be placed under a `bin/` directory alongside the
+[`Makefile`](Makefile).
+
+To avoid licensing and distribution legal issues openbmc-tools does not provide
+pre-built `opkg` binaries. You must build your own and integrate them into the
+hierarchy under `bin/` outlined above. To this end, [the
+`build-opkg` script is provided](build-opkg). It probably won't work without
+tinkering for your target distro, but it might get you 90% of the way there.
+
+### Maintenance in Practice
+
+Once you have built the required `opkg` binaries and integrated them into the
+hierarchy under `bin/` alongside the `Makefile`, run `make` to generate the
+final `ipkdbg` script. The output script can then be freely copied between
+machines supported by the embedded `opkg` binaries. The build system will handle
+stripping the `opkg` binaries to ensure their size is reduced as much as
+practical prior to archiving and compression. It is worth committing the
+binaries under `bin/` to ensure that the build for your output `ipkdbg` script
+is reproducible.
+
+### Theory of Integration
+
+For `ipkdbg` to do its job via `opkg` it must be possible for it to acquire an
+`opkg.conf` that describes the location of the package archive for your target
+environment. The `IPKDBG_CONF_*` variables help describe a `wget`-compatible URL
+where this configuration lives. `ipkdbg` will bootstrap by fetching this
+`opkg.conf` and then passing it as a parameter to all invocations of `opkg`,
+where these invocations of `opkg` populate a rootfs used as your `gdb` debugging
+environment.
+
+A consequence of this is you must also host `ipk` package archives whose URLs
+can be described in the acquired `opkg.conf`. You will need one complete package
+archive per BMC target environment (tag or release).
+
+The packages can be extracted from the bitbake build tree by archiving
+`tmp/deploy/ipk` once `bitbake obmc-phosphor-image && bitbake package-index` has
+exited successfully.
+
+Finally, `ipkdbg` works best when it has access to the image's installed
+package database.  This can be captured from the build tree when the build is
+configured with [`INC_IPK_IMAGE_GEN = "1"`][incremental-builds] by archiving
+the directory hierarchy under
+`./tmp/work/*/obmc-phosphor-image/1.0-r0/temp/saved` with the following
+incantation:
+
+```
+$ tar -cJf opkg-database.tar.xz \
+	-C ./tmp/work/*/obmc-phosphor-image/1.0-r0/temp/saved/target/ \
+	info lists status
+```
+
+`ipkdbg` assumes that the appropriate `opkg-database.tar.xz` can be fetched
+using `wget` and that its URL can be generated in the same manner as that for
+`opkg.conf`.
+
+[incremental-builds]: https://git.openembedded.org/openembedded-core/commit/?id=adf587e55c0f9bc74f0bef415273c937401baebb
diff --git a/ipkdbg/build-opkg b/ipkdbg/build-opkg
new file mode 100755
index 0000000..46a6874
--- /dev/null
+++ b/ipkdbg/build-opkg
@@ -0,0 +1,80 @@
+#!/bin/sh
+
+set -eu
+
+set -x
+
+# : ${OPKG_LIBS:="-llzma -lldap -llber -lz -pthread"}
+: ${OPKG_LIBS:="-lz -pthread"}
+
+generate_configure_id() {
+	echo "$@" | sha256sum | awk '{ printf "build-opkg-%s", $1 }'
+}
+
+mark_configured() {
+	rm -f build-opkg-*
+	touch $1
+}
+
+# libarchive
+[ -f libarchive-3.5.2.tar.gz ] || wget http://libarchive.org/downloads/libarchive-3.5.2.tar.gz
+[ -d libarchive-3.5.2 ] || tar -xvf libarchive-3.5.2.tar.gz
+cd libarchive-3.5.2
+LIBARCHIVE_OPTS="\
+--without-zlib \
+--without-bz2lib \
+--without-libb2 \
+--without-lz4 \
+--without-zstd \
+--without-lzo2 \
+--without-cng \
+--without-nettle \
+--without-xml2 \
+--without-expat \
+--disable-acl \
+--disable-xattr \
+--enable-posix-regex-lib=libc \
+--disable-rpath \
+--disable-bsdcat \
+--disable-bsdtar \
+--disable-bsdcpio \
+--with-pic"
+LIBARCHIVE_ID=$(generate_configure_id "$LIBARCHIVE_OPTS")
+[ -f $LIBARCHIVE_ID ] || ( ./configure $LIBARCHIVE_OPTS && mark_configured $LIBARCHIVE_ID )
+mkdir -p root && make -j$(nproc) install DESTDIR=$(realpath root)
+cd ..
+
+# curl
+[ -f curl-7.79.1.tar.bz2 ] || wget https://curl.haxx.se/download/curl-7.79.1.tar.bz2
+[ -d curl-7.79.1 ] || tar -xvf curl-7.79.1.tar.bz2
+cd curl-7.79.1
+CURL_OPTS=--with-openssl
+CURL_ID=$(generate_configure_id "$CURL_OPTS")
+[ -f $CURL_ID ] || ( ./configure $CURL_OPTS && mark_configured $CURL_ID )
+mkdir -p root && make -j$(nproc) install DESTDIR=$(realpath root)
+cd ..
+
+# opkg
+[ -f opkg-0.4.5.tar.gz ] || wget http://downloads.yoctoproject.org/releases/opkg/opkg-0.4.5.tar.gz
+[ -d opkg-0.4.5 ] || tar -xvf opkg-0.4.5.tar.gz
+cd opkg-0.4.5
+OPKG_OPTS="\
+--with-static-libopkg \
+--without-libsolv \
+--enable-curl \
+--enable-openssl \
+--disable-gpg \
+--disable-dependency-tracking"
+OPKG_ID=$(generate_configure_id "$OPKG_OPTS" "$OPKG_LIBS")
+[ -f $OPKG_ID ] || ( \
+	AR_FLAGS=Tcru \
+	PKG_CONFIG_PATH=$(realpath ../libarchive-3.5.2/root/usr/local/lib/pkgconfig/):$(realpath ../curl-7.79.1/root/usr/local/lib/pkgconfig/) \
+	CURL_CFLAGS=-I$(realpath ../curl-7.79.1/root/usr/local/include/) \
+	CURL_LIBS=$(realpath ../curl-7.79.1/root/usr/local/lib/libcurl.a) \
+	LIBARCHIVE_CFLAGS=-I$(realpath ../libarchive-3.5.2/root/usr/local/include/) \
+	LIBARCHIVE_LIBS=$(realpath ../libarchive-3.5.2/root/usr/local/lib/libarchive.a) \
+	LIBS="$OPKG_LIBS" \
+	./configure $OPKG_OPTS && mark_configured $OPKG_ID \
+)
+make -j$(nproc)
+cd ..
diff --git a/ipkdbg/ipkdbg.in b/ipkdbg/ipkdbg.in
new file mode 100644
index 0000000..ee5bb5c
--- /dev/null
+++ b/ipkdbg/ipkdbg.in
@@ -0,0 +1,430 @@
+#!/bin/sh
+
+set -eu
+
+: ${IPKDBG_OPKG_CACHE:=}
+: ${IPKDBG_CONF_HOST:=host.local}
+: ${IPKDBG_CONF_MNT:=mountpoint}
+: ${IPKDBG_CONF_LOC:=themoon}
+: ${IPKDBG_CONF_ROOT:=path}
+: ${IPKDBG_CONF_USER:=$USER}
+: ${IPKDBG_WGET_OPTS:="--quiet"}
+: ${IPKDBG_ZSTD:=zstd}
+
+ipkdbg_error() {
+    /bin/echo -e "$@" | fold >&2
+}
+
+ipkdbg_info() {
+    /bin/echo -e "$@" | fold
+}
+
+ipkdbg_help() {
+/bin/echo -e "\033[1mNAME\033[0m"
+/bin/echo -e "\tipkdbg - debug OpenBMC applications from an (internally) released firmware"
+/bin/echo -e
+/bin/echo -e "\033[1mSYNOPSIS\033[0m"
+/bin/echo -e "\tipkdbg [-q] RELEASE FILE CORE [PACKAGE...]"
+/bin/echo -e
+/bin/echo -e "\033[1mDESCRIPTION\033[0m"
+/bin/echo -e "\tRELEASE is the firmware release whose packages to install"
+/bin/echo -e "\tFILE is the absolute path to the binary of interest in the target environment"
+/bin/echo -e "\tCORE is an optional core file generated by FILE. Pass '-' for no core file"
+/bin/echo -e "\tPACKAGES will be used to populate a temporary rootfs for debugging FILE"
+/bin/echo -e
+/bin/echo -e "\033[1mOPTIONS\033[0m"
+/bin/echo -e "\t\033[1m-h\033[0m"
+/bin/echo -e "\tPrint this help."
+/bin/echo -e
+/bin/echo -e "\t\033[1m-q\033[0m"
+/bin/echo -e "\tQuit gdb once done. Intended for use in a scripting environment in combination"
+/bin/echo -e "\twith a core file, as the backtrace will be printed as an implicit first command."
+/bin/echo -e
+/bin/echo -e "\033[1mENVIRONMENT\033[0m"
+/bin/echo -e "\tThere are several important environment variables controlling the behaviour of"
+/bin/echo -e "\tthe script:"
+/bin/echo -e
+/bin/echo -e "\t\033[1mIPKDBG_OPKG_CACHE\033[0m"
+/bin/echo -e "\tA package cache directory for opkg. Defaults to empty, disabling the cache."
+/bin/echo -e
+/bin/echo -e "\t\033[1mIPKDBG_CONF_HOST\033[0m"
+/bin/echo -e "\tHostname for access to opkg.conf over the web interface"
+/bin/echo -e
+/bin/echo -e "\tDefaults to '${IPKDBG_CONF_HOST}'"
+/bin/echo -e
+/bin/echo -e "\t\033[1mIPKDBG_CONF_MNT\033[0m"
+/bin/echo -e "\tMount-point for access to opkg.conf"
+/bin/echo -e
+/bin/echo -e "\tDefaults to '${IPKDBG_CONF_MNT}'"
+/bin/echo -e
+/bin/echo -e "\t\033[1mIPKDBG_CONF_LOC\033[0m"
+/bin/echo -e "\tGeo-location for access to opkg.conf"
+/bin/echo -e
+/bin/echo -e "\tDefaults to '${IPKDBG_CONF_LOC}'"
+/bin/echo -e
+/bin/echo -e "\t\033[1mIPKDBG_CONF_ROOT\033[0m"
+/bin/echo -e "\tPath to the directory containing build artifacts, for access to opkg.conf"
+/bin/echo -e
+/bin/echo -e "\tDefaults to '${IPKDBG_CONF_ROOT}'"
+/bin/echo -e
+/bin/echo -e "\t\033[1mIPKDBG_CONF_USER\033[0m"
+/bin/echo -e "\tUsername for access to opkg.conf over the web interface"
+/bin/echo -e
+/bin/echo -e "\tDefaults to \$USER ($USER)"
+/bin/echo -e
+/bin/echo -e "\t\033[1mIPKDBG_GDB\033[0m"
+/bin/echo -e "\tThe gdb(1) binary to invoke. Automatically detected if unset."
+/bin/echo -e
+/bin/echo -e "\t\033[1mIPKDBG_WGET_OPTS\033[0m"
+/bin/echo -e "\tUser options to pass to wget(1) when fetching opkg.conf. Defaults to"
+/bin/echo -e "\t'$IPKDBG_WGET_OPTS'"
+/bin/echo -e
+/bin/echo -e "\t\033[1mIPKDBG_ZSTD\033[0m"
+/bin/echo -e "\tThe zstd(1) binary to extract the compressed core dump. Automatically"
+/bin/echo -e "\tdetected if unset."
+/bin/echo -e
+/bin/echo -e "\033[1mEXAMPLE\033[0m"
+/bin/echo -e "\tipkdbg 1020.2206.20220208a \\"
+/bin/echo -e "\t\t/usr/bin/nvmesensor - \\"
+/bin/echo -e "\t\tdbus-sensors dbus-sensors-dbg"
+}
+
+IPKDBG_OPT_QUIT=0
+
+while getopts hq f
+do
+    case $f in
+    q) IPKDBG_OPT_QUIT=1;;
+    h|\?) ipkdbg_help ; exit 1;;
+    esac
+done
+shift $(expr $OPTIND - 1)
+
+trap ipkdbg_help EXIT
+
+ipkdbg_core_extract()
+{
+    if [ "-" = "$1" ]
+    then
+        echo -
+    else
+        local src="$(realpath "$1")"
+        local dst="${src%.zst}"
+
+        command -v $IPKDBG_ZSTD > /dev/null
+        $IPKDBG_ZSTD --decompress --quiet --quiet --force -o "$dst" "$src" || true
+        echo "$dst"
+    fi
+}
+
+IPKDBG_BUILD=$1; shift
+IPKDBG_FILE=$1; shift
+IPKDBG_CORE=$(ipkdbg_core_extract "$1"); shift
+IPKDBG_PKGS=$@
+
+: ${IPKDBG_GDB:=}
+if [ -n "$IPKDBG_GDB" ]
+then
+    ipkdbg_info "Using provided gdb command '$IPKDBG_GDB'"
+else
+    os_id=$(. /etc/os-release; echo ${ID}-${VERSION_ID})
+    case $os_id in
+    rhel-8.6 | fedora*)
+        IPKDBG_GDB=gdb
+        if [ -z "$(command -v $IPKDBG_GDB)" ]
+        then
+            ipkdbg_error "Please install the gdb package:"
+            ipkdbg_error
+            ipkdbg_error "\tsudo dnf install gdb"
+            ipkdbg_error
+            exit 1
+        fi
+        ;;
+    rhel*)
+        IPKDBG_GDB=gdb-multiarch
+        if [ -z "$(command -v $IPKDBG_GDB)" ]
+        then
+            ipkdbg_error "Please install the gdb-multiarch package:"
+            ipkdbg_error
+            ipkdbg_error "\tsudo dnf install gdb-multiarch"
+            ipkdbg_error
+            exit 1
+        fi
+        ;;
+    ubuntu*)
+        IPKDBG_GDB=gdb-multiarch
+        if [ -z "$(command -v $IPKDBG_GDB)" ]
+        then
+            ipkdbg_error "Please Install the gdb-multiarch package"
+            ipkdbg_error
+            ipkdbg_error "\tsudo apt install gdb-multiarch"
+            ipkdbg_error
+            exit 1
+        fi
+        ;;
+    *)
+        ipkdbg_error "Unrecognised distribution $release_id. Please set IPKDBG_GDB or " \
+            "install an appropriate gdb binary to invoke"
+        exit 1
+        ;;
+    esac
+    ipkdbg_info "Using gdb command ${IPKDBG_GDB} ($(command -v $IPKDBG_GDB))"
+fi
+
+ipkdbg_archive_extract() {
+    local offset=$1
+    local work=$2
+    tail -n+$offset $0 | base64 --decode - | tar -xz -C $work
+}
+
+ipkdbg_opkg_path() {
+    local root=$1
+    local arch=$(uname -m)
+    local release_id=$(. /etc/os-release; echo $ID)
+    local release_version_id=$(. /etc/os-release; echo $VERSION_ID)
+    local p=${root}/bin/${arch}/${release_id}/${release_version_id}/opkg
+    if [ ! -x "$p" ]
+    then
+        ipkdbg_error "Unsupported environment:"
+        ipkdbg_error
+        ipkdbg_error "Architecture:\t$arch"
+        ipkdbg_error "Distribution ID:\t$release_id"
+        ipkdbg_error "Distribution Version:\t$release_version_id"
+        exit 1
+    fi
+    echo $p
+}
+
+if [ ! -f $0 ]
+then
+    ipkdbg_error "Please execute the script with a relative or absolute path"
+    exit 1
+fi
+
+IPKDBG_DATA=$(awk '/^__ARCHIVE_BEGIN__$/ { print NR + 1; exit 0 }' $0)
+IPKDBG_WORK=$(mktemp -t --directory ipkdbg.XXX)
+IPKDBG_BINS=${IPKDBG_WORK}/tools
+IPKDBG_ROOT=${IPKDBG_WORK}/root
+IPKDBG_CONF=${IPKDBG_WORK}/opkg.conf
+IPKDBG_DB=${IPKDBG_WORK}/database
+
+cleanup() {
+    rm -rf $IPKDBG_WORK
+}
+
+trap cleanup EXIT INT QUIT KILL
+
+mkdir $IPKDBG_BINS $IPKDBG_DB
+ipkdbg_archive_extract $IPKDBG_DATA $IPKDBG_BINS
+
+IPKDBG_OPKG_BIN=$(ipkdbg_opkg_path $IPKDBG_BINS)
+
+ipkdbg_build_gen_path() {
+    local build=$1
+    local component="$2"
+    echo /${IPKDBG_CONF_MNT}/${IPKDBG_CONF_LOC}/${IPKDBG_CONF_ROOT}/${build}/"$component"
+}
+
+ipkdbg_build_gen_url() {
+    local build=$1
+    local component="$2"
+    echo https://${IPKDBG_CONF_HOST}/${IPKDBG_CONF_MNT}/${IPKDBG_CONF_LOC}/${IPKDBG_CONF_ROOT}/${build}/${component}
+}
+
+ipkdbg_build_gen_cache() {
+    local build=$1
+    local component="$2"
+    echo "${HOME}/.cache/ipkdbg/builds/${build}/${component}"
+}
+
+ipkdbg_opkg_conf_gen_path() {
+    local build=$1
+    ipkdbg_build_gen_path $build bmc_ipk/opkg.conf
+}
+
+ipkdbg_opkg_fetch_path() {
+    local path=$1
+    local output=$2
+    cp "$path" "$output" > /dev/null 2>&1
+}
+
+ipkdbg_opkg_conf_gen_url() {
+    local build=$1
+    ipkdbg_build_gen_url $build bmc_ipk/opkg.conf
+}
+
+ipkdbg_opkg_fetch_url() {
+    local url=$1
+    local output=$2
+    # We don't want URL to wrap
+    ipkdbg_info "Authenticating as user $IPKDBG_CONF_USER"
+    if ! wget --http-user=$IPKDBG_CONF_USER \
+        --ask-password \
+        --output-document $output \
+        $IPKDBG_WGET_OPTS \
+        $url
+    then
+        ipkdbg_error "Failed to fetch resource"
+        exit 1
+    fi
+}
+
+ipkdbg_opkg_conf_gen_cache() {
+    local build=$1
+    ipkdbg_build_gen_cache $build opkg.conf
+}
+
+ipkdbg_opkg_conf_fetch_cache() {
+    local build=$1
+    local output=$2
+    local path="$(ipkdbg_opkg_conf_gen_cache $build)"
+    cp "$path" "$output" > /dev/null 2>&1
+}
+
+ipkdbg_opkg_conf_install() {
+    local build=$1
+    local output=$2
+    mkdir -p $(dirname $output)
+    if ! ipkdbg_opkg_conf_fetch_cache $build $output
+    then
+        local cache="$(ipkdbg_opkg_conf_gen_cache $build)"
+        mkdir -p $(dirname $cache)
+        url=
+        ipkdbg_opkg_fetch_path "$(ipkdbg_opkg_conf_gen_path $build)" $cache ||
+            (echo "Configuring opkg via $(ipkdbg_opkg_conf_gen_url $build)" &&
+                ipkdbg_opkg_fetch_url "$(ipkdbg_opkg_conf_gen_url $build)" $cache)
+        ipkdbg_opkg_conf_fetch_cache $build $output
+    fi
+}
+
+ipkdbg_opkg_db_gen_path() {
+    local build=$1
+    ipkdbg_build_gen_path $build bmc_ipk/opkg-database.tar.xz
+}
+
+ipkdbg_opkg_db_gen_url() {
+    local build=$1
+    ipkdbg_build_gen_url ${build} bmc_ipk/opkg-database.tar.xz
+}
+
+ipkdbg_opkg_db_gen_cache() {
+    local build=$1
+    ipkdbg_build_gen_cache $build opkg-database.tar.xz
+}
+
+ipkdbg_opkg_db_install() {
+    local build=$1
+    local root=$2
+    local state=${root}/var/lib/opkg
+    local cache="$(ipkdbg_opkg_db_gen_cache $build)"
+    mkdir -p $state
+    if ! [ -f $cache ]
+    then
+        mkdir -p $(dirname $cache)
+        ipkdbg_opkg_fetch_path "$(ipkdbg_opkg_db_gen_path $build)" $cache ||
+            ipkdbg_opkg_fetch_url "$(ipkdbg_opkg_db_gen_url $build)" $cache ||
+            rm -f $cache
+    fi
+    tar -xf $cache -C $state 2> /dev/null
+    mkdir -p ${root}/usr/local
+    ln -s ${root}/var ${root}/usr/local/var
+}
+
+ipkdbg_opkg() {
+    $IPKDBG_OPKG_BIN \
+        $([ -z "$IPKDBG_OPKG_CACHE" ] ||
+            echo --cache-dir $IPKDBG_OPKG_CACHE --host-cache-dir) \
+        -V1 -f $IPKDBG_CONF -o $IPKDBG_ROOT $@
+}
+
+ipkdbg_gdb_extract_bin() {
+    local core=$1
+    $IPKDBG_GDB --core $core -ex quit 2> /dev/null |
+        awk -F "[\`']" '/Core was generated by/ { print $2 }' |
+        awk -F " " '{ print $1 }' # Chop off the arguments, we only want the binary path
+}
+
+ipkdbg_opkg_find() {
+    ipkdbg_opkg find $@ | awk '{ print $1 }'
+}
+
+ipkdbg_opkg_find_extra() {
+    local pkg=$1
+
+    # Try appending -dbg and -src to the binary package name
+    extra_pkgs="$(ipkdbg_opkg_find ${pkg}-dbg) $(ipkdbg_opkg_find ${pkg}-src)"
+
+    # If that fails, we probably have a split binary package
+    if [ -z "$extra_pkgs" ]
+    then
+        # Strip the last component off as it's probably the split binary package name and
+        # try again
+        extra_pkgs="$(ipkdbg_opkg_find ${pkg%-*}-dbg) $(ipkdbg_opkg_find ${pkg%-*}-src)"
+    fi
+    echo $extra_pkgs
+}
+
+ipkdbg_opkg_conf_install $IPKDBG_BUILD $IPKDBG_CONF
+
+# Extract the binary path from the core
+if [ '-' = "$IPKDBG_FILE" -a '-' != "$IPKDBG_CORE" ]
+then
+    IPKDBG_FILE=$(ipkdbg_gdb_extract_bin $IPKDBG_CORE)
+fi
+
+# Update the package database before potentially looking up the debug packages
+ipkdbg_opkg update
+
+# Extract the package name for the binary
+if [ '-' != "$IPKDBG_CORE" ]
+then
+    if ipkdbg_opkg_db_install $IPKDBG_BUILD $IPKDBG_DB
+    then
+        # Look up the package for the binary
+        IPKDBG_CORE_PKG="$(IPKDBG_ROOT=$IPKDBG_DB ipkdbg_opkg search ${IPKDBG_DB}${IPKDBG_FILE} | awk '{ print $1 }')"
+        if [ -n "$IPKDBG_CORE_PKG" ]
+        then
+            # Look up the extra (debug, source) packages for the binary package
+            IPKDBG_PKGS="$IPKDBG_PKGS $IPKDBG_CORE_PKG"
+            IPKDBG_PKGS="$IPKDBG_PKGS $(ipkdbg_opkg_find_extra $IPKDBG_CORE_PKG)"
+        fi
+    fi
+
+    if [ -z "$IPKDBG_PKGS" ]
+    then
+        ipkdbg_error "Unable to determine package-set to install, please specify" \
+                 "appropriate packages on the command line"
+        exit 1
+    fi
+fi
+
+# Force installation of gcc-runtime-dbg to give us debug symbols for libstdc++
+IPKDBG_PKGS="gcc-runtime-dbg $IPKDBG_PKGS"
+
+if [ -n "$IPKDBG_OPKG_CACHE" ]
+then
+    mkdir -p "$IPKDBG_OPKG_CACHE"
+    ipkdbg_opkg install --download-only $IPKDBG_PKGS
+fi
+
+ipkdbg_opkg install $IPKDBG_PKGS | grep -vF 'Warning when extracting archive entry'
+
+cat <<EOF > ${IPKDBG_BINS}/opkg
+#!/bin/sh
+exec $IPKDBG_OPKG_BIN -f $IPKDBG_CONF -o $IPKDBG_ROOT \$@
+EOF
+chmod +x ${IPKDBG_BINS}/opkg
+
+PATH=${IPKDBG_BINS}:${PATH} $IPKDBG_GDB -q \
+    -iex "set solib-absolute-prefix $IPKDBG_ROOT" \
+    -iex "add-auto-load-safe-path $IPKDBG_ROOT" \
+    -iex "set directories $IPKDBG_ROOT" \
+    -iex "cd $IPKDBG_ROOT" \
+    $([ '-' = "$IPKDBG_CORE" ] || echo -ex bt) \
+    $([ 0 -eq $IPKDBG_OPT_QUIT ] || echo -ex quit) \
+    ${IPKDBG_ROOT}${IPKDBG_FILE} \
+    $([ '-' = "$IPKDBG_CORE" ] || echo $IPKDBG_CORE)
+
+exit 0
+
+__ARCHIVE_BEGIN__