subtree updates
poky: 348d9aba33..fc8e5d7c13:
Adithya Balakumar (1):
wic: implement reproducible Disk GUID
Adrian Freihofer (20):
cmake.bbclass: use --install
devtool: support plugins with plugins
devtool: refactor exec_fakeroot
devtool: refactor deploy to use exec_fakeroot_no_d
devtool: refactor deploy-target
recipetool: cleanup imports
oeqa: replace deprecated assertEquals
oeqa/selftest/recipetool: fix for python 3.12
oeqa/selftest/oelib/buildhistory: git default branch
scripts: python 3.12 regex
feature-microblaze-versions.inc: python 3.12 regex
meta/lib/oeqa: python 3.12 regex
meta/lib/patchtest: python 3.12 regex
meta/recipes: python 3.12 regex
bitbake: bitbake/lib/bs4/tests/test_tree.py: python 3.12 regex
devtool: new ide-sdk plugin
oe-selftest devtool: ide-sdk tests
devtool: ide-sdk make deploy-target quicker
vscode: drop .vscode folder
oe-init-build-env: generate .vscode from template
Aleksey Smirnov (2):
conf/machine: Add Power8 tune to PowerPC architecture
busybox: Explicitly specify tty device for serial consoles
Alex Kiernan (1):
wireless-regdb: Upgrade 2023.09.01 -> 2024.01.23
Alex Stewart (3):
opkg: upgrade to 0.6.3
opkg: add deprecation warning for internal solver
opkg-arch-config: update recipe HOMEPAGE
Alexander Kanavin (26):
sysroot user management postinsts: run with /bin/sh -e to report errors when they happen
classes/multilib: expand PACKAGE_WRITE_DEPS in addition to DEPENDS
classes/staging: capture output of sysroot postinsts into logs
classes/package_rpm: write file permissions and ownership explicitly into .spec
classes/package_rpm: use weak user/group dependencies
classes/package_rpm: set bogus locations for passwd/group files
oeqa/runtime/rpm: fail tests if test rpm file cannot be found
rpm: update 4.18.1 -> 4.19.1
classes/package_rpm: correctly escape percent characters
setftest/cdn tests: check for exceptions also in fetcher diagnostics
rpm: override curl executable search with just 'curl'
classes/package_rpm: additionally escape \ and " in filenames
classes/package_rpm: streamline the logic in one of the condition blocks
lzlib: add a recipe
file: enable additional internal compressor support
selftest/SStateCacheManagement: do not manipulate ERROR_QA
selftest/SStateCacheManagement: pre-populate the cache
shadow: add a packageconfig for logind support
meta/conf/templates/default/conf-notes.txt: remove
scripts/oe-setup-layers: write a list of layer paths into the checkout's top dir
meta/conf/templates/default/conf-summary.txt: add a template summary
meta/lib/bblayers/buildconf.py: add support for configuration summaries
scripts/oe-setup-builddir: add support for configuration summaries
oe-setup-build: add a tool for discovering config templates and setting up builds
meta-poky/conf/templates/default/conf-summary.txt: add a template summary
bitbake: Revert "bitbake: wget.py: always use the custom user agent"
Alexis Lothoré (3):
patchtest-send-results: remove unused variable
patchtest-send-results: properly parse test status
testimage: retrieve ptests directory when ptests fail
André Draszik (4):
sstate-cache-management: fix regex for 'sigdata' stamp files
bitbake: fetch/git2: support git's safe.bareRepository
bitbake: tests/fetch: support git's safe.bareRepository
bitbake: git-make-shallow: support git's safe.bareRepository
Anibal Limon (1):
ptest-runner: Bump to 2.4.3 (92c1b97)
Anuj Mittal (8):
enchant2: upgrade 2.6.5 -> 2.6.7
libproxy: upgrade 0.5.3 -> 0.5.4
sqlite3: upgrade 3.44.2 -> 3.45.1
orc: upgrade 0.4.36 -> 0.4.37
stress-ng: upgrade 0.17.04 -> 0.17.05
libcap-ng: fix build with swig 4.2.0
gstreamer1.0: upgrade 1.22.9 -> 1.22.10
swig: upgrade 4.1.1 -> 4.2.0
Bruce Ashfield (13):
lttng-modules: fix v6.8+ build
linux-yocto-dev: update to v6.8
linux-yocto/6.6: features/qat/qat.cfg: enable CONFIG_PCIEAER
linux-yocto/6.6: beaglebone: drop nonassignable kernel options
linux-yocto/6.6: update to v6.6.13
linux-yocto/6.6: update CVE exclusions
linux-yocto/6.6: can: drop obsolete CONFIG_PCH_CAN
linux-yocto/6.6: update to v6.6.15
linux-yocto/6.6: update CVE exclusions
yocto-bsp: update reference boards to v6.6.15
linux-yocto/6.6: update to v6.6.16
linux-yocto/6.6: update CVE exclusions
linux-yocto/6.6: qemuriscv: enable goldfish RTC
Chen Qi (5):
multilib_global.bbclass: fix parsing error with no kernel module split
gnupg: disable tests to avoid running target binaries at build time
bitbake: fetch2/git.py: fix a corner case in try_premirror
bitbake: tests/fetch.py: add test case for using premirror in restricted network
bitbake: fetch2/git.py: add comment in try_premirrors
Chi Xu (1):
xz: Add ptest support
Claus Stovgaard (2):
kernel-devsrc: fix RDEPENDS for make
kernel-devsrc: RDEPENDS on gawk
Clément Péron (1):
libpcap: extend with nativesdk
Colin McAllister (1):
initscripts: Add custom mount args for /var/lib
David Reyna (1):
bitbake: taskexp_ncurses: ncurses version of taskexp.py
Denys Dmytriyenko (3):
lttng-modules: upgrade 2.13.10 -> 2.13.11
zlib: upgrade 1.3 -> 1.3.1
xz: upgrade 5.4.5 -> 5.4.6
Enguerrand de Ribaucourt (3):
devtool: ide_sdk: Use bitbake's python3 for generated scripts
devtool: ide: vscode: Configure read-only files
meson: use absolute cross-compiler paths
Enrico Jörns (1):
rootfs-postcommands: remove make_zimage_symlink_relative()
Etienne Cordonnier (1):
dropbear: remove unnecessary line
Fabien Mahot (1):
ldconfig-native: Fix to point correctly on the DT_NEEDED entries in an ELF file
Fabio Estevam (3):
piglit: Update to latest revision
mesa: Upgrade 23.3.3 -> 23.3.4
mesa: Upgrade 23.3.4 -> 23.3.5
Jamin Lin (3):
uboot-sign: set load address and entrypoint
uboot-sign: Fix to install nonexistent dtb file
u-boot-sign:uboot-config: support to verify signed FIT image
Jermain Horsman (2):
bitbake-layers: Add ability to update the reference of repositories
bitbake-layers: Add test case layers setup for custom references
Joe Slater (1):
eudev: allow for predictable network interface names
Johannes Schneider (2):
initramfs-framework: overlayroot: fix kernel commandline clash
initramfs-framework: overlayroot: align bootparams with module name
Jon Mason (2):
tunes/sve: Add support for sve2 instructions
arm/armv*: add all the Arm tunes in GCC 13.2.0
Jonathan GUILLOT (3):
lib/oe/package: replace in place PN-locale-* packages in PACKAGES
lib/oe/package: add LOCALE_PATHS to add define all locations for locales
cups: use LOCALE_PATHS to split localized HTML templates
Jose Quaresma (3):
go: update 1.20.12 -> 1.20.13
systemd: pack pre-defined pcrlock files installed with tpm2
qemu: disbale AF_XDP network backend support
Joshua Watt (8):
bitbake: hashserv: Add Unihash Garbage Collection
bitbake: hashserv: sqlalchemy: Use _execute() helper
bitbake: hashserv: Add unihash-exists API
bitbake: asyncrpc: Add Client Pool object
bitbake: hashserv: Add Client Pool
bitbake: siggen: Add parallel query API
bitbake: siggen: Add parallel unihash exist API
sstatesig: Implement new siggen API
Kai Kang (2):
rpm: fix dependency for package config imaevm
ghostscript: correct LICENSE with AGPLv3
Khem Raj (27):
elfutils: Fix build with gcc trunk
python3: Initialize struct termios before calling tcgetattr()
qemu: Replace the basename patch with backport
xwayland: Upgrade 23.2.3 -> 23.2.4
armv8/armv9: Avoid using -march when -mcpu is chosen
kexec-tools: Fix build with gas 2.42
systemtap: Backport GCC-14 related calloc fixes
sdk/assimp.py: Fix build on 32bit arches with 64bit time_t
binutils: Upgrade to binutils 2.42
qemu-native: Use inherit_defer for including native class
syslinux: Disable error on implicit-function-declaration
glibc: Upgrade to 2.39
strace: Upgrade to 6.7
rust/cargo: Build fixes to rust for rv32 target
buildcpio.py: Switch to using cpio-2.15
ptest.bbclass: Handle the case when Makefile does not exist in do_install_ptest_base
kernel-devsrc: Add needed fixes for 6.1+ kernel build on target on RISCV
python3: Fix ptests with expat 2.6+
expat: Upgrade to 2.6.0
gcc-runtime: Move gdb pretty printer file to auto-load location
core-image-ptest: Increase disk size to 1.5G for strace ptest image
tcmode-default: Do not define LLVMVERSION
glibc: Update to latest on 2.39
glibc: Update to bring mips32/clone3 fix
piglit: Fix build with musl
llvm: Upgrade to LLVM-18 RC2
binutils: Update to tip of 2.42 release branch
Konrad Weihmann (1):
python3-yamllint: add missing dependency
Lee Chee Yang (1):
migration-guide: add release notes for 4.0.16
Maanya Goenka (2):
toolchain-shar-relocate: allow 'find' access to libraries in symlinked directories
bash: nativesdk-bash does not provide /bin/bash so don't claim to
Marek Vasut (1):
Revert "lzop: remove recipe from oe-core"
Mark Hatle (5):
qemu: Allow native and nativesdk versions on Linux older then 4.17
tune-cortexa78.inc: Add cortexa78 tune, based on cortexa77
feature-arm-vfp.inc: Allow hard-float on newer simd targets
tune-cortexr5: Add hard-float variant
tune-cortexr52: Add hard-float variant
Markus Volk (6):
gtk4: update 4.12.4 -> 4.12.5
mesa: update 23.3.5 -> 24.0.0
mesa: update 24.0.0 -> 24.0.1
libadwaita: update 1.4.2 -> 1.4.3
wayland-protocols: update 1.32 -> 1.33
ell: update 0.61 -> 0.62
Martin Jansa (5):
qemu: fix target build with ccache enabled
package_manager: ipk: add OPKG_MAKE_INDEX_EXTRA_PARAMS variable
package_rpm: add RPMBUILD_EXTRA_PARAMS variable
bitbake: bitbake-diffsigs: fix walking the task dependencies and show better error
bitbake: tests: fetch.py: use real subversion repository
Michael Opdenacker (9):
dev-manual: start: remove idle line
docs: remove support for mickledore (4.2) release
release-notes-4.3: fix spacing
alsa-lib: upgrade 1.2.10 -> 1.2.11
alsa-tools: upgrade 1.2.5 -> 1.2.11
alsa-ucm-conf: upgrade 1.2.10 -> 1.2.11
alsa-utils: upgrade 1.2.10 -> 1.2.11
oeqa/runtime/cases: fix typo in information message
bitbake: doc: README: simpler link to contributor guide
Michal Sieron (1):
sanity.bbclass: raise_sanity_error if /tmp is noexec
Nick Owens (1):
systemd: recommend libelf, libdw for elfutils flag
Ola x Nilsson (1):
python3-numpy: Use Large File Support version of fallocate
Paul Gortmaker (1):
bitbake: hashserv: improve the loglevel error message to be more helpful
Pavel Zhukov (3):
systemd.bbclass: Check for existence of the symlink too
bitbake: fetch2/git.py: Fetch mirror into HEAD
bitbake: tests/fetch.py: add multiple fetches test
Peter Kjellerstedt (12):
devtool: modify: Correct appending of type=git-dependency to URIs
devtool: standard: Add some missing whitespace
devtool: _extract_source: Correct the removal of an old backup directory
bitbake: tests/fetch: Make test_git_latest_versionstring support a max version
bitbake: fetch2/git: A bit of clean-up of latest_versionstring()
bitbake: fetch2/git: Make latest_versionstring extract tags with slashes correctly
lib/oe/patch: Make extractPatches() not extract ignored commits
lib/oe/patch: Add GitApplyTree.commitIgnored()
devtool: Make use of oe.patch.GitApplyTree.commitIgnored()
patch.bbclass: Make use of oe.patch.GitApplyTree.commitIgnored()
lib/oe/patch: Use git notes to store the filenames for the patches
insane.bbclass: Allow the warning about virtual/ to be disabled
Peter Marko (2):
openssl: Upgrade 3.2.0 -> 3.2.1
util-linux: add alternative link for scriptreplay
Petr Vorel (1):
ltp: Update to 20240129
Philip Lorenz (1):
ipk: Remove temporary package lists during SDK creation
Priyal Doshi (1):
tzdata : Upgrade to 2024a
Quentin Schulz (1):
u-boot: add missing dependency on pyelftools-native
Randolph Sapp (1):
mirrors.bbclass: add infraroot as an https mirror
Randy MacLeod (4):
valgrind: make ptest depend on all components
valgrind: update from 3.21.0 to 3.22.0
valgrind: skip 14 ptests in 3.22
valgrind: Skip 22 arm64 ptests
Richard Purdie (34):
oeqa/qemurunner: Handle rare shutdown race
pseudo: Update to pull in gcc14 fix and missing statvfs64 intercept
numactl: upgrade 2.0.16 -> 2.0.17
conf: Move selftest config to dedicated inc file
oeqa/selftest/bbtests: Tweak to use no-gplv3 inc file
python3-markupsafe: upgrade 2.1.3 -> 2.1.5
python3-markupsafe: Switch to python_setuptools_build_meta
qemu: Upgrade 8.2.0 -> 8.2.1
ltp: Enable extra test groups
ltp: Try re-enabling problematic test
meta-yocto-bsp: Remove accidentally added files
oeqa/runtime: Move files from oe-core to bsp layer
mirrors: Allow shallow glibc to work correctly
ptest-packagelists: Mark python3 as problematic on riscv64
kernel-devsrc: Clean up whitespace
selftest/recipetool: Factor tomllib test to a function
selftest/recipetool: Improve test failure output
layer.conf: Update for the scarthgap release series
layer.conf: Update for the scarthgap release series
bitbake: process: Add profile logging for main loop
bitbake: process/server: Fix typo
kernel-arch: Simplify strip support
insane: Clarify runtime/ warning
bitbake: runqueue: Improve performance for executing tasks
bitbake: runqueue: Optimise taskname lookups in next_buildable_task
bitbake: runqueue: Improve setcene performance when encoutering many 'hard' dependencies
openssh: Add a work around for ICE on mips/mips64
kernel-devsrc: Improve vdso-offsets handling for qemuriscv64
u-boot: Pass in prefix mapping variables to the compiler
testsdk: Avoid PATH contamination
oeqa/selftest/rust: Exclude failing riscv tests
bitbake: bitbake: Bump version to 2.7.3 for hashserv changes
sanity.conf: Require bitbake 2.7.3
python: Drop ${PYTHON_PN}
Robert Joslyn (2):
curl: Update to 8.6.0
gtk: Set CVE_PRODUCT
Robert Yang (1):
gnu-config: Update to latest version
Ross Burton (13):
grub2: ignore CVE-2023-4001, this is Red Hat-specific
openssl: backport fix for CVE-2023-6129
lib/oeqa: rename assertRaisesRegexp to assertRaisesRegex
oeqa/selftest/recipetool: downgrade meson version to not use pyproject.toml
recipetool: don't dump stack traces if a toml parser can't be found
xz: remove redundant PTEST_ENABLED conditional
libpam: remove redundant PTEST_ENABLED conditional
glib-2.0: backport memory monitor test fixes
python3: move dataclasses to python3-core
python3-unittest-automake-output: upgrade to 0.2
meson: remove TMPDIR workaround
meson: set the sysroot in the cross files
libffi: upgrade to 3.4.5
Simone Weiß (12):
gnutls: Upgrade 3.8.2 -> 3.8.3
maintainers.inc: Add self for libseccomp and gnutls
bsp-guide: correct formfactor recipe name
dev-manual: gen-tapdevs need iptables installed
gnutls: print log if ptest fails
patchtest: log errors and failures at end
grub2: ignore CVE-2024-1048, Redhat only issue
libgit2: update 1.7.1 -> 1.7.2
libuv: Upgrade 1.47.0 -> 1.48.0
qemu: Set CVE_STATUS for wrong CVEs
patchtest: Add selftest for test cve_check_ignore
patchtest: add stronger indication for failed tests
Siong W.LIM (1):
useradd.bbclass: Fix missing space when appending vardeps.
Thomas Perrot (2):
opensbi: append LDFLAGS to TARGET_CC_ARCH
bitbake: wget.py: always use the custom user agent
Tim Orling (13):
libxml-parser-perl: upgrade 2.46 -> 2.47
python3-pyyaml: add PACKAGECONFIG for libyaml
python3-pyyaml: enable ptest
python3-cryptography: upgrade 41.0.7 to 42.0.2
openssh: upgrade 9.5p1 -> 9.6p1
python3-poetry-core: upgrade 1.8.1 -> 1.9.0
python3-attrs: skip test failing with pytest-8
vim: upgrade from 9.0.2130 -> 9.1.0114
python3-pyproject-metadata: move from meta-python
python3-pyproject-metadata: HOMEPAGE; DESCRIPTION
python3-meson-python: move from meta-python
python_mesonpy.bbclass: move from meta-python
recipetool; add support for python_mesonpy class
Tobias Hagelborn (2):
sstate.bbclass: Only sign packages at the time of their creation
bitbake: bitbake: hashserv: Postgres adaptations for ignoring duplicate inserts
Toni Lammi (1):
bitbake: support temporary AWS credentials
Trevor Gamblin (7):
patchtest.README: update mailing list
cmake: upgrade 3.27.7 -> 3.28.3
python3-numpy: upgrade 1.26.3 -> 1.26.4
patchtest-send-results: Add 'References' header
patchtest-send-results: use Message-ID directly
patchtest: Fix grammar in log output
patchtest-send-results: add --debug option
Valek Andrej (1):
glibc: Refresh CVE status w.r.t 2.39 release
Vikas Katariya (1):
bmap-tools: Add missing runtime dependency
Wang Mingyu (36):
at-spi2-core: upgrade 2.50.0 -> 2.50.1
cpio: upgrade 2.14 -> 2.15
ethtool: upgrade 6.6 -> 6.7
iso-codes: upgrade 4.15.0 -> 4.16.0
libinput: upgrade 1.24.0 -> 1.25.0
libtest-warnings-perl: upgrade 0.032 -> 0.033
libwpe: upgrade 1.14.1 -> 1.14.2
lzip: upgrade 1.23 -> 1.24
createrepo-c: upgrade 1.0.2 -> 1.0.3
diffstat: upgrade 1.65 -> 1.66
dos2unix: upgrade 7.5.1 -> 7.5.2
ed: upgrade 1.19 -> 1.20
gnupg: upgrade 2.4.3 -> 2.4.4
gstreamer: upgrade 1.22.8 -> 1.22.9
libidn2: upgrade 2.3.4 -> 2.3.7
libpng: upgrade 1.6.40 -> 1.6.41
libsolv: upgrade 0.7.27 -> 0.7.28
liburi-perl: upgrade 5.21 -> 5.25
nghttp2: upgrade 1.58.0 -> 1.59.0
repo: upgrade 2.40 -> 2.41
orc: upgrade 0.4.34 -> 0.4.36
pkgconf: upgrade 2.0.3 -> 2.1.0
python3-sphinxcontrib-applehelp: upgrade 1.0.7 -> 1.0.8
python3-sphinxcontrib-devhelp: upgrade 1.0.5 -> 1.0.6
python3-sphinxcontrib-htmlhelp: upgrade 2.0.4 -> 2.0.5
python3-sphinxcontrib-qthelp: upgrade 1.0.6 -> 1.0.7
python3-sphinxcontrib-serializinghtml: upgrade 1.1.9 -> 1.1.10
python3-beartype: upgrade 0.16.4 -> 0.17.0
python3-mako: upgrade 1.3.0 -> 1.3.2
python3-hatchling: upgrade 1.21.0 -> 1.21.1
python3-hypothesis: upgrade 6.92.9 -> 6.97.3
python3-pluggy: upgrade 1.3.0 -> 1.4.0
python3-psutil: upgrade 5.9.7 -> 5.9.8
python3-pyopenssl: upgrade 23.3.0 -> 24.0.0
python3-pytz: upgrade 2023.3 -> 2023.4
python3-pytest: upgrade 7.4.4 -> 8.0.0
Xiangyu Chen (1):
bash: rebase the patch to fix ptest failure
Yi Zhao (2):
rpm: add missing dependencies for packageconfig
libsdl2: upgrade 2.28.5 -> 2.30.0
Yoann Congal (2):
kexec-tools: Replace a submitted patch by the backported one
waf.bbclass: Print waf output on unparsable version
Yogita Urade (1):
tiff: fix CVE-2023-52355 and CVE-2023-52356
baruch@tkos.co.il (3):
contributor-guide: fix lore URL
overlayfs: add missing closing parenthesis in selftest
overlayfs-etc: add option to skip creation of mount dirs
meta-arm: 6bb1fc8d8c..025f76a14f:
Ali Can Ozaslan (1):
arm-bsp/u-boot:corstone1000: Fix deployment of capsule files
Drew Reed (4):
bsp: Move Corstone-1000 U-Boot configuration entries
bsp: Move machine settings
bsp,ci: Switch to poky distro
bsp: Rename corstone1000-image
Harsimran Singh Tungal (2):
n1sdp:arm arm-bsp: fix tftf tests for n1sdp
arm-bsp/optee: upgrade optee to 4.1.0 for N1SDP
Jon Mason (3):
arm/opencsd: update to v1.5.1
arm/optee: update to 4.1
arm-bsp/optee: remove unused v3.22.0 recipes
Khem Raj (1):
layer.conf: Update for the scarthgap release series
Ross Burton (5):
CI: support extra kas files from environment
CI/cve.yml: add a CVE-checking Kas fragment
CI: add explanatory comments to variables
CI: allow the runner to set a NVD API key
CI: use https: to fetch meta-virtualization
Vincent Stehlé (1):
arm-bsp/documentation: corstone1000: fix typo
meta-security: b2e1511338..30e755c592:
Armin Kuster (3):
python3-pyinotify: do not rely on smtpd module
python3-fail2ban: remove unused distutils dependency
scap-security-guide: update to 0.1.71
BELOUARGA Mohamed (2):
checksec: Add more runtime dependencies to checksec tool
lynis: Add missing runtime dependencies
Leon Anavi (2):
linux-yocto%.bbappend: Add audit.cfg
integrity-image-minimal: Fix IMAGE_INSTALL
Mikko Rapeli (1):
parsec-tool: fix serialNumber check
Yi Zhao (1):
openscap: fix build with python 3.12
Yushi Sun (1):
meta-security: libhoth: SRCREV bump e520f8f...e482716
meta-raspberrypi: 9c901bf170..dbf1113a82:
Kevin Hao (1):
rpidistro-ffmpeg: Fix old override syntax
Khem Raj (3):
linux-raspberrypi_6.1.bb: Upgrade to 6.1.74
linux-raspberrypi: Upgrade to 6.1.77
layer.conf: Update for the scarthgap release series
Martin Jansa (1):
libcamera-apps: fix build with libcamera-0.2.0
Matthew Draws (1):
rpi-eeprom_git: v.2024.01.05-2712 Update recipe to latest rpi-eeprom repo This follows the current latest release of rpi-eeprom: https://github.com/raspberrypi/rpi-eeprom
Pascal Huerst (1):
rpi-base: Add missing hifiberry overlay
meta-openembedded: 9953ca1ac0..528f273006:
Alex Kiernan (3):
mdns: Fix SIGSEGV during DumpStateLog()
mdns: Upgrade 2200.60.25.0.4 -> 2200.80.16
c-ares: Upgrade 1.24.0 -> 1.26.0
Angelo Ribeiro (1):
flatcc: Add tool recipe
Angelo.Ribeiro (1):
e2tools: Add tool recipe
Archana Polampalli (1):
nodejs: update to latest v20 version 20.11.0
Beniamin Sandu (3):
mbedtls: upgrade 3.5.1 -> 3.5.2
mbedtls: upgrade 2.28.4 -> 2.28.7
opencv: upgrade 4.8.0 -> 4.9.0
Changqing Li (1):
cpuid: fix do_install
Chirag Shilwant (1):
kernel-selftest: Add few more testcases
Christophe Vu-Brugier (4):
dropwatch: add new recipe
switchtec-user: upgrade 4.1 -> 4.2
libnvme: upgrade 1.7.1 -> 1.8
nvme-cli: upgrade 2.7.1 -> 2.8
Clément Péron (2):
proj: extend class to native and nativesdk
proj: upgrade 9.3.0 -> 9.3.1
Denys Dmytriyenko (1):
libcamera: update 0.1.0 -> 0.2.0
Derek Straka (36):
python3-bandit: update to version 1.7.7
python3-web3: update to version 6.15.0
python3-argcomplete: update to version 3.2.2
python3-cytoolz: update to version 0.12.3
python3-pdm: update to version 2.12.2
python3-google-api-python-client: update to version 2.115.0
python3-coverage: update to version 7.4.1
python3-gmqtt: update to version 0.6.14
python3-colorlog: update to version 6.8.2
python3-argh: update to version 0.31.2
python3-luma-core: update to version 2.4.2
python-pdm: update to version 2.12.3
python3-parse: update to version 1.20.1
python3-grpcio: update to version 1.60.1
python3-dill: update to version 0.3.8
python3-types-setuptools: update to version 69.0.0.20240125
python3-pymisp: update to version 2.4.184
python3-cbor2: update to version 5.6.1
python3-sentry-sdk: update to version 1.40.0
python3-pytest-asyncio: update to version 0.23.4
python3-google-api-core: update to version 2.16.1
python3-google-api-python-client: update to version 2.116.0
python3-google-auth: update to version 2.27.0
python3-jsonrpcclient: update to version 4.0.3
python3-dnspython: update to version 2.5.0
python3-eventlet: update to version 0.35.1
python3-platformdirs: update to version 4.2.0
python3-ipython: update to version 8.21.0
python3-grpcio-tools: update to version 1.60.1
python3-cachecontrol: update to version 0.14.0
python3-binwalk: update the regex version for upstream checks
python3-pymodbus: update to version 3.6.3
python3-pyyaml-include: add initial recipe for version 1.3.2
python3-appdirs: add ptest into PTESTS_FAST_META_PYTHON items
python3-yarl: add ptest into PTESTS_FAST_META_PYTHON items
python3-ujson: add ptest into PTESTS_FAST_META_PYTHON items
Emil Kronborg (1):
php-fpm: fix systemd
Etienne Cordonnier (2):
uutils-coreutils: upgrade 0.0.23 -> 0.0.24
uutils_coreutils: merge .inc and .bb
Fathi Boudra (4):
whitenoise: add a new recipe
python3-django: upgrade to Django 4.2.10 LTS release
libtinyxml2: fix the homepage URL
libtinyxml2: allow to build both shared and static libraries
Geoff Parker (2):
python3-aiodns python3-pycares: Add native & nativesdk support
python3-aiohappyeyeballs: Add native & nativesdk support
Jean-Marc BOUCHE (1):
rtkit: missing files/directories in package
Jose Quaresma (1):
ostree: Upgrade 2023.8 -> 2024.1
Jörg Sommer (1):
bonnie++: New recipe for version 2.0
Khem Raj (18):
uftrace: Upgrade to 0.15.2
i2cdev: Set PV correctly
minicoredumper: Fix build with clang
python3-pytest-mock: Fix ptest failures with python 3.12
ndctl: Update to v78
vk-gl-cts: Disable Werror on amber external module
vulkan-cts: Upgrade to 1.3.7.3
uftrace: Adjust the summary to reflect rust and python support
libcamera: Fix build with clang-18
breakpad: Upgrade to 2023.06.01 release
bpftool: Add missing dep on elfutils-native
flatcc: Fix build warnings found with clang-18
Revert "lzop: add (from oe-core)"
can-isotp: Update to latest and skip it
openflow: Switch SRC_URI to github mirror
ot-br-posix: upgrade to latest trunk
libcereal: Disable c++11-narrowing-const-reference warning as error
ot-br-posix: Limit vla-cxx-extension option to clang >= 18
Li Wang (1):
radvd: add '--shell /sbin/nologin' to /etc/passwd
Mark Hatle (1):
opencv: Fix python3 package generation
Markus Volk (9):
luajit: allow to build on supported platforms
pipewire: fix build with libcamera-0.2
system-config-printer: fix runtime for system-config-printer
iwd: update 2.8 -> 2.13
pipewire: update 1.0.1 -> 1.0.3
flatpak: remove unneeded RDEPENDS
libosinfo: use hwdata for ids files
libnfs: update 5.0.2 -> 5.0.3
hwdata: update 0.378 -> 0.379
Martin Jansa (18):
libtalloc, libtevent, libtdb, libldb: set PYTHONARCHDIR for waf to respect python libdir
jack: fix build with python3 on host
redis: restore Upstream-Status
libvpx: restore Upstream-Status
python-jsonref: add missing Upstream-Status
flatcc: respect baselib
flatcc: drop 'r' from gitr and ${SRCPV}
recipes: drop ${SRCPV} usage
recipes: drop remaining +gitr cases
gitpkgv.bbclass: adjust the example in comment a bit
ne10: append +git instead of gitr+
evemu-tools: use better PV
nana: upgrade to latest commit from github
xfstests: upgrade to latest 2024.01.14
xfstests: add gawk to RDEPENDS
xfstests: use master branch instead of 'for-next'
xfstests: drop the upstream rejected install-sh hack
xfstests: fix make install race condition
Max Krummenacher (2):
libusbgx: fix usbgx.service stop / restart
libusbgx: uprev to the latest commit
Maxime Roussin-Belanger (1):
xdg-desktop-portal: add missing glib-2.0-native dependency
Maxime Roussin-Bélanger (1):
polkit: fix rules.d permissions
Ming Liu (1):
plymouth: uprev to 24.004.60
Niko Mauno (4):
python3-pybind11: Amend HOMEPAGE
python3-pybind11: Prune redundant inherit
python3-pybind11: Fix LICENSE
python3-pybind11: Cosmetic fixes
Pavel Zhukov (1):
python3-tzlocal: Add zoneinfo dependency
Peter Kjellerstedt (1):
xfstests: Only specify the main SRCREV once
Peter Marko (2):
syslog-ng: ignore CVE-2022-38725
libqmi: correct PV
Pratik Manvar (1):
python3-pybind11: Remove the Boost dependency
Richard Leitner (7):
python3-janus: add recipe for v1.0.0
python3-moteus: add recipe for v0.3.67
python3-socksio: add recipe for v1.0.0
python3-anyio: add recipe for v4.2.0
python3-sniffio: add recipe for v1.3.0
python3-httpcore: add recipe for v1.0.2
python3-httpx: add recipe for v0.26.0
Sascha Hauer (1):
signing.bbclass: make it work with eliptic curve keys
Simone Weiß (1):
scapy: Add difftools and logutils in RDEPENDS
Thomas Perrot (3):
dvb-apps: no longer skip ldflags QA
etcd-cpp-apiv3: no longer skip ldflags QA
kernel-selftest: no longer skip ldflags QA
Tim Orling (60):
python3-uritemplate: switch to pytest --automake
python3-unidiff: switch to pytest --automake
python3-ujson: switch to pytest --automake
python3-pytest-lazy-fixture: switch to pytest --automake
python3-fastjsonschema: switch to pytest --automake
python3-tomlkit: switch to pytest --automake
python3-inotify: switch to pytest --automake
python3-requests-file: switch to pytest --automake
python3-covdefaults: switch to pytest --automake
python3-dominate: switch to pytest --automake
python3-scrypt: switch to pytest --automake
python3-u-msgpack-python: switch to pytest --automake
python3-iso3166: switch to pytest --automake
python3-trustme: switch to pytest --automake
python3-asgiref: switch to pytest --automake
python3-html2text: switch to pytest --automake
python3-pyasn1-modules: switch to pytest --automake
python3-intervals: switch to pytest --automake
python3-py-cpuinfo: switch to pytest --automake
python3-backports-functools-lru-cache: drop folder
python3-whoosh: switch to pytest --automake
python3-xlrd: switch to pytest --automake
python3-dnspython: switch to pytest --automake
python3-prettytable: switch to pytest --automake
python3-ptyprocess: switch to pytest --automake
python3-gunicorn: switch to pytest --automake
python3-pytest-mock: switch to pytest --automake
python3-pyroute2: switch to pytest --automake
python3-smpplib: switch to pytest --automake
python3-pyzmq: switch to pytest --automake
python3-multidict: switch to pytest --automake
python3-geojson: switch to pytest --automake
python3-serpent: switch to pytest --automake
python3-soupsieve: switch to pytest --automake
python3-requests-toolbelt: switch to pytest --automake
python3-yarl: switch to pytest --automake
python3-cbor2: switch to pytest --automake
python3-ansicolors: switch to pytest --automake
python3-ipy: switch to pytest --automake
python3-sqlparse: switch to pytest --automake
python3-precise-runner: switch to pytest --automake
python3-parse-type: switch to pytest --automake
python3-inflection: switch to pytest --automake
python3-blinker: switch to pytest --automake
python3-service-identity: switch to pytest --automake
python3-cachetools: switch to pytest --automake
python3-simpleeval: switch to pytest --automake
python3-appdirs: switch to pytest --automake
python3-pillow: switch to pytest --automake
python3-semver: switch to pytest --automake
python3-platformdirs: switch to pytest --automake
python3-polyline: switch to pytest --automake
python3-betamax: switch to pytest --automake
python3-pytoml: switch to pytest --automake
python3-pyserial: switch to pytest --automake
python3-typeguard: switch to pytest --automake
python3-execnet: switch to pytest --automake
python3-pyyaml-include: switch to pytest --automake
python3-xxhash: switch to pytest --automake
python3-pylint: switch to pytest --automake
Tom Geelen (1):
python3-pychromecast: add missing RDEPENDS, and add initial recipe for dependency.
Wang Mingyu (90):
btop: upgrade 1.2.13 -> 1.3.0
ccid: upgrade 1.5.4 -> 1.5.5
ctags: upgrade 6.1.20231231.0 -> 6.1.20240114.0
gcr3: upgrade 3.41.1 -> 3.41.2
htop: upgrade 3.2.2 -> 3.3.0
hwdata: upgrade 0.377 -> 0.378
libdecor: upgrade 0.2.1 -> 0.2.2
libvpx: upgrade 1.13.1 -> 1.14.0
lldpd: upgrade 1.0.17 -> 1.0.18
gjs: upgrade 1.78.2 -> 1.78.3
wireshark: upgrade 4.2.0 -> 4.2.2
capnproto: upgrade 1.0.1.1 -> 1.0.2
dnfdragora: upgrade 2.1.5 -> 2.1.6
libyang: upgrade 2.1.128 -> 2.1.148
lshw: upgrade 02.19.2 -> 02.20
md4c: upgrade 0.4.8 -> 0.5.0
python3-apscheduler: add new recipe
redis: upgrade 7.2.3 -> 7.2.4
sanlock: upgrade 3.8.5 -> 3.9.0
python3-eth-keys: upgrade 0.4.0 -> 0.5.0
python3-xmlschema: upgrade 2.5.1 -> 3.0.1
plocate: upgrade 1.1.20 -> 1.1.22
python3-absl: upgrade 2.0.0 -> 2.1.0
python3-asyncinotify: upgrade 4.0.5 -> 4.0.6
python3-beautifulsoup4: upgrade 4.12.2 -> 4.12.3
python3-cantools: upgrade 39.4.2 -> 39.4.3
python3-cbor2: upgrade 5.5.1 -> 5.6.0
python3-dbus-fast: upgrade 2.21.0 -> 2.21.1
python3-django: upgrade 5.0 -> 5.0.1
python3-eth-abi: upgrade 4.2.1 -> 5.0.0
python3-eth-typing: upgrade 3.5.2 -> 4.0.0
python3-eth-utils: upgrade 2.3.1 -> 3.0.0
python3-eventlet: upgrade 0.34.2 -> 0.34.3
python3-flask: upgrade 3.0.0 -> 3.0.1
python3-git-pw: upgrade 2.5.0 -> 2.6.0
python3-google-api-python-client: upgrade 2.113.0 -> 2.114.0
python3-haversine: upgrade 2.8.0 -> 2.8.1
python3-ipython: upgrade 8.19.0 -> 8.20.0
python3-pdm: upgrade 2.11.2 -> 2.12.1
python3-pyatspi: upgrade 2.46.0 -> 2.46.1
python3-sentry-sdk: upgrade 1.39.1 -> 1.39.2
python3-robotframework: upgrade 6.1.1 -> 7.0
python3-pychromecast: upgrade 13.0.8 -> 13.1.0
python3-tox: upgrade 4.11.4 -> 4.12.1
python3-types-psutil: upgrade 5.9.5.17 -> 5.9.5.20240106
qpdf: upgrade 11.7.0 -> 11.8.0
smemstat: upgrade 0.02.12 -> 0.02.13
tesseract: upgrade 5.3.3 -> 5.3.4
libsmi: Fix buildpaths warning.
minicoredumper: upgrade 2.0.6 -> 2.0.7
cmocka: Fix install conflict when enable multilib.
czmq: Fix install conflict when enable multilib.
czmq: Fix buildpaths warning.
bdwgc: upgrade 8.2.4 -> 8.2.6
cmark: upgrade 0.30.3 -> 0.31.0
gensio: upgrade 2.8.2 -> 2.8.3
geos: upgrade 3.12.0 -> 3.12.1
imlib2: upgrade 1.12.1 -> 1.12.2
libcbor: upgrade 0.10.2 -> 0.11.0
libinih: upgrade 57 -> 58
libio-socket-ssl-perl: upgrade 2.084 -> 2.085
libjcat: upgrade 0.2.0 -> 0.2.1
libqmi: upgrade 1.35.1 -> 1.35.2
md4c: upgrade 0.5.0 -> 0.5.2
nanomsg: upgrade 1.2 -> 1.2.1
neatvnc: upgrade 0.7.1 -> 0.7.2
network-manager-applet: upgrade 1.34.0 -> 1.36.0
libgsf: upgrade 1.14.51 -> 1.14.52
ndisc6: upgrade 1.0.7 -> 1.0.8
squid: upgrade 6.6 -> 6.7
iotop: upgrade 1.25 -> 1.26
libblockdev: upgrade 3.0.4 -> 3.1.0
neon: upgrade 0.32.5 -> 0.33.0
pkcs11-provider: upgrade 0.2 -> 0.3
sanlock: upgrade 3.9.0 -> 3.9.1
satyr: upgrade 0.42 -> 0.43
python3-astroid: upgrade 3.0.2 -> 3.0.3
python3-elementpath: upgrade 4.1.5 -> 4.2.0
python3-flask: upgrade 3.0.1 -> 3.0.2
python3-google-api-core: upgrade 2.16.1 -> 2.16.2
python3-gspread: upgrade 5.12.4 -> 6.0.0
python3-path: upgrade 16.9.0 -> 16.10.0
python3-gcovr: upgrade 6.0 -> 7.0
python3-types-psutil: upgrade 5.9.5.20240106 -> 5.9.5.20240205
python3-waitress: upgrade 2.1.2 -> 3.0.0
rdma-core: upgrade 48.0 -> 50.0
ser2net: upgrade 4.6.0 -> 4.6.1
sip: upgrade 6.8.1 -> 6.8.2
span-lite: upgrade 0.10.3 -> 0.11.0
tcpslice: upgrade 1.6 -> 1.7
William A. Kennington III (3):
nanopb: Update 0.4.7 -> 0.4.8
nanopb: Split into 2 packages
nanopb-runtime: Enable shared library
Yoann Congal (6):
ibus: backport a reproducibility fix
radvd: Fix build in reproducible test
mariadb: Move useradd handling in target side of the recipe
kexec-tools-klibc: Fix building on x86_64 with binutils 2.41
freeradius: Add missing 'radiusd' static group id
ntp: Add missing 'ntp' static group id
alperak (18):
python3-flask-marshmallow: upgrade 0.15.0 -> 1.1.0
python3-netaddr: upgrade 0.10.0 -> 0.10.1
python3-toolz: upgrade 0.12.0 -> 0.12.1
python3-aiohappyeyeballs: add recipe
python3-aiohttp: upgrade 3.9.1 -> 3.9.2
python3-eth-rlp: upgrade 1.0.0 -> 1.0.1
python3-aiohttp: upgrade 3.9.2 -> 3.9.3
python3-google-auth-oauthlib: add recipe
python3-scikit-build: upgrade 0.16.7 -> 0.17.6
python3-eth-account: upgrade 0.10.0 -> 0.11.0
python3-pyunormalize: add recipe
python3-web3: upgrade 6.15.0 -> 6.15.1
python3-gspread: upgrade 6.0.0 -> 6.0.1
python3-strenum: add recipe
python3-flask-marshmallow: upgrade 1.1.0 -> 1.2.0
python3-werkzeug: upgrade 2.3.6 -> 3.0.1
python3-imageio: upgrade 2.33.1 -> 2.34.0
python3-werkzeug: add missing runtime dependencies
virendra thakur (1):
nodejs: Set CVE_PRODUCT to "node.js"
Change-Id: If9fadba6ede9e8de3b778d470bbd61f208f48e54
Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
diff --git a/poky/bitbake/lib/hashserv/client.py b/poky/bitbake/lib/hashserv/client.py
index 35a9768..b269879 100644
--- a/poky/bitbake/lib/hashserv/client.py
+++ b/poky/bitbake/lib/hashserv/client.py
@@ -16,6 +16,7 @@
class AsyncClient(bb.asyncrpc.AsyncClient):
MODE_NORMAL = 0
MODE_GET_STREAM = 1
+ MODE_EXIST_STREAM = 2
def __init__(self, username=None, password=None):
super().__init__("OEHASHEQUIV", "1.1", logger)
@@ -49,19 +50,36 @@
await self.socket.send("END")
return await self.socket.recv()
- if new_mode == self.MODE_NORMAL and self.mode == self.MODE_GET_STREAM:
+ async def normal_to_stream(command):
+ r = await self.invoke({command: None})
+ if r != "ok":
+ raise ConnectionError(
+ f"Unable to transition to stream mode: Bad response from server {r!r}"
+ )
+
+ self.logger.debug("Mode is now %s", command)
+
+ if new_mode == self.mode:
+ return
+
+ self.logger.debug("Transitioning mode %s -> %s", self.mode, new_mode)
+
+ # Always transition to normal mode before switching to any other mode
+ if self.mode != self.MODE_NORMAL:
r = await self._send_wrapper(stream_to_normal)
if r != "ok":
self.check_invoke_error(r)
- raise ConnectionError("Unable to transition to normal mode: Bad response from server %r" % r)
- elif new_mode == self.MODE_GET_STREAM and self.mode == self.MODE_NORMAL:
- r = await self.invoke({"get-stream": None})
- if r != "ok":
- raise ConnectionError("Unable to transition to stream mode: Bad response from server %r" % r)
- elif new_mode != self.mode:
- raise Exception(
- "Undefined mode transition %r -> %r" % (self.mode, new_mode)
- )
+ raise ConnectionError(
+ f"Unable to transition to normal mode: Bad response from server {r!r}"
+ )
+ self.logger.debug("Mode is now normal")
+
+ if new_mode == self.MODE_GET_STREAM:
+ await normal_to_stream("get-stream")
+ elif new_mode == self.MODE_EXIST_STREAM:
+ await normal_to_stream("exists-stream")
+ elif new_mode != self.MODE_NORMAL:
+ raise Exception("Undefined mode transition {self.mode!r} -> {new_mode!r}")
self.mode = new_mode
@@ -95,6 +113,11 @@
{"get": {"taskhash": taskhash, "method": method, "all": all_properties}}
)
+ async def unihash_exists(self, unihash):
+ await self._set_mode(self.MODE_EXIST_STREAM)
+ r = await self.send_stream(unihash)
+ return r == "true"
+
async def get_outhash(self, method, outhash, taskhash, with_unihash=True):
await self._set_mode(self.MODE_NORMAL)
return await self.invoke(
@@ -194,6 +217,34 @@
await self._set_mode(self.MODE_NORMAL)
return (await self.invoke({"get-db-query-columns": {}}))["columns"]
+ async def gc_status(self):
+ await self._set_mode(self.MODE_NORMAL)
+ return await self.invoke({"gc-status": {}})
+
+ async def gc_mark(self, mark, where):
+ """
+ Starts a new garbage collection operation identified by "mark". If
+ garbage collection is already in progress with "mark", the collection
+ is continued.
+
+ All unihash entries that match the "where" clause are marked to be
+ kept. In addition, any new entries added to the database after this
+ command will be automatically marked with "mark"
+ """
+ await self._set_mode(self.MODE_NORMAL)
+ return await self.invoke({"gc-mark": {"mark": mark, "where": where}})
+
+ async def gc_sweep(self, mark):
+ """
+ Finishes garbage collection for "mark". All unihash entries that have
+ not been marked will be deleted.
+
+ It is recommended to clean unused outhash entries after running this to
+ cleanup any dangling outhashes
+ """
+ await self._set_mode(self.MODE_NORMAL)
+ return await self.invoke({"gc-sweep": {"mark": mark}})
+
class Client(bb.asyncrpc.Client):
def __init__(self, username=None, password=None):
@@ -208,6 +259,7 @@
"report_unihash",
"report_unihash_equiv",
"get_taskhash",
+ "unihash_exists",
"get_outhash",
"get_stats",
"reset_stats",
@@ -224,7 +276,90 @@
"become_user",
"get_db_usage",
"get_db_query_columns",
+ "gc_status",
+ "gc_mark",
+ "gc_sweep",
)
def _get_async_client(self):
return AsyncClient(self.username, self.password)
+
+
+class ClientPool(bb.asyncrpc.ClientPool):
+ def __init__(
+ self,
+ address,
+ max_clients,
+ *,
+ username=None,
+ password=None,
+ become=None,
+ ):
+ super().__init__(max_clients)
+ self.address = address
+ self.username = username
+ self.password = password
+ self.become = become
+
+ async def _new_client(self):
+ client = await create_async_client(
+ self.address,
+ username=self.username,
+ password=self.password,
+ )
+ if self.become:
+ await client.become_user(self.become)
+ return client
+
+ def _run_key_tasks(self, queries, call):
+ results = {key: None for key in queries.keys()}
+
+ def make_task(key, args):
+ async def task(client):
+ nonlocal results
+ unihash = await call(client, args)
+ results[key] = unihash
+
+ return task
+
+ def gen_tasks():
+ for key, args in queries.items():
+ yield make_task(key, args)
+
+ self.run_tasks(gen_tasks())
+ return results
+
+ def get_unihashes(self, queries):
+ """
+ Query multiple unihashes in parallel.
+
+ The queries argument is a dictionary with arbitrary key. The values
+ must be a tuple of (method, taskhash).
+
+ Returns a dictionary with a corresponding key for each input key, and
+ the value is the queried unihash (which might be none if the query
+ failed)
+ """
+
+ async def call(client, args):
+ method, taskhash = args
+ return await client.get_unihash(method, taskhash)
+
+ return self._run_key_tasks(queries, call)
+
+ def unihashes_exist(self, queries):
+ """
+ Query multiple unihash existence checks in parallel.
+
+ The queries argument is a dictionary with arbitrary key. The values
+ must be a unihash.
+
+ Returns a dictionary with a corresponding key for each input key, and
+ the value is True or False if the unihash is known by the server (or
+ None if there was a failure)
+ """
+
+ async def call(client, unihash):
+ return await client.unihash_exists(unihash)
+
+ return self._run_key_tasks(queries, call)
diff --git a/poky/bitbake/lib/hashserv/server.py b/poky/bitbake/lib/hashserv/server.py
index a865078..68f64f9 100644
--- a/poky/bitbake/lib/hashserv/server.py
+++ b/poky/bitbake/lib/hashserv/server.py
@@ -199,7 +199,7 @@
if not self.user_has_permissions(*permissions, allow_anon=allow_anon):
if not self.user:
username = "Anonymous user"
- user_perms = self.anon_perms
+ user_perms = self.server.anon_perms
else:
username = self.user.username
user_perms = self.user.permissions
@@ -223,31 +223,18 @@
class ServerClient(bb.asyncrpc.AsyncServerConnection):
- def __init__(
- self,
- socket,
- db_engine,
- request_stats,
- backfill_queue,
- upstream,
- read_only,
- anon_perms,
- ):
- super().__init__(socket, "OEHASHEQUIV", logger)
- self.db_engine = db_engine
- self.request_stats = request_stats
+ def __init__(self, socket, server):
+ super().__init__(socket, "OEHASHEQUIV", server.logger)
+ self.server = server
self.max_chunk = bb.asyncrpc.DEFAULT_MAX_CHUNK
- self.backfill_queue = backfill_queue
- self.upstream = upstream
- self.read_only = read_only
self.user = None
- self.anon_perms = anon_perms
self.handlers.update(
{
"get": self.handle_get,
"get-outhash": self.handle_get_outhash,
"get-stream": self.handle_get_stream,
+ "exists-stream": self.handle_exists_stream,
"get-stats": self.handle_get_stats,
"get-db-usage": self.handle_get_db_usage,
"get-db-query-columns": self.handle_get_db_query_columns,
@@ -261,13 +248,16 @@
}
)
- if not read_only:
+ if not self.server.read_only:
self.handlers.update(
{
"report-equiv": self.handle_equivreport,
"reset-stats": self.handle_reset_stats,
"backfill-wait": self.handle_backfill_wait,
"remove": self.handle_remove,
+ "gc-mark": self.handle_gc_mark,
+ "gc-sweep": self.handle_gc_sweep,
+ "gc-status": self.handle_gc_status,
"clean-unused": self.handle_clean_unused,
"refresh-token": self.handle_refresh_token,
"set-user-perms": self.handle_set_perms,
@@ -282,10 +272,10 @@
def user_has_permissions(self, *permissions, allow_anon=True):
permissions = set(permissions)
if allow_anon:
- if ALL_PERM in self.anon_perms:
+ if ALL_PERM in self.server.anon_perms:
return True
- if not permissions - self.anon_perms:
+ if not permissions - self.server.anon_perms:
return True
if self.user is None:
@@ -303,10 +293,10 @@
return self.proto_version > (1, 0) and self.proto_version <= (1, 1)
async def process_requests(self):
- async with self.db_engine.connect(self.logger) as db:
+ async with self.server.db_engine.connect(self.logger) as db:
self.db = db
- if self.upstream is not None:
- self.upstream_client = await create_async_client(self.upstream)
+ if self.server.upstream is not None:
+ self.upstream_client = await create_async_client(self.server.upstream)
else:
self.upstream_client = None
@@ -323,7 +313,7 @@
if "stream" in k:
return await self.handlers[k](msg[k])
else:
- with self.request_stats.start_sample() as self.request_sample, self.request_sample.measure():
+ with self.server.request_stats.start_sample() as self.request_sample, self.request_sample.measure():
return await self.handlers[k](msg[k])
raise bb.asyncrpc.ClientError("Unrecognized command %r" % msg)
@@ -388,8 +378,7 @@
await self.db.insert_unihash(data["method"], data["taskhash"], data["unihash"])
await self.db.insert_outhash(data)
- @permissions(READ_PERM)
- async def handle_get_stream(self, request):
+ async def _stream_handler(self, handler):
await self.socket.send_message("ok")
while True:
@@ -404,42 +393,57 @@
# possible (which is why the request sample is handled manually
# instead of using 'with', and also why logging statements are
# commented out.
- self.request_sample = self.request_stats.start_sample()
+ self.request_sample = self.server.request_stats.start_sample()
request_measure = self.request_sample.measure()
request_measure.start()
if l == "END":
break
- (method, taskhash) = l.split()
- # self.logger.debug('Looking up %s %s' % (method, taskhash))
- row = await self.db.get_equivalent(method, taskhash)
-
- if row is not None:
- msg = row["unihash"]
- # self.logger.debug('Found equivalent task %s -> %s', (row['taskhash'], row['unihash']))
- elif self.upstream_client is not None:
- upstream = await self.upstream_client.get_unihash(method, taskhash)
- if upstream:
- msg = upstream
- else:
- msg = ""
- else:
- msg = ""
-
+ msg = await handler(l)
await self.socket.send(msg)
finally:
request_measure.end()
self.request_sample.end()
- # Post to the backfill queue after writing the result to minimize
- # the turn around time on a request
- if upstream is not None:
- await self.backfill_queue.put((method, taskhash))
-
await self.socket.send("ok")
return self.NO_RESPONSE
+ @permissions(READ_PERM)
+ async def handle_get_stream(self, request):
+ async def handler(l):
+ (method, taskhash) = l.split()
+ # self.logger.debug('Looking up %s %s' % (method, taskhash))
+ row = await self.db.get_equivalent(method, taskhash)
+
+ if row is not None:
+ # self.logger.debug('Found equivalent task %s -> %s', (row['taskhash'], row['unihash']))
+ return row["unihash"]
+
+ if self.upstream_client is not None:
+ upstream = await self.upstream_client.get_unihash(method, taskhash)
+ if upstream:
+ await self.server.backfill_queue.put((method, taskhash))
+ return upstream
+
+ return ""
+
+ return await self._stream_handler(handler)
+
+ @permissions(READ_PERM)
+ async def handle_exists_stream(self, request):
+ async def handler(l):
+ if await self.db.unihash_exists(l):
+ return "true"
+
+ if self.upstream_client is not None:
+ if await self.upstream_client.unihash_exists(l):
+ return "true"
+
+ return "false"
+
+ return await self._stream_handler(handler)
+
async def report_readonly(self, data):
method = data["method"]
outhash = data["outhash"]
@@ -461,7 +465,7 @@
# report is made inside the function
@permissions(READ_PERM)
async def handle_report(self, data):
- if self.read_only or not self.user_has_permissions(REPORT_PERM):
+ if self.server.read_only or not self.user_has_permissions(REPORT_PERM):
return await self.report_readonly(data)
outhash_data = {
@@ -538,24 +542,24 @@
@permissions(READ_PERM)
async def handle_get_stats(self, request):
return {
- "requests": self.request_stats.todict(),
+ "requests": self.server.request_stats.todict(),
}
@permissions(DB_ADMIN_PERM)
async def handle_reset_stats(self, request):
d = {
- "requests": self.request_stats.todict(),
+ "requests": self.server.request_stats.todict(),
}
- self.request_stats.reset()
+ self.server.request_stats.reset()
return d
@permissions(READ_PERM)
async def handle_backfill_wait(self, request):
d = {
- "tasks": self.backfill_queue.qsize(),
+ "tasks": self.server.backfill_queue.qsize(),
}
- await self.backfill_queue.join()
+ await self.server.backfill_queue.join()
return d
@permissions(DB_ADMIN_PERM)
@@ -567,6 +571,46 @@
return {"count": await self.db.remove(condition)}
@permissions(DB_ADMIN_PERM)
+ async def handle_gc_mark(self, request):
+ condition = request["where"]
+ mark = request["mark"]
+
+ if not isinstance(condition, dict):
+ raise TypeError("Bad condition type %s" % type(condition))
+
+ if not isinstance(mark, str):
+ raise TypeError("Bad mark type %s" % type(mark))
+
+ return {"count": await self.db.gc_mark(mark, condition)}
+
+ @permissions(DB_ADMIN_PERM)
+ async def handle_gc_sweep(self, request):
+ mark = request["mark"]
+
+ if not isinstance(mark, str):
+ raise TypeError("Bad mark type %s" % type(mark))
+
+ current_mark = await self.db.get_current_gc_mark()
+
+ if not current_mark or mark != current_mark:
+ raise bb.asyncrpc.InvokeError(
+ f"'{mark}' is not the current mark. Refusing to sweep"
+ )
+
+ count = await self.db.gc_sweep()
+
+ return {"count": count}
+
+ @permissions(DB_ADMIN_PERM)
+ async def handle_gc_status(self, request):
+ (keep_rows, remove_rows, current_mark) = await self.db.gc_status()
+ return {
+ "keep": keep_rows,
+ "remove": remove_rows,
+ "mark": current_mark,
+ }
+
+ @permissions(DB_ADMIN_PERM)
async def handle_clean_unused(self, request):
max_age = request["max_age_seconds"]
oldest = datetime.now() - timedelta(seconds=-max_age)
@@ -779,15 +823,7 @@
)
def accept_client(self, socket):
- return ServerClient(
- socket,
- self.db_engine,
- self.request_stats,
- self.backfill_queue,
- self.upstream,
- self.read_only,
- self.anon_perms,
- )
+ return ServerClient(socket, self)
async def create_admin_user(self):
admin_permissions = (ALL_PERM,)
diff --git a/poky/bitbake/lib/hashserv/sqlalchemy.py b/poky/bitbake/lib/hashserv/sqlalchemy.py
index cee04bf..fc3ae3d 100644
--- a/poky/bitbake/lib/hashserv/sqlalchemy.py
+++ b/poky/bitbake/lib/hashserv/sqlalchemy.py
@@ -28,24 +28,28 @@
delete,
update,
func,
+ inspect,
)
import sqlalchemy.engine
from sqlalchemy.orm import declarative_base
from sqlalchemy.exc import IntegrityError
+from sqlalchemy.dialects.postgresql import insert as postgres_insert
Base = declarative_base()
-class UnihashesV2(Base):
- __tablename__ = "unihashes_v2"
+class UnihashesV3(Base):
+ __tablename__ = "unihashes_v3"
id = Column(Integer, primary_key=True, autoincrement=True)
method = Column(Text, nullable=False)
taskhash = Column(Text, nullable=False)
unihash = Column(Text, nullable=False)
+ gc_mark = Column(Text, nullable=False)
__table_args__ = (
UniqueConstraint("method", "taskhash"),
- Index("taskhash_lookup_v3", "method", "taskhash"),
+ Index("taskhash_lookup_v4", "method", "taskhash"),
+ Index("unihash_lookup_v1", "unihash"),
)
@@ -79,6 +83,36 @@
__table_args__ = (UniqueConstraint("username"),)
+class Config(Base):
+ __tablename__ = "config"
+ id = Column(Integer, primary_key=True, autoincrement=True)
+ name = Column(Text, nullable=False)
+ value = Column(Text)
+ __table_args__ = (
+ UniqueConstraint("name"),
+ Index("config_lookup", "name"),
+ )
+
+
+#
+# Old table versions
+#
+DeprecatedBase = declarative_base()
+
+
+class UnihashesV2(DeprecatedBase):
+ __tablename__ = "unihashes_v2"
+ id = Column(Integer, primary_key=True, autoincrement=True)
+ method = Column(Text, nullable=False)
+ taskhash = Column(Text, nullable=False)
+ unihash = Column(Text, nullable=False)
+
+ __table_args__ = (
+ UniqueConstraint("method", "taskhash"),
+ Index("taskhash_lookup_v3", "method", "taskhash"),
+ )
+
+
class DatabaseEngine(object):
def __init__(self, url, username=None, password=None):
self.logger = logging.getLogger("hashserv.sqlalchemy")
@@ -91,6 +125,9 @@
self.url = self.url.set(password=password)
async def create(self):
+ def check_table_exists(conn, name):
+ return inspect(conn).has_table(name)
+
self.logger.info("Using database %s", self.url)
self.engine = create_async_engine(self.url, poolclass=NullPool)
@@ -99,6 +136,24 @@
self.logger.info("Creating tables...")
await conn.run_sync(Base.metadata.create_all)
+ if await conn.run_sync(check_table_exists, UnihashesV2.__tablename__):
+ self.logger.info("Upgrading Unihashes V2 -> V3...")
+ statement = insert(UnihashesV3).from_select(
+ ["id", "method", "unihash", "taskhash", "gc_mark"],
+ select(
+ UnihashesV2.id,
+ UnihashesV2.method,
+ UnihashesV2.unihash,
+ UnihashesV2.taskhash,
+ literal("").label("gc_mark"),
+ ),
+ )
+ self.logger.debug("%s", statement)
+ await conn.execute(statement)
+
+ await conn.run_sync(Base.metadata.drop_all, [UnihashesV2.__table__])
+ self.logger.info("Upgrade complete")
+
def connect(self, logger):
return Database(self.engine, logger)
@@ -118,6 +173,15 @@
)
+def _make_condition_statement(table, condition):
+ where = {}
+ for c in table.__table__.columns:
+ if c.key in condition and condition[c.key] is not None:
+ where[c] = condition[c.key]
+
+ return [(k == v) for k, v in where.items()]
+
+
class Database(object):
def __init__(self, engine, logger):
self.engine = engine
@@ -135,168 +199,265 @@
await self.db.close()
self.db = None
- async def get_unihash_by_taskhash_full(self, method, taskhash):
- statement = (
- select(
- OuthashesV2,
- UnihashesV2.unihash.label("unihash"),
- )
- .join(
- UnihashesV2,
- and_(
- UnihashesV2.method == OuthashesV2.method,
- UnihashesV2.taskhash == OuthashesV2.taskhash,
- ),
- )
- .where(
- OuthashesV2.method == method,
- OuthashesV2.taskhash == taskhash,
- )
- .order_by(
- OuthashesV2.created.asc(),
- )
- .limit(1)
- )
+ async def _execute(self, statement):
self.logger.debug("%s", statement)
+ return await self.db.execute(statement)
+
+ async def _set_config(self, name, value):
+ while True:
+ result = await self._execute(
+ update(Config).where(Config.name == name).values(value=value)
+ )
+
+ if result.rowcount == 0:
+ self.logger.debug("Config '%s' not found. Adding it", name)
+ try:
+ await self._execute(insert(Config).values(name=name, value=value))
+ except IntegrityError:
+ # Race. Try again
+ continue
+
+ break
+
+ def _get_config_subquery(self, name, default=None):
+ if default is not None:
+ return func.coalesce(
+ select(Config.value).where(Config.name == name).scalar_subquery(),
+ default,
+ )
+ return select(Config.value).where(Config.name == name).scalar_subquery()
+
+ async def _get_config(self, name):
+ result = await self._execute(select(Config.value).where(Config.name == name))
+ row = result.first()
+ if row is None:
+ return None
+ return row.value
+
+ async def get_unihash_by_taskhash_full(self, method, taskhash):
async with self.db.begin():
- result = await self.db.execute(statement)
+ result = await self._execute(
+ select(
+ OuthashesV2,
+ UnihashesV3.unihash.label("unihash"),
+ )
+ .join(
+ UnihashesV3,
+ and_(
+ UnihashesV3.method == OuthashesV2.method,
+ UnihashesV3.taskhash == OuthashesV2.taskhash,
+ ),
+ )
+ .where(
+ OuthashesV2.method == method,
+ OuthashesV2.taskhash == taskhash,
+ )
+ .order_by(
+ OuthashesV2.created.asc(),
+ )
+ .limit(1)
+ )
return map_row(result.first())
async def get_unihash_by_outhash(self, method, outhash):
- statement = (
- select(OuthashesV2, UnihashesV2.unihash.label("unihash"))
- .join(
- UnihashesV2,
- and_(
- UnihashesV2.method == OuthashesV2.method,
- UnihashesV2.taskhash == OuthashesV2.taskhash,
- ),
- )
- .where(
- OuthashesV2.method == method,
- OuthashesV2.outhash == outhash,
- )
- .order_by(
- OuthashesV2.created.asc(),
- )
- .limit(1)
- )
- self.logger.debug("%s", statement)
async with self.db.begin():
- result = await self.db.execute(statement)
+ result = await self._execute(
+ select(OuthashesV2, UnihashesV3.unihash.label("unihash"))
+ .join(
+ UnihashesV3,
+ and_(
+ UnihashesV3.method == OuthashesV2.method,
+ UnihashesV3.taskhash == OuthashesV2.taskhash,
+ ),
+ )
+ .where(
+ OuthashesV2.method == method,
+ OuthashesV2.outhash == outhash,
+ )
+ .order_by(
+ OuthashesV2.created.asc(),
+ )
+ .limit(1)
+ )
return map_row(result.first())
- async def get_outhash(self, method, outhash):
- statement = (
- select(OuthashesV2)
- .where(
- OuthashesV2.method == method,
- OuthashesV2.outhash == outhash,
- )
- .order_by(
- OuthashesV2.created.asc(),
- )
- .limit(1)
- )
-
- self.logger.debug("%s", statement)
+ async def unihash_exists(self, unihash):
async with self.db.begin():
- result = await self.db.execute(statement)
+ result = await self._execute(
+ select(UnihashesV3).where(UnihashesV3.unihash == unihash).limit(1)
+ )
+
+ return result.first() is not None
+
+ async def get_outhash(self, method, outhash):
+ async with self.db.begin():
+ result = await self._execute(
+ select(OuthashesV2)
+ .where(
+ OuthashesV2.method == method,
+ OuthashesV2.outhash == outhash,
+ )
+ .order_by(
+ OuthashesV2.created.asc(),
+ )
+ .limit(1)
+ )
return map_row(result.first())
async def get_equivalent_for_outhash(self, method, outhash, taskhash):
- statement = (
- select(
- OuthashesV2.taskhash.label("taskhash"),
- UnihashesV2.unihash.label("unihash"),
- )
- .join(
- UnihashesV2,
- and_(
- UnihashesV2.method == OuthashesV2.method,
- UnihashesV2.taskhash == OuthashesV2.taskhash,
- ),
- )
- .where(
- OuthashesV2.method == method,
- OuthashesV2.outhash == outhash,
- OuthashesV2.taskhash != taskhash,
- )
- .order_by(
- OuthashesV2.created.asc(),
- )
- .limit(1)
- )
- self.logger.debug("%s", statement)
async with self.db.begin():
- result = await self.db.execute(statement)
+ result = await self._execute(
+ select(
+ OuthashesV2.taskhash.label("taskhash"),
+ UnihashesV3.unihash.label("unihash"),
+ )
+ .join(
+ UnihashesV3,
+ and_(
+ UnihashesV3.method == OuthashesV2.method,
+ UnihashesV3.taskhash == OuthashesV2.taskhash,
+ ),
+ )
+ .where(
+ OuthashesV2.method == method,
+ OuthashesV2.outhash == outhash,
+ OuthashesV2.taskhash != taskhash,
+ )
+ .order_by(
+ OuthashesV2.created.asc(),
+ )
+ .limit(1)
+ )
return map_row(result.first())
async def get_equivalent(self, method, taskhash):
- statement = select(
- UnihashesV2.unihash,
- UnihashesV2.method,
- UnihashesV2.taskhash,
- ).where(
- UnihashesV2.method == method,
- UnihashesV2.taskhash == taskhash,
- )
- self.logger.debug("%s", statement)
async with self.db.begin():
- result = await self.db.execute(statement)
+ result = await self._execute(
+ select(
+ UnihashesV3.unihash,
+ UnihashesV3.method,
+ UnihashesV3.taskhash,
+ ).where(
+ UnihashesV3.method == method,
+ UnihashesV3.taskhash == taskhash,
+ )
+ )
return map_row(result.first())
async def remove(self, condition):
async def do_remove(table):
- where = {}
- for c in table.__table__.columns:
- if c.key in condition and condition[c.key] is not None:
- where[c] = condition[c.key]
-
+ where = _make_condition_statement(table, condition)
if where:
- statement = delete(table).where(*[(k == v) for k, v in where.items()])
- self.logger.debug("%s", statement)
async with self.db.begin():
- result = await self.db.execute(statement)
+ result = await self._execute(delete(table).where(*where))
return result.rowcount
return 0
count = 0
- count += await do_remove(UnihashesV2)
+ count += await do_remove(UnihashesV3)
count += await do_remove(OuthashesV2)
return count
- async def clean_unused(self, oldest):
- statement = delete(OuthashesV2).where(
- OuthashesV2.created < oldest,
- ~(
- select(UnihashesV2.id)
- .where(
- UnihashesV2.method == OuthashesV2.method,
- UnihashesV2.taskhash == OuthashesV2.taskhash,
- )
- .limit(1)
- .exists()
- ),
- )
- self.logger.debug("%s", statement)
+ async def get_current_gc_mark(self):
async with self.db.begin():
- result = await self.db.execute(statement)
+ return await self._get_config("gc-mark")
+
+ async def gc_status(self):
+ async with self.db.begin():
+ gc_mark_subquery = self._get_config_subquery("gc-mark", "")
+
+ result = await self._execute(
+ select(func.count())
+ .select_from(UnihashesV3)
+ .where(UnihashesV3.gc_mark == gc_mark_subquery)
+ )
+ keep_rows = result.scalar()
+
+ result = await self._execute(
+ select(func.count())
+ .select_from(UnihashesV3)
+ .where(UnihashesV3.gc_mark != gc_mark_subquery)
+ )
+ remove_rows = result.scalar()
+
+ return (keep_rows, remove_rows, await self._get_config("gc-mark"))
+
+ async def gc_mark(self, mark, condition):
+ async with self.db.begin():
+ await self._set_config("gc-mark", mark)
+
+ where = _make_condition_statement(UnihashesV3, condition)
+ if not where:
+ return 0
+
+ result = await self._execute(
+ update(UnihashesV3)
+ .values(gc_mark=self._get_config_subquery("gc-mark", ""))
+ .where(*where)
+ )
+ return result.rowcount
+
+ async def gc_sweep(self):
+ async with self.db.begin():
+ result = await self._execute(
+ delete(UnihashesV3).where(
+ # A sneaky conditional that provides some errant use
+ # protection: If the config mark is NULL, this will not
+ # match any rows because No default is specified in the
+ # select statement
+ UnihashesV3.gc_mark
+ != self._get_config_subquery("gc-mark")
+ )
+ )
+ await self._set_config("gc-mark", None)
+
+ return result.rowcount
+
+ async def clean_unused(self, oldest):
+ async with self.db.begin():
+ result = await self._execute(
+ delete(OuthashesV2).where(
+ OuthashesV2.created < oldest,
+ ~(
+ select(UnihashesV3.id)
+ .where(
+ UnihashesV3.method == OuthashesV2.method,
+ UnihashesV3.taskhash == OuthashesV2.taskhash,
+ )
+ .limit(1)
+ .exists()
+ ),
+ )
+ )
return result.rowcount
async def insert_unihash(self, method, taskhash, unihash):
- statement = insert(UnihashesV2).values(
- method=method,
- taskhash=taskhash,
- unihash=unihash,
- )
- self.logger.debug("%s", statement)
+ # Postgres specific ignore on insert duplicate
+ if self.engine.name == "postgresql":
+ statement = (
+ postgres_insert(UnihashesV3)
+ .values(
+ method=method,
+ taskhash=taskhash,
+ unihash=unihash,
+ gc_mark=self._get_config_subquery("gc-mark", ""),
+ )
+ .on_conflict_do_nothing(index_elements=("method", "taskhash"))
+ )
+ else:
+ statement = insert(UnihashesV3).values(
+ method=method,
+ taskhash=taskhash,
+ unihash=unihash,
+ gc_mark=self._get_config_subquery("gc-mark", ""),
+ )
+
try:
async with self.db.begin():
- await self.db.execute(statement)
- return True
+ result = await self._execute(statement)
+ return result.rowcount != 0
except IntegrityError:
self.logger.debug(
"%s, %s, %s already in unihash database", method, taskhash, unihash
@@ -311,12 +472,22 @@
if "created" in data and not isinstance(data["created"], datetime):
data["created"] = datetime.fromisoformat(data["created"])
- statement = insert(OuthashesV2).values(**data)
- self.logger.debug("%s", statement)
+ # Postgres specific ignore on insert duplicate
+ if self.engine.name == "postgresql":
+ statement = (
+ postgres_insert(OuthashesV2)
+ .values(**data)
+ .on_conflict_do_nothing(
+ index_elements=("method", "taskhash", "outhash")
+ )
+ )
+ else:
+ statement = insert(OuthashesV2).values(**data)
+
try:
async with self.db.begin():
- await self.db.execute(statement)
- return True
+ result = await self._execute(statement)
+ return result.rowcount != 0
except IntegrityError:
self.logger.debug(
"%s, %s already in outhash database", data["method"], data["outhash"]
@@ -324,16 +495,16 @@
return False
async def _get_user(self, username):
- statement = select(
- Users.username,
- Users.permissions,
- Users.token,
- ).where(
- Users.username == username,
- )
- self.logger.debug("%s", statement)
async with self.db.begin():
- result = await self.db.execute(statement)
+ result = await self._execute(
+ select(
+ Users.username,
+ Users.permissions,
+ Users.token,
+ ).where(
+ Users.username == username,
+ )
+ )
return result.first()
async def lookup_user_token(self, username):
@@ -346,70 +517,66 @@
return map_user(await self._get_user(username))
async def set_user_token(self, username, token):
- statement = (
- update(Users)
- .where(
- Users.username == username,
- )
- .values(
- token=token,
- )
- )
- self.logger.debug("%s", statement)
async with self.db.begin():
- result = await self.db.execute(statement)
+ result = await self._execute(
+ update(Users)
+ .where(
+ Users.username == username,
+ )
+ .values(
+ token=token,
+ )
+ )
return result.rowcount != 0
async def set_user_perms(self, username, permissions):
- statement = (
- update(Users)
- .where(Users.username == username)
- .values(permissions=" ".join(permissions))
- )
- self.logger.debug("%s", statement)
async with self.db.begin():
- result = await self.db.execute(statement)
+ result = await self._execute(
+ update(Users)
+ .where(Users.username == username)
+ .values(permissions=" ".join(permissions))
+ )
return result.rowcount != 0
async def get_all_users(self):
- statement = select(
- Users.username,
- Users.permissions,
- )
- self.logger.debug("%s", statement)
async with self.db.begin():
- result = await self.db.execute(statement)
+ result = await self._execute(
+ select(
+ Users.username,
+ Users.permissions,
+ )
+ )
return [map_user(row) for row in result]
async def new_user(self, username, permissions, token):
- statement = insert(Users).values(
- username=username,
- permissions=" ".join(permissions),
- token=token,
- )
- self.logger.debug("%s", statement)
try:
async with self.db.begin():
- await self.db.execute(statement)
+ await self._execute(
+ insert(Users).values(
+ username=username,
+ permissions=" ".join(permissions),
+ token=token,
+ )
+ )
return True
except IntegrityError as e:
self.logger.debug("Cannot create new user %s: %s", username, e)
return False
async def delete_user(self, username):
- statement = delete(Users).where(Users.username == username)
- self.logger.debug("%s", statement)
async with self.db.begin():
- result = await self.db.execute(statement)
+ result = await self._execute(
+ delete(Users).where(Users.username == username)
+ )
return result.rowcount != 0
async def get_usage(self):
usage = {}
async with self.db.begin() as session:
for name, table in Base.metadata.tables.items():
- statement = select(func.count()).select_from(table)
- self.logger.debug("%s", statement)
- result = await self.db.execute(statement)
+ result = await self._execute(
+ statement=select(func.count()).select_from(table)
+ )
usage[name] = {
"rows": result.scalar(),
}
@@ -418,7 +585,7 @@
async def get_query_columns(self):
columns = set()
- for table in (UnihashesV2, OuthashesV2):
+ for table in (UnihashesV3, OuthashesV2):
for c in table.__table__.columns:
if not isinstance(c.type, Text):
continue
diff --git a/poky/bitbake/lib/hashserv/sqlite.py b/poky/bitbake/lib/hashserv/sqlite.py
index f93cb2c..da2e844 100644
--- a/poky/bitbake/lib/hashserv/sqlite.py
+++ b/poky/bitbake/lib/hashserv/sqlite.py
@@ -15,6 +15,7 @@
("method", "TEXT NOT NULL", "UNIQUE"),
("taskhash", "TEXT NOT NULL", "UNIQUE"),
("unihash", "TEXT NOT NULL", ""),
+ ("gc_mark", "TEXT NOT NULL", ""),
)
UNIHASH_TABLE_COLUMNS = tuple(name for name, _, _ in UNIHASH_TABLE_DEFINITION)
@@ -44,6 +45,14 @@
USERS_TABLE_COLUMNS = tuple(name for name, _, _ in USERS_TABLE_DEFINITION)
+CONFIG_TABLE_DEFINITION = (
+ ("name", "TEXT NOT NULL", "UNIQUE"),
+ ("value", "TEXT", ""),
+)
+
+CONFIG_TABLE_COLUMNS = tuple(name for name, _, _ in CONFIG_TABLE_DEFINITION)
+
+
def _make_table(cursor, name, definition):
cursor.execute(
"""
@@ -71,6 +80,35 @@
)
+def _make_condition_statement(columns, condition):
+ where = {}
+ for c in columns:
+ if c in condition and condition[c] is not None:
+ where[c] = condition[c]
+
+ return where, " AND ".join("%s=:%s" % (k, k) for k in where.keys())
+
+
+def _get_sqlite_version(cursor):
+ cursor.execute("SELECT sqlite_version()")
+
+ version = []
+ for v in cursor.fetchone()[0].split("."):
+ try:
+ version.append(int(v))
+ except ValueError:
+ version.append(v)
+
+ return tuple(version)
+
+
+def _schema_table_name(version):
+ if version >= (3, 33):
+ return "sqlite_schema"
+
+ return "sqlite_master"
+
+
class DatabaseEngine(object):
def __init__(self, dbname, sync):
self.dbname = dbname
@@ -82,9 +120,10 @@
db.row_factory = sqlite3.Row
with closing(db.cursor()) as cursor:
- _make_table(cursor, "unihashes_v2", UNIHASH_TABLE_DEFINITION)
+ _make_table(cursor, "unihashes_v3", UNIHASH_TABLE_DEFINITION)
_make_table(cursor, "outhashes_v2", OUTHASH_TABLE_DEFINITION)
_make_table(cursor, "users", USERS_TABLE_DEFINITION)
+ _make_table(cursor, "config", CONFIG_TABLE_DEFINITION)
cursor.execute("PRAGMA journal_mode = WAL")
cursor.execute(
@@ -96,17 +135,41 @@
cursor.execute("DROP INDEX IF EXISTS outhash_lookup")
cursor.execute("DROP INDEX IF EXISTS taskhash_lookup_v2")
cursor.execute("DROP INDEX IF EXISTS outhash_lookup_v2")
+ cursor.execute("DROP INDEX IF EXISTS taskhash_lookup_v3")
# TODO: Upgrade from tasks_v2?
cursor.execute("DROP TABLE IF EXISTS tasks_v2")
# Create new indexes
cursor.execute(
- "CREATE INDEX IF NOT EXISTS taskhash_lookup_v3 ON unihashes_v2 (method, taskhash)"
+ "CREATE INDEX IF NOT EXISTS taskhash_lookup_v4 ON unihashes_v3 (method, taskhash)"
+ )
+ cursor.execute(
+ "CREATE INDEX IF NOT EXISTS unihash_lookup_v1 ON unihashes_v3 (unihash)"
)
cursor.execute(
"CREATE INDEX IF NOT EXISTS outhash_lookup_v3 ON outhashes_v2 (method, outhash)"
)
+ cursor.execute("CREATE INDEX IF NOT EXISTS config_lookup ON config (name)")
+
+ sqlite_version = _get_sqlite_version(cursor)
+
+ cursor.execute(
+ f"""
+ SELECT name FROM {_schema_table_name(sqlite_version)} WHERE type = 'table' AND name = 'unihashes_v2'
+ """
+ )
+ if cursor.fetchone():
+ self.logger.info("Upgrading Unihashes V2 -> V3...")
+ cursor.execute(
+ """
+ INSERT INTO unihashes_v3 (id, method, unihash, taskhash, gc_mark)
+ SELECT id, method, unihash, taskhash, '' FROM unihashes_v2
+ """
+ )
+ cursor.execute("DROP TABLE unihashes_v2")
+ db.commit()
+ self.logger.info("Upgrade complete")
def connect(self, logger):
return Database(logger, self.dbname, self.sync)
@@ -126,16 +189,7 @@
"PRAGMA synchronous = %s" % ("NORMAL" if sync else "OFF")
)
- cursor.execute("SELECT sqlite_version()")
-
- version = []
- for v in cursor.fetchone()[0].split("."):
- try:
- version.append(int(v))
- except ValueError:
- version.append(v)
-
- self.sqlite_version = tuple(version)
+ self.sqlite_version = _get_sqlite_version(cursor)
async def __aenter__(self):
return self
@@ -143,6 +197,30 @@
async def __aexit__(self, exc_type, exc_value, traceback):
await self.close()
+ async def _set_config(self, cursor, name, value):
+ cursor.execute(
+ """
+ INSERT OR REPLACE INTO config (id, name, value) VALUES
+ ((SELECT id FROM config WHERE name=:name), :name, :value)
+ """,
+ {
+ "name": name,
+ "value": value,
+ },
+ )
+
+ async def _get_config(self, cursor, name):
+ cursor.execute(
+ "SELECT value FROM config WHERE name=:name",
+ {
+ "name": name,
+ },
+ )
+ row = cursor.fetchone()
+ if row is None:
+ return None
+ return row["value"]
+
async def close(self):
self.db.close()
@@ -150,8 +228,8 @@
with closing(self.db.cursor()) as cursor:
cursor.execute(
"""
- SELECT *, unihashes_v2.unihash AS unihash FROM outhashes_v2
- INNER JOIN unihashes_v2 ON unihashes_v2.method=outhashes_v2.method AND unihashes_v2.taskhash=outhashes_v2.taskhash
+ SELECT *, unihashes_v3.unihash AS unihash FROM outhashes_v2
+ INNER JOIN unihashes_v3 ON unihashes_v3.method=outhashes_v2.method AND unihashes_v3.taskhash=outhashes_v2.taskhash
WHERE outhashes_v2.method=:method AND outhashes_v2.taskhash=:taskhash
ORDER BY outhashes_v2.created ASC
LIMIT 1
@@ -167,8 +245,8 @@
with closing(self.db.cursor()) as cursor:
cursor.execute(
"""
- SELECT *, unihashes_v2.unihash AS unihash FROM outhashes_v2
- INNER JOIN unihashes_v2 ON unihashes_v2.method=outhashes_v2.method AND unihashes_v2.taskhash=outhashes_v2.taskhash
+ SELECT *, unihashes_v3.unihash AS unihash FROM outhashes_v2
+ INNER JOIN unihashes_v3 ON unihashes_v3.method=outhashes_v2.method AND unihashes_v3.taskhash=outhashes_v2.taskhash
WHERE outhashes_v2.method=:method AND outhashes_v2.outhash=:outhash
ORDER BY outhashes_v2.created ASC
LIMIT 1
@@ -180,6 +258,19 @@
)
return cursor.fetchone()
+ async def unihash_exists(self, unihash):
+ with closing(self.db.cursor()) as cursor:
+ cursor.execute(
+ """
+ SELECT * FROM unihashes_v3 WHERE unihash=:unihash
+ LIMIT 1
+ """,
+ {
+ "unihash": unihash,
+ },
+ )
+ return cursor.fetchone() is not None
+
async def get_outhash(self, method, outhash):
with closing(self.db.cursor()) as cursor:
cursor.execute(
@@ -200,8 +291,8 @@
with closing(self.db.cursor()) as cursor:
cursor.execute(
"""
- SELECT outhashes_v2.taskhash AS taskhash, unihashes_v2.unihash AS unihash FROM outhashes_v2
- INNER JOIN unihashes_v2 ON unihashes_v2.method=outhashes_v2.method AND unihashes_v2.taskhash=outhashes_v2.taskhash
+ SELECT outhashes_v2.taskhash AS taskhash, unihashes_v3.unihash AS unihash FROM outhashes_v2
+ INNER JOIN unihashes_v3 ON unihashes_v3.method=outhashes_v2.method AND unihashes_v3.taskhash=outhashes_v2.taskhash
-- Select any matching output hash except the one we just inserted
WHERE outhashes_v2.method=:method AND outhashes_v2.outhash=:outhash AND outhashes_v2.taskhash!=:taskhash
-- Pick the oldest hash
@@ -219,7 +310,7 @@
async def get_equivalent(self, method, taskhash):
with closing(self.db.cursor()) as cursor:
cursor.execute(
- "SELECT taskhash, method, unihash FROM unihashes_v2 WHERE method=:method AND taskhash=:taskhash",
+ "SELECT taskhash, method, unihash FROM unihashes_v3 WHERE method=:method AND taskhash=:taskhash",
{
"method": method,
"taskhash": taskhash,
@@ -229,15 +320,9 @@
async def remove(self, condition):
def do_remove(columns, table_name, cursor):
- where = {}
- for c in columns:
- if c in condition and condition[c] is not None:
- where[c] = condition[c]
-
+ where, clause = _make_condition_statement(columns, condition)
if where:
- query = ("DELETE FROM %s WHERE " % table_name) + " AND ".join(
- "%s=:%s" % (k, k) for k in where.keys()
- )
+ query = f"DELETE FROM {table_name} WHERE {clause}"
cursor.execute(query, where)
return cursor.rowcount
@@ -246,17 +331,80 @@
count = 0
with closing(self.db.cursor()) as cursor:
count += do_remove(OUTHASH_TABLE_COLUMNS, "outhashes_v2", cursor)
- count += do_remove(UNIHASH_TABLE_COLUMNS, "unihashes_v2", cursor)
+ count += do_remove(UNIHASH_TABLE_COLUMNS, "unihashes_v3", cursor)
self.db.commit()
return count
+ async def get_current_gc_mark(self):
+ with closing(self.db.cursor()) as cursor:
+ return await self._get_config(cursor, "gc-mark")
+
+ async def gc_status(self):
+ with closing(self.db.cursor()) as cursor:
+ cursor.execute(
+ """
+ SELECT COUNT() FROM unihashes_v3 WHERE
+ gc_mark=COALESCE((SELECT value FROM config WHERE name='gc-mark'), '')
+ """
+ )
+ keep_rows = cursor.fetchone()[0]
+
+ cursor.execute(
+ """
+ SELECT COUNT() FROM unihashes_v3 WHERE
+ gc_mark!=COALESCE((SELECT value FROM config WHERE name='gc-mark'), '')
+ """
+ )
+ remove_rows = cursor.fetchone()[0]
+
+ current_mark = await self._get_config(cursor, "gc-mark")
+
+ return (keep_rows, remove_rows, current_mark)
+
+ async def gc_mark(self, mark, condition):
+ with closing(self.db.cursor()) as cursor:
+ await self._set_config(cursor, "gc-mark", mark)
+
+ where, clause = _make_condition_statement(UNIHASH_TABLE_COLUMNS, condition)
+
+ new_rows = 0
+ if where:
+ cursor.execute(
+ f"""
+ UPDATE unihashes_v3 SET
+ gc_mark=COALESCE((SELECT value FROM config WHERE name='gc-mark'), '')
+ WHERE {clause}
+ """,
+ where,
+ )
+ new_rows = cursor.rowcount
+
+ self.db.commit()
+ return new_rows
+
+ async def gc_sweep(self):
+ with closing(self.db.cursor()) as cursor:
+ # NOTE: COALESCE is not used in this query so that if the current
+ # mark is NULL, nothing will happen
+ cursor.execute(
+ """
+ DELETE FROM unihashes_v3 WHERE
+ gc_mark!=(SELECT value FROM config WHERE name='gc-mark')
+ """
+ )
+ count = cursor.rowcount
+ await self._set_config(cursor, "gc-mark", None)
+
+ self.db.commit()
+ return count
+
async def clean_unused(self, oldest):
with closing(self.db.cursor()) as cursor:
cursor.execute(
"""
DELETE FROM outhashes_v2 WHERE created<:oldest AND NOT EXISTS (
- SELECT unihashes_v2.id FROM unihashes_v2 WHERE unihashes_v2.method=outhashes_v2.method AND unihashes_v2.taskhash=outhashes_v2.taskhash LIMIT 1
+ SELECT unihashes_v3.id FROM unihashes_v3 WHERE unihashes_v3.method=outhashes_v2.method AND unihashes_v3.taskhash=outhashes_v2.taskhash LIMIT 1
)
""",
{
@@ -271,7 +419,13 @@
prevrowid = cursor.lastrowid
cursor.execute(
"""
- INSERT OR IGNORE INTO unihashes_v2 (method, taskhash, unihash) VALUES(:method, :taskhash, :unihash)
+ INSERT OR IGNORE INTO unihashes_v3 (method, taskhash, unihash, gc_mark) VALUES
+ (
+ :method,
+ :taskhash,
+ :unihash,
+ COALESCE((SELECT value FROM config WHERE name='gc-mark'), '')
+ )
""",
{
"method": method,
@@ -383,14 +537,9 @@
async def get_usage(self):
usage = {}
with closing(self.db.cursor()) as cursor:
- if self.sqlite_version >= (3, 33):
- table_name = "sqlite_schema"
- else:
- table_name = "sqlite_master"
-
cursor.execute(
f"""
- SELECT name FROM {table_name} WHERE type = 'table' AND name NOT LIKE 'sqlite_%'
+ SELECT name FROM {_schema_table_name(self.sqlite_version)} WHERE type = 'table' AND name NOT LIKE 'sqlite_%'
"""
)
for row in cursor.fetchall():
diff --git a/poky/bitbake/lib/hashserv/tests.py b/poky/bitbake/lib/hashserv/tests.py
index 869f763..0809453 100644
--- a/poky/bitbake/lib/hashserv/tests.py
+++ b/poky/bitbake/lib/hashserv/tests.py
@@ -8,6 +8,7 @@
from . import create_server, create_client
from .server import DEFAULT_ANON_PERMS, ALL_PERMISSIONS
from bb.asyncrpc import InvokeError
+from .client import ClientPool
import hashlib
import logging
import multiprocessing
@@ -442,6 +443,11 @@
self.assertEqual(result['taskhash'], taskhash9, 'Server failed to copy unihash from upstream')
self.assertEqual(result['method'], self.METHOD)
+ def test_unihash_exsits(self):
+ taskhash, outhash, unihash = self.create_test_hash(self.client)
+ self.assertTrue(self.client.unihash_exists(unihash))
+ self.assertFalse(self.client.unihash_exists('6662e699d6e3d894b24408ff9a4031ef9b038ee8'))
+
def test_ro_server(self):
rw_server = self.start_server()
rw_client = self.start_client(rw_server.address)
@@ -549,6 +555,88 @@
# shares a taskhash with Task 2
self.assertClientGetHash(self.client, taskhash2, unihash2)
+
+ def test_client_pool_get_unihashes(self):
+ TEST_INPUT = (
+ # taskhash outhash unihash
+ ('8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a', 'afe240a439959ce86f5e322f8c208e1fedefea9e813f2140c81af866cc9edf7e','218e57509998197d570e2c98512d0105985dffc9'),
+ # Duplicated taskhash with multiple output hashes and unihashes.
+ ('8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a', '0904a7fe3dc712d9fd8a74a616ddca2a825a8ee97adf0bd3fc86082c7639914d', 'ae9a7d252735f0dafcdb10e2e02561ca3a47314c'),
+ # Equivalent hash
+ ("044c2ec8aaf480685a00ff6ff49e6162e6ad34e1", '0904a7fe3dc712d9fd8a74a616ddca2a825a8ee97adf0bd3fc86082c7639914d', "def64766090d28f627e816454ed46894bb3aab36"),
+ ("e3da00593d6a7fb435c7e2114976c59c5fd6d561", "1cf8713e645f491eb9c959d20b5cae1c47133a292626dda9b10709857cbe688a", "3b5d3d83f07f259e9086fcb422c855286e18a57d"),
+ ('35788efcb8dfb0a02659d81cf2bfd695fb30faf9', '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3fce98fd7e5970796f', 'f46d3fbb439bd9b921095da657a4de906510d2cd'),
+ ('35788efcb8dfb0a02659d81cf2bfd695fb30fafa', '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3fce98fd7e5970796f', 'f46d3fbb439bd9b921095da657a4de906510d2ce'),
+ ('9d81d76242cc7cfaf7bf74b94b9cd2e29324ed74', '8470d56547eea6236d7c81a644ce74670ca0bbda998e13c629ef6bb3f0d60b69', '05d2a63c81e32f0a36542ca677e8ad852365c538'),
+ )
+ EXTRA_QUERIES = (
+ "6b6be7a84ab179b4240c4302518dc3f6",
+ )
+
+ with ClientPool(self.server_address, 10) as client_pool:
+ for taskhash, outhash, unihash in TEST_INPUT:
+ self.client.report_unihash(taskhash, self.METHOD, outhash, unihash)
+
+ query = {idx: (self.METHOD, data[0]) for idx, data in enumerate(TEST_INPUT)}
+ for idx, taskhash in enumerate(EXTRA_QUERIES):
+ query[idx + len(TEST_INPUT)] = (self.METHOD, taskhash)
+
+ result = client_pool.get_unihashes(query)
+
+ self.assertDictEqual(result, {
+ 0: "218e57509998197d570e2c98512d0105985dffc9",
+ 1: "218e57509998197d570e2c98512d0105985dffc9",
+ 2: "218e57509998197d570e2c98512d0105985dffc9",
+ 3: "3b5d3d83f07f259e9086fcb422c855286e18a57d",
+ 4: "f46d3fbb439bd9b921095da657a4de906510d2cd",
+ 5: "f46d3fbb439bd9b921095da657a4de906510d2cd",
+ 6: "05d2a63c81e32f0a36542ca677e8ad852365c538",
+ 7: None,
+ })
+
+ def test_client_pool_unihash_exists(self):
+ TEST_INPUT = (
+ # taskhash outhash unihash
+ ('8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a', 'afe240a439959ce86f5e322f8c208e1fedefea9e813f2140c81af866cc9edf7e','218e57509998197d570e2c98512d0105985dffc9'),
+ # Duplicated taskhash with multiple output hashes and unihashes.
+ ('8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a', '0904a7fe3dc712d9fd8a74a616ddca2a825a8ee97adf0bd3fc86082c7639914d', 'ae9a7d252735f0dafcdb10e2e02561ca3a47314c'),
+ # Equivalent hash
+ ("044c2ec8aaf480685a00ff6ff49e6162e6ad34e1", '0904a7fe3dc712d9fd8a74a616ddca2a825a8ee97adf0bd3fc86082c7639914d', "def64766090d28f627e816454ed46894bb3aab36"),
+ ("e3da00593d6a7fb435c7e2114976c59c5fd6d561", "1cf8713e645f491eb9c959d20b5cae1c47133a292626dda9b10709857cbe688a", "3b5d3d83f07f259e9086fcb422c855286e18a57d"),
+ ('35788efcb8dfb0a02659d81cf2bfd695fb30faf9', '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3fce98fd7e5970796f', 'f46d3fbb439bd9b921095da657a4de906510d2cd'),
+ ('35788efcb8dfb0a02659d81cf2bfd695fb30fafa', '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3fce98fd7e5970796f', 'f46d3fbb439bd9b921095da657a4de906510d2ce'),
+ ('9d81d76242cc7cfaf7bf74b94b9cd2e29324ed74', '8470d56547eea6236d7c81a644ce74670ca0bbda998e13c629ef6bb3f0d60b69', '05d2a63c81e32f0a36542ca677e8ad852365c538'),
+ )
+ EXTRA_QUERIES = (
+ "6b6be7a84ab179b4240c4302518dc3f6",
+ )
+
+ result_unihashes = set()
+
+
+ with ClientPool(self.server_address, 10) as client_pool:
+ for taskhash, outhash, unihash in TEST_INPUT:
+ result = self.client.report_unihash(taskhash, self.METHOD, outhash, unihash)
+ result_unihashes.add(result["unihash"])
+
+ query = {}
+ expected = {}
+
+ for _, _, unihash in TEST_INPUT:
+ idx = len(query)
+ query[idx] = unihash
+ expected[idx] = unihash in result_unihashes
+
+
+ for unihash in EXTRA_QUERIES:
+ idx = len(query)
+ query[idx] = unihash
+ expected[idx] = False
+
+ result = client_pool.unihashes_exist(query)
+ self.assertDictEqual(result, expected)
+
+
def test_auth_read_perms(self):
admin_client = self.start_auth_server()
@@ -810,6 +898,27 @@
with self.auth_perms("@user-admin") as client:
become = client.become_user(client.username)
+ def test_auth_gc(self):
+ admin_client = self.start_auth_server()
+
+ with self.auth_perms() as client, self.assertRaises(InvokeError):
+ client.gc_mark("ABC", {"unihash": "123"})
+
+ with self.auth_perms() as client, self.assertRaises(InvokeError):
+ client.gc_status()
+
+ with self.auth_perms() as client, self.assertRaises(InvokeError):
+ client.gc_sweep("ABC")
+
+ with self.auth_perms("@db-admin") as client:
+ client.gc_mark("ABC", {"unihash": "123"})
+
+ with self.auth_perms("@db-admin") as client:
+ client.gc_status()
+
+ with self.auth_perms("@db-admin") as client:
+ client.gc_sweep("ABC")
+
def test_get_db_usage(self):
usage = self.client.get_db_usage()
@@ -837,6 +946,147 @@
data = client.get_taskhash(self.METHOD, taskhash, True)
self.assertEqual(data["owner"], user["username"])
+ def test_gc(self):
+ taskhash = '53b8dce672cb6d0c73170be43f540460bfc347b4'
+ outhash = '5a9cb1649625f0bf41fc7791b635cd9c2d7118c7f021ba87dcd03f72b67ce7a8'
+ unihash = 'f37918cc02eb5a520b1aff86faacbc0a38124646'
+
+ result = self.client.report_unihash(taskhash, self.METHOD, outhash, unihash)
+ self.assertEqual(result['unihash'], unihash, 'Server returned bad unihash')
+
+ taskhash2 = '3bf6f1e89d26205aec90da04854fbdbf73afe6b4'
+ outhash2 = '77623a549b5b1a31e3732dfa8fe61d7ce5d44b3370f253c5360e136b852967b4'
+ unihash2 = 'af36b199320e611fbb16f1f277d3ee1d619ca58b'
+
+ result = self.client.report_unihash(taskhash2, self.METHOD, outhash2, unihash2)
+ self.assertClientGetHash(self.client, taskhash2, unihash2)
+
+ # Mark the first unihash to be kept
+ ret = self.client.gc_mark("ABC", {"unihash": unihash, "method": self.METHOD})
+ self.assertEqual(ret, {"count": 1})
+
+ ret = self.client.gc_status()
+ self.assertEqual(ret, {"mark": "ABC", "keep": 1, "remove": 1})
+
+ # Second hash is still there; mark doesn't delete hashes
+ self.assertClientGetHash(self.client, taskhash2, unihash2)
+
+ ret = self.client.gc_sweep("ABC")
+ self.assertEqual(ret, {"count": 1})
+
+ # Hash is gone. Taskhash is returned for second hash
+ self.assertClientGetHash(self.client, taskhash2, None)
+ # First hash is still present
+ self.assertClientGetHash(self.client, taskhash, unihash)
+
+ def test_gc_switch_mark(self):
+ taskhash = '53b8dce672cb6d0c73170be43f540460bfc347b4'
+ outhash = '5a9cb1649625f0bf41fc7791b635cd9c2d7118c7f021ba87dcd03f72b67ce7a8'
+ unihash = 'f37918cc02eb5a520b1aff86faacbc0a38124646'
+
+ result = self.client.report_unihash(taskhash, self.METHOD, outhash, unihash)
+ self.assertEqual(result['unihash'], unihash, 'Server returned bad unihash')
+
+ taskhash2 = '3bf6f1e89d26205aec90da04854fbdbf73afe6b4'
+ outhash2 = '77623a549b5b1a31e3732dfa8fe61d7ce5d44b3370f253c5360e136b852967b4'
+ unihash2 = 'af36b199320e611fbb16f1f277d3ee1d619ca58b'
+
+ result = self.client.report_unihash(taskhash2, self.METHOD, outhash2, unihash2)
+ self.assertClientGetHash(self.client, taskhash2, unihash2)
+
+ # Mark the first unihash to be kept
+ ret = self.client.gc_mark("ABC", {"unihash": unihash, "method": self.METHOD})
+ self.assertEqual(ret, {"count": 1})
+
+ ret = self.client.gc_status()
+ self.assertEqual(ret, {"mark": "ABC", "keep": 1, "remove": 1})
+
+ # Second hash is still there; mark doesn't delete hashes
+ self.assertClientGetHash(self.client, taskhash2, unihash2)
+
+ # Switch to a different mark and mark the second hash. This will start
+ # a new collection cycle
+ ret = self.client.gc_mark("DEF", {"unihash": unihash2, "method": self.METHOD})
+ self.assertEqual(ret, {"count": 1})
+
+ ret = self.client.gc_status()
+ self.assertEqual(ret, {"mark": "DEF", "keep": 1, "remove": 1})
+
+ # Both hashes are still present
+ self.assertClientGetHash(self.client, taskhash2, unihash2)
+ self.assertClientGetHash(self.client, taskhash, unihash)
+
+ # Sweep with the new mark
+ ret = self.client.gc_sweep("DEF")
+ self.assertEqual(ret, {"count": 1})
+
+ # First hash is gone, second is kept
+ self.assertClientGetHash(self.client, taskhash2, unihash2)
+ self.assertClientGetHash(self.client, taskhash, None)
+
+ def test_gc_switch_sweep_mark(self):
+ taskhash = '53b8dce672cb6d0c73170be43f540460bfc347b4'
+ outhash = '5a9cb1649625f0bf41fc7791b635cd9c2d7118c7f021ba87dcd03f72b67ce7a8'
+ unihash = 'f37918cc02eb5a520b1aff86faacbc0a38124646'
+
+ result = self.client.report_unihash(taskhash, self.METHOD, outhash, unihash)
+ self.assertEqual(result['unihash'], unihash, 'Server returned bad unihash')
+
+ taskhash2 = '3bf6f1e89d26205aec90da04854fbdbf73afe6b4'
+ outhash2 = '77623a549b5b1a31e3732dfa8fe61d7ce5d44b3370f253c5360e136b852967b4'
+ unihash2 = 'af36b199320e611fbb16f1f277d3ee1d619ca58b'
+
+ result = self.client.report_unihash(taskhash2, self.METHOD, outhash2, unihash2)
+ self.assertClientGetHash(self.client, taskhash2, unihash2)
+
+ # Mark the first unihash to be kept
+ ret = self.client.gc_mark("ABC", {"unihash": unihash, "method": self.METHOD})
+ self.assertEqual(ret, {"count": 1})
+
+ ret = self.client.gc_status()
+ self.assertEqual(ret, {"mark": "ABC", "keep": 1, "remove": 1})
+
+ # Sweeping with a different mark raises an error
+ with self.assertRaises(InvokeError):
+ self.client.gc_sweep("DEF")
+
+ # Both hashes are present
+ self.assertClientGetHash(self.client, taskhash2, unihash2)
+ self.assertClientGetHash(self.client, taskhash, unihash)
+
+ def test_gc_new_hashes(self):
+ taskhash = '53b8dce672cb6d0c73170be43f540460bfc347b4'
+ outhash = '5a9cb1649625f0bf41fc7791b635cd9c2d7118c7f021ba87dcd03f72b67ce7a8'
+ unihash = 'f37918cc02eb5a520b1aff86faacbc0a38124646'
+
+ result = self.client.report_unihash(taskhash, self.METHOD, outhash, unihash)
+ self.assertEqual(result['unihash'], unihash, 'Server returned bad unihash')
+
+ # Start a new garbage collection
+ ret = self.client.gc_mark("ABC", {"unihash": unihash, "method": self.METHOD})
+ self.assertEqual(ret, {"count": 1})
+
+ ret = self.client.gc_status()
+ self.assertEqual(ret, {"mark": "ABC", "keep": 1, "remove": 0})
+
+ # Add second hash. It should inherit the mark from the current garbage
+ # collection operation
+
+ taskhash2 = '3bf6f1e89d26205aec90da04854fbdbf73afe6b4'
+ outhash2 = '77623a549b5b1a31e3732dfa8fe61d7ce5d44b3370f253c5360e136b852967b4'
+ unihash2 = 'af36b199320e611fbb16f1f277d3ee1d619ca58b'
+
+ result = self.client.report_unihash(taskhash2, self.METHOD, outhash2, unihash2)
+ self.assertClientGetHash(self.client, taskhash2, unihash2)
+
+ # Sweep should remove nothing
+ ret = self.client.gc_sweep("ABC")
+ self.assertEqual(ret, {"count": 0})
+
+ # Both hashes are present
+ self.assertClientGetHash(self.client, taskhash2, unihash2)
+ self.assertClientGetHash(self.client, taskhash, unihash)
+
class TestHashEquivalenceClient(HashEquivalenceTestSetup, unittest.TestCase):
def get_server_addr(self, server_idx):
@@ -869,6 +1119,40 @@
def test_stress(self):
self.run_hashclient(["--address", self.server_address, "stress"], check=True)
+ def test_unihash_exsits(self):
+ taskhash, outhash, unihash = self.create_test_hash(self.client)
+
+ p = self.run_hashclient([
+ "--address", self.server_address,
+ "unihash-exists", unihash,
+ ], check=True)
+ self.assertEqual(p.stdout.strip(), "true")
+
+ p = self.run_hashclient([
+ "--address", self.server_address,
+ "unihash-exists", '6662e699d6e3d894b24408ff9a4031ef9b038ee8',
+ ], check=True)
+ self.assertEqual(p.stdout.strip(), "false")
+
+ def test_unihash_exsits_quiet(self):
+ taskhash, outhash, unihash = self.create_test_hash(self.client)
+
+ p = self.run_hashclient([
+ "--address", self.server_address,
+ "unihash-exists", unihash,
+ "--quiet",
+ ])
+ self.assertEqual(p.returncode, 0)
+ self.assertEqual(p.stdout.strip(), "")
+
+ p = self.run_hashclient([
+ "--address", self.server_address,
+ "unihash-exists", '6662e699d6e3d894b24408ff9a4031ef9b038ee8',
+ "--quiet",
+ ])
+ self.assertEqual(p.returncode, 1)
+ self.assertEqual(p.stdout.strip(), "")
+
def test_remove_taskhash(self):
taskhash, outhash, unihash = self.create_test_hash(self.client)
self.run_hashclient([
@@ -1086,6 +1370,42 @@
"get-db-query-columns",
], check=True)
+ def test_gc(self):
+ taskhash = '53b8dce672cb6d0c73170be43f540460bfc347b4'
+ outhash = '5a9cb1649625f0bf41fc7791b635cd9c2d7118c7f021ba87dcd03f72b67ce7a8'
+ unihash = 'f37918cc02eb5a520b1aff86faacbc0a38124646'
+
+ result = self.client.report_unihash(taskhash, self.METHOD, outhash, unihash)
+ self.assertEqual(result['unihash'], unihash, 'Server returned bad unihash')
+
+ taskhash2 = '3bf6f1e89d26205aec90da04854fbdbf73afe6b4'
+ outhash2 = '77623a549b5b1a31e3732dfa8fe61d7ce5d44b3370f253c5360e136b852967b4'
+ unihash2 = 'af36b199320e611fbb16f1f277d3ee1d619ca58b'
+
+ result = self.client.report_unihash(taskhash2, self.METHOD, outhash2, unihash2)
+ self.assertClientGetHash(self.client, taskhash2, unihash2)
+
+ # Mark the first unihash to be kept
+ self.run_hashclient([
+ "--address", self.server_address,
+ "gc-mark", "ABC",
+ "--where", "unihash", unihash,
+ "--where", "method", self.METHOD
+ ], check=True)
+
+ # Second hash is still there; mark doesn't delete hashes
+ self.assertClientGetHash(self.client, taskhash2, unihash2)
+
+ self.run_hashclient([
+ "--address", self.server_address,
+ "gc-sweep", "ABC",
+ ], check=True)
+
+ # Hash is gone. Taskhash is returned for second hash
+ self.assertClientGetHash(self.client, taskhash2, None)
+ # First hash is still present
+ self.assertClientGetHash(self.client, taskhash, unihash)
+
class TestHashEquivalenceUnixServer(HashEquivalenceTestSetup, HashEquivalenceCommonTests, unittest.TestCase):
def get_server_addr(self, server_idx):