master: subtree updates oct 12 2023
poky: e444d2bed0..8d0ba08aa6:
Alassane Yattara (1):
bitbake: toaster: Monitoring - implement Django logging system
Alexander Kanavin (1):
nghttp2: update 1.56.0 -> 1.57.0
Alexis Lothoré (2):
oeqa/utils/gitarchive: fix tag pattern searching
oeqa/utils/gitarchive: ensure tag matches regex before getting its fields
Andreas Cord-Landwehr (1):
wayland: consider pkgconfig sysroot for pkgdatadir
André Draszik (1):
wic: fix a typo in help output
Archana Polampalli (1):
gstreamer: upgrade 1.22.5 -> 1.22.6
Bruce Ashfield (1):
linux-yocto/6.5: integrate fixes for sanity issues
Chris Laplante (1):
recipetool/create_buildsys_python: use importlib instead of imp
Daniel McGregor (2):
meson: upgrade 1.2.1 -> 1.2.2
libtirpc: conditionally enable gssapi
Daniel Semkowicz (3):
uboot-extlinux-config.bbclass: Remove repeated space character
uboot-extlinux-config.bbclass: Uppercase "menu title" entry
uboot-extlinux-config.bbclass: Add menu title configuration
Fabio Estevam (1):
u-boot: Upgrade to 2023.10
Fahad Arslan (1):
linux-firmware: create separate package for cirrus and cnm firmwares
Jermain Horsman (1):
scripts/oe-setup-layers: Update how to determine if directory is git repo
Jose Quaresma (4):
curl: 8.3.0 -> 8.4.0
go: update 1.20.7 -> 1.20.8
go: update 1.20.8 -> 1.20.9
go: update 1.20.9 -> 1.20.10
Joshua Watt (6):
bitbake: hashserv: Add remove API
bitbake: bitbake-hashclient: Add remove subcommand
bitbake: hashserv: Extend get_outhash API to optionally include unihash
bitbake: hashserv: Add API to clean unused entries
bitbake: bitbake-hashclient: Add clean-unused subcommand
overview: Add note about non-reproducibility side effects
Julien Stephan (4):
bitbake.conf: include bblock.conf
sstatesig: add a new info level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK
scripts/bblock: add a script to lock/unlock recipes
oeqa/selftest/bblock: add self test for bblock tool
Khem Raj (1):
python3-docutils: Rename utilities to their canonical names
Lee Chee Yang (1):
migration-guides: add release notes for 4.0.13
Marcus Flyckt (1):
devtool/upgrade: check all git config locations
Markus Volk (2):
mesa: Upgrade 23.1.8 -> 23.2.1
gtk: Upgrade 4.12.1 -> 4.12.3
Marlon Rodriguez Garcia (1):
bitbake: toaster: update selenium version and code syntax
Martijn de Gouw (1):
busybox: Set PATH in syslog initscript
Michael Opdenacker (1):
ref-manual: releases.svg: Scarthgap is now version 5.0
Mikko Rapeli (2):
qemurunner.py: detect login prompt without without utf-8 conversion
openssh: drop sudo from ptest dependencies
Peter Kjellerstedt (4):
externalsrc.bbclass: Support specifying patterns in CONFIGURE_FILES
autotools.bbclass: Add *.m4 to CONFIGURE_FILES
packages.bbclass: Correct the check for conflicts with renamed packages
cmake.bbclass: Add *.cmake to CONFIGURE_FILES
Quentin Schulz (1):
uboot-extlinux-config.bbclass: fix missed override syntax migration
Randy MacLeod (1):
strace: skip so_peerpidfd test
Rasmus Villemoes (1):
openssh: update sshd_check_keys script to make use of 'sshd -G'
Richard Purdie (14):
wic: Add console parameters to qemux86 canned-wks
qemurunner: Skip conversion from/to binary data for logfile
qemurunner: Use backslashreplace with utf8 to make invalid characters clear
qemurunner: Log the second serial console as well as the first
qemurunner: Show both the login console log and all logging upon failure
oeqa/concurrencytest: Remove invalid buffering option
bitbake: selftest/fetch: Ensure top level directory timestamp doesn't break test
cryptodev: Update to latest git for 6.5 kernel fixes
qemux86/qemuarm: Drop kernel version overrides
oeqa/qemurunner: Add newlines serial workaround
runqemu/qemurunner: Use nodelay with tcp serial connections
oeqa/qemurunner: Add extra logging when console doesn't appear
poky/poky-tiny: Switch to the 6.5 kernel
oeqa/qemurunner: Ensure we retry after BrokenPipeError
Ross Burton (12):
python3-numpy: remove obsolete reproducible workaround
libx11: upgrade to 1.8.7
libxpm: upgrade to 3.5.17
qemuboot: reduce default size of software I/O translation buffer
sysvinit-initab: rewrite loop to generate inittab
ttyrun: add new recipe
sysvinit-inittab: use ttyrun to run getty only if the terminal exists
busybox: use ttyrun to run getty only if the terminal exists
oeqa/selftest: don't skip test_read_only_image on qemuarm64
meta/conf/machine: remove SERIAL_CONSOLES_CHECK
busybox-inittab: fix console handling
oeqa/runtime/_qemutiny: rewrite test to be functional
Sundeep KOKKONDA (1):
rust: reproducibility issue fix
Trevor Gamblin (1):
dev-manual: fix testimage usage instructions
Yogita Urade (1):
qemu: fix CVE-2023-42467
meta-openembedded: ea42cec2ec..62039a2c33:
Ahmad Fatoum (1):
signing.bbclass: don't export OPENSSL environment variables globally
Beniamin Sandu (1):
libnet: upgrade version v1.2 -> v1.3
Benjamin Bara (1):
libvpx: upgrade 1.13.0 -> 1.13.1
Chen Qi (1):
libblockdev: fix QA error in case of multilib
Christophe Vu-Brugier (3):
libnvme: upgrade 1.5 -> 1.6
nvme-cli: upgrade 2.5 -> 2.6
libnvme: apply patch already upstream to fix build with musl
Clément Péron (1):
Revert "protobuf: stage protoc binary to sysroot"
Daniel Klauer (1):
graphviz: Fix build to not use $prefix as search dir
Denys Zagorui (1):
libbpf: add arm, powerpc and mips64 to COMPATIBLE_HOST
Fabien Thomas (8):
meta-filesystems/layer.conf : Add meta-networking dependency
Add static-passwd and static-group files
Add static-passwd and static-group files
Add static-passwd and static-group files
Add static-passwd and static-group files
Add static-passwd and static-group files
Add static-passwd and static-group files
Add static-passwd and static-group files
Gianfranco Costamagna (6):
dlt-daemon: Make it work without systemd
dlt-daemon: Enable experimental coredumphandler feature
dlt-daemon: update patch 544.patch
dlt-daemon: do not disable dlt-system build when systemd is set to off
dlt-daemon: Add an additional fix for non-systemd builds
cpprestsdk: fix typo in comment, tag is actually 2.0.18
Jeffrey Pautler (1):
bolt: change product name used for CVE checking
Joe Slater (1):
nginx: add configure option
Johannes Kauffmann (1):
open62541: add Backport status and link to patch
Jörg Sommer (1):
collectd: Use https in SRC_URI, add HOMEPAGE
Khem Raj (16):
python3-pyroute2: Add missing dependency on sqlite3 for ptests
python3-pylint: Upgrade to 3.0.0
python3-lz4: use python3-unittest-automake-output
minicoredumber: Fix ptest reporting
images: Inherit from core-image-base
images: Delete layer specific base images
images: Rename <layer>-image to <layer>-image-all
images: Rename ptest images to rhyme with oe-core ptest images
ptest-image: Switch to using core-image-minimal
stressapptest: Upgrade to 1.0.11 release
klibc: Upgrade to 2.0.13 release
libnvme: Fix test builds on musl
kernel-selftest: Build bpf tests again
ptest-packagelists-meta-oe: Add kernel-selftest to x86/x86-64 images
kernel-selftest: Copy the .config from kernel build
kernel-selftest: Use clang options when clang is available
Markus Volk (3):
pugixml: Update 1.13 -> 1.14
pipewire: Upgrade 0.3.80 -> 0.3.81
gnome-control-center: Fix polkit gettext issue
Martin Jansa (1):
opencv: Fix build with protobuf v22 and dnn enabled
Mickael RAMILISON (1):
python3-rapidjson: add ptest
Pawel Langowski (1):
recipes-connectivity: Add tayga recipe
Philip-Dylan Gleonec (1):
cukinia: Fix license field
Thomas Roos (1):
python3-boto3, python3-botocore: remove recipes
Tom Hochstein (1):
libcamera: Avoid build break in signature recalculation
meta-arm: 95789365f7..e914891eee:
Jon Mason (1):
arm-bsp/u-boot: add recipe for 2023.07.02
meta-security: aca6d4a9e7..3f7d40b0fc:
Rasmus Villemoes (3):
fail2ban: add systemd support
fail2ban: change sqlite3 dependency to python3-sqlite3
fail2ban: add useful recommendations
Change-Id: I93672642f4e0392adc6223fdc4e073910b817bc9
Signed-off-by: Andrew Geissler <geissonator@yahoo.com>
diff --git a/poky/bitbake/lib/hashserv/client.py b/poky/bitbake/lib/hashserv/client.py
index b2aa102..f676d26 100644
--- a/poky/bitbake/lib/hashserv/client.py
+++ b/poky/bitbake/lib/hashserv/client.py
@@ -83,10 +83,10 @@
{"get": {"taskhash": taskhash, "method": method, "all": all_properties}}
)
- async def get_outhash(self, method, outhash, taskhash):
+ async def get_outhash(self, method, outhash, taskhash, with_unihash=True):
await self._set_mode(self.MODE_NORMAL)
return await self.send_message(
- {"get-outhash": {"outhash": outhash, "taskhash": taskhash, "method": method}}
+ {"get-outhash": {"outhash": outhash, "taskhash": taskhash, "method": method, "with_unihash": with_unihash}}
)
async def get_stats(self):
@@ -101,6 +101,14 @@
await self._set_mode(self.MODE_NORMAL)
return (await self.send_message({"backfill-wait": None}))["tasks"]
+ async def remove(self, where):
+ await self._set_mode(self.MODE_NORMAL)
+ return await self.send_message({"remove": {"where": where}})
+
+ async def clean_unused(self, max_age):
+ await self._set_mode(self.MODE_NORMAL)
+ return await self.send_message({"clean-unused": {"max_age_seconds": max_age}})
+
class Client(bb.asyncrpc.Client):
def __init__(self):
@@ -115,6 +123,8 @@
"get_stats",
"reset_stats",
"backfill_wait",
+ "remove",
+ "clean_unused",
)
def _get_async_client(self):
diff --git a/poky/bitbake/lib/hashserv/server.py b/poky/bitbake/lib/hashserv/server.py
index d40a2ab..45bf476 100644
--- a/poky/bitbake/lib/hashserv/server.py
+++ b/poky/bitbake/lib/hashserv/server.py
@@ -4,7 +4,7 @@
#
from contextlib import closing, contextmanager
-from datetime import datetime
+from datetime import datetime, timedelta
import enum
import asyncio
import logging
@@ -186,6 +186,8 @@
'report-equiv': self.handle_equivreport,
'reset-stats': self.handle_reset_stats,
'backfill-wait': self.handle_backfill_wait,
+ 'remove': self.handle_remove,
+ 'clean-unused': self.handle_clean_unused,
})
def validate_proto_version(self):
@@ -269,27 +271,42 @@
method = request['method']
outhash = request['outhash']
taskhash = request['taskhash']
+ with_unihash = request.get("with_unihash", True)
with closing(self.db.cursor()) as cursor:
- d = await self.get_outhash(cursor, method, outhash, taskhash)
+ d = await self.get_outhash(cursor, method, outhash, taskhash, with_unihash)
self.write_message(d)
- async def get_outhash(self, cursor, method, outhash, taskhash):
+ async def get_outhash(self, cursor, method, outhash, taskhash, with_unihash=True):
d = None
- cursor.execute(
- '''
- SELECT *, unihashes_v2.unihash AS unihash FROM outhashes_v2
- INNER JOIN unihashes_v2 ON unihashes_v2.method=outhashes_v2.method AND unihashes_v2.taskhash=outhashes_v2.taskhash
- WHERE outhashes_v2.method=:method AND outhashes_v2.outhash=:outhash
- ORDER BY outhashes_v2.created ASC
- LIMIT 1
- ''',
- {
- 'method': method,
- 'outhash': outhash,
- }
- )
+ if with_unihash:
+ cursor.execute(
+ '''
+ SELECT *, unihashes_v2.unihash AS unihash FROM outhashes_v2
+ INNER JOIN unihashes_v2 ON unihashes_v2.method=outhashes_v2.method AND unihashes_v2.taskhash=outhashes_v2.taskhash
+ WHERE outhashes_v2.method=:method AND outhashes_v2.outhash=:outhash
+ ORDER BY outhashes_v2.created ASC
+ LIMIT 1
+ ''',
+ {
+ 'method': method,
+ 'outhash': outhash,
+ }
+ )
+ else:
+ cursor.execute(
+ """
+ SELECT * FROM outhashes_v2
+ WHERE outhashes_v2.method=:method AND outhashes_v2.outhash=:outhash
+ ORDER BY outhashes_v2.created ASC
+ LIMIT 1
+ """,
+ {
+ 'method': method,
+ 'outhash': outhash,
+ }
+ )
row = cursor.fetchone()
if row is not None:
@@ -499,6 +516,50 @@
await self.backfill_queue.join()
self.write_message(d)
+ async def handle_remove(self, request):
+ condition = request["where"]
+ if not isinstance(condition, dict):
+ raise TypeError("Bad condition type %s" % type(condition))
+
+ def do_remove(columns, table_name, cursor):
+ nonlocal condition
+ where = {}
+ for c in columns:
+ if c in condition and condition[c] is not None:
+ where[c] = condition[c]
+
+ if where:
+ query = ('DELETE FROM %s WHERE ' % table_name) + ' AND '.join("%s=:%s" % (k, k) for k in where.keys())
+ cursor.execute(query, where)
+ return cursor.rowcount
+
+ return 0
+
+ count = 0
+ with closing(self.db.cursor()) as cursor:
+ count += do_remove(OUTHASH_TABLE_COLUMNS, "outhashes_v2", cursor)
+ count += do_remove(UNIHASH_TABLE_COLUMNS, "unihashes_v2", cursor)
+ self.db.commit()
+
+ self.write_message({"count": count})
+
+ async def handle_clean_unused(self, request):
+ max_age = request["max_age_seconds"]
+ with closing(self.db.cursor()) as cursor:
+ cursor.execute(
+ """
+ DELETE FROM outhashes_v2 WHERE created<:oldest AND NOT EXISTS (
+ SELECT unihashes_v2.id FROM unihashes_v2 WHERE unihashes_v2.method=outhashes_v2.method AND unihashes_v2.taskhash=outhashes_v2.taskhash LIMIT 1
+ )
+ """,
+ {
+ "oldest": datetime.now() - timedelta(seconds=-max_age)
+ }
+ )
+ count = cursor.rowcount
+
+ self.write_message({"count": count})
+
def query_equivalent(self, cursor, method, taskhash):
# This is part of the inner loop and must be as fast as possible
cursor.execute(
diff --git a/poky/bitbake/lib/hashserv/tests.py b/poky/bitbake/lib/hashserv/tests.py
index f6b85ae..f343c58 100644
--- a/poky/bitbake/lib/hashserv/tests.py
+++ b/poky/bitbake/lib/hashserv/tests.py
@@ -84,6 +84,7 @@
result = self.client.report_unihash(taskhash, self.METHOD, outhash, unihash)
self.assertEqual(result['unihash'], unihash, 'Server returned bad unihash')
+ return taskhash, outhash, unihash
def test_create_equivalent(self):
# Tests that a second reported task with the same outhash will be
@@ -125,6 +126,57 @@
self.assertClientGetHash(self.client, taskhash, unihash)
+ def test_remove_taskhash(self):
+ taskhash, outhash, unihash = self.test_create_hash()
+ result = self.client.remove({"taskhash": taskhash})
+ self.assertGreater(result["count"], 0)
+ self.assertClientGetHash(self.client, taskhash, None)
+
+ result_outhash = self.client.get_outhash(self.METHOD, outhash, taskhash)
+ self.assertIsNone(result_outhash)
+
+ def test_remove_unihash(self):
+ taskhash, outhash, unihash = self.test_create_hash()
+ result = self.client.remove({"unihash": unihash})
+ self.assertGreater(result["count"], 0)
+ self.assertClientGetHash(self.client, taskhash, None)
+
+ def test_remove_outhash(self):
+ taskhash, outhash, unihash = self.test_create_hash()
+ result = self.client.remove({"outhash": outhash})
+ self.assertGreater(result["count"], 0)
+
+ result_outhash = self.client.get_outhash(self.METHOD, outhash, taskhash)
+ self.assertIsNone(result_outhash)
+
+ def test_remove_method(self):
+ taskhash, outhash, unihash = self.test_create_hash()
+ result = self.client.remove({"method": self.METHOD})
+ self.assertGreater(result["count"], 0)
+ self.assertClientGetHash(self.client, taskhash, None)
+
+ result_outhash = self.client.get_outhash(self.METHOD, outhash, taskhash)
+ self.assertIsNone(result_outhash)
+
+ def test_clean_unused(self):
+ taskhash, outhash, unihash = self.test_create_hash()
+
+ # Clean the database, which should not remove anything because all hashes an in-use
+ result = self.client.clean_unused(0)
+ self.assertEqual(result["count"], 0)
+ self.assertClientGetHash(self.client, taskhash, unihash)
+
+ # Remove the unihash. The row in the outhash table should still be present
+ self.client.remove({"unihash": unihash})
+ result_outhash = self.client.get_outhash(self.METHOD, outhash, taskhash, False)
+ self.assertIsNotNone(result_outhash)
+
+ # Now clean with no minimum age which will remove the outhash
+ result = self.client.clean_unused(0)
+ self.assertEqual(result["count"], 1)
+ result_outhash = self.client.get_outhash(self.METHOD, outhash, taskhash, False)
+ self.assertIsNone(result_outhash)
+
def test_huge_message(self):
# Simple test that hashes can be created
taskhash = 'c665584ee6817aa99edfc77a44dd853828279370'