subtree updates
meta-arm: 025f76a14f..aba9250494:
Anusmita Dutta Mazumder (2):
arm-bsp/linux-yocto: Remove EOL Linux yocto kernel 6.1
arm-bsp/n1sdp: update to linux yocto kernel 6.6
Bence Balogh (1):
arm-bsp/trusted-firmware-m: disable libmetal doc generation
Drew Reed (5):
meta-arm: Support firmware building under a multiconfig
bsp,ci: Build Corstone-1000 firmware under multiconfig
bsp: Restore the ability to build firmware only
ci: Add back testing of firmware only builds
ci: Ensure tests are in the Corstone-1000 flash image
meta-raspberrypi: dbf1113a82..95a9103f91:
Khem Raj (1):
python3-sense-hat: Drop PYTHON_PN
Martin Jansa (2):
sdcard_image-rpi.bbclass: include ${IMAGE_NAME_SUFFIX} directly in both ${IMAGE_NAME} and ${IMAGE_LINK_NAME}
sdimage-raspberrypi.wks: increase /boot partition minimal size from 20 to 100
meta-openembedded: 528f273006..9f0e513211:
Andreas Mützel (1):
python3-pynacl: allow -native build
Chen Qi (1):
unixodbc: fix odbc.pc file generation
Daniel Ammann (1):
sdmon: add new package
Derek Straka (9):
python3-trustme: add runtime dependency for tests and re-add to ptest
python3-gunicorn: re-enable working ptests for the package
python-inotify: re-enable working ptests for the package
python3-license-expression: re-enable passing ptests for the package
python3-jdcal: re-add functional ptests
python3-msgpack: re-add functional ptests
python3-parse: re-add functional ptests
python3-typeguard: update ptest dependencies and re-enable functional tests
python3-service-identity: add missing ptest dependencies and re-enable functional tests
Jan Vermaete (1):
netdata: version bump 1.43.2 -> 1.44.3
Joerg Hofrichter (1):
python3-gevent: adding missing dependency to python3-zopeevent
Khawaja Shaheryar (2):
libdaq: add recipe
snort: add snort3 initial recipe
Khem Raj (25):
python3-pocketsphinx: Upgrade to 5.0.3
snort: Do not use llvm libunwind
snort3: Fix contains reference to TMPDIR [buildpaths] warnings
libcamera: Replace VLAs with alloca
dav1d: Inherit missing pkgconfig
webkitgtk3: Fix build on 32bit x86
ptest-packagelists-meta-oe: Remove oprofile for rv32/rv64
python3-jsmin: Fix ptests to run with python 3.12+
python3-ordered-set: Use automake formatter for ptest output
fuse3: Add missing runtime deps for ptests
python3-looseversion: Add recipe
sshfs-fuse: Fix ptest builds with python 3.12
meta-filesystems: Add meta-filesystems-image-ptest
meta-multimedia-image-ptest: Add images to enable BBCLASSEXTEND parallel execution
meta-networking-image-ptest: Add images to enable BBCLASSEXTEND parallel execution
python3-scapy: Add missing rdeps for ptests
ptest-packagelists-meta-oe.inc: Remove oprofile from PTESTS_PROBLEMS_META_OE
ptest-packagelists-meta-networking: firewalld hangs therefore disabled
ptest-packagelists-meta-perl.inc: Move couple of test to PTESTS_FAST_META_PERL
openhpi: Fix ptest run time failures
squid: Add missing bash dependency for ptest package
meta-networking: Express dependency on meta-python
ostree: Remove strace from ptest rdeps
python3-pydantic-core,python3-pydantic: Update to 2.16.3 and 2.6.3 respectively
python3-pydantic-core: Fix build for arches without 64bit atomics
Lei Maohui (1):
Fix install error when enable multilib.
Markus Volk (7):
iwd: update 2.13 -> 2.14
libgedit-gtksourceview: update 299.0.5 -> 299.1.0
gedit: update 46.1 -> 46.2
mutter: update 45.3 -> 45.4
gnome-shell: update 45.3 -> 45.4
gnome-control-center: update 45.2 -> 45.3
dav1d: update 1.3.0 -> 1.4.0
Martin Jansa (5):
python3-httpx: respect libdir in packaging
snort3: drop SRCPV from PV
snort3: fix snort.pc
gattlib: use python3native and depend on python3-packaging-native
networkmanager-fortisslvpn: use python3native and depend on python3-packaging-native
Mingli Yu (1):
mariadb: Upgrade to 10.11.7
Niko Mauno (2):
python3-pybind11: Migrate to python_setuptools_build_meta
python3-pybind11: Restore strip prevention patch
Oleh Matiusha (1):
yasm: improve reproducibility
Peter Marko (1):
dnsmasq: Upgrade 2.89 -> 2.90
Romain Naour (1):
wavemon: add recipe for version 0.9.5
Sascha Hauer (1):
signing.bbclass: fix wrong function name
Tim Orling (16):
python_mesonpy.bbclass: move to oe-core
python3-meson-python: move to oe-core
python3-pyproject-metadata: move to oe-core
meta-python: drop ${PYTHON_PN}
meta-oe: drop ${PYTHON_PN}
meta-filesystems: drop ${PYTHON_PN}
meta-networking: drop ${PYTHON_PN}
meta-gnome: drop ${PYTHON_PN}
python3-pytest-lazy-fixtures: add 1.0.5
python3-prettytable: upgrade 3.9.0 => 3.10.0; fix ptests
python3-pytest-lazy-fixture: drop recipe
meta-oe-image-ptest: add PTESTS_PROBLEMS_META_OE
meta-perl-image-ptest: add PTESTS_PROBLEMS_META_PERL
meta-python-image-ptest: add PTESTS_PROBLEMS_META_PYTHON
libencode-perl: drop recipe
libencode-locale-perl: drop recipe
Wang Mingyu (49):
babl: upgrade 0.1.106 -> 0.1.108
btop: upgrade 1.3.0 -> 1.3.2
gegl: upgrade 0.4.46 -> 0.4.48
gjs: upgrade 1.78.3 -> 1.78.4
gnome-bluetooth: upgrade 42.7 -> 42.8
gnome-keyring: upgrade 42.1 -> 46.1
isomd5sum: upgrade 1.2.3 -> 1.2.4
libei: upgrade 1.2.0 -> 1.2.1
libmanette: upgrade 0.2.6 -> 0.2.7
libmime-types-perl: upgrade 2.24 -> 2.26
logwatch: upgrade 7.9 -> 7.10
mpich: upgrade 4.1.2 -> 4.2.0
ostree: upgrade 2024.1 -> 2024.3
python3-aiohue: upgrade 4.7.0 -> 4.7.1
python3-awesomeversion: upgrade 23.11.0 -> 24.2.0
python3-bidict: upgrade 0.22.1 -> 0.23.0
python3-cantools: upgrade 39.4.3 -> 39.4.4
python3-cmake: upgrade 3.28.1 -> 3.28.3
python3-django: upgrade 5.0.1 -> 5.0.2
python3-dnspython: upgrade 2.5.0 -> 2.6.0
python3-elementpath: upgrade 4.2.0 -> 4.3.0
python3-engineio: upgrade 4.8.2 -> 4.9.0
python3-gevent: upgrade 23.9.1 -> 24.2.1
unbound: upgrade 1.19.0 -> 1.19.1
wireshark: upgrade 4.2.2 -> 4.2.3
protobuf: upgrade 4.25.2 -> 4.25.3
webkitgtk3: upgrade 2.42.4 -> 2.42.5
python3-tqdm: upgrade 4.66.1 -> 4.66.2
python3-google-api-python-client: upgrade 2.116.0 -> 2.118.0
python3-httpcore: upgrade 1.0.2 -> 1.0.3
python3-jsbeautifier: upgrade 1.14.11 -> 1.15.1
python3-langtable: upgrade 0.0.64 -> 0.0.65
python3-polyline: upgrade 2.0.1 -> 2.0.2
python3-protobuf: upgrade 4.25.2 -> 4.25.3
python3-pymisp: upgrade 2.4.184 -> 2.4.185
python3-pymodbus: upgrade 3.6.3 -> 3.6.4
python3-pytest-asyncio: upgrade 0.23.4 -> 0.23.5
python3-tox: upgrade 4.12.1 -> 4.13.0
python3-twine: upgrade 4.0.2 -> 5.0.0
python3-watchdog: upgrade 3.0.0 -> 4.0.0
python3-zopeinterface: upgrade 6.1 -> 6.2
remmina: upgrade 1.4.33 -> 1.4.34
sip: upgrade 6.8.2 -> 6.8.3
python3-google-auth: upgrade 2.27.0 -> 2.28.0
python3-gspread: upgrade 6.0.1 -> 6.0.2
python3-socketio: upgrade 5.11.0 -> 5.11.1
python3-sentry-sdk: upgrade 1.40.0 -> 1.40.4
python3-pydantic-core: upgrade 2.14.6 -> 2.16.1
python3-pydantic: upgrade 2.5.3 -> 2.6.0
William Lyu (1):
e2tools: Add ptest
Yi Zhao (1):
audit: upgrade 3.1.2 -> 4.0
Yoann Congal (2):
influxdb: Fix /etc files owner
influxdb: Add missing group to static id
chenheyun (1):
dropwatch: Use header files from sysroot instead of build host
poky: fc8e5d7c13..25d60ac6f6:
Adrian Freihofer (5):
devtool: ide-sdk python 3.12 escaping
sdk-manual: extensible.rst: cover devtool ide-sdk
devtool: ide-sdk launch.json per recipe only
devtool: ide-sdk prefer sources from workspace
oe-selftest devtool: ide-sdk tests
Alexander Kanavin (1):
dbus: disable assertions and enable only modular tests
Alexis Lothoré (7):
testimage: log exception when failing to retrieve artifacts
lib/oeqa: share get_json_result_dir helper
testimage: create a list of failed test post actions
oeqa/utils/postactions: isolate directory creation in dedicated action
oeqa/utils/postactions: add target disk usage stat as post action
oeqa/utils/postactions: testimage: add host disk usage stat as post action
oeqa/lib/utils/postactions: fix host disk usage stats retrieval
Bruce Ashfield (8):
linux-yocto/6.6: update to v6.6.17
linux-yocto/6.6: update CVE exclusions
linux-yocto/6.6: enable squashfs for selftests
linux-yocto/6.6: config: x86 tidy & consolidation
kern-tools: depend on git-replacement-native
linux-yocto/6.6: genericarm64 configuration/definition
linux-yocto/6.6: update to v6.6.18
linux-yocto/6.6: update CVE exclusions
Christoph Vogtländer (1):
overlayfs: add missing vardeps
Claus Stovgaard (1):
wpa-supplicant: Fix CVE-2023-52160
Eilís 'pidge' Ní Fhlannagáin (2):
creategroup*: Remove coreutils-native as a DEPENDS
selftest-users: Convoluted selftest for USERADD_DEPENDS
Emil Kronborg (1):
bluez5: remove configuration files from install task
Enguerrand de Ribaucourt (4):
devtool: ide: define compilerPath for meson projects
Revert "meson: use absolute cross-compiler paths"
bitbake: bitbake: progressbar: accept value over initial maxval
devtool: ide-sdk source mapping for vscode
Enrico Jörns (1):
wic: 'empty' plugin: fix typo in comment
Joe Slater (1):
qemuboot: predictable network interface names
Jonathan GUILLOT (2):
lib/oe/package: fix LOCALE_PATHS scan to create locale packages
glibc-locale: add an explicit dedicated package for locale.alias file
Jose Quaresma (1):
go: update 1.20.13 -> 1.20.14
Joshua Watt (1):
bitbake: asyncrpc: Add support for server headers
Khem Raj (6):
ncurses: Always pass -D_GNU_SOURCE
linux-yocto: Remove unused patch
ref-manual: variables: remove PYTHON_PN
python3-bcrypt: Fix build break on arches without 64 bit atomics
python3-maturin: Recognise riscv32 architecture
llvm: Update to 18.1.0 RC4
Lee Chee Yang (1):
migration-guide: add release notes for 4.3.3
Lei Maohui (1):
rpm: Fix the following error when run nativesdk-rpm in nativesdk environment.
Martin Jansa (1):
glib-2.0: backport a switch from distutils to packaging in codegen
Michael Halstead (1):
yocto-uninative: Update to 4.4 for glibc 2.39
Michael Opdenacker (5):
ref-manual: system-requirements: update packages to build docs
ref-manual: release-process: grammar fix
manuals: suppress excess use of "following" word
dev-manual: packages: clarify shared PR service constraint
dev-manual: packages: need enough free space
Munehisa Kamata (1):
kernel.bbclass: Set pkg-config variables for building modules
Nick Owens (1):
python3: dont disable readline module for editline
Philip Lorenz (1):
bitbake: fetch2: Ensure that git LFS objects are available
Piotr Łobacz (1):
useradd.bbclass: Fix order of postinst-useradd-*
Richard Purdie (6):
numactl: Upgrade 2.0.17 -> 2.0.18
lttng-ust: Upgrade 2.13.6 -> 2.13.7
oeqa/selftest/rust: Simplify the rust testsuite output gathering/processing
recipetool: Fix errors with meta-poky bbappend
bitbake: runqueue: Add support for BB_LOADFACTOR_MAX
mirrors: Switch llvm to use shallow cloning
Ross Burton (4):
base-files: add usage warning to motd
libexif: remove unused version_underscore
gstreamer1.0: skip a test that is known to be flaky
linux-firmware: split out more firmware pieces
Simone Weiß (6):
patchtest: provide further guidance for failed testcases
patchtest: Skip test for CVE_CHECK_IGNORE for older branches
meta: Remove some not needed CVE_STATUS
meta: Update CVE_STATUS for incorrect cpes
cve-check: Log if CVE_STATUS set but not reported for component
dev-manual: Rephrase spdx creation
Soumya Sambu (1):
bind: Upgrade 9.18.21 -> 9.18.24
Tim Orling (3):
bitbake: layerindexlib: fix missing layer branch backtrace
python3-cryptography{-vectors}: upgrade to 42.0.5
python3-attrs: disable Hypothesis deadline
Tobias Hagelborn (1):
bitbake: hashserv: Re-enable connection pooling with psycopg 3 driver
Trevor Gamblin (1):
python3-git: upgrade 3.1.41 -> 3.1.42
Trevor Woerner (1):
wic: allow imager-specific filename extensions
Ulrich Ölmann (1):
bitbake: taskexp_ncurses: fix execution example in introductory comment
Wang Mingyu (44):
bash-completion: upgrade 2.11 -> 2.12.0
ccache: upgrade 4.9 -> 4.9.1
createrepo-c: upgrade 1.0.3 -> 1.0.4
ed: upgrade 1.20 -> 1.20.1
efivar: upgrade 38 -> 39
gcr: upgrade 4.1.0 -> 4.2.0
git: upgrade 2.43.0 -> 2.44.0
libffi: upgrade 3.4.5 -> 3.4.6
libgpg-error: upgrade 1.47 -> 1.48
libhandy: upgrade 1.8.2 -> 1.8.3
libksba: upgrade 1.6.5 -> 1.6.6
libmicrohttpd: upgrade 0.9.77 -> 1.0.1
libpng: upgrade 1.6.41 -> 1.6.42
libsecret: upgrade 0.21.2 -> 0.21.4
libunistring: upgrade 1.1 -> 1.2
liburi-perl: upgrade 5.25 -> 5.27
libxext: upgrade 1.3.5 -> 1.3.6
libxkbfile: upgrade 1.1.2 -> 1.1.3
libxvmc: upgrade 1.0.13 -> 1.0.14
lighttpd: upgrade 1.4.73 -> 1.4.74
makedepend: upgrade 1.0.8 -> 1.0.9
mpg123: upgrade 1.32.4 -> 1.32.5
ofono: upgrade 2.3 -> 2.4
pango: upgrade 1.51.0 -> 1.52.0
pciutils: upgrade 3.10.0 -> 3.11.1
pkgconf: upgrade 2.1.0 -> 2.1.1
python3-beartype: upgrade 0.17.0 -> 0.17.2
python3-certifi: upgrade 2023.11.17 -> 2024.2.2
python3-dbusmock: upgrade 0.30.2 -> 0.31.1
python3-hypothesis: upgrade 6.97.3 -> 6.98.12
python3-pip: upgrade 23.3.2 -> 24.0
python3-pycairo: upgrade 1.25.1 -> 1.26.0
python3-pytest: upgrade 8.0.0 -> 8.0.2
python3-pytz: upgrade 2023.4 -> 2024.1
python3-setuptools-rust: upgrade 1.8.1 -> 1.9.0
python3-trove-classifiers: upgrade 2024.1.8 -> 2024.2.23
python3-typing-extensions: upgrade 4.9.0 -> 4.10.0
python3: upgrade 3.12.1 -> 3.12.2
python3-urllib3: upgrade 2.1.0 -> 2.2.1
python3-yamllint: upgrade 1.33.0 -> 1.35.1
swig: upgrade 4.2.0 -> 4.2.1
xkbcomp: upgrade 1.4.6 -> 1.4.7
xkeyboard-config: upgrade 2.40 -> 2.41
xprop: upgrade 1.2.6 -> 1.2.7
Xiangyu Chen (2):
systemd-systemctl: fix dead loop when multi services enable each other
libc-locale: fix ASCII compatible warning cause build failure.
Xiaotian Wu (2):
loongarch64: change -march to loongarch64
openssl: Match target name for loongarch64
Yash Shinde (3):
rust: Upgrade 1.74.1 -> 1.75.0
rust: Revert PGO to it's default
rust: reproducibility issue fix with v1.75
Yoann Congal (1):
waf: Improve version parsing to avoid failing on warnings
Change-Id: I6dfb848feb4ec8f5aae56a9ccbff475f4eb1edc6
Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
diff --git a/poky/bitbake/lib/bb/asyncrpc/client.py b/poky/bitbake/lib/bb/asyncrpc/client.py
index a6228bb..29a5ab7 100644
--- a/poky/bitbake/lib/bb/asyncrpc/client.py
+++ b/poky/bitbake/lib/bb/asyncrpc/client.py
@@ -17,13 +17,24 @@
class AsyncClient(object):
- def __init__(self, proto_name, proto_version, logger, timeout=30):
+ def __init__(
+ self,
+ proto_name,
+ proto_version,
+ logger,
+ timeout=30,
+ server_headers=False,
+ headers={},
+ ):
self.socket = None
self.max_chunk = DEFAULT_MAX_CHUNK
self.proto_name = proto_name
self.proto_version = proto_version
self.logger = logger
self.timeout = timeout
+ self.needs_server_headers = server_headers
+ self.server_headers = {}
+ self.headers = headers
async def connect_tcp(self, address, port):
async def connect_sock():
@@ -61,9 +72,29 @@
async def setup_connection(self):
# Send headers
await self.socket.send("%s %s" % (self.proto_name, self.proto_version))
+ await self.socket.send(
+ "needs-headers: %s" % ("true" if self.needs_server_headers else "false")
+ )
+ for k, v in self.headers.items():
+ await self.socket.send("%s: %s" % (k, v))
+
# End of headers
await self.socket.send("")
+ self.server_headers = {}
+ if self.needs_server_headers:
+ while True:
+ line = await self.socket.recv()
+ if not line:
+ # End headers
+ break
+ tag, value = line.split(":", 1)
+ self.server_headers[tag.lower()] = value.strip()
+
+ async def get_header(self, tag, default):
+ await self.connect()
+ return self.server_headers.get(tag, default)
+
async def connect(self):
if self.socket is None:
self.socket = await self._connect_sock()
diff --git a/poky/bitbake/lib/bb/asyncrpc/serv.py b/poky/bitbake/lib/bb/asyncrpc/serv.py
index f0be9a6..a66117a 100644
--- a/poky/bitbake/lib/bb/asyncrpc/serv.py
+++ b/poky/bitbake/lib/bb/asyncrpc/serv.py
@@ -39,10 +39,14 @@
"address": socket.address,
},
)
+ self.client_headers = {}
async def close(self):
await self.socket.close()
+ async def handle_headers(self, headers):
+ return {}
+
async def process_requests(self):
try:
self.logger.info("Client %r connected" % (self.socket.address,))
@@ -64,12 +68,20 @@
)
return
- # Read headers. Currently, no headers are implemented, so look for
- # an empty line to signal the end of the headers
+ # Read headers
+ self.client_headers = {}
while True:
header = await self.socket.recv()
if not header:
+ # Empty line. End of headers
break
+ tag, value = header.split(":", 1)
+ self.client_headers[tag.lower()] = value.strip()
+
+ if self.client_headers.get("needs-headers", "false") == "true":
+ for k, v in (await self.handle_headers(self.client_headers)).items():
+ await self.socket.send("%s: %s" % (k, v))
+ await self.socket.send("")
# Handle messages
while True:
diff --git a/poky/bitbake/lib/bb/fetch2/git.py b/poky/bitbake/lib/bb/fetch2/git.py
index df33fb6..b43ee0d 100644
--- a/poky/bitbake/lib/bb/fetch2/git.py
+++ b/poky/bitbake/lib/bb/fetch2/git.py
@@ -325,7 +325,10 @@
return ud.clonedir
def need_update(self, ud, d):
- return self.clonedir_need_update(ud, d) or self.shallow_tarball_need_update(ud) or self.tarball_need_update(ud)
+ return self.clonedir_need_update(ud, d) \
+ or self.shallow_tarball_need_update(ud) \
+ or self.tarball_need_update(ud) \
+ or self.lfs_need_update(ud, d)
def clonedir_need_update(self, ud, d):
if not os.path.exists(ud.clonedir):
@@ -337,6 +340,15 @@
return True
return False
+ def lfs_need_update(self, ud, d):
+ if self.clonedir_need_update(ud, d):
+ return True
+
+ for name in ud.names:
+ if not self._lfs_objects_downloaded(ud, d, name, ud.clonedir):
+ return True
+ return False
+
def clonedir_need_shallow_revs(self, ud, d):
for rev in ud.shallow_revs:
try:
@@ -467,7 +479,7 @@
if missing_rev:
raise bb.fetch2.FetchError("Unable to find revision %s even from upstream" % missing_rev)
- if self._contains_lfs(ud, d, ud.clonedir) and self._need_lfs(ud):
+ if self.lfs_need_update(ud, d):
# Unpack temporary working copy, use it to run 'git checkout' to force pre-fetching
# of all LFS blobs needed at the srcrev.
#
@@ -710,6 +722,35 @@
raise bb.fetch2.FetchError("The command '%s' gave output with more then 1 line unexpectedly, output: '%s'" % (cmd, output))
return output.split()[0] != "0"
+ def _lfs_objects_downloaded(self, ud, d, name, wd):
+ """
+ Verifies whether the LFS objects for requested revisions have already been downloaded
+ """
+ # Bail out early if this repository doesn't use LFS
+ if not self._need_lfs(ud) or not self._contains_lfs(ud, d, wd):
+ return True
+
+ # The Git LFS specification specifies ([1]) the LFS folder layout so it should be safe to check for file
+ # existence.
+ # [1] https://github.com/git-lfs/git-lfs/blob/main/docs/spec.md#intercepting-git
+ cmd = "%s lfs ls-files -l %s" \
+ % (ud.basecmd, ud.revisions[name])
+ output = runfetchcmd(cmd, d, quiet=True, workdir=wd).rstrip()
+ # Do not do any further matching if no objects are managed by LFS
+ if not output:
+ return True
+
+ # Match all lines beginning with the hexadecimal OID
+ oid_regex = re.compile("^(([a-fA-F0-9]{2})([a-fA-F0-9]{2})[A-Fa-f0-9]+)")
+ for line in output.split("\n"):
+ oid = re.search(oid_regex, line)
+ if not oid:
+ bb.warn("git lfs ls-files output '%s' did not match expected format." % line)
+ if not os.path.exists(os.path.join(wd, "lfs", "objects", oid.group(2), oid.group(3), oid.group(1))):
+ return False
+
+ return True
+
def _need_lfs(self, ud):
return ud.parm.get("lfs", "1") == "1"
diff --git a/poky/bitbake/lib/bb/runqueue.py b/poky/bitbake/lib/bb/runqueue.py
index e86ccd8..6987de3 100644
--- a/poky/bitbake/lib/bb/runqueue.py
+++ b/poky/bitbake/lib/bb/runqueue.py
@@ -220,6 +220,16 @@
bb.note("Pressure status changed to CPU: %s, IO: %s, Mem: %s (CPU: %s/%s, IO: %s/%s, Mem: %s/%s) - using %s/%s bitbake threads" % (pressure_state + pressure_values + (len(self.rq.runq_running.difference(self.rq.runq_complete)), self.rq.number_tasks)))
self.pressure_state = pressure_state
return (exceeds_cpu_pressure or exceeds_io_pressure or exceeds_memory_pressure)
+ elif self.rq.max_loadfactor:
+ limit = False
+ loadfactor = float(os.getloadavg()[0]) / os.cpu_count()
+ # bb.warn("Comparing %s to %s" % (loadfactor, self.rq.max_loadfactor))
+ if loadfactor > self.rq.max_loadfactor:
+ limit = True
+ if hasattr(self, "loadfactor_limit") and limit != self.loadfactor_limit:
+ bb.note("Load average limiting set to %s as load average: %s - using %s/%s bitbake threads" % (limit, loadfactor, len(self.rq.runq_running.difference(self.rq.runq_complete)), self.rq.number_tasks))
+ self.loadfactor_limit = limit
+ return limit
return False
def next_buildable_task(self):
@@ -1822,6 +1832,7 @@
self.max_cpu_pressure = self.cfgData.getVar("BB_PRESSURE_MAX_CPU")
self.max_io_pressure = self.cfgData.getVar("BB_PRESSURE_MAX_IO")
self.max_memory_pressure = self.cfgData.getVar("BB_PRESSURE_MAX_MEMORY")
+ self.max_loadfactor = self.cfgData.getVar("BB_LOADFACTOR_MAX")
self.sq_buildable = set()
self.sq_running = set()
@@ -1875,6 +1886,11 @@
bb.fatal("Invalid BB_PRESSURE_MAX_MEMORY %s, minimum value is %s." % (self.max_memory_pressure, lower_limit))
if self.max_memory_pressure > upper_limit:
bb.warn("Your build will be largely unregulated since BB_PRESSURE_MAX_MEMORY is set to %s. It is very unlikely that such high pressure will be experienced." % (self.max_io_pressure))
+
+ if self.max_loadfactor:
+ self.max_loadfactor = float(self.max_loadfactor)
+ if self.max_loadfactor <= 0:
+ bb.fatal("Invalid BB_LOADFACTOR_MAX %s, needs to be greater than zero." % (self.max_loadfactor))
# List of setscene tasks which we've covered
self.scenequeue_covered = set()
diff --git a/poky/bitbake/lib/bb/tests/fetch.py b/poky/bitbake/lib/bb/tests/fetch.py
index 5ed5b56..e988e26 100644
--- a/poky/bitbake/lib/bb/tests/fetch.py
+++ b/poky/bitbake/lib/bb/tests/fetch.py
@@ -6,6 +6,7 @@
# SPDX-License-Identifier: GPL-2.0-only
#
+import contextlib
import unittest
import hashlib
import tempfile
@@ -2261,10 +2262,14 @@
bb.utils.mkdirhier(self.srcdir)
self.git_init(cwd=self.srcdir)
- with open(os.path.join(self.srcdir, '.gitattributes'), 'wt') as attrs:
- attrs.write('*.mp3 filter=lfs -text')
- self.git(['add', '.gitattributes'], cwd=self.srcdir)
- self.git(['commit', '-m', "attributes", '.gitattributes'], cwd=self.srcdir)
+ self.commit_file('.gitattributes', '*.mp3 filter=lfs -text')
+
+ def commit_file(self, filename, content):
+ with open(os.path.join(self.srcdir, filename), "w") as f:
+ f.write(content)
+ self.git(["add", filename], cwd=self.srcdir)
+ self.git(["commit", "-m", "Change"], cwd=self.srcdir)
+ return self.git(["rev-parse", "HEAD"], cwd=self.srcdir).strip()
def fetch(self, uri=None, download=True):
uris = self.d.getVar('SRC_URI').split()
@@ -2285,6 +2290,44 @@
return unpacked_lfs_file
@skipIfNoGitLFS()
+ def test_fetch_lfs_on_srcrev_change(self):
+ """Test if fetch downloads missing LFS objects when a different revision within an existing repository is requested"""
+ self.git(["lfs", "install", "--local"], cwd=self.srcdir)
+
+ @contextlib.contextmanager
+ def hide_upstream_repository():
+ """Hide the upstream repository to make sure that git lfs cannot pull from it"""
+ temp_name = self.srcdir + ".bak"
+ os.rename(self.srcdir, temp_name)
+ try:
+ yield
+ finally:
+ os.rename(temp_name, self.srcdir)
+
+ def fetch_and_verify(revision, filename, content):
+ self.d.setVar('SRCREV', revision)
+ fetcher, ud = self.fetch()
+
+ with hide_upstream_repository():
+ workdir = self.d.getVar('WORKDIR')
+ fetcher.unpack(workdir)
+
+ with open(os.path.join(workdir, "git", filename)) as f:
+ self.assertEqual(f.read(), content)
+
+ commit_1 = self.commit_file("a.mp3", "version 1")
+ commit_2 = self.commit_file("a.mp3", "version 2")
+
+ self.d.setVar('SRC_URI', "git://%s;protocol=file;lfs=1;branch=master" % self.srcdir)
+
+ # Seed the local download folder by fetching the latest commit and verifying that the LFS contents are
+ # available even when the upstream repository disappears.
+ fetch_and_verify(commit_2, "a.mp3", "version 2")
+ # Verify that even when an older revision is fetched, the needed LFS objects are fetched into the download
+ # folder.
+ fetch_and_verify(commit_1, "a.mp3", "version 1")
+
+ @skipIfNoGitLFS()
@skipIfNoNetwork()
def test_real_git_lfs_repo_succeeds_without_lfs_param(self):
self.d.setVar('SRC_URI', "git://gitlab.com/gitlab-examples/lfs.git;protocol=https;branch=master")
diff --git a/poky/bitbake/lib/bb/ui/taskexp_ncurses.py b/poky/bitbake/lib/bb/ui/taskexp_ncurses.py
index dd91d26..ea94a49 100755
--- a/poky/bitbake/lib/bb/ui/taskexp_ncurses.py
+++ b/poky/bitbake/lib/bb/ui/taskexp_ncurses.py
@@ -12,7 +12,7 @@
#
# Execution example:
-# $ bitbake -g -u taskexp_ncurses.py acl zlib
+# $ bitbake -g -u taskexp_ncurses zlib acl
#
# Self-test example (executes a script of GUI actions):
# $ TASK_EXP_UNIT_TEST=1 bitbake -g -u taskexp_ncurses zlib acl
diff --git a/poky/bitbake/lib/hashserv/sqlalchemy.py b/poky/bitbake/lib/hashserv/sqlalchemy.py
index fc3ae3d..f7b0226 100644
--- a/poky/bitbake/lib/hashserv/sqlalchemy.py
+++ b/poky/bitbake/lib/hashserv/sqlalchemy.py
@@ -129,7 +129,11 @@
return inspect(conn).has_table(name)
self.logger.info("Using database %s", self.url)
- self.engine = create_async_engine(self.url, poolclass=NullPool)
+ if self.url.drivername == 'postgresql+psycopg':
+ # Psygopg 3 (psygopg) driver can handle async connection pooling
+ self.engine = create_async_engine(self.url, max_overflow=-1)
+ else:
+ self.engine = create_async_engine(self.url, poolclass=NullPool)
async with self.engine.begin() as conn:
# Create tables
diff --git a/poky/bitbake/lib/layerindexlib/__init__.py b/poky/bitbake/lib/layerindexlib/__init__.py
index ac03d89..c3265dd 100644
--- a/poky/bitbake/lib/layerindexlib/__init__.py
+++ b/poky/bitbake/lib/layerindexlib/__init__.py
@@ -178,9 +178,9 @@
'''Load the layerindex.
indexURI - An index to load. (Use multiple calls to load multiple indexes)
-
+
reload - If reload is True, then any previously loaded indexes will be forgotten.
-
+
load - List of elements to load. Default loads all items.
Note: plugs may ignore this.
@@ -383,7 +383,14 @@
# Get a list of dependencies and then recursively process them
for layerdependency in layerbranch.index.layerDependencies_layerBranchId[layerbranch.id]:
- deplayerbranch = layerdependency.dependency_layerBranch
+ try:
+ deplayerbranch = layerdependency.dependency_layerBranch
+ except AttributeError as e:
+ logger.error('LayerBranch does not exist for dependent layer {}:{}\n' \
+ ' Cannot continue successfully.\n' \
+ ' You might be able to resolve this by checking out the layer locally.\n' \
+ ' Consider reaching out the to the layer maintainers or the layerindex admins' \
+ .format(layerdependency.dependency.name, layerbranch.branch.name))
if ignores and deplayerbranch.layer.name in ignores:
continue
@@ -846,7 +853,7 @@
continue
for layerdependency in layerbranch.index.layerDependencies_layerBranchId[layerbranch.id]:
- deplayerbranch = layerdependency.dependency_layerBranch
+ deplayerbranch = layerdependency.dependency_layerBranch or None
if ignores and deplayerbranch.layer.name in ignores:
continue
diff --git a/poky/bitbake/lib/progressbar/progressbar.py b/poky/bitbake/lib/progressbar/progressbar.py
index e2b6ba1..d4da10a 100644
--- a/poky/bitbake/lib/progressbar/progressbar.py
+++ b/poky/bitbake/lib/progressbar/progressbar.py
@@ -253,7 +253,7 @@
if (self.maxval is not UnknownLength
and not 0 <= value <= self.maxval):
- raise ValueError('Value out of range')
+ self.maxval = value
self.currval = value