poky: subtree update:81f9e815d3..03d4d9d68f
Adrian Bunk (1):
json-c: Don't --enable-rdrand
Alessio Igor Bogani (2):
wic: Using the right rootfs size during prepare_rootfs
rootfs-postcommands: Avoid use of an hard-coded value
Alexander Kanavin (1):
binutils: drop UPSTREAM_VERSION_UNKNOWN
Alexandre Bard (1):
systemd: Expose resolv-conf alternative only when resolved is built
Andre McCurdy (1):
ffmpeg: enable more verbose build logs
André Draszik (4):
ruby: drop long-merged CVE patches
ruby: configure mis-detects isnan/isinf on musl
ruby: fix non-IPv6 support
packagegroup: fix a comment regarding PACKAGE_ARCH
Bruce Ashfield (6):
linux-yocto/5.2: update to v5.2.13
linux-yocto/4.19: update to v4.19.72
linux-yocto/5.2: update to v5.2.14
linux-yocto/5.2: update to v5.2.16
linux-yocto/5.2: update to v5.2.17
yocto-bsps: update to v5.2.17
Böszörményi Zoltán via Openembedded-core (1):
classes/image-live.bbclass: Don't hardcode cpio.gz
Changqing Li (2):
devtool.py: change to do clean before remove-layer
devtool.py: fix buildclean test
Chen Qi (1):
systemd: fix NFS regression
Dan Tran (1):
unzip: Fix CVE-2019-13232
David Reyna (2):
bitbake: toaster: issues in import layer when clicking 'add layer'
bitbake: toaster: improve warnings when adding dependency to packages
Diego Rondini (2):
initramfs-framework: fix var name
initramfs-framework: support PARTLABEL option
Douglas Royds (1):
icecc: Don't use icecc when INHIBIT_DEFAULT_DEPS is set
He Zhe (1):
ltp: Fix hang of cve test cases
Heiko Schocher (1):
kernel.fitimage.bbclass: remove ramdisk_ctype
Jacob Kroon (1):
bitbake: tests/data: Test combinations of _append together with override
Joe Slater (1):
bash-completion: add image feature
Jonathan Marler (1):
package: Multiple shlib_providers for the same file should error
Joshua Watt (8):
classes/reproducible_build: Move SDE deploy to another directory
oeqa: Test multiconfig parsing
bitbake: cookerdata: Add mc conffiles hashes to cache hash
bitbake: hashserve: Add missing import
bitbake: siggen: Fix attribute error when hashserver fails
bitbake: hashserv: Don't daemonize server process
local.conf.sample: Add Hash Equivalence
classes/reproducible_build: Create SDE destination
Khem Raj (7):
musl: Fix riscv64 CAS functions
qemuriscv: Do not blacklist clang anymore
sdk: Install nativesdk locales for all TCLIBC variants
strace: Upgrade to 5.3
packagegroups: All groups are not allarch
musl: Fix __riscv_mc* containers to match glibc
core-image-sato-sdk-ptest: Remove valgrind ptests for riscv
Konrad Scherer (1):
gen-lockedsig-cache: Replace glob lookup with hash to filename lookup
Lei Maohui (1):
bluez5: update patch to fix do_patch error when PATCHTOOL = "patch".
Li Zhou (1):
shadow: use relaxed usernames for all
Limeng (1):
u-boot: add CVE patches for u-boot
Nathan Rossi (2):
oeqa/core/utils/concurrencytest.py: Handle exceptions and details
oeqa/core/case.py: Encode binary data of log
Niclas Svensson (1):
devtool: finish: Keep patches ordered when updating bbappend
Otavio Salvador (1):
mesa: Add freedreno PACKAGECONFIG option
Peter Kjellerstedt (3):
systemd: Make it build with hwdb disabled
devtool: finish: Add suppport for the --no-clean option
lib/oe/lsb: Make sure the distro ID is always lowercased
Randy MacLeod (1):
ffmpeg: update from 4.2 to 4.2.1
Richard Purdie (17):
Revert "meta-extsdk: Either an sstate task is a proper task or it isn't"
sstatesig: Fix hash equivlanency locked signature issues
oeqa/selftest/signing: Fix for hash equivlance server
lib/sstatesig: Fix class inheritance problems
populate_sdk_ext: Fix for hash equiv
bitbake: runqueue: Fix task migration problems
bitbake: siggen: Ensure setscenetasks list is available to worker context
bitbake: runqueue: Change task migration behaviour for rerunning setscene tasks
bitbake: siggen/runqueue: Fix signature mismatch issues
bitbake: siggen: Avoid writing misleading sigdata files
bitbake: runqueue: Save unihashes more frequently
bitbake: runqueue: Small performance optimisation
bitbake: siggen: Remove full path from unitaskhashes keys
bitbake: tests/runqueue: Fix hashserve shutdown race
base: Improve module import error message
sanity.conf: Bump minimum bitbake version
bitbake: bitbake: Bump verison 1.43.1 -> 1.43.2
Robert Yang (6):
cases/bbtests.py: test_bitbake_g(): Check base-files rather than busybox
expect: Fix configure error for nativesdk
net-tools: Fix installed-vs-shipped for nativesdk
expect: Fix buffer overflow error when build in long path
apr: Check for libtoolize rather than libtool
lttng-ust: Fix for --enable-python-agent
Ross Burton (12):
oeqa/selftest/reproducible: test ipkgs too
distcc: clean up the UI install logic
distcc: use --enable-tcp-insecure instead of --make-me-a-botnet
distcc: split into client and server packages
json-c: clean up recipe
json-c: use GitHub for upstream release checking
bitbake: fetch2/git: refactor check for git-lfs command
bitbake: tests/fetch: add test case for git-lfs handling
python3: move runpy to core
pango: fix the failing testiter test case
opkg: remove redundant systemd inherit
lttng-ust: update patch Signed-off-by
Trevor Gamblin (5):
python3-subunit: ensure runtime dependencies are present
python3-pip: ensure pickle is installed
lighttpd: remove fam as a PACKAGECONFIG option
tiff: fix CVE-2019-14973
opkg: remove pathfinder PACKAGECONFIG option
Wang Quanyang (1):
kexec-tools: fix arm kexec failure for __NR_kexec_file_load
Yi Zhao (1):
python: add tk-lib as runtime dependency for python-tkinter
Change-Id: I0570125d49f7e4bc3bbf70508cbfd7e10bdbc032
Signed-off-by: Brad Bishop <bradleyb@fuzziesquirrel.com>
diff --git a/poky/bitbake/lib/bb/__init__.py b/poky/bitbake/lib/bb/__init__.py
index 322a1e0..f899691 100644
--- a/poky/bitbake/lib/bb/__init__.py
+++ b/poky/bitbake/lib/bb/__init__.py
@@ -9,7 +9,7 @@
# SPDX-License-Identifier: GPL-2.0-only
#
-__version__ = "1.43.1"
+__version__ = "1.43.2"
import sys
if sys.version_info < (3, 4, 0):
diff --git a/poky/bitbake/lib/bb/cooker.py b/poky/bitbake/lib/bb/cooker.py
index 0c54002..20ef04d 100644
--- a/poky/bitbake/lib/bb/cooker.py
+++ b/poky/bitbake/lib/bb/cooker.py
@@ -399,7 +399,6 @@
self.hashservaddr = "unix://%s/hashserve.sock" % self.data.getVar("TOPDIR")
self.hashserv = hashserv.create_server(self.hashservaddr, dbfile, sync=False)
self.hashserv.process = multiprocessing.Process(target=self.hashserv.serve_forever)
- self.hashserv.process.daemon = True
self.hashserv.process.start()
self.data.setVar("BB_HASHSERVE", self.hashservaddr)
self.databuilder.origdata.setVar("BB_HASHSERVE", self.hashservaddr)
diff --git a/poky/bitbake/lib/bb/cookerdata.py b/poky/bitbake/lib/bb/cookerdata.py
index 96a8e6b..472423f 100644
--- a/poky/bitbake/lib/bb/cookerdata.py
+++ b/poky/bitbake/lib/bb/cookerdata.py
@@ -13,6 +13,7 @@
import os
import re
import sys
+import hashlib
from functools import wraps
import bb
from bb import data
@@ -267,6 +268,7 @@
self.mcdata = {}
def parseBaseConfiguration(self):
+ data_hash = hashlib.sha256()
try:
self.data = self.parseConfigurationFiles(self.prefiles, self.postfiles)
@@ -290,7 +292,7 @@
bb.event.fire(bb.event.ConfigParsed(), self.data)
bb.parse.init_parser(self.data)
- self.data_hash = self.data.get_hash()
+ data_hash.update(self.data.get_hash().encode('utf-8'))
self.mcdata[''] = self.data
multiconfig = (self.data.getVar("BBMULTICONFIG") or "").split()
@@ -298,9 +300,11 @@
mcdata = self.parseConfigurationFiles(self.prefiles, self.postfiles, config)
bb.event.fire(bb.event.ConfigParsed(), mcdata)
self.mcdata[config] = mcdata
+ data_hash.update(mcdata.get_hash().encode('utf-8'))
if multiconfig:
bb.event.fire(bb.event.MultiConfigParsed(self.mcdata), self.data)
+ self.data_hash = data_hash.hexdigest()
except (SyntaxError, bb.BBHandledException):
raise bb.BBHandledException
except bb.data_smart.ExpansionError as e:
diff --git a/poky/bitbake/lib/bb/fetch2/git.py b/poky/bitbake/lib/bb/fetch2/git.py
index 5fd63b4..2d1d2ca 100644
--- a/poky/bitbake/lib/bb/fetch2/git.py
+++ b/poky/bitbake/lib/bb/fetch2/git.py
@@ -495,14 +495,8 @@
runfetchcmd("%s remote set-url origin %s" % (ud.basecmd, repourl), d, workdir=destdir)
if self._contains_lfs(ud, d, destdir):
- if need_lfs:
- path = d.getVar('PATH')
- if path:
- gitlfstool = bb.utils.which(path, "git-lfs", executable=True)
- if not gitlfstool:
- raise bb.fetch2.FetchError("Repository %s has LFS content, install git-lfs on host to download (or set lfs=0 to ignore it)" % (repourl))
- else:
- bb.note("Could not find 'PATH'")
+ if need_lfs and not self._find_git_lfs(d):
+ raise bb.fetch2.FetchError("Repository %s has LFS content, install git-lfs on host to download (or set lfs=0 to ignore it)" % (repourl))
else:
bb.note("Repository %s has LFS content but it is not being fetched" % (repourl))
@@ -570,6 +564,13 @@
pass
return False
+ def _find_git_lfs(self, d):
+ """
+ Return True if git-lfs can be found, False otherwise.
+ """
+ import shutil
+ return shutil.which("git-lfs", path=d.getVar('PATH')) is not None
+
def _get_repo_url(self, ud):
"""
Return the repository URL
diff --git a/poky/bitbake/lib/bb/runqueue.py b/poky/bitbake/lib/bb/runqueue.py
index d9a67a3..1804943 100644
--- a/poky/bitbake/lib/bb/runqueue.py
+++ b/poky/bitbake/lib/bb/runqueue.py
@@ -73,7 +73,7 @@
def pending_hash_index(tid, rqdata):
(mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
pn = rqdata.dataCaches[mc].pkg_fn[taskfn]
- h = rqdata.runtaskentries[tid].hash
+ h = rqdata.runtaskentries[tid].unihash
return pn + ":" + "taskname" + h
class RunQueueStats:
@@ -207,6 +207,8 @@
def newbuildable(self, task):
self.buildable.add(task)
+ # Once tasks are running we don't need to worry about them again
+ self.buildable.difference_update(self.rq.runq_running)
def removebuildable(self, task):
self.buildable.remove(task)
@@ -1162,6 +1164,8 @@
self.init_progress_reporter.next_stage()
+ bb.parse.siggen.set_setscene_tasks(self.runq_setscene_tids)
+
# Iterate over the task list and call into the siggen code
dealtwith = set()
todeal = set(self.runtaskentries)
@@ -1173,7 +1177,6 @@
self.prepare_task_hash(tid)
bb.parse.siggen.writeout_file_checksum_cache()
- bb.parse.siggen.set_setscene_tasks(self.runq_setscene_tids)
#self.dump_data()
return len(self.runtaskentries)
@@ -1442,6 +1445,7 @@
self.state = runQueueComplete
else:
self.state = runQueueSceneInit
+ bb.parse.siggen.save_unitaskhashes()
if self.state is runQueueSceneInit:
self.rqdata.init_progress_reporter.next_stage()
@@ -2299,11 +2303,12 @@
for tid in changed:
if tid not in self.rqdata.runq_setscene_tids:
continue
- valid = self.rq.validate_hashes(set([tid]), self.cooker.data, None, False)
- if not valid:
- continue
if tid in self.runq_running:
continue
+ if tid in self.scenequeue_covered:
+ # Potentially risky, should we report this hash as a match?
+ logger.info("Already covered setscene for %s so ignoring rehash" % (tid))
+ continue
if tid not in self.pending_migrations:
self.pending_migrations.add(tid)
@@ -2358,6 +2363,7 @@
logger.info("Setscene task %s now valid and being rerun" % tid)
self.sqdone = False
+ update_scenequeue_data([tid], self.sqdata, self.rqdata, self.rq, self.cooker, self.stampcache, self)
if changed:
self.holdoff_need_update = True
@@ -2674,64 +2680,77 @@
rqdata.init_progress_reporter.next_stage()
- multiconfigs = set()
+ sqdata.multiconfigs = set()
for tid in sqdata.sq_revdeps:
- multiconfigs.add(mc_from_tid(tid))
+ sqdata.multiconfigs.add(mc_from_tid(tid))
if len(sqdata.sq_revdeps[tid]) == 0:
sqrq.sq_buildable.add(tid)
rqdata.init_progress_reporter.finish()
- if rq.hashvalidate:
- noexec = []
- stamppresent = []
- tocheck = set()
+ sqdata.noexec = set()
+ sqdata.stamppresent = set()
+ sqdata.valid = set()
+ update_scenequeue_data(sqdata.sq_revdeps, sqdata, rqdata, rq, cooker, stampcache, sqrq)
+
+def update_scenequeue_data(tids, sqdata, rqdata, rq, cooker, stampcache, sqrq):
+
+ tocheck = set()
+
+ for tid in sorted(tids):
+ if tid in sqdata.stamppresent:
+ sqdata.stamppresent.remove(tid)
+ if tid in sqdata.valid:
+ sqdata.valid.remove(tid)
+
+ (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
+
+ taskdep = rqdata.dataCaches[mc].task_deps[taskfn]
+
+ if 'noexec' in taskdep and taskname in taskdep['noexec']:
+ sqdata.noexec.add(tid)
+ sqrq.sq_task_skip(tid)
+ bb.build.make_stamp(taskname + "_setscene", rqdata.dataCaches[mc], taskfn)
+ continue
+
+ if rq.check_stamp_task(tid, taskname + "_setscene", cache=stampcache):
+ logger.debug(2, 'Setscene stamp current for task %s', tid)
+ sqdata.stamppresent.add(tid)
+ sqrq.sq_task_skip(tid)
+ continue
+
+ if rq.check_stamp_task(tid, taskname, recurse = True, cache=stampcache):
+ logger.debug(2, 'Normal stamp current for task %s', tid)
+ sqdata.stamppresent.add(tid)
+ sqrq.sq_task_skip(tid)
+ continue
+
+ tocheck.add(tid)
+
+ sqdata.valid |= rq.validate_hashes(tocheck, cooker.data, len(sqdata.stamppresent), False)
+
+ sqdata.hashes = {}
+ for mc in sorted(sqdata.multiconfigs):
for tid in sorted(sqdata.sq_revdeps):
- (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
-
- taskdep = rqdata.dataCaches[mc].task_deps[taskfn]
-
- if 'noexec' in taskdep and taskname in taskdep['noexec']:
- noexec.append(tid)
- sqrq.sq_task_skip(tid)
- bb.build.make_stamp(taskname + "_setscene", rqdata.dataCaches[mc], taskfn)
- continue
-
- if rq.check_stamp_task(tid, taskname + "_setscene", cache=stampcache):
- logger.debug(2, 'Setscene stamp current for task %s', tid)
- stamppresent.append(tid)
- sqrq.sq_task_skip(tid)
- continue
-
- if rq.check_stamp_task(tid, taskname, recurse = True, cache=stampcache):
- logger.debug(2, 'Normal stamp current for task %s', tid)
- stamppresent.append(tid)
- sqrq.sq_task_skip(tid)
- continue
-
- tocheck.add(tid)
-
- valid = rq.validate_hashes(tocheck, cooker.data, len(stamppresent), False)
-
- valid_new = stamppresent
- for v in valid:
- valid_new.append(v)
-
- hashes = {}
- for mc in sorted(multiconfigs):
- for tid in sorted(sqdata.sq_revdeps):
if mc_from_tid(tid) != mc:
continue
- if tid not in valid_new and tid not in noexec and tid not in sqrq.scenequeue_notcovered:
- sqdata.outrightfail.add(tid)
+ if tid in sqdata.stamppresent:
+ continue
+ if tid in sqdata.valid:
+ continue
+ if tid in sqdata.noexec:
+ continue
+ if tid in sqrq.scenequeue_notcovered:
+ continue
+ sqdata.outrightfail.add(tid)
- h = pending_hash_index(tid, rqdata)
- if h not in hashes:
- hashes[h] = tid
- else:
- sqrq.sq_deferred[tid] = hashes[h]
- bb.warn("Deferring %s after %s" % (tid, hashes[h]))
+ h = pending_hash_index(tid, rqdata)
+ if h not in sqdata.hashes:
+ sqdata.hashes[h] = tid
+ else:
+ sqrq.sq_deferred[tid] = sqdata.hashes[h]
+ bb.warn("Deferring %s after %s" % (tid, sqdata.hashes[h]))
class TaskFailure(Exception):
diff --git a/poky/bitbake/lib/bb/siggen.py b/poky/bitbake/lib/bb/siggen.py
index e047c21..a4bb1ff 100644
--- a/poky/bitbake/lib/bb/siggen.py
+++ b/poky/bitbake/lib/bb/siggen.py
@@ -44,6 +44,7 @@
self.file_checksum_values = {}
self.taints = {}
self.unitaskhashes = {}
+ self.setscenetasks = {}
def finalise(self, fn, d, varient):
return
@@ -75,10 +76,10 @@
return
def get_taskdata(self):
- return (self.runtaskdeps, self.taskhash, self.file_checksum_values, self.taints, self.basehash, self.unitaskhashes)
+ return (self.runtaskdeps, self.taskhash, self.file_checksum_values, self.taints, self.basehash, self.unitaskhashes, self.setscenetasks)
def set_taskdata(self, data):
- self.runtaskdeps, self.taskhash, self.file_checksum_values, self.taints, self.basehash, self.unitaskhashes = data
+ self.runtaskdeps, self.taskhash, self.file_checksum_values, self.taints, self.basehash, self.unitaskhashes, self.setscenetasks = data
def reset(self, data):
self.__init__(data)
@@ -267,7 +268,7 @@
sigfile = stampbase
referencestamp = runtime[11:]
elif runtime and tid in self.taskhash:
- sigfile = stampbase + "." + task + ".sigdata" + "." + self.taskhash[tid]
+ sigfile = stampbase + "." + task + ".sigdata" + "." + self.get_unihash(tid)
else:
sigfile = stampbase + "." + task + ".sigbasedata" + "." + self.basehash[tid]
@@ -295,6 +296,7 @@
for dep in data['runtaskdeps']:
data['runtaskhashes'][dep] = self.get_unihash(dep)
data['taskhash'] = self.taskhash[tid]
+ data['unihash'] = self.get_unihash(tid)
taint = self.read_taint(fn, task, referencestamp)
if taint:
@@ -384,7 +386,7 @@
def __get_task_unihash_key(self, tid):
# TODO: The key only *needs* to be the taskhash, the tid is just
# convenient
- return '%s:%s' % (tid, self.taskhash[tid])
+ return '%s:%s' % (tid.rsplit("/", 1)[1], self.taskhash[tid])
def get_stampfile_hash(self, tid):
if tid in self.taskhash:
@@ -440,7 +442,7 @@
bb.debug((1, 2)[unihash == taskhash], 'Found unihash %s in place of %s for %s from %s' % (unihash, taskhash, tid, self.server))
else:
bb.debug(2, 'No reported unihash for %s:%s from %s' % (tid, taskhash, self.server))
- except hashserv.HashConnectionError as e:
+ except hashserv.client.HashConnectionError as e:
bb.warn('Error contacting Hash Equivalence Server %s: %s' % (self.server, str(e)))
self.unitaskhashes[key] = unihash
@@ -454,7 +456,11 @@
report_taskdata = d.getVar('SSTATE_HASHEQUIV_REPORT_TASKDATA') == '1'
tempdir = d.getVar('T')
fn = d.getVar('BB_FILENAME')
- key = fn + ':do_' + task + ':' + taskhash
+ tid = fn + ':do_' + task
+ key = tid.rsplit("/", 1)[1] + ':' + taskhash
+
+ if self.setscenetasks and tid not in self.setscenetasks:
+ return
# Sanity checks
cache_unihash = self.unitaskhashes.get(key, None)
@@ -504,7 +510,7 @@
bb.event.fire(bb.runqueue.taskUniHashUpdate(fn + ':do_' + task, new_unihash), d)
else:
bb.debug(1, 'Reported task %s as unihash %s to %s' % (taskhash, unihash, self.server))
- except hashserv.HashConnectionError as e:
+ except hashserv.client.HashConnectionError as e:
bb.warn('Error contacting Hash Equivalence Server %s: %s' % (self.server, str(e)))
finally:
if sigfile:
diff --git a/poky/bitbake/lib/bb/tests/data.py b/poky/bitbake/lib/bb/tests/data.py
index a9b0bdb..3e49984 100644
--- a/poky/bitbake/lib/bb/tests/data.py
+++ b/poky/bitbake/lib/bb/tests/data.py
@@ -381,6 +381,19 @@
self.d.setVar("OVERRIDES", "foo:bar:some_val")
self.assertEqual(self.d.getVar("TEST"), " testvalue5")
+ def test_append_and_override_1(self):
+ self.d.setVar("TEST_append", "testvalue2")
+ self.d.setVar("TEST_bar", "testvalue3")
+ self.assertEqual(self.d.getVar("TEST"), "testvalue3testvalue2")
+
+ def test_append_and_override_2(self):
+ self.d.setVar("TEST_append_bar", "testvalue2")
+ self.assertEqual(self.d.getVar("TEST"), "testvaluetestvalue2")
+
+ def test_append_and_override_3(self):
+ self.d.setVar("TEST_bar_append", "testvalue2")
+ self.assertEqual(self.d.getVar("TEST"), "testvalue2")
+
# Test an override with _<numeric> in it based on a real world OE issue
def test_underscore_override(self):
self.d.setVar("TARGET_ARCH", "x86_64")
diff --git a/poky/bitbake/lib/bb/tests/fetch.py b/poky/bitbake/lib/bb/tests/fetch.py
index 2ee0305..a0b656b 100644
--- a/poky/bitbake/lib/bb/tests/fetch.py
+++ b/poky/bitbake/lib/bb/tests/fetch.py
@@ -1908,3 +1908,83 @@
dir = os.listdir(self.unpackdir + "/git/")
self.assertIn("fstests.doap", dir)
+
+class GitLfsTest(FetcherTest):
+ def setUp(self):
+ FetcherTest.setUp(self)
+
+ self.gitdir = os.path.join(self.tempdir, 'git')
+ self.srcdir = os.path.join(self.tempdir, 'gitsource')
+
+ self.d.setVar('WORKDIR', self.tempdir)
+ self.d.setVar('S', self.gitdir)
+ self.d.delVar('PREMIRRORS')
+ self.d.delVar('MIRRORS')
+
+ self.d.setVar('SRCREV', '${AUTOREV}')
+ self.d.setVar('AUTOREV', '${@bb.fetch2.get_autorev(d)}')
+
+ bb.utils.mkdirhier(self.srcdir)
+ self.git('init', cwd=self.srcdir)
+ with open(os.path.join(self.srcdir, '.gitattributes'), 'wt') as attrs:
+ attrs.write('*.mp3 filter=lfs -text')
+ self.git(['add', '.gitattributes'], cwd=self.srcdir)
+ self.git(['commit', '-m', "attributes", '.gitattributes'], cwd=self.srcdir)
+
+ def git(self, cmd, cwd=None):
+ if isinstance(cmd, str):
+ cmd = 'git ' + cmd
+ else:
+ cmd = ['git'] + cmd
+ if cwd is None:
+ cwd = self.gitdir
+ return bb.process.run(cmd, cwd=cwd)[0]
+
+ def fetch(self, uri=None):
+ uris = self.d.getVar('SRC_URI').split()
+ uri = uris[0]
+ d = self.d
+
+ fetcher = bb.fetch2.Fetch(uris, d)
+ fetcher.download()
+ ud = fetcher.ud[uri]
+ return fetcher, ud
+
+ def test_lfs_enabled(self):
+ import shutil
+
+ uri = 'git://%s;protocol=file;subdir=${S};lfs=1' % self.srcdir
+ self.d.setVar('SRC_URI', uri)
+
+ fetcher, ud = self.fetch()
+ self.assertIsNotNone(ud.method._find_git_lfs)
+
+ # If git-lfs can be found, the unpack should be successful
+ ud.method._find_git_lfs = lambda d: True
+ shutil.rmtree(self.gitdir, ignore_errors=True)
+ fetcher.unpack(self.d.getVar('WORKDIR'))
+
+ # If git-lfs cannot be found, the unpack should throw an error
+ with self.assertRaises(bb.fetch2.FetchError):
+ ud.method._find_git_lfs = lambda d: False
+ shutil.rmtree(self.gitdir, ignore_errors=True)
+ fetcher.unpack(self.d.getVar('WORKDIR'))
+
+ def test_lfs_disabled(self):
+ import shutil
+
+ uri = 'git://%s;protocol=file;subdir=${S};lfs=0' % self.srcdir
+ self.d.setVar('SRC_URI', uri)
+
+ fetcher, ud = self.fetch()
+ self.assertIsNotNone(ud.method._find_git_lfs)
+
+ # If git-lfs can be found, the unpack should be successful
+ ud.method._find_git_lfs = lambda d: True
+ shutil.rmtree(self.gitdir, ignore_errors=True)
+ fetcher.unpack(self.d.getVar('WORKDIR'))
+
+ # If git-lfs cannot be found, the unpack should be successful
+ ud.method._find_git_lfs = lambda d: False
+ shutil.rmtree(self.gitdir, ignore_errors=True)
+ fetcher.unpack(self.d.getVar('WORKDIR'))
diff --git a/poky/bitbake/lib/bb/tests/runqueue.py b/poky/bitbake/lib/bb/tests/runqueue.py
index cb4d526..5e64391 100644
--- a/poky/bitbake/lib/bb/tests/runqueue.py
+++ b/poky/bitbake/lib/bb/tests/runqueue.py
@@ -12,6 +12,7 @@
import tempfile
import subprocess
import sys
+import time
#
# TODO:
@@ -257,6 +258,8 @@
'a1:package_write_ipk_setscene', 'a1:package_qa_setscene']
self.assertEqual(set(tasks), set(expected))
+ self.shutdown(tempdir)
+
@unittest.skipIf(sys.version_info < (3, 5, 0), 'Python 3.5 or later required')
def test_hashserv_double(self):
with tempfile.TemporaryDirectory(prefix="runqueuetest") as tempdir:
@@ -280,6 +283,7 @@
'a1:package_write_rpm_setscene', 'b1:package_write_ipk_setscene', 'a1:packagedata_setscene']
self.assertEqual(set(tasks), set(expected))
+ self.shutdown(tempdir)
@unittest.skipIf(sys.version_info < (3, 5, 0), 'Python 3.5 or later required')
def test_hashserv_multiple_setscene(self):
@@ -307,97 +311,13 @@
'e1:package_setscene']
self.assertEqual(set(tasks), set(expected))
for i in expected:
- if i in ["e1:package_setscene"]:
- self.assertEqual(tasks.count(i), 4, "%s not in task list four times" % i)
- else:
- self.assertEqual(tasks.count(i), 1, "%s not in task list once" % i)
+ self.assertEqual(tasks.count(i), 1, "%s not in task list once" % i)
- @unittest.skipIf(sys.version_info < (3, 5, 0), 'Python 3.5 or later required')
- def test_hashserv_partial_match(self):
- # e1:do_package matches initial built but not second hash value
- with tempfile.TemporaryDirectory(prefix="runqueuetest") as tempdir:
- extraenv = {
- "BB_HASHSERVE" : "auto",
- "BB_SIGNATURE_HANDLER" : "TestEquivHash"
- }
- cmd = ["bitbake", "a1", "b1"]
- setscenetasks = ['package_write_ipk_setscene', 'package_write_rpm_setscene', 'packagedata_setscene',
- 'populate_sysroot_setscene', 'package_qa_setscene']
- sstatevalid = ""
- tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid, extraenv=extraenv, cleanup=True)
- expected = ['a1:' + x for x in self.alltasks] + ['b1:' + x for x in self.alltasks]
- self.assertEqual(set(tasks), set(expected))
- with open(tempdir + "/stamps/a1.do_install.taint", "w") as f:
- f.write("d460a29e-903f-4b76-a96b-3bcc22a65994")
- with open(tempdir + "/stamps/b1.do_install.taint", "w") as f:
- f.write("ed36d46a-2977-458a-b3de-eef885bc1817")
- cmd = ["bitbake", "e1"]
- sstatevalid = "e1:do_package:685e69a026b2f029483fdefe6a11e1e06641dd2a0f6f86e27b9b550f8f21229d"
- tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid, extraenv=extraenv, cleanup=True)
- expected = ['a1:package', 'a1:install', 'b1:package', 'b1:install', 'a1:populate_sysroot', 'b1:populate_sysroot',
- 'a1:package_write_ipk_setscene', 'b1:packagedata_setscene', 'b1:package_write_rpm_setscene',
- 'a1:package_write_rpm_setscene', 'b1:package_write_ipk_setscene', 'a1:packagedata_setscene',
- 'e1:package_setscene'] + ['e1:' + x for x in self.alltasks]
- expected.remove('e1:package')
- self.assertEqual(set(tasks), set(expected))
+ self.shutdown(tempdir)
- @unittest.skipIf(sys.version_info < (3, 5, 0), 'Python 3.5 or later required')
- def test_hashserv_partial_match2(self):
- # e1:do_package + e1:do_populate_sysroot matches initial built but not second hash value
- with tempfile.TemporaryDirectory(prefix="runqueuetest") as tempdir:
- extraenv = {
- "BB_HASHSERVE" : "auto",
- "BB_SIGNATURE_HANDLER" : "TestEquivHash"
- }
- cmd = ["bitbake", "a1", "b1"]
- setscenetasks = ['package_write_ipk_setscene', 'package_write_rpm_setscene', 'packagedata_setscene',
- 'populate_sysroot_setscene', 'package_qa_setscene']
- sstatevalid = ""
- tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid, extraenv=extraenv, cleanup=True)
- expected = ['a1:' + x for x in self.alltasks] + ['b1:' + x for x in self.alltasks]
- self.assertEqual(set(tasks), set(expected))
- with open(tempdir + "/stamps/a1.do_install.taint", "w") as f:
- f.write("d460a29e-903f-4b76-a96b-3bcc22a65994")
- with open(tempdir + "/stamps/b1.do_install.taint", "w") as f:
- f.write("ed36d46a-2977-458a-b3de-eef885bc1817")
- cmd = ["bitbake", "e1"]
- sstatevalid = "e1:do_package:685e69a026b2f029483fdefe6a11e1e06641dd2a0f6f86e27b9b550f8f21229d e1:do_populate_sysroot:ef7dc0e2dd55d0534e75cba50731ff42f949818b6f29a65d72bc05856e56711d"
- tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid, extraenv=extraenv, cleanup=True)
- expected = ['a1:package', 'a1:install', 'b1:package', 'b1:install', 'a1:populate_sysroot', 'b1:populate_sysroot',
- 'a1:package_write_ipk_setscene', 'b1:packagedata_setscene', 'b1:package_write_rpm_setscene',
- 'a1:package_write_rpm_setscene', 'b1:package_write_ipk_setscene', 'a1:packagedata_setscene',
- 'e1:package_setscene', 'e1:populate_sysroot_setscene', 'e1:build', 'e1:package_qa', 'e1:package_write_rpm', 'e1:package_write_ipk', 'e1:packagedata']
- self.assertEqual(set(tasks), set(expected))
-
- @unittest.skipIf(sys.version_info < (3, 5, 0), 'Python 3.5 or later required')
- def test_hashserv_partial_match3(self):
- # e1:do_package is valid for a1 but not after b1
- # In former buggy code, this triggered e1:do_fetch, then e1:do_populate_sysroot to run
- # with none of the intermediate tasks which is a serious bug
- with tempfile.TemporaryDirectory(prefix="runqueuetest") as tempdir:
- extraenv = {
- "BB_HASHSERVE" : "auto",
- "BB_SIGNATURE_HANDLER" : "TestEquivHash"
- }
- cmd = ["bitbake", "a1", "b1"]
- setscenetasks = ['package_write_ipk_setscene', 'package_write_rpm_setscene', 'packagedata_setscene',
- 'populate_sysroot_setscene', 'package_qa_setscene']
- sstatevalid = ""
- tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid, extraenv=extraenv, cleanup=True)
- expected = ['a1:' + x for x in self.alltasks] + ['b1:' + x for x in self.alltasks]
- self.assertEqual(set(tasks), set(expected))
- with open(tempdir + "/stamps/a1.do_install.taint", "w") as f:
- f.write("d460a29e-903f-4b76-a96b-3bcc22a65994")
- with open(tempdir + "/stamps/b1.do_install.taint", "w") as f:
- f.write("ed36d46a-2977-458a-b3de-eef885bc1817")
- cmd = ["bitbake", "e1", "-DD"]
- sstatevalid = "e1:do_package:af056eae12a733a6a8c4f4da8c6757e588e13565852c94e2aad4d953a3989c13 e1:do_package:a3677703db82b22d28d57c1820a47851dd780104580863f5bd32e66e003a779d"
- tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid, extraenv=extraenv, cleanup=True, slowtasks="e1:fetch b1:install")
- expected = ['a1:package', 'a1:install', 'b1:package', 'b1:install', 'a1:populate_sysroot', 'b1:populate_sysroot',
- 'a1:package_write_ipk_setscene', 'b1:packagedata_setscene', 'b1:package_write_rpm_setscene',
- 'a1:package_write_rpm_setscene', 'b1:package_write_ipk_setscene', 'a1:packagedata_setscene',
- 'e1:package_setscene'] + ['e1:' + x for x in self.alltasks]
- expected.remove('e1:package')
- self.assertEqual(set(tasks), set(expected))
+ def shutdown(self, tempdir):
+ # Wait for the hashserve socket to disappear else we'll see races with the tempdir cleanup
+ while os.path.exists(tempdir + "/hashserve.sock"):
+ time.sleep(0.5)
diff --git a/poky/bitbake/lib/bb/ui/buildinfohelper.py b/poky/bitbake/lib/bb/ui/buildinfohelper.py
index f2151c2..5cbca97 100644
--- a/poky/bitbake/lib/bb/ui/buildinfohelper.py
+++ b/poky/bitbake/lib/bb/ui/buildinfohelper.py
@@ -646,6 +646,9 @@
Target_Installed_Package.objects.create(target = target_obj, package = packagedict[p]['object'])
packagedeps_objs = []
+ pattern_so = re.compile(r'.*\.so(\.\d*)?$')
+ pattern_lib = re.compile(r'.*\-suffix(\d*)?$')
+ pattern_ko = re.compile(r'^kernel-module-.*')
for p in packagedict:
for (px,deptype) in packagedict[p]['depends']:
if deptype == 'depends':
@@ -654,6 +657,13 @@
tdeptype = Package_Dependency.TYPE_TRECOMMENDS
try:
+ # Skip known non-package objects like libraries and kernel modules
+ if pattern_so.match(px) or pattern_lib.match(px):
+ logger.info("Toaster does not add library file dependencies to packages (%s,%s)", p, px)
+ continue
+ if pattern_ko.match(px):
+ logger.info("Toaster does not add kernel module dependencies to packages (%s,%s)", p, px)
+ continue
packagedeps_objs.append(Package_Dependency(
package = packagedict[p]['object'],
depends_on = packagedict[px]['object'],