poky: subtree update:745e38ff0f..81f9e815d3
Adrian Bunk (6):
openssl: Upgrade 1.1.1c -> 1.1.1d
glib-2.0: Upgrade 2.60.6 -> 2.60.7
lttng-modules: Upgrade 2.10.10 -> 2.10.11
lttng-ust: Upgrade 2.10.4 -> 2.10.5
squashfs-tools: Remove UPSTREAM_CHECK_COMMITS
libmpc: Remove dead UPSTREAM_CHECK_URI
Alexander Kanavin (2):
runqemu: decouple gtk and gl options
strace: add a timeout for running ptests
Alistair Francis (1):
gdb: Mark gdbserver as ALLOW_EMPTY for riscv32
Andre McCurdy (9):
busybox: drop unused mount.busybox and umount.busybox wrappers
busybox: drop inittab from SRC_URI ( now moved to busybox-inittab )
busybox-inittab: minor formatting tweaks
base-files: drop legacy empty file /etc/default/usbd
busybox: rcS and rcK should not be writeable by everyone
ffmpeg: add PACKAGECONFIG controls for alsa and zlib (enable by default)
libwebp: apply ARM specific config options to big endian ARM
initscripts: enable alignment.sh init script for big endian ARM
libunwind: apply configure over-ride to both big and little endian ARM
Andrew F. Davis (4):
libepoxy: Disable x11 when not building for x11
cogl: Set depends to the virtual needed not explicitly on Mesa
gtk+3: Set depends to the virtual needed not explicitly on Mesa
weston: Set depends to the virtual needed not explicitly on Mesa
Armin Kuster (1):
gcc: Security fix for CVE-2019-15847
Changhyeok Bae (1):
iw: upgrade to 5.3
Changqing Li (2):
classextend.py: don't extend file for file dependency
report-error.bbclass: add local.conf/auto.conf into error report
Chen Qi (1):
python-numpy: fix build for libn32
Daniel Gomez (1):
lttng-modules: Add missing SRCREV_FORMAT
Diego Rondini (1):
initramfs-framework: support PARTLABEL option
Dmitry Eremin-Solenikov (7):
image-uefi.conf: add config file holding configuration for UEFI images
grub-bootconf: switch to image-uefi.conf
grub-efi: switch to image-uefi.conf
grub-efi.bbclass: switch to image-uefi.conf
systemd-boot: switch to image-uefi.conf
systemd-boot.bbclass: switch to image-uefi.conf
live-vm-common.bbclass: provide efi population functions for live images
Hector Palacios (1):
udev-extraconf: skip mounting partitions already mounted by systemd
Henning Schild (6):
oe-git-proxy: allow setting SOCAT from outside
oeqa: add case for oe-git-proxy
Revert "oe-git-proxy: Avoid resolving NO_PROXY against local files"
oe-git-proxy: disable shell pathname expansion for the whole script
oe-git-proxy: NO_PROXY suffix matching without wildcard for match_host
oe-git-proxy: fix dash "Bad substitution"
Hongxu Jia (1):
elfutils: 0.176 -> 0.177
Jack Mitchell (1):
iptables: add systemd helper unit to load/restore rules
Jaewon Lee (1):
populate_sdk_ext: Introduce mechanism to keep nativesdk* sstate in esdk
Jason Wessel (1):
gnupg: Extend -native wrapper to fix gpgme-native's gpgconf problems
Jiang Lu (2):
glib-networking:enable glib-networking build as native package
libsoup:enable libsoup build as native package
Joshua Watt (4):
sstatesig: Update server URI
Remove SSTATE_HASHEQUIV_SERVER
bitbake: bitbake: Rework hash equivalence
classes/archiver: Fix WORKDIR for shared source
Kai Kang (1):
systemd: provides ${base_sbindir}/udevadm
Khem Raj (10):
ptrace: Drop ptrace aid for musl/ppc
elfutils: Fix build on ppc/musl
cogl: Do not depend PN-dev on empty PN
musl: Update to latest master
glibc: Move DISTRO_FEATURE specific do_install code for target recipe only
populate_sdk_base.bbclass: nativesdk-glibc-locale is required on musl too
nativesdk.bbclass: Clear out LIBCEXTENSION and ABIEXTENSION
openssl: Enable os option for with-rand-seed as well
weston-init: Add possibility to run weston as non-root user
layer.conf: Remove weston-conf from SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS
Li Zhou (1):
qemu: Security Advisory - qemu - CVE-2019-15890
Limeng (1):
tune-cortexa57-cortexa53: add tunes for ARM Cortex-A53-Cortex-A57
Martin Jansa (2):
perf: fix build on kernels which don't have ${S}/tools/include/linux/bits.h
bitbake: Revert "bitbake: cooker: Ensure bbappends are found in stable order"
Maxime Roussin-Bélanger (1):
meta: add missing descriptions and homepage in bsp
Mikko Rapeli (2):
busybox.inc: handle empty DEBUG_PREFIX_MAP
bitbake: svn fetcher: allow "svn propget svn:externals" to fail
Nathan Rossi (7):
resulttool: Handle multiple series containing ptestresults
gcc-cross.inc: Process binaries in build dir to be relocatable
oeqa/core/case.py: Add OEPTestResultTestCase for ptestresult helpers
oeqa/selftest: Rework toolchain tests to use OEPTestResultTestCase
glibc-testsuite: SkipRecipe if libc is not glibc
cmake: 3.15.2 -> 3.15.3
meson.bbclass: Handle microblaze* mapping to cpu family
Oleksandr Kravchuk (5):
python3-pygobject: update to 3.34.0
font-util: update to 1.3.2
expat: update to 2.2.8
curl: update to 7.66.0
python3-dbus: update to 1.2.12
Otavio Salvador (1):
mesa: Upgrade 19.1.1 -> 19.1.6
Peter Kjellerstedt (3):
glibc: Make it build without ldconfig in DISTRO_FEATURES
package_rpm.bbclass: Remove a misleading bb.note()
tzdata: Correct the packaging of /etc/localtime and /etc/timezone
Quentin Schulz (1):
externalsrc: stop rebuilds of 2+ externalsrc recipes sharing the same git repo
Randy MacLeod (4):
valgrind: enable ~500 more ptests
valgrind: make a few more ptests pass
valgrind: ptest improvements to run-ptest and more
valgrind: disable 256 ptests for aarch64
Richard Purdie (8):
bitbake: runqueue/siggen: Optimise hash equiv queries
runqemu: Mention snapshot in the help output
initramfs-framework: support PARTLABEL option
systemd: Handle slow to boot mips hwdb update timeouts
meta-extsdk: Either an sstate task is a proper task or it isn't
oeqa/concurrenttest: Use ionice to delete build directories
bitbake: utils: Add ionice option to prunedir
build-appliance-image: Update to master head revision
Robert Yang (2):
conf/multilib.conf: Add ovmf to NON_MULTILIB_RECIPES
bitbake: runqueue: validate_hashes(): currentcount should be a number
Ross Burton (16):
libtasn1: fix build with api-documentation enabled
gstreamer1.0-libav: enable gtk-doc again
python3: handle STAGING_LIBDIR/INCDIR being unset
mesa: no need to depend on target python3
adwaita-icon-theme: fix rare install race
oeqa/selftest/wic: improve assert messages in test_fixed_size
oeqa/selftest/imagefeatures: dump the JSON if it can't be parsed
libical: upgrade to 3.0.6
acpica: upgrade 20190509 -> 20190816
gdk-pixbuf: upgrade 2.38.1 -> 2.38.2
piglit: upgrade to latest revision
libinput: upgrade 1.14.0 -> 1.14.1
rootfs-postcommands: check /etc/gconf exists before working on it
systemd-systemctl-native: don't care about line endings
opkg-utils: respect SOURCE_DATE_EPOCH when building ipkgs
bitbake: fetch2/git: add git-lfs toggle option
Scott Murray (1):
systemd: upgrade to 243
Stefan Ghinea (1):
ghostscript: CVE-2019-14811, CVE-2019-14817
Tim Blechmann (1):
icecc: blacklist pixman
Yeoh Ee Peng (3):
bitbake: bitbake-layers: show-recipes: Show recipes only
bitbake: bitbake-layers: show-recipes: Select recipes from selected layer
bitbake: bitbake-layers: show-recipes: Enable bare output
Yi Zhao (3):
screen: add /etc/screenrc as global config file
nfs-utils: fix nfs mount error on 32bit nfs server
grub: remove diffutils and freetype runtime dependencies
Zang Ruochen (2):
btrfs-tools:upgrade 5.2.1 -> 5.2.2
timezone:upgrade 2019b -> 2019c
Change-Id: I1ec24480a8964e474cd99d60a0cb0975e49b46b8
Signed-off-by: Brad Bishop <bradleyb@fuzziesquirrel.com>
diff --git a/poky/bitbake/lib/hashserv/__init__.py b/poky/bitbake/lib/hashserv/__init__.py
index eb03c32..c331862 100644
--- a/poky/bitbake/lib/hashserv/__init__.py
+++ b/poky/bitbake/lib/hashserv/__init__.py
@@ -3,203 +3,21 @@
# SPDX-License-Identifier: GPL-2.0-only
#
-from http.server import BaseHTTPRequestHandler, HTTPServer
-import contextlib
-import urllib.parse
+from contextlib import closing
+import re
import sqlite3
-import json
-import traceback
-import logging
-import socketserver
-import queue
-import threading
-import signal
-import socket
-import struct
-from datetime import datetime
-logger = logging.getLogger('hashserv')
+UNIX_PREFIX = "unix://"
-class HashEquivalenceServer(BaseHTTPRequestHandler):
- def log_message(self, f, *args):
- logger.debug(f, *args)
+ADDR_TYPE_UNIX = 0
+ADDR_TYPE_TCP = 1
- def opendb(self):
- self.db = sqlite3.connect(self.dbname)
- self.db.row_factory = sqlite3.Row
- self.db.execute("PRAGMA synchronous = OFF;")
- self.db.execute("PRAGMA journal_mode = MEMORY;")
- def do_GET(self):
- try:
- if not self.db:
- self.opendb()
-
- p = urllib.parse.urlparse(self.path)
-
- if p.path != self.prefix + '/v1/equivalent':
- self.send_error(404)
- return
-
- query = urllib.parse.parse_qs(p.query, strict_parsing=True)
- method = query['method'][0]
- taskhash = query['taskhash'][0]
-
- d = None
- with contextlib.closing(self.db.cursor()) as cursor:
- cursor.execute('SELECT taskhash, method, unihash FROM tasks_v2 WHERE method=:method AND taskhash=:taskhash ORDER BY created ASC LIMIT 1',
- {'method': method, 'taskhash': taskhash})
-
- row = cursor.fetchone()
-
- if row is not None:
- logger.debug('Found equivalent task %s', row['taskhash'])
- d = {k: row[k] for k in ('taskhash', 'method', 'unihash')}
-
- self.send_response(200)
- self.send_header('Content-Type', 'application/json; charset=utf-8')
- self.end_headers()
- self.wfile.write(json.dumps(d).encode('utf-8'))
- except:
- logger.exception('Error in GET')
- self.send_error(400, explain=traceback.format_exc())
- return
-
- def do_POST(self):
- try:
- if not self.db:
- self.opendb()
-
- p = urllib.parse.urlparse(self.path)
-
- if p.path != self.prefix + '/v1/equivalent':
- self.send_error(404)
- return
-
- length = int(self.headers['content-length'])
- data = json.loads(self.rfile.read(length).decode('utf-8'))
-
- with contextlib.closing(self.db.cursor()) as cursor:
- cursor.execute('''
- -- Find tasks with a matching outhash (that is, tasks that
- -- are equivalent)
- SELECT taskhash, method, unihash FROM tasks_v2 WHERE method=:method AND outhash=:outhash
-
- -- If there is an exact match on the taskhash, return it.
- -- Otherwise return the oldest matching outhash of any
- -- taskhash
- ORDER BY CASE WHEN taskhash=:taskhash THEN 1 ELSE 2 END,
- created ASC
-
- -- Only return one row
- LIMIT 1
- ''', {k: data[k] for k in ('method', 'outhash', 'taskhash')})
-
- row = cursor.fetchone()
-
- # If no matching outhash was found, or one *was* found but it
- # wasn't an exact match on the taskhash, a new entry for this
- # taskhash should be added
- if row is None or row['taskhash'] != data['taskhash']:
- # If a row matching the outhash was found, the unihash for
- # the new taskhash should be the same as that one.
- # Otherwise the caller provided unihash is used.
- unihash = data['unihash']
- if row is not None:
- unihash = row['unihash']
-
- insert_data = {
- 'method': data['method'],
- 'outhash': data['outhash'],
- 'taskhash': data['taskhash'],
- 'unihash': unihash,
- 'created': datetime.now()
- }
-
- for k in ('owner', 'PN', 'PV', 'PR', 'task', 'outhash_siginfo'):
- if k in data:
- insert_data[k] = data[k]
-
- cursor.execute('''INSERT INTO tasks_v2 (%s) VALUES (%s)''' % (
- ', '.join(sorted(insert_data.keys())),
- ', '.join(':' + k for k in sorted(insert_data.keys()))),
- insert_data)
-
- logger.info('Adding taskhash %s with unihash %s', data['taskhash'], unihash)
-
- self.db.commit()
- d = {'taskhash': data['taskhash'], 'method': data['method'], 'unihash': unihash}
- else:
- d = {k: row[k] for k in ('taskhash', 'method', 'unihash')}
-
- self.send_response(200)
- self.send_header('Content-Type', 'application/json; charset=utf-8')
- self.end_headers()
- self.wfile.write(json.dumps(d).encode('utf-8'))
- except:
- logger.exception('Error in POST')
- self.send_error(400, explain=traceback.format_exc())
- return
-
-class ThreadedHTTPServer(HTTPServer):
- quit = False
-
- def serve_forever(self):
- self.requestqueue = queue.Queue()
- self.handlerthread = threading.Thread(target=self.process_request_thread)
- self.handlerthread.daemon = False
-
- self.handlerthread.start()
-
- signal.signal(signal.SIGTERM, self.sigterm_exception)
- super().serve_forever()
- os._exit(0)
-
- def sigterm_exception(self, signum, stackframe):
- self.server_close()
- os._exit(0)
-
- def server_bind(self):
- HTTPServer.server_bind(self)
- self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack('ii', 1, 0))
-
- def process_request_thread(self):
- while not self.quit:
- try:
- (request, client_address) = self.requestqueue.get(True)
- except queue.Empty:
- continue
- if request is None:
- continue
- try:
- self.finish_request(request, client_address)
- except Exception:
- self.handle_error(request, client_address)
- finally:
- self.shutdown_request(request)
- os._exit(0)
-
- def process_request(self, request, client_address):
- self.requestqueue.put((request, client_address))
-
- def server_close(self):
- super().server_close()
- self.quit = True
- self.requestqueue.put((None, None))
- self.handlerthread.join()
-
-def create_server(addr, dbname, prefix=''):
- class Handler(HashEquivalenceServer):
- pass
-
- db = sqlite3.connect(dbname)
+def setup_database(database, sync=True):
+ db = sqlite3.connect(database)
db.row_factory = sqlite3.Row
- Handler.prefix = prefix
- Handler.db = None
- Handler.dbname = dbname
-
- with contextlib.closing(db.cursor()) as cursor:
+ with closing(db.cursor()) as cursor:
cursor.execute('''
CREATE TABLE IF NOT EXISTS tasks_v2 (
id INTEGER PRIMARY KEY AUTOINCREMENT,
@@ -220,11 +38,56 @@
UNIQUE(method, outhash, taskhash)
)
''')
- cursor.execute('CREATE INDEX IF NOT EXISTS taskhash_lookup ON tasks_v2 (method, taskhash)')
- cursor.execute('CREATE INDEX IF NOT EXISTS outhash_lookup ON tasks_v2 (method, outhash)')
+ cursor.execute('PRAGMA journal_mode = WAL')
+ cursor.execute('PRAGMA synchronous = %s' % ('NORMAL' if sync else 'OFF'))
- ret = ThreadedHTTPServer(addr, Handler)
+ # Drop old indexes
+ cursor.execute('DROP INDEX IF EXISTS taskhash_lookup')
+ cursor.execute('DROP INDEX IF EXISTS outhash_lookup')
- logger.info('Starting server on %s\n', ret.server_port)
+ # Create new indexes
+ cursor.execute('CREATE INDEX IF NOT EXISTS taskhash_lookup_v2 ON tasks_v2 (method, taskhash, created)')
+ cursor.execute('CREATE INDEX IF NOT EXISTS outhash_lookup_v2 ON tasks_v2 (method, outhash)')
- return ret
+ return db
+
+
+def parse_address(addr):
+ if addr.startswith(UNIX_PREFIX):
+ return (ADDR_TYPE_UNIX, (addr[len(UNIX_PREFIX):],))
+ else:
+ m = re.match(r'\[(?P<host>[^\]]*)\]:(?P<port>\d+)$', addr)
+ if m is not None:
+ host = m.group('host')
+ port = m.group('port')
+ else:
+ host, port = addr.split(':')
+
+ return (ADDR_TYPE_TCP, (host, int(port)))
+
+
+def create_server(addr, dbname, *, sync=True):
+ from . import server
+ db = setup_database(dbname, sync=sync)
+ s = server.Server(db)
+
+ (typ, a) = parse_address(addr)
+ if typ == ADDR_TYPE_UNIX:
+ s.start_unix_server(*a)
+ else:
+ s.start_tcp_server(*a)
+
+ return s
+
+
+def create_client(addr):
+ from . import client
+ c = client.Client()
+
+ (typ, a) = parse_address(addr)
+ if typ == ADDR_TYPE_UNIX:
+ c.connect_unix(*a)
+ else:
+ c.connect_tcp(*a)
+
+ return c