master: subtree updates oct 12 2023
poky: e444d2bed0..8d0ba08aa6:
Alassane Yattara (1):
bitbake: toaster: Monitoring - implement Django logging system
Alexander Kanavin (1):
nghttp2: update 1.56.0 -> 1.57.0
Alexis Lothoré (2):
oeqa/utils/gitarchive: fix tag pattern searching
oeqa/utils/gitarchive: ensure tag matches regex before getting its fields
Andreas Cord-Landwehr (1):
wayland: consider pkgconfig sysroot for pkgdatadir
André Draszik (1):
wic: fix a typo in help output
Archana Polampalli (1):
gstreamer: upgrade 1.22.5 -> 1.22.6
Bruce Ashfield (1):
linux-yocto/6.5: integrate fixes for sanity issues
Chris Laplante (1):
recipetool/create_buildsys_python: use importlib instead of imp
Daniel McGregor (2):
meson: upgrade 1.2.1 -> 1.2.2
libtirpc: conditionally enable gssapi
Daniel Semkowicz (3):
uboot-extlinux-config.bbclass: Remove repeated space character
uboot-extlinux-config.bbclass: Uppercase "menu title" entry
uboot-extlinux-config.bbclass: Add menu title configuration
Fabio Estevam (1):
u-boot: Upgrade to 2023.10
Fahad Arslan (1):
linux-firmware: create separate package for cirrus and cnm firmwares
Jermain Horsman (1):
scripts/oe-setup-layers: Update how to determine if directory is git repo
Jose Quaresma (4):
curl: 8.3.0 -> 8.4.0
go: update 1.20.7 -> 1.20.8
go: update 1.20.8 -> 1.20.9
go: update 1.20.9 -> 1.20.10
Joshua Watt (6):
bitbake: hashserv: Add remove API
bitbake: bitbake-hashclient: Add remove subcommand
bitbake: hashserv: Extend get_outhash API to optionally include unihash
bitbake: hashserv: Add API to clean unused entries
bitbake: bitbake-hashclient: Add clean-unused subcommand
overview: Add note about non-reproducibility side effects
Julien Stephan (4):
bitbake.conf: include bblock.conf
sstatesig: add a new info level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK
scripts/bblock: add a script to lock/unlock recipes
oeqa/selftest/bblock: add self test for bblock tool
Khem Raj (1):
python3-docutils: Rename utilities to their canonical names
Lee Chee Yang (1):
migration-guides: add release notes for 4.0.13
Marcus Flyckt (1):
devtool/upgrade: check all git config locations
Markus Volk (2):
mesa: Upgrade 23.1.8 -> 23.2.1
gtk: Upgrade 4.12.1 -> 4.12.3
Marlon Rodriguez Garcia (1):
bitbake: toaster: update selenium version and code syntax
Martijn de Gouw (1):
busybox: Set PATH in syslog initscript
Michael Opdenacker (1):
ref-manual: releases.svg: Scarthgap is now version 5.0
Mikko Rapeli (2):
qemurunner.py: detect login prompt without without utf-8 conversion
openssh: drop sudo from ptest dependencies
Peter Kjellerstedt (4):
externalsrc.bbclass: Support specifying patterns in CONFIGURE_FILES
autotools.bbclass: Add *.m4 to CONFIGURE_FILES
packages.bbclass: Correct the check for conflicts with renamed packages
cmake.bbclass: Add *.cmake to CONFIGURE_FILES
Quentin Schulz (1):
uboot-extlinux-config.bbclass: fix missed override syntax migration
Randy MacLeod (1):
strace: skip so_peerpidfd test
Rasmus Villemoes (1):
openssh: update sshd_check_keys script to make use of 'sshd -G'
Richard Purdie (14):
wic: Add console parameters to qemux86 canned-wks
qemurunner: Skip conversion from/to binary data for logfile
qemurunner: Use backslashreplace with utf8 to make invalid characters clear
qemurunner: Log the second serial console as well as the first
qemurunner: Show both the login console log and all logging upon failure
oeqa/concurrencytest: Remove invalid buffering option
bitbake: selftest/fetch: Ensure top level directory timestamp doesn't break test
cryptodev: Update to latest git for 6.5 kernel fixes
qemux86/qemuarm: Drop kernel version overrides
oeqa/qemurunner: Add newlines serial workaround
runqemu/qemurunner: Use nodelay with tcp serial connections
oeqa/qemurunner: Add extra logging when console doesn't appear
poky/poky-tiny: Switch to the 6.5 kernel
oeqa/qemurunner: Ensure we retry after BrokenPipeError
Ross Burton (12):
python3-numpy: remove obsolete reproducible workaround
libx11: upgrade to 1.8.7
libxpm: upgrade to 3.5.17
qemuboot: reduce default size of software I/O translation buffer
sysvinit-initab: rewrite loop to generate inittab
ttyrun: add new recipe
sysvinit-inittab: use ttyrun to run getty only if the terminal exists
busybox: use ttyrun to run getty only if the terminal exists
oeqa/selftest: don't skip test_read_only_image on qemuarm64
meta/conf/machine: remove SERIAL_CONSOLES_CHECK
busybox-inittab: fix console handling
oeqa/runtime/_qemutiny: rewrite test to be functional
Sundeep KOKKONDA (1):
rust: reproducibility issue fix
Trevor Gamblin (1):
dev-manual: fix testimage usage instructions
Yogita Urade (1):
qemu: fix CVE-2023-42467
meta-openembedded: ea42cec2ec..62039a2c33:
Ahmad Fatoum (1):
signing.bbclass: don't export OPENSSL environment variables globally
Beniamin Sandu (1):
libnet: upgrade version v1.2 -> v1.3
Benjamin Bara (1):
libvpx: upgrade 1.13.0 -> 1.13.1
Chen Qi (1):
libblockdev: fix QA error in case of multilib
Christophe Vu-Brugier (3):
libnvme: upgrade 1.5 -> 1.6
nvme-cli: upgrade 2.5 -> 2.6
libnvme: apply patch already upstream to fix build with musl
Clément Péron (1):
Revert "protobuf: stage protoc binary to sysroot"
Daniel Klauer (1):
graphviz: Fix build to not use $prefix as search dir
Denys Zagorui (1):
libbpf: add arm, powerpc and mips64 to COMPATIBLE_HOST
Fabien Thomas (8):
meta-filesystems/layer.conf : Add meta-networking dependency
Add static-passwd and static-group files
Add static-passwd and static-group files
Add static-passwd and static-group files
Add static-passwd and static-group files
Add static-passwd and static-group files
Add static-passwd and static-group files
Add static-passwd and static-group files
Gianfranco Costamagna (6):
dlt-daemon: Make it work without systemd
dlt-daemon: Enable experimental coredumphandler feature
dlt-daemon: update patch 544.patch
dlt-daemon: do not disable dlt-system build when systemd is set to off
dlt-daemon: Add an additional fix for non-systemd builds
cpprestsdk: fix typo in comment, tag is actually 2.0.18
Jeffrey Pautler (1):
bolt: change product name used for CVE checking
Joe Slater (1):
nginx: add configure option
Johannes Kauffmann (1):
open62541: add Backport status and link to patch
Jörg Sommer (1):
collectd: Use https in SRC_URI, add HOMEPAGE
Khem Raj (16):
python3-pyroute2: Add missing dependency on sqlite3 for ptests
python3-pylint: Upgrade to 3.0.0
python3-lz4: use python3-unittest-automake-output
minicoredumber: Fix ptest reporting
images: Inherit from core-image-base
images: Delete layer specific base images
images: Rename <layer>-image to <layer>-image-all
images: Rename ptest images to rhyme with oe-core ptest images
ptest-image: Switch to using core-image-minimal
stressapptest: Upgrade to 1.0.11 release
klibc: Upgrade to 2.0.13 release
libnvme: Fix test builds on musl
kernel-selftest: Build bpf tests again
ptest-packagelists-meta-oe: Add kernel-selftest to x86/x86-64 images
kernel-selftest: Copy the .config from kernel build
kernel-selftest: Use clang options when clang is available
Markus Volk (3):
pugixml: Update 1.13 -> 1.14
pipewire: Upgrade 0.3.80 -> 0.3.81
gnome-control-center: Fix polkit gettext issue
Martin Jansa (1):
opencv: Fix build with protobuf v22 and dnn enabled
Mickael RAMILISON (1):
python3-rapidjson: add ptest
Pawel Langowski (1):
recipes-connectivity: Add tayga recipe
Philip-Dylan Gleonec (1):
cukinia: Fix license field
Thomas Roos (1):
python3-boto3, python3-botocore: remove recipes
Tom Hochstein (1):
libcamera: Avoid build break in signature recalculation
meta-arm: 95789365f7..e914891eee:
Jon Mason (1):
arm-bsp/u-boot: add recipe for 2023.07.02
meta-security: aca6d4a9e7..3f7d40b0fc:
Rasmus Villemoes (3):
fail2ban: add systemd support
fail2ban: change sqlite3 dependency to python3-sqlite3
fail2ban: add useful recommendations
Change-Id: I93672642f4e0392adc6223fdc4e073910b817bc9
Signed-off-by: Andrew Geissler <geissonator@yahoo.com>
diff --git a/poky/bitbake/lib/toaster/toastermain/logs.py b/poky/bitbake/lib/toaster/toastermain/logs.py
new file mode 100644
index 0000000..f995398
--- /dev/null
+++ b/poky/bitbake/lib/toaster/toastermain/logs.py
@@ -0,0 +1,153 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+
+import logging
+import json
+from pathlib import Path
+from django.http import HttpRequest
+
+BASE_DIR = Path(__file__).resolve(strict=True).parent.parent
+
+
+def log_api_request(request, response, view, logger_name='api'):
+ """Helper function for LogAPIMixin"""
+
+ repjson = {
+ 'view': view,
+ 'path': request.path,
+ 'method': request.method,
+ 'status': response.status_code
+ }
+
+ logger = logging.getLogger(logger_name)
+ logger.info(
+ json.dumps(repjson, indent=4, separators=(", ", " : "))
+ )
+
+
+def log_view_mixin(view):
+ def log_view_request(*args, **kwargs):
+ # get request from args else kwargs
+ request = None
+ if len(args) > 0:
+ for req in args:
+ if isinstance(req, HttpRequest):
+ request = req
+ break
+ elif request is None:
+ request = kwargs.get('request')
+
+ response = view(*args, **kwargs)
+ log_api_request(
+ request, response, request.resolver_match.view_name, 'toaster')
+ return response
+ return log_view_request
+
+
+
+class LogAPIMixin:
+ """Logs API requests
+
+ tested with:
+ - APIView
+ - ModelViewSet
+ - ReadOnlyModelViewSet
+ - GenericAPIView
+
+ Note: you can set `view_name` attribute in View to override get_view_name()
+ """
+
+ def get_view_name(self):
+ if hasattr(self, 'view_name'):
+ return self.view_name
+ return super().get_view_name()
+
+ def finalize_response(self, request, response, *args, **kwargs):
+ log_api_request(request, response, self.get_view_name())
+ return super().finalize_response(request, response, *args, **kwargs)
+
+
+LOGGING_SETTINGS = {
+ 'version': 1,
+ 'disable_existing_loggers': False,
+ 'filters': {
+ 'require_debug_false': {
+ '()': 'django.utils.log.RequireDebugFalse'
+ }
+ },
+ 'formatters': {
+ 'datetime': {
+ 'format': '%(asctime)s %(levelname)s %(message)s'
+ },
+ 'verbose': {
+ 'format': '{levelname} {asctime} {module} {name}.{funcName} {process:d} {thread:d} {message}',
+ 'datefmt': "%d/%b/%Y %H:%M:%S",
+ 'style': '{',
+ },
+ 'api': {
+ 'format': '\n{levelname} {asctime} {name}.{funcName}:\n{message}',
+ 'style': '{'
+ }
+ },
+ 'handlers': {
+ 'mail_admins': {
+ 'level': 'ERROR',
+ 'filters': ['require_debug_false'],
+ 'class': 'django.utils.log.AdminEmailHandler'
+ },
+ 'console': {
+ 'level': 'DEBUG',
+ 'class': 'logging.StreamHandler',
+ 'formatter': 'datetime',
+ },
+ 'file_django': {
+ 'level': 'INFO',
+ 'class': 'logging.handlers.TimedRotatingFileHandler',
+ 'filename': BASE_DIR / 'logs/django.log',
+ 'when': 'D', # interval type
+ 'interval': 1, # defaults to 1
+ 'backupCount': 10, # how many files to keep
+ 'formatter': 'verbose',
+ },
+ 'file_api': {
+ 'level': 'INFO',
+ 'class': 'logging.handlers.TimedRotatingFileHandler',
+ 'filename': BASE_DIR / 'logs/api.log',
+ 'when': 'D',
+ 'interval': 1,
+ 'backupCount': 10,
+ 'formatter': 'verbose',
+ },
+ 'file_toaster': {
+ 'level': 'INFO',
+ 'class': 'logging.handlers.TimedRotatingFileHandler',
+ 'filename': BASE_DIR / 'logs/toaster.log',
+ 'when': 'D',
+ 'interval': 1,
+ 'backupCount': 10,
+ 'formatter': 'verbose',
+ },
+ },
+ 'loggers': {
+ 'django.request': {
+ 'handlers': ['file_django', 'console'],
+ 'level': 'WARN',
+ 'propagate': True,
+ },
+ 'django': {
+ 'handlers': ['file_django', 'console'],
+ 'level': 'WARNING',
+ 'propogate': True,
+ },
+ 'toaster': {
+ 'handlers': ['file_toaster'],
+ 'level': 'INFO',
+ 'propagate': False,
+ },
+ 'api': {
+ 'handlers': ['file_api'],
+ 'level': 'INFO',
+ 'propagate': False,
+ }
+ }
+}
diff --git a/poky/bitbake/lib/toaster/toastermain/settings.py b/poky/bitbake/lib/toaster/toastermain/settings.py
index 609c85d..b083cf5 100644
--- a/poky/bitbake/lib/toaster/toastermain/settings.py
+++ b/poky/bitbake/lib/toaster/toastermain/settings.py
@@ -9,6 +9,8 @@
# Django settings for Toaster project.
import os
+from pathlib import Path
+from toastermain.logs import LOGGING_SETTINGS
DEBUG = True
@@ -186,7 +188,13 @@
'django.template.loaders.app_directories.Loader',
#'django.template.loaders.eggs.Loader',
],
- 'string_if_invalid': InvalidString("%s"),
+ # https://docs.djangoproject.com/en/4.2/ref/templates/api/#how-invalid-variables-are-handled
+ # Generally, string_if_invalid should only be enabled in order to debug
+ # a specific template problem, then cleared once debugging is complete.
+ # If you assign a value other than '' to string_if_invalid,
+ # you will experience rendering problems with these templates and sites.
+ # 'string_if_invalid': InvalidString("%s"),
+ 'string_if_invalid': "",
'debug': DEBUG,
},
},
@@ -242,6 +250,9 @@
'django.contrib.humanize',
'bldcollector',
'toastermain',
+
+ # 3rd-lib
+ "log_viewer",
)
@@ -302,43 +313,22 @@
# the site admins on every HTTP 500 error when DEBUG=False.
# See http://docs.djangoproject.com/en/dev/topics/logging for
# more details on how to customize your logging configuration.
-LOGGING = {
- 'version': 1,
- 'disable_existing_loggers': False,
- 'filters': {
- 'require_debug_false': {
- '()': 'django.utils.log.RequireDebugFalse'
- }
- },
- 'formatters': {
- 'datetime': {
- 'format': '%(asctime)s %(levelname)s %(message)s'
- }
- },
- 'handlers': {
- 'mail_admins': {
- 'level': 'ERROR',
- 'filters': ['require_debug_false'],
- 'class': 'django.utils.log.AdminEmailHandler'
- },
- 'console': {
- 'level': 'DEBUG',
- 'class': 'logging.StreamHandler',
- 'formatter': 'datetime',
- }
- },
- 'loggers': {
- 'toaster' : {
- 'handlers': ['console'],
- 'level': 'DEBUG',
- },
- 'django.request': {
- 'handlers': ['console'],
- 'level': 'WARN',
- 'propagate': True,
- },
- }
-}
+LOGGING = LOGGING_SETTINGS
+
+# Build paths inside the project like this: BASE_DIR / 'subdir'.
+BASE_DIR = Path(__file__).resolve(strict=True).parent.parent
+
+# LOG VIEWER
+# https://pypi.org/project/django-log-viewer/
+LOG_VIEWER_FILES_PATTERN = '*.log*'
+LOG_VIEWER_FILES_DIR = os.path.join(BASE_DIR, 'logs')
+LOG_VIEWER_PAGE_LENGTH = 25 # total log lines per-page
+LOG_VIEWER_MAX_READ_LINES = 100000 # total log lines will be read
+LOG_VIEWER_PATTERNS = ['INFO', 'DEBUG', 'WARNING', 'ERROR', 'CRITICAL']
+
+# Optionally you can set the next variables in order to customize the admin:
+LOG_VIEWER_FILE_LIST_TITLE = "Logs list"
+
if DEBUG and SQL_DEBUG:
LOGGING['loggers']['django.db.backends'] = {
diff --git a/poky/bitbake/lib/toaster/toastermain/urls.py b/poky/bitbake/lib/toaster/toastermain/urls.py
index 0360302..3be46fc 100644
--- a/poky/bitbake/lib/toaster/toastermain/urls.py
+++ b/poky/bitbake/lib/toaster/toastermain/urls.py
@@ -28,6 +28,8 @@
# url(r'^admin/doc/', include('django.contrib.admindocs.urls')),
+ url(r'^logs/', include('log_viewer.urls')),
+
# This is here to maintain backward compatibility and will be deprecated
# in the future.
url(r'^orm/eventfile$', bldcollector.views.eventfile),