poky: subtree update:26ae42ded7..5951cbcabe
Alex Kiernan (1):
recipetool: Fix list concatenation when using edit
Alexander Kanavin (4):
apr-util: make gdbm optional
gobject-introspection: add a patch to fix a build race
icu: merge .inc into main recipe
icu: make filtered data generation optional, serial and off by default
Alexandru N. Onea (3):
bitbake: perforce: add basic progress handler for perforce
bitbake: perforce: add local path handling SRC_URI options
bitbake: bitbake-user-manual: update perforce fetcher docs
Andreas M?ller (1):
meson.bbclass: avoid unexpected operating-system names
Andreas Müller (6):
boost: Add upstream patch to fix build on depending projects
libinput: upgrade 1.15.5 -> 1.15.6
sqlite3: upgrade 3.32.2 -> 3.32.3
desktop-file-utils: upgrade 0.24 -> 0.26
file: upgrade 5.38 -> 5.39
ffmpeg: upgrade 4.2.3 -> 4.3
Andrej Valek (1):
oeqa/runtime/cases/ptest: Make output content path absolute
Andrew Geissler (1):
meson: backport library ordering fix
Armin Kuster (1):
libuv: move from meta-oe to core for bind update
Arthur She (1):
igt-gpu-tools: add new package
Changqing Li (1):
mime.bbclass: fix post install scriptlet error
Chen Qi (1):
systemd-serialgetty: do not use BindsTo
Daniel McGregor (3):
sign_rpm.bbclass: ignore thread count
systemd-conf: Accept MTU from DHCP
buildhistory-collect-srcrevs: sort directories
He Zhe (1):
ltp: Fix copy_file_rang02 for 32-bit arches
Hongxu Jia (1):
libmodulemd: switch branch master -> main
Jacob Kroon (5):
bitbake: lib/bb/utils.py: Do not preserve TERM in the environment
bitbake: bitbake-user-manual: Remove TERM from BB_HASHBASE_WHITELIST example
bitbake.conf: Remove TERM from default BB_HASHBASE_WHITELIST
grub: Remove native version of grub-efi
distro_alias: Remove unused grub-efi distro aliases
Jens Rehsack (1):
u-boot: avoid blind merging all *.cfg
Joe Slater (1):
systemd: fix CVE-2020-13776
Joshua Watt (5):
sstatesig: Account for all dataCaches being passed
bitbake: bitbake: cache: Fix error message with bad multiconfig
wic: Fix error message when reporting invalid offset
classes/archiver: Create patched archive before configuring
bitbake: cache: Bump cache version
Konrad Weihmann (3):
oeqa/runtime: Add OERequirePackage decorator
bitbake: cookerdata: Add BBFILES_DYNAMIC inverse mode
bitbake: bitbake-user-manual: Add BBFILES_DYNAMIC
Mark Morton (2):
New source files and Makefile update for Test Manual
test-manual: Fixed codeblock formatting
Martin Jansa (1):
net-tools: backport a patch from upstream to use the same ifconfig format as debian/ubuntu
Mingli Yu (3):
python3: add the rdepends for python3-misc
python3: add rdepends for python3-idle
python3-dbusmock: add the missing rdepends
Otavio Salvador (2):
systemd: Sync systemd-serialgetty@.service with upstream
mtd-utils: Fix return value of ubiformat
Ovidiu Panait (2):
dbus-test: Remove EXTRA_OECONF_X configs
dbus,dbus-test: Move common parts to dbus.inc
Paul Barker (2):
bitbake: fetch2/gitsm: Mark srcrev as fetched once all submodules are processed
bitbake: fetch2/gitsm: Make need_update() process submodules
Paul Eggleton (5):
graph-tool: switch to argparse
graph-tool: add filter subcommand
dpkg-native: rebase and reinstate fix for "tar: file changed as we read it"
shadow-sysroot: drop unused SRC_URI checksums
devtool: fix typo
Peter Kjellerstedt (1):
relocatable.bbclass: Avoid an exception if an empty pkgconfig dir exist
Pierre-Jean Texier (3):
diffoscope: upgrade 146 -> 147
ell: upgrade 0.31 -> 0.32
curl: upgrade 7.70.0 -> 7.71.0
Rasmus Villemoes (1):
curl: add debug info
Richard Purdie (15):
buildhistory: Add simplistic file move detection
bitbake: bin/bitbake: Update to next series release version
perl: Fix host specific modules problems
sanity.conf: Require bitbake 1.47.0 as the minimum version
patchelf: Upgrade 0.10 -> 0.11
test-manual: Add SPDX license headers
Makefile: Drop obsolete edison/denzil branch conditionals
bitbake: tests/fetch: Switch from git.infradead.org to a YP mirror
pseudo: Fix attr errors due to incorrect library resolution issues
oeqa/selftest/runcmd: Add better debug for thread count mismatch failures
oeqa/utils/command: Improve stdin handling in runCmd
vulkan-headers: Fix upstream branch deletion issue
recipes: Fix Upstream-Status Accepted -> Backport
scripts/install-buildtools: Update to 3.2 M1 buildtools
scripts/install-buildtools: Handle new format checksum files
Robert P. J. Day (1):
python: use official "pypi.org" URLs for HOMEPAGE
Ross Burton (8):
install-buildtools: fail if an error occurs
install-buildtools: remove hardcoded x86-64 architecture
install-buildtools: add option to disable checksum validation
common-licenses: add BSD-2-Clause-Patent
gstreamer1.0-plugins-bad: add support for vdpau
go-binary-native: add binary Go to bootstrap
tcmode-default: use go-binary-native by default
go-native: merge bb/inc and add comment
Ryan Rowe (1):
python3: fix PGO for non-reproducible biniaries
Sakib Sajal (1):
qemu: uprev v4.2.0 -> v5.0.0
Samuli Piippo (2):
cmake: allow chainloading of the toolchain file
perl: use relative paths in the perl wrapper
Steve Sakoman (1):
buildtools-tarball: export OPENSSL_CONF in environment setup
Tanu Kaskinen (1):
pulseaudio: remove unnecessary libltdl copying
Trevor Gamblin (1):
python3-setuptools: patch entrypoints for faster initialization
Tuomas Salokanto (1):
recipetool: create: fix SRCBRANCH not being passed to params
Valentin Longchamp (2):
tools-profile: disable valgrind for powerpc soft-float
valgrind: disable it for powerpc soft-float
Wang Mingyu (5):
powertop: upgrade 2.12 -> 2.13
man-db: upgrade 2.9.2 -> 2.9.3
valgrind: upgrade 3.16.0 -> 3.16.1
man-pages: upgrade 5.06 -> 5.07
harfbuzz: upgrade 2.6.7 -> 2.6.8
Yi Zhao (2):
iptables: fix invalid symbolic link for ip6tables-apply
iptables: split iptables-apply to its own package
Yongxin Liu (1):
linux-firmware: add ice for Intel E800 series driver
Yuki Hoshino (1):
sysvinit-inittab: Add support for tty devices with 10 or more number.
akuster (9):
bind: update to 9.11.19
adt-manual: Add SPDX license headers
bsp-guide: Add SPDX license headers
brief-yoctoprojectsqa: Add SPDX license headers
dev-manual: Add SPDX License headers
kernel-dev: Add SPDX license headers
profile-manual: Add SPDX licence headers
sdk-manual: Add SPDX license headers
toaster-manaul: Add SPDX license headers
haiqing (1):
libpam: Remove option 'obscure' from common-password
hongxu (1):
kmod: add nativesdk support
zangrc (1):
ethtool:upgrade 5.6 -> 5.7
Signed-off-by: Andrew Geissler <geissonator@yahoo.com>
Change-Id: I1190ca17297b1167286cfc06033e8485396c7cce
diff --git a/poky/documentation/test-manual/test-manual-intro.xml b/poky/documentation/test-manual/test-manual-intro.xml
new file mode 100644
index 0000000..8e2c7cd
--- /dev/null
+++ b/poky/documentation/test-manual/test-manual-intro.xml
@@ -0,0 +1,624 @@
+<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
+"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
+[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
+<!--SPDX-License-Identifier: CC-BY-2.0-UK-->
+
+<chapter id='test-manual-intro'>
+
+<title>The Yocto Project Test Environment Manual</title>
+ <section id='test-welcome'>
+ <title>Welcome</title>
+
+ <para> Welcome to the Yocto Project Test Environment Manual! This manual is a work in
+ progress. The manual contains information about the testing environment used by the
+ Yocto Project to make sure each major and minor release works as intended. All the
+ project’s testing infrastructure and processes are publicly visible and available so
+ that the community can see what testing is being performed, how it’s being done and the
+ current status of the tests and the project at any given time. It is intended that Other
+ organizations can leverage off the process and testing environment used by the Yocto
+ Project to create their own automated, production test environment, building upon the
+ foundations from the project core. </para>
+
+ <para> Currently, the Yocto Project Test Environment Manual has no projected release date.
+ This manual is a work-in-progress and is being initially loaded with information from
+ the <ulink url="">README</ulink> files and notes from key engineers: <itemizedlist>
+ <listitem>
+ <para>
+ <emphasis><filename>yocto-autobuilder2</filename>:</emphasis> This <ulink
+ url="http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder2/tree/README.md"
+ ><filename>README.md</filename></ulink> is the main README which
+ detials how to set up the Yocto Project Autobuilder. The
+ <filename>yocto-autobuilder2</filename> repository represents the Yocto
+ Project's console UI plugin to Buildbot and the configuration necessary to
+ configure Buildbot to perform the testing the project requires. </para>
+ </listitem>
+ <listitem>
+ <para>
+ <emphasis><filename>yocto-autobuilder-helper</filename>:</emphasis> This
+ <ulink
+ url="http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder-helper/tree/README"
+ ><filename>README</filename></ulink> and repository contains Yocto
+ Project Autobuilder Helper scripts and configuration. The
+ <filename>yocto-autobuilder-helper</filename> repository contains the
+ "glue" logic that defines which tests to run and how to run them. As a
+ result, it can be used by any Continuous Improvement (CI) system to run
+ builds, support getting the correct code revisions, configure builds and
+ layers, run builds, and collect results. The code is independent of any CI
+ system, which means the code can work Buildbot, Jenkins, or others. This
+ repository has a branch per release of the project defining the tests to run
+ on a per release basis.</para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ </section>
+
+ <section id='test-yocto-project-autobuilder-overview'>
+ <title>Yocto Project Autobuilder Overview</title>
+
+ <para>The Yocto Project Autobuilder collectively refers to the software, tools, scripts, and
+ procedures used by the Yocto Project to test released software across supported hardware
+ in an automated and regular fashion. Basically, during the development of a Yocto
+ Project release, the Autobuilder tests if things work. The Autobuilder builds all test
+ targets and runs all the tests. </para>
+
+ <para>The Yocto Project uses now uses standard upstream <ulink
+ url="https://docs.buildbot.net/0.9.15.post1/">Buildbot</ulink> (version 9) to drive
+ its integration and testing. Buildbot Nine has a plug-in interface that the Yocto
+ Project customizes using code from the <filename>yocto-autobuilder2</filename>
+ repository, adding its own console UI plugin. The resulting UI plug-in allows you to
+ visualize builds in a way suited to the project's needs.</para>
+
+ <para>A <filename>helper</filename> layer provides configuration and job management through
+ scripts found in the <filename>yocto-autobuilder-helper</filename> repository. The
+ <filename>helper</filename> layer contains the bulk of the build configuration
+ information and is release-specific, which makes it highly customizable on a per-project
+ basis. The layer is CI system-agnostic and contains a number of Helper scripts that can
+ generate build configurations from simple JSON files. <note>
+ <para>The project uses Buildbot for historical reasons but also because many of the
+ project developers have knowledge of python. It is possible to use the outer
+ layers from another Continuous Integration (CI) system such as <ulink
+ url="https://en.wikipedia.org/wiki/Jenkins_(software)">Jenkins</ulink>
+ instead of Buildbot. </para>
+ </note>
+ </para>
+
+ <para> The following figure shows the Yocto Project Autobuilder stack with a topology that
+ includes a controller and a cluster of workers: <imagedata
+ fileref="figures/ab-test-cluster.png" width="4.6in" depth="4.35in" align="center"
+ scalefit="1"/>
+ </para>
+ </section>
+
+ <section id='test-project-tests'>
+ <title>Yocto Project Tests - Types of Testing Overview</title>
+
+ <para>The Autobuilder tests different elements of the project by using thefollowing types of
+ tests: <itemizedlist>
+ <listitem>
+ <para>
+ <emphasis>Build Testing:</emphasis> Tests whether specific configurations
+ build by varying <ulink url="&YOCTO_DOCS_REF_URL;#var-MACHINE"
+ ><filename>MACHINE</filename></ulink>, <ulink
+ url="&YOCTO_DOCS_REF_URL;#var-DISTRO"
+ ><filename>DISTRO</filename></ulink>, other configuration options, and
+ the specific target images being built (or world). Used to trigger builds of
+ all the different test configurations on the Autobuilder. Builds usually
+ cover many different targets for different architectures, machines, and
+ distributions, as well as different configurations, such as different init
+ systems. The Autobuilder tests literally hundreds of configurations and
+ targets. <itemizedlist>
+ <listitem>
+ <para>
+ <emphasis>Sanity Checks During the Build Process:</emphasis>
+ Tests initiated through the <ulink
+ url="&YOCTO_DOCS_REF_URL;#ref-classes-insane"
+ ><filename>insane</filename></ulink> class. These checks
+ ensure the output of the builds are correct. For example, does
+ the ELF architecture in the generated binaries match the target
+ system? ARM binaries would not work in a MIPS system! </para>
+ </listitem>
+ </itemizedlist></para>
+ </listitem>
+ <listitem>
+ <para>
+ <emphasis>Build Performance Testing:</emphasis> Tests whether or not
+ commonly used steps during builds work efficiently and avoid regressions.
+ Tests to time commonly used usage scenarios are run through
+ <filename>oe-build-perf-test</filename>. These tests are run on isolated
+ machines so that the time measurements of the tests are accurate and no
+ other processes interfere with the timing results. The project currently
+ tests performance on two different distributions, Fedora and Ubuntu, to
+ ensure we have no single point of failure and can ensure the different
+ distros work effectively. </para>
+ </listitem>
+ <listitem>
+ <para>
+ <emphasis>eSDK Testing:</emphasis> Image tests initiated through the
+ following command:
+ <literallayout class="monospaced">
+ $ bitbake <replaceable>image</replaceable> -c testsdkext
+ </literallayout>
+ The tests utilize the <filename>testsdkext</filename> class and the
+ <filename>do_testsdkext</filename> task. </para>
+ </listitem>
+ <listitem>
+ <para>
+ <emphasis>Feature Testing:</emphasis> Various scenario-based tests are run
+ through the <ulink url="&YOCTO_DOCS_REF_URL;#testing-and-quality-assurance"
+ >OpenEmbedded Self-Test</ulink> (oe-selftest). We test oe-selftest on
+ each of the main distrubutions we support. </para>
+ </listitem>
+ <listitem>
+ <para>
+ <emphasis>Image Testing:</emphasis> Image tests initiated through the
+ following command:
+ <literallayout class="monospaced">
+ $ bitbake <replaceable>image</replaceable> -c testimage
+ </literallayout>
+ The tests utilize the <ulink
+ url="&YOCTO_DOCS_REF_URL;#ref-classes-testimage*"
+ ><filename>testimage*</filename></ulink> classes and the <ulink
+ url="&YOCTO_DOCS_REF_URL;#ref-tasks-testimage"
+ ><filename>do_testimage</filename></ulink> task. </para>
+ </listitem>
+ <listitem>
+ <para>
+ <emphasis>Layer Testing:</emphasis> The Autobuilder has the possibility to
+ test whether specific layers work with the test of the system. The layers
+ tested may be selected by members of the project. Some key community layers
+ are also tested periodically.</para>
+ </listitem>
+ <listitem>
+ <para>
+ <emphasis>Package Testing:</emphasis> A Package Test (ptest) runs tests
+ against packages built by the OpenEmbedded build system on the target
+ machine. See the "<ulink
+ url="&YOCTO_DOCS_DEV_URL;#testing-packages-with-ptest">Testing Packages
+ With ptest</ulink>" section in the Yocto Project Development Tasks
+ Manual and the "<ulink url="&YOCTO_WIKI_URL;/wiki/Ptest">Ptest</ulink>" Wiki
+ page for more information on Ptest. </para>
+ </listitem>
+ <listitem>
+ <para>
+ <emphasis>SDK Testing:</emphasis> Image tests initiated through the
+ following command:
+ <literallayout class="monospaced">
+ $ bitbake <replaceable>image</replaceable> -c testsdk
+ </literallayout>
+ The tests utilize the <ulink url="&YOCTO_DOCS_REF_URL;#ref-classes-testsdk"
+ ><filename>testsdk</filename></ulink> class and the
+ <filename>do_testsdk</filename> task. </para>
+ </listitem>
+ <listitem>
+ <para>
+ <emphasis>Unit Testing:</emphasis> Unit tests on various components of the
+ system run through <filename>oe-selftest</filename> and <ulink
+ url="&YOCTO_DOCS_REF_URL;#testing-and-quality-assurance"
+ ><filename>bitbake-selftest</filename></ulink>. </para>
+ </listitem>
+ <listitem>
+ <para>
+ <emphasis>Automatic Upgrade Helper:</emphasis> This target tests whether new
+ versions of software are available and whether we can automatically upgrade
+ to those new versions. If so, this target emails the maintainers with a
+ patch to let them know this is possible.</para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ </section>
+
+ <section id='test-test-mapping'>
+ <title>How Tests Map to Areas of Code</title>
+
+ <para>
+ Tests map into the codebase as follows:
+ <itemizedlist>
+ <listitem><para>
+ <emphasis>bitbake-selftest</emphasis>: </para>
+ <para>These tests are self-contained and test BitBake as well as its APIs, which
+ include the fetchers. The tests are located in
+ <filename>bitbake/lib/*/tests</filename>. </para>
+ <para>From within the BitBake repository, run the following:
+ <literallayout class="monospaced">
+ $ bitbake-selftest
+ </literallayout>
+ </para>
+ <para>To skip tests that access the Internet, use the
+ <filename>BB_SKIP_NETTEST</filename> variable when running
+ "bitbake-selftest" as follows:
+ <literallayout class="monospaced">
+ $ BB_SKIP_NETTEST=yes bitbake-selftest
+ </literallayout></para>
+ <para>The default output is quiet and just prints a summary of what was run. To
+ see more information, there is a verbose
+ option:<literallayout class="monospaced">
+ $ bitbake-selftest -v
+ </literallayout></para>
+ <para>Use this option when you wish to skip tests that access the network, which
+ are mostly necessary to test the fetcher modules. To specify individual test
+ modules to run, append the test module name to the "bitbake-selftest"
+ command. For example, to specify the tests for the bb.data.module, run:
+ <literallayout class="monospaced">
+ $ bitbake-selftest bb.test.data.module
+ </literallayout>You
+ can also specify individual tests by defining the full name and module plus
+ the class path of the test, for example:
+ <literallayout class="monospaced">
+ $ bitbake-selftest bb.tests.data.TestOverrides.test_one_override
+ </literallayout></para>
+ <para>The tests are based on <ulink
+ url="https://docs.python.org/3/library/unittest.html">Python
+ unittest</ulink>. </para></listitem>
+ <listitem><para>
+ <emphasis>oe-selftest</emphasis>: <itemizedlist>
+ <listitem>
+ <para>These tests use OE to test the workflows, which include
+ testing specific features, behaviors of tasks, and API unit
+ tests. </para>
+ </listitem>
+ <listitem>
+ <para>The tests can take advantage of parallelism through the "-j"
+ option, which can specify a number of threads to spread the
+ tests across. Note that all tests from a given class of tests
+ will run in the same thread. To parallelize large numbers of
+ tests you can split the class into multiple units.</para>
+ </listitem>
+ <listitem>
+ <para>The tests are based on Python unittest. </para>
+ </listitem>
+ <listitem>
+ <para>The code for the tests resides in
+ <filename>meta/lib/oeqa/selftest/cases/</filename>. </para>
+ </listitem>
+ <listitem>
+ <para>To run all the tests, enter the following command:
+ <literallayout class="monospaced">
+ $ oe-selftest -a
+ </literallayout>
+ </para>
+ </listitem>
+ <listitem>
+ <para>To run a specific test, use the following command form where
+ <replaceable>testname</replaceable> is the name of the
+ specific test:
+ <literallayout class="monospaced">
+ $ oe-selftest -r <replaceable>testname</replaceable>
+ </literallayout>
+ For example, the following command would run the tinfoil getVar
+ API
+ test:<literallayout class="monospaced">
+ $ oe-selftest -r tinfoil.TinfoilTests.test_getvar
+ </literallayout>It
+ is also possible to run a set of tests. For example the
+ following command will run all of the tinfoil
+ tests:<literallayout class="monospaced">
+ $ oe-selftest -r tinfoil
+ </literallayout></para>
+ </listitem>
+ </itemizedlist>
+ </para></listitem>
+ <listitem><para>
+ <emphasis>testimage:</emphasis>
+ <itemizedlist>
+ <listitem><para>
+ These tests build an image, boot it, and run tests
+ against the image's content.
+ </para></listitem>
+ <listitem><para> The code for these tests resides in <filename>meta/lib/oeqa/runtime/cases/</filename>. </para></listitem>
+ <listitem><para>
+ You need to set the
+ <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_CLASSES'><filename>IMAGE_CLASSES</filename></ulink>
+ variable as follows:
+ <literallayout class='monospaced'>
+ IMAGE_CLASSES += "testimage"
+ </literallayout>
+ </para></listitem>
+ <listitem><para>
+ Run the tests using the following command form:
+ <literallayout class='monospaced'>
+ $ bitbake <replaceable>image</replaceable> -c testimage
+ </literallayout>
+ </para></listitem>
+ </itemizedlist>
+ </para></listitem>
+ <listitem><para>
+ <emphasis>testsdk:</emphasis>
+ <itemizedlist>
+ <listitem><para>These tests build an SDK, install it, and then run tests against that SDK. </para></listitem>
+ <listitem><para>The code for these tests resides in <filename>meta/lib/oeqa/sdk/cases/</filename>. </para></listitem>
+ <listitem><para>Run the test using the following command form:
+ <literallayout class="monospaced">
+ $ bitbake <replaceable>image</replaceable> -c testsdk
+ </literallayout>
+ </para></listitem>
+ </itemizedlist>
+ </para></listitem>
+ <listitem><para>
+ <emphasis>testsdk_ext:</emphasis>
+ <itemizedlist>
+ <listitem><para>These tests build an extended SDK (eSDK), install that eSDK, and run tests against the eSDK. </para></listitem>
+ <listitem><para>The code for these tests resides in <filename>meta/lib/oeqa/esdk</filename>. </para></listitem>
+ <listitem><para>To run the tests, use the following command form:
+ <literallayout class="monospaced">
+ $ bitbake <replaceable>image</replaceable> -c testsdkext
+ </literallayout>
+ </para></listitem>
+ </itemizedlist>
+ </para></listitem>
+
+
+ <listitem><para>
+ <emphasis>oe-build-perf-test:</emphasis>
+ <itemizedlist>
+ <listitem><para>These tests run through commonly used usage scenarios and measure the performance times. </para></listitem>
+ <listitem><para>The code for these tests resides in <filename>meta/lib/oeqa/buildperf</filename>. </para></listitem>
+ <listitem><para>To run the tests, use the following command form:
+ <literallayout class="monospaced">
+ $ oe-build-perf-test <replaceable>options</replaceable>
+ </literallayout>The
+ command takes a number of options, such as where to place the
+ test results. The Autobuilder Helper Scripts include the
+ <filename>build-perf-test-wrapper</filename> script with
+ examples of how to use the oe-build-perf-test from the command
+ line.</para>
+ <para>Use the <filename>oe-git-archive</filename> command to store
+ test results into a Git repository. </para>
+ <para>Use the <filename>oe-build-perf-report</filename> command to
+ generate text reports and HTML reports with graphs of the
+ performance data. For examples, see <link linkend=""
+ >http://downloads.yoctoproject.org/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.html</link>
+ and <link linkend=""
+ >http://downloads.yoctoproject.org/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.txt</link>.</para></listitem>
+ <listitem>
+ <para>The tests are contained in
+ <filename>lib/oeqa/buildperf/test_basic.py</filename>.</para>
+ </listitem>
+ </itemizedlist>
+ </para></listitem>
+
+
+
+
+ </itemizedlist>
+ </para>
+ </section>
+
+ <section id='test-examples'>
+ <title>Test Examples</title>
+
+ <para>This section provides example tests for each of the tests listed in the <link
+ linkend="test-test-mapping">How Tests Map to Areas of Code</link> section. </para>
+ <para>For oeqa tests, testcases for each area reside in the main test directory at
+ <filename>meta/lib/oeqa/selftest/cases</filename> directory.</para>
+ <para>For oe-selftest. bitbake testcases reside in the <filename>lib/bb/tests/</filename>
+ directory. </para>
+
+ <section id='bitbake-selftest-example'>
+ <title><filename>bitbake-selftest</filename></title>
+
+ <para>A simple test example from <filename>lib/bb/tests/data.py</filename> is:
+ <literallayout class="monospaced">
+ class DataExpansions(unittest.TestCase):
+ def setUp(self):
+ self.d = bb.data.init()
+ self.d["foo"] = "value_of_foo"
+ self.d["bar"] = "value_of_bar"
+ self.d["value_of_foo"] = "value_of_'value_of_foo'"
+
+ def test_one_var(self):
+ val = self.d.expand("${foo}")
+ self.assertEqual(str(val), "value_of_foo")
+ </literallayout>
+ </para>
+ <para>In this example, a <ulink url=""><filename>DataExpansions</filename></ulink> class
+ of tests is created, derived from standard python unittest. The class has a common
+ <filename>setUp</filename> function which is shared by all the tests in the
+ class. A simple test is then added to test that when a variable is expanded, the
+ correct value is found.</para>
+ <para>Bitbake selftests are straightforward python unittest. Refer to the Python
+ unittest documentation for additional information on writing these tests at: <link
+ linkend="">https://docs.python.org/3/library/unittest.html</link>.</para>
+ </section>
+
+ <section id='oe-selftest-example'>
+ <title><filename>oe-selftest</filename></title>
+
+ <para>These tests are more complex due to the setup required behind the scenes for full
+ builds. Rather than directly using Python's unittest, the code wraps most of the
+ standard objects. The tests can be simple, such as testing a command from within the
+ OE build environment using the following
+ example:<literallayout class="monospaced">
+ class BitbakeLayers(OESelftestTestCase):
+ def test_bitbakelayers_showcrossdepends(self):
+ result = runCmd('bitbake-layers show-cross-depends')
+ self.assertTrue('aspell' in result.output, msg = "No dependencies
+ were shown. bitbake-layers show-cross-depends output:
+ %s"% result.output)
+ </literallayout></para>
+ <para>This example, taken from
+ <filename>meta/lib/oeqa/selftest/cases/bblayers.py</filename>, creates a
+ testcase from the <ulink url=""><filename>OESelftestTestCase</filename></ulink>
+ class, derived from <filename>unittest.TestCase</filename>, which runs the
+ <filename>bitbake-layers</filename> command and checks the output to ensure it
+ contains something we know should be here.</para>
+ <para>The <filename>oeqa.utils.commands</filename> module contains Helpers which can
+ assist with common tasks, including:<itemizedlist>
+ <listitem>
+ <para><emphasis>Obtaining the value of a bitbake variable:</emphasis> Use
+ <filename>oeqa.utils.commands.get_bb_var()</filename> or use
+ <filename>oeqa.utils.commands.get_bb_vars()</filename> for more than
+ one variable</para>
+ </listitem>
+ <listitem>
+ <para><emphasis>Running a bitbake invocation for a build:</emphasis> Use
+ <filename>oeqa.utils.commands.bitbake()</filename></para>
+ </listitem>
+ <listitem>
+ <para><emphasis>Running a command:</emphasis> Use
+ <filename>oeqa.utils.commandsrunCmd()</filename></para>
+ </listitem>
+ </itemizedlist></para>
+ <para>There is also a <filename>oeqa.utils.commands.runqemu()</filename> function for
+ launching the <filename>runqemu</filename> command for testing things within a
+ running, virtualized image.</para>
+ <para>You can run these tests in parallel. Parallelism works per test class, so tests
+ within a given test class should always run in the same build, while tests in
+ different classes or modules may be split into different builds. There is no data
+ store available for these tests since the tests launch the
+ <filename>bitbake</filename> command and exist outside of its context. As a
+ result, common bitbake library functions (bb.*) are also unavailable.</para>
+ </section>
+
+ <section id='testimage-example'>
+ <title><filename>testimage</filename></title>
+
+ <para>These tests are run once an image is up and running, either on target hardware or
+ under QEMU. As a result, they are assumed to be running in a target image
+ environment, as opposed to a host build environment. A simple example from
+ <filename>meta/lib/oeqa/runtime/cases/python.py</filename> contains the
+ following:<literallayout class="monospaced">
+ class PythonTest(OERuntimeTestCase):
+ @OETestDepends(['ssh.SSHTest.test_ssh'])
+ @OEHasPackage(['python3-core'])
+ def test_python3(self):
+ cmd = "python3 -c \"import codecs; print(codecs.encode('Uryyb,
+ jbeyq', 'rot13'))\""
+ status, output = self.target.run(cmd)
+ msg = 'Exit status was not 0. Output: %s' % output
+ self.assertEqual(status, 0, msg=msg)
+ </literallayout></para>
+ <para>In this example, the <ulink url=""><filename>OERuntimeTestCase</filename></ulink>
+ class wraps <filename>unittest.TestCase</filename>. Within the test,
+ <filename>self.target</filename> represents the target system, where commands
+ can be run on it using the <filename>run()</filename> method. </para>
+ <para>To ensure certain test or package dependencies are met, you can use the
+ <filename>OETestDepends</filename> and <filename>OEHasPackage</filename>
+ decorators. For example, the test in this example would only make sense if
+ python3-core is installed in the image.</para>
+ </section>
+
+ <section id='testsdk_ext-example'>
+ <title><filename>testsdk_ext</filename></title>
+
+ <para>These tests are run against built extensible SDKs (eSDKs). The tests can assume
+ that the eSDK environment has already been setup. An example from
+ <filename>meta/lib/oeqa/sdk/cases/devtool.py</filename> contains the
+ following:<literallayout class="monospaced">
+ class DevtoolTest(OESDKExtTestCase):
+ @classmethod
+ def setUpClass(cls):
+ myapp_src = os.path.join(cls.tc.esdk_files_dir, "myapp")
+ cls.myapp_dst = os.path.join(cls.tc.sdk_dir, "myapp")
+ shutil.copytree(myapp_src, cls.myapp_dst)
+ subprocess.check_output(['git', 'init', '.'], cwd=cls.myapp_dst)
+ subprocess.check_output(['git', 'add', '.'], cwd=cls.myapp_dst)
+ subprocess.check_output(['git', 'commit', '-m', "'test commit'"], cwd=cls.myapp_dst)
+
+ @classmethod
+ def tearDownClass(cls):
+ shutil.rmtree(cls.myapp_dst)
+ def _test_devtool_build(self, directory):
+ self._run('devtool add myapp %s' % directory)
+ try:
+ self._run('devtool build myapp')
+ finally:
+ self._run('devtool reset myapp')
+ def test_devtool_build_make(self):
+ self._test_devtool_build(self.myapp_dst)
+ </literallayout>In
+ this example, the <filename>devtool</filename> command is tested to see whether a
+ sample application can be built with the <filename>devtool build</filename> command
+ within the eSDK.</para>
+ </section>
+
+ <section id='testsdk-example'>
+ <title><filename>testsdk</filename></title>
+
+ <para>These tests are run against built SDKs. The tests can assume that an SDK has
+ already been extracted and its environment file has been sourced. A simple example
+ from <filename>meta/lib/oeqa/sdk/cases/python2.py</filename> contains the
+ following:<literallayout class="monospaced">
+ class Python3Test(OESDKTestCase):
+ def setUp(self):
+ if not (self.tc.hasHostPackage("nativesdk-python3-core") or
+ self.tc.hasHostPackage("python3-core-native")):
+ raise unittest.SkipTest("No python3 package in the SDK")
+
+ def test_python3(self):
+ cmd = "python3 -c \"import codecs; print(codecs.encode('Uryyb, jbeyq', 'rot13'))\""
+ output = self._run(cmd)
+ self.assertEqual(output, "Hello, world\n")
+ </literallayout>In
+ this example, if nativesdk-python3-core has been installed into the SDK, the code
+ runs the python3 interpreter with a basic command to check it is working correctly.
+ The test would only run if python3 is installed in the SDK.</para>
+ </section>
+
+ <section id='oe-build-perf-test-example'>
+ <title><filename>oe-build-perf-test</filename></title>
+
+ <para>The performance tests usually measure how long operations take and the resource
+ utilisation as that happens. An example from
+ <filename>meta/lib/oeqa/buildperf/test_basic.py</filename> contains the
+ following:<literallayout class="monospaced">
+ class Test3(BuildPerfTestCase):
+
+ def test3(self):
+ """Bitbake parsing (bitbake -p)"""
+ # Drop all caches and parse
+ self.rm_cache()
+ oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)
+ self.measure_cmd_resources(['bitbake', '-p'], 'parse_1',
+ 'bitbake -p (no caches)')
+ # Drop tmp/cache
+ oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)
+ self.measure_cmd_resources(['bitbake', '-p'], 'parse_2',
+ 'bitbake -p (no tmp/cache)')
+ # Parse with fully cached data
+ self.measure_cmd_resources(['bitbake', '-p'], 'parse_3',
+ 'bitbake -p (cached)')
+ </literallayout>This
+ example shows how three specific parsing timings are measured, with and without
+ various caches, to show how BitBake’s parsing performance trends over time.</para>
+ </section>
+ </section>
+ <section id='test-writing-considerations'>
+ <title>Considerations When Writing Tests</title>
+ <para>When writing good tests, there are several things to keep in mind. Since things
+ running on the Autobuilder are accessed concurrently by multiple workers, consider the
+ following:</para>
+ <formalpara>
+ <title>Running "cleanall" is not permitted</title>
+ <para>This can delete files from DL_DIR which would potentially break other builds
+ running in parallel. If this is required, DL_DIR must be set to an isolated
+ directory.</para>
+ </formalpara>
+ <formalpara>
+ <title>Running "cleansstate" is not permitted</title>
+ <para>This can delete files from SSTATE_DIR which would potentially break other builds
+ running in parallel. If this is required, SSTATE_DIR must be set to an isolated
+ directory. Alternatively, you can use the "-f" option with the
+ <filename>bitbake</filename> command to "taint" tasks by changing the sstate
+ checksums to ensure sstate cache items will not be reused.</para>
+ </formalpara>
+ <formalpara>
+ <title>Tests should not change the metadata</title>
+ <para>This is particularly true for oe-selftests since these can run in parallel and
+ changing metadata leads to changing checksums, which confuses BitBake while running
+ in parallel. If this is necessary, copy layers to a temporary location and modify
+ them. Some tests need to change metadata, such as the devtool tests. To prevent the
+ metadate from changes, set up temporary copies of that data first.</para>
+ </formalpara>
+ </section>
+
+
+
+
+
+
+
+
+</chapter>
+<!--
+vim: expandtab tw=80 ts=4
+-->