blob: 6421dd53c9f2095404b91c765669a28e20d0959f [file] [log] [blame]
Andrew Geisslerf0343792020-11-18 10:42:21 -06001.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002
3*****************************************
4The Yocto Project Test Environment Manual
5*****************************************
6
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007Welcome
8=======
9
10Welcome to the Yocto Project Test Environment Manual! This manual is a
11work in progress. The manual contains information about the testing
12environment used by the Yocto Project to make sure each major and minor
13release works as intended. All the project's testing infrastructure and
14processes are publicly visible and available so that the community can
15see what testing is being performed, how it's being done and the current
16status of the tests and the project at any given time. It is intended
17that Other organizations can leverage off the process and testing
18environment used by the Yocto Project to create their own automated,
19production test environment, building upon the foundations from the
20project core.
21
22Currently, the Yocto Project Test Environment Manual has no projected
23release date. This manual is a work-in-progress and is being initially
24loaded with information from the README files and notes from key
25engineers:
26
27- *yocto-autobuilder2:* This
Andrew Geissler09209ee2020-12-13 08:44:15 -060028 :yocto_git:`README.md </yocto-autobuilder2/tree/README.md>`
Andrew Geissler3b8a17c2021-04-15 15:55:55 -050029 is the main README which details how to set up the Yocto Project
Andrew Geisslerc9f78652020-09-18 14:11:35 -050030 Autobuilder. The ``yocto-autobuilder2`` repository represents the
31 Yocto Project's console UI plugin to Buildbot and the configuration
32 necessary to configure Buildbot to perform the testing the project
33 requires.
34
Andrew Geissler09209ee2020-12-13 08:44:15 -060035- *yocto-autobuilder-helper:* This :yocto_git:`README </yocto-autobuilder-helper/tree/README/>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -050036 and repository contains Yocto Project Autobuilder Helper scripts and
37 configuration. The ``yocto-autobuilder-helper`` repository contains
38 the "glue" logic that defines which tests to run and how to run them.
39 As a result, it can be used by any Continuous Improvement (CI) system
40 to run builds, support getting the correct code revisions, configure
41 builds and layers, run builds, and collect results. The code is
42 independent of any CI system, which means the code can work `Buildbot <https://docs.buildbot.net/0.9.15.post1/>`__,
43 Jenkins, or others. This repository has a branch per release of the
44 project defining the tests to run on a per release basis.
45
Andrew Geisslerc9f78652020-09-18 14:11:35 -050046Yocto Project Autobuilder Overview
47==================================
48
49The Yocto Project Autobuilder collectively refers to the software,
50tools, scripts, and procedures used by the Yocto Project to test
51released software across supported hardware in an automated and regular
52fashion. Basically, during the development of a Yocto Project release,
53the Autobuilder tests if things work. The Autobuilder builds all test
54targets and runs all the tests.
55
56The Yocto Project uses now uses standard upstream
57`Buildbot <https://docs.buildbot.net/0.9.15.post1/>`__ (version 9) to
58drive its integration and testing. Buildbot Nine has a plug-in interface
59that the Yocto Project customizes using code from the
60``yocto-autobuilder2`` repository, adding its own console UI plugin. The
61resulting UI plug-in allows you to visualize builds in a way suited to
62the project's needs.
63
64A ``helper`` layer provides configuration and job management through
65scripts found in the ``yocto-autobuilder-helper`` repository. The
66``helper`` layer contains the bulk of the build configuration
67information and is release-specific, which makes it highly customizable
68on a per-project basis. The layer is CI system-agnostic and contains a
69number of Helper scripts that can generate build configurations from
70simple JSON files.
71
72.. note::
73
74 The project uses Buildbot for historical reasons but also because
Andrew Geisslerd5838332022-05-27 11:33:10 -050075 many of the project developers have knowledge of Python. It is
Andrew Geisslerc9f78652020-09-18 14:11:35 -050076 possible to use the outer layers from another Continuous Integration
77 (CI) system such as
78 `Jenkins <https://en.wikipedia.org/wiki/Jenkins_(software)>`__
79 instead of Buildbot.
80
81The following figure shows the Yocto Project Autobuilder stack with a
82topology that includes a controller and a cluster of workers:
83
84.. image:: figures/ab-test-cluster.png
85 :align: center
Andrew Geisslerd5838332022-05-27 11:33:10 -050086 :width: 70%
Andrew Geisslerc9f78652020-09-18 14:11:35 -050087
Andrew Geissler615f2f12022-07-15 14:00:58 -050088Yocto Project Tests --- Types of Testing Overview
89=================================================
Andrew Geisslerc9f78652020-09-18 14:11:35 -050090
91The Autobuilder tests different elements of the project by using
Andrew Geissler3b8a17c2021-04-15 15:55:55 -050092the following types of tests:
Andrew Geisslerc9f78652020-09-18 14:11:35 -050093
94- *Build Testing:* Tests whether specific configurations build by
95 varying :term:`MACHINE`,
96 :term:`DISTRO`, other configuration
97 options, and the specific target images being built (or world). Used
98 to trigger builds of all the different test configurations on the
99 Autobuilder. Builds usually cover many different targets for
100 different architectures, machines, and distributions, as well as
101 different configurations, such as different init systems. The
102 Autobuilder tests literally hundreds of configurations and targets.
103
104 - *Sanity Checks During the Build Process:* Tests initiated through
105 the :ref:`insane <ref-classes-insane>`
106 class. These checks ensure the output of the builds are correct.
107 For example, does the ELF architecture in the generated binaries
108 match the target system? ARM binaries would not work in a MIPS
109 system!
110
111- *Build Performance Testing:* Tests whether or not commonly used steps
112 during builds work efficiently and avoid regressions. Tests to time
113 commonly used usage scenarios are run through ``oe-build-perf-test``.
114 These tests are run on isolated machines so that the time
115 measurements of the tests are accurate and no other processes
116 interfere with the timing results. The project currently tests
117 performance on two different distributions, Fedora and Ubuntu, to
118 ensure we have no single point of failure and can ensure the
119 different distros work effectively.
120
121- *eSDK Testing:* Image tests initiated through the following command::
122
123 $ bitbake image -c testsdkext
124
Andrew Geissler595f6302022-01-24 19:11:47 +0000125 The tests utilize the :ref:`testsdkext <ref-classes-testsdk>` class and the ``do_testsdkext`` task.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500126
127- *Feature Testing:* Various scenario-based tests are run through the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500128 :ref:`OpenEmbedded Self test (oe-selftest) <ref-manual/release-process:Testing and Quality Assurance>`. We test oe-selftest on each of the main distributions
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500129 we support.
130
131- *Image Testing:* Image tests initiated through the following command::
132
133 $ bitbake image -c testimage
134
135 The tests utilize the :ref:`testimage* <ref-classes-testimage*>`
136 classes and the :ref:`ref-tasks-testimage` task.
137
138- *Layer Testing:* The Autobuilder has the possibility to test whether
139 specific layers work with the test of the system. The layers tested
140 may be selected by members of the project. Some key community layers
141 are also tested periodically.
142
143- *Package Testing:* A Package Test (ptest) runs tests against packages
144 built by the OpenEmbedded build system on the target machine. See the
145 :ref:`Testing Packages With
Andrew Geissler09209ee2020-12-13 08:44:15 -0600146 ptest <dev-manual/common-tasks:Testing Packages With ptest>` section
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500147 in the Yocto Project Development Tasks Manual and the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600148 ":yocto_wiki:`Ptest </Ptest>`" Wiki page for more
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500149 information on Ptest.
150
151- *SDK Testing:* Image tests initiated through the following command::
152
153 $ bitbake image -c testsdk
154
155 The tests utilize the :ref:`testsdk <ref-classes-testsdk>` class and
156 the ``do_testsdk`` task.
157
158- *Unit Testing:* Unit tests on various components of the system run
Andrew Geissler09209ee2020-12-13 08:44:15 -0600159 through :ref:`bitbake-selftest <ref-manual/release-process:Testing and Quality Assurance>` and
160 :ref:`oe-selftest <ref-manual/release-process:Testing and Quality Assurance>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500161
162- *Automatic Upgrade Helper:* This target tests whether new versions of
163 software are available and whether we can automatically upgrade to
164 those new versions. If so, this target emails the maintainers with a
165 patch to let them know this is possible.
166
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500167How Tests Map to Areas of Code
168==============================
169
170Tests map into the codebase as follows:
171
172- *bitbake-selftest:*
173
174 These tests are self-contained and test BitBake as well as its APIs,
175 which include the fetchers. The tests are located in
176 ``bitbake/lib/*/tests``.
177
Andrew Geissler78b72792022-06-14 06:47:25 -0500178 Some of these tests run the ``bitbake`` command, so ``bitbake/bin``
179 must be added to the ``PATH`` before running ``bitbake-selftest``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500180 From within the BitBake repository, run the following::
181
Andrew Geissler78b72792022-06-14 06:47:25 -0500182 $ export PATH=$PWD/bin:$PATH
183
184 After that, you can run the selftest script::
185
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500186 $ bitbake-selftest
187
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500188 The default output is quiet and just prints a summary of what was
189 run. To see more information, there is a verbose option::
190
191 $ bitbake-selftest -v
192
Andrew Geissler78b72792022-06-14 06:47:25 -0500193 To skip tests that access the Internet, use the ``BB_SKIP_NETTESTS``
194 variable when running "bitbake-selftest" as follows::
195
196 $ BB_SKIP_NETTESTS=yes bitbake-selftest
197
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500198 Use this option when you wish to skip tests that access the network,
199 which are mostly necessary to test the fetcher modules. To specify
200 individual test modules to run, append the test module name to the
201 "bitbake-selftest" command. For example, to specify the tests for the
202 bb.data.module, run::
203
204 $ bitbake-selftest bb.test.data.module
205
206 You can also specify individual tests by defining the full name and module
207 plus the class path of the test, for example::
208
209 $ bitbake-selftest bb.tests.data.TestOverrides.test_one_override
210
211 The tests are based on `Python
212 unittest <https://docs.python.org/3/library/unittest.html>`__.
213
214- *oe-selftest:*
215
216 - These tests use OE to test the workflows, which include testing
217 specific features, behaviors of tasks, and API unit tests.
218
219 - The tests can take advantage of parallelism through the "-j"
220 option, which can specify a number of threads to spread the tests
221 across. Note that all tests from a given class of tests will run
222 in the same thread. To parallelize large numbers of tests you can
223 split the class into multiple units.
224
225 - The tests are based on Python unittest.
226
227 - The code for the tests resides in
228 ``meta/lib/oeqa/selftest/cases/``.
229
230 - To run all the tests, enter the following command::
231
232 $ oe-selftest -a
233
234 - To run a specific test, use the following command form where
235 testname is the name of the specific test::
236
237 $ oe-selftest -r <testname>
238
239 For example, the following command would run the tinfoil
240 getVar API test::
241
242 $ oe-selftest -r tinfoil.TinfoilTests.test_getvar
243
244 It is also possible to run a set
245 of tests. For example the following command will run all of the
246 tinfoil tests::
247
248 $ oe-selftest -r tinfoil
249
250- *testimage:*
251
252 - These tests build an image, boot it, and run tests against the
253 image's content.
254
255 - The code for these tests resides in ``meta/lib/oeqa/runtime/cases/``.
256
257 - You need to set the :term:`IMAGE_CLASSES` variable as follows::
258
259 IMAGE_CLASSES += "testimage"
260
261 - Run the tests using the following command form::
262
263 $ bitbake image -c testimage
264
265- *testsdk:*
266
267 - These tests build an SDK, install it, and then run tests against
268 that SDK.
269
270 - The code for these tests resides in ``meta/lib/oeqa/sdk/cases/``.
271
272 - Run the test using the following command form::
273
274 $ bitbake image -c testsdk
275
276- *testsdk_ext:*
277
278 - These tests build an extended SDK (eSDK), install that eSDK, and
279 run tests against the eSDK.
280
281 - The code for these tests resides in ``meta/lib/oeqa/esdk``.
282
283 - To run the tests, use the following command form::
284
285 $ bitbake image -c testsdkext
286
287- *oe-build-perf-test:*
288
289 - These tests run through commonly used usage scenarios and measure
290 the performance times.
291
292 - The code for these tests resides in ``meta/lib/oeqa/buildperf``.
293
294 - To run the tests, use the following command form::
295
296 $ oe-build-perf-test <options>
297
298 The command takes a number of options,
299 such as where to place the test results. The Autobuilder Helper
300 Scripts include the ``build-perf-test-wrapper`` script with
301 examples of how to use the oe-build-perf-test from the command
302 line.
303
304 Use the ``oe-git-archive`` command to store test results into a
305 Git repository.
306
307 Use the ``oe-build-perf-report`` command to generate text reports
308 and HTML reports with graphs of the performance data. For
309 examples, see
310 :yocto_dl:`/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.html`
311 and
312 :yocto_dl:`/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.txt`.
313
314 - The tests are contained in ``lib/oeqa/buildperf/test_basic.py``.
315
316Test Examples
317=============
318
319This section provides example tests for each of the tests listed in the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600320:ref:`test-manual/intro:How Tests Map to Areas of Code` section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500321
322For oeqa tests, testcases for each area reside in the main test
323directory at ``meta/lib/oeqa/selftest/cases`` directory.
324
325For oe-selftest. bitbake testcases reside in the ``lib/bb/tests/``
326directory.
327
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500328``bitbake-selftest``
329--------------------
330
331A simple test example from ``lib/bb/tests/data.py`` is::
332
333 class DataExpansions(unittest.TestCase):
334 def setUp(self):
335 self.d = bb.data.init()
336 self.d["foo"] = "value_of_foo"
337 self.d["bar"] = "value_of_bar"
338 self.d["value_of_foo"] = "value_of_'value_of_foo'"
339
340 def test_one_var(self):
341 val = self.d.expand("${foo}")
342 self.assertEqual(str(val), "value_of_foo")
343
344In this example, a ``DataExpansions`` class of tests is created,
Andrew Geisslerd5838332022-05-27 11:33:10 -0500345derived from standard Python unittest. The class has a common ``setUp``
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500346function which is shared by all the tests in the class. A simple test is
347then added to test that when a variable is expanded, the correct value
348is found.
349
Andrew Geisslerd5838332022-05-27 11:33:10 -0500350BitBake selftests are straightforward Python unittest. Refer to the
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500351Python unittest documentation for additional information on writing
352these tests at: https://docs.python.org/3/library/unittest.html.
353
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500354``oe-selftest``
355---------------
356
357These tests are more complex due to the setup required behind the scenes
358for full builds. Rather than directly using Python's unittest, the code
359wraps most of the standard objects. The tests can be simple, such as
360testing a command from within the OE build environment using the
361following example::
362
363 class BitbakeLayers(OESelftestTestCase):
364 def test_bitbakelayers_showcrossdepends(self):
365 result = runCmd('bitbake-layers show-cross-depends')
366 self.assertTrue('aspell' in result.output, msg = "No dependencies were shown. bitbake-layers show-cross-depends output: %s"% result.output)
367
368This example, taken from ``meta/lib/oeqa/selftest/cases/bblayers.py``,
369creates a testcase from the ``OESelftestTestCase`` class, derived
370from ``unittest.TestCase``, which runs the ``bitbake-layers`` command
371and checks the output to ensure it contains something we know should be
372here.
373
374The ``oeqa.utils.commands`` module contains Helpers which can assist
375with common tasks, including:
376
377- *Obtaining the value of a bitbake variable:* Use
378 ``oeqa.utils.commands.get_bb_var()`` or use
379 ``oeqa.utils.commands.get_bb_vars()`` for more than one variable
380
381- *Running a bitbake invocation for a build:* Use
382 ``oeqa.utils.commands.bitbake()``
383
384- *Running a command:* Use ``oeqa.utils.commandsrunCmd()``
385
386There is also a ``oeqa.utils.commands.runqemu()`` function for launching
387the ``runqemu`` command for testing things within a running, virtualized
388image.
389
390You can run these tests in parallel. Parallelism works per test class,
391so tests within a given test class should always run in the same build,
392while tests in different classes or modules may be split into different
393builds. There is no data store available for these tests since the tests
394launch the ``bitbake`` command and exist outside of its context. As a
395result, common bitbake library functions (bb.\*) are also unavailable.
396
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500397``testimage``
398-------------
399
400These tests are run once an image is up and running, either on target
401hardware or under QEMU. As a result, they are assumed to be running in a
402target image environment, as opposed to a host build environment. A
403simple example from ``meta/lib/oeqa/runtime/cases/python.py`` contains
404the following::
405
406 class PythonTest(OERuntimeTestCase):
407 @OETestDepends(['ssh.SSHTest.test_ssh'])
408 @OEHasPackage(['python3-core'])
409 def test_python3(self):
410 cmd = "python3 -c \\"import codecs; print(codecs.encode('Uryyb, jbeyq', 'rot13'))\""
411 status, output = self.target.run(cmd)
412 msg = 'Exit status was not 0. Output: %s' % output
413 self.assertEqual(status, 0, msg=msg)
414
415In this example, the ``OERuntimeTestCase`` class wraps
416``unittest.TestCase``. Within the test, ``self.target`` represents the
417target system, where commands can be run on it using the ``run()``
418method.
419
420To ensure certain test or package dependencies are met, you can use the
421``OETestDepends`` and ``OEHasPackage`` decorators. For example, the test
422in this example would only make sense if python3-core is installed in
423the image.
424
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500425``testsdk_ext``
426---------------
427
428These tests are run against built extensible SDKs (eSDKs). The tests can
429assume that the eSDK environment has already been setup. An example from
430``meta/lib/oeqa/sdk/cases/devtool.py`` contains the following::
431
432 class DevtoolTest(OESDKExtTestCase):
433 @classmethod def setUpClass(cls):
434 myapp_src = os.path.join(cls.tc.esdk_files_dir, "myapp")
435 cls.myapp_dst = os.path.join(cls.tc.sdk_dir, "myapp")
436 shutil.copytree(myapp_src, cls.myapp_dst)
437 subprocess.check_output(['git', 'init', '.'], cwd=cls.myapp_dst)
438 subprocess.check_output(['git', 'add', '.'], cwd=cls.myapp_dst)
439 subprocess.check_output(['git', 'commit', '-m', "'test commit'"], cwd=cls.myapp_dst)
440
441 @classmethod
442 def tearDownClass(cls):
443 shutil.rmtree(cls.myapp_dst)
444 def _test_devtool_build(self, directory):
445 self._run('devtool add myapp %s' % directory)
446 try:
447 self._run('devtool build myapp')
448 finally:
449 self._run('devtool reset myapp')
450 def test_devtool_build_make(self):
451 self._test_devtool_build(self.myapp_dst)
452
453In this example, the ``devtool``
454command is tested to see whether a sample application can be built with
455the ``devtool build`` command within the eSDK.
456
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500457``testsdk``
458-----------
459
460These tests are run against built SDKs. The tests can assume that an SDK
461has already been extracted and its environment file has been sourced. A
462simple example from ``meta/lib/oeqa/sdk/cases/python2.py`` contains the
463following::
464
465 class Python3Test(OESDKTestCase):
466 def setUp(self):
467 if not (self.tc.hasHostPackage("nativesdk-python3-core") or
468 self.tc.hasHostPackage("python3-core-native")):
469 raise unittest.SkipTest("No python3 package in the SDK")
470
471 def test_python3(self):
472 cmd = "python3 -c \\"import codecs; print(codecs.encode('Uryyb, jbeyq', 'rot13'))\""
473 output = self._run(cmd)
474 self.assertEqual(output, "Hello, world\n")
475
476In this example, if nativesdk-python3-core has been installed into the SDK, the code runs
477the python3 interpreter with a basic command to check it is working
Andrew Geisslerd5838332022-05-27 11:33:10 -0500478correctly. The test would only run if Python3 is installed in the SDK.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500479
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500480``oe-build-perf-test``
481----------------------
482
483The performance tests usually measure how long operations take and the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500484resource utilization as that happens. An example from
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500485``meta/lib/oeqa/buildperf/test_basic.py`` contains the following::
486
487 class Test3(BuildPerfTestCase):
488 def test3(self):
489 """Bitbake parsing (bitbake -p)"""
490 # Drop all caches and parse
491 self.rm_cache()
492 oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)
493 self.measure_cmd_resources(['bitbake', '-p'], 'parse_1',
494 'bitbake -p (no caches)')
495 # Drop tmp/cache
496 oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)
497 self.measure_cmd_resources(['bitbake', '-p'], 'parse_2',
498 'bitbake -p (no tmp/cache)')
499 # Parse with fully cached data
500 self.measure_cmd_resources(['bitbake', '-p'], 'parse_3',
501 'bitbake -p (cached)')
502
503This example shows how three specific parsing timings are
504measured, with and without various caches, to show how BitBake's parsing
505performance trends over time.
506
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500507Considerations When Writing Tests
508=================================
509
510When writing good tests, there are several things to keep in mind. Since
511things running on the Autobuilder are accessed concurrently by multiple
512workers, consider the following:
513
514**Running "cleanall" is not permitted.**
515
516This can delete files from DL_DIR which would potentially break other
517builds running in parallel. If this is required, DL_DIR must be set to
518an isolated directory.
519
520**Running "cleansstate" is not permitted.**
521
522This can delete files from SSTATE_DIR which would potentially break
523other builds running in parallel. If this is required, SSTATE_DIR must
524be set to an isolated directory. Alternatively, you can use the "-f"
525option with the ``bitbake`` command to "taint" tasks by changing the
526sstate checksums to ensure sstate cache items will not be reused.
527
528**Tests should not change the metadata.**
529
530This is particularly true for oe-selftests since these can run in
531parallel and changing metadata leads to changing checksums, which
532confuses BitBake while running in parallel. If this is necessary, copy
533layers to a temporary location and modify them. Some tests need to
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500534change metadata, such as the devtool tests. To protect the metadata from
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500535changes, set up temporary copies of that data first.