blob: 0831f80466b5e77def4380ea6641b872eed575ad [file] [log] [blame]
Andrew Geisslerf0343792020-11-18 10:42:21 -06001.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002
3*****************************************
4The Yocto Project Test Environment Manual
5*****************************************
6
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007Welcome
8=======
9
10Welcome to the Yocto Project Test Environment Manual! This manual is a
11work in progress. The manual contains information about the testing
12environment used by the Yocto Project to make sure each major and minor
13release works as intended. All the project's testing infrastructure and
14processes are publicly visible and available so that the community can
15see what testing is being performed, how it's being done and the current
16status of the tests and the project at any given time. It is intended
17that Other organizations can leverage off the process and testing
18environment used by the Yocto Project to create their own automated,
19production test environment, building upon the foundations from the
20project core.
21
22Currently, the Yocto Project Test Environment Manual has no projected
23release date. This manual is a work-in-progress and is being initially
24loaded with information from the README files and notes from key
25engineers:
26
27- *yocto-autobuilder2:* This
Andrew Geissler09209ee2020-12-13 08:44:15 -060028 :yocto_git:`README.md </yocto-autobuilder2/tree/README.md>`
Andrew Geissler3b8a17c2021-04-15 15:55:55 -050029 is the main README which details how to set up the Yocto Project
Andrew Geisslerc9f78652020-09-18 14:11:35 -050030 Autobuilder. The ``yocto-autobuilder2`` repository represents the
31 Yocto Project's console UI plugin to Buildbot and the configuration
32 necessary to configure Buildbot to perform the testing the project
33 requires.
34
Andrew Geissler09209ee2020-12-13 08:44:15 -060035- *yocto-autobuilder-helper:* This :yocto_git:`README </yocto-autobuilder-helper/tree/README/>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -050036 and repository contains Yocto Project Autobuilder Helper scripts and
37 configuration. The ``yocto-autobuilder-helper`` repository contains
38 the "glue" logic that defines which tests to run and how to run them.
39 As a result, it can be used by any Continuous Improvement (CI) system
40 to run builds, support getting the correct code revisions, configure
41 builds and layers, run builds, and collect results. The code is
42 independent of any CI system, which means the code can work `Buildbot <https://docs.buildbot.net/0.9.15.post1/>`__,
43 Jenkins, or others. This repository has a branch per release of the
44 project defining the tests to run on a per release basis.
45
Andrew Geisslerc9f78652020-09-18 14:11:35 -050046Yocto Project Autobuilder Overview
47==================================
48
49The Yocto Project Autobuilder collectively refers to the software,
50tools, scripts, and procedures used by the Yocto Project to test
51released software across supported hardware in an automated and regular
52fashion. Basically, during the development of a Yocto Project release,
53the Autobuilder tests if things work. The Autobuilder builds all test
54targets and runs all the tests.
55
56The Yocto Project uses now uses standard upstream
57`Buildbot <https://docs.buildbot.net/0.9.15.post1/>`__ (version 9) to
58drive its integration and testing. Buildbot Nine has a plug-in interface
59that the Yocto Project customizes using code from the
60``yocto-autobuilder2`` repository, adding its own console UI plugin. The
61resulting UI plug-in allows you to visualize builds in a way suited to
62the project's needs.
63
64A ``helper`` layer provides configuration and job management through
65scripts found in the ``yocto-autobuilder-helper`` repository. The
66``helper`` layer contains the bulk of the build configuration
67information and is release-specific, which makes it highly customizable
68on a per-project basis. The layer is CI system-agnostic and contains a
69number of Helper scripts that can generate build configurations from
70simple JSON files.
71
72.. note::
73
74 The project uses Buildbot for historical reasons but also because
Andrew Geisslerd5838332022-05-27 11:33:10 -050075 many of the project developers have knowledge of Python. It is
Andrew Geisslerc9f78652020-09-18 14:11:35 -050076 possible to use the outer layers from another Continuous Integration
Patrick Williams7784c422022-11-17 07:29:11 -060077 (CI) system such as :wikipedia:`Jenkins <Jenkins_(software)>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -050078 instead of Buildbot.
79
80The following figure shows the Yocto Project Autobuilder stack with a
81topology that includes a controller and a cluster of workers:
82
83.. image:: figures/ab-test-cluster.png
84 :align: center
Andrew Geisslerd5838332022-05-27 11:33:10 -050085 :width: 70%
Andrew Geisslerc9f78652020-09-18 14:11:35 -050086
Andrew Geissler615f2f12022-07-15 14:00:58 -050087Yocto Project Tests --- Types of Testing Overview
88=================================================
Andrew Geisslerc9f78652020-09-18 14:11:35 -050089
90The Autobuilder tests different elements of the project by using
Andrew Geissler3b8a17c2021-04-15 15:55:55 -050091the following types of tests:
Andrew Geisslerc9f78652020-09-18 14:11:35 -050092
93- *Build Testing:* Tests whether specific configurations build by
94 varying :term:`MACHINE`,
95 :term:`DISTRO`, other configuration
96 options, and the specific target images being built (or world). Used
97 to trigger builds of all the different test configurations on the
98 Autobuilder. Builds usually cover many different targets for
99 different architectures, machines, and distributions, as well as
100 different configurations, such as different init systems. The
101 Autobuilder tests literally hundreds of configurations and targets.
102
103 - *Sanity Checks During the Build Process:* Tests initiated through
104 the :ref:`insane <ref-classes-insane>`
105 class. These checks ensure the output of the builds are correct.
106 For example, does the ELF architecture in the generated binaries
107 match the target system? ARM binaries would not work in a MIPS
108 system!
109
110- *Build Performance Testing:* Tests whether or not commonly used steps
111 during builds work efficiently and avoid regressions. Tests to time
112 commonly used usage scenarios are run through ``oe-build-perf-test``.
113 These tests are run on isolated machines so that the time
114 measurements of the tests are accurate and no other processes
115 interfere with the timing results. The project currently tests
116 performance on two different distributions, Fedora and Ubuntu, to
117 ensure we have no single point of failure and can ensure the
118 different distros work effectively.
119
120- *eSDK Testing:* Image tests initiated through the following command::
121
122 $ bitbake image -c testsdkext
123
Andrew Geissler595f6302022-01-24 19:11:47 +0000124 The tests utilize the :ref:`testsdkext <ref-classes-testsdk>` class and the ``do_testsdkext`` task.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500125
126- *Feature Testing:* Various scenario-based tests are run through the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500127 :ref:`OpenEmbedded Self test (oe-selftest) <ref-manual/release-process:Testing and Quality Assurance>`. We test oe-selftest on each of the main distributions
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500128 we support.
129
130- *Image Testing:* Image tests initiated through the following command::
131
132 $ bitbake image -c testimage
133
Patrick Williams975a06f2022-10-21 14:42:47 -0500134 The tests utilize the :ref:`testimage <ref-classes-testimage>`
135 class and the :ref:`ref-tasks-testimage` task.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500136
137- *Layer Testing:* The Autobuilder has the possibility to test whether
138 specific layers work with the test of the system. The layers tested
139 may be selected by members of the project. Some key community layers
140 are also tested periodically.
141
142- *Package Testing:* A Package Test (ptest) runs tests against packages
143 built by the OpenEmbedded build system on the target machine. See the
144 :ref:`Testing Packages With
Andrew Geissler09209ee2020-12-13 08:44:15 -0600145 ptest <dev-manual/common-tasks:Testing Packages With ptest>` section
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500146 in the Yocto Project Development Tasks Manual and the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600147 ":yocto_wiki:`Ptest </Ptest>`" Wiki page for more
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500148 information on Ptest.
149
150- *SDK Testing:* Image tests initiated through the following command::
151
152 $ bitbake image -c testsdk
153
154 The tests utilize the :ref:`testsdk <ref-classes-testsdk>` class and
155 the ``do_testsdk`` task.
156
157- *Unit Testing:* Unit tests on various components of the system run
Andrew Geissler09209ee2020-12-13 08:44:15 -0600158 through :ref:`bitbake-selftest <ref-manual/release-process:Testing and Quality Assurance>` and
159 :ref:`oe-selftest <ref-manual/release-process:Testing and Quality Assurance>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500160
161- *Automatic Upgrade Helper:* This target tests whether new versions of
162 software are available and whether we can automatically upgrade to
163 those new versions. If so, this target emails the maintainers with a
164 patch to let them know this is possible.
165
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500166How Tests Map to Areas of Code
167==============================
168
169Tests map into the codebase as follows:
170
171- *bitbake-selftest:*
172
173 These tests are self-contained and test BitBake as well as its APIs,
174 which include the fetchers. The tests are located in
175 ``bitbake/lib/*/tests``.
176
Andrew Geissler78b72792022-06-14 06:47:25 -0500177 Some of these tests run the ``bitbake`` command, so ``bitbake/bin``
178 must be added to the ``PATH`` before running ``bitbake-selftest``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500179 From within the BitBake repository, run the following::
180
Andrew Geissler78b72792022-06-14 06:47:25 -0500181 $ export PATH=$PWD/bin:$PATH
182
183 After that, you can run the selftest script::
184
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500185 $ bitbake-selftest
186
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500187 The default output is quiet and just prints a summary of what was
188 run. To see more information, there is a verbose option::
189
190 $ bitbake-selftest -v
191
Andrew Geissler78b72792022-06-14 06:47:25 -0500192 To skip tests that access the Internet, use the ``BB_SKIP_NETTESTS``
193 variable when running "bitbake-selftest" as follows::
194
195 $ BB_SKIP_NETTESTS=yes bitbake-selftest
196
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500197 Use this option when you wish to skip tests that access the network,
198 which are mostly necessary to test the fetcher modules. To specify
199 individual test modules to run, append the test module name to the
200 "bitbake-selftest" command. For example, to specify the tests for the
201 bb.data.module, run::
202
203 $ bitbake-selftest bb.test.data.module
204
205 You can also specify individual tests by defining the full name and module
206 plus the class path of the test, for example::
207
208 $ bitbake-selftest bb.tests.data.TestOverrides.test_one_override
209
210 The tests are based on `Python
211 unittest <https://docs.python.org/3/library/unittest.html>`__.
212
213- *oe-selftest:*
214
215 - These tests use OE to test the workflows, which include testing
216 specific features, behaviors of tasks, and API unit tests.
217
218 - The tests can take advantage of parallelism through the "-j"
219 option, which can specify a number of threads to spread the tests
220 across. Note that all tests from a given class of tests will run
221 in the same thread. To parallelize large numbers of tests you can
222 split the class into multiple units.
223
224 - The tests are based on Python unittest.
225
226 - The code for the tests resides in
227 ``meta/lib/oeqa/selftest/cases/``.
228
229 - To run all the tests, enter the following command::
230
231 $ oe-selftest -a
232
233 - To run a specific test, use the following command form where
234 testname is the name of the specific test::
235
236 $ oe-selftest -r <testname>
237
238 For example, the following command would run the tinfoil
239 getVar API test::
240
241 $ oe-selftest -r tinfoil.TinfoilTests.test_getvar
242
243 It is also possible to run a set
244 of tests. For example the following command will run all of the
245 tinfoil tests::
246
247 $ oe-selftest -r tinfoil
248
249- *testimage:*
250
251 - These tests build an image, boot it, and run tests against the
252 image's content.
253
254 - The code for these tests resides in ``meta/lib/oeqa/runtime/cases/``.
255
256 - You need to set the :term:`IMAGE_CLASSES` variable as follows::
257
258 IMAGE_CLASSES += "testimage"
259
260 - Run the tests using the following command form::
261
262 $ bitbake image -c testimage
263
264- *testsdk:*
265
266 - These tests build an SDK, install it, and then run tests against
267 that SDK.
268
269 - The code for these tests resides in ``meta/lib/oeqa/sdk/cases/``.
270
271 - Run the test using the following command form::
272
273 $ bitbake image -c testsdk
274
275- *testsdk_ext:*
276
277 - These tests build an extended SDK (eSDK), install that eSDK, and
278 run tests against the eSDK.
279
280 - The code for these tests resides in ``meta/lib/oeqa/esdk``.
281
282 - To run the tests, use the following command form::
283
284 $ bitbake image -c testsdkext
285
286- *oe-build-perf-test:*
287
288 - These tests run through commonly used usage scenarios and measure
289 the performance times.
290
291 - The code for these tests resides in ``meta/lib/oeqa/buildperf``.
292
293 - To run the tests, use the following command form::
294
295 $ oe-build-perf-test <options>
296
297 The command takes a number of options,
298 such as where to place the test results. The Autobuilder Helper
299 Scripts include the ``build-perf-test-wrapper`` script with
300 examples of how to use the oe-build-perf-test from the command
301 line.
302
303 Use the ``oe-git-archive`` command to store test results into a
304 Git repository.
305
306 Use the ``oe-build-perf-report`` command to generate text reports
307 and HTML reports with graphs of the performance data. For
308 examples, see
309 :yocto_dl:`/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.html`
310 and
311 :yocto_dl:`/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.txt`.
312
313 - The tests are contained in ``lib/oeqa/buildperf/test_basic.py``.
314
315Test Examples
316=============
317
318This section provides example tests for each of the tests listed in the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600319:ref:`test-manual/intro:How Tests Map to Areas of Code` section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500320
321For oeqa tests, testcases for each area reside in the main test
322directory at ``meta/lib/oeqa/selftest/cases`` directory.
323
324For oe-selftest. bitbake testcases reside in the ``lib/bb/tests/``
325directory.
326
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500327``bitbake-selftest``
328--------------------
329
330A simple test example from ``lib/bb/tests/data.py`` is::
331
332 class DataExpansions(unittest.TestCase):
333 def setUp(self):
334 self.d = bb.data.init()
335 self.d["foo"] = "value_of_foo"
336 self.d["bar"] = "value_of_bar"
337 self.d["value_of_foo"] = "value_of_'value_of_foo'"
338
339 def test_one_var(self):
340 val = self.d.expand("${foo}")
341 self.assertEqual(str(val), "value_of_foo")
342
343In this example, a ``DataExpansions`` class of tests is created,
Andrew Geisslerd5838332022-05-27 11:33:10 -0500344derived from standard Python unittest. The class has a common ``setUp``
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500345function which is shared by all the tests in the class. A simple test is
346then added to test that when a variable is expanded, the correct value
347is found.
348
Andrew Geisslerd5838332022-05-27 11:33:10 -0500349BitBake selftests are straightforward Python unittest. Refer to the
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500350Python unittest documentation for additional information on writing
351these tests at: https://docs.python.org/3/library/unittest.html.
352
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500353``oe-selftest``
354---------------
355
356These tests are more complex due to the setup required behind the scenes
357for full builds. Rather than directly using Python's unittest, the code
358wraps most of the standard objects. The tests can be simple, such as
359testing a command from within the OE build environment using the
360following example::
361
362 class BitbakeLayers(OESelftestTestCase):
363 def test_bitbakelayers_showcrossdepends(self):
364 result = runCmd('bitbake-layers show-cross-depends')
365 self.assertTrue('aspell' in result.output, msg = "No dependencies were shown. bitbake-layers show-cross-depends output: %s"% result.output)
366
367This example, taken from ``meta/lib/oeqa/selftest/cases/bblayers.py``,
368creates a testcase from the ``OESelftestTestCase`` class, derived
369from ``unittest.TestCase``, which runs the ``bitbake-layers`` command
370and checks the output to ensure it contains something we know should be
371here.
372
373The ``oeqa.utils.commands`` module contains Helpers which can assist
374with common tasks, including:
375
376- *Obtaining the value of a bitbake variable:* Use
377 ``oeqa.utils.commands.get_bb_var()`` or use
378 ``oeqa.utils.commands.get_bb_vars()`` for more than one variable
379
380- *Running a bitbake invocation for a build:* Use
381 ``oeqa.utils.commands.bitbake()``
382
383- *Running a command:* Use ``oeqa.utils.commandsrunCmd()``
384
385There is also a ``oeqa.utils.commands.runqemu()`` function for launching
386the ``runqemu`` command for testing things within a running, virtualized
387image.
388
389You can run these tests in parallel. Parallelism works per test class,
390so tests within a given test class should always run in the same build,
391while tests in different classes or modules may be split into different
392builds. There is no data store available for these tests since the tests
393launch the ``bitbake`` command and exist outside of its context. As a
394result, common bitbake library functions (bb.\*) are also unavailable.
395
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500396``testimage``
397-------------
398
399These tests are run once an image is up and running, either on target
400hardware or under QEMU. As a result, they are assumed to be running in a
401target image environment, as opposed to a host build environment. A
402simple example from ``meta/lib/oeqa/runtime/cases/python.py`` contains
403the following::
404
405 class PythonTest(OERuntimeTestCase):
406 @OETestDepends(['ssh.SSHTest.test_ssh'])
407 @OEHasPackage(['python3-core'])
408 def test_python3(self):
409 cmd = "python3 -c \\"import codecs; print(codecs.encode('Uryyb, jbeyq', 'rot13'))\""
410 status, output = self.target.run(cmd)
411 msg = 'Exit status was not 0. Output: %s' % output
412 self.assertEqual(status, 0, msg=msg)
413
414In this example, the ``OERuntimeTestCase`` class wraps
415``unittest.TestCase``. Within the test, ``self.target`` represents the
416target system, where commands can be run on it using the ``run()``
417method.
418
419To ensure certain test or package dependencies are met, you can use the
420``OETestDepends`` and ``OEHasPackage`` decorators. For example, the test
421in this example would only make sense if python3-core is installed in
422the image.
423
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500424``testsdk_ext``
425---------------
426
427These tests are run against built extensible SDKs (eSDKs). The tests can
428assume that the eSDK environment has already been setup. An example from
429``meta/lib/oeqa/sdk/cases/devtool.py`` contains the following::
430
431 class DevtoolTest(OESDKExtTestCase):
432 @classmethod def setUpClass(cls):
433 myapp_src = os.path.join(cls.tc.esdk_files_dir, "myapp")
434 cls.myapp_dst = os.path.join(cls.tc.sdk_dir, "myapp")
435 shutil.copytree(myapp_src, cls.myapp_dst)
436 subprocess.check_output(['git', 'init', '.'], cwd=cls.myapp_dst)
437 subprocess.check_output(['git', 'add', '.'], cwd=cls.myapp_dst)
438 subprocess.check_output(['git', 'commit', '-m', "'test commit'"], cwd=cls.myapp_dst)
439
440 @classmethod
441 def tearDownClass(cls):
442 shutil.rmtree(cls.myapp_dst)
443 def _test_devtool_build(self, directory):
444 self._run('devtool add myapp %s' % directory)
445 try:
446 self._run('devtool build myapp')
447 finally:
448 self._run('devtool reset myapp')
449 def test_devtool_build_make(self):
450 self._test_devtool_build(self.myapp_dst)
451
452In this example, the ``devtool``
453command is tested to see whether a sample application can be built with
454the ``devtool build`` command within the eSDK.
455
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500456``testsdk``
457-----------
458
459These tests are run against built SDKs. The tests can assume that an SDK
460has already been extracted and its environment file has been sourced. A
461simple example from ``meta/lib/oeqa/sdk/cases/python2.py`` contains the
462following::
463
464 class Python3Test(OESDKTestCase):
465 def setUp(self):
466 if not (self.tc.hasHostPackage("nativesdk-python3-core") or
467 self.tc.hasHostPackage("python3-core-native")):
468 raise unittest.SkipTest("No python3 package in the SDK")
469
470 def test_python3(self):
471 cmd = "python3 -c \\"import codecs; print(codecs.encode('Uryyb, jbeyq', 'rot13'))\""
472 output = self._run(cmd)
473 self.assertEqual(output, "Hello, world\n")
474
475In this example, if nativesdk-python3-core has been installed into the SDK, the code runs
476the python3 interpreter with a basic command to check it is working
Andrew Geisslerd5838332022-05-27 11:33:10 -0500477correctly. The test would only run if Python3 is installed in the SDK.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500478
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500479``oe-build-perf-test``
480----------------------
481
482The performance tests usually measure how long operations take and the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500483resource utilization as that happens. An example from
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500484``meta/lib/oeqa/buildperf/test_basic.py`` contains the following::
485
486 class Test3(BuildPerfTestCase):
487 def test3(self):
488 """Bitbake parsing (bitbake -p)"""
489 # Drop all caches and parse
490 self.rm_cache()
491 oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)
492 self.measure_cmd_resources(['bitbake', '-p'], 'parse_1',
493 'bitbake -p (no caches)')
494 # Drop tmp/cache
495 oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)
496 self.measure_cmd_resources(['bitbake', '-p'], 'parse_2',
497 'bitbake -p (no tmp/cache)')
498 # Parse with fully cached data
499 self.measure_cmd_resources(['bitbake', '-p'], 'parse_3',
500 'bitbake -p (cached)')
501
502This example shows how three specific parsing timings are
503measured, with and without various caches, to show how BitBake's parsing
504performance trends over time.
505
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500506Considerations When Writing Tests
507=================================
508
509When writing good tests, there are several things to keep in mind. Since
510things running on the Autobuilder are accessed concurrently by multiple
511workers, consider the following:
512
513**Running "cleanall" is not permitted.**
514
515This can delete files from DL_DIR which would potentially break other
516builds running in parallel. If this is required, DL_DIR must be set to
517an isolated directory.
518
519**Running "cleansstate" is not permitted.**
520
521This can delete files from SSTATE_DIR which would potentially break
522other builds running in parallel. If this is required, SSTATE_DIR must
523be set to an isolated directory. Alternatively, you can use the "-f"
524option with the ``bitbake`` command to "taint" tasks by changing the
525sstate checksums to ensure sstate cache items will not be reused.
526
527**Tests should not change the metadata.**
528
529This is particularly true for oe-selftests since these can run in
530parallel and changing metadata leads to changing checksums, which
531confuses BitBake while running in parallel. If this is necessary, copy
532layers to a temporary location and modify them. Some tests need to
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500533change metadata, such as the devtool tests. To protect the metadata from
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500534changes, set up temporary copies of that data first.