blob: 101d28366522f690b5bcbd4d7df71da9fdf716fb [file] [log] [blame]
Andrew Geisslerf0343792020-11-18 10:42:21 -06001.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002
3*****************************************
4The Yocto Project Test Environment Manual
5*****************************************
6
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007Welcome
8=======
9
10Welcome to the Yocto Project Test Environment Manual! This manual is a
11work in progress. The manual contains information about the testing
12environment used by the Yocto Project to make sure each major and minor
13release works as intended. All the project's testing infrastructure and
14processes are publicly visible and available so that the community can
15see what testing is being performed, how it's being done and the current
16status of the tests and the project at any given time. It is intended
17that Other organizations can leverage off the process and testing
18environment used by the Yocto Project to create their own automated,
19production test environment, building upon the foundations from the
20project core.
21
22Currently, the Yocto Project Test Environment Manual has no projected
23release date. This manual is a work-in-progress and is being initially
24loaded with information from the README files and notes from key
25engineers:
26
27- *yocto-autobuilder2:* This
Andrew Geissler09209ee2020-12-13 08:44:15 -060028 :yocto_git:`README.md </yocto-autobuilder2/tree/README.md>`
Andrew Geissler3b8a17c2021-04-15 15:55:55 -050029 is the main README which details how to set up the Yocto Project
Andrew Geisslerc9f78652020-09-18 14:11:35 -050030 Autobuilder. The ``yocto-autobuilder2`` repository represents the
31 Yocto Project's console UI plugin to Buildbot and the configuration
32 necessary to configure Buildbot to perform the testing the project
33 requires.
34
Andrew Geissler09209ee2020-12-13 08:44:15 -060035- *yocto-autobuilder-helper:* This :yocto_git:`README </yocto-autobuilder-helper/tree/README/>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -050036 and repository contains Yocto Project Autobuilder Helper scripts and
37 configuration. The ``yocto-autobuilder-helper`` repository contains
38 the "glue" logic that defines which tests to run and how to run them.
39 As a result, it can be used by any Continuous Improvement (CI) system
40 to run builds, support getting the correct code revisions, configure
41 builds and layers, run builds, and collect results. The code is
42 independent of any CI system, which means the code can work `Buildbot <https://docs.buildbot.net/0.9.15.post1/>`__,
43 Jenkins, or others. This repository has a branch per release of the
44 project defining the tests to run on a per release basis.
45
Andrew Geisslerc9f78652020-09-18 14:11:35 -050046Yocto Project Autobuilder Overview
47==================================
48
49The Yocto Project Autobuilder collectively refers to the software,
50tools, scripts, and procedures used by the Yocto Project to test
51released software across supported hardware in an automated and regular
52fashion. Basically, during the development of a Yocto Project release,
53the Autobuilder tests if things work. The Autobuilder builds all test
54targets and runs all the tests.
55
56The Yocto Project uses now uses standard upstream
57`Buildbot <https://docs.buildbot.net/0.9.15.post1/>`__ (version 9) to
58drive its integration and testing. Buildbot Nine has a plug-in interface
59that the Yocto Project customizes using code from the
60``yocto-autobuilder2`` repository, adding its own console UI plugin. The
61resulting UI plug-in allows you to visualize builds in a way suited to
62the project's needs.
63
64A ``helper`` layer provides configuration and job management through
65scripts found in the ``yocto-autobuilder-helper`` repository. The
66``helper`` layer contains the bulk of the build configuration
67information and is release-specific, which makes it highly customizable
68on a per-project basis. The layer is CI system-agnostic and contains a
69number of Helper scripts that can generate build configurations from
70simple JSON files.
71
72.. note::
73
74 The project uses Buildbot for historical reasons but also because
75 many of the project developers have knowledge of python. It is
76 possible to use the outer layers from another Continuous Integration
77 (CI) system such as
78 `Jenkins <https://en.wikipedia.org/wiki/Jenkins_(software)>`__
79 instead of Buildbot.
80
81The following figure shows the Yocto Project Autobuilder stack with a
82topology that includes a controller and a cluster of workers:
83
84.. image:: figures/ab-test-cluster.png
85 :align: center
86
Andrew Geisslerc9f78652020-09-18 14:11:35 -050087Yocto Project Tests - Types of Testing Overview
88===============================================
89
90The Autobuilder tests different elements of the project by using
Andrew Geissler3b8a17c2021-04-15 15:55:55 -050091the following types of tests:
Andrew Geisslerc9f78652020-09-18 14:11:35 -050092
93- *Build Testing:* Tests whether specific configurations build by
94 varying :term:`MACHINE`,
95 :term:`DISTRO`, other configuration
96 options, and the specific target images being built (or world). Used
97 to trigger builds of all the different test configurations on the
98 Autobuilder. Builds usually cover many different targets for
99 different architectures, machines, and distributions, as well as
100 different configurations, such as different init systems. The
101 Autobuilder tests literally hundreds of configurations and targets.
102
103 - *Sanity Checks During the Build Process:* Tests initiated through
104 the :ref:`insane <ref-classes-insane>`
105 class. These checks ensure the output of the builds are correct.
106 For example, does the ELF architecture in the generated binaries
107 match the target system? ARM binaries would not work in a MIPS
108 system!
109
110- *Build Performance Testing:* Tests whether or not commonly used steps
111 during builds work efficiently and avoid regressions. Tests to time
112 commonly used usage scenarios are run through ``oe-build-perf-test``.
113 These tests are run on isolated machines so that the time
114 measurements of the tests are accurate and no other processes
115 interfere with the timing results. The project currently tests
116 performance on two different distributions, Fedora and Ubuntu, to
117 ensure we have no single point of failure and can ensure the
118 different distros work effectively.
119
120- *eSDK Testing:* Image tests initiated through the following command::
121
122 $ bitbake image -c testsdkext
123
124 The tests utilize the ``testsdkext`` class and the ``do_testsdkext`` task.
125
126- *Feature Testing:* Various scenario-based tests are run through the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500127 :ref:`OpenEmbedded Self test (oe-selftest) <ref-manual/release-process:Testing and Quality Assurance>`. We test oe-selftest on each of the main distributions
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500128 we support.
129
130- *Image Testing:* Image tests initiated through the following command::
131
132 $ bitbake image -c testimage
133
134 The tests utilize the :ref:`testimage* <ref-classes-testimage*>`
135 classes and the :ref:`ref-tasks-testimage` task.
136
137- *Layer Testing:* The Autobuilder has the possibility to test whether
138 specific layers work with the test of the system. The layers tested
139 may be selected by members of the project. Some key community layers
140 are also tested periodically.
141
142- *Package Testing:* A Package Test (ptest) runs tests against packages
143 built by the OpenEmbedded build system on the target machine. See the
144 :ref:`Testing Packages With
Andrew Geissler09209ee2020-12-13 08:44:15 -0600145 ptest <dev-manual/common-tasks:Testing Packages With ptest>` section
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500146 in the Yocto Project Development Tasks Manual and the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600147 ":yocto_wiki:`Ptest </Ptest>`" Wiki page for more
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500148 information on Ptest.
149
150- *SDK Testing:* Image tests initiated through the following command::
151
152 $ bitbake image -c testsdk
153
154 The tests utilize the :ref:`testsdk <ref-classes-testsdk>` class and
155 the ``do_testsdk`` task.
156
157- *Unit Testing:* Unit tests on various components of the system run
Andrew Geissler09209ee2020-12-13 08:44:15 -0600158 through :ref:`bitbake-selftest <ref-manual/release-process:Testing and Quality Assurance>` and
159 :ref:`oe-selftest <ref-manual/release-process:Testing and Quality Assurance>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500160
161- *Automatic Upgrade Helper:* This target tests whether new versions of
162 software are available and whether we can automatically upgrade to
163 those new versions. If so, this target emails the maintainers with a
164 patch to let them know this is possible.
165
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500166How Tests Map to Areas of Code
167==============================
168
169Tests map into the codebase as follows:
170
171- *bitbake-selftest:*
172
173 These tests are self-contained and test BitBake as well as its APIs,
174 which include the fetchers. The tests are located in
175 ``bitbake/lib/*/tests``.
176
177 From within the BitBake repository, run the following::
178
179 $ bitbake-selftest
180
181 To skip tests that access the Internet, use the ``BB_SKIP_NETTEST``
182 variable when running "bitbake-selftest" as follows::
183
184 $ BB_SKIP_NETTEST=yes bitbake-selftest
185
186 The default output is quiet and just prints a summary of what was
187 run. To see more information, there is a verbose option::
188
189 $ bitbake-selftest -v
190
191 Use this option when you wish to skip tests that access the network,
192 which are mostly necessary to test the fetcher modules. To specify
193 individual test modules to run, append the test module name to the
194 "bitbake-selftest" command. For example, to specify the tests for the
195 bb.data.module, run::
196
197 $ bitbake-selftest bb.test.data.module
198
199 You can also specify individual tests by defining the full name and module
200 plus the class path of the test, for example::
201
202 $ bitbake-selftest bb.tests.data.TestOverrides.test_one_override
203
204 The tests are based on `Python
205 unittest <https://docs.python.org/3/library/unittest.html>`__.
206
207- *oe-selftest:*
208
209 - These tests use OE to test the workflows, which include testing
210 specific features, behaviors of tasks, and API unit tests.
211
212 - The tests can take advantage of parallelism through the "-j"
213 option, which can specify a number of threads to spread the tests
214 across. Note that all tests from a given class of tests will run
215 in the same thread. To parallelize large numbers of tests you can
216 split the class into multiple units.
217
218 - The tests are based on Python unittest.
219
220 - The code for the tests resides in
221 ``meta/lib/oeqa/selftest/cases/``.
222
223 - To run all the tests, enter the following command::
224
225 $ oe-selftest -a
226
227 - To run a specific test, use the following command form where
228 testname is the name of the specific test::
229
230 $ oe-selftest -r <testname>
231
232 For example, the following command would run the tinfoil
233 getVar API test::
234
235 $ oe-selftest -r tinfoil.TinfoilTests.test_getvar
236
237 It is also possible to run a set
238 of tests. For example the following command will run all of the
239 tinfoil tests::
240
241 $ oe-selftest -r tinfoil
242
243- *testimage:*
244
245 - These tests build an image, boot it, and run tests against the
246 image's content.
247
248 - The code for these tests resides in ``meta/lib/oeqa/runtime/cases/``.
249
250 - You need to set the :term:`IMAGE_CLASSES` variable as follows::
251
252 IMAGE_CLASSES += "testimage"
253
254 - Run the tests using the following command form::
255
256 $ bitbake image -c testimage
257
258- *testsdk:*
259
260 - These tests build an SDK, install it, and then run tests against
261 that SDK.
262
263 - The code for these tests resides in ``meta/lib/oeqa/sdk/cases/``.
264
265 - Run the test using the following command form::
266
267 $ bitbake image -c testsdk
268
269- *testsdk_ext:*
270
271 - These tests build an extended SDK (eSDK), install that eSDK, and
272 run tests against the eSDK.
273
274 - The code for these tests resides in ``meta/lib/oeqa/esdk``.
275
276 - To run the tests, use the following command form::
277
278 $ bitbake image -c testsdkext
279
280- *oe-build-perf-test:*
281
282 - These tests run through commonly used usage scenarios and measure
283 the performance times.
284
285 - The code for these tests resides in ``meta/lib/oeqa/buildperf``.
286
287 - To run the tests, use the following command form::
288
289 $ oe-build-perf-test <options>
290
291 The command takes a number of options,
292 such as where to place the test results. The Autobuilder Helper
293 Scripts include the ``build-perf-test-wrapper`` script with
294 examples of how to use the oe-build-perf-test from the command
295 line.
296
297 Use the ``oe-git-archive`` command to store test results into a
298 Git repository.
299
300 Use the ``oe-build-perf-report`` command to generate text reports
301 and HTML reports with graphs of the performance data. For
302 examples, see
303 :yocto_dl:`/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.html`
304 and
305 :yocto_dl:`/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.txt`.
306
307 - The tests are contained in ``lib/oeqa/buildperf/test_basic.py``.
308
309Test Examples
310=============
311
312This section provides example tests for each of the tests listed in the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600313:ref:`test-manual/intro:How Tests Map to Areas of Code` section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500314
315For oeqa tests, testcases for each area reside in the main test
316directory at ``meta/lib/oeqa/selftest/cases`` directory.
317
318For oe-selftest. bitbake testcases reside in the ``lib/bb/tests/``
319directory.
320
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500321``bitbake-selftest``
322--------------------
323
324A simple test example from ``lib/bb/tests/data.py`` is::
325
326 class DataExpansions(unittest.TestCase):
327 def setUp(self):
328 self.d = bb.data.init()
329 self.d["foo"] = "value_of_foo"
330 self.d["bar"] = "value_of_bar"
331 self.d["value_of_foo"] = "value_of_'value_of_foo'"
332
333 def test_one_var(self):
334 val = self.d.expand("${foo}")
335 self.assertEqual(str(val), "value_of_foo")
336
337In this example, a ``DataExpansions`` class of tests is created,
338derived from standard python unittest. The class has a common ``setUp``
339function which is shared by all the tests in the class. A simple test is
340then added to test that when a variable is expanded, the correct value
341is found.
342
343Bitbake selftests are straightforward python unittest. Refer to the
344Python unittest documentation for additional information on writing
345these tests at: https://docs.python.org/3/library/unittest.html.
346
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500347``oe-selftest``
348---------------
349
350These tests are more complex due to the setup required behind the scenes
351for full builds. Rather than directly using Python's unittest, the code
352wraps most of the standard objects. The tests can be simple, such as
353testing a command from within the OE build environment using the
354following example::
355
356 class BitbakeLayers(OESelftestTestCase):
357 def test_bitbakelayers_showcrossdepends(self):
358 result = runCmd('bitbake-layers show-cross-depends')
359 self.assertTrue('aspell' in result.output, msg = "No dependencies were shown. bitbake-layers show-cross-depends output: %s"% result.output)
360
361This example, taken from ``meta/lib/oeqa/selftest/cases/bblayers.py``,
362creates a testcase from the ``OESelftestTestCase`` class, derived
363from ``unittest.TestCase``, which runs the ``bitbake-layers`` command
364and checks the output to ensure it contains something we know should be
365here.
366
367The ``oeqa.utils.commands`` module contains Helpers which can assist
368with common tasks, including:
369
370- *Obtaining the value of a bitbake variable:* Use
371 ``oeqa.utils.commands.get_bb_var()`` or use
372 ``oeqa.utils.commands.get_bb_vars()`` for more than one variable
373
374- *Running a bitbake invocation for a build:* Use
375 ``oeqa.utils.commands.bitbake()``
376
377- *Running a command:* Use ``oeqa.utils.commandsrunCmd()``
378
379There is also a ``oeqa.utils.commands.runqemu()`` function for launching
380the ``runqemu`` command for testing things within a running, virtualized
381image.
382
383You can run these tests in parallel. Parallelism works per test class,
384so tests within a given test class should always run in the same build,
385while tests in different classes or modules may be split into different
386builds. There is no data store available for these tests since the tests
387launch the ``bitbake`` command and exist outside of its context. As a
388result, common bitbake library functions (bb.\*) are also unavailable.
389
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500390``testimage``
391-------------
392
393These tests are run once an image is up and running, either on target
394hardware or under QEMU. As a result, they are assumed to be running in a
395target image environment, as opposed to a host build environment. A
396simple example from ``meta/lib/oeqa/runtime/cases/python.py`` contains
397the following::
398
399 class PythonTest(OERuntimeTestCase):
400 @OETestDepends(['ssh.SSHTest.test_ssh'])
401 @OEHasPackage(['python3-core'])
402 def test_python3(self):
403 cmd = "python3 -c \\"import codecs; print(codecs.encode('Uryyb, jbeyq', 'rot13'))\""
404 status, output = self.target.run(cmd)
405 msg = 'Exit status was not 0. Output: %s' % output
406 self.assertEqual(status, 0, msg=msg)
407
408In this example, the ``OERuntimeTestCase`` class wraps
409``unittest.TestCase``. Within the test, ``self.target`` represents the
410target system, where commands can be run on it using the ``run()``
411method.
412
413To ensure certain test or package dependencies are met, you can use the
414``OETestDepends`` and ``OEHasPackage`` decorators. For example, the test
415in this example would only make sense if python3-core is installed in
416the image.
417
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500418``testsdk_ext``
419---------------
420
421These tests are run against built extensible SDKs (eSDKs). The tests can
422assume that the eSDK environment has already been setup. An example from
423``meta/lib/oeqa/sdk/cases/devtool.py`` contains the following::
424
425 class DevtoolTest(OESDKExtTestCase):
426 @classmethod def setUpClass(cls):
427 myapp_src = os.path.join(cls.tc.esdk_files_dir, "myapp")
428 cls.myapp_dst = os.path.join(cls.tc.sdk_dir, "myapp")
429 shutil.copytree(myapp_src, cls.myapp_dst)
430 subprocess.check_output(['git', 'init', '.'], cwd=cls.myapp_dst)
431 subprocess.check_output(['git', 'add', '.'], cwd=cls.myapp_dst)
432 subprocess.check_output(['git', 'commit', '-m', "'test commit'"], cwd=cls.myapp_dst)
433
434 @classmethod
435 def tearDownClass(cls):
436 shutil.rmtree(cls.myapp_dst)
437 def _test_devtool_build(self, directory):
438 self._run('devtool add myapp %s' % directory)
439 try:
440 self._run('devtool build myapp')
441 finally:
442 self._run('devtool reset myapp')
443 def test_devtool_build_make(self):
444 self._test_devtool_build(self.myapp_dst)
445
446In this example, the ``devtool``
447command is tested to see whether a sample application can be built with
448the ``devtool build`` command within the eSDK.
449
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500450``testsdk``
451-----------
452
453These tests are run against built SDKs. The tests can assume that an SDK
454has already been extracted and its environment file has been sourced. A
455simple example from ``meta/lib/oeqa/sdk/cases/python2.py`` contains the
456following::
457
458 class Python3Test(OESDKTestCase):
459 def setUp(self):
460 if not (self.tc.hasHostPackage("nativesdk-python3-core") or
461 self.tc.hasHostPackage("python3-core-native")):
462 raise unittest.SkipTest("No python3 package in the SDK")
463
464 def test_python3(self):
465 cmd = "python3 -c \\"import codecs; print(codecs.encode('Uryyb, jbeyq', 'rot13'))\""
466 output = self._run(cmd)
467 self.assertEqual(output, "Hello, world\n")
468
469In this example, if nativesdk-python3-core has been installed into the SDK, the code runs
470the python3 interpreter with a basic command to check it is working
471correctly. The test would only run if python3 is installed in the SDK.
472
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500473``oe-build-perf-test``
474----------------------
475
476The performance tests usually measure how long operations take and the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500477resource utilization as that happens. An example from
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500478``meta/lib/oeqa/buildperf/test_basic.py`` contains the following::
479
480 class Test3(BuildPerfTestCase):
481 def test3(self):
482 """Bitbake parsing (bitbake -p)"""
483 # Drop all caches and parse
484 self.rm_cache()
485 oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)
486 self.measure_cmd_resources(['bitbake', '-p'], 'parse_1',
487 'bitbake -p (no caches)')
488 # Drop tmp/cache
489 oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)
490 self.measure_cmd_resources(['bitbake', '-p'], 'parse_2',
491 'bitbake -p (no tmp/cache)')
492 # Parse with fully cached data
493 self.measure_cmd_resources(['bitbake', '-p'], 'parse_3',
494 'bitbake -p (cached)')
495
496This example shows how three specific parsing timings are
497measured, with and without various caches, to show how BitBake's parsing
498performance trends over time.
499
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500500Considerations When Writing Tests
501=================================
502
503When writing good tests, there are several things to keep in mind. Since
504things running on the Autobuilder are accessed concurrently by multiple
505workers, consider the following:
506
507**Running "cleanall" is not permitted.**
508
509This can delete files from DL_DIR which would potentially break other
510builds running in parallel. If this is required, DL_DIR must be set to
511an isolated directory.
512
513**Running "cleansstate" is not permitted.**
514
515This can delete files from SSTATE_DIR which would potentially break
516other builds running in parallel. If this is required, SSTATE_DIR must
517be set to an isolated directory. Alternatively, you can use the "-f"
518option with the ``bitbake`` command to "taint" tasks by changing the
519sstate checksums to ensure sstate cache items will not be reused.
520
521**Tests should not change the metadata.**
522
523This is particularly true for oe-selftests since these can run in
524parallel and changing metadata leads to changing checksums, which
525confuses BitBake while running in parallel. If this is necessary, copy
526layers to a temporary location and modify them. Some tests need to
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500527change metadata, such as the devtool tests. To protect the metadata from
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500528changes, set up temporary copies of that data first.