blob: 2444333761b5b4a65e1bd81a49374e45941b7450 [file] [log] [blame]
Andrew Geisslerf0343792020-11-18 10:42:21 -06001.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002
3*******************************************
4Understanding the Yocto Project Autobuilder
5*******************************************
6
7Execution Flow within the Autobuilder
8=====================================
9
10The "a-full" and "a-quick" targets are the usual entry points into the
11Autobuilder and it makes sense to follow the process through the system
12starting there. This is best visualised from the Autobuilder Console
13view (:yocto_ab:`/typhoon/#/console`).
14
15Each item along the top of that view represents some "target build" and
16these targets are all run in parallel. The 'full' build will trigger the
17majority of them, the "quick" build will trigger some subset of them.
18The Autobuilder effectively runs whichever configuration is defined for
19each of those targets on a seperate buildbot worker. To understand the
20configuration, you need to look at the entry on ``config.json`` file
21within the ``yocto-autobuilder-helper`` repository. The targets are
22defined in the ‘overrides' section, a quick example could be qemux86-64
23which looks like::
24
25 "qemux86-64" : {
26 "MACHINE" : "qemux86-64",
27 "TEMPLATE" : "arch-qemu",
28 "step1" : {
29 "extravars" : [
30 "IMAGE_FSTYPES_append = ' wic wic.bmap'"
31 ]
32 }
33 },
34
35And to expand that, you need the "arch-qemu" entry from
36the "templates" section, which looks like::
37
38 "arch-qemu" : {
39 "BUILDINFO" : true,
40 "BUILDHISTORY" : true,
41 "step1" : {
42 "BBTARGETS" : "core-image-sato core-image-sato-dev core-image-sato-sdk core-image-minimal core-image-minimal-dev core-image-sato:do_populate_sdk",
43 "SANITYTARGETS" : "core-image-minimal:do_testimage core-image-sato:do_testimage core-image-sato-sdk:do_testimage core-image-sato:do_testsdk"
44 },
45 "step2" : {
46 "SDKMACHINE" : "x86_64",
47 "BBTARGETS" : "core-image-sato:do_populate_sdk core-image-minimal:do_populate_sdk_ext core-image-sato:do_populate_sdk_ext",
48 "SANITYTARGETS" : "core-image-sato:do_testsdk core-image-minimal:do_testsdkext core-image-sato:do_testsdkext"
49 },
50 "step3" : {
51 "BUILDHISTORY" : false,
52 "EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest ${HELPERSTMACHTARGS} -j 15"],
53 "ADDLAYER" : ["${BUILDDIR}/../meta-selftest"]
54 }
55 },
56
57Combining these two entries you can see that "qemux86-64" is a three step build where the
58``bitbake BBTARGETS`` would be run, then ``bitbake SANITYTARGETS`` for each step; all for
59``MACHINE="qemx86-64"`` but with differing SDKMACHINE settings. In step
601 an extra variable is added to the ``auto.conf`` file to enable wic
61image generation.
62
63While not every detail of this is covered here, you can see how the
64template mechanism allows quite complex configurations to be built up
65yet allows duplication and repetition to be kept to a minimum.
66
67The different build targets are designed to allow for parallelisation,
68so different machines are usually built in parallel, operations using
69the same machine and metadata are built sequentially, with the aim of
70trying to optimise build efficiency as much as possible.
71
72The ``config.json`` file is processed by the scripts in the Helper
73repository in the ``scripts`` directory. The following section details
74how this works.
75
76.. _test-autobuilder-target-exec-overview:
77
78Autobuilder Target Execution Overview
79=====================================
80
81For each given target in a build, the Autobuilder executes several
82steps. These are configured in ``yocto-autobuilder2/builders.py`` and
83roughly consist of:
84
85#. *Run clobberdir*.
86
87 This cleans out any previous build. Old builds are left around to
88 allow easier debugging of failed builds. For additional information,
89 see :ref:`test-manual/test-manual-understand-autobuilder:clobberdir`.
90
91#. *Obtain yocto-autobuilder-helper*
92
93 This step clones the ``yocto-autobuilder-helper`` git repository.
94 This is necessary to prevent the requirement to maintain all the
95 release or project-specific code within Buildbot. The branch chosen
96 matches the release being built so we can support older releases and
97 still make changes in newer ones.
98
99#. *Write layerinfo.json*
100
101 This transfers data in the Buildbot UI when the build was configured
102 to the Helper.
103
104#. *Call scripts/shared-repo-unpack*
105
106 This is a call into the Helper scripts to set up a checkout of all
107 the pieces this build might need. It might clone the BitBake
108 repository and the OpenEmbedded-Core repository. It may clone the
109 Poky repository, as well as additional layers. It will use the data
110 from the ``layerinfo.json`` file to help understand the
111 configuration. It will also use a local cache of repositories to
112 speed up the clone checkouts. For additional information, see
113 :ref:`test-manual/test-manual-understand-autobuilder:Autobuilder Clone Cache`.
114
115 This step has two possible modes of operation. If the build is part
116 of a parent build, its possible that all the repositories needed may
117 already be available, ready in a pre-prepared directory. An "a-quick"
118 or "a-full" build would prepare this before starting the other
119 sub-target builds. This is done for two reasons:
120
121 - the upstream may change during a build, for example, from a forced
122 push and this ensures we have matching content for the whole build
123
124 - if 15 Workers all tried to pull the same data from the same repos,
125 we can hit resource limits on upstream servers as they can think
126 they are under some kind of network attack
127
128 This pre-prepared directory is shared among the Workers over NFS. If
129 the build is an individual build and there is no "shared" directory
130 available, it would clone from the cache and the upstreams as
131 necessary. This is considered the fallback mode.
132
133#. *Call scripts/run-config*
134
135 This is another call into the Helper scripts where its expected that
136 the main functionality of this target will be executed.
137
138.. _test-autobuilder-tech:
139
140Autobuilder Technology
141======================
142
143The Autobuilder has Yocto Project-specific functionality to allow builds
144to operate with increased efficiency and speed.
145
146.. _test-clobberdir:
147
148clobberdir
149----------
150
151When deleting files, the Autobuilder uses ``clobberdir``, which is a
152special script that moves files to a special location, rather than
153deleting them. Files in this location are deleted by an ``rm`` command,
154which is run under ``ionice -c 3``. For example, the deletion only
155happens when there is idle IO capacity on the Worker. The Autobuilder
156Worker Janitor runs this deletion. See :ref:`test-manual/test-manual-understand-autobuilder:Autobuilder Worker Janitor`.
157
158.. _test-autobuilder-clone-cache:
159
160Autobuilder Clone Cache
161-----------------------
162
163Cloning repositories from scratch each time they are required was slow
164on the Autobuilder. We therefore have a stash of commonly used
165repositories pre-cloned on the Workers. Data is fetched from these
166during clones first, then "topped up" with later revisions from any
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500167upstream when necessary. The cache is maintained by the Autobuilder
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500168Worker Janitor. See :ref:`test-manual/test-manual-understand-autobuilder:Autobuilder Worker Janitor`.
169
170.. _test-autobuilder-worker-janitor:
171
172Autobuilder Worker Janitor
173--------------------------
174
175This is a process running on each Worker that performs two basic
176operations, including background file deletion at IO idle (see :ref:`test-manual/test-manual-understand-autobuilder:Autobuilder Target Execution Overview`: Run clobberdir) and
177maintainenance of a cache of cloned repositories to improve the speed
178the system can checkout repositories.
179
180.. _test-shared-dl-dir:
181
182Shared DL_DIR
183-------------
184
185The Workers are all connected over NFS which allows DL_DIR to be shared
186between them. This reduces network accesses from the system and allows
187the build to be sped up. Usage of the directory within the build system
188is designed to be able to be shared over NFS.
189
190.. _test-shared-sstate-cache:
191
192Shared SSTATE_DIR
193-----------------
194
195The Workers are all connected over NFS which allows the ``sstate``
196directory to be shared between them. This means once a Worker has built
197an artifact, all the others can benefit from it. Usage of the directory
198within the directory is designed for sharing over NFS.
199
200.. _test-resulttool:
201
202Resulttool
203----------
204
205All of the different tests run as part of the build generate output into
206``testresults.json`` files. This allows us to determine which tests ran
207in a given build and their status. Additional information, such as
208failure logs or the time taken to run the tests, may also be included.
209
210Resulttool is part of OpenEmbedded-Core and is used to manipulate these
211json results files. It has the ability to merge files together, display
212reports of the test results and compare different result files.
213
214For details, see :yocto_wiki:`/wiki/Resulttool`.
215
216.. _test-run-config-tgt-execution:
217
218run-config Target Execution
219===========================
220
221The ``scripts/run-config`` execution is where most of the work within
222the Autobuilder happens. It runs through a number of steps; the first
223are general setup steps that are run once and include:
224
225#. Set up any ``buildtools-tarball`` if configured.
226
227#. Call "buildhistory-init" if buildhistory is configured.
228
229For each step that is configured in ``config.json``, it will perform the
230following:
231
232#. Add any layers that are specified using the
233 ``bitbake-layers add-layer`` command (logging as stepXa)
234
235#. Call the ``scripts/setup-config`` script to generate the necessary
236 ``auto.conf`` configuration file for the build
237
238#. Run the ``bitbake BBTARGETS`` command (logging as stepXb)
239
240#. Run the ``bitbake SANITYTARGETS`` command (logging as stepXc)
241
242#. Run the ``EXTRACMDS`` command, which are run within the BitBake build
243 environment (logging as stepXd)
244
245#. Run the ``EXTRAPLAINCMDS`` command(s), which are run outside the
246 BitBake build environment (logging as stepXd)
247
248#. Remove any layers added in step
249 1 using the ``bitbake-layers remove-layer`` command (logging as stepXa)
250
251Once the execution steps above complete, ``run-config`` executes a set
252of post-build steps, including:
253
254#. Call ``scripts/publish-artifacts`` to collect any output which is to
255 be saved from the build.
256
257#. Call ``scripts/collect-results`` to collect any test results to be
258 saved from the build.
259
260#. Call ``scripts/upload-error-reports`` to send any error reports
261 generated to the remote server.
262
263#. Cleanup the build directory using
264 :ref:`test-manual/test-manual-understand-autobuilder:clobberdir` if the build was successful,
265 else rename it to "build-renamed" for potential future debugging.
266
267.. _test-deploying-yp-autobuilder:
268
269Deploying Yocto Autobuilder
270===========================
271
272The most up to date information about how to setup and deploy your own
273Autbuilder can be found in README.md in the ``yocto-autobuilder2``
274repository.
275
276We hope that people can use the ``yocto-autobuilder2`` code directly but
277it is inevitable that users will end up needing to heavily customise the
278``yocto-autobuilder-helper`` repository, particularly the
279``config.json`` file as they will want to define their own test matrix.
280
281The Autobuilder supports wo customization options:
282
283- variable substitution
284
285- overlaying configuration files
286
287The standard ``config.json`` minimally attempts to allow substitution of
288the paths. The Helper script repository includes a
289``local-example.json`` file to show how you could override these from a
290separate configuration file. Pass the following into the environment of
291the Autobuilder::
292
293 $ ABHELPER_JSON="config.json local-example.json"
294
295As another example, you could also pass the following into the
296environment::
297
298 $ ABHELPER_JSON="config.json /some/location/local.json"
299
300One issue users often run into is validation of the ``config.json`` files. A
301tip for minimizing issues from invalid json files is to use a Git
302``pre-commit-hook.sh`` script to verify the JSON file before committing
303it. Create a symbolic link as follows::
304
305 $ ln -s ../../scripts/pre-commit-hook.sh .git/hooks/pre-commit