Andrew Geissler | f034379 | 2020-11-18 10:42:21 -0600 | [diff] [blame] | 1 | .. SPDX-License-Identifier: CC-BY-SA-2.0-UK |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 2 | |
| 3 | ******************************************* |
| 4 | Understanding the Yocto Project Autobuilder |
| 5 | ******************************************* |
| 6 | |
| 7 | Execution Flow within the Autobuilder |
| 8 | ===================================== |
| 9 | |
| 10 | The "a-full" and "a-quick" targets are the usual entry points into the |
| 11 | Autobuilder and it makes sense to follow the process through the system |
Patrick Williams | 169d7bc | 2024-01-05 11:33:25 -0600 | [diff] [blame] | 12 | starting there. This is best visualized from the :yocto_ab:`Autobuilder |
| 13 | Console view </typhoon/#/console>`. |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 14 | |
| 15 | Each item along the top of that view represents some "target build" and |
| 16 | these targets are all run in parallel. The 'full' build will trigger the |
| 17 | majority of them, the "quick" build will trigger some subset of them. |
| 18 | The Autobuilder effectively runs whichever configuration is defined for |
Andrew Geissler | 3b8a17c | 2021-04-15 15:55:55 -0500 | [diff] [blame] | 19 | each of those targets on a separate buildbot worker. To understand the |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 20 | configuration, you need to look at the entry on ``config.json`` file |
Patrick Williams | 169d7bc | 2024-01-05 11:33:25 -0600 | [diff] [blame] | 21 | within the :yocto_git:`yocto-autobuilder-helper </yocto-autobuilder-helper>` |
| 22 | repository. The targets are defined in the ``overrides`` section, a quick |
| 23 | example could be ``qemux86-64`` which looks like:: |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 24 | |
| 25 | "qemux86-64" : { |
| 26 | "MACHINE" : "qemux86-64", |
| 27 | "TEMPLATE" : "arch-qemu", |
| 28 | "step1" : { |
| 29 | "extravars" : [ |
Patrick Williams | 0ca19cc | 2021-08-16 14:03:13 -0500 | [diff] [blame] | 30 | "IMAGE_FSTYPES:append = ' wic wic.bmap'" |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 31 | ] |
| 32 | } |
| 33 | }, |
| 34 | |
Patrick Williams | 169d7bc | 2024-01-05 11:33:25 -0600 | [diff] [blame] | 35 | And to expand that, you need the ``arch-qemu`` entry from |
| 36 | the ``templates`` section, which looks like:: |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 37 | |
| 38 | "arch-qemu" : { |
| 39 | "BUILDINFO" : true, |
| 40 | "BUILDHISTORY" : true, |
| 41 | "step1" : { |
| 42 | "BBTARGETS" : "core-image-sato core-image-sato-dev core-image-sato-sdk core-image-minimal core-image-minimal-dev core-image-sato:do_populate_sdk", |
| 43 | "SANITYTARGETS" : "core-image-minimal:do_testimage core-image-sato:do_testimage core-image-sato-sdk:do_testimage core-image-sato:do_testsdk" |
| 44 | }, |
| 45 | "step2" : { |
| 46 | "SDKMACHINE" : "x86_64", |
| 47 | "BBTARGETS" : "core-image-sato:do_populate_sdk core-image-minimal:do_populate_sdk_ext core-image-sato:do_populate_sdk_ext", |
| 48 | "SANITYTARGETS" : "core-image-sato:do_testsdk core-image-minimal:do_testsdkext core-image-sato:do_testsdkext" |
| 49 | }, |
| 50 | "step3" : { |
| 51 | "BUILDHISTORY" : false, |
| 52 | "EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest ${HELPERSTMACHTARGS} -j 15"], |
| 53 | "ADDLAYER" : ["${BUILDDIR}/../meta-selftest"] |
| 54 | } |
| 55 | }, |
| 56 | |
Patrick Williams | 169d7bc | 2024-01-05 11:33:25 -0600 | [diff] [blame] | 57 | Combining these two entries you can see that ``qemux86-64`` is a three step |
| 58 | build where ``bitbake BBTARGETS`` would be run, then ``bitbake SANITYTARGETS`` |
| 59 | for each step; all for ``MACHINE="qemux86-64"`` but with differing |
| 60 | :term:`SDKMACHINE` settings. In step 1, an extra variable is added to the |
| 61 | ``auto.conf`` file to enable wic image generation. |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 62 | |
| 63 | While not every detail of this is covered here, you can see how the |
| 64 | template mechanism allows quite complex configurations to be built up |
| 65 | yet allows duplication and repetition to be kept to a minimum. |
| 66 | |
Andrew Geissler | 3b8a17c | 2021-04-15 15:55:55 -0500 | [diff] [blame] | 67 | The different build targets are designed to allow for parallelization, |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 68 | so different machines are usually built in parallel, operations using |
| 69 | the same machine and metadata are built sequentially, with the aim of |
Andrew Geissler | 3b8a17c | 2021-04-15 15:55:55 -0500 | [diff] [blame] | 70 | trying to optimize build efficiency as much as possible. |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 71 | |
| 72 | The ``config.json`` file is processed by the scripts in the Helper |
| 73 | repository in the ``scripts`` directory. The following section details |
| 74 | how this works. |
| 75 | |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 76 | Autobuilder Target Execution Overview |
| 77 | ===================================== |
| 78 | |
| 79 | For each given target in a build, the Autobuilder executes several |
| 80 | steps. These are configured in ``yocto-autobuilder2/builders.py`` and |
| 81 | roughly consist of: |
| 82 | |
| 83 | #. *Run clobberdir*. |
| 84 | |
| 85 | This cleans out any previous build. Old builds are left around to |
| 86 | allow easier debugging of failed builds. For additional information, |
Andrew Geissler | 09209ee | 2020-12-13 08:44:15 -0600 | [diff] [blame] | 87 | see :ref:`test-manual/understand-autobuilder:clobberdir`. |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 88 | |
| 89 | #. *Obtain yocto-autobuilder-helper* |
| 90 | |
Patrick Williams | 169d7bc | 2024-01-05 11:33:25 -0600 | [diff] [blame] | 91 | This step clones the :yocto_git:`yocto-autobuilder-helper </yocto-autobuilder-helper>` |
| 92 | git repository. This is necessary to avoid the requirement to maintain all |
| 93 | the release or project-specific code within Buildbot. The branch chosen |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 94 | matches the release being built so we can support older releases and |
| 95 | still make changes in newer ones. |
| 96 | |
| 97 | #. *Write layerinfo.json* |
| 98 | |
| 99 | This transfers data in the Buildbot UI when the build was configured |
| 100 | to the Helper. |
| 101 | |
| 102 | #. *Call scripts/shared-repo-unpack* |
| 103 | |
| 104 | This is a call into the Helper scripts to set up a checkout of all |
| 105 | the pieces this build might need. It might clone the BitBake |
| 106 | repository and the OpenEmbedded-Core repository. It may clone the |
| 107 | Poky repository, as well as additional layers. It will use the data |
| 108 | from the ``layerinfo.json`` file to help understand the |
| 109 | configuration. It will also use a local cache of repositories to |
| 110 | speed up the clone checkouts. For additional information, see |
Andrew Geissler | 09209ee | 2020-12-13 08:44:15 -0600 | [diff] [blame] | 111 | :ref:`test-manual/understand-autobuilder:Autobuilder Clone Cache`. |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 112 | |
| 113 | This step has two possible modes of operation. If the build is part |
Andrew Geissler | 3b8a17c | 2021-04-15 15:55:55 -0500 | [diff] [blame] | 114 | of a parent build, it's possible that all the repositories needed may |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 115 | already be available, ready in a pre-prepared directory. An "a-quick" |
| 116 | or "a-full" build would prepare this before starting the other |
| 117 | sub-target builds. This is done for two reasons: |
| 118 | |
| 119 | - the upstream may change during a build, for example, from a forced |
| 120 | push and this ensures we have matching content for the whole build |
| 121 | |
| 122 | - if 15 Workers all tried to pull the same data from the same repos, |
| 123 | we can hit resource limits on upstream servers as they can think |
| 124 | they are under some kind of network attack |
| 125 | |
| 126 | This pre-prepared directory is shared among the Workers over NFS. If |
| 127 | the build is an individual build and there is no "shared" directory |
| 128 | available, it would clone from the cache and the upstreams as |
| 129 | necessary. This is considered the fallback mode. |
| 130 | |
| 131 | #. *Call scripts/run-config* |
| 132 | |
Andrew Geissler | 3b8a17c | 2021-04-15 15:55:55 -0500 | [diff] [blame] | 133 | This is another call into the Helper scripts where it's expected that |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 134 | the main functionality of this target will be executed. |
| 135 | |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 136 | Autobuilder Technology |
| 137 | ====================== |
| 138 | |
| 139 | The Autobuilder has Yocto Project-specific functionality to allow builds |
| 140 | to operate with increased efficiency and speed. |
| 141 | |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 142 | clobberdir |
| 143 | ---------- |
| 144 | |
| 145 | When deleting files, the Autobuilder uses ``clobberdir``, which is a |
| 146 | special script that moves files to a special location, rather than |
| 147 | deleting them. Files in this location are deleted by an ``rm`` command, |
| 148 | which is run under ``ionice -c 3``. For example, the deletion only |
| 149 | happens when there is idle IO capacity on the Worker. The Autobuilder |
Andrew Geissler | 09209ee | 2020-12-13 08:44:15 -0600 | [diff] [blame] | 150 | Worker Janitor runs this deletion. See :ref:`test-manual/understand-autobuilder:Autobuilder Worker Janitor`. |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 151 | |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 152 | Autobuilder Clone Cache |
| 153 | ----------------------- |
| 154 | |
| 155 | Cloning repositories from scratch each time they are required was slow |
| 156 | on the Autobuilder. We therefore have a stash of commonly used |
| 157 | repositories pre-cloned on the Workers. Data is fetched from these |
| 158 | during clones first, then "topped up" with later revisions from any |
Andrew Geissler | 4c19ea1 | 2020-10-27 13:52:24 -0500 | [diff] [blame] | 159 | upstream when necessary. The cache is maintained by the Autobuilder |
Andrew Geissler | 09209ee | 2020-12-13 08:44:15 -0600 | [diff] [blame] | 160 | Worker Janitor. See :ref:`test-manual/understand-autobuilder:Autobuilder Worker Janitor`. |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 161 | |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 162 | Autobuilder Worker Janitor |
| 163 | -------------------------- |
| 164 | |
| 165 | This is a process running on each Worker that performs two basic |
Patrick Williams | 169d7bc | 2024-01-05 11:33:25 -0600 | [diff] [blame] | 166 | operations, including background file deletion at IO idle (see |
| 167 | "Run clobberdir" in :ref:`test-manual/understand-autobuilder:Autobuilder Target Execution Overview`) |
| 168 | and maintenance of a cache of cloned repositories to improve the speed |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 169 | the system can checkout repositories. |
| 170 | |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 171 | Shared DL_DIR |
| 172 | ------------- |
| 173 | |
Andrew Geissler | 517393d | 2023-01-13 08:55:19 -0600 | [diff] [blame] | 174 | The Workers are all connected over NFS which allows :term:`DL_DIR` to be shared |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 175 | between them. This reduces network accesses from the system and allows |
Patrick Williams | 169d7bc | 2024-01-05 11:33:25 -0600 | [diff] [blame] | 176 | the build to be sped up. The usage of the directory within the build system |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 177 | is designed to be able to be shared over NFS. |
| 178 | |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 179 | Shared SSTATE_DIR |
| 180 | ----------------- |
| 181 | |
| 182 | The Workers are all connected over NFS which allows the ``sstate`` |
| 183 | directory to be shared between them. This means once a Worker has built |
Patrick Williams | 169d7bc | 2024-01-05 11:33:25 -0600 | [diff] [blame] | 184 | an artifact, all the others can benefit from it. The usage of the directory |
| 185 | within the build system is designed for sharing over NFS. |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 186 | |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 187 | Resulttool |
| 188 | ---------- |
| 189 | |
| 190 | All of the different tests run as part of the build generate output into |
| 191 | ``testresults.json`` files. This allows us to determine which tests ran |
| 192 | in a given build and their status. Additional information, such as |
| 193 | failure logs or the time taken to run the tests, may also be included. |
| 194 | |
| 195 | Resulttool is part of OpenEmbedded-Core and is used to manipulate these |
Patrick Williams | 169d7bc | 2024-01-05 11:33:25 -0600 | [diff] [blame] | 196 | JSON results files. It has the ability to merge files together, display |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 197 | reports of the test results and compare different result files. |
| 198 | |
Andrew Geissler | 09209ee | 2020-12-13 08:44:15 -0600 | [diff] [blame] | 199 | For details, see :yocto_wiki:`/Resulttool`. |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 200 | |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 201 | run-config Target Execution |
| 202 | =========================== |
| 203 | |
| 204 | The ``scripts/run-config`` execution is where most of the work within |
| 205 | the Autobuilder happens. It runs through a number of steps; the first |
| 206 | are general setup steps that are run once and include: |
| 207 | |
Andrew Geissler | 517393d | 2023-01-13 08:55:19 -0600 | [diff] [blame] | 208 | #. Set up any :term:`buildtools` tarball if configured. |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 209 | |
Patrick Williams | 169d7bc | 2024-01-05 11:33:25 -0600 | [diff] [blame] | 210 | #. Call ``buildhistory-init`` if :ref:`ref-classes-buildhistory` is configured. |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 211 | |
| 212 | For each step that is configured in ``config.json``, it will perform the |
| 213 | following: |
| 214 | |
| 215 | #. Add any layers that are specified using the |
| 216 | ``bitbake-layers add-layer`` command (logging as stepXa) |
| 217 | |
| 218 | #. Call the ``scripts/setup-config`` script to generate the necessary |
| 219 | ``auto.conf`` configuration file for the build |
| 220 | |
| 221 | #. Run the ``bitbake BBTARGETS`` command (logging as stepXb) |
| 222 | |
| 223 | #. Run the ``bitbake SANITYTARGETS`` command (logging as stepXc) |
| 224 | |
| 225 | #. Run the ``EXTRACMDS`` command, which are run within the BitBake build |
| 226 | environment (logging as stepXd) |
| 227 | |
| 228 | #. Run the ``EXTRAPLAINCMDS`` command(s), which are run outside the |
| 229 | BitBake build environment (logging as stepXd) |
| 230 | |
| 231 | #. Remove any layers added in step |
| 232 | 1 using the ``bitbake-layers remove-layer`` command (logging as stepXa) |
| 233 | |
| 234 | Once the execution steps above complete, ``run-config`` executes a set |
| 235 | of post-build steps, including: |
| 236 | |
| 237 | #. Call ``scripts/publish-artifacts`` to collect any output which is to |
| 238 | be saved from the build. |
| 239 | |
| 240 | #. Call ``scripts/collect-results`` to collect any test results to be |
| 241 | saved from the build. |
| 242 | |
| 243 | #. Call ``scripts/upload-error-reports`` to send any error reports |
| 244 | generated to the remote server. |
| 245 | |
Patrick Williams | 2390b1b | 2022-11-03 13:47:49 -0500 | [diff] [blame] | 246 | #. Cleanup the :term:`Build Directory` using |
Andrew Geissler | 09209ee | 2020-12-13 08:44:15 -0600 | [diff] [blame] | 247 | :ref:`test-manual/understand-autobuilder:clobberdir` if the build was successful, |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 248 | else rename it to "build-renamed" for potential future debugging. |
| 249 | |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 250 | Deploying Yocto Autobuilder |
| 251 | =========================== |
| 252 | |
| 253 | The most up to date information about how to setup and deploy your own |
Patrick Williams | 169d7bc | 2024-01-05 11:33:25 -0600 | [diff] [blame] | 254 | Autobuilder can be found in :yocto_git:`README.md </yocto-autobuilder2/tree/README.md>` |
| 255 | in the :yocto_git:`yocto-autobuilder2 </yocto-autobuilder2>` repository. |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 256 | |
Patrick Williams | 169d7bc | 2024-01-05 11:33:25 -0600 | [diff] [blame] | 257 | We hope that people can use the :yocto_git:`yocto-autobuilder2 </yocto-autobuilder2>` |
| 258 | code directly but it is inevitable that users will end up needing to heavily |
| 259 | customize the :yocto_git:`yocto-autobuilder-helper </yocto-autobuilder-helper>` |
| 260 | repository, particularly the ``config.json`` file as they will want to define |
| 261 | their own test matrix. |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 262 | |
Patrick Williams | 169d7bc | 2024-01-05 11:33:25 -0600 | [diff] [blame] | 263 | The Autobuilder supports two customization options: |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 264 | |
| 265 | - variable substitution |
| 266 | |
| 267 | - overlaying configuration files |
| 268 | |
| 269 | The standard ``config.json`` minimally attempts to allow substitution of |
| 270 | the paths. The Helper script repository includes a |
| 271 | ``local-example.json`` file to show how you could override these from a |
| 272 | separate configuration file. Pass the following into the environment of |
| 273 | the Autobuilder:: |
| 274 | |
| 275 | $ ABHELPER_JSON="config.json local-example.json" |
| 276 | |
| 277 | As another example, you could also pass the following into the |
| 278 | environment:: |
| 279 | |
| 280 | $ ABHELPER_JSON="config.json /some/location/local.json" |
| 281 | |
| 282 | One issue users often run into is validation of the ``config.json`` files. A |
Patrick Williams | 169d7bc | 2024-01-05 11:33:25 -0600 | [diff] [blame] | 283 | tip for minimizing issues from invalid JSON files is to use a Git |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 284 | ``pre-commit-hook.sh`` script to verify the JSON file before committing |
| 285 | it. Create a symbolic link as follows:: |
| 286 | |
| 287 | $ ln -s ../../scripts/pre-commit-hook.sh .git/hooks/pre-commit |