| <!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" |
| "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd" |
| [<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] > |
| <!--SPDX-License-Identifier: CC-BY-2.0-UK--> |
| |
| <chapter id='test-manual-understand-autobuilder'> |
| |
| <title>Understanding the Yocto Project Autobuilder</title> |
| <section> |
| <title>Execution Flow within the Autobuilder</title> |
| <para>The "a-full" and "a-quick" targets are the usual entry points into the Autobuilder and |
| it makes sense to follow the process through the system starting there. This is best |
| visualised from the Autobuilder Console view (<link linkend="" |
| >https://autobuilder.yoctoproject.org/typhoon/#/console</link>). </para> |
| <para>Each item along the top of that view represents some "target build" and these targets |
| are all run in parallel. The 'full' build will trigger the majority of them, the "quick" |
| build will trigger some subset of them. The Autobuilder effectively runs whichever |
| configuration is defined for each of those targets on a seperate buildbot worker. To |
| understand the configuration, you need to look at the entry on |
| <filename>config.json</filename> file within the |
| <filename>yocto-autobuilder-helper</filename> repository. The targets are defined in |
| the ‘overrides' section, a quick example could be qemux86-64 which looks |
| like:<literallayout class="monospaced"> |
| "qemux86-64" : { |
| "MACHINE" : "qemux86-64", |
| "TEMPLATE" : "arch-qemu", |
| "step1" : { |
| "extravars" : [ |
| "IMAGE_FSTYPES_append = ' wic wic.bmap'" |
| ] |
| } |
| }, |
| </literallayout>And |
| to expand that, you need the "arch-qemu" entry from the "templates" section, which looks |
| like:<literallayout class="monospaced"> |
| "arch-qemu" : { |
| "BUILDINFO" : true, |
| "BUILDHISTORY" : true, |
| "step1" : { |
| "BBTARGETS" : "core-image-sato core-image-sato-dev core-image-sato-sdk core-image-minimal core-image-minimal-dev core-image-sato:do_populate_sdk", |
| "SANITYTARGETS" : "core-image-minimal:do_testimage core-image-sato:do_testimage core-image-sato-sdk:do_testimage core-image-sato:do_testsdk" |
| }, |
| "step2" : { |
| "SDKMACHINE" : "x86_64", |
| "BBTARGETS" : "core-image-sato:do_populate_sdk core-image-minimal:do_populate_sdk_ext core-image-sato:do_populate_sdk_ext", |
| "SANITYTARGETS" : "core-image-sato:do_testsdk core-image-minimal:do_testsdkext core-image-sato:do_testsdkext" |
| }, |
| "step3" : { |
| "BUILDHISTORY" : false, |
| "EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest ${HELPERSTMACHTARGS} -j 15"], |
| "ADDLAYER" : ["${BUILDDIR}/../meta-selftest"] |
| } |
| }, |
| </literallayout>Combining |
| these two entries you can see that "qemux86-64" is a three step build where the |
| <filename>bitbake BBTARGETS</filename> would be run, then <filename>bitbake |
| SANITYTARGETS</filename> for each step; all for |
| <filename>MACHINE="qemx86-64"</filename> but with differing SDKMACHINE settings. In |
| step 1 an extra variable is added to the <filename>auto.conf</filename> file to enable |
| wic image generation.</para> |
| <para>While not every detail of this is covered here, you can see how the templating |
| mechanism allows quite complex configurations to be built up yet allows duplication and |
| repetition to be kept to a minimum.</para> |
| <para>The different build targets are designed to allow for parallelisation, so different |
| machines are usually built in parallel, operations using the same machine and metadata |
| are built sequentially, with the aim of trying to optimise build efficiency as much as |
| possible.</para> |
| <para>The <filename>config.json</filename> file is processed by the scripts in the Helper |
| repository in the <filename>scripts</filename> directory. The following section details |
| how this works.</para> |
| </section> |
| |
| <section id='test-autobuilder-target-exec-overview'> |
| <title>Autobuilder Target Execution Overview</title> |
| |
| <para>For each given target in a build, the Autobuilder executes several steps. These are |
| configured in <filename>yocto-autobuilder2/builders.py</filename> and roughly consist |
| of: <orderedlist> |
| <listitem id='test-list-tgt-exec-clobberdir'> |
| <para><emphasis>Run <filename>clobberdir</filename></emphasis></para> |
| <para>This cleans out any previous build. Old builds are left around to allow |
| easier debugging of failed builds. For additional information, see <link |
| linkend="test-clobberdir"><filename>clobberdir</filename></link>.</para> |
| </listitem> |
| <listitem> |
| <para><emphasis>Obtain yocto-autobuilder-helper</emphasis></para> |
| <para>This step clones the <filename>yocto-autobuilder-helper</filename> git |
| repository. This is necessary to prevent the requirement to maintain all the |
| release or project-specific code within Buildbot. The branch chosen matches |
| the release being built so we can support older releases and still make |
| changes in newer ones.</para> |
| </listitem> |
| <listitem> |
| <para><emphasis>Write layerinfo.json</emphasis></para> |
| <para>This transfers data in the Buildbot UI when the build was configured to |
| the Helper.</para> |
| </listitem> |
| <listitem> |
| <para><emphasis>Call scripts/shared-repo-unpack</emphasis></para> |
| <para>This is a call into the Helper scripts to set up a checkout of all the |
| pieces this build might need. It might clone the BitBake repository and the |
| OpenEmbedded-Core repository. It may clone the Poky repository, as well as |
| additional layers. It will use the data from the |
| <filename>layerinfo.json</filename> file to help understand the |
| configuration. It will also use a local cache of repositories to speed up |
| the clone checkouts. For additional information, see <link |
| linkend="test-autobuilder-clone-cache">Autobuilder Clone |
| Cache</link>.</para> |
| <para>This step has two possible modes of operation. If the build is part of a |
| parent build, its possible that all the repositories needed may already be |
| available, ready in a pre-prepared directory. An "a-quick" or "a-full" build |
| would prepare this before starting the other sub-target builds. This is done |
| for two reasons:<itemizedlist> |
| <listitem> |
| <para>the upstream may change during a build, for example, from a |
| forced push and this ensures we have matching content for the |
| whole build</para> |
| </listitem> |
| <listitem> |
| <para>if 15 Workers all tried to pull the same data from the same |
| repos, we can hit resource limits on upstream servers as they |
| can think they are under some kind of network attack</para> |
| </listitem> |
| </itemizedlist>This pre-prepared directory is shared among the Workers over |
| NFS. If the build is an individual build and there is no "shared" directory |
| available, it would clone from the cache and the upstreams as necessary. |
| This is considered the fallback mode.</para> |
| </listitem> |
| <listitem> |
| <para><emphasis>Call scripts/run-config</emphasis></para> |
| <para>This is another call into the Helper scripts where its expected that the |
| main functionality of this target will be executed.</para> |
| </listitem> |
| </orderedlist></para> |
| </section> |
| <section id='test-autobuilder-tech'> |
| <title>Autobuilder Technology</title> |
| <para>The Autobuilder has Yocto Project-specific functionality to allow builds to operate |
| with increased efficiency and speed.</para> |
| <section id='test-clobberdir'> |
| <title>clobberdir</title> |
| <para>When deleting files, the Autobuilder uses <filename>clobberdir</filename>, which |
| is a special script that moves files to a special location, rather than deleting |
| them. Files in this location are deleted by an <filename>rm</filename> command, |
| which is run under <filename>ionice -c 3</filename>. For example, the deletion only |
| happens when there is idle IO capacity on the Worker. The Autobuilder Worker Janitor |
| runs this deletion. See <link linkend="test-autobuilder-worker-janitor">Autobuilder |
| Worker Janitor</link>.</para> |
| </section> |
| <section id='test-autobuilder-clone-cache'> |
| <title>Autobuilder Clone Cache</title> |
| <para>Cloning repositories from scratch each time they are required was slow on the |
| Autobuilder. We therefore have a stash of commonly used repositories pre-cloned on |
| the Workers. Data is fetched from these during clones first, then "topped up" with |
| later revisions from any upstream when necesary. The cache is maintained by the |
| Autobuilder Worker Janitor. See <link linkend="test-autobuilder-worker-janitor" |
| >Autobuilder Worker Janitor</link>.</para> |
| </section> |
| <section id='test-autobuilder-worker-janitor'> |
| <title>Autobuilder Worker Janitor</title> |
| <para>This is a process running on each Worker that performs two basic operations, |
| including background file deletion at IO idle (see <link |
| linkend="test-list-tgt-exec-clobberdir">Target Execution: clobberdir</link>) and |
| maintainenance of a cache of cloned repositories to improve the speed the system can |
| checkout repositories.</para> |
| </section> |
| <section id='test-shared-dl-dir'> |
| <title>Shared DL_DIR</title> |
| <para>The Workers are all connected over NFS which allows DL_DIR to be shared between |
| them. This reduces network accesses from the system and allows the build to be sped |
| up. Usage of the directory within the build system is designed to be able to be |
| shared over NFS.</para> |
| </section> |
| <section id='test-shared-sstate-cache'> |
| <title>Shared SSTATE_DIR</title> |
| <para>The Workers are all connected over NFS which allows the |
| <filename>sstate</filename> directory to be shared between them. This means once |
| a Worker has built an artefact, all the others can benefit from it. Usage of the |
| directory within the directory is designed for sharing over NFS.</para> |
| </section> |
| <section id='test-resulttool'> |
| <title>Resulttool</title> |
| <para>All of the different tests run as part of the build generate output into |
| <filename>testresults.json</filename> files. This allows us to determine which |
| tests ran in a given build and their status. Additional information, such as failure |
| logs or the time taken to run the tests, may also be included.</para> |
| <para>Resulttool is part of OpenEmbedded-Core and is used to manipulate these json |
| results files. It has the ability to merge files together, display reports of the |
| test results and compare different result files.</para> |
| <para>For details, see <link linkend="" |
| >https://wiki.yoctoproject.org/wiki/Resulttool</link>.</para> |
| </section> |
| </section> |
| <section id='test-run-config-tgt-execution'> |
| <title>run-config Target Execution</title> |
| <para>The <filename>scripts/run-config</filename> execution is where most of the work within |
| the Autobuilder happens. It runs through a number of steps; the first are general setup |
| steps that are run once and include:<orderedlist> |
| <listitem> |
| <para>Set up any <filename>buildtools-tarball</filename> if configured.</para> |
| </listitem> |
| <listitem> |
| <para>Call "buildhistory-init" if buildhistory is configured.</para> |
| </listitem> |
| </orderedlist></para> |
| <para>For each step that is configured in <filename>config.json</filename>, it will perform |
| the following:</para> |
| <para> |
| <remark>## WRITER's question: What does "logging in as stepXa" and others refer to |
| below? ##</remark> |
| <orderedlist> |
| <listitem id="test-run-config-add-layers-step"> |
| <para dir="ltr">Add any layers that are specified using the |
| <filename>bitbake-layers add-layer</filename> command (logging as |
| stepXa)</para> |
| </listitem> |
| <listitem> |
| <para dir="ltr">Call the <filename>scripts/setup-config</filename> script to |
| generate the necessary <filename>auto.conf</filename> configuration file for |
| the build</para> |
| </listitem> |
| <listitem> |
| <para dir="ltr">Run the <filename>bitbake BBTARGETS</filename> command (logging |
| as stepXb)</para> |
| </listitem> |
| <listitem> |
| <para dir="ltr">Run the <filename>bitbake SANITYTARGETS</filename> command |
| (logging as stepXc)</para> |
| </listitem> |
| <listitem> |
| <para dir="ltr">Run the <filename>EXTRACMDS</filename> command, which are run |
| within the BitBake build environment (logging as stepXd)</para> |
| </listitem> |
| <listitem> |
| <para dir="ltr">Run the <filename>EXTRAPLAINCMDS</filename> command(s), which |
| are run outside the BitBake build environment (logging as stepXd)</para> |
| </listitem> |
| <listitem> |
| <para dir="ltr">Remove any layers added in <link |
| linkend="test-run-config-add-layers-step">step 1</link> using the |
| <filename>bitbake-layers remove-layer</filename> command (logging as |
| stepXa)</para> |
| </listitem> |
| </orderedlist> |
| </para> |
| <para>Once the execution steps above complete, <filename>run-config</filename> executes a |
| set of post-build steps, including:<orderedlist> |
| <listitem> |
| <para dir="ltr">Call <filename>scripts/publish-artifacts</filename> to collect |
| any output which is to be saved from the build.</para> |
| </listitem> |
| <listitem> |
| <para dir="ltr">Call <filename>scripts/collect-results</filename> to collect any |
| test results to be saved from the build.</para> |
| </listitem> |
| <listitem> |
| <para dir="ltr">Call <filename>scripts/upload-error-reports</filename> to send |
| any error reports generated to the remote server.</para> |
| </listitem> |
| <listitem> |
| <para dir="ltr">Cleanup the build directory using <link |
| linkend="test-clobberdir"><filename>clobberdir</filename></link> if the |
| build was successful, else rename it to "build-renamed" for potential future |
| debugging.</para> |
| </listitem> |
| </orderedlist></para> |
| </section> |
| <section id='test-deploying-yp-autobuilder'> |
| <title>Deploying Yocto Autobuilder</title> |
| <para>The most up to date information about how to setup and deploy your own Autbuilder can |
| be found in README.md in the <filename>yocto-autobuilder2</filename> repository.</para> |
| <para>We hope that people can use the <filename>yocto-autobuilder2</filename> code directly |
| but it is inevitable that users will end up needing to heavily customise the |
| <filename>yocto-autobuilder-helper</filename> repository, particularly the |
| <filename>config.json</filename> file as they will want to define their own test |
| matrix.</para> |
| <para>The Autobuilder supports wo customization options: <itemizedlist> |
| <listitem> |
| <para>variable substitution</para> |
| </listitem> |
| <listitem> |
| <para>overlaying configuration files</para> |
| </listitem> |
| </itemizedlist>The standard <filename>config.json</filename> minimally attempts to allow |
| substitution of the paths. The Helper script repository includes a |
| <filename>local-example.json</filename> file to show how you could override these |
| from a separate configuration file. Pass the following into the environment of the |
| Autobuilder:<literallayout class="monospaced"> |
| $ ABHELPER_JSON="config.json local-example.json" |
| </literallayout>As |
| another example, you could also pass the following into the |
| environment:<literallayout class="monospaced"> |
| $ ABHELPER_JSON="config.json <replaceable>/some/location/</replaceable>local.json" |
| </literallayout>One |
| issue users often run into is validation of the <filename>config.json</filename> files. |
| A tip for minimizing issues from invalid json files is to use a Git |
| <filename>pre-commit-hook.sh</filename> script to verify the JSON file before |
| committing it. Create a symbolic link as |
| follows:<literallayout class="monospaced"> |
| $ ln -s ../../scripts/pre-commit-hook.sh .git/hooks/pre-commit |
| </literallayout></para> |
| </section> |
| |
| |
| |
| |
| |
| |
| |
| |
| </chapter> |
| <!-- |
| vim: expandtab tw=80 ts=4 |
| --> |