blob: 6c341976f912834b08cca7d680270ff06b857783 [file] [log] [blame]
Andrew Geisslerf0343792020-11-18 10:42:21 -06001.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002
3**********************
4Yocto Project Concepts
5**********************
6
7This chapter provides explanations for Yocto Project concepts that go
8beyond the surface of "how-to" information and reference (or look-up)
9material. Concepts such as components, the :term:`OpenEmbedded Build System`
10workflow,
11cross-development toolchains, shared state cache, and so forth are
12explained.
13
14Yocto Project Components
15========================
16
17The :term:`BitBake` task executor
18together with various types of configuration files form the
19:term:`OpenEmbedded-Core (OE-Core)`. This section
20overviews these components by describing their use and how they
21interact.
22
23BitBake handles the parsing and execution of the data files. The data
24itself is of various types:
25
26- *Recipes:* Provides details about particular pieces of software.
27
28- *Class Data:* Abstracts common build information (e.g. how to build a
29 Linux kernel).
30
31- *Configuration Data:* Defines machine-specific settings, policy
32 decisions, and so forth. Configuration data acts as the glue to bind
33 everything together.
34
35BitBake knows how to combine multiple data sources together and refers
36to each data source as a layer. For information on layers, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -060037":ref:`dev-manual/common-tasks:understanding and creating layers`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050038section of the Yocto Project Development Tasks Manual.
39
40Following are some brief details on these core components. For
41additional information on how these components interact during a build,
42see the
Andrew Geissler09209ee2020-12-13 08:44:15 -060043":ref:`overview-manual/concepts:openembedded build system concepts`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050044section.
45
Andrew Geisslerc9f78652020-09-18 14:11:35 -050046BitBake
47-------
48
49BitBake is the tool at the heart of the :term:`OpenEmbedded Build System`
50and is responsible
51for parsing the :term:`Metadata`, generating
52a list of tasks from it, and then executing those tasks.
53
54This section briefly introduces BitBake. If you want more information on
55BitBake, see the :doc:`BitBake User Manual <bitbake:index>`.
56
57To see a list of the options BitBake supports, use either of the
Andrew Geisslerc926e172021-05-07 16:11:35 -050058following commands::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050059
60 $ bitbake -h
61 $ bitbake --help
62
63The most common usage for BitBake is ``bitbake recipename``, where
64``recipename`` is the name of the recipe you want to build (referred
65to as the "target"). The target often equates to the first part of a
66recipe's filename (e.g. "foo" for a recipe named ``foo_1.3.0-r0.bb``).
67So, to process the ``matchbox-desktop_1.2.3.bb`` recipe file, you might
Andrew Geisslerc926e172021-05-07 16:11:35 -050068type the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050069
70 $ bitbake matchbox-desktop
71
72Several different
73versions of ``matchbox-desktop`` might exist. BitBake chooses the one
74selected by the distribution configuration. You can get more details
75about how BitBake chooses between different target versions and
76providers in the
Patrick Williams213cb262021-08-07 19:21:33 -050077":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-execution:preferences`" section
Andrew Geisslerc9f78652020-09-18 14:11:35 -050078of the BitBake User Manual.
79
80BitBake also tries to execute any dependent tasks first. So for example,
81before building ``matchbox-desktop``, BitBake would build a cross
82compiler and ``glibc`` if they had not already been built.
83
84A useful BitBake option to consider is the ``-k`` or ``--continue``
85option. This option instructs BitBake to try and continue processing the
86job as long as possible even after encountering an error. When an error
87occurs, the target that failed and those that depend on it cannot be
88remade. However, when you use this option other dependencies can still
89be processed.
90
Andrew Geisslerc9f78652020-09-18 14:11:35 -050091Recipes
92-------
93
94Files that have the ``.bb`` suffix are "recipes" files. In general, a
95recipe contains information about a single piece of software. This
96information includes the location from which to download the unaltered
97source, any source patches to be applied to that source (if needed),
98which special configuration options to apply, how to compile the source
99files, and how to package the compiled output.
100
101The term "package" is sometimes used to refer to recipes. However, since
102the word "package" is used for the packaged output from the OpenEmbedded
103build system (i.e. ``.ipk`` or ``.deb`` files), this document avoids
104using the term "package" when referring to recipes.
105
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500106Classes
107-------
108
109Class files (``.bbclass``) contain information that is useful to share
110between recipes files. An example is the
111:ref:`autotools <ref-classes-autotools>` class,
Patrick Williams03907ee2022-05-01 06:28:52 -0500112which contains common settings for any application that is built with
113the `GNU Autotools <https://en.wikipedia.org/wiki/GNU_Autotools>`__.
114The ":ref:`ref-manual/classes:Classes`" chapter in the Yocto Project
115Reference Manual provides details about classes and how to use them.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500116
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500117Configurations
118--------------
119
120The configuration files (``.conf``) define various configuration
121variables that govern the OpenEmbedded build process. These files fall
122into several areas that define machine configuration options,
123distribution configuration options, compiler tuning options, general
124common configuration options, and user configuration options in
125``conf/local.conf``, which is found in the :term:`Build Directory`.
126
127
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500128Layers
129======
130
131Layers are repositories that contain related metadata (i.e. sets of
132instructions) that tell the OpenEmbedded build system how to build a
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500133target. :ref:`overview-manual/yp-intro:the yocto project layer model`
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500134facilitates collaboration, sharing, customization, and reuse within the
135Yocto Project development environment. Layers logically separate
136information for your project. For example, you can use a layer to hold
137all the configurations for a particular piece of hardware. Isolating
138hardware-specific configurations allows you to share other metadata by
139using a different layer where that metadata might be common across
140several pieces of hardware.
141
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700142There are many layers working in the Yocto Project development environment. The
Patrick Williams03907ee2022-05-01 06:28:52 -0500143:yocto_home:`Yocto Project Compatible Layer Index </software-overview/layers/>`
Andrew Geisslerd1e89492021-02-12 15:35:20 -0600144and :oe_layerindex:`OpenEmbedded Layer Index <>` both contain layers from
145which you can use or leverage.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500146
147By convention, layers in the Yocto Project follow a specific form.
148Conforming to a known structure allows BitBake to make assumptions
149during builds on where to find types of metadata. You can find
150procedures and learn about tools (i.e. ``bitbake-layers``) for creating
151layers suitable for the Yocto Project in the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600152":ref:`dev-manual/common-tasks:understanding and creating layers`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500153section of the Yocto Project Development Tasks Manual.
154
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500155OpenEmbedded Build System Concepts
156==================================
157
158This section takes a more detailed look inside the build process used by
159the :term:`OpenEmbedded Build System`,
160which is the build
161system specific to the Yocto Project. At the heart of the build system
162is BitBake, the task executor.
163
164The following diagram represents the high-level workflow of a build. The
165remainder of this section expands on the fundamental input, output,
166process, and metadata logical blocks that make up the workflow.
167
168.. image:: figures/YP-flow-diagram.png
169 :align: center
170
171In general, the build's workflow consists of several functional areas:
172
173- *User Configuration:* metadata you can use to control the build
174 process.
175
176- *Metadata Layers:* Various layers that provide software, machine, and
177 distro metadata.
178
179- *Source Files:* Upstream releases, local projects, and SCMs.
180
181- *Build System:* Processes under the control of
182 :term:`BitBake`. This block expands
183 on how BitBake fetches source, applies patches, completes
184 compilation, analyzes output for package generation, creates and
185 tests packages, generates images, and generates cross-development
186 tools.
187
188- *Package Feeds:* Directories containing output packages (RPM, DEB or
189 IPK), which are subsequently used in the construction of an image or
190 Software Development Kit (SDK), produced by the build system. These
191 feeds can also be copied and shared using a web server or other means
192 to facilitate extending or updating existing images on devices at
193 runtime if runtime package management is enabled.
194
195- *Images:* Images produced by the workflow.
196
197- *Application Development SDK:* Cross-development tools that are
198 produced along with an image or separately with BitBake.
199
200User Configuration
201------------------
202
203User configuration helps define the build. Through user configuration,
204you can tell BitBake the target architecture for which you are building
205the image, where to store downloaded source, and other build properties.
206
207The following figure shows an expanded representation of the "User
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500208Configuration" box of the :ref:`general workflow
209figure <overview-manual/concepts:openembedded build system concepts>`:
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500210
211.. image:: figures/user-configuration.png
212 :align: center
213
214BitBake needs some basic configuration files in order to complete a
215build. These files are ``*.conf`` files. The minimally necessary ones
216reside as example files in the ``build/conf`` directory of the
217:term:`Source Directory`. For simplicity,
218this section refers to the Source Directory as the "Poky Directory."
219
220When you clone the :term:`Poky` Git repository
221or you download and unpack a Yocto Project release, you can set up the
222Source Directory to be named anything you want. For this discussion, the
223cloned repository uses the default name ``poky``.
224
225.. note::
226
227 The Poky repository is primarily an aggregation of existing
228 repositories. It is not a canonical upstream source.
229
230The ``meta-poky`` layer inside Poky contains a ``conf`` directory that
231has example configuration files. These example files are used as a basis
232for creating actual configuration files when you source
233:ref:`structure-core-script`, which is the
234build environment script.
235
236Sourcing the build environment script creates a
237:term:`Build Directory` if one does not
238already exist. BitBake uses the Build Directory for all its work during
239builds. The Build Directory has a ``conf`` directory that contains
240default versions of your ``local.conf`` and ``bblayers.conf``
241configuration files. These default configuration files are created only
242if versions do not already exist in the Build Directory at the time you
243source the build environment setup script.
244
245Because the Poky repository is fundamentally an aggregation of existing
246repositories, some users might be familiar with running the
247:ref:`structure-core-script` script in the context of separate
248:term:`OpenEmbedded-Core (OE-Core)` and BitBake
249repositories rather than a single Poky repository. This discussion
250assumes the script is executed from within a cloned or unpacked version
251of Poky.
252
253Depending on where the script is sourced, different sub-scripts are
254called to set up the Build Directory (Yocto or OpenEmbedded).
255Specifically, the script ``scripts/oe-setup-builddir`` inside the poky
256directory sets up the Build Directory and seeds the directory (if
257necessary) with configuration files appropriate for the Yocto Project
258development environment.
259
260.. note::
261
262 The
263 scripts/oe-setup-builddir
264 script uses the
265 ``$TEMPLATECONF``
266 variable to determine which sample configuration files to locate.
267
268The ``local.conf`` file provides many basic variables that define a
269build environment. Here is a list of a few. To see the default
270configurations in a ``local.conf`` file created by the build environment
271script, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600272:yocto_git:`local.conf.sample </poky/tree/meta-poky/conf/local.conf.sample>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500273in the ``meta-poky`` layer:
274
275- *Target Machine Selection:* Controlled by the
276 :term:`MACHINE` variable.
277
278- *Download Directory:* Controlled by the
279 :term:`DL_DIR` variable.
280
281- *Shared State Directory:* Controlled by the
282 :term:`SSTATE_DIR` variable.
283
284- *Build Output:* Controlled by the
285 :term:`TMPDIR` variable.
286
287- *Distribution Policy:* Controlled by the
288 :term:`DISTRO` variable.
289
290- *Packaging Format:* Controlled by the
291 :term:`PACKAGE_CLASSES`
292 variable.
293
294- *SDK Target Architecture:* Controlled by the
295 :term:`SDKMACHINE` variable.
296
297- *Extra Image Packages:* Controlled by the
298 :term:`EXTRA_IMAGE_FEATURES`
299 variable.
300
301.. note::
302
Andrew Geissler9aee5002022-03-30 16:27:02 +0000303 Configurations set in the ``conf/local.conf`` file can also be set
304 in the ``conf/site.conf`` and ``conf/auto.conf`` configuration files.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500305
306The ``bblayers.conf`` file tells BitBake what layers you want considered
307during the build. By default, the layers listed in this file include
308layers minimally needed by the build system. However, you must manually
309add any custom layers you have created. You can find more information on
310working with the ``bblayers.conf`` file in the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600311":ref:`dev-manual/common-tasks:enabling your layer`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500312section in the Yocto Project Development Tasks Manual.
313
314The files ``site.conf`` and ``auto.conf`` are not created by the
315environment initialization script. If you want the ``site.conf`` file,
316you need to create that yourself. The ``auto.conf`` file is typically
317created by an autobuilder:
318
319- *site.conf:* You can use the ``conf/site.conf`` configuration
320 file to configure multiple build directories. For example, suppose
321 you had several build environments and they shared some common
322 features. You can set these default build properties here. A good
323 example is perhaps the packaging format to use through the
324 :term:`PACKAGE_CLASSES`
325 variable.
326
327 One useful scenario for using the ``conf/site.conf`` file is to
328 extend your :term:`BBPATH` variable
329 to include the path to a ``conf/site.conf``. Then, when BitBake looks
Andrew Geissler09036742021-06-25 14:25:14 -0500330 for Metadata using :term:`BBPATH`, it finds the ``conf/site.conf`` file
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500331 and applies your common configurations found in the file. To override
332 configurations in a particular build directory, alter the similar
333 configurations within that build directory's ``conf/local.conf``
334 file.
335
336- *auto.conf:* The file is usually created and written to by an
337 autobuilder. The settings put into the file are typically the same as
338 you would find in the ``conf/local.conf`` or the ``conf/site.conf``
339 files.
340
341You can edit all configuration files to further define any particular
342build environment. This process is represented by the "User
343Configuration Edits" box in the figure.
344
345When you launch your build with the ``bitbake target`` command, BitBake
346sorts out the configurations to ultimately define your build
347environment. It is important to understand that the
348:term:`OpenEmbedded Build System` reads the
349configuration files in a specific order: ``site.conf``, ``auto.conf``,
350and ``local.conf``. And, the build system applies the normal assignment
351statement rules as described in the
352":doc:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata`" chapter
353of the BitBake User Manual. Because the files are parsed in a specific
354order, variable assignments for the same variable could be affected. For
355example, if the ``auto.conf`` file and the ``local.conf`` set variable1
356to different values, because the build system parses ``local.conf``
357after ``auto.conf``, variable1 is assigned the value from the
358``local.conf`` file.
359
360Metadata, Machine Configuration, and Policy Configuration
361---------------------------------------------------------
362
363The previous section described the user configurations that define
364BitBake's global behavior. This section takes a closer look at the
365layers the build system uses to further control the build. These layers
366provide Metadata for the software, machine, and policies.
367
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700368In general, there are three types of layer input. You can see them below
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500369the "User Configuration" box in the `general workflow
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500370figure <overview-manual/concepts:openembedded build system concepts>`:
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500371
372- *Metadata (.bb + Patches):* Software layers containing
373 user-supplied recipe files, patches, and append files. A good example
Andrew Geisslerd1e89492021-02-12 15:35:20 -0600374 of a software layer might be the :oe_layer:`meta-qt5 layer </meta-qt5>`
375 from the :oe_layerindex:`OpenEmbedded Layer Index <>`. This layer is for
376 version 5.0 of the popular `Qt <https://wiki.qt.io/About_Qt>`__
377 cross-platform application development framework for desktop, embedded and
378 mobile.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500379
380- *Machine BSP Configuration:* Board Support Package (BSP) layers (i.e.
381 "BSP Layer" in the following figure) providing machine-specific
382 configurations. This type of information is specific to a particular
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500383 target architecture. A good example of a BSP layer from the
384 :ref:`overview-manual/yp-intro:reference distribution (poky)` is the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600385 :yocto_git:`meta-yocto-bsp </poky/tree/meta-yocto-bsp>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500386 layer.
387
388- *Policy Configuration:* Distribution Layers (i.e. "Distro Layer" in
389 the following figure) providing top-level or general policies for the
390 images or SDKs being built for a particular distribution. For
391 example, in the Poky Reference Distribution the distro layer is the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600392 :yocto_git:`meta-poky </poky/tree/meta-poky>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500393 layer. Within the distro layer is a ``conf/distro`` directory that
394 contains distro configuration files (e.g.
Andrew Geissler09209ee2020-12-13 08:44:15 -0600395 :yocto_git:`poky.conf </poky/tree/meta-poky/conf/distro/poky.conf>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500396 that contain many policy configurations for the Poky distribution.
397
398The following figure shows an expanded representation of these three
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500399layers from the :ref:`general workflow figure
400<overview-manual/concepts:openembedded build system concepts>`:
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500401
402.. image:: figures/layer-input.png
403 :align: center
404
405In general, all layers have a similar structure. They all contain a
406licensing file (e.g. ``COPYING.MIT``) if the layer is to be distributed,
407a ``README`` file as good practice and especially if the layer is to be
408distributed, a configuration directory, and recipe directories. You can
409learn about the general structure for layers used with the Yocto Project
410in the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600411":ref:`dev-manual/common-tasks:creating your own layer`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500412section in the
413Yocto Project Development Tasks Manual. For a general discussion on
414layers and the many layers from which you can draw, see the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500415":ref:`overview-manual/concepts:layers`" and
416":ref:`overview-manual/yp-intro:the yocto project layer model`" sections both
417earlier in this manual.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500418
419If you explored the previous links, you discovered some areas where many
Andrew Geissler09209ee2020-12-13 08:44:15 -0600420layers that work with the Yocto Project exist. The :yocto_git:`Source
421Repositories <>` also shows layers categorized under "Yocto Metadata Layers."
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500422
423.. note::
424
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700425 There are layers in the Yocto Project Source Repositories that cannot be
426 found in the OpenEmbedded Layer Index. Such layers are either
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500427 deprecated or experimental in nature.
428
429BitBake uses the ``conf/bblayers.conf`` file, which is part of the user
430configuration, to find what layers it should be using as part of the
431build.
432
433Distro Layer
434~~~~~~~~~~~~
435
436The distribution layer provides policy configurations for your
437distribution. Best practices dictate that you isolate these types of
438configurations into their own layer. Settings you provide in
439``conf/distro/distro.conf`` override similar settings that BitBake finds
440in your ``conf/local.conf`` file in the Build Directory.
441
442The following list provides some explanation and references for what you
443typically find in the distribution layer:
444
445- *classes:* Class files (``.bbclass``) hold common functionality that
446 can be shared among recipes in the distribution. When your recipes
447 inherit a class, they take on the settings and functions for that
448 class. You can read more about class files in the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600449 ":ref:`ref-manual/classes:Classes`" chapter of the Yocto
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500450 Reference Manual.
451
452- *conf:* This area holds configuration files for the layer
453 (``conf/layer.conf``), the distribution
454 (``conf/distro/distro.conf``), and any distribution-wide include
455 files.
456
457- *recipes-*:* Recipes and append files that affect common
458 functionality across the distribution. This area could include
459 recipes and append files to add distribution-specific configuration,
460 initialization scripts, custom image recipes, and so forth. Examples
461 of ``recipes-*`` directories are ``recipes-core`` and
462 ``recipes-extra``. Hierarchy and contents within a ``recipes-*``
463 directory can vary. Generally, these directories contain recipe files
464 (``*.bb``), recipe append files (``*.bbappend``), directories that
465 are distro-specific for configuration files, and so forth.
466
467BSP Layer
468~~~~~~~~~
469
470The BSP Layer provides machine configurations that target specific
471hardware. Everything in this layer is specific to the machine for which
472you are building the image or the SDK. A common structure or form is
473defined for BSP layers. You can learn more about this structure in the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600474:doc:`/bsp-guide/index`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500475
476.. note::
477
478 In order for a BSP layer to be considered compliant with the Yocto
479 Project, it must meet some structural requirements.
480
481The BSP Layer's configuration directory contains configuration files for
482the machine (``conf/machine/machine.conf``) and, of course, the layer
483(``conf/layer.conf``).
484
485The remainder of the layer is dedicated to specific recipes by function:
486``recipes-bsp``, ``recipes-core``, ``recipes-graphics``,
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700487``recipes-kernel``, and so forth. There can be metadata for multiple
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500488formfactors, graphics support systems, and so forth.
489
490.. note::
491
492 While the figure shows several
493 recipes-\*
494 directories, not all these directories appear in all BSP layers.
495
496Software Layer
497~~~~~~~~~~~~~~
498
499The software layer provides the Metadata for additional software
500packages used during the build. This layer does not include Metadata
501that is specific to the distribution or the machine, which are found in
502their respective layers.
503
504This layer contains any recipes, append files, and patches, that your
505project needs.
506
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500507Sources
508-------
509
510In order for the OpenEmbedded build system to create an image or any
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500511target, it must be able to access source files. The :ref:`general workflow
512figure <overview-manual/concepts:openembedded build system concepts>`
513represents source files using the "Upstream Project Releases", "Local
514Projects", and "SCMs (optional)" boxes. The figure represents mirrors,
515which also play a role in locating source files, with the "Source
516Materials" box.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500517
518The method by which source files are ultimately organized is a function
519of the project. For example, for released software, projects tend to use
520tarballs or other archived files that can capture the state of a release
521guaranteeing that it is statically represented. On the other hand, for a
522project that is more dynamic or experimental in nature, a project might
523keep source files in a repository controlled by a Source Control Manager
524(SCM) such as Git. Pulling source from a repository allows you to
525control the point in the repository (the revision) from which you want
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700526to build software. A combination of the two is also possible.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500527
528BitBake uses the :term:`SRC_URI`
529variable to point to source files regardless of their location. Each
Andrew Geissler09036742021-06-25 14:25:14 -0500530recipe must have a :term:`SRC_URI` variable that points to the source.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500531
532Another area that plays a significant role in where source files come
533from is pointed to by the
534:term:`DL_DIR` variable. This area is
535a cache that can hold previously downloaded source. You can also
536instruct the OpenEmbedded build system to create tarballs from Git
537repositories, which is not the default behavior, and store them in the
Andrew Geissler09036742021-06-25 14:25:14 -0500538:term:`DL_DIR` by using the
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500539:term:`BB_GENERATE_MIRROR_TARBALLS`
540variable.
541
Andrew Geissler09036742021-06-25 14:25:14 -0500542Judicious use of a :term:`DL_DIR` directory can save the build system a trip
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500543across the Internet when looking for files. A good method for using a
Andrew Geissler09036742021-06-25 14:25:14 -0500544download directory is to have :term:`DL_DIR` point to an area outside of
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500545your Build Directory. Doing so allows you to safely delete the Build
546Directory if needed without fear of removing any downloaded source file.
547
548The remainder of this section provides a deeper look into the source
549files and the mirrors. Here is a more detailed look at the source file
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500550area of the :ref:`general workflow figure <overview-manual/concepts:openembedded build system concepts>`:
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500551
552.. image:: figures/source-input.png
553 :align: center
554
555Upstream Project Releases
556~~~~~~~~~~~~~~~~~~~~~~~~~
557
558Upstream project releases exist anywhere in the form of an archived file
559(e.g. tarball or zip file). These files correspond to individual
560recipes. For example, the figure uses specific releases each for
561BusyBox, Qt, and Dbus. An archive file can be for any released product
562that can be built using a recipe.
563
564Local Projects
565~~~~~~~~~~~~~~
566
567Local projects are custom bits of software the user provides. These bits
568reside somewhere local to a project - perhaps a directory into which the
569user checks in items (e.g. a local directory containing a development
570source tree used by the group).
571
572The canonical method through which to include a local project is to use
573the :ref:`externalsrc <ref-classes-externalsrc>`
574class to include that local project. You use either the ``local.conf``
575or a recipe's append file to override or set the recipe to point to the
576local directory on your disk to pull in the whole source tree.
577
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500578Source Control Managers (Optional)
579~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
580
581Another place from which the build system can get source files is with
Patrick Williams213cb262021-08-07 19:21:33 -0500582:ref:`bitbake:bitbake-user-manual/bitbake-user-manual-fetching:fetchers` employing various Source
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500583Control Managers (SCMs) such as Git or Subversion. In such cases, a
584repository is cloned or checked out. The
585:ref:`ref-tasks-fetch` task inside
586BitBake uses the :term:`SRC_URI`
587variable and the argument's prefix to determine the correct fetcher
588module.
589
590.. note::
591
592 For information on how to have the OpenEmbedded build system generate
593 tarballs for Git repositories and place them in the
594 DL_DIR
595 directory, see the :term:`BB_GENERATE_MIRROR_TARBALLS`
596 variable in the Yocto Project Reference Manual.
597
598When fetching a repository, BitBake uses the
599:term:`SRCREV` variable to determine
600the specific revision from which to build.
601
602Source Mirror(s)
603~~~~~~~~~~~~~~~~
604
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700605There are two kinds of mirrors: pre-mirrors and regular mirrors. The
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500606:term:`PREMIRRORS` and
607:term:`MIRRORS` variables point to
608these, respectively. BitBake checks pre-mirrors before looking upstream
609for any source files. Pre-mirrors are appropriate when you have a shared
610directory that is not a directory defined by the
611:term:`DL_DIR` variable. A Pre-mirror
612typically points to a shared directory that is local to your
613organization.
614
615Regular mirrors can be any site across the Internet that is used as an
616alternative location for source code should the primary site not be
617functioning for some reason or another.
618
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500619Package Feeds
620-------------
621
622When the OpenEmbedded build system generates an image or an SDK, it gets
623the packages from a package feed area located in the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500624:term:`Build Directory`. The :ref:`general workflow figure
625<overview-manual/concepts:openembedded build system concepts>`
626shows this package feeds area in the upper-right corner.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500627
628This section looks a little closer into the package feeds area used by
629the build system. Here is a more detailed look at the area:
630
631.. image:: figures/package-feeds.png
632 :align: center
633
634Package feeds are an intermediary step in the build process. The
635OpenEmbedded build system provides classes to generate different package
636types, and you specify which classes to enable through the
637:term:`PACKAGE_CLASSES`
638variable. Before placing the packages into package feeds, the build
639process validates them with generated output quality assurance checks
640through the :ref:`insane <ref-classes-insane>`
641class.
642
643The package feed area resides in the Build Directory. The directory the
644build system uses to temporarily store packages is determined by a
645combination of variables and the particular package manager in use. See
646the "Package Feeds" box in the illustration and note the information to
647the right of that area. In particular, the following defines where
648package files are kept:
649
650- :term:`DEPLOY_DIR`: Defined as
651 ``tmp/deploy`` in the Build Directory.
652
653- ``DEPLOY_DIR_*``: Depending on the package manager used, the package
654 type sub-folder. Given RPM, IPK, or DEB packaging and tarball
655 creation, the
656 :term:`DEPLOY_DIR_RPM`,
657 :term:`DEPLOY_DIR_IPK`,
658 :term:`DEPLOY_DIR_DEB`, or
659 :term:`DEPLOY_DIR_TAR`,
660 variables are used, respectively.
661
662- :term:`PACKAGE_ARCH`: Defines
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700663 architecture-specific sub-folders. For example, packages could be
664 available for the i586 or qemux86 architectures.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500665
666BitBake uses the
667:ref:`do_package_write_* <ref-tasks-package_write_deb>`
668tasks to generate packages and place them into the package holding area
669(e.g. ``do_package_write_ipk`` for IPK packages). See the
670":ref:`ref-tasks-package_write_deb`",
671":ref:`ref-tasks-package_write_ipk`",
672":ref:`ref-tasks-package_write_rpm`",
673and
674":ref:`ref-tasks-package_write_tar`"
675sections in the Yocto Project Reference Manual for additional
676information. As an example, consider a scenario where an IPK packaging
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700677manager is being used and there is package architecture support for both
678i586 and qemux86. Packages for the i586 architecture are placed in
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500679``build/tmp/deploy/ipk/i586``, while packages for the qemux86
680architecture are placed in ``build/tmp/deploy/ipk/qemux86``.
681
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500682BitBake Tool
683------------
684
685The OpenEmbedded build system uses
686:term:`BitBake` to produce images and
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500687Software Development Kits (SDKs). You can see from the :ref:`general workflow
688figure <overview-manual/concepts:openembedded build system concepts>`,
689the BitBake area consists of several functional areas. This section takes a
690closer look at each of those areas.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500691
692.. note::
693
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700694 Documentation for the BitBake tool is available separately. See the
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500695 BitBake User Manual
696 for reference material on BitBake.
697
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500698Source Fetching
699~~~~~~~~~~~~~~~
700
701The first stages of building a recipe are to fetch and unpack the source
702code:
703
704.. image:: figures/source-fetching.png
705 :align: center
706
707The :ref:`ref-tasks-fetch` and
708:ref:`ref-tasks-unpack` tasks fetch
709the source files and unpack them into the
710:term:`Build Directory`.
711
712.. note::
713
714 For every local file (e.g.
715 file://
716 ) that is part of a recipe's
717 SRC_URI
718 statement, the OpenEmbedded build system takes a checksum of the file
719 for the recipe and inserts the checksum into the signature for the
720 do_fetch
721 task. If any local file has been modified, the
722 do_fetch
723 task and all tasks that depend on it are re-executed.
724
725By default, everything is accomplished in the Build Directory, which has
726a defined structure. For additional general information on the Build
727Directory, see the ":ref:`structure-core-build`" section in
728the Yocto Project Reference Manual.
729
730Each recipe has an area in the Build Directory where the unpacked source
731code resides. The :term:`S` variable points
732to this area for a recipe's unpacked source code. The name of that
733directory for any given recipe is defined from several different
734variables. The preceding figure and the following list describe the
735Build Directory's hierarchy:
736
737- :term:`TMPDIR`: The base directory
738 where the OpenEmbedded build system performs all its work during the
739 build. The default base directory is the ``tmp`` directory.
740
741- :term:`PACKAGE_ARCH`: The
742 architecture of the built package or packages. Depending on the
743 eventual destination of the package or packages (i.e. machine
744 architecture, :term:`Build Host`, SDK, or
Andrew Geissler09036742021-06-25 14:25:14 -0500745 specific machine), :term:`PACKAGE_ARCH` varies. See the variable's
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500746 description for details.
747
748- :term:`TARGET_OS`: The operating
749 system of the target device. A typical value would be "linux" (e.g.
750 "qemux86-poky-linux").
751
752- :term:`PN`: The name of the recipe used
753 to build the package. This variable can have multiple meanings.
Andrew Geissler09036742021-06-25 14:25:14 -0500754 However, when used in the context of input files, :term:`PN` represents
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500755 the name of the recipe.
756
757- :term:`WORKDIR`: The location
758 where the OpenEmbedded build system builds a recipe (i.e. does the
759 work to create the package).
760
761 - :term:`PV`: The version of the
762 recipe used to build the package.
763
764 - :term:`PR`: The revision of the
765 recipe used to build the package.
766
767- :term:`S`: Contains the unpacked source
768 files for a given recipe.
769
770 - :term:`BPN`: The name of the recipe
Andrew Geissler09036742021-06-25 14:25:14 -0500771 used to build the package. The :term:`BPN` variable is a version of
Andrew Geissler5f350902021-07-23 13:09:54 -0400772 the :term:`PN` variable but with common prefixes and suffixes removed.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500773
774 - :term:`PV`: The version of the
775 recipe used to build the package.
776
777.. note::
778
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700779 In the previous figure, notice that there are two sample hierarchies:
780 one based on package architecture (i.e. :term:`PACKAGE_ARCH`)
781 and one based on a machine (i.e. :term:`MACHINE`).
782 The underlying structures are identical. The differentiator being
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500783 what the OpenEmbedded build system is using as a build target (e.g.
784 general architecture, a build host, an SDK, or a specific machine).
785
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500786Patching
787~~~~~~~~
788
789Once source code is fetched and unpacked, BitBake locates patch files
790and applies them to the source files:
791
792.. image:: figures/patching.png
793 :align: center
794
795The :ref:`ref-tasks-patch` task uses a
796recipe's :term:`SRC_URI` statements
797and the :term:`FILESPATH` variable
798to locate applicable patch files.
799
800Default processing for patch files assumes the files have either
Andrew Geissler09036742021-06-25 14:25:14 -0500801``*.patch`` or ``*.diff`` file types. You can use :term:`SRC_URI` parameters
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500802to change the way the build system recognizes patch files. See the
803:ref:`ref-tasks-patch` task for more
804information.
805
806BitBake finds and applies multiple patches for a single recipe in the
Andrew Geissler09036742021-06-25 14:25:14 -0500807order in which it locates the patches. The :term:`FILESPATH` variable
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500808defines the default set of directories that the build system uses to
809search for patch files. Once found, patches are applied to the recipe's
810source files, which are located in the
811:term:`S` directory.
812
813For more information on how the source directories are created, see the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500814":ref:`overview-manual/concepts:source fetching`" section. For
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500815more information on how to create patches and how the build system
816processes patches, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600817":ref:`dev-manual/common-tasks:patching code`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500818section in the
819Yocto Project Development Tasks Manual. You can also see the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600820":ref:`sdk-manual/extensible:use \`\`devtool modify\`\` to modify the source of an existing component`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500821section in the Yocto Project Application Development and the Extensible
822Software Development Kit (SDK) manual and the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600823":ref:`kernel-dev/common:using traditional kernel development to patch the kernel`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500824section in the Yocto Project Linux Kernel Development Manual.
825
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500826Configuration, Compilation, and Staging
827~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
828
829After source code is patched, BitBake executes tasks that configure and
830compile the source code. Once compilation occurs, the files are copied
831to a holding area (staged) in preparation for packaging:
832
833.. image:: figures/configuration-compile-autoreconf.png
834 :align: center
835
836This step in the build process consists of the following tasks:
837
838- :ref:`ref-tasks-prepare_recipe_sysroot`:
839 This task sets up the two sysroots in
840 ``${``\ :term:`WORKDIR`\ ``}``
841 (i.e. ``recipe-sysroot`` and ``recipe-sysroot-native``) so that
842 during the packaging phase the sysroots can contain the contents of
843 the
844 :ref:`ref-tasks-populate_sysroot`
845 tasks of the recipes on which the recipe containing the tasks
846 depends. A sysroot exists for both the target and for the native
847 binaries, which run on the host system.
848
849- *do_configure*: This task configures the source by enabling and
850 disabling any build-time and configuration options for the software
851 being built. Configurations can come from the recipe itself as well
852 as from an inherited class. Additionally, the software itself might
853 configure itself depending on the target for which it is being built.
854
855 The configurations handled by the
856 :ref:`ref-tasks-configure` task
857 are specific to configurations for the source code being built by the
858 recipe.
859
860 If you are using the
861 :ref:`autotools <ref-classes-autotools>` class,
862 you can add additional configuration options by using the
863 :term:`EXTRA_OECONF` or
864 :term:`PACKAGECONFIG_CONFARGS`
865 variables. For information on how this variable works within that
866 class, see the
867 :ref:`autotools <ref-classes-autotools>` class
Andrew Geissler09209ee2020-12-13 08:44:15 -0600868 :yocto_git:`here </poky/tree/meta/classes/autotools.bbclass>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500869
870- *do_compile*: Once a configuration task has been satisfied,
871 BitBake compiles the source using the
872 :ref:`ref-tasks-compile` task.
873 Compilation occurs in the directory pointed to by the
874 :term:`B` variable. Realize that the
Andrew Geissler09036742021-06-25 14:25:14 -0500875 :term:`B` directory is, by default, the same as the
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500876 :term:`S` directory.
877
878- *do_install*: After compilation completes, BitBake executes the
879 :ref:`ref-tasks-install` task.
Andrew Geissler09036742021-06-25 14:25:14 -0500880 This task copies files from the :term:`B` directory and places them in a
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500881 holding area pointed to by the :term:`D`
882 variable. Packaging occurs later using files from this holding
883 directory.
884
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500885Package Splitting
886~~~~~~~~~~~~~~~~~
887
888After source code is configured, compiled, and staged, the build system
889analyzes the results and splits the output into packages:
890
891.. image:: figures/analysis-for-package-splitting.png
892 :align: center
893
894The :ref:`ref-tasks-package` and
895:ref:`ref-tasks-packagedata`
896tasks combine to analyze the files found in the
897:term:`D` directory and split them into
898subsets based on available packages and files. Analysis involves the
899following as well as other items: splitting out debugging symbols,
900looking at shared library dependencies between packages, and looking at
901package relationships.
902
903The ``do_packagedata`` task creates package metadata based on the
904analysis such that the build system can generate the final packages. The
905:ref:`ref-tasks-populate_sysroot`
906task stages (copies) a subset of the files installed by the
907:ref:`ref-tasks-install` task into
908the appropriate sysroot. Working, staged, and intermediate results of
909the analysis and package splitting process use several areas:
910
911- :term:`PKGD`: The destination
912 directory (i.e. ``package``) for packages before they are split into
913 individual packages.
914
915- :term:`PKGDESTWORK`: A
916 temporary work area (i.e. ``pkgdata``) used by the ``do_package``
917 task to save package metadata.
918
919- :term:`PKGDEST`: The parent
920 directory (i.e. ``packages-split``) for packages after they have been
921 split.
922
923- :term:`PKGDATA_DIR`: A shared,
924 global-state directory that holds packaging metadata generated during
925 the packaging process. The packaging process copies metadata from
Andrew Geissler09036742021-06-25 14:25:14 -0500926 :term:`PKGDESTWORK` to the :term:`PKGDATA_DIR` area where it becomes globally
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500927 available.
928
929- :term:`STAGING_DIR_HOST`:
930 The path for the sysroot for the system on which a component is built
931 to run (i.e. ``recipe-sysroot``).
932
933- :term:`STAGING_DIR_NATIVE`:
934 The path for the sysroot used when building components for the build
935 host (i.e. ``recipe-sysroot-native``).
936
937- :term:`STAGING_DIR_TARGET`:
938 The path for the sysroot used when a component that is built to
939 execute on a system and it generates code for yet another machine
940 (e.g. cross-canadian recipes).
941
942The :term:`FILES` variable defines the
943files that go into each package in
944:term:`PACKAGES`. If you want
945details on how this is accomplished, you can look at
Andrew Geissler09209ee2020-12-13 08:44:15 -0600946:yocto_git:`package.bbclass </poky/tree/meta/classes/package.bbclass>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500947
948Depending on the type of packages being created (RPM, DEB, or IPK), the
949:ref:`do_package_write_* <ref-tasks-package_write_deb>`
950task creates the actual packages and places them in the Package Feed
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500951area, which is ``${TMPDIR}/deploy``. You can see the
952":ref:`overview-manual/concepts:package feeds`" section for more detail on
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500953that part of the build process.
954
955.. note::
956
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700957 Support for creating feeds directly from the ``deploy/*``
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500958 directories does not exist. Creating such feeds usually requires some
959 kind of feed maintenance mechanism that would upload the new packages
960 into an official package feed (e.g. the Ångström distribution). This
961 functionality is highly distribution-specific and thus is not
962 provided out of the box.
963
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500964Image Generation
965~~~~~~~~~~~~~~~~
966
967Once packages are split and stored in the Package Feeds area, the build
968system uses BitBake to generate the root filesystem image:
969
970.. image:: figures/image-generation.png
971 :align: center
972
973The image generation process consists of several stages and depends on
974several tasks and variables. The
975:ref:`ref-tasks-rootfs` task creates
976the root filesystem (file and directory structure) for an image. This
977task uses several key variables to help create the list of packages to
978actually install:
979
980- :term:`IMAGE_INSTALL`: Lists
981 out the base set of packages from which to install from the Package
982 Feeds area.
983
984- :term:`PACKAGE_EXCLUDE`:
985 Specifies packages that should not be installed into the image.
986
987- :term:`IMAGE_FEATURES`:
988 Specifies features to include in the image. Most of these features
989 map to additional packages for installation.
990
991- :term:`PACKAGE_CLASSES`:
992 Specifies the package backend (e.g. RPM, DEB, or IPK) to use and
993 consequently helps determine where to locate packages within the
994 Package Feeds area.
995
996- :term:`IMAGE_LINGUAS`:
997 Determines the language(s) for which additional language support
998 packages are installed.
999
1000- :term:`PACKAGE_INSTALL`:
1001 The final list of packages passed to the package manager for
1002 installation into the image.
1003
1004With :term:`IMAGE_ROOTFS`
1005pointing to the location of the filesystem under construction and the
Andrew Geissler09036742021-06-25 14:25:14 -05001006:term:`PACKAGE_INSTALL` variable providing the final list of packages to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001007install, the root file system is created.
1008
1009Package installation is under control of the package manager (e.g.
1010dnf/rpm, opkg, or apt/dpkg) regardless of whether or not package
1011management is enabled for the target. At the end of the process, if
1012package management is not enabled for the target, the package manager's
1013data files are deleted from the root filesystem. As part of the final
1014stage of package installation, post installation scripts that are part
1015of the packages are run. Any scripts that fail to run on the build host
1016are run on the target when the target system is first booted. If you are
1017using a
Andrew Geissler09209ee2020-12-13 08:44:15 -06001018:ref:`read-only root filesystem <dev-manual/common-tasks:creating a read-only root filesystem>`,
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001019all the post installation scripts must succeed on the build host during
1020the package installation phase since the root filesystem on the target
1021is read-only.
1022
1023The final stages of the ``do_rootfs`` task handle post processing. Post
1024processing includes creation of a manifest file and optimizations.
1025
1026The manifest file (``.manifest``) resides in the same directory as the
1027root filesystem image. This file lists out, line-by-line, the installed
1028packages. The manifest file is useful for the
1029:ref:`testimage <ref-classes-testimage*>` class,
1030for example, to determine whether or not to run specific tests. See the
1031:term:`IMAGE_MANIFEST`
1032variable for additional information.
1033
Andrew Geissler9aee5002022-03-30 16:27:02 +00001034Optimizing processes that are run across the image include ``mklibs``
1035and any other post-processing commands as defined by the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001036:term:`ROOTFS_POSTPROCESS_COMMAND`
Andrew Geissler9aee5002022-03-30 16:27:02 +00001037variable. The ``mklibs`` process optimizes the size of the libraries.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001038
1039After the root filesystem is built, processing begins on the image
1040through the :ref:`ref-tasks-image`
1041task. The build system runs any pre-processing commands as defined by
1042the
1043:term:`IMAGE_PREPROCESS_COMMAND`
1044variable. This variable specifies a list of functions to call before the
1045build system creates the final image output files.
1046
1047The build system dynamically creates ``do_image_*`` tasks as needed,
1048based on the image types specified in the
1049:term:`IMAGE_FSTYPES` variable.
1050The process turns everything into an image file or a set of image files
1051and can compress the root filesystem image to reduce the overall size of
1052the image. The formats used for the root filesystem depend on the
Andrew Geissler09036742021-06-25 14:25:14 -05001053:term:`IMAGE_FSTYPES` variable. Compression depends on whether the formats
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001054support compression.
1055
1056As an example, a dynamically created task when creating a particular
Andrew Geisslerc926e172021-05-07 16:11:35 -05001057image type would take the following form::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001058
1059 do_image_type
1060
1061So, if the type
Andrew Geissler09036742021-06-25 14:25:14 -05001062as specified by the :term:`IMAGE_FSTYPES` were ``ext4``, the dynamically
Andrew Geisslerc926e172021-05-07 16:11:35 -05001063generated task would be as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001064
1065 do_image_ext4
1066
1067The final task involved in image creation is the
1068:ref:`do_image_complete <ref-tasks-image-complete>`
1069task. This task completes the image by applying any image post
1070processing as defined through the
1071:term:`IMAGE_POSTPROCESS_COMMAND`
1072variable. The variable specifies a list of functions to call once the
1073build system has created the final image output files.
1074
1075.. note::
1076
1077 The entire image generation process is run under
1078 Pseudo. Running under Pseudo ensures that the files in the root filesystem
1079 have correct ownership.
1080
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001081SDK Generation
1082~~~~~~~~~~~~~~
1083
1084The OpenEmbedded build system uses BitBake to generate the Software
1085Development Kit (SDK) installer scripts for both the standard SDK and
1086the extensible SDK (eSDK):
1087
1088.. image:: figures/sdk-generation.png
1089 :align: center
1090
1091.. note::
1092
1093 For more information on the cross-development toolchain generation,
Andrew Geissler09209ee2020-12-13 08:44:15 -06001094 see the ":ref:`overview-manual/concepts:cross-development toolchain generation`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001095 section. For information on advantages gained when building a
1096 cross-development toolchain using the do_populate_sdk task, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001097 ":ref:`sdk-manual/appendix-obtain:building an sdk installer`" section in
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001098 the Yocto Project Application Development and the Extensible Software
1099 Development Kit (eSDK) manual.
1100
1101Like image generation, the SDK script process consists of several stages
1102and depends on many variables. The
1103:ref:`ref-tasks-populate_sdk`
1104and
1105:ref:`ref-tasks-populate_sdk_ext`
1106tasks use these key variables to help create the list of packages to
1107actually install. For information on the variables listed in the figure,
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001108see the ":ref:`overview-manual/concepts:application development sdk`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001109section.
1110
1111The ``do_populate_sdk`` task helps create the standard SDK and handles
1112two parts: a target part and a host part. The target part is the part
1113built for the target hardware and includes libraries and headers. The
1114host part is the part of the SDK that runs on the
1115:term:`SDKMACHINE`.
1116
1117The ``do_populate_sdk_ext`` task helps create the extensible SDK and
1118handles host and target parts differently than its counter part does for
1119the standard SDK. For the extensible SDK, the task encapsulates the
1120build system, which includes everything needed (host and target) for the
1121SDK.
1122
1123Regardless of the type of SDK being constructed, the tasks perform some
1124cleanup after which a cross-development environment setup script and any
1125needed configuration files are created. The final output is the
1126Cross-development toolchain installation script (``.sh`` file), which
1127includes the environment setup script.
1128
1129Stamp Files and the Rerunning of Tasks
1130~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1131
1132For each task that completes successfully, BitBake writes a stamp file
1133into the :term:`STAMPS_DIR`
1134directory. The beginning of the stamp file's filename is determined by
1135the :term:`STAMP` variable, and the end
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001136of the name consists of the task's name and current :ref:`input
1137checksum <overview-manual/concepts:checksums (signatures)>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001138
1139.. note::
1140
1141 This naming scheme assumes that
1142 BB_SIGNATURE_HANDLER
1143 is "OEBasicHash", which is almost always the case in current
1144 OpenEmbedded.
1145
1146To determine if a task needs to be rerun, BitBake checks if a stamp file
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001147with a matching input checksum exists for the task. In this case,
1148the task's output is assumed to exist and still be valid. Otherwise,
1149the task is rerun.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001150
1151.. note::
1152
1153 The stamp mechanism is more general than the shared state (sstate)
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001154 cache mechanism described in the
1155 ":ref:`overview-manual/concepts:setscene tasks and shared state`" section.
1156 BitBake avoids rerunning any task that has a valid stamp file, not just
1157 tasks that can be accelerated through the sstate cache.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001158
1159 However, you should realize that stamp files only serve as a marker
1160 that some work has been done and that these files do not record task
1161 output. The actual task output would usually be somewhere in
1162 :term:`TMPDIR` (e.g. in some
1163 recipe's :term:`WORKDIR`.) What
1164 the sstate cache mechanism adds is a way to cache task output that
1165 can then be shared between build machines.
1166
Andrew Geissler09036742021-06-25 14:25:14 -05001167Since :term:`STAMPS_DIR` is usually a subdirectory of :term:`TMPDIR`, removing
1168:term:`TMPDIR` will also remove :term:`STAMPS_DIR`, which means tasks will
1169properly be rerun to repopulate :term:`TMPDIR`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001170
1171If you want some task to always be considered "out of date", you can
1172mark it with the :ref:`nostamp <bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`
1173varflag. If some other task depends on such a task, then that task will
1174also always be considered out of date, which might not be what you want.
1175
1176For details on how to view information about a task's signature, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001177":ref:`dev-manual/common-tasks:viewing task variable dependencies`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001178section in the Yocto Project Development Tasks Manual.
1179
1180Setscene Tasks and Shared State
1181~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1182
1183The description of tasks so far assumes that BitBake needs to build
1184everything and no available prebuilt objects exist. BitBake does support
1185skipping tasks if prebuilt objects are available. These objects are
1186usually made available in the form of a shared state (sstate) cache.
1187
1188.. note::
1189
1190 For information on variables affecting sstate, see the
1191 :term:`SSTATE_DIR`
1192 and
1193 :term:`SSTATE_MIRRORS`
1194 variables.
1195
Andrew Geisslereff27472021-10-29 15:35:00 -05001196The idea of a setscene task (i.e ``do_taskname_setscene``) is a
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001197version of the task where instead of building something, BitBake can
1198skip to the end result and simply place a set of files into specific
1199locations as needed. In some cases, it makes sense to have a setscene
1200task variant (e.g. generating package files in the
1201:ref:`do_package_write_* <ref-tasks-package_write_deb>`
1202task). In other cases, it does not make sense (e.g. a
1203:ref:`ref-tasks-patch` task or a
1204:ref:`ref-tasks-unpack` task) since
1205the work involved would be equal to or greater than the underlying task.
1206
1207In the build system, the common tasks that have setscene variants are
1208:ref:`ref-tasks-package`,
1209``do_package_write_*``,
1210:ref:`ref-tasks-deploy`,
1211:ref:`ref-tasks-packagedata`, and
1212:ref:`ref-tasks-populate_sysroot`.
1213Notice that these tasks represent most of the tasks whose output is an
1214end result.
1215
1216The build system has knowledge of the relationship between these tasks
1217and other preceding tasks. For example, if BitBake runs
1218``do_populate_sysroot_setscene`` for something, it does not make sense
1219to run any of the ``do_fetch``, ``do_unpack``, ``do_patch``,
1220``do_configure``, ``do_compile``, and ``do_install`` tasks. However, if
1221``do_package`` needs to be run, BitBake needs to run those other tasks.
1222
1223It becomes more complicated if everything can come from an sstate cache
1224because some objects are simply not required at all. For example, you do
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001225not need a compiler or native tools, such as quilt, if there isn't anything
1226to compile or patch. If the ``do_package_write_*`` packages are available
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001227from sstate, BitBake does not need the ``do_package`` task data.
1228
1229To handle all these complexities, BitBake runs in two phases. The first
1230is the "setscene" stage. During this stage, BitBake first checks the
1231sstate cache for any targets it is planning to build. BitBake does a
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001232fast check to see if the object exists rather than doing a complete download.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001233If nothing exists, the second phase, which is the setscene stage,
1234completes and the main build proceeds.
1235
1236If objects are found in the sstate cache, the build system works
1237backwards from the end targets specified by the user. For example, if an
1238image is being built, the build system first looks for the packages
1239needed for that image and the tools needed to construct an image. If
1240those are available, the compiler is not needed. Thus, the compiler is
1241not even downloaded. If something was found to be unavailable, or the
1242download or setscene task fails, the build system then tries to install
1243dependencies, such as the compiler, from the cache.
1244
1245The availability of objects in the sstate cache is handled by the
Patrick Williams213cb262021-08-07 19:21:33 -05001246function specified by the :term:`BB_HASHCHECK_FUNCTION`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001247variable and returns a list of available objects. The function specified
Patrick Williams213cb262021-08-07 19:21:33 -05001248by the :term:`BB_SETSCENE_DEPVALID`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001249variable is the function that determines whether a given dependency
1250needs to be followed, and whether for any given relationship the
1251function needs to be passed. The function returns a True or False value.
1252
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001253Images
1254------
1255
1256The images produced by the build system are compressed forms of the root
1257filesystem and are ready to boot on a target device. You can see from
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001258the :ref:`general workflow figure
1259<overview-manual/concepts:openembedded build system concepts>` that BitBake
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001260output, in part, consists of images. This section takes a closer look at
1261this output:
1262
1263.. image:: figures/images.png
1264 :align: center
1265
1266.. note::
1267
1268 For a list of example images that the Yocto Project provides, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001269 ":doc:`/ref-manual/images`" chapter in the Yocto Project Reference
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001270 Manual.
1271
1272The build process writes images out to the :term:`Build Directory`
1273inside the
1274``tmp/deploy/images/machine/`` folder as shown in the figure. This
1275folder contains any files expected to be loaded on the target device.
1276The :term:`DEPLOY_DIR` variable
1277points to the ``deploy`` directory, while the
1278:term:`DEPLOY_DIR_IMAGE`
1279variable points to the appropriate directory containing images for the
1280current configuration.
1281
1282- kernel-image: A kernel binary file. The
1283 :term:`KERNEL_IMAGETYPE`
1284 variable determines the naming scheme for the kernel image file.
1285 Depending on this variable, the file could begin with a variety of
1286 naming strings. The ``deploy/images/``\ machine directory can contain
1287 multiple image files for the machine.
1288
1289- root-filesystem-image: Root filesystems for the target device (e.g.
1290 ``*.ext3`` or ``*.bz2`` files). The
1291 :term:`IMAGE_FSTYPES`
1292 variable determines the root filesystem image type. The
1293 ``deploy/images/``\ machine directory can contain multiple root
1294 filesystems for the machine.
1295
1296- kernel-modules: Tarballs that contain all the modules built for the
1297 kernel. Kernel module tarballs exist for legacy purposes and can be
1298 suppressed by setting the
1299 :term:`MODULE_TARBALL_DEPLOY`
1300 variable to "0". The ``deploy/images/``\ machine directory can
1301 contain multiple kernel module tarballs for the machine.
1302
1303- bootloaders: If applicable to the target machine, bootloaders
1304 supporting the image. The ``deploy/images/``\ machine directory can
1305 contain multiple bootloaders for the machine.
1306
1307- symlinks: The ``deploy/images/``\ machine folder contains a symbolic
1308 link that points to the most recently built file for each machine.
1309 These links might be useful for external scripts that need to obtain
1310 the latest version of each file.
1311
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001312Application Development SDK
1313---------------------------
1314
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001315In the :ref:`general workflow figure
1316<overview-manual/concepts:openembedded build system concepts>`, the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001317output labeled "Application Development SDK" represents an SDK. The SDK
1318generation process differs depending on whether you build an extensible
1319SDK (e.g. ``bitbake -c populate_sdk_ext`` imagename) or a standard SDK
1320(e.g. ``bitbake -c populate_sdk`` imagename). This section takes a
1321closer look at this output:
1322
1323.. image:: figures/sdk.png
1324 :align: center
1325
1326The specific form of this output is a set of files that includes a
1327self-extracting SDK installer (``*.sh``), host and target manifest
1328files, and files used for SDK testing. When the SDK installer file is
1329run, it installs the SDK. The SDK consists of a cross-development
1330toolchain, a set of libraries and headers, and an SDK environment setup
1331script. Running this installer essentially sets up your
1332cross-development environment. You can think of the cross-toolchain as
1333the "host" part because it runs on the SDK machine. You can think of the
1334libraries and headers as the "target" part because they are built for
1335the target hardware. The environment setup script is added so that you
1336can initialize the environment before using the tools.
1337
1338.. note::
1339
1340 - The Yocto Project supports several methods by which you can set up
1341 this cross-development environment. These methods include
1342 downloading pre-built SDK installers or building and installing
1343 your own SDK installer.
1344
1345 - For background information on cross-development toolchains in the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001346 Yocto Project development environment, see the
1347 ":ref:`overview-manual/concepts:cross-development toolchain generation`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001348 section.
1349
1350 - For information on setting up a cross-development environment, see
Andrew Geissler09209ee2020-12-13 08:44:15 -06001351 the :doc:`/sdk-manual/index` manual.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001352
1353All the output files for an SDK are written to the ``deploy/sdk`` folder
1354inside the :term:`Build Directory` as
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001355shown in the previous figure. Depending on the type of SDK, there are
1356several variables to configure these files. Here are the variables
1357associated with an extensible SDK:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001358
1359- :term:`DEPLOY_DIR`: Points to
1360 the ``deploy`` directory.
1361
1362- :term:`SDK_EXT_TYPE`:
1363 Controls whether or not shared state artifacts are copied into the
1364 extensible SDK. By default, all required shared state artifacts are
1365 copied into the SDK.
1366
1367- :term:`SDK_INCLUDE_PKGDATA`:
1368 Specifies whether or not packagedata is included in the extensible
1369 SDK for all recipes in the "world" target.
1370
1371- :term:`SDK_INCLUDE_TOOLCHAIN`:
1372 Specifies whether or not the toolchain is included when building the
1373 extensible SDK.
1374
Andrew Geissler9aee5002022-03-30 16:27:02 +00001375- :term:`ESDK_LOCALCONF_ALLOW`:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001376 A list of variables allowed through from the build system
1377 configuration into the extensible SDK configuration.
1378
Andrew Geissler9aee5002022-03-30 16:27:02 +00001379- :term:`ESDK_LOCALCONF_REMOVE`:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001380 A list of variables not allowed through from the build system
1381 configuration into the extensible SDK configuration.
1382
Andrew Geissler9aee5002022-03-30 16:27:02 +00001383- :term:`ESDK_CLASS_INHERIT_DISABLE`:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001384 A list of classes to remove from the
1385 :term:`INHERIT` value globally
1386 within the extensible SDK configuration.
1387
1388This next list, shows the variables associated with a standard SDK:
1389
1390- :term:`DEPLOY_DIR`: Points to
1391 the ``deploy`` directory.
1392
1393- :term:`SDKMACHINE`: Specifies
1394 the architecture of the machine on which the cross-development tools
1395 are run to create packages for the target hardware.
1396
1397- :term:`SDKIMAGE_FEATURES`:
1398 Lists the features to include in the "target" part of the SDK.
1399
1400- :term:`TOOLCHAIN_HOST_TASK`:
1401 Lists packages that make up the host part of the SDK (i.e. the part
Andrew Geissler09036742021-06-25 14:25:14 -05001402 that runs on the :term:`SDKMACHINE`). When you use
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001403 ``bitbake -c populate_sdk imagename`` to create the SDK, a set of
1404 default packages apply. This variable allows you to add more
1405 packages.
1406
1407- :term:`TOOLCHAIN_TARGET_TASK`:
1408 Lists packages that make up the target part of the SDK (i.e. the part
1409 built for the target hardware).
1410
1411- :term:`SDKPATH`: Defines the
1412 default SDK installation path offered by the installation script.
1413
1414- :term:`SDK_HOST_MANIFEST`:
1415 Lists all the installed packages that make up the host part of the
1416 SDK. This variable also plays a minor role for extensible SDK
1417 development as well. However, it is mainly used for the standard SDK.
1418
1419- :term:`SDK_TARGET_MANIFEST`:
1420 Lists all the installed packages that make up the target part of the
1421 SDK. This variable also plays a minor role for extensible SDK
1422 development as well. However, it is mainly used for the standard SDK.
1423
1424Cross-Development Toolchain Generation
1425======================================
1426
1427The Yocto Project does most of the work for you when it comes to
Andrew Geissler09209ee2020-12-13 08:44:15 -06001428creating :ref:`sdk-manual/intro:the cross-development toolchain`. This
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001429section provides some technical background on how cross-development
1430toolchains are created and used. For more information on toolchains, you
Andrew Geissler09209ee2020-12-13 08:44:15 -06001431can also see the :doc:`/sdk-manual/index` manual.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001432
1433In the Yocto Project development environment, cross-development
1434toolchains are used to build images and applications that run on the
1435target hardware. With just a few commands, the OpenEmbedded build system
1436creates these necessary toolchains for you.
1437
1438The following figure shows a high-level build environment regarding
1439toolchain construction and use.
1440
1441.. image:: figures/cross-development-toolchains.png
1442 :align: center
1443
1444Most of the work occurs on the Build Host. This is the machine used to
1445build images and generally work within the the Yocto Project
1446environment. When you run
1447:term:`BitBake` to create an image, the
1448OpenEmbedded build system uses the host ``gcc`` compiler to bootstrap a
1449cross-compiler named ``gcc-cross``. The ``gcc-cross`` compiler is what
1450BitBake uses to compile source files when creating the target image. You
1451can think of ``gcc-cross`` simply as an automatically generated
1452cross-compiler that is used internally within BitBake only.
1453
1454.. note::
1455
Andrew Geissler09036742021-06-25 14:25:14 -05001456 The extensible SDK does not use ``gcc-cross-canadian``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001457 since this SDK ships a copy of the OpenEmbedded build system and the
Andrew Geissler09036742021-06-25 14:25:14 -05001458 sysroot within it contains ``gcc-cross``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001459
Andrew Geisslerc926e172021-05-07 16:11:35 -05001460The chain of events that occurs when the standard toolchain is bootstrapped::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001461
Andrew Geissler6ce62a22020-11-30 19:58:47 -06001462 binutils-cross -> linux-libc-headers -> gcc-cross -> libgcc-initial -> glibc -> libgcc -> gcc-runtime
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001463
Andrew Geissler6ce62a22020-11-30 19:58:47 -06001464- ``gcc``: The compiler, GNU Compiler Collection (GCC).
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001465
Andrew Geissler6ce62a22020-11-30 19:58:47 -06001466- ``binutils-cross``: The binary utilities needed in order
1467 to run the ``gcc-cross`` phase of the bootstrap operation and build the
1468 headers for the C library.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001469
Andrew Geissler6ce62a22020-11-30 19:58:47 -06001470- ``linux-libc-headers``: Headers needed for the cross-compiler and C library build.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001471
Andrew Geissler6ce62a22020-11-30 19:58:47 -06001472- ``libgcc-initial``: An initial version of the gcc support library needed
1473 to bootstrap ``glibc``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001474
Andrew Geissler6ce62a22020-11-30 19:58:47 -06001475- ``libgcc``: The final version of the gcc support library which
1476 can only be built once there is a C library to link against.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001477
1478- ``glibc``: The GNU C Library.
1479
1480- ``gcc-cross``: The final stage of the bootstrap process for the
1481 cross-compiler. This stage results in the actual cross-compiler that
1482 BitBake uses when it builds an image for a targeted device.
1483
Andrew Geissler6ce62a22020-11-30 19:58:47 -06001484 This tool is a "native" tool (i.e. it is designed to run on
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001485 the build host).
1486
1487- ``gcc-runtime``: Runtime libraries resulting from the toolchain
1488 bootstrapping process. This tool produces a binary that consists of
1489 the runtime libraries need for the targeted device.
1490
1491You can use the OpenEmbedded build system to build an installer for the
1492relocatable SDK used to develop applications. When you run the
1493installer, it installs the toolchain, which contains the development
1494tools (e.g., ``gcc-cross-canadian``, ``binutils-cross-canadian``, and
1495other ``nativesdk-*`` tools), which are tools native to the SDK (i.e.
1496native to :term:`SDK_ARCH`), you
1497need to cross-compile and test your software. The figure shows the
1498commands you use to easily build out this toolchain. This
1499cross-development toolchain is built to execute on the
1500:term:`SDKMACHINE`, which might or
1501might not be the same machine as the Build Host.
1502
1503.. note::
1504
1505 If your target architecture is supported by the Yocto Project, you
1506 can take advantage of pre-built images that ship with the Yocto
1507 Project and already contain cross-development toolchain installers.
1508
Andrew Geisslerc926e172021-05-07 16:11:35 -05001509Here is the bootstrap process for the relocatable toolchain::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001510
1511 gcc -> binutils-crosssdk -> gcc-crosssdk-initial -> linux-libc-headers -> glibc-initial -> nativesdk-glibc -> gcc-crosssdk -> gcc-cross-canadian
1512
1513- ``gcc``: The build host's GNU Compiler Collection (GCC).
1514
1515- ``binutils-crosssdk``: The bare minimum binary utilities needed in
1516 order to run the ``gcc-crosssdk-initial`` phase of the bootstrap
1517 operation.
1518
1519- ``gcc-crosssdk-initial``: An early stage of the bootstrap process for
1520 creating the cross-compiler. This stage builds enough of the
1521 ``gcc-crosssdk`` and supporting pieces so that the final stage of the
1522 bootstrap process can produce the finished cross-compiler. This tool
1523 is a "native" binary that runs on the build host.
1524
1525- ``linux-libc-headers``: Headers needed for the cross-compiler.
1526
1527- ``glibc-initial``: An initial version of the Embedded GLIBC needed to
1528 bootstrap ``nativesdk-glibc``.
1529
1530- ``nativesdk-glibc``: The Embedded GLIBC needed to bootstrap the
1531 ``gcc-crosssdk``.
1532
1533- ``gcc-crosssdk``: The final stage of the bootstrap process for the
1534 relocatable cross-compiler. The ``gcc-crosssdk`` is a transitory
1535 compiler and never leaves the build host. Its purpose is to help in
1536 the bootstrap process to create the eventual ``gcc-cross-canadian``
1537 compiler, which is relocatable. This tool is also a "native" package
1538 (i.e. it is designed to run on the build host).
1539
1540- ``gcc-cross-canadian``: The final relocatable cross-compiler. When
1541 run on the :term:`SDKMACHINE`,
1542 this tool produces executable code that runs on the target device.
1543 Only one cross-canadian compiler is produced per architecture since
1544 they can be targeted at different processor optimizations using
1545 configurations passed to the compiler through the compile commands.
1546 This circumvents the need for multiple compilers and thus reduces the
1547 size of the toolchains.
1548
1549.. note::
1550
1551 For information on advantages gained when building a
1552 cross-development toolchain installer, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001553 ":ref:`sdk-manual/appendix-obtain:building an sdk installer`" appendix
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001554 in the Yocto Project Application Development and the
1555 Extensible Software Development Kit (eSDK) manual.
1556
1557Shared State Cache
1558==================
1559
1560By design, the OpenEmbedded build system builds everything from scratch
1561unless :term:`BitBake` can determine
1562that parts do not need to be rebuilt. Fundamentally, building from
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001563scratch is attractive as it means all parts are built fresh and there is
1564no possibility of stale data that can cause problems. When
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001565developers hit problems, they typically default back to building from
Andrew Geissler595f6302022-01-24 19:11:47 +00001566scratch so they have a known state from the start.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001567
1568Building an image from scratch is both an advantage and a disadvantage
1569to the process. As mentioned in the previous paragraph, building from
1570scratch ensures that everything is current and starts from a known
1571state. However, building from scratch also takes much longer as it
1572generally means rebuilding things that do not necessarily need to be
1573rebuilt.
1574
1575The Yocto Project implements shared state code that supports incremental
1576builds. The implementation of the shared state code answers the
1577following questions that were fundamental roadblocks within the
1578OpenEmbedded incremental build support system:
1579
1580- What pieces of the system have changed and what pieces have not
1581 changed?
1582
1583- How are changed pieces of software removed and replaced?
1584
1585- How are pre-built components that do not need to be rebuilt from
1586 scratch used when they are available?
1587
1588For the first question, the build system detects changes in the "inputs"
1589to a given task by creating a checksum (or signature) of the task's
1590inputs. If the checksum changes, the system assumes the inputs have
1591changed and the task needs to be rerun. For the second question, the
1592shared state (sstate) code tracks which tasks add which output to the
1593build process. This means the output from a given task can be removed,
1594upgraded or otherwise manipulated. The third question is partly
1595addressed by the solution for the second question assuming the build
1596system can fetch the sstate objects from remote locations and install
1597them if they are deemed to be valid.
1598
1599.. note::
1600
1601 - The build system does not maintain
1602 :term:`PR` information as part of
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001603 the shared state packages. Consequently, there are considerations that
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001604 affect maintaining shared state feeds. For information on how the
Andrew Geissler09036742021-06-25 14:25:14 -05001605 build system works with packages and can track incrementing :term:`PR`
Andrew Geissler09209ee2020-12-13 08:44:15 -06001606 information, see the ":ref:`dev-manual/common-tasks:automatically incrementing a package version number`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001607 section in the Yocto Project Development Tasks Manual.
1608
1609 - The code in the build system that supports incremental builds is
Andrew Geisslereff27472021-10-29 15:35:00 -05001610 complex. For techniques that help you work around issues
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001611 related to shared state code, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001612 ":ref:`dev-manual/common-tasks:viewing metadata used to create the input signature of a shared state task`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001613 and
Andrew Geissler09209ee2020-12-13 08:44:15 -06001614 ":ref:`dev-manual/common-tasks:invalidating shared state to force a task to run`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001615 sections both in the Yocto Project Development Tasks Manual.
1616
1617The rest of this section goes into detail about the overall incremental
1618build architecture, the checksums (signatures), and shared state.
1619
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001620Overall Architecture
1621--------------------
1622
1623When determining what parts of the system need to be built, BitBake
1624works on a per-task basis rather than a per-recipe basis. You might
1625wonder why using a per-task basis is preferred over a per-recipe basis.
1626To help explain, consider having the IPK packaging backend enabled and
1627then switching to DEB. In this case, the
1628:ref:`ref-tasks-install` and
1629:ref:`ref-tasks-package` task outputs
1630are still valid. However, with a per-recipe approach, the build would
1631not include the ``.deb`` files. Consequently, you would have to
1632invalidate the whole build and rerun it. Rerunning everything is not the
1633best solution. Also, in this case, the core must be "taught" much about
1634specific tasks. This methodology does not scale well and does not allow
1635users to easily add new tasks in layers or as external recipes without
1636touching the packaged-staging core.
1637
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001638Checksums (Signatures)
1639----------------------
1640
1641The shared state code uses a checksum, which is a unique signature of a
1642task's inputs, to determine if a task needs to be run again. Because it
1643is a change in a task's inputs that triggers a rerun, the process needs
1644to detect all the inputs to a given task. For shell tasks, this turns
1645out to be fairly easy because the build process generates a "run" shell
1646script for each task and it is possible to create a checksum that gives
1647you a good idea of when the task's data changes.
1648
1649To complicate the problem, there are things that should not be included
1650in the checksum. First, there is the actual specific build path of a
1651given task - the :term:`WORKDIR`. It
1652does not matter if the work directory changes because it should not
1653affect the output for target packages. Also, the build process has the
1654objective of making native or cross packages relocatable.
1655
1656.. note::
1657
1658 Both native and cross packages run on the
1659 build host. However, cross packages generate output for the target
1660 architecture.
1661
Andrew Geissler09036742021-06-25 14:25:14 -05001662The checksum therefore needs to exclude :term:`WORKDIR`. The simplistic
1663approach for excluding the work directory is to set :term:`WORKDIR` to some
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001664fixed value and create the checksum for the "run" script.
1665
1666Another problem results from the "run" scripts containing functions that
1667might or might not get called. The incremental build solution contains
1668code that figures out dependencies between shell functions. This code is
1669used to prune the "run" scripts down to the minimum set, thereby
1670alleviating this problem and making the "run" scripts much more readable
1671as a bonus.
1672
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001673So far, there are solutions for shell scripts. What about Python tasks? The
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001674same approach applies even though these tasks are more difficult. The
1675process needs to figure out what variables a Python function accesses
1676and what functions it calls. Again, the incremental build solution
1677contains code that first figures out the variable and function
1678dependencies, and then creates a checksum for the data used as the input
1679to the task.
1680
Andrew Geissler09036742021-06-25 14:25:14 -05001681Like the :term:`WORKDIR` case, there can be situations where dependencies should be
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001682ignored. For these situations, you can instruct the build process to
Andrew Geisslerc926e172021-05-07 16:11:35 -05001683ignore a dependency by using a line like the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001684
1685 PACKAGE_ARCHS[vardepsexclude] = "MACHINE"
1686
1687This example ensures that the :term:`PACKAGE_ARCHS` variable
1688does not depend on the value of :term:`MACHINE`, even if it does
1689reference it.
1690
1691Equally, there are cases where you need to add dependencies BitBake is
1692not able to find. You can accomplish this by using a line like the
Andrew Geisslerc926e172021-05-07 16:11:35 -05001693following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001694
1695 PACKAGE_ARCHS[vardeps] = "MACHINE"
1696
1697This example explicitly
Andrew Geissler09036742021-06-25 14:25:14 -05001698adds the :term:`MACHINE` variable as a dependency for :term:`PACKAGE_ARCHS`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001699
1700As an example, consider a case with in-line Python where BitBake is not
1701able to figure out dependencies. When running in debug mode (i.e. using
1702``-DDD``), BitBake produces output when it discovers something for which
1703it cannot figure out dependencies. The Yocto Project team has currently
1704not managed to cover those dependencies in detail and is aware of the
1705need to fix this situation.
1706
1707Thus far, this section has limited discussion to the direct inputs into
1708a task. Information based on direct inputs is referred to as the
1709"basehash" in the code. However, the question of a task's indirect
1710inputs still exits - items already built and present in the
1711:term:`Build Directory`. The checksum (or
1712signature) for a particular task needs to add the hashes of all the
1713tasks on which the particular task depends. Choosing which dependencies
Andrew Geissler595f6302022-01-24 19:11:47 +00001714to add is a policy decision. However, the effect is to generate a
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001715checksum that combines the basehash and the hashes of the task's
1716dependencies.
1717
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001718At the code level, there are multiple ways by which both the basehash
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001719and the dependent task hashes can be influenced. Within the BitBake
1720configuration file, you can give BitBake some extra information to help
1721it construct the basehash. The following statement effectively results
1722in a list of global variable dependency excludes (i.e. variables never
Andrew Geisslerc926e172021-05-07 16:11:35 -05001723included in any checksum)::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001724
Andrew Geissler9aee5002022-03-30 16:27:02 +00001725 BB_BASEHASH_IGNORE_VARS ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \\
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001726 SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \\
1727 USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \\
1728 PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \\
1729 CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX"
1730
Andrew Geissler595f6302022-01-24 19:11:47 +00001731The previous example does not include :term:`WORKDIR` since that variable is
1732actually constructed as a path within :term:`TMPDIR`, which is included above.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001733
1734The rules for deciding which hashes of dependent tasks to include
1735through dependency chains are more complex and are generally
1736accomplished with a Python function. The code in
1737``meta/lib/oe/sstatesig.py`` shows two examples of this and also
1738illustrates how you can insert your own policy into the system if so
1739desired. This file defines the two basic signature generators
1740:term:`OpenEmbedded-Core (OE-Core)` uses: "OEBasic" and
1741"OEBasicHash". By default, a dummy "noop" signature handler is enabled
1742in BitBake. This means that behavior is unchanged from previous
1743versions. OE-Core uses the "OEBasicHash" signature handler by default
Andrew Geisslerc926e172021-05-07 16:11:35 -05001744through this setting in the ``bitbake.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001745
1746 BB_SIGNATURE_HANDLER ?= "OEBasicHash"
1747
Andrew Geissler09036742021-06-25 14:25:14 -05001748The "OEBasicHash" :term:`BB_SIGNATURE_HANDLER` is the same
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001749as the "OEBasic" version but adds the task hash to the :ref:`stamp
1750files <overview-manual/concepts:stamp files and the rerunning of tasks>`. This
1751results in any metadata change that changes the task hash, automatically causing
1752the task to be run again. This removes the need to bump
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001753:term:`PR` values, and changes to metadata
1754automatically ripple across the build.
1755
1756It is also worth noting that the end result of these signature
1757generators is to make some dependency and hash information available to
1758the build. This information includes:
1759
Patrick Williams0ca19cc2021-08-16 14:03:13 -05001760- ``BB_BASEHASH:task-``\ taskname: The base hashes for each task in the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001761 recipe.
1762
1763- ``BB_BASEHASH_``\ filename\ ``:``\ taskname: The base hashes for each
1764 dependent task.
1765
Andrew Geissler09036742021-06-25 14:25:14 -05001766- :term:`BB_TASKHASH`: The hash of the currently running task.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001767
1768Shared State
1769------------
1770
1771Checksums and dependencies, as discussed in the previous section, solve
1772half the problem of supporting a shared state. The other half of the
1773problem is being able to use checksum information during the build and
1774being able to reuse or rebuild specific components.
1775
1776The :ref:`sstate <ref-classes-sstate>` class is a
1777relatively generic implementation of how to "capture" a snapshot of a
1778given task. The idea is that the build process does not care about the
1779source of a task's output. Output could be freshly built or it could be
1780downloaded and unpacked from somewhere. In other words, the build
1781process does not need to worry about its origin.
1782
1783Two types of output exist. One type is just about creating a directory
1784in :term:`WORKDIR`. A good example is
1785the output of either
1786:ref:`ref-tasks-install` or
1787:ref:`ref-tasks-package`. The other
1788type of output occurs when a set of data is merged into a shared
1789directory tree such as the sysroot.
1790
1791The Yocto Project team has tried to keep the details of the
Andrew Geissler595f6302022-01-24 19:11:47 +00001792implementation hidden in the :ref:`sstate <ref-classes-sstate>` class. From a user's perspective,
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001793adding shared state wrapping to a task is as simple as this
1794:ref:`ref-tasks-deploy` example taken
Andrew Geisslerc926e172021-05-07 16:11:35 -05001795from the :ref:`deploy <ref-classes-deploy>` class::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001796
1797 DEPLOYDIR = "${WORKDIR}/deploy-${PN}"
1798 SSTATETASKS += "do_deploy"
1799 do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"
1800 do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
1801
1802 python do_deploy_setscene () {
1803 sstate_setscene(d)
1804 }
1805 addtask do_deploy_setscene
1806 do_deploy[dirs] = "${DEPLOYDIR} ${B}"
1807 do_deploy[stamp-extra-info] = "${MACHINE_ARCH}"
1808
1809The following list explains the previous example:
1810
1811- Adding "do_deploy" to ``SSTATETASKS`` adds some required
1812 sstate-related processing, which is implemented in the
1813 :ref:`sstate <ref-classes-sstate>` class, to
1814 before and after the
1815 :ref:`ref-tasks-deploy` task.
1816
1817- The ``do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"`` declares that
1818 ``do_deploy`` places its output in ``${DEPLOYDIR}`` when run normally
1819 (i.e. when not using the sstate cache). This output becomes the input
1820 to the shared state cache.
1821
1822- The ``do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"`` line
1823 causes the contents of the shared state cache to be copied to
1824 ``${DEPLOY_DIR_IMAGE}``.
1825
1826 .. note::
1827
1828 If ``do_deploy`` is not already in the shared state cache or if its input
1829 checksum (signature) has changed from when the output was cached, the task
1830 runs to populate the shared state cache, after which the contents of the
1831 shared state cache is copied to ${:term:`DEPLOY_DIR_IMAGE`}. If
1832 ``do_deploy`` is in the shared state cache and its signature indicates
1833 that the cached output is still valid (i.e. if no relevant task inputs
1834 have changed), then the contents of the shared state cache copies
Andrew Geissler09036742021-06-25 14:25:14 -05001835 directly to ${:term:`DEPLOY_DIR_IMAGE`} by the ``do_deploy_setscene`` task
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001836 instead, skipping the ``do_deploy`` task.
1837
1838- The following task definition is glue logic needed to make the
Andrew Geisslerc926e172021-05-07 16:11:35 -05001839 previous settings effective::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001840
1841 python do_deploy_setscene () {
1842 sstate_setscene(d)
1843 }
1844 addtask do_deploy_setscene
1845
1846 ``sstate_setscene()`` takes the flags above as input and accelerates the ``do_deploy`` task
1847 through the shared state cache if possible. If the task was
1848 accelerated, ``sstate_setscene()`` returns True. Otherwise, it
1849 returns False, and the normal ``do_deploy`` task runs. For more
Patrick Williams213cb262021-08-07 19:21:33 -05001850 information, see the ":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-execution:setscene`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001851 section in the BitBake User Manual.
1852
1853- The ``do_deploy[dirs] = "${DEPLOYDIR} ${B}"`` line creates
1854 ``${DEPLOYDIR}`` and ``${B}`` before the ``do_deploy`` task runs, and
1855 also sets the current working directory of ``do_deploy`` to ``${B}``.
1856 For more information, see the ":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags`"
1857 section in the BitBake
1858 User Manual.
1859
1860 .. note::
1861
1862 In cases where ``sstate-inputdirs`` and ``sstate-outputdirs`` would be
1863 the same, you can use ``sstate-plaindirs``. For example, to preserve the
1864 ${:term:`PKGD`} and ${:term:`PKGDEST`} output from the ``do_package``
Andrew Geisslerc926e172021-05-07 16:11:35 -05001865 task, use the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001866
1867 do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}"
1868
1869
1870- The ``do_deploy[stamp-extra-info] = "${MACHINE_ARCH}"`` line appends
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001871 extra metadata to the :ref:`stamp
1872 file <overview-manual/concepts:stamp files and the rerunning of tasks>`. In
1873 this case, the metadata makes the task specific to a machine's architecture.
1874 See
Andrew Geissler09209ee2020-12-13 08:44:15 -06001875 ":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-execution:the task list`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001876 section in the BitBake User Manual for more information on the
1877 ``stamp-extra-info`` flag.
1878
1879- ``sstate-inputdirs`` and ``sstate-outputdirs`` can also be used with
1880 multiple directories. For example, the following declares
Andrew Geissler09036742021-06-25 14:25:14 -05001881 :term:`PKGDESTWORK` and ``SHLIBWORK`` as shared state input directories,
1882 which populates the shared state cache, and :term:`PKGDATA_DIR` and
Andrew Geisslerc926e172021-05-07 16:11:35 -05001883 ``SHLIBSDIR`` as the corresponding shared state output directories::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001884
1885 do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}"
1886 do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}"
1887
1888- These methods also include the ability to take a lockfile when
1889 manipulating shared state directory structures, for cases where file
Andrew Geisslerc926e172021-05-07 16:11:35 -05001890 additions or removals are sensitive::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001891
1892 do_package[sstate-lockfile] = "${PACKAGELOCK}"
1893
1894Behind the scenes, the shared state code works by looking in
1895:term:`SSTATE_DIR` and
1896:term:`SSTATE_MIRRORS` for
Andrew Geisslerc926e172021-05-07 16:11:35 -05001897shared state files. Here is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001898
1899 SSTATE_MIRRORS ?= "\
Andrew Geissler7e0e3c02022-02-25 20:34:39 +00001900 file://.* https://someserver.tld/share/sstate/PATH;downloadfilename=PATH \
Patrick Williams93c203f2021-10-06 16:15:23 -05001901 file://.* file:///some/local/dir/sstate/PATH"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001902
1903.. note::
1904
Andrew Geissler5f350902021-07-23 13:09:54 -04001905 The shared state directory (:term:`SSTATE_DIR`) is organized into two-character
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001906 subdirectories, where the subdirectory names are based on the first two
1907 characters of the hash.
1908 If the shared state directory structure for a mirror has the same structure
Andrew Geissler09036742021-06-25 14:25:14 -05001909 as :term:`SSTATE_DIR`, you must specify "PATH" as part of the URI to enable the build
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001910 system to map to the appropriate subdirectory.
1911
1912The shared state package validity can be detected just by looking at the
1913filename since the filename contains the task checksum (or signature) as
1914described earlier in this section. If a valid shared state package is
1915found, the build process downloads it and uses it to accelerate the
1916task.
1917
1918The build processes use the ``*_setscene`` tasks for the task
1919acceleration phase. BitBake goes through this phase before the main
1920execution code and tries to accelerate any tasks for which it can find
1921shared state packages. If a shared state package for a task is
1922available, the shared state package is used. This means the task and any
1923tasks on which it is dependent are not executed.
1924
1925As a real world example, the aim is when building an IPK-based image,
1926only the
1927:ref:`ref-tasks-package_write_ipk`
1928tasks would have their shared state packages fetched and extracted.
1929Since the sysroot is not used, it would never get extracted. This is
1930another reason why a task-based approach is preferred over a
1931recipe-based approach, which would have to install the output from every
1932task.
1933
Andrew Geissler595f6302022-01-24 19:11:47 +00001934Hash Equivalence
1935----------------
1936
1937The above section explained how BitBake skips the execution of tasks
1938whose output can already be found in the Shared State cache.
1939
1940During a build, it may often be the case that the output / result of a task might
1941be unchanged despite changes in the task's input values. An example might be
1942whitespace changes in some input C code. In project terms, this is what we define
1943as "equivalence".
1944
1945To keep track of such equivalence, BitBake has to manage three hashes
1946for each task:
1947
1948- The *task hash* explained earlier: computed from the recipe metadata,
1949 the task code and the task hash values from its dependencies.
1950 When changes are made, these task hashes are therefore modified,
1951 causing the task to re-execute. The task hashes of tasks depending on this
1952 task are therefore modified too, causing the whole dependency
1953 chain to re-execute.
1954
1955- The *output hash*, a new hash computed from the output of Shared State tasks,
1956 tasks that save their resulting output to a Shared State tarball.
1957 The mapping between the task hash and its output hash is reported
1958 to a new *Hash Equivalence* server. This mapping is stored in a database
1959 by the server for future reference.
1960
1961- The *unihash*, a new hash, initially set to the task hash for the task.
1962 This is used to track the *unicity* of task output, and we will explain
1963 how its value is maintained.
1964
1965When Hash Equivalence is enabled, BitBake computes the task hash
1966for each task by using the unihash of its dependencies, instead
1967of their task hash.
1968
1969Now, imagine that a Shared State task is modified because of a change in
1970its code or metadata, or because of a change in its dependencies.
1971Since this modifies its task hash, this task will need re-executing.
1972Its output hash will therefore be computed again.
1973
1974Then, the new mapping between the new task hash and its output hash
1975will be reported to the Hash Equivalence server. The server will
1976let BitBake know whether this output hash is the same as a previously
1977reported output hash, for a different task hash.
1978
1979If the output hash is already known, BitBake will update the task's
1980unihash to match the original task hash that generated that output.
1981Thanks to this, the depending tasks will keep a previously recorded
1982task hash, and BitBake will be able to retrieve their output from
1983the Shared State cache, instead of re-executing them. Similarly, the
1984output of further downstream tasks can also be retrieved from Shared
1985Shate.
1986
1987If the output hash is unknown, a new entry will be created on the Hash
1988Equivalence server, matching the task hash to that output.
1989The depending tasks, still having a new task hash because of the
1990change, will need to re-execute as expected. The change propagates
1991to the depending tasks.
1992
1993To summarize, when Hash Equivalence is enabled, a change in one of the
1994tasks in BitBake's run queue doesn't have to propagate to all the
1995downstream tasks that depend on the output of this task, causing a
1996full rebuild of such tasks, and so on with the next depending tasks.
1997Instead, when the output of this task remains identical to previously
1998recorded output, BitBake can safely retrieve all the downstream
1999task output from the Shared State cache.
2000
2001.. note::
2002
2003 Having :doc:`/test-manual/reproducible-builds` is a key ingredient for
2004 the stability of the task's output hash. Therefore, the effectiveness
2005 of Hash Equivalence strongly depends on it.
2006
2007This applies to multiple scenarios:
2008
2009- A "trivial" change to a recipe that doesn't impact its generated output,
2010 such as whitespace changes, modifications to unused code paths or
2011 in the ordering of variables.
2012
2013- Shared library updates, for example to fix a security vulnerability.
2014 For sure, the programs using such a library should be rebuilt, but
2015 their new binaries should remain identical. The corresponding tasks should
2016 have a different output hash because of the change in the hash of their
2017 library dependency, but thanks to their output being identical, Hash
2018 Equivalence will stop the propagation down the dependency chain.
2019
2020- Native tool updates. Though the depending tasks should be rebuilt,
2021 it's likely that they will generate the same output and be marked
2022 as equivalent.
2023
2024This mechanism is enabled by default in Poky, and is controlled by three
2025variables:
2026
2027- :term:`bitbake:BB_HASHSERVE`, specifying a local or remote Hash
2028 Equivalence server to use.
2029
2030- :term:`BB_HASHSERVE_UPSTREAM`, when ``BB_HASHSERVE = "auto"``,
2031 allowing to connect the local server to an upstream one.
2032
2033- :term:`bitbake:BB_SIGNATURE_HANDLER`, which must be set to ``OEEquivHash``.
2034
2035Therefore, the default configuration in Poky corresponds to the
2036below settings::
2037
2038 BB_HASHSERVE = "auto"
2039 BB_SIGNATURE_HANDLER = "OEEquivHash"
2040
2041Rather than starting a local server, another possibility is to rely
2042on a Hash Equivalence server on a network, by setting::
2043
2044 BB_HASHSERVE = "<HOSTNAME>:<PORT>"
2045
2046.. note::
2047
2048 The shared Hash Equivalence server needs to be maintained together with the
2049 Shared State cache. Otherwise, the server could report Shared State hashes
2050 that only exist on specific clients.
2051
2052 We therefore recommend that one Hash Equivalence server be set up to
2053 correspond with a given Shared State cache, and to start this server
2054 in *read-only mode*, so that it doesn't store equivalences for
2055 Shared State caches that are local to clients.
2056
2057 See the :term:`BB_HASHSERVE` reference for details about starting
2058 a Hash Equivalence server.
2059
2060See the `video <https://www.youtube.com/watch?v=zXEdqGS62Wc>`__
2061of Joshua Watt's `Hash Equivalence and Reproducible Builds
2062<https://elinux.org/images/3/37/Hash_Equivalence_and_Reproducible_Builds.pdf>`__
2063presentation at ELC 2020 for a very synthetic introduction to the
2064Hash Equivalence implementation in the Yocto Project.
2065
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002066Automatically Added Runtime Dependencies
2067========================================
2068
2069The OpenEmbedded build system automatically adds common types of runtime
2070dependencies between packages, which means that you do not need to
2071explicitly declare the packages using
William A. Kennington IIIac69b482021-06-02 12:28:27 -07002072:term:`RDEPENDS`. There are three automatic
2073mechanisms (``shlibdeps``, ``pcdeps``, and ``depchains``) that
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002074handle shared libraries, package configuration (pkg-config) modules, and
2075``-dev`` and ``-dbg`` packages, respectively. For other types of runtime
2076dependencies, you must manually declare the dependencies.
2077
2078- ``shlibdeps``: During the
2079 :ref:`ref-tasks-package` task of
2080 each recipe, all shared libraries installed by the recipe are
2081 located. For each shared library, the package that contains the
2082 shared library is registered as providing the shared library. More
2083 specifically, the package is registered as providing the
2084 `soname <https://en.wikipedia.org/wiki/Soname>`__ of the library. The
2085 resulting shared-library-to-package mapping is saved globally in
2086 :term:`PKGDATA_DIR` by the
2087 :ref:`ref-tasks-packagedata`
2088 task.
2089
2090 Simultaneously, all executables and shared libraries installed by the
2091 recipe are inspected to see what shared libraries they link against.
Andrew Geissler09036742021-06-25 14:25:14 -05002092 For each shared library dependency that is found, :term:`PKGDATA_DIR` is
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002093 queried to see if some package (likely from a different recipe)
2094 contains the shared library. If such a package is found, a runtime
2095 dependency is added from the package that depends on the shared
2096 library to the package that contains the library.
2097
2098 The automatically added runtime dependency also includes a version
2099 restriction. This version restriction specifies that at least the
2100 current version of the package that provides the shared library must
Andrew Geissler09036742021-06-25 14:25:14 -05002101 be used, as if "package (>= version)" had been added to :term:`RDEPENDS`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002102 This forces an upgrade of the package containing the shared library
2103 when installing the package that depends on the library, if needed.
2104
2105 If you want to avoid a package being registered as providing a
2106 particular shared library (e.g. because the library is for internal
2107 use only), then add the library to
2108 :term:`PRIVATE_LIBS` inside
2109 the package's recipe.
2110
2111- ``pcdeps``: During the ``do_package`` task of each recipe, all
2112 pkg-config modules (``*.pc`` files) installed by the recipe are
2113 located. For each module, the package that contains the module is
2114 registered as providing the module. The resulting module-to-package
Andrew Geissler09036742021-06-25 14:25:14 -05002115 mapping is saved globally in :term:`PKGDATA_DIR` by the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002116 ``do_packagedata`` task.
2117
2118 Simultaneously, all pkg-config modules installed by the recipe are
2119 inspected to see what other pkg-config modules they depend on. A
2120 module is seen as depending on another module if it contains a
2121 "Requires:" line that specifies the other module. For each module
Andrew Geissler09036742021-06-25 14:25:14 -05002122 dependency, :term:`PKGDATA_DIR` is queried to see if some package
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002123 contains the module. If such a package is found, a runtime dependency
2124 is added from the package that depends on the module to the package
2125 that contains the module.
2126
2127 .. note::
2128
2129 The
2130 pcdeps
2131 mechanism most often infers dependencies between
2132 -dev
2133 packages.
2134
2135- ``depchains``: If a package ``foo`` depends on a package ``bar``,
2136 then ``foo-dev`` and ``foo-dbg`` are also made to depend on
2137 ``bar-dev`` and ``bar-dbg``, respectively. Taking the ``-dev``
2138 packages as an example, the ``bar-dev`` package might provide headers
2139 and shared library symlinks needed by ``foo-dev``, which shows the
2140 need for a dependency between the packages.
2141
2142 The dependencies added by ``depchains`` are in the form of
2143 :term:`RRECOMMENDS`.
2144
2145 .. note::
2146
Andrew Geissler5f350902021-07-23 13:09:54 -04002147 By default, ``foo-dev`` also has an :term:`RDEPENDS`-style dependency on
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002148 ``foo``, because the default value of ``RDEPENDS:${PN}-dev`` (set in
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002149 bitbake.conf) includes "${PN}".
2150
2151 To ensure that the dependency chain is never broken, ``-dev`` and
2152 ``-dbg`` packages are always generated by default, even if the
2153 packages turn out to be empty. See the
2154 :term:`ALLOW_EMPTY` variable
2155 for more information.
2156
2157The ``do_package`` task depends on the ``do_packagedata`` task of each
2158recipe in :term:`DEPENDS` through use
2159of a ``[``\ :ref:`deptask <bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ``]``
2160declaration, which guarantees that the required
2161shared-library/module-to-package mapping information will be available
Andrew Geissler09036742021-06-25 14:25:14 -05002162when needed as long as :term:`DEPENDS` has been correctly set.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002163
2164Fakeroot and Pseudo
2165===================
2166
2167Some tasks are easier to implement when allowed to perform certain
2168operations that are normally reserved for the root user (e.g.
2169:ref:`ref-tasks-install`,
2170:ref:`do_package_write* <ref-tasks-package_write_deb>`,
2171:ref:`ref-tasks-rootfs`, and
2172:ref:`do_image* <ref-tasks-image>`). For example,
2173the ``do_install`` task benefits from being able to set the UID and GID
2174of installed files to arbitrary values.
2175
2176One approach to allowing tasks to perform root-only operations would be
2177to require :term:`BitBake` to run as
2178root. However, this method is cumbersome and has security issues. The
2179approach that is actually used is to run tasks that benefit from root
2180privileges in a "fake" root environment. Within this environment, the
2181task and its child processes believe that they are running as the root
2182user, and see an internally consistent view of the filesystem. As long
2183as generating the final output (e.g. a package or an image) does not
2184require root privileges, the fact that some earlier steps ran in a fake
2185root environment does not cause problems.
2186
2187The capability to run tasks in a fake root environment is known as
2188"`fakeroot <http://man.he.net/man1/fakeroot>`__", which is derived from
2189the BitBake keyword/variable flag that requests a fake root environment
2190for a task.
2191
Andrew Geisslerd1e89492021-02-12 15:35:20 -06002192In the :term:`OpenEmbedded Build System`, the program that implements
2193fakeroot is known as :yocto_home:`Pseudo </software-item/pseudo/>`. Pseudo
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002194overrides system calls by using the environment variable ``LD_PRELOAD``,
2195which results in the illusion of running as root. To keep track of
2196"fake" file ownership and permissions resulting from operations that
2197require root permissions, Pseudo uses an SQLite 3 database. This
2198database is stored in
2199``${``\ :term:`WORKDIR`\ ``}/pseudo/files.db``
2200for individual recipes. Storing the database in a file as opposed to in
2201memory gives persistence between tasks and builds, which is not
2202accomplished using fakeroot.
2203
2204.. note::
2205
2206 If you add your own task that manipulates the same files or
2207 directories as a fakeroot task, then that task also needs to run
2208 under fakeroot. Otherwise, the task cannot run root-only operations,
2209 and cannot see the fake file ownership and permissions set by the
2210 other task. You need to also add a dependency on
Andrew Geisslerc926e172021-05-07 16:11:35 -05002211 ``virtual/fakeroot-native:do_populate_sysroot``, giving the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002212
2213 fakeroot do_mytask () {
2214 ...
2215 }
2216 do_mytask[depends] += "virtual/fakeroot-native:do_populate_sysroot"
2217
2218
2219For more information, see the
2220:term:`FAKEROOT* <bitbake:FAKEROOT>` variables in the
2221BitBake User Manual. You can also reference the "`Why Not
2222Fakeroot? <https://github.com/wrpseudo/pseudo/wiki/WhyNotFakeroot>`__"
2223article for background information on Fakeroot and Pseudo.