blob: e5bdcdad2a359743cd1bd119273e3be0ee714a04 [file] [log] [blame]
Andrew Geisslerf0343792020-11-18 10:42:21 -06001.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002
3**********************
4Yocto Project Concepts
5**********************
6
7This chapter provides explanations for Yocto Project concepts that go
8beyond the surface of "how-to" information and reference (or look-up)
9material. Concepts such as components, the :term:`OpenEmbedded Build System`
10workflow,
11cross-development toolchains, shared state cache, and so forth are
12explained.
13
14Yocto Project Components
15========================
16
17The :term:`BitBake` task executor
18together with various types of configuration files form the
19:term:`OpenEmbedded-Core (OE-Core)`. This section
20overviews these components by describing their use and how they
21interact.
22
23BitBake handles the parsing and execution of the data files. The data
24itself is of various types:
25
26- *Recipes:* Provides details about particular pieces of software.
27
28- *Class Data:* Abstracts common build information (e.g. how to build a
29 Linux kernel).
30
31- *Configuration Data:* Defines machine-specific settings, policy
32 decisions, and so forth. Configuration data acts as the glue to bind
33 everything together.
34
35BitBake knows how to combine multiple data sources together and refers
36to each data source as a layer. For information on layers, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -060037":ref:`dev-manual/common-tasks:understanding and creating layers`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050038section of the Yocto Project Development Tasks Manual.
39
40Following are some brief details on these core components. For
41additional information on how these components interact during a build,
42see the
Andrew Geissler09209ee2020-12-13 08:44:15 -060043":ref:`overview-manual/concepts:openembedded build system concepts`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050044section.
45
Andrew Geisslerc9f78652020-09-18 14:11:35 -050046BitBake
47-------
48
49BitBake is the tool at the heart of the :term:`OpenEmbedded Build System`
50and is responsible
51for parsing the :term:`Metadata`, generating
52a list of tasks from it, and then executing those tasks.
53
54This section briefly introduces BitBake. If you want more information on
55BitBake, see the :doc:`BitBake User Manual <bitbake:index>`.
56
57To see a list of the options BitBake supports, use either of the
Andrew Geisslerc926e172021-05-07 16:11:35 -050058following commands::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050059
60 $ bitbake -h
61 $ bitbake --help
62
63The most common usage for BitBake is ``bitbake recipename``, where
64``recipename`` is the name of the recipe you want to build (referred
65to as the "target"). The target often equates to the first part of a
66recipe's filename (e.g. "foo" for a recipe named ``foo_1.3.0-r0.bb``).
67So, to process the ``matchbox-desktop_1.2.3.bb`` recipe file, you might
Andrew Geisslerc926e172021-05-07 16:11:35 -050068type the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050069
70 $ bitbake matchbox-desktop
71
72Several different
73versions of ``matchbox-desktop`` might exist. BitBake chooses the one
74selected by the distribution configuration. You can get more details
75about how BitBake chooses between different target versions and
76providers in the
Andrew Geissler09209ee2020-12-13 08:44:15 -060077":ref:`Preferences <bitbake:bitbake-user-manual/bitbake-user-manual-execution:preferences>`" section
Andrew Geisslerc9f78652020-09-18 14:11:35 -050078of the BitBake User Manual.
79
80BitBake also tries to execute any dependent tasks first. So for example,
81before building ``matchbox-desktop``, BitBake would build a cross
82compiler and ``glibc`` if they had not already been built.
83
84A useful BitBake option to consider is the ``-k`` or ``--continue``
85option. This option instructs BitBake to try and continue processing the
86job as long as possible even after encountering an error. When an error
87occurs, the target that failed and those that depend on it cannot be
88remade. However, when you use this option other dependencies can still
89be processed.
90
Andrew Geisslerc9f78652020-09-18 14:11:35 -050091Recipes
92-------
93
94Files that have the ``.bb`` suffix are "recipes" files. In general, a
95recipe contains information about a single piece of software. This
96information includes the location from which to download the unaltered
97source, any source patches to be applied to that source (if needed),
98which special configuration options to apply, how to compile the source
99files, and how to package the compiled output.
100
101The term "package" is sometimes used to refer to recipes. However, since
102the word "package" is used for the packaged output from the OpenEmbedded
103build system (i.e. ``.ipk`` or ``.deb`` files), this document avoids
104using the term "package" when referring to recipes.
105
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500106Classes
107-------
108
109Class files (``.bbclass``) contain information that is useful to share
110between recipes files. An example is the
111:ref:`autotools <ref-classes-autotools>` class,
112which contains common settings for any application that Autotools uses.
Andrew Geissler09209ee2020-12-13 08:44:15 -0600113The ":ref:`ref-manual/classes:Classes`" chapter in the
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500114Yocto Project Reference Manual provides details about classes and how to
115use them.
116
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500117Configurations
118--------------
119
120The configuration files (``.conf``) define various configuration
121variables that govern the OpenEmbedded build process. These files fall
122into several areas that define machine configuration options,
123distribution configuration options, compiler tuning options, general
124common configuration options, and user configuration options in
125``conf/local.conf``, which is found in the :term:`Build Directory`.
126
127
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500128Layers
129======
130
131Layers are repositories that contain related metadata (i.e. sets of
132instructions) that tell the OpenEmbedded build system how to build a
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500133target. :ref:`overview-manual/yp-intro:the yocto project layer model`
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500134facilitates collaboration, sharing, customization, and reuse within the
135Yocto Project development environment. Layers logically separate
136information for your project. For example, you can use a layer to hold
137all the configurations for a particular piece of hardware. Isolating
138hardware-specific configurations allows you to share other metadata by
139using a different layer where that metadata might be common across
140several pieces of hardware.
141
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700142There are many layers working in the Yocto Project development environment. The
Andrew Geisslerd1e89492021-02-12 15:35:20 -0600143:yocto_home:`Yocto Project Curated Layer Index </software-overview/layers/>`
144and :oe_layerindex:`OpenEmbedded Layer Index <>` both contain layers from
145which you can use or leverage.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500146
147By convention, layers in the Yocto Project follow a specific form.
148Conforming to a known structure allows BitBake to make assumptions
149during builds on where to find types of metadata. You can find
150procedures and learn about tools (i.e. ``bitbake-layers``) for creating
151layers suitable for the Yocto Project in the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600152":ref:`dev-manual/common-tasks:understanding and creating layers`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500153section of the Yocto Project Development Tasks Manual.
154
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500155OpenEmbedded Build System Concepts
156==================================
157
158This section takes a more detailed look inside the build process used by
159the :term:`OpenEmbedded Build System`,
160which is the build
161system specific to the Yocto Project. At the heart of the build system
162is BitBake, the task executor.
163
164The following diagram represents the high-level workflow of a build. The
165remainder of this section expands on the fundamental input, output,
166process, and metadata logical blocks that make up the workflow.
167
168.. image:: figures/YP-flow-diagram.png
169 :align: center
170
171In general, the build's workflow consists of several functional areas:
172
173- *User Configuration:* metadata you can use to control the build
174 process.
175
176- *Metadata Layers:* Various layers that provide software, machine, and
177 distro metadata.
178
179- *Source Files:* Upstream releases, local projects, and SCMs.
180
181- *Build System:* Processes under the control of
182 :term:`BitBake`. This block expands
183 on how BitBake fetches source, applies patches, completes
184 compilation, analyzes output for package generation, creates and
185 tests packages, generates images, and generates cross-development
186 tools.
187
188- *Package Feeds:* Directories containing output packages (RPM, DEB or
189 IPK), which are subsequently used in the construction of an image or
190 Software Development Kit (SDK), produced by the build system. These
191 feeds can also be copied and shared using a web server or other means
192 to facilitate extending or updating existing images on devices at
193 runtime if runtime package management is enabled.
194
195- *Images:* Images produced by the workflow.
196
197- *Application Development SDK:* Cross-development tools that are
198 produced along with an image or separately with BitBake.
199
200User Configuration
201------------------
202
203User configuration helps define the build. Through user configuration,
204you can tell BitBake the target architecture for which you are building
205the image, where to store downloaded source, and other build properties.
206
207The following figure shows an expanded representation of the "User
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500208Configuration" box of the :ref:`general workflow
209figure <overview-manual/concepts:openembedded build system concepts>`:
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500210
211.. image:: figures/user-configuration.png
212 :align: center
213
214BitBake needs some basic configuration files in order to complete a
215build. These files are ``*.conf`` files. The minimally necessary ones
216reside as example files in the ``build/conf`` directory of the
217:term:`Source Directory`. For simplicity,
218this section refers to the Source Directory as the "Poky Directory."
219
220When you clone the :term:`Poky` Git repository
221or you download and unpack a Yocto Project release, you can set up the
222Source Directory to be named anything you want. For this discussion, the
223cloned repository uses the default name ``poky``.
224
225.. note::
226
227 The Poky repository is primarily an aggregation of existing
228 repositories. It is not a canonical upstream source.
229
230The ``meta-poky`` layer inside Poky contains a ``conf`` directory that
231has example configuration files. These example files are used as a basis
232for creating actual configuration files when you source
233:ref:`structure-core-script`, which is the
234build environment script.
235
236Sourcing the build environment script creates a
237:term:`Build Directory` if one does not
238already exist. BitBake uses the Build Directory for all its work during
239builds. The Build Directory has a ``conf`` directory that contains
240default versions of your ``local.conf`` and ``bblayers.conf``
241configuration files. These default configuration files are created only
242if versions do not already exist in the Build Directory at the time you
243source the build environment setup script.
244
245Because the Poky repository is fundamentally an aggregation of existing
246repositories, some users might be familiar with running the
247:ref:`structure-core-script` script in the context of separate
248:term:`OpenEmbedded-Core (OE-Core)` and BitBake
249repositories rather than a single Poky repository. This discussion
250assumes the script is executed from within a cloned or unpacked version
251of Poky.
252
253Depending on where the script is sourced, different sub-scripts are
254called to set up the Build Directory (Yocto or OpenEmbedded).
255Specifically, the script ``scripts/oe-setup-builddir`` inside the poky
256directory sets up the Build Directory and seeds the directory (if
257necessary) with configuration files appropriate for the Yocto Project
258development environment.
259
260.. note::
261
262 The
263 scripts/oe-setup-builddir
264 script uses the
265 ``$TEMPLATECONF``
266 variable to determine which sample configuration files to locate.
267
268The ``local.conf`` file provides many basic variables that define a
269build environment. Here is a list of a few. To see the default
270configurations in a ``local.conf`` file created by the build environment
271script, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600272:yocto_git:`local.conf.sample </poky/tree/meta-poky/conf/local.conf.sample>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500273in the ``meta-poky`` layer:
274
275- *Target Machine Selection:* Controlled by the
276 :term:`MACHINE` variable.
277
278- *Download Directory:* Controlled by the
279 :term:`DL_DIR` variable.
280
281- *Shared State Directory:* Controlled by the
282 :term:`SSTATE_DIR` variable.
283
284- *Build Output:* Controlled by the
285 :term:`TMPDIR` variable.
286
287- *Distribution Policy:* Controlled by the
288 :term:`DISTRO` variable.
289
290- *Packaging Format:* Controlled by the
291 :term:`PACKAGE_CLASSES`
292 variable.
293
294- *SDK Target Architecture:* Controlled by the
295 :term:`SDKMACHINE` variable.
296
297- *Extra Image Packages:* Controlled by the
298 :term:`EXTRA_IMAGE_FEATURES`
299 variable.
300
301.. note::
302
303 Configurations set in the
304 conf/local.conf
305 file can also be set in the
306 conf/site.conf
307 and
308 conf/auto.conf
309 configuration files.
310
311The ``bblayers.conf`` file tells BitBake what layers you want considered
312during the build. By default, the layers listed in this file include
313layers minimally needed by the build system. However, you must manually
314add any custom layers you have created. You can find more information on
315working with the ``bblayers.conf`` file in the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600316":ref:`dev-manual/common-tasks:enabling your layer`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500317section in the Yocto Project Development Tasks Manual.
318
319The files ``site.conf`` and ``auto.conf`` are not created by the
320environment initialization script. If you want the ``site.conf`` file,
321you need to create that yourself. The ``auto.conf`` file is typically
322created by an autobuilder:
323
324- *site.conf:* You can use the ``conf/site.conf`` configuration
325 file to configure multiple build directories. For example, suppose
326 you had several build environments and they shared some common
327 features. You can set these default build properties here. A good
328 example is perhaps the packaging format to use through the
329 :term:`PACKAGE_CLASSES`
330 variable.
331
332 One useful scenario for using the ``conf/site.conf`` file is to
333 extend your :term:`BBPATH` variable
334 to include the path to a ``conf/site.conf``. Then, when BitBake looks
335 for Metadata using ``BBPATH``, it finds the ``conf/site.conf`` file
336 and applies your common configurations found in the file. To override
337 configurations in a particular build directory, alter the similar
338 configurations within that build directory's ``conf/local.conf``
339 file.
340
341- *auto.conf:* The file is usually created and written to by an
342 autobuilder. The settings put into the file are typically the same as
343 you would find in the ``conf/local.conf`` or the ``conf/site.conf``
344 files.
345
346You can edit all configuration files to further define any particular
347build environment. This process is represented by the "User
348Configuration Edits" box in the figure.
349
350When you launch your build with the ``bitbake target`` command, BitBake
351sorts out the configurations to ultimately define your build
352environment. It is important to understand that the
353:term:`OpenEmbedded Build System` reads the
354configuration files in a specific order: ``site.conf``, ``auto.conf``,
355and ``local.conf``. And, the build system applies the normal assignment
356statement rules as described in the
357":doc:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata`" chapter
358of the BitBake User Manual. Because the files are parsed in a specific
359order, variable assignments for the same variable could be affected. For
360example, if the ``auto.conf`` file and the ``local.conf`` set variable1
361to different values, because the build system parses ``local.conf``
362after ``auto.conf``, variable1 is assigned the value from the
363``local.conf`` file.
364
365Metadata, Machine Configuration, and Policy Configuration
366---------------------------------------------------------
367
368The previous section described the user configurations that define
369BitBake's global behavior. This section takes a closer look at the
370layers the build system uses to further control the build. These layers
371provide Metadata for the software, machine, and policies.
372
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700373In general, there are three types of layer input. You can see them below
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500374the "User Configuration" box in the `general workflow
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500375figure <overview-manual/concepts:openembedded build system concepts>`:
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500376
377- *Metadata (.bb + Patches):* Software layers containing
378 user-supplied recipe files, patches, and append files. A good example
Andrew Geisslerd1e89492021-02-12 15:35:20 -0600379 of a software layer might be the :oe_layer:`meta-qt5 layer </meta-qt5>`
380 from the :oe_layerindex:`OpenEmbedded Layer Index <>`. This layer is for
381 version 5.0 of the popular `Qt <https://wiki.qt.io/About_Qt>`__
382 cross-platform application development framework for desktop, embedded and
383 mobile.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500384
385- *Machine BSP Configuration:* Board Support Package (BSP) layers (i.e.
386 "BSP Layer" in the following figure) providing machine-specific
387 configurations. This type of information is specific to a particular
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500388 target architecture. A good example of a BSP layer from the
389 :ref:`overview-manual/yp-intro:reference distribution (poky)` is the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600390 :yocto_git:`meta-yocto-bsp </poky/tree/meta-yocto-bsp>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500391 layer.
392
393- *Policy Configuration:* Distribution Layers (i.e. "Distro Layer" in
394 the following figure) providing top-level or general policies for the
395 images or SDKs being built for a particular distribution. For
396 example, in the Poky Reference Distribution the distro layer is the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600397 :yocto_git:`meta-poky </poky/tree/meta-poky>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500398 layer. Within the distro layer is a ``conf/distro`` directory that
399 contains distro configuration files (e.g.
Andrew Geissler09209ee2020-12-13 08:44:15 -0600400 :yocto_git:`poky.conf </poky/tree/meta-poky/conf/distro/poky.conf>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500401 that contain many policy configurations for the Poky distribution.
402
403The following figure shows an expanded representation of these three
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500404layers from the :ref:`general workflow figure
405<overview-manual/concepts:openembedded build system concepts>`:
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500406
407.. image:: figures/layer-input.png
408 :align: center
409
410In general, all layers have a similar structure. They all contain a
411licensing file (e.g. ``COPYING.MIT``) if the layer is to be distributed,
412a ``README`` file as good practice and especially if the layer is to be
413distributed, a configuration directory, and recipe directories. You can
414learn about the general structure for layers used with the Yocto Project
415in the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600416":ref:`dev-manual/common-tasks:creating your own layer`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500417section in the
418Yocto Project Development Tasks Manual. For a general discussion on
419layers and the many layers from which you can draw, see the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500420":ref:`overview-manual/concepts:layers`" and
421":ref:`overview-manual/yp-intro:the yocto project layer model`" sections both
422earlier in this manual.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500423
424If you explored the previous links, you discovered some areas where many
Andrew Geissler09209ee2020-12-13 08:44:15 -0600425layers that work with the Yocto Project exist. The :yocto_git:`Source
426Repositories <>` also shows layers categorized under "Yocto Metadata Layers."
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500427
428.. note::
429
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700430 There are layers in the Yocto Project Source Repositories that cannot be
431 found in the OpenEmbedded Layer Index. Such layers are either
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500432 deprecated or experimental in nature.
433
434BitBake uses the ``conf/bblayers.conf`` file, which is part of the user
435configuration, to find what layers it should be using as part of the
436build.
437
438Distro Layer
439~~~~~~~~~~~~
440
441The distribution layer provides policy configurations for your
442distribution. Best practices dictate that you isolate these types of
443configurations into their own layer. Settings you provide in
444``conf/distro/distro.conf`` override similar settings that BitBake finds
445in your ``conf/local.conf`` file in the Build Directory.
446
447The following list provides some explanation and references for what you
448typically find in the distribution layer:
449
450- *classes:* Class files (``.bbclass``) hold common functionality that
451 can be shared among recipes in the distribution. When your recipes
452 inherit a class, they take on the settings and functions for that
453 class. You can read more about class files in the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600454 ":ref:`ref-manual/classes:Classes`" chapter of the Yocto
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500455 Reference Manual.
456
457- *conf:* This area holds configuration files for the layer
458 (``conf/layer.conf``), the distribution
459 (``conf/distro/distro.conf``), and any distribution-wide include
460 files.
461
462- *recipes-*:* Recipes and append files that affect common
463 functionality across the distribution. This area could include
464 recipes and append files to add distribution-specific configuration,
465 initialization scripts, custom image recipes, and so forth. Examples
466 of ``recipes-*`` directories are ``recipes-core`` and
467 ``recipes-extra``. Hierarchy and contents within a ``recipes-*``
468 directory can vary. Generally, these directories contain recipe files
469 (``*.bb``), recipe append files (``*.bbappend``), directories that
470 are distro-specific for configuration files, and so forth.
471
472BSP Layer
473~~~~~~~~~
474
475The BSP Layer provides machine configurations that target specific
476hardware. Everything in this layer is specific to the machine for which
477you are building the image or the SDK. A common structure or form is
478defined for BSP layers. You can learn more about this structure in the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600479:doc:`/bsp-guide/index`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500480
481.. note::
482
483 In order for a BSP layer to be considered compliant with the Yocto
484 Project, it must meet some structural requirements.
485
486The BSP Layer's configuration directory contains configuration files for
487the machine (``conf/machine/machine.conf``) and, of course, the layer
488(``conf/layer.conf``).
489
490The remainder of the layer is dedicated to specific recipes by function:
491``recipes-bsp``, ``recipes-core``, ``recipes-graphics``,
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700492``recipes-kernel``, and so forth. There can be metadata for multiple
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500493formfactors, graphics support systems, and so forth.
494
495.. note::
496
497 While the figure shows several
498 recipes-\*
499 directories, not all these directories appear in all BSP layers.
500
501Software Layer
502~~~~~~~~~~~~~~
503
504The software layer provides the Metadata for additional software
505packages used during the build. This layer does not include Metadata
506that is specific to the distribution or the machine, which are found in
507their respective layers.
508
509This layer contains any recipes, append files, and patches, that your
510project needs.
511
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500512Sources
513-------
514
515In order for the OpenEmbedded build system to create an image or any
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500516target, it must be able to access source files. The :ref:`general workflow
517figure <overview-manual/concepts:openembedded build system concepts>`
518represents source files using the "Upstream Project Releases", "Local
519Projects", and "SCMs (optional)" boxes. The figure represents mirrors,
520which also play a role in locating source files, with the "Source
521Materials" box.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500522
523The method by which source files are ultimately organized is a function
524of the project. For example, for released software, projects tend to use
525tarballs or other archived files that can capture the state of a release
526guaranteeing that it is statically represented. On the other hand, for a
527project that is more dynamic or experimental in nature, a project might
528keep source files in a repository controlled by a Source Control Manager
529(SCM) such as Git. Pulling source from a repository allows you to
530control the point in the repository (the revision) from which you want
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700531to build software. A combination of the two is also possible.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500532
533BitBake uses the :term:`SRC_URI`
534variable to point to source files regardless of their location. Each
535recipe must have a ``SRC_URI`` variable that points to the source.
536
537Another area that plays a significant role in where source files come
538from is pointed to by the
539:term:`DL_DIR` variable. This area is
540a cache that can hold previously downloaded source. You can also
541instruct the OpenEmbedded build system to create tarballs from Git
542repositories, which is not the default behavior, and store them in the
543``DL_DIR`` by using the
544:term:`BB_GENERATE_MIRROR_TARBALLS`
545variable.
546
547Judicious use of a ``DL_DIR`` directory can save the build system a trip
548across the Internet when looking for files. A good method for using a
549download directory is to have ``DL_DIR`` point to an area outside of
550your Build Directory. Doing so allows you to safely delete the Build
551Directory if needed without fear of removing any downloaded source file.
552
553The remainder of this section provides a deeper look into the source
554files and the mirrors. Here is a more detailed look at the source file
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500555area of the :ref:`general workflow figure <overview-manual/concepts:openembedded build system concepts>`:
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500556
557.. image:: figures/source-input.png
558 :align: center
559
560Upstream Project Releases
561~~~~~~~~~~~~~~~~~~~~~~~~~
562
563Upstream project releases exist anywhere in the form of an archived file
564(e.g. tarball or zip file). These files correspond to individual
565recipes. For example, the figure uses specific releases each for
566BusyBox, Qt, and Dbus. An archive file can be for any released product
567that can be built using a recipe.
568
569Local Projects
570~~~~~~~~~~~~~~
571
572Local projects are custom bits of software the user provides. These bits
573reside somewhere local to a project - perhaps a directory into which the
574user checks in items (e.g. a local directory containing a development
575source tree used by the group).
576
577The canonical method through which to include a local project is to use
578the :ref:`externalsrc <ref-classes-externalsrc>`
579class to include that local project. You use either the ``local.conf``
580or a recipe's append file to override or set the recipe to point to the
581local directory on your disk to pull in the whole source tree.
582
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500583Source Control Managers (Optional)
584~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
585
586Another place from which the build system can get source files is with
Andrew Geissler09209ee2020-12-13 08:44:15 -0600587:ref:`fetchers <bitbake:bitbake-user-manual/bitbake-user-manual-fetching:fetchers>` employing various Source
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500588Control Managers (SCMs) such as Git or Subversion. In such cases, a
589repository is cloned or checked out. The
590:ref:`ref-tasks-fetch` task inside
591BitBake uses the :term:`SRC_URI`
592variable and the argument's prefix to determine the correct fetcher
593module.
594
595.. note::
596
597 For information on how to have the OpenEmbedded build system generate
598 tarballs for Git repositories and place them in the
599 DL_DIR
600 directory, see the :term:`BB_GENERATE_MIRROR_TARBALLS`
601 variable in the Yocto Project Reference Manual.
602
603When fetching a repository, BitBake uses the
604:term:`SRCREV` variable to determine
605the specific revision from which to build.
606
607Source Mirror(s)
608~~~~~~~~~~~~~~~~
609
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700610There are two kinds of mirrors: pre-mirrors and regular mirrors. The
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500611:term:`PREMIRRORS` and
612:term:`MIRRORS` variables point to
613these, respectively. BitBake checks pre-mirrors before looking upstream
614for any source files. Pre-mirrors are appropriate when you have a shared
615directory that is not a directory defined by the
616:term:`DL_DIR` variable. A Pre-mirror
617typically points to a shared directory that is local to your
618organization.
619
620Regular mirrors can be any site across the Internet that is used as an
621alternative location for source code should the primary site not be
622functioning for some reason or another.
623
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500624Package Feeds
625-------------
626
627When the OpenEmbedded build system generates an image or an SDK, it gets
628the packages from a package feed area located in the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500629:term:`Build Directory`. The :ref:`general workflow figure
630<overview-manual/concepts:openembedded build system concepts>`
631shows this package feeds area in the upper-right corner.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500632
633This section looks a little closer into the package feeds area used by
634the build system. Here is a more detailed look at the area:
635
636.. image:: figures/package-feeds.png
637 :align: center
638
639Package feeds are an intermediary step in the build process. The
640OpenEmbedded build system provides classes to generate different package
641types, and you specify which classes to enable through the
642:term:`PACKAGE_CLASSES`
643variable. Before placing the packages into package feeds, the build
644process validates them with generated output quality assurance checks
645through the :ref:`insane <ref-classes-insane>`
646class.
647
648The package feed area resides in the Build Directory. The directory the
649build system uses to temporarily store packages is determined by a
650combination of variables and the particular package manager in use. See
651the "Package Feeds" box in the illustration and note the information to
652the right of that area. In particular, the following defines where
653package files are kept:
654
655- :term:`DEPLOY_DIR`: Defined as
656 ``tmp/deploy`` in the Build Directory.
657
658- ``DEPLOY_DIR_*``: Depending on the package manager used, the package
659 type sub-folder. Given RPM, IPK, or DEB packaging and tarball
660 creation, the
661 :term:`DEPLOY_DIR_RPM`,
662 :term:`DEPLOY_DIR_IPK`,
663 :term:`DEPLOY_DIR_DEB`, or
664 :term:`DEPLOY_DIR_TAR`,
665 variables are used, respectively.
666
667- :term:`PACKAGE_ARCH`: Defines
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700668 architecture-specific sub-folders. For example, packages could be
669 available for the i586 or qemux86 architectures.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500670
671BitBake uses the
672:ref:`do_package_write_* <ref-tasks-package_write_deb>`
673tasks to generate packages and place them into the package holding area
674(e.g. ``do_package_write_ipk`` for IPK packages). See the
675":ref:`ref-tasks-package_write_deb`",
676":ref:`ref-tasks-package_write_ipk`",
677":ref:`ref-tasks-package_write_rpm`",
678and
679":ref:`ref-tasks-package_write_tar`"
680sections in the Yocto Project Reference Manual for additional
681information. As an example, consider a scenario where an IPK packaging
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700682manager is being used and there is package architecture support for both
683i586 and qemux86. Packages for the i586 architecture are placed in
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500684``build/tmp/deploy/ipk/i586``, while packages for the qemux86
685architecture are placed in ``build/tmp/deploy/ipk/qemux86``.
686
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500687BitBake Tool
688------------
689
690The OpenEmbedded build system uses
691:term:`BitBake` to produce images and
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500692Software Development Kits (SDKs). You can see from the :ref:`general workflow
693figure <overview-manual/concepts:openembedded build system concepts>`,
694the BitBake area consists of several functional areas. This section takes a
695closer look at each of those areas.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500696
697.. note::
698
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700699 Documentation for the BitBake tool is available separately. See the
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500700 BitBake User Manual
701 for reference material on BitBake.
702
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500703Source Fetching
704~~~~~~~~~~~~~~~
705
706The first stages of building a recipe are to fetch and unpack the source
707code:
708
709.. image:: figures/source-fetching.png
710 :align: center
711
712The :ref:`ref-tasks-fetch` and
713:ref:`ref-tasks-unpack` tasks fetch
714the source files and unpack them into the
715:term:`Build Directory`.
716
717.. note::
718
719 For every local file (e.g.
720 file://
721 ) that is part of a recipe's
722 SRC_URI
723 statement, the OpenEmbedded build system takes a checksum of the file
724 for the recipe and inserts the checksum into the signature for the
725 do_fetch
726 task. If any local file has been modified, the
727 do_fetch
728 task and all tasks that depend on it are re-executed.
729
730By default, everything is accomplished in the Build Directory, which has
731a defined structure. For additional general information on the Build
732Directory, see the ":ref:`structure-core-build`" section in
733the Yocto Project Reference Manual.
734
735Each recipe has an area in the Build Directory where the unpacked source
736code resides. The :term:`S` variable points
737to this area for a recipe's unpacked source code. The name of that
738directory for any given recipe is defined from several different
739variables. The preceding figure and the following list describe the
740Build Directory's hierarchy:
741
742- :term:`TMPDIR`: The base directory
743 where the OpenEmbedded build system performs all its work during the
744 build. The default base directory is the ``tmp`` directory.
745
746- :term:`PACKAGE_ARCH`: The
747 architecture of the built package or packages. Depending on the
748 eventual destination of the package or packages (i.e. machine
749 architecture, :term:`Build Host`, SDK, or
750 specific machine), ``PACKAGE_ARCH`` varies. See the variable's
751 description for details.
752
753- :term:`TARGET_OS`: The operating
754 system of the target device. A typical value would be "linux" (e.g.
755 "qemux86-poky-linux").
756
757- :term:`PN`: The name of the recipe used
758 to build the package. This variable can have multiple meanings.
759 However, when used in the context of input files, ``PN`` represents
760 the name of the recipe.
761
762- :term:`WORKDIR`: The location
763 where the OpenEmbedded build system builds a recipe (i.e. does the
764 work to create the package).
765
766 - :term:`PV`: The version of the
767 recipe used to build the package.
768
769 - :term:`PR`: The revision of the
770 recipe used to build the package.
771
772- :term:`S`: Contains the unpacked source
773 files for a given recipe.
774
775 - :term:`BPN`: The name of the recipe
776 used to build the package. The ``BPN`` variable is a version of
777 the ``PN`` variable but with common prefixes and suffixes removed.
778
779 - :term:`PV`: The version of the
780 recipe used to build the package.
781
782.. note::
783
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700784 In the previous figure, notice that there are two sample hierarchies:
785 one based on package architecture (i.e. :term:`PACKAGE_ARCH`)
786 and one based on a machine (i.e. :term:`MACHINE`).
787 The underlying structures are identical. The differentiator being
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500788 what the OpenEmbedded build system is using as a build target (e.g.
789 general architecture, a build host, an SDK, or a specific machine).
790
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500791Patching
792~~~~~~~~
793
794Once source code is fetched and unpacked, BitBake locates patch files
795and applies them to the source files:
796
797.. image:: figures/patching.png
798 :align: center
799
800The :ref:`ref-tasks-patch` task uses a
801recipe's :term:`SRC_URI` statements
802and the :term:`FILESPATH` variable
803to locate applicable patch files.
804
805Default processing for patch files assumes the files have either
806``*.patch`` or ``*.diff`` file types. You can use ``SRC_URI`` parameters
807to change the way the build system recognizes patch files. See the
808:ref:`ref-tasks-patch` task for more
809information.
810
811BitBake finds and applies multiple patches for a single recipe in the
812order in which it locates the patches. The ``FILESPATH`` variable
813defines the default set of directories that the build system uses to
814search for patch files. Once found, patches are applied to the recipe's
815source files, which are located in the
816:term:`S` directory.
817
818For more information on how the source directories are created, see the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500819":ref:`overview-manual/concepts:source fetching`" section. For
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500820more information on how to create patches and how the build system
821processes patches, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600822":ref:`dev-manual/common-tasks:patching code`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500823section in the
824Yocto Project Development Tasks Manual. You can also see the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600825":ref:`sdk-manual/extensible:use \`\`devtool modify\`\` to modify the source of an existing component`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500826section in the Yocto Project Application Development and the Extensible
827Software Development Kit (SDK) manual and the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600828":ref:`kernel-dev/common:using traditional kernel development to patch the kernel`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500829section in the Yocto Project Linux Kernel Development Manual.
830
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500831Configuration, Compilation, and Staging
832~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
833
834After source code is patched, BitBake executes tasks that configure and
835compile the source code. Once compilation occurs, the files are copied
836to a holding area (staged) in preparation for packaging:
837
838.. image:: figures/configuration-compile-autoreconf.png
839 :align: center
840
841This step in the build process consists of the following tasks:
842
843- :ref:`ref-tasks-prepare_recipe_sysroot`:
844 This task sets up the two sysroots in
845 ``${``\ :term:`WORKDIR`\ ``}``
846 (i.e. ``recipe-sysroot`` and ``recipe-sysroot-native``) so that
847 during the packaging phase the sysroots can contain the contents of
848 the
849 :ref:`ref-tasks-populate_sysroot`
850 tasks of the recipes on which the recipe containing the tasks
851 depends. A sysroot exists for both the target and for the native
852 binaries, which run on the host system.
853
854- *do_configure*: This task configures the source by enabling and
855 disabling any build-time and configuration options for the software
856 being built. Configurations can come from the recipe itself as well
857 as from an inherited class. Additionally, the software itself might
858 configure itself depending on the target for which it is being built.
859
860 The configurations handled by the
861 :ref:`ref-tasks-configure` task
862 are specific to configurations for the source code being built by the
863 recipe.
864
865 If you are using the
866 :ref:`autotools <ref-classes-autotools>` class,
867 you can add additional configuration options by using the
868 :term:`EXTRA_OECONF` or
869 :term:`PACKAGECONFIG_CONFARGS`
870 variables. For information on how this variable works within that
871 class, see the
872 :ref:`autotools <ref-classes-autotools>` class
Andrew Geissler09209ee2020-12-13 08:44:15 -0600873 :yocto_git:`here </poky/tree/meta/classes/autotools.bbclass>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500874
875- *do_compile*: Once a configuration task has been satisfied,
876 BitBake compiles the source using the
877 :ref:`ref-tasks-compile` task.
878 Compilation occurs in the directory pointed to by the
879 :term:`B` variable. Realize that the
880 ``B`` directory is, by default, the same as the
881 :term:`S` directory.
882
883- *do_install*: After compilation completes, BitBake executes the
884 :ref:`ref-tasks-install` task.
885 This task copies files from the ``B`` directory and places them in a
886 holding area pointed to by the :term:`D`
887 variable. Packaging occurs later using files from this holding
888 directory.
889
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500890Package Splitting
891~~~~~~~~~~~~~~~~~
892
893After source code is configured, compiled, and staged, the build system
894analyzes the results and splits the output into packages:
895
896.. image:: figures/analysis-for-package-splitting.png
897 :align: center
898
899The :ref:`ref-tasks-package` and
900:ref:`ref-tasks-packagedata`
901tasks combine to analyze the files found in the
902:term:`D` directory and split them into
903subsets based on available packages and files. Analysis involves the
904following as well as other items: splitting out debugging symbols,
905looking at shared library dependencies between packages, and looking at
906package relationships.
907
908The ``do_packagedata`` task creates package metadata based on the
909analysis such that the build system can generate the final packages. The
910:ref:`ref-tasks-populate_sysroot`
911task stages (copies) a subset of the files installed by the
912:ref:`ref-tasks-install` task into
913the appropriate sysroot. Working, staged, and intermediate results of
914the analysis and package splitting process use several areas:
915
916- :term:`PKGD`: The destination
917 directory (i.e. ``package``) for packages before they are split into
918 individual packages.
919
920- :term:`PKGDESTWORK`: A
921 temporary work area (i.e. ``pkgdata``) used by the ``do_package``
922 task to save package metadata.
923
924- :term:`PKGDEST`: The parent
925 directory (i.e. ``packages-split``) for packages after they have been
926 split.
927
928- :term:`PKGDATA_DIR`: A shared,
929 global-state directory that holds packaging metadata generated during
930 the packaging process. The packaging process copies metadata from
931 ``PKGDESTWORK`` to the ``PKGDATA_DIR`` area where it becomes globally
932 available.
933
934- :term:`STAGING_DIR_HOST`:
935 The path for the sysroot for the system on which a component is built
936 to run (i.e. ``recipe-sysroot``).
937
938- :term:`STAGING_DIR_NATIVE`:
939 The path for the sysroot used when building components for the build
940 host (i.e. ``recipe-sysroot-native``).
941
942- :term:`STAGING_DIR_TARGET`:
943 The path for the sysroot used when a component that is built to
944 execute on a system and it generates code for yet another machine
945 (e.g. cross-canadian recipes).
946
947The :term:`FILES` variable defines the
948files that go into each package in
949:term:`PACKAGES`. If you want
950details on how this is accomplished, you can look at
Andrew Geissler09209ee2020-12-13 08:44:15 -0600951:yocto_git:`package.bbclass </poky/tree/meta/classes/package.bbclass>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500952
953Depending on the type of packages being created (RPM, DEB, or IPK), the
954:ref:`do_package_write_* <ref-tasks-package_write_deb>`
955task creates the actual packages and places them in the Package Feed
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500956area, which is ``${TMPDIR}/deploy``. You can see the
957":ref:`overview-manual/concepts:package feeds`" section for more detail on
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500958that part of the build process.
959
960.. note::
961
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700962 Support for creating feeds directly from the ``deploy/*``
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500963 directories does not exist. Creating such feeds usually requires some
964 kind of feed maintenance mechanism that would upload the new packages
965 into an official package feed (e.g. the Ångström distribution). This
966 functionality is highly distribution-specific and thus is not
967 provided out of the box.
968
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500969Image Generation
970~~~~~~~~~~~~~~~~
971
972Once packages are split and stored in the Package Feeds area, the build
973system uses BitBake to generate the root filesystem image:
974
975.. image:: figures/image-generation.png
976 :align: center
977
978The image generation process consists of several stages and depends on
979several tasks and variables. The
980:ref:`ref-tasks-rootfs` task creates
981the root filesystem (file and directory structure) for an image. This
982task uses several key variables to help create the list of packages to
983actually install:
984
985- :term:`IMAGE_INSTALL`: Lists
986 out the base set of packages from which to install from the Package
987 Feeds area.
988
989- :term:`PACKAGE_EXCLUDE`:
990 Specifies packages that should not be installed into the image.
991
992- :term:`IMAGE_FEATURES`:
993 Specifies features to include in the image. Most of these features
994 map to additional packages for installation.
995
996- :term:`PACKAGE_CLASSES`:
997 Specifies the package backend (e.g. RPM, DEB, or IPK) to use and
998 consequently helps determine where to locate packages within the
999 Package Feeds area.
1000
1001- :term:`IMAGE_LINGUAS`:
1002 Determines the language(s) for which additional language support
1003 packages are installed.
1004
1005- :term:`PACKAGE_INSTALL`:
1006 The final list of packages passed to the package manager for
1007 installation into the image.
1008
1009With :term:`IMAGE_ROOTFS`
1010pointing to the location of the filesystem under construction and the
1011``PACKAGE_INSTALL`` variable providing the final list of packages to
1012install, the root file system is created.
1013
1014Package installation is under control of the package manager (e.g.
1015dnf/rpm, opkg, or apt/dpkg) regardless of whether or not package
1016management is enabled for the target. At the end of the process, if
1017package management is not enabled for the target, the package manager's
1018data files are deleted from the root filesystem. As part of the final
1019stage of package installation, post installation scripts that are part
1020of the packages are run. Any scripts that fail to run on the build host
1021are run on the target when the target system is first booted. If you are
1022using a
Andrew Geissler09209ee2020-12-13 08:44:15 -06001023:ref:`read-only root filesystem <dev-manual/common-tasks:creating a read-only root filesystem>`,
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001024all the post installation scripts must succeed on the build host during
1025the package installation phase since the root filesystem on the target
1026is read-only.
1027
1028The final stages of the ``do_rootfs`` task handle post processing. Post
1029processing includes creation of a manifest file and optimizations.
1030
1031The manifest file (``.manifest``) resides in the same directory as the
1032root filesystem image. This file lists out, line-by-line, the installed
1033packages. The manifest file is useful for the
1034:ref:`testimage <ref-classes-testimage*>` class,
1035for example, to determine whether or not to run specific tests. See the
1036:term:`IMAGE_MANIFEST`
1037variable for additional information.
1038
1039Optimizing processes that are run across the image include ``mklibs``,
1040``prelink``, and any other post-processing commands as defined by the
1041:term:`ROOTFS_POSTPROCESS_COMMAND`
1042variable. The ``mklibs`` process optimizes the size of the libraries,
1043while the ``prelink`` process optimizes the dynamic linking of shared
1044libraries to reduce start up time of executables.
1045
1046After the root filesystem is built, processing begins on the image
1047through the :ref:`ref-tasks-image`
1048task. The build system runs any pre-processing commands as defined by
1049the
1050:term:`IMAGE_PREPROCESS_COMMAND`
1051variable. This variable specifies a list of functions to call before the
1052build system creates the final image output files.
1053
1054The build system dynamically creates ``do_image_*`` tasks as needed,
1055based on the image types specified in the
1056:term:`IMAGE_FSTYPES` variable.
1057The process turns everything into an image file or a set of image files
1058and can compress the root filesystem image to reduce the overall size of
1059the image. The formats used for the root filesystem depend on the
1060``IMAGE_FSTYPES`` variable. Compression depends on whether the formats
1061support compression.
1062
1063As an example, a dynamically created task when creating a particular
Andrew Geisslerc926e172021-05-07 16:11:35 -05001064image type would take the following form::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001065
1066 do_image_type
1067
1068So, if the type
1069as specified by the ``IMAGE_FSTYPES`` were ``ext4``, the dynamically
Andrew Geisslerc926e172021-05-07 16:11:35 -05001070generated task would be as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001071
1072 do_image_ext4
1073
1074The final task involved in image creation is the
1075:ref:`do_image_complete <ref-tasks-image-complete>`
1076task. This task completes the image by applying any image post
1077processing as defined through the
1078:term:`IMAGE_POSTPROCESS_COMMAND`
1079variable. The variable specifies a list of functions to call once the
1080build system has created the final image output files.
1081
1082.. note::
1083
1084 The entire image generation process is run under
1085 Pseudo. Running under Pseudo ensures that the files in the root filesystem
1086 have correct ownership.
1087
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001088SDK Generation
1089~~~~~~~~~~~~~~
1090
1091The OpenEmbedded build system uses BitBake to generate the Software
1092Development Kit (SDK) installer scripts for both the standard SDK and
1093the extensible SDK (eSDK):
1094
1095.. image:: figures/sdk-generation.png
1096 :align: center
1097
1098.. note::
1099
1100 For more information on the cross-development toolchain generation,
Andrew Geissler09209ee2020-12-13 08:44:15 -06001101 see the ":ref:`overview-manual/concepts:cross-development toolchain generation`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001102 section. For information on advantages gained when building a
1103 cross-development toolchain using the do_populate_sdk task, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001104 ":ref:`sdk-manual/appendix-obtain:building an sdk installer`" section in
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001105 the Yocto Project Application Development and the Extensible Software
1106 Development Kit (eSDK) manual.
1107
1108Like image generation, the SDK script process consists of several stages
1109and depends on many variables. The
1110:ref:`ref-tasks-populate_sdk`
1111and
1112:ref:`ref-tasks-populate_sdk_ext`
1113tasks use these key variables to help create the list of packages to
1114actually install. For information on the variables listed in the figure,
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001115see the ":ref:`overview-manual/concepts:application development sdk`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001116section.
1117
1118The ``do_populate_sdk`` task helps create the standard SDK and handles
1119two parts: a target part and a host part. The target part is the part
1120built for the target hardware and includes libraries and headers. The
1121host part is the part of the SDK that runs on the
1122:term:`SDKMACHINE`.
1123
1124The ``do_populate_sdk_ext`` task helps create the extensible SDK and
1125handles host and target parts differently than its counter part does for
1126the standard SDK. For the extensible SDK, the task encapsulates the
1127build system, which includes everything needed (host and target) for the
1128SDK.
1129
1130Regardless of the type of SDK being constructed, the tasks perform some
1131cleanup after which a cross-development environment setup script and any
1132needed configuration files are created. The final output is the
1133Cross-development toolchain installation script (``.sh`` file), which
1134includes the environment setup script.
1135
1136Stamp Files and the Rerunning of Tasks
1137~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1138
1139For each task that completes successfully, BitBake writes a stamp file
1140into the :term:`STAMPS_DIR`
1141directory. The beginning of the stamp file's filename is determined by
1142the :term:`STAMP` variable, and the end
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001143of the name consists of the task's name and current :ref:`input
1144checksum <overview-manual/concepts:checksums (signatures)>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001145
1146.. note::
1147
1148 This naming scheme assumes that
1149 BB_SIGNATURE_HANDLER
1150 is "OEBasicHash", which is almost always the case in current
1151 OpenEmbedded.
1152
1153To determine if a task needs to be rerun, BitBake checks if a stamp file
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001154with a matching input checksum exists for the task. In this case,
1155the task's output is assumed to exist and still be valid. Otherwise,
1156the task is rerun.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001157
1158.. note::
1159
1160 The stamp mechanism is more general than the shared state (sstate)
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001161 cache mechanism described in the
1162 ":ref:`overview-manual/concepts:setscene tasks and shared state`" section.
1163 BitBake avoids rerunning any task that has a valid stamp file, not just
1164 tasks that can be accelerated through the sstate cache.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001165
1166 However, you should realize that stamp files only serve as a marker
1167 that some work has been done and that these files do not record task
1168 output. The actual task output would usually be somewhere in
1169 :term:`TMPDIR` (e.g. in some
1170 recipe's :term:`WORKDIR`.) What
1171 the sstate cache mechanism adds is a way to cache task output that
1172 can then be shared between build machines.
1173
1174Since ``STAMPS_DIR`` is usually a subdirectory of ``TMPDIR``, removing
1175``TMPDIR`` will also remove ``STAMPS_DIR``, which means tasks will
1176properly be rerun to repopulate ``TMPDIR``.
1177
1178If you want some task to always be considered "out of date", you can
1179mark it with the :ref:`nostamp <bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`
1180varflag. If some other task depends on such a task, then that task will
1181also always be considered out of date, which might not be what you want.
1182
1183For details on how to view information about a task's signature, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001184":ref:`dev-manual/common-tasks:viewing task variable dependencies`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001185section in the Yocto Project Development Tasks Manual.
1186
1187Setscene Tasks and Shared State
1188~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1189
1190The description of tasks so far assumes that BitBake needs to build
1191everything and no available prebuilt objects exist. BitBake does support
1192skipping tasks if prebuilt objects are available. These objects are
1193usually made available in the form of a shared state (sstate) cache.
1194
1195.. note::
1196
1197 For information on variables affecting sstate, see the
1198 :term:`SSTATE_DIR`
1199 and
1200 :term:`SSTATE_MIRRORS`
1201 variables.
1202
1203The idea of a setscene task (i.e ``do_``\ taskname\ ``_setscene``) is a
1204version of the task where instead of building something, BitBake can
1205skip to the end result and simply place a set of files into specific
1206locations as needed. In some cases, it makes sense to have a setscene
1207task variant (e.g. generating package files in the
1208:ref:`do_package_write_* <ref-tasks-package_write_deb>`
1209task). In other cases, it does not make sense (e.g. a
1210:ref:`ref-tasks-patch` task or a
1211:ref:`ref-tasks-unpack` task) since
1212the work involved would be equal to or greater than the underlying task.
1213
1214In the build system, the common tasks that have setscene variants are
1215:ref:`ref-tasks-package`,
1216``do_package_write_*``,
1217:ref:`ref-tasks-deploy`,
1218:ref:`ref-tasks-packagedata`, and
1219:ref:`ref-tasks-populate_sysroot`.
1220Notice that these tasks represent most of the tasks whose output is an
1221end result.
1222
1223The build system has knowledge of the relationship between these tasks
1224and other preceding tasks. For example, if BitBake runs
1225``do_populate_sysroot_setscene`` for something, it does not make sense
1226to run any of the ``do_fetch``, ``do_unpack``, ``do_patch``,
1227``do_configure``, ``do_compile``, and ``do_install`` tasks. However, if
1228``do_package`` needs to be run, BitBake needs to run those other tasks.
1229
1230It becomes more complicated if everything can come from an sstate cache
1231because some objects are simply not required at all. For example, you do
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001232not need a compiler or native tools, such as quilt, if there isn't anything
1233to compile or patch. If the ``do_package_write_*`` packages are available
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001234from sstate, BitBake does not need the ``do_package`` task data.
1235
1236To handle all these complexities, BitBake runs in two phases. The first
1237is the "setscene" stage. During this stage, BitBake first checks the
1238sstate cache for any targets it is planning to build. BitBake does a
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001239fast check to see if the object exists rather than doing a complete download.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001240If nothing exists, the second phase, which is the setscene stage,
1241completes and the main build proceeds.
1242
1243If objects are found in the sstate cache, the build system works
1244backwards from the end targets specified by the user. For example, if an
1245image is being built, the build system first looks for the packages
1246needed for that image and the tools needed to construct an image. If
1247those are available, the compiler is not needed. Thus, the compiler is
1248not even downloaded. If something was found to be unavailable, or the
1249download or setscene task fails, the build system then tries to install
1250dependencies, such as the compiler, from the cache.
1251
1252The availability of objects in the sstate cache is handled by the
1253function specified by the
1254:term:`bitbake:BB_HASHCHECK_FUNCTION`
1255variable and returns a list of available objects. The function specified
1256by the
1257:term:`bitbake:BB_SETSCENE_DEPVALID`
1258variable is the function that determines whether a given dependency
1259needs to be followed, and whether for any given relationship the
1260function needs to be passed. The function returns a True or False value.
1261
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001262Images
1263------
1264
1265The images produced by the build system are compressed forms of the root
1266filesystem and are ready to boot on a target device. You can see from
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001267the :ref:`general workflow figure
1268<overview-manual/concepts:openembedded build system concepts>` that BitBake
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001269output, in part, consists of images. This section takes a closer look at
1270this output:
1271
1272.. image:: figures/images.png
1273 :align: center
1274
1275.. note::
1276
1277 For a list of example images that the Yocto Project provides, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001278 ":doc:`/ref-manual/images`" chapter in the Yocto Project Reference
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001279 Manual.
1280
1281The build process writes images out to the :term:`Build Directory`
1282inside the
1283``tmp/deploy/images/machine/`` folder as shown in the figure. This
1284folder contains any files expected to be loaded on the target device.
1285The :term:`DEPLOY_DIR` variable
1286points to the ``deploy`` directory, while the
1287:term:`DEPLOY_DIR_IMAGE`
1288variable points to the appropriate directory containing images for the
1289current configuration.
1290
1291- kernel-image: A kernel binary file. The
1292 :term:`KERNEL_IMAGETYPE`
1293 variable determines the naming scheme for the kernel image file.
1294 Depending on this variable, the file could begin with a variety of
1295 naming strings. The ``deploy/images/``\ machine directory can contain
1296 multiple image files for the machine.
1297
1298- root-filesystem-image: Root filesystems for the target device (e.g.
1299 ``*.ext3`` or ``*.bz2`` files). The
1300 :term:`IMAGE_FSTYPES`
1301 variable determines the root filesystem image type. The
1302 ``deploy/images/``\ machine directory can contain multiple root
1303 filesystems for the machine.
1304
1305- kernel-modules: Tarballs that contain all the modules built for the
1306 kernel. Kernel module tarballs exist for legacy purposes and can be
1307 suppressed by setting the
1308 :term:`MODULE_TARBALL_DEPLOY`
1309 variable to "0". The ``deploy/images/``\ machine directory can
1310 contain multiple kernel module tarballs for the machine.
1311
1312- bootloaders: If applicable to the target machine, bootloaders
1313 supporting the image. The ``deploy/images/``\ machine directory can
1314 contain multiple bootloaders for the machine.
1315
1316- symlinks: The ``deploy/images/``\ machine folder contains a symbolic
1317 link that points to the most recently built file for each machine.
1318 These links might be useful for external scripts that need to obtain
1319 the latest version of each file.
1320
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001321Application Development SDK
1322---------------------------
1323
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001324In the :ref:`general workflow figure
1325<overview-manual/concepts:openembedded build system concepts>`, the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001326output labeled "Application Development SDK" represents an SDK. The SDK
1327generation process differs depending on whether you build an extensible
1328SDK (e.g. ``bitbake -c populate_sdk_ext`` imagename) or a standard SDK
1329(e.g. ``bitbake -c populate_sdk`` imagename). This section takes a
1330closer look at this output:
1331
1332.. image:: figures/sdk.png
1333 :align: center
1334
1335The specific form of this output is a set of files that includes a
1336self-extracting SDK installer (``*.sh``), host and target manifest
1337files, and files used for SDK testing. When the SDK installer file is
1338run, it installs the SDK. The SDK consists of a cross-development
1339toolchain, a set of libraries and headers, and an SDK environment setup
1340script. Running this installer essentially sets up your
1341cross-development environment. You can think of the cross-toolchain as
1342the "host" part because it runs on the SDK machine. You can think of the
1343libraries and headers as the "target" part because they are built for
1344the target hardware. The environment setup script is added so that you
1345can initialize the environment before using the tools.
1346
1347.. note::
1348
1349 - The Yocto Project supports several methods by which you can set up
1350 this cross-development environment. These methods include
1351 downloading pre-built SDK installers or building and installing
1352 your own SDK installer.
1353
1354 - For background information on cross-development toolchains in the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001355 Yocto Project development environment, see the
1356 ":ref:`overview-manual/concepts:cross-development toolchain generation`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001357 section.
1358
1359 - For information on setting up a cross-development environment, see
Andrew Geissler09209ee2020-12-13 08:44:15 -06001360 the :doc:`/sdk-manual/index` manual.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001361
1362All the output files for an SDK are written to the ``deploy/sdk`` folder
1363inside the :term:`Build Directory` as
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001364shown in the previous figure. Depending on the type of SDK, there are
1365several variables to configure these files. Here are the variables
1366associated with an extensible SDK:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001367
1368- :term:`DEPLOY_DIR`: Points to
1369 the ``deploy`` directory.
1370
1371- :term:`SDK_EXT_TYPE`:
1372 Controls whether or not shared state artifacts are copied into the
1373 extensible SDK. By default, all required shared state artifacts are
1374 copied into the SDK.
1375
1376- :term:`SDK_INCLUDE_PKGDATA`:
1377 Specifies whether or not packagedata is included in the extensible
1378 SDK for all recipes in the "world" target.
1379
1380- :term:`SDK_INCLUDE_TOOLCHAIN`:
1381 Specifies whether or not the toolchain is included when building the
1382 extensible SDK.
1383
1384- :term:`SDK_LOCAL_CONF_WHITELIST`:
1385 A list of variables allowed through from the build system
1386 configuration into the extensible SDK configuration.
1387
1388- :term:`SDK_LOCAL_CONF_BLACKLIST`:
1389 A list of variables not allowed through from the build system
1390 configuration into the extensible SDK configuration.
1391
1392- :term:`SDK_INHERIT_BLACKLIST`:
1393 A list of classes to remove from the
1394 :term:`INHERIT` value globally
1395 within the extensible SDK configuration.
1396
1397This next list, shows the variables associated with a standard SDK:
1398
1399- :term:`DEPLOY_DIR`: Points to
1400 the ``deploy`` directory.
1401
1402- :term:`SDKMACHINE`: Specifies
1403 the architecture of the machine on which the cross-development tools
1404 are run to create packages for the target hardware.
1405
1406- :term:`SDKIMAGE_FEATURES`:
1407 Lists the features to include in the "target" part of the SDK.
1408
1409- :term:`TOOLCHAIN_HOST_TASK`:
1410 Lists packages that make up the host part of the SDK (i.e. the part
1411 that runs on the ``SDKMACHINE``). When you use
1412 ``bitbake -c populate_sdk imagename`` to create the SDK, a set of
1413 default packages apply. This variable allows you to add more
1414 packages.
1415
1416- :term:`TOOLCHAIN_TARGET_TASK`:
1417 Lists packages that make up the target part of the SDK (i.e. the part
1418 built for the target hardware).
1419
1420- :term:`SDKPATH`: Defines the
1421 default SDK installation path offered by the installation script.
1422
1423- :term:`SDK_HOST_MANIFEST`:
1424 Lists all the installed packages that make up the host part of the
1425 SDK. This variable also plays a minor role for extensible SDK
1426 development as well. However, it is mainly used for the standard SDK.
1427
1428- :term:`SDK_TARGET_MANIFEST`:
1429 Lists all the installed packages that make up the target part of the
1430 SDK. This variable also plays a minor role for extensible SDK
1431 development as well. However, it is mainly used for the standard SDK.
1432
1433Cross-Development Toolchain Generation
1434======================================
1435
1436The Yocto Project does most of the work for you when it comes to
Andrew Geissler09209ee2020-12-13 08:44:15 -06001437creating :ref:`sdk-manual/intro:the cross-development toolchain`. This
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001438section provides some technical background on how cross-development
1439toolchains are created and used. For more information on toolchains, you
Andrew Geissler09209ee2020-12-13 08:44:15 -06001440can also see the :doc:`/sdk-manual/index` manual.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001441
1442In the Yocto Project development environment, cross-development
1443toolchains are used to build images and applications that run on the
1444target hardware. With just a few commands, the OpenEmbedded build system
1445creates these necessary toolchains for you.
1446
1447The following figure shows a high-level build environment regarding
1448toolchain construction and use.
1449
1450.. image:: figures/cross-development-toolchains.png
1451 :align: center
1452
1453Most of the work occurs on the Build Host. This is the machine used to
1454build images and generally work within the the Yocto Project
1455environment. When you run
1456:term:`BitBake` to create an image, the
1457OpenEmbedded build system uses the host ``gcc`` compiler to bootstrap a
1458cross-compiler named ``gcc-cross``. The ``gcc-cross`` compiler is what
1459BitBake uses to compile source files when creating the target image. You
1460can think of ``gcc-cross`` simply as an automatically generated
1461cross-compiler that is used internally within BitBake only.
1462
1463.. note::
1464
1465 The extensible SDK does not use
1466 gcc-cross-canadian
1467 since this SDK ships a copy of the OpenEmbedded build system and the
1468 sysroot within it contains
1469 gcc-cross
1470 .
1471
Andrew Geisslerc926e172021-05-07 16:11:35 -05001472The chain of events that occurs when the standard toolchain is bootstrapped::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001473
Andrew Geissler6ce62a22020-11-30 19:58:47 -06001474 binutils-cross -> linux-libc-headers -> gcc-cross -> libgcc-initial -> glibc -> libgcc -> gcc-runtime
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001475
Andrew Geissler6ce62a22020-11-30 19:58:47 -06001476- ``gcc``: The compiler, GNU Compiler Collection (GCC).
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001477
Andrew Geissler6ce62a22020-11-30 19:58:47 -06001478- ``binutils-cross``: The binary utilities needed in order
1479 to run the ``gcc-cross`` phase of the bootstrap operation and build the
1480 headers for the C library.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001481
Andrew Geissler6ce62a22020-11-30 19:58:47 -06001482- ``linux-libc-headers``: Headers needed for the cross-compiler and C library build.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001483
Andrew Geissler6ce62a22020-11-30 19:58:47 -06001484- ``libgcc-initial``: An initial version of the gcc support library needed
1485 to bootstrap ``glibc``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001486
Andrew Geissler6ce62a22020-11-30 19:58:47 -06001487- ``libgcc``: The final version of the gcc support library which
1488 can only be built once there is a C library to link against.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001489
1490- ``glibc``: The GNU C Library.
1491
1492- ``gcc-cross``: The final stage of the bootstrap process for the
1493 cross-compiler. This stage results in the actual cross-compiler that
1494 BitBake uses when it builds an image for a targeted device.
1495
Andrew Geissler6ce62a22020-11-30 19:58:47 -06001496 This tool is a "native" tool (i.e. it is designed to run on
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001497 the build host).
1498
1499- ``gcc-runtime``: Runtime libraries resulting from the toolchain
1500 bootstrapping process. This tool produces a binary that consists of
1501 the runtime libraries need for the targeted device.
1502
1503You can use the OpenEmbedded build system to build an installer for the
1504relocatable SDK used to develop applications. When you run the
1505installer, it installs the toolchain, which contains the development
1506tools (e.g., ``gcc-cross-canadian``, ``binutils-cross-canadian``, and
1507other ``nativesdk-*`` tools), which are tools native to the SDK (i.e.
1508native to :term:`SDK_ARCH`), you
1509need to cross-compile and test your software. The figure shows the
1510commands you use to easily build out this toolchain. This
1511cross-development toolchain is built to execute on the
1512:term:`SDKMACHINE`, which might or
1513might not be the same machine as the Build Host.
1514
1515.. note::
1516
1517 If your target architecture is supported by the Yocto Project, you
1518 can take advantage of pre-built images that ship with the Yocto
1519 Project and already contain cross-development toolchain installers.
1520
Andrew Geisslerc926e172021-05-07 16:11:35 -05001521Here is the bootstrap process for the relocatable toolchain::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001522
1523 gcc -> binutils-crosssdk -> gcc-crosssdk-initial -> linux-libc-headers -> glibc-initial -> nativesdk-glibc -> gcc-crosssdk -> gcc-cross-canadian
1524
1525- ``gcc``: The build host's GNU Compiler Collection (GCC).
1526
1527- ``binutils-crosssdk``: The bare minimum binary utilities needed in
1528 order to run the ``gcc-crosssdk-initial`` phase of the bootstrap
1529 operation.
1530
1531- ``gcc-crosssdk-initial``: An early stage of the bootstrap process for
1532 creating the cross-compiler. This stage builds enough of the
1533 ``gcc-crosssdk`` and supporting pieces so that the final stage of the
1534 bootstrap process can produce the finished cross-compiler. This tool
1535 is a "native" binary that runs on the build host.
1536
1537- ``linux-libc-headers``: Headers needed for the cross-compiler.
1538
1539- ``glibc-initial``: An initial version of the Embedded GLIBC needed to
1540 bootstrap ``nativesdk-glibc``.
1541
1542- ``nativesdk-glibc``: The Embedded GLIBC needed to bootstrap the
1543 ``gcc-crosssdk``.
1544
1545- ``gcc-crosssdk``: The final stage of the bootstrap process for the
1546 relocatable cross-compiler. The ``gcc-crosssdk`` is a transitory
1547 compiler and never leaves the build host. Its purpose is to help in
1548 the bootstrap process to create the eventual ``gcc-cross-canadian``
1549 compiler, which is relocatable. This tool is also a "native" package
1550 (i.e. it is designed to run on the build host).
1551
1552- ``gcc-cross-canadian``: The final relocatable cross-compiler. When
1553 run on the :term:`SDKMACHINE`,
1554 this tool produces executable code that runs on the target device.
1555 Only one cross-canadian compiler is produced per architecture since
1556 they can be targeted at different processor optimizations using
1557 configurations passed to the compiler through the compile commands.
1558 This circumvents the need for multiple compilers and thus reduces the
1559 size of the toolchains.
1560
1561.. note::
1562
1563 For information on advantages gained when building a
1564 cross-development toolchain installer, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001565 ":ref:`sdk-manual/appendix-obtain:building an sdk installer`" appendix
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001566 in the Yocto Project Application Development and the
1567 Extensible Software Development Kit (eSDK) manual.
1568
1569Shared State Cache
1570==================
1571
1572By design, the OpenEmbedded build system builds everything from scratch
1573unless :term:`BitBake` can determine
1574that parts do not need to be rebuilt. Fundamentally, building from
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001575scratch is attractive as it means all parts are built fresh and there is
1576no possibility of stale data that can cause problems. When
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001577developers hit problems, they typically default back to building from
1578scratch so they have a know state from the start.
1579
1580Building an image from scratch is both an advantage and a disadvantage
1581to the process. As mentioned in the previous paragraph, building from
1582scratch ensures that everything is current and starts from a known
1583state. However, building from scratch also takes much longer as it
1584generally means rebuilding things that do not necessarily need to be
1585rebuilt.
1586
1587The Yocto Project implements shared state code that supports incremental
1588builds. The implementation of the shared state code answers the
1589following questions that were fundamental roadblocks within the
1590OpenEmbedded incremental build support system:
1591
1592- What pieces of the system have changed and what pieces have not
1593 changed?
1594
1595- How are changed pieces of software removed and replaced?
1596
1597- How are pre-built components that do not need to be rebuilt from
1598 scratch used when they are available?
1599
1600For the first question, the build system detects changes in the "inputs"
1601to a given task by creating a checksum (or signature) of the task's
1602inputs. If the checksum changes, the system assumes the inputs have
1603changed and the task needs to be rerun. For the second question, the
1604shared state (sstate) code tracks which tasks add which output to the
1605build process. This means the output from a given task can be removed,
1606upgraded or otherwise manipulated. The third question is partly
1607addressed by the solution for the second question assuming the build
1608system can fetch the sstate objects from remote locations and install
1609them if they are deemed to be valid.
1610
1611.. note::
1612
1613 - The build system does not maintain
1614 :term:`PR` information as part of
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001615 the shared state packages. Consequently, there are considerations that
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001616 affect maintaining shared state feeds. For information on how the
1617 build system works with packages and can track incrementing ``PR``
Andrew Geissler09209ee2020-12-13 08:44:15 -06001618 information, see the ":ref:`dev-manual/common-tasks:automatically incrementing a package version number`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001619 section in the Yocto Project Development Tasks Manual.
1620
1621 - The code in the build system that supports incremental builds is
1622 not simple code. For techniques that help you work around issues
1623 related to shared state code, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001624 ":ref:`dev-manual/common-tasks:viewing metadata used to create the input signature of a shared state task`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001625 and
Andrew Geissler09209ee2020-12-13 08:44:15 -06001626 ":ref:`dev-manual/common-tasks:invalidating shared state to force a task to run`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001627 sections both in the Yocto Project Development Tasks Manual.
1628
1629The rest of this section goes into detail about the overall incremental
1630build architecture, the checksums (signatures), and shared state.
1631
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001632Overall Architecture
1633--------------------
1634
1635When determining what parts of the system need to be built, BitBake
1636works on a per-task basis rather than a per-recipe basis. You might
1637wonder why using a per-task basis is preferred over a per-recipe basis.
1638To help explain, consider having the IPK packaging backend enabled and
1639then switching to DEB. In this case, the
1640:ref:`ref-tasks-install` and
1641:ref:`ref-tasks-package` task outputs
1642are still valid. However, with a per-recipe approach, the build would
1643not include the ``.deb`` files. Consequently, you would have to
1644invalidate the whole build and rerun it. Rerunning everything is not the
1645best solution. Also, in this case, the core must be "taught" much about
1646specific tasks. This methodology does not scale well and does not allow
1647users to easily add new tasks in layers or as external recipes without
1648touching the packaged-staging core.
1649
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001650Checksums (Signatures)
1651----------------------
1652
1653The shared state code uses a checksum, which is a unique signature of a
1654task's inputs, to determine if a task needs to be run again. Because it
1655is a change in a task's inputs that triggers a rerun, the process needs
1656to detect all the inputs to a given task. For shell tasks, this turns
1657out to be fairly easy because the build process generates a "run" shell
1658script for each task and it is possible to create a checksum that gives
1659you a good idea of when the task's data changes.
1660
1661To complicate the problem, there are things that should not be included
1662in the checksum. First, there is the actual specific build path of a
1663given task - the :term:`WORKDIR`. It
1664does not matter if the work directory changes because it should not
1665affect the output for target packages. Also, the build process has the
1666objective of making native or cross packages relocatable.
1667
1668.. note::
1669
1670 Both native and cross packages run on the
1671 build host. However, cross packages generate output for the target
1672 architecture.
1673
1674The checksum therefore needs to exclude ``WORKDIR``. The simplistic
1675approach for excluding the work directory is to set ``WORKDIR`` to some
1676fixed value and create the checksum for the "run" script.
1677
1678Another problem results from the "run" scripts containing functions that
1679might or might not get called. The incremental build solution contains
1680code that figures out dependencies between shell functions. This code is
1681used to prune the "run" scripts down to the minimum set, thereby
1682alleviating this problem and making the "run" scripts much more readable
1683as a bonus.
1684
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001685So far, there are solutions for shell scripts. What about Python tasks? The
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001686same approach applies even though these tasks are more difficult. The
1687process needs to figure out what variables a Python function accesses
1688and what functions it calls. Again, the incremental build solution
1689contains code that first figures out the variable and function
1690dependencies, and then creates a checksum for the data used as the input
1691to the task.
1692
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001693Like the ``WORKDIR`` case, there can be situations where dependencies should be
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001694ignored. For these situations, you can instruct the build process to
Andrew Geisslerc926e172021-05-07 16:11:35 -05001695ignore a dependency by using a line like the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001696
1697 PACKAGE_ARCHS[vardepsexclude] = "MACHINE"
1698
1699This example ensures that the :term:`PACKAGE_ARCHS` variable
1700does not depend on the value of :term:`MACHINE`, even if it does
1701reference it.
1702
1703Equally, there are cases where you need to add dependencies BitBake is
1704not able to find. You can accomplish this by using a line like the
Andrew Geisslerc926e172021-05-07 16:11:35 -05001705following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001706
1707 PACKAGE_ARCHS[vardeps] = "MACHINE"
1708
1709This example explicitly
1710adds the ``MACHINE`` variable as a dependency for ``PACKAGE_ARCHS``.
1711
1712As an example, consider a case with in-line Python where BitBake is not
1713able to figure out dependencies. When running in debug mode (i.e. using
1714``-DDD``), BitBake produces output when it discovers something for which
1715it cannot figure out dependencies. The Yocto Project team has currently
1716not managed to cover those dependencies in detail and is aware of the
1717need to fix this situation.
1718
1719Thus far, this section has limited discussion to the direct inputs into
1720a task. Information based on direct inputs is referred to as the
1721"basehash" in the code. However, the question of a task's indirect
1722inputs still exits - items already built and present in the
1723:term:`Build Directory`. The checksum (or
1724signature) for a particular task needs to add the hashes of all the
1725tasks on which the particular task depends. Choosing which dependencies
1726to add is a policy decision. However, the effect is to generate a master
1727checksum that combines the basehash and the hashes of the task's
1728dependencies.
1729
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001730At the code level, there are multiple ways by which both the basehash
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001731and the dependent task hashes can be influenced. Within the BitBake
1732configuration file, you can give BitBake some extra information to help
1733it construct the basehash. The following statement effectively results
1734in a list of global variable dependency excludes (i.e. variables never
Andrew Geisslerc926e172021-05-07 16:11:35 -05001735included in any checksum)::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001736
1737 BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \\
1738 SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \\
1739 USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \\
1740 PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \\
1741 CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX"
1742
1743The
1744previous example excludes
1745:term:`WORKDIR` since that variable
1746is actually constructed as a path within
1747:term:`TMPDIR`, which is on the
1748whitelist.
1749
1750The rules for deciding which hashes of dependent tasks to include
1751through dependency chains are more complex and are generally
1752accomplished with a Python function. The code in
1753``meta/lib/oe/sstatesig.py`` shows two examples of this and also
1754illustrates how you can insert your own policy into the system if so
1755desired. This file defines the two basic signature generators
1756:term:`OpenEmbedded-Core (OE-Core)` uses: "OEBasic" and
1757"OEBasicHash". By default, a dummy "noop" signature handler is enabled
1758in BitBake. This means that behavior is unchanged from previous
1759versions. OE-Core uses the "OEBasicHash" signature handler by default
Andrew Geisslerc926e172021-05-07 16:11:35 -05001760through this setting in the ``bitbake.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001761
1762 BB_SIGNATURE_HANDLER ?= "OEBasicHash"
1763
1764The "OEBasicHash" ``BB_SIGNATURE_HANDLER`` is the same
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001765as the "OEBasic" version but adds the task hash to the :ref:`stamp
1766files <overview-manual/concepts:stamp files and the rerunning of tasks>`. This
1767results in any metadata change that changes the task hash, automatically causing
1768the task to be run again. This removes the need to bump
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001769:term:`PR` values, and changes to metadata
1770automatically ripple across the build.
1771
1772It is also worth noting that the end result of these signature
1773generators is to make some dependency and hash information available to
1774the build. This information includes:
1775
1776- ``BB_BASEHASH_task-``\ taskname: The base hashes for each task in the
1777 recipe.
1778
1779- ``BB_BASEHASH_``\ filename\ ``:``\ taskname: The base hashes for each
1780 dependent task.
1781
1782- ``BBHASHDEPS_``\ filename\ ``:``\ taskname: The task dependencies for
1783 each task.
1784
1785- ``BB_TASKHASH``: The hash of the currently running task.
1786
1787Shared State
1788------------
1789
1790Checksums and dependencies, as discussed in the previous section, solve
1791half the problem of supporting a shared state. The other half of the
1792problem is being able to use checksum information during the build and
1793being able to reuse or rebuild specific components.
1794
1795The :ref:`sstate <ref-classes-sstate>` class is a
1796relatively generic implementation of how to "capture" a snapshot of a
1797given task. The idea is that the build process does not care about the
1798source of a task's output. Output could be freshly built or it could be
1799downloaded and unpacked from somewhere. In other words, the build
1800process does not need to worry about its origin.
1801
1802Two types of output exist. One type is just about creating a directory
1803in :term:`WORKDIR`. A good example is
1804the output of either
1805:ref:`ref-tasks-install` or
1806:ref:`ref-tasks-package`. The other
1807type of output occurs when a set of data is merged into a shared
1808directory tree such as the sysroot.
1809
1810The Yocto Project team has tried to keep the details of the
1811implementation hidden in ``sstate`` class. From a user's perspective,
1812adding shared state wrapping to a task is as simple as this
1813:ref:`ref-tasks-deploy` example taken
Andrew Geisslerc926e172021-05-07 16:11:35 -05001814from the :ref:`deploy <ref-classes-deploy>` class::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001815
1816 DEPLOYDIR = "${WORKDIR}/deploy-${PN}"
1817 SSTATETASKS += "do_deploy"
1818 do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"
1819 do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
1820
1821 python do_deploy_setscene () {
1822 sstate_setscene(d)
1823 }
1824 addtask do_deploy_setscene
1825 do_deploy[dirs] = "${DEPLOYDIR} ${B}"
1826 do_deploy[stamp-extra-info] = "${MACHINE_ARCH}"
1827
1828The following list explains the previous example:
1829
1830- Adding "do_deploy" to ``SSTATETASKS`` adds some required
1831 sstate-related processing, which is implemented in the
1832 :ref:`sstate <ref-classes-sstate>` class, to
1833 before and after the
1834 :ref:`ref-tasks-deploy` task.
1835
1836- The ``do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"`` declares that
1837 ``do_deploy`` places its output in ``${DEPLOYDIR}`` when run normally
1838 (i.e. when not using the sstate cache). This output becomes the input
1839 to the shared state cache.
1840
1841- The ``do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"`` line
1842 causes the contents of the shared state cache to be copied to
1843 ``${DEPLOY_DIR_IMAGE}``.
1844
1845 .. note::
1846
1847 If ``do_deploy`` is not already in the shared state cache or if its input
1848 checksum (signature) has changed from when the output was cached, the task
1849 runs to populate the shared state cache, after which the contents of the
1850 shared state cache is copied to ${:term:`DEPLOY_DIR_IMAGE`}. If
1851 ``do_deploy`` is in the shared state cache and its signature indicates
1852 that the cached output is still valid (i.e. if no relevant task inputs
1853 have changed), then the contents of the shared state cache copies
1854 directly to ${``DEPLOY_DIR_IMAGE``} by the ``do_deploy_setscene`` task
1855 instead, skipping the ``do_deploy`` task.
1856
1857- The following task definition is glue logic needed to make the
Andrew Geisslerc926e172021-05-07 16:11:35 -05001858 previous settings effective::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001859
1860 python do_deploy_setscene () {
1861 sstate_setscene(d)
1862 }
1863 addtask do_deploy_setscene
1864
1865 ``sstate_setscene()`` takes the flags above as input and accelerates the ``do_deploy`` task
1866 through the shared state cache if possible. If the task was
1867 accelerated, ``sstate_setscene()`` returns True. Otherwise, it
1868 returns False, and the normal ``do_deploy`` task runs. For more
1869 information, see the ":ref:`setscene <bitbake:bitbake-user-manual/bitbake-user-manual-execution:setscene>`"
1870 section in the BitBake User Manual.
1871
1872- The ``do_deploy[dirs] = "${DEPLOYDIR} ${B}"`` line creates
1873 ``${DEPLOYDIR}`` and ``${B}`` before the ``do_deploy`` task runs, and
1874 also sets the current working directory of ``do_deploy`` to ``${B}``.
1875 For more information, see the ":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags`"
1876 section in the BitBake
1877 User Manual.
1878
1879 .. note::
1880
1881 In cases where ``sstate-inputdirs`` and ``sstate-outputdirs`` would be
1882 the same, you can use ``sstate-plaindirs``. For example, to preserve the
1883 ${:term:`PKGD`} and ${:term:`PKGDEST`} output from the ``do_package``
Andrew Geisslerc926e172021-05-07 16:11:35 -05001884 task, use the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001885
1886 do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}"
1887
1888
1889- The ``do_deploy[stamp-extra-info] = "${MACHINE_ARCH}"`` line appends
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001890 extra metadata to the :ref:`stamp
1891 file <overview-manual/concepts:stamp files and the rerunning of tasks>`. In
1892 this case, the metadata makes the task specific to a machine's architecture.
1893 See
Andrew Geissler09209ee2020-12-13 08:44:15 -06001894 ":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-execution:the task list`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001895 section in the BitBake User Manual for more information on the
1896 ``stamp-extra-info`` flag.
1897
1898- ``sstate-inputdirs`` and ``sstate-outputdirs`` can also be used with
1899 multiple directories. For example, the following declares
1900 ``PKGDESTWORK`` and ``SHLIBWORK`` as shared state input directories,
1901 which populates the shared state cache, and ``PKGDATA_DIR`` and
Andrew Geisslerc926e172021-05-07 16:11:35 -05001902 ``SHLIBSDIR`` as the corresponding shared state output directories::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001903
1904 do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}"
1905 do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}"
1906
1907- These methods also include the ability to take a lockfile when
1908 manipulating shared state directory structures, for cases where file
Andrew Geisslerc926e172021-05-07 16:11:35 -05001909 additions or removals are sensitive::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001910
1911 do_package[sstate-lockfile] = "${PACKAGELOCK}"
1912
1913Behind the scenes, the shared state code works by looking in
1914:term:`SSTATE_DIR` and
1915:term:`SSTATE_MIRRORS` for
Andrew Geisslerc926e172021-05-07 16:11:35 -05001916shared state files. Here is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001917
1918 SSTATE_MIRRORS ?= "\
1919 file://.\* http://someserver.tld/share/sstate/PATH;downloadfilename=PATH \n \
1920 file://.\* file:///some/local/dir/sstate/PATH"
1921
1922.. note::
1923
1924 The shared state directory (``SSTATE_DIR``) is organized into two-character
1925 subdirectories, where the subdirectory names are based on the first two
1926 characters of the hash.
1927 If the shared state directory structure for a mirror has the same structure
1928 as ``SSTATE_DIR``, you must specify "PATH" as part of the URI to enable the build
1929 system to map to the appropriate subdirectory.
1930
1931The shared state package validity can be detected just by looking at the
1932filename since the filename contains the task checksum (or signature) as
1933described earlier in this section. If a valid shared state package is
1934found, the build process downloads it and uses it to accelerate the
1935task.
1936
1937The build processes use the ``*_setscene`` tasks for the task
1938acceleration phase. BitBake goes through this phase before the main
1939execution code and tries to accelerate any tasks for which it can find
1940shared state packages. If a shared state package for a task is
1941available, the shared state package is used. This means the task and any
1942tasks on which it is dependent are not executed.
1943
1944As a real world example, the aim is when building an IPK-based image,
1945only the
1946:ref:`ref-tasks-package_write_ipk`
1947tasks would have their shared state packages fetched and extracted.
1948Since the sysroot is not used, it would never get extracted. This is
1949another reason why a task-based approach is preferred over a
1950recipe-based approach, which would have to install the output from every
1951task.
1952
1953Automatically Added Runtime Dependencies
1954========================================
1955
1956The OpenEmbedded build system automatically adds common types of runtime
1957dependencies between packages, which means that you do not need to
1958explicitly declare the packages using
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001959:term:`RDEPENDS`. There are three automatic
1960mechanisms (``shlibdeps``, ``pcdeps``, and ``depchains``) that
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001961handle shared libraries, package configuration (pkg-config) modules, and
1962``-dev`` and ``-dbg`` packages, respectively. For other types of runtime
1963dependencies, you must manually declare the dependencies.
1964
1965- ``shlibdeps``: During the
1966 :ref:`ref-tasks-package` task of
1967 each recipe, all shared libraries installed by the recipe are
1968 located. For each shared library, the package that contains the
1969 shared library is registered as providing the shared library. More
1970 specifically, the package is registered as providing the
1971 `soname <https://en.wikipedia.org/wiki/Soname>`__ of the library. The
1972 resulting shared-library-to-package mapping is saved globally in
1973 :term:`PKGDATA_DIR` by the
1974 :ref:`ref-tasks-packagedata`
1975 task.
1976
1977 Simultaneously, all executables and shared libraries installed by the
1978 recipe are inspected to see what shared libraries they link against.
1979 For each shared library dependency that is found, ``PKGDATA_DIR`` is
1980 queried to see if some package (likely from a different recipe)
1981 contains the shared library. If such a package is found, a runtime
1982 dependency is added from the package that depends on the shared
1983 library to the package that contains the library.
1984
1985 The automatically added runtime dependency also includes a version
1986 restriction. This version restriction specifies that at least the
1987 current version of the package that provides the shared library must
1988 be used, as if "package (>= version)" had been added to ``RDEPENDS``.
1989 This forces an upgrade of the package containing the shared library
1990 when installing the package that depends on the library, if needed.
1991
1992 If you want to avoid a package being registered as providing a
1993 particular shared library (e.g. because the library is for internal
1994 use only), then add the library to
1995 :term:`PRIVATE_LIBS` inside
1996 the package's recipe.
1997
1998- ``pcdeps``: During the ``do_package`` task of each recipe, all
1999 pkg-config modules (``*.pc`` files) installed by the recipe are
2000 located. For each module, the package that contains the module is
2001 registered as providing the module. The resulting module-to-package
2002 mapping is saved globally in ``PKGDATA_DIR`` by the
2003 ``do_packagedata`` task.
2004
2005 Simultaneously, all pkg-config modules installed by the recipe are
2006 inspected to see what other pkg-config modules they depend on. A
2007 module is seen as depending on another module if it contains a
2008 "Requires:" line that specifies the other module. For each module
2009 dependency, ``PKGDATA_DIR`` is queried to see if some package
2010 contains the module. If such a package is found, a runtime dependency
2011 is added from the package that depends on the module to the package
2012 that contains the module.
2013
2014 .. note::
2015
2016 The
2017 pcdeps
2018 mechanism most often infers dependencies between
2019 -dev
2020 packages.
2021
2022- ``depchains``: If a package ``foo`` depends on a package ``bar``,
2023 then ``foo-dev`` and ``foo-dbg`` are also made to depend on
2024 ``bar-dev`` and ``bar-dbg``, respectively. Taking the ``-dev``
2025 packages as an example, the ``bar-dev`` package might provide headers
2026 and shared library symlinks needed by ``foo-dev``, which shows the
2027 need for a dependency between the packages.
2028
2029 The dependencies added by ``depchains`` are in the form of
2030 :term:`RRECOMMENDS`.
2031
2032 .. note::
2033
2034 By default, ``foo-dev`` also has an ``RDEPENDS``-style dependency on
2035 ``foo``, because the default value of ``RDEPENDS_${PN}-dev`` (set in
2036 bitbake.conf) includes "${PN}".
2037
2038 To ensure that the dependency chain is never broken, ``-dev`` and
2039 ``-dbg`` packages are always generated by default, even if the
2040 packages turn out to be empty. See the
2041 :term:`ALLOW_EMPTY` variable
2042 for more information.
2043
2044The ``do_package`` task depends on the ``do_packagedata`` task of each
2045recipe in :term:`DEPENDS` through use
2046of a ``[``\ :ref:`deptask <bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ``]``
2047declaration, which guarantees that the required
2048shared-library/module-to-package mapping information will be available
2049when needed as long as ``DEPENDS`` has been correctly set.
2050
2051Fakeroot and Pseudo
2052===================
2053
2054Some tasks are easier to implement when allowed to perform certain
2055operations that are normally reserved for the root user (e.g.
2056:ref:`ref-tasks-install`,
2057:ref:`do_package_write* <ref-tasks-package_write_deb>`,
2058:ref:`ref-tasks-rootfs`, and
2059:ref:`do_image* <ref-tasks-image>`). For example,
2060the ``do_install`` task benefits from being able to set the UID and GID
2061of installed files to arbitrary values.
2062
2063One approach to allowing tasks to perform root-only operations would be
2064to require :term:`BitBake` to run as
2065root. However, this method is cumbersome and has security issues. The
2066approach that is actually used is to run tasks that benefit from root
2067privileges in a "fake" root environment. Within this environment, the
2068task and its child processes believe that they are running as the root
2069user, and see an internally consistent view of the filesystem. As long
2070as generating the final output (e.g. a package or an image) does not
2071require root privileges, the fact that some earlier steps ran in a fake
2072root environment does not cause problems.
2073
2074The capability to run tasks in a fake root environment is known as
2075"`fakeroot <http://man.he.net/man1/fakeroot>`__", which is derived from
2076the BitBake keyword/variable flag that requests a fake root environment
2077for a task.
2078
Andrew Geisslerd1e89492021-02-12 15:35:20 -06002079In the :term:`OpenEmbedded Build System`, the program that implements
2080fakeroot is known as :yocto_home:`Pseudo </software-item/pseudo/>`. Pseudo
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002081overrides system calls by using the environment variable ``LD_PRELOAD``,
2082which results in the illusion of running as root. To keep track of
2083"fake" file ownership and permissions resulting from operations that
2084require root permissions, Pseudo uses an SQLite 3 database. This
2085database is stored in
2086``${``\ :term:`WORKDIR`\ ``}/pseudo/files.db``
2087for individual recipes. Storing the database in a file as opposed to in
2088memory gives persistence between tasks and builds, which is not
2089accomplished using fakeroot.
2090
2091.. note::
2092
2093 If you add your own task that manipulates the same files or
2094 directories as a fakeroot task, then that task also needs to run
2095 under fakeroot. Otherwise, the task cannot run root-only operations,
2096 and cannot see the fake file ownership and permissions set by the
2097 other task. You need to also add a dependency on
Andrew Geisslerc926e172021-05-07 16:11:35 -05002098 ``virtual/fakeroot-native:do_populate_sysroot``, giving the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002099
2100 fakeroot do_mytask () {
2101 ...
2102 }
2103 do_mytask[depends] += "virtual/fakeroot-native:do_populate_sysroot"
2104
2105
2106For more information, see the
2107:term:`FAKEROOT* <bitbake:FAKEROOT>` variables in the
2108BitBake User Manual. You can also reference the "`Why Not
2109Fakeroot? <https://github.com/wrpseudo/pseudo/wiki/WhyNotFakeroot>`__"
2110article for background information on Fakeroot and Pseudo.