blob: 1e5f0f90399768b0fdb4643051b1162311136c57 [file] [log] [blame]
Andrew Geisslerf0343792020-11-18 10:42:21 -06001.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002
3**********************
4Yocto Project Concepts
5**********************
6
7This chapter provides explanations for Yocto Project concepts that go
8beyond the surface of "how-to" information and reference (or look-up)
9material. Concepts such as components, the :term:`OpenEmbedded Build System`
10workflow,
11cross-development toolchains, shared state cache, and so forth are
12explained.
13
14Yocto Project Components
15========================
16
17The :term:`BitBake` task executor
18together with various types of configuration files form the
19:term:`OpenEmbedded-Core (OE-Core)`. This section
20overviews these components by describing their use and how they
21interact.
22
23BitBake handles the parsing and execution of the data files. The data
24itself is of various types:
25
26- *Recipes:* Provides details about particular pieces of software.
27
28- *Class Data:* Abstracts common build information (e.g. how to build a
29 Linux kernel).
30
31- *Configuration Data:* Defines machine-specific settings, policy
32 decisions, and so forth. Configuration data acts as the glue to bind
33 everything together.
34
35BitBake knows how to combine multiple data sources together and refers
36to each data source as a layer. For information on layers, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -060037":ref:`dev-manual/common-tasks:understanding and creating layers`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050038section of the Yocto Project Development Tasks Manual.
39
40Following are some brief details on these core components. For
41additional information on how these components interact during a build,
42see the
Andrew Geissler09209ee2020-12-13 08:44:15 -060043":ref:`overview-manual/concepts:openembedded build system concepts`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050044section.
45
Andrew Geisslerc9f78652020-09-18 14:11:35 -050046BitBake
47-------
48
49BitBake is the tool at the heart of the :term:`OpenEmbedded Build System`
50and is responsible
51for parsing the :term:`Metadata`, generating
52a list of tasks from it, and then executing those tasks.
53
54This section briefly introduces BitBake. If you want more information on
55BitBake, see the :doc:`BitBake User Manual <bitbake:index>`.
56
57To see a list of the options BitBake supports, use either of the
Andrew Geisslerc926e172021-05-07 16:11:35 -050058following commands::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050059
60 $ bitbake -h
61 $ bitbake --help
62
63The most common usage for BitBake is ``bitbake recipename``, where
64``recipename`` is the name of the recipe you want to build (referred
65to as the "target"). The target often equates to the first part of a
66recipe's filename (e.g. "foo" for a recipe named ``foo_1.3.0-r0.bb``).
67So, to process the ``matchbox-desktop_1.2.3.bb`` recipe file, you might
Andrew Geisslerc926e172021-05-07 16:11:35 -050068type the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050069
70 $ bitbake matchbox-desktop
71
72Several different
73versions of ``matchbox-desktop`` might exist. BitBake chooses the one
74selected by the distribution configuration. You can get more details
75about how BitBake chooses between different target versions and
76providers in the
Andrew Geissler09209ee2020-12-13 08:44:15 -060077":ref:`Preferences <bitbake:bitbake-user-manual/bitbake-user-manual-execution:preferences>`" section
Andrew Geisslerc9f78652020-09-18 14:11:35 -050078of the BitBake User Manual.
79
80BitBake also tries to execute any dependent tasks first. So for example,
81before building ``matchbox-desktop``, BitBake would build a cross
82compiler and ``glibc`` if they had not already been built.
83
84A useful BitBake option to consider is the ``-k`` or ``--continue``
85option. This option instructs BitBake to try and continue processing the
86job as long as possible even after encountering an error. When an error
87occurs, the target that failed and those that depend on it cannot be
88remade. However, when you use this option other dependencies can still
89be processed.
90
Andrew Geisslerc9f78652020-09-18 14:11:35 -050091Recipes
92-------
93
94Files that have the ``.bb`` suffix are "recipes" files. In general, a
95recipe contains information about a single piece of software. This
96information includes the location from which to download the unaltered
97source, any source patches to be applied to that source (if needed),
98which special configuration options to apply, how to compile the source
99files, and how to package the compiled output.
100
101The term "package" is sometimes used to refer to recipes. However, since
102the word "package" is used for the packaged output from the OpenEmbedded
103build system (i.e. ``.ipk`` or ``.deb`` files), this document avoids
104using the term "package" when referring to recipes.
105
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500106Classes
107-------
108
109Class files (``.bbclass``) contain information that is useful to share
110between recipes files. An example is the
111:ref:`autotools <ref-classes-autotools>` class,
112which contains common settings for any application that Autotools uses.
Andrew Geissler09209ee2020-12-13 08:44:15 -0600113The ":ref:`ref-manual/classes:Classes`" chapter in the
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500114Yocto Project Reference Manual provides details about classes and how to
115use them.
116
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500117Configurations
118--------------
119
120The configuration files (``.conf``) define various configuration
121variables that govern the OpenEmbedded build process. These files fall
122into several areas that define machine configuration options,
123distribution configuration options, compiler tuning options, general
124common configuration options, and user configuration options in
125``conf/local.conf``, which is found in the :term:`Build Directory`.
126
127
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500128Layers
129======
130
131Layers are repositories that contain related metadata (i.e. sets of
132instructions) that tell the OpenEmbedded build system how to build a
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500133target. :ref:`overview-manual/yp-intro:the yocto project layer model`
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500134facilitates collaboration, sharing, customization, and reuse within the
135Yocto Project development environment. Layers logically separate
136information for your project. For example, you can use a layer to hold
137all the configurations for a particular piece of hardware. Isolating
138hardware-specific configurations allows you to share other metadata by
139using a different layer where that metadata might be common across
140several pieces of hardware.
141
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700142There are many layers working in the Yocto Project development environment. The
Andrew Geisslerd1e89492021-02-12 15:35:20 -0600143:yocto_home:`Yocto Project Curated Layer Index </software-overview/layers/>`
144and :oe_layerindex:`OpenEmbedded Layer Index <>` both contain layers from
145which you can use or leverage.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500146
147By convention, layers in the Yocto Project follow a specific form.
148Conforming to a known structure allows BitBake to make assumptions
149during builds on where to find types of metadata. You can find
150procedures and learn about tools (i.e. ``bitbake-layers``) for creating
151layers suitable for the Yocto Project in the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600152":ref:`dev-manual/common-tasks:understanding and creating layers`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500153section of the Yocto Project Development Tasks Manual.
154
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500155OpenEmbedded Build System Concepts
156==================================
157
158This section takes a more detailed look inside the build process used by
159the :term:`OpenEmbedded Build System`,
160which is the build
161system specific to the Yocto Project. At the heart of the build system
162is BitBake, the task executor.
163
164The following diagram represents the high-level workflow of a build. The
165remainder of this section expands on the fundamental input, output,
166process, and metadata logical blocks that make up the workflow.
167
168.. image:: figures/YP-flow-diagram.png
169 :align: center
170
171In general, the build's workflow consists of several functional areas:
172
173- *User Configuration:* metadata you can use to control the build
174 process.
175
176- *Metadata Layers:* Various layers that provide software, machine, and
177 distro metadata.
178
179- *Source Files:* Upstream releases, local projects, and SCMs.
180
181- *Build System:* Processes under the control of
182 :term:`BitBake`. This block expands
183 on how BitBake fetches source, applies patches, completes
184 compilation, analyzes output for package generation, creates and
185 tests packages, generates images, and generates cross-development
186 tools.
187
188- *Package Feeds:* Directories containing output packages (RPM, DEB or
189 IPK), which are subsequently used in the construction of an image or
190 Software Development Kit (SDK), produced by the build system. These
191 feeds can also be copied and shared using a web server or other means
192 to facilitate extending or updating existing images on devices at
193 runtime if runtime package management is enabled.
194
195- *Images:* Images produced by the workflow.
196
197- *Application Development SDK:* Cross-development tools that are
198 produced along with an image or separately with BitBake.
199
200User Configuration
201------------------
202
203User configuration helps define the build. Through user configuration,
204you can tell BitBake the target architecture for which you are building
205the image, where to store downloaded source, and other build properties.
206
207The following figure shows an expanded representation of the "User
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500208Configuration" box of the :ref:`general workflow
209figure <overview-manual/concepts:openembedded build system concepts>`:
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500210
211.. image:: figures/user-configuration.png
212 :align: center
213
214BitBake needs some basic configuration files in order to complete a
215build. These files are ``*.conf`` files. The minimally necessary ones
216reside as example files in the ``build/conf`` directory of the
217:term:`Source Directory`. For simplicity,
218this section refers to the Source Directory as the "Poky Directory."
219
220When you clone the :term:`Poky` Git repository
221or you download and unpack a Yocto Project release, you can set up the
222Source Directory to be named anything you want. For this discussion, the
223cloned repository uses the default name ``poky``.
224
225.. note::
226
227 The Poky repository is primarily an aggregation of existing
228 repositories. It is not a canonical upstream source.
229
230The ``meta-poky`` layer inside Poky contains a ``conf`` directory that
231has example configuration files. These example files are used as a basis
232for creating actual configuration files when you source
233:ref:`structure-core-script`, which is the
234build environment script.
235
236Sourcing the build environment script creates a
237:term:`Build Directory` if one does not
238already exist. BitBake uses the Build Directory for all its work during
239builds. The Build Directory has a ``conf`` directory that contains
240default versions of your ``local.conf`` and ``bblayers.conf``
241configuration files. These default configuration files are created only
242if versions do not already exist in the Build Directory at the time you
243source the build environment setup script.
244
245Because the Poky repository is fundamentally an aggregation of existing
246repositories, some users might be familiar with running the
247:ref:`structure-core-script` script in the context of separate
248:term:`OpenEmbedded-Core (OE-Core)` and BitBake
249repositories rather than a single Poky repository. This discussion
250assumes the script is executed from within a cloned or unpacked version
251of Poky.
252
253Depending on where the script is sourced, different sub-scripts are
254called to set up the Build Directory (Yocto or OpenEmbedded).
255Specifically, the script ``scripts/oe-setup-builddir`` inside the poky
256directory sets up the Build Directory and seeds the directory (if
257necessary) with configuration files appropriate for the Yocto Project
258development environment.
259
260.. note::
261
262 The
263 scripts/oe-setup-builddir
264 script uses the
265 ``$TEMPLATECONF``
266 variable to determine which sample configuration files to locate.
267
268The ``local.conf`` file provides many basic variables that define a
269build environment. Here is a list of a few. To see the default
270configurations in a ``local.conf`` file created by the build environment
271script, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600272:yocto_git:`local.conf.sample </poky/tree/meta-poky/conf/local.conf.sample>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500273in the ``meta-poky`` layer:
274
275- *Target Machine Selection:* Controlled by the
276 :term:`MACHINE` variable.
277
278- *Download Directory:* Controlled by the
279 :term:`DL_DIR` variable.
280
281- *Shared State Directory:* Controlled by the
282 :term:`SSTATE_DIR` variable.
283
284- *Build Output:* Controlled by the
285 :term:`TMPDIR` variable.
286
287- *Distribution Policy:* Controlled by the
288 :term:`DISTRO` variable.
289
290- *Packaging Format:* Controlled by the
291 :term:`PACKAGE_CLASSES`
292 variable.
293
294- *SDK Target Architecture:* Controlled by the
295 :term:`SDKMACHINE` variable.
296
297- *Extra Image Packages:* Controlled by the
298 :term:`EXTRA_IMAGE_FEATURES`
299 variable.
300
301.. note::
302
303 Configurations set in the
304 conf/local.conf
305 file can also be set in the
306 conf/site.conf
307 and
308 conf/auto.conf
309 configuration files.
310
311The ``bblayers.conf`` file tells BitBake what layers you want considered
312during the build. By default, the layers listed in this file include
313layers minimally needed by the build system. However, you must manually
314add any custom layers you have created. You can find more information on
315working with the ``bblayers.conf`` file in the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600316":ref:`dev-manual/common-tasks:enabling your layer`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500317section in the Yocto Project Development Tasks Manual.
318
319The files ``site.conf`` and ``auto.conf`` are not created by the
320environment initialization script. If you want the ``site.conf`` file,
321you need to create that yourself. The ``auto.conf`` file is typically
322created by an autobuilder:
323
324- *site.conf:* You can use the ``conf/site.conf`` configuration
325 file to configure multiple build directories. For example, suppose
326 you had several build environments and they shared some common
327 features. You can set these default build properties here. A good
328 example is perhaps the packaging format to use through the
329 :term:`PACKAGE_CLASSES`
330 variable.
331
332 One useful scenario for using the ``conf/site.conf`` file is to
333 extend your :term:`BBPATH` variable
334 to include the path to a ``conf/site.conf``. Then, when BitBake looks
Andrew Geissler09036742021-06-25 14:25:14 -0500335 for Metadata using :term:`BBPATH`, it finds the ``conf/site.conf`` file
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500336 and applies your common configurations found in the file. To override
337 configurations in a particular build directory, alter the similar
338 configurations within that build directory's ``conf/local.conf``
339 file.
340
341- *auto.conf:* The file is usually created and written to by an
342 autobuilder. The settings put into the file are typically the same as
343 you would find in the ``conf/local.conf`` or the ``conf/site.conf``
344 files.
345
346You can edit all configuration files to further define any particular
347build environment. This process is represented by the "User
348Configuration Edits" box in the figure.
349
350When you launch your build with the ``bitbake target`` command, BitBake
351sorts out the configurations to ultimately define your build
352environment. It is important to understand that the
353:term:`OpenEmbedded Build System` reads the
354configuration files in a specific order: ``site.conf``, ``auto.conf``,
355and ``local.conf``. And, the build system applies the normal assignment
356statement rules as described in the
357":doc:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata`" chapter
358of the BitBake User Manual. Because the files are parsed in a specific
359order, variable assignments for the same variable could be affected. For
360example, if the ``auto.conf`` file and the ``local.conf`` set variable1
361to different values, because the build system parses ``local.conf``
362after ``auto.conf``, variable1 is assigned the value from the
363``local.conf`` file.
364
365Metadata, Machine Configuration, and Policy Configuration
366---------------------------------------------------------
367
368The previous section described the user configurations that define
369BitBake's global behavior. This section takes a closer look at the
370layers the build system uses to further control the build. These layers
371provide Metadata for the software, machine, and policies.
372
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700373In general, there are three types of layer input. You can see them below
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500374the "User Configuration" box in the `general workflow
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500375figure <overview-manual/concepts:openembedded build system concepts>`:
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500376
377- *Metadata (.bb + Patches):* Software layers containing
378 user-supplied recipe files, patches, and append files. A good example
Andrew Geisslerd1e89492021-02-12 15:35:20 -0600379 of a software layer might be the :oe_layer:`meta-qt5 layer </meta-qt5>`
380 from the :oe_layerindex:`OpenEmbedded Layer Index <>`. This layer is for
381 version 5.0 of the popular `Qt <https://wiki.qt.io/About_Qt>`__
382 cross-platform application development framework for desktop, embedded and
383 mobile.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500384
385- *Machine BSP Configuration:* Board Support Package (BSP) layers (i.e.
386 "BSP Layer" in the following figure) providing machine-specific
387 configurations. This type of information is specific to a particular
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500388 target architecture. A good example of a BSP layer from the
389 :ref:`overview-manual/yp-intro:reference distribution (poky)` is the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600390 :yocto_git:`meta-yocto-bsp </poky/tree/meta-yocto-bsp>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500391 layer.
392
393- *Policy Configuration:* Distribution Layers (i.e. "Distro Layer" in
394 the following figure) providing top-level or general policies for the
395 images or SDKs being built for a particular distribution. For
396 example, in the Poky Reference Distribution the distro layer is the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600397 :yocto_git:`meta-poky </poky/tree/meta-poky>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500398 layer. Within the distro layer is a ``conf/distro`` directory that
399 contains distro configuration files (e.g.
Andrew Geissler09209ee2020-12-13 08:44:15 -0600400 :yocto_git:`poky.conf </poky/tree/meta-poky/conf/distro/poky.conf>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500401 that contain many policy configurations for the Poky distribution.
402
403The following figure shows an expanded representation of these three
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500404layers from the :ref:`general workflow figure
405<overview-manual/concepts:openembedded build system concepts>`:
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500406
407.. image:: figures/layer-input.png
408 :align: center
409
410In general, all layers have a similar structure. They all contain a
411licensing file (e.g. ``COPYING.MIT``) if the layer is to be distributed,
412a ``README`` file as good practice and especially if the layer is to be
413distributed, a configuration directory, and recipe directories. You can
414learn about the general structure for layers used with the Yocto Project
415in the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600416":ref:`dev-manual/common-tasks:creating your own layer`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500417section in the
418Yocto Project Development Tasks Manual. For a general discussion on
419layers and the many layers from which you can draw, see the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500420":ref:`overview-manual/concepts:layers`" and
421":ref:`overview-manual/yp-intro:the yocto project layer model`" sections both
422earlier in this manual.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500423
424If you explored the previous links, you discovered some areas where many
Andrew Geissler09209ee2020-12-13 08:44:15 -0600425layers that work with the Yocto Project exist. The :yocto_git:`Source
426Repositories <>` also shows layers categorized under "Yocto Metadata Layers."
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500427
428.. note::
429
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700430 There are layers in the Yocto Project Source Repositories that cannot be
431 found in the OpenEmbedded Layer Index. Such layers are either
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500432 deprecated or experimental in nature.
433
434BitBake uses the ``conf/bblayers.conf`` file, which is part of the user
435configuration, to find what layers it should be using as part of the
436build.
437
438Distro Layer
439~~~~~~~~~~~~
440
441The distribution layer provides policy configurations for your
442distribution. Best practices dictate that you isolate these types of
443configurations into their own layer. Settings you provide in
444``conf/distro/distro.conf`` override similar settings that BitBake finds
445in your ``conf/local.conf`` file in the Build Directory.
446
447The following list provides some explanation and references for what you
448typically find in the distribution layer:
449
450- *classes:* Class files (``.bbclass``) hold common functionality that
451 can be shared among recipes in the distribution. When your recipes
452 inherit a class, they take on the settings and functions for that
453 class. You can read more about class files in the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600454 ":ref:`ref-manual/classes:Classes`" chapter of the Yocto
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500455 Reference Manual.
456
457- *conf:* This area holds configuration files for the layer
458 (``conf/layer.conf``), the distribution
459 (``conf/distro/distro.conf``), and any distribution-wide include
460 files.
461
462- *recipes-*:* Recipes and append files that affect common
463 functionality across the distribution. This area could include
464 recipes and append files to add distribution-specific configuration,
465 initialization scripts, custom image recipes, and so forth. Examples
466 of ``recipes-*`` directories are ``recipes-core`` and
467 ``recipes-extra``. Hierarchy and contents within a ``recipes-*``
468 directory can vary. Generally, these directories contain recipe files
469 (``*.bb``), recipe append files (``*.bbappend``), directories that
470 are distro-specific for configuration files, and so forth.
471
472BSP Layer
473~~~~~~~~~
474
475The BSP Layer provides machine configurations that target specific
476hardware. Everything in this layer is specific to the machine for which
477you are building the image or the SDK. A common structure or form is
478defined for BSP layers. You can learn more about this structure in the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600479:doc:`/bsp-guide/index`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500480
481.. note::
482
483 In order for a BSP layer to be considered compliant with the Yocto
484 Project, it must meet some structural requirements.
485
486The BSP Layer's configuration directory contains configuration files for
487the machine (``conf/machine/machine.conf``) and, of course, the layer
488(``conf/layer.conf``).
489
490The remainder of the layer is dedicated to specific recipes by function:
491``recipes-bsp``, ``recipes-core``, ``recipes-graphics``,
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700492``recipes-kernel``, and so forth. There can be metadata for multiple
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500493formfactors, graphics support systems, and so forth.
494
495.. note::
496
497 While the figure shows several
498 recipes-\*
499 directories, not all these directories appear in all BSP layers.
500
501Software Layer
502~~~~~~~~~~~~~~
503
504The software layer provides the Metadata for additional software
505packages used during the build. This layer does not include Metadata
506that is specific to the distribution or the machine, which are found in
507their respective layers.
508
509This layer contains any recipes, append files, and patches, that your
510project needs.
511
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500512Sources
513-------
514
515In order for the OpenEmbedded build system to create an image or any
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500516target, it must be able to access source files. The :ref:`general workflow
517figure <overview-manual/concepts:openembedded build system concepts>`
518represents source files using the "Upstream Project Releases", "Local
519Projects", and "SCMs (optional)" boxes. The figure represents mirrors,
520which also play a role in locating source files, with the "Source
521Materials" box.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500522
523The method by which source files are ultimately organized is a function
524of the project. For example, for released software, projects tend to use
525tarballs or other archived files that can capture the state of a release
526guaranteeing that it is statically represented. On the other hand, for a
527project that is more dynamic or experimental in nature, a project might
528keep source files in a repository controlled by a Source Control Manager
529(SCM) such as Git. Pulling source from a repository allows you to
530control the point in the repository (the revision) from which you want
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700531to build software. A combination of the two is also possible.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500532
533BitBake uses the :term:`SRC_URI`
534variable to point to source files regardless of their location. Each
Andrew Geissler09036742021-06-25 14:25:14 -0500535recipe must have a :term:`SRC_URI` variable that points to the source.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500536
537Another area that plays a significant role in where source files come
538from is pointed to by the
539:term:`DL_DIR` variable. This area is
540a cache that can hold previously downloaded source. You can also
541instruct the OpenEmbedded build system to create tarballs from Git
542repositories, which is not the default behavior, and store them in the
Andrew Geissler09036742021-06-25 14:25:14 -0500543:term:`DL_DIR` by using the
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500544:term:`BB_GENERATE_MIRROR_TARBALLS`
545variable.
546
Andrew Geissler09036742021-06-25 14:25:14 -0500547Judicious use of a :term:`DL_DIR` directory can save the build system a trip
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500548across the Internet when looking for files. A good method for using a
Andrew Geissler09036742021-06-25 14:25:14 -0500549download directory is to have :term:`DL_DIR` point to an area outside of
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500550your Build Directory. Doing so allows you to safely delete the Build
551Directory if needed without fear of removing any downloaded source file.
552
553The remainder of this section provides a deeper look into the source
554files and the mirrors. Here is a more detailed look at the source file
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500555area of the :ref:`general workflow figure <overview-manual/concepts:openembedded build system concepts>`:
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500556
557.. image:: figures/source-input.png
558 :align: center
559
560Upstream Project Releases
561~~~~~~~~~~~~~~~~~~~~~~~~~
562
563Upstream project releases exist anywhere in the form of an archived file
564(e.g. tarball or zip file). These files correspond to individual
565recipes. For example, the figure uses specific releases each for
566BusyBox, Qt, and Dbus. An archive file can be for any released product
567that can be built using a recipe.
568
569Local Projects
570~~~~~~~~~~~~~~
571
572Local projects are custom bits of software the user provides. These bits
573reside somewhere local to a project - perhaps a directory into which the
574user checks in items (e.g. a local directory containing a development
575source tree used by the group).
576
577The canonical method through which to include a local project is to use
578the :ref:`externalsrc <ref-classes-externalsrc>`
579class to include that local project. You use either the ``local.conf``
580or a recipe's append file to override or set the recipe to point to the
581local directory on your disk to pull in the whole source tree.
582
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500583Source Control Managers (Optional)
584~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
585
586Another place from which the build system can get source files is with
Andrew Geissler09209ee2020-12-13 08:44:15 -0600587:ref:`fetchers <bitbake:bitbake-user-manual/bitbake-user-manual-fetching:fetchers>` employing various Source
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500588Control Managers (SCMs) such as Git or Subversion. In such cases, a
589repository is cloned or checked out. The
590:ref:`ref-tasks-fetch` task inside
591BitBake uses the :term:`SRC_URI`
592variable and the argument's prefix to determine the correct fetcher
593module.
594
595.. note::
596
597 For information on how to have the OpenEmbedded build system generate
598 tarballs for Git repositories and place them in the
599 DL_DIR
600 directory, see the :term:`BB_GENERATE_MIRROR_TARBALLS`
601 variable in the Yocto Project Reference Manual.
602
603When fetching a repository, BitBake uses the
604:term:`SRCREV` variable to determine
605the specific revision from which to build.
606
607Source Mirror(s)
608~~~~~~~~~~~~~~~~
609
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700610There are two kinds of mirrors: pre-mirrors and regular mirrors. The
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500611:term:`PREMIRRORS` and
612:term:`MIRRORS` variables point to
613these, respectively. BitBake checks pre-mirrors before looking upstream
614for any source files. Pre-mirrors are appropriate when you have a shared
615directory that is not a directory defined by the
616:term:`DL_DIR` variable. A Pre-mirror
617typically points to a shared directory that is local to your
618organization.
619
620Regular mirrors can be any site across the Internet that is used as an
621alternative location for source code should the primary site not be
622functioning for some reason or another.
623
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500624Package Feeds
625-------------
626
627When the OpenEmbedded build system generates an image or an SDK, it gets
628the packages from a package feed area located in the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500629:term:`Build Directory`. The :ref:`general workflow figure
630<overview-manual/concepts:openembedded build system concepts>`
631shows this package feeds area in the upper-right corner.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500632
633This section looks a little closer into the package feeds area used by
634the build system. Here is a more detailed look at the area:
635
636.. image:: figures/package-feeds.png
637 :align: center
638
639Package feeds are an intermediary step in the build process. The
640OpenEmbedded build system provides classes to generate different package
641types, and you specify which classes to enable through the
642:term:`PACKAGE_CLASSES`
643variable. Before placing the packages into package feeds, the build
644process validates them with generated output quality assurance checks
645through the :ref:`insane <ref-classes-insane>`
646class.
647
648The package feed area resides in the Build Directory. The directory the
649build system uses to temporarily store packages is determined by a
650combination of variables and the particular package manager in use. See
651the "Package Feeds" box in the illustration and note the information to
652the right of that area. In particular, the following defines where
653package files are kept:
654
655- :term:`DEPLOY_DIR`: Defined as
656 ``tmp/deploy`` in the Build Directory.
657
658- ``DEPLOY_DIR_*``: Depending on the package manager used, the package
659 type sub-folder. Given RPM, IPK, or DEB packaging and tarball
660 creation, the
661 :term:`DEPLOY_DIR_RPM`,
662 :term:`DEPLOY_DIR_IPK`,
663 :term:`DEPLOY_DIR_DEB`, or
664 :term:`DEPLOY_DIR_TAR`,
665 variables are used, respectively.
666
667- :term:`PACKAGE_ARCH`: Defines
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700668 architecture-specific sub-folders. For example, packages could be
669 available for the i586 or qemux86 architectures.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500670
671BitBake uses the
672:ref:`do_package_write_* <ref-tasks-package_write_deb>`
673tasks to generate packages and place them into the package holding area
674(e.g. ``do_package_write_ipk`` for IPK packages). See the
675":ref:`ref-tasks-package_write_deb`",
676":ref:`ref-tasks-package_write_ipk`",
677":ref:`ref-tasks-package_write_rpm`",
678and
679":ref:`ref-tasks-package_write_tar`"
680sections in the Yocto Project Reference Manual for additional
681information. As an example, consider a scenario where an IPK packaging
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700682manager is being used and there is package architecture support for both
683i586 and qemux86. Packages for the i586 architecture are placed in
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500684``build/tmp/deploy/ipk/i586``, while packages for the qemux86
685architecture are placed in ``build/tmp/deploy/ipk/qemux86``.
686
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500687BitBake Tool
688------------
689
690The OpenEmbedded build system uses
691:term:`BitBake` to produce images and
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500692Software Development Kits (SDKs). You can see from the :ref:`general workflow
693figure <overview-manual/concepts:openembedded build system concepts>`,
694the BitBake area consists of several functional areas. This section takes a
695closer look at each of those areas.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500696
697.. note::
698
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700699 Documentation for the BitBake tool is available separately. See the
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500700 BitBake User Manual
701 for reference material on BitBake.
702
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500703Source Fetching
704~~~~~~~~~~~~~~~
705
706The first stages of building a recipe are to fetch and unpack the source
707code:
708
709.. image:: figures/source-fetching.png
710 :align: center
711
712The :ref:`ref-tasks-fetch` and
713:ref:`ref-tasks-unpack` tasks fetch
714the source files and unpack them into the
715:term:`Build Directory`.
716
717.. note::
718
719 For every local file (e.g.
720 file://
721 ) that is part of a recipe's
722 SRC_URI
723 statement, the OpenEmbedded build system takes a checksum of the file
724 for the recipe and inserts the checksum into the signature for the
725 do_fetch
726 task. If any local file has been modified, the
727 do_fetch
728 task and all tasks that depend on it are re-executed.
729
730By default, everything is accomplished in the Build Directory, which has
731a defined structure. For additional general information on the Build
732Directory, see the ":ref:`structure-core-build`" section in
733the Yocto Project Reference Manual.
734
735Each recipe has an area in the Build Directory where the unpacked source
736code resides. The :term:`S` variable points
737to this area for a recipe's unpacked source code. The name of that
738directory for any given recipe is defined from several different
739variables. The preceding figure and the following list describe the
740Build Directory's hierarchy:
741
742- :term:`TMPDIR`: The base directory
743 where the OpenEmbedded build system performs all its work during the
744 build. The default base directory is the ``tmp`` directory.
745
746- :term:`PACKAGE_ARCH`: The
747 architecture of the built package or packages. Depending on the
748 eventual destination of the package or packages (i.e. machine
749 architecture, :term:`Build Host`, SDK, or
Andrew Geissler09036742021-06-25 14:25:14 -0500750 specific machine), :term:`PACKAGE_ARCH` varies. See the variable's
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500751 description for details.
752
753- :term:`TARGET_OS`: The operating
754 system of the target device. A typical value would be "linux" (e.g.
755 "qemux86-poky-linux").
756
757- :term:`PN`: The name of the recipe used
758 to build the package. This variable can have multiple meanings.
Andrew Geissler09036742021-06-25 14:25:14 -0500759 However, when used in the context of input files, :term:`PN` represents
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500760 the name of the recipe.
761
762- :term:`WORKDIR`: The location
763 where the OpenEmbedded build system builds a recipe (i.e. does the
764 work to create the package).
765
766 - :term:`PV`: The version of the
767 recipe used to build the package.
768
769 - :term:`PR`: The revision of the
770 recipe used to build the package.
771
772- :term:`S`: Contains the unpacked source
773 files for a given recipe.
774
775 - :term:`BPN`: The name of the recipe
Andrew Geissler09036742021-06-25 14:25:14 -0500776 used to build the package. The :term:`BPN` variable is a version of
Andrew Geissler5f350902021-07-23 13:09:54 -0400777 the :term:`PN` variable but with common prefixes and suffixes removed.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500778
779 - :term:`PV`: The version of the
780 recipe used to build the package.
781
782.. note::
783
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700784 In the previous figure, notice that there are two sample hierarchies:
785 one based on package architecture (i.e. :term:`PACKAGE_ARCH`)
786 and one based on a machine (i.e. :term:`MACHINE`).
787 The underlying structures are identical. The differentiator being
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500788 what the OpenEmbedded build system is using as a build target (e.g.
789 general architecture, a build host, an SDK, or a specific machine).
790
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500791Patching
792~~~~~~~~
793
794Once source code is fetched and unpacked, BitBake locates patch files
795and applies them to the source files:
796
797.. image:: figures/patching.png
798 :align: center
799
800The :ref:`ref-tasks-patch` task uses a
801recipe's :term:`SRC_URI` statements
802and the :term:`FILESPATH` variable
803to locate applicable patch files.
804
805Default processing for patch files assumes the files have either
Andrew Geissler09036742021-06-25 14:25:14 -0500806``*.patch`` or ``*.diff`` file types. You can use :term:`SRC_URI` parameters
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500807to change the way the build system recognizes patch files. See the
808:ref:`ref-tasks-patch` task for more
809information.
810
811BitBake finds and applies multiple patches for a single recipe in the
Andrew Geissler09036742021-06-25 14:25:14 -0500812order in which it locates the patches. The :term:`FILESPATH` variable
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500813defines the default set of directories that the build system uses to
814search for patch files. Once found, patches are applied to the recipe's
815source files, which are located in the
816:term:`S` directory.
817
818For more information on how the source directories are created, see the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500819":ref:`overview-manual/concepts:source fetching`" section. For
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500820more information on how to create patches and how the build system
821processes patches, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600822":ref:`dev-manual/common-tasks:patching code`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500823section in the
824Yocto Project Development Tasks Manual. You can also see the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600825":ref:`sdk-manual/extensible:use \`\`devtool modify\`\` to modify the source of an existing component`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500826section in the Yocto Project Application Development and the Extensible
827Software Development Kit (SDK) manual and the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600828":ref:`kernel-dev/common:using traditional kernel development to patch the kernel`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500829section in the Yocto Project Linux Kernel Development Manual.
830
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500831Configuration, Compilation, and Staging
832~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
833
834After source code is patched, BitBake executes tasks that configure and
835compile the source code. Once compilation occurs, the files are copied
836to a holding area (staged) in preparation for packaging:
837
838.. image:: figures/configuration-compile-autoreconf.png
839 :align: center
840
841This step in the build process consists of the following tasks:
842
843- :ref:`ref-tasks-prepare_recipe_sysroot`:
844 This task sets up the two sysroots in
845 ``${``\ :term:`WORKDIR`\ ``}``
846 (i.e. ``recipe-sysroot`` and ``recipe-sysroot-native``) so that
847 during the packaging phase the sysroots can contain the contents of
848 the
849 :ref:`ref-tasks-populate_sysroot`
850 tasks of the recipes on which the recipe containing the tasks
851 depends. A sysroot exists for both the target and for the native
852 binaries, which run on the host system.
853
854- *do_configure*: This task configures the source by enabling and
855 disabling any build-time and configuration options for the software
856 being built. Configurations can come from the recipe itself as well
857 as from an inherited class. Additionally, the software itself might
858 configure itself depending on the target for which it is being built.
859
860 The configurations handled by the
861 :ref:`ref-tasks-configure` task
862 are specific to configurations for the source code being built by the
863 recipe.
864
865 If you are using the
866 :ref:`autotools <ref-classes-autotools>` class,
867 you can add additional configuration options by using the
868 :term:`EXTRA_OECONF` or
869 :term:`PACKAGECONFIG_CONFARGS`
870 variables. For information on how this variable works within that
871 class, see the
872 :ref:`autotools <ref-classes-autotools>` class
Andrew Geissler09209ee2020-12-13 08:44:15 -0600873 :yocto_git:`here </poky/tree/meta/classes/autotools.bbclass>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500874
875- *do_compile*: Once a configuration task has been satisfied,
876 BitBake compiles the source using the
877 :ref:`ref-tasks-compile` task.
878 Compilation occurs in the directory pointed to by the
879 :term:`B` variable. Realize that the
Andrew Geissler09036742021-06-25 14:25:14 -0500880 :term:`B` directory is, by default, the same as the
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500881 :term:`S` directory.
882
883- *do_install*: After compilation completes, BitBake executes the
884 :ref:`ref-tasks-install` task.
Andrew Geissler09036742021-06-25 14:25:14 -0500885 This task copies files from the :term:`B` directory and places them in a
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500886 holding area pointed to by the :term:`D`
887 variable. Packaging occurs later using files from this holding
888 directory.
889
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500890Package Splitting
891~~~~~~~~~~~~~~~~~
892
893After source code is configured, compiled, and staged, the build system
894analyzes the results and splits the output into packages:
895
896.. image:: figures/analysis-for-package-splitting.png
897 :align: center
898
899The :ref:`ref-tasks-package` and
900:ref:`ref-tasks-packagedata`
901tasks combine to analyze the files found in the
902:term:`D` directory and split them into
903subsets based on available packages and files. Analysis involves the
904following as well as other items: splitting out debugging symbols,
905looking at shared library dependencies between packages, and looking at
906package relationships.
907
908The ``do_packagedata`` task creates package metadata based on the
909analysis such that the build system can generate the final packages. The
910:ref:`ref-tasks-populate_sysroot`
911task stages (copies) a subset of the files installed by the
912:ref:`ref-tasks-install` task into
913the appropriate sysroot. Working, staged, and intermediate results of
914the analysis and package splitting process use several areas:
915
916- :term:`PKGD`: The destination
917 directory (i.e. ``package``) for packages before they are split into
918 individual packages.
919
920- :term:`PKGDESTWORK`: A
921 temporary work area (i.e. ``pkgdata``) used by the ``do_package``
922 task to save package metadata.
923
924- :term:`PKGDEST`: The parent
925 directory (i.e. ``packages-split``) for packages after they have been
926 split.
927
928- :term:`PKGDATA_DIR`: A shared,
929 global-state directory that holds packaging metadata generated during
930 the packaging process. The packaging process copies metadata from
Andrew Geissler09036742021-06-25 14:25:14 -0500931 :term:`PKGDESTWORK` to the :term:`PKGDATA_DIR` area where it becomes globally
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500932 available.
933
934- :term:`STAGING_DIR_HOST`:
935 The path for the sysroot for the system on which a component is built
936 to run (i.e. ``recipe-sysroot``).
937
938- :term:`STAGING_DIR_NATIVE`:
939 The path for the sysroot used when building components for the build
940 host (i.e. ``recipe-sysroot-native``).
941
942- :term:`STAGING_DIR_TARGET`:
943 The path for the sysroot used when a component that is built to
944 execute on a system and it generates code for yet another machine
945 (e.g. cross-canadian recipes).
946
947The :term:`FILES` variable defines the
948files that go into each package in
949:term:`PACKAGES`. If you want
950details on how this is accomplished, you can look at
Andrew Geissler09209ee2020-12-13 08:44:15 -0600951:yocto_git:`package.bbclass </poky/tree/meta/classes/package.bbclass>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500952
953Depending on the type of packages being created (RPM, DEB, or IPK), the
954:ref:`do_package_write_* <ref-tasks-package_write_deb>`
955task creates the actual packages and places them in the Package Feed
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500956area, which is ``${TMPDIR}/deploy``. You can see the
957":ref:`overview-manual/concepts:package feeds`" section for more detail on
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500958that part of the build process.
959
960.. note::
961
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700962 Support for creating feeds directly from the ``deploy/*``
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500963 directories does not exist. Creating such feeds usually requires some
964 kind of feed maintenance mechanism that would upload the new packages
965 into an official package feed (e.g. the Ångström distribution). This
966 functionality is highly distribution-specific and thus is not
967 provided out of the box.
968
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500969Image Generation
970~~~~~~~~~~~~~~~~
971
972Once packages are split and stored in the Package Feeds area, the build
973system uses BitBake to generate the root filesystem image:
974
975.. image:: figures/image-generation.png
976 :align: center
977
978The image generation process consists of several stages and depends on
979several tasks and variables. The
980:ref:`ref-tasks-rootfs` task creates
981the root filesystem (file and directory structure) for an image. This
982task uses several key variables to help create the list of packages to
983actually install:
984
985- :term:`IMAGE_INSTALL`: Lists
986 out the base set of packages from which to install from the Package
987 Feeds area.
988
989- :term:`PACKAGE_EXCLUDE`:
990 Specifies packages that should not be installed into the image.
991
992- :term:`IMAGE_FEATURES`:
993 Specifies features to include in the image. Most of these features
994 map to additional packages for installation.
995
996- :term:`PACKAGE_CLASSES`:
997 Specifies the package backend (e.g. RPM, DEB, or IPK) to use and
998 consequently helps determine where to locate packages within the
999 Package Feeds area.
1000
1001- :term:`IMAGE_LINGUAS`:
1002 Determines the language(s) for which additional language support
1003 packages are installed.
1004
1005- :term:`PACKAGE_INSTALL`:
1006 The final list of packages passed to the package manager for
1007 installation into the image.
1008
1009With :term:`IMAGE_ROOTFS`
1010pointing to the location of the filesystem under construction and the
Andrew Geissler09036742021-06-25 14:25:14 -05001011:term:`PACKAGE_INSTALL` variable providing the final list of packages to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001012install, the root file system is created.
1013
1014Package installation is under control of the package manager (e.g.
1015dnf/rpm, opkg, or apt/dpkg) regardless of whether or not package
1016management is enabled for the target. At the end of the process, if
1017package management is not enabled for the target, the package manager's
1018data files are deleted from the root filesystem. As part of the final
1019stage of package installation, post installation scripts that are part
1020of the packages are run. Any scripts that fail to run on the build host
1021are run on the target when the target system is first booted. If you are
1022using a
Andrew Geissler09209ee2020-12-13 08:44:15 -06001023:ref:`read-only root filesystem <dev-manual/common-tasks:creating a read-only root filesystem>`,
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001024all the post installation scripts must succeed on the build host during
1025the package installation phase since the root filesystem on the target
1026is read-only.
1027
1028The final stages of the ``do_rootfs`` task handle post processing. Post
1029processing includes creation of a manifest file and optimizations.
1030
1031The manifest file (``.manifest``) resides in the same directory as the
1032root filesystem image. This file lists out, line-by-line, the installed
1033packages. The manifest file is useful for the
1034:ref:`testimage <ref-classes-testimage*>` class,
1035for example, to determine whether or not to run specific tests. See the
1036:term:`IMAGE_MANIFEST`
1037variable for additional information.
1038
1039Optimizing processes that are run across the image include ``mklibs``,
1040``prelink``, and any other post-processing commands as defined by the
1041:term:`ROOTFS_POSTPROCESS_COMMAND`
1042variable. The ``mklibs`` process optimizes the size of the libraries,
1043while the ``prelink`` process optimizes the dynamic linking of shared
1044libraries to reduce start up time of executables.
1045
1046After the root filesystem is built, processing begins on the image
1047through the :ref:`ref-tasks-image`
1048task. The build system runs any pre-processing commands as defined by
1049the
1050:term:`IMAGE_PREPROCESS_COMMAND`
1051variable. This variable specifies a list of functions to call before the
1052build system creates the final image output files.
1053
1054The build system dynamically creates ``do_image_*`` tasks as needed,
1055based on the image types specified in the
1056:term:`IMAGE_FSTYPES` variable.
1057The process turns everything into an image file or a set of image files
1058and can compress the root filesystem image to reduce the overall size of
1059the image. The formats used for the root filesystem depend on the
Andrew Geissler09036742021-06-25 14:25:14 -05001060:term:`IMAGE_FSTYPES` variable. Compression depends on whether the formats
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001061support compression.
1062
1063As an example, a dynamically created task when creating a particular
Andrew Geisslerc926e172021-05-07 16:11:35 -05001064image type would take the following form::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001065
1066 do_image_type
1067
1068So, if the type
Andrew Geissler09036742021-06-25 14:25:14 -05001069as specified by the :term:`IMAGE_FSTYPES` were ``ext4``, the dynamically
Andrew Geisslerc926e172021-05-07 16:11:35 -05001070generated task would be as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001071
1072 do_image_ext4
1073
1074The final task involved in image creation is the
1075:ref:`do_image_complete <ref-tasks-image-complete>`
1076task. This task completes the image by applying any image post
1077processing as defined through the
1078:term:`IMAGE_POSTPROCESS_COMMAND`
1079variable. The variable specifies a list of functions to call once the
1080build system has created the final image output files.
1081
1082.. note::
1083
1084 The entire image generation process is run under
1085 Pseudo. Running under Pseudo ensures that the files in the root filesystem
1086 have correct ownership.
1087
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001088SDK Generation
1089~~~~~~~~~~~~~~
1090
1091The OpenEmbedded build system uses BitBake to generate the Software
1092Development Kit (SDK) installer scripts for both the standard SDK and
1093the extensible SDK (eSDK):
1094
1095.. image:: figures/sdk-generation.png
1096 :align: center
1097
1098.. note::
1099
1100 For more information on the cross-development toolchain generation,
Andrew Geissler09209ee2020-12-13 08:44:15 -06001101 see the ":ref:`overview-manual/concepts:cross-development toolchain generation`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001102 section. For information on advantages gained when building a
1103 cross-development toolchain using the do_populate_sdk task, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001104 ":ref:`sdk-manual/appendix-obtain:building an sdk installer`" section in
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001105 the Yocto Project Application Development and the Extensible Software
1106 Development Kit (eSDK) manual.
1107
1108Like image generation, the SDK script process consists of several stages
1109and depends on many variables. The
1110:ref:`ref-tasks-populate_sdk`
1111and
1112:ref:`ref-tasks-populate_sdk_ext`
1113tasks use these key variables to help create the list of packages to
1114actually install. For information on the variables listed in the figure,
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001115see the ":ref:`overview-manual/concepts:application development sdk`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001116section.
1117
1118The ``do_populate_sdk`` task helps create the standard SDK and handles
1119two parts: a target part and a host part. The target part is the part
1120built for the target hardware and includes libraries and headers. The
1121host part is the part of the SDK that runs on the
1122:term:`SDKMACHINE`.
1123
1124The ``do_populate_sdk_ext`` task helps create the extensible SDK and
1125handles host and target parts differently than its counter part does for
1126the standard SDK. For the extensible SDK, the task encapsulates the
1127build system, which includes everything needed (host and target) for the
1128SDK.
1129
1130Regardless of the type of SDK being constructed, the tasks perform some
1131cleanup after which a cross-development environment setup script and any
1132needed configuration files are created. The final output is the
1133Cross-development toolchain installation script (``.sh`` file), which
1134includes the environment setup script.
1135
1136Stamp Files and the Rerunning of Tasks
1137~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1138
1139For each task that completes successfully, BitBake writes a stamp file
1140into the :term:`STAMPS_DIR`
1141directory. The beginning of the stamp file's filename is determined by
1142the :term:`STAMP` variable, and the end
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001143of the name consists of the task's name and current :ref:`input
1144checksum <overview-manual/concepts:checksums (signatures)>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001145
1146.. note::
1147
1148 This naming scheme assumes that
1149 BB_SIGNATURE_HANDLER
1150 is "OEBasicHash", which is almost always the case in current
1151 OpenEmbedded.
1152
1153To determine if a task needs to be rerun, BitBake checks if a stamp file
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001154with a matching input checksum exists for the task. In this case,
1155the task's output is assumed to exist and still be valid. Otherwise,
1156the task is rerun.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001157
1158.. note::
1159
1160 The stamp mechanism is more general than the shared state (sstate)
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001161 cache mechanism described in the
1162 ":ref:`overview-manual/concepts:setscene tasks and shared state`" section.
1163 BitBake avoids rerunning any task that has a valid stamp file, not just
1164 tasks that can be accelerated through the sstate cache.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001165
1166 However, you should realize that stamp files only serve as a marker
1167 that some work has been done and that these files do not record task
1168 output. The actual task output would usually be somewhere in
1169 :term:`TMPDIR` (e.g. in some
1170 recipe's :term:`WORKDIR`.) What
1171 the sstate cache mechanism adds is a way to cache task output that
1172 can then be shared between build machines.
1173
Andrew Geissler09036742021-06-25 14:25:14 -05001174Since :term:`STAMPS_DIR` is usually a subdirectory of :term:`TMPDIR`, removing
1175:term:`TMPDIR` will also remove :term:`STAMPS_DIR`, which means tasks will
1176properly be rerun to repopulate :term:`TMPDIR`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001177
1178If you want some task to always be considered "out of date", you can
1179mark it with the :ref:`nostamp <bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`
1180varflag. If some other task depends on such a task, then that task will
1181also always be considered out of date, which might not be what you want.
1182
1183For details on how to view information about a task's signature, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001184":ref:`dev-manual/common-tasks:viewing task variable dependencies`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001185section in the Yocto Project Development Tasks Manual.
1186
1187Setscene Tasks and Shared State
1188~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1189
1190The description of tasks so far assumes that BitBake needs to build
1191everything and no available prebuilt objects exist. BitBake does support
1192skipping tasks if prebuilt objects are available. These objects are
1193usually made available in the form of a shared state (sstate) cache.
1194
1195.. note::
1196
1197 For information on variables affecting sstate, see the
1198 :term:`SSTATE_DIR`
1199 and
1200 :term:`SSTATE_MIRRORS`
1201 variables.
1202
1203The idea of a setscene task (i.e ``do_``\ taskname\ ``_setscene``) is a
1204version of the task where instead of building something, BitBake can
1205skip to the end result and simply place a set of files into specific
1206locations as needed. In some cases, it makes sense to have a setscene
1207task variant (e.g. generating package files in the
1208:ref:`do_package_write_* <ref-tasks-package_write_deb>`
1209task). In other cases, it does not make sense (e.g. a
1210:ref:`ref-tasks-patch` task or a
1211:ref:`ref-tasks-unpack` task) since
1212the work involved would be equal to or greater than the underlying task.
1213
1214In the build system, the common tasks that have setscene variants are
1215:ref:`ref-tasks-package`,
1216``do_package_write_*``,
1217:ref:`ref-tasks-deploy`,
1218:ref:`ref-tasks-packagedata`, and
1219:ref:`ref-tasks-populate_sysroot`.
1220Notice that these tasks represent most of the tasks whose output is an
1221end result.
1222
1223The build system has knowledge of the relationship between these tasks
1224and other preceding tasks. For example, if BitBake runs
1225``do_populate_sysroot_setscene`` for something, it does not make sense
1226to run any of the ``do_fetch``, ``do_unpack``, ``do_patch``,
1227``do_configure``, ``do_compile``, and ``do_install`` tasks. However, if
1228``do_package`` needs to be run, BitBake needs to run those other tasks.
1229
1230It becomes more complicated if everything can come from an sstate cache
1231because some objects are simply not required at all. For example, you do
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001232not need a compiler or native tools, such as quilt, if there isn't anything
1233to compile or patch. If the ``do_package_write_*`` packages are available
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001234from sstate, BitBake does not need the ``do_package`` task data.
1235
1236To handle all these complexities, BitBake runs in two phases. The first
1237is the "setscene" stage. During this stage, BitBake first checks the
1238sstate cache for any targets it is planning to build. BitBake does a
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001239fast check to see if the object exists rather than doing a complete download.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001240If nothing exists, the second phase, which is the setscene stage,
1241completes and the main build proceeds.
1242
1243If objects are found in the sstate cache, the build system works
1244backwards from the end targets specified by the user. For example, if an
1245image is being built, the build system first looks for the packages
1246needed for that image and the tools needed to construct an image. If
1247those are available, the compiler is not needed. Thus, the compiler is
1248not even downloaded. If something was found to be unavailable, or the
1249download or setscene task fails, the build system then tries to install
1250dependencies, such as the compiler, from the cache.
1251
1252The availability of objects in the sstate cache is handled by the
1253function specified by the
1254:term:`bitbake:BB_HASHCHECK_FUNCTION`
1255variable and returns a list of available objects. The function specified
1256by the
1257:term:`bitbake:BB_SETSCENE_DEPVALID`
1258variable is the function that determines whether a given dependency
1259needs to be followed, and whether for any given relationship the
1260function needs to be passed. The function returns a True or False value.
1261
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001262Images
1263------
1264
1265The images produced by the build system are compressed forms of the root
1266filesystem and are ready to boot on a target device. You can see from
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001267the :ref:`general workflow figure
1268<overview-manual/concepts:openembedded build system concepts>` that BitBake
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001269output, in part, consists of images. This section takes a closer look at
1270this output:
1271
1272.. image:: figures/images.png
1273 :align: center
1274
1275.. note::
1276
1277 For a list of example images that the Yocto Project provides, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001278 ":doc:`/ref-manual/images`" chapter in the Yocto Project Reference
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001279 Manual.
1280
1281The build process writes images out to the :term:`Build Directory`
1282inside the
1283``tmp/deploy/images/machine/`` folder as shown in the figure. This
1284folder contains any files expected to be loaded on the target device.
1285The :term:`DEPLOY_DIR` variable
1286points to the ``deploy`` directory, while the
1287:term:`DEPLOY_DIR_IMAGE`
1288variable points to the appropriate directory containing images for the
1289current configuration.
1290
1291- kernel-image: A kernel binary file. The
1292 :term:`KERNEL_IMAGETYPE`
1293 variable determines the naming scheme for the kernel image file.
1294 Depending on this variable, the file could begin with a variety of
1295 naming strings. The ``deploy/images/``\ machine directory can contain
1296 multiple image files for the machine.
1297
1298- root-filesystem-image: Root filesystems for the target device (e.g.
1299 ``*.ext3`` or ``*.bz2`` files). The
1300 :term:`IMAGE_FSTYPES`
1301 variable determines the root filesystem image type. The
1302 ``deploy/images/``\ machine directory can contain multiple root
1303 filesystems for the machine.
1304
1305- kernel-modules: Tarballs that contain all the modules built for the
1306 kernel. Kernel module tarballs exist for legacy purposes and can be
1307 suppressed by setting the
1308 :term:`MODULE_TARBALL_DEPLOY`
1309 variable to "0". The ``deploy/images/``\ machine directory can
1310 contain multiple kernel module tarballs for the machine.
1311
1312- bootloaders: If applicable to the target machine, bootloaders
1313 supporting the image. The ``deploy/images/``\ machine directory can
1314 contain multiple bootloaders for the machine.
1315
1316- symlinks: The ``deploy/images/``\ machine folder contains a symbolic
1317 link that points to the most recently built file for each machine.
1318 These links might be useful for external scripts that need to obtain
1319 the latest version of each file.
1320
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001321Application Development SDK
1322---------------------------
1323
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001324In the :ref:`general workflow figure
1325<overview-manual/concepts:openembedded build system concepts>`, the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001326output labeled "Application Development SDK" represents an SDK. The SDK
1327generation process differs depending on whether you build an extensible
1328SDK (e.g. ``bitbake -c populate_sdk_ext`` imagename) or a standard SDK
1329(e.g. ``bitbake -c populate_sdk`` imagename). This section takes a
1330closer look at this output:
1331
1332.. image:: figures/sdk.png
1333 :align: center
1334
1335The specific form of this output is a set of files that includes a
1336self-extracting SDK installer (``*.sh``), host and target manifest
1337files, and files used for SDK testing. When the SDK installer file is
1338run, it installs the SDK. The SDK consists of a cross-development
1339toolchain, a set of libraries and headers, and an SDK environment setup
1340script. Running this installer essentially sets up your
1341cross-development environment. You can think of the cross-toolchain as
1342the "host" part because it runs on the SDK machine. You can think of the
1343libraries and headers as the "target" part because they are built for
1344the target hardware. The environment setup script is added so that you
1345can initialize the environment before using the tools.
1346
1347.. note::
1348
1349 - The Yocto Project supports several methods by which you can set up
1350 this cross-development environment. These methods include
1351 downloading pre-built SDK installers or building and installing
1352 your own SDK installer.
1353
1354 - For background information on cross-development toolchains in the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001355 Yocto Project development environment, see the
1356 ":ref:`overview-manual/concepts:cross-development toolchain generation`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001357 section.
1358
1359 - For information on setting up a cross-development environment, see
Andrew Geissler09209ee2020-12-13 08:44:15 -06001360 the :doc:`/sdk-manual/index` manual.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001361
1362All the output files for an SDK are written to the ``deploy/sdk`` folder
1363inside the :term:`Build Directory` as
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001364shown in the previous figure. Depending on the type of SDK, there are
1365several variables to configure these files. Here are the variables
1366associated with an extensible SDK:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001367
1368- :term:`DEPLOY_DIR`: Points to
1369 the ``deploy`` directory.
1370
1371- :term:`SDK_EXT_TYPE`:
1372 Controls whether or not shared state artifacts are copied into the
1373 extensible SDK. By default, all required shared state artifacts are
1374 copied into the SDK.
1375
1376- :term:`SDK_INCLUDE_PKGDATA`:
1377 Specifies whether or not packagedata is included in the extensible
1378 SDK for all recipes in the "world" target.
1379
1380- :term:`SDK_INCLUDE_TOOLCHAIN`:
1381 Specifies whether or not the toolchain is included when building the
1382 extensible SDK.
1383
1384- :term:`SDK_LOCAL_CONF_WHITELIST`:
1385 A list of variables allowed through from the build system
1386 configuration into the extensible SDK configuration.
1387
1388- :term:`SDK_LOCAL_CONF_BLACKLIST`:
1389 A list of variables not allowed through from the build system
1390 configuration into the extensible SDK configuration.
1391
1392- :term:`SDK_INHERIT_BLACKLIST`:
1393 A list of classes to remove from the
1394 :term:`INHERIT` value globally
1395 within the extensible SDK configuration.
1396
1397This next list, shows the variables associated with a standard SDK:
1398
1399- :term:`DEPLOY_DIR`: Points to
1400 the ``deploy`` directory.
1401
1402- :term:`SDKMACHINE`: Specifies
1403 the architecture of the machine on which the cross-development tools
1404 are run to create packages for the target hardware.
1405
1406- :term:`SDKIMAGE_FEATURES`:
1407 Lists the features to include in the "target" part of the SDK.
1408
1409- :term:`TOOLCHAIN_HOST_TASK`:
1410 Lists packages that make up the host part of the SDK (i.e. the part
Andrew Geissler09036742021-06-25 14:25:14 -05001411 that runs on the :term:`SDKMACHINE`). When you use
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001412 ``bitbake -c populate_sdk imagename`` to create the SDK, a set of
1413 default packages apply. This variable allows you to add more
1414 packages.
1415
1416- :term:`TOOLCHAIN_TARGET_TASK`:
1417 Lists packages that make up the target part of the SDK (i.e. the part
1418 built for the target hardware).
1419
1420- :term:`SDKPATH`: Defines the
1421 default SDK installation path offered by the installation script.
1422
1423- :term:`SDK_HOST_MANIFEST`:
1424 Lists all the installed packages that make up the host part of the
1425 SDK. This variable also plays a minor role for extensible SDK
1426 development as well. However, it is mainly used for the standard SDK.
1427
1428- :term:`SDK_TARGET_MANIFEST`:
1429 Lists all the installed packages that make up the target part of the
1430 SDK. This variable also plays a minor role for extensible SDK
1431 development as well. However, it is mainly used for the standard SDK.
1432
1433Cross-Development Toolchain Generation
1434======================================
1435
1436The Yocto Project does most of the work for you when it comes to
Andrew Geissler09209ee2020-12-13 08:44:15 -06001437creating :ref:`sdk-manual/intro:the cross-development toolchain`. This
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001438section provides some technical background on how cross-development
1439toolchains are created and used. For more information on toolchains, you
Andrew Geissler09209ee2020-12-13 08:44:15 -06001440can also see the :doc:`/sdk-manual/index` manual.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001441
1442In the Yocto Project development environment, cross-development
1443toolchains are used to build images and applications that run on the
1444target hardware. With just a few commands, the OpenEmbedded build system
1445creates these necessary toolchains for you.
1446
1447The following figure shows a high-level build environment regarding
1448toolchain construction and use.
1449
1450.. image:: figures/cross-development-toolchains.png
1451 :align: center
1452
1453Most of the work occurs on the Build Host. This is the machine used to
1454build images and generally work within the the Yocto Project
1455environment. When you run
1456:term:`BitBake` to create an image, the
1457OpenEmbedded build system uses the host ``gcc`` compiler to bootstrap a
1458cross-compiler named ``gcc-cross``. The ``gcc-cross`` compiler is what
1459BitBake uses to compile source files when creating the target image. You
1460can think of ``gcc-cross`` simply as an automatically generated
1461cross-compiler that is used internally within BitBake only.
1462
1463.. note::
1464
Andrew Geissler09036742021-06-25 14:25:14 -05001465 The extensible SDK does not use ``gcc-cross-canadian``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001466 since this SDK ships a copy of the OpenEmbedded build system and the
Andrew Geissler09036742021-06-25 14:25:14 -05001467 sysroot within it contains ``gcc-cross``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001468
Andrew Geisslerc926e172021-05-07 16:11:35 -05001469The chain of events that occurs when the standard toolchain is bootstrapped::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001470
Andrew Geissler6ce62a22020-11-30 19:58:47 -06001471 binutils-cross -> linux-libc-headers -> gcc-cross -> libgcc-initial -> glibc -> libgcc -> gcc-runtime
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001472
Andrew Geissler6ce62a22020-11-30 19:58:47 -06001473- ``gcc``: The compiler, GNU Compiler Collection (GCC).
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001474
Andrew Geissler6ce62a22020-11-30 19:58:47 -06001475- ``binutils-cross``: The binary utilities needed in order
1476 to run the ``gcc-cross`` phase of the bootstrap operation and build the
1477 headers for the C library.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001478
Andrew Geissler6ce62a22020-11-30 19:58:47 -06001479- ``linux-libc-headers``: Headers needed for the cross-compiler and C library build.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001480
Andrew Geissler6ce62a22020-11-30 19:58:47 -06001481- ``libgcc-initial``: An initial version of the gcc support library needed
1482 to bootstrap ``glibc``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001483
Andrew Geissler6ce62a22020-11-30 19:58:47 -06001484- ``libgcc``: The final version of the gcc support library which
1485 can only be built once there is a C library to link against.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001486
1487- ``glibc``: The GNU C Library.
1488
1489- ``gcc-cross``: The final stage of the bootstrap process for the
1490 cross-compiler. This stage results in the actual cross-compiler that
1491 BitBake uses when it builds an image for a targeted device.
1492
Andrew Geissler6ce62a22020-11-30 19:58:47 -06001493 This tool is a "native" tool (i.e. it is designed to run on
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001494 the build host).
1495
1496- ``gcc-runtime``: Runtime libraries resulting from the toolchain
1497 bootstrapping process. This tool produces a binary that consists of
1498 the runtime libraries need for the targeted device.
1499
1500You can use the OpenEmbedded build system to build an installer for the
1501relocatable SDK used to develop applications. When you run the
1502installer, it installs the toolchain, which contains the development
1503tools (e.g., ``gcc-cross-canadian``, ``binutils-cross-canadian``, and
1504other ``nativesdk-*`` tools), which are tools native to the SDK (i.e.
1505native to :term:`SDK_ARCH`), you
1506need to cross-compile and test your software. The figure shows the
1507commands you use to easily build out this toolchain. This
1508cross-development toolchain is built to execute on the
1509:term:`SDKMACHINE`, which might or
1510might not be the same machine as the Build Host.
1511
1512.. note::
1513
1514 If your target architecture is supported by the Yocto Project, you
1515 can take advantage of pre-built images that ship with the Yocto
1516 Project and already contain cross-development toolchain installers.
1517
Andrew Geisslerc926e172021-05-07 16:11:35 -05001518Here is the bootstrap process for the relocatable toolchain::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001519
1520 gcc -> binutils-crosssdk -> gcc-crosssdk-initial -> linux-libc-headers -> glibc-initial -> nativesdk-glibc -> gcc-crosssdk -> gcc-cross-canadian
1521
1522- ``gcc``: The build host's GNU Compiler Collection (GCC).
1523
1524- ``binutils-crosssdk``: The bare minimum binary utilities needed in
1525 order to run the ``gcc-crosssdk-initial`` phase of the bootstrap
1526 operation.
1527
1528- ``gcc-crosssdk-initial``: An early stage of the bootstrap process for
1529 creating the cross-compiler. This stage builds enough of the
1530 ``gcc-crosssdk`` and supporting pieces so that the final stage of the
1531 bootstrap process can produce the finished cross-compiler. This tool
1532 is a "native" binary that runs on the build host.
1533
1534- ``linux-libc-headers``: Headers needed for the cross-compiler.
1535
1536- ``glibc-initial``: An initial version of the Embedded GLIBC needed to
1537 bootstrap ``nativesdk-glibc``.
1538
1539- ``nativesdk-glibc``: The Embedded GLIBC needed to bootstrap the
1540 ``gcc-crosssdk``.
1541
1542- ``gcc-crosssdk``: The final stage of the bootstrap process for the
1543 relocatable cross-compiler. The ``gcc-crosssdk`` is a transitory
1544 compiler and never leaves the build host. Its purpose is to help in
1545 the bootstrap process to create the eventual ``gcc-cross-canadian``
1546 compiler, which is relocatable. This tool is also a "native" package
1547 (i.e. it is designed to run on the build host).
1548
1549- ``gcc-cross-canadian``: The final relocatable cross-compiler. When
1550 run on the :term:`SDKMACHINE`,
1551 this tool produces executable code that runs on the target device.
1552 Only one cross-canadian compiler is produced per architecture since
1553 they can be targeted at different processor optimizations using
1554 configurations passed to the compiler through the compile commands.
1555 This circumvents the need for multiple compilers and thus reduces the
1556 size of the toolchains.
1557
1558.. note::
1559
1560 For information on advantages gained when building a
1561 cross-development toolchain installer, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001562 ":ref:`sdk-manual/appendix-obtain:building an sdk installer`" appendix
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001563 in the Yocto Project Application Development and the
1564 Extensible Software Development Kit (eSDK) manual.
1565
1566Shared State Cache
1567==================
1568
1569By design, the OpenEmbedded build system builds everything from scratch
1570unless :term:`BitBake` can determine
1571that parts do not need to be rebuilt. Fundamentally, building from
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001572scratch is attractive as it means all parts are built fresh and there is
1573no possibility of stale data that can cause problems. When
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001574developers hit problems, they typically default back to building from
1575scratch so they have a know state from the start.
1576
1577Building an image from scratch is both an advantage and a disadvantage
1578to the process. As mentioned in the previous paragraph, building from
1579scratch ensures that everything is current and starts from a known
1580state. However, building from scratch also takes much longer as it
1581generally means rebuilding things that do not necessarily need to be
1582rebuilt.
1583
1584The Yocto Project implements shared state code that supports incremental
1585builds. The implementation of the shared state code answers the
1586following questions that were fundamental roadblocks within the
1587OpenEmbedded incremental build support system:
1588
1589- What pieces of the system have changed and what pieces have not
1590 changed?
1591
1592- How are changed pieces of software removed and replaced?
1593
1594- How are pre-built components that do not need to be rebuilt from
1595 scratch used when they are available?
1596
1597For the first question, the build system detects changes in the "inputs"
1598to a given task by creating a checksum (or signature) of the task's
1599inputs. If the checksum changes, the system assumes the inputs have
1600changed and the task needs to be rerun. For the second question, the
1601shared state (sstate) code tracks which tasks add which output to the
1602build process. This means the output from a given task can be removed,
1603upgraded or otherwise manipulated. The third question is partly
1604addressed by the solution for the second question assuming the build
1605system can fetch the sstate objects from remote locations and install
1606them if they are deemed to be valid.
1607
1608.. note::
1609
1610 - The build system does not maintain
1611 :term:`PR` information as part of
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001612 the shared state packages. Consequently, there are considerations that
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001613 affect maintaining shared state feeds. For information on how the
Andrew Geissler09036742021-06-25 14:25:14 -05001614 build system works with packages and can track incrementing :term:`PR`
Andrew Geissler09209ee2020-12-13 08:44:15 -06001615 information, see the ":ref:`dev-manual/common-tasks:automatically incrementing a package version number`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001616 section in the Yocto Project Development Tasks Manual.
1617
1618 - The code in the build system that supports incremental builds is
1619 not simple code. For techniques that help you work around issues
1620 related to shared state code, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001621 ":ref:`dev-manual/common-tasks:viewing metadata used to create the input signature of a shared state task`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001622 and
Andrew Geissler09209ee2020-12-13 08:44:15 -06001623 ":ref:`dev-manual/common-tasks:invalidating shared state to force a task to run`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001624 sections both in the Yocto Project Development Tasks Manual.
1625
1626The rest of this section goes into detail about the overall incremental
1627build architecture, the checksums (signatures), and shared state.
1628
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001629Overall Architecture
1630--------------------
1631
1632When determining what parts of the system need to be built, BitBake
1633works on a per-task basis rather than a per-recipe basis. You might
1634wonder why using a per-task basis is preferred over a per-recipe basis.
1635To help explain, consider having the IPK packaging backend enabled and
1636then switching to DEB. In this case, the
1637:ref:`ref-tasks-install` and
1638:ref:`ref-tasks-package` task outputs
1639are still valid. However, with a per-recipe approach, the build would
1640not include the ``.deb`` files. Consequently, you would have to
1641invalidate the whole build and rerun it. Rerunning everything is not the
1642best solution. Also, in this case, the core must be "taught" much about
1643specific tasks. This methodology does not scale well and does not allow
1644users to easily add new tasks in layers or as external recipes without
1645touching the packaged-staging core.
1646
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001647Checksums (Signatures)
1648----------------------
1649
1650The shared state code uses a checksum, which is a unique signature of a
1651task's inputs, to determine if a task needs to be run again. Because it
1652is a change in a task's inputs that triggers a rerun, the process needs
1653to detect all the inputs to a given task. For shell tasks, this turns
1654out to be fairly easy because the build process generates a "run" shell
1655script for each task and it is possible to create a checksum that gives
1656you a good idea of when the task's data changes.
1657
1658To complicate the problem, there are things that should not be included
1659in the checksum. First, there is the actual specific build path of a
1660given task - the :term:`WORKDIR`. It
1661does not matter if the work directory changes because it should not
1662affect the output for target packages. Also, the build process has the
1663objective of making native or cross packages relocatable.
1664
1665.. note::
1666
1667 Both native and cross packages run on the
1668 build host. However, cross packages generate output for the target
1669 architecture.
1670
Andrew Geissler09036742021-06-25 14:25:14 -05001671The checksum therefore needs to exclude :term:`WORKDIR`. The simplistic
1672approach for excluding the work directory is to set :term:`WORKDIR` to some
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001673fixed value and create the checksum for the "run" script.
1674
1675Another problem results from the "run" scripts containing functions that
1676might or might not get called. The incremental build solution contains
1677code that figures out dependencies between shell functions. This code is
1678used to prune the "run" scripts down to the minimum set, thereby
1679alleviating this problem and making the "run" scripts much more readable
1680as a bonus.
1681
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001682So far, there are solutions for shell scripts. What about Python tasks? The
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001683same approach applies even though these tasks are more difficult. The
1684process needs to figure out what variables a Python function accesses
1685and what functions it calls. Again, the incremental build solution
1686contains code that first figures out the variable and function
1687dependencies, and then creates a checksum for the data used as the input
1688to the task.
1689
Andrew Geissler09036742021-06-25 14:25:14 -05001690Like the :term:`WORKDIR` case, there can be situations where dependencies should be
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001691ignored. For these situations, you can instruct the build process to
Andrew Geisslerc926e172021-05-07 16:11:35 -05001692ignore a dependency by using a line like the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001693
1694 PACKAGE_ARCHS[vardepsexclude] = "MACHINE"
1695
1696This example ensures that the :term:`PACKAGE_ARCHS` variable
1697does not depend on the value of :term:`MACHINE`, even if it does
1698reference it.
1699
1700Equally, there are cases where you need to add dependencies BitBake is
1701not able to find. You can accomplish this by using a line like the
Andrew Geisslerc926e172021-05-07 16:11:35 -05001702following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001703
1704 PACKAGE_ARCHS[vardeps] = "MACHINE"
1705
1706This example explicitly
Andrew Geissler09036742021-06-25 14:25:14 -05001707adds the :term:`MACHINE` variable as a dependency for :term:`PACKAGE_ARCHS`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001708
1709As an example, consider a case with in-line Python where BitBake is not
1710able to figure out dependencies. When running in debug mode (i.e. using
1711``-DDD``), BitBake produces output when it discovers something for which
1712it cannot figure out dependencies. The Yocto Project team has currently
1713not managed to cover those dependencies in detail and is aware of the
1714need to fix this situation.
1715
1716Thus far, this section has limited discussion to the direct inputs into
1717a task. Information based on direct inputs is referred to as the
1718"basehash" in the code. However, the question of a task's indirect
1719inputs still exits - items already built and present in the
1720:term:`Build Directory`. The checksum (or
1721signature) for a particular task needs to add the hashes of all the
1722tasks on which the particular task depends. Choosing which dependencies
1723to add is a policy decision. However, the effect is to generate a master
1724checksum that combines the basehash and the hashes of the task's
1725dependencies.
1726
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001727At the code level, there are multiple ways by which both the basehash
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001728and the dependent task hashes can be influenced. Within the BitBake
1729configuration file, you can give BitBake some extra information to help
1730it construct the basehash. The following statement effectively results
1731in a list of global variable dependency excludes (i.e. variables never
Andrew Geisslerc926e172021-05-07 16:11:35 -05001732included in any checksum)::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001733
1734 BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \\
1735 SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \\
1736 USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \\
1737 PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \\
1738 CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX"
1739
1740The
1741previous example excludes
1742:term:`WORKDIR` since that variable
1743is actually constructed as a path within
1744:term:`TMPDIR`, which is on the
1745whitelist.
1746
1747The rules for deciding which hashes of dependent tasks to include
1748through dependency chains are more complex and are generally
1749accomplished with a Python function. The code in
1750``meta/lib/oe/sstatesig.py`` shows two examples of this and also
1751illustrates how you can insert your own policy into the system if so
1752desired. This file defines the two basic signature generators
1753:term:`OpenEmbedded-Core (OE-Core)` uses: "OEBasic" and
1754"OEBasicHash". By default, a dummy "noop" signature handler is enabled
1755in BitBake. This means that behavior is unchanged from previous
1756versions. OE-Core uses the "OEBasicHash" signature handler by default
Andrew Geisslerc926e172021-05-07 16:11:35 -05001757through this setting in the ``bitbake.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001758
1759 BB_SIGNATURE_HANDLER ?= "OEBasicHash"
1760
Andrew Geissler09036742021-06-25 14:25:14 -05001761The "OEBasicHash" :term:`BB_SIGNATURE_HANDLER` is the same
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001762as the "OEBasic" version but adds the task hash to the :ref:`stamp
1763files <overview-manual/concepts:stamp files and the rerunning of tasks>`. This
1764results in any metadata change that changes the task hash, automatically causing
1765the task to be run again. This removes the need to bump
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001766:term:`PR` values, and changes to metadata
1767automatically ripple across the build.
1768
1769It is also worth noting that the end result of these signature
1770generators is to make some dependency and hash information available to
1771the build. This information includes:
1772
1773- ``BB_BASEHASH_task-``\ taskname: The base hashes for each task in the
1774 recipe.
1775
1776- ``BB_BASEHASH_``\ filename\ ``:``\ taskname: The base hashes for each
1777 dependent task.
1778
1779- ``BBHASHDEPS_``\ filename\ ``:``\ taskname: The task dependencies for
1780 each task.
1781
Andrew Geissler09036742021-06-25 14:25:14 -05001782- :term:`BB_TASKHASH`: The hash of the currently running task.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001783
1784Shared State
1785------------
1786
1787Checksums and dependencies, as discussed in the previous section, solve
1788half the problem of supporting a shared state. The other half of the
1789problem is being able to use checksum information during the build and
1790being able to reuse or rebuild specific components.
1791
1792The :ref:`sstate <ref-classes-sstate>` class is a
1793relatively generic implementation of how to "capture" a snapshot of a
1794given task. The idea is that the build process does not care about the
1795source of a task's output. Output could be freshly built or it could be
1796downloaded and unpacked from somewhere. In other words, the build
1797process does not need to worry about its origin.
1798
1799Two types of output exist. One type is just about creating a directory
1800in :term:`WORKDIR`. A good example is
1801the output of either
1802:ref:`ref-tasks-install` or
1803:ref:`ref-tasks-package`. The other
1804type of output occurs when a set of data is merged into a shared
1805directory tree such as the sysroot.
1806
1807The Yocto Project team has tried to keep the details of the
1808implementation hidden in ``sstate`` class. From a user's perspective,
1809adding shared state wrapping to a task is as simple as this
1810:ref:`ref-tasks-deploy` example taken
Andrew Geisslerc926e172021-05-07 16:11:35 -05001811from the :ref:`deploy <ref-classes-deploy>` class::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001812
1813 DEPLOYDIR = "${WORKDIR}/deploy-${PN}"
1814 SSTATETASKS += "do_deploy"
1815 do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"
1816 do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
1817
1818 python do_deploy_setscene () {
1819 sstate_setscene(d)
1820 }
1821 addtask do_deploy_setscene
1822 do_deploy[dirs] = "${DEPLOYDIR} ${B}"
1823 do_deploy[stamp-extra-info] = "${MACHINE_ARCH}"
1824
1825The following list explains the previous example:
1826
1827- Adding "do_deploy" to ``SSTATETASKS`` adds some required
1828 sstate-related processing, which is implemented in the
1829 :ref:`sstate <ref-classes-sstate>` class, to
1830 before and after the
1831 :ref:`ref-tasks-deploy` task.
1832
1833- The ``do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"`` declares that
1834 ``do_deploy`` places its output in ``${DEPLOYDIR}`` when run normally
1835 (i.e. when not using the sstate cache). This output becomes the input
1836 to the shared state cache.
1837
1838- The ``do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"`` line
1839 causes the contents of the shared state cache to be copied to
1840 ``${DEPLOY_DIR_IMAGE}``.
1841
1842 .. note::
1843
1844 If ``do_deploy`` is not already in the shared state cache or if its input
1845 checksum (signature) has changed from when the output was cached, the task
1846 runs to populate the shared state cache, after which the contents of the
1847 shared state cache is copied to ${:term:`DEPLOY_DIR_IMAGE`}. If
1848 ``do_deploy`` is in the shared state cache and its signature indicates
1849 that the cached output is still valid (i.e. if no relevant task inputs
1850 have changed), then the contents of the shared state cache copies
Andrew Geissler09036742021-06-25 14:25:14 -05001851 directly to ${:term:`DEPLOY_DIR_IMAGE`} by the ``do_deploy_setscene`` task
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001852 instead, skipping the ``do_deploy`` task.
1853
1854- The following task definition is glue logic needed to make the
Andrew Geisslerc926e172021-05-07 16:11:35 -05001855 previous settings effective::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001856
1857 python do_deploy_setscene () {
1858 sstate_setscene(d)
1859 }
1860 addtask do_deploy_setscene
1861
1862 ``sstate_setscene()`` takes the flags above as input and accelerates the ``do_deploy`` task
1863 through the shared state cache if possible. If the task was
1864 accelerated, ``sstate_setscene()`` returns True. Otherwise, it
1865 returns False, and the normal ``do_deploy`` task runs. For more
1866 information, see the ":ref:`setscene <bitbake:bitbake-user-manual/bitbake-user-manual-execution:setscene>`"
1867 section in the BitBake User Manual.
1868
1869- The ``do_deploy[dirs] = "${DEPLOYDIR} ${B}"`` line creates
1870 ``${DEPLOYDIR}`` and ``${B}`` before the ``do_deploy`` task runs, and
1871 also sets the current working directory of ``do_deploy`` to ``${B}``.
1872 For more information, see the ":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags`"
1873 section in the BitBake
1874 User Manual.
1875
1876 .. note::
1877
1878 In cases where ``sstate-inputdirs`` and ``sstate-outputdirs`` would be
1879 the same, you can use ``sstate-plaindirs``. For example, to preserve the
1880 ${:term:`PKGD`} and ${:term:`PKGDEST`} output from the ``do_package``
Andrew Geisslerc926e172021-05-07 16:11:35 -05001881 task, use the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001882
1883 do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}"
1884
1885
1886- The ``do_deploy[stamp-extra-info] = "${MACHINE_ARCH}"`` line appends
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001887 extra metadata to the :ref:`stamp
1888 file <overview-manual/concepts:stamp files and the rerunning of tasks>`. In
1889 this case, the metadata makes the task specific to a machine's architecture.
1890 See
Andrew Geissler09209ee2020-12-13 08:44:15 -06001891 ":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-execution:the task list`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001892 section in the BitBake User Manual for more information on the
1893 ``stamp-extra-info`` flag.
1894
1895- ``sstate-inputdirs`` and ``sstate-outputdirs`` can also be used with
1896 multiple directories. For example, the following declares
Andrew Geissler09036742021-06-25 14:25:14 -05001897 :term:`PKGDESTWORK` and ``SHLIBWORK`` as shared state input directories,
1898 which populates the shared state cache, and :term:`PKGDATA_DIR` and
Andrew Geisslerc926e172021-05-07 16:11:35 -05001899 ``SHLIBSDIR`` as the corresponding shared state output directories::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001900
1901 do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}"
1902 do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}"
1903
1904- These methods also include the ability to take a lockfile when
1905 manipulating shared state directory structures, for cases where file
Andrew Geisslerc926e172021-05-07 16:11:35 -05001906 additions or removals are sensitive::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001907
1908 do_package[sstate-lockfile] = "${PACKAGELOCK}"
1909
1910Behind the scenes, the shared state code works by looking in
1911:term:`SSTATE_DIR` and
1912:term:`SSTATE_MIRRORS` for
Andrew Geisslerc926e172021-05-07 16:11:35 -05001913shared state files. Here is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001914
1915 SSTATE_MIRRORS ?= "\
1916 file://.\* http://someserver.tld/share/sstate/PATH;downloadfilename=PATH \n \
1917 file://.\* file:///some/local/dir/sstate/PATH"
1918
1919.. note::
1920
Andrew Geissler5f350902021-07-23 13:09:54 -04001921 The shared state directory (:term:`SSTATE_DIR`) is organized into two-character
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001922 subdirectories, where the subdirectory names are based on the first two
1923 characters of the hash.
1924 If the shared state directory structure for a mirror has the same structure
Andrew Geissler09036742021-06-25 14:25:14 -05001925 as :term:`SSTATE_DIR`, you must specify "PATH" as part of the URI to enable the build
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001926 system to map to the appropriate subdirectory.
1927
1928The shared state package validity can be detected just by looking at the
1929filename since the filename contains the task checksum (or signature) as
1930described earlier in this section. If a valid shared state package is
1931found, the build process downloads it and uses it to accelerate the
1932task.
1933
1934The build processes use the ``*_setscene`` tasks for the task
1935acceleration phase. BitBake goes through this phase before the main
1936execution code and tries to accelerate any tasks for which it can find
1937shared state packages. If a shared state package for a task is
1938available, the shared state package is used. This means the task and any
1939tasks on which it is dependent are not executed.
1940
1941As a real world example, the aim is when building an IPK-based image,
1942only the
1943:ref:`ref-tasks-package_write_ipk`
1944tasks would have their shared state packages fetched and extracted.
1945Since the sysroot is not used, it would never get extracted. This is
1946another reason why a task-based approach is preferred over a
1947recipe-based approach, which would have to install the output from every
1948task.
1949
1950Automatically Added Runtime Dependencies
1951========================================
1952
1953The OpenEmbedded build system automatically adds common types of runtime
1954dependencies between packages, which means that you do not need to
1955explicitly declare the packages using
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001956:term:`RDEPENDS`. There are three automatic
1957mechanisms (``shlibdeps``, ``pcdeps``, and ``depchains``) that
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001958handle shared libraries, package configuration (pkg-config) modules, and
1959``-dev`` and ``-dbg`` packages, respectively. For other types of runtime
1960dependencies, you must manually declare the dependencies.
1961
1962- ``shlibdeps``: During the
1963 :ref:`ref-tasks-package` task of
1964 each recipe, all shared libraries installed by the recipe are
1965 located. For each shared library, the package that contains the
1966 shared library is registered as providing the shared library. More
1967 specifically, the package is registered as providing the
1968 `soname <https://en.wikipedia.org/wiki/Soname>`__ of the library. The
1969 resulting shared-library-to-package mapping is saved globally in
1970 :term:`PKGDATA_DIR` by the
1971 :ref:`ref-tasks-packagedata`
1972 task.
1973
1974 Simultaneously, all executables and shared libraries installed by the
1975 recipe are inspected to see what shared libraries they link against.
Andrew Geissler09036742021-06-25 14:25:14 -05001976 For each shared library dependency that is found, :term:`PKGDATA_DIR` is
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001977 queried to see if some package (likely from a different recipe)
1978 contains the shared library. If such a package is found, a runtime
1979 dependency is added from the package that depends on the shared
1980 library to the package that contains the library.
1981
1982 The automatically added runtime dependency also includes a version
1983 restriction. This version restriction specifies that at least the
1984 current version of the package that provides the shared library must
Andrew Geissler09036742021-06-25 14:25:14 -05001985 be used, as if "package (>= version)" had been added to :term:`RDEPENDS`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001986 This forces an upgrade of the package containing the shared library
1987 when installing the package that depends on the library, if needed.
1988
1989 If you want to avoid a package being registered as providing a
1990 particular shared library (e.g. because the library is for internal
1991 use only), then add the library to
1992 :term:`PRIVATE_LIBS` inside
1993 the package's recipe.
1994
1995- ``pcdeps``: During the ``do_package`` task of each recipe, all
1996 pkg-config modules (``*.pc`` files) installed by the recipe are
1997 located. For each module, the package that contains the module is
1998 registered as providing the module. The resulting module-to-package
Andrew Geissler09036742021-06-25 14:25:14 -05001999 mapping is saved globally in :term:`PKGDATA_DIR` by the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002000 ``do_packagedata`` task.
2001
2002 Simultaneously, all pkg-config modules installed by the recipe are
2003 inspected to see what other pkg-config modules they depend on. A
2004 module is seen as depending on another module if it contains a
2005 "Requires:" line that specifies the other module. For each module
Andrew Geissler09036742021-06-25 14:25:14 -05002006 dependency, :term:`PKGDATA_DIR` is queried to see if some package
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002007 contains the module. If such a package is found, a runtime dependency
2008 is added from the package that depends on the module to the package
2009 that contains the module.
2010
2011 .. note::
2012
2013 The
2014 pcdeps
2015 mechanism most often infers dependencies between
2016 -dev
2017 packages.
2018
2019- ``depchains``: If a package ``foo`` depends on a package ``bar``,
2020 then ``foo-dev`` and ``foo-dbg`` are also made to depend on
2021 ``bar-dev`` and ``bar-dbg``, respectively. Taking the ``-dev``
2022 packages as an example, the ``bar-dev`` package might provide headers
2023 and shared library symlinks needed by ``foo-dev``, which shows the
2024 need for a dependency between the packages.
2025
2026 The dependencies added by ``depchains`` are in the form of
2027 :term:`RRECOMMENDS`.
2028
2029 .. note::
2030
Andrew Geissler5f350902021-07-23 13:09:54 -04002031 By default, ``foo-dev`` also has an :term:`RDEPENDS`-style dependency on
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002032 ``foo``, because the default value of ``RDEPENDS_${PN}-dev`` (set in
2033 bitbake.conf) includes "${PN}".
2034
2035 To ensure that the dependency chain is never broken, ``-dev`` and
2036 ``-dbg`` packages are always generated by default, even if the
2037 packages turn out to be empty. See the
2038 :term:`ALLOW_EMPTY` variable
2039 for more information.
2040
2041The ``do_package`` task depends on the ``do_packagedata`` task of each
2042recipe in :term:`DEPENDS` through use
2043of a ``[``\ :ref:`deptask <bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ``]``
2044declaration, which guarantees that the required
2045shared-library/module-to-package mapping information will be available
Andrew Geissler09036742021-06-25 14:25:14 -05002046when needed as long as :term:`DEPENDS` has been correctly set.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002047
2048Fakeroot and Pseudo
2049===================
2050
2051Some tasks are easier to implement when allowed to perform certain
2052operations that are normally reserved for the root user (e.g.
2053:ref:`ref-tasks-install`,
2054:ref:`do_package_write* <ref-tasks-package_write_deb>`,
2055:ref:`ref-tasks-rootfs`, and
2056:ref:`do_image* <ref-tasks-image>`). For example,
2057the ``do_install`` task benefits from being able to set the UID and GID
2058of installed files to arbitrary values.
2059
2060One approach to allowing tasks to perform root-only operations would be
2061to require :term:`BitBake` to run as
2062root. However, this method is cumbersome and has security issues. The
2063approach that is actually used is to run tasks that benefit from root
2064privileges in a "fake" root environment. Within this environment, the
2065task and its child processes believe that they are running as the root
2066user, and see an internally consistent view of the filesystem. As long
2067as generating the final output (e.g. a package or an image) does not
2068require root privileges, the fact that some earlier steps ran in a fake
2069root environment does not cause problems.
2070
2071The capability to run tasks in a fake root environment is known as
2072"`fakeroot <http://man.he.net/man1/fakeroot>`__", which is derived from
2073the BitBake keyword/variable flag that requests a fake root environment
2074for a task.
2075
Andrew Geisslerd1e89492021-02-12 15:35:20 -06002076In the :term:`OpenEmbedded Build System`, the program that implements
2077fakeroot is known as :yocto_home:`Pseudo </software-item/pseudo/>`. Pseudo
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002078overrides system calls by using the environment variable ``LD_PRELOAD``,
2079which results in the illusion of running as root. To keep track of
2080"fake" file ownership and permissions resulting from operations that
2081require root permissions, Pseudo uses an SQLite 3 database. This
2082database is stored in
2083``${``\ :term:`WORKDIR`\ ``}/pseudo/files.db``
2084for individual recipes. Storing the database in a file as opposed to in
2085memory gives persistence between tasks and builds, which is not
2086accomplished using fakeroot.
2087
2088.. note::
2089
2090 If you add your own task that manipulates the same files or
2091 directories as a fakeroot task, then that task also needs to run
2092 under fakeroot. Otherwise, the task cannot run root-only operations,
2093 and cannot see the fake file ownership and permissions set by the
2094 other task. You need to also add a dependency on
Andrew Geisslerc926e172021-05-07 16:11:35 -05002095 ``virtual/fakeroot-native:do_populate_sysroot``, giving the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002096
2097 fakeroot do_mytask () {
2098 ...
2099 }
2100 do_mytask[depends] += "virtual/fakeroot-native:do_populate_sysroot"
2101
2102
2103For more information, see the
2104:term:`FAKEROOT* <bitbake:FAKEROOT>` variables in the
2105BitBake User Manual. You can also reference the "`Why Not
2106Fakeroot? <https://github.com/wrpseudo/pseudo/wiki/WhyNotFakeroot>`__"
2107article for background information on Fakeroot and Pseudo.