blob: 4f8e6cb041b05be94adc115f876b8dc402664cc4 [file] [log] [blame]
Andrew Geisslerf0343792020-11-18 10:42:21 -06001.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002
3**********************
4Yocto Project Concepts
5**********************
6
7This chapter provides explanations for Yocto Project concepts that go
8beyond the surface of "how-to" information and reference (or look-up)
9material. Concepts such as components, the :term:`OpenEmbedded Build System`
10workflow,
11cross-development toolchains, shared state cache, and so forth are
12explained.
13
14Yocto Project Components
15========================
16
17The :term:`BitBake` task executor
18together with various types of configuration files form the
19:term:`OpenEmbedded-Core (OE-Core)`. This section
20overviews these components by describing their use and how they
21interact.
22
23BitBake handles the parsing and execution of the data files. The data
24itself is of various types:
25
26- *Recipes:* Provides details about particular pieces of software.
27
28- *Class Data:* Abstracts common build information (e.g. how to build a
29 Linux kernel).
30
31- *Configuration Data:* Defines machine-specific settings, policy
32 decisions, and so forth. Configuration data acts as the glue to bind
33 everything together.
34
35BitBake knows how to combine multiple data sources together and refers
36to each data source as a layer. For information on layers, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -060037":ref:`dev-manual/common-tasks:understanding and creating layers`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050038section of the Yocto Project Development Tasks Manual.
39
40Following are some brief details on these core components. For
41additional information on how these components interact during a build,
42see the
Andrew Geissler09209ee2020-12-13 08:44:15 -060043":ref:`overview-manual/concepts:openembedded build system concepts`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050044section.
45
Andrew Geisslerc9f78652020-09-18 14:11:35 -050046BitBake
47-------
48
49BitBake is the tool at the heart of the :term:`OpenEmbedded Build System`
50and is responsible
51for parsing the :term:`Metadata`, generating
52a list of tasks from it, and then executing those tasks.
53
54This section briefly introduces BitBake. If you want more information on
55BitBake, see the :doc:`BitBake User Manual <bitbake:index>`.
56
57To see a list of the options BitBake supports, use either of the
Andrew Geisslerc926e172021-05-07 16:11:35 -050058following commands::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050059
60 $ bitbake -h
61 $ bitbake --help
62
63The most common usage for BitBake is ``bitbake recipename``, where
64``recipename`` is the name of the recipe you want to build (referred
65to as the "target"). The target often equates to the first part of a
66recipe's filename (e.g. "foo" for a recipe named ``foo_1.3.0-r0.bb``).
67So, to process the ``matchbox-desktop_1.2.3.bb`` recipe file, you might
Andrew Geisslerc926e172021-05-07 16:11:35 -050068type the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050069
70 $ bitbake matchbox-desktop
71
72Several different
73versions of ``matchbox-desktop`` might exist. BitBake chooses the one
74selected by the distribution configuration. You can get more details
75about how BitBake chooses between different target versions and
76providers in the
Patrick Williams213cb262021-08-07 19:21:33 -050077":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-execution:preferences`" section
Andrew Geisslerc9f78652020-09-18 14:11:35 -050078of the BitBake User Manual.
79
80BitBake also tries to execute any dependent tasks first. So for example,
81before building ``matchbox-desktop``, BitBake would build a cross
82compiler and ``glibc`` if they had not already been built.
83
84A useful BitBake option to consider is the ``-k`` or ``--continue``
85option. This option instructs BitBake to try and continue processing the
86job as long as possible even after encountering an error. When an error
87occurs, the target that failed and those that depend on it cannot be
88remade. However, when you use this option other dependencies can still
89be processed.
90
Andrew Geisslerc9f78652020-09-18 14:11:35 -050091Recipes
92-------
93
94Files that have the ``.bb`` suffix are "recipes" files. In general, a
95recipe contains information about a single piece of software. This
96information includes the location from which to download the unaltered
97source, any source patches to be applied to that source (if needed),
98which special configuration options to apply, how to compile the source
99files, and how to package the compiled output.
100
101The term "package" is sometimes used to refer to recipes. However, since
102the word "package" is used for the packaged output from the OpenEmbedded
103build system (i.e. ``.ipk`` or ``.deb`` files), this document avoids
104using the term "package" when referring to recipes.
105
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500106Classes
107-------
108
109Class files (``.bbclass``) contain information that is useful to share
110between recipes files. An example is the
111:ref:`autotools <ref-classes-autotools>` class,
112which contains common settings for any application that Autotools uses.
Andrew Geissler09209ee2020-12-13 08:44:15 -0600113The ":ref:`ref-manual/classes:Classes`" chapter in the
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500114Yocto Project Reference Manual provides details about classes and how to
115use them.
116
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500117Configurations
118--------------
119
120The configuration files (``.conf``) define various configuration
121variables that govern the OpenEmbedded build process. These files fall
122into several areas that define machine configuration options,
123distribution configuration options, compiler tuning options, general
124common configuration options, and user configuration options in
125``conf/local.conf``, which is found in the :term:`Build Directory`.
126
127
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500128Layers
129======
130
131Layers are repositories that contain related metadata (i.e. sets of
132instructions) that tell the OpenEmbedded build system how to build a
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500133target. :ref:`overview-manual/yp-intro:the yocto project layer model`
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500134facilitates collaboration, sharing, customization, and reuse within the
135Yocto Project development environment. Layers logically separate
136information for your project. For example, you can use a layer to hold
137all the configurations for a particular piece of hardware. Isolating
138hardware-specific configurations allows you to share other metadata by
139using a different layer where that metadata might be common across
140several pieces of hardware.
141
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700142There are many layers working in the Yocto Project development environment. The
Andrew Geisslerd1e89492021-02-12 15:35:20 -0600143:yocto_home:`Yocto Project Curated Layer Index </software-overview/layers/>`
144and :oe_layerindex:`OpenEmbedded Layer Index <>` both contain layers from
145which you can use or leverage.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500146
147By convention, layers in the Yocto Project follow a specific form.
148Conforming to a known structure allows BitBake to make assumptions
149during builds on where to find types of metadata. You can find
150procedures and learn about tools (i.e. ``bitbake-layers``) for creating
151layers suitable for the Yocto Project in the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600152":ref:`dev-manual/common-tasks:understanding and creating layers`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500153section of the Yocto Project Development Tasks Manual.
154
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500155OpenEmbedded Build System Concepts
156==================================
157
158This section takes a more detailed look inside the build process used by
159the :term:`OpenEmbedded Build System`,
160which is the build
161system specific to the Yocto Project. At the heart of the build system
162is BitBake, the task executor.
163
164The following diagram represents the high-level workflow of a build. The
165remainder of this section expands on the fundamental input, output,
166process, and metadata logical blocks that make up the workflow.
167
168.. image:: figures/YP-flow-diagram.png
169 :align: center
170
171In general, the build's workflow consists of several functional areas:
172
173- *User Configuration:* metadata you can use to control the build
174 process.
175
176- *Metadata Layers:* Various layers that provide software, machine, and
177 distro metadata.
178
179- *Source Files:* Upstream releases, local projects, and SCMs.
180
181- *Build System:* Processes under the control of
182 :term:`BitBake`. This block expands
183 on how BitBake fetches source, applies patches, completes
184 compilation, analyzes output for package generation, creates and
185 tests packages, generates images, and generates cross-development
186 tools.
187
188- *Package Feeds:* Directories containing output packages (RPM, DEB or
189 IPK), which are subsequently used in the construction of an image or
190 Software Development Kit (SDK), produced by the build system. These
191 feeds can also be copied and shared using a web server or other means
192 to facilitate extending or updating existing images on devices at
193 runtime if runtime package management is enabled.
194
195- *Images:* Images produced by the workflow.
196
197- *Application Development SDK:* Cross-development tools that are
198 produced along with an image or separately with BitBake.
199
200User Configuration
201------------------
202
203User configuration helps define the build. Through user configuration,
204you can tell BitBake the target architecture for which you are building
205the image, where to store downloaded source, and other build properties.
206
207The following figure shows an expanded representation of the "User
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500208Configuration" box of the :ref:`general workflow
209figure <overview-manual/concepts:openembedded build system concepts>`:
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500210
211.. image:: figures/user-configuration.png
212 :align: center
213
214BitBake needs some basic configuration files in order to complete a
215build. These files are ``*.conf`` files. The minimally necessary ones
216reside as example files in the ``build/conf`` directory of the
217:term:`Source Directory`. For simplicity,
218this section refers to the Source Directory as the "Poky Directory."
219
220When you clone the :term:`Poky` Git repository
221or you download and unpack a Yocto Project release, you can set up the
222Source Directory to be named anything you want. For this discussion, the
223cloned repository uses the default name ``poky``.
224
225.. note::
226
227 The Poky repository is primarily an aggregation of existing
228 repositories. It is not a canonical upstream source.
229
230The ``meta-poky`` layer inside Poky contains a ``conf`` directory that
231has example configuration files. These example files are used as a basis
232for creating actual configuration files when you source
233:ref:`structure-core-script`, which is the
234build environment script.
235
236Sourcing the build environment script creates a
237:term:`Build Directory` if one does not
238already exist. BitBake uses the Build Directory for all its work during
239builds. The Build Directory has a ``conf`` directory that contains
240default versions of your ``local.conf`` and ``bblayers.conf``
241configuration files. These default configuration files are created only
242if versions do not already exist in the Build Directory at the time you
243source the build environment setup script.
244
245Because the Poky repository is fundamentally an aggregation of existing
246repositories, some users might be familiar with running the
247:ref:`structure-core-script` script in the context of separate
248:term:`OpenEmbedded-Core (OE-Core)` and BitBake
249repositories rather than a single Poky repository. This discussion
250assumes the script is executed from within a cloned or unpacked version
251of Poky.
252
253Depending on where the script is sourced, different sub-scripts are
254called to set up the Build Directory (Yocto or OpenEmbedded).
255Specifically, the script ``scripts/oe-setup-builddir`` inside the poky
256directory sets up the Build Directory and seeds the directory (if
257necessary) with configuration files appropriate for the Yocto Project
258development environment.
259
260.. note::
261
262 The
263 scripts/oe-setup-builddir
264 script uses the
265 ``$TEMPLATECONF``
266 variable to determine which sample configuration files to locate.
267
268The ``local.conf`` file provides many basic variables that define a
269build environment. Here is a list of a few. To see the default
270configurations in a ``local.conf`` file created by the build environment
271script, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600272:yocto_git:`local.conf.sample </poky/tree/meta-poky/conf/local.conf.sample>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500273in the ``meta-poky`` layer:
274
275- *Target Machine Selection:* Controlled by the
276 :term:`MACHINE` variable.
277
278- *Download Directory:* Controlled by the
279 :term:`DL_DIR` variable.
280
281- *Shared State Directory:* Controlled by the
282 :term:`SSTATE_DIR` variable.
283
284- *Build Output:* Controlled by the
285 :term:`TMPDIR` variable.
286
287- *Distribution Policy:* Controlled by the
288 :term:`DISTRO` variable.
289
290- *Packaging Format:* Controlled by the
291 :term:`PACKAGE_CLASSES`
292 variable.
293
294- *SDK Target Architecture:* Controlled by the
295 :term:`SDKMACHINE` variable.
296
297- *Extra Image Packages:* Controlled by the
298 :term:`EXTRA_IMAGE_FEATURES`
299 variable.
300
301.. note::
302
303 Configurations set in the
304 conf/local.conf
305 file can also be set in the
306 conf/site.conf
307 and
308 conf/auto.conf
309 configuration files.
310
311The ``bblayers.conf`` file tells BitBake what layers you want considered
312during the build. By default, the layers listed in this file include
313layers minimally needed by the build system. However, you must manually
314add any custom layers you have created. You can find more information on
315working with the ``bblayers.conf`` file in the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600316":ref:`dev-manual/common-tasks:enabling your layer`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500317section in the Yocto Project Development Tasks Manual.
318
319The files ``site.conf`` and ``auto.conf`` are not created by the
320environment initialization script. If you want the ``site.conf`` file,
321you need to create that yourself. The ``auto.conf`` file is typically
322created by an autobuilder:
323
324- *site.conf:* You can use the ``conf/site.conf`` configuration
325 file to configure multiple build directories. For example, suppose
326 you had several build environments and they shared some common
327 features. You can set these default build properties here. A good
328 example is perhaps the packaging format to use through the
329 :term:`PACKAGE_CLASSES`
330 variable.
331
332 One useful scenario for using the ``conf/site.conf`` file is to
333 extend your :term:`BBPATH` variable
334 to include the path to a ``conf/site.conf``. Then, when BitBake looks
Andrew Geissler09036742021-06-25 14:25:14 -0500335 for Metadata using :term:`BBPATH`, it finds the ``conf/site.conf`` file
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500336 and applies your common configurations found in the file. To override
337 configurations in a particular build directory, alter the similar
338 configurations within that build directory's ``conf/local.conf``
339 file.
340
341- *auto.conf:* The file is usually created and written to by an
342 autobuilder. The settings put into the file are typically the same as
343 you would find in the ``conf/local.conf`` or the ``conf/site.conf``
344 files.
345
346You can edit all configuration files to further define any particular
347build environment. This process is represented by the "User
348Configuration Edits" box in the figure.
349
350When you launch your build with the ``bitbake target`` command, BitBake
351sorts out the configurations to ultimately define your build
352environment. It is important to understand that the
353:term:`OpenEmbedded Build System` reads the
354configuration files in a specific order: ``site.conf``, ``auto.conf``,
355and ``local.conf``. And, the build system applies the normal assignment
356statement rules as described in the
357":doc:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata`" chapter
358of the BitBake User Manual. Because the files are parsed in a specific
359order, variable assignments for the same variable could be affected. For
360example, if the ``auto.conf`` file and the ``local.conf`` set variable1
361to different values, because the build system parses ``local.conf``
362after ``auto.conf``, variable1 is assigned the value from the
363``local.conf`` file.
364
365Metadata, Machine Configuration, and Policy Configuration
366---------------------------------------------------------
367
368The previous section described the user configurations that define
369BitBake's global behavior. This section takes a closer look at the
370layers the build system uses to further control the build. These layers
371provide Metadata for the software, machine, and policies.
372
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700373In general, there are three types of layer input. You can see them below
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500374the "User Configuration" box in the `general workflow
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500375figure <overview-manual/concepts:openembedded build system concepts>`:
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500376
377- *Metadata (.bb + Patches):* Software layers containing
378 user-supplied recipe files, patches, and append files. A good example
Andrew Geisslerd1e89492021-02-12 15:35:20 -0600379 of a software layer might be the :oe_layer:`meta-qt5 layer </meta-qt5>`
380 from the :oe_layerindex:`OpenEmbedded Layer Index <>`. This layer is for
381 version 5.0 of the popular `Qt <https://wiki.qt.io/About_Qt>`__
382 cross-platform application development framework for desktop, embedded and
383 mobile.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500384
385- *Machine BSP Configuration:* Board Support Package (BSP) layers (i.e.
386 "BSP Layer" in the following figure) providing machine-specific
387 configurations. This type of information is specific to a particular
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500388 target architecture. A good example of a BSP layer from the
389 :ref:`overview-manual/yp-intro:reference distribution (poky)` is the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600390 :yocto_git:`meta-yocto-bsp </poky/tree/meta-yocto-bsp>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500391 layer.
392
393- *Policy Configuration:* Distribution Layers (i.e. "Distro Layer" in
394 the following figure) providing top-level or general policies for the
395 images or SDKs being built for a particular distribution. For
396 example, in the Poky Reference Distribution the distro layer is the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600397 :yocto_git:`meta-poky </poky/tree/meta-poky>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500398 layer. Within the distro layer is a ``conf/distro`` directory that
399 contains distro configuration files (e.g.
Andrew Geissler09209ee2020-12-13 08:44:15 -0600400 :yocto_git:`poky.conf </poky/tree/meta-poky/conf/distro/poky.conf>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500401 that contain many policy configurations for the Poky distribution.
402
403The following figure shows an expanded representation of these three
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500404layers from the :ref:`general workflow figure
405<overview-manual/concepts:openembedded build system concepts>`:
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500406
407.. image:: figures/layer-input.png
408 :align: center
409
410In general, all layers have a similar structure. They all contain a
411licensing file (e.g. ``COPYING.MIT``) if the layer is to be distributed,
412a ``README`` file as good practice and especially if the layer is to be
413distributed, a configuration directory, and recipe directories. You can
414learn about the general structure for layers used with the Yocto Project
415in the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600416":ref:`dev-manual/common-tasks:creating your own layer`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500417section in the
418Yocto Project Development Tasks Manual. For a general discussion on
419layers and the many layers from which you can draw, see the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500420":ref:`overview-manual/concepts:layers`" and
421":ref:`overview-manual/yp-intro:the yocto project layer model`" sections both
422earlier in this manual.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500423
424If you explored the previous links, you discovered some areas where many
Andrew Geissler09209ee2020-12-13 08:44:15 -0600425layers that work with the Yocto Project exist. The :yocto_git:`Source
426Repositories <>` also shows layers categorized under "Yocto Metadata Layers."
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500427
428.. note::
429
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700430 There are layers in the Yocto Project Source Repositories that cannot be
431 found in the OpenEmbedded Layer Index. Such layers are either
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500432 deprecated or experimental in nature.
433
434BitBake uses the ``conf/bblayers.conf`` file, which is part of the user
435configuration, to find what layers it should be using as part of the
436build.
437
438Distro Layer
439~~~~~~~~~~~~
440
441The distribution layer provides policy configurations for your
442distribution. Best practices dictate that you isolate these types of
443configurations into their own layer. Settings you provide in
444``conf/distro/distro.conf`` override similar settings that BitBake finds
445in your ``conf/local.conf`` file in the Build Directory.
446
447The following list provides some explanation and references for what you
448typically find in the distribution layer:
449
450- *classes:* Class files (``.bbclass``) hold common functionality that
451 can be shared among recipes in the distribution. When your recipes
452 inherit a class, they take on the settings and functions for that
453 class. You can read more about class files in the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600454 ":ref:`ref-manual/classes:Classes`" chapter of the Yocto
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500455 Reference Manual.
456
457- *conf:* This area holds configuration files for the layer
458 (``conf/layer.conf``), the distribution
459 (``conf/distro/distro.conf``), and any distribution-wide include
460 files.
461
462- *recipes-*:* Recipes and append files that affect common
463 functionality across the distribution. This area could include
464 recipes and append files to add distribution-specific configuration,
465 initialization scripts, custom image recipes, and so forth. Examples
466 of ``recipes-*`` directories are ``recipes-core`` and
467 ``recipes-extra``. Hierarchy and contents within a ``recipes-*``
468 directory can vary. Generally, these directories contain recipe files
469 (``*.bb``), recipe append files (``*.bbappend``), directories that
470 are distro-specific for configuration files, and so forth.
471
472BSP Layer
473~~~~~~~~~
474
475The BSP Layer provides machine configurations that target specific
476hardware. Everything in this layer is specific to the machine for which
477you are building the image or the SDK. A common structure or form is
478defined for BSP layers. You can learn more about this structure in the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600479:doc:`/bsp-guide/index`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500480
481.. note::
482
483 In order for a BSP layer to be considered compliant with the Yocto
484 Project, it must meet some structural requirements.
485
486The BSP Layer's configuration directory contains configuration files for
487the machine (``conf/machine/machine.conf``) and, of course, the layer
488(``conf/layer.conf``).
489
490The remainder of the layer is dedicated to specific recipes by function:
491``recipes-bsp``, ``recipes-core``, ``recipes-graphics``,
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700492``recipes-kernel``, and so forth. There can be metadata for multiple
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500493formfactors, graphics support systems, and so forth.
494
495.. note::
496
497 While the figure shows several
498 recipes-\*
499 directories, not all these directories appear in all BSP layers.
500
501Software Layer
502~~~~~~~~~~~~~~
503
504The software layer provides the Metadata for additional software
505packages used during the build. This layer does not include Metadata
506that is specific to the distribution or the machine, which are found in
507their respective layers.
508
509This layer contains any recipes, append files, and patches, that your
510project needs.
511
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500512Sources
513-------
514
515In order for the OpenEmbedded build system to create an image or any
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500516target, it must be able to access source files. The :ref:`general workflow
517figure <overview-manual/concepts:openembedded build system concepts>`
518represents source files using the "Upstream Project Releases", "Local
519Projects", and "SCMs (optional)" boxes. The figure represents mirrors,
520which also play a role in locating source files, with the "Source
521Materials" box.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500522
523The method by which source files are ultimately organized is a function
524of the project. For example, for released software, projects tend to use
525tarballs or other archived files that can capture the state of a release
526guaranteeing that it is statically represented. On the other hand, for a
527project that is more dynamic or experimental in nature, a project might
528keep source files in a repository controlled by a Source Control Manager
529(SCM) such as Git. Pulling source from a repository allows you to
530control the point in the repository (the revision) from which you want
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700531to build software. A combination of the two is also possible.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500532
533BitBake uses the :term:`SRC_URI`
534variable to point to source files regardless of their location. Each
Andrew Geissler09036742021-06-25 14:25:14 -0500535recipe must have a :term:`SRC_URI` variable that points to the source.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500536
537Another area that plays a significant role in where source files come
538from is pointed to by the
539:term:`DL_DIR` variable. This area is
540a cache that can hold previously downloaded source. You can also
541instruct the OpenEmbedded build system to create tarballs from Git
542repositories, which is not the default behavior, and store them in the
Andrew Geissler09036742021-06-25 14:25:14 -0500543:term:`DL_DIR` by using the
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500544:term:`BB_GENERATE_MIRROR_TARBALLS`
545variable.
546
Andrew Geissler09036742021-06-25 14:25:14 -0500547Judicious use of a :term:`DL_DIR` directory can save the build system a trip
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500548across the Internet when looking for files. A good method for using a
Andrew Geissler09036742021-06-25 14:25:14 -0500549download directory is to have :term:`DL_DIR` point to an area outside of
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500550your Build Directory. Doing so allows you to safely delete the Build
551Directory if needed without fear of removing any downloaded source file.
552
553The remainder of this section provides a deeper look into the source
554files and the mirrors. Here is a more detailed look at the source file
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500555area of the :ref:`general workflow figure <overview-manual/concepts:openembedded build system concepts>`:
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500556
557.. image:: figures/source-input.png
558 :align: center
559
560Upstream Project Releases
561~~~~~~~~~~~~~~~~~~~~~~~~~
562
563Upstream project releases exist anywhere in the form of an archived file
564(e.g. tarball or zip file). These files correspond to individual
565recipes. For example, the figure uses specific releases each for
566BusyBox, Qt, and Dbus. An archive file can be for any released product
567that can be built using a recipe.
568
569Local Projects
570~~~~~~~~~~~~~~
571
572Local projects are custom bits of software the user provides. These bits
573reside somewhere local to a project - perhaps a directory into which the
574user checks in items (e.g. a local directory containing a development
575source tree used by the group).
576
577The canonical method through which to include a local project is to use
578the :ref:`externalsrc <ref-classes-externalsrc>`
579class to include that local project. You use either the ``local.conf``
580or a recipe's append file to override or set the recipe to point to the
581local directory on your disk to pull in the whole source tree.
582
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500583Source Control Managers (Optional)
584~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
585
586Another place from which the build system can get source files is with
Patrick Williams213cb262021-08-07 19:21:33 -0500587:ref:`bitbake:bitbake-user-manual/bitbake-user-manual-fetching:fetchers` employing various Source
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500588Control Managers (SCMs) such as Git or Subversion. In such cases, a
589repository is cloned or checked out. The
590:ref:`ref-tasks-fetch` task inside
591BitBake uses the :term:`SRC_URI`
592variable and the argument's prefix to determine the correct fetcher
593module.
594
595.. note::
596
597 For information on how to have the OpenEmbedded build system generate
598 tarballs for Git repositories and place them in the
599 DL_DIR
600 directory, see the :term:`BB_GENERATE_MIRROR_TARBALLS`
601 variable in the Yocto Project Reference Manual.
602
603When fetching a repository, BitBake uses the
604:term:`SRCREV` variable to determine
605the specific revision from which to build.
606
607Source Mirror(s)
608~~~~~~~~~~~~~~~~
609
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700610There are two kinds of mirrors: pre-mirrors and regular mirrors. The
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500611:term:`PREMIRRORS` and
612:term:`MIRRORS` variables point to
613these, respectively. BitBake checks pre-mirrors before looking upstream
614for any source files. Pre-mirrors are appropriate when you have a shared
615directory that is not a directory defined by the
616:term:`DL_DIR` variable. A Pre-mirror
617typically points to a shared directory that is local to your
618organization.
619
620Regular mirrors can be any site across the Internet that is used as an
621alternative location for source code should the primary site not be
622functioning for some reason or another.
623
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500624Package Feeds
625-------------
626
627When the OpenEmbedded build system generates an image or an SDK, it gets
628the packages from a package feed area located in the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500629:term:`Build Directory`. The :ref:`general workflow figure
630<overview-manual/concepts:openembedded build system concepts>`
631shows this package feeds area in the upper-right corner.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500632
633This section looks a little closer into the package feeds area used by
634the build system. Here is a more detailed look at the area:
635
636.. image:: figures/package-feeds.png
637 :align: center
638
639Package feeds are an intermediary step in the build process. The
640OpenEmbedded build system provides classes to generate different package
641types, and you specify which classes to enable through the
642:term:`PACKAGE_CLASSES`
643variable. Before placing the packages into package feeds, the build
644process validates them with generated output quality assurance checks
645through the :ref:`insane <ref-classes-insane>`
646class.
647
648The package feed area resides in the Build Directory. The directory the
649build system uses to temporarily store packages is determined by a
650combination of variables and the particular package manager in use. See
651the "Package Feeds" box in the illustration and note the information to
652the right of that area. In particular, the following defines where
653package files are kept:
654
655- :term:`DEPLOY_DIR`: Defined as
656 ``tmp/deploy`` in the Build Directory.
657
658- ``DEPLOY_DIR_*``: Depending on the package manager used, the package
659 type sub-folder. Given RPM, IPK, or DEB packaging and tarball
660 creation, the
661 :term:`DEPLOY_DIR_RPM`,
662 :term:`DEPLOY_DIR_IPK`,
663 :term:`DEPLOY_DIR_DEB`, or
664 :term:`DEPLOY_DIR_TAR`,
665 variables are used, respectively.
666
667- :term:`PACKAGE_ARCH`: Defines
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700668 architecture-specific sub-folders. For example, packages could be
669 available for the i586 or qemux86 architectures.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500670
671BitBake uses the
672:ref:`do_package_write_* <ref-tasks-package_write_deb>`
673tasks to generate packages and place them into the package holding area
674(e.g. ``do_package_write_ipk`` for IPK packages). See the
675":ref:`ref-tasks-package_write_deb`",
676":ref:`ref-tasks-package_write_ipk`",
677":ref:`ref-tasks-package_write_rpm`",
678and
679":ref:`ref-tasks-package_write_tar`"
680sections in the Yocto Project Reference Manual for additional
681information. As an example, consider a scenario where an IPK packaging
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700682manager is being used and there is package architecture support for both
683i586 and qemux86. Packages for the i586 architecture are placed in
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500684``build/tmp/deploy/ipk/i586``, while packages for the qemux86
685architecture are placed in ``build/tmp/deploy/ipk/qemux86``.
686
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500687BitBake Tool
688------------
689
690The OpenEmbedded build system uses
691:term:`BitBake` to produce images and
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500692Software Development Kits (SDKs). You can see from the :ref:`general workflow
693figure <overview-manual/concepts:openembedded build system concepts>`,
694the BitBake area consists of several functional areas. This section takes a
695closer look at each of those areas.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500696
697.. note::
698
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700699 Documentation for the BitBake tool is available separately. See the
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500700 BitBake User Manual
701 for reference material on BitBake.
702
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500703Source Fetching
704~~~~~~~~~~~~~~~
705
706The first stages of building a recipe are to fetch and unpack the source
707code:
708
709.. image:: figures/source-fetching.png
710 :align: center
711
712The :ref:`ref-tasks-fetch` and
713:ref:`ref-tasks-unpack` tasks fetch
714the source files and unpack them into the
715:term:`Build Directory`.
716
717.. note::
718
719 For every local file (e.g.
720 file://
721 ) that is part of a recipe's
722 SRC_URI
723 statement, the OpenEmbedded build system takes a checksum of the file
724 for the recipe and inserts the checksum into the signature for the
725 do_fetch
726 task. If any local file has been modified, the
727 do_fetch
728 task and all tasks that depend on it are re-executed.
729
730By default, everything is accomplished in the Build Directory, which has
731a defined structure. For additional general information on the Build
732Directory, see the ":ref:`structure-core-build`" section in
733the Yocto Project Reference Manual.
734
735Each recipe has an area in the Build Directory where the unpacked source
736code resides. The :term:`S` variable points
737to this area for a recipe's unpacked source code. The name of that
738directory for any given recipe is defined from several different
739variables. The preceding figure and the following list describe the
740Build Directory's hierarchy:
741
742- :term:`TMPDIR`: The base directory
743 where the OpenEmbedded build system performs all its work during the
744 build. The default base directory is the ``tmp`` directory.
745
746- :term:`PACKAGE_ARCH`: The
747 architecture of the built package or packages. Depending on the
748 eventual destination of the package or packages (i.e. machine
749 architecture, :term:`Build Host`, SDK, or
Andrew Geissler09036742021-06-25 14:25:14 -0500750 specific machine), :term:`PACKAGE_ARCH` varies. See the variable's
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500751 description for details.
752
753- :term:`TARGET_OS`: The operating
754 system of the target device. A typical value would be "linux" (e.g.
755 "qemux86-poky-linux").
756
757- :term:`PN`: The name of the recipe used
758 to build the package. This variable can have multiple meanings.
Andrew Geissler09036742021-06-25 14:25:14 -0500759 However, when used in the context of input files, :term:`PN` represents
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500760 the name of the recipe.
761
762- :term:`WORKDIR`: The location
763 where the OpenEmbedded build system builds a recipe (i.e. does the
764 work to create the package).
765
766 - :term:`PV`: The version of the
767 recipe used to build the package.
768
769 - :term:`PR`: The revision of the
770 recipe used to build the package.
771
772- :term:`S`: Contains the unpacked source
773 files for a given recipe.
774
775 - :term:`BPN`: The name of the recipe
Andrew Geissler09036742021-06-25 14:25:14 -0500776 used to build the package. The :term:`BPN` variable is a version of
Andrew Geissler5f350902021-07-23 13:09:54 -0400777 the :term:`PN` variable but with common prefixes and suffixes removed.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500778
779 - :term:`PV`: The version of the
780 recipe used to build the package.
781
782.. note::
783
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700784 In the previous figure, notice that there are two sample hierarchies:
785 one based on package architecture (i.e. :term:`PACKAGE_ARCH`)
786 and one based on a machine (i.e. :term:`MACHINE`).
787 The underlying structures are identical. The differentiator being
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500788 what the OpenEmbedded build system is using as a build target (e.g.
789 general architecture, a build host, an SDK, or a specific machine).
790
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500791Patching
792~~~~~~~~
793
794Once source code is fetched and unpacked, BitBake locates patch files
795and applies them to the source files:
796
797.. image:: figures/patching.png
798 :align: center
799
800The :ref:`ref-tasks-patch` task uses a
801recipe's :term:`SRC_URI` statements
802and the :term:`FILESPATH` variable
803to locate applicable patch files.
804
805Default processing for patch files assumes the files have either
Andrew Geissler09036742021-06-25 14:25:14 -0500806``*.patch`` or ``*.diff`` file types. You can use :term:`SRC_URI` parameters
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500807to change the way the build system recognizes patch files. See the
808:ref:`ref-tasks-patch` task for more
809information.
810
811BitBake finds and applies multiple patches for a single recipe in the
Andrew Geissler09036742021-06-25 14:25:14 -0500812order in which it locates the patches. The :term:`FILESPATH` variable
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500813defines the default set of directories that the build system uses to
814search for patch files. Once found, patches are applied to the recipe's
815source files, which are located in the
816:term:`S` directory.
817
818For more information on how the source directories are created, see the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500819":ref:`overview-manual/concepts:source fetching`" section. For
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500820more information on how to create patches and how the build system
821processes patches, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600822":ref:`dev-manual/common-tasks:patching code`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500823section in the
824Yocto Project Development Tasks Manual. You can also see the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600825":ref:`sdk-manual/extensible:use \`\`devtool modify\`\` to modify the source of an existing component`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500826section in the Yocto Project Application Development and the Extensible
827Software Development Kit (SDK) manual and the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600828":ref:`kernel-dev/common:using traditional kernel development to patch the kernel`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500829section in the Yocto Project Linux Kernel Development Manual.
830
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500831Configuration, Compilation, and Staging
832~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
833
834After source code is patched, BitBake executes tasks that configure and
835compile the source code. Once compilation occurs, the files are copied
836to a holding area (staged) in preparation for packaging:
837
838.. image:: figures/configuration-compile-autoreconf.png
839 :align: center
840
841This step in the build process consists of the following tasks:
842
843- :ref:`ref-tasks-prepare_recipe_sysroot`:
844 This task sets up the two sysroots in
845 ``${``\ :term:`WORKDIR`\ ``}``
846 (i.e. ``recipe-sysroot`` and ``recipe-sysroot-native``) so that
847 during the packaging phase the sysroots can contain the contents of
848 the
849 :ref:`ref-tasks-populate_sysroot`
850 tasks of the recipes on which the recipe containing the tasks
851 depends. A sysroot exists for both the target and for the native
852 binaries, which run on the host system.
853
854- *do_configure*: This task configures the source by enabling and
855 disabling any build-time and configuration options for the software
856 being built. Configurations can come from the recipe itself as well
857 as from an inherited class. Additionally, the software itself might
858 configure itself depending on the target for which it is being built.
859
860 The configurations handled by the
861 :ref:`ref-tasks-configure` task
862 are specific to configurations for the source code being built by the
863 recipe.
864
865 If you are using the
866 :ref:`autotools <ref-classes-autotools>` class,
867 you can add additional configuration options by using the
868 :term:`EXTRA_OECONF` or
869 :term:`PACKAGECONFIG_CONFARGS`
870 variables. For information on how this variable works within that
871 class, see the
872 :ref:`autotools <ref-classes-autotools>` class
Andrew Geissler09209ee2020-12-13 08:44:15 -0600873 :yocto_git:`here </poky/tree/meta/classes/autotools.bbclass>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500874
875- *do_compile*: Once a configuration task has been satisfied,
876 BitBake compiles the source using the
877 :ref:`ref-tasks-compile` task.
878 Compilation occurs in the directory pointed to by the
879 :term:`B` variable. Realize that the
Andrew Geissler09036742021-06-25 14:25:14 -0500880 :term:`B` directory is, by default, the same as the
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500881 :term:`S` directory.
882
883- *do_install*: After compilation completes, BitBake executes the
884 :ref:`ref-tasks-install` task.
Andrew Geissler09036742021-06-25 14:25:14 -0500885 This task copies files from the :term:`B` directory and places them in a
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500886 holding area pointed to by the :term:`D`
887 variable. Packaging occurs later using files from this holding
888 directory.
889
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500890Package Splitting
891~~~~~~~~~~~~~~~~~
892
893After source code is configured, compiled, and staged, the build system
894analyzes the results and splits the output into packages:
895
896.. image:: figures/analysis-for-package-splitting.png
897 :align: center
898
899The :ref:`ref-tasks-package` and
900:ref:`ref-tasks-packagedata`
901tasks combine to analyze the files found in the
902:term:`D` directory and split them into
903subsets based on available packages and files. Analysis involves the
904following as well as other items: splitting out debugging symbols,
905looking at shared library dependencies between packages, and looking at
906package relationships.
907
908The ``do_packagedata`` task creates package metadata based on the
909analysis such that the build system can generate the final packages. The
910:ref:`ref-tasks-populate_sysroot`
911task stages (copies) a subset of the files installed by the
912:ref:`ref-tasks-install` task into
913the appropriate sysroot. Working, staged, and intermediate results of
914the analysis and package splitting process use several areas:
915
916- :term:`PKGD`: The destination
917 directory (i.e. ``package``) for packages before they are split into
918 individual packages.
919
920- :term:`PKGDESTWORK`: A
921 temporary work area (i.e. ``pkgdata``) used by the ``do_package``
922 task to save package metadata.
923
924- :term:`PKGDEST`: The parent
925 directory (i.e. ``packages-split``) for packages after they have been
926 split.
927
928- :term:`PKGDATA_DIR`: A shared,
929 global-state directory that holds packaging metadata generated during
930 the packaging process. The packaging process copies metadata from
Andrew Geissler09036742021-06-25 14:25:14 -0500931 :term:`PKGDESTWORK` to the :term:`PKGDATA_DIR` area where it becomes globally
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500932 available.
933
934- :term:`STAGING_DIR_HOST`:
935 The path for the sysroot for the system on which a component is built
936 to run (i.e. ``recipe-sysroot``).
937
938- :term:`STAGING_DIR_NATIVE`:
939 The path for the sysroot used when building components for the build
940 host (i.e. ``recipe-sysroot-native``).
941
942- :term:`STAGING_DIR_TARGET`:
943 The path for the sysroot used when a component that is built to
944 execute on a system and it generates code for yet another machine
945 (e.g. cross-canadian recipes).
946
947The :term:`FILES` variable defines the
948files that go into each package in
949:term:`PACKAGES`. If you want
950details on how this is accomplished, you can look at
Andrew Geissler09209ee2020-12-13 08:44:15 -0600951:yocto_git:`package.bbclass </poky/tree/meta/classes/package.bbclass>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500952
953Depending on the type of packages being created (RPM, DEB, or IPK), the
954:ref:`do_package_write_* <ref-tasks-package_write_deb>`
955task creates the actual packages and places them in the Package Feed
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500956area, which is ``${TMPDIR}/deploy``. You can see the
957":ref:`overview-manual/concepts:package feeds`" section for more detail on
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500958that part of the build process.
959
960.. note::
961
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700962 Support for creating feeds directly from the ``deploy/*``
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500963 directories does not exist. Creating such feeds usually requires some
964 kind of feed maintenance mechanism that would upload the new packages
965 into an official package feed (e.g. the Ångström distribution). This
966 functionality is highly distribution-specific and thus is not
967 provided out of the box.
968
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500969Image Generation
970~~~~~~~~~~~~~~~~
971
972Once packages are split and stored in the Package Feeds area, the build
973system uses BitBake to generate the root filesystem image:
974
975.. image:: figures/image-generation.png
976 :align: center
977
978The image generation process consists of several stages and depends on
979several tasks and variables. The
980:ref:`ref-tasks-rootfs` task creates
981the root filesystem (file and directory structure) for an image. This
982task uses several key variables to help create the list of packages to
983actually install:
984
985- :term:`IMAGE_INSTALL`: Lists
986 out the base set of packages from which to install from the Package
987 Feeds area.
988
989- :term:`PACKAGE_EXCLUDE`:
990 Specifies packages that should not be installed into the image.
991
992- :term:`IMAGE_FEATURES`:
993 Specifies features to include in the image. Most of these features
994 map to additional packages for installation.
995
996- :term:`PACKAGE_CLASSES`:
997 Specifies the package backend (e.g. RPM, DEB, or IPK) to use and
998 consequently helps determine where to locate packages within the
999 Package Feeds area.
1000
1001- :term:`IMAGE_LINGUAS`:
1002 Determines the language(s) for which additional language support
1003 packages are installed.
1004
1005- :term:`PACKAGE_INSTALL`:
1006 The final list of packages passed to the package manager for
1007 installation into the image.
1008
1009With :term:`IMAGE_ROOTFS`
1010pointing to the location of the filesystem under construction and the
Andrew Geissler09036742021-06-25 14:25:14 -05001011:term:`PACKAGE_INSTALL` variable providing the final list of packages to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001012install, the root file system is created.
1013
1014Package installation is under control of the package manager (e.g.
1015dnf/rpm, opkg, or apt/dpkg) regardless of whether or not package
1016management is enabled for the target. At the end of the process, if
1017package management is not enabled for the target, the package manager's
1018data files are deleted from the root filesystem. As part of the final
1019stage of package installation, post installation scripts that are part
1020of the packages are run. Any scripts that fail to run on the build host
1021are run on the target when the target system is first booted. If you are
1022using a
Andrew Geissler09209ee2020-12-13 08:44:15 -06001023:ref:`read-only root filesystem <dev-manual/common-tasks:creating a read-only root filesystem>`,
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001024all the post installation scripts must succeed on the build host during
1025the package installation phase since the root filesystem on the target
1026is read-only.
1027
1028The final stages of the ``do_rootfs`` task handle post processing. Post
1029processing includes creation of a manifest file and optimizations.
1030
1031The manifest file (``.manifest``) resides in the same directory as the
1032root filesystem image. This file lists out, line-by-line, the installed
1033packages. The manifest file is useful for the
1034:ref:`testimage <ref-classes-testimage*>` class,
1035for example, to determine whether or not to run specific tests. See the
1036:term:`IMAGE_MANIFEST`
1037variable for additional information.
1038
1039Optimizing processes that are run across the image include ``mklibs``,
1040``prelink``, and any other post-processing commands as defined by the
1041:term:`ROOTFS_POSTPROCESS_COMMAND`
1042variable. The ``mklibs`` process optimizes the size of the libraries,
1043while the ``prelink`` process optimizes the dynamic linking of shared
1044libraries to reduce start up time of executables.
1045
1046After the root filesystem is built, processing begins on the image
1047through the :ref:`ref-tasks-image`
1048task. The build system runs any pre-processing commands as defined by
1049the
1050:term:`IMAGE_PREPROCESS_COMMAND`
1051variable. This variable specifies a list of functions to call before the
1052build system creates the final image output files.
1053
1054The build system dynamically creates ``do_image_*`` tasks as needed,
1055based on the image types specified in the
1056:term:`IMAGE_FSTYPES` variable.
1057The process turns everything into an image file or a set of image files
1058and can compress the root filesystem image to reduce the overall size of
1059the image. The formats used for the root filesystem depend on the
Andrew Geissler09036742021-06-25 14:25:14 -05001060:term:`IMAGE_FSTYPES` variable. Compression depends on whether the formats
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001061support compression.
1062
1063As an example, a dynamically created task when creating a particular
Andrew Geisslerc926e172021-05-07 16:11:35 -05001064image type would take the following form::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001065
1066 do_image_type
1067
1068So, if the type
Andrew Geissler09036742021-06-25 14:25:14 -05001069as specified by the :term:`IMAGE_FSTYPES` were ``ext4``, the dynamically
Andrew Geisslerc926e172021-05-07 16:11:35 -05001070generated task would be as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001071
1072 do_image_ext4
1073
1074The final task involved in image creation is the
1075:ref:`do_image_complete <ref-tasks-image-complete>`
1076task. This task completes the image by applying any image post
1077processing as defined through the
1078:term:`IMAGE_POSTPROCESS_COMMAND`
1079variable. The variable specifies a list of functions to call once the
1080build system has created the final image output files.
1081
1082.. note::
1083
1084 The entire image generation process is run under
1085 Pseudo. Running under Pseudo ensures that the files in the root filesystem
1086 have correct ownership.
1087
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001088SDK Generation
1089~~~~~~~~~~~~~~
1090
1091The OpenEmbedded build system uses BitBake to generate the Software
1092Development Kit (SDK) installer scripts for both the standard SDK and
1093the extensible SDK (eSDK):
1094
1095.. image:: figures/sdk-generation.png
1096 :align: center
1097
1098.. note::
1099
1100 For more information on the cross-development toolchain generation,
Andrew Geissler09209ee2020-12-13 08:44:15 -06001101 see the ":ref:`overview-manual/concepts:cross-development toolchain generation`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001102 section. For information on advantages gained when building a
1103 cross-development toolchain using the do_populate_sdk task, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001104 ":ref:`sdk-manual/appendix-obtain:building an sdk installer`" section in
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001105 the Yocto Project Application Development and the Extensible Software
1106 Development Kit (eSDK) manual.
1107
1108Like image generation, the SDK script process consists of several stages
1109and depends on many variables. The
1110:ref:`ref-tasks-populate_sdk`
1111and
1112:ref:`ref-tasks-populate_sdk_ext`
1113tasks use these key variables to help create the list of packages to
1114actually install. For information on the variables listed in the figure,
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001115see the ":ref:`overview-manual/concepts:application development sdk`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001116section.
1117
1118The ``do_populate_sdk`` task helps create the standard SDK and handles
1119two parts: a target part and a host part. The target part is the part
1120built for the target hardware and includes libraries and headers. The
1121host part is the part of the SDK that runs on the
1122:term:`SDKMACHINE`.
1123
1124The ``do_populate_sdk_ext`` task helps create the extensible SDK and
1125handles host and target parts differently than its counter part does for
1126the standard SDK. For the extensible SDK, the task encapsulates the
1127build system, which includes everything needed (host and target) for the
1128SDK.
1129
1130Regardless of the type of SDK being constructed, the tasks perform some
1131cleanup after which a cross-development environment setup script and any
1132needed configuration files are created. The final output is the
1133Cross-development toolchain installation script (``.sh`` file), which
1134includes the environment setup script.
1135
1136Stamp Files and the Rerunning of Tasks
1137~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1138
1139For each task that completes successfully, BitBake writes a stamp file
1140into the :term:`STAMPS_DIR`
1141directory. The beginning of the stamp file's filename is determined by
1142the :term:`STAMP` variable, and the end
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001143of the name consists of the task's name and current :ref:`input
1144checksum <overview-manual/concepts:checksums (signatures)>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001145
1146.. note::
1147
1148 This naming scheme assumes that
1149 BB_SIGNATURE_HANDLER
1150 is "OEBasicHash", which is almost always the case in current
1151 OpenEmbedded.
1152
1153To determine if a task needs to be rerun, BitBake checks if a stamp file
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001154with a matching input checksum exists for the task. In this case,
1155the task's output is assumed to exist and still be valid. Otherwise,
1156the task is rerun.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001157
1158.. note::
1159
1160 The stamp mechanism is more general than the shared state (sstate)
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001161 cache mechanism described in the
1162 ":ref:`overview-manual/concepts:setscene tasks and shared state`" section.
1163 BitBake avoids rerunning any task that has a valid stamp file, not just
1164 tasks that can be accelerated through the sstate cache.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001165
1166 However, you should realize that stamp files only serve as a marker
1167 that some work has been done and that these files do not record task
1168 output. The actual task output would usually be somewhere in
1169 :term:`TMPDIR` (e.g. in some
1170 recipe's :term:`WORKDIR`.) What
1171 the sstate cache mechanism adds is a way to cache task output that
1172 can then be shared between build machines.
1173
Andrew Geissler09036742021-06-25 14:25:14 -05001174Since :term:`STAMPS_DIR` is usually a subdirectory of :term:`TMPDIR`, removing
1175:term:`TMPDIR` will also remove :term:`STAMPS_DIR`, which means tasks will
1176properly be rerun to repopulate :term:`TMPDIR`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001177
1178If you want some task to always be considered "out of date", you can
1179mark it with the :ref:`nostamp <bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`
1180varflag. If some other task depends on such a task, then that task will
1181also always be considered out of date, which might not be what you want.
1182
1183For details on how to view information about a task's signature, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001184":ref:`dev-manual/common-tasks:viewing task variable dependencies`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001185section in the Yocto Project Development Tasks Manual.
1186
1187Setscene Tasks and Shared State
1188~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1189
1190The description of tasks so far assumes that BitBake needs to build
1191everything and no available prebuilt objects exist. BitBake does support
1192skipping tasks if prebuilt objects are available. These objects are
1193usually made available in the form of a shared state (sstate) cache.
1194
1195.. note::
1196
1197 For information on variables affecting sstate, see the
1198 :term:`SSTATE_DIR`
1199 and
1200 :term:`SSTATE_MIRRORS`
1201 variables.
1202
1203The idea of a setscene task (i.e ``do_``\ taskname\ ``_setscene``) is a
1204version of the task where instead of building something, BitBake can
1205skip to the end result and simply place a set of files into specific
1206locations as needed. In some cases, it makes sense to have a setscene
1207task variant (e.g. generating package files in the
1208:ref:`do_package_write_* <ref-tasks-package_write_deb>`
1209task). In other cases, it does not make sense (e.g. a
1210:ref:`ref-tasks-patch` task or a
1211:ref:`ref-tasks-unpack` task) since
1212the work involved would be equal to or greater than the underlying task.
1213
1214In the build system, the common tasks that have setscene variants are
1215:ref:`ref-tasks-package`,
1216``do_package_write_*``,
1217:ref:`ref-tasks-deploy`,
1218:ref:`ref-tasks-packagedata`, and
1219:ref:`ref-tasks-populate_sysroot`.
1220Notice that these tasks represent most of the tasks whose output is an
1221end result.
1222
1223The build system has knowledge of the relationship between these tasks
1224and other preceding tasks. For example, if BitBake runs
1225``do_populate_sysroot_setscene`` for something, it does not make sense
1226to run any of the ``do_fetch``, ``do_unpack``, ``do_patch``,
1227``do_configure``, ``do_compile``, and ``do_install`` tasks. However, if
1228``do_package`` needs to be run, BitBake needs to run those other tasks.
1229
1230It becomes more complicated if everything can come from an sstate cache
1231because some objects are simply not required at all. For example, you do
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001232not need a compiler or native tools, such as quilt, if there isn't anything
1233to compile or patch. If the ``do_package_write_*`` packages are available
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001234from sstate, BitBake does not need the ``do_package`` task data.
1235
1236To handle all these complexities, BitBake runs in two phases. The first
1237is the "setscene" stage. During this stage, BitBake first checks the
1238sstate cache for any targets it is planning to build. BitBake does a
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001239fast check to see if the object exists rather than doing a complete download.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001240If nothing exists, the second phase, which is the setscene stage,
1241completes and the main build proceeds.
1242
1243If objects are found in the sstate cache, the build system works
1244backwards from the end targets specified by the user. For example, if an
1245image is being built, the build system first looks for the packages
1246needed for that image and the tools needed to construct an image. If
1247those are available, the compiler is not needed. Thus, the compiler is
1248not even downloaded. If something was found to be unavailable, or the
1249download or setscene task fails, the build system then tries to install
1250dependencies, such as the compiler, from the cache.
1251
1252The availability of objects in the sstate cache is handled by the
Patrick Williams213cb262021-08-07 19:21:33 -05001253function specified by the :term:`BB_HASHCHECK_FUNCTION`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001254variable and returns a list of available objects. The function specified
Patrick Williams213cb262021-08-07 19:21:33 -05001255by the :term:`BB_SETSCENE_DEPVALID`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001256variable is the function that determines whether a given dependency
1257needs to be followed, and whether for any given relationship the
1258function needs to be passed. The function returns a True or False value.
1259
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001260Images
1261------
1262
1263The images produced by the build system are compressed forms of the root
1264filesystem and are ready to boot on a target device. You can see from
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001265the :ref:`general workflow figure
1266<overview-manual/concepts:openembedded build system concepts>` that BitBake
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001267output, in part, consists of images. This section takes a closer look at
1268this output:
1269
1270.. image:: figures/images.png
1271 :align: center
1272
1273.. note::
1274
1275 For a list of example images that the Yocto Project provides, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001276 ":doc:`/ref-manual/images`" chapter in the Yocto Project Reference
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001277 Manual.
1278
1279The build process writes images out to the :term:`Build Directory`
1280inside the
1281``tmp/deploy/images/machine/`` folder as shown in the figure. This
1282folder contains any files expected to be loaded on the target device.
1283The :term:`DEPLOY_DIR` variable
1284points to the ``deploy`` directory, while the
1285:term:`DEPLOY_DIR_IMAGE`
1286variable points to the appropriate directory containing images for the
1287current configuration.
1288
1289- kernel-image: A kernel binary file. The
1290 :term:`KERNEL_IMAGETYPE`
1291 variable determines the naming scheme for the kernel image file.
1292 Depending on this variable, the file could begin with a variety of
1293 naming strings. The ``deploy/images/``\ machine directory can contain
1294 multiple image files for the machine.
1295
1296- root-filesystem-image: Root filesystems for the target device (e.g.
1297 ``*.ext3`` or ``*.bz2`` files). The
1298 :term:`IMAGE_FSTYPES`
1299 variable determines the root filesystem image type. The
1300 ``deploy/images/``\ machine directory can contain multiple root
1301 filesystems for the machine.
1302
1303- kernel-modules: Tarballs that contain all the modules built for the
1304 kernel. Kernel module tarballs exist for legacy purposes and can be
1305 suppressed by setting the
1306 :term:`MODULE_TARBALL_DEPLOY`
1307 variable to "0". The ``deploy/images/``\ machine directory can
1308 contain multiple kernel module tarballs for the machine.
1309
1310- bootloaders: If applicable to the target machine, bootloaders
1311 supporting the image. The ``deploy/images/``\ machine directory can
1312 contain multiple bootloaders for the machine.
1313
1314- symlinks: The ``deploy/images/``\ machine folder contains a symbolic
1315 link that points to the most recently built file for each machine.
1316 These links might be useful for external scripts that need to obtain
1317 the latest version of each file.
1318
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001319Application Development SDK
1320---------------------------
1321
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001322In the :ref:`general workflow figure
1323<overview-manual/concepts:openembedded build system concepts>`, the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001324output labeled "Application Development SDK" represents an SDK. The SDK
1325generation process differs depending on whether you build an extensible
1326SDK (e.g. ``bitbake -c populate_sdk_ext`` imagename) or a standard SDK
1327(e.g. ``bitbake -c populate_sdk`` imagename). This section takes a
1328closer look at this output:
1329
1330.. image:: figures/sdk.png
1331 :align: center
1332
1333The specific form of this output is a set of files that includes a
1334self-extracting SDK installer (``*.sh``), host and target manifest
1335files, and files used for SDK testing. When the SDK installer file is
1336run, it installs the SDK. The SDK consists of a cross-development
1337toolchain, a set of libraries and headers, and an SDK environment setup
1338script. Running this installer essentially sets up your
1339cross-development environment. You can think of the cross-toolchain as
1340the "host" part because it runs on the SDK machine. You can think of the
1341libraries and headers as the "target" part because they are built for
1342the target hardware. The environment setup script is added so that you
1343can initialize the environment before using the tools.
1344
1345.. note::
1346
1347 - The Yocto Project supports several methods by which you can set up
1348 this cross-development environment. These methods include
1349 downloading pre-built SDK installers or building and installing
1350 your own SDK installer.
1351
1352 - For background information on cross-development toolchains in the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001353 Yocto Project development environment, see the
1354 ":ref:`overview-manual/concepts:cross-development toolchain generation`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001355 section.
1356
1357 - For information on setting up a cross-development environment, see
Andrew Geissler09209ee2020-12-13 08:44:15 -06001358 the :doc:`/sdk-manual/index` manual.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001359
1360All the output files for an SDK are written to the ``deploy/sdk`` folder
1361inside the :term:`Build Directory` as
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001362shown in the previous figure. Depending on the type of SDK, there are
1363several variables to configure these files. Here are the variables
1364associated with an extensible SDK:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001365
1366- :term:`DEPLOY_DIR`: Points to
1367 the ``deploy`` directory.
1368
1369- :term:`SDK_EXT_TYPE`:
1370 Controls whether or not shared state artifacts are copied into the
1371 extensible SDK. By default, all required shared state artifacts are
1372 copied into the SDK.
1373
1374- :term:`SDK_INCLUDE_PKGDATA`:
1375 Specifies whether or not packagedata is included in the extensible
1376 SDK for all recipes in the "world" target.
1377
1378- :term:`SDK_INCLUDE_TOOLCHAIN`:
1379 Specifies whether or not the toolchain is included when building the
1380 extensible SDK.
1381
1382- :term:`SDK_LOCAL_CONF_WHITELIST`:
1383 A list of variables allowed through from the build system
1384 configuration into the extensible SDK configuration.
1385
1386- :term:`SDK_LOCAL_CONF_BLACKLIST`:
1387 A list of variables not allowed through from the build system
1388 configuration into the extensible SDK configuration.
1389
1390- :term:`SDK_INHERIT_BLACKLIST`:
1391 A list of classes to remove from the
1392 :term:`INHERIT` value globally
1393 within the extensible SDK configuration.
1394
1395This next list, shows the variables associated with a standard SDK:
1396
1397- :term:`DEPLOY_DIR`: Points to
1398 the ``deploy`` directory.
1399
1400- :term:`SDKMACHINE`: Specifies
1401 the architecture of the machine on which the cross-development tools
1402 are run to create packages for the target hardware.
1403
1404- :term:`SDKIMAGE_FEATURES`:
1405 Lists the features to include in the "target" part of the SDK.
1406
1407- :term:`TOOLCHAIN_HOST_TASK`:
1408 Lists packages that make up the host part of the SDK (i.e. the part
Andrew Geissler09036742021-06-25 14:25:14 -05001409 that runs on the :term:`SDKMACHINE`). When you use
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001410 ``bitbake -c populate_sdk imagename`` to create the SDK, a set of
1411 default packages apply. This variable allows you to add more
1412 packages.
1413
1414- :term:`TOOLCHAIN_TARGET_TASK`:
1415 Lists packages that make up the target part of the SDK (i.e. the part
1416 built for the target hardware).
1417
1418- :term:`SDKPATH`: Defines the
1419 default SDK installation path offered by the installation script.
1420
1421- :term:`SDK_HOST_MANIFEST`:
1422 Lists all the installed packages that make up the host part of the
1423 SDK. This variable also plays a minor role for extensible SDK
1424 development as well. However, it is mainly used for the standard SDK.
1425
1426- :term:`SDK_TARGET_MANIFEST`:
1427 Lists all the installed packages that make up the target part of the
1428 SDK. This variable also plays a minor role for extensible SDK
1429 development as well. However, it is mainly used for the standard SDK.
1430
1431Cross-Development Toolchain Generation
1432======================================
1433
1434The Yocto Project does most of the work for you when it comes to
Andrew Geissler09209ee2020-12-13 08:44:15 -06001435creating :ref:`sdk-manual/intro:the cross-development toolchain`. This
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001436section provides some technical background on how cross-development
1437toolchains are created and used. For more information on toolchains, you
Andrew Geissler09209ee2020-12-13 08:44:15 -06001438can also see the :doc:`/sdk-manual/index` manual.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001439
1440In the Yocto Project development environment, cross-development
1441toolchains are used to build images and applications that run on the
1442target hardware. With just a few commands, the OpenEmbedded build system
1443creates these necessary toolchains for you.
1444
1445The following figure shows a high-level build environment regarding
1446toolchain construction and use.
1447
1448.. image:: figures/cross-development-toolchains.png
1449 :align: center
1450
1451Most of the work occurs on the Build Host. This is the machine used to
1452build images and generally work within the the Yocto Project
1453environment. When you run
1454:term:`BitBake` to create an image, the
1455OpenEmbedded build system uses the host ``gcc`` compiler to bootstrap a
1456cross-compiler named ``gcc-cross``. The ``gcc-cross`` compiler is what
1457BitBake uses to compile source files when creating the target image. You
1458can think of ``gcc-cross`` simply as an automatically generated
1459cross-compiler that is used internally within BitBake only.
1460
1461.. note::
1462
Andrew Geissler09036742021-06-25 14:25:14 -05001463 The extensible SDK does not use ``gcc-cross-canadian``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001464 since this SDK ships a copy of the OpenEmbedded build system and the
Andrew Geissler09036742021-06-25 14:25:14 -05001465 sysroot within it contains ``gcc-cross``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001466
Andrew Geisslerc926e172021-05-07 16:11:35 -05001467The chain of events that occurs when the standard toolchain is bootstrapped::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001468
Andrew Geissler6ce62a22020-11-30 19:58:47 -06001469 binutils-cross -> linux-libc-headers -> gcc-cross -> libgcc-initial -> glibc -> libgcc -> gcc-runtime
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001470
Andrew Geissler6ce62a22020-11-30 19:58:47 -06001471- ``gcc``: The compiler, GNU Compiler Collection (GCC).
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001472
Andrew Geissler6ce62a22020-11-30 19:58:47 -06001473- ``binutils-cross``: The binary utilities needed in order
1474 to run the ``gcc-cross`` phase of the bootstrap operation and build the
1475 headers for the C library.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001476
Andrew Geissler6ce62a22020-11-30 19:58:47 -06001477- ``linux-libc-headers``: Headers needed for the cross-compiler and C library build.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001478
Andrew Geissler6ce62a22020-11-30 19:58:47 -06001479- ``libgcc-initial``: An initial version of the gcc support library needed
1480 to bootstrap ``glibc``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001481
Andrew Geissler6ce62a22020-11-30 19:58:47 -06001482- ``libgcc``: The final version of the gcc support library which
1483 can only be built once there is a C library to link against.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001484
1485- ``glibc``: The GNU C Library.
1486
1487- ``gcc-cross``: The final stage of the bootstrap process for the
1488 cross-compiler. This stage results in the actual cross-compiler that
1489 BitBake uses when it builds an image for a targeted device.
1490
Andrew Geissler6ce62a22020-11-30 19:58:47 -06001491 This tool is a "native" tool (i.e. it is designed to run on
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001492 the build host).
1493
1494- ``gcc-runtime``: Runtime libraries resulting from the toolchain
1495 bootstrapping process. This tool produces a binary that consists of
1496 the runtime libraries need for the targeted device.
1497
1498You can use the OpenEmbedded build system to build an installer for the
1499relocatable SDK used to develop applications. When you run the
1500installer, it installs the toolchain, which contains the development
1501tools (e.g., ``gcc-cross-canadian``, ``binutils-cross-canadian``, and
1502other ``nativesdk-*`` tools), which are tools native to the SDK (i.e.
1503native to :term:`SDK_ARCH`), you
1504need to cross-compile and test your software. The figure shows the
1505commands you use to easily build out this toolchain. This
1506cross-development toolchain is built to execute on the
1507:term:`SDKMACHINE`, which might or
1508might not be the same machine as the Build Host.
1509
1510.. note::
1511
1512 If your target architecture is supported by the Yocto Project, you
1513 can take advantage of pre-built images that ship with the Yocto
1514 Project and already contain cross-development toolchain installers.
1515
Andrew Geisslerc926e172021-05-07 16:11:35 -05001516Here is the bootstrap process for the relocatable toolchain::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001517
1518 gcc -> binutils-crosssdk -> gcc-crosssdk-initial -> linux-libc-headers -> glibc-initial -> nativesdk-glibc -> gcc-crosssdk -> gcc-cross-canadian
1519
1520- ``gcc``: The build host's GNU Compiler Collection (GCC).
1521
1522- ``binutils-crosssdk``: The bare minimum binary utilities needed in
1523 order to run the ``gcc-crosssdk-initial`` phase of the bootstrap
1524 operation.
1525
1526- ``gcc-crosssdk-initial``: An early stage of the bootstrap process for
1527 creating the cross-compiler. This stage builds enough of the
1528 ``gcc-crosssdk`` and supporting pieces so that the final stage of the
1529 bootstrap process can produce the finished cross-compiler. This tool
1530 is a "native" binary that runs on the build host.
1531
1532- ``linux-libc-headers``: Headers needed for the cross-compiler.
1533
1534- ``glibc-initial``: An initial version of the Embedded GLIBC needed to
1535 bootstrap ``nativesdk-glibc``.
1536
1537- ``nativesdk-glibc``: The Embedded GLIBC needed to bootstrap the
1538 ``gcc-crosssdk``.
1539
1540- ``gcc-crosssdk``: The final stage of the bootstrap process for the
1541 relocatable cross-compiler. The ``gcc-crosssdk`` is a transitory
1542 compiler and never leaves the build host. Its purpose is to help in
1543 the bootstrap process to create the eventual ``gcc-cross-canadian``
1544 compiler, which is relocatable. This tool is also a "native" package
1545 (i.e. it is designed to run on the build host).
1546
1547- ``gcc-cross-canadian``: The final relocatable cross-compiler. When
1548 run on the :term:`SDKMACHINE`,
1549 this tool produces executable code that runs on the target device.
1550 Only one cross-canadian compiler is produced per architecture since
1551 they can be targeted at different processor optimizations using
1552 configurations passed to the compiler through the compile commands.
1553 This circumvents the need for multiple compilers and thus reduces the
1554 size of the toolchains.
1555
1556.. note::
1557
1558 For information on advantages gained when building a
1559 cross-development toolchain installer, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001560 ":ref:`sdk-manual/appendix-obtain:building an sdk installer`" appendix
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001561 in the Yocto Project Application Development and the
1562 Extensible Software Development Kit (eSDK) manual.
1563
1564Shared State Cache
1565==================
1566
1567By design, the OpenEmbedded build system builds everything from scratch
1568unless :term:`BitBake` can determine
1569that parts do not need to be rebuilt. Fundamentally, building from
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001570scratch is attractive as it means all parts are built fresh and there is
1571no possibility of stale data that can cause problems. When
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001572developers hit problems, they typically default back to building from
1573scratch so they have a know state from the start.
1574
1575Building an image from scratch is both an advantage and a disadvantage
1576to the process. As mentioned in the previous paragraph, building from
1577scratch ensures that everything is current and starts from a known
1578state. However, building from scratch also takes much longer as it
1579generally means rebuilding things that do not necessarily need to be
1580rebuilt.
1581
1582The Yocto Project implements shared state code that supports incremental
1583builds. The implementation of the shared state code answers the
1584following questions that were fundamental roadblocks within the
1585OpenEmbedded incremental build support system:
1586
1587- What pieces of the system have changed and what pieces have not
1588 changed?
1589
1590- How are changed pieces of software removed and replaced?
1591
1592- How are pre-built components that do not need to be rebuilt from
1593 scratch used when they are available?
1594
1595For the first question, the build system detects changes in the "inputs"
1596to a given task by creating a checksum (or signature) of the task's
1597inputs. If the checksum changes, the system assumes the inputs have
1598changed and the task needs to be rerun. For the second question, the
1599shared state (sstate) code tracks which tasks add which output to the
1600build process. This means the output from a given task can be removed,
1601upgraded or otherwise manipulated. The third question is partly
1602addressed by the solution for the second question assuming the build
1603system can fetch the sstate objects from remote locations and install
1604them if they are deemed to be valid.
1605
1606.. note::
1607
1608 - The build system does not maintain
1609 :term:`PR` information as part of
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001610 the shared state packages. Consequently, there are considerations that
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001611 affect maintaining shared state feeds. For information on how the
Andrew Geissler09036742021-06-25 14:25:14 -05001612 build system works with packages and can track incrementing :term:`PR`
Andrew Geissler09209ee2020-12-13 08:44:15 -06001613 information, see the ":ref:`dev-manual/common-tasks:automatically incrementing a package version number`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001614 section in the Yocto Project Development Tasks Manual.
1615
1616 - The code in the build system that supports incremental builds is
1617 not simple code. For techniques that help you work around issues
1618 related to shared state code, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001619 ":ref:`dev-manual/common-tasks:viewing metadata used to create the input signature of a shared state task`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001620 and
Andrew Geissler09209ee2020-12-13 08:44:15 -06001621 ":ref:`dev-manual/common-tasks:invalidating shared state to force a task to run`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001622 sections both in the Yocto Project Development Tasks Manual.
1623
1624The rest of this section goes into detail about the overall incremental
1625build architecture, the checksums (signatures), and shared state.
1626
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001627Overall Architecture
1628--------------------
1629
1630When determining what parts of the system need to be built, BitBake
1631works on a per-task basis rather than a per-recipe basis. You might
1632wonder why using a per-task basis is preferred over a per-recipe basis.
1633To help explain, consider having the IPK packaging backend enabled and
1634then switching to DEB. In this case, the
1635:ref:`ref-tasks-install` and
1636:ref:`ref-tasks-package` task outputs
1637are still valid. However, with a per-recipe approach, the build would
1638not include the ``.deb`` files. Consequently, you would have to
1639invalidate the whole build and rerun it. Rerunning everything is not the
1640best solution. Also, in this case, the core must be "taught" much about
1641specific tasks. This methodology does not scale well and does not allow
1642users to easily add new tasks in layers or as external recipes without
1643touching the packaged-staging core.
1644
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001645Checksums (Signatures)
1646----------------------
1647
1648The shared state code uses a checksum, which is a unique signature of a
1649task's inputs, to determine if a task needs to be run again. Because it
1650is a change in a task's inputs that triggers a rerun, the process needs
1651to detect all the inputs to a given task. For shell tasks, this turns
1652out to be fairly easy because the build process generates a "run" shell
1653script for each task and it is possible to create a checksum that gives
1654you a good idea of when the task's data changes.
1655
1656To complicate the problem, there are things that should not be included
1657in the checksum. First, there is the actual specific build path of a
1658given task - the :term:`WORKDIR`. It
1659does not matter if the work directory changes because it should not
1660affect the output for target packages. Also, the build process has the
1661objective of making native or cross packages relocatable.
1662
1663.. note::
1664
1665 Both native and cross packages run on the
1666 build host. However, cross packages generate output for the target
1667 architecture.
1668
Andrew Geissler09036742021-06-25 14:25:14 -05001669The checksum therefore needs to exclude :term:`WORKDIR`. The simplistic
1670approach for excluding the work directory is to set :term:`WORKDIR` to some
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001671fixed value and create the checksum for the "run" script.
1672
1673Another problem results from the "run" scripts containing functions that
1674might or might not get called. The incremental build solution contains
1675code that figures out dependencies between shell functions. This code is
1676used to prune the "run" scripts down to the minimum set, thereby
1677alleviating this problem and making the "run" scripts much more readable
1678as a bonus.
1679
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001680So far, there are solutions for shell scripts. What about Python tasks? The
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001681same approach applies even though these tasks are more difficult. The
1682process needs to figure out what variables a Python function accesses
1683and what functions it calls. Again, the incremental build solution
1684contains code that first figures out the variable and function
1685dependencies, and then creates a checksum for the data used as the input
1686to the task.
1687
Andrew Geissler09036742021-06-25 14:25:14 -05001688Like the :term:`WORKDIR` case, there can be situations where dependencies should be
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001689ignored. For these situations, you can instruct the build process to
Andrew Geisslerc926e172021-05-07 16:11:35 -05001690ignore a dependency by using a line like the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001691
1692 PACKAGE_ARCHS[vardepsexclude] = "MACHINE"
1693
1694This example ensures that the :term:`PACKAGE_ARCHS` variable
1695does not depend on the value of :term:`MACHINE`, even if it does
1696reference it.
1697
1698Equally, there are cases where you need to add dependencies BitBake is
1699not able to find. You can accomplish this by using a line like the
Andrew Geisslerc926e172021-05-07 16:11:35 -05001700following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001701
1702 PACKAGE_ARCHS[vardeps] = "MACHINE"
1703
1704This example explicitly
Andrew Geissler09036742021-06-25 14:25:14 -05001705adds the :term:`MACHINE` variable as a dependency for :term:`PACKAGE_ARCHS`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001706
1707As an example, consider a case with in-line Python where BitBake is not
1708able to figure out dependencies. When running in debug mode (i.e. using
1709``-DDD``), BitBake produces output when it discovers something for which
1710it cannot figure out dependencies. The Yocto Project team has currently
1711not managed to cover those dependencies in detail and is aware of the
1712need to fix this situation.
1713
1714Thus far, this section has limited discussion to the direct inputs into
1715a task. Information based on direct inputs is referred to as the
1716"basehash" in the code. However, the question of a task's indirect
1717inputs still exits - items already built and present in the
1718:term:`Build Directory`. The checksum (or
1719signature) for a particular task needs to add the hashes of all the
1720tasks on which the particular task depends. Choosing which dependencies
1721to add is a policy decision. However, the effect is to generate a master
1722checksum that combines the basehash and the hashes of the task's
1723dependencies.
1724
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001725At the code level, there are multiple ways by which both the basehash
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001726and the dependent task hashes can be influenced. Within the BitBake
1727configuration file, you can give BitBake some extra information to help
1728it construct the basehash. The following statement effectively results
1729in a list of global variable dependency excludes (i.e. variables never
Andrew Geisslerc926e172021-05-07 16:11:35 -05001730included in any checksum)::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001731
1732 BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \\
1733 SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \\
1734 USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \\
1735 PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \\
1736 CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX"
1737
1738The
1739previous example excludes
1740:term:`WORKDIR` since that variable
1741is actually constructed as a path within
1742:term:`TMPDIR`, which is on the
1743whitelist.
1744
1745The rules for deciding which hashes of dependent tasks to include
1746through dependency chains are more complex and are generally
1747accomplished with a Python function. The code in
1748``meta/lib/oe/sstatesig.py`` shows two examples of this and also
1749illustrates how you can insert your own policy into the system if so
1750desired. This file defines the two basic signature generators
1751:term:`OpenEmbedded-Core (OE-Core)` uses: "OEBasic" and
1752"OEBasicHash". By default, a dummy "noop" signature handler is enabled
1753in BitBake. This means that behavior is unchanged from previous
1754versions. OE-Core uses the "OEBasicHash" signature handler by default
Andrew Geisslerc926e172021-05-07 16:11:35 -05001755through this setting in the ``bitbake.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001756
1757 BB_SIGNATURE_HANDLER ?= "OEBasicHash"
1758
Andrew Geissler09036742021-06-25 14:25:14 -05001759The "OEBasicHash" :term:`BB_SIGNATURE_HANDLER` is the same
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001760as the "OEBasic" version but adds the task hash to the :ref:`stamp
1761files <overview-manual/concepts:stamp files and the rerunning of tasks>`. This
1762results in any metadata change that changes the task hash, automatically causing
1763the task to be run again. This removes the need to bump
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001764:term:`PR` values, and changes to metadata
1765automatically ripple across the build.
1766
1767It is also worth noting that the end result of these signature
1768generators is to make some dependency and hash information available to
1769the build. This information includes:
1770
1771- ``BB_BASEHASH_task-``\ taskname: The base hashes for each task in the
1772 recipe.
1773
1774- ``BB_BASEHASH_``\ filename\ ``:``\ taskname: The base hashes for each
1775 dependent task.
1776
1777- ``BBHASHDEPS_``\ filename\ ``:``\ taskname: The task dependencies for
1778 each task.
1779
Andrew Geissler09036742021-06-25 14:25:14 -05001780- :term:`BB_TASKHASH`: The hash of the currently running task.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001781
1782Shared State
1783------------
1784
1785Checksums and dependencies, as discussed in the previous section, solve
1786half the problem of supporting a shared state. The other half of the
1787problem is being able to use checksum information during the build and
1788being able to reuse or rebuild specific components.
1789
1790The :ref:`sstate <ref-classes-sstate>` class is a
1791relatively generic implementation of how to "capture" a snapshot of a
1792given task. The idea is that the build process does not care about the
1793source of a task's output. Output could be freshly built or it could be
1794downloaded and unpacked from somewhere. In other words, the build
1795process does not need to worry about its origin.
1796
1797Two types of output exist. One type is just about creating a directory
1798in :term:`WORKDIR`. A good example is
1799the output of either
1800:ref:`ref-tasks-install` or
1801:ref:`ref-tasks-package`. The other
1802type of output occurs when a set of data is merged into a shared
1803directory tree such as the sysroot.
1804
1805The Yocto Project team has tried to keep the details of the
1806implementation hidden in ``sstate`` class. From a user's perspective,
1807adding shared state wrapping to a task is as simple as this
1808:ref:`ref-tasks-deploy` example taken
Andrew Geisslerc926e172021-05-07 16:11:35 -05001809from the :ref:`deploy <ref-classes-deploy>` class::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001810
1811 DEPLOYDIR = "${WORKDIR}/deploy-${PN}"
1812 SSTATETASKS += "do_deploy"
1813 do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"
1814 do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
1815
1816 python do_deploy_setscene () {
1817 sstate_setscene(d)
1818 }
1819 addtask do_deploy_setscene
1820 do_deploy[dirs] = "${DEPLOYDIR} ${B}"
1821 do_deploy[stamp-extra-info] = "${MACHINE_ARCH}"
1822
1823The following list explains the previous example:
1824
1825- Adding "do_deploy" to ``SSTATETASKS`` adds some required
1826 sstate-related processing, which is implemented in the
1827 :ref:`sstate <ref-classes-sstate>` class, to
1828 before and after the
1829 :ref:`ref-tasks-deploy` task.
1830
1831- The ``do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"`` declares that
1832 ``do_deploy`` places its output in ``${DEPLOYDIR}`` when run normally
1833 (i.e. when not using the sstate cache). This output becomes the input
1834 to the shared state cache.
1835
1836- The ``do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"`` line
1837 causes the contents of the shared state cache to be copied to
1838 ``${DEPLOY_DIR_IMAGE}``.
1839
1840 .. note::
1841
1842 If ``do_deploy`` is not already in the shared state cache or if its input
1843 checksum (signature) has changed from when the output was cached, the task
1844 runs to populate the shared state cache, after which the contents of the
1845 shared state cache is copied to ${:term:`DEPLOY_DIR_IMAGE`}. If
1846 ``do_deploy`` is in the shared state cache and its signature indicates
1847 that the cached output is still valid (i.e. if no relevant task inputs
1848 have changed), then the contents of the shared state cache copies
Andrew Geissler09036742021-06-25 14:25:14 -05001849 directly to ${:term:`DEPLOY_DIR_IMAGE`} by the ``do_deploy_setscene`` task
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001850 instead, skipping the ``do_deploy`` task.
1851
1852- The following task definition is glue logic needed to make the
Andrew Geisslerc926e172021-05-07 16:11:35 -05001853 previous settings effective::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001854
1855 python do_deploy_setscene () {
1856 sstate_setscene(d)
1857 }
1858 addtask do_deploy_setscene
1859
1860 ``sstate_setscene()`` takes the flags above as input and accelerates the ``do_deploy`` task
1861 through the shared state cache if possible. If the task was
1862 accelerated, ``sstate_setscene()`` returns True. Otherwise, it
1863 returns False, and the normal ``do_deploy`` task runs. For more
Patrick Williams213cb262021-08-07 19:21:33 -05001864 information, see the ":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-execution:setscene`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001865 section in the BitBake User Manual.
1866
1867- The ``do_deploy[dirs] = "${DEPLOYDIR} ${B}"`` line creates
1868 ``${DEPLOYDIR}`` and ``${B}`` before the ``do_deploy`` task runs, and
1869 also sets the current working directory of ``do_deploy`` to ``${B}``.
1870 For more information, see the ":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags`"
1871 section in the BitBake
1872 User Manual.
1873
1874 .. note::
1875
1876 In cases where ``sstate-inputdirs`` and ``sstate-outputdirs`` would be
1877 the same, you can use ``sstate-plaindirs``. For example, to preserve the
1878 ${:term:`PKGD`} and ${:term:`PKGDEST`} output from the ``do_package``
Andrew Geisslerc926e172021-05-07 16:11:35 -05001879 task, use the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001880
1881 do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}"
1882
1883
1884- The ``do_deploy[stamp-extra-info] = "${MACHINE_ARCH}"`` line appends
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001885 extra metadata to the :ref:`stamp
1886 file <overview-manual/concepts:stamp files and the rerunning of tasks>`. In
1887 this case, the metadata makes the task specific to a machine's architecture.
1888 See
Andrew Geissler09209ee2020-12-13 08:44:15 -06001889 ":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-execution:the task list`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001890 section in the BitBake User Manual for more information on the
1891 ``stamp-extra-info`` flag.
1892
1893- ``sstate-inputdirs`` and ``sstate-outputdirs`` can also be used with
1894 multiple directories. For example, the following declares
Andrew Geissler09036742021-06-25 14:25:14 -05001895 :term:`PKGDESTWORK` and ``SHLIBWORK`` as shared state input directories,
1896 which populates the shared state cache, and :term:`PKGDATA_DIR` and
Andrew Geisslerc926e172021-05-07 16:11:35 -05001897 ``SHLIBSDIR`` as the corresponding shared state output directories::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001898
1899 do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}"
1900 do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}"
1901
1902- These methods also include the ability to take a lockfile when
1903 manipulating shared state directory structures, for cases where file
Andrew Geisslerc926e172021-05-07 16:11:35 -05001904 additions or removals are sensitive::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001905
1906 do_package[sstate-lockfile] = "${PACKAGELOCK}"
1907
1908Behind the scenes, the shared state code works by looking in
1909:term:`SSTATE_DIR` and
1910:term:`SSTATE_MIRRORS` for
Andrew Geisslerc926e172021-05-07 16:11:35 -05001911shared state files. Here is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001912
1913 SSTATE_MIRRORS ?= "\
1914 file://.\* http://someserver.tld/share/sstate/PATH;downloadfilename=PATH \n \
1915 file://.\* file:///some/local/dir/sstate/PATH"
1916
1917.. note::
1918
Andrew Geissler5f350902021-07-23 13:09:54 -04001919 The shared state directory (:term:`SSTATE_DIR`) is organized into two-character
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001920 subdirectories, where the subdirectory names are based on the first two
1921 characters of the hash.
1922 If the shared state directory structure for a mirror has the same structure
Andrew Geissler09036742021-06-25 14:25:14 -05001923 as :term:`SSTATE_DIR`, you must specify "PATH" as part of the URI to enable the build
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001924 system to map to the appropriate subdirectory.
1925
1926The shared state package validity can be detected just by looking at the
1927filename since the filename contains the task checksum (or signature) as
1928described earlier in this section. If a valid shared state package is
1929found, the build process downloads it and uses it to accelerate the
1930task.
1931
1932The build processes use the ``*_setscene`` tasks for the task
1933acceleration phase. BitBake goes through this phase before the main
1934execution code and tries to accelerate any tasks for which it can find
1935shared state packages. If a shared state package for a task is
1936available, the shared state package is used. This means the task and any
1937tasks on which it is dependent are not executed.
1938
1939As a real world example, the aim is when building an IPK-based image,
1940only the
1941:ref:`ref-tasks-package_write_ipk`
1942tasks would have their shared state packages fetched and extracted.
1943Since the sysroot is not used, it would never get extracted. This is
1944another reason why a task-based approach is preferred over a
1945recipe-based approach, which would have to install the output from every
1946task.
1947
1948Automatically Added Runtime Dependencies
1949========================================
1950
1951The OpenEmbedded build system automatically adds common types of runtime
1952dependencies between packages, which means that you do not need to
1953explicitly declare the packages using
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001954:term:`RDEPENDS`. There are three automatic
1955mechanisms (``shlibdeps``, ``pcdeps``, and ``depchains``) that
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001956handle shared libraries, package configuration (pkg-config) modules, and
1957``-dev`` and ``-dbg`` packages, respectively. For other types of runtime
1958dependencies, you must manually declare the dependencies.
1959
1960- ``shlibdeps``: During the
1961 :ref:`ref-tasks-package` task of
1962 each recipe, all shared libraries installed by the recipe are
1963 located. For each shared library, the package that contains the
1964 shared library is registered as providing the shared library. More
1965 specifically, the package is registered as providing the
1966 `soname <https://en.wikipedia.org/wiki/Soname>`__ of the library. The
1967 resulting shared-library-to-package mapping is saved globally in
1968 :term:`PKGDATA_DIR` by the
1969 :ref:`ref-tasks-packagedata`
1970 task.
1971
1972 Simultaneously, all executables and shared libraries installed by the
1973 recipe are inspected to see what shared libraries they link against.
Andrew Geissler09036742021-06-25 14:25:14 -05001974 For each shared library dependency that is found, :term:`PKGDATA_DIR` is
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001975 queried to see if some package (likely from a different recipe)
1976 contains the shared library. If such a package is found, a runtime
1977 dependency is added from the package that depends on the shared
1978 library to the package that contains the library.
1979
1980 The automatically added runtime dependency also includes a version
1981 restriction. This version restriction specifies that at least the
1982 current version of the package that provides the shared library must
Andrew Geissler09036742021-06-25 14:25:14 -05001983 be used, as if "package (>= version)" had been added to :term:`RDEPENDS`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001984 This forces an upgrade of the package containing the shared library
1985 when installing the package that depends on the library, if needed.
1986
1987 If you want to avoid a package being registered as providing a
1988 particular shared library (e.g. because the library is for internal
1989 use only), then add the library to
1990 :term:`PRIVATE_LIBS` inside
1991 the package's recipe.
1992
1993- ``pcdeps``: During the ``do_package`` task of each recipe, all
1994 pkg-config modules (``*.pc`` files) installed by the recipe are
1995 located. For each module, the package that contains the module is
1996 registered as providing the module. The resulting module-to-package
Andrew Geissler09036742021-06-25 14:25:14 -05001997 mapping is saved globally in :term:`PKGDATA_DIR` by the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001998 ``do_packagedata`` task.
1999
2000 Simultaneously, all pkg-config modules installed by the recipe are
2001 inspected to see what other pkg-config modules they depend on. A
2002 module is seen as depending on another module if it contains a
2003 "Requires:" line that specifies the other module. For each module
Andrew Geissler09036742021-06-25 14:25:14 -05002004 dependency, :term:`PKGDATA_DIR` is queried to see if some package
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002005 contains the module. If such a package is found, a runtime dependency
2006 is added from the package that depends on the module to the package
2007 that contains the module.
2008
2009 .. note::
2010
2011 The
2012 pcdeps
2013 mechanism most often infers dependencies between
2014 -dev
2015 packages.
2016
2017- ``depchains``: If a package ``foo`` depends on a package ``bar``,
2018 then ``foo-dev`` and ``foo-dbg`` are also made to depend on
2019 ``bar-dev`` and ``bar-dbg``, respectively. Taking the ``-dev``
2020 packages as an example, the ``bar-dev`` package might provide headers
2021 and shared library symlinks needed by ``foo-dev``, which shows the
2022 need for a dependency between the packages.
2023
2024 The dependencies added by ``depchains`` are in the form of
2025 :term:`RRECOMMENDS`.
2026
2027 .. note::
2028
Andrew Geissler5f350902021-07-23 13:09:54 -04002029 By default, ``foo-dev`` also has an :term:`RDEPENDS`-style dependency on
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002030 ``foo``, because the default value of ``RDEPENDS_${PN}-dev`` (set in
2031 bitbake.conf) includes "${PN}".
2032
2033 To ensure that the dependency chain is never broken, ``-dev`` and
2034 ``-dbg`` packages are always generated by default, even if the
2035 packages turn out to be empty. See the
2036 :term:`ALLOW_EMPTY` variable
2037 for more information.
2038
2039The ``do_package`` task depends on the ``do_packagedata`` task of each
2040recipe in :term:`DEPENDS` through use
2041of a ``[``\ :ref:`deptask <bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ``]``
2042declaration, which guarantees that the required
2043shared-library/module-to-package mapping information will be available
Andrew Geissler09036742021-06-25 14:25:14 -05002044when needed as long as :term:`DEPENDS` has been correctly set.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002045
2046Fakeroot and Pseudo
2047===================
2048
2049Some tasks are easier to implement when allowed to perform certain
2050operations that are normally reserved for the root user (e.g.
2051:ref:`ref-tasks-install`,
2052:ref:`do_package_write* <ref-tasks-package_write_deb>`,
2053:ref:`ref-tasks-rootfs`, and
2054:ref:`do_image* <ref-tasks-image>`). For example,
2055the ``do_install`` task benefits from being able to set the UID and GID
2056of installed files to arbitrary values.
2057
2058One approach to allowing tasks to perform root-only operations would be
2059to require :term:`BitBake` to run as
2060root. However, this method is cumbersome and has security issues. The
2061approach that is actually used is to run tasks that benefit from root
2062privileges in a "fake" root environment. Within this environment, the
2063task and its child processes believe that they are running as the root
2064user, and see an internally consistent view of the filesystem. As long
2065as generating the final output (e.g. a package or an image) does not
2066require root privileges, the fact that some earlier steps ran in a fake
2067root environment does not cause problems.
2068
2069The capability to run tasks in a fake root environment is known as
2070"`fakeroot <http://man.he.net/man1/fakeroot>`__", which is derived from
2071the BitBake keyword/variable flag that requests a fake root environment
2072for a task.
2073
Andrew Geisslerd1e89492021-02-12 15:35:20 -06002074In the :term:`OpenEmbedded Build System`, the program that implements
2075fakeroot is known as :yocto_home:`Pseudo </software-item/pseudo/>`. Pseudo
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002076overrides system calls by using the environment variable ``LD_PRELOAD``,
2077which results in the illusion of running as root. To keep track of
2078"fake" file ownership and permissions resulting from operations that
2079require root permissions, Pseudo uses an SQLite 3 database. This
2080database is stored in
2081``${``\ :term:`WORKDIR`\ ``}/pseudo/files.db``
2082for individual recipes. Storing the database in a file as opposed to in
2083memory gives persistence between tasks and builds, which is not
2084accomplished using fakeroot.
2085
2086.. note::
2087
2088 If you add your own task that manipulates the same files or
2089 directories as a fakeroot task, then that task also needs to run
2090 under fakeroot. Otherwise, the task cannot run root-only operations,
2091 and cannot see the fake file ownership and permissions set by the
2092 other task. You need to also add a dependency on
Andrew Geisslerc926e172021-05-07 16:11:35 -05002093 ``virtual/fakeroot-native:do_populate_sysroot``, giving the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002094
2095 fakeroot do_mytask () {
2096 ...
2097 }
2098 do_mytask[depends] += "virtual/fakeroot-native:do_populate_sysroot"
2099
2100
2101For more information, see the
2102:term:`FAKEROOT* <bitbake:FAKEROOT>` variables in the
2103BitBake User Manual. You can also reference the "`Why Not
2104Fakeroot? <https://github.com/wrpseudo/pseudo/wiki/WhyNotFakeroot>`__"
2105article for background information on Fakeroot and Pseudo.