blob: 8fbbabbac52a66ac05a4e502e06ab80f6c74f4c1 [file] [log] [blame]
Andrew Geisslerf0343792020-11-18 10:42:21 -06001.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002
3**********************
4Yocto Project Concepts
5**********************
6
7This chapter provides explanations for Yocto Project concepts that go
8beyond the surface of "how-to" information and reference (or look-up)
9material. Concepts such as components, the :term:`OpenEmbedded Build System`
10workflow,
11cross-development toolchains, shared state cache, and so forth are
12explained.
13
14Yocto Project Components
15========================
16
17The :term:`BitBake` task executor
18together with various types of configuration files form the
19:term:`OpenEmbedded-Core (OE-Core)`. This section
20overviews these components by describing their use and how they
21interact.
22
23BitBake handles the parsing and execution of the data files. The data
24itself is of various types:
25
26- *Recipes:* Provides details about particular pieces of software.
27
28- *Class Data:* Abstracts common build information (e.g. how to build a
29 Linux kernel).
30
31- *Configuration Data:* Defines machine-specific settings, policy
32 decisions, and so forth. Configuration data acts as the glue to bind
33 everything together.
34
35BitBake knows how to combine multiple data sources together and refers
36to each data source as a layer. For information on layers, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -060037":ref:`dev-manual/common-tasks:understanding and creating layers`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050038section of the Yocto Project Development Tasks Manual.
39
40Following are some brief details on these core components. For
41additional information on how these components interact during a build,
42see the
Andrew Geissler09209ee2020-12-13 08:44:15 -060043":ref:`overview-manual/concepts:openembedded build system concepts`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050044section.
45
Andrew Geisslerc9f78652020-09-18 14:11:35 -050046BitBake
47-------
48
49BitBake is the tool at the heart of the :term:`OpenEmbedded Build System`
50and is responsible
51for parsing the :term:`Metadata`, generating
52a list of tasks from it, and then executing those tasks.
53
54This section briefly introduces BitBake. If you want more information on
55BitBake, see the :doc:`BitBake User Manual <bitbake:index>`.
56
57To see a list of the options BitBake supports, use either of the
58following commands:
59::
60
61 $ bitbake -h
62 $ bitbake --help
63
64The most common usage for BitBake is ``bitbake recipename``, where
65``recipename`` is the name of the recipe you want to build (referred
66to as the "target"). The target often equates to the first part of a
67recipe's filename (e.g. "foo" for a recipe named ``foo_1.3.0-r0.bb``).
68So, to process the ``matchbox-desktop_1.2.3.bb`` recipe file, you might
69type the following:
70::
71
72 $ bitbake matchbox-desktop
73
74Several different
75versions of ``matchbox-desktop`` might exist. BitBake chooses the one
76selected by the distribution configuration. You can get more details
77about how BitBake chooses between different target versions and
78providers in the
Andrew Geissler09209ee2020-12-13 08:44:15 -060079":ref:`Preferences <bitbake:bitbake-user-manual/bitbake-user-manual-execution:preferences>`" section
Andrew Geisslerc9f78652020-09-18 14:11:35 -050080of the BitBake User Manual.
81
82BitBake also tries to execute any dependent tasks first. So for example,
83before building ``matchbox-desktop``, BitBake would build a cross
84compiler and ``glibc`` if they had not already been built.
85
86A useful BitBake option to consider is the ``-k`` or ``--continue``
87option. This option instructs BitBake to try and continue processing the
88job as long as possible even after encountering an error. When an error
89occurs, the target that failed and those that depend on it cannot be
90remade. However, when you use this option other dependencies can still
91be processed.
92
Andrew Geisslerc9f78652020-09-18 14:11:35 -050093Recipes
94-------
95
96Files that have the ``.bb`` suffix are "recipes" files. In general, a
97recipe contains information about a single piece of software. This
98information includes the location from which to download the unaltered
99source, any source patches to be applied to that source (if needed),
100which special configuration options to apply, how to compile the source
101files, and how to package the compiled output.
102
103The term "package" is sometimes used to refer to recipes. However, since
104the word "package" is used for the packaged output from the OpenEmbedded
105build system (i.e. ``.ipk`` or ``.deb`` files), this document avoids
106using the term "package" when referring to recipes.
107
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500108Classes
109-------
110
111Class files (``.bbclass``) contain information that is useful to share
112between recipes files. An example is the
113:ref:`autotools <ref-classes-autotools>` class,
114which contains common settings for any application that Autotools uses.
Andrew Geissler09209ee2020-12-13 08:44:15 -0600115The ":ref:`ref-manual/classes:Classes`" chapter in the
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500116Yocto Project Reference Manual provides details about classes and how to
117use them.
118
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500119Configurations
120--------------
121
122The configuration files (``.conf``) define various configuration
123variables that govern the OpenEmbedded build process. These files fall
124into several areas that define machine configuration options,
125distribution configuration options, compiler tuning options, general
126common configuration options, and user configuration options in
127``conf/local.conf``, which is found in the :term:`Build Directory`.
128
129
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500130Layers
131======
132
133Layers are repositories that contain related metadata (i.e. sets of
134instructions) that tell the OpenEmbedded build system how to build a
135target. Yocto Project's `layer model <#the-yocto-project-layer-model>`__
136facilitates collaboration, sharing, customization, and reuse within the
137Yocto Project development environment. Layers logically separate
138information for your project. For example, you can use a layer to hold
139all the configurations for a particular piece of hardware. Isolating
140hardware-specific configurations allows you to share other metadata by
141using a different layer where that metadata might be common across
142several pieces of hardware.
143
144Many layers exist that work in the Yocto Project development
145environment. The `Yocto Project Curated Layer
Andrew Geisslerc3d88e42020-10-02 09:45:00 -0500146Index <https://www.yoctoproject.org/software-overview/layers/>`__
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500147and `OpenEmbedded Layer
148Index <http://layers.openembedded.org/layerindex/branch/master/layers/>`__
149both contain layers from which you can use or leverage.
150
151By convention, layers in the Yocto Project follow a specific form.
152Conforming to a known structure allows BitBake to make assumptions
153during builds on where to find types of metadata. You can find
154procedures and learn about tools (i.e. ``bitbake-layers``) for creating
155layers suitable for the Yocto Project in the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600156":ref:`dev-manual/common-tasks:understanding and creating layers`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500157section of the Yocto Project Development Tasks Manual.
158
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500159OpenEmbedded Build System Concepts
160==================================
161
162This section takes a more detailed look inside the build process used by
163the :term:`OpenEmbedded Build System`,
164which is the build
165system specific to the Yocto Project. At the heart of the build system
166is BitBake, the task executor.
167
168The following diagram represents the high-level workflow of a build. The
169remainder of this section expands on the fundamental input, output,
170process, and metadata logical blocks that make up the workflow.
171
172.. image:: figures/YP-flow-diagram.png
173 :align: center
174
175In general, the build's workflow consists of several functional areas:
176
177- *User Configuration:* metadata you can use to control the build
178 process.
179
180- *Metadata Layers:* Various layers that provide software, machine, and
181 distro metadata.
182
183- *Source Files:* Upstream releases, local projects, and SCMs.
184
185- *Build System:* Processes under the control of
186 :term:`BitBake`. This block expands
187 on how BitBake fetches source, applies patches, completes
188 compilation, analyzes output for package generation, creates and
189 tests packages, generates images, and generates cross-development
190 tools.
191
192- *Package Feeds:* Directories containing output packages (RPM, DEB or
193 IPK), which are subsequently used in the construction of an image or
194 Software Development Kit (SDK), produced by the build system. These
195 feeds can also be copied and shared using a web server or other means
196 to facilitate extending or updating existing images on devices at
197 runtime if runtime package management is enabled.
198
199- *Images:* Images produced by the workflow.
200
201- *Application Development SDK:* Cross-development tools that are
202 produced along with an image or separately with BitBake.
203
204User Configuration
205------------------
206
207User configuration helps define the build. Through user configuration,
208you can tell BitBake the target architecture for which you are building
209the image, where to store downloaded source, and other build properties.
210
211The following figure shows an expanded representation of the "User
212Configuration" box of the `general workflow
213figure <#general-workflow-figure>`__:
214
215.. image:: figures/user-configuration.png
216 :align: center
217
218BitBake needs some basic configuration files in order to complete a
219build. These files are ``*.conf`` files. The minimally necessary ones
220reside as example files in the ``build/conf`` directory of the
221:term:`Source Directory`. For simplicity,
222this section refers to the Source Directory as the "Poky Directory."
223
224When you clone the :term:`Poky` Git repository
225or you download and unpack a Yocto Project release, you can set up the
226Source Directory to be named anything you want. For this discussion, the
227cloned repository uses the default name ``poky``.
228
229.. note::
230
231 The Poky repository is primarily an aggregation of existing
232 repositories. It is not a canonical upstream source.
233
234The ``meta-poky`` layer inside Poky contains a ``conf`` directory that
235has example configuration files. These example files are used as a basis
236for creating actual configuration files when you source
237:ref:`structure-core-script`, which is the
238build environment script.
239
240Sourcing the build environment script creates a
241:term:`Build Directory` if one does not
242already exist. BitBake uses the Build Directory for all its work during
243builds. The Build Directory has a ``conf`` directory that contains
244default versions of your ``local.conf`` and ``bblayers.conf``
245configuration files. These default configuration files are created only
246if versions do not already exist in the Build Directory at the time you
247source the build environment setup script.
248
249Because the Poky repository is fundamentally an aggregation of existing
250repositories, some users might be familiar with running the
251:ref:`structure-core-script` script in the context of separate
252:term:`OpenEmbedded-Core (OE-Core)` and BitBake
253repositories rather than a single Poky repository. This discussion
254assumes the script is executed from within a cloned or unpacked version
255of Poky.
256
257Depending on where the script is sourced, different sub-scripts are
258called to set up the Build Directory (Yocto or OpenEmbedded).
259Specifically, the script ``scripts/oe-setup-builddir`` inside the poky
260directory sets up the Build Directory and seeds the directory (if
261necessary) with configuration files appropriate for the Yocto Project
262development environment.
263
264.. note::
265
266 The
267 scripts/oe-setup-builddir
268 script uses the
269 ``$TEMPLATECONF``
270 variable to determine which sample configuration files to locate.
271
272The ``local.conf`` file provides many basic variables that define a
273build environment. Here is a list of a few. To see the default
274configurations in a ``local.conf`` file created by the build environment
275script, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600276:yocto_git:`local.conf.sample </poky/tree/meta-poky/conf/local.conf.sample>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500277in the ``meta-poky`` layer:
278
279- *Target Machine Selection:* Controlled by the
280 :term:`MACHINE` variable.
281
282- *Download Directory:* Controlled by the
283 :term:`DL_DIR` variable.
284
285- *Shared State Directory:* Controlled by the
286 :term:`SSTATE_DIR` variable.
287
288- *Build Output:* Controlled by the
289 :term:`TMPDIR` variable.
290
291- *Distribution Policy:* Controlled by the
292 :term:`DISTRO` variable.
293
294- *Packaging Format:* Controlled by the
295 :term:`PACKAGE_CLASSES`
296 variable.
297
298- *SDK Target Architecture:* Controlled by the
299 :term:`SDKMACHINE` variable.
300
301- *Extra Image Packages:* Controlled by the
302 :term:`EXTRA_IMAGE_FEATURES`
303 variable.
304
305.. note::
306
307 Configurations set in the
308 conf/local.conf
309 file can also be set in the
310 conf/site.conf
311 and
312 conf/auto.conf
313 configuration files.
314
315The ``bblayers.conf`` file tells BitBake what layers you want considered
316during the build. By default, the layers listed in this file include
317layers minimally needed by the build system. However, you must manually
318add any custom layers you have created. You can find more information on
319working with the ``bblayers.conf`` file in the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600320":ref:`dev-manual/common-tasks:enabling your layer`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500321section in the Yocto Project Development Tasks Manual.
322
323The files ``site.conf`` and ``auto.conf`` are not created by the
324environment initialization script. If you want the ``site.conf`` file,
325you need to create that yourself. The ``auto.conf`` file is typically
326created by an autobuilder:
327
328- *site.conf:* You can use the ``conf/site.conf`` configuration
329 file to configure multiple build directories. For example, suppose
330 you had several build environments and they shared some common
331 features. You can set these default build properties here. A good
332 example is perhaps the packaging format to use through the
333 :term:`PACKAGE_CLASSES`
334 variable.
335
336 One useful scenario for using the ``conf/site.conf`` file is to
337 extend your :term:`BBPATH` variable
338 to include the path to a ``conf/site.conf``. Then, when BitBake looks
339 for Metadata using ``BBPATH``, it finds the ``conf/site.conf`` file
340 and applies your common configurations found in the file. To override
341 configurations in a particular build directory, alter the similar
342 configurations within that build directory's ``conf/local.conf``
343 file.
344
345- *auto.conf:* The file is usually created and written to by an
346 autobuilder. The settings put into the file are typically the same as
347 you would find in the ``conf/local.conf`` or the ``conf/site.conf``
348 files.
349
350You can edit all configuration files to further define any particular
351build environment. This process is represented by the "User
352Configuration Edits" box in the figure.
353
354When you launch your build with the ``bitbake target`` command, BitBake
355sorts out the configurations to ultimately define your build
356environment. It is important to understand that the
357:term:`OpenEmbedded Build System` reads the
358configuration files in a specific order: ``site.conf``, ``auto.conf``,
359and ``local.conf``. And, the build system applies the normal assignment
360statement rules as described in the
361":doc:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata`" chapter
362of the BitBake User Manual. Because the files are parsed in a specific
363order, variable assignments for the same variable could be affected. For
364example, if the ``auto.conf`` file and the ``local.conf`` set variable1
365to different values, because the build system parses ``local.conf``
366after ``auto.conf``, variable1 is assigned the value from the
367``local.conf`` file.
368
369Metadata, Machine Configuration, and Policy Configuration
370---------------------------------------------------------
371
372The previous section described the user configurations that define
373BitBake's global behavior. This section takes a closer look at the
374layers the build system uses to further control the build. These layers
375provide Metadata for the software, machine, and policies.
376
377In general, three types of layer input exists. You can see them below
378the "User Configuration" box in the `general workflow
379figure <#general-workflow-figure>`__:
380
381- *Metadata (.bb + Patches):* Software layers containing
382 user-supplied recipe files, patches, and append files. A good example
383 of a software layer might be the
384 `meta-qt5 layer <https://github.com/meta-qt5/meta-qt5>`__ from
385 the `OpenEmbedded Layer
386 Index <http://layers.openembedded.org/layerindex/branch/master/layers/>`__.
387 This layer is for version 5.0 of the popular
388 `Qt <https://wiki.qt.io/About_Qt>`__ cross-platform application
389 development framework for desktop, embedded and mobile.
390
391- *Machine BSP Configuration:* Board Support Package (BSP) layers (i.e.
392 "BSP Layer" in the following figure) providing machine-specific
393 configurations. This type of information is specific to a particular
394 target architecture. A good example of a BSP layer from the `Poky
395 Reference Distribution <#gs-reference-distribution-poky>`__ is the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600396 :yocto_git:`meta-yocto-bsp </poky/tree/meta-yocto-bsp>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500397 layer.
398
399- *Policy Configuration:* Distribution Layers (i.e. "Distro Layer" in
400 the following figure) providing top-level or general policies for the
401 images or SDKs being built for a particular distribution. For
402 example, in the Poky Reference Distribution the distro layer is the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600403 :yocto_git:`meta-poky </poky/tree/meta-poky>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500404 layer. Within the distro layer is a ``conf/distro`` directory that
405 contains distro configuration files (e.g.
Andrew Geissler09209ee2020-12-13 08:44:15 -0600406 :yocto_git:`poky.conf </poky/tree/meta-poky/conf/distro/poky.conf>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500407 that contain many policy configurations for the Poky distribution.
408
409The following figure shows an expanded representation of these three
410layers from the `general workflow figure <#general-workflow-figure>`__:
411
412.. image:: figures/layer-input.png
413 :align: center
414
415In general, all layers have a similar structure. They all contain a
416licensing file (e.g. ``COPYING.MIT``) if the layer is to be distributed,
417a ``README`` file as good practice and especially if the layer is to be
418distributed, a configuration directory, and recipe directories. You can
419learn about the general structure for layers used with the Yocto Project
420in the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600421":ref:`dev-manual/common-tasks:creating your own layer`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500422section in the
423Yocto Project Development Tasks Manual. For a general discussion on
424layers and the many layers from which you can draw, see the
425"`Layers <#overview-layers>`__" and "`The Yocto Project Layer
426Model <#the-yocto-project-layer-model>`__" sections both earlier in this
427manual.
428
429If you explored the previous links, you discovered some areas where many
Andrew Geissler09209ee2020-12-13 08:44:15 -0600430layers that work with the Yocto Project exist. The :yocto_git:`Source
431Repositories <>` also shows layers categorized under "Yocto Metadata Layers."
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500432
433.. note::
434
435 Layers exist in the Yocto Project Source Repositories that cannot be
436 found in the OpenEmbedded Layer Index. These layers are either
437 deprecated or experimental in nature.
438
439BitBake uses the ``conf/bblayers.conf`` file, which is part of the user
440configuration, to find what layers it should be using as part of the
441build.
442
443Distro Layer
444~~~~~~~~~~~~
445
446The distribution layer provides policy configurations for your
447distribution. Best practices dictate that you isolate these types of
448configurations into their own layer. Settings you provide in
449``conf/distro/distro.conf`` override similar settings that BitBake finds
450in your ``conf/local.conf`` file in the Build Directory.
451
452The following list provides some explanation and references for what you
453typically find in the distribution layer:
454
455- *classes:* Class files (``.bbclass``) hold common functionality that
456 can be shared among recipes in the distribution. When your recipes
457 inherit a class, they take on the settings and functions for that
458 class. You can read more about class files in the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600459 ":ref:`ref-manual/classes:Classes`" chapter of the Yocto
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500460 Reference Manual.
461
462- *conf:* This area holds configuration files for the layer
463 (``conf/layer.conf``), the distribution
464 (``conf/distro/distro.conf``), and any distribution-wide include
465 files.
466
467- *recipes-*:* Recipes and append files that affect common
468 functionality across the distribution. This area could include
469 recipes and append files to add distribution-specific configuration,
470 initialization scripts, custom image recipes, and so forth. Examples
471 of ``recipes-*`` directories are ``recipes-core`` and
472 ``recipes-extra``. Hierarchy and contents within a ``recipes-*``
473 directory can vary. Generally, these directories contain recipe files
474 (``*.bb``), recipe append files (``*.bbappend``), directories that
475 are distro-specific for configuration files, and so forth.
476
477BSP Layer
478~~~~~~~~~
479
480The BSP Layer provides machine configurations that target specific
481hardware. Everything in this layer is specific to the machine for which
482you are building the image or the SDK. A common structure or form is
483defined for BSP layers. You can learn more about this structure in the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600484:doc:`/bsp-guide/index`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500485
486.. note::
487
488 In order for a BSP layer to be considered compliant with the Yocto
489 Project, it must meet some structural requirements.
490
491The BSP Layer's configuration directory contains configuration files for
492the machine (``conf/machine/machine.conf``) and, of course, the layer
493(``conf/layer.conf``).
494
495The remainder of the layer is dedicated to specific recipes by function:
496``recipes-bsp``, ``recipes-core``, ``recipes-graphics``,
497``recipes-kernel``, and so forth. Metadata can exist for multiple
498formfactors, graphics support systems, and so forth.
499
500.. note::
501
502 While the figure shows several
503 recipes-\*
504 directories, not all these directories appear in all BSP layers.
505
506Software Layer
507~~~~~~~~~~~~~~
508
509The software layer provides the Metadata for additional software
510packages used during the build. This layer does not include Metadata
511that is specific to the distribution or the machine, which are found in
512their respective layers.
513
514This layer contains any recipes, append files, and patches, that your
515project needs.
516
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500517Sources
518-------
519
520In order for the OpenEmbedded build system to create an image or any
521target, it must be able to access source files. The `general workflow
522figure <#general-workflow-figure>`__ represents source files using the
523"Upstream Project Releases", "Local Projects", and "SCMs (optional)"
524boxes. The figure represents mirrors, which also play a role in locating
525source files, with the "Source Materials" box.
526
527The method by which source files are ultimately organized is a function
528of the project. For example, for released software, projects tend to use
529tarballs or other archived files that can capture the state of a release
530guaranteeing that it is statically represented. On the other hand, for a
531project that is more dynamic or experimental in nature, a project might
532keep source files in a repository controlled by a Source Control Manager
533(SCM) such as Git. Pulling source from a repository allows you to
534control the point in the repository (the revision) from which you want
535to build software. Finally, a combination of the two might exist, which
536would give the consumer a choice when deciding where to get source
537files.
538
539BitBake uses the :term:`SRC_URI`
540variable to point to source files regardless of their location. Each
541recipe must have a ``SRC_URI`` variable that points to the source.
542
543Another area that plays a significant role in where source files come
544from is pointed to by the
545:term:`DL_DIR` variable. This area is
546a cache that can hold previously downloaded source. You can also
547instruct the OpenEmbedded build system to create tarballs from Git
548repositories, which is not the default behavior, and store them in the
549``DL_DIR`` by using the
550:term:`BB_GENERATE_MIRROR_TARBALLS`
551variable.
552
553Judicious use of a ``DL_DIR`` directory can save the build system a trip
554across the Internet when looking for files. A good method for using a
555download directory is to have ``DL_DIR`` point to an area outside of
556your Build Directory. Doing so allows you to safely delete the Build
557Directory if needed without fear of removing any downloaded source file.
558
559The remainder of this section provides a deeper look into the source
560files and the mirrors. Here is a more detailed look at the source file
561area of the `general workflow figure <#general-workflow-figure>`__:
562
563.. image:: figures/source-input.png
564 :align: center
565
566Upstream Project Releases
567~~~~~~~~~~~~~~~~~~~~~~~~~
568
569Upstream project releases exist anywhere in the form of an archived file
570(e.g. tarball or zip file). These files correspond to individual
571recipes. For example, the figure uses specific releases each for
572BusyBox, Qt, and Dbus. An archive file can be for any released product
573that can be built using a recipe.
574
575Local Projects
576~~~~~~~~~~~~~~
577
578Local projects are custom bits of software the user provides. These bits
579reside somewhere local to a project - perhaps a directory into which the
580user checks in items (e.g. a local directory containing a development
581source tree used by the group).
582
583The canonical method through which to include a local project is to use
584the :ref:`externalsrc <ref-classes-externalsrc>`
585class to include that local project. You use either the ``local.conf``
586or a recipe's append file to override or set the recipe to point to the
587local directory on your disk to pull in the whole source tree.
588
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500589Source Control Managers (Optional)
590~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
591
592Another place from which the build system can get source files is with
Andrew Geissler09209ee2020-12-13 08:44:15 -0600593:ref:`fetchers <bitbake:bitbake-user-manual/bitbake-user-manual-fetching:fetchers>` employing various Source
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500594Control Managers (SCMs) such as Git or Subversion. In such cases, a
595repository is cloned or checked out. The
596:ref:`ref-tasks-fetch` task inside
597BitBake uses the :term:`SRC_URI`
598variable and the argument's prefix to determine the correct fetcher
599module.
600
601.. note::
602
603 For information on how to have the OpenEmbedded build system generate
604 tarballs for Git repositories and place them in the
605 DL_DIR
606 directory, see the :term:`BB_GENERATE_MIRROR_TARBALLS`
607 variable in the Yocto Project Reference Manual.
608
609When fetching a repository, BitBake uses the
610:term:`SRCREV` variable to determine
611the specific revision from which to build.
612
613Source Mirror(s)
614~~~~~~~~~~~~~~~~
615
616Two kinds of mirrors exist: pre-mirrors and regular mirrors. The
617:term:`PREMIRRORS` and
618:term:`MIRRORS` variables point to
619these, respectively. BitBake checks pre-mirrors before looking upstream
620for any source files. Pre-mirrors are appropriate when you have a shared
621directory that is not a directory defined by the
622:term:`DL_DIR` variable. A Pre-mirror
623typically points to a shared directory that is local to your
624organization.
625
626Regular mirrors can be any site across the Internet that is used as an
627alternative location for source code should the primary site not be
628functioning for some reason or another.
629
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500630Package Feeds
631-------------
632
633When the OpenEmbedded build system generates an image or an SDK, it gets
634the packages from a package feed area located in the
635:term:`Build Directory`. The `general
636workflow figure <#general-workflow-figure>`__ shows this package feeds
637area in the upper-right corner.
638
639This section looks a little closer into the package feeds area used by
640the build system. Here is a more detailed look at the area:
641
642.. image:: figures/package-feeds.png
643 :align: center
644
645Package feeds are an intermediary step in the build process. The
646OpenEmbedded build system provides classes to generate different package
647types, and you specify which classes to enable through the
648:term:`PACKAGE_CLASSES`
649variable. Before placing the packages into package feeds, the build
650process validates them with generated output quality assurance checks
651through the :ref:`insane <ref-classes-insane>`
652class.
653
654The package feed area resides in the Build Directory. The directory the
655build system uses to temporarily store packages is determined by a
656combination of variables and the particular package manager in use. See
657the "Package Feeds" box in the illustration and note the information to
658the right of that area. In particular, the following defines where
659package files are kept:
660
661- :term:`DEPLOY_DIR`: Defined as
662 ``tmp/deploy`` in the Build Directory.
663
664- ``DEPLOY_DIR_*``: Depending on the package manager used, the package
665 type sub-folder. Given RPM, IPK, or DEB packaging and tarball
666 creation, the
667 :term:`DEPLOY_DIR_RPM`,
668 :term:`DEPLOY_DIR_IPK`,
669 :term:`DEPLOY_DIR_DEB`, or
670 :term:`DEPLOY_DIR_TAR`,
671 variables are used, respectively.
672
673- :term:`PACKAGE_ARCH`: Defines
674 architecture-specific sub-folders. For example, packages could exist
675 for the i586 or qemux86 architectures.
676
677BitBake uses the
678:ref:`do_package_write_* <ref-tasks-package_write_deb>`
679tasks to generate packages and place them into the package holding area
680(e.g. ``do_package_write_ipk`` for IPK packages). See the
681":ref:`ref-tasks-package_write_deb`",
682":ref:`ref-tasks-package_write_ipk`",
683":ref:`ref-tasks-package_write_rpm`",
684and
685":ref:`ref-tasks-package_write_tar`"
686sections in the Yocto Project Reference Manual for additional
687information. As an example, consider a scenario where an IPK packaging
688manager is being used and package architecture support for both i586 and
689qemux86 exist. Packages for the i586 architecture are placed in
690``build/tmp/deploy/ipk/i586``, while packages for the qemux86
691architecture are placed in ``build/tmp/deploy/ipk/qemux86``.
692
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500693BitBake Tool
694------------
695
696The OpenEmbedded build system uses
697:term:`BitBake` to produce images and
698Software Development Kits (SDKs). You can see from the `general workflow
699figure <#general-workflow-figure>`__, the BitBake area consists of
700several functional areas. This section takes a closer look at each of
701those areas.
702
703.. note::
704
705 Separate documentation exists for the BitBake tool. See the
706 BitBake User Manual
707 for reference material on BitBake.
708
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500709Source Fetching
710~~~~~~~~~~~~~~~
711
712The first stages of building a recipe are to fetch and unpack the source
713code:
714
715.. image:: figures/source-fetching.png
716 :align: center
717
718The :ref:`ref-tasks-fetch` and
719:ref:`ref-tasks-unpack` tasks fetch
720the source files and unpack them into the
721:term:`Build Directory`.
722
723.. note::
724
725 For every local file (e.g.
726 file://
727 ) that is part of a recipe's
728 SRC_URI
729 statement, the OpenEmbedded build system takes a checksum of the file
730 for the recipe and inserts the checksum into the signature for the
731 do_fetch
732 task. If any local file has been modified, the
733 do_fetch
734 task and all tasks that depend on it are re-executed.
735
736By default, everything is accomplished in the Build Directory, which has
737a defined structure. For additional general information on the Build
738Directory, see the ":ref:`structure-core-build`" section in
739the Yocto Project Reference Manual.
740
741Each recipe has an area in the Build Directory where the unpacked source
742code resides. The :term:`S` variable points
743to this area for a recipe's unpacked source code. The name of that
744directory for any given recipe is defined from several different
745variables. The preceding figure and the following list describe the
746Build Directory's hierarchy:
747
748- :term:`TMPDIR`: The base directory
749 where the OpenEmbedded build system performs all its work during the
750 build. The default base directory is the ``tmp`` directory.
751
752- :term:`PACKAGE_ARCH`: The
753 architecture of the built package or packages. Depending on the
754 eventual destination of the package or packages (i.e. machine
755 architecture, :term:`Build Host`, SDK, or
756 specific machine), ``PACKAGE_ARCH`` varies. See the variable's
757 description for details.
758
759- :term:`TARGET_OS`: The operating
760 system of the target device. A typical value would be "linux" (e.g.
761 "qemux86-poky-linux").
762
763- :term:`PN`: The name of the recipe used
764 to build the package. This variable can have multiple meanings.
765 However, when used in the context of input files, ``PN`` represents
766 the name of the recipe.
767
768- :term:`WORKDIR`: The location
769 where the OpenEmbedded build system builds a recipe (i.e. does the
770 work to create the package).
771
772 - :term:`PV`: The version of the
773 recipe used to build the package.
774
775 - :term:`PR`: The revision of the
776 recipe used to build the package.
777
778- :term:`S`: Contains the unpacked source
779 files for a given recipe.
780
781 - :term:`BPN`: The name of the recipe
782 used to build the package. The ``BPN`` variable is a version of
783 the ``PN`` variable but with common prefixes and suffixes removed.
784
785 - :term:`PV`: The version of the
786 recipe used to build the package.
787
788.. note::
789
790 In the previous figure, notice that two sample hierarchies exist: one
791 based on package architecture (i.e.
792 PACKAGE_ARCH
793 ) and one based on a machine (i.e.
794 MACHINE
795 ). The underlying structures are identical. The differentiator being
796 what the OpenEmbedded build system is using as a build target (e.g.
797 general architecture, a build host, an SDK, or a specific machine).
798
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500799Patching
800~~~~~~~~
801
802Once source code is fetched and unpacked, BitBake locates patch files
803and applies them to the source files:
804
805.. image:: figures/patching.png
806 :align: center
807
808The :ref:`ref-tasks-patch` task uses a
809recipe's :term:`SRC_URI` statements
810and the :term:`FILESPATH` variable
811to locate applicable patch files.
812
813Default processing for patch files assumes the files have either
814``*.patch`` or ``*.diff`` file types. You can use ``SRC_URI`` parameters
815to change the way the build system recognizes patch files. See the
816:ref:`ref-tasks-patch` task for more
817information.
818
819BitBake finds and applies multiple patches for a single recipe in the
820order in which it locates the patches. The ``FILESPATH`` variable
821defines the default set of directories that the build system uses to
822search for patch files. Once found, patches are applied to the recipe's
823source files, which are located in the
824:term:`S` directory.
825
826For more information on how the source directories are created, see the
827"`Source Fetching <#source-fetching-dev-environment>`__" section. For
828more information on how to create patches and how the build system
829processes patches, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600830":ref:`dev-manual/common-tasks:patching code`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500831section in the
832Yocto Project Development Tasks Manual. You can also see the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600833":ref:`sdk-manual/extensible:use \`\`devtool modify\`\` to modify the source of an existing component`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500834section in the Yocto Project Application Development and the Extensible
835Software Development Kit (SDK) manual and the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600836":ref:`kernel-dev/common:using traditional kernel development to patch the kernel`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500837section in the Yocto Project Linux Kernel Development Manual.
838
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500839Configuration, Compilation, and Staging
840~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
841
842After source code is patched, BitBake executes tasks that configure and
843compile the source code. Once compilation occurs, the files are copied
844to a holding area (staged) in preparation for packaging:
845
846.. image:: figures/configuration-compile-autoreconf.png
847 :align: center
848
849This step in the build process consists of the following tasks:
850
851- :ref:`ref-tasks-prepare_recipe_sysroot`:
852 This task sets up the two sysroots in
853 ``${``\ :term:`WORKDIR`\ ``}``
854 (i.e. ``recipe-sysroot`` and ``recipe-sysroot-native``) so that
855 during the packaging phase the sysroots can contain the contents of
856 the
857 :ref:`ref-tasks-populate_sysroot`
858 tasks of the recipes on which the recipe containing the tasks
859 depends. A sysroot exists for both the target and for the native
860 binaries, which run on the host system.
861
862- *do_configure*: This task configures the source by enabling and
863 disabling any build-time and configuration options for the software
864 being built. Configurations can come from the recipe itself as well
865 as from an inherited class. Additionally, the software itself might
866 configure itself depending on the target for which it is being built.
867
868 The configurations handled by the
869 :ref:`ref-tasks-configure` task
870 are specific to configurations for the source code being built by the
871 recipe.
872
873 If you are using the
874 :ref:`autotools <ref-classes-autotools>` class,
875 you can add additional configuration options by using the
876 :term:`EXTRA_OECONF` or
877 :term:`PACKAGECONFIG_CONFARGS`
878 variables. For information on how this variable works within that
879 class, see the
880 :ref:`autotools <ref-classes-autotools>` class
Andrew Geissler09209ee2020-12-13 08:44:15 -0600881 :yocto_git:`here </poky/tree/meta/classes/autotools.bbclass>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500882
883- *do_compile*: Once a configuration task has been satisfied,
884 BitBake compiles the source using the
885 :ref:`ref-tasks-compile` task.
886 Compilation occurs in the directory pointed to by the
887 :term:`B` variable. Realize that the
888 ``B`` directory is, by default, the same as the
889 :term:`S` directory.
890
891- *do_install*: After compilation completes, BitBake executes the
892 :ref:`ref-tasks-install` task.
893 This task copies files from the ``B`` directory and places them in a
894 holding area pointed to by the :term:`D`
895 variable. Packaging occurs later using files from this holding
896 directory.
897
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500898Package Splitting
899~~~~~~~~~~~~~~~~~
900
901After source code is configured, compiled, and staged, the build system
902analyzes the results and splits the output into packages:
903
904.. image:: figures/analysis-for-package-splitting.png
905 :align: center
906
907The :ref:`ref-tasks-package` and
908:ref:`ref-tasks-packagedata`
909tasks combine to analyze the files found in the
910:term:`D` directory and split them into
911subsets based on available packages and files. Analysis involves the
912following as well as other items: splitting out debugging symbols,
913looking at shared library dependencies between packages, and looking at
914package relationships.
915
916The ``do_packagedata`` task creates package metadata based on the
917analysis such that the build system can generate the final packages. The
918:ref:`ref-tasks-populate_sysroot`
919task stages (copies) a subset of the files installed by the
920:ref:`ref-tasks-install` task into
921the appropriate sysroot. Working, staged, and intermediate results of
922the analysis and package splitting process use several areas:
923
924- :term:`PKGD`: The destination
925 directory (i.e. ``package``) for packages before they are split into
926 individual packages.
927
928- :term:`PKGDESTWORK`: A
929 temporary work area (i.e. ``pkgdata``) used by the ``do_package``
930 task to save package metadata.
931
932- :term:`PKGDEST`: The parent
933 directory (i.e. ``packages-split``) for packages after they have been
934 split.
935
936- :term:`PKGDATA_DIR`: A shared,
937 global-state directory that holds packaging metadata generated during
938 the packaging process. The packaging process copies metadata from
939 ``PKGDESTWORK`` to the ``PKGDATA_DIR`` area where it becomes globally
940 available.
941
942- :term:`STAGING_DIR_HOST`:
943 The path for the sysroot for the system on which a component is built
944 to run (i.e. ``recipe-sysroot``).
945
946- :term:`STAGING_DIR_NATIVE`:
947 The path for the sysroot used when building components for the build
948 host (i.e. ``recipe-sysroot-native``).
949
950- :term:`STAGING_DIR_TARGET`:
951 The path for the sysroot used when a component that is built to
952 execute on a system and it generates code for yet another machine
953 (e.g. cross-canadian recipes).
954
955The :term:`FILES` variable defines the
956files that go into each package in
957:term:`PACKAGES`. If you want
958details on how this is accomplished, you can look at
Andrew Geissler09209ee2020-12-13 08:44:15 -0600959:yocto_git:`package.bbclass </poky/tree/meta/classes/package.bbclass>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500960
961Depending on the type of packages being created (RPM, DEB, or IPK), the
962:ref:`do_package_write_* <ref-tasks-package_write_deb>`
963task creates the actual packages and places them in the Package Feed
964area, which is ``${TMPDIR}/deploy``. You can see the "`Package
965Feeds <#package-feeds-dev-environment>`__" section for more detail on
966that part of the build process.
967
968.. note::
969
970 Support for creating feeds directly from the
971 deploy/\*
972 directories does not exist. Creating such feeds usually requires some
973 kind of feed maintenance mechanism that would upload the new packages
974 into an official package feed (e.g. the Ångström distribution). This
975 functionality is highly distribution-specific and thus is not
976 provided out of the box.
977
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500978Image Generation
979~~~~~~~~~~~~~~~~
980
981Once packages are split and stored in the Package Feeds area, the build
982system uses BitBake to generate the root filesystem image:
983
984.. image:: figures/image-generation.png
985 :align: center
986
987The image generation process consists of several stages and depends on
988several tasks and variables. The
989:ref:`ref-tasks-rootfs` task creates
990the root filesystem (file and directory structure) for an image. This
991task uses several key variables to help create the list of packages to
992actually install:
993
994- :term:`IMAGE_INSTALL`: Lists
995 out the base set of packages from which to install from the Package
996 Feeds area.
997
998- :term:`PACKAGE_EXCLUDE`:
999 Specifies packages that should not be installed into the image.
1000
1001- :term:`IMAGE_FEATURES`:
1002 Specifies features to include in the image. Most of these features
1003 map to additional packages for installation.
1004
1005- :term:`PACKAGE_CLASSES`:
1006 Specifies the package backend (e.g. RPM, DEB, or IPK) to use and
1007 consequently helps determine where to locate packages within the
1008 Package Feeds area.
1009
1010- :term:`IMAGE_LINGUAS`:
1011 Determines the language(s) for which additional language support
1012 packages are installed.
1013
1014- :term:`PACKAGE_INSTALL`:
1015 The final list of packages passed to the package manager for
1016 installation into the image.
1017
1018With :term:`IMAGE_ROOTFS`
1019pointing to the location of the filesystem under construction and the
1020``PACKAGE_INSTALL`` variable providing the final list of packages to
1021install, the root file system is created.
1022
1023Package installation is under control of the package manager (e.g.
1024dnf/rpm, opkg, or apt/dpkg) regardless of whether or not package
1025management is enabled for the target. At the end of the process, if
1026package management is not enabled for the target, the package manager's
1027data files are deleted from the root filesystem. As part of the final
1028stage of package installation, post installation scripts that are part
1029of the packages are run. Any scripts that fail to run on the build host
1030are run on the target when the target system is first booted. If you are
1031using a
Andrew Geissler09209ee2020-12-13 08:44:15 -06001032:ref:`read-only root filesystem <dev-manual/common-tasks:creating a read-only root filesystem>`,
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001033all the post installation scripts must succeed on the build host during
1034the package installation phase since the root filesystem on the target
1035is read-only.
1036
1037The final stages of the ``do_rootfs`` task handle post processing. Post
1038processing includes creation of a manifest file and optimizations.
1039
1040The manifest file (``.manifest``) resides in the same directory as the
1041root filesystem image. This file lists out, line-by-line, the installed
1042packages. The manifest file is useful for the
1043:ref:`testimage <ref-classes-testimage*>` class,
1044for example, to determine whether or not to run specific tests. See the
1045:term:`IMAGE_MANIFEST`
1046variable for additional information.
1047
1048Optimizing processes that are run across the image include ``mklibs``,
1049``prelink``, and any other post-processing commands as defined by the
1050:term:`ROOTFS_POSTPROCESS_COMMAND`
1051variable. The ``mklibs`` process optimizes the size of the libraries,
1052while the ``prelink`` process optimizes the dynamic linking of shared
1053libraries to reduce start up time of executables.
1054
1055After the root filesystem is built, processing begins on the image
1056through the :ref:`ref-tasks-image`
1057task. The build system runs any pre-processing commands as defined by
1058the
1059:term:`IMAGE_PREPROCESS_COMMAND`
1060variable. This variable specifies a list of functions to call before the
1061build system creates the final image output files.
1062
1063The build system dynamically creates ``do_image_*`` tasks as needed,
1064based on the image types specified in the
1065:term:`IMAGE_FSTYPES` variable.
1066The process turns everything into an image file or a set of image files
1067and can compress the root filesystem image to reduce the overall size of
1068the image. The formats used for the root filesystem depend on the
1069``IMAGE_FSTYPES`` variable. Compression depends on whether the formats
1070support compression.
1071
1072As an example, a dynamically created task when creating a particular
1073image type would take the following form:
1074::
1075
1076 do_image_type
1077
1078So, if the type
1079as specified by the ``IMAGE_FSTYPES`` were ``ext4``, the dynamically
1080generated task would be as follows:
1081::
1082
1083 do_image_ext4
1084
1085The final task involved in image creation is the
1086:ref:`do_image_complete <ref-tasks-image-complete>`
1087task. This task completes the image by applying any image post
1088processing as defined through the
1089:term:`IMAGE_POSTPROCESS_COMMAND`
1090variable. The variable specifies a list of functions to call once the
1091build system has created the final image output files.
1092
1093.. note::
1094
1095 The entire image generation process is run under
1096 Pseudo. Running under Pseudo ensures that the files in the root filesystem
1097 have correct ownership.
1098
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001099SDK Generation
1100~~~~~~~~~~~~~~
1101
1102The OpenEmbedded build system uses BitBake to generate the Software
1103Development Kit (SDK) installer scripts for both the standard SDK and
1104the extensible SDK (eSDK):
1105
1106.. image:: figures/sdk-generation.png
1107 :align: center
1108
1109.. note::
1110
1111 For more information on the cross-development toolchain generation,
Andrew Geissler09209ee2020-12-13 08:44:15 -06001112 see the ":ref:`overview-manual/concepts:cross-development toolchain generation`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001113 section. For information on advantages gained when building a
1114 cross-development toolchain using the do_populate_sdk task, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001115 ":ref:`sdk-manual/appendix-obtain:building an sdk installer`" section in
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001116 the Yocto Project Application Development and the Extensible Software
1117 Development Kit (eSDK) manual.
1118
1119Like image generation, the SDK script process consists of several stages
1120and depends on many variables. The
1121:ref:`ref-tasks-populate_sdk`
1122and
1123:ref:`ref-tasks-populate_sdk_ext`
1124tasks use these key variables to help create the list of packages to
1125actually install. For information on the variables listed in the figure,
1126see the "`Application Development SDK <#sdk-dev-environment>`__"
1127section.
1128
1129The ``do_populate_sdk`` task helps create the standard SDK and handles
1130two parts: a target part and a host part. The target part is the part
1131built for the target hardware and includes libraries and headers. The
1132host part is the part of the SDK that runs on the
1133:term:`SDKMACHINE`.
1134
1135The ``do_populate_sdk_ext`` task helps create the extensible SDK and
1136handles host and target parts differently than its counter part does for
1137the standard SDK. For the extensible SDK, the task encapsulates the
1138build system, which includes everything needed (host and target) for the
1139SDK.
1140
1141Regardless of the type of SDK being constructed, the tasks perform some
1142cleanup after which a cross-development environment setup script and any
1143needed configuration files are created. The final output is the
1144Cross-development toolchain installation script (``.sh`` file), which
1145includes the environment setup script.
1146
1147Stamp Files and the Rerunning of Tasks
1148~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1149
1150For each task that completes successfully, BitBake writes a stamp file
1151into the :term:`STAMPS_DIR`
1152directory. The beginning of the stamp file's filename is determined by
1153the :term:`STAMP` variable, and the end
1154of the name consists of the task's name and current `input
1155checksum <#overview-checksums>`__.
1156
1157.. note::
1158
1159 This naming scheme assumes that
1160 BB_SIGNATURE_HANDLER
1161 is "OEBasicHash", which is almost always the case in current
1162 OpenEmbedded.
1163
1164To determine if a task needs to be rerun, BitBake checks if a stamp file
1165with a matching input checksum exists for the task. If such a stamp file
1166exists, the task's output is assumed to exist and still be valid. If the
1167file does not exist, the task is rerun.
1168
1169.. note::
1170
1171 The stamp mechanism is more general than the shared state (sstate)
1172 cache mechanism described in the "`Setscene Tasks and Shared
1173 State <#setscene-tasks-and-shared-state>`__" section. BitBake avoids
1174 rerunning any task that has a valid stamp file, not just tasks that
1175 can be accelerated through the sstate cache.
1176
1177 However, you should realize that stamp files only serve as a marker
1178 that some work has been done and that these files do not record task
1179 output. The actual task output would usually be somewhere in
1180 :term:`TMPDIR` (e.g. in some
1181 recipe's :term:`WORKDIR`.) What
1182 the sstate cache mechanism adds is a way to cache task output that
1183 can then be shared between build machines.
1184
1185Since ``STAMPS_DIR`` is usually a subdirectory of ``TMPDIR``, removing
1186``TMPDIR`` will also remove ``STAMPS_DIR``, which means tasks will
1187properly be rerun to repopulate ``TMPDIR``.
1188
1189If you want some task to always be considered "out of date", you can
1190mark it with the :ref:`nostamp <bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`
1191varflag. If some other task depends on such a task, then that task will
1192also always be considered out of date, which might not be what you want.
1193
1194For details on how to view information about a task's signature, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001195":ref:`dev-manual/common-tasks:viewing task variable dependencies`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001196section in the Yocto Project Development Tasks Manual.
1197
1198Setscene Tasks and Shared State
1199~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1200
1201The description of tasks so far assumes that BitBake needs to build
1202everything and no available prebuilt objects exist. BitBake does support
1203skipping tasks if prebuilt objects are available. These objects are
1204usually made available in the form of a shared state (sstate) cache.
1205
1206.. note::
1207
1208 For information on variables affecting sstate, see the
1209 :term:`SSTATE_DIR`
1210 and
1211 :term:`SSTATE_MIRRORS`
1212 variables.
1213
1214The idea of a setscene task (i.e ``do_``\ taskname\ ``_setscene``) is a
1215version of the task where instead of building something, BitBake can
1216skip to the end result and simply place a set of files into specific
1217locations as needed. In some cases, it makes sense to have a setscene
1218task variant (e.g. generating package files in the
1219:ref:`do_package_write_* <ref-tasks-package_write_deb>`
1220task). In other cases, it does not make sense (e.g. a
1221:ref:`ref-tasks-patch` task or a
1222:ref:`ref-tasks-unpack` task) since
1223the work involved would be equal to or greater than the underlying task.
1224
1225In the build system, the common tasks that have setscene variants are
1226:ref:`ref-tasks-package`,
1227``do_package_write_*``,
1228:ref:`ref-tasks-deploy`,
1229:ref:`ref-tasks-packagedata`, and
1230:ref:`ref-tasks-populate_sysroot`.
1231Notice that these tasks represent most of the tasks whose output is an
1232end result.
1233
1234The build system has knowledge of the relationship between these tasks
1235and other preceding tasks. For example, if BitBake runs
1236``do_populate_sysroot_setscene`` for something, it does not make sense
1237to run any of the ``do_fetch``, ``do_unpack``, ``do_patch``,
1238``do_configure``, ``do_compile``, and ``do_install`` tasks. However, if
1239``do_package`` needs to be run, BitBake needs to run those other tasks.
1240
1241It becomes more complicated if everything can come from an sstate cache
1242because some objects are simply not required at all. For example, you do
1243not need a compiler or native tools, such as quilt, if nothing exists to
1244compile or patch. If the ``do_package_write_*`` packages are available
1245from sstate, BitBake does not need the ``do_package`` task data.
1246
1247To handle all these complexities, BitBake runs in two phases. The first
1248is the "setscene" stage. During this stage, BitBake first checks the
1249sstate cache for any targets it is planning to build. BitBake does a
1250fast check to see if the object exists rather than a complete download.
1251If nothing exists, the second phase, which is the setscene stage,
1252completes and the main build proceeds.
1253
1254If objects are found in the sstate cache, the build system works
1255backwards from the end targets specified by the user. For example, if an
1256image is being built, the build system first looks for the packages
1257needed for that image and the tools needed to construct an image. If
1258those are available, the compiler is not needed. Thus, the compiler is
1259not even downloaded. If something was found to be unavailable, or the
1260download or setscene task fails, the build system then tries to install
1261dependencies, such as the compiler, from the cache.
1262
1263The availability of objects in the sstate cache is handled by the
1264function specified by the
1265:term:`bitbake:BB_HASHCHECK_FUNCTION`
1266variable and returns a list of available objects. The function specified
1267by the
1268:term:`bitbake:BB_SETSCENE_DEPVALID`
1269variable is the function that determines whether a given dependency
1270needs to be followed, and whether for any given relationship the
1271function needs to be passed. The function returns a True or False value.
1272
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001273Images
1274------
1275
1276The images produced by the build system are compressed forms of the root
1277filesystem and are ready to boot on a target device. You can see from
1278the `general workflow figure <#general-workflow-figure>`__ that BitBake
1279output, in part, consists of images. This section takes a closer look at
1280this output:
1281
1282.. image:: figures/images.png
1283 :align: center
1284
1285.. note::
1286
1287 For a list of example images that the Yocto Project provides, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001288 ":doc:`/ref-manual/images`" chapter in the Yocto Project Reference
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001289 Manual.
1290
1291The build process writes images out to the :term:`Build Directory`
1292inside the
1293``tmp/deploy/images/machine/`` folder as shown in the figure. This
1294folder contains any files expected to be loaded on the target device.
1295The :term:`DEPLOY_DIR` variable
1296points to the ``deploy`` directory, while the
1297:term:`DEPLOY_DIR_IMAGE`
1298variable points to the appropriate directory containing images for the
1299current configuration.
1300
1301- kernel-image: A kernel binary file. The
1302 :term:`KERNEL_IMAGETYPE`
1303 variable determines the naming scheme for the kernel image file.
1304 Depending on this variable, the file could begin with a variety of
1305 naming strings. The ``deploy/images/``\ machine directory can contain
1306 multiple image files for the machine.
1307
1308- root-filesystem-image: Root filesystems for the target device (e.g.
1309 ``*.ext3`` or ``*.bz2`` files). The
1310 :term:`IMAGE_FSTYPES`
1311 variable determines the root filesystem image type. The
1312 ``deploy/images/``\ machine directory can contain multiple root
1313 filesystems for the machine.
1314
1315- kernel-modules: Tarballs that contain all the modules built for the
1316 kernel. Kernel module tarballs exist for legacy purposes and can be
1317 suppressed by setting the
1318 :term:`MODULE_TARBALL_DEPLOY`
1319 variable to "0". The ``deploy/images/``\ machine directory can
1320 contain multiple kernel module tarballs for the machine.
1321
1322- bootloaders: If applicable to the target machine, bootloaders
1323 supporting the image. The ``deploy/images/``\ machine directory can
1324 contain multiple bootloaders for the machine.
1325
1326- symlinks: The ``deploy/images/``\ machine folder contains a symbolic
1327 link that points to the most recently built file for each machine.
1328 These links might be useful for external scripts that need to obtain
1329 the latest version of each file.
1330
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001331Application Development SDK
1332---------------------------
1333
1334In the `general workflow figure <#general-workflow-figure>`__, the
1335output labeled "Application Development SDK" represents an SDK. The SDK
1336generation process differs depending on whether you build an extensible
1337SDK (e.g. ``bitbake -c populate_sdk_ext`` imagename) or a standard SDK
1338(e.g. ``bitbake -c populate_sdk`` imagename). This section takes a
1339closer look at this output:
1340
1341.. image:: figures/sdk.png
1342 :align: center
1343
1344The specific form of this output is a set of files that includes a
1345self-extracting SDK installer (``*.sh``), host and target manifest
1346files, and files used for SDK testing. When the SDK installer file is
1347run, it installs the SDK. The SDK consists of a cross-development
1348toolchain, a set of libraries and headers, and an SDK environment setup
1349script. Running this installer essentially sets up your
1350cross-development environment. You can think of the cross-toolchain as
1351the "host" part because it runs on the SDK machine. You can think of the
1352libraries and headers as the "target" part because they are built for
1353the target hardware. The environment setup script is added so that you
1354can initialize the environment before using the tools.
1355
1356.. note::
1357
1358 - The Yocto Project supports several methods by which you can set up
1359 this cross-development environment. These methods include
1360 downloading pre-built SDK installers or building and installing
1361 your own SDK installer.
1362
1363 - For background information on cross-development toolchains in the
1364 Yocto Project development environment, see the "`Cross-Development
1365 Toolchain Generation <#cross-development-toolchain-generation>`__"
1366 section.
1367
1368 - For information on setting up a cross-development environment, see
Andrew Geissler09209ee2020-12-13 08:44:15 -06001369 the :doc:`/sdk-manual/index` manual.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001370
1371All the output files for an SDK are written to the ``deploy/sdk`` folder
1372inside the :term:`Build Directory` as
1373shown in the previous figure. Depending on the type of SDK, several
1374variables exist that help configure these files. The following list
1375shows the variables associated with an extensible SDK:
1376
1377- :term:`DEPLOY_DIR`: Points to
1378 the ``deploy`` directory.
1379
1380- :term:`SDK_EXT_TYPE`:
1381 Controls whether or not shared state artifacts are copied into the
1382 extensible SDK. By default, all required shared state artifacts are
1383 copied into the SDK.
1384
1385- :term:`SDK_INCLUDE_PKGDATA`:
1386 Specifies whether or not packagedata is included in the extensible
1387 SDK for all recipes in the "world" target.
1388
1389- :term:`SDK_INCLUDE_TOOLCHAIN`:
1390 Specifies whether or not the toolchain is included when building the
1391 extensible SDK.
1392
1393- :term:`SDK_LOCAL_CONF_WHITELIST`:
1394 A list of variables allowed through from the build system
1395 configuration into the extensible SDK configuration.
1396
1397- :term:`SDK_LOCAL_CONF_BLACKLIST`:
1398 A list of variables not allowed through from the build system
1399 configuration into the extensible SDK configuration.
1400
1401- :term:`SDK_INHERIT_BLACKLIST`:
1402 A list of classes to remove from the
1403 :term:`INHERIT` value globally
1404 within the extensible SDK configuration.
1405
1406This next list, shows the variables associated with a standard SDK:
1407
1408- :term:`DEPLOY_DIR`: Points to
1409 the ``deploy`` directory.
1410
1411- :term:`SDKMACHINE`: Specifies
1412 the architecture of the machine on which the cross-development tools
1413 are run to create packages for the target hardware.
1414
1415- :term:`SDKIMAGE_FEATURES`:
1416 Lists the features to include in the "target" part of the SDK.
1417
1418- :term:`TOOLCHAIN_HOST_TASK`:
1419 Lists packages that make up the host part of the SDK (i.e. the part
1420 that runs on the ``SDKMACHINE``). When you use
1421 ``bitbake -c populate_sdk imagename`` to create the SDK, a set of
1422 default packages apply. This variable allows you to add more
1423 packages.
1424
1425- :term:`TOOLCHAIN_TARGET_TASK`:
1426 Lists packages that make up the target part of the SDK (i.e. the part
1427 built for the target hardware).
1428
1429- :term:`SDKPATH`: Defines the
1430 default SDK installation path offered by the installation script.
1431
1432- :term:`SDK_HOST_MANIFEST`:
1433 Lists all the installed packages that make up the host part of the
1434 SDK. This variable also plays a minor role for extensible SDK
1435 development as well. However, it is mainly used for the standard SDK.
1436
1437- :term:`SDK_TARGET_MANIFEST`:
1438 Lists all the installed packages that make up the target part of the
1439 SDK. This variable also plays a minor role for extensible SDK
1440 development as well. However, it is mainly used for the standard SDK.
1441
1442Cross-Development Toolchain Generation
1443======================================
1444
1445The Yocto Project does most of the work for you when it comes to
Andrew Geissler09209ee2020-12-13 08:44:15 -06001446creating :ref:`sdk-manual/intro:the cross-development toolchain`. This
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001447section provides some technical background on how cross-development
1448toolchains are created and used. For more information on toolchains, you
Andrew Geissler09209ee2020-12-13 08:44:15 -06001449can also see the :doc:`/sdk-manual/index` manual.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001450
1451In the Yocto Project development environment, cross-development
1452toolchains are used to build images and applications that run on the
1453target hardware. With just a few commands, the OpenEmbedded build system
1454creates these necessary toolchains for you.
1455
1456The following figure shows a high-level build environment regarding
1457toolchain construction and use.
1458
1459.. image:: figures/cross-development-toolchains.png
1460 :align: center
1461
1462Most of the work occurs on the Build Host. This is the machine used to
1463build images and generally work within the the Yocto Project
1464environment. When you run
1465:term:`BitBake` to create an image, the
1466OpenEmbedded build system uses the host ``gcc`` compiler to bootstrap a
1467cross-compiler named ``gcc-cross``. The ``gcc-cross`` compiler is what
1468BitBake uses to compile source files when creating the target image. You
1469can think of ``gcc-cross`` simply as an automatically generated
1470cross-compiler that is used internally within BitBake only.
1471
1472.. note::
1473
1474 The extensible SDK does not use
1475 gcc-cross-canadian
1476 since this SDK ships a copy of the OpenEmbedded build system and the
1477 sysroot within it contains
1478 gcc-cross
1479 .
1480
Andrew Geissler6ce62a22020-11-30 19:58:47 -06001481The chain of events that occurs when the standard toolchain is bootstrapped:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001482::
1483
Andrew Geissler6ce62a22020-11-30 19:58:47 -06001484 binutils-cross -> linux-libc-headers -> gcc-cross -> libgcc-initial -> glibc -> libgcc -> gcc-runtime
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001485
Andrew Geissler6ce62a22020-11-30 19:58:47 -06001486- ``gcc``: The compiler, GNU Compiler Collection (GCC).
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001487
Andrew Geissler6ce62a22020-11-30 19:58:47 -06001488- ``binutils-cross``: The binary utilities needed in order
1489 to run the ``gcc-cross`` phase of the bootstrap operation and build the
1490 headers for the C library.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001491
Andrew Geissler6ce62a22020-11-30 19:58:47 -06001492- ``linux-libc-headers``: Headers needed for the cross-compiler and C library build.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001493
Andrew Geissler6ce62a22020-11-30 19:58:47 -06001494- ``libgcc-initial``: An initial version of the gcc support library needed
1495 to bootstrap ``glibc``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001496
Andrew Geissler6ce62a22020-11-30 19:58:47 -06001497- ``libgcc``: The final version of the gcc support library which
1498 can only be built once there is a C library to link against.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001499
1500- ``glibc``: The GNU C Library.
1501
1502- ``gcc-cross``: The final stage of the bootstrap process for the
1503 cross-compiler. This stage results in the actual cross-compiler that
1504 BitBake uses when it builds an image for a targeted device.
1505
Andrew Geissler6ce62a22020-11-30 19:58:47 -06001506 This tool is a "native" tool (i.e. it is designed to run on
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001507 the build host).
1508
1509- ``gcc-runtime``: Runtime libraries resulting from the toolchain
1510 bootstrapping process. This tool produces a binary that consists of
1511 the runtime libraries need for the targeted device.
1512
1513You can use the OpenEmbedded build system to build an installer for the
1514relocatable SDK used to develop applications. When you run the
1515installer, it installs the toolchain, which contains the development
1516tools (e.g., ``gcc-cross-canadian``, ``binutils-cross-canadian``, and
1517other ``nativesdk-*`` tools), which are tools native to the SDK (i.e.
1518native to :term:`SDK_ARCH`), you
1519need to cross-compile and test your software. The figure shows the
1520commands you use to easily build out this toolchain. This
1521cross-development toolchain is built to execute on the
1522:term:`SDKMACHINE`, which might or
1523might not be the same machine as the Build Host.
1524
1525.. note::
1526
1527 If your target architecture is supported by the Yocto Project, you
1528 can take advantage of pre-built images that ship with the Yocto
1529 Project and already contain cross-development toolchain installers.
1530
1531Here is the bootstrap process for the relocatable toolchain:
1532::
1533
1534 gcc -> binutils-crosssdk -> gcc-crosssdk-initial -> linux-libc-headers -> glibc-initial -> nativesdk-glibc -> gcc-crosssdk -> gcc-cross-canadian
1535
1536- ``gcc``: The build host's GNU Compiler Collection (GCC).
1537
1538- ``binutils-crosssdk``: The bare minimum binary utilities needed in
1539 order to run the ``gcc-crosssdk-initial`` phase of the bootstrap
1540 operation.
1541
1542- ``gcc-crosssdk-initial``: An early stage of the bootstrap process for
1543 creating the cross-compiler. This stage builds enough of the
1544 ``gcc-crosssdk`` and supporting pieces so that the final stage of the
1545 bootstrap process can produce the finished cross-compiler. This tool
1546 is a "native" binary that runs on the build host.
1547
1548- ``linux-libc-headers``: Headers needed for the cross-compiler.
1549
1550- ``glibc-initial``: An initial version of the Embedded GLIBC needed to
1551 bootstrap ``nativesdk-glibc``.
1552
1553- ``nativesdk-glibc``: The Embedded GLIBC needed to bootstrap the
1554 ``gcc-crosssdk``.
1555
1556- ``gcc-crosssdk``: The final stage of the bootstrap process for the
1557 relocatable cross-compiler. The ``gcc-crosssdk`` is a transitory
1558 compiler and never leaves the build host. Its purpose is to help in
1559 the bootstrap process to create the eventual ``gcc-cross-canadian``
1560 compiler, which is relocatable. This tool is also a "native" package
1561 (i.e. it is designed to run on the build host).
1562
1563- ``gcc-cross-canadian``: The final relocatable cross-compiler. When
1564 run on the :term:`SDKMACHINE`,
1565 this tool produces executable code that runs on the target device.
1566 Only one cross-canadian compiler is produced per architecture since
1567 they can be targeted at different processor optimizations using
1568 configurations passed to the compiler through the compile commands.
1569 This circumvents the need for multiple compilers and thus reduces the
1570 size of the toolchains.
1571
1572.. note::
1573
1574 For information on advantages gained when building a
1575 cross-development toolchain installer, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001576 ":ref:`sdk-manual/appendix-obtain:building an sdk installer`" appendix
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001577 in the Yocto Project Application Development and the
1578 Extensible Software Development Kit (eSDK) manual.
1579
1580Shared State Cache
1581==================
1582
1583By design, the OpenEmbedded build system builds everything from scratch
1584unless :term:`BitBake` can determine
1585that parts do not need to be rebuilt. Fundamentally, building from
1586scratch is attractive as it means all parts are built fresh and no
1587possibility of stale data exists that can cause problems. When
1588developers hit problems, they typically default back to building from
1589scratch so they have a know state from the start.
1590
1591Building an image from scratch is both an advantage and a disadvantage
1592to the process. As mentioned in the previous paragraph, building from
1593scratch ensures that everything is current and starts from a known
1594state. However, building from scratch also takes much longer as it
1595generally means rebuilding things that do not necessarily need to be
1596rebuilt.
1597
1598The Yocto Project implements shared state code that supports incremental
1599builds. The implementation of the shared state code answers the
1600following questions that were fundamental roadblocks within the
1601OpenEmbedded incremental build support system:
1602
1603- What pieces of the system have changed and what pieces have not
1604 changed?
1605
1606- How are changed pieces of software removed and replaced?
1607
1608- How are pre-built components that do not need to be rebuilt from
1609 scratch used when they are available?
1610
1611For the first question, the build system detects changes in the "inputs"
1612to a given task by creating a checksum (or signature) of the task's
1613inputs. If the checksum changes, the system assumes the inputs have
1614changed and the task needs to be rerun. For the second question, the
1615shared state (sstate) code tracks which tasks add which output to the
1616build process. This means the output from a given task can be removed,
1617upgraded or otherwise manipulated. The third question is partly
1618addressed by the solution for the second question assuming the build
1619system can fetch the sstate objects from remote locations and install
1620them if they are deemed to be valid.
1621
1622.. note::
1623
1624 - The build system does not maintain
1625 :term:`PR` information as part of
1626 the shared state packages. Consequently, considerations exist that
1627 affect maintaining shared state feeds. For information on how the
1628 build system works with packages and can track incrementing ``PR``
Andrew Geissler09209ee2020-12-13 08:44:15 -06001629 information, see the ":ref:`dev-manual/common-tasks:automatically incrementing a package version number`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001630 section in the Yocto Project Development Tasks Manual.
1631
1632 - The code in the build system that supports incremental builds is
1633 not simple code. For techniques that help you work around issues
1634 related to shared state code, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001635 ":ref:`dev-manual/common-tasks:viewing metadata used to create the input signature of a shared state task`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001636 and
Andrew Geissler09209ee2020-12-13 08:44:15 -06001637 ":ref:`dev-manual/common-tasks:invalidating shared state to force a task to run`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001638 sections both in the Yocto Project Development Tasks Manual.
1639
1640The rest of this section goes into detail about the overall incremental
1641build architecture, the checksums (signatures), and shared state.
1642
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001643Overall Architecture
1644--------------------
1645
1646When determining what parts of the system need to be built, BitBake
1647works on a per-task basis rather than a per-recipe basis. You might
1648wonder why using a per-task basis is preferred over a per-recipe basis.
1649To help explain, consider having the IPK packaging backend enabled and
1650then switching to DEB. In this case, the
1651:ref:`ref-tasks-install` and
1652:ref:`ref-tasks-package` task outputs
1653are still valid. However, with a per-recipe approach, the build would
1654not include the ``.deb`` files. Consequently, you would have to
1655invalidate the whole build and rerun it. Rerunning everything is not the
1656best solution. Also, in this case, the core must be "taught" much about
1657specific tasks. This methodology does not scale well and does not allow
1658users to easily add new tasks in layers or as external recipes without
1659touching the packaged-staging core.
1660
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001661Checksums (Signatures)
1662----------------------
1663
1664The shared state code uses a checksum, which is a unique signature of a
1665task's inputs, to determine if a task needs to be run again. Because it
1666is a change in a task's inputs that triggers a rerun, the process needs
1667to detect all the inputs to a given task. For shell tasks, this turns
1668out to be fairly easy because the build process generates a "run" shell
1669script for each task and it is possible to create a checksum that gives
1670you a good idea of when the task's data changes.
1671
1672To complicate the problem, there are things that should not be included
1673in the checksum. First, there is the actual specific build path of a
1674given task - the :term:`WORKDIR`. It
1675does not matter if the work directory changes because it should not
1676affect the output for target packages. Also, the build process has the
1677objective of making native or cross packages relocatable.
1678
1679.. note::
1680
1681 Both native and cross packages run on the
1682 build host. However, cross packages generate output for the target
1683 architecture.
1684
1685The checksum therefore needs to exclude ``WORKDIR``. The simplistic
1686approach for excluding the work directory is to set ``WORKDIR`` to some
1687fixed value and create the checksum for the "run" script.
1688
1689Another problem results from the "run" scripts containing functions that
1690might or might not get called. The incremental build solution contains
1691code that figures out dependencies between shell functions. This code is
1692used to prune the "run" scripts down to the minimum set, thereby
1693alleviating this problem and making the "run" scripts much more readable
1694as a bonus.
1695
1696So far, solutions for shell scripts exist. What about Python tasks? The
1697same approach applies even though these tasks are more difficult. The
1698process needs to figure out what variables a Python function accesses
1699and what functions it calls. Again, the incremental build solution
1700contains code that first figures out the variable and function
1701dependencies, and then creates a checksum for the data used as the input
1702to the task.
1703
1704Like the ``WORKDIR`` case, situations exist where dependencies should be
1705ignored. For these situations, you can instruct the build process to
1706ignore a dependency by using a line like the following:
1707::
1708
1709 PACKAGE_ARCHS[vardepsexclude] = "MACHINE"
1710
1711This example ensures that the :term:`PACKAGE_ARCHS` variable
1712does not depend on the value of :term:`MACHINE`, even if it does
1713reference it.
1714
1715Equally, there are cases where you need to add dependencies BitBake is
1716not able to find. You can accomplish this by using a line like the
1717following:
1718::
1719
1720 PACKAGE_ARCHS[vardeps] = "MACHINE"
1721
1722This example explicitly
1723adds the ``MACHINE`` variable as a dependency for ``PACKAGE_ARCHS``.
1724
1725As an example, consider a case with in-line Python where BitBake is not
1726able to figure out dependencies. When running in debug mode (i.e. using
1727``-DDD``), BitBake produces output when it discovers something for which
1728it cannot figure out dependencies. The Yocto Project team has currently
1729not managed to cover those dependencies in detail and is aware of the
1730need to fix this situation.
1731
1732Thus far, this section has limited discussion to the direct inputs into
1733a task. Information based on direct inputs is referred to as the
1734"basehash" in the code. However, the question of a task's indirect
1735inputs still exits - items already built and present in the
1736:term:`Build Directory`. The checksum (or
1737signature) for a particular task needs to add the hashes of all the
1738tasks on which the particular task depends. Choosing which dependencies
1739to add is a policy decision. However, the effect is to generate a master
1740checksum that combines the basehash and the hashes of the task's
1741dependencies.
1742
1743At the code level, a variety of ways exist by which both the basehash
1744and the dependent task hashes can be influenced. Within the BitBake
1745configuration file, you can give BitBake some extra information to help
1746it construct the basehash. The following statement effectively results
1747in a list of global variable dependency excludes (i.e. variables never
1748included in any checksum):
1749::
1750
1751 BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \\
1752 SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \\
1753 USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \\
1754 PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \\
1755 CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX"
1756
1757The
1758previous example excludes
1759:term:`WORKDIR` since that variable
1760is actually constructed as a path within
1761:term:`TMPDIR`, which is on the
1762whitelist.
1763
1764The rules for deciding which hashes of dependent tasks to include
1765through dependency chains are more complex and are generally
1766accomplished with a Python function. The code in
1767``meta/lib/oe/sstatesig.py`` shows two examples of this and also
1768illustrates how you can insert your own policy into the system if so
1769desired. This file defines the two basic signature generators
1770:term:`OpenEmbedded-Core (OE-Core)` uses: "OEBasic" and
1771"OEBasicHash". By default, a dummy "noop" signature handler is enabled
1772in BitBake. This means that behavior is unchanged from previous
1773versions. OE-Core uses the "OEBasicHash" signature handler by default
1774through this setting in the ``bitbake.conf`` file:
1775::
1776
1777 BB_SIGNATURE_HANDLER ?= "OEBasicHash"
1778
1779The "OEBasicHash" ``BB_SIGNATURE_HANDLER`` is the same
1780as the "OEBasic" version but adds the task hash to the `stamp
1781files <#stamp-files-and-the-rerunning-of-tasks>`__. This results in any
1782metadata change that changes the task hash, automatically causing the
1783task to be run again. This removes the need to bump
1784:term:`PR` values, and changes to metadata
1785automatically ripple across the build.
1786
1787It is also worth noting that the end result of these signature
1788generators is to make some dependency and hash information available to
1789the build. This information includes:
1790
1791- ``BB_BASEHASH_task-``\ taskname: The base hashes for each task in the
1792 recipe.
1793
1794- ``BB_BASEHASH_``\ filename\ ``:``\ taskname: The base hashes for each
1795 dependent task.
1796
1797- ``BBHASHDEPS_``\ filename\ ``:``\ taskname: The task dependencies for
1798 each task.
1799
1800- ``BB_TASKHASH``: The hash of the currently running task.
1801
1802Shared State
1803------------
1804
1805Checksums and dependencies, as discussed in the previous section, solve
1806half the problem of supporting a shared state. The other half of the
1807problem is being able to use checksum information during the build and
1808being able to reuse or rebuild specific components.
1809
1810The :ref:`sstate <ref-classes-sstate>` class is a
1811relatively generic implementation of how to "capture" a snapshot of a
1812given task. The idea is that the build process does not care about the
1813source of a task's output. Output could be freshly built or it could be
1814downloaded and unpacked from somewhere. In other words, the build
1815process does not need to worry about its origin.
1816
1817Two types of output exist. One type is just about creating a directory
1818in :term:`WORKDIR`. A good example is
1819the output of either
1820:ref:`ref-tasks-install` or
1821:ref:`ref-tasks-package`. The other
1822type of output occurs when a set of data is merged into a shared
1823directory tree such as the sysroot.
1824
1825The Yocto Project team has tried to keep the details of the
1826implementation hidden in ``sstate`` class. From a user's perspective,
1827adding shared state wrapping to a task is as simple as this
1828:ref:`ref-tasks-deploy` example taken
1829from the :ref:`deploy <ref-classes-deploy>` class:
1830::
1831
1832 DEPLOYDIR = "${WORKDIR}/deploy-${PN}"
1833 SSTATETASKS += "do_deploy"
1834 do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"
1835 do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
1836
1837 python do_deploy_setscene () {
1838 sstate_setscene(d)
1839 }
1840 addtask do_deploy_setscene
1841 do_deploy[dirs] = "${DEPLOYDIR} ${B}"
1842 do_deploy[stamp-extra-info] = "${MACHINE_ARCH}"
1843
1844The following list explains the previous example:
1845
1846- Adding "do_deploy" to ``SSTATETASKS`` adds some required
1847 sstate-related processing, which is implemented in the
1848 :ref:`sstate <ref-classes-sstate>` class, to
1849 before and after the
1850 :ref:`ref-tasks-deploy` task.
1851
1852- The ``do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"`` declares that
1853 ``do_deploy`` places its output in ``${DEPLOYDIR}`` when run normally
1854 (i.e. when not using the sstate cache). This output becomes the input
1855 to the shared state cache.
1856
1857- The ``do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"`` line
1858 causes the contents of the shared state cache to be copied to
1859 ``${DEPLOY_DIR_IMAGE}``.
1860
1861 .. note::
1862
1863 If ``do_deploy`` is not already in the shared state cache or if its input
1864 checksum (signature) has changed from when the output was cached, the task
1865 runs to populate the shared state cache, after which the contents of the
1866 shared state cache is copied to ${:term:`DEPLOY_DIR_IMAGE`}. If
1867 ``do_deploy`` is in the shared state cache and its signature indicates
1868 that the cached output is still valid (i.e. if no relevant task inputs
1869 have changed), then the contents of the shared state cache copies
1870 directly to ${``DEPLOY_DIR_IMAGE``} by the ``do_deploy_setscene`` task
1871 instead, skipping the ``do_deploy`` task.
1872
1873- The following task definition is glue logic needed to make the
1874 previous settings effective:
1875 ::
1876
1877 python do_deploy_setscene () {
1878 sstate_setscene(d)
1879 }
1880 addtask do_deploy_setscene
1881
1882 ``sstate_setscene()`` takes the flags above as input and accelerates the ``do_deploy`` task
1883 through the shared state cache if possible. If the task was
1884 accelerated, ``sstate_setscene()`` returns True. Otherwise, it
1885 returns False, and the normal ``do_deploy`` task runs. For more
1886 information, see the ":ref:`setscene <bitbake:bitbake-user-manual/bitbake-user-manual-execution:setscene>`"
1887 section in the BitBake User Manual.
1888
1889- The ``do_deploy[dirs] = "${DEPLOYDIR} ${B}"`` line creates
1890 ``${DEPLOYDIR}`` and ``${B}`` before the ``do_deploy`` task runs, and
1891 also sets the current working directory of ``do_deploy`` to ``${B}``.
1892 For more information, see the ":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags`"
1893 section in the BitBake
1894 User Manual.
1895
1896 .. note::
1897
1898 In cases where ``sstate-inputdirs`` and ``sstate-outputdirs`` would be
1899 the same, you can use ``sstate-plaindirs``. For example, to preserve the
1900 ${:term:`PKGD`} and ${:term:`PKGDEST`} output from the ``do_package``
1901 task, use the following:
1902 ::
1903
1904 do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}"
1905
1906
1907- The ``do_deploy[stamp-extra-info] = "${MACHINE_ARCH}"`` line appends
1908 extra metadata to the `stamp
1909 file <#stamp-files-and-the-rerunning-of-tasks>`__. In this case, the
1910 metadata makes the task specific to a machine's architecture. See
Andrew Geissler09209ee2020-12-13 08:44:15 -06001911 ":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-execution:the task list`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001912 section in the BitBake User Manual for more information on the
1913 ``stamp-extra-info`` flag.
1914
1915- ``sstate-inputdirs`` and ``sstate-outputdirs`` can also be used with
1916 multiple directories. For example, the following declares
1917 ``PKGDESTWORK`` and ``SHLIBWORK`` as shared state input directories,
1918 which populates the shared state cache, and ``PKGDATA_DIR`` and
1919 ``SHLIBSDIR`` as the corresponding shared state output directories:
1920 ::
1921
1922 do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}"
1923 do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}"
1924
1925- These methods also include the ability to take a lockfile when
1926 manipulating shared state directory structures, for cases where file
1927 additions or removals are sensitive:
1928 ::
1929
1930 do_package[sstate-lockfile] = "${PACKAGELOCK}"
1931
1932Behind the scenes, the shared state code works by looking in
1933:term:`SSTATE_DIR` and
1934:term:`SSTATE_MIRRORS` for
1935shared state files. Here is an example:
1936::
1937
1938 SSTATE_MIRRORS ?= "\
1939 file://.\* http://someserver.tld/share/sstate/PATH;downloadfilename=PATH \n \
1940 file://.\* file:///some/local/dir/sstate/PATH"
1941
1942.. note::
1943
1944 The shared state directory (``SSTATE_DIR``) is organized into two-character
1945 subdirectories, where the subdirectory names are based on the first two
1946 characters of the hash.
1947 If the shared state directory structure for a mirror has the same structure
1948 as ``SSTATE_DIR``, you must specify "PATH" as part of the URI to enable the build
1949 system to map to the appropriate subdirectory.
1950
1951The shared state package validity can be detected just by looking at the
1952filename since the filename contains the task checksum (or signature) as
1953described earlier in this section. If a valid shared state package is
1954found, the build process downloads it and uses it to accelerate the
1955task.
1956
1957The build processes use the ``*_setscene`` tasks for the task
1958acceleration phase. BitBake goes through this phase before the main
1959execution code and tries to accelerate any tasks for which it can find
1960shared state packages. If a shared state package for a task is
1961available, the shared state package is used. This means the task and any
1962tasks on which it is dependent are not executed.
1963
1964As a real world example, the aim is when building an IPK-based image,
1965only the
1966:ref:`ref-tasks-package_write_ipk`
1967tasks would have their shared state packages fetched and extracted.
1968Since the sysroot is not used, it would never get extracted. This is
1969another reason why a task-based approach is preferred over a
1970recipe-based approach, which would have to install the output from every
1971task.
1972
1973Automatically Added Runtime Dependencies
1974========================================
1975
1976The OpenEmbedded build system automatically adds common types of runtime
1977dependencies between packages, which means that you do not need to
1978explicitly declare the packages using
1979:term:`RDEPENDS`. Three automatic
1980mechanisms exist (``shlibdeps``, ``pcdeps``, and ``depchains``) that
1981handle shared libraries, package configuration (pkg-config) modules, and
1982``-dev`` and ``-dbg`` packages, respectively. For other types of runtime
1983dependencies, you must manually declare the dependencies.
1984
1985- ``shlibdeps``: During the
1986 :ref:`ref-tasks-package` task of
1987 each recipe, all shared libraries installed by the recipe are
1988 located. For each shared library, the package that contains the
1989 shared library is registered as providing the shared library. More
1990 specifically, the package is registered as providing the
1991 `soname <https://en.wikipedia.org/wiki/Soname>`__ of the library. The
1992 resulting shared-library-to-package mapping is saved globally in
1993 :term:`PKGDATA_DIR` by the
1994 :ref:`ref-tasks-packagedata`
1995 task.
1996
1997 Simultaneously, all executables and shared libraries installed by the
1998 recipe are inspected to see what shared libraries they link against.
1999 For each shared library dependency that is found, ``PKGDATA_DIR`` is
2000 queried to see if some package (likely from a different recipe)
2001 contains the shared library. If such a package is found, a runtime
2002 dependency is added from the package that depends on the shared
2003 library to the package that contains the library.
2004
2005 The automatically added runtime dependency also includes a version
2006 restriction. This version restriction specifies that at least the
2007 current version of the package that provides the shared library must
2008 be used, as if "package (>= version)" had been added to ``RDEPENDS``.
2009 This forces an upgrade of the package containing the shared library
2010 when installing the package that depends on the library, if needed.
2011
2012 If you want to avoid a package being registered as providing a
2013 particular shared library (e.g. because the library is for internal
2014 use only), then add the library to
2015 :term:`PRIVATE_LIBS` inside
2016 the package's recipe.
2017
2018- ``pcdeps``: During the ``do_package`` task of each recipe, all
2019 pkg-config modules (``*.pc`` files) installed by the recipe are
2020 located. For each module, the package that contains the module is
2021 registered as providing the module. The resulting module-to-package
2022 mapping is saved globally in ``PKGDATA_DIR`` by the
2023 ``do_packagedata`` task.
2024
2025 Simultaneously, all pkg-config modules installed by the recipe are
2026 inspected to see what other pkg-config modules they depend on. A
2027 module is seen as depending on another module if it contains a
2028 "Requires:" line that specifies the other module. For each module
2029 dependency, ``PKGDATA_DIR`` is queried to see if some package
2030 contains the module. If such a package is found, a runtime dependency
2031 is added from the package that depends on the module to the package
2032 that contains the module.
2033
2034 .. note::
2035
2036 The
2037 pcdeps
2038 mechanism most often infers dependencies between
2039 -dev
2040 packages.
2041
2042- ``depchains``: If a package ``foo`` depends on a package ``bar``,
2043 then ``foo-dev`` and ``foo-dbg`` are also made to depend on
2044 ``bar-dev`` and ``bar-dbg``, respectively. Taking the ``-dev``
2045 packages as an example, the ``bar-dev`` package might provide headers
2046 and shared library symlinks needed by ``foo-dev``, which shows the
2047 need for a dependency between the packages.
2048
2049 The dependencies added by ``depchains`` are in the form of
2050 :term:`RRECOMMENDS`.
2051
2052 .. note::
2053
2054 By default, ``foo-dev`` also has an ``RDEPENDS``-style dependency on
2055 ``foo``, because the default value of ``RDEPENDS_${PN}-dev`` (set in
2056 bitbake.conf) includes "${PN}".
2057
2058 To ensure that the dependency chain is never broken, ``-dev`` and
2059 ``-dbg`` packages are always generated by default, even if the
2060 packages turn out to be empty. See the
2061 :term:`ALLOW_EMPTY` variable
2062 for more information.
2063
2064The ``do_package`` task depends on the ``do_packagedata`` task of each
2065recipe in :term:`DEPENDS` through use
2066of a ``[``\ :ref:`deptask <bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ``]``
2067declaration, which guarantees that the required
2068shared-library/module-to-package mapping information will be available
2069when needed as long as ``DEPENDS`` has been correctly set.
2070
2071Fakeroot and Pseudo
2072===================
2073
2074Some tasks are easier to implement when allowed to perform certain
2075operations that are normally reserved for the root user (e.g.
2076:ref:`ref-tasks-install`,
2077:ref:`do_package_write* <ref-tasks-package_write_deb>`,
2078:ref:`ref-tasks-rootfs`, and
2079:ref:`do_image* <ref-tasks-image>`). For example,
2080the ``do_install`` task benefits from being able to set the UID and GID
2081of installed files to arbitrary values.
2082
2083One approach to allowing tasks to perform root-only operations would be
2084to require :term:`BitBake` to run as
2085root. However, this method is cumbersome and has security issues. The
2086approach that is actually used is to run tasks that benefit from root
2087privileges in a "fake" root environment. Within this environment, the
2088task and its child processes believe that they are running as the root
2089user, and see an internally consistent view of the filesystem. As long
2090as generating the final output (e.g. a package or an image) does not
2091require root privileges, the fact that some earlier steps ran in a fake
2092root environment does not cause problems.
2093
2094The capability to run tasks in a fake root environment is known as
2095"`fakeroot <http://man.he.net/man1/fakeroot>`__", which is derived from
2096the BitBake keyword/variable flag that requests a fake root environment
2097for a task.
2098
2099In the :term:`OpenEmbedded Build System`,
2100the program that
2101implements fakeroot is known as
2102`Pseudo <https://www.yoctoproject.org/software-item/pseudo/>`__. Pseudo
2103overrides system calls by using the environment variable ``LD_PRELOAD``,
2104which results in the illusion of running as root. To keep track of
2105"fake" file ownership and permissions resulting from operations that
2106require root permissions, Pseudo uses an SQLite 3 database. This
2107database is stored in
2108``${``\ :term:`WORKDIR`\ ``}/pseudo/files.db``
2109for individual recipes. Storing the database in a file as opposed to in
2110memory gives persistence between tasks and builds, which is not
2111accomplished using fakeroot.
2112
2113.. note::
2114
2115 If you add your own task that manipulates the same files or
2116 directories as a fakeroot task, then that task also needs to run
2117 under fakeroot. Otherwise, the task cannot run root-only operations,
2118 and cannot see the fake file ownership and permissions set by the
2119 other task. You need to also add a dependency on
2120 virtual/fakeroot-native:do_populate_sysroot
2121 , giving the following:
2122 ::
2123
2124 fakeroot do_mytask () {
2125 ...
2126 }
2127 do_mytask[depends] += "virtual/fakeroot-native:do_populate_sysroot"
2128
2129
2130For more information, see the
2131:term:`FAKEROOT* <bitbake:FAKEROOT>` variables in the
2132BitBake User Manual. You can also reference the "`Why Not
2133Fakeroot? <https://github.com/wrpseudo/pseudo/wiki/WhyNotFakeroot>`__"
2134article for background information on Fakeroot and Pseudo.