blob: 736fd9591e14b4e10bbc9d007aa1efaeb07abfe5 [file] [log] [blame]
Andrew Geisslerf0343792020-11-18 10:42:21 -06001.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002
3**********************
4Yocto Project Concepts
5**********************
6
7This chapter provides explanations for Yocto Project concepts that go
8beyond the surface of "how-to" information and reference (or look-up)
9material. Concepts such as components, the :term:`OpenEmbedded Build System`
10workflow,
11cross-development toolchains, shared state cache, and so forth are
12explained.
13
14Yocto Project Components
15========================
16
17The :term:`BitBake` task executor
18together with various types of configuration files form the
19:term:`OpenEmbedded-Core (OE-Core)`. This section
20overviews these components by describing their use and how they
21interact.
22
23BitBake handles the parsing and execution of the data files. The data
24itself is of various types:
25
26- *Recipes:* Provides details about particular pieces of software.
27
28- *Class Data:* Abstracts common build information (e.g. how to build a
29 Linux kernel).
30
31- *Configuration Data:* Defines machine-specific settings, policy
32 decisions, and so forth. Configuration data acts as the glue to bind
33 everything together.
34
35BitBake knows how to combine multiple data sources together and refers
36to each data source as a layer. For information on layers, see the
37":ref:`dev-manual/dev-manual-common-tasks:understanding and creating layers`"
38section of the Yocto Project Development Tasks Manual.
39
40Following are some brief details on these core components. For
41additional information on how these components interact during a build,
42see the
43":ref:`overview-manual/overview-manual-concepts:openembedded build system concepts`"
44section.
45
46.. _usingpoky-components-bitbake:
47
48BitBake
49-------
50
51BitBake is the tool at the heart of the :term:`OpenEmbedded Build System`
52and is responsible
53for parsing the :term:`Metadata`, generating
54a list of tasks from it, and then executing those tasks.
55
56This section briefly introduces BitBake. If you want more information on
57BitBake, see the :doc:`BitBake User Manual <bitbake:index>`.
58
59To see a list of the options BitBake supports, use either of the
60following commands:
61::
62
63 $ bitbake -h
64 $ bitbake --help
65
66The most common usage for BitBake is ``bitbake recipename``, where
67``recipename`` is the name of the recipe you want to build (referred
68to as the "target"). The target often equates to the first part of a
69recipe's filename (e.g. "foo" for a recipe named ``foo_1.3.0-r0.bb``).
70So, to process the ``matchbox-desktop_1.2.3.bb`` recipe file, you might
71type the following:
72::
73
74 $ bitbake matchbox-desktop
75
76Several different
77versions of ``matchbox-desktop`` might exist. BitBake chooses the one
78selected by the distribution configuration. You can get more details
79about how BitBake chooses between different target versions and
80providers in the
81":ref:`Preferences <bitbake:bb-bitbake-preferences>`" section
82of the BitBake User Manual.
83
84BitBake also tries to execute any dependent tasks first. So for example,
85before building ``matchbox-desktop``, BitBake would build a cross
86compiler and ``glibc`` if they had not already been built.
87
88A useful BitBake option to consider is the ``-k`` or ``--continue``
89option. This option instructs BitBake to try and continue processing the
90job as long as possible even after encountering an error. When an error
91occurs, the target that failed and those that depend on it cannot be
92remade. However, when you use this option other dependencies can still
93be processed.
94
95.. _overview-components-recipes:
96
97Recipes
98-------
99
100Files that have the ``.bb`` suffix are "recipes" files. In general, a
101recipe contains information about a single piece of software. This
102information includes the location from which to download the unaltered
103source, any source patches to be applied to that source (if needed),
104which special configuration options to apply, how to compile the source
105files, and how to package the compiled output.
106
107The term "package" is sometimes used to refer to recipes. However, since
108the word "package" is used for the packaged output from the OpenEmbedded
109build system (i.e. ``.ipk`` or ``.deb`` files), this document avoids
110using the term "package" when referring to recipes.
111
112.. _overview-components-classes:
113
114Classes
115-------
116
117Class files (``.bbclass``) contain information that is useful to share
118between recipes files. An example is the
119:ref:`autotools <ref-classes-autotools>` class,
120which contains common settings for any application that Autotools uses.
121The ":ref:`ref-manual/ref-classes:Classes`" chapter in the
122Yocto Project Reference Manual provides details about classes and how to
123use them.
124
125.. _overview-components-configurations:
126
127Configurations
128--------------
129
130The configuration files (``.conf``) define various configuration
131variables that govern the OpenEmbedded build process. These files fall
132into several areas that define machine configuration options,
133distribution configuration options, compiler tuning options, general
134common configuration options, and user configuration options in
135``conf/local.conf``, which is found in the :term:`Build Directory`.
136
137
138.. _overview-layers:
139
140Layers
141======
142
143Layers are repositories that contain related metadata (i.e. sets of
144instructions) that tell the OpenEmbedded build system how to build a
145target. Yocto Project's `layer model <#the-yocto-project-layer-model>`__
146facilitates collaboration, sharing, customization, and reuse within the
147Yocto Project development environment. Layers logically separate
148information for your project. For example, you can use a layer to hold
149all the configurations for a particular piece of hardware. Isolating
150hardware-specific configurations allows you to share other metadata by
151using a different layer where that metadata might be common across
152several pieces of hardware.
153
154Many layers exist that work in the Yocto Project development
155environment. The `Yocto Project Curated Layer
Andrew Geisslerc3d88e42020-10-02 09:45:00 -0500156Index <https://www.yoctoproject.org/software-overview/layers/>`__
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500157and `OpenEmbedded Layer
158Index <http://layers.openembedded.org/layerindex/branch/master/layers/>`__
159both contain layers from which you can use or leverage.
160
161By convention, layers in the Yocto Project follow a specific form.
162Conforming to a known structure allows BitBake to make assumptions
163during builds on where to find types of metadata. You can find
164procedures and learn about tools (i.e. ``bitbake-layers``) for creating
165layers suitable for the Yocto Project in the
166":ref:`dev-manual/dev-manual-common-tasks:understanding and creating layers`"
167section of the Yocto Project Development Tasks Manual.
168
169.. _openembedded-build-system-build-concepts:
170
171OpenEmbedded Build System Concepts
172==================================
173
174This section takes a more detailed look inside the build process used by
175the :term:`OpenEmbedded Build System`,
176which is the build
177system specific to the Yocto Project. At the heart of the build system
178is BitBake, the task executor.
179
180The following diagram represents the high-level workflow of a build. The
181remainder of this section expands on the fundamental input, output,
182process, and metadata logical blocks that make up the workflow.
183
184.. image:: figures/YP-flow-diagram.png
185 :align: center
186
187In general, the build's workflow consists of several functional areas:
188
189- *User Configuration:* metadata you can use to control the build
190 process.
191
192- *Metadata Layers:* Various layers that provide software, machine, and
193 distro metadata.
194
195- *Source Files:* Upstream releases, local projects, and SCMs.
196
197- *Build System:* Processes under the control of
198 :term:`BitBake`. This block expands
199 on how BitBake fetches source, applies patches, completes
200 compilation, analyzes output for package generation, creates and
201 tests packages, generates images, and generates cross-development
202 tools.
203
204- *Package Feeds:* Directories containing output packages (RPM, DEB or
205 IPK), which are subsequently used in the construction of an image or
206 Software Development Kit (SDK), produced by the build system. These
207 feeds can also be copied and shared using a web server or other means
208 to facilitate extending or updating existing images on devices at
209 runtime if runtime package management is enabled.
210
211- *Images:* Images produced by the workflow.
212
213- *Application Development SDK:* Cross-development tools that are
214 produced along with an image or separately with BitBake.
215
216User Configuration
217------------------
218
219User configuration helps define the build. Through user configuration,
220you can tell BitBake the target architecture for which you are building
221the image, where to store downloaded source, and other build properties.
222
223The following figure shows an expanded representation of the "User
224Configuration" box of the `general workflow
225figure <#general-workflow-figure>`__:
226
227.. image:: figures/user-configuration.png
228 :align: center
229
230BitBake needs some basic configuration files in order to complete a
231build. These files are ``*.conf`` files. The minimally necessary ones
232reside as example files in the ``build/conf`` directory of the
233:term:`Source Directory`. For simplicity,
234this section refers to the Source Directory as the "Poky Directory."
235
236When you clone the :term:`Poky` Git repository
237or you download and unpack a Yocto Project release, you can set up the
238Source Directory to be named anything you want. For this discussion, the
239cloned repository uses the default name ``poky``.
240
241.. note::
242
243 The Poky repository is primarily an aggregation of existing
244 repositories. It is not a canonical upstream source.
245
246The ``meta-poky`` layer inside Poky contains a ``conf`` directory that
247has example configuration files. These example files are used as a basis
248for creating actual configuration files when you source
249:ref:`structure-core-script`, which is the
250build environment script.
251
252Sourcing the build environment script creates a
253:term:`Build Directory` if one does not
254already exist. BitBake uses the Build Directory for all its work during
255builds. The Build Directory has a ``conf`` directory that contains
256default versions of your ``local.conf`` and ``bblayers.conf``
257configuration files. These default configuration files are created only
258if versions do not already exist in the Build Directory at the time you
259source the build environment setup script.
260
261Because the Poky repository is fundamentally an aggregation of existing
262repositories, some users might be familiar with running the
263:ref:`structure-core-script` script in the context of separate
264:term:`OpenEmbedded-Core (OE-Core)` and BitBake
265repositories rather than a single Poky repository. This discussion
266assumes the script is executed from within a cloned or unpacked version
267of Poky.
268
269Depending on where the script is sourced, different sub-scripts are
270called to set up the Build Directory (Yocto or OpenEmbedded).
271Specifically, the script ``scripts/oe-setup-builddir`` inside the poky
272directory sets up the Build Directory and seeds the directory (if
273necessary) with configuration files appropriate for the Yocto Project
274development environment.
275
276.. note::
277
278 The
279 scripts/oe-setup-builddir
280 script uses the
281 ``$TEMPLATECONF``
282 variable to determine which sample configuration files to locate.
283
284The ``local.conf`` file provides many basic variables that define a
285build environment. Here is a list of a few. To see the default
286configurations in a ``local.conf`` file created by the build environment
287script, see the
288:yocto_git:`local.conf.sample </cgit/cgit.cgi/poky/tree/meta-poky/conf/local.conf.sample>`
289in the ``meta-poky`` layer:
290
291- *Target Machine Selection:* Controlled by the
292 :term:`MACHINE` variable.
293
294- *Download Directory:* Controlled by the
295 :term:`DL_DIR` variable.
296
297- *Shared State Directory:* Controlled by the
298 :term:`SSTATE_DIR` variable.
299
300- *Build Output:* Controlled by the
301 :term:`TMPDIR` variable.
302
303- *Distribution Policy:* Controlled by the
304 :term:`DISTRO` variable.
305
306- *Packaging Format:* Controlled by the
307 :term:`PACKAGE_CLASSES`
308 variable.
309
310- *SDK Target Architecture:* Controlled by the
311 :term:`SDKMACHINE` variable.
312
313- *Extra Image Packages:* Controlled by the
314 :term:`EXTRA_IMAGE_FEATURES`
315 variable.
316
317.. note::
318
319 Configurations set in the
320 conf/local.conf
321 file can also be set in the
322 conf/site.conf
323 and
324 conf/auto.conf
325 configuration files.
326
327The ``bblayers.conf`` file tells BitBake what layers you want considered
328during the build. By default, the layers listed in this file include
329layers minimally needed by the build system. However, you must manually
330add any custom layers you have created. You can find more information on
331working with the ``bblayers.conf`` file in the
332":ref:`dev-manual/dev-manual-common-tasks:enabling your layer`"
333section in the Yocto Project Development Tasks Manual.
334
335The files ``site.conf`` and ``auto.conf`` are not created by the
336environment initialization script. If you want the ``site.conf`` file,
337you need to create that yourself. The ``auto.conf`` file is typically
338created by an autobuilder:
339
340- *site.conf:* You can use the ``conf/site.conf`` configuration
341 file to configure multiple build directories. For example, suppose
342 you had several build environments and they shared some common
343 features. You can set these default build properties here. A good
344 example is perhaps the packaging format to use through the
345 :term:`PACKAGE_CLASSES`
346 variable.
347
348 One useful scenario for using the ``conf/site.conf`` file is to
349 extend your :term:`BBPATH` variable
350 to include the path to a ``conf/site.conf``. Then, when BitBake looks
351 for Metadata using ``BBPATH``, it finds the ``conf/site.conf`` file
352 and applies your common configurations found in the file. To override
353 configurations in a particular build directory, alter the similar
354 configurations within that build directory's ``conf/local.conf``
355 file.
356
357- *auto.conf:* The file is usually created and written to by an
358 autobuilder. The settings put into the file are typically the same as
359 you would find in the ``conf/local.conf`` or the ``conf/site.conf``
360 files.
361
362You can edit all configuration files to further define any particular
363build environment. This process is represented by the "User
364Configuration Edits" box in the figure.
365
366When you launch your build with the ``bitbake target`` command, BitBake
367sorts out the configurations to ultimately define your build
368environment. It is important to understand that the
369:term:`OpenEmbedded Build System` reads the
370configuration files in a specific order: ``site.conf``, ``auto.conf``,
371and ``local.conf``. And, the build system applies the normal assignment
372statement rules as described in the
373":doc:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata`" chapter
374of the BitBake User Manual. Because the files are parsed in a specific
375order, variable assignments for the same variable could be affected. For
376example, if the ``auto.conf`` file and the ``local.conf`` set variable1
377to different values, because the build system parses ``local.conf``
378after ``auto.conf``, variable1 is assigned the value from the
379``local.conf`` file.
380
381Metadata, Machine Configuration, and Policy Configuration
382---------------------------------------------------------
383
384The previous section described the user configurations that define
385BitBake's global behavior. This section takes a closer look at the
386layers the build system uses to further control the build. These layers
387provide Metadata for the software, machine, and policies.
388
389In general, three types of layer input exists. You can see them below
390the "User Configuration" box in the `general workflow
391figure <#general-workflow-figure>`__:
392
393- *Metadata (.bb + Patches):* Software layers containing
394 user-supplied recipe files, patches, and append files. A good example
395 of a software layer might be the
396 `meta-qt5 layer <https://github.com/meta-qt5/meta-qt5>`__ from
397 the `OpenEmbedded Layer
398 Index <http://layers.openembedded.org/layerindex/branch/master/layers/>`__.
399 This layer is for version 5.0 of the popular
400 `Qt <https://wiki.qt.io/About_Qt>`__ cross-platform application
401 development framework for desktop, embedded and mobile.
402
403- *Machine BSP Configuration:* Board Support Package (BSP) layers (i.e.
404 "BSP Layer" in the following figure) providing machine-specific
405 configurations. This type of information is specific to a particular
406 target architecture. A good example of a BSP layer from the `Poky
407 Reference Distribution <#gs-reference-distribution-poky>`__ is the
408 :yocto_git:`meta-yocto-bsp </cgit/cgit.cgi/poky/tree/meta-yocto-bsp>`
409 layer.
410
411- *Policy Configuration:* Distribution Layers (i.e. "Distro Layer" in
412 the following figure) providing top-level or general policies for the
413 images or SDKs being built for a particular distribution. For
414 example, in the Poky Reference Distribution the distro layer is the
415 :yocto_git:`meta-poky </cgit/cgit.cgi/poky/tree/meta-poky>`
416 layer. Within the distro layer is a ``conf/distro`` directory that
417 contains distro configuration files (e.g.
418 :yocto_git:`poky.conf </cgit/cgit.cgi/poky/tree/meta-poky/conf/distro/poky.conf>`
419 that contain many policy configurations for the Poky distribution.
420
421The following figure shows an expanded representation of these three
422layers from the `general workflow figure <#general-workflow-figure>`__:
423
424.. image:: figures/layer-input.png
425 :align: center
426
427In general, all layers have a similar structure. They all contain a
428licensing file (e.g. ``COPYING.MIT``) if the layer is to be distributed,
429a ``README`` file as good practice and especially if the layer is to be
430distributed, a configuration directory, and recipe directories. You can
431learn about the general structure for layers used with the Yocto Project
432in the
433":ref:`dev-manual/dev-manual-common-tasks:creating your own layer`"
434section in the
435Yocto Project Development Tasks Manual. For a general discussion on
436layers and the many layers from which you can draw, see the
437"`Layers <#overview-layers>`__" and "`The Yocto Project Layer
438Model <#the-yocto-project-layer-model>`__" sections both earlier in this
439manual.
440
441If you explored the previous links, you discovered some areas where many
442layers that work with the Yocto Project exist. The `Source
443Repositories <http://git.yoctoproject.org/>`__ also shows layers
444categorized under "Yocto Metadata Layers."
445
446.. note::
447
448 Layers exist in the Yocto Project Source Repositories that cannot be
449 found in the OpenEmbedded Layer Index. These layers are either
450 deprecated or experimental in nature.
451
452BitBake uses the ``conf/bblayers.conf`` file, which is part of the user
453configuration, to find what layers it should be using as part of the
454build.
455
456Distro Layer
457~~~~~~~~~~~~
458
459The distribution layer provides policy configurations for your
460distribution. Best practices dictate that you isolate these types of
461configurations into their own layer. Settings you provide in
462``conf/distro/distro.conf`` override similar settings that BitBake finds
463in your ``conf/local.conf`` file in the Build Directory.
464
465The following list provides some explanation and references for what you
466typically find in the distribution layer:
467
468- *classes:* Class files (``.bbclass``) hold common functionality that
469 can be shared among recipes in the distribution. When your recipes
470 inherit a class, they take on the settings and functions for that
471 class. You can read more about class files in the
472 ":ref:`ref-manual/ref-classes:Classes`" chapter of the Yocto
473 Reference Manual.
474
475- *conf:* This area holds configuration files for the layer
476 (``conf/layer.conf``), the distribution
477 (``conf/distro/distro.conf``), and any distribution-wide include
478 files.
479
480- *recipes-*:* Recipes and append files that affect common
481 functionality across the distribution. This area could include
482 recipes and append files to add distribution-specific configuration,
483 initialization scripts, custom image recipes, and so forth. Examples
484 of ``recipes-*`` directories are ``recipes-core`` and
485 ``recipes-extra``. Hierarchy and contents within a ``recipes-*``
486 directory can vary. Generally, these directories contain recipe files
487 (``*.bb``), recipe append files (``*.bbappend``), directories that
488 are distro-specific for configuration files, and so forth.
489
490BSP Layer
491~~~~~~~~~
492
493The BSP Layer provides machine configurations that target specific
494hardware. Everything in this layer is specific to the machine for which
495you are building the image or the SDK. A common structure or form is
496defined for BSP layers. You can learn more about this structure in the
497:doc:`../bsp-guide/bsp-guide`.
498
499.. note::
500
501 In order for a BSP layer to be considered compliant with the Yocto
502 Project, it must meet some structural requirements.
503
504The BSP Layer's configuration directory contains configuration files for
505the machine (``conf/machine/machine.conf``) and, of course, the layer
506(``conf/layer.conf``).
507
508The remainder of the layer is dedicated to specific recipes by function:
509``recipes-bsp``, ``recipes-core``, ``recipes-graphics``,
510``recipes-kernel``, and so forth. Metadata can exist for multiple
511formfactors, graphics support systems, and so forth.
512
513.. note::
514
515 While the figure shows several
516 recipes-\*
517 directories, not all these directories appear in all BSP layers.
518
519Software Layer
520~~~~~~~~~~~~~~
521
522The software layer provides the Metadata for additional software
523packages used during the build. This layer does not include Metadata
524that is specific to the distribution or the machine, which are found in
525their respective layers.
526
527This layer contains any recipes, append files, and patches, that your
528project needs.
529
530.. _sources-dev-environment:
531
532Sources
533-------
534
535In order for the OpenEmbedded build system to create an image or any
536target, it must be able to access source files. The `general workflow
537figure <#general-workflow-figure>`__ represents source files using the
538"Upstream Project Releases", "Local Projects", and "SCMs (optional)"
539boxes. The figure represents mirrors, which also play a role in locating
540source files, with the "Source Materials" box.
541
542The method by which source files are ultimately organized is a function
543of the project. For example, for released software, projects tend to use
544tarballs or other archived files that can capture the state of a release
545guaranteeing that it is statically represented. On the other hand, for a
546project that is more dynamic or experimental in nature, a project might
547keep source files in a repository controlled by a Source Control Manager
548(SCM) such as Git. Pulling source from a repository allows you to
549control the point in the repository (the revision) from which you want
550to build software. Finally, a combination of the two might exist, which
551would give the consumer a choice when deciding where to get source
552files.
553
554BitBake uses the :term:`SRC_URI`
555variable to point to source files regardless of their location. Each
556recipe must have a ``SRC_URI`` variable that points to the source.
557
558Another area that plays a significant role in where source files come
559from is pointed to by the
560:term:`DL_DIR` variable. This area is
561a cache that can hold previously downloaded source. You can also
562instruct the OpenEmbedded build system to create tarballs from Git
563repositories, which is not the default behavior, and store them in the
564``DL_DIR`` by using the
565:term:`BB_GENERATE_MIRROR_TARBALLS`
566variable.
567
568Judicious use of a ``DL_DIR`` directory can save the build system a trip
569across the Internet when looking for files. A good method for using a
570download directory is to have ``DL_DIR`` point to an area outside of
571your Build Directory. Doing so allows you to safely delete the Build
572Directory if needed without fear of removing any downloaded source file.
573
574The remainder of this section provides a deeper look into the source
575files and the mirrors. Here is a more detailed look at the source file
576area of the `general workflow figure <#general-workflow-figure>`__:
577
578.. image:: figures/source-input.png
579 :align: center
580
581Upstream Project Releases
582~~~~~~~~~~~~~~~~~~~~~~~~~
583
584Upstream project releases exist anywhere in the form of an archived file
585(e.g. tarball or zip file). These files correspond to individual
586recipes. For example, the figure uses specific releases each for
587BusyBox, Qt, and Dbus. An archive file can be for any released product
588that can be built using a recipe.
589
590Local Projects
591~~~~~~~~~~~~~~
592
593Local projects are custom bits of software the user provides. These bits
594reside somewhere local to a project - perhaps a directory into which the
595user checks in items (e.g. a local directory containing a development
596source tree used by the group).
597
598The canonical method through which to include a local project is to use
599the :ref:`externalsrc <ref-classes-externalsrc>`
600class to include that local project. You use either the ``local.conf``
601or a recipe's append file to override or set the recipe to point to the
602local directory on your disk to pull in the whole source tree.
603
604.. _scms:
605
606Source Control Managers (Optional)
607~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
608
609Another place from which the build system can get source files is with
610:ref:`fetchers <bitbake:bb-fetchers>` employing various Source
611Control Managers (SCMs) such as Git or Subversion. In such cases, a
612repository is cloned or checked out. The
613:ref:`ref-tasks-fetch` task inside
614BitBake uses the :term:`SRC_URI`
615variable and the argument's prefix to determine the correct fetcher
616module.
617
618.. note::
619
620 For information on how to have the OpenEmbedded build system generate
621 tarballs for Git repositories and place them in the
622 DL_DIR
623 directory, see the :term:`BB_GENERATE_MIRROR_TARBALLS`
624 variable in the Yocto Project Reference Manual.
625
626When fetching a repository, BitBake uses the
627:term:`SRCREV` variable to determine
628the specific revision from which to build.
629
630Source Mirror(s)
631~~~~~~~~~~~~~~~~
632
633Two kinds of mirrors exist: pre-mirrors and regular mirrors. The
634:term:`PREMIRRORS` and
635:term:`MIRRORS` variables point to
636these, respectively. BitBake checks pre-mirrors before looking upstream
637for any source files. Pre-mirrors are appropriate when you have a shared
638directory that is not a directory defined by the
639:term:`DL_DIR` variable. A Pre-mirror
640typically points to a shared directory that is local to your
641organization.
642
643Regular mirrors can be any site across the Internet that is used as an
644alternative location for source code should the primary site not be
645functioning for some reason or another.
646
647.. _package-feeds-dev-environment:
648
649Package Feeds
650-------------
651
652When the OpenEmbedded build system generates an image or an SDK, it gets
653the packages from a package feed area located in the
654:term:`Build Directory`. The `general
655workflow figure <#general-workflow-figure>`__ shows this package feeds
656area in the upper-right corner.
657
658This section looks a little closer into the package feeds area used by
659the build system. Here is a more detailed look at the area:
660
661.. image:: figures/package-feeds.png
662 :align: center
663
664Package feeds are an intermediary step in the build process. The
665OpenEmbedded build system provides classes to generate different package
666types, and you specify which classes to enable through the
667:term:`PACKAGE_CLASSES`
668variable. Before placing the packages into package feeds, the build
669process validates them with generated output quality assurance checks
670through the :ref:`insane <ref-classes-insane>`
671class.
672
673The package feed area resides in the Build Directory. The directory the
674build system uses to temporarily store packages is determined by a
675combination of variables and the particular package manager in use. See
676the "Package Feeds" box in the illustration and note the information to
677the right of that area. In particular, the following defines where
678package files are kept:
679
680- :term:`DEPLOY_DIR`: Defined as
681 ``tmp/deploy`` in the Build Directory.
682
683- ``DEPLOY_DIR_*``: Depending on the package manager used, the package
684 type sub-folder. Given RPM, IPK, or DEB packaging and tarball
685 creation, the
686 :term:`DEPLOY_DIR_RPM`,
687 :term:`DEPLOY_DIR_IPK`,
688 :term:`DEPLOY_DIR_DEB`, or
689 :term:`DEPLOY_DIR_TAR`,
690 variables are used, respectively.
691
692- :term:`PACKAGE_ARCH`: Defines
693 architecture-specific sub-folders. For example, packages could exist
694 for the i586 or qemux86 architectures.
695
696BitBake uses the
697:ref:`do_package_write_* <ref-tasks-package_write_deb>`
698tasks to generate packages and place them into the package holding area
699(e.g. ``do_package_write_ipk`` for IPK packages). See the
700":ref:`ref-tasks-package_write_deb`",
701":ref:`ref-tasks-package_write_ipk`",
702":ref:`ref-tasks-package_write_rpm`",
703and
704":ref:`ref-tasks-package_write_tar`"
705sections in the Yocto Project Reference Manual for additional
706information. As an example, consider a scenario where an IPK packaging
707manager is being used and package architecture support for both i586 and
708qemux86 exist. Packages for the i586 architecture are placed in
709``build/tmp/deploy/ipk/i586``, while packages for the qemux86
710architecture are placed in ``build/tmp/deploy/ipk/qemux86``.
711
712.. _bitbake-dev-environment:
713
714BitBake Tool
715------------
716
717The OpenEmbedded build system uses
718:term:`BitBake` to produce images and
719Software Development Kits (SDKs). You can see from the `general workflow
720figure <#general-workflow-figure>`__, the BitBake area consists of
721several functional areas. This section takes a closer look at each of
722those areas.
723
724.. note::
725
726 Separate documentation exists for the BitBake tool. See the
727 BitBake User Manual
728 for reference material on BitBake.
729
730.. _source-fetching-dev-environment:
731
732Source Fetching
733~~~~~~~~~~~~~~~
734
735The first stages of building a recipe are to fetch and unpack the source
736code:
737
738.. image:: figures/source-fetching.png
739 :align: center
740
741The :ref:`ref-tasks-fetch` and
742:ref:`ref-tasks-unpack` tasks fetch
743the source files and unpack them into the
744:term:`Build Directory`.
745
746.. note::
747
748 For every local file (e.g.
749 file://
750 ) that is part of a recipe's
751 SRC_URI
752 statement, the OpenEmbedded build system takes a checksum of the file
753 for the recipe and inserts the checksum into the signature for the
754 do_fetch
755 task. If any local file has been modified, the
756 do_fetch
757 task and all tasks that depend on it are re-executed.
758
759By default, everything is accomplished in the Build Directory, which has
760a defined structure. For additional general information on the Build
761Directory, see the ":ref:`structure-core-build`" section in
762the Yocto Project Reference Manual.
763
764Each recipe has an area in the Build Directory where the unpacked source
765code resides. The :term:`S` variable points
766to this area for a recipe's unpacked source code. The name of that
767directory for any given recipe is defined from several different
768variables. The preceding figure and the following list describe the
769Build Directory's hierarchy:
770
771- :term:`TMPDIR`: The base directory
772 where the OpenEmbedded build system performs all its work during the
773 build. The default base directory is the ``tmp`` directory.
774
775- :term:`PACKAGE_ARCH`: The
776 architecture of the built package or packages. Depending on the
777 eventual destination of the package or packages (i.e. machine
778 architecture, :term:`Build Host`, SDK, or
779 specific machine), ``PACKAGE_ARCH`` varies. See the variable's
780 description for details.
781
782- :term:`TARGET_OS`: The operating
783 system of the target device. A typical value would be "linux" (e.g.
784 "qemux86-poky-linux").
785
786- :term:`PN`: The name of the recipe used
787 to build the package. This variable can have multiple meanings.
788 However, when used in the context of input files, ``PN`` represents
789 the name of the recipe.
790
791- :term:`WORKDIR`: The location
792 where the OpenEmbedded build system builds a recipe (i.e. does the
793 work to create the package).
794
795 - :term:`PV`: The version of the
796 recipe used to build the package.
797
798 - :term:`PR`: The revision of the
799 recipe used to build the package.
800
801- :term:`S`: Contains the unpacked source
802 files for a given recipe.
803
804 - :term:`BPN`: The name of the recipe
805 used to build the package. The ``BPN`` variable is a version of
806 the ``PN`` variable but with common prefixes and suffixes removed.
807
808 - :term:`PV`: The version of the
809 recipe used to build the package.
810
811.. note::
812
813 In the previous figure, notice that two sample hierarchies exist: one
814 based on package architecture (i.e.
815 PACKAGE_ARCH
816 ) and one based on a machine (i.e.
817 MACHINE
818 ). The underlying structures are identical. The differentiator being
819 what the OpenEmbedded build system is using as a build target (e.g.
820 general architecture, a build host, an SDK, or a specific machine).
821
822.. _patching-dev-environment:
823
824Patching
825~~~~~~~~
826
827Once source code is fetched and unpacked, BitBake locates patch files
828and applies them to the source files:
829
830.. image:: figures/patching.png
831 :align: center
832
833The :ref:`ref-tasks-patch` task uses a
834recipe's :term:`SRC_URI` statements
835and the :term:`FILESPATH` variable
836to locate applicable patch files.
837
838Default processing for patch files assumes the files have either
839``*.patch`` or ``*.diff`` file types. You can use ``SRC_URI`` parameters
840to change the way the build system recognizes patch files. See the
841:ref:`ref-tasks-patch` task for more
842information.
843
844BitBake finds and applies multiple patches for a single recipe in the
845order in which it locates the patches. The ``FILESPATH`` variable
846defines the default set of directories that the build system uses to
847search for patch files. Once found, patches are applied to the recipe's
848source files, which are located in the
849:term:`S` directory.
850
851For more information on how the source directories are created, see the
852"`Source Fetching <#source-fetching-dev-environment>`__" section. For
853more information on how to create patches and how the build system
854processes patches, see the
855":ref:`dev-manual/dev-manual-common-tasks:patching code`"
856section in the
857Yocto Project Development Tasks Manual. You can also see the
858":ref:`sdk-manual/sdk-extensible:use \`\`devtool modify\`\` to modify the source of an existing component`"
859section in the Yocto Project Application Development and the Extensible
860Software Development Kit (SDK) manual and the
861":ref:`kernel-dev/kernel-dev-common:using traditional kernel development to patch the kernel`"
862section in the Yocto Project Linux Kernel Development Manual.
863
864.. _configuration-compilation-and-staging-dev-environment:
865
866Configuration, Compilation, and Staging
867~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
868
869After source code is patched, BitBake executes tasks that configure and
870compile the source code. Once compilation occurs, the files are copied
871to a holding area (staged) in preparation for packaging:
872
873.. image:: figures/configuration-compile-autoreconf.png
874 :align: center
875
876This step in the build process consists of the following tasks:
877
878- :ref:`ref-tasks-prepare_recipe_sysroot`:
879 This task sets up the two sysroots in
880 ``${``\ :term:`WORKDIR`\ ``}``
881 (i.e. ``recipe-sysroot`` and ``recipe-sysroot-native``) so that
882 during the packaging phase the sysroots can contain the contents of
883 the
884 :ref:`ref-tasks-populate_sysroot`
885 tasks of the recipes on which the recipe containing the tasks
886 depends. A sysroot exists for both the target and for the native
887 binaries, which run on the host system.
888
889- *do_configure*: This task configures the source by enabling and
890 disabling any build-time and configuration options for the software
891 being built. Configurations can come from the recipe itself as well
892 as from an inherited class. Additionally, the software itself might
893 configure itself depending on the target for which it is being built.
894
895 The configurations handled by the
896 :ref:`ref-tasks-configure` task
897 are specific to configurations for the source code being built by the
898 recipe.
899
900 If you are using the
901 :ref:`autotools <ref-classes-autotools>` class,
902 you can add additional configuration options by using the
903 :term:`EXTRA_OECONF` or
904 :term:`PACKAGECONFIG_CONFARGS`
905 variables. For information on how this variable works within that
906 class, see the
907 :ref:`autotools <ref-classes-autotools>` class
908 :yocto_git:`here </cgit/cgit.cgi/poky/tree/meta/classes/autotools.bbclass>`.
909
910- *do_compile*: Once a configuration task has been satisfied,
911 BitBake compiles the source using the
912 :ref:`ref-tasks-compile` task.
913 Compilation occurs in the directory pointed to by the
914 :term:`B` variable. Realize that the
915 ``B`` directory is, by default, the same as the
916 :term:`S` directory.
917
918- *do_install*: After compilation completes, BitBake executes the
919 :ref:`ref-tasks-install` task.
920 This task copies files from the ``B`` directory and places them in a
921 holding area pointed to by the :term:`D`
922 variable. Packaging occurs later using files from this holding
923 directory.
924
925.. _package-splitting-dev-environment:
926
927Package Splitting
928~~~~~~~~~~~~~~~~~
929
930After source code is configured, compiled, and staged, the build system
931analyzes the results and splits the output into packages:
932
933.. image:: figures/analysis-for-package-splitting.png
934 :align: center
935
936The :ref:`ref-tasks-package` and
937:ref:`ref-tasks-packagedata`
938tasks combine to analyze the files found in the
939:term:`D` directory and split them into
940subsets based on available packages and files. Analysis involves the
941following as well as other items: splitting out debugging symbols,
942looking at shared library dependencies between packages, and looking at
943package relationships.
944
945The ``do_packagedata`` task creates package metadata based on the
946analysis such that the build system can generate the final packages. The
947:ref:`ref-tasks-populate_sysroot`
948task stages (copies) a subset of the files installed by the
949:ref:`ref-tasks-install` task into
950the appropriate sysroot. Working, staged, and intermediate results of
951the analysis and package splitting process use several areas:
952
953- :term:`PKGD`: The destination
954 directory (i.e. ``package``) for packages before they are split into
955 individual packages.
956
957- :term:`PKGDESTWORK`: A
958 temporary work area (i.e. ``pkgdata``) used by the ``do_package``
959 task to save package metadata.
960
961- :term:`PKGDEST`: The parent
962 directory (i.e. ``packages-split``) for packages after they have been
963 split.
964
965- :term:`PKGDATA_DIR`: A shared,
966 global-state directory that holds packaging metadata generated during
967 the packaging process. The packaging process copies metadata from
968 ``PKGDESTWORK`` to the ``PKGDATA_DIR`` area where it becomes globally
969 available.
970
971- :term:`STAGING_DIR_HOST`:
972 The path for the sysroot for the system on which a component is built
973 to run (i.e. ``recipe-sysroot``).
974
975- :term:`STAGING_DIR_NATIVE`:
976 The path for the sysroot used when building components for the build
977 host (i.e. ``recipe-sysroot-native``).
978
979- :term:`STAGING_DIR_TARGET`:
980 The path for the sysroot used when a component that is built to
981 execute on a system and it generates code for yet another machine
982 (e.g. cross-canadian recipes).
983
984The :term:`FILES` variable defines the
985files that go into each package in
986:term:`PACKAGES`. If you want
987details on how this is accomplished, you can look at
988:yocto_git:`package.bbclass </cgit/cgit.cgi/poky/tree/meta/classes/package.bbclass>`.
989
990Depending on the type of packages being created (RPM, DEB, or IPK), the
991:ref:`do_package_write_* <ref-tasks-package_write_deb>`
992task creates the actual packages and places them in the Package Feed
993area, which is ``${TMPDIR}/deploy``. You can see the "`Package
994Feeds <#package-feeds-dev-environment>`__" section for more detail on
995that part of the build process.
996
997.. note::
998
999 Support for creating feeds directly from the
1000 deploy/\*
1001 directories does not exist. Creating such feeds usually requires some
1002 kind of feed maintenance mechanism that would upload the new packages
1003 into an official package feed (e.g. the Ångström distribution). This
1004 functionality is highly distribution-specific and thus is not
1005 provided out of the box.
1006
1007.. _image-generation-dev-environment:
1008
1009Image Generation
1010~~~~~~~~~~~~~~~~
1011
1012Once packages are split and stored in the Package Feeds area, the build
1013system uses BitBake to generate the root filesystem image:
1014
1015.. image:: figures/image-generation.png
1016 :align: center
1017
1018The image generation process consists of several stages and depends on
1019several tasks and variables. The
1020:ref:`ref-tasks-rootfs` task creates
1021the root filesystem (file and directory structure) for an image. This
1022task uses several key variables to help create the list of packages to
1023actually install:
1024
1025- :term:`IMAGE_INSTALL`: Lists
1026 out the base set of packages from which to install from the Package
1027 Feeds area.
1028
1029- :term:`PACKAGE_EXCLUDE`:
1030 Specifies packages that should not be installed into the image.
1031
1032- :term:`IMAGE_FEATURES`:
1033 Specifies features to include in the image. Most of these features
1034 map to additional packages for installation.
1035
1036- :term:`PACKAGE_CLASSES`:
1037 Specifies the package backend (e.g. RPM, DEB, or IPK) to use and
1038 consequently helps determine where to locate packages within the
1039 Package Feeds area.
1040
1041- :term:`IMAGE_LINGUAS`:
1042 Determines the language(s) for which additional language support
1043 packages are installed.
1044
1045- :term:`PACKAGE_INSTALL`:
1046 The final list of packages passed to the package manager for
1047 installation into the image.
1048
1049With :term:`IMAGE_ROOTFS`
1050pointing to the location of the filesystem under construction and the
1051``PACKAGE_INSTALL`` variable providing the final list of packages to
1052install, the root file system is created.
1053
1054Package installation is under control of the package manager (e.g.
1055dnf/rpm, opkg, or apt/dpkg) regardless of whether or not package
1056management is enabled for the target. At the end of the process, if
1057package management is not enabled for the target, the package manager's
1058data files are deleted from the root filesystem. As part of the final
1059stage of package installation, post installation scripts that are part
1060of the packages are run. Any scripts that fail to run on the build host
1061are run on the target when the target system is first booted. If you are
1062using a
1063:ref:`read-only root filesystem <dev-manual/dev-manual-common-tasks:creating a read-only root filesystem>`,
1064all the post installation scripts must succeed on the build host during
1065the package installation phase since the root filesystem on the target
1066is read-only.
1067
1068The final stages of the ``do_rootfs`` task handle post processing. Post
1069processing includes creation of a manifest file and optimizations.
1070
1071The manifest file (``.manifest``) resides in the same directory as the
1072root filesystem image. This file lists out, line-by-line, the installed
1073packages. The manifest file is useful for the
1074:ref:`testimage <ref-classes-testimage*>` class,
1075for example, to determine whether or not to run specific tests. See the
1076:term:`IMAGE_MANIFEST`
1077variable for additional information.
1078
1079Optimizing processes that are run across the image include ``mklibs``,
1080``prelink``, and any other post-processing commands as defined by the
1081:term:`ROOTFS_POSTPROCESS_COMMAND`
1082variable. The ``mklibs`` process optimizes the size of the libraries,
1083while the ``prelink`` process optimizes the dynamic linking of shared
1084libraries to reduce start up time of executables.
1085
1086After the root filesystem is built, processing begins on the image
1087through the :ref:`ref-tasks-image`
1088task. The build system runs any pre-processing commands as defined by
1089the
1090:term:`IMAGE_PREPROCESS_COMMAND`
1091variable. This variable specifies a list of functions to call before the
1092build system creates the final image output files.
1093
1094The build system dynamically creates ``do_image_*`` tasks as needed,
1095based on the image types specified in the
1096:term:`IMAGE_FSTYPES` variable.
1097The process turns everything into an image file or a set of image files
1098and can compress the root filesystem image to reduce the overall size of
1099the image. The formats used for the root filesystem depend on the
1100``IMAGE_FSTYPES`` variable. Compression depends on whether the formats
1101support compression.
1102
1103As an example, a dynamically created task when creating a particular
1104image type would take the following form:
1105::
1106
1107 do_image_type
1108
1109So, if the type
1110as specified by the ``IMAGE_FSTYPES`` were ``ext4``, the dynamically
1111generated task would be as follows:
1112::
1113
1114 do_image_ext4
1115
1116The final task involved in image creation is the
1117:ref:`do_image_complete <ref-tasks-image-complete>`
1118task. This task completes the image by applying any image post
1119processing as defined through the
1120:term:`IMAGE_POSTPROCESS_COMMAND`
1121variable. The variable specifies a list of functions to call once the
1122build system has created the final image output files.
1123
1124.. note::
1125
1126 The entire image generation process is run under
1127 Pseudo. Running under Pseudo ensures that the files in the root filesystem
1128 have correct ownership.
1129
1130.. _sdk-generation-dev-environment:
1131
1132SDK Generation
1133~~~~~~~~~~~~~~
1134
1135The OpenEmbedded build system uses BitBake to generate the Software
1136Development Kit (SDK) installer scripts for both the standard SDK and
1137the extensible SDK (eSDK):
1138
1139.. image:: figures/sdk-generation.png
1140 :align: center
1141
1142.. note::
1143
1144 For more information on the cross-development toolchain generation,
1145 see the ":ref:`overview-manual/overview-manual-concepts:cross-development toolchain generation`"
1146 section. For information on advantages gained when building a
1147 cross-development toolchain using the do_populate_sdk task, see the
1148 ":ref:`sdk-manual/sdk-appendix-obtain:building an sdk installer`" section in
1149 the Yocto Project Application Development and the Extensible Software
1150 Development Kit (eSDK) manual.
1151
1152Like image generation, the SDK script process consists of several stages
1153and depends on many variables. The
1154:ref:`ref-tasks-populate_sdk`
1155and
1156:ref:`ref-tasks-populate_sdk_ext`
1157tasks use these key variables to help create the list of packages to
1158actually install. For information on the variables listed in the figure,
1159see the "`Application Development SDK <#sdk-dev-environment>`__"
1160section.
1161
1162The ``do_populate_sdk`` task helps create the standard SDK and handles
1163two parts: a target part and a host part. The target part is the part
1164built for the target hardware and includes libraries and headers. The
1165host part is the part of the SDK that runs on the
1166:term:`SDKMACHINE`.
1167
1168The ``do_populate_sdk_ext`` task helps create the extensible SDK and
1169handles host and target parts differently than its counter part does for
1170the standard SDK. For the extensible SDK, the task encapsulates the
1171build system, which includes everything needed (host and target) for the
1172SDK.
1173
1174Regardless of the type of SDK being constructed, the tasks perform some
1175cleanup after which a cross-development environment setup script and any
1176needed configuration files are created. The final output is the
1177Cross-development toolchain installation script (``.sh`` file), which
1178includes the environment setup script.
1179
1180Stamp Files and the Rerunning of Tasks
1181~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1182
1183For each task that completes successfully, BitBake writes a stamp file
1184into the :term:`STAMPS_DIR`
1185directory. The beginning of the stamp file's filename is determined by
1186the :term:`STAMP` variable, and the end
1187of the name consists of the task's name and current `input
1188checksum <#overview-checksums>`__.
1189
1190.. note::
1191
1192 This naming scheme assumes that
1193 BB_SIGNATURE_HANDLER
1194 is "OEBasicHash", which is almost always the case in current
1195 OpenEmbedded.
1196
1197To determine if a task needs to be rerun, BitBake checks if a stamp file
1198with a matching input checksum exists for the task. If such a stamp file
1199exists, the task's output is assumed to exist and still be valid. If the
1200file does not exist, the task is rerun.
1201
1202.. note::
1203
1204 The stamp mechanism is more general than the shared state (sstate)
1205 cache mechanism described in the "`Setscene Tasks and Shared
1206 State <#setscene-tasks-and-shared-state>`__" section. BitBake avoids
1207 rerunning any task that has a valid stamp file, not just tasks that
1208 can be accelerated through the sstate cache.
1209
1210 However, you should realize that stamp files only serve as a marker
1211 that some work has been done and that these files do not record task
1212 output. The actual task output would usually be somewhere in
1213 :term:`TMPDIR` (e.g. in some
1214 recipe's :term:`WORKDIR`.) What
1215 the sstate cache mechanism adds is a way to cache task output that
1216 can then be shared between build machines.
1217
1218Since ``STAMPS_DIR`` is usually a subdirectory of ``TMPDIR``, removing
1219``TMPDIR`` will also remove ``STAMPS_DIR``, which means tasks will
1220properly be rerun to repopulate ``TMPDIR``.
1221
1222If you want some task to always be considered "out of date", you can
1223mark it with the :ref:`nostamp <bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`
1224varflag. If some other task depends on such a task, then that task will
1225also always be considered out of date, which might not be what you want.
1226
1227For details on how to view information about a task's signature, see the
1228":ref:`dev-manual/dev-manual-common-tasks:viewing task variable dependencies`"
1229section in the Yocto Project Development Tasks Manual.
1230
1231Setscene Tasks and Shared State
1232~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1233
1234The description of tasks so far assumes that BitBake needs to build
1235everything and no available prebuilt objects exist. BitBake does support
1236skipping tasks if prebuilt objects are available. These objects are
1237usually made available in the form of a shared state (sstate) cache.
1238
1239.. note::
1240
1241 For information on variables affecting sstate, see the
1242 :term:`SSTATE_DIR`
1243 and
1244 :term:`SSTATE_MIRRORS`
1245 variables.
1246
1247The idea of a setscene task (i.e ``do_``\ taskname\ ``_setscene``) is a
1248version of the task where instead of building something, BitBake can
1249skip to the end result and simply place a set of files into specific
1250locations as needed. In some cases, it makes sense to have a setscene
1251task variant (e.g. generating package files in the
1252:ref:`do_package_write_* <ref-tasks-package_write_deb>`
1253task). In other cases, it does not make sense (e.g. a
1254:ref:`ref-tasks-patch` task or a
1255:ref:`ref-tasks-unpack` task) since
1256the work involved would be equal to or greater than the underlying task.
1257
1258In the build system, the common tasks that have setscene variants are
1259:ref:`ref-tasks-package`,
1260``do_package_write_*``,
1261:ref:`ref-tasks-deploy`,
1262:ref:`ref-tasks-packagedata`, and
1263:ref:`ref-tasks-populate_sysroot`.
1264Notice that these tasks represent most of the tasks whose output is an
1265end result.
1266
1267The build system has knowledge of the relationship between these tasks
1268and other preceding tasks. For example, if BitBake runs
1269``do_populate_sysroot_setscene`` for something, it does not make sense
1270to run any of the ``do_fetch``, ``do_unpack``, ``do_patch``,
1271``do_configure``, ``do_compile``, and ``do_install`` tasks. However, if
1272``do_package`` needs to be run, BitBake needs to run those other tasks.
1273
1274It becomes more complicated if everything can come from an sstate cache
1275because some objects are simply not required at all. For example, you do
1276not need a compiler or native tools, such as quilt, if nothing exists to
1277compile or patch. If the ``do_package_write_*`` packages are available
1278from sstate, BitBake does not need the ``do_package`` task data.
1279
1280To handle all these complexities, BitBake runs in two phases. The first
1281is the "setscene" stage. During this stage, BitBake first checks the
1282sstate cache for any targets it is planning to build. BitBake does a
1283fast check to see if the object exists rather than a complete download.
1284If nothing exists, the second phase, which is the setscene stage,
1285completes and the main build proceeds.
1286
1287If objects are found in the sstate cache, the build system works
1288backwards from the end targets specified by the user. For example, if an
1289image is being built, the build system first looks for the packages
1290needed for that image and the tools needed to construct an image. If
1291those are available, the compiler is not needed. Thus, the compiler is
1292not even downloaded. If something was found to be unavailable, or the
1293download or setscene task fails, the build system then tries to install
1294dependencies, such as the compiler, from the cache.
1295
1296The availability of objects in the sstate cache is handled by the
1297function specified by the
1298:term:`bitbake:BB_HASHCHECK_FUNCTION`
1299variable and returns a list of available objects. The function specified
1300by the
1301:term:`bitbake:BB_SETSCENE_DEPVALID`
1302variable is the function that determines whether a given dependency
1303needs to be followed, and whether for any given relationship the
1304function needs to be passed. The function returns a True or False value.
1305
1306.. _images-dev-environment:
1307
1308Images
1309------
1310
1311The images produced by the build system are compressed forms of the root
1312filesystem and are ready to boot on a target device. You can see from
1313the `general workflow figure <#general-workflow-figure>`__ that BitBake
1314output, in part, consists of images. This section takes a closer look at
1315this output:
1316
1317.. image:: figures/images.png
1318 :align: center
1319
1320.. note::
1321
1322 For a list of example images that the Yocto Project provides, see the
1323 ":doc:`../ref-manual/ref-images`" chapter in the Yocto Project Reference
1324 Manual.
1325
1326The build process writes images out to the :term:`Build Directory`
1327inside the
1328``tmp/deploy/images/machine/`` folder as shown in the figure. This
1329folder contains any files expected to be loaded on the target device.
1330The :term:`DEPLOY_DIR` variable
1331points to the ``deploy`` directory, while the
1332:term:`DEPLOY_DIR_IMAGE`
1333variable points to the appropriate directory containing images for the
1334current configuration.
1335
1336- kernel-image: A kernel binary file. The
1337 :term:`KERNEL_IMAGETYPE`
1338 variable determines the naming scheme for the kernel image file.
1339 Depending on this variable, the file could begin with a variety of
1340 naming strings. The ``deploy/images/``\ machine directory can contain
1341 multiple image files for the machine.
1342
1343- root-filesystem-image: Root filesystems for the target device (e.g.
1344 ``*.ext3`` or ``*.bz2`` files). The
1345 :term:`IMAGE_FSTYPES`
1346 variable determines the root filesystem image type. The
1347 ``deploy/images/``\ machine directory can contain multiple root
1348 filesystems for the machine.
1349
1350- kernel-modules: Tarballs that contain all the modules built for the
1351 kernel. Kernel module tarballs exist for legacy purposes and can be
1352 suppressed by setting the
1353 :term:`MODULE_TARBALL_DEPLOY`
1354 variable to "0". The ``deploy/images/``\ machine directory can
1355 contain multiple kernel module tarballs for the machine.
1356
1357- bootloaders: If applicable to the target machine, bootloaders
1358 supporting the image. The ``deploy/images/``\ machine directory can
1359 contain multiple bootloaders for the machine.
1360
1361- symlinks: The ``deploy/images/``\ machine folder contains a symbolic
1362 link that points to the most recently built file for each machine.
1363 These links might be useful for external scripts that need to obtain
1364 the latest version of each file.
1365
1366.. _sdk-dev-environment:
1367
1368Application Development SDK
1369---------------------------
1370
1371In the `general workflow figure <#general-workflow-figure>`__, the
1372output labeled "Application Development SDK" represents an SDK. The SDK
1373generation process differs depending on whether you build an extensible
1374SDK (e.g. ``bitbake -c populate_sdk_ext`` imagename) or a standard SDK
1375(e.g. ``bitbake -c populate_sdk`` imagename). This section takes a
1376closer look at this output:
1377
1378.. image:: figures/sdk.png
1379 :align: center
1380
1381The specific form of this output is a set of files that includes a
1382self-extracting SDK installer (``*.sh``), host and target manifest
1383files, and files used for SDK testing. When the SDK installer file is
1384run, it installs the SDK. The SDK consists of a cross-development
1385toolchain, a set of libraries and headers, and an SDK environment setup
1386script. Running this installer essentially sets up your
1387cross-development environment. You can think of the cross-toolchain as
1388the "host" part because it runs on the SDK machine. You can think of the
1389libraries and headers as the "target" part because they are built for
1390the target hardware. The environment setup script is added so that you
1391can initialize the environment before using the tools.
1392
1393.. note::
1394
1395 - The Yocto Project supports several methods by which you can set up
1396 this cross-development environment. These methods include
1397 downloading pre-built SDK installers or building and installing
1398 your own SDK installer.
1399
1400 - For background information on cross-development toolchains in the
1401 Yocto Project development environment, see the "`Cross-Development
1402 Toolchain Generation <#cross-development-toolchain-generation>`__"
1403 section.
1404
1405 - For information on setting up a cross-development environment, see
1406 the :doc:`../sdk-manual/sdk-manual` manual.
1407
1408All the output files for an SDK are written to the ``deploy/sdk`` folder
1409inside the :term:`Build Directory` as
1410shown in the previous figure. Depending on the type of SDK, several
1411variables exist that help configure these files. The following list
1412shows the variables associated with an extensible SDK:
1413
1414- :term:`DEPLOY_DIR`: Points to
1415 the ``deploy`` directory.
1416
1417- :term:`SDK_EXT_TYPE`:
1418 Controls whether or not shared state artifacts are copied into the
1419 extensible SDK. By default, all required shared state artifacts are
1420 copied into the SDK.
1421
1422- :term:`SDK_INCLUDE_PKGDATA`:
1423 Specifies whether or not packagedata is included in the extensible
1424 SDK for all recipes in the "world" target.
1425
1426- :term:`SDK_INCLUDE_TOOLCHAIN`:
1427 Specifies whether or not the toolchain is included when building the
1428 extensible SDK.
1429
1430- :term:`SDK_LOCAL_CONF_WHITELIST`:
1431 A list of variables allowed through from the build system
1432 configuration into the extensible SDK configuration.
1433
1434- :term:`SDK_LOCAL_CONF_BLACKLIST`:
1435 A list of variables not allowed through from the build system
1436 configuration into the extensible SDK configuration.
1437
1438- :term:`SDK_INHERIT_BLACKLIST`:
1439 A list of classes to remove from the
1440 :term:`INHERIT` value globally
1441 within the extensible SDK configuration.
1442
1443This next list, shows the variables associated with a standard SDK:
1444
1445- :term:`DEPLOY_DIR`: Points to
1446 the ``deploy`` directory.
1447
1448- :term:`SDKMACHINE`: Specifies
1449 the architecture of the machine on which the cross-development tools
1450 are run to create packages for the target hardware.
1451
1452- :term:`SDKIMAGE_FEATURES`:
1453 Lists the features to include in the "target" part of the SDK.
1454
1455- :term:`TOOLCHAIN_HOST_TASK`:
1456 Lists packages that make up the host part of the SDK (i.e. the part
1457 that runs on the ``SDKMACHINE``). When you use
1458 ``bitbake -c populate_sdk imagename`` to create the SDK, a set of
1459 default packages apply. This variable allows you to add more
1460 packages.
1461
1462- :term:`TOOLCHAIN_TARGET_TASK`:
1463 Lists packages that make up the target part of the SDK (i.e. the part
1464 built for the target hardware).
1465
1466- :term:`SDKPATH`: Defines the
1467 default SDK installation path offered by the installation script.
1468
1469- :term:`SDK_HOST_MANIFEST`:
1470 Lists all the installed packages that make up the host part of the
1471 SDK. This variable also plays a minor role for extensible SDK
1472 development as well. However, it is mainly used for the standard SDK.
1473
1474- :term:`SDK_TARGET_MANIFEST`:
1475 Lists all the installed packages that make up the target part of the
1476 SDK. This variable also plays a minor role for extensible SDK
1477 development as well. However, it is mainly used for the standard SDK.
1478
1479Cross-Development Toolchain Generation
1480======================================
1481
1482The Yocto Project does most of the work for you when it comes to
1483creating :ref:`sdk-manual/sdk-intro:the cross-development toolchain`. This
1484section provides some technical background on how cross-development
1485toolchains are created and used. For more information on toolchains, you
1486can also see the :doc:`../sdk-manual/sdk-manual` manual.
1487
1488In the Yocto Project development environment, cross-development
1489toolchains are used to build images and applications that run on the
1490target hardware. With just a few commands, the OpenEmbedded build system
1491creates these necessary toolchains for you.
1492
1493The following figure shows a high-level build environment regarding
1494toolchain construction and use.
1495
1496.. image:: figures/cross-development-toolchains.png
1497 :align: center
1498
1499Most of the work occurs on the Build Host. This is the machine used to
1500build images and generally work within the the Yocto Project
1501environment. When you run
1502:term:`BitBake` to create an image, the
1503OpenEmbedded build system uses the host ``gcc`` compiler to bootstrap a
1504cross-compiler named ``gcc-cross``. The ``gcc-cross`` compiler is what
1505BitBake uses to compile source files when creating the target image. You
1506can think of ``gcc-cross`` simply as an automatically generated
1507cross-compiler that is used internally within BitBake only.
1508
1509.. note::
1510
1511 The extensible SDK does not use
1512 gcc-cross-canadian
1513 since this SDK ships a copy of the OpenEmbedded build system and the
1514 sysroot within it contains
1515 gcc-cross
1516 .
1517
Andrew Geissler6ce62a22020-11-30 19:58:47 -06001518The chain of events that occurs when the standard toolchain is bootstrapped:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001519::
1520
Andrew Geissler6ce62a22020-11-30 19:58:47 -06001521 binutils-cross -> linux-libc-headers -> gcc-cross -> libgcc-initial -> glibc -> libgcc -> gcc-runtime
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001522
Andrew Geissler6ce62a22020-11-30 19:58:47 -06001523- ``gcc``: The compiler, GNU Compiler Collection (GCC).
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001524
Andrew Geissler6ce62a22020-11-30 19:58:47 -06001525- ``binutils-cross``: The binary utilities needed in order
1526 to run the ``gcc-cross`` phase of the bootstrap operation and build the
1527 headers for the C library.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001528
Andrew Geissler6ce62a22020-11-30 19:58:47 -06001529- ``linux-libc-headers``: Headers needed for the cross-compiler and C library build.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001530
Andrew Geissler6ce62a22020-11-30 19:58:47 -06001531- ``libgcc-initial``: An initial version of the gcc support library needed
1532 to bootstrap ``glibc``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001533
Andrew Geissler6ce62a22020-11-30 19:58:47 -06001534- ``libgcc``: The final version of the gcc support library which
1535 can only be built once there is a C library to link against.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001536
1537- ``glibc``: The GNU C Library.
1538
1539- ``gcc-cross``: The final stage of the bootstrap process for the
1540 cross-compiler. This stage results in the actual cross-compiler that
1541 BitBake uses when it builds an image for a targeted device.
1542
Andrew Geissler6ce62a22020-11-30 19:58:47 -06001543 This tool is a "native" tool (i.e. it is designed to run on
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001544 the build host).
1545
1546- ``gcc-runtime``: Runtime libraries resulting from the toolchain
1547 bootstrapping process. This tool produces a binary that consists of
1548 the runtime libraries need for the targeted device.
1549
1550You can use the OpenEmbedded build system to build an installer for the
1551relocatable SDK used to develop applications. When you run the
1552installer, it installs the toolchain, which contains the development
1553tools (e.g., ``gcc-cross-canadian``, ``binutils-cross-canadian``, and
1554other ``nativesdk-*`` tools), which are tools native to the SDK (i.e.
1555native to :term:`SDK_ARCH`), you
1556need to cross-compile and test your software. The figure shows the
1557commands you use to easily build out this toolchain. This
1558cross-development toolchain is built to execute on the
1559:term:`SDKMACHINE`, which might or
1560might not be the same machine as the Build Host.
1561
1562.. note::
1563
1564 If your target architecture is supported by the Yocto Project, you
1565 can take advantage of pre-built images that ship with the Yocto
1566 Project and already contain cross-development toolchain installers.
1567
1568Here is the bootstrap process for the relocatable toolchain:
1569::
1570
1571 gcc -> binutils-crosssdk -> gcc-crosssdk-initial -> linux-libc-headers -> glibc-initial -> nativesdk-glibc -> gcc-crosssdk -> gcc-cross-canadian
1572
1573- ``gcc``: The build host's GNU Compiler Collection (GCC).
1574
1575- ``binutils-crosssdk``: The bare minimum binary utilities needed in
1576 order to run the ``gcc-crosssdk-initial`` phase of the bootstrap
1577 operation.
1578
1579- ``gcc-crosssdk-initial``: An early stage of the bootstrap process for
1580 creating the cross-compiler. This stage builds enough of the
1581 ``gcc-crosssdk`` and supporting pieces so that the final stage of the
1582 bootstrap process can produce the finished cross-compiler. This tool
1583 is a "native" binary that runs on the build host.
1584
1585- ``linux-libc-headers``: Headers needed for the cross-compiler.
1586
1587- ``glibc-initial``: An initial version of the Embedded GLIBC needed to
1588 bootstrap ``nativesdk-glibc``.
1589
1590- ``nativesdk-glibc``: The Embedded GLIBC needed to bootstrap the
1591 ``gcc-crosssdk``.
1592
1593- ``gcc-crosssdk``: The final stage of the bootstrap process for the
1594 relocatable cross-compiler. The ``gcc-crosssdk`` is a transitory
1595 compiler and never leaves the build host. Its purpose is to help in
1596 the bootstrap process to create the eventual ``gcc-cross-canadian``
1597 compiler, which is relocatable. This tool is also a "native" package
1598 (i.e. it is designed to run on the build host).
1599
1600- ``gcc-cross-canadian``: The final relocatable cross-compiler. When
1601 run on the :term:`SDKMACHINE`,
1602 this tool produces executable code that runs on the target device.
1603 Only one cross-canadian compiler is produced per architecture since
1604 they can be targeted at different processor optimizations using
1605 configurations passed to the compiler through the compile commands.
1606 This circumvents the need for multiple compilers and thus reduces the
1607 size of the toolchains.
1608
1609.. note::
1610
1611 For information on advantages gained when building a
1612 cross-development toolchain installer, see the
1613 ":ref:`sdk-manual/sdk-appendix-obtain:building an sdk installer`" appendix
1614 in the Yocto Project Application Development and the
1615 Extensible Software Development Kit (eSDK) manual.
1616
1617Shared State Cache
1618==================
1619
1620By design, the OpenEmbedded build system builds everything from scratch
1621unless :term:`BitBake` can determine
1622that parts do not need to be rebuilt. Fundamentally, building from
1623scratch is attractive as it means all parts are built fresh and no
1624possibility of stale data exists that can cause problems. When
1625developers hit problems, they typically default back to building from
1626scratch so they have a know state from the start.
1627
1628Building an image from scratch is both an advantage and a disadvantage
1629to the process. As mentioned in the previous paragraph, building from
1630scratch ensures that everything is current and starts from a known
1631state. However, building from scratch also takes much longer as it
1632generally means rebuilding things that do not necessarily need to be
1633rebuilt.
1634
1635The Yocto Project implements shared state code that supports incremental
1636builds. The implementation of the shared state code answers the
1637following questions that were fundamental roadblocks within the
1638OpenEmbedded incremental build support system:
1639
1640- What pieces of the system have changed and what pieces have not
1641 changed?
1642
1643- How are changed pieces of software removed and replaced?
1644
1645- How are pre-built components that do not need to be rebuilt from
1646 scratch used when they are available?
1647
1648For the first question, the build system detects changes in the "inputs"
1649to a given task by creating a checksum (or signature) of the task's
1650inputs. If the checksum changes, the system assumes the inputs have
1651changed and the task needs to be rerun. For the second question, the
1652shared state (sstate) code tracks which tasks add which output to the
1653build process. This means the output from a given task can be removed,
1654upgraded or otherwise manipulated. The third question is partly
1655addressed by the solution for the second question assuming the build
1656system can fetch the sstate objects from remote locations and install
1657them if they are deemed to be valid.
1658
1659.. note::
1660
1661 - The build system does not maintain
1662 :term:`PR` information as part of
1663 the shared state packages. Consequently, considerations exist that
1664 affect maintaining shared state feeds. For information on how the
1665 build system works with packages and can track incrementing ``PR``
1666 information, see the ":ref:`dev-manual/dev-manual-common-tasks:automatically incrementing a package version number`"
1667 section in the Yocto Project Development Tasks Manual.
1668
1669 - The code in the build system that supports incremental builds is
1670 not simple code. For techniques that help you work around issues
1671 related to shared state code, see the
1672 ":ref:`dev-manual/dev-manual-common-tasks:viewing metadata used to create the input signature of a shared state task`"
1673 and
1674 ":ref:`dev-manual/dev-manual-common-tasks:invalidating shared state to force a task to run`"
1675 sections both in the Yocto Project Development Tasks Manual.
1676
1677The rest of this section goes into detail about the overall incremental
1678build architecture, the checksums (signatures), and shared state.
1679
1680.. _concepts-overall-architecture:
1681
1682Overall Architecture
1683--------------------
1684
1685When determining what parts of the system need to be built, BitBake
1686works on a per-task basis rather than a per-recipe basis. You might
1687wonder why using a per-task basis is preferred over a per-recipe basis.
1688To help explain, consider having the IPK packaging backend enabled and
1689then switching to DEB. In this case, the
1690:ref:`ref-tasks-install` and
1691:ref:`ref-tasks-package` task outputs
1692are still valid. However, with a per-recipe approach, the build would
1693not include the ``.deb`` files. Consequently, you would have to
1694invalidate the whole build and rerun it. Rerunning everything is not the
1695best solution. Also, in this case, the core must be "taught" much about
1696specific tasks. This methodology does not scale well and does not allow
1697users to easily add new tasks in layers or as external recipes without
1698touching the packaged-staging core.
1699
1700.. _overview-checksums:
1701
1702Checksums (Signatures)
1703----------------------
1704
1705The shared state code uses a checksum, which is a unique signature of a
1706task's inputs, to determine if a task needs to be run again. Because it
1707is a change in a task's inputs that triggers a rerun, the process needs
1708to detect all the inputs to a given task. For shell tasks, this turns
1709out to be fairly easy because the build process generates a "run" shell
1710script for each task and it is possible to create a checksum that gives
1711you a good idea of when the task's data changes.
1712
1713To complicate the problem, there are things that should not be included
1714in the checksum. First, there is the actual specific build path of a
1715given task - the :term:`WORKDIR`. It
1716does not matter if the work directory changes because it should not
1717affect the output for target packages. Also, the build process has the
1718objective of making native or cross packages relocatable.
1719
1720.. note::
1721
1722 Both native and cross packages run on the
1723 build host. However, cross packages generate output for the target
1724 architecture.
1725
1726The checksum therefore needs to exclude ``WORKDIR``. The simplistic
1727approach for excluding the work directory is to set ``WORKDIR`` to some
1728fixed value and create the checksum for the "run" script.
1729
1730Another problem results from the "run" scripts containing functions that
1731might or might not get called. The incremental build solution contains
1732code that figures out dependencies between shell functions. This code is
1733used to prune the "run" scripts down to the minimum set, thereby
1734alleviating this problem and making the "run" scripts much more readable
1735as a bonus.
1736
1737So far, solutions for shell scripts exist. What about Python tasks? The
1738same approach applies even though these tasks are more difficult. The
1739process needs to figure out what variables a Python function accesses
1740and what functions it calls. Again, the incremental build solution
1741contains code that first figures out the variable and function
1742dependencies, and then creates a checksum for the data used as the input
1743to the task.
1744
1745Like the ``WORKDIR`` case, situations exist where dependencies should be
1746ignored. For these situations, you can instruct the build process to
1747ignore a dependency by using a line like the following:
1748::
1749
1750 PACKAGE_ARCHS[vardepsexclude] = "MACHINE"
1751
1752This example ensures that the :term:`PACKAGE_ARCHS` variable
1753does not depend on the value of :term:`MACHINE`, even if it does
1754reference it.
1755
1756Equally, there are cases where you need to add dependencies BitBake is
1757not able to find. You can accomplish this by using a line like the
1758following:
1759::
1760
1761 PACKAGE_ARCHS[vardeps] = "MACHINE"
1762
1763This example explicitly
1764adds the ``MACHINE`` variable as a dependency for ``PACKAGE_ARCHS``.
1765
1766As an example, consider a case with in-line Python where BitBake is not
1767able to figure out dependencies. When running in debug mode (i.e. using
1768``-DDD``), BitBake produces output when it discovers something for which
1769it cannot figure out dependencies. The Yocto Project team has currently
1770not managed to cover those dependencies in detail and is aware of the
1771need to fix this situation.
1772
1773Thus far, this section has limited discussion to the direct inputs into
1774a task. Information based on direct inputs is referred to as the
1775"basehash" in the code. However, the question of a task's indirect
1776inputs still exits - items already built and present in the
1777:term:`Build Directory`. The checksum (or
1778signature) for a particular task needs to add the hashes of all the
1779tasks on which the particular task depends. Choosing which dependencies
1780to add is a policy decision. However, the effect is to generate a master
1781checksum that combines the basehash and the hashes of the task's
1782dependencies.
1783
1784At the code level, a variety of ways exist by which both the basehash
1785and the dependent task hashes can be influenced. Within the BitBake
1786configuration file, you can give BitBake some extra information to help
1787it construct the basehash. The following statement effectively results
1788in a list of global variable dependency excludes (i.e. variables never
1789included in any checksum):
1790::
1791
1792 BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \\
1793 SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \\
1794 USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \\
1795 PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \\
1796 CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX"
1797
1798The
1799previous example excludes
1800:term:`WORKDIR` since that variable
1801is actually constructed as a path within
1802:term:`TMPDIR`, which is on the
1803whitelist.
1804
1805The rules for deciding which hashes of dependent tasks to include
1806through dependency chains are more complex and are generally
1807accomplished with a Python function. The code in
1808``meta/lib/oe/sstatesig.py`` shows two examples of this and also
1809illustrates how you can insert your own policy into the system if so
1810desired. This file defines the two basic signature generators
1811:term:`OpenEmbedded-Core (OE-Core)` uses: "OEBasic" and
1812"OEBasicHash". By default, a dummy "noop" signature handler is enabled
1813in BitBake. This means that behavior is unchanged from previous
1814versions. OE-Core uses the "OEBasicHash" signature handler by default
1815through this setting in the ``bitbake.conf`` file:
1816::
1817
1818 BB_SIGNATURE_HANDLER ?= "OEBasicHash"
1819
1820The "OEBasicHash" ``BB_SIGNATURE_HANDLER`` is the same
1821as the "OEBasic" version but adds the task hash to the `stamp
1822files <#stamp-files-and-the-rerunning-of-tasks>`__. This results in any
1823metadata change that changes the task hash, automatically causing the
1824task to be run again. This removes the need to bump
1825:term:`PR` values, and changes to metadata
1826automatically ripple across the build.
1827
1828It is also worth noting that the end result of these signature
1829generators is to make some dependency and hash information available to
1830the build. This information includes:
1831
1832- ``BB_BASEHASH_task-``\ taskname: The base hashes for each task in the
1833 recipe.
1834
1835- ``BB_BASEHASH_``\ filename\ ``:``\ taskname: The base hashes for each
1836 dependent task.
1837
1838- ``BBHASHDEPS_``\ filename\ ``:``\ taskname: The task dependencies for
1839 each task.
1840
1841- ``BB_TASKHASH``: The hash of the currently running task.
1842
1843Shared State
1844------------
1845
1846Checksums and dependencies, as discussed in the previous section, solve
1847half the problem of supporting a shared state. The other half of the
1848problem is being able to use checksum information during the build and
1849being able to reuse or rebuild specific components.
1850
1851The :ref:`sstate <ref-classes-sstate>` class is a
1852relatively generic implementation of how to "capture" a snapshot of a
1853given task. The idea is that the build process does not care about the
1854source of a task's output. Output could be freshly built or it could be
1855downloaded and unpacked from somewhere. In other words, the build
1856process does not need to worry about its origin.
1857
1858Two types of output exist. One type is just about creating a directory
1859in :term:`WORKDIR`. A good example is
1860the output of either
1861:ref:`ref-tasks-install` or
1862:ref:`ref-tasks-package`. The other
1863type of output occurs when a set of data is merged into a shared
1864directory tree such as the sysroot.
1865
1866The Yocto Project team has tried to keep the details of the
1867implementation hidden in ``sstate`` class. From a user's perspective,
1868adding shared state wrapping to a task is as simple as this
1869:ref:`ref-tasks-deploy` example taken
1870from the :ref:`deploy <ref-classes-deploy>` class:
1871::
1872
1873 DEPLOYDIR = "${WORKDIR}/deploy-${PN}"
1874 SSTATETASKS += "do_deploy"
1875 do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"
1876 do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
1877
1878 python do_deploy_setscene () {
1879 sstate_setscene(d)
1880 }
1881 addtask do_deploy_setscene
1882 do_deploy[dirs] = "${DEPLOYDIR} ${B}"
1883 do_deploy[stamp-extra-info] = "${MACHINE_ARCH}"
1884
1885The following list explains the previous example:
1886
1887- Adding "do_deploy" to ``SSTATETASKS`` adds some required
1888 sstate-related processing, which is implemented in the
1889 :ref:`sstate <ref-classes-sstate>` class, to
1890 before and after the
1891 :ref:`ref-tasks-deploy` task.
1892
1893- The ``do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"`` declares that
1894 ``do_deploy`` places its output in ``${DEPLOYDIR}`` when run normally
1895 (i.e. when not using the sstate cache). This output becomes the input
1896 to the shared state cache.
1897
1898- The ``do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"`` line
1899 causes the contents of the shared state cache to be copied to
1900 ``${DEPLOY_DIR_IMAGE}``.
1901
1902 .. note::
1903
1904 If ``do_deploy`` is not already in the shared state cache or if its input
1905 checksum (signature) has changed from when the output was cached, the task
1906 runs to populate the shared state cache, after which the contents of the
1907 shared state cache is copied to ${:term:`DEPLOY_DIR_IMAGE`}. If
1908 ``do_deploy`` is in the shared state cache and its signature indicates
1909 that the cached output is still valid (i.e. if no relevant task inputs
1910 have changed), then the contents of the shared state cache copies
1911 directly to ${``DEPLOY_DIR_IMAGE``} by the ``do_deploy_setscene`` task
1912 instead, skipping the ``do_deploy`` task.
1913
1914- The following task definition is glue logic needed to make the
1915 previous settings effective:
1916 ::
1917
1918 python do_deploy_setscene () {
1919 sstate_setscene(d)
1920 }
1921 addtask do_deploy_setscene
1922
1923 ``sstate_setscene()`` takes the flags above as input and accelerates the ``do_deploy`` task
1924 through the shared state cache if possible. If the task was
1925 accelerated, ``sstate_setscene()`` returns True. Otherwise, it
1926 returns False, and the normal ``do_deploy`` task runs. For more
1927 information, see the ":ref:`setscene <bitbake:bitbake-user-manual/bitbake-user-manual-execution:setscene>`"
1928 section in the BitBake User Manual.
1929
1930- The ``do_deploy[dirs] = "${DEPLOYDIR} ${B}"`` line creates
1931 ``${DEPLOYDIR}`` and ``${B}`` before the ``do_deploy`` task runs, and
1932 also sets the current working directory of ``do_deploy`` to ``${B}``.
1933 For more information, see the ":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags`"
1934 section in the BitBake
1935 User Manual.
1936
1937 .. note::
1938
1939 In cases where ``sstate-inputdirs`` and ``sstate-outputdirs`` would be
1940 the same, you can use ``sstate-plaindirs``. For example, to preserve the
1941 ${:term:`PKGD`} and ${:term:`PKGDEST`} output from the ``do_package``
1942 task, use the following:
1943 ::
1944
1945 do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}"
1946
1947
1948- The ``do_deploy[stamp-extra-info] = "${MACHINE_ARCH}"`` line appends
1949 extra metadata to the `stamp
1950 file <#stamp-files-and-the-rerunning-of-tasks>`__. In this case, the
1951 metadata makes the task specific to a machine's architecture. See
1952 ":ref:`bitbake:ref-bitbake-tasklist`"
1953 section in the BitBake User Manual for more information on the
1954 ``stamp-extra-info`` flag.
1955
1956- ``sstate-inputdirs`` and ``sstate-outputdirs`` can also be used with
1957 multiple directories. For example, the following declares
1958 ``PKGDESTWORK`` and ``SHLIBWORK`` as shared state input directories,
1959 which populates the shared state cache, and ``PKGDATA_DIR`` and
1960 ``SHLIBSDIR`` as the corresponding shared state output directories:
1961 ::
1962
1963 do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}"
1964 do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}"
1965
1966- These methods also include the ability to take a lockfile when
1967 manipulating shared state directory structures, for cases where file
1968 additions or removals are sensitive:
1969 ::
1970
1971 do_package[sstate-lockfile] = "${PACKAGELOCK}"
1972
1973Behind the scenes, the shared state code works by looking in
1974:term:`SSTATE_DIR` and
1975:term:`SSTATE_MIRRORS` for
1976shared state files. Here is an example:
1977::
1978
1979 SSTATE_MIRRORS ?= "\
1980 file://.\* http://someserver.tld/share/sstate/PATH;downloadfilename=PATH \n \
1981 file://.\* file:///some/local/dir/sstate/PATH"
1982
1983.. note::
1984
1985 The shared state directory (``SSTATE_DIR``) is organized into two-character
1986 subdirectories, where the subdirectory names are based on the first two
1987 characters of the hash.
1988 If the shared state directory structure for a mirror has the same structure
1989 as ``SSTATE_DIR``, you must specify "PATH" as part of the URI to enable the build
1990 system to map to the appropriate subdirectory.
1991
1992The shared state package validity can be detected just by looking at the
1993filename since the filename contains the task checksum (or signature) as
1994described earlier in this section. If a valid shared state package is
1995found, the build process downloads it and uses it to accelerate the
1996task.
1997
1998The build processes use the ``*_setscene`` tasks for the task
1999acceleration phase. BitBake goes through this phase before the main
2000execution code and tries to accelerate any tasks for which it can find
2001shared state packages. If a shared state package for a task is
2002available, the shared state package is used. This means the task and any
2003tasks on which it is dependent are not executed.
2004
2005As a real world example, the aim is when building an IPK-based image,
2006only the
2007:ref:`ref-tasks-package_write_ipk`
2008tasks would have their shared state packages fetched and extracted.
2009Since the sysroot is not used, it would never get extracted. This is
2010another reason why a task-based approach is preferred over a
2011recipe-based approach, which would have to install the output from every
2012task.
2013
2014Automatically Added Runtime Dependencies
2015========================================
2016
2017The OpenEmbedded build system automatically adds common types of runtime
2018dependencies between packages, which means that you do not need to
2019explicitly declare the packages using
2020:term:`RDEPENDS`. Three automatic
2021mechanisms exist (``shlibdeps``, ``pcdeps``, and ``depchains``) that
2022handle shared libraries, package configuration (pkg-config) modules, and
2023``-dev`` and ``-dbg`` packages, respectively. For other types of runtime
2024dependencies, you must manually declare the dependencies.
2025
2026- ``shlibdeps``: During the
2027 :ref:`ref-tasks-package` task of
2028 each recipe, all shared libraries installed by the recipe are
2029 located. For each shared library, the package that contains the
2030 shared library is registered as providing the shared library. More
2031 specifically, the package is registered as providing the
2032 `soname <https://en.wikipedia.org/wiki/Soname>`__ of the library. The
2033 resulting shared-library-to-package mapping is saved globally in
2034 :term:`PKGDATA_DIR` by the
2035 :ref:`ref-tasks-packagedata`
2036 task.
2037
2038 Simultaneously, all executables and shared libraries installed by the
2039 recipe are inspected to see what shared libraries they link against.
2040 For each shared library dependency that is found, ``PKGDATA_DIR`` is
2041 queried to see if some package (likely from a different recipe)
2042 contains the shared library. If such a package is found, a runtime
2043 dependency is added from the package that depends on the shared
2044 library to the package that contains the library.
2045
2046 The automatically added runtime dependency also includes a version
2047 restriction. This version restriction specifies that at least the
2048 current version of the package that provides the shared library must
2049 be used, as if "package (>= version)" had been added to ``RDEPENDS``.
2050 This forces an upgrade of the package containing the shared library
2051 when installing the package that depends on the library, if needed.
2052
2053 If you want to avoid a package being registered as providing a
2054 particular shared library (e.g. because the library is for internal
2055 use only), then add the library to
2056 :term:`PRIVATE_LIBS` inside
2057 the package's recipe.
2058
2059- ``pcdeps``: During the ``do_package`` task of each recipe, all
2060 pkg-config modules (``*.pc`` files) installed by the recipe are
2061 located. For each module, the package that contains the module is
2062 registered as providing the module. The resulting module-to-package
2063 mapping is saved globally in ``PKGDATA_DIR`` by the
2064 ``do_packagedata`` task.
2065
2066 Simultaneously, all pkg-config modules installed by the recipe are
2067 inspected to see what other pkg-config modules they depend on. A
2068 module is seen as depending on another module if it contains a
2069 "Requires:" line that specifies the other module. For each module
2070 dependency, ``PKGDATA_DIR`` is queried to see if some package
2071 contains the module. If such a package is found, a runtime dependency
2072 is added from the package that depends on the module to the package
2073 that contains the module.
2074
2075 .. note::
2076
2077 The
2078 pcdeps
2079 mechanism most often infers dependencies between
2080 -dev
2081 packages.
2082
2083- ``depchains``: If a package ``foo`` depends on a package ``bar``,
2084 then ``foo-dev`` and ``foo-dbg`` are also made to depend on
2085 ``bar-dev`` and ``bar-dbg``, respectively. Taking the ``-dev``
2086 packages as an example, the ``bar-dev`` package might provide headers
2087 and shared library symlinks needed by ``foo-dev``, which shows the
2088 need for a dependency between the packages.
2089
2090 The dependencies added by ``depchains`` are in the form of
2091 :term:`RRECOMMENDS`.
2092
2093 .. note::
2094
2095 By default, ``foo-dev`` also has an ``RDEPENDS``-style dependency on
2096 ``foo``, because the default value of ``RDEPENDS_${PN}-dev`` (set in
2097 bitbake.conf) includes "${PN}".
2098
2099 To ensure that the dependency chain is never broken, ``-dev`` and
2100 ``-dbg`` packages are always generated by default, even if the
2101 packages turn out to be empty. See the
2102 :term:`ALLOW_EMPTY` variable
2103 for more information.
2104
2105The ``do_package`` task depends on the ``do_packagedata`` task of each
2106recipe in :term:`DEPENDS` through use
2107of a ``[``\ :ref:`deptask <bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ``]``
2108declaration, which guarantees that the required
2109shared-library/module-to-package mapping information will be available
2110when needed as long as ``DEPENDS`` has been correctly set.
2111
2112Fakeroot and Pseudo
2113===================
2114
2115Some tasks are easier to implement when allowed to perform certain
2116operations that are normally reserved for the root user (e.g.
2117:ref:`ref-tasks-install`,
2118:ref:`do_package_write* <ref-tasks-package_write_deb>`,
2119:ref:`ref-tasks-rootfs`, and
2120:ref:`do_image* <ref-tasks-image>`). For example,
2121the ``do_install`` task benefits from being able to set the UID and GID
2122of installed files to arbitrary values.
2123
2124One approach to allowing tasks to perform root-only operations would be
2125to require :term:`BitBake` to run as
2126root. However, this method is cumbersome and has security issues. The
2127approach that is actually used is to run tasks that benefit from root
2128privileges in a "fake" root environment. Within this environment, the
2129task and its child processes believe that they are running as the root
2130user, and see an internally consistent view of the filesystem. As long
2131as generating the final output (e.g. a package or an image) does not
2132require root privileges, the fact that some earlier steps ran in a fake
2133root environment does not cause problems.
2134
2135The capability to run tasks in a fake root environment is known as
2136"`fakeroot <http://man.he.net/man1/fakeroot>`__", which is derived from
2137the BitBake keyword/variable flag that requests a fake root environment
2138for a task.
2139
2140In the :term:`OpenEmbedded Build System`,
2141the program that
2142implements fakeroot is known as
2143`Pseudo <https://www.yoctoproject.org/software-item/pseudo/>`__. Pseudo
2144overrides system calls by using the environment variable ``LD_PRELOAD``,
2145which results in the illusion of running as root. To keep track of
2146"fake" file ownership and permissions resulting from operations that
2147require root permissions, Pseudo uses an SQLite 3 database. This
2148database is stored in
2149``${``\ :term:`WORKDIR`\ ``}/pseudo/files.db``
2150for individual recipes. Storing the database in a file as opposed to in
2151memory gives persistence between tasks and builds, which is not
2152accomplished using fakeroot.
2153
2154.. note::
2155
2156 If you add your own task that manipulates the same files or
2157 directories as a fakeroot task, then that task also needs to run
2158 under fakeroot. Otherwise, the task cannot run root-only operations,
2159 and cannot see the fake file ownership and permissions set by the
2160 other task. You need to also add a dependency on
2161 virtual/fakeroot-native:do_populate_sysroot
2162 , giving the following:
2163 ::
2164
2165 fakeroot do_mytask () {
2166 ...
2167 }
2168 do_mytask[depends] += "virtual/fakeroot-native:do_populate_sysroot"
2169
2170
2171For more information, see the
2172:term:`FAKEROOT* <bitbake:FAKEROOT>` variables in the
2173BitBake User Manual. You can also reference the "`Why Not
2174Fakeroot? <https://github.com/wrpseudo/pseudo/wiki/WhyNotFakeroot>`__"
2175article for background information on Fakeroot and Pseudo.