blob: a4b1efbe8bd7f71f5dec908d9e2efb921f2e3d7e [file] [log] [blame]
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001.. SPDX-License-Identifier: CC-BY-2.5
2
3=========
4Execution
5=========
6
7|
8
9The primary purpose for running BitBake is to produce some kind of
10output such as a single installable package, a kernel, a software
11development kit, or even a full, board-specific bootable Linux image,
12complete with bootloader, kernel, and root filesystem. Of course, you
13can execute the ``bitbake`` command with options that cause it to
14execute single tasks, compile single recipe files, capture or clear
15data, or simply return information about the execution environment.
16
17This chapter describes BitBake's execution process from start to finish
18when you use it to create an image. The execution process is launched
Andrew Geisslerc926e172021-05-07 16:11:35 -050019using the following command form::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050020
21 $ bitbake target
22
23For information on
24the BitBake command and its options, see ":ref:`The BitBake Command
25<bitbake-user-manual-command>`" section.
26
27.. note::
28
29 Prior to executing BitBake, you should take advantage of available
30 parallel thread execution on your build host by setting the
31 :term:`BB_NUMBER_THREADS` variable in
32 your project's ``local.conf`` configuration file.
33
34 A common method to determine this value for your build host is to run
Andrew Geisslerc926e172021-05-07 16:11:35 -050035 the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050036
37 $ grep processor /proc/cpuinfo
38
39 This command returns
40 the number of processors, which takes into account hyper-threading.
41 Thus, a quad-core build host with hyper-threading most likely shows
42 eight processors, which is the value you would then assign to
Patrick Williams213cb262021-08-07 19:21:33 -050043 :term:`BB_NUMBER_THREADS`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050044
45 A possibly simpler solution is that some Linux distributions (e.g.
46 Debian and Ubuntu) provide the ``ncpus`` command.
47
48Parsing the Base Configuration Metadata
49=======================================
50
51The first thing BitBake does is parse base configuration metadata. Base
52configuration metadata consists of your project's ``bblayers.conf`` file
53to determine what layers BitBake needs to recognize, all necessary
54``layer.conf`` files (one from each layer), and ``bitbake.conf``. The
55data itself is of various types:
56
57- **Recipes:** Details about particular pieces of software.
58
59- **Class Data:** An abstraction of common build information (e.g. how to
60 build a Linux kernel).
61
62- **Configuration Data:** Machine-specific settings, policy decisions,
63 and so forth. Configuration data acts as the glue to bind everything
64 together.
65
66The ``layer.conf`` files are used to construct key variables such as
67:term:`BBPATH` and :term:`BBFILES`.
Patrick Williams213cb262021-08-07 19:21:33 -050068:term:`BBPATH` is used to search for configuration and class files under the
69``conf`` and ``classes`` directories, respectively. :term:`BBFILES` is used
Andrew Geisslerc9f78652020-09-18 14:11:35 -050070to locate both recipe and recipe append files (``.bb`` and
71``.bbappend``). If there is no ``bblayers.conf`` file, it is assumed the
Patrick Williams213cb262021-08-07 19:21:33 -050072user has set the :term:`BBPATH` and :term:`BBFILES` directly in the environment.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050073
Patrick Williams213cb262021-08-07 19:21:33 -050074Next, the ``bitbake.conf`` file is located using the :term:`BBPATH` variable
Andrew Geisslerc9f78652020-09-18 14:11:35 -050075that was just constructed. The ``bitbake.conf`` file may also include
76other configuration files using the ``include`` or ``require``
77directives.
78
79Prior to parsing configuration files, BitBake looks at certain
80variables, including:
81
82- :term:`BB_ENV_WHITELIST`
83- :term:`BB_ENV_EXTRAWHITE`
84- :term:`BB_PRESERVE_ENV`
85- :term:`BB_ORIGENV`
86- :term:`BITBAKE_UI`
87
88The first four variables in this list relate to how BitBake treats shell
89environment variables during task execution. By default, BitBake cleans
90the environment variables and provides tight control over the shell
91execution environment. However, through the use of these first four
92variables, you can apply your control regarding the environment
93variables allowed to be used by BitBake in the shell during execution of
94tasks. See the
95":ref:`bitbake-user-manual/bitbake-user-manual-metadata:Passing Information Into the Build Task Environment`"
96section and the information about these variables in the variable
97glossary for more information on how they work and on how to use them.
98
99The base configuration metadata is global and therefore affects all
100recipes and tasks that are executed.
101
102BitBake first searches the current working directory for an optional
103``conf/bblayers.conf`` configuration file. This file is expected to
104contain a :term:`BBLAYERS` variable that is a
105space-delimited list of 'layer' directories. Recall that if BitBake
106cannot find a ``bblayers.conf`` file, then it is assumed the user has
Patrick Williams213cb262021-08-07 19:21:33 -0500107set the :term:`BBPATH` and :term:`BBFILES` variables directly in the
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500108environment.
109
110For each directory (layer) in this list, a ``conf/layer.conf`` file is
111located and parsed with the :term:`LAYERDIR` variable
112being set to the directory where the layer was found. The idea is these
113files automatically set up :term:`BBPATH` and other
114variables correctly for a given build directory.
115
116BitBake then expects to find the ``conf/bitbake.conf`` file somewhere in
Patrick Williams213cb262021-08-07 19:21:33 -0500117the user-specified :term:`BBPATH`. That configuration file generally has
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500118include directives to pull in any other metadata such as files specific
119to the architecture, the machine, the local environment, and so forth.
120
121Only variable definitions and include directives are allowed in BitBake
122``.conf`` files. Some variables directly influence BitBake's behavior.
123These variables might have been set from the environment depending on
124the environment variables previously mentioned or set in the
125configuration files. The ":ref:`bitbake-user-manual/bitbake-user-manual-ref-variables:Variables Glossary`"
126chapter presents a full list of
127variables.
128
129After parsing configuration files, BitBake uses its rudimentary
130inheritance mechanism, which is through class files, to inherit some
131standard classes. BitBake parses a class when the inherit directive
132responsible for getting that class is encountered.
133
134The ``base.bbclass`` file is always included. Other classes that are
135specified in the configuration using the
136:term:`INHERIT` variable are also included. BitBake
137searches for class files in a ``classes`` subdirectory under the paths
Patrick Williams213cb262021-08-07 19:21:33 -0500138in :term:`BBPATH` in the same way as configuration files.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500139
140A good way to get an idea of the configuration files and the class files
141used in your execution environment is to run the following BitBake
Andrew Geisslerc926e172021-05-07 16:11:35 -0500142command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500143
144 $ bitbake -e > mybb.log
145
146Examining the top of the ``mybb.log``
147shows you the many configuration files and class files used in your
148execution environment.
149
150.. note::
151
152 You need to be aware of how BitBake parses curly braces. If a recipe
153 uses a closing curly brace within the function and the character has
154 no leading spaces, BitBake produces a parsing error. If you use a
155 pair of curly braces in a shell function, the closing curly brace
156 must not be located at the start of the line without leading spaces.
157
Andrew Geisslerc926e172021-05-07 16:11:35 -0500158 Here is an example that causes BitBake to produce a parsing error::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500159
160 fakeroot create_shar() {
161 cat << "EOF" > ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh
162 usage()
163 {
164 echo "test"
165 ###### The following "}" at the start of the line causes a parsing error ######
166 }
167 EOF
168 }
169
170 Writing the recipe this way avoids the error:
171 fakeroot create_shar() {
172 cat << "EOF" > ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh
173 usage()
174 {
175 echo "test"
176 ###### The following "}" with a leading space at the start of the line avoids the error ######
177 }
178 EOF
179 }
180
181Locating and Parsing Recipes
182============================
183
184During the configuration phase, BitBake will have set
185:term:`BBFILES`. BitBake now uses it to construct a
186list of recipes to parse, along with any append files (``.bbappend``) to
Patrick Williams213cb262021-08-07 19:21:33 -0500187apply. :term:`BBFILES` is a space-separated list of available files and
Andrew Geisslerc926e172021-05-07 16:11:35 -0500188supports wildcards. An example would be::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500189
190 BBFILES = "/path/to/bbfiles/*.bb /path/to/appends/*.bbappend"
191
192BitBake parses each
Patrick Williams213cb262021-08-07 19:21:33 -0500193recipe and append file located with :term:`BBFILES` and stores the values of
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500194various variables into the datastore.
195
196.. note::
197
198 Append files are applied in the order they are encountered in BBFILES.
199
200For each file, a fresh copy of the base configuration is made, then the
201recipe is parsed line by line. Any inherit statements cause BitBake to
202find and then parse class files (``.bbclass``) using
203:term:`BBPATH` as the search path. Finally, BitBake
Patrick Williams213cb262021-08-07 19:21:33 -0500204parses in order any append files found in :term:`BBFILES`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500205
206One common convention is to use the recipe filename to define pieces of
207metadata. For example, in ``bitbake.conf`` the recipe name and version
208are used to set the variables :term:`PN` and
Andrew Geisslerc926e172021-05-07 16:11:35 -0500209:term:`PV`::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500210
Andrew Geissler5199d832021-09-24 16:47:35 -0500211 PN = "${@bb.parse.vars_from_file(d.getVar('FILE', False),d)[0] or 'defaultpkgname'}"
212 PV = "${@bb.parse.vars_from_file(d.getVar('FILE', False),d)[1] or '1.0'}"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500213
214In this example, a recipe called "something_1.2.3.bb" would set
Patrick Williams213cb262021-08-07 19:21:33 -0500215:term:`PN` to "something" and :term:`PV` to "1.2.3".
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500216
217By the time parsing is complete for a recipe, BitBake has a list of
218tasks that the recipe defines and a set of data consisting of keys and
219values as well as dependency information about the tasks.
220
221BitBake does not need all of this information. It only needs a small
222subset of the information to make decisions about the recipe.
223Consequently, BitBake caches the values in which it is interested and
224does not store the rest of the information. Experience has shown it is
225faster to re-parse the metadata than to try and write it out to the disk
226and then reload it.
227
228Where possible, subsequent BitBake commands reuse this cache of recipe
229information. The validity of this cache is determined by first computing
230a checksum of the base configuration data (see
231:term:`BB_HASHCONFIG_WHITELIST`) and
232then checking if the checksum matches. If that checksum matches what is
233in the cache and the recipe and class files have not changed, BitBake is
234able to use the cache. BitBake then reloads the cached information about
235the recipe instead of reparsing it from scratch.
236
237Recipe file collections exist to allow the user to have multiple
238repositories of ``.bb`` files that contain the same exact package. For
239example, one could easily use them to make one's own local copy of an
240upstream repository, but with custom modifications that one does not
Andrew Geisslerc926e172021-05-07 16:11:35 -0500241want upstream. Here is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500242
243 BBFILES = "/stuff/openembedded/*/*.bb /stuff/openembedded.modified/*/*.bb"
244 BBFILE_COLLECTIONS = "upstream local"
245 BBFILE_PATTERN_upstream = "^/stuff/openembedded/"
246 BBFILE_PATTERN_local = "^/stuff/openembedded.modified/"
Andrew Geissler95ac1b82021-03-31 14:34:31 -0500247 BBFILE_PRIORITY_upstream = "5"
248 BBFILE_PRIORITY_local = "10"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500249
250.. note::
251
252 The layers mechanism is now the preferred method of collecting code.
253 While the collections code remains, its main use is to set layer
254 priorities and to deal with overlap (conflicts) between layers.
255
256.. _bb-bitbake-providers:
257
258Providers
259=========
260
261Assuming BitBake has been instructed to execute a target and that all
262the recipe files have been parsed, BitBake starts to figure out how to
Patrick Williams213cb262021-08-07 19:21:33 -0500263build the target. BitBake looks through the :term:`PROVIDES` list for each
264of the recipes. A :term:`PROVIDES` list is the list of names by which the
265recipe can be known. Each recipe's :term:`PROVIDES` list is created
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500266implicitly through the recipe's :term:`PN` variable and
267explicitly through the recipe's :term:`PROVIDES`
268variable, which is optional.
269
Patrick Williams213cb262021-08-07 19:21:33 -0500270When a recipe uses :term:`PROVIDES`, that recipe's functionality can be
271found under an alternative name or names other than the implicit :term:`PN`
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500272name. As an example, suppose a recipe named ``keyboard_1.0.bb``
Andrew Geisslerc926e172021-05-07 16:11:35 -0500273contained the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500274
275 PROVIDES += "fullkeyboard"
276
Patrick Williams213cb262021-08-07 19:21:33 -0500277The :term:`PROVIDES`
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500278list for this recipe becomes "keyboard", which is implicit, and
279"fullkeyboard", which is explicit. Consequently, the functionality found
280in ``keyboard_1.0.bb`` can be found under two different names.
281
282.. _bb-bitbake-preferences:
283
284Preferences
285===========
286
Patrick Williams213cb262021-08-07 19:21:33 -0500287The :term:`PROVIDES` list is only part of the solution for figuring out a
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500288target's recipes. Because targets might have multiple providers, BitBake
289needs to prioritize providers by determining provider preferences.
290
291A common example in which a target has multiple providers is
Patrick Williams213cb262021-08-07 19:21:33 -0500292"virtual/kernel", which is on the :term:`PROVIDES` list for each kernel
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500293recipe. Each machine often selects the best kernel provider by using a
Andrew Geisslerc926e172021-05-07 16:11:35 -0500294line similar to the following in the machine configuration file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500295
296 PREFERRED_PROVIDER_virtual/kernel = "linux-yocto"
297
298The default :term:`PREFERRED_PROVIDER` is the provider
299with the same name as the target. BitBake iterates through each target
300it needs to build and resolves them and their dependencies using this
301process.
302
303Understanding how providers are chosen is made complicated by the fact
304that multiple versions might exist for a given provider. BitBake
305defaults to the highest version of a provider. Version comparisons are
306made using the same method as Debian. You can use the
307:term:`PREFERRED_VERSION` variable to
308specify a particular version. You can influence the order by using the
309:term:`DEFAULT_PREFERENCE` variable.
310
311By default, files have a preference of "0". Setting
Patrick Williams213cb262021-08-07 19:21:33 -0500312:term:`DEFAULT_PREFERENCE` to "-1" makes the recipe unlikely to be used
313unless it is explicitly referenced. Setting :term:`DEFAULT_PREFERENCE` to
314"1" makes it likely the recipe is used. :term:`PREFERRED_VERSION` overrides
315any :term:`DEFAULT_PREFERENCE` setting. :term:`DEFAULT_PREFERENCE` is often used
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500316to mark newer and more experimental recipe versions until they have
317undergone sufficient testing to be considered stable.
318
Andrew Geisslerf0343792020-11-18 10:42:21 -0600319When there are multiple "versions" of a given recipe, BitBake defaults
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500320to selecting the most recent version, unless otherwise specified. If the
321recipe in question has a
322:term:`DEFAULT_PREFERENCE` set lower than
323the other recipes (default is 0), then it will not be selected. This
324allows the person or persons maintaining the repository of recipe files
325to specify their preference for the default selected version.
326Additionally, the user can specify their preferred version.
327
328If the first recipe is named ``a_1.1.bb``, then the
Andrew Geisslerf0343792020-11-18 10:42:21 -0600329:term:`PN` variable will be set to "a", and the
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500330:term:`PV` variable will be set to 1.1.
331
332Thus, if a recipe named ``a_1.2.bb`` exists, BitBake will choose 1.2 by
333default. However, if you define the following variable in a ``.conf``
Andrew Geisslerc926e172021-05-07 16:11:35 -0500334file that BitBake parses, you can change that preference::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500335
336 PREFERRED_VERSION_a = "1.1"
337
338.. note::
339
340 It is common for a recipe to provide two versions -- a stable,
341 numbered (and preferred) version, and a version that is automatically
342 checked out from a source code repository that is considered more
343 "bleeding edge" but can be selected only explicitly.
344
345 For example, in the OpenEmbedded codebase, there is a standard,
346 versioned recipe file for BusyBox, ``busybox_1.22.1.bb``, but there
347 is also a Git-based version, ``busybox_git.bb``, which explicitly
348 contains the line ::
349
350 DEFAULT_PREFERENCE = "-1"
351
352 to ensure that the
353 numbered, stable version is always preferred unless the developer
354 selects otherwise.
355
356.. _bb-bitbake-dependencies:
357
358Dependencies
359============
360
361Each target BitBake builds consists of multiple tasks such as ``fetch``,
362``unpack``, ``patch``, ``configure``, and ``compile``. For best
363performance on multi-core systems, BitBake considers each task as an
364independent entity with its own set of dependencies.
365
366Dependencies are defined through several variables. You can find
367information about variables BitBake uses in the
368:doc:`bitbake-user-manual-ref-variables` near the end of this manual. At a
369basic level, it is sufficient to know that BitBake uses the
370:term:`DEPENDS` and
371:term:`RDEPENDS` variables when calculating
372dependencies.
373
374For more information on how BitBake handles dependencies, see the
375:ref:`bitbake-user-manual/bitbake-user-manual-metadata:Dependencies`
376section.
377
378.. _ref-bitbake-tasklist:
379
380The Task List
381=============
382
383Based on the generated list of providers and the dependency information,
384BitBake can now calculate exactly what tasks it needs to run and in what
385order it needs to run them. The
386:ref:`bitbake-user-manual/bitbake-user-manual-execution:executing tasks`
387section has more information on how BitBake chooses which task to
388execute next.
389
390The build now starts with BitBake forking off threads up to the limit
391set in the :term:`BB_NUMBER_THREADS`
392variable. BitBake continues to fork threads as long as there are tasks
393ready to run, those tasks have all their dependencies met, and the
394thread threshold has not been exceeded.
395
396It is worth noting that you can greatly speed up the build time by
Patrick Williams213cb262021-08-07 19:21:33 -0500397properly setting the :term:`BB_NUMBER_THREADS` variable.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500398
399As each task completes, a timestamp is written to the directory
400specified by the :term:`STAMP` variable. On subsequent
401runs, BitBake looks in the build directory within ``tmp/stamps`` and
402does not rerun tasks that are already completed unless a timestamp is
403found to be invalid. Currently, invalid timestamps are only considered
404on a per recipe file basis. So, for example, if the configure stamp has
405a timestamp greater than the compile timestamp for a given target, then
406the compile task would rerun. Running the compile task again, however,
407has no effect on other providers that depend on that target.
408
409The exact format of the stamps is partly configurable. In modern
410versions of BitBake, a hash is appended to the stamp so that if the
411configuration changes, the stamp becomes invalid and the task is
412automatically rerun. This hash, or signature used, is governed by the
413signature policy that is configured (see the
414:ref:`bitbake-user-manual/bitbake-user-manual-execution:checksums (signatures)`
415section for information). It is also
416possible to append extra metadata to the stamp using the
417``[stamp-extra-info]`` task flag. For example, OpenEmbedded uses this
418flag to make some tasks machine-specific.
419
420.. note::
421
422 Some tasks are marked as "nostamp" tasks. No timestamp file is
423 created when these tasks are run. Consequently, "nostamp" tasks are
424 always rerun.
425
426For more information on tasks, see the
427:ref:`bitbake-user-manual/bitbake-user-manual-metadata:tasks` section.
428
429Executing Tasks
430===============
431
432Tasks can be either a shell task or a Python task. For shell tasks,
433BitBake writes a shell script to
434``${``\ :term:`T`\ ``}/run.do_taskname.pid`` and then
435executes the script. The generated shell script contains all the
436exported variables, and the shell functions with all variables expanded.
437Output from the shell script goes to the file
438``${T}/log.do_taskname.pid``. Looking at the expanded shell functions in
439the run file and the output in the log files is a useful debugging
440technique.
441
442For Python tasks, BitBake executes the task internally and logs
443information to the controlling terminal. Future versions of BitBake will
444write the functions to files similar to the way shell tasks are handled.
445Logging will be handled in a way similar to shell tasks as well.
446
447The order in which BitBake runs the tasks is controlled by its task
448scheduler. It is possible to configure the scheduler and define custom
449implementations for specific use cases. For more information, see these
450variables that control the behavior:
451
452- :term:`BB_SCHEDULER`
453
454- :term:`BB_SCHEDULERS`
455
456It is possible to have functions run before and after a task's main
457function. This is done using the ``[prefuncs]`` and ``[postfuncs]``
458flags of the task that lists the functions to run.
459
460.. _checksums:
461
462Checksums (Signatures)
463======================
464
465A checksum is a unique signature of a task's inputs. The signature of a
466task can be used to determine if a task needs to be run. Because it is a
467change in a task's inputs that triggers running the task, BitBake needs
468to detect all the inputs to a given task. For shell tasks, this turns
469out to be fairly easy because BitBake generates a "run" shell script for
470each task and it is possible to create a checksum that gives you a good
471idea of when the task's data changes.
472
473To complicate the problem, some things should not be included in the
474checksum. First, there is the actual specific build path of a given task
475- the working directory. It does not matter if the working directory
476changes because it should not affect the output for target packages. The
477simplistic approach for excluding the working directory is to set it to
478some fixed value and create the checksum for the "run" script. BitBake
479goes one step better and uses the
480:term:`BB_HASHBASE_WHITELIST` variable
481to define a list of variables that should never be included when
482generating the signatures.
483
484Another problem results from the "run" scripts containing functions that
485might or might not get called. The incremental build solution contains
486code that figures out dependencies between shell functions. This code is
487used to prune the "run" scripts down to the minimum set, thereby
488alleviating this problem and making the "run" scripts much more readable
489as a bonus.
490
491So far we have solutions for shell scripts. What about Python tasks? The
492same approach applies even though these tasks are more difficult. The
493process needs to figure out what variables a Python function accesses
494and what functions it calls. Again, the incremental build solution
495contains code that first figures out the variable and function
496dependencies, and then creates a checksum for the data used as the input
497to the task.
498
499Like the working directory case, situations exist where dependencies
500should be ignored. For these cases, you can instruct the build process
Andrew Geisslerc926e172021-05-07 16:11:35 -0500501to ignore a dependency by using a line like the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500502
503 PACKAGE_ARCHS[vardepsexclude] = "MACHINE"
504
505This example ensures that the
506``PACKAGE_ARCHS`` variable does not depend on the value of ``MACHINE``,
507even if it does reference it.
508
509Equally, there are cases where we need to add dependencies BitBake is
510not able to find. You can accomplish this by using a line like the
Andrew Geisslerc926e172021-05-07 16:11:35 -0500511following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500512
513 PACKAGE_ARCHS[vardeps] = "MACHINE"
514
515This example explicitly
516adds the ``MACHINE`` variable as a dependency for ``PACKAGE_ARCHS``.
517
518Consider a case with in-line Python, for example, where BitBake is not
519able to figure out dependencies. When running in debug mode (i.e. using
520``-DDD``), BitBake produces output when it discovers something for which
521it cannot figure out dependencies.
522
523Thus far, this section has limited discussion to the direct inputs into
524a task. Information based on direct inputs is referred to as the
525"basehash" in the code. However, there is still the question of a task's
526indirect inputs - the things that were already built and present in the
527build directory. The checksum (or signature) for a particular task needs
528to add the hashes of all the tasks on which the particular task depends.
529Choosing which dependencies to add is a policy decision. However, the
530effect is to generate a master checksum that combines the basehash and
531the hashes of the task's dependencies.
532
533At the code level, there are a variety of ways both the basehash and the
534dependent task hashes can be influenced. Within the BitBake
535configuration file, we can give BitBake some extra information to help
536it construct the basehash. The following statement effectively results
537in a list of global variable dependency excludes - variables never
538included in any checksum. This example uses variables from OpenEmbedded
Andrew Geisslerc926e172021-05-07 16:11:35 -0500539to help illustrate the concept::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500540
541 BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \
542 SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL \
543 USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \
544 PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \
545 CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX"
546
547The previous example excludes the work directory, which is part of
548``TMPDIR``.
549
550The rules for deciding which hashes of dependent tasks to include
551through dependency chains are more complex and are generally
552accomplished with a Python function. The code in
553``meta/lib/oe/sstatesig.py`` shows two examples of this and also
554illustrates how you can insert your own policy into the system if so
555desired. This file defines the two basic signature generators
556OpenEmbedded-Core uses: "OEBasic" and "OEBasicHash". By default, there
557is a dummy "noop" signature handler enabled in BitBake. This means that
558behavior is unchanged from previous versions. ``OE-Core`` uses the
559"OEBasicHash" signature handler by default through this setting in the
Andrew Geisslerc926e172021-05-07 16:11:35 -0500560``bitbake.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500561
562 BB_SIGNATURE_HANDLER ?= "OEBasicHash"
563
Patrick Williams213cb262021-08-07 19:21:33 -0500564The "OEBasicHash" :term:`BB_SIGNATURE_HANDLER` is the same as the "OEBasic"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500565version but adds the task hash to the stamp files. This results in any
566metadata change that changes the task hash, automatically causing the
567task to be run again. This removes the need to bump
568:term:`PR` values, and changes to metadata automatically
569ripple across the build.
570
571It is also worth noting that the end result of these signature
572generators is to make some dependency and hash information available to
573the build. This information includes:
574
575- ``BB_BASEHASH_task-``\ *taskname*: The base hashes for each task in the
576 recipe.
577
578- ``BB_BASEHASH_``\ *filename:taskname*: The base hashes for each
579 dependent task.
580
Patrick Williams213cb262021-08-07 19:21:33 -0500581- :term:`BB_TASKHASH`: The hash of the currently running task.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500582
583It is worth noting that BitBake's "-S" option lets you debug BitBake's
584processing of signatures. The options passed to -S allow different
585debugging modes to be used, either using BitBake's own debug functions
586or possibly those defined in the metadata/signature handler itself. The
587simplest parameter to pass is "none", which causes a set of signature
588information to be written out into ``STAMPS_DIR`` corresponding to the
589targets specified. The other currently available parameter is
590"printdiff", which causes BitBake to try to establish the closest
591signature match it can (e.g. in the sstate cache) and then run
592``bitbake-diffsigs`` over the matches to determine the stamps and delta
593where these two stamp trees diverge.
594
595.. note::
596
597 It is likely that future versions of BitBake will provide other
598 signature handlers triggered through additional "-S" parameters.
599
600You can find more information on checksum metadata in the
601:ref:`bitbake-user-manual/bitbake-user-manual-metadata:task checksums and setscene`
602section.
603
604Setscene
605========
606
607The setscene process enables BitBake to handle "pre-built" artifacts.
608The ability to handle and reuse these artifacts allows BitBake the
609luxury of not having to build something from scratch every time.
610Instead, BitBake can use, when possible, existing build artifacts.
611
612BitBake needs to have reliable data indicating whether or not an
613artifact is compatible. Signatures, described in the previous section,
614provide an ideal way of representing whether an artifact is compatible.
615If a signature is the same, an object can be reused.
616
617If an object can be reused, the problem then becomes how to replace a
618given task or set of tasks with the pre-built artifact. BitBake solves
619the problem with the "setscene" process.
620
621When BitBake is asked to build a given target, before building anything,
622it first asks whether cached information is available for any of the
623targets it's building, or any of the intermediate targets. If cached
624information is available, BitBake uses this information instead of
625running the main tasks.
626
627BitBake first calls the function defined by the
628:term:`BB_HASHCHECK_FUNCTION` variable
629with a list of tasks and corresponding hashes it wants to build. This
630function is designed to be fast and returns a list of the tasks for
631which it believes in can obtain artifacts.
632
633Next, for each of the tasks that were returned as possibilities, BitBake
634executes a setscene version of the task that the possible artifact
635covers. Setscene versions of a task have the string "_setscene" appended
636to the task name. So, for example, the task with the name ``xxx`` has a
637setscene task named ``xxx_setscene``. The setscene version of the task
638executes and provides the necessary artifacts returning either success
639or failure.
640
641As previously mentioned, an artifact can cover more than one task. For
642example, it is pointless to obtain a compiler if you already have the
643compiled binary. To handle this, BitBake calls the
644:term:`BB_SETSCENE_DEPVALID` function for
645each successful setscene task to know whether or not it needs to obtain
646the dependencies of that task.
647
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500648You can find more information on setscene metadata in the
649:ref:`bitbake-user-manual/bitbake-user-manual-metadata:task checksums and setscene`
650section.
651
652Logging
653=======
654
655In addition to the standard command line option to control how verbose
656builds are when execute, bitbake also supports user defined
657configuration of the `Python
658logging <https://docs.python.org/3/library/logging.html>`__ facilities
659through the :term:`BB_LOGCONFIG` variable. This
660variable defines a json or yaml `logging
661configuration <https://docs.python.org/3/library/logging.config.html>`__
662that will be intelligently merged into the default configuration. The
663logging configuration is merged using the following rules:
664
665- The user defined configuration will completely replace the default
666 configuration if top level key ``bitbake_merge`` is set to the value
667 ``False``. In this case, all other rules are ignored.
668
669- The user configuration must have a top level ``version`` which must
670 match the value of the default configuration.
671
672- Any keys defined in the ``handlers``, ``formatters``, or ``filters``,
673 will be merged into the same section in the default configuration,
674 with the user specified keys taking replacing a default one if there
675 is a conflict. In practice, this means that if both the default
676 configuration and user configuration specify a handler named
677 ``myhandler``, the user defined one will replace the default. To
678 prevent the user from inadvertently replacing a default handler,
679 formatter, or filter, all of the default ones are named with a prefix
680 of "``BitBake.``"
681
682- If a logger is defined by the user with the key ``bitbake_merge`` set
683 to ``False``, that logger will be completely replaced by user
684 configuration. In this case, no other rules will apply to that
685 logger.
686
687- All user defined ``filter`` and ``handlers`` properties for a given
688 logger will be merged with corresponding properties from the default
689 logger. For example, if the user configuration adds a filter called
690 ``myFilter`` to the ``BitBake.SigGen``, and the default configuration
691 adds a filter called ``BitBake.defaultFilter``, both filters will be
692 applied to the logger
693
694As an example, consider the following user logging configuration file
695which logs all Hash Equivalence related messages of VERBOSE or higher to
696a file called ``hashequiv.log`` ::
697
698 {
699 "version": 1,
700 "handlers": {
701 "autobuilderlog": {
702 "class": "logging.FileHandler",
703 "formatter": "logfileFormatter",
704 "level": "DEBUG",
705 "filename": "hashequiv.log",
706 "mode": "w"
707 }
708 },
709 "formatters": {
710 "logfileFormatter": {
711 "format": "%(name)s: %(levelname)s: %(message)s"
712 }
713 },
714 "loggers": {
715 "BitBake.SigGen.HashEquiv": {
716 "level": "VERBOSE",
717 "handlers": ["autobuilderlog"]
718 },
719 "BitBake.RunQueue.HashEquiv": {
720 "level": "VERBOSE",
721 "handlers": ["autobuilderlog"]
722 }
723 }
724 }