blob: d74e768f692dd7b19475e9ecb581f2e652f17dd7 [file] [log] [blame]
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001.. SPDX-License-Identifier: CC-BY-2.5
2
3=========
4Execution
5=========
6
7|
8
9The primary purpose for running BitBake is to produce some kind of
10output such as a single installable package, a kernel, a software
11development kit, or even a full, board-specific bootable Linux image,
12complete with bootloader, kernel, and root filesystem. Of course, you
13can execute the ``bitbake`` command with options that cause it to
14execute single tasks, compile single recipe files, capture or clear
15data, or simply return information about the execution environment.
16
17This chapter describes BitBake's execution process from start to finish
18when you use it to create an image. The execution process is launched
19using the following command form: ::
20
21 $ bitbake target
22
23For information on
24the BitBake command and its options, see ":ref:`The BitBake Command
25<bitbake-user-manual-command>`" section.
26
27.. note::
28
29 Prior to executing BitBake, you should take advantage of available
30 parallel thread execution on your build host by setting the
31 :term:`BB_NUMBER_THREADS` variable in
32 your project's ``local.conf`` configuration file.
33
34 A common method to determine this value for your build host is to run
35 the following: ::
36
37 $ grep processor /proc/cpuinfo
38
39 This command returns
40 the number of processors, which takes into account hyper-threading.
41 Thus, a quad-core build host with hyper-threading most likely shows
42 eight processors, which is the value you would then assign to
43 ``BB_NUMBER_THREADS``.
44
45 A possibly simpler solution is that some Linux distributions (e.g.
46 Debian and Ubuntu) provide the ``ncpus`` command.
47
48Parsing the Base Configuration Metadata
49=======================================
50
51The first thing BitBake does is parse base configuration metadata. Base
52configuration metadata consists of your project's ``bblayers.conf`` file
53to determine what layers BitBake needs to recognize, all necessary
54``layer.conf`` files (one from each layer), and ``bitbake.conf``. The
55data itself is of various types:
56
57- **Recipes:** Details about particular pieces of software.
58
59- **Class Data:** An abstraction of common build information (e.g. how to
60 build a Linux kernel).
61
62- **Configuration Data:** Machine-specific settings, policy decisions,
63 and so forth. Configuration data acts as the glue to bind everything
64 together.
65
66The ``layer.conf`` files are used to construct key variables such as
67:term:`BBPATH` and :term:`BBFILES`.
68``BBPATH`` is used to search for configuration and class files under the
69``conf`` and ``classes`` directories, respectively. ``BBFILES`` is used
70to locate both recipe and recipe append files (``.bb`` and
71``.bbappend``). If there is no ``bblayers.conf`` file, it is assumed the
72user has set the ``BBPATH`` and ``BBFILES`` directly in the environment.
73
74Next, the ``bitbake.conf`` file is located using the ``BBPATH`` variable
75that was just constructed. The ``bitbake.conf`` file may also include
76other configuration files using the ``include`` or ``require``
77directives.
78
79Prior to parsing configuration files, BitBake looks at certain
80variables, including:
81
82- :term:`BB_ENV_WHITELIST`
83- :term:`BB_ENV_EXTRAWHITE`
84- :term:`BB_PRESERVE_ENV`
85- :term:`BB_ORIGENV`
86- :term:`BITBAKE_UI`
87
88The first four variables in this list relate to how BitBake treats shell
89environment variables during task execution. By default, BitBake cleans
90the environment variables and provides tight control over the shell
91execution environment. However, through the use of these first four
92variables, you can apply your control regarding the environment
93variables allowed to be used by BitBake in the shell during execution of
94tasks. See the
95":ref:`bitbake-user-manual/bitbake-user-manual-metadata:Passing Information Into the Build Task Environment`"
96section and the information about these variables in the variable
97glossary for more information on how they work and on how to use them.
98
99The base configuration metadata is global and therefore affects all
100recipes and tasks that are executed.
101
102BitBake first searches the current working directory for an optional
103``conf/bblayers.conf`` configuration file. This file is expected to
104contain a :term:`BBLAYERS` variable that is a
105space-delimited list of 'layer' directories. Recall that if BitBake
106cannot find a ``bblayers.conf`` file, then it is assumed the user has
107set the ``BBPATH`` and ``BBFILES`` variables directly in the
108environment.
109
110For each directory (layer) in this list, a ``conf/layer.conf`` file is
111located and parsed with the :term:`LAYERDIR` variable
112being set to the directory where the layer was found. The idea is these
113files automatically set up :term:`BBPATH` and other
114variables correctly for a given build directory.
115
116BitBake then expects to find the ``conf/bitbake.conf`` file somewhere in
117the user-specified ``BBPATH``. That configuration file generally has
118include directives to pull in any other metadata such as files specific
119to the architecture, the machine, the local environment, and so forth.
120
121Only variable definitions and include directives are allowed in BitBake
122``.conf`` files. Some variables directly influence BitBake's behavior.
123These variables might have been set from the environment depending on
124the environment variables previously mentioned or set in the
125configuration files. The ":ref:`bitbake-user-manual/bitbake-user-manual-ref-variables:Variables Glossary`"
126chapter presents a full list of
127variables.
128
129After parsing configuration files, BitBake uses its rudimentary
130inheritance mechanism, which is through class files, to inherit some
131standard classes. BitBake parses a class when the inherit directive
132responsible for getting that class is encountered.
133
134The ``base.bbclass`` file is always included. Other classes that are
135specified in the configuration using the
136:term:`INHERIT` variable are also included. BitBake
137searches for class files in a ``classes`` subdirectory under the paths
138in ``BBPATH`` in the same way as configuration files.
139
140A good way to get an idea of the configuration files and the class files
141used in your execution environment is to run the following BitBake
142command: ::
143
144 $ bitbake -e > mybb.log
145
146Examining the top of the ``mybb.log``
147shows you the many configuration files and class files used in your
148execution environment.
149
150.. note::
151
152 You need to be aware of how BitBake parses curly braces. If a recipe
153 uses a closing curly brace within the function and the character has
154 no leading spaces, BitBake produces a parsing error. If you use a
155 pair of curly braces in a shell function, the closing curly brace
156 must not be located at the start of the line without leading spaces.
157
158 Here is an example that causes BitBake to produce a parsing error: ::
159
160 fakeroot create_shar() {
161 cat << "EOF" > ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh
162 usage()
163 {
164 echo "test"
165 ###### The following "}" at the start of the line causes a parsing error ######
166 }
167 EOF
168 }
169
170 Writing the recipe this way avoids the error:
171 fakeroot create_shar() {
172 cat << "EOF" > ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh
173 usage()
174 {
175 echo "test"
176 ###### The following "}" with a leading space at the start of the line avoids the error ######
177 }
178 EOF
179 }
180
181Locating and Parsing Recipes
182============================
183
184During the configuration phase, BitBake will have set
185:term:`BBFILES`. BitBake now uses it to construct a
186list of recipes to parse, along with any append files (``.bbappend``) to
187apply. ``BBFILES`` is a space-separated list of available files and
188supports wildcards. An example would be: ::
189
190 BBFILES = "/path/to/bbfiles/*.bb /path/to/appends/*.bbappend"
191
192BitBake parses each
193recipe and append file located with ``BBFILES`` and stores the values of
194various variables into the datastore.
195
196.. note::
197
198 Append files are applied in the order they are encountered in BBFILES.
199
200For each file, a fresh copy of the base configuration is made, then the
201recipe is parsed line by line. Any inherit statements cause BitBake to
202find and then parse class files (``.bbclass``) using
203:term:`BBPATH` as the search path. Finally, BitBake
204parses in order any append files found in ``BBFILES``.
205
206One common convention is to use the recipe filename to define pieces of
207metadata. For example, in ``bitbake.conf`` the recipe name and version
208are used to set the variables :term:`PN` and
209:term:`PV`: ::
210
211 PN = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0] or 'defaultpkgname'}"
212 PV = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[1] or '1.0'}"
213
214In this example, a recipe called "something_1.2.3.bb" would set
215``PN`` to "something" and ``PV`` to "1.2.3".
216
217By the time parsing is complete for a recipe, BitBake has a list of
218tasks that the recipe defines and a set of data consisting of keys and
219values as well as dependency information about the tasks.
220
221BitBake does not need all of this information. It only needs a small
222subset of the information to make decisions about the recipe.
223Consequently, BitBake caches the values in which it is interested and
224does not store the rest of the information. Experience has shown it is
225faster to re-parse the metadata than to try and write it out to the disk
226and then reload it.
227
228Where possible, subsequent BitBake commands reuse this cache of recipe
229information. The validity of this cache is determined by first computing
230a checksum of the base configuration data (see
231:term:`BB_HASHCONFIG_WHITELIST`) and
232then checking if the checksum matches. If that checksum matches what is
233in the cache and the recipe and class files have not changed, BitBake is
234able to use the cache. BitBake then reloads the cached information about
235the recipe instead of reparsing it from scratch.
236
237Recipe file collections exist to allow the user to have multiple
238repositories of ``.bb`` files that contain the same exact package. For
239example, one could easily use them to make one's own local copy of an
240upstream repository, but with custom modifications that one does not
241want upstream. Here is an example: ::
242
243 BBFILES = "/stuff/openembedded/*/*.bb /stuff/openembedded.modified/*/*.bb"
244 BBFILE_COLLECTIONS = "upstream local"
245 BBFILE_PATTERN_upstream = "^/stuff/openembedded/"
246 BBFILE_PATTERN_local = "^/stuff/openembedded.modified/"
247 BBFILE_PRIORITY_upstream = "5" BBFILE_PRIORITY_local = "10"
248
249.. note::
250
251 The layers mechanism is now the preferred method of collecting code.
252 While the collections code remains, its main use is to set layer
253 priorities and to deal with overlap (conflicts) between layers.
254
255.. _bb-bitbake-providers:
256
257Providers
258=========
259
260Assuming BitBake has been instructed to execute a target and that all
261the recipe files have been parsed, BitBake starts to figure out how to
262build the target. BitBake looks through the ``PROVIDES`` list for each
263of the recipes. A ``PROVIDES`` list is the list of names by which the
264recipe can be known. Each recipe's ``PROVIDES`` list is created
265implicitly through the recipe's :term:`PN` variable and
266explicitly through the recipe's :term:`PROVIDES`
267variable, which is optional.
268
269When a recipe uses ``PROVIDES``, that recipe's functionality can be
270found under an alternative name or names other than the implicit ``PN``
271name. As an example, suppose a recipe named ``keyboard_1.0.bb``
272contained the following: ::
273
274 PROVIDES += "fullkeyboard"
275
276The ``PROVIDES``
277list for this recipe becomes "keyboard", which is implicit, and
278"fullkeyboard", which is explicit. Consequently, the functionality found
279in ``keyboard_1.0.bb`` can be found under two different names.
280
281.. _bb-bitbake-preferences:
282
283Preferences
284===========
285
286The ``PROVIDES`` list is only part of the solution for figuring out a
287target's recipes. Because targets might have multiple providers, BitBake
288needs to prioritize providers by determining provider preferences.
289
290A common example in which a target has multiple providers is
291"virtual/kernel", which is on the ``PROVIDES`` list for each kernel
292recipe. Each machine often selects the best kernel provider by using a
293line similar to the following in the machine configuration file: ::
294
295 PREFERRED_PROVIDER_virtual/kernel = "linux-yocto"
296
297The default :term:`PREFERRED_PROVIDER` is the provider
298with the same name as the target. BitBake iterates through each target
299it needs to build and resolves them and their dependencies using this
300process.
301
302Understanding how providers are chosen is made complicated by the fact
303that multiple versions might exist for a given provider. BitBake
304defaults to the highest version of a provider. Version comparisons are
305made using the same method as Debian. You can use the
306:term:`PREFERRED_VERSION` variable to
307specify a particular version. You can influence the order by using the
308:term:`DEFAULT_PREFERENCE` variable.
309
310By default, files have a preference of "0". Setting
311``DEFAULT_PREFERENCE`` to "-1" makes the recipe unlikely to be used
312unless it is explicitly referenced. Setting ``DEFAULT_PREFERENCE`` to
313"1" makes it likely the recipe is used. ``PREFERRED_VERSION`` overrides
314any ``DEFAULT_PREFERENCE`` setting. ``DEFAULT_PREFERENCE`` is often used
315to mark newer and more experimental recipe versions until they have
316undergone sufficient testing to be considered stable.
317
Andrew Geisslerf0343792020-11-18 10:42:21 -0600318When there are multiple "versions" of a given recipe, BitBake defaults
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500319to selecting the most recent version, unless otherwise specified. If the
320recipe in question has a
321:term:`DEFAULT_PREFERENCE` set lower than
322the other recipes (default is 0), then it will not be selected. This
323allows the person or persons maintaining the repository of recipe files
324to specify their preference for the default selected version.
325Additionally, the user can specify their preferred version.
326
327If the first recipe is named ``a_1.1.bb``, then the
Andrew Geisslerf0343792020-11-18 10:42:21 -0600328:term:`PN` variable will be set to "a", and the
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500329:term:`PV` variable will be set to 1.1.
330
331Thus, if a recipe named ``a_1.2.bb`` exists, BitBake will choose 1.2 by
332default. However, if you define the following variable in a ``.conf``
333file that BitBake parses, you can change that preference: ::
334
335 PREFERRED_VERSION_a = "1.1"
336
337.. note::
338
339 It is common for a recipe to provide two versions -- a stable,
340 numbered (and preferred) version, and a version that is automatically
341 checked out from a source code repository that is considered more
342 "bleeding edge" but can be selected only explicitly.
343
344 For example, in the OpenEmbedded codebase, there is a standard,
345 versioned recipe file for BusyBox, ``busybox_1.22.1.bb``, but there
346 is also a Git-based version, ``busybox_git.bb``, which explicitly
347 contains the line ::
348
349 DEFAULT_PREFERENCE = "-1"
350
351 to ensure that the
352 numbered, stable version is always preferred unless the developer
353 selects otherwise.
354
355.. _bb-bitbake-dependencies:
356
357Dependencies
358============
359
360Each target BitBake builds consists of multiple tasks such as ``fetch``,
361``unpack``, ``patch``, ``configure``, and ``compile``. For best
362performance on multi-core systems, BitBake considers each task as an
363independent entity with its own set of dependencies.
364
365Dependencies are defined through several variables. You can find
366information about variables BitBake uses in the
367:doc:`bitbake-user-manual-ref-variables` near the end of this manual. At a
368basic level, it is sufficient to know that BitBake uses the
369:term:`DEPENDS` and
370:term:`RDEPENDS` variables when calculating
371dependencies.
372
373For more information on how BitBake handles dependencies, see the
374:ref:`bitbake-user-manual/bitbake-user-manual-metadata:Dependencies`
375section.
376
377.. _ref-bitbake-tasklist:
378
379The Task List
380=============
381
382Based on the generated list of providers and the dependency information,
383BitBake can now calculate exactly what tasks it needs to run and in what
384order it needs to run them. The
385:ref:`bitbake-user-manual/bitbake-user-manual-execution:executing tasks`
386section has more information on how BitBake chooses which task to
387execute next.
388
389The build now starts with BitBake forking off threads up to the limit
390set in the :term:`BB_NUMBER_THREADS`
391variable. BitBake continues to fork threads as long as there are tasks
392ready to run, those tasks have all their dependencies met, and the
393thread threshold has not been exceeded.
394
395It is worth noting that you can greatly speed up the build time by
396properly setting the ``BB_NUMBER_THREADS`` variable.
397
398As each task completes, a timestamp is written to the directory
399specified by the :term:`STAMP` variable. On subsequent
400runs, BitBake looks in the build directory within ``tmp/stamps`` and
401does not rerun tasks that are already completed unless a timestamp is
402found to be invalid. Currently, invalid timestamps are only considered
403on a per recipe file basis. So, for example, if the configure stamp has
404a timestamp greater than the compile timestamp for a given target, then
405the compile task would rerun. Running the compile task again, however,
406has no effect on other providers that depend on that target.
407
408The exact format of the stamps is partly configurable. In modern
409versions of BitBake, a hash is appended to the stamp so that if the
410configuration changes, the stamp becomes invalid and the task is
411automatically rerun. This hash, or signature used, is governed by the
412signature policy that is configured (see the
413:ref:`bitbake-user-manual/bitbake-user-manual-execution:checksums (signatures)`
414section for information). It is also
415possible to append extra metadata to the stamp using the
416``[stamp-extra-info]`` task flag. For example, OpenEmbedded uses this
417flag to make some tasks machine-specific.
418
419.. note::
420
421 Some tasks are marked as "nostamp" tasks. No timestamp file is
422 created when these tasks are run. Consequently, "nostamp" tasks are
423 always rerun.
424
425For more information on tasks, see the
426:ref:`bitbake-user-manual/bitbake-user-manual-metadata:tasks` section.
427
428Executing Tasks
429===============
430
431Tasks can be either a shell task or a Python task. For shell tasks,
432BitBake writes a shell script to
433``${``\ :term:`T`\ ``}/run.do_taskname.pid`` and then
434executes the script. The generated shell script contains all the
435exported variables, and the shell functions with all variables expanded.
436Output from the shell script goes to the file
437``${T}/log.do_taskname.pid``. Looking at the expanded shell functions in
438the run file and the output in the log files is a useful debugging
439technique.
440
441For Python tasks, BitBake executes the task internally and logs
442information to the controlling terminal. Future versions of BitBake will
443write the functions to files similar to the way shell tasks are handled.
444Logging will be handled in a way similar to shell tasks as well.
445
446The order in which BitBake runs the tasks is controlled by its task
447scheduler. It is possible to configure the scheduler and define custom
448implementations for specific use cases. For more information, see these
449variables that control the behavior:
450
451- :term:`BB_SCHEDULER`
452
453- :term:`BB_SCHEDULERS`
454
455It is possible to have functions run before and after a task's main
456function. This is done using the ``[prefuncs]`` and ``[postfuncs]``
457flags of the task that lists the functions to run.
458
459.. _checksums:
460
461Checksums (Signatures)
462======================
463
464A checksum is a unique signature of a task's inputs. The signature of a
465task can be used to determine if a task needs to be run. Because it is a
466change in a task's inputs that triggers running the task, BitBake needs
467to detect all the inputs to a given task. For shell tasks, this turns
468out to be fairly easy because BitBake generates a "run" shell script for
469each task and it is possible to create a checksum that gives you a good
470idea of when the task's data changes.
471
472To complicate the problem, some things should not be included in the
473checksum. First, there is the actual specific build path of a given task
474- the working directory. It does not matter if the working directory
475changes because it should not affect the output for target packages. The
476simplistic approach for excluding the working directory is to set it to
477some fixed value and create the checksum for the "run" script. BitBake
478goes one step better and uses the
479:term:`BB_HASHBASE_WHITELIST` variable
480to define a list of variables that should never be included when
481generating the signatures.
482
483Another problem results from the "run" scripts containing functions that
484might or might not get called. The incremental build solution contains
485code that figures out dependencies between shell functions. This code is
486used to prune the "run" scripts down to the minimum set, thereby
487alleviating this problem and making the "run" scripts much more readable
488as a bonus.
489
490So far we have solutions for shell scripts. What about Python tasks? The
491same approach applies even though these tasks are more difficult. The
492process needs to figure out what variables a Python function accesses
493and what functions it calls. Again, the incremental build solution
494contains code that first figures out the variable and function
495dependencies, and then creates a checksum for the data used as the input
496to the task.
497
498Like the working directory case, situations exist where dependencies
499should be ignored. For these cases, you can instruct the build process
500to ignore a dependency by using a line like the following: ::
501
502 PACKAGE_ARCHS[vardepsexclude] = "MACHINE"
503
504This example ensures that the
505``PACKAGE_ARCHS`` variable does not depend on the value of ``MACHINE``,
506even if it does reference it.
507
508Equally, there are cases where we need to add dependencies BitBake is
509not able to find. You can accomplish this by using a line like the
510following: ::
511
512 PACKAGE_ARCHS[vardeps] = "MACHINE"
513
514This example explicitly
515adds the ``MACHINE`` variable as a dependency for ``PACKAGE_ARCHS``.
516
517Consider a case with in-line Python, for example, where BitBake is not
518able to figure out dependencies. When running in debug mode (i.e. using
519``-DDD``), BitBake produces output when it discovers something for which
520it cannot figure out dependencies.
521
522Thus far, this section has limited discussion to the direct inputs into
523a task. Information based on direct inputs is referred to as the
524"basehash" in the code. However, there is still the question of a task's
525indirect inputs - the things that were already built and present in the
526build directory. The checksum (or signature) for a particular task needs
527to add the hashes of all the tasks on which the particular task depends.
528Choosing which dependencies to add is a policy decision. However, the
529effect is to generate a master checksum that combines the basehash and
530the hashes of the task's dependencies.
531
532At the code level, there are a variety of ways both the basehash and the
533dependent task hashes can be influenced. Within the BitBake
534configuration file, we can give BitBake some extra information to help
535it construct the basehash. The following statement effectively results
536in a list of global variable dependency excludes - variables never
537included in any checksum. This example uses variables from OpenEmbedded
538to help illustrate the concept: ::
539
540 BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \
541 SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL \
542 USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \
543 PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \
544 CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX"
545
546The previous example excludes the work directory, which is part of
547``TMPDIR``.
548
549The rules for deciding which hashes of dependent tasks to include
550through dependency chains are more complex and are generally
551accomplished with a Python function. The code in
552``meta/lib/oe/sstatesig.py`` shows two examples of this and also
553illustrates how you can insert your own policy into the system if so
554desired. This file defines the two basic signature generators
555OpenEmbedded-Core uses: "OEBasic" and "OEBasicHash". By default, there
556is a dummy "noop" signature handler enabled in BitBake. This means that
557behavior is unchanged from previous versions. ``OE-Core`` uses the
558"OEBasicHash" signature handler by default through this setting in the
559``bitbake.conf`` file: ::
560
561 BB_SIGNATURE_HANDLER ?= "OEBasicHash"
562
563The "OEBasicHash" ``BB_SIGNATURE_HANDLER`` is the same as the "OEBasic"
564version but adds the task hash to the stamp files. This results in any
565metadata change that changes the task hash, automatically causing the
566task to be run again. This removes the need to bump
567:term:`PR` values, and changes to metadata automatically
568ripple across the build.
569
570It is also worth noting that the end result of these signature
571generators is to make some dependency and hash information available to
572the build. This information includes:
573
574- ``BB_BASEHASH_task-``\ *taskname*: The base hashes for each task in the
575 recipe.
576
577- ``BB_BASEHASH_``\ *filename:taskname*: The base hashes for each
578 dependent task.
579
580- ``BBHASHDEPS_``\ *filename:taskname*: The task dependencies for
581 each task.
582
583- ``BB_TASKHASH``: The hash of the currently running task.
584
585It is worth noting that BitBake's "-S" option lets you debug BitBake's
586processing of signatures. The options passed to -S allow different
587debugging modes to be used, either using BitBake's own debug functions
588or possibly those defined in the metadata/signature handler itself. The
589simplest parameter to pass is "none", which causes a set of signature
590information to be written out into ``STAMPS_DIR`` corresponding to the
591targets specified. The other currently available parameter is
592"printdiff", which causes BitBake to try to establish the closest
593signature match it can (e.g. in the sstate cache) and then run
594``bitbake-diffsigs`` over the matches to determine the stamps and delta
595where these two stamp trees diverge.
596
597.. note::
598
599 It is likely that future versions of BitBake will provide other
600 signature handlers triggered through additional "-S" parameters.
601
602You can find more information on checksum metadata in the
603:ref:`bitbake-user-manual/bitbake-user-manual-metadata:task checksums and setscene`
604section.
605
606Setscene
607========
608
609The setscene process enables BitBake to handle "pre-built" artifacts.
610The ability to handle and reuse these artifacts allows BitBake the
611luxury of not having to build something from scratch every time.
612Instead, BitBake can use, when possible, existing build artifacts.
613
614BitBake needs to have reliable data indicating whether or not an
615artifact is compatible. Signatures, described in the previous section,
616provide an ideal way of representing whether an artifact is compatible.
617If a signature is the same, an object can be reused.
618
619If an object can be reused, the problem then becomes how to replace a
620given task or set of tasks with the pre-built artifact. BitBake solves
621the problem with the "setscene" process.
622
623When BitBake is asked to build a given target, before building anything,
624it first asks whether cached information is available for any of the
625targets it's building, or any of the intermediate targets. If cached
626information is available, BitBake uses this information instead of
627running the main tasks.
628
629BitBake first calls the function defined by the
630:term:`BB_HASHCHECK_FUNCTION` variable
631with a list of tasks and corresponding hashes it wants to build. This
632function is designed to be fast and returns a list of the tasks for
633which it believes in can obtain artifacts.
634
635Next, for each of the tasks that were returned as possibilities, BitBake
636executes a setscene version of the task that the possible artifact
637covers. Setscene versions of a task have the string "_setscene" appended
638to the task name. So, for example, the task with the name ``xxx`` has a
639setscene task named ``xxx_setscene``. The setscene version of the task
640executes and provides the necessary artifacts returning either success
641or failure.
642
643As previously mentioned, an artifact can cover more than one task. For
644example, it is pointless to obtain a compiler if you already have the
645compiled binary. To handle this, BitBake calls the
646:term:`BB_SETSCENE_DEPVALID` function for
647each successful setscene task to know whether or not it needs to obtain
648the dependencies of that task.
649
650Finally, after all the setscene tasks have executed, BitBake calls the
651function listed in
652:term:`BB_SETSCENE_VERIFY_FUNCTION2`
653with the list of tasks BitBake thinks has been "covered". The metadata
654can then ensure that this list is correct and can inform BitBake that it
655wants specific tasks to be run regardless of the setscene result.
656
657You can find more information on setscene metadata in the
658:ref:`bitbake-user-manual/bitbake-user-manual-metadata:task checksums and setscene`
659section.
660
661Logging
662=======
663
664In addition to the standard command line option to control how verbose
665builds are when execute, bitbake also supports user defined
666configuration of the `Python
667logging <https://docs.python.org/3/library/logging.html>`__ facilities
668through the :term:`BB_LOGCONFIG` variable. This
669variable defines a json or yaml `logging
670configuration <https://docs.python.org/3/library/logging.config.html>`__
671that will be intelligently merged into the default configuration. The
672logging configuration is merged using the following rules:
673
674- The user defined configuration will completely replace the default
675 configuration if top level key ``bitbake_merge`` is set to the value
676 ``False``. In this case, all other rules are ignored.
677
678- The user configuration must have a top level ``version`` which must
679 match the value of the default configuration.
680
681- Any keys defined in the ``handlers``, ``formatters``, or ``filters``,
682 will be merged into the same section in the default configuration,
683 with the user specified keys taking replacing a default one if there
684 is a conflict. In practice, this means that if both the default
685 configuration and user configuration specify a handler named
686 ``myhandler``, the user defined one will replace the default. To
687 prevent the user from inadvertently replacing a default handler,
688 formatter, or filter, all of the default ones are named with a prefix
689 of "``BitBake.``"
690
691- If a logger is defined by the user with the key ``bitbake_merge`` set
692 to ``False``, that logger will be completely replaced by user
693 configuration. In this case, no other rules will apply to that
694 logger.
695
696- All user defined ``filter`` and ``handlers`` properties for a given
697 logger will be merged with corresponding properties from the default
698 logger. For example, if the user configuration adds a filter called
699 ``myFilter`` to the ``BitBake.SigGen``, and the default configuration
700 adds a filter called ``BitBake.defaultFilter``, both filters will be
701 applied to the logger
702
703As an example, consider the following user logging configuration file
704which logs all Hash Equivalence related messages of VERBOSE or higher to
705a file called ``hashequiv.log`` ::
706
707 {
708 "version": 1,
709 "handlers": {
710 "autobuilderlog": {
711 "class": "logging.FileHandler",
712 "formatter": "logfileFormatter",
713 "level": "DEBUG",
714 "filename": "hashequiv.log",
715 "mode": "w"
716 }
717 },
718 "formatters": {
719 "logfileFormatter": {
720 "format": "%(name)s: %(levelname)s: %(message)s"
721 }
722 },
723 "loggers": {
724 "BitBake.SigGen.HashEquiv": {
725 "level": "VERBOSE",
726 "handlers": ["autobuilderlog"]
727 },
728 "BitBake.RunQueue.HashEquiv": {
729 "level": "VERBOSE",
730 "handlers": ["autobuilderlog"]
731 }
732 }
733 }