Gitiles
Code Review Sign In
gerrit.openbmc.org / mdmillerii / openbmc / 20f9b94e782e4db6a1e897bcd2c58bb033de7a1f / . / poky / bitbake / doc / bitbake-user-manual / bitbake-user-manual-execution.xml
blob: 6c5234ed993317195a1a22c2a87832db3aadd575 [file] [log] [blame]
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">
<chapter id="bitbake-user-manual-execution">
<title>Execution</title>
<para>
The primary purpose for running BitBake is to produce some kind
of output such as a single installable package, a kernel, a software
development kit, or even a full, board-specific bootable Linux image,
complete with bootloader, kernel, and root filesystem.
Of course, you can execute the <filename>bitbake</filename>
command with options that cause it to execute single tasks,
compile single recipe files, capture or clear data, or simply
return information about the execution environment.
</para>
<para>
This chapter describes BitBake's execution process from start
to finish when you use it to create an image.
The execution process is launched using the following command
form:
<literallayout class='monospaced'>
$ bitbake <replaceable>target</replaceable>
</literallayout>
For information on the BitBake command and its options,
see
"<link linkend='bitbake-user-manual-command'>The BitBake Command</link>"
section.
<note>
<para>
Prior to executing BitBake, you should take advantage of available
parallel thread execution on your build host by setting the
<link linkend='var-bb-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
variable in your project's <filename>local.conf</filename>
configuration file.
</para>
<para>
A common method to determine this value for your build host is to run
the following:
<literallayout class='monospaced'>
$ grep processor /proc/cpuinfo
</literallayout>
This command returns the number of processors, which takes into
account hyper-threading.
Thus, a quad-core build host with hyper-threading most likely
shows eight processors, which is the value you would then assign to
<filename>BB_NUMBER_THREADS</filename>.
</para>
<para>
A possibly simpler solution is that some Linux distributions
(e.g. Debian and Ubuntu) provide the <filename>ncpus</filename> command.
</para>
</note>
</para>
<section id='parsing-the-base-configuration-metadata'>
<title>Parsing the Base Configuration Metadata</title>
<para>
The first thing BitBake does is parse base configuration
metadata.
Base configuration metadata consists of your project's
<filename>bblayers.conf</filename> file to determine what
layers BitBake needs to recognize, all necessary
<filename>layer.conf</filename> files (one from each layer),
and <filename>bitbake.conf</filename>.
The data itself is of various types:
<itemizedlist>
<listitem><para><emphasis>Recipes:</emphasis>
Details about particular pieces of software.
</para></listitem>
<listitem><para><emphasis>Class Data:</emphasis>
An abstraction of common build information
(e.g. how to build a Linux kernel).
</para></listitem>
<listitem><para><emphasis>Configuration Data:</emphasis>
Machine-specific settings, policy decisions,
and so forth.
Configuration data acts as the glue to bind everything
together.</para></listitem>
</itemizedlist>
</para>
<para>
The <filename>layer.conf</filename> files are used to
construct key variables such as
<link linkend='var-bb-BBPATH'><filename>BBPATH</filename></link>
and
<link linkend='var-bb-BBFILES'><filename>BBFILES</filename></link>.
<filename>BBPATH</filename> is used to search for
configuration and class files under the
<filename>conf</filename> and <filename>classes</filename>
directories, respectively.
<filename>BBFILES</filename> is used to locate both recipe
and recipe append files
(<filename>.bb</filename> and <filename>.bbappend</filename>).
If there is no <filename>bblayers.conf</filename> file,
it is assumed the user has set the <filename>BBPATH</filename>
and <filename>BBFILES</filename> directly in the environment.
</para>
<para>
Next, the <filename>bitbake.conf</filename> file is located
using the <filename>BBPATH</filename> variable that was
just constructed.
The <filename>bitbake.conf</filename> file may also include other
configuration files using the
<filename>include</filename> or
<filename>require</filename> directives.
</para>
<para>
Prior to parsing configuration files, BitBake looks
at certain variables, including:
<itemizedlist>
<listitem><para>
<link linkend='var-bb-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link>
</para></listitem>
<listitem><para>
<link linkend='var-bb-BB_ENV_EXTRAWHITE'><filename>BB_ENV_EXTRAWHITE</filename></link>
</para></listitem>
<listitem><para>
<link linkend='var-bb-BB_PRESERVE_ENV'><filename>BB_PRESERVE_ENV</filename></link>
</para></listitem>
<listitem><para>
<link linkend='var-bb-BB_ORIGENV'><filename>BB_ORIGENV</filename></link>
</para></listitem>
<listitem><para>
<link linkend='var-bb-BITBAKE_UI'><filename>BITBAKE_UI</filename></link>
</para></listitem>
</itemizedlist>
The first four variables in this list relate to how BitBake treats shell
environment variables during task execution.
By default, BitBake cleans the environment variables and provides tight
control over the shell execution environment.
However, through the use of these first four variables, you can
apply your control regarding the
environment variables allowed to be used by BitBake in the shell
during execution of tasks.
See the
"<link linkend='passing-information-into-the-build-task-environment'>Passing Information Into the Build Task Environment</link>"
section and the information about these variables in the
variable glossary for more information on how they work and
on how to use them.
</para>
<para>
The base configuration metadata is global
and therefore affects all recipes and tasks that are executed.
</para>
<para>
BitBake first searches the current working directory for an
optional <filename>conf/bblayers.conf</filename> configuration file.
This file is expected to contain a
<link linkend='var-bb-BBLAYERS'><filename>BBLAYERS</filename></link>
variable that is a space-delimited list of 'layer' directories.
Recall that if BitBake cannot find a <filename>bblayers.conf</filename>
file, then it is assumed the user has set the <filename>BBPATH</filename>
and <filename>BBFILES</filename> variables directly in the environment.
</para>
<para>
For each directory (layer) in this list, a <filename>conf/layer.conf</filename>
file is located and parsed with the
<link linkend='var-bb-LAYERDIR'><filename>LAYERDIR</filename></link>
variable being set to the directory where the layer was found.
The idea is these files automatically set up
<link linkend='var-bb-BBPATH'><filename>BBPATH</filename></link>
and other variables correctly for a given build directory.
</para>
<para>
BitBake then expects to find the <filename>conf/bitbake.conf</filename>
file somewhere in the user-specified <filename>BBPATH</filename>.
That configuration file generally has include directives to pull
in any other metadata such as files specific to the architecture,
the machine, the local environment, and so forth.
</para>
<para>
Only variable definitions and include directives are allowed
in BitBake <filename>.conf</filename> files.
Some variables directly influence BitBake's behavior.
These variables might have been set from the environment
depending on the environment variables previously
mentioned or set in the configuration files.
The
"<link linkend='ref-bb-variables-glos'>Variables Glossary</link>"
chapter presents a full list of variables.
</para>
<para>
After parsing configuration files, BitBake uses its rudimentary
inheritance mechanism, which is through class files, to inherit
some standard classes.
BitBake parses a class when the inherit directive responsible
for getting that class is encountered.
</para>
<para>
The <filename>base.bbclass</filename> file is always included.
Other classes that are specified in the configuration using the
<link linkend='var-bb-INHERIT'><filename>INHERIT</filename></link>
variable are also included.
BitBake searches for class files in a
<filename>classes</filename> subdirectory under
the paths in <filename>BBPATH</filename> in the same way as
configuration files.
</para>
<para>
A good way to get an idea of the configuration files and
the class files used in your execution environment is to
run the following BitBake command:
<literallayout class='monospaced'>
$ bitbake -e > mybb.log
</literallayout>
Examining the top of the <filename>mybb.log</filename>
shows you the many configuration files and class files
used in your execution environment.
</para>
<note>
<para>
You need to be aware of how BitBake parses curly braces.
If a recipe uses a closing curly brace within the function and
the character has no leading spaces, BitBake produces a parsing
error.
If you use a pair of curly braces in a shell function, the
closing curly brace must not be located at the start of the line
without leading spaces.
</para>
<para>
Here is an example that causes BitBake to produce a parsing
error:
<literallayout class='monospaced'>
fakeroot create_shar() {
cat &lt;&lt; "EOF" &gt; ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh
usage()
{
echo "test"
###### The following "}" at the start of the line causes a parsing error ######
}
EOF
}
</literallayout>
Writing the recipe this way avoids the error:
<literallayout class='monospaced'>
fakeroot create_shar() {
cat &lt;&lt; "EOF" &gt; ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh
usage()
{
echo "test"
######The following "}" with a leading space at the start of the line avoids the error ######
}
EOF
}
</literallayout>
</para>
</note>
</section>
<section id='locating-and-parsing-recipes'>
<title>Locating and Parsing Recipes</title>
<para>
During the configuration phase, BitBake will have set
<link linkend='var-bb-BBFILES'><filename>BBFILES</filename></link>.
BitBake now uses it to construct a list of recipes to parse,
along with any append files (<filename>.bbappend</filename>)
to apply.
<filename>BBFILES</filename> is a space-separated list of
available files and supports wildcards.
An example would be:
<literallayout class='monospaced'>
BBFILES = "/path/to/bbfiles/*.bb /path/to/appends/*.bbappend"
</literallayout>
BitBake parses each recipe and append file located
with <filename>BBFILES</filename> and stores the values of
various variables into the datastore.
<note>
Append files are applied in the order they are encountered in
<filename>BBFILES</filename>.
</note>
For each file, a fresh copy of the base configuration is
made, then the recipe is parsed line by line.
Any inherit statements cause BitBake to find and
then parse class files (<filename>.bbclass</filename>)
using
<link linkend='var-bb-BBPATH'><filename>BBPATH</filename></link>
as the search path.
Finally, BitBake parses in order any append files found in
<filename>BBFILES</filename>.
</para>
<para>
One common convention is to use the recipe filename to define
pieces of metadata.
For example, in <filename>bitbake.conf</filename> the recipe
name and version are used to set the variables
<link linkend='var-bb-PN'><filename>PN</filename></link> and
<link linkend='var-bb-PV'><filename>PV</filename></link>:
<literallayout class='monospaced'>
PN = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0] or 'defaultpkgname'}"
PV = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[1] or '1.0'}"
</literallayout>
In this example, a recipe called "something_1.2.3.bb" would set
<filename>PN</filename> to "something" and
<filename>PV</filename> to "1.2.3".
</para>
<para>
By the time parsing is complete for a recipe, BitBake
has a list of tasks that the recipe defines and a set of
data consisting of keys and values as well as
dependency information about the tasks.
</para>
<para>
BitBake does not need all of this information.
It only needs a small subset of the information to make
decisions about the recipe.
Consequently, BitBake caches the values in which it is
interested and does not store the rest of the information.
Experience has shown it is faster to re-parse the metadata than to
try and write it out to the disk and then reload it.
</para>
<para>
Where possible, subsequent BitBake commands reuse this cache of
recipe information.
The validity of this cache is determined by first computing a
checksum of the base configuration data (see
<link linkend='var-bb-BB_HASHCONFIG_WHITELIST'><filename>BB_HASHCONFIG_WHITELIST</filename></link>)
and then checking if the checksum matches.
If that checksum matches what is in the cache and the recipe
and class files have not changed, BitBake is able to use
the cache.
BitBake then reloads the cached information about the recipe
instead of reparsing it from scratch.
</para>
<para>
Recipe file collections exist to allow the user to
have multiple repositories of
<filename>.bb</filename> files that contain the same
exact package.
For example, one could easily use them to make one's
own local copy of an upstream repository, but with
custom modifications that one does not want upstream.
Here is an example:
<literallayout class='monospaced'>
BBFILES = "/stuff/openembedded/*/*.bb /stuff/openembedded.modified/*/*.bb"
BBFILE_COLLECTIONS = "upstream local"
BBFILE_PATTERN_upstream = "^/stuff/openembedded/"
BBFILE_PATTERN_local = "^/stuff/openembedded.modified/"
BBFILE_PRIORITY_upstream = "5"
BBFILE_PRIORITY_local = "10"
</literallayout>
<note>
The layers mechanism is now the preferred method of collecting
code.
While the collections code remains, its main use is to set layer
priorities and to deal with overlap (conflicts) between layers.
</note>
</para>
</section>
<section id='bb-bitbake-providers'>
<title>Providers</title>
<para>
Assuming BitBake has been instructed to execute a target
and that all the recipe files have been parsed, BitBake
starts to figure out how to build the target.
BitBake looks through the <filename>PROVIDES</filename> list
for each of the recipes.
A <filename>PROVIDES</filename> list is the list of names by which
the recipe can be known.
Each recipe's <filename>PROVIDES</filename> list is created
implicitly through the recipe's
<link linkend='var-bb-PN'><filename>PN</filename></link> variable
and explicitly through the recipe's
<link linkend='var-bb-PROVIDES'><filename>PROVIDES</filename></link>
variable, which is optional.
</para>
<para>
When a recipe uses <filename>PROVIDES</filename>, that recipe's
functionality can be found under an alternative name or names other
than the implicit <filename>PN</filename> name.
As an example, suppose a recipe named <filename>keyboard_1.0.bb</filename>
contained the following:
<literallayout class='monospaced'>
PROVIDES += "fullkeyboard"
</literallayout>
The <filename>PROVIDES</filename> list for this recipe becomes
"keyboard", which is implicit, and "fullkeyboard", which is explicit.
Consequently, the functionality found in
<filename>keyboard_1.0.bb</filename> can be found under two
different names.
</para>
</section>
<section id='bb-bitbake-preferences'>
<title>Preferences</title>
<para>
The <filename>PROVIDES</filename> list is only part of the solution
for figuring out a target's recipes.
Because targets might have multiple providers, BitBake needs
to prioritize providers by determining provider preferences.
</para>
<para>
A common example in which a target has multiple providers
is "virtual/kernel", which is on the
<filename>PROVIDES</filename> list for each kernel recipe.
Each machine often selects the best kernel provider by using a
line similar to the following in the machine configuration file:
<literallayout class='monospaced'>
PREFERRED_PROVIDER_virtual/kernel = "linux-yocto"
</literallayout>
The default
<link linkend='var-bb-PREFERRED_PROVIDER'><filename>PREFERRED_PROVIDER</filename></link>
is the provider with the same name as the target.
BitBake iterates through each target it needs to build and
resolves them and their dependencies using this process.
</para>
<para>
Understanding how providers are chosen is made complicated by the fact
that multiple versions might exist for a given provider.
BitBake defaults to the highest version of a provider.
Version comparisons are made using the same method as Debian.
You can use the
<link linkend='var-bb-PREFERRED_VERSION'><filename>PREFERRED_VERSION</filename></link>
variable to specify a particular version.
You can influence the order by using the
<link linkend='var-bb-DEFAULT_PREFERENCE'><filename>DEFAULT_PREFERENCE</filename></link>
variable.
</para>
<para>
By default, files have a preference of "0".
Setting <filename>DEFAULT_PREFERENCE</filename> to "-1" makes the
recipe unlikely to be used unless it is explicitly referenced.
Setting <filename>DEFAULT_PREFERENCE</filename> to "1" makes it
likely the recipe is used.
<filename>PREFERRED_VERSION</filename> overrides any
<filename>DEFAULT_PREFERENCE</filename> setting.
<filename>DEFAULT_PREFERENCE</filename> is often used to mark newer
and more experimental recipe versions until they have undergone
sufficient testing to be considered stable.
</para>
<para>
When there are multiple “versions” of a given recipe,
BitBake defaults to selecting the most recent
version, unless otherwise specified.
If the recipe in question has a
<link linkend='var-bb-DEFAULT_PREFERENCE'><filename>DEFAULT_PREFERENCE</filename></link>
set lower than the other recipes (default is 0), then
it will not be selected.
This allows the person or persons maintaining
the repository of recipe files to specify
their preference for the default selected version.
Additionally, the user can specify their preferred version.
</para>
<para>
If the first recipe is named <filename>a_1.1.bb</filename>, then the
<link linkend='var-bb-PN'><filename>PN</filename></link> variable
will be set to “a”, and the
<link linkend='var-bb-PV'><filename>PV</filename></link>
variable will be set to 1.1.
</para>
<para>
Thus, if a recipe named <filename>a_1.2.bb</filename> exists, BitBake
will choose 1.2 by default.
However, if you define the following variable in a
<filename>.conf</filename> file that BitBake parses, you
can change that preference:
<literallayout class='monospaced'>
PREFERRED_VERSION_a = "1.1"
</literallayout>
</para>
<note>
<para>
It is common for a recipe to provide two versions -- a stable,
numbered (and preferred) version, and a version that is
automatically checked out from a source code repository that
is considered more "bleeding edge" but can be selected only
explicitly.
</para>
<para>
For example, in the OpenEmbedded codebase, there is a standard,
versioned recipe file for BusyBox,
<filename>busybox_1.22.1.bb</filename>,
but there is also a Git-based version,
<filename>busybox_git.bb</filename>, which explicitly contains the line
<literallayout class='monospaced'>
DEFAULT_PREFERENCE = "-1"
</literallayout>
to ensure that the numbered, stable version is always preferred
unless the developer selects otherwise.
</para>
</note>
</section>
<section id='bb-bitbake-dependencies'>
<title>Dependencies</title>
<para>
Each target BitBake builds consists of multiple tasks such as
<filename>fetch</filename>, <filename>unpack</filename>,
<filename>patch</filename>, <filename>configure</filename>,
and <filename>compile</filename>.
For best performance on multi-core systems, BitBake considers each
task as an independent
entity with its own set of dependencies.
</para>
<para>
Dependencies are defined through several variables.
You can find information about variables BitBake uses in
the <link linkend='ref-bb-variables-glos'>Variables Glossary</link>
near the end of this manual.
At a basic level, it is sufficient to know that BitBake uses the
<link linkend='var-bb-DEPENDS'><filename>DEPENDS</filename></link> and
<link linkend='var-bb-RDEPENDS'><filename>RDEPENDS</filename></link> variables when
calculating dependencies.
</para>
<para>
For more information on how BitBake handles dependencies, see the
"<link linkend='dependencies'>Dependencies</link>" section.
</para>
</section>
<section id='ref-bitbake-tasklist'>
<title>The Task List</title>
<para>
Based on the generated list of providers and the dependency information,
BitBake can now calculate exactly what tasks it needs to run and in what
order it needs to run them.
The
"<link linkend='executing-tasks'>Executing Tasks</link>" section has more
information on how BitBake chooses which task to execute next.
</para>
<para>
The build now starts with BitBake forking off threads up to the limit set in the
<link linkend='var-bb-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
variable.
BitBake continues to fork threads as long as there are tasks ready to run,
those tasks have all their dependencies met, and the thread threshold has not been
exceeded.
</para>
<para>
It is worth noting that you can greatly speed up the build time by properly setting
the <filename>BB_NUMBER_THREADS</filename> variable.
</para>
<para>
As each task completes, a timestamp is written to the directory specified by the
<link linkend='var-bb-STAMP'><filename>STAMP</filename></link> variable.
On subsequent runs, BitBake looks in the build directory within
<filename>tmp/stamps</filename> and does not rerun
tasks that are already completed unless a timestamp is found to be invalid.
Currently, invalid timestamps are only considered on a per
recipe file basis.
So, for example, if the configure stamp has a timestamp greater than the
compile timestamp for a given target, then the compile task would rerun.
Running the compile task again, however, has no effect on other providers
that depend on that target.
</para>
<para>
The exact format of the stamps is partly configurable.
In modern versions of BitBake, a hash is appended to the
stamp so that if the configuration changes, the stamp becomes
invalid and the task is automatically rerun.
This hash, or signature used, is governed by the signature policy
that is configured (see the
"<link linkend='checksums'>Checksums (Signatures)</link>"
section for information).
It is also possible to append extra metadata to the stamp using
the <filename>[stamp-extra-info]</filename> task flag.
For example, OpenEmbedded uses this flag to make some tasks machine-specific.
</para>
<note>
Some tasks are marked as "nostamp" tasks.
No timestamp file is created when these tasks are run.
Consequently, "nostamp" tasks are always rerun.
</note>
<para>
For more information on tasks, see the
"<link linkend='tasks'>Tasks</link>" section.
</para>
</section>
<section id='executing-tasks'>
<title>Executing Tasks</title>
<para>
Tasks can be either a shell task or a Python task.
For shell tasks, BitBake writes a shell script to
<filename>${</filename><link linkend='var-bb-T'><filename>T</filename></link><filename>}/run.do_taskname.pid</filename>
and then executes the script.
The generated shell script contains all the exported variables,
and the shell functions with all variables expanded.
Output from the shell script goes to the file
<filename>${T}/log.do_taskname.pid</filename>.
Looking at the expanded shell functions in the run file and
the output in the log files is a useful debugging technique.
</para>
<para>
For Python tasks, BitBake executes the task internally and logs
information to the controlling terminal.
Future versions of BitBake will write the functions to files
similar to the way shell tasks are handled.
Logging will be handled in a way similar to shell tasks as well.
</para>
<para>
The order in which BitBake runs the tasks is controlled by its
task scheduler.
It is possible to configure the scheduler and define custom
implementations for specific use cases.
For more information, see these variables that control the
behavior:
<itemizedlist>
<listitem><para>
<link linkend='var-bb-BB_SCHEDULER'><filename>BB_SCHEDULER</filename></link>
</para></listitem>
<listitem><para>
<link linkend='var-bb-BB_SCHEDULERS'><filename>BB_SCHEDULERS</filename></link>
</para></listitem>
</itemizedlist>
It is possible to have functions run before and after a task's main
function.
This is done using the <filename>[prefuncs]</filename>
and <filename>[postfuncs]</filename> flags of the task
that lists the functions to run.
</para>
</section>
<section id='checksums'>
<title>Checksums (Signatures)</title>
<para>
A checksum is a unique signature of a task's inputs.
The signature of a task can be used to determine if a task
needs to be run.
Because it is a change in a task's inputs that triggers running
the task, BitBake needs to detect all the inputs to a given task.
For shell tasks, this turns out to be fairly easy because
BitBake generates a "run" shell script for each task and
it is possible to create a checksum that gives you a good idea of when
the task's data changes.
</para>
<para>
To complicate the problem, some things should not be included in
the checksum.
First, there is the actual specific build path of a given task -
the working directory.
It does not matter if the working directory changes because it should not
affect the output for target packages.
The simplistic approach for excluding the working directory is to set
it to some fixed value and create the checksum for the "run" script.
BitBake goes one step better and uses the
<link linkend='var-bb-BB_HASHBASE_WHITELIST'><filename>BB_HASHBASE_WHITELIST</filename></link>
variable to define a list of variables that should never be included
when generating the signatures.
</para>
<para>
Another problem results from the "run" scripts containing functions that
might or might not get called.
The incremental build solution contains code that figures out dependencies
between shell functions.
This code is used to prune the "run" scripts down to the minimum set,
thereby alleviating this problem and making the "run" scripts much more
readable as a bonus.
</para>
<para>
So far we have solutions for shell scripts.
What about Python tasks?
The same approach applies even though these tasks are more difficult.
The process needs to figure out what variables a Python function accesses
and what functions it calls.
Again, the incremental build solution contains code that first figures out
the variable and function dependencies, and then creates a checksum for the data
used as the input to the task.
</para>
<para>
Like the working directory case, situations exist where dependencies
should be ignored.
For these cases, you can instruct the build process to ignore a dependency
by using a line like the following:
<literallayout class='monospaced'>
PACKAGE_ARCHS[vardepsexclude] = "MACHINE"
</literallayout>
This example ensures that the <filename>PACKAGE_ARCHS</filename> variable does not
depend on the value of <filename>MACHINE</filename>, even if it does reference it.
</para>
<para>
Equally, there are cases where we need to add dependencies BitBake
is not able to find.
You can accomplish this by using a line like the following:
<literallayout class='monospaced'>
PACKAGE_ARCHS[vardeps] = "MACHINE"
</literallayout>
This example explicitly adds the <filename>MACHINE</filename> variable as a
dependency for <filename>PACKAGE_ARCHS</filename>.
</para>
<para>
Consider a case with in-line Python, for example, where BitBake is not
able to figure out dependencies.
When running in debug mode (i.e. using <filename>-DDD</filename>), BitBake
produces output when it discovers something for which it cannot figure out
dependencies.
</para>
<para>
Thus far, this section has limited discussion to the direct inputs into a task.
Information based on direct inputs is referred to as the "basehash" in the
code.
However, there is still the question of a task's indirect inputs - the
things that were already built and present in the build directory.
The checksum (or signature) for a particular task needs to add the hashes
of all the tasks on which the particular task depends.
Choosing which dependencies to add is a policy decision.
However, the effect is to generate a master checksum that combines the basehash
and the hashes of the task's dependencies.
</para>
<para>
At the code level, there are a variety of ways both the basehash and the
dependent task hashes can be influenced.
Within the BitBake configuration file, we can give BitBake some extra information
to help it construct the basehash.
The following statement effectively results in a list of global variable
dependency excludes - variables never included in any checksum.
This example uses variables from OpenEmbedded to help illustrate
the concept:
<literallayout class='monospaced'>
BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \
SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \
USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \
PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \
CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX"
</literallayout>
The previous example excludes the work directory, which is part of
<filename>TMPDIR</filename>.
</para>
<para>
The rules for deciding which hashes of dependent tasks to include through
dependency chains are more complex and are generally accomplished with a
Python function.
The code in <filename>meta/lib/oe/sstatesig.py</filename> shows two examples
of this and also illustrates how you can insert your own policy into the system
if so desired.
This file defines the two basic signature generators OpenEmbedded-Core
uses: "OEBasic" and "OEBasicHash".
By default, there is a dummy "noop" signature handler enabled in BitBake.
This means that behavior is unchanged from previous versions.
<filename>OE-Core</filename> uses the "OEBasicHash" signature handler by default
through this setting in the <filename>bitbake.conf</filename> file:
<literallayout class='monospaced'>
BB_SIGNATURE_HANDLER ?= "OEBasicHash"
</literallayout>
The "OEBasicHash" <filename>BB_SIGNATURE_HANDLER</filename> is the same as the
"OEBasic" version but adds the task hash to the stamp files.
This results in any metadata change that changes the task hash, automatically
causing the task to be run again.
This removes the need to bump
<link linkend='var-bb-PR'><filename>PR</filename></link>
values, and changes to metadata automatically ripple across the build.
</para>
<para>
It is also worth noting that the end result of these signature generators is to
make some dependency and hash information available to the build.
This information includes:
<itemizedlist>
<listitem><para><filename>BB_BASEHASH_task-</filename><replaceable>taskname</replaceable>:
The base hashes for each task in the recipe.
</para></listitem>
<listitem><para><filename>BB_BASEHASH_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>:
The base hashes for each dependent task.
</para></listitem>
<listitem><para><filename>BBHASHDEPS_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>:
The task dependencies for each task.
</para></listitem>
<listitem><para><filename>BB_TASKHASH</filename>:
The hash of the currently running task.
</para></listitem>
</itemizedlist>
</para>
<para>
It is worth noting that BitBake's "-S" option lets you
debug BitBake's processing of signatures.
The options passed to -S allow different debugging modes
to be used, either using BitBake's own debug functions
or possibly those defined in the metadata/signature handler
itself.
The simplest parameter to pass is "none", which causes a
set of signature information to be written out into
<filename>STAMPS_DIR</filename>
corresponding to the targets specified.
The other currently available parameter is "printdiff",
which causes BitBake to try to establish the closest
signature match it can (e.g. in the sstate cache) and then
run <filename>bitbake-diffsigs</filename> over the matches
to determine the stamps and delta where these two
stamp trees diverge.
<note>
It is likely that future versions of BitBake will
provide other signature handlers triggered through
additional "-S" parameters.
</note>
</para>
<para>
You can find more information on checksum metadata in the
"<link linkend='task-checksums-and-setscene'>Task Checksums and Setscene</link>"
section.
</para>
</section>
<section id='setscene'>
<title>Setscene</title>
<para>
The setscene process enables BitBake to handle "pre-built" artifacts.
The ability to handle and reuse these artifacts allows BitBake
the luxury of not having to build something from scratch every time.
Instead, BitBake can use, when possible, existing build artifacts.
</para>
<para>
BitBake needs to have reliable data indicating whether or not an
artifact is compatible.
Signatures, described in the previous section, provide an ideal
way of representing whether an artifact is compatible.
If a signature is the same, an object can be reused.
</para>
<para>
If an object can be reused, the problem then becomes how to
replace a given task or set of tasks with the pre-built artifact.
BitBake solves the problem with the "setscene" process.
</para>
<para>
When BitBake is asked to build a given target, before building anything,
it first asks whether cached information is available for any of the
targets it's building, or any of the intermediate targets.
If cached information is available, BitBake uses this information instead of
running the main tasks.
</para>
<para>
BitBake first calls the function defined by the
<link linkend='var-bb-BB_HASHCHECK_FUNCTION'><filename>BB_HASHCHECK_FUNCTION</filename></link>
variable with a list of tasks and corresponding
hashes it wants to build.
This function is designed to be fast and returns a list
of the tasks for which it believes in can obtain artifacts.
</para>
<para>
Next, for each of the tasks that were returned as possibilities,
BitBake executes a setscene version of the task that the possible
artifact covers.
Setscene versions of a task have the string "_setscene" appended to the
task name.
So, for example, the task with the name <filename>xxx</filename> has
a setscene task named <filename>xxx_setscene</filename>.
The setscene version of the task executes and provides the necessary
artifacts returning either success or failure.
</para>
<para>
As previously mentioned, an artifact can cover more than one task.
For example, it is pointless to obtain a compiler if you
already have the compiled binary.
To handle this, BitBake calls the
<link linkend='var-bb-BB_SETSCENE_DEPVALID'><filename>BB_SETSCENE_DEPVALID</filename></link>
function for each successful setscene task to know whether or not it needs
to obtain the dependencies of that task.
</para>
<para>
Finally, after all the setscene tasks have executed, BitBake calls the
function listed in
<link linkend='var-bb-BB_SETSCENE_VERIFY_FUNCTION2'><filename>BB_SETSCENE_VERIFY_FUNCTION2</filename></link>
with the list of tasks BitBake thinks has been "covered".
The metadata can then ensure that this list is correct and can
inform BitBake that it wants specific tasks to be run regardless
of the setscene result.
</para>
<para>
You can find more information on setscene metadata in the
"<link linkend='task-checksums-and-setscene'>Task Checksums and Setscene</link>"
section.
</para>
</section>
<section id="logging">
<title>Logging</title>
<para>
In addition to the standard command line option to control how
verbose builds are when execute, bitbake also supports user defined
configuration of the
<ulink url='https://docs.python.org/3/library/logging.html'>Python logging</ulink>
facilities through the
<link linkend="var-bb-BB_LOGCONFIG"><filename>BB_LOGCONFIG</filename></link>
variable. This variable defines a json or yaml
<ulink url='https://docs.python.org/3/library/logging.config.html'>logging configuration</ulink>
that will be intelligently merged into the default configuration.
The logging configuration is merged using the following rules:
<itemizedlist>
<listitem><para>
The user defined configuration will completely replace the default
configuration if top level key
<filename>bitbake_merge</filename> is set to the value
<filename>False</filename>. In this case, all other rules
are ignored.
</para></listitem>
<listitem><para>
The user configuration must have a top level
<filename>version</filename> which must match the value of
the default configuration.
</para></listitem>
<listitem><para>
Any keys defined in the <filename>handlers</filename>,
<filename>formatters</filename>, or <filename>filters</filename>,
will be merged into the same section in the default
configuration, with the user specified keys taking
replacing a default one if there is a conflict. In
practice, this means that if both the default configuration
and user configuration specify a handler named
<filename>myhandler</filename>, the user defined one will
replace the default. To prevent the user from inadvertently
replacing a default handler, formatter, or filter, all of
the default ones are named with a prefix of
"<filename>BitBake.</filename>"
</para></listitem>
<listitem><para>
If a logger is defined by the user with the key
<filename>bitbake_merge</filename> set to
<filename>False</filename>, that logger will be completely
replaced by user configuration. In this case, no other
rules will apply to that logger.
</para></listitem>
<listitem><para>
All user defined <filename>filter</filename> and
<filename>handlers</filename> properties for a given logger
will be merged with corresponding properties from the
default logger. For example, if the user configuration adds
a filter called <filename>myFilter</filename> to the
<filename>BitBake.SigGen</filename>, and the default
configuration adds a filter called
<filename>BitBake.defaultFilter</filename>, both filters
will be applied to the logger
</para></listitem>
</itemizedlist>
</para>
<para>
As an example, consider the following user logging configuration
file which logs all Hash Equivalence related messages of VERBOSE or
higher to a file called <filename>hashequiv.log</filename>
<literallayout class='monospaced'>
{
"version": 1,
"handlers": {
"autobuilderlog": {
"class": "logging.FileHandler",
"formatter": "logfileFormatter",
"level": "DEBUG",
"filename": "hashequiv.log",
"mode": "w"
}
},
"formatters": {
"logfileFormatter": {
"format": "%(name)s: %(levelname)s: %(message)s"
}
},
"loggers": {
"BitBake.SigGen.HashEquiv": {
"level": "VERBOSE",
"handlers": ["autobuilderlog"]
},
"BitBake.RunQueue.HashEquiv": {
"level": "VERBOSE",
"handlers": ["autobuilderlog"]
}
}
}
</literallayout>
</para>
</section>
</chapter>