Revert "poky: subtree update:b23aa6b753..ad30a6d470"
This reverts commit af5e4ef732faedf66c6dc1756432e9de2ac72988.
This commit introduced openbmc/openbmc#3720 and no solution has been
forthcoming. Revert until we can get to the bottom of this.
Change-Id: I2fb0d81eb26cf3dadb2f2abdd1a1bb7a95eaf03c
diff --git a/poky/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml b/poky/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml
new file mode 100644
index 0000000..04c5a26
--- /dev/null
+++ b/poky/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml
@@ -0,0 +1,1029 @@
+<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
+"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">
+
+<chapter id="bitbake-user-manual-execution">
+ <title>Execution</title>
+
+ <para>
+ The primary purpose for running BitBake is to produce some kind
+ of output such as a single installable package, a kernel, a software
+ development kit, or even a full, board-specific bootable Linux image,
+ complete with bootloader, kernel, and root filesystem.
+ Of course, you can execute the <filename>bitbake</filename>
+ command with options that cause it to execute single tasks,
+ compile single recipe files, capture or clear data, or simply
+ return information about the execution environment.
+ </para>
+
+ <para>
+ This chapter describes BitBake's execution process from start
+ to finish when you use it to create an image.
+ The execution process is launched using the following command
+ form:
+ <literallayout class='monospaced'>
+ $ bitbake <replaceable>target</replaceable>
+ </literallayout>
+ For information on the BitBake command and its options,
+ see
+ "<link linkend='bitbake-user-manual-command'>The BitBake Command</link>"
+ section.
+ <note>
+ <para>
+ Prior to executing BitBake, you should take advantage of available
+ parallel thread execution on your build host by setting the
+ <link linkend='var-bb-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
+ variable in your project's <filename>local.conf</filename>
+ configuration file.
+ </para>
+
+ <para>
+ A common method to determine this value for your build host is to run
+ the following:
+ <literallayout class='monospaced'>
+ $ grep processor /proc/cpuinfo
+ </literallayout>
+ This command returns the number of processors, which takes into
+ account hyper-threading.
+ Thus, a quad-core build host with hyper-threading most likely
+ shows eight processors, which is the value you would then assign to
+ <filename>BB_NUMBER_THREADS</filename>.
+ </para>
+
+ <para>
+ A possibly simpler solution is that some Linux distributions
+ (e.g. Debian and Ubuntu) provide the <filename>ncpus</filename> command.
+ </para>
+ </note>
+ </para>
+
+ <section id='parsing-the-base-configuration-metadata'>
+ <title>Parsing the Base Configuration Metadata</title>
+
+ <para>
+ The first thing BitBake does is parse base configuration
+ metadata.
+ Base configuration metadata consists of your project's
+ <filename>bblayers.conf</filename> file to determine what
+ layers BitBake needs to recognize, all necessary
+ <filename>layer.conf</filename> files (one from each layer),
+ and <filename>bitbake.conf</filename>.
+ The data itself is of various types:
+ <itemizedlist>
+ <listitem><para><emphasis>Recipes:</emphasis>
+ Details about particular pieces of software.
+ </para></listitem>
+ <listitem><para><emphasis>Class Data:</emphasis>
+ An abstraction of common build information
+ (e.g. how to build a Linux kernel).
+ </para></listitem>
+ <listitem><para><emphasis>Configuration Data:</emphasis>
+ Machine-specific settings, policy decisions,
+ and so forth.
+ Configuration data acts as the glue to bind everything
+ together.</para></listitem>
+ </itemizedlist>
+ </para>
+
+ <para>
+ The <filename>layer.conf</filename> files are used to
+ construct key variables such as
+ <link linkend='var-bb-BBPATH'><filename>BBPATH</filename></link>
+ and
+ <link linkend='var-bb-BBFILES'><filename>BBFILES</filename></link>.
+ <filename>BBPATH</filename> is used to search for
+ configuration and class files under the
+ <filename>conf</filename> and <filename>classes</filename>
+ directories, respectively.
+ <filename>BBFILES</filename> is used to locate both recipe
+ and recipe append files
+ (<filename>.bb</filename> and <filename>.bbappend</filename>).
+ If there is no <filename>bblayers.conf</filename> file,
+ it is assumed the user has set the <filename>BBPATH</filename>
+ and <filename>BBFILES</filename> directly in the environment.
+ </para>
+
+ <para>
+ Next, the <filename>bitbake.conf</filename> file is located
+ using the <filename>BBPATH</filename> variable that was
+ just constructed.
+ The <filename>bitbake.conf</filename> file may also include other
+ configuration files using the
+ <filename>include</filename> or
+ <filename>require</filename> directives.
+ </para>
+
+ <para>
+ Prior to parsing configuration files, BitBake looks
+ at certain variables, including:
+ <itemizedlist>
+ <listitem><para>
+ <link linkend='var-bb-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link>
+ </para></listitem>
+ <listitem><para>
+ <link linkend='var-bb-BB_ENV_EXTRAWHITE'><filename>BB_ENV_EXTRAWHITE</filename></link>
+ </para></listitem>
+ <listitem><para>
+ <link linkend='var-bb-BB_PRESERVE_ENV'><filename>BB_PRESERVE_ENV</filename></link>
+ </para></listitem>
+ <listitem><para>
+ <link linkend='var-bb-BB_ORIGENV'><filename>BB_ORIGENV</filename></link>
+ </para></listitem>
+ <listitem><para>
+ <link linkend='var-bb-BITBAKE_UI'><filename>BITBAKE_UI</filename></link>
+ </para></listitem>
+ </itemizedlist>
+ The first four variables in this list relate to how BitBake treats shell
+ environment variables during task execution.
+ By default, BitBake cleans the environment variables and provides tight
+ control over the shell execution environment.
+ However, through the use of these first four variables, you can
+ apply your control regarding the
+ environment variables allowed to be used by BitBake in the shell
+ during execution of tasks.
+ See the
+ "<link linkend='passing-information-into-the-build-task-environment'>Passing Information Into the Build Task Environment</link>"
+ section and the information about these variables in the
+ variable glossary for more information on how they work and
+ on how to use them.
+ </para>
+
+ <para>
+ The base configuration metadata is global
+ and therefore affects all recipes and tasks that are executed.
+ </para>
+
+ <para>
+ BitBake first searches the current working directory for an
+ optional <filename>conf/bblayers.conf</filename> configuration file.
+ This file is expected to contain a
+ <link linkend='var-bb-BBLAYERS'><filename>BBLAYERS</filename></link>
+ variable that is a space-delimited list of 'layer' directories.
+ Recall that if BitBake cannot find a <filename>bblayers.conf</filename>
+ file, then it is assumed the user has set the <filename>BBPATH</filename>
+ and <filename>BBFILES</filename> variables directly in the environment.
+ </para>
+
+ <para>
+ For each directory (layer) in this list, a <filename>conf/layer.conf</filename>
+ file is located and parsed with the
+ <link linkend='var-bb-LAYERDIR'><filename>LAYERDIR</filename></link>
+ variable being set to the directory where the layer was found.
+ The idea is these files automatically set up
+ <link linkend='var-bb-BBPATH'><filename>BBPATH</filename></link>
+ and other variables correctly for a given build directory.
+ </para>
+
+ <para>
+ BitBake then expects to find the <filename>conf/bitbake.conf</filename>
+ file somewhere in the user-specified <filename>BBPATH</filename>.
+ That configuration file generally has include directives to pull
+ in any other metadata such as files specific to the architecture,
+ the machine, the local environment, and so forth.
+ </para>
+
+ <para>
+ Only variable definitions and include directives are allowed
+ in BitBake <filename>.conf</filename> files.
+ Some variables directly influence BitBake's behavior.
+ These variables might have been set from the environment
+ depending on the environment variables previously
+ mentioned or set in the configuration files.
+ The
+ "<link linkend='ref-bb-variables-glos'>Variables Glossary</link>"
+ chapter presents a full list of variables.
+ </para>
+
+ <para>
+ After parsing configuration files, BitBake uses its rudimentary
+ inheritance mechanism, which is through class files, to inherit
+ some standard classes.
+ BitBake parses a class when the inherit directive responsible
+ for getting that class is encountered.
+ </para>
+
+ <para>
+ The <filename>base.bbclass</filename> file is always included.
+ Other classes that are specified in the configuration using the
+ <link linkend='var-bb-INHERIT'><filename>INHERIT</filename></link>
+ variable are also included.
+ BitBake searches for class files in a
+ <filename>classes</filename> subdirectory under
+ the paths in <filename>BBPATH</filename> in the same way as
+ configuration files.
+ </para>
+
+ <para>
+ A good way to get an idea of the configuration files and
+ the class files used in your execution environment is to
+ run the following BitBake command:
+ <literallayout class='monospaced'>
+ $ bitbake -e > mybb.log
+ </literallayout>
+ Examining the top of the <filename>mybb.log</filename>
+ shows you the many configuration files and class files
+ used in your execution environment.
+ </para>
+
+ <note>
+ <para>
+ You need to be aware of how BitBake parses curly braces.
+ If a recipe uses a closing curly brace within the function and
+ the character has no leading spaces, BitBake produces a parsing
+ error.
+ If you use a pair of curly braces in a shell function, the
+ closing curly brace must not be located at the start of the line
+ without leading spaces.
+ </para>
+
+ <para>
+ Here is an example that causes BitBake to produce a parsing
+ error:
+ <literallayout class='monospaced'>
+ fakeroot create_shar() {
+ cat << "EOF" > ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh
+ usage()
+ {
+ echo "test"
+ ###### The following "}" at the start of the line causes a parsing error ######
+ }
+ EOF
+ }
+ </literallayout>
+ Writing the recipe this way avoids the error:
+ <literallayout class='monospaced'>
+ fakeroot create_shar() {
+ cat << "EOF" > ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh
+ usage()
+ {
+ echo "test"
+ ######The following "}" with a leading space at the start of the line avoids the error ######
+ }
+ EOF
+ }
+ </literallayout>
+ </para>
+ </note>
+ </section>
+
+ <section id='locating-and-parsing-recipes'>
+ <title>Locating and Parsing Recipes</title>
+
+ <para>
+ During the configuration phase, BitBake will have set
+ <link linkend='var-bb-BBFILES'><filename>BBFILES</filename></link>.
+ BitBake now uses it to construct a list of recipes to parse,
+ along with any append files (<filename>.bbappend</filename>)
+ to apply.
+ <filename>BBFILES</filename> is a space-separated list of
+ available files and supports wildcards.
+ An example would be:
+ <literallayout class='monospaced'>
+ BBFILES = "/path/to/bbfiles/*.bb /path/to/appends/*.bbappend"
+ </literallayout>
+ BitBake parses each recipe and append file located
+ with <filename>BBFILES</filename> and stores the values of
+ various variables into the datastore.
+ <note>
+ Append files are applied in the order they are encountered in
+ <filename>BBFILES</filename>.
+ </note>
+ For each file, a fresh copy of the base configuration is
+ made, then the recipe is parsed line by line.
+ Any inherit statements cause BitBake to find and
+ then parse class files (<filename>.bbclass</filename>)
+ using
+ <link linkend='var-bb-BBPATH'><filename>BBPATH</filename></link>
+ as the search path.
+ Finally, BitBake parses in order any append files found in
+ <filename>BBFILES</filename>.
+ </para>
+
+ <para>
+ One common convention is to use the recipe filename to define
+ pieces of metadata.
+ For example, in <filename>bitbake.conf</filename> the recipe
+ name and version are used to set the variables
+ <link linkend='var-bb-PN'><filename>PN</filename></link> and
+ <link linkend='var-bb-PV'><filename>PV</filename></link>:
+ <literallayout class='monospaced'>
+ PN = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0] or 'defaultpkgname'}"
+ PV = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[1] or '1.0'}"
+ </literallayout>
+ In this example, a recipe called "something_1.2.3.bb" would set
+ <filename>PN</filename> to "something" and
+ <filename>PV</filename> to "1.2.3".
+ </para>
+
+ <para>
+ By the time parsing is complete for a recipe, BitBake
+ has a list of tasks that the recipe defines and a set of
+ data consisting of keys and values as well as
+ dependency information about the tasks.
+ </para>
+
+ <para>
+ BitBake does not need all of this information.
+ It only needs a small subset of the information to make
+ decisions about the recipe.
+ Consequently, BitBake caches the values in which it is
+ interested and does not store the rest of the information.
+ Experience has shown it is faster to re-parse the metadata than to
+ try and write it out to the disk and then reload it.
+ </para>
+
+ <para>
+ Where possible, subsequent BitBake commands reuse this cache of
+ recipe information.
+ The validity of this cache is determined by first computing a
+ checksum of the base configuration data (see
+ <link linkend='var-bb-BB_HASHCONFIG_WHITELIST'><filename>BB_HASHCONFIG_WHITELIST</filename></link>)
+ and then checking if the checksum matches.
+ If that checksum matches what is in the cache and the recipe
+ and class files have not changed, BitBake is able to use
+ the cache.
+ BitBake then reloads the cached information about the recipe
+ instead of reparsing it from scratch.
+ </para>
+
+ <para>
+ Recipe file collections exist to allow the user to
+ have multiple repositories of
+ <filename>.bb</filename> files that contain the same
+ exact package.
+ For example, one could easily use them to make one's
+ own local copy of an upstream repository, but with
+ custom modifications that one does not want upstream.
+ Here is an example:
+ <literallayout class='monospaced'>
+ BBFILES = "/stuff/openembedded/*/*.bb /stuff/openembedded.modified/*/*.bb"
+ BBFILE_COLLECTIONS = "upstream local"
+ BBFILE_PATTERN_upstream = "^/stuff/openembedded/"
+ BBFILE_PATTERN_local = "^/stuff/openembedded.modified/"
+ BBFILE_PRIORITY_upstream = "5"
+ BBFILE_PRIORITY_local = "10"
+ </literallayout>
+ <note>
+ The layers mechanism is now the preferred method of collecting
+ code.
+ While the collections code remains, its main use is to set layer
+ priorities and to deal with overlap (conflicts) between layers.
+ </note>
+ </para>
+ </section>
+
+ <section id='bb-bitbake-providers'>
+ <title>Providers</title>
+
+ <para>
+ Assuming BitBake has been instructed to execute a target
+ and that all the recipe files have been parsed, BitBake
+ starts to figure out how to build the target.
+ BitBake looks through the <filename>PROVIDES</filename> list
+ for each of the recipes.
+ A <filename>PROVIDES</filename> list is the list of names by which
+ the recipe can be known.
+ Each recipe's <filename>PROVIDES</filename> list is created
+ implicitly through the recipe's
+ <link linkend='var-bb-PN'><filename>PN</filename></link> variable
+ and explicitly through the recipe's
+ <link linkend='var-bb-PROVIDES'><filename>PROVIDES</filename></link>
+ variable, which is optional.
+ </para>
+
+ <para>
+ When a recipe uses <filename>PROVIDES</filename>, that recipe's
+ functionality can be found under an alternative name or names other
+ than the implicit <filename>PN</filename> name.
+ As an example, suppose a recipe named <filename>keyboard_1.0.bb</filename>
+ contained the following:
+ <literallayout class='monospaced'>
+ PROVIDES += "fullkeyboard"
+ </literallayout>
+ The <filename>PROVIDES</filename> list for this recipe becomes
+ "keyboard", which is implicit, and "fullkeyboard", which is explicit.
+ Consequently, the functionality found in
+ <filename>keyboard_1.0.bb</filename> can be found under two
+ different names.
+ </para>
+ </section>
+
+ <section id='bb-bitbake-preferences'>
+ <title>Preferences</title>
+
+ <para>
+ The <filename>PROVIDES</filename> list is only part of the solution
+ for figuring out a target's recipes.
+ Because targets might have multiple providers, BitBake needs
+ to prioritize providers by determining provider preferences.
+ </para>
+
+ <para>
+ A common example in which a target has multiple providers
+ is "virtual/kernel", which is on the
+ <filename>PROVIDES</filename> list for each kernel recipe.
+ Each machine often selects the best kernel provider by using a
+ line similar to the following in the machine configuration file:
+ <literallayout class='monospaced'>
+ PREFERRED_PROVIDER_virtual/kernel = "linux-yocto"
+ </literallayout>
+ The default
+ <link linkend='var-bb-PREFERRED_PROVIDER'><filename>PREFERRED_PROVIDER</filename></link>
+ is the provider with the same name as the target.
+ BitBake iterates through each target it needs to build and
+ resolves them and their dependencies using this process.
+ </para>
+
+ <para>
+ Understanding how providers are chosen is made complicated by the fact
+ that multiple versions might exist for a given provider.
+ BitBake defaults to the highest version of a provider.
+ Version comparisons are made using the same method as Debian.
+ You can use the
+ <link linkend='var-bb-PREFERRED_VERSION'><filename>PREFERRED_VERSION</filename></link>
+ variable to specify a particular version.
+ You can influence the order by using the
+ <link linkend='var-bb-DEFAULT_PREFERENCE'><filename>DEFAULT_PREFERENCE</filename></link>
+ variable.
+ </para>
+
+ <para>
+ By default, files have a preference of "0".
+ Setting <filename>DEFAULT_PREFERENCE</filename> to "-1" makes the
+ recipe unlikely to be used unless it is explicitly referenced.
+ Setting <filename>DEFAULT_PREFERENCE</filename> to "1" makes it
+ likely the recipe is used.
+ <filename>PREFERRED_VERSION</filename> overrides any
+ <filename>DEFAULT_PREFERENCE</filename> setting.
+ <filename>DEFAULT_PREFERENCE</filename> is often used to mark newer
+ and more experimental recipe versions until they have undergone
+ sufficient testing to be considered stable.
+ </para>
+
+ <para>
+ When there are multiple “versions” of a given recipe,
+ BitBake defaults to selecting the most recent
+ version, unless otherwise specified.
+ If the recipe in question has a
+ <link linkend='var-bb-DEFAULT_PREFERENCE'><filename>DEFAULT_PREFERENCE</filename></link>
+ set lower than the other recipes (default is 0), then
+ it will not be selected.
+ This allows the person or persons maintaining
+ the repository of recipe files to specify
+ their preference for the default selected version.
+ Additionally, the user can specify their preferred version.
+ </para>
+
+ <para>
+ If the first recipe is named <filename>a_1.1.bb</filename>, then the
+ <link linkend='var-bb-PN'><filename>PN</filename></link> variable
+ will be set to “a”, and the
+ <link linkend='var-bb-PV'><filename>PV</filename></link>
+ variable will be set to 1.1.
+ </para>
+
+ <para>
+ Thus, if a recipe named <filename>a_1.2.bb</filename> exists, BitBake
+ will choose 1.2 by default.
+ However, if you define the following variable in a
+ <filename>.conf</filename> file that BitBake parses, you
+ can change that preference:
+ <literallayout class='monospaced'>
+ PREFERRED_VERSION_a = "1.1"
+ </literallayout>
+ </para>
+
+ <note>
+ <para>
+ It is common for a recipe to provide two versions -- a stable,
+ numbered (and preferred) version, and a version that is
+ automatically checked out from a source code repository that
+ is considered more "bleeding edge" but can be selected only
+ explicitly.
+ </para>
+
+ <para>
+ For example, in the OpenEmbedded codebase, there is a standard,
+ versioned recipe file for BusyBox,
+ <filename>busybox_1.22.1.bb</filename>,
+ but there is also a Git-based version,
+ <filename>busybox_git.bb</filename>, which explicitly contains the line
+ <literallayout class='monospaced'>
+ DEFAULT_PREFERENCE = "-1"
+ </literallayout>
+ to ensure that the numbered, stable version is always preferred
+ unless the developer selects otherwise.
+ </para>
+ </note>
+ </section>
+
+ <section id='bb-bitbake-dependencies'>
+ <title>Dependencies</title>
+
+ <para>
+ Each target BitBake builds consists of multiple tasks such as
+ <filename>fetch</filename>, <filename>unpack</filename>,
+ <filename>patch</filename>, <filename>configure</filename>,
+ and <filename>compile</filename>.
+ For best performance on multi-core systems, BitBake considers each
+ task as an independent
+ entity with its own set of dependencies.
+ </para>
+
+ <para>
+ Dependencies are defined through several variables.
+ You can find information about variables BitBake uses in
+ the <link linkend='ref-bb-variables-glos'>Variables Glossary</link>
+ near the end of this manual.
+ At a basic level, it is sufficient to know that BitBake uses the
+ <link linkend='var-bb-DEPENDS'><filename>DEPENDS</filename></link> and
+ <link linkend='var-bb-RDEPENDS'><filename>RDEPENDS</filename></link> variables when
+ calculating dependencies.
+ </para>
+
+ <para>
+ For more information on how BitBake handles dependencies, see the
+ "<link linkend='dependencies'>Dependencies</link>" section.
+ </para>
+ </section>
+
+ <section id='ref-bitbake-tasklist'>
+ <title>The Task List</title>
+
+ <para>
+ Based on the generated list of providers and the dependency information,
+ BitBake can now calculate exactly what tasks it needs to run and in what
+ order it needs to run them.
+ The
+ "<link linkend='executing-tasks'>Executing Tasks</link>" section has more
+ information on how BitBake chooses which task to execute next.
+ </para>
+
+ <para>
+ The build now starts with BitBake forking off threads up to the limit set in the
+ <link linkend='var-bb-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
+ variable.
+ BitBake continues to fork threads as long as there are tasks ready to run,
+ those tasks have all their dependencies met, and the thread threshold has not been
+ exceeded.
+ </para>
+
+ <para>
+ It is worth noting that you can greatly speed up the build time by properly setting
+ the <filename>BB_NUMBER_THREADS</filename> variable.
+ </para>
+
+ <para>
+ As each task completes, a timestamp is written to the directory specified by the
+ <link linkend='var-bb-STAMP'><filename>STAMP</filename></link> variable.
+ On subsequent runs, BitBake looks in the build directory within
+ <filename>tmp/stamps</filename> and does not rerun
+ tasks that are already completed unless a timestamp is found to be invalid.
+ Currently, invalid timestamps are only considered on a per
+ recipe file basis.
+ So, for example, if the configure stamp has a timestamp greater than the
+ compile timestamp for a given target, then the compile task would rerun.
+ Running the compile task again, however, has no effect on other providers
+ that depend on that target.
+ </para>
+
+ <para>
+ The exact format of the stamps is partly configurable.
+ In modern versions of BitBake, a hash is appended to the
+ stamp so that if the configuration changes, the stamp becomes
+ invalid and the task is automatically rerun.
+ This hash, or signature used, is governed by the signature policy
+ that is configured (see the
+ "<link linkend='checksums'>Checksums (Signatures)</link>"
+ section for information).
+ It is also possible to append extra metadata to the stamp using
+ the <filename>[stamp-extra-info]</filename> task flag.
+ For example, OpenEmbedded uses this flag to make some tasks machine-specific.
+ </para>
+
+ <note>
+ Some tasks are marked as "nostamp" tasks.
+ No timestamp file is created when these tasks are run.
+ Consequently, "nostamp" tasks are always rerun.
+ </note>
+
+ <para>
+ For more information on tasks, see the
+ "<link linkend='tasks'>Tasks</link>" section.
+ </para>
+ </section>
+
+ <section id='executing-tasks'>
+ <title>Executing Tasks</title>
+
+ <para>
+ Tasks can be either a shell task or a Python task.
+ For shell tasks, BitBake writes a shell script to
+ <filename>${</filename><link linkend='var-bb-T'><filename>T</filename></link><filename>}/run.do_taskname.<replaceable>pid</replaceable></filename>
+ and then executes the script.
+ The generated shell script contains all the exported variables,
+ and the shell functions with all variables expanded.
+ Output from the shell script goes to the file
+ <filename>${T}/log.do_taskname.<replaceable>pid</replaceable></filename>.
+ Looking at the expanded shell functions in the run file and
+ the output in the log files is a useful debugging technique.
+ </para>
+
+ <para>
+ For Python tasks, BitBake executes the task internally and logs
+ information to the controlling terminal.
+ Future versions of BitBake will write the functions to files
+ similar to the way shell tasks are handled.
+ Logging will be handled in a way similar to shell tasks as well.
+ </para>
+
+ <para>
+ The order in which BitBake runs the tasks is controlled by its
+ task scheduler.
+ It is possible to configure the scheduler and define custom
+ implementations for specific use cases.
+ For more information, see these variables that control the
+ behavior:
+ <itemizedlist>
+ <listitem><para>
+ <link linkend='var-bb-BB_SCHEDULER'><filename>BB_SCHEDULER</filename></link>
+ </para></listitem>
+ <listitem><para>
+ <link linkend='var-bb-BB_SCHEDULERS'><filename>BB_SCHEDULERS</filename></link>
+ </para></listitem>
+ </itemizedlist>
+ It is possible to have functions run before and after a task's main
+ function.
+ This is done using the <filename>[prefuncs]</filename>
+ and <filename>[postfuncs]</filename> flags of the task
+ that lists the functions to run.
+ </para>
+ </section>
+
+ <section id='checksums'>
+ <title>Checksums (Signatures)</title>
+
+ <para>
+ A checksum is a unique signature of a task's inputs.
+ The signature of a task can be used to determine if a task
+ needs to be run.
+ Because it is a change in a task's inputs that triggers running
+ the task, BitBake needs to detect all the inputs to a given task.
+ For shell tasks, this turns out to be fairly easy because
+ BitBake generates a "run" shell script for each task and
+ it is possible to create a checksum that gives you a good idea of when
+ the task's data changes.
+ </para>
+
+ <para>
+ To complicate the problem, some things should not be included in
+ the checksum.
+ First, there is the actual specific build path of a given task -
+ the working directory.
+ It does not matter if the working directory changes because it should not
+ affect the output for target packages.
+ The simplistic approach for excluding the working directory is to set
+ it to some fixed value and create the checksum for the "run" script.
+ BitBake goes one step better and uses the
+ <link linkend='var-bb-BB_HASHBASE_WHITELIST'><filename>BB_HASHBASE_WHITELIST</filename></link>
+ variable to define a list of variables that should never be included
+ when generating the signatures.
+ </para>
+
+ <para>
+ Another problem results from the "run" scripts containing functions that
+ might or might not get called.
+ The incremental build solution contains code that figures out dependencies
+ between shell functions.
+ This code is used to prune the "run" scripts down to the minimum set,
+ thereby alleviating this problem and making the "run" scripts much more
+ readable as a bonus.
+ </para>
+
+ <para>
+ So far we have solutions for shell scripts.
+ What about Python tasks?
+ The same approach applies even though these tasks are more difficult.
+ The process needs to figure out what variables a Python function accesses
+ and what functions it calls.
+ Again, the incremental build solution contains code that first figures out
+ the variable and function dependencies, and then creates a checksum for the data
+ used as the input to the task.
+ </para>
+
+ <para>
+ Like the working directory case, situations exist where dependencies
+ should be ignored.
+ For these cases, you can instruct the build process to ignore a dependency
+ by using a line like the following:
+ <literallayout class='monospaced'>
+ PACKAGE_ARCHS[vardepsexclude] = "MACHINE"
+ </literallayout>
+ This example ensures that the <filename>PACKAGE_ARCHS</filename> variable does not
+ depend on the value of <filename>MACHINE</filename>, even if it does reference it.
+ </para>
+
+ <para>
+ Equally, there are cases where we need to add dependencies BitBake
+ is not able to find.
+ You can accomplish this by using a line like the following:
+ <literallayout class='monospaced'>
+ PACKAGE_ARCHS[vardeps] = "MACHINE"
+ </literallayout>
+ This example explicitly adds the <filename>MACHINE</filename> variable as a
+ dependency for <filename>PACKAGE_ARCHS</filename>.
+ </para>
+
+ <para>
+ Consider a case with in-line Python, for example, where BitBake is not
+ able to figure out dependencies.
+ When running in debug mode (i.e. using <filename>-DDD</filename>), BitBake
+ produces output when it discovers something for which it cannot figure out
+ dependencies.
+ </para>
+
+ <para>
+ Thus far, this section has limited discussion to the direct inputs into a task.
+ Information based on direct inputs is referred to as the "basehash" in the
+ code.
+ However, there is still the question of a task's indirect inputs - the
+ things that were already built and present in the build directory.
+ The checksum (or signature) for a particular task needs to add the hashes
+ of all the tasks on which the particular task depends.
+ Choosing which dependencies to add is a policy decision.
+ However, the effect is to generate a master checksum that combines the basehash
+ and the hashes of the task's dependencies.
+ </para>
+
+ <para>
+ At the code level, there are a variety of ways both the basehash and the
+ dependent task hashes can be influenced.
+ Within the BitBake configuration file, we can give BitBake some extra information
+ to help it construct the basehash.
+ The following statement effectively results in a list of global variable
+ dependency excludes - variables never included in any checksum.
+ This example uses variables from OpenEmbedded to help illustrate
+ the concept:
+ <literallayout class='monospaced'>
+ BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \
+ SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL \
+ USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \
+ PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \
+ CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX"
+ </literallayout>
+ The previous example excludes the work directory, which is part of
+ <filename>TMPDIR</filename>.
+ </para>
+
+ <para>
+ The rules for deciding which hashes of dependent tasks to include through
+ dependency chains are more complex and are generally accomplished with a
+ Python function.
+ The code in <filename>meta/lib/oe/sstatesig.py</filename> shows two examples
+ of this and also illustrates how you can insert your own policy into the system
+ if so desired.
+ This file defines the two basic signature generators OpenEmbedded-Core
+ uses: "OEBasic" and "OEBasicHash".
+ By default, there is a dummy "noop" signature handler enabled in BitBake.
+ This means that behavior is unchanged from previous versions.
+ <filename>OE-Core</filename> uses the "OEBasicHash" signature handler by default
+ through this setting in the <filename>bitbake.conf</filename> file:
+ <literallayout class='monospaced'>
+ BB_SIGNATURE_HANDLER ?= "OEBasicHash"
+ </literallayout>
+ The "OEBasicHash" <filename>BB_SIGNATURE_HANDLER</filename> is the same as the
+ "OEBasic" version but adds the task hash to the stamp files.
+ This results in any metadata change that changes the task hash, automatically
+ causing the task to be run again.
+ This removes the need to bump
+ <link linkend='var-bb-PR'><filename>PR</filename></link>
+ values, and changes to metadata automatically ripple across the build.
+ </para>
+
+ <para>
+ It is also worth noting that the end result of these signature generators is to
+ make some dependency and hash information available to the build.
+ This information includes:
+ <itemizedlist>
+ <listitem><para><filename>BB_BASEHASH_task-</filename><replaceable>taskname</replaceable>:
+ The base hashes for each task in the recipe.
+ </para></listitem>
+ <listitem><para><filename>BB_BASEHASH_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>:
+ The base hashes for each dependent task.
+ </para></listitem>
+ <listitem><para><filename>BBHASHDEPS_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>:
+ The task dependencies for each task.
+ </para></listitem>
+ <listitem><para><filename>BB_TASKHASH</filename>:
+ The hash of the currently running task.
+ </para></listitem>
+ </itemizedlist>
+ </para>
+
+ <para>
+ It is worth noting that BitBake's "-S" option lets you
+ debug BitBake's processing of signatures.
+ The options passed to -S allow different debugging modes
+ to be used, either using BitBake's own debug functions
+ or possibly those defined in the metadata/signature handler
+ itself.
+ The simplest parameter to pass is "none", which causes a
+ set of signature information to be written out into
+ <filename>STAMPS_DIR</filename>
+ corresponding to the targets specified.
+ The other currently available parameter is "printdiff",
+ which causes BitBake to try to establish the closest
+ signature match it can (e.g. in the sstate cache) and then
+ run <filename>bitbake-diffsigs</filename> over the matches
+ to determine the stamps and delta where these two
+ stamp trees diverge.
+ <note>
+ It is likely that future versions of BitBake will
+ provide other signature handlers triggered through
+ additional "-S" parameters.
+ </note>
+ </para>
+
+ <para>
+ You can find more information on checksum metadata in the
+ "<link linkend='task-checksums-and-setscene'>Task Checksums and Setscene</link>"
+ section.
+ </para>
+ </section>
+
+ <section id='setscene'>
+ <title>Setscene</title>
+
+ <para>
+ The setscene process enables BitBake to handle "pre-built" artifacts.
+ The ability to handle and reuse these artifacts allows BitBake
+ the luxury of not having to build something from scratch every time.
+ Instead, BitBake can use, when possible, existing build artifacts.
+ </para>
+
+ <para>
+ BitBake needs to have reliable data indicating whether or not an
+ artifact is compatible.
+ Signatures, described in the previous section, provide an ideal
+ way of representing whether an artifact is compatible.
+ If a signature is the same, an object can be reused.
+ </para>
+
+ <para>
+ If an object can be reused, the problem then becomes how to
+ replace a given task or set of tasks with the pre-built artifact.
+ BitBake solves the problem with the "setscene" process.
+ </para>
+
+ <para>
+ When BitBake is asked to build a given target, before building anything,
+ it first asks whether cached information is available for any of the
+ targets it's building, or any of the intermediate targets.
+ If cached information is available, BitBake uses this information instead of
+ running the main tasks.
+ </para>
+
+ <para>
+ BitBake first calls the function defined by the
+ <link linkend='var-bb-BB_HASHCHECK_FUNCTION'><filename>BB_HASHCHECK_FUNCTION</filename></link>
+ variable with a list of tasks and corresponding
+ hashes it wants to build.
+ This function is designed to be fast and returns a list
+ of the tasks for which it believes in can obtain artifacts.
+ </para>
+
+ <para>
+ Next, for each of the tasks that were returned as possibilities,
+ BitBake executes a setscene version of the task that the possible
+ artifact covers.
+ Setscene versions of a task have the string "_setscene" appended to the
+ task name.
+ So, for example, the task with the name <filename>xxx</filename> has
+ a setscene task named <filename>xxx_setscene</filename>.
+ The setscene version of the task executes and provides the necessary
+ artifacts returning either success or failure.
+ </para>
+
+ <para>
+ As previously mentioned, an artifact can cover more than one task.
+ For example, it is pointless to obtain a compiler if you
+ already have the compiled binary.
+ To handle this, BitBake calls the
+ <link linkend='var-bb-BB_SETSCENE_DEPVALID'><filename>BB_SETSCENE_DEPVALID</filename></link>
+ function for each successful setscene task to know whether or not it needs
+ to obtain the dependencies of that task.
+ </para>
+
+ <para>
+ Finally, after all the setscene tasks have executed, BitBake calls the
+ function listed in
+ <link linkend='var-bb-BB_SETSCENE_VERIFY_FUNCTION2'><filename>BB_SETSCENE_VERIFY_FUNCTION2</filename></link>
+ with the list of tasks BitBake thinks has been "covered".
+ The metadata can then ensure that this list is correct and can
+ inform BitBake that it wants specific tasks to be run regardless
+ of the setscene result.
+ </para>
+
+ <para>
+ You can find more information on setscene metadata in the
+ "<link linkend='task-checksums-and-setscene'>Task Checksums and Setscene</link>"
+ section.
+ </para>
+ </section>
+
+ <section id="logging">
+ <title>Logging</title>
+ <para>
+ In addition to the standard command line option to control how
+ verbose builds are when execute, bitbake also supports user defined
+ configuration of the
+ <ulink url='https://docs.python.org/3/library/logging.html'>Python logging</ulink>
+ facilities through the
+ <link linkend="var-bb-BB_LOGCONFIG"><filename>BB_LOGCONFIG</filename></link>
+ variable. This variable defines a json or yaml
+ <ulink url='https://docs.python.org/3/library/logging.config.html'>logging configuration</ulink>
+ that will be intelligently merged into the default configuration.
+ The logging configuration is merged using the following rules:
+ <itemizedlist>
+ <listitem><para>
+ The user defined configuration will completely replace the default
+ configuration if top level key
+ <filename>bitbake_merge</filename> is set to the value
+ <filename>False</filename>. In this case, all other rules
+ are ignored.
+ </para></listitem>
+ <listitem><para>
+ The user configuration must have a top level
+ <filename>version</filename> which must match the value of
+ the default configuration.
+ </para></listitem>
+ <listitem><para>
+ Any keys defined in the <filename>handlers</filename>,
+ <filename>formatters</filename>, or <filename>filters</filename>,
+ will be merged into the same section in the default
+ configuration, with the user specified keys taking
+ replacing a default one if there is a conflict. In
+ practice, this means that if both the default configuration
+ and user configuration specify a handler named
+ <filename>myhandler</filename>, the user defined one will
+ replace the default. To prevent the user from inadvertently
+ replacing a default handler, formatter, or filter, all of
+ the default ones are named with a prefix of
+ "<filename>BitBake.</filename>"
+ </para></listitem>
+ <listitem><para>
+ If a logger is defined by the user with the key
+ <filename>bitbake_merge</filename> set to
+ <filename>False</filename>, that logger will be completely
+ replaced by user configuration. In this case, no other
+ rules will apply to that logger.
+ </para></listitem>
+ <listitem><para>
+ All user defined <filename>filter</filename> and
+ <filename>handlers</filename> properties for a given logger
+ will be merged with corresponding properties from the
+ default logger. For example, if the user configuration adds
+ a filter called <filename>myFilter</filename> to the
+ <filename>BitBake.SigGen</filename>, and the default
+ configuration adds a filter called
+ <filename>BitBake.defaultFilter</filename>, both filters
+ will be applied to the logger
+ </para></listitem>
+ </itemizedlist>
+ </para>
+
+ <para>
+ As an example, consider the following user logging configuration
+ file which logs all Hash Equivalence related messages of VERBOSE or
+ higher to a file called <filename>hashequiv.log</filename>
+ <literallayout class='monospaced'>
+ {
+ "version": 1,
+ "handlers": {
+ "autobuilderlog": {
+ "class": "logging.FileHandler",
+ "formatter": "logfileFormatter",
+ "level": "DEBUG",
+ "filename": "hashequiv.log",
+ "mode": "w"
+ }
+ },
+ "formatters": {
+ "logfileFormatter": {
+ "format": "%(name)s: %(levelname)s: %(message)s"
+ }
+ },
+ "loggers": {
+ "BitBake.SigGen.HashEquiv": {
+ "level": "VERBOSE",
+ "handlers": ["autobuilderlog"]
+ },
+ "BitBake.RunQueue.HashEquiv": {
+ "level": "VERBOSE",
+ "handlers": ["autobuilderlog"]
+ }
+ }
+ }
+ </literallayout>
+ </para>
+ </section>
+</chapter>