| <!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" |
| "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"> |
| |
| <chapter id="bitbake-user-manual-execution"> |
| <title>Execution</title> |
| |
| <para> |
| The primary purpose for running BitBake is to produce some kind |
| of output such as a single installable package, a kernel, a software |
| development kit, or even a full, board-specific bootable Linux image, |
| complete with bootloader, kernel, and root filesystem. |
| Of course, you can execute the <filename>bitbake</filename> |
| command with options that cause it to execute single tasks, |
| compile single recipe files, capture or clear data, or simply |
| return information about the execution environment. |
| </para> |
| |
| <para> |
| This chapter describes BitBake's execution process from start |
| to finish when you use it to create an image. |
| The execution process is launched using the following command |
| form: |
| <literallayout class='monospaced'> |
| $ bitbake <replaceable>target</replaceable> |
| </literallayout> |
| For information on the BitBake command and its options, |
| see |
| "<link linkend='bitbake-user-manual-command'>The BitBake Command</link>" |
| section. |
| <note> |
| <para> |
| Prior to executing BitBake, you should take advantage of available |
| parallel thread execution on your build host by setting the |
| <link linkend='var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link> |
| variable in your project's <filename>local.conf</filename> |
| configuration file. |
| </para> |
| |
| <para> |
| A common method to determine this value for your build host is to run |
| the following: |
| <literallayout class='monospaced'> |
| $ grep processor /proc/cpuinfo |
| </literallayout> |
| This command returns the number of processors, which takes into |
| account hyper-threading. |
| Thus, a quad-core build host with hyper-threading most likely |
| shows eight processors, which is the value you would then assign to |
| <filename>BB_NUMBER_THREADS</filename>. |
| </para> |
| |
| <para> |
| A possibly simpler solution is that some Linux distributions |
| (e.g. Debian and Ubuntu) provide the <filename>ncpus</filename> command. |
| </para> |
| </note> |
| </para> |
| |
| <section id='parsing-the-base-configuration-metadata'> |
| <title>Parsing the Base Configuration Metadata</title> |
| |
| <para> |
| The first thing BitBake does is parse base configuration |
| metadata. |
| Base configuration metadata consists of your project's |
| <filename>bblayers.conf</filename> file to determine what |
| layers BitBake needs to recognize, all necessary |
| <filename>layer.conf</filename> files (one from each layer), |
| and <filename>bitbake.conf</filename>. |
| The data itself is of various types: |
| <itemizedlist> |
| <listitem><para><emphasis>Recipes:</emphasis> |
| Details about particular pieces of software. |
| </para></listitem> |
| <listitem><para><emphasis>Class Data:</emphasis> |
| An abstraction of common build information |
| (e.g. how to build a Linux kernel). |
| </para></listitem> |
| <listitem><para><emphasis>Configuration Data:</emphasis> |
| Machine-specific settings, policy decisions, |
| and so forth. |
| Configuration data acts as the glue to bind everything |
| together.</para></listitem> |
| </itemizedlist> |
| </para> |
| |
| <para> |
| The <filename>layer.conf</filename> files are used to |
| construct key variables such as |
| <link linkend='var-BBPATH'><filename>BBPATH</filename></link> |
| and |
| <link linkend='var-BBFILES'><filename>BBFILES</filename></link>. |
| <filename>BBPATH</filename> is used to search for |
| configuration and class files under the |
| <filename>conf</filename> and <filename>classes</filename> |
| directories, respectively. |
| <filename>BBFILES</filename> is used to locate both recipe |
| and recipe append files |
| (<filename>.bb</filename> and <filename>.bbappend</filename>). |
| If there is no <filename>bblayers.conf</filename> file, |
| it is assumed the user has set the <filename>BBPATH</filename> |
| and <filename>BBFILES</filename> directly in the environment. |
| </para> |
| |
| <para> |
| Next, the <filename>bitbake.conf</filename> file is located |
| using the <filename>BBPATH</filename> variable that was |
| just constructed. |
| The <filename>bitbake.conf</filename> file may also include other |
| configuration files using the |
| <filename>include</filename> or |
| <filename>require</filename> directives. |
| </para> |
| |
| <para> |
| Prior to parsing configuration files, Bitbake looks |
| at certain variables, including: |
| <itemizedlist> |
| <listitem><para><link linkend='var-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link></para></listitem> |
| <listitem><para><link linkend='var-BB_PRESERVE_ENV'><filename>BB_PRESERVE_ENV</filename></link></para></listitem> |
| <listitem><para><link linkend='var-BB_ENV_EXTRAWHITE'><filename>BB_ENV_EXTRAWHITE</filename></link></para></listitem> |
| <listitem><para> |
| <link linkend='var-BITBAKE_UI'><filename>BITBAKE_UI</filename></link> |
| </para></listitem> |
| </itemizedlist> |
| You can find information on how to pass environment variables into the BitBake |
| execution environment in the |
| "<link linkend='passing-information-into-the-build-task-environment'>Passing Information Into the Build Task Environment</link>" section. |
| </para> |
| |
| <para> |
| The base configuration metadata is global |
| and therefore affects all recipes and tasks that are executed. |
| </para> |
| |
| <para> |
| BitBake first searches the current working directory for an |
| optional <filename>conf/bblayers.conf</filename> configuration file. |
| This file is expected to contain a |
| <link linkend='var-BBLAYERS'><filename>BBLAYERS</filename></link> |
| variable that is a space-delimited list of 'layer' directories. |
| Recall that if BitBake cannot find a <filename>bblayers.conf</filename> |
| file, then it is assumed the user has set the <filename>BBPATH</filename> |
| and <filename>BBFILES</filename> variables directly in the environment. |
| </para> |
| |
| <para> |
| For each directory (layer) in this list, a <filename>conf/layer.conf</filename> |
| file is located and parsed with the |
| <link linkend='var-LAYERDIR'><filename>LAYERDIR</filename></link> |
| variable being set to the directory where the layer was found. |
| The idea is these files automatically set up |
| <link linkend='var-BBPATH'><filename>BBPATH</filename></link> |
| and other variables correctly for a given build directory. |
| </para> |
| |
| <para> |
| BitBake then expects to find the <filename>conf/bitbake.conf</filename> |
| file somewhere in the user-specified <filename>BBPATH</filename>. |
| That configuration file generally has include directives to pull |
| in any other metadata such as files specific to the architecture, |
| the machine, the local environment, and so forth. |
| </para> |
| |
| <para> |
| Only variable definitions and include directives are allowed |
| in BitBake <filename>.conf</filename> files. |
| Some variables directly influence BitBake's behavior. |
| These variables might have been set from the environment |
| depending on the environment variables previously |
| mentioned or set in the configuration files. |
| The |
| "<link linkend='ref-variables-glos'>Variables Glossary</link>" |
| chapter presents a full list of variables. |
| </para> |
| |
| <para> |
| After parsing configuration files, BitBake uses its rudimentary |
| inheritance mechanism, which is through class files, to inherit |
| some standard classes. |
| BitBake parses a class when the inherit directive responsible |
| for getting that class is encountered. |
| </para> |
| |
| <para> |
| The <filename>base.bbclass</filename> file is always included. |
| Other classes that are specified in the configuration using the |
| <link linkend='var-INHERIT'><filename>INHERIT</filename></link> |
| variable are also included. |
| BitBake searches for class files in a |
| <filename>classes</filename> subdirectory under |
| the paths in <filename>BBPATH</filename> in the same way as |
| configuration files. |
| </para> |
| |
| <para> |
| A good way to get an idea of the configuration files and |
| the class files used in your execution environment is to |
| run the following BitBake command: |
| <literallayout class='monospaced'> |
| $ bitbake -e > mybb.log |
| </literallayout> |
| Examining the top of the <filename>mybb.log</filename> |
| shows you the many configuration files and class files |
| used in your execution environment. |
| </para> |
| |
| <note> |
| <para> |
| You need to be aware of how BitBake parses curly braces. |
| If a recipe uses a closing curly brace within the function and |
| the character has no leading spaces, BitBake produces a parsing |
| error. |
| If you use a pair of curly braces in a shell function, the |
| closing curly brace must not be located at the start of the line |
| without leading spaces. |
| </para> |
| |
| <para> |
| Here is an example that causes BitBake to produce a parsing |
| error: |
| <literallayout class='monospaced'> |
| fakeroot create_shar() { |
| cat << "EOF" > ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh |
| usage() |
| { |
| echo "test" |
| ###### The following "}" at the start of the line causes a parsing error ###### |
| } |
| EOF |
| } |
| </literallayout> |
| Writing the recipe this way avoids the error: |
| <literallayout class='monospaced'> |
| fakeroot create_shar() { |
| cat << "EOF" > ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh |
| usage() |
| { |
| echo "test" |
| ######The following "}" with a leading space at the start of the line avoids the error ###### |
| } |
| EOF |
| } |
| </literallayout> |
| </para> |
| </note> |
| </section> |
| |
| <section id='locating-and-parsing-recipes'> |
| <title>Locating and Parsing Recipes</title> |
| |
| <para> |
| During the configuration phase, BitBake will have set |
| <link linkend='var-BBFILES'><filename>BBFILES</filename></link>. |
| BitBake now uses it to construct a list of recipes to parse, |
| along with any append files (<filename>.bbappend</filename>) |
| to apply. |
| <filename>BBFILES</filename> is a space-separated list of |
| available files and supports wildcards. |
| An example would be: |
| <literallayout class='monospaced'> |
| BBFILES = "/path/to/bbfiles/*.bb /path/to/appends/*.bbappend" |
| </literallayout> |
| BitBake parses each recipe and append file located |
| with <filename>BBFILES</filename> and stores the values of |
| various variables into the datastore. |
| <note> |
| Append files are applied in the order they are encountered in |
| <filename>BBFILES</filename>. |
| </note> |
| For each file, a fresh copy of the base configuration is |
| made, then the recipe is parsed line by line. |
| Any inherit statements cause BitBake to find and |
| then parse class files (<filename>.bbclass</filename>) |
| using |
| <link linkend='var-BBPATH'><filename>BBPATH</filename></link> |
| as the search path. |
| Finally, BitBake parses in order any append files found in |
| <filename>BBFILES</filename>. |
| </para> |
| |
| <para> |
| One common convention is to use the recipe filename to define |
| pieces of metadata. |
| For example, in <filename>bitbake.conf</filename> the recipe |
| name and version are used to set the variables |
| <link linkend='var-PN'><filename>PN</filename></link> and |
| <link linkend='var-PV'><filename>PV</filename></link>: |
| <literallayout class='monospaced'> |
| PN = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0] or 'defaultpkgname'}" |
| PV = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[1] or '1.0'}" |
| </literallayout> |
| In this example, a recipe called "something_1.2.3.bb" would set |
| <filename>PN</filename> to "something" and |
| <filename>PV</filename> to "1.2.3". |
| </para> |
| |
| <para> |
| By the time parsing is complete for a recipe, BitBake |
| has a list of tasks that the recipe defines and a set of |
| data consisting of keys and values as well as |
| dependency information about the tasks. |
| </para> |
| |
| <para> |
| BitBake does not need all of this information. |
| It only needs a small subset of the information to make |
| decisions about the recipe. |
| Consequently, BitBake caches the values in which it is |
| interested and does not store the rest of the information. |
| Experience has shown it is faster to re-parse the metadata than to |
| try and write it out to the disk and then reload it. |
| </para> |
| |
| <para> |
| Where possible, subsequent BitBake commands reuse this cache of |
| recipe information. |
| The validity of this cache is determined by first computing a |
| checksum of the base configuration data (see |
| <link linkend='var-BB_HASHCONFIG_WHITELIST'><filename>BB_HASHCONFIG_WHITELIST</filename></link>) |
| and then checking if the checksum matches. |
| If that checksum matches what is in the cache and the recipe |
| and class files have not changed, Bitbake is able to use |
| the cache. |
| BitBake then reloads the cached information about the recipe |
| instead of reparsing it from scratch. |
| </para> |
| |
| <para> |
| Recipe file collections exist to allow the user to |
| have multiple repositories of |
| <filename>.bb</filename> files that contain the same |
| exact package. |
| For example, one could easily use them to make one's |
| own local copy of an upstream repository, but with |
| custom modifications that one does not want upstream. |
| Here is an example: |
| <literallayout class='monospaced'> |
| BBFILES = "/stuff/openembedded/*/*.bb /stuff/openembedded.modified/*/*.bb" |
| BBFILE_COLLECTIONS = "upstream local" |
| BBFILE_PATTERN_upstream = "^/stuff/openembedded/" |
| BBFILE_PATTERN_local = "^/stuff/openembedded.modified/" |
| BBFILE_PRIORITY_upstream = "5" |
| BBFILE_PRIORITY_local = "10" |
| </literallayout> |
| <note> |
| The layers mechanism is now the preferred method of collecting |
| code. |
| While the collections code remains, its main use is to set layer |
| priorities and to deal with overlap (conflicts) between layers. |
| </note> |
| </para> |
| </section> |
| |
| <section id='bb-bitbake-providers'> |
| <title>Providers</title> |
| |
| <para> |
| Assuming BitBake has been instructed to execute a target |
| and that all the recipe files have been parsed, BitBake |
| starts to figure out how to build the target. |
| BitBake looks through the <filename>PROVIDES</filename> list |
| for each of the recipes. |
| A <filename>PROVIDES</filename> list is the list of names by which |
| the recipe can be known. |
| Each recipe's <filename>PROVIDES</filename> list is created |
| implicitly through the recipe's |
| <link linkend='var-PN'><filename>PN</filename></link> variable |
| and explicitly through the recipe's |
| <link linkend='var-PROVIDES'><filename>PROVIDES</filename></link> |
| variable, which is optional. |
| </para> |
| |
| <para> |
| When a recipe uses <filename>PROVIDES</filename>, that recipe's |
| functionality can be found under an alternative name or names other |
| than the implicit <filename>PN</filename> name. |
| As an example, suppose a recipe named <filename>keyboard_1.0.bb</filename> |
| contained the following: |
| <literallayout class='monospaced'> |
| PROVIDES += "fullkeyboard" |
| </literallayout> |
| The <filename>PROVIDES</filename> list for this recipe becomes |
| "keyboard", which is implicit, and "fullkeyboard", which is explicit. |
| Consequently, the functionality found in |
| <filename>keyboard_1.0.bb</filename> can be found under two |
| different names. |
| </para> |
| </section> |
| |
| <section id='bb-bitbake-preferences'> |
| <title>Preferences</title> |
| |
| <para> |
| The <filename>PROVIDES</filename> list is only part of the solution |
| for figuring out a target's recipes. |
| Because targets might have multiple providers, BitBake needs |
| to prioritize providers by determining provider preferences. |
| </para> |
| |
| <para> |
| A common example in which a target has multiple providers |
| is "virtual/kernel", which is on the |
| <filename>PROVIDES</filename> list for each kernel recipe. |
| Each machine often selects the best kernel provider by using a |
| line similar to the following in the machine configuration file: |
| <literallayout class='monospaced'> |
| PREFERRED_PROVIDER_virtual/kernel = "linux-yocto" |
| </literallayout> |
| The default |
| <link linkend='var-PREFERRED_PROVIDER'><filename>PREFERRED_PROVIDER</filename></link> |
| is the provider with the same name as the target. |
| Bitbake iterates through each target it needs to build and |
| resolves them and their dependencies using this process. |
| </para> |
| |
| <para> |
| Understanding how providers are chosen is made complicated by the fact |
| that multiple versions might exist for a given provider. |
| BitBake defaults to the highest version of a provider. |
| Version comparisons are made using the same method as Debian. |
| You can use the |
| <link linkend='var-PREFERRED_VERSION'><filename>PREFERRED_VERSION</filename></link> |
| variable to specify a particular version. |
| You can influence the order by using the |
| <link linkend='var-DEFAULT_PREFERENCE'><filename>DEFAULT_PREFERENCE</filename></link> |
| variable. |
| </para> |
| |
| <para> |
| By default, files have a preference of "0". |
| Setting <filename>DEFAULT_PREFERENCE</filename> to "-1" makes the |
| recipe unlikely to be used unless it is explicitly referenced. |
| Setting <filename>DEFAULT_PREFERENCE</filename> to "1" makes it |
| likely the recipe is used. |
| <filename>PREFERRED_VERSION</filename> overrides any |
| <filename>DEFAULT_PREFERENCE</filename> setting. |
| <filename>DEFAULT_PREFERENCE</filename> is often used to mark newer |
| and more experimental recipe versions until they have undergone |
| sufficient testing to be considered stable. |
| </para> |
| |
| <para> |
| When there are multiple “versions” of a given recipe, |
| BitBake defaults to selecting the most recent |
| version, unless otherwise specified. |
| If the recipe in question has a |
| <link linkend='var-DEFAULT_PREFERENCE'><filename>DEFAULT_PREFERENCE</filename></link> |
| set lower than the other recipes (default is 0), then |
| it will not be selected. |
| This allows the person or persons maintaining |
| the repository of recipe files to specify |
| their preference for the default selected version. |
| Additionally, the user can specify their preferred version. |
| </para> |
| |
| <para> |
| If the first recipe is named <filename>a_1.1.bb</filename>, then the |
| <link linkend='var-PN'><filename>PN</filename></link> variable |
| will be set to “a”, and the |
| <link linkend='var-PV'><filename>PV</filename></link> |
| variable will be set to 1.1. |
| </para> |
| |
| <para> |
| Thus, if a recipe named <filename>a_1.2.bb</filename> exists, BitBake |
| will choose 1.2 by default. |
| However, if you define the following variable in a |
| <filename>.conf</filename> file that BitBake parses, you |
| can change that preference: |
| <literallayout class='monospaced'> |
| PREFERRED_VERSION_a = "1.1" |
| </literallayout> |
| </para> |
| |
| <note> |
| <para> |
| It is common for a recipe to provide two versions -- a stable, |
| numbered (and preferred) version, and a version that is |
| automatically checked out from a source code repository that |
| is considered more "bleeding edge" but can be selected only |
| explicitly. |
| </para> |
| |
| <para> |
| For example, in the OpenEmbedded codebase, there is a standard, |
| versioned recipe file for BusyBox, |
| <filename>busybox_1.22.1.bb</filename>, |
| but there is also a Git-based version, |
| <filename>busybox_git.bb</filename>, which explicitly contains the line |
| <literallayout class='monospaced'> |
| DEFAULT_PREFERENCE = "-1" |
| </literallayout> |
| to ensure that the numbered, stable version is always preferred |
| unless the developer selects otherwise. |
| </para> |
| </note> |
| </section> |
| |
| <section id='bb-bitbake-dependencies'> |
| <title>Dependencies</title> |
| |
| <para> |
| Each target BitBake builds consists of multiple tasks such as |
| <filename>fetch</filename>, <filename>unpack</filename>, |
| <filename>patch</filename>, <filename>configure</filename>, |
| and <filename>compile</filename>. |
| For best performance on multi-core systems, BitBake considers each |
| task as an independent |
| entity with its own set of dependencies. |
| </para> |
| |
| <para> |
| Dependencies are defined through several variables. |
| You can find information about variables BitBake uses in |
| the <link linkend='ref-variables-glos'>Variables Glossary</link> |
| near the end of this manual. |
| At a basic level, it is sufficient to know that BitBake uses the |
| <link linkend='var-DEPENDS'><filename>DEPENDS</filename></link> and |
| <link linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link> variables when |
| calculating dependencies. |
| </para> |
| |
| <para> |
| For more information on how BitBake handles dependencies, see the |
| "<link linkend='dependencies'>Dependencies</link>" section. |
| </para> |
| </section> |
| |
| <section id='ref-bitbake-tasklist'> |
| <title>The Task List</title> |
| |
| <para> |
| Based on the generated list of providers and the dependency information, |
| BitBake can now calculate exactly what tasks it needs to run and in what |
| order it needs to run them. |
| The |
| "<link linkend='executing-tasks'>Executing Tasks</link>" section has more |
| information on how BitBake chooses which task to execute next. |
| </para> |
| |
| <para> |
| The build now starts with BitBake forking off threads up to the limit set in the |
| <link linkend='var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link> |
| variable. |
| BitBake continues to fork threads as long as there are tasks ready to run, |
| those tasks have all their dependencies met, and the thread threshold has not been |
| exceeded. |
| </para> |
| |
| <para> |
| It is worth noting that you can greatly speed up the build time by properly setting |
| the <filename>BB_NUMBER_THREADS</filename> variable. |
| </para> |
| |
| <para> |
| As each task completes, a timestamp is written to the directory specified by the |
| <link linkend='var-STAMP'><filename>STAMP</filename></link> variable. |
| On subsequent runs, BitBake looks in the build directory within |
| <filename>tmp/stamps</filename> and does not rerun |
| tasks that are already completed unless a timestamp is found to be invalid. |
| Currently, invalid timestamps are only considered on a per |
| recipe file basis. |
| So, for example, if the configure stamp has a timestamp greater than the |
| compile timestamp for a given target, then the compile task would rerun. |
| Running the compile task again, however, has no effect on other providers |
| that depend on that target. |
| </para> |
| |
| <para> |
| The exact format of the stamps is partly configurable. |
| In modern versions of BitBake, a hash is appended to the |
| stamp so that if the configuration changes, the stamp becomes |
| invalid and the task is automatically rerun. |
| This hash, or signature used, is governed by the signature policy |
| that is configured (see the |
| "<link linkend='checksums'>Checksums (Signatures)</link>" |
| section for information). |
| It is also possible to append extra metadata to the stamp using |
| the "stamp-extra-info" task flag. |
| For example, OpenEmbedded uses this flag to make some tasks machine-specific. |
| </para> |
| |
| <note> |
| Some tasks are marked as "nostamp" tasks. |
| No timestamp file is created when these tasks are run. |
| Consequently, "nostamp" tasks are always rerun. |
| </note> |
| |
| <para> |
| For more information on tasks, see the |
| "<link linkend='tasks'>Tasks</link>" section. |
| </para> |
| </section> |
| |
| <section id='executing-tasks'> |
| <title>Executing Tasks</title> |
| |
| <para> |
| Tasks can be either a shell task or a Python task. |
| For shell tasks, BitBake writes a shell script to |
| <filename>${</filename><link linkend='var-T'><filename>T</filename></link><filename>}/run.do_taskname.pid</filename> |
| and then executes the script. |
| The generated shell script contains all the exported variables, |
| and the shell functions with all variables expanded. |
| Output from the shell script goes to the file |
| <filename>${T}/log.do_taskname.pid</filename>. |
| Looking at the expanded shell functions in the run file and |
| the output in the log files is a useful debugging technique. |
| </para> |
| |
| <para> |
| For Python tasks, BitBake executes the task internally and logs |
| information to the controlling terminal. |
| Future versions of BitBake will write the functions to files |
| similar to the way shell tasks are handled. |
| Logging will be handled in a way similar to shell tasks as well. |
| </para> |
| |
| <para> |
| The order in which BitBake runs the tasks is controlled by its |
| task scheduler. |
| It is possible to configure the scheduler and define custom |
| implementations for specific use cases. |
| For more information, see these variables that control the |
| behavior: |
| <itemizedlist> |
| <listitem><para> |
| <link linkend='var-BB_SCHEDULER'><filename>BB_SCHEDULER</filename></link> |
| </para></listitem> |
| <listitem><para> |
| <link linkend='var-BB_SCHEDULERS'><filename>BB_SCHEDULERS</filename></link> |
| </para></listitem> |
| </itemizedlist> |
| It is possible to have functions run before and after a task's main |
| function. |
| This is done using the "prefuncs" and "postfuncs" flags of the task |
| that lists the functions to run. |
| </para> |
| </section> |
| |
| <section id='checksums'> |
| <title>Checksums (Signatures)</title> |
| |
| <para> |
| A checksum is a unique signature of a task's inputs. |
| The signature of a task can be used to determine if a task |
| needs to be run. |
| Because it is a change in a task's inputs that triggers running |
| the task, BitBake needs to detect all the inputs to a given task. |
| For shell tasks, this turns out to be fairly easy because |
| BitBake generates a "run" shell script for each task and |
| it is possible to create a checksum that gives you a good idea of when |
| the task's data changes. |
| </para> |
| |
| <para> |
| To complicate the problem, some things should not be included in |
| the checksum. |
| First, there is the actual specific build path of a given task - |
| the working directory. |
| It does not matter if the working directory changes because it should not |
| affect the output for target packages. |
| The simplistic approach for excluding the working directory is to set |
| it to some fixed value and create the checksum for the "run" script. |
| BitBake goes one step better and uses the |
| <link linkend='var-BB_HASHBASE_WHITELIST'><filename>BB_HASHBASE_WHITELIST</filename></link> |
| variable to define a list of variables that should never be included |
| when generating the signatures. |
| </para> |
| |
| <para> |
| Another problem results from the "run" scripts containing functions that |
| might or might not get called. |
| The incremental build solution contains code that figures out dependencies |
| between shell functions. |
| This code is used to prune the "run" scripts down to the minimum set, |
| thereby alleviating this problem and making the "run" scripts much more |
| readable as a bonus. |
| </para> |
| |
| <para> |
| So far we have solutions for shell scripts. |
| What about Python tasks? |
| The same approach applies even though these tasks are more difficult. |
| The process needs to figure out what variables a Python function accesses |
| and what functions it calls. |
| Again, the incremental build solution contains code that first figures out |
| the variable and function dependencies, and then creates a checksum for the data |
| used as the input to the task. |
| </para> |
| |
| <para> |
| Like the working directory case, situations exist where dependencies |
| should be ignored. |
| For these cases, you can instruct the build process to ignore a dependency |
| by using a line like the following: |
| <literallayout class='monospaced'> |
| PACKAGE_ARCHS[vardepsexclude] = "MACHINE" |
| </literallayout> |
| This example ensures that the <filename>PACKAGE_ARCHS</filename> variable does not |
| depend on the value of <filename>MACHINE</filename>, even if it does reference it. |
| </para> |
| |
| <para> |
| Equally, there are cases where we need to add dependencies BitBake |
| is not able to find. |
| You can accomplish this by using a line like the following: |
| <literallayout class='monospaced'> |
| PACKAGE_ARCHS[vardeps] = "MACHINE" |
| </literallayout> |
| This example explicitly adds the <filename>MACHINE</filename> variable as a |
| dependency for <filename>PACKAGE_ARCHS</filename>. |
| </para> |
| |
| <para> |
| Consider a case with in-line Python, for example, where BitBake is not |
| able to figure out dependencies. |
| When running in debug mode (i.e. using <filename>-DDD</filename>), BitBake |
| produces output when it discovers something for which it cannot figure out |
| dependencies. |
| </para> |
| |
| <para> |
| Thus far, this section has limited discussion to the direct inputs into a task. |
| Information based on direct inputs is referred to as the "basehash" in the |
| code. |
| However, there is still the question of a task's indirect inputs - the |
| things that were already built and present in the build directory. |
| The checksum (or signature) for a particular task needs to add the hashes |
| of all the tasks on which the particular task depends. |
| Choosing which dependencies to add is a policy decision. |
| However, the effect is to generate a master checksum that combines the basehash |
| and the hashes of the task's dependencies. |
| </para> |
| |
| <para> |
| At the code level, there are a variety of ways both the basehash and the |
| dependent task hashes can be influenced. |
| Within the BitBake configuration file, we can give BitBake some extra information |
| to help it construct the basehash. |
| The following statement effectively results in a list of global variable |
| dependency excludes - variables never included in any checksum. |
| This example uses variables from OpenEmbedded to help illustrate |
| the concept: |
| <literallayout class='monospaced'> |
| BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \ |
| SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \ |
| USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \ |
| PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \ |
| CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX" |
| </literallayout> |
| The previous example excludes the work directory, which is part of |
| <filename>TMPDIR</filename>. |
| </para> |
| |
| <para> |
| The rules for deciding which hashes of dependent tasks to include through |
| dependency chains are more complex and are generally accomplished with a |
| Python function. |
| The code in <filename>meta/lib/oe/sstatesig.py</filename> shows two examples |
| of this and also illustrates how you can insert your own policy into the system |
| if so desired. |
| This file defines the two basic signature generators OpenEmbedded Core |
| uses: "OEBasic" and "OEBasicHash". |
| By default, there is a dummy "noop" signature handler enabled in BitBake. |
| This means that behavior is unchanged from previous versions. |
| <filename>OE-Core</filename> uses the "OEBasicHash" signature handler by default |
| through this setting in the <filename>bitbake.conf</filename> file: |
| <literallayout class='monospaced'> |
| BB_SIGNATURE_HANDLER ?= "OEBasicHash" |
| </literallayout> |
| The "OEBasicHash" <filename>BB_SIGNATURE_HANDLER</filename> is the same as the |
| "OEBasic" version but adds the task hash to the stamp files. |
| This results in any metadata change that changes the task hash, automatically |
| causing the task to be run again. |
| This removes the need to bump |
| <link linkend='var-PR'><filename>PR</filename></link> |
| values, and changes to metadata automatically ripple across the build. |
| </para> |
| |
| <para> |
| It is also worth noting that the end result of these signature generators is to |
| make some dependency and hash information available to the build. |
| This information includes: |
| <itemizedlist> |
| <listitem><para><filename>BB_BASEHASH_task-</filename><replaceable>taskname</replaceable>: |
| The base hashes for each task in the recipe. |
| </para></listitem> |
| <listitem><para><filename>BB_BASEHASH_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>: |
| The base hashes for each dependent task. |
| </para></listitem> |
| <listitem><para><filename>BBHASHDEPS_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>: |
| The task dependencies for each task. |
| </para></listitem> |
| <listitem><para><filename>BB_TASKHASH</filename>: |
| The hash of the currently running task. |
| </para></listitem> |
| </itemizedlist> |
| </para> |
| |
| <para> |
| It is worth noting that BitBake's "-S" option lets you |
| debug Bitbake's processing of signatures. |
| The options passed to -S allow different debugging modes |
| to be used, either using BitBake's own debug functions |
| or possibly those defined in the metadata/signature handler |
| itself. |
| The simplest parameter to pass is "none", which causes a |
| set of signature information to be written out into |
| <filename>STAMP_DIR</filename> |
| corresponding to the targets specified. |
| The other currently available parameter is "printdiff", |
| which causes BitBake to try to establish the closest |
| signature match it can (e.g. in the sstate cache) and then |
| run <filename>bitbake-diffsigs</filename> over the matches |
| to determine the stamps and delta where these two |
| stamp trees diverge. |
| <note> |
| It is likely that future versions of BitBake will |
| provide other signature handlers triggered through |
| additional "-S" parameters. |
| </note> |
| </para> |
| |
| <para> |
| You can find more information on checksum metadata in the |
| "<link linkend='task-checksums-and-setscene'>Task Checksums and Setscene</link>" |
| section. |
| </para> |
| </section> |
| |
| <section id='setscene'> |
| <title>Setscene</title> |
| |
| <para> |
| The setscene process enables BitBake to handle "pre-built" artifacts. |
| The ability to handle and reuse these artifacts allows BitBake |
| the luxury of not having to build something from scratch every time. |
| Instead, BitBake can use, when possible, existing build artifacts. |
| </para> |
| |
| <para> |
| BitBake needs to have reliable data indicating whether or not an |
| artifact is compatible. |
| Signatures, described in the previous section, provide an ideal |
| way of representing whether an artifact is compatible. |
| If a signature is the same, an object can be reused. |
| </para> |
| |
| <para> |
| If an object can be reused, the problem then becomes how to |
| replace a given task or set of tasks with the pre-built artifact. |
| BitBake solves the problem with the "setscene" process. |
| </para> |
| |
| <para> |
| When BitBake is asked to build a given target, before building anything, |
| it first asks whether cached information is available for any of the |
| targets it's building, or any of the intermediate targets. |
| If cached information is available, BitBake uses this information instead of |
| running the main tasks. |
| </para> |
| |
| <para> |
| BitBake first calls the function defined by the |
| <link linkend='var-BB_HASHCHECK_FUNCTION'><filename>BB_HASHCHECK_FUNCTION</filename></link> |
| variable with a list of tasks and corresponding |
| hashes it wants to build. |
| This function is designed to be fast and returns a list |
| of the tasks for which it believes in can obtain artifacts. |
| </para> |
| |
| <para> |
| Next, for each of the tasks that were returned as possibilities, |
| BitBake executes a setscene version of the task that the possible |
| artifact covers. |
| Setscene versions of a task have the string "_setscene" appended to the |
| task name. |
| So, for example, the task with the name <filename>xxx</filename> has |
| a setscene task named <filename>xxx_setscene</filename>. |
| The setscene version of the task executes and provides the necessary |
| artifacts returning either success or failure. |
| </para> |
| |
| <para> |
| As previously mentioned, an artifact can cover more than one task. |
| For example, it is pointless to obtain a compiler if you |
| already have the compiled binary. |
| To handle this, BitBake calls the |
| <link linkend='var-BB_SETSCENE_DEPVALID'><filename>BB_SETSCENE_DEPVALID</filename></link> |
| function for each successful setscene task to know whether or not it needs |
| to obtain the dependencies of that task. |
| </para> |
| |
| <para> |
| Finally, after all the setscene tasks have executed, BitBake calls the |
| function listed in |
| <link linkend='var-BB_SETSCENE_VERIFY_FUNCTION'><filename>BB_SETSCENE_VERIFY_FUNCTION</filename></link> |
| with the list of tasks BitBake thinks has been "covered". |
| The metadata can then ensure that this list is correct and can |
| inform BitBake that it wants specific tasks to be run regardless |
| of the setscene result. |
| </para> |
| |
| <para> |
| You can find more information on setscene metadata in the |
| "<link linkend='task-checksums-and-setscene'>Task Checksums and Setscene</link>" |
| section. |
| </para> |
| </section> |
| </chapter> |