blob: 995c2fa7bf48d1047b3088436848088e920ee59a [file] [log] [blame]
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
<chapter id="bitbake-user-manual-intro">
Welcome to the BitBake User Manual.
This manual provides information on the BitBake tool.
The information attempts to be as independent as possible regarding
systems that use BitBake, such as OpenEmbedded and the
Yocto Project.
In some cases, scenarios or examples within the context of
a build system are used in the manual to help with understanding.
For these cases, the manual clearly states the context.
<section id="intro">
Fundamentally, BitBake is a generic task execution
engine that allows shell and Python tasks to be run
efficiently and in parallel while working within
complex inter-task dependency constraints.
One of BitBake's main users, OpenEmbedded, takes this core
and builds embedded Linux software stacks using
a task-oriented approach.
Conceptually, BitBake is similar to GNU Make in
some regards but has significant differences:
BitBake executes tasks according to provided
metadata that builds up the tasks.
Metadata is stored in recipe (<filename>.bb</filename>)
and related recipe "append" (<filename>.bbappend</filename>)
files, configuration (<filename>.conf</filename>) and
underlying include (<filename>.inc</filename>) files, and
in class (<filename>.bbclass</filename>) files.
The metadata provides
BitBake with instructions on what tasks to run and
the dependencies between those tasks.
BitBake includes a fetcher library for obtaining source
code from various places such as local files, source control
systems, or websites.
The instructions for each unit to be built (e.g. a piece
of software) are known as "recipe" files and
contain all the information about the unit
(dependencies, source file locations, checksums, description
and so on).
BitBake includes a client/server abstraction and can
be used from a command line or used as a service over
XML-RPC and has several different user interfaces.
<section id="history-and-goals">
<title>History and Goals</title>
BitBake was originally a part of the OpenEmbedded project.
It was inspired by the Portage package management system
used by the Gentoo Linux distribution.
On December 7, 2004, OpenEmbedded project team member
Chris Larson split the project into two distinct pieces:
<listitem><para>BitBake, a generic task executor</para></listitem>
<listitem><para>OpenEmbedded, a metadata set utilized by
Today, BitBake is the primary basis of the
<ulink url="">OpenEmbedded</ulink>
project, which is being used to build and maintain Linux
distributions such as the
<ulink url=''>Angstrom Distribution</ulink>,
and which is also being used as the build tool for Linux projects
such as the
<ulink url=''>Yocto Project</ulink>.
Prior to BitBake, no other build tool adequately met the needs of
an aspiring embedded Linux distribution.
All of the build systems used by traditional desktop Linux
distributions lacked important functionality, and none of the
ad hoc Buildroot-based systems, prevalent in the
embedded space, were scalable or maintainable.
Some important original goals for BitBake were:
Handle cross-compilation.
Handle inter-package dependencies (build time on
target architecture, build time on native
architecture, and runtime).
Support running any number of tasks within a given
package, including, but not limited to, fetching
upstream sources, unpacking them, patching them,
configuring them, and so forth.
Be Linux distribution agnostic for both build and
target systems.
Be architecture agnostic.
Support multiple build and target operating systems
(e.g. Cygwin, the BSDs, and so forth).
Be self-contained, rather than tightly
integrated into the build machine's root
Handle conditional metadata on the target architecture,
operating system, distribution, and machine.
Be easy to use the tools to supply local metadata and packages
against which to operate.
Be easy to use BitBake to collaborate between multiple
projects for their builds.
Provide an inheritance mechanism to share
common metadata between many packages.
Over time it became apparent that some further requirements
were necessary:
Handle variants of a base recipe (e.g. native, sdk,
and multilib).
Split metadata into layers and allow layers
to enhance or override other layers.
Allow representation of a given set of input variables
to a task as a checksum.
Based on that checksum, allow acceleration of builds
with prebuilt components.
BitBake satisfies all the original requirements and many more
with extensions being made to the basic functionality to
reflect the additional requirements.
Flexibility and power have always been the priorities.
BitBake is highly extensible and supports embedded Python code and
execution of any arbitrary tasks.
<section id="Concepts">
BitBake is a program written in the Python language.
At the highest level, BitBake interprets metadata, decides
what tasks are required to run, and executes those tasks.
Similar to GNU Make, BitBake controls how software is
GNU Make achieves its control through "makefiles", while
BitBake uses "recipes".
BitBake extends the capabilities of a simple
tool like GNU Make by allowing for the definition of much more
complex tasks, such as assembling entire embedded Linux
The remainder of this section introduces several concepts
that should be understood in order to better leverage
the power of BitBake.
<section id='recipes'>
BitBake Recipes, which are denoted by the file extension
<filename>.bb</filename>, are the most basic metadata files.
These recipe files provide BitBake with the following:
<listitem><para>Descriptive information about the
package (author, homepage, license, and so on)</para></listitem>
<listitem><para>The version of the recipe</para></listitem>
<listitem><para>Existing dependencies (both build
and runtime dependencies)</para></listitem>
<listitem><para>Where the source code resides and
how to fetch it</para></listitem>
<listitem><para>Whether the source code requires
any patches, where to find them, and how to apply
<listitem><para>How to configure and compile the
source code</para></listitem>
<listitem><para>How to assemble the generated artifacts into
one or more installable packages</para></listitem>
<listitem><para>Where on the target machine to install the
package or packages created</para></listitem>
Within the context of BitBake, or any project utilizing BitBake
as its build system, files with the <filename>.bb</filename>
extension are referred to as <firstterm>recipes</firstterm>.
The term "package" is also commonly used to describe recipes.
However, since the same word is used to describe packaged
output from a project, it is best to maintain a single
descriptive term - "recipes".
Put another way, a single "recipe" file is quite capable
of generating a number of related but separately installable
In fact, that ability is fairly common.
<section id='configuration-files'>
<title>Configuration Files</title>
Configuration files, which are denoted by the
<filename>.conf</filename> extension, define
various configuration variables that govern the project's build
These files fall into several areas that define
machine configuration, distribution configuration,
possible compiler tuning, general common
configuration, and user configuration.
The main configuration file is the sample
<filename>bitbake.conf</filename> file, which is
located within the BitBake source tree
<filename>conf</filename> directory.
<section id='classes'>
Class files, which are denoted by the
<filename>.bbclass</filename> extension, contain
information that is useful to share between metadata files.
The BitBake source tree currently comes with one class metadata file
called <filename>base.bbclass</filename>.
You can find this file in the
<filename>classes</filename> directory.
The <filename>base.bbclass</filename> class files is special since it
is always included automatically for all recipes
and classes.
This class contains definitions for standard basic tasks such
as fetching, unpacking, configuring (empty by default),
compiling (runs any Makefile present), installing (empty by
default) and packaging (empty by default).
These tasks are often overridden or extended by other classes
added during the project development process.
<section id='layers'>
Layers allow you to isolate different types of
customizations from each other.
While you might find it tempting to keep everything in one layer
when working on a single project, the more modular
your metadata, the easier it is to cope with future changes.
To illustrate how you can use layers to keep things modular,
consider customizations you might make to support a specific target machine.
These types of customizations typically reside in a special layer,
rather than a general layer, called a <firstterm>Board Support Package</firstterm> (BSP)
Furthermore, the machine customizations should be isolated from
recipes and metadata that support a new GUI environment, for
This situation gives you a couple of layers: one for the machine
configurations and one for the GUI environment.
It is important to understand, however, that the BSP layer can still
make machine-specific additions to recipes within
the GUI environment layer without polluting the GUI layer itself
with those machine-specific changes.
You can accomplish this through a recipe that is a BitBake append
(<filename>.bbappend</filename>) file.
<section id='append-bbappend-files'>
<title>Append Files</title>
Append files, which are files that have the
<filename>.bbappend</filename> file extension, extend or
override information in an existing recipe file.
BitBake expects every append file to have a corresponding recipe file.
Furthermore, the append file and corresponding recipe file
must use the same root filename.
The filenames can differ only in the file type suffix used
(e.g. <filename></filename> and
Information in append files extends or
overrides the information in the underlying,
similarly-named recipe files.
When you name an append file, you can use the
"<filename>%</filename>" wildcard character to allow for matching
recipe names.
For example, suppose you have an append file named
as follows:
<literallayout class='monospaced'>
That append file would match any <filename>busybox_1.21.</filename><replaceable>x</replaceable><filename>.bb</filename>
version of the recipe.
So, the append file would match the following recipe names:
<literallayout class='monospaced'>
The use of the "<filename>%</filename>" character
is limited in that it only works directly in front of the
<filename>.bbappend</filename> portion of the append file's
You cannot use the wildcard character in any other
location of the name.
If the <filename>busybox</filename> recipe was updated to
<filename></filename>, the append name would not
However, if you named the append file
<filename>busybox_1.%.bbappend</filename>, then you would have a match.
In the most general case, you could name the append file something as
simple as <filename>busybox_%.bbappend</filename> to be entirely
version independent.
<section id='obtaining-bitbake'>
<title>Obtaining BitBake</title>
You can obtain BitBake several different ways:
<listitem><para><emphasis>Cloning BitBake:</emphasis>
Using Git to clone the BitBake source code repository
is the recommended method for obtaining BitBake.
Cloning the repository makes it easy to get bug fixes
and have access to stable branches and the master
Once you have cloned BitBake, you should use
the latest stable
branch for development since the master branch is for
BitBake development and might contain less stable changes.
<para>You usually need a version of BitBake
that matches the metadata you are using.
The metadata is generally backwards compatible but
not forward compatible.</para>
<para>Here is an example that clones the BitBake repository:
<literallayout class='monospaced'>
$ git clone git://
This command clones the BitBake Git repository into a
directory called <filename>bitbake</filename>.
Alternatively, you can
designate a directory after the
<filename>git clone</filename> command
if you want to call the new directory something
other than <filename>bitbake</filename>.
Here is an example that names the directory
<literallayout class='monospaced'>
$ git clone git:// bbdev
<listitem><para><emphasis>Installation using your Distribution
Package Management System:</emphasis>
This method is not
recommended because the BitBake version that is
provided by your distribution, in most cases,
is several
releases behind a snapshot of the BitBake repository.
<listitem><para><emphasis>Taking a snapshot of BitBake:</emphasis>
Downloading a snapshot of BitBake from the
source code repository gives you access to a known
branch or release of BitBake.
Cloning the Git repository, as described earlier,
is the preferred method for getting BitBake.
Cloning the repository makes it easier to update as
patches are added to the stable branches.
<para>The following example downloads a snapshot of
BitBake version 1.17.0:
<literallayout class='monospaced'>
$ wget
$ tar zxpvf bitbake-1.17.0.tar.gz
After extraction of the tarball using the tar utility,
you have a directory entitled
<listitem><para><emphasis>Using the BitBake that Comes With Your
Build Checkout:</emphasis>
A final possibility for getting a copy of BitBake is that it
already comes with your checkout of a larger BitBake-based build
system, such as Poky.
Rather than manually checking out individual layers and
gluing them together yourself, you can check
out an entire build system.
The checkout will already include a version of BitBake that
has been thoroughly tested for compatibility with the other
For information on how to check out a particular BitBake-based
build system, consult that build system's supporting documentation.
<section id="bitbake-user-manual-command">
<title>The BitBake Command</title>
The <filename>bitbake</filename> command is the primary interface
to the BitBake tool.
This section presents the BitBake command syntax and provides
several execution examples.
<section id='usage-and-syntax'>
<title>Usage and syntax</title>
Following is the usage and syntax for BitBake:
<literallayout class='monospaced'>
$ bitbake -h
Usage: bitbake [options] [recipename/target recipe:do_task ...]
Executes the specified task (default is 'build') for a given set of target recipes (.bb files).
It is assumed there is a conf/bblayers.conf available in cwd or in BBPATH which
will provide the layer, BBFILES and other configuration information.
--version show program's version number and exit
-h, --help show this help message and exit
Execute tasks from a specific .bb recipe directly.
WARNING: Does not handle any dependencies from other
-k, --continue Continue as much as possible after an error. While the
target that failed and anything depending on it cannot
be built, as much as possible will be built before
-f, --force Force the specified targets/task to run (invalidating
any existing stamp file).
-c CMD, --cmd=CMD Specify the task to execute. The exact options
available depend on the metadata. Some examples might
be 'compile' or 'populate_sysroot' or 'listtasks' may
give a list of the tasks available.
Invalidate the stamp for the specified task such as
'compile' and then run the default task for the
specified target(s).
Read the specified file before bitbake.conf.
Read the specified file after bitbake.conf.
-v, --verbose Enable tracing of shell tasks (with 'set -x'). Also
print bb.note(...) messages to stdout (in addition to
writing them to ${T}/log.do_&lt;task&gt;).
-D, --debug Increase the debug level. You can specify this more
than once. -D sets the debug level to 1, where only
bb.debug(1, ...) messages are printed to stdout; -DD
sets the debug level to 2, where both bb.debug(1, ...)
and bb.debug(2, ...) messages are printed; etc.
Without -D, no debug messages are printed. Note that
-D only affects output to stdout. All debug messages
are written to ${T}/log.do_taskname, regardless of the
debug level.
-q, --quiet Output less log message data to the terminal. You can
specify this more than once.
-n, --dry-run Don't execute, just go through the motions.
Dump out the signature construction information, with
no task execution. The SIGNATURE_HANDLER parameter is
passed to the handler. Two common values are none and
printdiff but the handler may define more/less. none
means only dump the signature, printdiff means compare
the dumped signature with the cached one.
-p, --parse-only Quit after parsing the BB recipes.
-s, --show-versions Show current and preferred versions of all recipes.
-e, --environment Show the global or per-recipe environment complete
with information about where variables were
-g, --graphviz Save dependency tree information for the specified
targets in the dot syntax.
Assume these dependencies don't exist and are already
provided (equivalent to ASSUME_PROVIDED). Useful to
make dependency graphs more appealing
Show debug logging for the specified logging domains
-P, --profile Profile the command and save reports.
-u UI, --ui=UI The user interface to use (knotty, ncurses or taskexp
- default knotty).
--token=XMLRPCTOKEN Specify the connection token to be used when
connecting to a remote server.
--revisions-changed Set the exit code depending on whether upstream
floating revisions have changed or not.
--server-only Run bitbake without a UI, only starting a server
(cooker) process.
-B BIND, --bind=BIND The name/address for the bitbake xmlrpc server to bind
Set timeout to unload bitbake server due to
inactivity, set to -1 means no unload, default:
Environment variable BB_SERVER_TIMEOUT.
--no-setscene Do not run any setscene tasks. sstate will be ignored
and everything needed, built.
--setscene-only Only run setscene tasks, don't run any real tasks.
Connect to the specified server.
-m, --kill-server Terminate any running bitbake server.
--observe-only Connect to a server as an observing-only client.
--status-only Check the status of the remote bitbake server.
Writes the event log of the build to a bitbake event
json file. Use '' (empty string) to assign the name
--runall=RUNALL Run the specified task for any recipe in the taskgraph
of the specified target (even if it wouldn't otherwise
have run).
--runonly=RUNONLY Run only the specified task within the taskgraph of
the specified targets (and any task dependencies those
tasks may have).
<section id='bitbake-examples'>
This section presents some examples showing how to use BitBake.
<section id='example-executing-a-task-against-a-single-recipe'>
<title>Executing a Task Against a Single Recipe</title>
Executing tasks for a single recipe file is relatively simple.
You specify the file in question, and BitBake parses
it and executes the specified task.
If you do not specify a task, BitBake executes the default
task, which is "build”.
BitBake obeys inter-task dependencies when doing
The following command runs the build task, which is
the default task, on the <filename></filename>
recipe file:
<literallayout class='monospaced'>
$ bitbake -b
The following command runs the clean task on the
<filename></filename> recipe file:
<literallayout class='monospaced'>
$ bitbake -b -c clean
The "-b" option explicitly does not handle recipe
Other than for debugging purposes, it is instead
recommended that you use the syntax presented in the
next section.
<section id='executing-tasks-against-a-set-of-recipe-files'>
<title>Executing Tasks Against a Set of Recipe Files</title>
There are a number of additional complexities introduced
when one wants to manage multiple <filename>.bb</filename>
Clearly there needs to be a way to tell BitBake what
files are available and, of those, which you
want to execute.
There also needs to be a way for each recipe
to express its dependencies, both for build-time and
There must be a way for you to express recipe preferences
when multiple recipes provide the same functionality, or when
there are multiple versions of a recipe.
The <filename>bitbake</filename> command, when not using
"--buildfile" or "-b" only accepts a "PROVIDES".
You cannot provide anything else.
By default, a recipe file generally "PROVIDES" its
"packagename" as shown in the following example:
<literallayout class='monospaced'>
$ bitbake foo
This next example "PROVIDES" the package name and also uses
the "-c" option to tell BitBake to just execute the
<filename>do_clean</filename> task:
<literallayout class='monospaced'>
$ bitbake -c clean foo
<section id='executing-a-list-of-task-and-recipe-combinations'>
<title>Executing a List of Task and Recipe Combinations</title>
The BitBake command line supports specifying different
tasks for individual targets when you specify multiple
For example, suppose you had two targets (or recipes)
<filename>myfirstrecipe</filename> and
<filename>mysecondrecipe</filename> and you needed
BitBake to run <filename>taskA</filename> for the first
recipe and <filename>taskB</filename> for the second
<literallayout class='monospaced'>
$ bitbake myfirstrecipe:do_taskA mysecondrecipe:do_taskB
<section id='generating-dependency-graphs'>
<title>Generating Dependency Graphs</title>
BitBake is able to generate dependency graphs using
the <filename>dot</filename> syntax.
You can convert these graphs into images using the
<filename>dot</filename> tool from
<ulink url=''>Graphviz</ulink>.
When you generate a dependency graph, BitBake writes two files
to the current working directory:
Shows dependencies between tasks.
These dependencies match BitBake's internal task execution list.
Shows a simple list of targets that are to be built.
To stop depending on common depends, use the "-I" depend
option and BitBake omits them from the graph.
Leaving this information out can produce more readable graphs.
This way, you can remove from the graph
<filename>DEPENDS</filename> from inherited classes
such as <filename>base.bbclass</filename>.
Here are two examples that create dependency graphs.
The second example omits depends common in OpenEmbedded from
the graph:
<literallayout class='monospaced'>
$ bitbake -g foo
$ bitbake -g -I virtual/kernel -I eglibc foo
<section id='executing-a-multiple-configuration-build'>
<title>Executing a Multiple Configuration Build</title>
BitBake is able to build multiple images or packages
using a single command where the different targets
require different configurations (multiple configuration
Each target, in this scenario, is referred to as a
To accomplish a multiple configuration build, you must
define each target's configuration separately using
a parallel configuration file in the build directory.
The location for these multiconfig configuration files
is specific.
They must reside in the current build directory in
a sub-directory of <filename>conf</filename> named
Following is an example for two separate targets:
<imagedata fileref="figures/bb_multiconfig_files.png" align="center" width="4in" depth="3in" />
The reason for this required file hierarchy
is because the <filename>BBPATH</filename> variable
is not constructed until the layers are parsed.
Consequently, using the configuration file as a
pre-configuration file is not possible unless it is
located in the current working directory.
Minimally, each configuration file must define the
machine and the temporary directory BitBake uses
for the build.
Suggested practice dictates that you do not
overlap the temporary directories used during the
Aside from separate configuration files for each
target, you must also enable BitBake to perform multiple
configuration builds.
Enabling is accomplished by setting the
<link linkend='var-bb-BBMULTICONFIG'><filename>BBMULTICONFIG</filename></link>
variable in the <filename>local.conf</filename>
configuration file.
As an example, suppose you had configuration files
for <filename>target1</filename> and
<filename>target2</filename> defined in the build
The following statement in the
<filename>local.conf</filename> file both enables
BitBake to perform multiple configuration builds and
specifies the two extra multiconfigs:
<literallayout class='monospaced'>
BBMULTICONFIG = "target1 target2"
Once the target configuration files are in place and
BitBake has been enabled to perform multiple configuration
builds, use the following command form to start the
<literallayout class='monospaced'>
$ bitbake [mc:<replaceable>multiconfigname</replaceable>:]<replaceable>target</replaceable> [[[mc:<replaceable>multiconfigname</replaceable>:]<replaceable>target</replaceable>] ... ]
Here is an example for two extra multiconfigs:
<filename>target1</filename> and
<literallayout class='monospaced'>
$ bitbake mc::<replaceable>target</replaceable> mc:target1:<replaceable>target</replaceable> mc:target2:<replaceable>target</replaceable>
<section id='bb-enabling-multiple-configuration-build-dependencies'>
<title>Enabling Multiple Configuration Build Dependencies</title>
Sometimes dependencies can exist between targets
(multiconfigs) in a multiple configuration build.
For example, suppose that in order to build an image
for a particular architecture, the root filesystem of
another build for a different architecture needs to
In other words, the image for the first multiconfig depends
on the root filesystem of the second multiconfig.
This dependency is essentially that the task in the recipe
that builds one multiconfig is dependent on the
completion of the task in the recipe that builds
another multiconfig.
To enable dependencies in a multiple configuration
build, you must declare the dependencies in the recipe
using the following statement form:
<literallayout class='monospaced'>
<replaceable>task_or_package</replaceable>[mcdepends] = "mc:<replaceable>from_multiconfig</replaceable>:<replaceable>to_multiconfig</replaceable>:<replaceable>recipe_name</replaceable>:<replaceable>task_on_which_to_depend</replaceable>"
To better show how to use this statement, consider an
example with two multiconfigs: <filename>target1</filename>
and <filename>target2</filename>:
<literallayout class='monospaced'>
<replaceable>image_task</replaceable>[mcdepends] = "mc:target1:target2:<replaceable>image2</replaceable>:<replaceable>rootfs_task</replaceable>"
In this example, the
<replaceable>from_multiconfig</replaceable> is "target1" and
the <replaceable>to_multiconfig</replaceable> is "target2".
The task on which the image whose recipe contains
<replaceable>image_task</replaceable> depends on the
completion of the <replaceable>rootfs_task</replaceable>
used to build out <replaceable>image2</replaceable>, which
is associated with the "target2" multiconfig.
Once you set up this dependency, you can build the
"target1" multiconfig using a BitBake command as follows:
<literallayout class='monospaced'>
$ bitbake mc:target1:<replaceable>image1</replaceable>
This command executes all the tasks needed to create
<replaceable>image1</replaceable> for the "target1"
Because of the dependency, BitBake also executes through
the <replaceable>rootfs_task</replaceable> for the "target2"
multiconfig build.
Having a recipe depend on the root filesystem of another
build might not seem that useful.
Consider this change to the statement in the
<replaceable>image1</replaceable> recipe:
<literallayout class='monospaced'>
<replaceable>image_task</replaceable>[mcdepends] = "mc:target1:target2:<replaceable>image2</replaceable>:<replaceable>image_task</replaceable>"
In this case, BitBake must create
<replaceable>image2</replaceable> for the "target2"
build since the "target1" build depends on it.
Because "target1" and "target2" are enabled for multiple
configuration builds and have separate configuration
files, BitBake places the artifacts for each build in the
respective temporary build directories.