| <!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" |
| "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"> |
| |
| <chapter id="bitbake-user-manual-intro"> |
| <title>Overview</title> |
| |
| <para> |
| Welcome to the BitBake User Manual. |
| This manual provides information on the BitBake tool. |
| The information attempts to be as independent as possible regarding |
| systems that use BitBake, such as OpenEmbedded and the |
| Yocto Project. |
| In some cases, scenarios or examples within the context of |
| a build system are used in the manual to help with understanding. |
| For these cases, the manual clearly states the context. |
| </para> |
| |
| <section id="intro"> |
| <title>Introduction</title> |
| |
| <para> |
| Fundamentally, BitBake is a generic task execution |
| engine that allows shell and Python tasks to be run |
| efficiently and in parallel while working within |
| complex inter-task dependency constraints. |
| One of BitBake's main users, OpenEmbedded, takes this core |
| and builds embedded Linux software stacks using |
| a task-oriented approach. |
| </para> |
| |
| <para> |
| Conceptually, BitBake is similar to GNU Make in |
| some regards but has significant differences: |
| <itemizedlist> |
| <listitem><para> |
| BitBake executes tasks according to provided |
| metadata that builds up the tasks. |
| Metadata is stored in recipe (<filename>.bb</filename>) |
| and related recipe "append" (<filename>.bbappend</filename>) |
| files, configuration (<filename>.conf</filename>) and |
| underlying include (<filename>.inc</filename>) files, and |
| in class (<filename>.bbclass</filename>) files. |
| The metadata provides |
| BitBake with instructions on what tasks to run and |
| the dependencies between those tasks. |
| </para></listitem> |
| <listitem><para> |
| BitBake includes a fetcher library for obtaining source |
| code from various places such as local files, source control |
| systems, or websites. |
| </para></listitem> |
| <listitem><para> |
| The instructions for each unit to be built (e.g. a piece |
| of software) are known as "recipe" files and |
| contain all the information about the unit |
| (dependencies, source file locations, checksums, description |
| and so on). |
| </para></listitem> |
| <listitem><para> |
| BitBake includes a client/server abstraction and can |
| be used from a command line or used as a service over |
| XML-RPC and has several different user interfaces. |
| </para></listitem> |
| </itemizedlist> |
| </para> |
| </section> |
| |
| <section id="history-and-goals"> |
| <title>History and Goals</title> |
| |
| <para> |
| BitBake was originally a part of the OpenEmbedded project. |
| It was inspired by the Portage package management system |
| used by the Gentoo Linux distribution. |
| On December 7, 2004, OpenEmbedded project team member |
| Chris Larson split the project into two distinct pieces: |
| <itemizedlist> |
| <listitem><para>BitBake, a generic task executor</para></listitem> |
| <listitem><para>OpenEmbedded, a metadata set utilized by |
| BitBake</para></listitem> |
| </itemizedlist> |
| Today, BitBake is the primary basis of the |
| <ulink url="http://www.openembedded.org/">OpenEmbedded</ulink> |
| project, which is being used to build and maintain Linux |
| distributions such as the |
| <ulink url='http://www.angstrom-distribution.org/'>Angstrom Distribution</ulink>, |
| and which is also being used as the build tool for Linux projects |
| such as the |
| <ulink url='http://www.yoctoproject.org'>Yocto Project</ulink>. |
| </para> |
| |
| <para> |
| Prior to BitBake, no other build tool adequately met the needs of |
| an aspiring embedded Linux distribution. |
| All of the build systems used by traditional desktop Linux |
| distributions lacked important functionality, and none of the |
| ad hoc Buildroot-based systems, prevalent in the |
| embedded space, were scalable or maintainable. |
| </para> |
| |
| <para> |
| Some important original goals for BitBake were: |
| <itemizedlist> |
| <listitem><para> |
| Handle cross-compilation. |
| </para></listitem> |
| <listitem><para> |
| Handle inter-package dependencies (build time on |
| target architecture, build time on native |
| architecture, and runtime). |
| </para></listitem> |
| <listitem><para> |
| Support running any number of tasks within a given |
| package, including, but not limited to, fetching |
| upstream sources, unpacking them, patching them, |
| configuring them, and so forth. |
| </para></listitem> |
| <listitem><para> |
| Be Linux distribution agnostic for both build and |
| target systems. |
| </para></listitem> |
| <listitem><para> |
| Be architecture agnostic. |
| </para></listitem> |
| <listitem><para> |
| Support multiple build and target operating systems |
| (e.g. Cygwin, the BSDs, and so forth). |
| </para></listitem> |
| <listitem><para> |
| Be self contained, rather than tightly |
| integrated into the build machine's root |
| filesystem. |
| </para></listitem> |
| <listitem><para> |
| Handle conditional metadata on the target architecture, |
| operating system, distribution, and machine. |
| </para></listitem> |
| <listitem><para> |
| Be easy to use the tools to supply local metadata and packages |
| against which to operate. |
| </para></listitem> |
| <listitem><para> |
| Be easy to use BitBake to collaborate between multiple |
| projects for their builds. |
| </para></listitem> |
| <listitem><para> |
| Provide an inheritance mechanism to share |
| common metadata between many packages. |
| </para></listitem> |
| </itemizedlist> |
| Over time it became apparent that some further requirements |
| were necessary: |
| <itemizedlist> |
| <listitem><para> |
| Handle variants of a base recipe (e.g. native, sdk, |
| and multilib). |
| </para></listitem> |
| <listitem><para> |
| Split metadata into layers and allow layers |
| to enhance or override other layers. |
| </para></listitem> |
| <listitem><para> |
| Allow representation of a given set of input variables |
| to a task as a checksum. |
| Based on that checksum, allow acceleration of builds |
| with prebuilt components. |
| </para></listitem> |
| </itemizedlist> |
| BitBake satisfies all the original requirements and many more |
| with extensions being made to the basic functionality to |
| reflect the additional requirements. |
| Flexibility and power have always been the priorities. |
| BitBake is highly extensible and supports embedded Python code and |
| execution of any arbitrary tasks. |
| </para> |
| </section> |
| |
| <section id="Concepts"> |
| <title>Concepts</title> |
| |
| <para> |
| BitBake is a program written in the Python language. |
| At the highest level, BitBake interprets metadata, decides |
| what tasks are required to run, and executes those tasks. |
| Similar to GNU Make, BitBake controls how software is |
| built. |
| GNU Make achieves its control through "makefiles", while |
| BitBake uses "recipes". |
| </para> |
| |
| <para> |
| BitBake extends the capabilities of a simple |
| tool like GNU Make by allowing for the definition of much more |
| complex tasks, such as assembling entire embedded Linux |
| distributions. |
| </para> |
| |
| <para> |
| The remainder of this section introduces several concepts |
| that should be understood in order to better leverage |
| the power of BitBake. |
| </para> |
| |
| <section id='recipes'> |
| <title>Recipes</title> |
| |
| <para> |
| BitBake Recipes, which are denoted by the file extension |
| <filename>.bb</filename>, are the most basic metadata files. |
| These recipe files provide BitBake with the following: |
| <itemizedlist> |
| <listitem><para>Descriptive information about the |
| package (author, homepage, license, and so on)</para></listitem> |
| <listitem><para>The version of the recipe</para></listitem> |
| <listitem><para>Existing dependencies (both build |
| and runtime dependencies)</para></listitem> |
| <listitem><para>Where the source code resides and |
| how to fetch it</para></listitem> |
| <listitem><para>Whether the source code requires |
| any patches, where to find them, and how to apply |
| them</para></listitem> |
| <listitem><para>How to configure and compile the |
| source code</para></listitem> |
| <listitem><para>Where on the target machine to install the |
| package or packages created</para></listitem> |
| </itemizedlist> |
| </para> |
| |
| <para> |
| Within the context of BitBake, or any project utilizing BitBake |
| as its build system, files with the <filename>.bb</filename> |
| extension are referred to as recipes. |
| <note> |
| The term "package" is also commonly used to describe recipes. |
| However, since the same word is used to describe packaged |
| output from a project, it is best to maintain a single |
| descriptive term - "recipes". |
| Put another way, a single "recipe" file is quite capable |
| of generating a number of related but separately installable |
| "packages". |
| In fact, that ability is fairly common. |
| </note> |
| </para> |
| </section> |
| |
| <section id='configuration-files'> |
| <title>Configuration Files</title> |
| |
| <para> |
| Configuration files, which are denoted by the |
| <filename>.conf</filename> extension, define |
| various configuration variables that govern the project's build |
| process. |
| These files fall into several areas that define |
| machine configuration options, distribution configuration |
| options, compiler tuning options, general common |
| configuration options, and user configuration options. |
| The main configuration file is the sample |
| <filename>bitbake.conf</filename> file, which is |
| located within the BitBake source tree |
| <filename>conf</filename> directory. |
| </para> |
| </section> |
| |
| <section id='classes'> |
| <title>Classes</title> |
| |
| <para> |
| Class files, which are denoted by the |
| <filename>.bbclass</filename> extension, contain |
| information that is useful to share between metadata files. |
| The BitBake source tree currently comes with one class metadata file |
| called <filename>base.bbclass</filename>. |
| You can find this file in the |
| <filename>classes</filename> directory. |
| The <filename>base.bbclass</filename> class files is special since it |
| is always included automatically for all recipes |
| and classes. |
| This class contains definitions for standard basic tasks such |
| as fetching, unpacking, configuring (empty by default), |
| compiling (runs any Makefile present), installing (empty by |
| default) and packaging (empty by default). |
| These tasks are often overridden or extended by other classes |
| added during the project development process. |
| </para> |
| </section> |
| |
| <section id='layers'> |
| <title>Layers</title> |
| |
| <para> |
| Layers allow you to isolate different types of |
| customizations from each other. |
| While you might find it tempting to keep everything in one layer |
| when working on a single project, the more modular you organize |
| your metadata, the easier it is to cope with future changes. |
| </para> |
| |
| <para> |
| To illustrate how you can use layers to keep things modular, |
| consider customizations you might make to support a specific target machine. |
| These types of customizations typically reside in a special layer, |
| rather than a general layer, called a Board Support Package (BSP) |
| Layer. |
| Furthermore, the machine customizations should be isolated from |
| recipes and metadata that support a new GUI environment, for |
| example. |
| This situation gives you a couple of layers: one for the machine |
| configurations and one for the GUI environment. |
| It is important to understand, however, that the BSP layer can still |
| make machine-specific additions to recipes within |
| the GUI environment layer without polluting the GUI layer itself |
| with those machine-specific changes. |
| You can accomplish this through a recipe that is a BitBake append |
| (<filename>.bbappend</filename>) file. |
| </para> |
| </section> |
| |
| <section id='append-bbappend-files'> |
| <title>Append Files</title> |
| |
| <para> |
| Append files, which are files that have the |
| <filename>.bbappend</filename> file extension, extend or |
| override information in an existing recipe file. |
| </para> |
| |
| <para> |
| BitBake expects every append file to have a corresponding recipe file. |
| Furthermore, the append file and corresponding recipe file |
| must use the same root filename. |
| The filenames can differ only in the file type suffix used |
| (e.g. <filename>formfactor_0.0.bb</filename> and |
| <filename>formfactor_0.0.bbappend</filename>). |
| </para> |
| |
| <para> |
| Information in append files extends or |
| overrides the information in the underlying, |
| similarly-named recipe files. |
| </para> |
| |
| <para> |
| When you name an append file, you can use the |
| wildcard character (%) to allow for matching recipe names. |
| For example, suppose you have an append file named |
| as follows: |
| <literallayout class='monospaced'> |
| busybox_1.21.%.bbappend |
| </literallayout> |
| That append file would match any <filename>busybox_1.21.x.bb</filename> |
| version of the recipe. |
| So, the append file would match the following recipe names: |
| <literallayout class='monospaced'> |
| busybox_1.21.1.bb |
| busybox_1.21.2.bb |
| busybox_1.21.3.bb |
| </literallayout> |
| If the <filename>busybox</filename> recipe was updated to |
| <filename>busybox_1.3.0.bb</filename>, the append name would not |
| match. |
| However, if you named the append file |
| <filename>busybox_1.%.bbappend</filename>, then you would have a match. |
| </para> |
| |
| <para> |
| In the most general case, you could name the append file something as |
| simple as <filename>busybox_%.bbappend</filename> to be entirely |
| version independent. |
| </para> |
| </section> |
| </section> |
| |
| <section id='obtaining-bitbake'> |
| <title>Obtaining BitBake</title> |
| |
| <para> |
| You can obtain BitBake several different ways: |
| <itemizedlist> |
| <listitem><para><emphasis>Cloning BitBake:</emphasis> |
| Using Git to clone the BitBake source code repository |
| is the recommended method for obtaining BitBake. |
| Cloning the repository makes it easy to get bug fixes |
| and have access to stable branches and the master |
| branch. |
| Once you have cloned BitBake, you should use |
| the latest stable |
| branch for development since the master branch is for |
| BitBake development and might contain less stable changes. |
| </para> |
| <para>You usually need a version of BitBake |
| that matches the metadata you are using. |
| The metadata is generally backwards compatible but |
| not forward compatible.</para> |
| <para>Here is an example that clones the BitBake repository: |
| <literallayout class='monospaced'> |
| $ git clone git://git.openembedded.org/bitbake |
| </literallayout> |
| This command clones the BitBake Git repository into a |
| directory called <filename>bitbake</filename>. |
| Alternatively, you can |
| designate a directory after the |
| <filename>git clone</filename> command |
| if you want to call the new directory something |
| other than <filename>bitbake</filename>. |
| Here is an example that names the directory |
| <filename>bbdev</filename>: |
| <literallayout class='monospaced'> |
| $ git clone git://git.openembedded.org/bitbake bbdev |
| </literallayout></para></listitem> |
| <listitem><para><emphasis>Installation using your Distribution |
| Package Management System:</emphasis> |
| This method is not |
| recommended because the BitBake version that is |
| provided by your distribution, in most cases, |
| is several |
| releases behind a snapshot of the BitBake repository. |
| </para></listitem> |
| <listitem><para><emphasis>Taking a snapshot of BitBake:</emphasis> |
| Downloading a snapshot of BitBake from the |
| source code repository gives you access to a known |
| branch or release of BitBake. |
| <note> |
| Cloning the Git repository, as described earlier, |
| is the preferred method for getting BitBake. |
| Cloning the repository makes it easier to update as |
| patches are added to the stable branches. |
| </note></para> |
| <para>The following example downloads a snapshot of |
| BitBake version 1.17.0: |
| <literallayout class='monospaced'> |
| $ wget http://git.openembedded.org/bitbake/snapshot/bitbake-1.17.0.tar.gz |
| $ tar zxpvf bitbake-1.17.0.tar.gz |
| </literallayout> |
| After extraction of the tarball using the tar utility, |
| you have a directory entitled |
| <filename>bitbake-1.17.0</filename>. |
| </para></listitem> |
| <listitem><para><emphasis>Using the BitBake that Comes With Your |
| Build Checkout:</emphasis> |
| A final possibility for getting a copy of BitBake is that it |
| already comes with your checkout of a larger Bitbake-based build |
| system, such as Poky or Yocto Project. |
| Rather than manually checking out individual layers and |
| gluing them together yourself, you can check |
| out an entire build system. |
| The checkout will already include a version of BitBake that |
| has been thoroughly tested for compatibility with the other |
| components. |
| For information on how to check out a particular BitBake-based |
| build system, consult that build system's supporting documentation. |
| </para></listitem> |
| </itemizedlist> |
| </para> |
| </section> |
| |
| <section id="bitbake-user-manual-command"> |
| <title>The BitBake Command</title> |
| |
| <para> |
| The <filename>bitbake</filename> command is the primary interface |
| to the BitBake tool. |
| This section presents the BitBake command syntax and provides |
| several execution examples. |
| </para> |
| |
| <section id='usage-and-syntax'> |
| <title>Usage and syntax</title> |
| |
| <para> |
| Following is the usage and syntax for BitBake: |
| <literallayout class='monospaced'> |
| $ bitbake -h |
| Usage: bitbake [options] [recipename/target recipe:do_task ...] |
| |
| Executes the specified task (default is 'build') for a given set of target recipes (.bb files). |
| It is assumed there is a conf/bblayers.conf available in cwd or in BBPATH which |
| will provide the layer, BBFILES and other configuration information. |
| |
| Options: |
| --version show program's version number and exit |
| -h, --help show this help message and exit |
| -b BUILDFILE, --buildfile=BUILDFILE |
| Execute tasks from a specific .bb recipe directly. |
| WARNING: Does not handle any dependencies from other |
| recipes. |
| -k, --continue Continue as much as possible after an error. While the |
| target that failed and anything depending on it cannot |
| be built, as much as possible will be built before |
| stopping. |
| -a, --tryaltconfigs Continue with builds by trying to use alternative |
| providers where possible. |
| -f, --force Force the specified targets/task to run (invalidating |
| any existing stamp file). |
| -c CMD, --cmd=CMD Specify the task to execute. The exact options |
| available depend on the metadata. Some examples might |
| be 'compile' or 'populate_sysroot' or 'listtasks' may |
| give a list of the tasks available. |
| -C INVALIDATE_STAMP, --clear-stamp=INVALIDATE_STAMP |
| Invalidate the stamp for the specified task such as |
| 'compile' and then run the default task for the |
| specified target(s). |
| -r PREFILE, --read=PREFILE |
| Read the specified file before bitbake.conf. |
| -R POSTFILE, --postread=POSTFILE |
| Read the specified file after bitbake.conf. |
| -v, --verbose Output more log message data to the terminal. |
| -D, --debug Increase the debug level. You can specify this more |
| than once. |
| -n, --dry-run Don't execute, just go through the motions. |
| -S SIGNATURE_HANDLER, --dump-signatures=SIGNATURE_HANDLER |
| Dump out the signature construction information, with |
| no task execution. The SIGNATURE_HANDLER parameter is |
| passed to the handler. Two common values are none and |
| printdiff but the handler may define more/less. none |
| means only dump the signature, printdiff means compare |
| the dumped signature with the cached one. |
| -p, --parse-only Quit after parsing the BB recipes. |
| -s, --show-versions Show current and preferred versions of all recipes. |
| -e, --environment Show the global or per-recipe environment complete |
| with information about where variables were |
| set/changed. |
| -g, --graphviz Save dependency tree information for the specified |
| targets in the dot syntax. |
| -I EXTRA_ASSUME_PROVIDED, --ignore-deps=EXTRA_ASSUME_PROVIDED |
| Assume these dependencies don't exist and are already |
| provided (equivalent to ASSUME_PROVIDED). Useful to |
| make dependency graphs more appealing |
| -l DEBUG_DOMAINS, --log-domains=DEBUG_DOMAINS |
| Show debug logging for the specified logging domains |
| -P, --profile Profile the command and save reports. |
| -u UI, --ui=UI The user interface to use (depexp, goggle, hob, knotty |
| or ncurses - default knotty). |
| -t SERVERTYPE, --servertype=SERVERTYPE |
| Choose which server type to use (process or xmlrpc - |
| default process). |
| --token=XMLRPCTOKEN Specify the connection token to be used when |
| connecting to a remote server. |
| --revisions-changed Set the exit code depending on whether upstream |
| floating revisions have changed or not. |
| --server-only Run bitbake without a UI, only starting a server |
| (cooker) process. |
| -B BIND, --bind=BIND The name/address for the bitbake server to bind to. |
| --no-setscene Do not run any setscene tasks. sstate will be ignored |
| and everything needed, built. |
| --remote-server=REMOTE_SERVER |
| Connect to the specified server. |
| -m, --kill-server Terminate the remote server. |
| --observe-only Connect to a server as an observing-only client. |
| --status-only Check the status of the remote bitbake server. |
| -w WRITEEVENTLOG, --write-log=WRITEEVENTLOG |
| Writes the event log of the build to a bitbake event |
| json file. Use '' (empty string) to assign the name |
| automatically. |
| </literallayout> |
| </para> |
| </section> |
| |
| <section id='bitbake-examples'> |
| <title>Examples</title> |
| |
| <para> |
| This section presents some examples showing how to use BitBake. |
| </para> |
| |
| <section id='example-executing-a-task-against-a-single-recipe'> |
| <title>Executing a Task Against a Single Recipe</title> |
| |
| <para> |
| Executing tasks for a single recipe file is relatively simple. |
| You specify the file in question, and BitBake parses |
| it and executes the specified task. |
| If you do not specify a task, BitBake executes the default |
| task, which is "build”. |
| BitBake obeys inter-task dependencies when doing |
| so. |
| </para> |
| |
| <para> |
| The following command runs the build task, which is |
| the default task, on the <filename>foo_1.0.bb</filename> |
| recipe file: |
| <literallayout class='monospaced'> |
| $ bitbake -b foo_1.0.bb |
| </literallayout> |
| The following command runs the clean task on the |
| <filename>foo.bb</filename> recipe file: |
| <literallayout class='monospaced'> |
| $ bitbake -b foo.bb -c clean |
| </literallayout> |
| <note> |
| The "-b" option explicitly does not handle recipe |
| dependencies. |
| Other than for debugging purposes, it is instead |
| recommended that you use the syntax presented in the |
| next section. |
| </note> |
| </para> |
| </section> |
| |
| <section id='executing-tasks-against-a-set-of-recipe-files'> |
| <title>Executing Tasks Against a Set of Recipe Files</title> |
| |
| <para> |
| There are a number of additional complexities introduced |
| when one wants to manage multiple <filename>.bb</filename> |
| files. |
| Clearly there needs to be a way to tell BitBake what |
| files are available and, of those, which you |
| want to execute. |
| There also needs to be a way for each recipe |
| to express its dependencies, both for build-time and |
| runtime. |
| There must be a way for you to express recipe preferences |
| when multiple recipes provide the same functionality, or when |
| there are multiple versions of a recipe. |
| </para> |
| |
| <para> |
| The <filename>bitbake</filename> command, when not using |
| "--buildfile" or "-b" only accepts a "PROVIDES". |
| You cannot provide anything else. |
| By default, a recipe file generally "PROVIDES" its |
| "packagename" as shown in the following example: |
| <literallayout class='monospaced'> |
| $ bitbake foo |
| </literallayout> |
| This next example "PROVIDES" the package name and also uses |
| the "-c" option to tell BitBake to just execute the |
| <filename>do_clean</filename> task: |
| <literallayout class='monospaced'> |
| $ bitbake -c clean foo |
| </literallayout> |
| </para> |
| </section> |
| |
| <section id='generating-dependency-graphs'> |
| <title>Generating Dependency Graphs</title> |
| |
| <para> |
| BitBake is able to generate dependency graphs using |
| the <filename>dot</filename> syntax. |
| You can convert these graphs into images using the |
| <filename>dot</filename> tool from |
| <ulink url='http://www.graphviz.org'>Graphviz</ulink>. |
| </para> |
| |
| <para> |
| When you generate a dependency graph, BitBake writes four files |
| to the current working directory: |
| <itemizedlist> |
| <listitem><para><emphasis><filename>package-depends.dot</filename>:</emphasis> |
| Shows BitBake's knowledge of dependencies between |
| runtime targets. |
| </para></listitem> |
| <listitem><para><emphasis><filename>pn-depends.dot</filename>:</emphasis> |
| Shows dependencies between build-time targets |
| (i.e. recipes). |
| </para></listitem> |
| <listitem><para><emphasis><filename>task-depends.dot</filename>:</emphasis> |
| Shows dependencies between tasks. |
| </para></listitem> |
| <listitem><para><emphasis><filename>pn-buildlist</filename>:</emphasis> |
| Shows a simple list of targets that are to be built. |
| </para></listitem> |
| </itemizedlist> |
| </para> |
| |
| <para> |
| To stop depending on common depends, use the "-I" depend |
| option and BitBake omits them from the graph. |
| Leaving this information out can produce more readable graphs. |
| This way, you can remove from the graph |
| <filename>DEPENDS</filename> from inherited classes |
| such as <filename>base.bbclass</filename>. |
| </para> |
| |
| <para> |
| Here are two examples that create dependency graphs. |
| The second example omits depends common in OpenEmbedded from |
| the graph: |
| <literallayout class='monospaced'> |
| $ bitbake -g foo |
| |
| $ bitbake -g -I virtual/kernel -I eglibc foo |
| </literallayout> |
| </para> |
| </section> |
| </section> |
| </section> |
| </chapter> |