blob: 6f9d3929356ed3727a2129f5170a91c2cf194269 [file] [log] [blame]
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001.. SPDX-License-Identifier: CC-BY-2.5
2
3========
4Overview
5========
6
7|
8
9Welcome to the BitBake User Manual. This manual provides information on
10the BitBake tool. The information attempts to be as independent as
11possible regarding systems that use BitBake, such as OpenEmbedded and
12the Yocto Project. In some cases, scenarios or examples within the
13context of a build system are used in the manual to help with
14understanding. For these cases, the manual clearly states the context.
15
16.. _intro:
17
18Introduction
19============
20
21Fundamentally, BitBake is a generic task execution engine that allows
22shell and Python tasks to be run efficiently and in parallel while
23working within complex inter-task dependency constraints. One of
24BitBake's main users, OpenEmbedded, takes this core and builds embedded
25Linux software stacks using a task-oriented approach.
26
27Conceptually, BitBake is similar to GNU Make in some regards but has
28significant differences:
29
30- BitBake executes tasks according to provided metadata that builds up
31 the tasks. Metadata is stored in recipe (``.bb``) and related recipe
32 "append" (``.bbappend``) files, configuration (``.conf``) and
33 underlying include (``.inc``) files, and in class (``.bbclass``)
34 files. The metadata provides BitBake with instructions on what tasks
35 to run and the dependencies between those tasks.
36
37- BitBake includes a fetcher library for obtaining source code from
38 various places such as local files, source control systems, or
39 websites.
40
41- The instructions for each unit to be built (e.g. a piece of software)
42 are known as "recipe" files and contain all the information about the
43 unit (dependencies, source file locations, checksums, description and
44 so on).
45
46- BitBake includes a client/server abstraction and can be used from a
47 command line or used as a service over XML-RPC and has several
48 different user interfaces.
49
50History and Goals
51=================
52
53BitBake was originally a part of the OpenEmbedded project. It was
54inspired by the Portage package management system used by the Gentoo
55Linux distribution. On December 7, 2004, OpenEmbedded project team
56member Chris Larson split the project into two distinct pieces:
57
58- BitBake, a generic task executor
59
60- OpenEmbedded, a metadata set utilized by BitBake
61
62Today, BitBake is the primary basis of the
63`OpenEmbedded <http://www.openembedded.org/>`__ project, which is being
64used to build and maintain Linux distributions such as the `Angstrom
65Distribution <http://www.angstrom-distribution.org/>`__, and which is
66also being used as the build tool for Linux projects such as the `Yocto
67Project <http://www.yoctoproject.org>`__.
68
69Prior to BitBake, no other build tool adequately met the needs of an
70aspiring embedded Linux distribution. All of the build systems used by
71traditional desktop Linux distributions lacked important functionality,
72and none of the ad hoc Buildroot-based systems, prevalent in the
73embedded space, were scalable or maintainable.
74
75Some important original goals for BitBake were:
76
77- Handle cross-compilation.
78
79- Handle inter-package dependencies (build time on target architecture,
80 build time on native architecture, and runtime).
81
82- Support running any number of tasks within a given package,
83 including, but not limited to, fetching upstream sources, unpacking
84 them, patching them, configuring them, and so forth.
85
86- Be Linux distribution agnostic for both build and target systems.
87
88- Be architecture agnostic.
89
90- Support multiple build and target operating systems (e.g. Cygwin, the
91 BSDs, and so forth).
92
93- Be self-contained, rather than tightly integrated into the build
94 machine's root filesystem.
95
96- Handle conditional metadata on the target architecture, operating
97 system, distribution, and machine.
98
99- Be easy to use the tools to supply local metadata and packages
100 against which to operate.
101
102- Be easy to use BitBake to collaborate between multiple projects for
103 their builds.
104
105- Provide an inheritance mechanism to share common metadata between
106 many packages.
107
108Over time it became apparent that some further requirements were
109necessary:
110
111- Handle variants of a base recipe (e.g. native, sdk, and multilib).
112
113- Split metadata into layers and allow layers to enhance or override
114 other layers.
115
116- Allow representation of a given set of input variables to a task as a
117 checksum. Based on that checksum, allow acceleration of builds with
118 prebuilt components.
119
120BitBake satisfies all the original requirements and many more with
121extensions being made to the basic functionality to reflect the
122additional requirements. Flexibility and power have always been the
123priorities. BitBake is highly extensible and supports embedded Python
124code and execution of any arbitrary tasks.
125
126.. _Concepts:
127
128Concepts
129========
130
131BitBake is a program written in the Python language. At the highest
132level, BitBake interprets metadata, decides what tasks are required to
133run, and executes those tasks. Similar to GNU Make, BitBake controls how
134software is built. GNU Make achieves its control through "makefiles",
135while BitBake uses "recipes".
136
137BitBake extends the capabilities of a simple tool like GNU Make by
138allowing for the definition of much more complex tasks, such as
139assembling entire embedded Linux distributions.
140
141The remainder of this section introduces several concepts that should be
142understood in order to better leverage the power of BitBake.
143
144Recipes
145-------
146
147BitBake Recipes, which are denoted by the file extension ``.bb``, are
148the most basic metadata files. These recipe files provide BitBake with
149the following:
150
151- Descriptive information about the package (author, homepage, license,
152 and so on)
153
154- The version of the recipe
155
156- Existing dependencies (both build and runtime dependencies)
157
158- Where the source code resides and how to fetch it
159
160- Whether the source code requires any patches, where to find them, and
161 how to apply them
162
163- How to configure and compile the source code
164
165- How to assemble the generated artifacts into one or more installable
166 packages
167
168- Where on the target machine to install the package or packages
169 created
170
171Within the context of BitBake, or any project utilizing BitBake as its
172build system, files with the ``.bb`` extension are referred to as
173recipes.
174
175.. note::
176
177 The term "package" is also commonly used to describe recipes.
178 However, since the same word is used to describe packaged output from
179 a project, it is best to maintain a single descriptive term -
180 "recipes". Put another way, a single "recipe" file is quite capable
181 of generating a number of related but separately installable
182 "packages". In fact, that ability is fairly common.
183
184Configuration Files
185-------------------
186
187Configuration files, which are denoted by the ``.conf`` extension,
188define various configuration variables that govern the project's build
189process. These files fall into several areas that define machine
190configuration, distribution configuration, possible compiler tuning,
191general common configuration, and user configuration. The main
192configuration file is the sample ``bitbake.conf`` file, which is located
193within the BitBake source tree ``conf`` directory.
194
195Classes
196-------
197
198Class files, which are denoted by the ``.bbclass`` extension, contain
199information that is useful to share between metadata files. The BitBake
200source tree currently comes with one class metadata file called
201``base.bbclass``. You can find this file in the ``classes`` directory.
202The ``base.bbclass`` class files is special since it is always included
203automatically for all recipes and classes. This class contains
204definitions for standard basic tasks such as fetching, unpacking,
205configuring (empty by default), compiling (runs any Makefile present),
206installing (empty by default) and packaging (empty by default). These
207tasks are often overridden or extended by other classes added during the
208project development process.
209
210Layers
211------
212
213Layers allow you to isolate different types of customizations from each
214other. While you might find it tempting to keep everything in one layer
215when working on a single project, the more modular your metadata, the
216easier it is to cope with future changes.
217
218To illustrate how you can use layers to keep things modular, consider
219customizations you might make to support a specific target machine.
220These types of customizations typically reside in a special layer,
221rather than a general layer, called a Board Support Package (BSP) layer.
222Furthermore, the machine customizations should be isolated from recipes
223and metadata that support a new GUI environment, for example. This
224situation gives you a couple of layers: one for the machine
225configurations and one for the GUI environment. It is important to
226understand, however, that the BSP layer can still make machine-specific
227additions to recipes within the GUI environment layer without polluting
228the GUI layer itself with those machine-specific changes. You can
229accomplish this through a recipe that is a BitBake append
230(``.bbappend``) file.
231
232.. _append-bbappend-files:
233
234Append Files
235------------
236
237Append files, which are files that have the ``.bbappend`` file
238extension, extend or override information in an existing recipe file.
239
240BitBake expects every append file to have a corresponding recipe file.
241Furthermore, the append file and corresponding recipe file must use the
242same root filename. The filenames can differ only in the file type
243suffix used (e.g. ``formfactor_0.0.bb`` and
244``formfactor_0.0.bbappend``).
245
246Information in append files extends or overrides the information in the
247underlying, similarly-named recipe files.
248
249When you name an append file, you can use the "``%``" wildcard character
250to allow for matching recipe names. For example, suppose you have an
251append file named as follows: ::
252
253 busybox_1.21.%.bbappend
254
255That append file
256would match any ``busybox_1.21.``\ x\ ``.bb`` version of the recipe. So,
257the append file would match the following recipe names: ::
258
259 busybox_1.21.1.bb
260 busybox_1.21.2.bb
261 busybox_1.21.3.bb
262
263.. note::
264
265 The use of the " % " character is limited in that it only works directly in
266 front of the .bbappend portion of the append file's name. You cannot use the
267 wildcard character in any other location of the name.
268
269If the ``busybox`` recipe was updated to ``busybox_1.3.0.bb``, the
270append name would not match. However, if you named the append file
271``busybox_1.%.bbappend``, then you would have a match.
272
273In the most general case, you could name the append file something as
274simple as ``busybox_%.bbappend`` to be entirely version independent.
275
276Obtaining BitBake
277=================
278
279You can obtain BitBake several different ways:
280
281- **Cloning BitBake:** Using Git to clone the BitBake source code
282 repository is the recommended method for obtaining BitBake. Cloning
283 the repository makes it easy to get bug fixes and have access to
284 stable branches and the master branch. Once you have cloned BitBake,
285 you should use the latest stable branch for development since the
286 master branch is for BitBake development and might contain less
287 stable changes.
288
289 You usually need a version of BitBake that matches the metadata you
290 are using. The metadata is generally backwards compatible but not
291 forward compatible.
292
293 Here is an example that clones the BitBake repository: ::
294
295 $ git clone git://git.openembedded.org/bitbake
296
297 This command clones the BitBake
298 Git repository into a directory called ``bitbake``. Alternatively,
299 you can designate a directory after the ``git clone`` command if you
300 want to call the new directory something other than ``bitbake``. Here
301 is an example that names the directory ``bbdev``: ::
302
303 $ git clone git://git.openembedded.org/bitbake bbdev
304
305- **Installation using your Distribution Package Management System:**
306 This method is not recommended because the BitBake version that is
307 provided by your distribution, in most cases, is several releases
308 behind a snapshot of the BitBake repository.
309
310- **Taking a snapshot of BitBake:** Downloading a snapshot of BitBake
311 from the source code repository gives you access to a known branch or
312 release of BitBake.
313
314 .. note::
315
316 Cloning the Git repository, as described earlier, is the preferred
317 method for getting BitBake. Cloning the repository makes it easier
318 to update as patches are added to the stable branches.
319
320 The following example downloads a snapshot of BitBake version 1.17.0: ::
321
322 $ wget http://git.openembedded.org/bitbake/snapshot/bitbake-1.17.0.tar.gz
323 $ tar zxpvf bitbake-1.17.0.tar.gz
324
325 After extraction of the tarball using
326 the tar utility, you have a directory entitled ``bitbake-1.17.0``.
327
328- **Using the BitBake that Comes With Your Build Checkout:** A final
329 possibility for getting a copy of BitBake is that it already comes
330 with your checkout of a larger BitBake-based build system, such as
331 Poky. Rather than manually checking out individual layers and gluing
332 them together yourself, you can check out an entire build system. The
333 checkout will already include a version of BitBake that has been
334 thoroughly tested for compatibility with the other components. For
335 information on how to check out a particular BitBake-based build
336 system, consult that build system's supporting documentation.
337
338.. _bitbake-user-manual-command:
339
340The BitBake Command
341===================
342
343The ``bitbake`` command is the primary interface to the BitBake tool.
344This section presents the BitBake command syntax and provides several
345execution examples.
346
347Usage and syntax
348----------------
349
350Following is the usage and syntax for BitBake: ::
351
352 $ bitbake -h
353 Usage: bitbake [options] [recipename/target recipe:do_task ...]
354
355 Executes the specified task (default is 'build') for a given set of target recipes (.bb files).
356 It is assumed there is a conf/bblayers.conf available in cwd or in BBPATH which
357 will provide the layer, BBFILES and other configuration information.
358
359 Options:
360 --version show program's version number and exit
361 -h, --help show this help message and exit
362 -b BUILDFILE, --buildfile=BUILDFILE
363 Execute tasks from a specific .bb recipe directly.
364 WARNING: Does not handle any dependencies from other
365 recipes.
366 -k, --continue Continue as much as possible after an error. While the
367 target that failed and anything depending on it cannot
368 be built, as much as possible will be built before
369 stopping.
370 -f, --force Force the specified targets/task to run (invalidating
371 any existing stamp file).
372 -c CMD, --cmd=CMD Specify the task to execute. The exact options
373 available depend on the metadata. Some examples might
374 be 'compile' or 'populate_sysroot' or 'listtasks' may
375 give a list of the tasks available.
376 -C INVALIDATE_STAMP, --clear-stamp=INVALIDATE_STAMP
377 Invalidate the stamp for the specified task such as
378 'compile' and then run the default task for the
379 specified target(s).
380 -r PREFILE, --read=PREFILE
381 Read the specified file before bitbake.conf.
382 -R POSTFILE, --postread=POSTFILE
383 Read the specified file after bitbake.conf.
384 -v, --verbose Enable tracing of shell tasks (with 'set -x'). Also
385 print bb.note(...) messages to stdout (in addition to
386 writing them to ${T}/log.do_&lt;task&gt;).
387 -D, --debug Increase the debug level. You can specify this more
388 than once. -D sets the debug level to 1, where only
389 bb.debug(1, ...) messages are printed to stdout; -DD
390 sets the debug level to 2, where both bb.debug(1, ...)
391 and bb.debug(2, ...) messages are printed; etc.
392 Without -D, no debug messages are printed. Note that
393 -D only affects output to stdout. All debug messages
394 are written to ${T}/log.do_taskname, regardless of the
395 debug level.
396 -q, --quiet Output less log message data to the terminal. You can
397 specify this more than once.
398 -n, --dry-run Don't execute, just go through the motions.
399 -S SIGNATURE_HANDLER, --dump-signatures=SIGNATURE_HANDLER
400 Dump out the signature construction information, with
401 no task execution. The SIGNATURE_HANDLER parameter is
402 passed to the handler. Two common values are none and
403 printdiff but the handler may define more/less. none
404 means only dump the signature, printdiff means compare
405 the dumped signature with the cached one.
406 -p, --parse-only Quit after parsing the BB recipes.
407 -s, --show-versions Show current and preferred versions of all recipes.
408 -e, --environment Show the global or per-recipe environment complete
409 with information about where variables were
410 set/changed.
411 -g, --graphviz Save dependency tree information for the specified
412 targets in the dot syntax.
413 -I EXTRA_ASSUME_PROVIDED, --ignore-deps=EXTRA_ASSUME_PROVIDED
414 Assume these dependencies don't exist and are already
415 provided (equivalent to ASSUME_PROVIDED). Useful to
416 make dependency graphs more appealing
417 -l DEBUG_DOMAINS, --log-domains=DEBUG_DOMAINS
418 Show debug logging for the specified logging domains
419 -P, --profile Profile the command and save reports.
420 -u UI, --ui=UI The user interface to use (knotty, ncurses or taskexp
421 - default knotty).
422 --token=XMLRPCTOKEN Specify the connection token to be used when
423 connecting to a remote server.
424 --revisions-changed Set the exit code depending on whether upstream
425 floating revisions have changed or not.
426 --server-only Run bitbake without a UI, only starting a server
427 (cooker) process.
428 -B BIND, --bind=BIND The name/address for the bitbake xmlrpc server to bind
429 to.
430 -T SERVER_TIMEOUT, --idle-timeout=SERVER_TIMEOUT
431 Set timeout to unload bitbake server due to
432 inactivity, set to -1 means no unload, default:
433 Environment variable BB_SERVER_TIMEOUT.
434 --no-setscene Do not run any setscene tasks. sstate will be ignored
435 and everything needed, built.
436 --setscene-only Only run setscene tasks, don't run any real tasks.
437 --remote-server=REMOTE_SERVER
438 Connect to the specified server.
439 -m, --kill-server Terminate any running bitbake server.
440 --observe-only Connect to a server as an observing-only client.
441 --status-only Check the status of the remote bitbake server.
442 -w WRITEEVENTLOG, --write-log=WRITEEVENTLOG
443 Writes the event log of the build to a bitbake event
444 json file. Use '' (empty string) to assign the name
445 automatically.
446 --runall=RUNALL Run the specified task for any recipe in the taskgraph
447 of the specified target (even if it wouldn't otherwise
448 have run).
449 --runonly=RUNONLY Run only the specified task within the taskgraph of
450 the specified targets (and any task dependencies those
451 tasks may have).
452
453.. _bitbake-examples:
454
455Examples
456--------
457
458This section presents some examples showing how to use BitBake.
459
460.. _example-executing-a-task-against-a-single-recipe:
461
462Executing a Task Against a Single Recipe
463~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
464
465Executing tasks for a single recipe file is relatively simple. You
466specify the file in question, and BitBake parses it and executes the
467specified task. If you do not specify a task, BitBake executes the
Andrew Geisslerf0343792020-11-18 10:42:21 -0600468default task, which is "build". BitBake obeys inter-task dependencies
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500469when doing so.
470
471The following command runs the build task, which is the default task, on
472the ``foo_1.0.bb`` recipe file: ::
473
474 $ bitbake -b foo_1.0.bb
475
476The following command runs the clean task on the ``foo.bb`` recipe file: ::
477
478 $ bitbake -b foo.bb -c clean
479
480.. note::
481
482 The "-b" option explicitly does not handle recipe dependencies. Other
483 than for debugging purposes, it is instead recommended that you use
484 the syntax presented in the next section.
485
486Executing Tasks Against a Set of Recipe Files
487~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
488
489There are a number of additional complexities introduced when one wants
490to manage multiple ``.bb`` files. Clearly there needs to be a way to
491tell BitBake what files are available and, of those, which you want to
492execute. There also needs to be a way for each recipe to express its
493dependencies, both for build-time and runtime. There must be a way for
494you to express recipe preferences when multiple recipes provide the same
495functionality, or when there are multiple versions of a recipe.
496
497The ``bitbake`` command, when not using "--buildfile" or "-b" only
498accepts a "PROVIDES". You cannot provide anything else. By default, a
499recipe file generally "PROVIDES" its "packagename" as shown in the
500following example: ::
501
502 $ bitbake foo
503
504This next example "PROVIDES" the
505package name and also uses the "-c" option to tell BitBake to just
506execute the ``do_clean`` task: ::
507
508 $ bitbake -c clean foo
509
510Executing a List of Task and Recipe Combinations
511~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
512
513The BitBake command line supports specifying different tasks for
514individual targets when you specify multiple targets. For example,
515suppose you had two targets (or recipes) ``myfirstrecipe`` and
516``mysecondrecipe`` and you needed BitBake to run ``taskA`` for the first
517recipe and ``taskB`` for the second recipe: ::
518
519 $ bitbake myfirstrecipe:do_taskA mysecondrecipe:do_taskB
520
521Generating Dependency Graphs
522~~~~~~~~~~~~~~~~~~~~~~~~~~~~
523
524BitBake is able to generate dependency graphs using the ``dot`` syntax.
525You can convert these graphs into images using the ``dot`` tool from
526`Graphviz <http://www.graphviz.org>`__.
527
528When you generate a dependency graph, BitBake writes two files to the
529current working directory:
530
531- ``task-depends.dot``: Shows dependencies between tasks. These
532 dependencies match BitBake's internal task execution list.
533
534- ``pn-buildlist``: Shows a simple list of targets that are to be
535 built.
536
537To stop depending on common depends, use the "-I" depend option and
538BitBake omits them from the graph. Leaving this information out can
539produce more readable graphs. This way, you can remove from the graph
540``DEPENDS`` from inherited classes such as ``base.bbclass``.
541
542Here are two examples that create dependency graphs. The second example
543omits depends common in OpenEmbedded from the graph: ::
544
545 $ bitbake -g foo
546
547 $ bitbake -g -I virtual/kernel -I eglibc foo
548
549Executing a Multiple Configuration Build
550~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
551
552BitBake is able to build multiple images or packages using a single
553command where the different targets require different configurations
554(multiple configuration builds). Each target, in this scenario, is
555referred to as a "multiconfig".
556
557To accomplish a multiple configuration build, you must define each
558target's configuration separately using a parallel configuration file in
559the build directory. The location for these multiconfig configuration
560files is specific. They must reside in the current build directory in a
561sub-directory of ``conf`` named ``multiconfig``. Following is an example
562for two separate targets:
563
564.. image:: figures/bb_multiconfig_files.png
565 :align: center
566
567The reason for this required file hierarchy is because the ``BBPATH``
568variable is not constructed until the layers are parsed. Consequently,
569using the configuration file as a pre-configuration file is not possible
570unless it is located in the current working directory.
571
572Minimally, each configuration file must define the machine and the
573temporary directory BitBake uses for the build. Suggested practice
574dictates that you do not overlap the temporary directories used during
575the builds.
576
577Aside from separate configuration files for each target, you must also
578enable BitBake to perform multiple configuration builds. Enabling is
579accomplished by setting the
580:term:`BBMULTICONFIG` variable in the
581``local.conf`` configuration file. As an example, suppose you had
582configuration files for ``target1`` and ``target2`` defined in the build
583directory. The following statement in the ``local.conf`` file both
584enables BitBake to perform multiple configuration builds and specifies
585the two extra multiconfigs: ::
586
587 BBMULTICONFIG = "target1 target2"
588
589Once the target configuration files are in place and BitBake has been
590enabled to perform multiple configuration builds, use the following
591command form to start the builds: ::
592
593 $ bitbake [mc:multiconfigname:]target [[[mc:multiconfigname:]target] ... ]
594
595Here is an example for two extra multiconfigs: ``target1`` and ``target2``: ::
596
597 $ bitbake mc::target mc:target1:target mc:target2:target
598
599.. _bb-enabling-multiple-configuration-build-dependencies:
600
601Enabling Multiple Configuration Build Dependencies
602~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
603
604Sometimes dependencies can exist between targets (multiconfigs) in a
605multiple configuration build. For example, suppose that in order to
606build an image for a particular architecture, the root filesystem of
607another build for a different architecture needs to exist. In other
608words, the image for the first multiconfig depends on the root
609filesystem of the second multiconfig. This dependency is essentially
610that the task in the recipe that builds one multiconfig is dependent on
611the completion of the task in the recipe that builds another
612multiconfig.
613
614To enable dependencies in a multiple configuration build, you must
615declare the dependencies in the recipe using the following statement
616form: ::
617
618 task_or_package[mcdepends] = "mc:from_multiconfig:to_multiconfig:recipe_name:task_on_which_to_depend"
619
620To better show how to use this statement, consider an example with two
621multiconfigs: ``target1`` and ``target2``: ::
622
623 image_task[mcdepends] = "mc:target1:target2:image2:rootfs_task"
624
625In this example, the
626``from_multiconfig`` is "target1" and the ``to_multiconfig`` is "target2". The
627task on which the image whose recipe contains image_task depends on the
628completion of the rootfs_task used to build out image2, which is
629associated with the "target2" multiconfig.
630
631Once you set up this dependency, you can build the "target1" multiconfig
632using a BitBake command as follows: ::
633
634 $ bitbake mc:target1:image1
635
636This command executes all the tasks needed to create ``image1`` for the "target1"
637multiconfig. Because of the dependency, BitBake also executes through
638the ``rootfs_task`` for the "target2" multiconfig build.
639
640Having a recipe depend on the root filesystem of another build might not
641seem that useful. Consider this change to the statement in the image1
642recipe: ::
643
644 image_task[mcdepends] = "mc:target1:target2:image2:image_task"
645
646In this case, BitBake must create ``image2`` for the "target2" build since
647the "target1" build depends on it.
648
649Because "target1" and "target2" are enabled for multiple configuration
650builds and have separate configuration files, BitBake places the
651artifacts for each build in the respective temporary build directories.