Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 1 | .. SPDX-License-Identifier: CC-BY-2.5 |
| 2 | |
| 3 | ========= |
| 4 | Execution |
| 5 | ========= |
| 6 | |
| 7 | | |
| 8 | |
| 9 | The primary purpose for running BitBake is to produce some kind of |
| 10 | output such as a single installable package, a kernel, a software |
| 11 | development kit, or even a full, board-specific bootable Linux image, |
| 12 | complete with bootloader, kernel, and root filesystem. Of course, you |
| 13 | can execute the ``bitbake`` command with options that cause it to |
| 14 | execute single tasks, compile single recipe files, capture or clear |
| 15 | data, or simply return information about the execution environment. |
| 16 | |
| 17 | This chapter describes BitBake's execution process from start to finish |
| 18 | when you use it to create an image. The execution process is launched |
Andrew Geissler | c926e17 | 2021-05-07 16:11:35 -0500 | [diff] [blame] | 19 | using the following command form:: |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 20 | |
| 21 | $ bitbake target |
| 22 | |
| 23 | For information on |
| 24 | the BitBake command and its options, see ":ref:`The BitBake Command |
| 25 | <bitbake-user-manual-command>`" section. |
| 26 | |
| 27 | .. note:: |
| 28 | |
| 29 | Prior to executing BitBake, you should take advantage of available |
| 30 | parallel thread execution on your build host by setting the |
| 31 | :term:`BB_NUMBER_THREADS` variable in |
| 32 | your project's ``local.conf`` configuration file. |
| 33 | |
| 34 | A common method to determine this value for your build host is to run |
Andrew Geissler | c926e17 | 2021-05-07 16:11:35 -0500 | [diff] [blame] | 35 | the following:: |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 36 | |
| 37 | $ grep processor /proc/cpuinfo |
| 38 | |
| 39 | This command returns |
| 40 | the number of processors, which takes into account hyper-threading. |
| 41 | Thus, a quad-core build host with hyper-threading most likely shows |
| 42 | eight processors, which is the value you would then assign to |
| 43 | ``BB_NUMBER_THREADS``. |
| 44 | |
| 45 | A possibly simpler solution is that some Linux distributions (e.g. |
| 46 | Debian and Ubuntu) provide the ``ncpus`` command. |
| 47 | |
| 48 | Parsing the Base Configuration Metadata |
| 49 | ======================================= |
| 50 | |
| 51 | The first thing BitBake does is parse base configuration metadata. Base |
| 52 | configuration metadata consists of your project's ``bblayers.conf`` file |
| 53 | to determine what layers BitBake needs to recognize, all necessary |
| 54 | ``layer.conf`` files (one from each layer), and ``bitbake.conf``. The |
| 55 | data itself is of various types: |
| 56 | |
| 57 | - **Recipes:** Details about particular pieces of software. |
| 58 | |
| 59 | - **Class Data:** An abstraction of common build information (e.g. how to |
| 60 | build a Linux kernel). |
| 61 | |
| 62 | - **Configuration Data:** Machine-specific settings, policy decisions, |
| 63 | and so forth. Configuration data acts as the glue to bind everything |
| 64 | together. |
| 65 | |
| 66 | The ``layer.conf`` files are used to construct key variables such as |
| 67 | :term:`BBPATH` and :term:`BBFILES`. |
| 68 | ``BBPATH`` is used to search for configuration and class files under the |
| 69 | ``conf`` and ``classes`` directories, respectively. ``BBFILES`` is used |
| 70 | to locate both recipe and recipe append files (``.bb`` and |
| 71 | ``.bbappend``). If there is no ``bblayers.conf`` file, it is assumed the |
| 72 | user has set the ``BBPATH`` and ``BBFILES`` directly in the environment. |
| 73 | |
| 74 | Next, the ``bitbake.conf`` file is located using the ``BBPATH`` variable |
| 75 | that was just constructed. The ``bitbake.conf`` file may also include |
| 76 | other configuration files using the ``include`` or ``require`` |
| 77 | directives. |
| 78 | |
| 79 | Prior to parsing configuration files, BitBake looks at certain |
| 80 | variables, including: |
| 81 | |
| 82 | - :term:`BB_ENV_WHITELIST` |
| 83 | - :term:`BB_ENV_EXTRAWHITE` |
| 84 | - :term:`BB_PRESERVE_ENV` |
| 85 | - :term:`BB_ORIGENV` |
| 86 | - :term:`BITBAKE_UI` |
| 87 | |
| 88 | The first four variables in this list relate to how BitBake treats shell |
| 89 | environment variables during task execution. By default, BitBake cleans |
| 90 | the environment variables and provides tight control over the shell |
| 91 | execution environment. However, through the use of these first four |
| 92 | variables, you can apply your control regarding the environment |
| 93 | variables allowed to be used by BitBake in the shell during execution of |
| 94 | tasks. See the |
| 95 | ":ref:`bitbake-user-manual/bitbake-user-manual-metadata:Passing Information Into the Build Task Environment`" |
| 96 | section and the information about these variables in the variable |
| 97 | glossary for more information on how they work and on how to use them. |
| 98 | |
| 99 | The base configuration metadata is global and therefore affects all |
| 100 | recipes and tasks that are executed. |
| 101 | |
| 102 | BitBake first searches the current working directory for an optional |
| 103 | ``conf/bblayers.conf`` configuration file. This file is expected to |
| 104 | contain a :term:`BBLAYERS` variable that is a |
| 105 | space-delimited list of 'layer' directories. Recall that if BitBake |
| 106 | cannot find a ``bblayers.conf`` file, then it is assumed the user has |
| 107 | set the ``BBPATH`` and ``BBFILES`` variables directly in the |
| 108 | environment. |
| 109 | |
| 110 | For each directory (layer) in this list, a ``conf/layer.conf`` file is |
| 111 | located and parsed with the :term:`LAYERDIR` variable |
| 112 | being set to the directory where the layer was found. The idea is these |
| 113 | files automatically set up :term:`BBPATH` and other |
| 114 | variables correctly for a given build directory. |
| 115 | |
| 116 | BitBake then expects to find the ``conf/bitbake.conf`` file somewhere in |
| 117 | the user-specified ``BBPATH``. That configuration file generally has |
| 118 | include directives to pull in any other metadata such as files specific |
| 119 | to the architecture, the machine, the local environment, and so forth. |
| 120 | |
| 121 | Only variable definitions and include directives are allowed in BitBake |
| 122 | ``.conf`` files. Some variables directly influence BitBake's behavior. |
| 123 | These variables might have been set from the environment depending on |
| 124 | the environment variables previously mentioned or set in the |
| 125 | configuration files. The ":ref:`bitbake-user-manual/bitbake-user-manual-ref-variables:Variables Glossary`" |
| 126 | chapter presents a full list of |
| 127 | variables. |
| 128 | |
| 129 | After parsing configuration files, BitBake uses its rudimentary |
| 130 | inheritance mechanism, which is through class files, to inherit some |
| 131 | standard classes. BitBake parses a class when the inherit directive |
| 132 | responsible for getting that class is encountered. |
| 133 | |
| 134 | The ``base.bbclass`` file is always included. Other classes that are |
| 135 | specified in the configuration using the |
| 136 | :term:`INHERIT` variable are also included. BitBake |
| 137 | searches for class files in a ``classes`` subdirectory under the paths |
| 138 | in ``BBPATH`` in the same way as configuration files. |
| 139 | |
| 140 | A good way to get an idea of the configuration files and the class files |
| 141 | used in your execution environment is to run the following BitBake |
Andrew Geissler | c926e17 | 2021-05-07 16:11:35 -0500 | [diff] [blame] | 142 | command:: |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 143 | |
| 144 | $ bitbake -e > mybb.log |
| 145 | |
| 146 | Examining the top of the ``mybb.log`` |
| 147 | shows you the many configuration files and class files used in your |
| 148 | execution environment. |
| 149 | |
| 150 | .. note:: |
| 151 | |
| 152 | You need to be aware of how BitBake parses curly braces. If a recipe |
| 153 | uses a closing curly brace within the function and the character has |
| 154 | no leading spaces, BitBake produces a parsing error. If you use a |
| 155 | pair of curly braces in a shell function, the closing curly brace |
| 156 | must not be located at the start of the line without leading spaces. |
| 157 | |
Andrew Geissler | c926e17 | 2021-05-07 16:11:35 -0500 | [diff] [blame] | 158 | Here is an example that causes BitBake to produce a parsing error:: |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 159 | |
| 160 | fakeroot create_shar() { |
| 161 | cat << "EOF" > ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh |
| 162 | usage() |
| 163 | { |
| 164 | echo "test" |
| 165 | ###### The following "}" at the start of the line causes a parsing error ###### |
| 166 | } |
| 167 | EOF |
| 168 | } |
| 169 | |
| 170 | Writing the recipe this way avoids the error: |
| 171 | fakeroot create_shar() { |
| 172 | cat << "EOF" > ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh |
| 173 | usage() |
| 174 | { |
| 175 | echo "test" |
| 176 | ###### The following "}" with a leading space at the start of the line avoids the error ###### |
| 177 | } |
| 178 | EOF |
| 179 | } |
| 180 | |
| 181 | Locating and Parsing Recipes |
| 182 | ============================ |
| 183 | |
| 184 | During the configuration phase, BitBake will have set |
| 185 | :term:`BBFILES`. BitBake now uses it to construct a |
| 186 | list of recipes to parse, along with any append files (``.bbappend``) to |
| 187 | apply. ``BBFILES`` is a space-separated list of available files and |
Andrew Geissler | c926e17 | 2021-05-07 16:11:35 -0500 | [diff] [blame] | 188 | supports wildcards. An example would be:: |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 189 | |
| 190 | BBFILES = "/path/to/bbfiles/*.bb /path/to/appends/*.bbappend" |
| 191 | |
| 192 | BitBake parses each |
| 193 | recipe and append file located with ``BBFILES`` and stores the values of |
| 194 | various variables into the datastore. |
| 195 | |
| 196 | .. note:: |
| 197 | |
| 198 | Append files are applied in the order they are encountered in BBFILES. |
| 199 | |
| 200 | For each file, a fresh copy of the base configuration is made, then the |
| 201 | recipe is parsed line by line. Any inherit statements cause BitBake to |
| 202 | find and then parse class files (``.bbclass``) using |
| 203 | :term:`BBPATH` as the search path. Finally, BitBake |
| 204 | parses in order any append files found in ``BBFILES``. |
| 205 | |
| 206 | One common convention is to use the recipe filename to define pieces of |
| 207 | metadata. For example, in ``bitbake.conf`` the recipe name and version |
| 208 | are used to set the variables :term:`PN` and |
Andrew Geissler | c926e17 | 2021-05-07 16:11:35 -0500 | [diff] [blame] | 209 | :term:`PV`:: |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 210 | |
| 211 | PN = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0] or 'defaultpkgname'}" |
| 212 | PV = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[1] or '1.0'}" |
| 213 | |
| 214 | In this example, a recipe called "something_1.2.3.bb" would set |
| 215 | ``PN`` to "something" and ``PV`` to "1.2.3". |
| 216 | |
| 217 | By the time parsing is complete for a recipe, BitBake has a list of |
| 218 | tasks that the recipe defines and a set of data consisting of keys and |
| 219 | values as well as dependency information about the tasks. |
| 220 | |
| 221 | BitBake does not need all of this information. It only needs a small |
| 222 | subset of the information to make decisions about the recipe. |
| 223 | Consequently, BitBake caches the values in which it is interested and |
| 224 | does not store the rest of the information. Experience has shown it is |
| 225 | faster to re-parse the metadata than to try and write it out to the disk |
| 226 | and then reload it. |
| 227 | |
| 228 | Where possible, subsequent BitBake commands reuse this cache of recipe |
| 229 | information. The validity of this cache is determined by first computing |
| 230 | a checksum of the base configuration data (see |
| 231 | :term:`BB_HASHCONFIG_WHITELIST`) and |
| 232 | then checking if the checksum matches. If that checksum matches what is |
| 233 | in the cache and the recipe and class files have not changed, BitBake is |
| 234 | able to use the cache. BitBake then reloads the cached information about |
| 235 | the recipe instead of reparsing it from scratch. |
| 236 | |
| 237 | Recipe file collections exist to allow the user to have multiple |
| 238 | repositories of ``.bb`` files that contain the same exact package. For |
| 239 | example, one could easily use them to make one's own local copy of an |
| 240 | upstream repository, but with custom modifications that one does not |
Andrew Geissler | c926e17 | 2021-05-07 16:11:35 -0500 | [diff] [blame] | 241 | want upstream. Here is an example:: |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 242 | |
| 243 | BBFILES = "/stuff/openembedded/*/*.bb /stuff/openembedded.modified/*/*.bb" |
| 244 | BBFILE_COLLECTIONS = "upstream local" |
| 245 | BBFILE_PATTERN_upstream = "^/stuff/openembedded/" |
| 246 | BBFILE_PATTERN_local = "^/stuff/openembedded.modified/" |
Andrew Geissler | 95ac1b8 | 2021-03-31 14:34:31 -0500 | [diff] [blame] | 247 | BBFILE_PRIORITY_upstream = "5" |
| 248 | BBFILE_PRIORITY_local = "10" |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 249 | |
| 250 | .. note:: |
| 251 | |
| 252 | The layers mechanism is now the preferred method of collecting code. |
| 253 | While the collections code remains, its main use is to set layer |
| 254 | priorities and to deal with overlap (conflicts) between layers. |
| 255 | |
| 256 | .. _bb-bitbake-providers: |
| 257 | |
| 258 | Providers |
| 259 | ========= |
| 260 | |
| 261 | Assuming BitBake has been instructed to execute a target and that all |
| 262 | the recipe files have been parsed, BitBake starts to figure out how to |
| 263 | build the target. BitBake looks through the ``PROVIDES`` list for each |
| 264 | of the recipes. A ``PROVIDES`` list is the list of names by which the |
| 265 | recipe can be known. Each recipe's ``PROVIDES`` list is created |
| 266 | implicitly through the recipe's :term:`PN` variable and |
| 267 | explicitly through the recipe's :term:`PROVIDES` |
| 268 | variable, which is optional. |
| 269 | |
| 270 | When a recipe uses ``PROVIDES``, that recipe's functionality can be |
| 271 | found under an alternative name or names other than the implicit ``PN`` |
| 272 | name. As an example, suppose a recipe named ``keyboard_1.0.bb`` |
Andrew Geissler | c926e17 | 2021-05-07 16:11:35 -0500 | [diff] [blame] | 273 | contained the following:: |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 274 | |
| 275 | PROVIDES += "fullkeyboard" |
| 276 | |
| 277 | The ``PROVIDES`` |
| 278 | list for this recipe becomes "keyboard", which is implicit, and |
| 279 | "fullkeyboard", which is explicit. Consequently, the functionality found |
| 280 | in ``keyboard_1.0.bb`` can be found under two different names. |
| 281 | |
| 282 | .. _bb-bitbake-preferences: |
| 283 | |
| 284 | Preferences |
| 285 | =========== |
| 286 | |
| 287 | The ``PROVIDES`` list is only part of the solution for figuring out a |
| 288 | target's recipes. Because targets might have multiple providers, BitBake |
| 289 | needs to prioritize providers by determining provider preferences. |
| 290 | |
| 291 | A common example in which a target has multiple providers is |
| 292 | "virtual/kernel", which is on the ``PROVIDES`` list for each kernel |
| 293 | recipe. Each machine often selects the best kernel provider by using a |
Andrew Geissler | c926e17 | 2021-05-07 16:11:35 -0500 | [diff] [blame] | 294 | line similar to the following in the machine configuration file:: |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 295 | |
| 296 | PREFERRED_PROVIDER_virtual/kernel = "linux-yocto" |
| 297 | |
| 298 | The default :term:`PREFERRED_PROVIDER` is the provider |
| 299 | with the same name as the target. BitBake iterates through each target |
| 300 | it needs to build and resolves them and their dependencies using this |
| 301 | process. |
| 302 | |
| 303 | Understanding how providers are chosen is made complicated by the fact |
| 304 | that multiple versions might exist for a given provider. BitBake |
| 305 | defaults to the highest version of a provider. Version comparisons are |
| 306 | made using the same method as Debian. You can use the |
| 307 | :term:`PREFERRED_VERSION` variable to |
| 308 | specify a particular version. You can influence the order by using the |
| 309 | :term:`DEFAULT_PREFERENCE` variable. |
| 310 | |
| 311 | By default, files have a preference of "0". Setting |
| 312 | ``DEFAULT_PREFERENCE`` to "-1" makes the recipe unlikely to be used |
| 313 | unless it is explicitly referenced. Setting ``DEFAULT_PREFERENCE`` to |
| 314 | "1" makes it likely the recipe is used. ``PREFERRED_VERSION`` overrides |
| 315 | any ``DEFAULT_PREFERENCE`` setting. ``DEFAULT_PREFERENCE`` is often used |
| 316 | to mark newer and more experimental recipe versions until they have |
| 317 | undergone sufficient testing to be considered stable. |
| 318 | |
Andrew Geissler | f034379 | 2020-11-18 10:42:21 -0600 | [diff] [blame] | 319 | When there are multiple "versions" of a given recipe, BitBake defaults |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 320 | to selecting the most recent version, unless otherwise specified. If the |
| 321 | recipe in question has a |
| 322 | :term:`DEFAULT_PREFERENCE` set lower than |
| 323 | the other recipes (default is 0), then it will not be selected. This |
| 324 | allows the person or persons maintaining the repository of recipe files |
| 325 | to specify their preference for the default selected version. |
| 326 | Additionally, the user can specify their preferred version. |
| 327 | |
| 328 | If the first recipe is named ``a_1.1.bb``, then the |
Andrew Geissler | f034379 | 2020-11-18 10:42:21 -0600 | [diff] [blame] | 329 | :term:`PN` variable will be set to "a", and the |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 330 | :term:`PV` variable will be set to 1.1. |
| 331 | |
| 332 | Thus, if a recipe named ``a_1.2.bb`` exists, BitBake will choose 1.2 by |
| 333 | default. However, if you define the following variable in a ``.conf`` |
Andrew Geissler | c926e17 | 2021-05-07 16:11:35 -0500 | [diff] [blame] | 334 | file that BitBake parses, you can change that preference:: |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 335 | |
| 336 | PREFERRED_VERSION_a = "1.1" |
| 337 | |
| 338 | .. note:: |
| 339 | |
| 340 | It is common for a recipe to provide two versions -- a stable, |
| 341 | numbered (and preferred) version, and a version that is automatically |
| 342 | checked out from a source code repository that is considered more |
| 343 | "bleeding edge" but can be selected only explicitly. |
| 344 | |
| 345 | For example, in the OpenEmbedded codebase, there is a standard, |
| 346 | versioned recipe file for BusyBox, ``busybox_1.22.1.bb``, but there |
| 347 | is also a Git-based version, ``busybox_git.bb``, which explicitly |
| 348 | contains the line :: |
| 349 | |
| 350 | DEFAULT_PREFERENCE = "-1" |
| 351 | |
| 352 | to ensure that the |
| 353 | numbered, stable version is always preferred unless the developer |
| 354 | selects otherwise. |
| 355 | |
| 356 | .. _bb-bitbake-dependencies: |
| 357 | |
| 358 | Dependencies |
| 359 | ============ |
| 360 | |
| 361 | Each target BitBake builds consists of multiple tasks such as ``fetch``, |
| 362 | ``unpack``, ``patch``, ``configure``, and ``compile``. For best |
| 363 | performance on multi-core systems, BitBake considers each task as an |
| 364 | independent entity with its own set of dependencies. |
| 365 | |
| 366 | Dependencies are defined through several variables. You can find |
| 367 | information about variables BitBake uses in the |
| 368 | :doc:`bitbake-user-manual-ref-variables` near the end of this manual. At a |
| 369 | basic level, it is sufficient to know that BitBake uses the |
| 370 | :term:`DEPENDS` and |
| 371 | :term:`RDEPENDS` variables when calculating |
| 372 | dependencies. |
| 373 | |
| 374 | For more information on how BitBake handles dependencies, see the |
| 375 | :ref:`bitbake-user-manual/bitbake-user-manual-metadata:Dependencies` |
| 376 | section. |
| 377 | |
| 378 | .. _ref-bitbake-tasklist: |
| 379 | |
| 380 | The Task List |
| 381 | ============= |
| 382 | |
| 383 | Based on the generated list of providers and the dependency information, |
| 384 | BitBake can now calculate exactly what tasks it needs to run and in what |
| 385 | order it needs to run them. The |
| 386 | :ref:`bitbake-user-manual/bitbake-user-manual-execution:executing tasks` |
| 387 | section has more information on how BitBake chooses which task to |
| 388 | execute next. |
| 389 | |
| 390 | The build now starts with BitBake forking off threads up to the limit |
| 391 | set in the :term:`BB_NUMBER_THREADS` |
| 392 | variable. BitBake continues to fork threads as long as there are tasks |
| 393 | ready to run, those tasks have all their dependencies met, and the |
| 394 | thread threshold has not been exceeded. |
| 395 | |
| 396 | It is worth noting that you can greatly speed up the build time by |
| 397 | properly setting the ``BB_NUMBER_THREADS`` variable. |
| 398 | |
| 399 | As each task completes, a timestamp is written to the directory |
| 400 | specified by the :term:`STAMP` variable. On subsequent |
| 401 | runs, BitBake looks in the build directory within ``tmp/stamps`` and |
| 402 | does not rerun tasks that are already completed unless a timestamp is |
| 403 | found to be invalid. Currently, invalid timestamps are only considered |
| 404 | on a per recipe file basis. So, for example, if the configure stamp has |
| 405 | a timestamp greater than the compile timestamp for a given target, then |
| 406 | the compile task would rerun. Running the compile task again, however, |
| 407 | has no effect on other providers that depend on that target. |
| 408 | |
| 409 | The exact format of the stamps is partly configurable. In modern |
| 410 | versions of BitBake, a hash is appended to the stamp so that if the |
| 411 | configuration changes, the stamp becomes invalid and the task is |
| 412 | automatically rerun. This hash, or signature used, is governed by the |
| 413 | signature policy that is configured (see the |
| 414 | :ref:`bitbake-user-manual/bitbake-user-manual-execution:checksums (signatures)` |
| 415 | section for information). It is also |
| 416 | possible to append extra metadata to the stamp using the |
| 417 | ``[stamp-extra-info]`` task flag. For example, OpenEmbedded uses this |
| 418 | flag to make some tasks machine-specific. |
| 419 | |
| 420 | .. note:: |
| 421 | |
| 422 | Some tasks are marked as "nostamp" tasks. No timestamp file is |
| 423 | created when these tasks are run. Consequently, "nostamp" tasks are |
| 424 | always rerun. |
| 425 | |
| 426 | For more information on tasks, see the |
| 427 | :ref:`bitbake-user-manual/bitbake-user-manual-metadata:tasks` section. |
| 428 | |
| 429 | Executing Tasks |
| 430 | =============== |
| 431 | |
| 432 | Tasks can be either a shell task or a Python task. For shell tasks, |
| 433 | BitBake writes a shell script to |
| 434 | ``${``\ :term:`T`\ ``}/run.do_taskname.pid`` and then |
| 435 | executes the script. The generated shell script contains all the |
| 436 | exported variables, and the shell functions with all variables expanded. |
| 437 | Output from the shell script goes to the file |
| 438 | ``${T}/log.do_taskname.pid``. Looking at the expanded shell functions in |
| 439 | the run file and the output in the log files is a useful debugging |
| 440 | technique. |
| 441 | |
| 442 | For Python tasks, BitBake executes the task internally and logs |
| 443 | information to the controlling terminal. Future versions of BitBake will |
| 444 | write the functions to files similar to the way shell tasks are handled. |
| 445 | Logging will be handled in a way similar to shell tasks as well. |
| 446 | |
| 447 | The order in which BitBake runs the tasks is controlled by its task |
| 448 | scheduler. It is possible to configure the scheduler and define custom |
| 449 | implementations for specific use cases. For more information, see these |
| 450 | variables that control the behavior: |
| 451 | |
| 452 | - :term:`BB_SCHEDULER` |
| 453 | |
| 454 | - :term:`BB_SCHEDULERS` |
| 455 | |
| 456 | It is possible to have functions run before and after a task's main |
| 457 | function. This is done using the ``[prefuncs]`` and ``[postfuncs]`` |
| 458 | flags of the task that lists the functions to run. |
| 459 | |
| 460 | .. _checksums: |
| 461 | |
| 462 | Checksums (Signatures) |
| 463 | ====================== |
| 464 | |
| 465 | A checksum is a unique signature of a task's inputs. The signature of a |
| 466 | task can be used to determine if a task needs to be run. Because it is a |
| 467 | change in a task's inputs that triggers running the task, BitBake needs |
| 468 | to detect all the inputs to a given task. For shell tasks, this turns |
| 469 | out to be fairly easy because BitBake generates a "run" shell script for |
| 470 | each task and it is possible to create a checksum that gives you a good |
| 471 | idea of when the task's data changes. |
| 472 | |
| 473 | To complicate the problem, some things should not be included in the |
| 474 | checksum. First, there is the actual specific build path of a given task |
| 475 | - the working directory. It does not matter if the working directory |
| 476 | changes because it should not affect the output for target packages. The |
| 477 | simplistic approach for excluding the working directory is to set it to |
| 478 | some fixed value and create the checksum for the "run" script. BitBake |
| 479 | goes one step better and uses the |
| 480 | :term:`BB_HASHBASE_WHITELIST` variable |
| 481 | to define a list of variables that should never be included when |
| 482 | generating the signatures. |
| 483 | |
| 484 | Another problem results from the "run" scripts containing functions that |
| 485 | might or might not get called. The incremental build solution contains |
| 486 | code that figures out dependencies between shell functions. This code is |
| 487 | used to prune the "run" scripts down to the minimum set, thereby |
| 488 | alleviating this problem and making the "run" scripts much more readable |
| 489 | as a bonus. |
| 490 | |
| 491 | So far we have solutions for shell scripts. What about Python tasks? The |
| 492 | same approach applies even though these tasks are more difficult. The |
| 493 | process needs to figure out what variables a Python function accesses |
| 494 | and what functions it calls. Again, the incremental build solution |
| 495 | contains code that first figures out the variable and function |
| 496 | dependencies, and then creates a checksum for the data used as the input |
| 497 | to the task. |
| 498 | |
| 499 | Like the working directory case, situations exist where dependencies |
| 500 | should be ignored. For these cases, you can instruct the build process |
Andrew Geissler | c926e17 | 2021-05-07 16:11:35 -0500 | [diff] [blame] | 501 | to ignore a dependency by using a line like the following:: |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 502 | |
| 503 | PACKAGE_ARCHS[vardepsexclude] = "MACHINE" |
| 504 | |
| 505 | This example ensures that the |
| 506 | ``PACKAGE_ARCHS`` variable does not depend on the value of ``MACHINE``, |
| 507 | even if it does reference it. |
| 508 | |
| 509 | Equally, there are cases where we need to add dependencies BitBake is |
| 510 | not able to find. You can accomplish this by using a line like the |
Andrew Geissler | c926e17 | 2021-05-07 16:11:35 -0500 | [diff] [blame] | 511 | following:: |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 512 | |
| 513 | PACKAGE_ARCHS[vardeps] = "MACHINE" |
| 514 | |
| 515 | This example explicitly |
| 516 | adds the ``MACHINE`` variable as a dependency for ``PACKAGE_ARCHS``. |
| 517 | |
| 518 | Consider a case with in-line Python, for example, where BitBake is not |
| 519 | able to figure out dependencies. When running in debug mode (i.e. using |
| 520 | ``-DDD``), BitBake produces output when it discovers something for which |
| 521 | it cannot figure out dependencies. |
| 522 | |
| 523 | Thus far, this section has limited discussion to the direct inputs into |
| 524 | a task. Information based on direct inputs is referred to as the |
| 525 | "basehash" in the code. However, there is still the question of a task's |
| 526 | indirect inputs - the things that were already built and present in the |
| 527 | build directory. The checksum (or signature) for a particular task needs |
| 528 | to add the hashes of all the tasks on which the particular task depends. |
| 529 | Choosing which dependencies to add is a policy decision. However, the |
| 530 | effect is to generate a master checksum that combines the basehash and |
| 531 | the hashes of the task's dependencies. |
| 532 | |
| 533 | At the code level, there are a variety of ways both the basehash and the |
| 534 | dependent task hashes can be influenced. Within the BitBake |
| 535 | configuration file, we can give BitBake some extra information to help |
| 536 | it construct the basehash. The following statement effectively results |
| 537 | in a list of global variable dependency excludes - variables never |
| 538 | included in any checksum. This example uses variables from OpenEmbedded |
Andrew Geissler | c926e17 | 2021-05-07 16:11:35 -0500 | [diff] [blame] | 539 | to help illustrate the concept:: |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 540 | |
| 541 | BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \ |
| 542 | SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL \ |
| 543 | USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \ |
| 544 | PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \ |
| 545 | CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX" |
| 546 | |
| 547 | The previous example excludes the work directory, which is part of |
| 548 | ``TMPDIR``. |
| 549 | |
| 550 | The rules for deciding which hashes of dependent tasks to include |
| 551 | through dependency chains are more complex and are generally |
| 552 | accomplished with a Python function. The code in |
| 553 | ``meta/lib/oe/sstatesig.py`` shows two examples of this and also |
| 554 | illustrates how you can insert your own policy into the system if so |
| 555 | desired. This file defines the two basic signature generators |
| 556 | OpenEmbedded-Core uses: "OEBasic" and "OEBasicHash". By default, there |
| 557 | is a dummy "noop" signature handler enabled in BitBake. This means that |
| 558 | behavior is unchanged from previous versions. ``OE-Core`` uses the |
| 559 | "OEBasicHash" signature handler by default through this setting in the |
Andrew Geissler | c926e17 | 2021-05-07 16:11:35 -0500 | [diff] [blame] | 560 | ``bitbake.conf`` file:: |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 561 | |
| 562 | BB_SIGNATURE_HANDLER ?= "OEBasicHash" |
| 563 | |
| 564 | The "OEBasicHash" ``BB_SIGNATURE_HANDLER`` is the same as the "OEBasic" |
| 565 | version but adds the task hash to the stamp files. This results in any |
| 566 | metadata change that changes the task hash, automatically causing the |
| 567 | task to be run again. This removes the need to bump |
| 568 | :term:`PR` values, and changes to metadata automatically |
| 569 | ripple across the build. |
| 570 | |
| 571 | It is also worth noting that the end result of these signature |
| 572 | generators is to make some dependency and hash information available to |
| 573 | the build. This information includes: |
| 574 | |
| 575 | - ``BB_BASEHASH_task-``\ *taskname*: The base hashes for each task in the |
| 576 | recipe. |
| 577 | |
| 578 | - ``BB_BASEHASH_``\ *filename:taskname*: The base hashes for each |
| 579 | dependent task. |
| 580 | |
| 581 | - ``BBHASHDEPS_``\ *filename:taskname*: The task dependencies for |
| 582 | each task. |
| 583 | |
| 584 | - ``BB_TASKHASH``: The hash of the currently running task. |
| 585 | |
| 586 | It is worth noting that BitBake's "-S" option lets you debug BitBake's |
| 587 | processing of signatures. The options passed to -S allow different |
| 588 | debugging modes to be used, either using BitBake's own debug functions |
| 589 | or possibly those defined in the metadata/signature handler itself. The |
| 590 | simplest parameter to pass is "none", which causes a set of signature |
| 591 | information to be written out into ``STAMPS_DIR`` corresponding to the |
| 592 | targets specified. The other currently available parameter is |
| 593 | "printdiff", which causes BitBake to try to establish the closest |
| 594 | signature match it can (e.g. in the sstate cache) and then run |
| 595 | ``bitbake-diffsigs`` over the matches to determine the stamps and delta |
| 596 | where these two stamp trees diverge. |
| 597 | |
| 598 | .. note:: |
| 599 | |
| 600 | It is likely that future versions of BitBake will provide other |
| 601 | signature handlers triggered through additional "-S" parameters. |
| 602 | |
| 603 | You can find more information on checksum metadata in the |
| 604 | :ref:`bitbake-user-manual/bitbake-user-manual-metadata:task checksums and setscene` |
| 605 | section. |
| 606 | |
| 607 | Setscene |
| 608 | ======== |
| 609 | |
| 610 | The setscene process enables BitBake to handle "pre-built" artifacts. |
| 611 | The ability to handle and reuse these artifacts allows BitBake the |
| 612 | luxury of not having to build something from scratch every time. |
| 613 | Instead, BitBake can use, when possible, existing build artifacts. |
| 614 | |
| 615 | BitBake needs to have reliable data indicating whether or not an |
| 616 | artifact is compatible. Signatures, described in the previous section, |
| 617 | provide an ideal way of representing whether an artifact is compatible. |
| 618 | If a signature is the same, an object can be reused. |
| 619 | |
| 620 | If an object can be reused, the problem then becomes how to replace a |
| 621 | given task or set of tasks with the pre-built artifact. BitBake solves |
| 622 | the problem with the "setscene" process. |
| 623 | |
| 624 | When BitBake is asked to build a given target, before building anything, |
| 625 | it first asks whether cached information is available for any of the |
| 626 | targets it's building, or any of the intermediate targets. If cached |
| 627 | information is available, BitBake uses this information instead of |
| 628 | running the main tasks. |
| 629 | |
| 630 | BitBake first calls the function defined by the |
| 631 | :term:`BB_HASHCHECK_FUNCTION` variable |
| 632 | with a list of tasks and corresponding hashes it wants to build. This |
| 633 | function is designed to be fast and returns a list of the tasks for |
| 634 | which it believes in can obtain artifacts. |
| 635 | |
| 636 | Next, for each of the tasks that were returned as possibilities, BitBake |
| 637 | executes a setscene version of the task that the possible artifact |
| 638 | covers. Setscene versions of a task have the string "_setscene" appended |
| 639 | to the task name. So, for example, the task with the name ``xxx`` has a |
| 640 | setscene task named ``xxx_setscene``. The setscene version of the task |
| 641 | executes and provides the necessary artifacts returning either success |
| 642 | or failure. |
| 643 | |
| 644 | As previously mentioned, an artifact can cover more than one task. For |
| 645 | example, it is pointless to obtain a compiler if you already have the |
| 646 | compiled binary. To handle this, BitBake calls the |
| 647 | :term:`BB_SETSCENE_DEPVALID` function for |
| 648 | each successful setscene task to know whether or not it needs to obtain |
| 649 | the dependencies of that task. |
| 650 | |
Andrew Geissler | c9f7865 | 2020-09-18 14:11:35 -0500 | [diff] [blame] | 651 | You can find more information on setscene metadata in the |
| 652 | :ref:`bitbake-user-manual/bitbake-user-manual-metadata:task checksums and setscene` |
| 653 | section. |
| 654 | |
| 655 | Logging |
| 656 | ======= |
| 657 | |
| 658 | In addition to the standard command line option to control how verbose |
| 659 | builds are when execute, bitbake also supports user defined |
| 660 | configuration of the `Python |
| 661 | logging <https://docs.python.org/3/library/logging.html>`__ facilities |
| 662 | through the :term:`BB_LOGCONFIG` variable. This |
| 663 | variable defines a json or yaml `logging |
| 664 | configuration <https://docs.python.org/3/library/logging.config.html>`__ |
| 665 | that will be intelligently merged into the default configuration. The |
| 666 | logging configuration is merged using the following rules: |
| 667 | |
| 668 | - The user defined configuration will completely replace the default |
| 669 | configuration if top level key ``bitbake_merge`` is set to the value |
| 670 | ``False``. In this case, all other rules are ignored. |
| 671 | |
| 672 | - The user configuration must have a top level ``version`` which must |
| 673 | match the value of the default configuration. |
| 674 | |
| 675 | - Any keys defined in the ``handlers``, ``formatters``, or ``filters``, |
| 676 | will be merged into the same section in the default configuration, |
| 677 | with the user specified keys taking replacing a default one if there |
| 678 | is a conflict. In practice, this means that if both the default |
| 679 | configuration and user configuration specify a handler named |
| 680 | ``myhandler``, the user defined one will replace the default. To |
| 681 | prevent the user from inadvertently replacing a default handler, |
| 682 | formatter, or filter, all of the default ones are named with a prefix |
| 683 | of "``BitBake.``" |
| 684 | |
| 685 | - If a logger is defined by the user with the key ``bitbake_merge`` set |
| 686 | to ``False``, that logger will be completely replaced by user |
| 687 | configuration. In this case, no other rules will apply to that |
| 688 | logger. |
| 689 | |
| 690 | - All user defined ``filter`` and ``handlers`` properties for a given |
| 691 | logger will be merged with corresponding properties from the default |
| 692 | logger. For example, if the user configuration adds a filter called |
| 693 | ``myFilter`` to the ``BitBake.SigGen``, and the default configuration |
| 694 | adds a filter called ``BitBake.defaultFilter``, both filters will be |
| 695 | applied to the logger |
| 696 | |
| 697 | As an example, consider the following user logging configuration file |
| 698 | which logs all Hash Equivalence related messages of VERBOSE or higher to |
| 699 | a file called ``hashequiv.log`` :: |
| 700 | |
| 701 | { |
| 702 | "version": 1, |
| 703 | "handlers": { |
| 704 | "autobuilderlog": { |
| 705 | "class": "logging.FileHandler", |
| 706 | "formatter": "logfileFormatter", |
| 707 | "level": "DEBUG", |
| 708 | "filename": "hashequiv.log", |
| 709 | "mode": "w" |
| 710 | } |
| 711 | }, |
| 712 | "formatters": { |
| 713 | "logfileFormatter": { |
| 714 | "format": "%(name)s: %(levelname)s: %(message)s" |
| 715 | } |
| 716 | }, |
| 717 | "loggers": { |
| 718 | "BitBake.SigGen.HashEquiv": { |
| 719 | "level": "VERBOSE", |
| 720 | "handlers": ["autobuilderlog"] |
| 721 | }, |
| 722 | "BitBake.RunQueue.HashEquiv": { |
| 723 | "level": "VERBOSE", |
| 724 | "handlers": ["autobuilderlog"] |
| 725 | } |
| 726 | } |
| 727 | } |