Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 1 | <!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" |
| 2 | "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd" |
| 3 | [<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] > |
| 4 | |
| 5 | <chapter id='technical-details'> |
| 6 | <title>Technical Details</title> |
| 7 | |
| 8 | <para> |
| 9 | This chapter provides technical details for various parts of the |
| 10 | Yocto Project. |
| 11 | Currently, topics include Yocto Project components, |
| 12 | cross-toolchain generation, shared state (sstate) cache, |
| 13 | x32, Wayland support, and Licenses. |
| 14 | </para> |
| 15 | |
| 16 | <section id='usingpoky-components'> |
| 17 | <title>Yocto Project Components</title> |
| 18 | |
| 19 | <para> |
| 20 | The |
Brad Bishop | d7bf8c1 | 2018-02-25 22:55:05 -0500 | [diff] [blame] | 21 | <link linkend='bitbake-term'>BitBake</link> |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 22 | task executor together with various types of configuration files form |
| 23 | the OpenEmbedded Core. |
| 24 | This section overviews these components by describing their use and |
| 25 | how they interact. |
| 26 | </para> |
| 27 | |
| 28 | <para> |
| 29 | BitBake handles the parsing and execution of the data files. |
| 30 | The data itself is of various types: |
| 31 | <itemizedlist> |
| 32 | <listitem><para><emphasis>Recipes:</emphasis> Provides details |
| 33 | about particular pieces of software. |
| 34 | </para></listitem> |
| 35 | <listitem><para><emphasis>Class Data:</emphasis> Abstracts |
| 36 | common build information (e.g. how to build a Linux kernel). |
| 37 | </para></listitem> |
| 38 | <listitem><para><emphasis>Configuration Data:</emphasis> Defines |
| 39 | machine-specific settings, policy decisions, and so forth. |
| 40 | Configuration data acts as the glue to bind everything |
| 41 | together. |
| 42 | </para></listitem> |
| 43 | </itemizedlist> |
| 44 | </para> |
| 45 | |
| 46 | <para> |
| 47 | BitBake knows how to combine multiple data sources together and refers |
| 48 | to each data source as a layer. |
| 49 | For information on layers, see the |
| 50 | "<ulink url='&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers'>Understanding and |
Brad Bishop | d7bf8c1 | 2018-02-25 22:55:05 -0500 | [diff] [blame] | 51 | Creating Layers</ulink>" section of the Yocto Project Development |
| 52 | Tasks Manual. |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 53 | </para> |
| 54 | |
| 55 | <para> |
| 56 | Following are some brief details on these core components. |
| 57 | For additional information on how these components interact during |
| 58 | a build, see the |
Brad Bishop | d7bf8c1 | 2018-02-25 22:55:05 -0500 | [diff] [blame] | 59 | "<link linkend='development-concepts'>Development Concepts</link>" |
| 60 | section. |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 61 | </para> |
| 62 | |
| 63 | <section id='usingpoky-components-bitbake'> |
| 64 | <title>BitBake</title> |
| 65 | |
| 66 | <para> |
| 67 | BitBake is the tool at the heart of the OpenEmbedded build system |
| 68 | and is responsible for parsing the |
Brad Bishop | d7bf8c1 | 2018-02-25 22:55:05 -0500 | [diff] [blame] | 69 | <link linkend='metadata'>Metadata</link>, |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 70 | generating a list of tasks from it, and then executing those tasks. |
| 71 | </para> |
| 72 | |
| 73 | <para> |
| 74 | This section briefly introduces BitBake. |
| 75 | If you want more information on BitBake, see the |
| 76 | <ulink url='&YOCTO_DOCS_BB_URL;#bitbake-user-manual'>BitBake User Manual</ulink>. |
| 77 | </para> |
| 78 | |
| 79 | <para> |
| 80 | To see a list of the options BitBake supports, use either of |
| 81 | the following commands: |
| 82 | <literallayout class='monospaced'> |
| 83 | $ bitbake -h |
| 84 | $ bitbake --help |
| 85 | </literallayout> |
| 86 | </para> |
| 87 | |
| 88 | <para> |
| 89 | The most common usage for BitBake is <filename>bitbake <replaceable>packagename</replaceable></filename>, where |
| 90 | <filename>packagename</filename> is the name of the package you want to build |
| 91 | (referred to as the "target" in this manual). |
| 92 | The target often equates to the first part of a recipe's filename |
| 93 | (e.g. "foo" for a recipe named |
| 94 | <filename>foo_1.3.0-r0.bb</filename>). |
| 95 | So, to process the <filename>matchbox-desktop_1.2.3.bb</filename> recipe file, you |
| 96 | might type the following: |
| 97 | <literallayout class='monospaced'> |
| 98 | $ bitbake matchbox-desktop |
| 99 | </literallayout> |
| 100 | Several different versions of <filename>matchbox-desktop</filename> might exist. |
| 101 | BitBake chooses the one selected by the distribution configuration. |
| 102 | You can get more details about how BitBake chooses between different |
| 103 | target versions and providers in the |
| 104 | "<ulink url='&YOCTO_DOCS_BB_URL;#bb-bitbake-preferences'>Preferences</ulink>" |
| 105 | section of the BitBake User Manual. |
| 106 | </para> |
| 107 | |
| 108 | <para> |
| 109 | BitBake also tries to execute any dependent tasks first. |
| 110 | So for example, before building <filename>matchbox-desktop</filename>, BitBake |
| 111 | would build a cross compiler and <filename>glibc</filename> if they had not already |
| 112 | been built. |
| 113 | </para> |
| 114 | |
| 115 | <para> |
| 116 | A useful BitBake option to consider is the <filename>-k</filename> or |
| 117 | <filename>--continue</filename> option. |
| 118 | This option instructs BitBake to try and continue processing the job |
| 119 | as long as possible even after encountering an error. |
| 120 | When an error occurs, the target that |
| 121 | failed and those that depend on it cannot be remade. |
| 122 | However, when you use this option other dependencies can still be |
| 123 | processed. |
| 124 | </para> |
| 125 | </section> |
| 126 | |
| 127 | <section id='usingpoky-components-metadata'> |
| 128 | <title>Metadata (Recipes)</title> |
| 129 | |
| 130 | <para> |
| 131 | Files that have the <filename>.bb</filename> suffix are "recipes" |
| 132 | files. |
| 133 | In general, a recipe contains information about a single piece of |
| 134 | software. |
| 135 | This information includes the location from which to download the |
| 136 | unaltered source, any source patches to be applied to that source |
| 137 | (if needed), which special configuration options to apply, |
| 138 | how to compile the source files, and how to package the compiled |
| 139 | output. |
| 140 | </para> |
| 141 | |
| 142 | <para> |
| 143 | The term "package" is sometimes used to refer to recipes. However, |
| 144 | since the word "package" is used for the packaged output from the OpenEmbedded |
| 145 | build system (i.e. <filename>.ipk</filename> or <filename>.deb</filename> files), |
| 146 | this document avoids using the term "package" when referring to recipes. |
| 147 | </para> |
| 148 | </section> |
| 149 | |
Brad Bishop | d7bf8c1 | 2018-02-25 22:55:05 -0500 | [diff] [blame] | 150 | <section id='metadata-virtual-providers'> |
| 151 | <title>Metadata (Virtual Providers)</title> |
| 152 | |
| 153 | <para> |
| 154 | Prior to the build, if you know that several different recipes |
| 155 | provide the same functionality, you can use a virtual provider |
| 156 | (i.e. <filename>virtual/*</filename>) as a placeholder for the |
| 157 | actual provider. |
| 158 | The actual provider would be determined at build |
| 159 | time. |
| 160 | In this case, you should add <filename>virtual/*</filename> |
| 161 | to <link linkend='var-DEPENDS'><filename>DEPENDS</filename></link>, |
| 162 | rather than listing the specified provider. |
| 163 | You would select the actual provider by setting the |
| 164 | <link linkend='var-PREFERRED_PROVIDER'><filename>PREFERRED_PROVIDER</filename></link> |
| 165 | variable (i.e. <filename>PREFERRED_PROVIDER_virtual/*</filename>) |
| 166 | in the build's configuration file (e.g. |
| 167 | <filename>poky/build/conf/local.conf</filename>). |
| 168 | <note> |
| 169 | Any recipe that PROVIDES a <filename>virtual/*</filename> item |
| 170 | that is ultimately not selected through |
| 171 | <filename>PREFERRED_PROVIDER</filename> does not get built. |
| 172 | Preventing these recipes from building is usually the desired |
| 173 | behavior since this mechanism's purpose is to select between |
| 174 | mutually exclusive alternative providers. |
| 175 | </note> |
| 176 | </para> |
| 177 | |
| 178 | <para> |
| 179 | The following lists specific examples of virtual providers: |
| 180 | <itemizedlist> |
| 181 | <listitem><para> |
| 182 | <filename>virtual/mesa</filename>: |
| 183 | Provides <filename>gbm.pc</filename>. |
| 184 | </para></listitem> |
| 185 | <listitem><para> |
| 186 | <filename>virtual/egl</filename>: |
| 187 | Provides <filename>egl.pc</filename> and possibly |
| 188 | <filename>wayland-egl.pc</filename>. |
| 189 | </para></listitem> |
| 190 | <listitem><para> |
| 191 | <filename>virtual/libgl</filename>: |
| 192 | Provides <filename>gl.pc</filename> (i.e. libGL). |
| 193 | </para></listitem> |
| 194 | <listitem><para> |
| 195 | <filename>virtual/libgles1</filename>: |
| 196 | Provides <filename>glesv1_cm.pc</filename> |
| 197 | (i.e. libGLESv1_CM). |
| 198 | </para></listitem> |
| 199 | <listitem><para> |
| 200 | <filename>virtual/libgles2</filename>: |
| 201 | Provides <filename>glesv2.pc</filename> (i.e. libGLESv2). |
| 202 | </para></listitem> |
| 203 | </itemizedlist> |
| 204 | </para> |
| 205 | </section> |
| 206 | |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 207 | <section id='usingpoky-components-classes'> |
| 208 | <title>Classes</title> |
| 209 | |
| 210 | <para> |
| 211 | Class files (<filename>.bbclass</filename>) contain information that |
| 212 | is useful to share between |
Brad Bishop | d7bf8c1 | 2018-02-25 22:55:05 -0500 | [diff] [blame] | 213 | <link linkend='metadata'>Metadata</link> files. |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 214 | An example is the |
| 215 | <link linkend='ref-classes-autotools'><filename>autotools</filename></link> |
| 216 | class, which contains common settings for any application that |
| 217 | Autotools uses. |
| 218 | The "<link linkend='ref-classes'>Classes</link>" chapter provides |
| 219 | details about classes and how to use them. |
| 220 | </para> |
| 221 | </section> |
| 222 | |
| 223 | <section id='usingpoky-components-configuration'> |
| 224 | <title>Configuration</title> |
| 225 | |
| 226 | <para> |
| 227 | The configuration files (<filename>.conf</filename>) define various configuration variables |
| 228 | that govern the OpenEmbedded build process. |
| 229 | These files fall into several areas that define machine configuration options, |
| 230 | distribution configuration options, compiler tuning options, general common configuration |
| 231 | options, and user configuration options in <filename>local.conf</filename>, which is found |
| 232 | in the |
Brad Bishop | d7bf8c1 | 2018-02-25 22:55:05 -0500 | [diff] [blame] | 233 | <link linkend='build-directory'>Build Directory</link>. |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 234 | </para> |
| 235 | </section> |
| 236 | </section> |
| 237 | |
| 238 | <section id="cross-development-toolchain-generation"> |
| 239 | <title>Cross-Development Toolchain Generation</title> |
| 240 | |
| 241 | <para> |
| 242 | The Yocto Project does most of the work for you when it comes to |
| 243 | creating |
Brad Bishop | d7bf8c1 | 2018-02-25 22:55:05 -0500 | [diff] [blame] | 244 | <link linkend='cross-development-toolchain'>cross-development toolchains</link>. |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 245 | This section provides some technical background on how |
| 246 | cross-development toolchains are created and used. |
| 247 | For more information on toolchains, you can also see the |
Brad Bishop | d7bf8c1 | 2018-02-25 22:55:05 -0500 | [diff] [blame] | 248 | <ulink url='&YOCTO_DOCS_SDK_URL;'>Yocto Project Application Development and the Extensible Software Development Kit (eSDK)</ulink> |
| 249 | manual. |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 250 | </para> |
| 251 | |
| 252 | <para> |
| 253 | In the Yocto Project development environment, cross-development |
| 254 | toolchains are used to build the image and applications that run on the |
| 255 | target hardware. |
| 256 | With just a few commands, the OpenEmbedded build system creates |
| 257 | these necessary toolchains for you. |
| 258 | </para> |
| 259 | |
| 260 | <para> |
| 261 | The following figure shows a high-level build environment regarding |
| 262 | toolchain construction and use. |
| 263 | </para> |
| 264 | |
| 265 | <para> |
| 266 | <imagedata fileref="figures/cross-development-toolchains.png" width="8in" depth="6in" align="center" /> |
| 267 | </para> |
| 268 | |
| 269 | <para> |
| 270 | Most of the work occurs on the Build Host. |
| 271 | This is the machine used to build images and generally work within the |
| 272 | the Yocto Project environment. |
| 273 | When you run BitBake to create an image, the OpenEmbedded build system |
| 274 | uses the host <filename>gcc</filename> compiler to bootstrap a |
| 275 | cross-compiler named <filename>gcc-cross</filename>. |
| 276 | The <filename>gcc-cross</filename> compiler is what BitBake uses to |
| 277 | compile source files when creating the target image. |
| 278 | You can think of <filename>gcc-cross</filename> simply as an |
| 279 | automatically generated cross-compiler that is used internally within |
| 280 | BitBake only. |
Patrick Williams | d8c66bc | 2016-06-20 12:57:21 -0500 | [diff] [blame] | 281 | <note> |
| 282 | The extensible SDK does not use |
| 283 | <filename>gcc-cross-canadian</filename> since this SDK |
| 284 | ships a copy of the OpenEmbedded build system and the sysroot |
| 285 | within it contains <filename>gcc-cross</filename>. |
| 286 | </note> |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 287 | </para> |
| 288 | |
| 289 | <para> |
| 290 | The chain of events that occurs when <filename>gcc-cross</filename> is |
| 291 | bootstrapped is as follows: |
| 292 | <literallayout class='monospaced'> |
| 293 | gcc -> binutils-cross -> gcc-cross-initial -> linux-libc-headers -> glibc-initial -> glibc -> gcc-cross -> gcc-runtime |
| 294 | </literallayout> |
| 295 | <itemizedlist> |
| 296 | <listitem><para><filename>gcc</filename>: |
| 297 | The build host's GNU Compiler Collection (GCC). |
| 298 | </para></listitem> |
| 299 | <listitem><para><filename>binutils-cross</filename>: |
| 300 | The bare minimum binary utilities needed in order to run |
| 301 | the <filename>gcc-cross-initial</filename> phase of the |
| 302 | bootstrap operation. |
| 303 | </para></listitem> |
| 304 | <listitem><para><filename>gcc-cross-initial</filename>: |
| 305 | An early stage of the bootstrap process for creating |
| 306 | the cross-compiler. |
| 307 | This stage builds enough of the <filename>gcc-cross</filename>, |
| 308 | the C library, and other pieces needed to finish building the |
| 309 | final cross-compiler in later stages. |
| 310 | This tool is a "native" package (i.e. it is designed to run on |
| 311 | the build host). |
| 312 | </para></listitem> |
| 313 | <listitem><para><filename>linux-libc-headers</filename>: |
| 314 | Headers needed for the cross-compiler. |
| 315 | </para></listitem> |
| 316 | <listitem><para><filename>glibc-initial</filename>: |
| 317 | An initial version of the Embedded GLIBC needed to bootstrap |
| 318 | <filename>glibc</filename>. |
| 319 | </para></listitem> |
| 320 | <listitem><para><filename>gcc-cross</filename>: |
| 321 | The final stage of the bootstrap process for the |
| 322 | cross-compiler. |
| 323 | This stage results in the actual cross-compiler that |
| 324 | BitBake uses when it builds an image for a targeted |
| 325 | device. |
| 326 | <note> |
| 327 | If you are replacing this cross compiler toolchain |
| 328 | with a custom version, you must replace |
| 329 | <filename>gcc-cross</filename>. |
| 330 | </note> |
| 331 | This tool is also a "native" package (i.e. it is |
| 332 | designed to run on the build host). |
| 333 | </para></listitem> |
| 334 | <listitem><para><filename>gcc-runtime</filename>: |
| 335 | Runtime libraries resulting from the toolchain bootstrapping |
| 336 | process. |
| 337 | This tool produces a binary that consists of the |
| 338 | runtime libraries need for the targeted device. |
| 339 | </para></listitem> |
| 340 | </itemizedlist> |
| 341 | </para> |
| 342 | |
| 343 | <para> |
| 344 | You can use the OpenEmbedded build system to build an installer for |
| 345 | the relocatable SDK used to develop applications. |
| 346 | When you run the installer, it installs the toolchain, which contains |
| 347 | the development tools (e.g., the |
| 348 | <filename>gcc-cross-canadian</filename>), |
| 349 | <filename>binutils-cross-canadian</filename>, and other |
Patrick Williams | d8c66bc | 2016-06-20 12:57:21 -0500 | [diff] [blame] | 350 | <filename>nativesdk-*</filename> tools, |
| 351 | which are tools native to the SDK (i.e. native to |
| 352 | <link linkend='var-SDK_ARCH'><filename>SDK_ARCH</filename></link>), |
| 353 | you need to cross-compile and test your software. |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 354 | The figure shows the commands you use to easily build out this |
| 355 | toolchain. |
| 356 | This cross-development toolchain is built to execute on the |
| 357 | <link linkend='var-SDKMACHINE'><filename>SDKMACHINE</filename></link>, |
| 358 | which might or might not be the same |
| 359 | machine as the Build Host. |
| 360 | <note> |
| 361 | If your target architecture is supported by the Yocto Project, |
| 362 | you can take advantage of pre-built images that ship with the |
| 363 | Yocto Project and already contain cross-development toolchain |
| 364 | installers. |
| 365 | </note> |
| 366 | </para> |
| 367 | |
| 368 | <para> |
| 369 | Here is the bootstrap process for the relocatable toolchain: |
| 370 | <literallayout class='monospaced'> |
| 371 | gcc -> binutils-crosssdk -> gcc-crosssdk-initial -> linux-libc-headers -> |
| 372 | glibc-initial -> nativesdk-glibc -> gcc-crosssdk -> gcc-cross-canadian |
| 373 | </literallayout> |
| 374 | <itemizedlist> |
| 375 | <listitem><para><filename>gcc</filename>: |
| 376 | The build host's GNU Compiler Collection (GCC). |
| 377 | </para></listitem> |
| 378 | <listitem><para><filename>binutils-crosssdk</filename>: |
| 379 | The bare minimum binary utilities needed in order to run |
| 380 | the <filename>gcc-crosssdk-initial</filename> phase of the |
| 381 | bootstrap operation. |
| 382 | </para></listitem> |
| 383 | <listitem><para><filename>gcc-crosssdk-initial</filename>: |
| 384 | An early stage of the bootstrap process for creating |
| 385 | the cross-compiler. |
| 386 | This stage builds enough of the |
| 387 | <filename>gcc-crosssdk</filename> and supporting pieces so that |
| 388 | the final stage of the bootstrap process can produce the |
| 389 | finished cross-compiler. |
| 390 | This tool is a "native" binary that runs on the build host. |
| 391 | </para></listitem> |
| 392 | <listitem><para><filename>linux-libc-headers</filename>: |
| 393 | Headers needed for the cross-compiler. |
| 394 | </para></listitem> |
| 395 | <listitem><para><filename>glibc-initial</filename>: |
| 396 | An initial version of the Embedded GLIBC needed to bootstrap |
| 397 | <filename>nativesdk-glibc</filename>. |
| 398 | </para></listitem> |
| 399 | <listitem><para><filename>nativesdk-glibc</filename>: |
| 400 | The Embedded GLIBC needed to bootstrap the |
| 401 | <filename>gcc-crosssdk</filename>. |
| 402 | </para></listitem> |
| 403 | <listitem><para><filename>gcc-crosssdk</filename>: |
| 404 | The final stage of the bootstrap process for the |
| 405 | relocatable cross-compiler. |
| 406 | The <filename>gcc-crosssdk</filename> is a transitory compiler |
| 407 | and never leaves the build host. |
| 408 | Its purpose is to help in the bootstrap process to create the |
| 409 | eventual relocatable <filename>gcc-cross-canadian</filename> |
| 410 | compiler, which is relocatable. |
| 411 | This tool is also a "native" package (i.e. it is |
| 412 | designed to run on the build host). |
| 413 | </para></listitem> |
| 414 | <listitem><para><filename>gcc-cross-canadian</filename>: |
| 415 | The final relocatable cross-compiler. |
| 416 | When run on the |
| 417 | <link linkend='var-SDKMACHINE'><filename>SDKMACHINE</filename></link>, |
| 418 | this tool |
| 419 | produces executable code that runs on the target device. |
| 420 | Only one cross-canadian compiler is produced per architecture |
| 421 | since they can be targeted at different processor optimizations |
| 422 | using configurations passed to the compiler through the |
| 423 | compile commands. |
| 424 | This circumvents the need for multiple compilers and thus |
| 425 | reduces the size of the toolchains. |
| 426 | </para></listitem> |
| 427 | </itemizedlist> |
| 428 | </para> |
| 429 | |
| 430 | <note> |
| 431 | For information on advantages gained when building a |
| 432 | cross-development toolchain installer, see the |
Patrick Williams | d8c66bc | 2016-06-20 12:57:21 -0500 | [diff] [blame] | 433 | "<ulink url='&YOCTO_DOCS_SDK_URL;#sdk-building-an-sdk-installer'>Building an SDK Installer</ulink>" |
Brad Bishop | d7bf8c1 | 2018-02-25 22:55:05 -0500 | [diff] [blame] | 434 | section in the Yocto Project Application Development and the |
| 435 | Extensible Software Development Kit (eSDK) manual. |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 436 | </note> |
| 437 | </section> |
| 438 | |
| 439 | <section id="shared-state-cache"> |
| 440 | <title>Shared State Cache</title> |
| 441 | |
| 442 | <para> |
| 443 | By design, the OpenEmbedded build system builds everything from scratch unless |
| 444 | BitBake can determine that parts do not need to be rebuilt. |
| 445 | Fundamentally, building from scratch is attractive as it means all parts are |
| 446 | built fresh and there is no possibility of stale data causing problems. |
| 447 | When developers hit problems, they typically default back to building from scratch |
| 448 | so they know the state of things from the start. |
| 449 | </para> |
| 450 | |
| 451 | <para> |
| 452 | Building an image from scratch is both an advantage and a disadvantage to the process. |
| 453 | As mentioned in the previous paragraph, building from scratch ensures that |
| 454 | everything is current and starts from a known state. |
| 455 | However, building from scratch also takes much longer as it generally means |
| 456 | rebuilding things that do not necessarily need to be rebuilt. |
| 457 | </para> |
| 458 | |
| 459 | <para> |
| 460 | The Yocto Project implements shared state code that supports incremental builds. |
| 461 | The implementation of the shared state code answers the following questions that |
| 462 | were fundamental roadblocks within the OpenEmbedded incremental build support system: |
| 463 | <itemizedlist> |
| 464 | <listitem><para>What pieces of the system have changed and what pieces have |
| 465 | not changed?</para></listitem> |
| 466 | <listitem><para>How are changed pieces of software removed and replaced?</para></listitem> |
| 467 | <listitem><para>How are pre-built components that do not need to be rebuilt from scratch |
| 468 | used when they are available?</para></listitem> |
| 469 | </itemizedlist> |
| 470 | </para> |
| 471 | |
| 472 | <para> |
| 473 | For the first question, the build system detects changes in the "inputs" to a given task by |
| 474 | creating a checksum (or signature) of the task's inputs. |
| 475 | If the checksum changes, the system assumes the inputs have changed and the task needs to be |
| 476 | rerun. |
| 477 | For the second question, the shared state (sstate) code tracks which tasks add which output |
| 478 | to the build process. |
| 479 | This means the output from a given task can be removed, upgraded or otherwise manipulated. |
| 480 | The third question is partly addressed by the solution for the second question |
| 481 | assuming the build system can fetch the sstate objects from remote locations and |
| 482 | install them if they are deemed to be valid. |
| 483 | </para> |
| 484 | |
| 485 | <note> |
| 486 | The OpenEmbedded build system does not maintain |
| 487 | <link linkend='var-PR'><filename>PR</filename></link> information |
| 488 | as part of the shared state packages. |
| 489 | Consequently, considerations exist that affect maintaining shared |
| 490 | state feeds. |
| 491 | For information on how the OpenEmbedded build system |
| 492 | works with packages and can |
| 493 | track incrementing <filename>PR</filename> information, see the |
Brad Bishop | 6e60e8b | 2018-02-01 10:27:11 -0500 | [diff] [blame] | 494 | "<ulink url='&YOCTO_DOCS_DEV_URL;#automatically-incrementing-a-binary-package-revision-number'>Automatically Incrementing a Binary Package Revision Number</ulink>" |
Brad Bishop | d7bf8c1 | 2018-02-25 22:55:05 -0500 | [diff] [blame] | 495 | section in the Yocto Project Development Tasks Manual. |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 496 | </note> |
| 497 | |
| 498 | <para> |
| 499 | The rest of this section goes into detail about the overall incremental build |
| 500 | architecture, the checksums (signatures), shared state, and some tips and tricks. |
| 501 | </para> |
| 502 | |
| 503 | <section id='overall-architecture'> |
| 504 | <title>Overall Architecture</title> |
| 505 | |
| 506 | <para> |
| 507 | When determining what parts of the system need to be built, BitBake |
| 508 | works on a per-task basis rather than a per-recipe basis. |
| 509 | You might wonder why using a per-task basis is preferred over a per-recipe basis. |
| 510 | To help explain, consider having the IPK packaging backend enabled and then switching to DEB. |
| 511 | In this case, the |
| 512 | <link linkend='ref-tasks-install'><filename>do_install</filename></link> |
| 513 | and |
| 514 | <link linkend='ref-tasks-package'><filename>do_package</filename></link> |
| 515 | task outputs are still valid. |
| 516 | However, with a per-recipe approach, the build would not include the |
| 517 | <filename>.deb</filename> files. |
| 518 | Consequently, you would have to invalidate the whole build and rerun it. |
| 519 | Rerunning everything is not the best solution. |
| 520 | Also, in this case, the core must be "taught" much about specific tasks. |
| 521 | This methodology does not scale well and does not allow users to easily add new tasks |
| 522 | in layers or as external recipes without touching the packaged-staging core. |
| 523 | </para> |
| 524 | </section> |
| 525 | |
| 526 | <section id='checksums'> |
| 527 | <title>Checksums (Signatures)</title> |
| 528 | |
| 529 | <para> |
| 530 | The shared state code uses a checksum, which is a unique signature of a task's |
| 531 | inputs, to determine if a task needs to be run again. |
| 532 | Because it is a change in a task's inputs that triggers a rerun, the process |
| 533 | needs to detect all the inputs to a given task. |
| 534 | For shell tasks, this turns out to be fairly easy because |
| 535 | the build process generates a "run" shell script for each task and |
| 536 | it is possible to create a checksum that gives you a good idea of when |
| 537 | the task's data changes. |
| 538 | </para> |
| 539 | |
| 540 | <para> |
Patrick Williams | d8c66bc | 2016-06-20 12:57:21 -0500 | [diff] [blame] | 541 | To complicate the problem, there are things that should not be |
| 542 | included in the checksum. |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 543 | First, there is the actual specific build path of a given task - |
| 544 | the <link linkend='var-WORKDIR'><filename>WORKDIR</filename></link>. |
Patrick Williams | d8c66bc | 2016-06-20 12:57:21 -0500 | [diff] [blame] | 545 | It does not matter if the work directory changes because it should |
| 546 | not affect the output for target packages. |
| 547 | Also, the build process has the objective of making native |
| 548 | or cross packages relocatable. |
| 549 | <note> |
| 550 | Both native and cross packages run on the build host. |
| 551 | However, cross packages generate output for the target |
| 552 | architecture. |
| 553 | </note> |
| 554 | The checksum therefore needs to exclude |
| 555 | <filename>WORKDIR</filename>. |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 556 | The simplistic approach for excluding the work directory is to set |
Patrick Williams | d8c66bc | 2016-06-20 12:57:21 -0500 | [diff] [blame] | 557 | <filename>WORKDIR</filename> to some fixed value and create the |
| 558 | checksum for the "run" script. |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 559 | </para> |
| 560 | |
| 561 | <para> |
| 562 | Another problem results from the "run" scripts containing functions that |
| 563 | might or might not get called. |
| 564 | The incremental build solution contains code that figures out dependencies |
| 565 | between shell functions. |
| 566 | This code is used to prune the "run" scripts down to the minimum set, |
| 567 | thereby alleviating this problem and making the "run" scripts much more |
| 568 | readable as a bonus. |
| 569 | </para> |
| 570 | |
| 571 | <para> |
| 572 | So far we have solutions for shell scripts. |
| 573 | What about Python tasks? |
| 574 | The same approach applies even though these tasks are more difficult. |
| 575 | The process needs to figure out what variables a Python function accesses |
| 576 | and what functions it calls. |
| 577 | Again, the incremental build solution contains code that first figures out |
| 578 | the variable and function dependencies, and then creates a checksum for the data |
| 579 | used as the input to the task. |
| 580 | </para> |
| 581 | |
| 582 | <para> |
| 583 | Like the <filename>WORKDIR</filename> case, situations exist where dependencies |
| 584 | should be ignored. |
| 585 | For these cases, you can instruct the build process to ignore a dependency |
| 586 | by using a line like the following: |
| 587 | <literallayout class='monospaced'> |
| 588 | PACKAGE_ARCHS[vardepsexclude] = "MACHINE" |
| 589 | </literallayout> |
| 590 | This example ensures that the |
| 591 | <link linkend='var-PACKAGE_ARCHS'><filename>PACKAGE_ARCHS</filename></link> |
| 592 | variable does not |
| 593 | depend on the value of |
| 594 | <link linkend='var-MACHINE'><filename>MACHINE</filename></link>, |
| 595 | even if it does reference it. |
| 596 | </para> |
| 597 | |
| 598 | <para> |
| 599 | Equally, there are cases where we need to add dependencies BitBake is not able to find. |
| 600 | You can accomplish this by using a line like the following: |
| 601 | <literallayout class='monospaced'> |
| 602 | PACKAGE_ARCHS[vardeps] = "MACHINE" |
| 603 | </literallayout> |
| 604 | This example explicitly adds the <filename>MACHINE</filename> variable as a |
| 605 | dependency for <filename>PACKAGE_ARCHS</filename>. |
| 606 | </para> |
| 607 | |
| 608 | <para> |
| 609 | Consider a case with in-line Python, for example, where BitBake is not |
| 610 | able to figure out dependencies. |
| 611 | When running in debug mode (i.e. using <filename>-DDD</filename>), BitBake |
| 612 | produces output when it discovers something for which it cannot figure out |
| 613 | dependencies. |
| 614 | The Yocto Project team has currently not managed to cover those dependencies |
| 615 | in detail and is aware of the need to fix this situation. |
| 616 | </para> |
| 617 | |
| 618 | <para> |
| 619 | Thus far, this section has limited discussion to the direct inputs into a task. |
| 620 | Information based on direct inputs is referred to as the "basehash" in the |
| 621 | code. |
| 622 | However, there is still the question of a task's indirect inputs - the |
| 623 | things that were already built and present in the |
Brad Bishop | d7bf8c1 | 2018-02-25 22:55:05 -0500 | [diff] [blame] | 624 | <link linkend='build-directory'>Build Directory</link>. |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 625 | The checksum (or signature) for a particular task needs to add the hashes |
| 626 | of all the tasks on which the particular task depends. |
| 627 | Choosing which dependencies to add is a policy decision. |
| 628 | However, the effect is to generate a master checksum that combines the basehash |
| 629 | and the hashes of the task's dependencies. |
| 630 | </para> |
| 631 | |
| 632 | <para> |
| 633 | At the code level, there are a variety of ways both the basehash and the |
| 634 | dependent task hashes can be influenced. |
| 635 | Within the BitBake configuration file, we can give BitBake some extra information |
| 636 | to help it construct the basehash. |
| 637 | The following statement effectively results in a list of global variable |
| 638 | dependency excludes - variables never included in any checksum: |
| 639 | <literallayout class='monospaced'> |
| 640 | BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \ |
| 641 | SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \ |
| 642 | USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \ |
| 643 | PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \ |
| 644 | CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX" |
| 645 | </literallayout> |
| 646 | The previous example excludes |
| 647 | <link linkend='var-WORKDIR'><filename>WORKDIR</filename></link> |
| 648 | since that variable is actually constructed as a path within |
| 649 | <link linkend='var-TMPDIR'><filename>TMPDIR</filename></link>, which is on |
| 650 | the whitelist. |
| 651 | </para> |
| 652 | |
| 653 | <para> |
| 654 | The rules for deciding which hashes of dependent tasks to include through |
| 655 | dependency chains are more complex and are generally accomplished with a |
| 656 | Python function. |
| 657 | The code in <filename>meta/lib/oe/sstatesig.py</filename> shows two examples |
| 658 | of this and also illustrates how you can insert your own policy into the system |
| 659 | if so desired. |
Brad Bishop | d7bf8c1 | 2018-02-25 22:55:05 -0500 | [diff] [blame] | 660 | This file defines the two basic signature generators |
| 661 | <link linkend='oe-core'>OE-Core</link> uses: "OEBasic" and |
| 662 | "OEBasicHash". |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 663 | By default, there is a dummy "noop" signature handler enabled in BitBake. |
| 664 | This means that behavior is unchanged from previous versions. |
Brad Bishop | d7bf8c1 | 2018-02-25 22:55:05 -0500 | [diff] [blame] | 665 | OE-Core uses the "OEBasicHash" signature handler by default |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 666 | through this setting in the <filename>bitbake.conf</filename> file: |
| 667 | <literallayout class='monospaced'> |
| 668 | BB_SIGNATURE_HANDLER ?= "OEBasicHash" |
| 669 | </literallayout> |
| 670 | The "OEBasicHash" <filename>BB_SIGNATURE_HANDLER</filename> is the same as the |
| 671 | "OEBasic" version but adds the task hash to the stamp files. |
| 672 | This results in any |
Brad Bishop | d7bf8c1 | 2018-02-25 22:55:05 -0500 | [diff] [blame] | 673 | <link linkend='metadata'>Metadata</link> |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 674 | change that changes the task hash, automatically |
| 675 | causing the task to be run again. |
| 676 | This removes the need to bump <link linkend='var-PR'><filename>PR</filename></link> |
| 677 | values, and changes to Metadata automatically ripple across the build. |
| 678 | </para> |
| 679 | |
| 680 | <para> |
| 681 | It is also worth noting that the end result of these signature generators is to |
| 682 | make some dependency and hash information available to the build. |
| 683 | This information includes: |
| 684 | <itemizedlist> |
| 685 | <listitem><para><filename>BB_BASEHASH_task-</filename><replaceable>taskname</replaceable>: |
| 686 | The base hashes for each task in the recipe. |
| 687 | </para></listitem> |
| 688 | <listitem><para><filename>BB_BASEHASH_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>: |
| 689 | The base hashes for each dependent task. |
| 690 | </para></listitem> |
| 691 | <listitem><para><filename>BBHASHDEPS_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>: |
| 692 | The task dependencies for each task. |
| 693 | </para></listitem> |
| 694 | <listitem><para><filename>BB_TASKHASH</filename>: |
| 695 | The hash of the currently running task. |
| 696 | </para></listitem> |
| 697 | </itemizedlist> |
| 698 | </para> |
| 699 | </section> |
| 700 | |
| 701 | <section id='shared-state'> |
| 702 | <title>Shared State</title> |
| 703 | |
| 704 | <para> |
| 705 | Checksums and dependencies, as discussed in the previous section, solve half the |
| 706 | problem of supporting a shared state. |
| 707 | The other part of the problem is being able to use checksum information during the build |
| 708 | and being able to reuse or rebuild specific components. |
| 709 | </para> |
| 710 | |
| 711 | <para> |
| 712 | The |
| 713 | <link linkend='ref-classes-sstate'><filename>sstate</filename></link> |
| 714 | class is a relatively generic implementation of how to "capture" |
| 715 | a snapshot of a given task. |
| 716 | The idea is that the build process does not care about the source of a task's output. |
| 717 | Output could be freshly built or it could be downloaded and unpacked from |
| 718 | somewhere - the build process does not need to worry about its origin. |
| 719 | </para> |
| 720 | |
| 721 | <para> |
| 722 | There are two types of output, one is just about creating a directory |
| 723 | in <link linkend='var-WORKDIR'><filename>WORKDIR</filename></link>. |
| 724 | A good example is the output of either |
| 725 | <link linkend='ref-tasks-install'><filename>do_install</filename></link> |
| 726 | or |
| 727 | <link linkend='ref-tasks-package'><filename>do_package</filename></link>. |
| 728 | The other type of output occurs when a set of data is merged into a shared directory |
| 729 | tree such as the sysroot. |
| 730 | </para> |
| 731 | |
| 732 | <para> |
| 733 | The Yocto Project team has tried to keep the details of the |
| 734 | implementation hidden in <filename>sstate</filename> class. |
| 735 | From a user's perspective, adding shared state wrapping to a task |
| 736 | is as simple as this |
| 737 | <link linkend='ref-tasks-deploy'><filename>do_deploy</filename></link> |
| 738 | example taken from the |
| 739 | <link linkend='ref-classes-deploy'><filename>deploy</filename></link> |
| 740 | class: |
| 741 | <literallayout class='monospaced'> |
| 742 | DEPLOYDIR = "${WORKDIR}/deploy-${PN}" |
| 743 | SSTATETASKS += "do_deploy" |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 744 | do_deploy[sstate-inputdirs] = "${DEPLOYDIR}" |
| 745 | do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" |
| 746 | |
| 747 | python do_deploy_setscene () { |
| 748 | sstate_setscene(d) |
| 749 | } |
| 750 | addtask do_deploy_setscene |
| 751 | do_deploy[dirs] = "${DEPLOYDIR} ${B}" |
| 752 | </literallayout> |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 753 | The following list explains the previous example: |
| 754 | <itemizedlist> |
| 755 | <listitem><para> |
| 756 | Adding "do_deploy" to <filename>SSTATETASKS</filename> |
| 757 | adds some required sstate-related processing, which is |
| 758 | implemented in the |
| 759 | <link linkend='ref-classes-sstate'><filename>sstate</filename></link> |
| 760 | class, to before and after the |
| 761 | <link linkend='ref-tasks-deploy'><filename>do_deploy</filename></link> |
| 762 | task. |
| 763 | </para></listitem> |
| 764 | <listitem><para> |
| 765 | The |
| 766 | <filename>do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"</filename> |
| 767 | declares that <filename>do_deploy</filename> places its |
| 768 | output in <filename>${DEPLOYDIR}</filename> when run |
| 769 | normally (i.e. when not using the sstate cache). |
| 770 | This output becomes the input to the shared state cache. |
| 771 | </para></listitem> |
| 772 | <listitem><para> |
| 773 | The |
| 774 | <filename>do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"</filename> |
| 775 | line causes the contents of the shared state cache to be |
| 776 | copied to <filename>${DEPLOY_DIR_IMAGE}</filename>. |
| 777 | <note> |
| 778 | If <filename>do_deploy</filename> is not already in |
| 779 | the shared state cache or if its input checksum |
| 780 | (signature) has changed from when the output was |
| 781 | cached, the task will be run to populate the shared |
| 782 | state cache, after which the contents of the shared |
| 783 | state cache is copied to |
| 784 | <filename>${DEPLOY_DIR_IMAGE}</filename>. |
| 785 | If <filename>do_deploy</filename> is in the shared |
| 786 | state cache and its signature indicates that the |
| 787 | cached output is still valid (i.e. if no |
| 788 | relevant task inputs have changed), then the contents |
| 789 | of the shared state cache will be copied directly to |
| 790 | <filename>${DEPLOY_DIR_IMAGE}</filename> by the |
| 791 | <filename>do_deploy_setscene</filename> task instead, |
| 792 | skipping the <filename>do_deploy</filename> task. |
| 793 | </note> |
| 794 | </para></listitem> |
| 795 | <listitem><para> |
| 796 | The following task definition is glue logic needed to make |
| 797 | the previous settings effective: |
| 798 | <literallayout class='monospaced'> |
| 799 | python do_deploy_setscene () { |
| 800 | sstate_setscene(d) |
| 801 | } |
| 802 | addtask do_deploy_setscene |
| 803 | </literallayout> |
| 804 | <filename>sstate_setscene()</filename> takes the flags |
| 805 | above as input and accelerates the |
| 806 | <filename>do_deploy</filename> task through the |
| 807 | shared state cache if possible. |
| 808 | If the task was accelerated, |
| 809 | <filename>sstate_setscene()</filename> returns True. |
| 810 | Otherwise, it returns False, and the normal |
| 811 | <filename>do_deploy</filename> task runs. |
| 812 | For more information, see the |
| 813 | "<ulink url='&YOCTO_DOCS_BB_URL;#setscene'>setscene</ulink>" |
| 814 | section in the BitBake User Manual. |
| 815 | </para></listitem> |
| 816 | <listitem><para> |
| 817 | The <filename>do_deploy[dirs] = "${DEPLOYDIR} ${B}"</filename> |
| 818 | line creates <filename>${DEPLOYDIR}</filename> and |
| 819 | <filename>${B}</filename> before the |
| 820 | <filename>do_deploy</filename> task runs, and also sets |
| 821 | the current working directory of |
| 822 | <filename>do_deploy</filename> to |
| 823 | <filename>${B}</filename>. |
| 824 | For more information, see the |
| 825 | "<ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'>Variable Flags</ulink>" |
| 826 | section in the BitBake User Manual. |
| 827 | <note> |
| 828 | In cases where |
| 829 | <filename>sstate-inputdirs</filename> and |
| 830 | <filename>sstate-outputdirs</filename> would be the |
| 831 | same, you can use |
| 832 | <filename>sstate-plaindirs</filename>. |
| 833 | For example, to preserve the |
| 834 | <filename>${PKGD}</filename> and |
| 835 | <filename>${PKGDEST}</filename> output from the |
| 836 | <link linkend='ref-tasks-package'><filename>do_package</filename></link> |
| 837 | task, use the following: |
| 838 | <literallayout class='monospaced'> |
| 839 | do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}" |
| 840 | </literallayout> |
| 841 | </note> |
| 842 | </para></listitem> |
| 843 | <listitem><para> |
| 844 | <filename>sstate-inputdirs</filename> and |
| 845 | <filename>sstate-outputdirs</filename> can also be used |
| 846 | with multiple directories. |
| 847 | For example, the following declares |
| 848 | <filename>PKGDESTWORK</filename> and |
| 849 | <filename>SHLIBWORK</filename> as shared state |
| 850 | input directories, which populates the shared state |
| 851 | cache, and <filename>PKGDATA_DIR</filename> and |
| 852 | <filename>SHLIBSDIR</filename> as the corresponding |
| 853 | shared state output directories: |
| 854 | <literallayout class='monospaced'> |
| 855 | do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}" |
| 856 | do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}" |
| 857 | </literallayout> |
| 858 | </para></listitem> |
| 859 | <listitem><para> |
| 860 | These methods also include the ability to take a lockfile |
| 861 | when manipulating shared state directory structures, |
| 862 | for cases where file additions or removals are sensitive: |
| 863 | <literallayout class='monospaced'> |
| 864 | do_package[sstate-lockfile] = "${PACKAGELOCK}" |
| 865 | </literallayout> |
| 866 | </para></listitem> |
| 867 | </itemizedlist> |
| 868 | </para> |
| 869 | |
| 870 | <!-- |
| 871 | <para> |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 872 | In this example, we add some extra flags to the task, a name field ("deploy"), an |
| 873 | input directory where the task sends data, and the output |
| 874 | directory where the data from the task should eventually be copied. |
| 875 | We also add a <filename>_setscene</filename> variant of the task and add the task |
| 876 | name to the <filename>SSTATETASKS</filename> list. |
| 877 | </para> |
| 878 | |
| 879 | <para> |
| 880 | If you have a directory whose contents you need to preserve, you can do this with |
| 881 | a line like the following: |
| 882 | <literallayout class='monospaced'> |
| 883 | do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}" |
| 884 | </literallayout> |
| 885 | This method, as well as the following example, also works for multiple directories. |
| 886 | <literallayout class='monospaced'> |
| 887 | do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}" |
| 888 | do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}" |
| 889 | do_package[sstate-lockfile] = "${PACKAGELOCK}" |
| 890 | </literallayout> |
| 891 | These methods also include the ability to take a lockfile when manipulating |
| 892 | shared state directory structures since some cases are sensitive to file |
| 893 | additions or removals. |
| 894 | </para> |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 895 | --> |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 896 | |
| 897 | <para> |
| 898 | Behind the scenes, the shared state code works by looking in |
| 899 | <link linkend='var-SSTATE_DIR'><filename>SSTATE_DIR</filename></link> and |
| 900 | <link linkend='var-SSTATE_MIRRORS'><filename>SSTATE_MIRRORS</filename></link> |
| 901 | for shared state files. |
| 902 | Here is an example: |
| 903 | <literallayout class='monospaced'> |
| 904 | SSTATE_MIRRORS ?= "\ |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 905 | file://.* http://someserver.tld/share/sstate/PATH;downloadfilename=PATH \n \ |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 906 | file://.* file:///some/local/dir/sstate/PATH" |
| 907 | </literallayout> |
| 908 | <note> |
| 909 | The shared state directory (<filename>SSTATE_DIR</filename>) is |
| 910 | organized into two-character subdirectories, where the subdirectory |
| 911 | names are based on the first two characters of the hash. |
| 912 | If the shared state directory structure for a mirror has the |
| 913 | same structure as <filename>SSTATE_DIR</filename>, you must |
| 914 | specify "PATH" as part of the URI to enable the build system |
| 915 | to map to the appropriate subdirectory. |
| 916 | </note> |
| 917 | </para> |
| 918 | |
| 919 | <para> |
| 920 | The shared state package validity can be detected just by looking at the |
| 921 | filename since the filename contains the task checksum (or signature) as |
| 922 | described earlier in this section. |
| 923 | If a valid shared state package is found, the build process downloads it |
| 924 | and uses it to accelerate the task. |
| 925 | </para> |
| 926 | |
| 927 | <para> |
| 928 | The build processes use the <filename>*_setscene</filename> tasks |
| 929 | for the task acceleration phase. |
| 930 | BitBake goes through this phase before the main execution code and tries |
| 931 | to accelerate any tasks for which it can find shared state packages. |
| 932 | If a shared state package for a task is available, the shared state |
| 933 | package is used. |
| 934 | This means the task and any tasks on which it is dependent are not |
| 935 | executed. |
| 936 | </para> |
| 937 | |
| 938 | <para> |
| 939 | As a real world example, the aim is when building an IPK-based image, |
| 940 | only the |
| 941 | <link linkend='ref-tasks-package_write_ipk'><filename>do_package_write_ipk</filename></link> |
| 942 | tasks would have their |
| 943 | shared state packages fetched and extracted. |
| 944 | Since the sysroot is not used, it would never get extracted. |
| 945 | This is another reason why a task-based approach is preferred over a |
| 946 | recipe-based approach, which would have to install the output from every task. |
| 947 | </para> |
| 948 | </section> |
| 949 | |
| 950 | <section id='tips-and-tricks'> |
| 951 | <title>Tips and Tricks</title> |
| 952 | |
| 953 | <para> |
| 954 | The code in the build system that supports incremental builds is not |
| 955 | simple code. |
| 956 | This section presents some tips and tricks that help you work around |
| 957 | issues related to shared state code. |
| 958 | </para> |
| 959 | |
| 960 | <section id='debugging'> |
| 961 | <title>Debugging</title> |
| 962 | |
| 963 | <para> |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 964 | Seeing what metadata went into creating the input signature |
| 965 | of a shared state (sstate) task can be a useful debugging aid. |
| 966 | This information is available in signature information |
| 967 | (<filename>siginfo</filename>) files in |
| 968 | <link linkend='var-SSTATE_DIR'><filename>SSTATE_DIR</filename></link>. |
| 969 | For information on how to view and interpret information in |
| 970 | <filename>siginfo</filename> files, see the |
| 971 | "<link linkend='usingpoky-viewing-task-variable-dependencies'>Viewing Task Variable Dependencies</link>" |
| 972 | section. |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 973 | </para> |
| 974 | </section> |
| 975 | |
| 976 | <section id='invalidating-shared-state'> |
| 977 | <title>Invalidating Shared State</title> |
| 978 | |
| 979 | <para> |
| 980 | The OpenEmbedded build system uses checksums and shared state |
| 981 | cache to avoid unnecessarily rebuilding tasks. |
| 982 | Collectively, this scheme is known as "shared state code." |
| 983 | </para> |
| 984 | |
| 985 | <para> |
| 986 | As with all schemes, this one has some drawbacks. |
| 987 | It is possible that you could make implicit changes to your |
| 988 | code that the checksum calculations do not take into |
| 989 | account. |
| 990 | These implicit changes affect a task's output but do not trigger |
| 991 | the shared state code into rebuilding a recipe. |
| 992 | Consider an example during which a tool changes its output. |
| 993 | Assume that the output of <filename>rpmdeps</filename> changes. |
| 994 | The result of the change should be that all the |
| 995 | <filename>package</filename> and |
| 996 | <filename>package_write_rpm</filename> shared state cache |
| 997 | items become invalid. |
| 998 | However, because the change to the output is |
| 999 | external to the code and therefore implicit, |
| 1000 | the associated shared state cache items do not become |
| 1001 | invalidated. |
| 1002 | In this case, the build process uses the cached items rather |
| 1003 | than running the task again. |
| 1004 | Obviously, these types of implicit changes can cause problems. |
| 1005 | </para> |
| 1006 | |
| 1007 | <para> |
| 1008 | To avoid these problems during the build, you need to |
| 1009 | understand the effects of any changes you make. |
| 1010 | Realize that changes you make directly to a function |
| 1011 | are automatically factored into the checksum calculation. |
| 1012 | Thus, these explicit changes invalidate the associated area of |
| 1013 | shared state cache. |
| 1014 | However, you need to be aware of any implicit changes that |
| 1015 | are not obvious changes to the code and could affect the output |
| 1016 | of a given task. |
| 1017 | </para> |
| 1018 | |
| 1019 | <para> |
| 1020 | When you identify an implicit change, you can easily take steps |
| 1021 | to invalidate the cache and force the tasks to run. |
| 1022 | The steps you can take are as simple as changing a function's |
| 1023 | comments in the source code. |
| 1024 | For example, to invalidate package shared state files, change |
| 1025 | the comment statements of |
| 1026 | <link linkend='ref-tasks-package'><filename>do_package</filename></link> |
| 1027 | or the comments of one of the functions it calls. |
| 1028 | Even though the change is purely cosmetic, it causes the |
| 1029 | checksum to be recalculated and forces the OpenEmbedded build |
| 1030 | system to run the task again. |
| 1031 | </para> |
| 1032 | |
| 1033 | <note> |
| 1034 | For an example of a commit that makes a cosmetic change to |
| 1035 | invalidate shared state, see this |
| 1036 | <ulink url='&YOCTO_GIT_URL;/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54'>commit</ulink>. |
| 1037 | </note> |
| 1038 | </section> |
| 1039 | </section> |
| 1040 | </section> |
| 1041 | |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 1042 | <section id='automatically-added-runtime-dependencies'> |
| 1043 | <title>Automatically Added Runtime Dependencies</title> |
| 1044 | |
| 1045 | <para> |
| 1046 | The OpenEmbedded build system automatically adds common types of |
| 1047 | runtime dependencies between packages, which means that you do not |
| 1048 | need to explicitly declare the packages using |
| 1049 | <link linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link>. |
| 1050 | Three automatic mechanisms exist (<filename>shlibdeps</filename>, |
| 1051 | <filename>pcdeps</filename>, and <filename>depchains</filename>) that |
| 1052 | handle shared libraries, package configuration (pkg-config) modules, |
| 1053 | and <filename>-dev</filename> and <filename>-dbg</filename> packages, |
| 1054 | respectively. |
| 1055 | For other types of runtime dependencies, you must manually declare |
| 1056 | the dependencies. |
| 1057 | <itemizedlist> |
| 1058 | <listitem><para> |
| 1059 | <filename>shlibdeps</filename>: |
| 1060 | During the |
| 1061 | <link linkend='ref-tasks-package'><filename>do_package</filename></link> |
| 1062 | task of each recipe, all shared libraries installed by the |
| 1063 | recipe are located. |
| 1064 | For each shared library, the package that contains the shared |
| 1065 | library is registered as providing the shared library. |
| 1066 | More specifically, the package is registered as providing the |
| 1067 | <ulink url='https://en.wikipedia.org/wiki/Soname'>soname</ulink> |
| 1068 | of the library. |
| 1069 | The resulting shared-library-to-package mapping |
| 1070 | is saved globally in |
| 1071 | <link linkend='var-PKGDATA_DIR'><filename>PKGDATA_DIR</filename></link> |
| 1072 | by the |
| 1073 | <link linkend='ref-tasks-packagedata'><filename>do_packagedata</filename></link> |
| 1074 | task.</para> |
| 1075 | |
| 1076 | <para>Simultaneously, all executables and shared libraries |
| 1077 | installed by the recipe are inspected to see what shared |
| 1078 | libraries they link against. |
| 1079 | For each shared library dependency that is found, |
| 1080 | <filename>PKGDATA_DIR</filename> is queried to |
| 1081 | see if some package (likely from a different recipe) contains |
| 1082 | the shared library. |
| 1083 | If such a package is found, a runtime dependency is added from |
| 1084 | the package that depends on the shared library to the package |
| 1085 | that contains the library.</para> |
| 1086 | |
| 1087 | <para>The automatically added runtime dependency also includes |
| 1088 | a version restriction. |
| 1089 | This version restriction specifies that at least the current |
| 1090 | version of the package that provides the shared library must be |
| 1091 | used, as if |
| 1092 | "<replaceable>package</replaceable> (>= <replaceable>version</replaceable>)" |
| 1093 | had been added to |
| 1094 | <link linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link>. |
| 1095 | This forces an upgrade of the package containing the shared |
| 1096 | library when installing the package that depends on the |
| 1097 | library, if needed.</para> |
| 1098 | |
| 1099 | <para>If you want to avoid a package being registered as |
| 1100 | providing a particular shared library (e.g. because the library |
| 1101 | is for internal use only), then add the library to |
| 1102 | <link linkend='var-PRIVATE_LIBS'><filename>PRIVATE_LIBS</filename></link> |
| 1103 | inside the package's recipe. |
| 1104 | </para></listitem> |
| 1105 | <listitem><para> |
| 1106 | <filename>pcdeps</filename>: |
| 1107 | During the |
| 1108 | <link linkend='ref-tasks-package'><filename>do_package</filename></link> |
| 1109 | task of each recipe, all pkg-config modules |
| 1110 | (<filename>*.pc</filename> files) installed by the recipe are |
| 1111 | located. |
| 1112 | For each module, the package that contains the module is |
| 1113 | registered as providing the module. |
| 1114 | The resulting module-to-package mapping is saved globally in |
| 1115 | <link linkend='var-PKGDATA_DIR'><filename>PKGDATA_DIR</filename></link> |
| 1116 | by the |
| 1117 | <link linkend='ref-tasks-packagedata'><filename>do_packagedata</filename></link> |
| 1118 | task.</para> |
| 1119 | |
| 1120 | <para>Simultaneously, all pkg-config modules installed by the |
| 1121 | recipe are inspected to see what other pkg-config modules they |
| 1122 | depend on. |
| 1123 | A module is seen as depending on another module if it contains |
| 1124 | a "Requires:" line that specifies the other module. |
| 1125 | For each module dependency, |
| 1126 | <filename>PKGDATA_DIR</filename> is queried to see if some |
| 1127 | package contains the module. |
| 1128 | If such a package is found, a runtime dependency is added from |
| 1129 | the package that depends on the module to the package that |
| 1130 | contains the module. |
| 1131 | <note> |
| 1132 | The <filename>pcdeps</filename> mechanism most often infers |
| 1133 | dependencies between <filename>-dev</filename> packages. |
| 1134 | </note> |
| 1135 | </para></listitem> |
| 1136 | <listitem><para> |
| 1137 | <filename>depchains</filename>: |
| 1138 | If a package <filename>foo</filename> depends on a package |
| 1139 | <filename>bar</filename>, then <filename>foo-dev</filename> |
| 1140 | and <filename>foo-dbg</filename> are also made to depend on |
| 1141 | <filename>bar-dev</filename> and <filename>bar-dbg</filename>, |
| 1142 | respectively. |
| 1143 | Taking the <filename>-dev</filename> packages as an example, |
| 1144 | the <filename>bar-dev</filename> package might provide |
| 1145 | headers and shared library symlinks needed by |
| 1146 | <filename>foo-dev</filename>, which shows the need |
| 1147 | for a dependency between the packages.</para> |
| 1148 | |
| 1149 | <para>The dependencies added by <filename>depchains</filename> |
| 1150 | are in the form of |
| 1151 | <link linkend='var-RRECOMMENDS'><filename>RRECOMMENDS</filename></link>. |
| 1152 | <note> |
| 1153 | By default, <filename>foo-dev</filename> also has an |
| 1154 | <filename>RDEPENDS</filename>-style dependency on |
| 1155 | <filename>foo</filename>, because the default value of |
| 1156 | <filename>RDEPENDS_${PN}-dev</filename> (set in |
| 1157 | <filename>bitbake.conf</filename>) includes |
| 1158 | "${PN}". |
| 1159 | </note></para> |
| 1160 | |
| 1161 | <para>To ensure that the dependency chain is never broken, |
| 1162 | <filename>-dev</filename> and <filename>-dbg</filename> |
| 1163 | packages are always generated by default, even if the packages |
| 1164 | turn out to be empty. |
| 1165 | See the |
| 1166 | <link linkend='var-ALLOW_EMPTY'><filename>ALLOW_EMPTY</filename></link> |
| 1167 | variable for more information. |
| 1168 | </para></listitem> |
| 1169 | </itemizedlist> |
| 1170 | </para> |
| 1171 | |
| 1172 | <para> |
| 1173 | The <filename>do_package</filename> task depends on the |
| 1174 | <link linkend='ref-tasks-packagedata'><filename>do_packagedata</filename></link> |
| 1175 | task of each recipe in |
| 1176 | <link linkend='var-DEPENDS'><filename>DEPENDS</filename></link> |
| 1177 | through use of a |
| 1178 | <filename>[</filename><ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'><filename>deptask</filename></ulink><filename>]</filename> |
| 1179 | declaration, which guarantees that the required |
| 1180 | shared-library/module-to-package mapping information will be available |
| 1181 | when needed as long as <filename>DEPENDS</filename> has been |
| 1182 | correctly set. |
| 1183 | </para> |
| 1184 | </section> |
| 1185 | |
| 1186 | <section id='fakeroot-and-pseudo'> |
| 1187 | <title>Fakeroot and Pseudo</title> |
| 1188 | |
| 1189 | <para> |
| 1190 | Some tasks are easier to implement when allowed to perform certain |
| 1191 | operations that are normally reserved for the root user. |
| 1192 | For example, the |
| 1193 | <link linkend='ref-tasks-install'><filename>do_install</filename></link> |
| 1194 | task benefits from being able to set the UID and GID of installed files |
| 1195 | to arbitrary values. |
| 1196 | </para> |
| 1197 | |
| 1198 | <para> |
| 1199 | One approach to allowing tasks to perform root-only operations |
| 1200 | would be to require BitBake to run as root. |
| 1201 | However, this method is cumbersome and has security issues. |
| 1202 | The approach that is actually used is to run tasks that benefit from |
| 1203 | root privileges in a "fake" root environment. |
| 1204 | Within this environment, the task and its child processes believe that |
| 1205 | they are running as the root user, and see an internally consistent |
| 1206 | view of the filesystem. |
| 1207 | As long as generating the final output (e.g. a package or an image) |
| 1208 | does not require root privileges, the fact that some earlier steps ran |
| 1209 | in a fake root environment does not cause problems. |
| 1210 | </para> |
| 1211 | |
| 1212 | <para> |
| 1213 | The capability to run tasks in a fake root environment is known as |
| 1214 | "fakeroot", which is derived from the BitBake keyword/variable |
| 1215 | flag that requests a fake root environment for a task. |
| 1216 | In current versions of the OpenEmbedded build system, |
| 1217 | the program that implements fakeroot is known as Pseudo. |
| 1218 | </para> |
| 1219 | |
| 1220 | <para> |
| 1221 | Pseudo overrides system calls through the |
| 1222 | <filename>LD_PRELOAD</filename> mechanism to give the |
| 1223 | illusion of running as root. |
| 1224 | To keep track of "fake" file ownership and permissions resulting from |
| 1225 | operations that require root permissions, an sqlite3 |
| 1226 | database is used. |
| 1227 | This database is stored in |
| 1228 | <filename>${</filename><link linkend='var-WORKDIR'><filename>WORKDIR</filename></link><filename>}/pseudo/files.db</filename> |
| 1229 | for individual recipes. |
| 1230 | Storing the database in a file as opposed to in memory |
| 1231 | gives persistence between tasks, and even between builds. |
| 1232 | <note><title>Caution</title> |
| 1233 | If you add your own task that manipulates the same files or |
| 1234 | directories as a fakeroot task, then that task should also run |
| 1235 | under fakeroot. |
| 1236 | Otherwise, the task will not be able to run root-only operations, |
| 1237 | and will not see the fake file ownership and permissions set by the |
| 1238 | other task. |
| 1239 | You should also add a dependency on |
| 1240 | <filename>virtual/fakeroot-native:do_populate_sysroot</filename>, |
| 1241 | giving the following: |
| 1242 | <literallayout class='monospaced'> |
| 1243 | fakeroot do_mytask () { |
| 1244 | ... |
| 1245 | } |
| 1246 | do_mytask[depends] += "virtual/fakeroot-native:do_populate_sysroot" |
| 1247 | </literallayout> |
| 1248 | </note> |
| 1249 | For more information, see the |
| 1250 | <ulink url='&YOCTO_DOCS_BB_URL;#var-FAKEROOT'><filename>FAKEROOT*</filename></ulink> |
| 1251 | variables in the BitBake User Manual. |
| 1252 | You can also reference this |
| 1253 | <ulink url='http://www.ibm.com/developerworks/opensource/library/os-aapseudo1/index.html'>Pseudo</ulink> |
| 1254 | article. |
| 1255 | </para> |
| 1256 | </section> |
| 1257 | |
Brad Bishop | d7bf8c1 | 2018-02-25 22:55:05 -0500 | [diff] [blame] | 1258 | <section id='wic-plug-ins-interface'> |
| 1259 | <title>Wic Plug-Ins Interface</title> |
| 1260 | |
| 1261 | <para> |
| 1262 | You can extend and specialize Wic functionality by using |
| 1263 | Wic plug-ins. |
| 1264 | This section explains the Wic plug-in interface. |
| 1265 | For information on using Wic in general, see the |
| 1266 | "<ulink url='&YOCTO_DOCS_DEV_URL;#creating-partitioned-images-using-wic'>Creating Partitioned Images Using Wic</ulink>" |
| 1267 | section in the Yocto Project Development Tasks Manual. |
| 1268 | <note> |
| 1269 | Wic plug-ins consist of "source" and "imager" plug-ins. |
| 1270 | Imager plug-ins are beyond the scope of this section. |
| 1271 | </note> |
| 1272 | </para> |
| 1273 | |
| 1274 | <para> |
| 1275 | Source plug-ins provide a mechanism to customize partition |
| 1276 | content during the Wic image generation process. |
| 1277 | You can use source plug-ins to map values that you specify |
| 1278 | using <filename>--source</filename> commands in kickstart |
| 1279 | files (i.e. <filename>*.wks</filename>) to a plug-in |
| 1280 | implementation used to populate a given partition. |
| 1281 | <note> |
| 1282 | If you use plug-ins that have build-time dependencies |
| 1283 | (e.g. native tools, bootloaders, and so forth) |
| 1284 | when building a Wic image, you need to specify those |
| 1285 | dependencies using the |
| 1286 | <link linkend='var-WKS_FILE_DEPENDS'><filename>WKS_FILE_DEPENDS</filename></link> |
| 1287 | variable. |
| 1288 | </note> |
| 1289 | </para> |
| 1290 | |
| 1291 | <para> |
| 1292 | Source plug-ins are subclasses defined in plug-in files. |
| 1293 | As shipped, the Yocto Project provides several plug-in |
| 1294 | files. |
| 1295 | You can see the source plug-in files that ship with the |
| 1296 | Yocto Project |
| 1297 | <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/scripts/lib/wic/plugins/source'>here</ulink>. |
| 1298 | Each of these plug-in files contain source plug-ins that |
| 1299 | are designed to populate a specific Wic image partition. |
| 1300 | </para> |
| 1301 | |
| 1302 | <para> |
| 1303 | Source plug-ins are subclasses of the |
| 1304 | <filename>SourcePlugin</filename> class, which is |
| 1305 | defined in the |
| 1306 | <filename>poky/scripts/lib/wic/pluginbase.py</filename> |
| 1307 | file. |
| 1308 | For example, the <filename>BootimgEFIPlugin</filename> |
| 1309 | source plug-in found in the |
| 1310 | <filename>bootimg-efi.py</filename> file is a subclass of |
| 1311 | the <filename>SourcePlugin</filename> class, which is found |
| 1312 | in the <filename>pluginbase.py</filename> file. |
| 1313 | </para> |
| 1314 | |
| 1315 | <para> |
| 1316 | You can also implement source plug-ins in a layer outside |
| 1317 | of the Source Repositories (external layer). |
| 1318 | To do so, be sure that your plug-in files are located in |
| 1319 | a directory whose path is |
| 1320 | <filename>scripts/lib/wic/plugins/source/</filename> |
| 1321 | within your external layer. |
| 1322 | When the plug-in files are located there, the source |
| 1323 | plug-ins they contain are made available to Wic. |
| 1324 | </para> |
| 1325 | |
| 1326 | <para> |
| 1327 | When the Wic implementation needs to invoke a |
| 1328 | partition-specific implementation, it looks for the plug-in |
| 1329 | with the same name as the <filename>--source</filename> |
| 1330 | parameter used in the kickstart file given to that |
| 1331 | partition. |
| 1332 | For example, if the partition is set up using the following |
| 1333 | command in a kickstart file: |
| 1334 | <literallayout class='monospaced'> |
| 1335 | part /boot --source bootimg-pcbios --ondisk sda --label boot --active --align 1024 |
| 1336 | </literallayout> |
| 1337 | The methods defined as class members of the matching |
| 1338 | source plug-in (i.e. <filename>bootimg-pcbios</filename>) |
| 1339 | in the <filename>bootimg-pcbios.py</filename> plug-in file |
| 1340 | are used. |
| 1341 | </para> |
| 1342 | |
| 1343 | <para> |
| 1344 | To be more concrete, here is the corresponding plug-in |
| 1345 | definition from the <filename>bootimg-pcbios.py</filename> |
| 1346 | file for the previous command along with an example |
| 1347 | method called by the Wic implementation when it needs to |
| 1348 | prepare a partition using an implementation-specific |
| 1349 | function: |
| 1350 | <literallayout class='monospaced'> |
| 1351 | bootimg-pcbios.py |
| 1352 | . |
| 1353 | . |
| 1354 | . |
| 1355 | class BootimgPcbiosPlugin(SourcePlugin): |
| 1356 | """ |
| 1357 | Create MBR boot partition and install syslinux on it. |
| 1358 | """ |
| 1359 | |
| 1360 | name = 'bootimg-pcbios' |
| 1361 | . |
| 1362 | . |
| 1363 | . |
| 1364 | @classmethod |
| 1365 | def do_prepare_partition(cls, part, source_params, creator, cr_workdir, |
| 1366 | oe_builddir, bootimg_dir, kernel_dir, |
| 1367 | rootfs_dir, native_sysroot): |
| 1368 | """ |
| 1369 | Called to do the actual content population for a partition i.e. it |
| 1370 | 'prepares' the partition to be incorporated into the image. |
| 1371 | In this case, prepare content for legacy bios boot partition. |
| 1372 | """ |
| 1373 | . |
| 1374 | . |
| 1375 | . |
| 1376 | </literallayout> |
| 1377 | If a subclass (plug-in) itself does not implement a |
| 1378 | particular function, Wic locates and uses the default |
| 1379 | version in the superclass. |
| 1380 | It is for this reason that all source plug-ins are derived |
| 1381 | from the <filename>SourcePlugin</filename> class. |
| 1382 | </para> |
| 1383 | |
| 1384 | <para> |
| 1385 | The <filename>SourcePlugin</filename> class defined in |
| 1386 | the <filename>pluginbase.py</filename> file defines |
| 1387 | a set of methods that source plug-ins can implement or |
| 1388 | override. |
| 1389 | Any plug-ins (subclass of |
| 1390 | <filename>SourcePlugin</filename>) that do not implement |
| 1391 | a particular method inherit the implementation of the |
| 1392 | method from the <filename>SourcePlugin</filename> class. |
| 1393 | For more information, see the |
| 1394 | <filename>SourcePlugin</filename> class in the |
| 1395 | <filename>pluginbase.py</filename> file for details: |
| 1396 | </para> |
| 1397 | |
| 1398 | <para> |
| 1399 | The following list describes the methods implemented in the |
| 1400 | <filename>SourcePlugin</filename> class: |
| 1401 | <itemizedlist> |
| 1402 | <listitem><para> |
| 1403 | <emphasis><filename>do_prepare_partition()</filename>:</emphasis> |
| 1404 | Called to populate a partition with actual content. |
| 1405 | In other words, the method prepares the final |
| 1406 | partition image that is incorporated into the |
| 1407 | disk image. |
| 1408 | </para></listitem> |
| 1409 | <listitem><para> |
| 1410 | <emphasis><filename>do_configure_partition()</filename>:</emphasis> |
| 1411 | Called before |
| 1412 | <filename>do_prepare_partition()</filename> to |
| 1413 | create custom configuration files for a partition |
| 1414 | (e.g. syslinux or grub configuration files). |
| 1415 | </para></listitem> |
| 1416 | <listitem><para> |
| 1417 | <emphasis><filename>do_install_disk()</filename>:</emphasis> |
| 1418 | Called after all partitions have been prepared and |
| 1419 | assembled into a disk image. |
| 1420 | This method provides a hook to allow finalization |
| 1421 | of a disk image (e.g. writing an MBR). |
| 1422 | </para></listitem> |
| 1423 | <listitem><para> |
| 1424 | <emphasis><filename>do_stage_partition()</filename>:</emphasis> |
| 1425 | Special content-staging hook called before |
| 1426 | <filename>do_prepare_partition()</filename>. |
| 1427 | This method is normally empty.</para> |
| 1428 | |
| 1429 | <para>Typically, a partition just uses the passed-in |
| 1430 | parameters (e.g. the unmodified value of |
| 1431 | <filename>bootimg_dir</filename>). |
| 1432 | However, in some cases, things might need to be |
| 1433 | more tailored. |
| 1434 | As an example, certain files might additionally |
| 1435 | need to be taken from |
| 1436 | <filename>bootimg_dir + /boot</filename>. |
| 1437 | This hook allows those files to be staged in a |
| 1438 | customized fashion. |
| 1439 | <note> |
| 1440 | <filename>get_bitbake_var()</filename> |
| 1441 | allows you to access non-standard variables |
| 1442 | that you might want to use for this |
| 1443 | behavior. |
| 1444 | </note> |
| 1445 | </para></listitem> |
| 1446 | </itemizedlist> |
| 1447 | </para> |
| 1448 | |
| 1449 | <para> |
| 1450 | You can extend the source plug-in mechanism. |
| 1451 | To add more hooks, create more source plug-in methods |
| 1452 | within <filename>SourcePlugin</filename> and the |
| 1453 | corresponding derived subclasses. |
| 1454 | The code that calls the plug-in methods uses the |
| 1455 | <filename>plugin.get_source_plugin_methods()</filename> |
| 1456 | function to find the method or methods needed by the call. |
| 1457 | Retrieval of those methods is accomplished by filling up |
| 1458 | a dict with keys that contain the method names of interest. |
| 1459 | On success, these will be filled in with the actual |
| 1460 | methods. |
| 1461 | See the Wic implementation for examples and details. |
| 1462 | </para> |
| 1463 | </section> |
| 1464 | |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 1465 | <section id='x32'> |
| 1466 | <title>x32</title> |
| 1467 | |
| 1468 | <para> |
| 1469 | x32 is a processor-specific Application Binary Interface (psABI) for x86_64. |
| 1470 | An ABI defines the calling conventions between functions in a processing environment. |
| 1471 | The interface determines what registers are used and what the sizes are for various C data types. |
| 1472 | </para> |
| 1473 | |
| 1474 | <para> |
| 1475 | Some processing environments prefer using 32-bit applications even when running |
| 1476 | on Intel 64-bit platforms. |
| 1477 | Consider the i386 psABI, which is a very old 32-bit ABI for Intel 64-bit platforms. |
| 1478 | The i386 psABI does not provide efficient use and access of the Intel 64-bit processor resources, |
| 1479 | leaving the system underutilized. |
| 1480 | Now consider the x86_64 psABI. |
| 1481 | This ABI is newer and uses 64-bits for data sizes and program pointers. |
| 1482 | The extra bits increase the footprint size of the programs, libraries, |
| 1483 | and also increases the memory and file system size requirements. |
| 1484 | Executing under the x32 psABI enables user programs to utilize CPU and system resources |
| 1485 | more efficiently while keeping the memory footprint of the applications low. |
| 1486 | Extra bits are used for registers but not for addressing mechanisms. |
| 1487 | </para> |
| 1488 | |
| 1489 | <section id='support'> |
| 1490 | <title>Support</title> |
| 1491 | |
| 1492 | <para> |
| 1493 | This Yocto Project release supports the final specifications of x32 |
| 1494 | psABI. |
| 1495 | Support for x32 psABI exists as follows: |
| 1496 | <itemizedlist> |
| 1497 | <listitem><para>You can create packages and images in x32 psABI format on x86_64 architecture targets. |
| 1498 | </para></listitem> |
| 1499 | <listitem><para>You can successfully build many recipes with the x32 toolchain.</para></listitem> |
| 1500 | <listitem><para>You can create and boot <filename>core-image-minimal</filename> and |
| 1501 | <filename>core-image-sato</filename> images.</para></listitem> |
| 1502 | </itemizedlist> |
| 1503 | </para> |
| 1504 | </section> |
| 1505 | |
| 1506 | <section id='completing-x32'> |
| 1507 | <title>Completing x32</title> |
| 1508 | |
| 1509 | <para> |
| 1510 | Future Plans for the x32 psABI in the Yocto Project include the following: |
| 1511 | <itemizedlist> |
| 1512 | <listitem><para>Enhance and fix the few remaining recipes so they |
| 1513 | work with and support x32 toolchains.</para></listitem> |
| 1514 | <listitem><para>Enhance RPM Package Manager (RPM) support for x32 binaries.</para></listitem> |
| 1515 | <listitem><para>Support larger images.</para></listitem> |
| 1516 | </itemizedlist> |
| 1517 | </para> |
| 1518 | </section> |
| 1519 | |
| 1520 | <section id='using-x32-right-now'> |
| 1521 | <title>Using x32 Right Now</title> |
| 1522 | |
| 1523 | <para> |
| 1524 | Follow these steps to use the x32 spABI: |
| 1525 | <itemizedlist> |
| 1526 | <listitem><para>Enable the x32 psABI tuning file for <filename>x86_64</filename> |
| 1527 | machines by editing the <filename>conf/local.conf</filename> like this: |
| 1528 | <literallayout class='monospaced'> |
| 1529 | MACHINE = "qemux86-64" |
| 1530 | DEFAULTTUNE = "x86-64-x32" |
| 1531 | baselib = "${@d.getVar('BASE_LIB_tune-' + (d.getVar('DEFAULTTUNE', True) \ |
| 1532 | or 'INVALID'), True) or 'lib'}" |
| 1533 | #MACHINE = "genericx86" |
| 1534 | #DEFAULTTUNE = "core2-64-x32" |
| 1535 | </literallayout></para></listitem> |
| 1536 | <listitem><para>As usual, use BitBake to build an image that supports the x32 psABI. |
| 1537 | Here is an example: |
| 1538 | <literallayout class='monospaced'> |
| 1539 | $ bitbake core-image-sato |
| 1540 | </literallayout></para></listitem> |
| 1541 | <listitem><para>As usual, run your image using QEMU: |
| 1542 | <literallayout class='monospaced'> |
| 1543 | $ runqemu qemux86-64 core-image-sato |
| 1544 | </literallayout></para></listitem> |
| 1545 | </itemizedlist> |
| 1546 | </para> |
| 1547 | </section> |
| 1548 | </section> |
| 1549 | |
| 1550 | <section id="wayland"> |
| 1551 | <title>Wayland</title> |
| 1552 | |
| 1553 | <para> |
| 1554 | <ulink url='http://en.wikipedia.org/wiki/Wayland_(display_server_protocol)'>Wayland</ulink> |
| 1555 | is a computer display server protocol that |
| 1556 | provides a method for compositing window managers to communicate |
| 1557 | directly with applications and video hardware and expects them to |
| 1558 | communicate with input hardware using other libraries. |
| 1559 | Using Wayland with supporting targets can result in better control |
| 1560 | over graphics frame rendering than an application might otherwise |
| 1561 | achieve. |
| 1562 | </para> |
| 1563 | |
| 1564 | <para> |
| 1565 | The Yocto Project provides the Wayland protocol libraries and the |
| 1566 | reference |
| 1567 | <ulink url='http://en.wikipedia.org/wiki/Wayland_(display_server_protocol)#Weston'>Weston</ulink> |
| 1568 | compositor as part of its release. |
| 1569 | This section describes what you need to do to implement Wayland and |
| 1570 | use the compositor when building an image for a supporting target. |
| 1571 | </para> |
| 1572 | |
| 1573 | <section id="wayland-support"> |
| 1574 | <title>Support</title> |
| 1575 | |
| 1576 | <para> |
| 1577 | The Wayland protocol libraries and the reference Weston compositor |
| 1578 | ship as integrated packages in the <filename>meta</filename> layer |
| 1579 | of the |
Brad Bishop | d7bf8c1 | 2018-02-25 22:55:05 -0500 | [diff] [blame] | 1580 | <link linkend='source-directory'>Source Directory</link>. |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 1581 | Specifically, you can find the recipes that build both Wayland |
| 1582 | and Weston at <filename>meta/recipes-graphics/wayland</filename>. |
| 1583 | </para> |
| 1584 | |
| 1585 | <para> |
| 1586 | You can build both the Wayland and Weston packages for use only |
| 1587 | with targets that accept the |
| 1588 | <ulink url='http://dri.freedesktop.org/wiki/'>Mesa 3D and Direct Rendering Infrastructure</ulink>, |
| 1589 | which is also known as Mesa DRI. |
| 1590 | This implies that you cannot build and use the packages if your |
| 1591 | target uses, for example, the |
| 1592 | <trademark class='registered'>Intel</trademark> Embedded Media and |
| 1593 | Graphics Driver (<trademark class='registered'>Intel</trademark> |
| 1594 | EMGD) that overrides Mesa DRI. |
| 1595 | </para> |
| 1596 | |
| 1597 | <note> |
| 1598 | Due to lack of EGL support, Weston 1.0.3 will not run directly on |
| 1599 | the emulated QEMU hardware. |
| 1600 | However, this version of Weston will run under X emulation without |
| 1601 | issues. |
| 1602 | </note> |
| 1603 | </section> |
| 1604 | |
| 1605 | <section id="enabling-wayland-in-an-image"> |
| 1606 | <title>Enabling Wayland in an Image</title> |
| 1607 | |
| 1608 | <para> |
| 1609 | To enable Wayland, you need to enable it to be built and enable |
| 1610 | it to be included in the image. |
| 1611 | </para> |
| 1612 | |
| 1613 | <section id="enable-building"> |
| 1614 | <title>Building</title> |
| 1615 | |
| 1616 | <para> |
| 1617 | To cause Mesa to build the <filename>wayland-egl</filename> |
| 1618 | platform and Weston to build Wayland with Kernel Mode |
| 1619 | Setting |
| 1620 | (<ulink url='https://wiki.archlinux.org/index.php/Kernel_Mode_Setting'>KMS</ulink>) |
| 1621 | support, include the "wayland" flag in the |
| 1622 | <link linkend="var-DISTRO_FEATURES"><filename>DISTRO_FEATURES</filename></link> |
| 1623 | statement in your <filename>local.conf</filename> file: |
| 1624 | <literallayout class='monospaced'> |
| 1625 | DISTRO_FEATURES_append = " wayland" |
| 1626 | </literallayout> |
| 1627 | </para> |
| 1628 | |
| 1629 | <note> |
| 1630 | If X11 has been enabled elsewhere, Weston will build Wayland |
| 1631 | with X11 support |
| 1632 | </note> |
| 1633 | </section> |
| 1634 | |
| 1635 | <section id="enable-installation-in-an-image"> |
| 1636 | <title>Installing</title> |
| 1637 | |
| 1638 | <para> |
| 1639 | To install the Wayland feature into an image, you must |
| 1640 | include the following |
| 1641 | <link linkend='var-CORE_IMAGE_EXTRA_INSTALL'><filename>CORE_IMAGE_EXTRA_INSTALL</filename></link> |
| 1642 | statement in your <filename>local.conf</filename> file: |
| 1643 | <literallayout class='monospaced'> |
| 1644 | CORE_IMAGE_EXTRA_INSTALL += "wayland weston" |
| 1645 | </literallayout> |
| 1646 | </para> |
| 1647 | </section> |
| 1648 | </section> |
| 1649 | |
| 1650 | <section id="running-weston"> |
| 1651 | <title>Running Weston</title> |
| 1652 | |
| 1653 | <para> |
| 1654 | To run Weston inside X11, enabling it as described earlier and |
| 1655 | building a Sato image is sufficient. |
| 1656 | If you are running your image under Sato, a Weston Launcher appears |
| 1657 | in the "Utility" category. |
| 1658 | </para> |
| 1659 | |
| 1660 | <para> |
| 1661 | Alternatively, you can run Weston through the command-line |
| 1662 | interpretor (CLI), which is better suited for development work. |
| 1663 | To run Weston under the CLI, you need to do the following after |
| 1664 | your image is built: |
| 1665 | <orderedlist> |
| 1666 | <listitem><para>Run these commands to export |
| 1667 | <filename>XDG_RUNTIME_DIR</filename>: |
| 1668 | <literallayout class='monospaced'> |
| 1669 | mkdir -p /tmp/$USER-weston |
| 1670 | chmod 0700 /tmp/$USER-weston |
| 1671 | export XDG_RUNTIME_DIR=/tmp/$USER-weston |
| 1672 | </literallayout></para></listitem> |
| 1673 | <listitem><para>Launch Weston in the shell: |
| 1674 | <literallayout class='monospaced'> |
| 1675 | weston |
| 1676 | </literallayout></para></listitem> |
| 1677 | </orderedlist> |
| 1678 | </para> |
| 1679 | </section> |
| 1680 | </section> |
| 1681 | |
| 1682 | <section id="licenses"> |
| 1683 | <title>Licenses</title> |
| 1684 | |
| 1685 | <para> |
| 1686 | This section describes the mechanism by which the OpenEmbedded build system |
| 1687 | tracks changes to licensing text. |
| 1688 | The section also describes how to enable commercially licensed recipes, |
| 1689 | which by default are disabled. |
| 1690 | </para> |
| 1691 | |
| 1692 | <para> |
| 1693 | For information that can help you maintain compliance with various open |
| 1694 | source licensing during the lifecycle of the product, see the |
Brad Bishop | d7bf8c1 | 2018-02-25 22:55:05 -0500 | [diff] [blame] | 1695 | "<ulink url='&YOCTO_DOCS_DEV_URL;#maintaining-open-source-license-compliance-during-your-products-lifecycle'>Maintaining Open Source License Compliance During Your Project's Lifecycle</ulink>" |
| 1696 | section in the Yocto Project Development Tasks Manual. |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 1697 | </para> |
| 1698 | |
| 1699 | <section id="usingpoky-configuring-LIC_FILES_CHKSUM"> |
| 1700 | <title>Tracking License Changes</title> |
| 1701 | |
| 1702 | <para> |
| 1703 | The license of an upstream project might change in the future. |
| 1704 | In order to prevent these changes going unnoticed, the |
| 1705 | <filename><link linkend='var-LIC_FILES_CHKSUM'>LIC_FILES_CHKSUM</link></filename> |
| 1706 | variable tracks changes to the license text. The checksums are validated at the end of the |
| 1707 | configure step, and if the checksums do not match, the build will fail. |
| 1708 | </para> |
| 1709 | |
| 1710 | <section id="usingpoky-specifying-LIC_FILES_CHKSUM"> |
| 1711 | <title>Specifying the <filename>LIC_FILES_CHKSUM</filename> Variable</title> |
| 1712 | |
| 1713 | <para> |
| 1714 | The <filename>LIC_FILES_CHKSUM</filename> |
Brad Bishop | 6e60e8b | 2018-02-01 10:27:11 -0500 | [diff] [blame] | 1715 | variable contains checksums of the license text in the source |
| 1716 | code for the recipe. |
| 1717 | Following is an example of how to specify |
| 1718 | <filename>LIC_FILES_CHKSUM</filename>: |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 1719 | <literallayout class='monospaced'> |
| 1720 | LIC_FILES_CHKSUM = "file://COPYING;md5=xxxx \ |
| 1721 | file://licfile1.txt;beginline=5;endline=29;md5=yyyy \ |
| 1722 | file://licfile2.txt;endline=50;md5=zzzz \ |
| 1723 | ..." |
| 1724 | </literallayout> |
Brad Bishop | 6e60e8b | 2018-02-01 10:27:11 -0500 | [diff] [blame] | 1725 | <note><title>Notes</title> |
| 1726 | <itemizedlist> |
| 1727 | <listitem><para> |
| 1728 | When using "beginline" and "endline", realize that |
| 1729 | line numbering begins with one and not zero. |
| 1730 | Also, the included lines are inclusive (i.e. lines |
| 1731 | five through and including 29 in the previous |
| 1732 | example for <filename>licfile1.txt</filename>). |
| 1733 | </para></listitem> |
| 1734 | <listitem><para> |
| 1735 | When a license check fails, the selected license |
| 1736 | text is included as part of the QA message. |
| 1737 | Using this output, you can determine the exact |
| 1738 | start and finish for the needed license text. |
| 1739 | </para></listitem> |
| 1740 | </itemizedlist> |
| 1741 | </note> |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 1742 | </para> |
| 1743 | |
| 1744 | <para> |
| 1745 | The build system uses the |
| 1746 | <filename><link linkend='var-S'>S</link></filename> variable as |
| 1747 | the default directory when searching files listed in |
| 1748 | <filename>LIC_FILES_CHKSUM</filename>. |
| 1749 | The previous example employs the default directory. |
| 1750 | </para> |
| 1751 | |
| 1752 | <para> |
| 1753 | Consider this next example: |
| 1754 | <literallayout class='monospaced'> |
| 1755 | LIC_FILES_CHKSUM = "file://src/ls.c;beginline=5;endline=16;\ |
| 1756 | md5=bb14ed3c4cda583abc85401304b5cd4e" |
| 1757 | LIC_FILES_CHKSUM = "file://${WORKDIR}/license.html;md5=5c94767cedb5d6987c902ac850ded2c6" |
| 1758 | </literallayout> |
| 1759 | </para> |
| 1760 | |
| 1761 | <para> |
| 1762 | The first line locates a file in |
Brad Bishop | 6e60e8b | 2018-02-01 10:27:11 -0500 | [diff] [blame] | 1763 | <filename>${S}/src/ls.c</filename> and isolates lines five |
| 1764 | through 16 as license text. |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 1765 | The second line refers to a file in |
| 1766 | <filename><link linkend='var-WORKDIR'>WORKDIR</link></filename>. |
| 1767 | </para> |
| 1768 | <para> |
| 1769 | Note that <filename>LIC_FILES_CHKSUM</filename> variable is |
| 1770 | mandatory for all recipes, unless the |
| 1771 | <filename>LICENSE</filename> variable is set to "CLOSED". |
| 1772 | </para> |
| 1773 | </section> |
| 1774 | |
| 1775 | <section id="usingpoky-LIC_FILES_CHKSUM-explanation-of-syntax"> |
| 1776 | <title>Explanation of Syntax</title> |
| 1777 | <para> |
| 1778 | As mentioned in the previous section, the |
| 1779 | <filename>LIC_FILES_CHKSUM</filename> variable lists all the |
| 1780 | important files that contain the license text for the source code. |
| 1781 | It is possible to specify a checksum for an entire file, or a specific section of a |
| 1782 | file (specified by beginning and ending line numbers with the "beginline" and "endline" |
| 1783 | parameters, respectively). |
| 1784 | The latter is useful for source files with a license notice header, |
| 1785 | README documents, and so forth. |
| 1786 | If you do not use the "beginline" parameter, then it is assumed that the text begins on the |
| 1787 | first line of the file. |
| 1788 | Similarly, if you do not use the "endline" parameter, it is assumed that the license text |
| 1789 | ends with the last line of the file. |
| 1790 | </para> |
| 1791 | |
| 1792 | <para> |
| 1793 | The "md5" parameter stores the md5 checksum of the license text. |
| 1794 | If the license text changes in any way as compared to this parameter |
| 1795 | then a mismatch occurs. |
| 1796 | This mismatch triggers a build failure and notifies the developer. |
| 1797 | Notification allows the developer to review and address the license text changes. |
| 1798 | Also note that if a mismatch occurs during the build, the correct md5 |
| 1799 | checksum is placed in the build log and can be easily copied to the recipe. |
| 1800 | </para> |
| 1801 | |
| 1802 | <para> |
| 1803 | There is no limit to how many files you can specify using the |
| 1804 | <filename>LIC_FILES_CHKSUM</filename> variable. |
| 1805 | Generally, however, every project requires a few specifications for license tracking. |
| 1806 | Many projects have a "COPYING" file that stores the license information for all the source |
| 1807 | code files. |
| 1808 | This practice allows you to just track the "COPYING" file as long as it is kept up to date. |
| 1809 | </para> |
| 1810 | |
| 1811 | <tip> |
| 1812 | If you specify an empty or invalid "md5" parameter, BitBake returns an md5 mis-match |
| 1813 | error and displays the correct "md5" parameter value during the build. |
| 1814 | The correct parameter is also captured in the build log. |
| 1815 | </tip> |
| 1816 | |
| 1817 | <tip> |
| 1818 | If the whole file contains only license text, you do not need to use the "beginline" and |
| 1819 | "endline" parameters. |
| 1820 | </tip> |
| 1821 | </section> |
| 1822 | </section> |
| 1823 | |
| 1824 | <section id="enabling-commercially-licensed-recipes"> |
| 1825 | <title>Enabling Commercially Licensed Recipes</title> |
| 1826 | |
| 1827 | <para> |
| 1828 | By default, the OpenEmbedded build system disables |
| 1829 | components that have commercial or other special licensing |
| 1830 | requirements. |
| 1831 | Such requirements are defined on a |
| 1832 | recipe-by-recipe basis through the |
| 1833 | <link linkend='var-LICENSE_FLAGS'><filename>LICENSE_FLAGS</filename></link> |
| 1834 | variable definition in the affected recipe. |
| 1835 | For instance, the |
| 1836 | <filename>poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly</filename> |
| 1837 | recipe contains the following statement: |
| 1838 | <literallayout class='monospaced'> |
| 1839 | LICENSE_FLAGS = "commercial" |
| 1840 | </literallayout> |
| 1841 | Here is a slightly more complicated example that contains both an |
| 1842 | explicit recipe name and version (after variable expansion): |
| 1843 | <literallayout class='monospaced'> |
| 1844 | LICENSE_FLAGS = "license_${PN}_${PV}" |
| 1845 | </literallayout> |
| 1846 | In order for a component restricted by a <filename>LICENSE_FLAGS</filename> |
| 1847 | definition to be enabled and included in an image, it |
| 1848 | needs to have a matching entry in the global |
| 1849 | <link linkend='var-LICENSE_FLAGS_WHITELIST'><filename>LICENSE_FLAGS_WHITELIST</filename></link> |
| 1850 | variable, which is a variable |
| 1851 | typically defined in your <filename>local.conf</filename> file. |
| 1852 | For example, to enable |
| 1853 | the <filename>poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly</filename> |
| 1854 | package, you could add either the string |
| 1855 | "commercial_gst-plugins-ugly" or the more general string |
| 1856 | "commercial" to <filename>LICENSE_FLAGS_WHITELIST</filename>. |
| 1857 | See the |
| 1858 | "<link linkend='license-flag-matching'>License Flag Matching</link>" section |
| 1859 | for a full explanation of how <filename>LICENSE_FLAGS</filename> matching works. |
| 1860 | Here is the example: |
| 1861 | <literallayout class='monospaced'> |
| 1862 | LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly" |
| 1863 | </literallayout> |
| 1864 | Likewise, to additionally enable the package built from the recipe containing |
| 1865 | <filename>LICENSE_FLAGS = "license_${PN}_${PV}"</filename>, and assuming |
| 1866 | that the actual recipe name was <filename>emgd_1.10.bb</filename>, |
| 1867 | the following string would enable that package as well as |
| 1868 | the original <filename>gst-plugins-ugly</filename> package: |
| 1869 | <literallayout class='monospaced'> |
| 1870 | LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly license_emgd_1.10" |
| 1871 | </literallayout> |
| 1872 | As a convenience, you do not need to specify the complete license string |
| 1873 | in the whitelist for every package. |
| 1874 | You can use an abbreviated form, which consists |
| 1875 | of just the first portion or portions of the license string before |
| 1876 | the initial underscore character or characters. |
| 1877 | A partial string will match |
| 1878 | any license that contains the given string as the first |
| 1879 | portion of its license. |
| 1880 | For example, the following |
| 1881 | whitelist string will also match both of the packages |
| 1882 | previously mentioned as well as any other packages that have |
| 1883 | licenses starting with "commercial" or "license". |
| 1884 | <literallayout class='monospaced'> |
| 1885 | LICENSE_FLAGS_WHITELIST = "commercial license" |
| 1886 | </literallayout> |
| 1887 | </para> |
| 1888 | |
| 1889 | <section id="license-flag-matching"> |
| 1890 | <title>License Flag Matching</title> |
| 1891 | |
| 1892 | <para> |
| 1893 | License flag matching allows you to control what recipes the |
| 1894 | OpenEmbedded build system includes in the build. |
| 1895 | Fundamentally, the build system attempts to match |
| 1896 | <link linkend='var-LICENSE_FLAGS'><filename>LICENSE_FLAGS</filename></link> |
| 1897 | strings found in recipes against |
| 1898 | <link linkend='var-LICENSE_FLAGS_WHITELIST'><filename>LICENSE_FLAGS_WHITELIST</filename></link> |
| 1899 | strings found in the whitelist. |
| 1900 | A match causes the build system to include a recipe in the |
| 1901 | build, while failure to find a match causes the build system to |
| 1902 | exclude a recipe. |
| 1903 | </para> |
| 1904 | |
| 1905 | <para> |
| 1906 | In general, license flag matching is simple. |
| 1907 | However, understanding some concepts will help you |
| 1908 | correctly and effectively use matching. |
| 1909 | </para> |
| 1910 | |
| 1911 | <para> |
| 1912 | Before a flag |
| 1913 | defined by a particular recipe is tested against the |
| 1914 | contents of the whitelist, the expanded string |
| 1915 | <filename>_${PN}</filename> is appended to the flag. |
| 1916 | This expansion makes each <filename>LICENSE_FLAGS</filename> |
| 1917 | value recipe-specific. |
| 1918 | After expansion, the string is then matched against the |
| 1919 | whitelist. |
| 1920 | Thus, specifying |
| 1921 | <filename>LICENSE_FLAGS = "commercial"</filename> |
| 1922 | in recipe "foo", for example, results in the string |
| 1923 | <filename>"commercial_foo"</filename>. |
| 1924 | And, to create a match, that string must appear in the |
| 1925 | whitelist. |
| 1926 | </para> |
| 1927 | |
| 1928 | <para> |
| 1929 | Judicious use of the <filename>LICENSE_FLAGS</filename> |
| 1930 | strings and the contents of the |
| 1931 | <filename>LICENSE_FLAGS_WHITELIST</filename> variable |
| 1932 | allows you a lot of flexibility for including or excluding |
| 1933 | recipes based on licensing. |
| 1934 | For example, you can broaden the matching capabilities by |
| 1935 | using license flags string subsets in the whitelist. |
| 1936 | <note>When using a string subset, be sure to use the part of |
| 1937 | the expanded string that precedes the appended underscore |
| 1938 | character (e.g. <filename>usethispart_1.3</filename>, |
| 1939 | <filename>usethispart_1.4</filename>, and so forth). |
| 1940 | </note> |
| 1941 | For example, simply specifying the string "commercial" in |
| 1942 | the whitelist matches any expanded |
| 1943 | <filename>LICENSE_FLAGS</filename> definition that starts with |
| 1944 | the string "commercial" such as "commercial_foo" and |
| 1945 | "commercial_bar", which are the strings the build system |
| 1946 | automatically generates for hypothetical recipes named |
| 1947 | "foo" and "bar" assuming those recipes simply specify the |
| 1948 | following: |
| 1949 | <literallayout class='monospaced'> |
| 1950 | LICENSE_FLAGS = "commercial" |
| 1951 | </literallayout> |
| 1952 | Thus, you can choose to exhaustively |
| 1953 | enumerate each license flag in the whitelist and |
| 1954 | allow only specific recipes into the image, or |
| 1955 | you can use a string subset that causes a broader range of |
| 1956 | matches to allow a range of recipes into the image. |
| 1957 | </para> |
| 1958 | |
| 1959 | <para> |
| 1960 | This scheme works even if the |
| 1961 | <filename>LICENSE_FLAGS</filename> string already |
| 1962 | has <filename>_${PN}</filename> appended. |
| 1963 | For example, the build system turns the license flag |
| 1964 | "commercial_1.2_foo" into "commercial_1.2_foo_foo" and would |
| 1965 | match both the general "commercial" and the specific |
| 1966 | "commercial_1.2_foo" strings found in the whitelist, as |
| 1967 | expected. |
| 1968 | </para> |
| 1969 | |
| 1970 | <para> |
| 1971 | Here are some other scenarios: |
| 1972 | <itemizedlist> |
| 1973 | <listitem><para>You can specify a versioned string in the |
| 1974 | recipe such as "commercial_foo_1.2" in a "foo" recipe. |
| 1975 | The build system expands this string to |
| 1976 | "commercial_foo_1.2_foo". |
| 1977 | Combine this license flag with a whitelist that has |
| 1978 | the string "commercial" and you match the flag along |
| 1979 | with any other flag that starts with the string |
| 1980 | "commercial".</para></listitem> |
| 1981 | <listitem><para>Under the same circumstances, you can |
| 1982 | use "commercial_foo" in the whitelist and the |
| 1983 | build system not only matches "commercial_foo_1.2" but |
| 1984 | also matches any license flag with the string |
| 1985 | "commercial_foo", regardless of the version. |
| 1986 | </para></listitem> |
| 1987 | <listitem><para>You can be very specific and use both the |
| 1988 | package and version parts in the whitelist (e.g. |
| 1989 | "commercial_foo_1.2") to specifically match a |
| 1990 | versioned recipe.</para></listitem> |
| 1991 | </itemizedlist> |
| 1992 | </para> |
| 1993 | </section> |
| 1994 | |
| 1995 | <section id="other-variables-related-to-commercial-licenses"> |
| 1996 | <title>Other Variables Related to Commercial Licenses</title> |
| 1997 | |
| 1998 | <para> |
| 1999 | Other helpful variables related to commercial |
| 2000 | license handling exist and are defined in the |
| 2001 | <filename>poky/meta/conf/distro/include/default-distrovars.inc</filename> file: |
| 2002 | <literallayout class='monospaced'> |
| 2003 | COMMERCIAL_AUDIO_PLUGINS ?= "" |
| 2004 | COMMERCIAL_VIDEO_PLUGINS ?= "" |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 2005 | </literallayout> |
| 2006 | If you want to enable these components, you can do so by making sure you have |
| 2007 | statements similar to the following |
| 2008 | in your <filename>local.conf</filename> configuration file: |
| 2009 | <literallayout class='monospaced'> |
| 2010 | COMMERCIAL_AUDIO_PLUGINS = "gst-plugins-ugly-mad \ |
| 2011 | gst-plugins-ugly-mpegaudioparse" |
| 2012 | COMMERCIAL_VIDEO_PLUGINS = "gst-plugins-ugly-mpeg2dec \ |
| 2013 | gst-plugins-ugly-mpegstream gst-plugins-bad-mpegvideoparse" |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 2014 | LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly commercial_gst-plugins-bad commercial_qmmp" |
| 2015 | </literallayout> |
| 2016 | Of course, you could also create a matching whitelist |
| 2017 | for those components using the more general "commercial" |
| 2018 | in the whitelist, but that would also enable all the |
| 2019 | other packages with |
| 2020 | <link linkend='var-LICENSE_FLAGS'><filename>LICENSE_FLAGS</filename></link> |
| 2021 | containing "commercial", which you may or may not want: |
| 2022 | <literallayout class='monospaced'> |
| 2023 | LICENSE_FLAGS_WHITELIST = "commercial" |
| 2024 | </literallayout> |
| 2025 | </para> |
| 2026 | |
| 2027 | <para> |
| 2028 | Specifying audio and video plug-ins as part of the |
| 2029 | <filename>COMMERCIAL_AUDIO_PLUGINS</filename> and |
| 2030 | <filename>COMMERCIAL_VIDEO_PLUGINS</filename> statements |
Patrick Williams | d8c66bc | 2016-06-20 12:57:21 -0500 | [diff] [blame] | 2031 | (along with the enabling |
| 2032 | <filename>LICENSE_FLAGS_WHITELIST</filename>) includes the |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 2033 | plug-ins or components into built images, thus adding |
| 2034 | support for media formats or components. |
| 2035 | </para> |
| 2036 | </section> |
| 2037 | </section> |
| 2038 | </section> |
| 2039 | </chapter> |
| 2040 | <!-- |
| 2041 | vim: expandtab tw=80 ts=4 |
| 2042 | --> |