blob: e9e76e46d74084728c4dd8486c7681aed9897b8d [file] [log] [blame]
Patrick Williamsc124f4f2015-09-15 14:41:29 -05001<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
2"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
3[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
4
5<chapter id='technical-details'>
6<title>Technical Details</title>
7
8 <para>
9 This chapter provides technical details for various parts of the
10 Yocto Project.
11 Currently, topics include Yocto Project components,
12 cross-toolchain generation, shared state (sstate) cache,
13 x32, Wayland support, and Licenses.
14 </para>
15
16<section id='usingpoky-components'>
17 <title>Yocto Project Components</title>
18
19 <para>
20 The
Brad Bishopd7bf8c12018-02-25 22:55:05 -050021 <link linkend='bitbake-term'>BitBake</link>
Patrick Williamsc124f4f2015-09-15 14:41:29 -050022 task executor together with various types of configuration files form
23 the OpenEmbedded Core.
24 This section overviews these components by describing their use and
25 how they interact.
26 </para>
27
28 <para>
29 BitBake handles the parsing and execution of the data files.
30 The data itself is of various types:
31 <itemizedlist>
32 <listitem><para><emphasis>Recipes:</emphasis> Provides details
33 about particular pieces of software.
34 </para></listitem>
35 <listitem><para><emphasis>Class Data:</emphasis> Abstracts
36 common build information (e.g. how to build a Linux kernel).
37 </para></listitem>
38 <listitem><para><emphasis>Configuration Data:</emphasis> Defines
39 machine-specific settings, policy decisions, and so forth.
40 Configuration data acts as the glue to bind everything
41 together.
42 </para></listitem>
43 </itemizedlist>
44 </para>
45
46 <para>
47 BitBake knows how to combine multiple data sources together and refers
48 to each data source as a layer.
49 For information on layers, see the
50 "<ulink url='&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers'>Understanding and
Brad Bishopd7bf8c12018-02-25 22:55:05 -050051 Creating Layers</ulink>" section of the Yocto Project Development
52 Tasks Manual.
Patrick Williamsc124f4f2015-09-15 14:41:29 -050053 </para>
54
55 <para>
56 Following are some brief details on these core components.
57 For additional information on how these components interact during
58 a build, see the
Brad Bishopd7bf8c12018-02-25 22:55:05 -050059 "<link linkend='development-concepts'>Development Concepts</link>"
60 section.
Patrick Williamsc124f4f2015-09-15 14:41:29 -050061 </para>
62
63 <section id='usingpoky-components-bitbake'>
64 <title>BitBake</title>
65
66 <para>
67 BitBake is the tool at the heart of the OpenEmbedded build system
68 and is responsible for parsing the
Brad Bishopd7bf8c12018-02-25 22:55:05 -050069 <link linkend='metadata'>Metadata</link>,
Patrick Williamsc124f4f2015-09-15 14:41:29 -050070 generating a list of tasks from it, and then executing those tasks.
71 </para>
72
73 <para>
74 This section briefly introduces BitBake.
75 If you want more information on BitBake, see the
76 <ulink url='&YOCTO_DOCS_BB_URL;#bitbake-user-manual'>BitBake User Manual</ulink>.
77 </para>
78
79 <para>
80 To see a list of the options BitBake supports, use either of
81 the following commands:
82 <literallayout class='monospaced'>
83 $ bitbake -h
84 $ bitbake --help
85 </literallayout>
86 </para>
87
88 <para>
89 The most common usage for BitBake is <filename>bitbake <replaceable>packagename</replaceable></filename>, where
90 <filename>packagename</filename> is the name of the package you want to build
91 (referred to as the "target" in this manual).
92 The target often equates to the first part of a recipe's filename
93 (e.g. "foo" for a recipe named
94 <filename>foo_1.3.0-r0.bb</filename>).
95 So, to process the <filename>matchbox-desktop_1.2.3.bb</filename> recipe file, you
96 might type the following:
97 <literallayout class='monospaced'>
98 $ bitbake matchbox-desktop
99 </literallayout>
100 Several different versions of <filename>matchbox-desktop</filename> might exist.
101 BitBake chooses the one selected by the distribution configuration.
102 You can get more details about how BitBake chooses between different
103 target versions and providers in the
104 "<ulink url='&YOCTO_DOCS_BB_URL;#bb-bitbake-preferences'>Preferences</ulink>"
105 section of the BitBake User Manual.
106 </para>
107
108 <para>
109 BitBake also tries to execute any dependent tasks first.
110 So for example, before building <filename>matchbox-desktop</filename>, BitBake
111 would build a cross compiler and <filename>glibc</filename> if they had not already
112 been built.
113 </para>
114
115 <para>
116 A useful BitBake option to consider is the <filename>-k</filename> or
117 <filename>--continue</filename> option.
118 This option instructs BitBake to try and continue processing the job
119 as long as possible even after encountering an error.
120 When an error occurs, the target that
121 failed and those that depend on it cannot be remade.
122 However, when you use this option other dependencies can still be
123 processed.
124 </para>
125 </section>
126
127 <section id='usingpoky-components-metadata'>
128 <title>Metadata (Recipes)</title>
129
130 <para>
131 Files that have the <filename>.bb</filename> suffix are "recipes"
132 files.
133 In general, a recipe contains information about a single piece of
134 software.
135 This information includes the location from which to download the
136 unaltered source, any source patches to be applied to that source
137 (if needed), which special configuration options to apply,
138 how to compile the source files, and how to package the compiled
139 output.
140 </para>
141
142 <para>
143 The term "package" is sometimes used to refer to recipes. However,
144 since the word "package" is used for the packaged output from the OpenEmbedded
145 build system (i.e. <filename>.ipk</filename> or <filename>.deb</filename> files),
146 this document avoids using the term "package" when referring to recipes.
147 </para>
148 </section>
149
Brad Bishopd7bf8c12018-02-25 22:55:05 -0500150 <section id='metadata-virtual-providers'>
151 <title>Metadata (Virtual Providers)</title>
152
153 <para>
154 Prior to the build, if you know that several different recipes
155 provide the same functionality, you can use a virtual provider
156 (i.e. <filename>virtual/*</filename>) as a placeholder for the
157 actual provider.
158 The actual provider would be determined at build
159 time.
160 In this case, you should add <filename>virtual/*</filename>
161 to <link linkend='var-DEPENDS'><filename>DEPENDS</filename></link>,
162 rather than listing the specified provider.
163 You would select the actual provider by setting the
164 <link linkend='var-PREFERRED_PROVIDER'><filename>PREFERRED_PROVIDER</filename></link>
165 variable (i.e. <filename>PREFERRED_PROVIDER_virtual/*</filename>)
166 in the build's configuration file (e.g.
167 <filename>poky/build/conf/local.conf</filename>).
168 <note>
169 Any recipe that PROVIDES a <filename>virtual/*</filename> item
170 that is ultimately not selected through
171 <filename>PREFERRED_PROVIDER</filename> does not get built.
172 Preventing these recipes from building is usually the desired
173 behavior since this mechanism's purpose is to select between
174 mutually exclusive alternative providers.
175 </note>
176 </para>
177
178 <para>
179 The following lists specific examples of virtual providers:
180 <itemizedlist>
181 <listitem><para>
182 <filename>virtual/mesa</filename>:
183 Provides <filename>gbm.pc</filename>.
184 </para></listitem>
185 <listitem><para>
186 <filename>virtual/egl</filename>:
187 Provides <filename>egl.pc</filename> and possibly
188 <filename>wayland-egl.pc</filename>.
189 </para></listitem>
190 <listitem><para>
191 <filename>virtual/libgl</filename>:
192 Provides <filename>gl.pc</filename> (i.e. libGL).
193 </para></listitem>
194 <listitem><para>
195 <filename>virtual/libgles1</filename>:
196 Provides <filename>glesv1_cm.pc</filename>
197 (i.e. libGLESv1_CM).
198 </para></listitem>
199 <listitem><para>
200 <filename>virtual/libgles2</filename>:
201 Provides <filename>glesv2.pc</filename> (i.e. libGLESv2).
202 </para></listitem>
203 </itemizedlist>
204 </para>
205 </section>
206
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500207 <section id='usingpoky-components-classes'>
208 <title>Classes</title>
209
210 <para>
211 Class files (<filename>.bbclass</filename>) contain information that
212 is useful to share between
Brad Bishopd7bf8c12018-02-25 22:55:05 -0500213 <link linkend='metadata'>Metadata</link> files.
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500214 An example is the
215 <link linkend='ref-classes-autotools'><filename>autotools</filename></link>
216 class, which contains common settings for any application that
217 Autotools uses.
218 The "<link linkend='ref-classes'>Classes</link>" chapter provides
219 details about classes and how to use them.
220 </para>
221 </section>
222
223 <section id='usingpoky-components-configuration'>
224 <title>Configuration</title>
225
226 <para>
227 The configuration files (<filename>.conf</filename>) define various configuration variables
228 that govern the OpenEmbedded build process.
229 These files fall into several areas that define machine configuration options,
230 distribution configuration options, compiler tuning options, general common configuration
231 options, and user configuration options in <filename>local.conf</filename>, which is found
232 in the
Brad Bishopd7bf8c12018-02-25 22:55:05 -0500233 <link linkend='build-directory'>Build Directory</link>.
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500234 </para>
235 </section>
236</section>
237
238<section id="cross-development-toolchain-generation">
239 <title>Cross-Development Toolchain Generation</title>
240
241 <para>
242 The Yocto Project does most of the work for you when it comes to
243 creating
Brad Bishopd7bf8c12018-02-25 22:55:05 -0500244 <link linkend='cross-development-toolchain'>cross-development toolchains</link>.
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500245 This section provides some technical background on how
246 cross-development toolchains are created and used.
247 For more information on toolchains, you can also see the
Brad Bishopd7bf8c12018-02-25 22:55:05 -0500248 <ulink url='&YOCTO_DOCS_SDK_URL;'>Yocto Project Application Development and the Extensible Software Development Kit (eSDK)</ulink>
249 manual.
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500250 </para>
251
252 <para>
253 In the Yocto Project development environment, cross-development
254 toolchains are used to build the image and applications that run on the
255 target hardware.
256 With just a few commands, the OpenEmbedded build system creates
257 these necessary toolchains for you.
258 </para>
259
260 <para>
261 The following figure shows a high-level build environment regarding
262 toolchain construction and use.
263 </para>
264
265 <para>
266 <imagedata fileref="figures/cross-development-toolchains.png" width="8in" depth="6in" align="center" />
267 </para>
268
269 <para>
270 Most of the work occurs on the Build Host.
271 This is the machine used to build images and generally work within the
272 the Yocto Project environment.
273 When you run BitBake to create an image, the OpenEmbedded build system
274 uses the host <filename>gcc</filename> compiler to bootstrap a
275 cross-compiler named <filename>gcc-cross</filename>.
276 The <filename>gcc-cross</filename> compiler is what BitBake uses to
277 compile source files when creating the target image.
278 You can think of <filename>gcc-cross</filename> simply as an
279 automatically generated cross-compiler that is used internally within
280 BitBake only.
Patrick Williamsd8c66bc2016-06-20 12:57:21 -0500281 <note>
282 The extensible SDK does not use
283 <filename>gcc-cross-canadian</filename> since this SDK
284 ships a copy of the OpenEmbedded build system and the sysroot
285 within it contains <filename>gcc-cross</filename>.
286 </note>
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500287 </para>
288
289 <para>
290 The chain of events that occurs when <filename>gcc-cross</filename> is
291 bootstrapped is as follows:
292 <literallayout class='monospaced'>
293 gcc -> binutils-cross -> gcc-cross-initial -> linux-libc-headers -> glibc-initial -> glibc -> gcc-cross -> gcc-runtime
294 </literallayout>
295 <itemizedlist>
296 <listitem><para><filename>gcc</filename>:
297 The build host's GNU Compiler Collection (GCC).
298 </para></listitem>
299 <listitem><para><filename>binutils-cross</filename>:
300 The bare minimum binary utilities needed in order to run
301 the <filename>gcc-cross-initial</filename> phase of the
302 bootstrap operation.
303 </para></listitem>
304 <listitem><para><filename>gcc-cross-initial</filename>:
305 An early stage of the bootstrap process for creating
306 the cross-compiler.
307 This stage builds enough of the <filename>gcc-cross</filename>,
308 the C library, and other pieces needed to finish building the
309 final cross-compiler in later stages.
310 This tool is a "native" package (i.e. it is designed to run on
311 the build host).
312 </para></listitem>
313 <listitem><para><filename>linux-libc-headers</filename>:
314 Headers needed for the cross-compiler.
315 </para></listitem>
316 <listitem><para><filename>glibc-initial</filename>:
317 An initial version of the Embedded GLIBC needed to bootstrap
318 <filename>glibc</filename>.
319 </para></listitem>
320 <listitem><para><filename>gcc-cross</filename>:
321 The final stage of the bootstrap process for the
322 cross-compiler.
323 This stage results in the actual cross-compiler that
324 BitBake uses when it builds an image for a targeted
325 device.
326 <note>
327 If you are replacing this cross compiler toolchain
328 with a custom version, you must replace
329 <filename>gcc-cross</filename>.
330 </note>
331 This tool is also a "native" package (i.e. it is
332 designed to run on the build host).
333 </para></listitem>
334 <listitem><para><filename>gcc-runtime</filename>:
335 Runtime libraries resulting from the toolchain bootstrapping
336 process.
337 This tool produces a binary that consists of the
338 runtime libraries need for the targeted device.
339 </para></listitem>
340 </itemizedlist>
341 </para>
342
343 <para>
344 You can use the OpenEmbedded build system to build an installer for
345 the relocatable SDK used to develop applications.
346 When you run the installer, it installs the toolchain, which contains
347 the development tools (e.g., the
348 <filename>gcc-cross-canadian</filename>),
349 <filename>binutils-cross-canadian</filename>, and other
Patrick Williamsd8c66bc2016-06-20 12:57:21 -0500350 <filename>nativesdk-*</filename> tools,
351 which are tools native to the SDK (i.e. native to
352 <link linkend='var-SDK_ARCH'><filename>SDK_ARCH</filename></link>),
353 you need to cross-compile and test your software.
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500354 The figure shows the commands you use to easily build out this
355 toolchain.
356 This cross-development toolchain is built to execute on the
357 <link linkend='var-SDKMACHINE'><filename>SDKMACHINE</filename></link>,
358 which might or might not be the same
359 machine as the Build Host.
360 <note>
361 If your target architecture is supported by the Yocto Project,
362 you can take advantage of pre-built images that ship with the
363 Yocto Project and already contain cross-development toolchain
364 installers.
365 </note>
366 </para>
367
368 <para>
369 Here is the bootstrap process for the relocatable toolchain:
370 <literallayout class='monospaced'>
371 gcc -> binutils-crosssdk -> gcc-crosssdk-initial -> linux-libc-headers ->
372 glibc-initial -> nativesdk-glibc -> gcc-crosssdk -> gcc-cross-canadian
373 </literallayout>
374 <itemizedlist>
375 <listitem><para><filename>gcc</filename>:
376 The build host's GNU Compiler Collection (GCC).
377 </para></listitem>
378 <listitem><para><filename>binutils-crosssdk</filename>:
379 The bare minimum binary utilities needed in order to run
380 the <filename>gcc-crosssdk-initial</filename> phase of the
381 bootstrap operation.
382 </para></listitem>
383 <listitem><para><filename>gcc-crosssdk-initial</filename>:
384 An early stage of the bootstrap process for creating
385 the cross-compiler.
386 This stage builds enough of the
387 <filename>gcc-crosssdk</filename> and supporting pieces so that
388 the final stage of the bootstrap process can produce the
389 finished cross-compiler.
390 This tool is a "native" binary that runs on the build host.
391 </para></listitem>
392 <listitem><para><filename>linux-libc-headers</filename>:
393 Headers needed for the cross-compiler.
394 </para></listitem>
395 <listitem><para><filename>glibc-initial</filename>:
396 An initial version of the Embedded GLIBC needed to bootstrap
397 <filename>nativesdk-glibc</filename>.
398 </para></listitem>
399 <listitem><para><filename>nativesdk-glibc</filename>:
400 The Embedded GLIBC needed to bootstrap the
401 <filename>gcc-crosssdk</filename>.
402 </para></listitem>
403 <listitem><para><filename>gcc-crosssdk</filename>:
404 The final stage of the bootstrap process for the
405 relocatable cross-compiler.
406 The <filename>gcc-crosssdk</filename> is a transitory compiler
407 and never leaves the build host.
408 Its purpose is to help in the bootstrap process to create the
409 eventual relocatable <filename>gcc-cross-canadian</filename>
410 compiler, which is relocatable.
411 This tool is also a "native" package (i.e. it is
412 designed to run on the build host).
413 </para></listitem>
414 <listitem><para><filename>gcc-cross-canadian</filename>:
415 The final relocatable cross-compiler.
416 When run on the
417 <link linkend='var-SDKMACHINE'><filename>SDKMACHINE</filename></link>,
418 this tool
419 produces executable code that runs on the target device.
420 Only one cross-canadian compiler is produced per architecture
421 since they can be targeted at different processor optimizations
422 using configurations passed to the compiler through the
423 compile commands.
424 This circumvents the need for multiple compilers and thus
425 reduces the size of the toolchains.
426 </para></listitem>
427 </itemizedlist>
428 </para>
429
430 <note>
431 For information on advantages gained when building a
432 cross-development toolchain installer, see the
Patrick Williamsd8c66bc2016-06-20 12:57:21 -0500433 "<ulink url='&YOCTO_DOCS_SDK_URL;#sdk-building-an-sdk-installer'>Building an SDK Installer</ulink>"
Brad Bishopd7bf8c12018-02-25 22:55:05 -0500434 section in the Yocto Project Application Development and the
435 Extensible Software Development Kit (eSDK) manual.
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500436 </note>
437</section>
438
439<section id="shared-state-cache">
440 <title>Shared State Cache</title>
441
442 <para>
443 By design, the OpenEmbedded build system builds everything from scratch unless
444 BitBake can determine that parts do not need to be rebuilt.
445 Fundamentally, building from scratch is attractive as it means all parts are
446 built fresh and there is no possibility of stale data causing problems.
447 When developers hit problems, they typically default back to building from scratch
448 so they know the state of things from the start.
449 </para>
450
451 <para>
452 Building an image from scratch is both an advantage and a disadvantage to the process.
453 As mentioned in the previous paragraph, building from scratch ensures that
454 everything is current and starts from a known state.
455 However, building from scratch also takes much longer as it generally means
456 rebuilding things that do not necessarily need to be rebuilt.
457 </para>
458
459 <para>
460 The Yocto Project implements shared state code that supports incremental builds.
461 The implementation of the shared state code answers the following questions that
462 were fundamental roadblocks within the OpenEmbedded incremental build support system:
463 <itemizedlist>
464 <listitem><para>What pieces of the system have changed and what pieces have
465 not changed?</para></listitem>
466 <listitem><para>How are changed pieces of software removed and replaced?</para></listitem>
467 <listitem><para>How are pre-built components that do not need to be rebuilt from scratch
468 used when they are available?</para></listitem>
469 </itemizedlist>
470 </para>
471
472 <para>
473 For the first question, the build system detects changes in the "inputs" to a given task by
474 creating a checksum (or signature) of the task's inputs.
475 If the checksum changes, the system assumes the inputs have changed and the task needs to be
476 rerun.
477 For the second question, the shared state (sstate) code tracks which tasks add which output
478 to the build process.
479 This means the output from a given task can be removed, upgraded or otherwise manipulated.
480 The third question is partly addressed by the solution for the second question
481 assuming the build system can fetch the sstate objects from remote locations and
482 install them if they are deemed to be valid.
483 </para>
484
485 <note>
486 The OpenEmbedded build system does not maintain
487 <link linkend='var-PR'><filename>PR</filename></link> information
488 as part of the shared state packages.
489 Consequently, considerations exist that affect maintaining shared
490 state feeds.
491 For information on how the OpenEmbedded build system
492 works with packages and can
493 track incrementing <filename>PR</filename> information, see the
Brad Bishop6e60e8b2018-02-01 10:27:11 -0500494 "<ulink url='&YOCTO_DOCS_DEV_URL;#automatically-incrementing-a-binary-package-revision-number'>Automatically Incrementing a Binary Package Revision Number</ulink>"
Brad Bishopd7bf8c12018-02-25 22:55:05 -0500495 section in the Yocto Project Development Tasks Manual.
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500496 </note>
497
498 <para>
499 The rest of this section goes into detail about the overall incremental build
500 architecture, the checksums (signatures), shared state, and some tips and tricks.
501 </para>
502
503 <section id='overall-architecture'>
504 <title>Overall Architecture</title>
505
506 <para>
507 When determining what parts of the system need to be built, BitBake
508 works on a per-task basis rather than a per-recipe basis.
509 You might wonder why using a per-task basis is preferred over a per-recipe basis.
510 To help explain, consider having the IPK packaging backend enabled and then switching to DEB.
511 In this case, the
512 <link linkend='ref-tasks-install'><filename>do_install</filename></link>
513 and
514 <link linkend='ref-tasks-package'><filename>do_package</filename></link>
515 task outputs are still valid.
516 However, with a per-recipe approach, the build would not include the
517 <filename>.deb</filename> files.
518 Consequently, you would have to invalidate the whole build and rerun it.
519 Rerunning everything is not the best solution.
520 Also, in this case, the core must be "taught" much about specific tasks.
521 This methodology does not scale well and does not allow users to easily add new tasks
522 in layers or as external recipes without touching the packaged-staging core.
523 </para>
524 </section>
525
526 <section id='checksums'>
527 <title>Checksums (Signatures)</title>
528
529 <para>
530 The shared state code uses a checksum, which is a unique signature of a task's
531 inputs, to determine if a task needs to be run again.
532 Because it is a change in a task's inputs that triggers a rerun, the process
533 needs to detect all the inputs to a given task.
534 For shell tasks, this turns out to be fairly easy because
535 the build process generates a "run" shell script for each task and
536 it is possible to create a checksum that gives you a good idea of when
537 the task's data changes.
538 </para>
539
540 <para>
Patrick Williamsd8c66bc2016-06-20 12:57:21 -0500541 To complicate the problem, there are things that should not be
542 included in the checksum.
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500543 First, there is the actual specific build path of a given task -
544 the <link linkend='var-WORKDIR'><filename>WORKDIR</filename></link>.
Patrick Williamsd8c66bc2016-06-20 12:57:21 -0500545 It does not matter if the work directory changes because it should
546 not affect the output for target packages.
547 Also, the build process has the objective of making native
548 or cross packages relocatable.
549 <note>
550 Both native and cross packages run on the build host.
551 However, cross packages generate output for the target
552 architecture.
553 </note>
554 The checksum therefore needs to exclude
555 <filename>WORKDIR</filename>.
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500556 The simplistic approach for excluding the work directory is to set
Patrick Williamsd8c66bc2016-06-20 12:57:21 -0500557 <filename>WORKDIR</filename> to some fixed value and create the
558 checksum for the "run" script.
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500559 </para>
560
561 <para>
562 Another problem results from the "run" scripts containing functions that
563 might or might not get called.
564 The incremental build solution contains code that figures out dependencies
565 between shell functions.
566 This code is used to prune the "run" scripts down to the minimum set,
567 thereby alleviating this problem and making the "run" scripts much more
568 readable as a bonus.
569 </para>
570
571 <para>
572 So far we have solutions for shell scripts.
573 What about Python tasks?
574 The same approach applies even though these tasks are more difficult.
575 The process needs to figure out what variables a Python function accesses
576 and what functions it calls.
577 Again, the incremental build solution contains code that first figures out
578 the variable and function dependencies, and then creates a checksum for the data
579 used as the input to the task.
580 </para>
581
582 <para>
583 Like the <filename>WORKDIR</filename> case, situations exist where dependencies
584 should be ignored.
585 For these cases, you can instruct the build process to ignore a dependency
586 by using a line like the following:
587 <literallayout class='monospaced'>
588 PACKAGE_ARCHS[vardepsexclude] = "MACHINE"
589 </literallayout>
590 This example ensures that the
591 <link linkend='var-PACKAGE_ARCHS'><filename>PACKAGE_ARCHS</filename></link>
592 variable does not
593 depend on the value of
594 <link linkend='var-MACHINE'><filename>MACHINE</filename></link>,
595 even if it does reference it.
596 </para>
597
598 <para>
599 Equally, there are cases where we need to add dependencies BitBake is not able to find.
600 You can accomplish this by using a line like the following:
601 <literallayout class='monospaced'>
602 PACKAGE_ARCHS[vardeps] = "MACHINE"
603 </literallayout>
604 This example explicitly adds the <filename>MACHINE</filename> variable as a
605 dependency for <filename>PACKAGE_ARCHS</filename>.
606 </para>
607
608 <para>
609 Consider a case with in-line Python, for example, where BitBake is not
610 able to figure out dependencies.
611 When running in debug mode (i.e. using <filename>-DDD</filename>), BitBake
612 produces output when it discovers something for which it cannot figure out
613 dependencies.
614 The Yocto Project team has currently not managed to cover those dependencies
615 in detail and is aware of the need to fix this situation.
616 </para>
617
618 <para>
619 Thus far, this section has limited discussion to the direct inputs into a task.
620 Information based on direct inputs is referred to as the "basehash" in the
621 code.
622 However, there is still the question of a task's indirect inputs - the
623 things that were already built and present in the
Brad Bishopd7bf8c12018-02-25 22:55:05 -0500624 <link linkend='build-directory'>Build Directory</link>.
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500625 The checksum (or signature) for a particular task needs to add the hashes
626 of all the tasks on which the particular task depends.
627 Choosing which dependencies to add is a policy decision.
628 However, the effect is to generate a master checksum that combines the basehash
629 and the hashes of the task's dependencies.
630 </para>
631
632 <para>
633 At the code level, there are a variety of ways both the basehash and the
634 dependent task hashes can be influenced.
635 Within the BitBake configuration file, we can give BitBake some extra information
636 to help it construct the basehash.
637 The following statement effectively results in a list of global variable
638 dependency excludes - variables never included in any checksum:
639 <literallayout class='monospaced'>
640 BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \
641 SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \
642 USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \
643 PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \
644 CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX"
645 </literallayout>
646 The previous example excludes
647 <link linkend='var-WORKDIR'><filename>WORKDIR</filename></link>
648 since that variable is actually constructed as a path within
649 <link linkend='var-TMPDIR'><filename>TMPDIR</filename></link>, which is on
650 the whitelist.
651 </para>
652
653 <para>
654 The rules for deciding which hashes of dependent tasks to include through
655 dependency chains are more complex and are generally accomplished with a
656 Python function.
657 The code in <filename>meta/lib/oe/sstatesig.py</filename> shows two examples
658 of this and also illustrates how you can insert your own policy into the system
659 if so desired.
Brad Bishopd7bf8c12018-02-25 22:55:05 -0500660 This file defines the two basic signature generators
661 <link linkend='oe-core'>OE-Core</link> uses: "OEBasic" and
662 "OEBasicHash".
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500663 By default, there is a dummy "noop" signature handler enabled in BitBake.
664 This means that behavior is unchanged from previous versions.
Brad Bishopd7bf8c12018-02-25 22:55:05 -0500665 OE-Core uses the "OEBasicHash" signature handler by default
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500666 through this setting in the <filename>bitbake.conf</filename> file:
667 <literallayout class='monospaced'>
668 BB_SIGNATURE_HANDLER ?= "OEBasicHash"
669 </literallayout>
670 The "OEBasicHash" <filename>BB_SIGNATURE_HANDLER</filename> is the same as the
671 "OEBasic" version but adds the task hash to the stamp files.
672 This results in any
Brad Bishopd7bf8c12018-02-25 22:55:05 -0500673 <link linkend='metadata'>Metadata</link>
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500674 change that changes the task hash, automatically
675 causing the task to be run again.
676 This removes the need to bump <link linkend='var-PR'><filename>PR</filename></link>
677 values, and changes to Metadata automatically ripple across the build.
678 </para>
679
680 <para>
681 It is also worth noting that the end result of these signature generators is to
682 make some dependency and hash information available to the build.
683 This information includes:
684 <itemizedlist>
685 <listitem><para><filename>BB_BASEHASH_task-</filename><replaceable>taskname</replaceable>:
686 The base hashes for each task in the recipe.
687 </para></listitem>
688 <listitem><para><filename>BB_BASEHASH_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>:
689 The base hashes for each dependent task.
690 </para></listitem>
691 <listitem><para><filename>BBHASHDEPS_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>:
692 The task dependencies for each task.
693 </para></listitem>
694 <listitem><para><filename>BB_TASKHASH</filename>:
695 The hash of the currently running task.
696 </para></listitem>
697 </itemizedlist>
698 </para>
699 </section>
700
701 <section id='shared-state'>
702 <title>Shared State</title>
703
704 <para>
705 Checksums and dependencies, as discussed in the previous section, solve half the
706 problem of supporting a shared state.
707 The other part of the problem is being able to use checksum information during the build
708 and being able to reuse or rebuild specific components.
709 </para>
710
711 <para>
712 The
713 <link linkend='ref-classes-sstate'><filename>sstate</filename></link>
714 class is a relatively generic implementation of how to "capture"
715 a snapshot of a given task.
716 The idea is that the build process does not care about the source of a task's output.
717 Output could be freshly built or it could be downloaded and unpacked from
718 somewhere - the build process does not need to worry about its origin.
719 </para>
720
721 <para>
722 There are two types of output, one is just about creating a directory
723 in <link linkend='var-WORKDIR'><filename>WORKDIR</filename></link>.
724 A good example is the output of either
725 <link linkend='ref-tasks-install'><filename>do_install</filename></link>
726 or
727 <link linkend='ref-tasks-package'><filename>do_package</filename></link>.
728 The other type of output occurs when a set of data is merged into a shared directory
729 tree such as the sysroot.
730 </para>
731
732 <para>
733 The Yocto Project team has tried to keep the details of the
734 implementation hidden in <filename>sstate</filename> class.
735 From a user's perspective, adding shared state wrapping to a task
736 is as simple as this
737 <link linkend='ref-tasks-deploy'><filename>do_deploy</filename></link>
738 example taken from the
739 <link linkend='ref-classes-deploy'><filename>deploy</filename></link>
740 class:
741 <literallayout class='monospaced'>
742 DEPLOYDIR = "${WORKDIR}/deploy-${PN}"
743 SSTATETASKS += "do_deploy"
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500744 do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"
745 do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
746
747 python do_deploy_setscene () {
748 sstate_setscene(d)
749 }
750 addtask do_deploy_setscene
751 do_deploy[dirs] = "${DEPLOYDIR} ${B}"
752 </literallayout>
Patrick Williamsc0f7c042017-02-23 20:41:17 -0600753 The following list explains the previous example:
754 <itemizedlist>
755 <listitem><para>
756 Adding "do_deploy" to <filename>SSTATETASKS</filename>
757 adds some required sstate-related processing, which is
758 implemented in the
759 <link linkend='ref-classes-sstate'><filename>sstate</filename></link>
760 class, to before and after the
761 <link linkend='ref-tasks-deploy'><filename>do_deploy</filename></link>
762 task.
763 </para></listitem>
764 <listitem><para>
765 The
766 <filename>do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"</filename>
767 declares that <filename>do_deploy</filename> places its
768 output in <filename>${DEPLOYDIR}</filename> when run
769 normally (i.e. when not using the sstate cache).
770 This output becomes the input to the shared state cache.
771 </para></listitem>
772 <listitem><para>
773 The
774 <filename>do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"</filename>
775 line causes the contents of the shared state cache to be
776 copied to <filename>${DEPLOY_DIR_IMAGE}</filename>.
777 <note>
778 If <filename>do_deploy</filename> is not already in
779 the shared state cache or if its input checksum
780 (signature) has changed from when the output was
781 cached, the task will be run to populate the shared
782 state cache, after which the contents of the shared
783 state cache is copied to
784 <filename>${DEPLOY_DIR_IMAGE}</filename>.
785 If <filename>do_deploy</filename> is in the shared
786 state cache and its signature indicates that the
787 cached output is still valid (i.e. if no
788 relevant task inputs have changed), then the contents
789 of the shared state cache will be copied directly to
790 <filename>${DEPLOY_DIR_IMAGE}</filename> by the
791 <filename>do_deploy_setscene</filename> task instead,
792 skipping the <filename>do_deploy</filename> task.
793 </note>
794 </para></listitem>
795 <listitem><para>
796 The following task definition is glue logic needed to make
797 the previous settings effective:
798 <literallayout class='monospaced'>
799 python do_deploy_setscene () {
800 sstate_setscene(d)
801 }
802 addtask do_deploy_setscene
803 </literallayout>
804 <filename>sstate_setscene()</filename> takes the flags
805 above as input and accelerates the
806 <filename>do_deploy</filename> task through the
807 shared state cache if possible.
808 If the task was accelerated,
809 <filename>sstate_setscene()</filename> returns True.
810 Otherwise, it returns False, and the normal
811 <filename>do_deploy</filename> task runs.
812 For more information, see the
813 "<ulink url='&YOCTO_DOCS_BB_URL;#setscene'>setscene</ulink>"
814 section in the BitBake User Manual.
815 </para></listitem>
816 <listitem><para>
817 The <filename>do_deploy[dirs] = "${DEPLOYDIR} ${B}"</filename>
818 line creates <filename>${DEPLOYDIR}</filename> and
819 <filename>${B}</filename> before the
820 <filename>do_deploy</filename> task runs, and also sets
821 the current working directory of
822 <filename>do_deploy</filename> to
823 <filename>${B}</filename>.
824 For more information, see the
825 "<ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'>Variable Flags</ulink>"
826 section in the BitBake User Manual.
827 <note>
828 In cases where
829 <filename>sstate-inputdirs</filename> and
830 <filename>sstate-outputdirs</filename> would be the
831 same, you can use
832 <filename>sstate-plaindirs</filename>.
833 For example, to preserve the
834 <filename>${PKGD}</filename> and
835 <filename>${PKGDEST}</filename> output from the
836 <link linkend='ref-tasks-package'><filename>do_package</filename></link>
837 task, use the following:
838 <literallayout class='monospaced'>
839 do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}"
840 </literallayout>
841 </note>
842 </para></listitem>
843 <listitem><para>
844 <filename>sstate-inputdirs</filename> and
845 <filename>sstate-outputdirs</filename> can also be used
846 with multiple directories.
847 For example, the following declares
848 <filename>PKGDESTWORK</filename> and
849 <filename>SHLIBWORK</filename> as shared state
850 input directories, which populates the shared state
851 cache, and <filename>PKGDATA_DIR</filename> and
852 <filename>SHLIBSDIR</filename> as the corresponding
853 shared state output directories:
854 <literallayout class='monospaced'>
855 do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}"
856 do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}"
857 </literallayout>
858 </para></listitem>
859 <listitem><para>
860 These methods also include the ability to take a lockfile
861 when manipulating shared state directory structures,
862 for cases where file additions or removals are sensitive:
863 <literallayout class='monospaced'>
864 do_package[sstate-lockfile] = "${PACKAGELOCK}"
865 </literallayout>
866 </para></listitem>
867 </itemizedlist>
868 </para>
869
870<!--
871 <para>
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500872 In this example, we add some extra flags to the task, a name field ("deploy"), an
873 input directory where the task sends data, and the output
874 directory where the data from the task should eventually be copied.
875 We also add a <filename>_setscene</filename> variant of the task and add the task
876 name to the <filename>SSTATETASKS</filename> list.
877 </para>
878
879 <para>
880 If you have a directory whose contents you need to preserve, you can do this with
881 a line like the following:
882 <literallayout class='monospaced'>
883 do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}"
884 </literallayout>
885 This method, as well as the following example, also works for multiple directories.
886 <literallayout class='monospaced'>
887 do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}"
888 do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}"
889 do_package[sstate-lockfile] = "${PACKAGELOCK}"
890 </literallayout>
891 These methods also include the ability to take a lockfile when manipulating
892 shared state directory structures since some cases are sensitive to file
893 additions or removals.
894 </para>
Patrick Williamsc0f7c042017-02-23 20:41:17 -0600895-->
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500896
897 <para>
898 Behind the scenes, the shared state code works by looking in
899 <link linkend='var-SSTATE_DIR'><filename>SSTATE_DIR</filename></link> and
900 <link linkend='var-SSTATE_MIRRORS'><filename>SSTATE_MIRRORS</filename></link>
901 for shared state files.
902 Here is an example:
903 <literallayout class='monospaced'>
904 SSTATE_MIRRORS ?= "\
Patrick Williamsc0f7c042017-02-23 20:41:17 -0600905 file://.* http://someserver.tld/share/sstate/PATH;downloadfilename=PATH \n \
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500906 file://.* file:///some/local/dir/sstate/PATH"
907 </literallayout>
908 <note>
909 The shared state directory (<filename>SSTATE_DIR</filename>) is
910 organized into two-character subdirectories, where the subdirectory
911 names are based on the first two characters of the hash.
912 If the shared state directory structure for a mirror has the
913 same structure as <filename>SSTATE_DIR</filename>, you must
914 specify "PATH" as part of the URI to enable the build system
915 to map to the appropriate subdirectory.
916 </note>
917 </para>
918
919 <para>
920 The shared state package validity can be detected just by looking at the
921 filename since the filename contains the task checksum (or signature) as
922 described earlier in this section.
923 If a valid shared state package is found, the build process downloads it
924 and uses it to accelerate the task.
925 </para>
926
927 <para>
928 The build processes use the <filename>*_setscene</filename> tasks
929 for the task acceleration phase.
930 BitBake goes through this phase before the main execution code and tries
931 to accelerate any tasks for which it can find shared state packages.
932 If a shared state package for a task is available, the shared state
933 package is used.
934 This means the task and any tasks on which it is dependent are not
935 executed.
936 </para>
937
938 <para>
939 As a real world example, the aim is when building an IPK-based image,
940 only the
941 <link linkend='ref-tasks-package_write_ipk'><filename>do_package_write_ipk</filename></link>
942 tasks would have their
943 shared state packages fetched and extracted.
944 Since the sysroot is not used, it would never get extracted.
945 This is another reason why a task-based approach is preferred over a
946 recipe-based approach, which would have to install the output from every task.
947 </para>
948 </section>
949
950 <section id='tips-and-tricks'>
951 <title>Tips and Tricks</title>
952
953 <para>
954 The code in the build system that supports incremental builds is not
955 simple code.
956 This section presents some tips and tricks that help you work around
957 issues related to shared state code.
958 </para>
959
960 <section id='debugging'>
961 <title>Debugging</title>
962
963 <para>
Patrick Williamsc0f7c042017-02-23 20:41:17 -0600964 Seeing what metadata went into creating the input signature
965 of a shared state (sstate) task can be a useful debugging aid.
966 This information is available in signature information
967 (<filename>siginfo</filename>) files in
968 <link linkend='var-SSTATE_DIR'><filename>SSTATE_DIR</filename></link>.
969 For information on how to view and interpret information in
970 <filename>siginfo</filename> files, see the
971 "<link linkend='usingpoky-viewing-task-variable-dependencies'>Viewing Task Variable Dependencies</link>"
972 section.
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500973 </para>
974 </section>
975
976 <section id='invalidating-shared-state'>
977 <title>Invalidating Shared State</title>
978
979 <para>
980 The OpenEmbedded build system uses checksums and shared state
981 cache to avoid unnecessarily rebuilding tasks.
982 Collectively, this scheme is known as "shared state code."
983 </para>
984
985 <para>
986 As with all schemes, this one has some drawbacks.
987 It is possible that you could make implicit changes to your
988 code that the checksum calculations do not take into
989 account.
990 These implicit changes affect a task's output but do not trigger
991 the shared state code into rebuilding a recipe.
992 Consider an example during which a tool changes its output.
993 Assume that the output of <filename>rpmdeps</filename> changes.
994 The result of the change should be that all the
995 <filename>package</filename> and
996 <filename>package_write_rpm</filename> shared state cache
997 items become invalid.
998 However, because the change to the output is
999 external to the code and therefore implicit,
1000 the associated shared state cache items do not become
1001 invalidated.
1002 In this case, the build process uses the cached items rather
1003 than running the task again.
1004 Obviously, these types of implicit changes can cause problems.
1005 </para>
1006
1007 <para>
1008 To avoid these problems during the build, you need to
1009 understand the effects of any changes you make.
1010 Realize that changes you make directly to a function
1011 are automatically factored into the checksum calculation.
1012 Thus, these explicit changes invalidate the associated area of
1013 shared state cache.
1014 However, you need to be aware of any implicit changes that
1015 are not obvious changes to the code and could affect the output
1016 of a given task.
1017 </para>
1018
1019 <para>
1020 When you identify an implicit change, you can easily take steps
1021 to invalidate the cache and force the tasks to run.
1022 The steps you can take are as simple as changing a function's
1023 comments in the source code.
1024 For example, to invalidate package shared state files, change
1025 the comment statements of
1026 <link linkend='ref-tasks-package'><filename>do_package</filename></link>
1027 or the comments of one of the functions it calls.
1028 Even though the change is purely cosmetic, it causes the
1029 checksum to be recalculated and forces the OpenEmbedded build
1030 system to run the task again.
1031 </para>
1032
1033 <note>
1034 For an example of a commit that makes a cosmetic change to
1035 invalidate shared state, see this
1036 <ulink url='&YOCTO_GIT_URL;/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54'>commit</ulink>.
1037 </note>
1038 </section>
1039 </section>
1040</section>
1041
Patrick Williamsc0f7c042017-02-23 20:41:17 -06001042<section id='automatically-added-runtime-dependencies'>
1043 <title>Automatically Added Runtime Dependencies</title>
1044
1045 <para>
1046 The OpenEmbedded build system automatically adds common types of
1047 runtime dependencies between packages, which means that you do not
1048 need to explicitly declare the packages using
1049 <link linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link>.
1050 Three automatic mechanisms exist (<filename>shlibdeps</filename>,
1051 <filename>pcdeps</filename>, and <filename>depchains</filename>) that
1052 handle shared libraries, package configuration (pkg-config) modules,
1053 and <filename>-dev</filename> and <filename>-dbg</filename> packages,
1054 respectively.
1055 For other types of runtime dependencies, you must manually declare
1056 the dependencies.
1057 <itemizedlist>
1058 <listitem><para>
1059 <filename>shlibdeps</filename>:
1060 During the
1061 <link linkend='ref-tasks-package'><filename>do_package</filename></link>
1062 task of each recipe, all shared libraries installed by the
1063 recipe are located.
1064 For each shared library, the package that contains the shared
1065 library is registered as providing the shared library.
1066 More specifically, the package is registered as providing the
1067 <ulink url='https://en.wikipedia.org/wiki/Soname'>soname</ulink>
1068 of the library.
1069 The resulting shared-library-to-package mapping
1070 is saved globally in
1071 <link linkend='var-PKGDATA_DIR'><filename>PKGDATA_DIR</filename></link>
1072 by the
1073 <link linkend='ref-tasks-packagedata'><filename>do_packagedata</filename></link>
1074 task.</para>
1075
1076 <para>Simultaneously, all executables and shared libraries
1077 installed by the recipe are inspected to see what shared
1078 libraries they link against.
1079 For each shared library dependency that is found,
1080 <filename>PKGDATA_DIR</filename> is queried to
1081 see if some package (likely from a different recipe) contains
1082 the shared library.
1083 If such a package is found, a runtime dependency is added from
1084 the package that depends on the shared library to the package
1085 that contains the library.</para>
1086
1087 <para>The automatically added runtime dependency also includes
1088 a version restriction.
1089 This version restriction specifies that at least the current
1090 version of the package that provides the shared library must be
1091 used, as if
1092 "<replaceable>package</replaceable> (>= <replaceable>version</replaceable>)"
1093 had been added to
1094 <link linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link>.
1095 This forces an upgrade of the package containing the shared
1096 library when installing the package that depends on the
1097 library, if needed.</para>
1098
1099 <para>If you want to avoid a package being registered as
1100 providing a particular shared library (e.g. because the library
1101 is for internal use only), then add the library to
1102 <link linkend='var-PRIVATE_LIBS'><filename>PRIVATE_LIBS</filename></link>
1103 inside the package's recipe.
1104 </para></listitem>
1105 <listitem><para>
1106 <filename>pcdeps</filename>:
1107 During the
1108 <link linkend='ref-tasks-package'><filename>do_package</filename></link>
1109 task of each recipe, all pkg-config modules
1110 (<filename>*.pc</filename> files) installed by the recipe are
1111 located.
1112 For each module, the package that contains the module is
1113 registered as providing the module.
1114 The resulting module-to-package mapping is saved globally in
1115 <link linkend='var-PKGDATA_DIR'><filename>PKGDATA_DIR</filename></link>
1116 by the
1117 <link linkend='ref-tasks-packagedata'><filename>do_packagedata</filename></link>
1118 task.</para>
1119
1120 <para>Simultaneously, all pkg-config modules installed by the
1121 recipe are inspected to see what other pkg-config modules they
1122 depend on.
1123 A module is seen as depending on another module if it contains
1124 a "Requires:" line that specifies the other module.
1125 For each module dependency,
1126 <filename>PKGDATA_DIR</filename> is queried to see if some
1127 package contains the module.
1128 If such a package is found, a runtime dependency is added from
1129 the package that depends on the module to the package that
1130 contains the module.
1131 <note>
1132 The <filename>pcdeps</filename> mechanism most often infers
1133 dependencies between <filename>-dev</filename> packages.
1134 </note>
1135 </para></listitem>
1136 <listitem><para>
1137 <filename>depchains</filename>:
1138 If a package <filename>foo</filename> depends on a package
1139 <filename>bar</filename>, then <filename>foo-dev</filename>
1140 and <filename>foo-dbg</filename> are also made to depend on
1141 <filename>bar-dev</filename> and <filename>bar-dbg</filename>,
1142 respectively.
1143 Taking the <filename>-dev</filename> packages as an example,
1144 the <filename>bar-dev</filename> package might provide
1145 headers and shared library symlinks needed by
1146 <filename>foo-dev</filename>, which shows the need
1147 for a dependency between the packages.</para>
1148
1149 <para>The dependencies added by <filename>depchains</filename>
1150 are in the form of
1151 <link linkend='var-RRECOMMENDS'><filename>RRECOMMENDS</filename></link>.
1152 <note>
1153 By default, <filename>foo-dev</filename> also has an
1154 <filename>RDEPENDS</filename>-style dependency on
1155 <filename>foo</filename>, because the default value of
1156 <filename>RDEPENDS_${PN}-dev</filename> (set in
1157 <filename>bitbake.conf</filename>) includes
1158 "${PN}".
1159 </note></para>
1160
1161 <para>To ensure that the dependency chain is never broken,
1162 <filename>-dev</filename> and <filename>-dbg</filename>
1163 packages are always generated by default, even if the packages
1164 turn out to be empty.
1165 See the
1166 <link linkend='var-ALLOW_EMPTY'><filename>ALLOW_EMPTY</filename></link>
1167 variable for more information.
1168 </para></listitem>
1169 </itemizedlist>
1170 </para>
1171
1172 <para>
1173 The <filename>do_package</filename> task depends on the
1174 <link linkend='ref-tasks-packagedata'><filename>do_packagedata</filename></link>
1175 task of each recipe in
1176 <link linkend='var-DEPENDS'><filename>DEPENDS</filename></link>
1177 through use of a
1178 <filename>[</filename><ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'><filename>deptask</filename></ulink><filename>]</filename>
1179 declaration, which guarantees that the required
1180 shared-library/module-to-package mapping information will be available
1181 when needed as long as <filename>DEPENDS</filename> has been
1182 correctly set.
1183 </para>
1184</section>
1185
1186<section id='fakeroot-and-pseudo'>
1187 <title>Fakeroot and Pseudo</title>
1188
1189 <para>
1190 Some tasks are easier to implement when allowed to perform certain
1191 operations that are normally reserved for the root user.
1192 For example, the
1193 <link linkend='ref-tasks-install'><filename>do_install</filename></link>
1194 task benefits from being able to set the UID and GID of installed files
1195 to arbitrary values.
1196 </para>
1197
1198 <para>
1199 One approach to allowing tasks to perform root-only operations
1200 would be to require BitBake to run as root.
1201 However, this method is cumbersome and has security issues.
1202 The approach that is actually used is to run tasks that benefit from
1203 root privileges in a "fake" root environment.
1204 Within this environment, the task and its child processes believe that
1205 they are running as the root user, and see an internally consistent
1206 view of the filesystem.
1207 As long as generating the final output (e.g. a package or an image)
1208 does not require root privileges, the fact that some earlier steps ran
1209 in a fake root environment does not cause problems.
1210 </para>
1211
1212 <para>
1213 The capability to run tasks in a fake root environment is known as
1214 "fakeroot", which is derived from the BitBake keyword/variable
1215 flag that requests a fake root environment for a task.
1216 In current versions of the OpenEmbedded build system,
1217 the program that implements fakeroot is known as Pseudo.
1218 </para>
1219
1220 <para>
1221 Pseudo overrides system calls through the
1222 <filename>LD_PRELOAD</filename> mechanism to give the
1223 illusion of running as root.
1224 To keep track of "fake" file ownership and permissions resulting from
1225 operations that require root permissions, an sqlite3
1226 database is used.
1227 This database is stored in
1228 <filename>${</filename><link linkend='var-WORKDIR'><filename>WORKDIR</filename></link><filename>}/pseudo/files.db</filename>
1229 for individual recipes.
1230 Storing the database in a file as opposed to in memory
1231 gives persistence between tasks, and even between builds.
1232 <note><title>Caution</title>
1233 If you add your own task that manipulates the same files or
1234 directories as a fakeroot task, then that task should also run
1235 under fakeroot.
1236 Otherwise, the task will not be able to run root-only operations,
1237 and will not see the fake file ownership and permissions set by the
1238 other task.
1239 You should also add a dependency on
1240 <filename>virtual/fakeroot-native:do_populate_sysroot</filename>,
1241 giving the following:
1242 <literallayout class='monospaced'>
1243 fakeroot do_mytask () {
1244 ...
1245 }
1246 do_mytask[depends] += "virtual/fakeroot-native:do_populate_sysroot"
1247 </literallayout>
1248 </note>
1249 For more information, see the
1250 <ulink url='&YOCTO_DOCS_BB_URL;#var-FAKEROOT'><filename>FAKEROOT*</filename></ulink>
1251 variables in the BitBake User Manual.
1252 You can also reference this
1253 <ulink url='http://www.ibm.com/developerworks/opensource/library/os-aapseudo1/index.html'>Pseudo</ulink>
1254 article.
1255 </para>
1256</section>
1257
Brad Bishopd7bf8c12018-02-25 22:55:05 -05001258<section id='wic-plug-ins-interface'>
1259 <title>Wic Plug-Ins Interface</title>
1260
1261 <para>
1262 You can extend and specialize Wic functionality by using
1263 Wic plug-ins.
1264 This section explains the Wic plug-in interface.
1265 For information on using Wic in general, see the
1266 "<ulink url='&YOCTO_DOCS_DEV_URL;#creating-partitioned-images-using-wic'>Creating Partitioned Images Using Wic</ulink>"
1267 section in the Yocto Project Development Tasks Manual.
1268 <note>
1269 Wic plug-ins consist of "source" and "imager" plug-ins.
1270 Imager plug-ins are beyond the scope of this section.
1271 </note>
1272 </para>
1273
1274 <para>
1275 Source plug-ins provide a mechanism to customize partition
1276 content during the Wic image generation process.
1277 You can use source plug-ins to map values that you specify
1278 using <filename>--source</filename> commands in kickstart
1279 files (i.e. <filename>*.wks</filename>) to a plug-in
1280 implementation used to populate a given partition.
1281 <note>
1282 If you use plug-ins that have build-time dependencies
1283 (e.g. native tools, bootloaders, and so forth)
1284 when building a Wic image, you need to specify those
1285 dependencies using the
1286 <link linkend='var-WKS_FILE_DEPENDS'><filename>WKS_FILE_DEPENDS</filename></link>
1287 variable.
1288 </note>
1289 </para>
1290
1291 <para>
1292 Source plug-ins are subclasses defined in plug-in files.
1293 As shipped, the Yocto Project provides several plug-in
1294 files.
1295 You can see the source plug-in files that ship with the
1296 Yocto Project
1297 <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/scripts/lib/wic/plugins/source'>here</ulink>.
1298 Each of these plug-in files contain source plug-ins that
1299 are designed to populate a specific Wic image partition.
1300 </para>
1301
1302 <para>
1303 Source plug-ins are subclasses of the
1304 <filename>SourcePlugin</filename> class, which is
1305 defined in the
1306 <filename>poky/scripts/lib/wic/pluginbase.py</filename>
1307 file.
1308 For example, the <filename>BootimgEFIPlugin</filename>
1309 source plug-in found in the
1310 <filename>bootimg-efi.py</filename> file is a subclass of
1311 the <filename>SourcePlugin</filename> class, which is found
1312 in the <filename>pluginbase.py</filename> file.
1313 </para>
1314
1315 <para>
1316 You can also implement source plug-ins in a layer outside
1317 of the Source Repositories (external layer).
1318 To do so, be sure that your plug-in files are located in
1319 a directory whose path is
1320 <filename>scripts/lib/wic/plugins/source/</filename>
1321 within your external layer.
1322 When the plug-in files are located there, the source
1323 plug-ins they contain are made available to Wic.
1324 </para>
1325
1326 <para>
1327 When the Wic implementation needs to invoke a
1328 partition-specific implementation, it looks for the plug-in
1329 with the same name as the <filename>--source</filename>
1330 parameter used in the kickstart file given to that
1331 partition.
1332 For example, if the partition is set up using the following
1333 command in a kickstart file:
1334 <literallayout class='monospaced'>
1335 part /boot --source bootimg-pcbios --ondisk sda --label boot --active --align 1024
1336 </literallayout>
1337 The methods defined as class members of the matching
1338 source plug-in (i.e. <filename>bootimg-pcbios</filename>)
1339 in the <filename>bootimg-pcbios.py</filename> plug-in file
1340 are used.
1341 </para>
1342
1343 <para>
1344 To be more concrete, here is the corresponding plug-in
1345 definition from the <filename>bootimg-pcbios.py</filename>
1346 file for the previous command along with an example
1347 method called by the Wic implementation when it needs to
1348 prepare a partition using an implementation-specific
1349 function:
1350 <literallayout class='monospaced'>
1351 bootimg-pcbios.py
1352 .
1353 .
1354 .
1355 class BootimgPcbiosPlugin(SourcePlugin):
1356 """
1357 Create MBR boot partition and install syslinux on it.
1358 """
1359
1360 name = 'bootimg-pcbios'
1361 .
1362 .
1363 .
1364 @classmethod
1365 def do_prepare_partition(cls, part, source_params, creator, cr_workdir,
1366 oe_builddir, bootimg_dir, kernel_dir,
1367 rootfs_dir, native_sysroot):
1368 """
1369 Called to do the actual content population for a partition i.e. it
1370 'prepares' the partition to be incorporated into the image.
1371 In this case, prepare content for legacy bios boot partition.
1372 """
1373 .
1374 .
1375 .
1376 </literallayout>
1377 If a subclass (plug-in) itself does not implement a
1378 particular function, Wic locates and uses the default
1379 version in the superclass.
1380 It is for this reason that all source plug-ins are derived
1381 from the <filename>SourcePlugin</filename> class.
1382 </para>
1383
1384 <para>
1385 The <filename>SourcePlugin</filename> class defined in
1386 the <filename>pluginbase.py</filename> file defines
1387 a set of methods that source plug-ins can implement or
1388 override.
1389 Any plug-ins (subclass of
1390 <filename>SourcePlugin</filename>) that do not implement
1391 a particular method inherit the implementation of the
1392 method from the <filename>SourcePlugin</filename> class.
1393 For more information, see the
1394 <filename>SourcePlugin</filename> class in the
1395 <filename>pluginbase.py</filename> file for details:
1396 </para>
1397
1398 <para>
1399 The following list describes the methods implemented in the
1400 <filename>SourcePlugin</filename> class:
1401 <itemizedlist>
1402 <listitem><para>
1403 <emphasis><filename>do_prepare_partition()</filename>:</emphasis>
1404 Called to populate a partition with actual content.
1405 In other words, the method prepares the final
1406 partition image that is incorporated into the
1407 disk image.
1408 </para></listitem>
1409 <listitem><para>
1410 <emphasis><filename>do_configure_partition()</filename>:</emphasis>
1411 Called before
1412 <filename>do_prepare_partition()</filename> to
1413 create custom configuration files for a partition
1414 (e.g. syslinux or grub configuration files).
1415 </para></listitem>
1416 <listitem><para>
1417 <emphasis><filename>do_install_disk()</filename>:</emphasis>
1418 Called after all partitions have been prepared and
1419 assembled into a disk image.
1420 This method provides a hook to allow finalization
1421 of a disk image (e.g. writing an MBR).
1422 </para></listitem>
1423 <listitem><para>
1424 <emphasis><filename>do_stage_partition()</filename>:</emphasis>
1425 Special content-staging hook called before
1426 <filename>do_prepare_partition()</filename>.
1427 This method is normally empty.</para>
1428
1429 <para>Typically, a partition just uses the passed-in
1430 parameters (e.g. the unmodified value of
1431 <filename>bootimg_dir</filename>).
1432 However, in some cases, things might need to be
1433 more tailored.
1434 As an example, certain files might additionally
1435 need to be taken from
1436 <filename>bootimg_dir + /boot</filename>.
1437 This hook allows those files to be staged in a
1438 customized fashion.
1439 <note>
1440 <filename>get_bitbake_var()</filename>
1441 allows you to access non-standard variables
1442 that you might want to use for this
1443 behavior.
1444 </note>
1445 </para></listitem>
1446 </itemizedlist>
1447 </para>
1448
1449 <para>
1450 You can extend the source plug-in mechanism.
1451 To add more hooks, create more source plug-in methods
1452 within <filename>SourcePlugin</filename> and the
1453 corresponding derived subclasses.
1454 The code that calls the plug-in methods uses the
1455 <filename>plugin.get_source_plugin_methods()</filename>
1456 function to find the method or methods needed by the call.
1457 Retrieval of those methods is accomplished by filling up
1458 a dict with keys that contain the method names of interest.
1459 On success, these will be filled in with the actual
1460 methods.
1461 See the Wic implementation for examples and details.
1462 </para>
1463</section>
1464
Patrick Williamsc124f4f2015-09-15 14:41:29 -05001465<section id='x32'>
1466 <title>x32</title>
1467
1468 <para>
1469 x32 is a processor-specific Application Binary Interface (psABI) for x86_64.
1470 An ABI defines the calling conventions between functions in a processing environment.
1471 The interface determines what registers are used and what the sizes are for various C data types.
1472 </para>
1473
1474 <para>
1475 Some processing environments prefer using 32-bit applications even when running
1476 on Intel 64-bit platforms.
1477 Consider the i386 psABI, which is a very old 32-bit ABI for Intel 64-bit platforms.
1478 The i386 psABI does not provide efficient use and access of the Intel 64-bit processor resources,
1479 leaving the system underutilized.
1480 Now consider the x86_64 psABI.
1481 This ABI is newer and uses 64-bits for data sizes and program pointers.
1482 The extra bits increase the footprint size of the programs, libraries,
1483 and also increases the memory and file system size requirements.
1484 Executing under the x32 psABI enables user programs to utilize CPU and system resources
1485 more efficiently while keeping the memory footprint of the applications low.
1486 Extra bits are used for registers but not for addressing mechanisms.
1487 </para>
1488
1489 <section id='support'>
1490 <title>Support</title>
1491
1492 <para>
1493 This Yocto Project release supports the final specifications of x32
1494 psABI.
1495 Support for x32 psABI exists as follows:
1496 <itemizedlist>
1497 <listitem><para>You can create packages and images in x32 psABI format on x86_64 architecture targets.
1498 </para></listitem>
1499 <listitem><para>You can successfully build many recipes with the x32 toolchain.</para></listitem>
1500 <listitem><para>You can create and boot <filename>core-image-minimal</filename> and
1501 <filename>core-image-sato</filename> images.</para></listitem>
1502 </itemizedlist>
1503 </para>
1504 </section>
1505
1506 <section id='completing-x32'>
1507 <title>Completing x32</title>
1508
1509 <para>
1510 Future Plans for the x32 psABI in the Yocto Project include the following:
1511 <itemizedlist>
1512 <listitem><para>Enhance and fix the few remaining recipes so they
1513 work with and support x32 toolchains.</para></listitem>
1514 <listitem><para>Enhance RPM Package Manager (RPM) support for x32 binaries.</para></listitem>
1515 <listitem><para>Support larger images.</para></listitem>
1516 </itemizedlist>
1517 </para>
1518 </section>
1519
1520 <section id='using-x32-right-now'>
1521 <title>Using x32 Right Now</title>
1522
1523 <para>
1524 Follow these steps to use the x32 spABI:
1525 <itemizedlist>
1526 <listitem><para>Enable the x32 psABI tuning file for <filename>x86_64</filename>
1527 machines by editing the <filename>conf/local.conf</filename> like this:
1528 <literallayout class='monospaced'>
1529 MACHINE = "qemux86-64"
1530 DEFAULTTUNE = "x86-64-x32"
1531 baselib = "${@d.getVar('BASE_LIB_tune-' + (d.getVar('DEFAULTTUNE', True) \
1532 or 'INVALID'), True) or 'lib'}"
1533 #MACHINE = "genericx86"
1534 #DEFAULTTUNE = "core2-64-x32"
1535 </literallayout></para></listitem>
1536 <listitem><para>As usual, use BitBake to build an image that supports the x32 psABI.
1537 Here is an example:
1538 <literallayout class='monospaced'>
1539 $ bitbake core-image-sato
1540 </literallayout></para></listitem>
1541 <listitem><para>As usual, run your image using QEMU:
1542 <literallayout class='monospaced'>
1543 $ runqemu qemux86-64 core-image-sato
1544 </literallayout></para></listitem>
1545 </itemizedlist>
1546 </para>
1547 </section>
1548</section>
1549
1550<section id="wayland">
1551 <title>Wayland</title>
1552
1553 <para>
1554 <ulink url='http://en.wikipedia.org/wiki/Wayland_(display_server_protocol)'>Wayland</ulink>
1555 is a computer display server protocol that
1556 provides a method for compositing window managers to communicate
1557 directly with applications and video hardware and expects them to
1558 communicate with input hardware using other libraries.
1559 Using Wayland with supporting targets can result in better control
1560 over graphics frame rendering than an application might otherwise
1561 achieve.
1562 </para>
1563
1564 <para>
1565 The Yocto Project provides the Wayland protocol libraries and the
1566 reference
1567 <ulink url='http://en.wikipedia.org/wiki/Wayland_(display_server_protocol)#Weston'>Weston</ulink>
1568 compositor as part of its release.
1569 This section describes what you need to do to implement Wayland and
1570 use the compositor when building an image for a supporting target.
1571 </para>
1572
1573 <section id="wayland-support">
1574 <title>Support</title>
1575
1576 <para>
1577 The Wayland protocol libraries and the reference Weston compositor
1578 ship as integrated packages in the <filename>meta</filename> layer
1579 of the
Brad Bishopd7bf8c12018-02-25 22:55:05 -05001580 <link linkend='source-directory'>Source Directory</link>.
Patrick Williamsc124f4f2015-09-15 14:41:29 -05001581 Specifically, you can find the recipes that build both Wayland
1582 and Weston at <filename>meta/recipes-graphics/wayland</filename>.
1583 </para>
1584
1585 <para>
1586 You can build both the Wayland and Weston packages for use only
1587 with targets that accept the
1588 <ulink url='http://dri.freedesktop.org/wiki/'>Mesa 3D and Direct Rendering Infrastructure</ulink>,
1589 which is also known as Mesa DRI.
1590 This implies that you cannot build and use the packages if your
1591 target uses, for example, the
1592 <trademark class='registered'>Intel</trademark> Embedded Media and
1593 Graphics Driver (<trademark class='registered'>Intel</trademark>
1594 EMGD) that overrides Mesa DRI.
1595 </para>
1596
1597 <note>
1598 Due to lack of EGL support, Weston 1.0.3 will not run directly on
1599 the emulated QEMU hardware.
1600 However, this version of Weston will run under X emulation without
1601 issues.
1602 </note>
1603 </section>
1604
1605 <section id="enabling-wayland-in-an-image">
1606 <title>Enabling Wayland in an Image</title>
1607
1608 <para>
1609 To enable Wayland, you need to enable it to be built and enable
1610 it to be included in the image.
1611 </para>
1612
1613 <section id="enable-building">
1614 <title>Building</title>
1615
1616 <para>
1617 To cause Mesa to build the <filename>wayland-egl</filename>
1618 platform and Weston to build Wayland with Kernel Mode
1619 Setting
1620 (<ulink url='https://wiki.archlinux.org/index.php/Kernel_Mode_Setting'>KMS</ulink>)
1621 support, include the "wayland" flag in the
1622 <link linkend="var-DISTRO_FEATURES"><filename>DISTRO_FEATURES</filename></link>
1623 statement in your <filename>local.conf</filename> file:
1624 <literallayout class='monospaced'>
1625 DISTRO_FEATURES_append = " wayland"
1626 </literallayout>
1627 </para>
1628
1629 <note>
1630 If X11 has been enabled elsewhere, Weston will build Wayland
1631 with X11 support
1632 </note>
1633 </section>
1634
1635 <section id="enable-installation-in-an-image">
1636 <title>Installing</title>
1637
1638 <para>
1639 To install the Wayland feature into an image, you must
1640 include the following
1641 <link linkend='var-CORE_IMAGE_EXTRA_INSTALL'><filename>CORE_IMAGE_EXTRA_INSTALL</filename></link>
1642 statement in your <filename>local.conf</filename> file:
1643 <literallayout class='monospaced'>
1644 CORE_IMAGE_EXTRA_INSTALL += "wayland weston"
1645 </literallayout>
1646 </para>
1647 </section>
1648 </section>
1649
1650 <section id="running-weston">
1651 <title>Running Weston</title>
1652
1653 <para>
1654 To run Weston inside X11, enabling it as described earlier and
1655 building a Sato image is sufficient.
1656 If you are running your image under Sato, a Weston Launcher appears
1657 in the "Utility" category.
1658 </para>
1659
1660 <para>
1661 Alternatively, you can run Weston through the command-line
1662 interpretor (CLI), which is better suited for development work.
1663 To run Weston under the CLI, you need to do the following after
1664 your image is built:
1665 <orderedlist>
1666 <listitem><para>Run these commands to export
1667 <filename>XDG_RUNTIME_DIR</filename>:
1668 <literallayout class='monospaced'>
1669 mkdir -p /tmp/$USER-weston
1670 chmod 0700 /tmp/$USER-weston
1671 export XDG_RUNTIME_DIR=/tmp/$USER-weston
1672 </literallayout></para></listitem>
1673 <listitem><para>Launch Weston in the shell:
1674 <literallayout class='monospaced'>
1675 weston
1676 </literallayout></para></listitem>
1677 </orderedlist>
1678 </para>
1679 </section>
1680</section>
1681
1682<section id="licenses">
1683 <title>Licenses</title>
1684
1685 <para>
1686 This section describes the mechanism by which the OpenEmbedded build system
1687 tracks changes to licensing text.
1688 The section also describes how to enable commercially licensed recipes,
1689 which by default are disabled.
1690 </para>
1691
1692 <para>
1693 For information that can help you maintain compliance with various open
1694 source licensing during the lifecycle of the product, see the
Brad Bishopd7bf8c12018-02-25 22:55:05 -05001695 "<ulink url='&YOCTO_DOCS_DEV_URL;#maintaining-open-source-license-compliance-during-your-products-lifecycle'>Maintaining Open Source License Compliance During Your Project's Lifecycle</ulink>"
1696 section in the Yocto Project Development Tasks Manual.
Patrick Williamsc124f4f2015-09-15 14:41:29 -05001697 </para>
1698
1699 <section id="usingpoky-configuring-LIC_FILES_CHKSUM">
1700 <title>Tracking License Changes</title>
1701
1702 <para>
1703 The license of an upstream project might change in the future.
1704 In order to prevent these changes going unnoticed, the
1705 <filename><link linkend='var-LIC_FILES_CHKSUM'>LIC_FILES_CHKSUM</link></filename>
1706 variable tracks changes to the license text. The checksums are validated at the end of the
1707 configure step, and if the checksums do not match, the build will fail.
1708 </para>
1709
1710 <section id="usingpoky-specifying-LIC_FILES_CHKSUM">
1711 <title>Specifying the <filename>LIC_FILES_CHKSUM</filename> Variable</title>
1712
1713 <para>
1714 The <filename>LIC_FILES_CHKSUM</filename>
Brad Bishop6e60e8b2018-02-01 10:27:11 -05001715 variable contains checksums of the license text in the source
1716 code for the recipe.
1717 Following is an example of how to specify
1718 <filename>LIC_FILES_CHKSUM</filename>:
Patrick Williamsc124f4f2015-09-15 14:41:29 -05001719 <literallayout class='monospaced'>
1720 LIC_FILES_CHKSUM = "file://COPYING;md5=xxxx \
1721 file://licfile1.txt;beginline=5;endline=29;md5=yyyy \
1722 file://licfile2.txt;endline=50;md5=zzzz \
1723 ..."
1724 </literallayout>
Brad Bishop6e60e8b2018-02-01 10:27:11 -05001725 <note><title>Notes</title>
1726 <itemizedlist>
1727 <listitem><para>
1728 When using "beginline" and "endline", realize that
1729 line numbering begins with one and not zero.
1730 Also, the included lines are inclusive (i.e. lines
1731 five through and including 29 in the previous
1732 example for <filename>licfile1.txt</filename>).
1733 </para></listitem>
1734 <listitem><para>
1735 When a license check fails, the selected license
1736 text is included as part of the QA message.
1737 Using this output, you can determine the exact
1738 start and finish for the needed license text.
1739 </para></listitem>
1740 </itemizedlist>
1741 </note>
Patrick Williamsc124f4f2015-09-15 14:41:29 -05001742 </para>
1743
1744 <para>
1745 The build system uses the
1746 <filename><link linkend='var-S'>S</link></filename> variable as
1747 the default directory when searching files listed in
1748 <filename>LIC_FILES_CHKSUM</filename>.
1749 The previous example employs the default directory.
1750 </para>
1751
1752 <para>
1753 Consider this next example:
1754 <literallayout class='monospaced'>
1755 LIC_FILES_CHKSUM = "file://src/ls.c;beginline=5;endline=16;\
1756 md5=bb14ed3c4cda583abc85401304b5cd4e"
1757 LIC_FILES_CHKSUM = "file://${WORKDIR}/license.html;md5=5c94767cedb5d6987c902ac850ded2c6"
1758 </literallayout>
1759 </para>
1760
1761 <para>
1762 The first line locates a file in
Brad Bishop6e60e8b2018-02-01 10:27:11 -05001763 <filename>${S}/src/ls.c</filename> and isolates lines five
1764 through 16 as license text.
Patrick Williamsc124f4f2015-09-15 14:41:29 -05001765 The second line refers to a file in
1766 <filename><link linkend='var-WORKDIR'>WORKDIR</link></filename>.
1767 </para>
1768 <para>
1769 Note that <filename>LIC_FILES_CHKSUM</filename> variable is
1770 mandatory for all recipes, unless the
1771 <filename>LICENSE</filename> variable is set to "CLOSED".
1772 </para>
1773 </section>
1774
1775 <section id="usingpoky-LIC_FILES_CHKSUM-explanation-of-syntax">
1776 <title>Explanation of Syntax</title>
1777 <para>
1778 As mentioned in the previous section, the
1779 <filename>LIC_FILES_CHKSUM</filename> variable lists all the
1780 important files that contain the license text for the source code.
1781 It is possible to specify a checksum for an entire file, or a specific section of a
1782 file (specified by beginning and ending line numbers with the "beginline" and "endline"
1783 parameters, respectively).
1784 The latter is useful for source files with a license notice header,
1785 README documents, and so forth.
1786 If you do not use the "beginline" parameter, then it is assumed that the text begins on the
1787 first line of the file.
1788 Similarly, if you do not use the "endline" parameter, it is assumed that the license text
1789 ends with the last line of the file.
1790 </para>
1791
1792 <para>
1793 The "md5" parameter stores the md5 checksum of the license text.
1794 If the license text changes in any way as compared to this parameter
1795 then a mismatch occurs.
1796 This mismatch triggers a build failure and notifies the developer.
1797 Notification allows the developer to review and address the license text changes.
1798 Also note that if a mismatch occurs during the build, the correct md5
1799 checksum is placed in the build log and can be easily copied to the recipe.
1800 </para>
1801
1802 <para>
1803 There is no limit to how many files you can specify using the
1804 <filename>LIC_FILES_CHKSUM</filename> variable.
1805 Generally, however, every project requires a few specifications for license tracking.
1806 Many projects have a "COPYING" file that stores the license information for all the source
1807 code files.
1808 This practice allows you to just track the "COPYING" file as long as it is kept up to date.
1809 </para>
1810
1811 <tip>
1812 If you specify an empty or invalid "md5" parameter, BitBake returns an md5 mis-match
1813 error and displays the correct "md5" parameter value during the build.
1814 The correct parameter is also captured in the build log.
1815 </tip>
1816
1817 <tip>
1818 If the whole file contains only license text, you do not need to use the "beginline" and
1819 "endline" parameters.
1820 </tip>
1821 </section>
1822 </section>
1823
1824 <section id="enabling-commercially-licensed-recipes">
1825 <title>Enabling Commercially Licensed Recipes</title>
1826
1827 <para>
1828 By default, the OpenEmbedded build system disables
1829 components that have commercial or other special licensing
1830 requirements.
1831 Such requirements are defined on a
1832 recipe-by-recipe basis through the
1833 <link linkend='var-LICENSE_FLAGS'><filename>LICENSE_FLAGS</filename></link>
1834 variable definition in the affected recipe.
1835 For instance, the
1836 <filename>poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly</filename>
1837 recipe contains the following statement:
1838 <literallayout class='monospaced'>
1839 LICENSE_FLAGS = "commercial"
1840 </literallayout>
1841 Here is a slightly more complicated example that contains both an
1842 explicit recipe name and version (after variable expansion):
1843 <literallayout class='monospaced'>
1844 LICENSE_FLAGS = "license_${PN}_${PV}"
1845 </literallayout>
1846 In order for a component restricted by a <filename>LICENSE_FLAGS</filename>
1847 definition to be enabled and included in an image, it
1848 needs to have a matching entry in the global
1849 <link linkend='var-LICENSE_FLAGS_WHITELIST'><filename>LICENSE_FLAGS_WHITELIST</filename></link>
1850 variable, which is a variable
1851 typically defined in your <filename>local.conf</filename> file.
1852 For example, to enable
1853 the <filename>poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly</filename>
1854 package, you could add either the string
1855 "commercial_gst-plugins-ugly" or the more general string
1856 "commercial" to <filename>LICENSE_FLAGS_WHITELIST</filename>.
1857 See the
1858 "<link linkend='license-flag-matching'>License Flag Matching</link>" section
1859 for a full explanation of how <filename>LICENSE_FLAGS</filename> matching works.
1860 Here is the example:
1861 <literallayout class='monospaced'>
1862 LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly"
1863 </literallayout>
1864 Likewise, to additionally enable the package built from the recipe containing
1865 <filename>LICENSE_FLAGS = "license_${PN}_${PV}"</filename>, and assuming
1866 that the actual recipe name was <filename>emgd_1.10.bb</filename>,
1867 the following string would enable that package as well as
1868 the original <filename>gst-plugins-ugly</filename> package:
1869 <literallayout class='monospaced'>
1870 LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly license_emgd_1.10"
1871 </literallayout>
1872 As a convenience, you do not need to specify the complete license string
1873 in the whitelist for every package.
1874 You can use an abbreviated form, which consists
1875 of just the first portion or portions of the license string before
1876 the initial underscore character or characters.
1877 A partial string will match
1878 any license that contains the given string as the first
1879 portion of its license.
1880 For example, the following
1881 whitelist string will also match both of the packages
1882 previously mentioned as well as any other packages that have
1883 licenses starting with "commercial" or "license".
1884 <literallayout class='monospaced'>
1885 LICENSE_FLAGS_WHITELIST = "commercial license"
1886 </literallayout>
1887 </para>
1888
1889 <section id="license-flag-matching">
1890 <title>License Flag Matching</title>
1891
1892 <para>
1893 License flag matching allows you to control what recipes the
1894 OpenEmbedded build system includes in the build.
1895 Fundamentally, the build system attempts to match
1896 <link linkend='var-LICENSE_FLAGS'><filename>LICENSE_FLAGS</filename></link>
1897 strings found in recipes against
1898 <link linkend='var-LICENSE_FLAGS_WHITELIST'><filename>LICENSE_FLAGS_WHITELIST</filename></link>
1899 strings found in the whitelist.
1900 A match causes the build system to include a recipe in the
1901 build, while failure to find a match causes the build system to
1902 exclude a recipe.
1903 </para>
1904
1905 <para>
1906 In general, license flag matching is simple.
1907 However, understanding some concepts will help you
1908 correctly and effectively use matching.
1909 </para>
1910
1911 <para>
1912 Before a flag
1913 defined by a particular recipe is tested against the
1914 contents of the whitelist, the expanded string
1915 <filename>_${PN}</filename> is appended to the flag.
1916 This expansion makes each <filename>LICENSE_FLAGS</filename>
1917 value recipe-specific.
1918 After expansion, the string is then matched against the
1919 whitelist.
1920 Thus, specifying
1921 <filename>LICENSE_FLAGS = "commercial"</filename>
1922 in recipe "foo", for example, results in the string
1923 <filename>"commercial_foo"</filename>.
1924 And, to create a match, that string must appear in the
1925 whitelist.
1926 </para>
1927
1928 <para>
1929 Judicious use of the <filename>LICENSE_FLAGS</filename>
1930 strings and the contents of the
1931 <filename>LICENSE_FLAGS_WHITELIST</filename> variable
1932 allows you a lot of flexibility for including or excluding
1933 recipes based on licensing.
1934 For example, you can broaden the matching capabilities by
1935 using license flags string subsets in the whitelist.
1936 <note>When using a string subset, be sure to use the part of
1937 the expanded string that precedes the appended underscore
1938 character (e.g. <filename>usethispart_1.3</filename>,
1939 <filename>usethispart_1.4</filename>, and so forth).
1940 </note>
1941 For example, simply specifying the string "commercial" in
1942 the whitelist matches any expanded
1943 <filename>LICENSE_FLAGS</filename> definition that starts with
1944 the string "commercial" such as "commercial_foo" and
1945 "commercial_bar", which are the strings the build system
1946 automatically generates for hypothetical recipes named
1947 "foo" and "bar" assuming those recipes simply specify the
1948 following:
1949 <literallayout class='monospaced'>
1950 LICENSE_FLAGS = "commercial"
1951 </literallayout>
1952 Thus, you can choose to exhaustively
1953 enumerate each license flag in the whitelist and
1954 allow only specific recipes into the image, or
1955 you can use a string subset that causes a broader range of
1956 matches to allow a range of recipes into the image.
1957 </para>
1958
1959 <para>
1960 This scheme works even if the
1961 <filename>LICENSE_FLAGS</filename> string already
1962 has <filename>_${PN}</filename> appended.
1963 For example, the build system turns the license flag
1964 "commercial_1.2_foo" into "commercial_1.2_foo_foo" and would
1965 match both the general "commercial" and the specific
1966 "commercial_1.2_foo" strings found in the whitelist, as
1967 expected.
1968 </para>
1969
1970 <para>
1971 Here are some other scenarios:
1972 <itemizedlist>
1973 <listitem><para>You can specify a versioned string in the
1974 recipe such as "commercial_foo_1.2" in a "foo" recipe.
1975 The build system expands this string to
1976 "commercial_foo_1.2_foo".
1977 Combine this license flag with a whitelist that has
1978 the string "commercial" and you match the flag along
1979 with any other flag that starts with the string
1980 "commercial".</para></listitem>
1981 <listitem><para>Under the same circumstances, you can
1982 use "commercial_foo" in the whitelist and the
1983 build system not only matches "commercial_foo_1.2" but
1984 also matches any license flag with the string
1985 "commercial_foo", regardless of the version.
1986 </para></listitem>
1987 <listitem><para>You can be very specific and use both the
1988 package and version parts in the whitelist (e.g.
1989 "commercial_foo_1.2") to specifically match a
1990 versioned recipe.</para></listitem>
1991 </itemizedlist>
1992 </para>
1993 </section>
1994
1995 <section id="other-variables-related-to-commercial-licenses">
1996 <title>Other Variables Related to Commercial Licenses</title>
1997
1998 <para>
1999 Other helpful variables related to commercial
2000 license handling exist and are defined in the
2001 <filename>poky/meta/conf/distro/include/default-distrovars.inc</filename> file:
2002 <literallayout class='monospaced'>
2003 COMMERCIAL_AUDIO_PLUGINS ?= ""
2004 COMMERCIAL_VIDEO_PLUGINS ?= ""
Patrick Williamsc124f4f2015-09-15 14:41:29 -05002005 </literallayout>
2006 If you want to enable these components, you can do so by making sure you have
2007 statements similar to the following
2008 in your <filename>local.conf</filename> configuration file:
2009 <literallayout class='monospaced'>
2010 COMMERCIAL_AUDIO_PLUGINS = "gst-plugins-ugly-mad \
2011 gst-plugins-ugly-mpegaudioparse"
2012 COMMERCIAL_VIDEO_PLUGINS = "gst-plugins-ugly-mpeg2dec \
2013 gst-plugins-ugly-mpegstream gst-plugins-bad-mpegvideoparse"
Patrick Williamsc124f4f2015-09-15 14:41:29 -05002014 LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly commercial_gst-plugins-bad commercial_qmmp"
2015 </literallayout>
2016 Of course, you could also create a matching whitelist
2017 for those components using the more general "commercial"
2018 in the whitelist, but that would also enable all the
2019 other packages with
2020 <link linkend='var-LICENSE_FLAGS'><filename>LICENSE_FLAGS</filename></link>
2021 containing "commercial", which you may or may not want:
2022 <literallayout class='monospaced'>
2023 LICENSE_FLAGS_WHITELIST = "commercial"
2024 </literallayout>
2025 </para>
2026
2027 <para>
2028 Specifying audio and video plug-ins as part of the
2029 <filename>COMMERCIAL_AUDIO_PLUGINS</filename> and
2030 <filename>COMMERCIAL_VIDEO_PLUGINS</filename> statements
Patrick Williamsd8c66bc2016-06-20 12:57:21 -05002031 (along with the enabling
2032 <filename>LICENSE_FLAGS_WHITELIST</filename>) includes the
Patrick Williamsc124f4f2015-09-15 14:41:29 -05002033 plug-ins or components into built images, thus adding
2034 support for media formats or components.
2035 </para>
2036 </section>
2037 </section>
2038</section>
2039</chapter>
2040<!--
2041vim: expandtab tw=80 ts=4
2042-->