blob: 338a190ae235d31f392033683341f25565882d14 [file] [log] [blame]
Brad Bishop316dfdd2018-06-25 12:45:53 -04001<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
2"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
3[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
4
5<chapter id=' overview-manual-concepts'>
6<title>Yocto Project Concepts</title>
7
8 <para>
9 This chapter provides explanations for Yocto Project concepts that
10 go beyond the surface of "how-to" information and reference (or
11 look-up) material.
12 Concepts such as components, the
13 <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink>
14 workflow, cross-development toolchains, shared state cache, and so
15 forth are explained.
16 </para>
17
18 <section id='yocto-project-components'>
19 <title>Yocto Project Components</title>
20
21 <para>
22 The
23 <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink>
24 task executor together with various types of configuration files
25 form the
26 <ulink url='&YOCTO_DOCS_REF_URL;#oe-core'>OpenEmbedded-Core</ulink>.
27 This section overviews these components by describing their use and
28 how they interact.
29 </para>
30
31 <para>
32 BitBake handles the parsing and execution of the data files.
33 The data itself is of various types:
34 <itemizedlist>
35 <listitem><para>
36 <emphasis>Recipes:</emphasis>
37 Provides details about particular pieces of software.
38 </para></listitem>
39 <listitem><para>
40 <emphasis>Class Data:</emphasis>
41 Abstracts common build information (e.g. how to build a
42 Linux kernel).
43 </para></listitem>
44 <listitem><para>
45 <emphasis>Configuration Data:</emphasis>
46 Defines machine-specific settings, policy decisions, and
47 so forth.
48 Configuration data acts as the glue to bind everything
49 together.
50 </para></listitem>
51 </itemizedlist>
52 </para>
53
54 <para>
55 BitBake knows how to combine multiple data sources together and
56 refers to each data source as a layer.
57 For information on layers, see the
58 "<ulink url='&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers'>Understanding and Creating Layers</ulink>"
59 section of the Yocto Project Development Tasks Manual.
60 </para>
61
62 <para>
63 Following are some brief details on these core components.
64 For additional information on how these components interact during
65 a build, see the
66 "<link linkend='openembedded-build-system-build-concepts'>OpenEmbedded Build System Concepts</link>"
67 section.
68 </para>
69
70 <section id='usingpoky-components-bitbake'>
71 <title>BitBake</title>
72
73 <para>
74 BitBake is the tool at the heart of the
75 <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink>
76 and is responsible for parsing the
77 <ulink url='&YOCTO_DOCS_REF_URL;#metadata'>Metadata</ulink>,
78 generating a list of tasks from it, and then executing those
79 tasks.
80 </para>
81
82 <para>
83 This section briefly introduces BitBake.
84 If you want more information on BitBake, see the
85 <ulink url='&YOCTO_DOCS_BB_URL;#bitbake-user-manual'>BitBake User Manual</ulink>.
86 </para>
87
88 <para>
89 To see a list of the options BitBake supports, use either of
90 the following commands:
91 <literallayout class='monospaced'>
92 $ bitbake -h
93 $ bitbake --help
94 </literallayout>
95 </para>
96
97 <para>
98 The most common usage for BitBake is
99 <filename>bitbake <replaceable>packagename</replaceable></filename>,
100 where <filename>packagename</filename> is the name of the
101 package you want to build (referred to as the "target").
102 The target often equates to the first part of a recipe's
103 filename (e.g. "foo" for a recipe named
104 <filename>foo_1.3.0-r0.bb</filename>).
105 So, to process the
106 <filename>matchbox-desktop_1.2.3.bb</filename> recipe file, you
107 might type the following:
108 <literallayout class='monospaced'>
109 $ bitbake matchbox-desktop
110 </literallayout>
111 Several different versions of
112 <filename>matchbox-desktop</filename> might exist.
113 BitBake chooses the one selected by the distribution
114 configuration.
115 You can get more details about how BitBake chooses between
116 different target versions and providers in the
117 "<ulink url='&YOCTO_DOCS_BB_URL;#bb-bitbake-preferences'>Preferences</ulink>"
118 section of the BitBake User Manual.
119 </para>
120
121 <para>
122 BitBake also tries to execute any dependent tasks first.
123 So for example, before building
124 <filename>matchbox-desktop</filename>, BitBake would build a
125 cross compiler and <filename>glibc</filename> if they had not
126 already been built.
127 </para>
128
129 <para>
130 A useful BitBake option to consider is the
131 <filename>-k</filename> or <filename>--continue</filename>
132 option.
133 This option instructs BitBake to try and continue processing
134 the job as long as possible even after encountering an error.
135 When an error occurs, the target that failed and those that
136 depend on it cannot be remade.
137 However, when you use this option other dependencies can
138 still be processed.
139 </para>
140 </section>
141
142 <section id='overview-components-recipes'>
143 <title>Recipes</title>
144
145 <para>
146 Files that have the <filename>.bb</filename> suffix are
147 "recipes" files.
148 In general, a recipe contains information about a single piece
149 of software.
150 This information includes the location from which to download
151 the unaltered source, any source patches to be applied to that
152 source (if needed), which special configuration options to
153 apply, how to compile the source files, and how to package the
154 compiled output.
155 </para>
156
157 <para>
158 The term "package" is sometimes used to refer to recipes.
159 However, since the word "package" is used for the packaged
160 output from the OpenEmbedded build system (i.e.
161 <filename>.ipk</filename> or <filename>.deb</filename> files),
162 this document avoids using the term "package" when referring
163 to recipes.
164 </para>
165 </section>
166
167 <section id='overview-components-classes'>
168 <title>Classes</title>
169
170 <para>
171 Class files (<filename>.bbclass</filename>) contain information
172 that is useful to share between recipes files.
173 An example is the
174 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-autotools'><filename>autotools</filename></ulink>
175 class, which contains common settings for any application that
176 Autotools uses.
177 The
178 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-classes'>Classes</ulink>"
179 chapter in the Yocto Project Reference Manual provides
180 details about classes and how to use them.
181 </para>
182 </section>
183
184 <section id='overview-components-configurations'>
185 <title>Configurations</title>
186
187 <para>
188 The configuration files (<filename>.conf</filename>) define
189 various configuration variables that govern the OpenEmbedded
190 build process.
191 These files fall into several areas that define machine
192 configuration options, distribution configuration options,
193 compiler tuning options, general common configuration options,
194 and user configuration options in
195 <filename>conf/local.conf</filename>, which is found in the
196 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>.
197 </para>
198 </section>
199 </section>
200
201 <section id='overview-layers'>
202 <title>Layers</title>
203
204 <para>
205 Layers are repositories that contain related metadata (i.e.
206 sets of instructions) that tell the OpenEmbedded build system how
207 to build a target.
208 Yocto Project's
209 <link linkend='the-yocto-project-layer-model'>layer model</link>
210 facilitates collaboration, sharing, customization, and reuse
211 within the Yocto Project development environment.
212 Layers logically separate information for your project.
213 For example, you can use a layer to hold all the configurations
214 for a particular piece of hardware.
215 Isolating hardware-specific configurations allows you to share
216 other metadata by using a different layer where that metadata
217 might be common across several pieces of hardware.
218 </para>
219
220 <para>
221 Many layers exist that work in the Yocto Project development
222 environment.
223 The
224 <ulink url='https://caffelli-staging.yoctoproject.org/software-overview/layers/'>Yocto Project Curated Layer Index</ulink>
225 and
226 <ulink url='http://layers.openembedded.org/layerindex/branch/master/layers/'>OpenEmbedded Layer Index</ulink>
227 both contain layers from which you can use or leverage.
228 </para>
229
230 <para>
231 By convention, layers in the Yocto Project follow a specific form.
232 Conforming to a known structure allows BitBake to make assumptions
233 during builds on where to find types of metadata.
234 You can find procedures and learn about tools (i.e.
235 <filename>bitbake-layers</filename>) for creating layers suitable
236 for the Yocto Project in the
237 "<ulink url='&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers'>Understanding and Creating Layers</ulink>"
238 section of the Yocto Project Development Tasks Manual.
239 </para>
240 </section>
241
242 <section id="openembedded-build-system-build-concepts">
243 <title>OpenEmbedded Build System Concepts</title>
244
245 <para>
246 This section takes a more detailed look inside the build
247 process used by the
248 <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink>,
249 which is the build system specific to the Yocto Project.
250 At the heart of the build system is BitBake, the task executor.
251 </para>
252
253 <para>
254 The following diagram represents the high-level workflow of a
255 build.
256 The remainder of this section expands on the fundamental input,
257 output, process, and metadata logical blocks that make up the
258 workflow.
259 </para>
260
261 <para id='general-workflow-figure'>
262 <imagedata fileref="figures/YP-flow-diagram.png" format="PNG" align='center' width="8in"/>
263 </para>
264
265 <para>
266 In general, the build's workflow consists of several functional
267 areas:
268 <itemizedlist>
269 <listitem><para>
270 <emphasis>User Configuration:</emphasis>
271 metadata you can use to control the build process.
272 </para></listitem>
273 <listitem><para>
274 <emphasis>Metadata Layers:</emphasis>
275 Various layers that provide software, machine, and
276 distro metadata.
277 </para></listitem>
278 <listitem><para>
279 <emphasis>Source Files:</emphasis>
280 Upstream releases, local projects, and SCMs.
281 </para></listitem>
282 <listitem><para>
283 <emphasis>Build System:</emphasis>
284 Processes under the control of
285 <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink>.
286 This block expands on how BitBake fetches source, applies
287 patches, completes compilation, analyzes output for package
288 generation, creates and tests packages, generates images,
289 and generates cross-development tools.
290 </para></listitem>
291 <listitem><para>
292 <emphasis>Package Feeds:</emphasis>
293 Directories containing output packages (RPM, DEB or IPK),
294 which are subsequently used in the construction of an
295 image or Software Development Kit (SDK), produced by the
296 build system.
297 These feeds can also be copied and shared using a web
298 server or other means to facilitate extending or updating
299 existing images on devices at runtime if runtime package
300 management is enabled.
301 </para></listitem>
302 <listitem><para>
303 <emphasis>Images:</emphasis>
304 Images produced by the workflow.
305 </para></listitem>
306 <listitem><para>
307 <emphasis>Application Development SDK:</emphasis>
308 Cross-development tools that are produced along with
309 an image or separately with BitBake.
310 </para></listitem>
311 </itemizedlist>
312 </para>
313
314 <section id="user-configuration">
315 <title>User Configuration</title>
316
317 <para>
318 User configuration helps define the build.
319 Through user configuration, you can tell BitBake the
320 target architecture for which you are building the image,
321 where to store downloaded source, and other build properties.
322 </para>
323
324 <para>
325 The following figure shows an expanded representation of the
326 "User Configuration" box of the
327 <link linkend='general-workflow-figure'>general workflow figure</link>:
328 </para>
329
330 <para>
331 <imagedata fileref="figures/user-configuration.png" align="center" width="8in" depth="4.5in" />
332 </para>
333
334 <para>
335 BitBake needs some basic configuration files in order to
336 complete a build.
337 These files are <filename>*.conf</filename> files.
338 The minimally necessary ones reside as example files in the
339 <filename>build/conf</filename> directory of the
340 <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>.
341 For simplicity, this section refers to the Source Directory as
342 the "Poky Directory."
343 </para>
344
345 <para>
346 When you clone the
347 <ulink url='&YOCTO_DOCS_REF_URL;#poky'>Poky</ulink>
348 Git repository or you download and unpack a Yocto Project
349 release, you can set up the Source Directory to be named
350 anything you want.
351 For this discussion, the cloned repository uses the default
352 name <filename>poky</filename>.
353 <note>
354 The Poky repository is primarily an aggregation of existing
355 repositories.
356 It is not a canonical upstream source.
357 </note>
358 </para>
359
360 <para>
361 The <filename>meta-poky</filename> layer inside Poky contains
362 a <filename>conf</filename> directory that has example
363 configuration files.
364 These example files are used as a basis for creating actual
365 configuration files when you source
366 <ulink url='&YOCTO_DOCS_REF_URL;#structure-core-script'><filename>&OE_INIT_FILE;</filename></ulink>,
367 which is the build environment script.
368 </para>
369
370 <para>
371 Sourcing the build environment script creates a
372 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>
373 if one does not already exist.
374 BitBake uses the Build Directory for all its work during
375 builds.
376 The Build Directory has a <filename>conf</filename> directory
377 that contains default versions of your
378 <filename>local.conf</filename> and
379 <filename>bblayers.conf</filename> configuration files.
380 These default configuration files are created only if versions
381 do not already exist in the Build Directory at the time you
382 source the build environment setup script.
383 </para>
384
385 <para>
386 Because the Poky repository is fundamentally an aggregation of
387 existing repositories, some users might be familiar with
388 running the <filename>&OE_INIT_FILE;</filename> script
389 in the context of separate
390 <ulink url='&YOCTO_DOCS_REF_URL;#oe-core'>OpenEmbedded-Core</ulink>
391 and BitBake repositories rather than a single Poky repository.
392 This discussion assumes the script is executed from
393 within a cloned or unpacked version of Poky.
394 </para>
395
396 <para>
397 Depending on where the script is sourced, different
398 sub-scripts are called to set up the Build Directory
399 (Yocto or OpenEmbedded).
400 Specifically, the script
401 <filename>scripts/oe-setup-builddir</filename> inside the
402 poky directory sets up the Build Directory and seeds the
403 directory (if necessary) with configuration files appropriate
404 for the Yocto Project development environment.
405 <note>
406 The <filename>scripts/oe-setup-builddir</filename> script
407 uses the <filename>$TEMPLATECONF</filename> variable to
408 determine which sample configuration files to locate.
409 </note>
410 </para>
411
412 <para>
413 The <filename>local.conf</filename> file provides many
414 basic variables that define a build environment.
415 Here is a list of a few.
416 To see the default configurations in a
417 <filename>local.conf</filename> file created by the build
418 environment script, see the
419 <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta-poky/conf/local.conf.sample'><filename>local.conf.sample</filename></ulink>
420 in the <filename>meta-poky</filename> layer:
421 <itemizedlist>
422 <listitem><para>
423 <emphasis>Target Machine Selection:</emphasis>
424 Controlled by the
425 <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink>
426 variable.
427 </para></listitem>
428 <listitem><para>
429 <emphasis>Download Directory:</emphasis>
430 Controlled by the
431 <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink>
432 variable.
433 </para></listitem>
434 <listitem><para>
435 <emphasis>Shared State Directory:</emphasis>
436 Controlled by the
437 <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink>
438 variable.
439 </para></listitem>
440 <listitem><para>
441 <emphasis>Build Output:</emphasis>
442 Controlled by the
443 <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink>
444 variable.
445 </para></listitem>
446 <listitem><para>
447 <emphasis>Distribution Policy:</emphasis>
448 Controlled by the
449 <ulink url='&YOCTO_DOCS_REF_URL;#var-DISTRO'><filename>DISTRO</filename></ulink>
450 variable.
451 </para></listitem>
452 <listitem><para>
453 <emphasis>Packaging Format:</emphasis>
454 Controlled by the
455 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></ulink>
456 variable.
457 </para></listitem>
458 <listitem><para>
459 <emphasis>SDK Target Architecture:</emphasis>
460 Controlled by the
461 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKMACHINE'><filename>SDKMACHINE</filename></ulink>
462 variable.
463 </para></listitem>
464 <listitem><para>
465 <emphasis>Extra Image Packages:</emphasis>
466 Controlled by the
467 <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTRA_IMAGE_FEATURES'><filename>EXTRA_IMAGE_FEATURES</filename></ulink>
468 variable.
469 </para></listitem>
470 </itemizedlist>
471 <note>
472 Configurations set in the
473 <filename>conf/local.conf</filename> file can also be set
474 in the <filename>conf/site.conf</filename> and
475 <filename>conf/auto.conf</filename> configuration files.
476 </note>
477 </para>
478
479 <para>
480 The <filename>bblayers.conf</filename> file tells BitBake what
481 layers you want considered during the build.
482 By default, the layers listed in this file include layers
483 minimally needed by the build system.
484 However, you must manually add any custom layers you have
485 created.
486 You can find more information on working with the
487 <filename>bblayers.conf</filename> file in the
488 "<ulink url='&YOCTO_DOCS_DEV_URL;#enabling-your-layer'>Enabling Your Layer</ulink>"
489 section in the Yocto Project Development Tasks Manual.
490 </para>
491
492 <para>
493 The files <filename>site.conf</filename> and
494 <filename>auto.conf</filename> are not created by the
495 environment initialization script.
496 If you want the <filename>site.conf</filename> file, you
497 need to create that yourself.
498 The <filename>auto.conf</filename> file is typically created by
499 an autobuilder:
500 <itemizedlist>
501 <listitem><para>
502 <emphasis><filename>site.conf</filename>:</emphasis>
503 You can use the <filename>conf/site.conf</filename>
504 configuration file to configure multiple
505 build directories.
506 For example, suppose you had several build environments
507 and they shared some common features.
508 You can set these default build properties here.
509 A good example is perhaps the packaging format to use
510 through the
511 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></ulink>
512 variable.</para>
513
514 <para>One useful scenario for using the
515 <filename>conf/site.conf</filename> file is to extend
516 your
517 <ulink url='&YOCTO_DOCS_REF_URL;#var-BBPATH'><filename>BBPATH</filename></ulink>
518 variable to include the path to a
519 <filename>conf/site.conf</filename>.
520 Then, when BitBake looks for Metadata using
521 <filename>BBPATH</filename>, it finds the
522 <filename>conf/site.conf</filename> file and applies
523 your common configurations found in the file.
524 To override configurations in a particular build
525 directory, alter the similar configurations within
526 that build directory's
527 <filename>conf/local.conf</filename> file.
528 </para></listitem>
529 <listitem><para>
530 <emphasis><filename>auto.conf</filename>:</emphasis>
531 The file is usually created and written to by
532 an autobuilder.
533 The settings put into the file are typically the
534 same as you would find in the
535 <filename>conf/local.conf</filename> or the
536 <filename>conf/site.conf</filename> files.
537 </para></listitem>
538 </itemizedlist>
539 </para>
540
541 <para>
542 You can edit all configuration files to further define
543 any particular build environment.
544 This process is represented by the "User Configuration Edits"
545 box in the figure.
546 </para>
547
548 <para>
549 When you launch your build with the
550 <filename>bitbake <replaceable>target</replaceable></filename>
551 command, BitBake sorts out the configurations to ultimately
552 define your build environment.
553 It is important to understand that the
554 <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink>
555 reads the configuration files in a specific order:
556 <filename>site.conf</filename>, <filename>auto.conf</filename>,
557 and <filename>local.conf</filename>.
558 And, the build system applies the normal assignment statement
559 rules as described in the
560 "<ulink url='&YOCTO_DOCS_BB_URL;#bitbake-user-manual-metadata'>Syntax and Operators</ulink>"
561 chapter of the BitBake User Manual.
562 Because the files are parsed in a specific order, variable
563 assignments for the same variable could be affected.
564 For example, if the <filename>auto.conf</filename> file and
565 the <filename>local.conf</filename> set
566 <replaceable>variable1</replaceable> to different values,
567 because the build system parses <filename>local.conf</filename>
568 after <filename>auto.conf</filename>,
569 <replaceable>variable1</replaceable> is assigned the value from
570 the <filename>local.conf</filename> file.
571 </para>
572 </section>
573
574 <section id="metadata-machine-configuration-and-policy-configuration">
575 <title>Metadata, Machine Configuration, and Policy Configuration</title>
576
577 <para>
578 The previous section described the user configurations that
579 define BitBake's global behavior.
580 This section takes a closer look at the layers the build system
581 uses to further control the build.
582 These layers provide Metadata for the software, machine, and
583 policies.
584 </para>
585
586 <para>
587 In general, three types of layer input exists.
588 You can see them below the "User Configuration" box in the
589 <link linkend='general-workflow-figure'>general workflow figure</link>:
590 <itemizedlist>
591 <listitem><para>
592 <emphasis>Metadata (<filename>.bb</filename> + Patches):</emphasis>
593 Software layers containing user-supplied recipe files,
594 patches, and append files.
595 A good example of a software layer might be the
596 <ulink url='https://github.com/meta-qt5/meta-qt5'><filename>meta-qt5</filename></ulink>
597 layer from the
598 <ulink url='http://layers.openembedded.org/layerindex/branch/master/layers/'>OpenEmbedded Layer Index</ulink>.
599 This layer is for version 5.0 of the popular
600 <ulink url='https://wiki.qt.io/About_Qt'>Qt</ulink>
601 cross-platform application development framework for
602 desktop, embedded and mobile.
603 </para></listitem>
604 <listitem><para>
605 <emphasis>Machine BSP Configuration:</emphasis>
606 Board Support Package (BSP) layers (i.e. "BSP Layer"
607 in the following figure) providing machine-specific
608 configurations.
609 This type of information is specific to a particular
610 target architecture.
611 A good example of a BSP layer from the
612 <link linkend='gs-reference-distribution-poky'>Poky Reference Distribution</link>
613 is the
614 <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta-yocto-bsp'><filename>meta-yocto-bsp</filename></ulink>
615 layer.
616 </para></listitem>
617 <listitem><para>
618 <emphasis>Policy Configuration:</emphasis>
619 Distribution Layers (i.e. "Distro Layer" in the
620 following figure) providing top-level or general
621 policies for the images or SDKs being built for a
622 particular distribution.
623 For example, in the Poky Reference Distribution the
624 distro layer is the
625 <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta-poky'><filename>meta-poky</filename></ulink>
626 layer.
627 Within the distro layer is a
628 <filename>conf/distro</filename> directory that
629 contains distro configuration files (e.g.
630 <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta-poky/conf/distro/poky.conf'><filename>poky.conf</filename></ulink>
631 that contain many policy configurations for the
632 Poky distribution.
633 </para></listitem>
634 </itemizedlist>
635 </para>
636
637 <para>
638 The following figure shows an expanded representation of
639 these three layers from the
640 <link linkend='general-workflow-figure'>general workflow figure</link>:
641 </para>
642
643 <para>
644 <imagedata fileref="figures/layer-input.png" align="center" width="8in" depth="8in" />
645 </para>
646
647 <para>
648 In general, all layers have a similar structure.
649 They all contain a licensing file
650 (e.g. <filename>COPYING.MIT</filename>) if the layer is to be
651 distributed, a <filename>README</filename> file as good
652 practice and especially if the layer is to be distributed, a
653 configuration directory, and recipe directories.
654 You can learn about the general structure for layers used with
655 the Yocto Project in the
656 "<ulink url='&YOCTO_DOCS_DEV_URL;#creating-your-own-layer'>Creating Your Own Layer</ulink>"
657 section in the Yocto Project Development Tasks Manual.
658 For a general discussion on layers and the many layers from
659 which you can draw, see the
660 "<link linkend='overview-layers'>Layers</link>" and
661 "<link linkend='the-yocto-project-layer-model'>The Yocto Project Layer Model</link>"
662 sections both earlier in this manual.
663 </para>
664
665 <para>
666 If you explored the previous links, you discovered some
667 areas where many layers that work with the Yocto Project
668 exist.
669 The
670 <ulink url="http://git.yoctoproject.org/">Source Repositories</ulink>
671 also shows layers categorized under "Yocto Metadata Layers."
672 <note>
673 Layers exist in the Yocto Project Source Repositories that
674 cannot be found in the OpenEmbedded Layer Index.
675 These layers are either deprecated or experimental
676 in nature.
677 </note>
678 </para>
679
680 <para>
681 BitBake uses the <filename>conf/bblayers.conf</filename> file,
682 which is part of the user configuration, to find what layers it
683 should be using as part of the build.
684 </para>
685
686 <section id="distro-layer">
687 <title>Distro Layer</title>
688
689 <para>
690 The distribution layer provides policy configurations for
691 your distribution.
692 Best practices dictate that you isolate these types of
693 configurations into their own layer.
694 Settings you provide in
695 <filename>conf/distro/<replaceable>distro</replaceable>.conf</filename> override
696 similar settings that BitBake finds in your
697 <filename>conf/local.conf</filename> file in the Build
698 Directory.
699 </para>
700
701 <para>
702 The following list provides some explanation and references
703 for what you typically find in the distribution layer:
704 <itemizedlist>
705 <listitem><para>
706 <emphasis>classes:</emphasis>
707 Class files (<filename>.bbclass</filename>) hold
708 common functionality that can be shared among
709 recipes in the distribution.
710 When your recipes inherit a class, they take on the
711 settings and functions for that class.
712 You can read more about class files in the
713 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-classes'>Classes</ulink>"
714 chapter of the Yocto Reference Manual.
715 </para></listitem>
716 <listitem><para>
717 <emphasis>conf:</emphasis>
718 This area holds configuration files for the
719 layer (<filename>conf/layer.conf</filename>),
720 the distribution
721 (<filename>conf/distro/<replaceable>distro</replaceable>.conf</filename>),
722 and any distribution-wide include files.
723 </para></listitem>
724 <listitem><para>
725 <emphasis>recipes-*:</emphasis>
726 Recipes and append files that affect common
727 functionality across the distribution.
728 This area could include recipes and append files
729 to add distribution-specific configuration,
730 initialization scripts, custom image recipes,
731 and so forth.
732 Examples of <filename>recipes-*</filename>
733 directories are <filename>recipes-core</filename>
734 and <filename>recipes-extra</filename>.
735 Hierarchy and contents within a
736 <filename>recipes-*</filename> directory can vary.
737 Generally, these directories contain recipe files
738 (<filename>*.bb</filename>), recipe append files
739 (<filename>*.bbappend</filename>), directories
740 that are distro-specific for configuration files,
741 and so forth.
742 </para></listitem>
743 </itemizedlist>
744 </para>
745 </section>
746
747 <section id="bsp-layer">
748 <title>BSP Layer</title>
749
750 <para>
751 The BSP Layer provides machine configurations that
752 target specific hardware.
753 Everything in this layer is specific to the machine for
754 which you are building the image or the SDK.
755 A common structure or form is defined for BSP layers.
756 You can learn more about this structure in the
757 <ulink url='&YOCTO_DOCS_BSP_URL;'>Yocto Project Board Support Package (BSP) Developer's Guide</ulink>.
758 <note>
759 In order for a BSP layer to be considered compliant
760 with the Yocto Project, it must meet some structural
761 requirements.
762 </note>
763 </para>
764
765 <para>
766 The BSP Layer's configuration directory contains
767 configuration files for the machine
768 (<filename>conf/machine/<replaceable>machine</replaceable>.conf</filename>)
769 and, of course, the layer
770 (<filename>conf/layer.conf</filename>).
771 </para>
772
773 <para>
774 The remainder of the layer is dedicated to specific recipes
775 by function: <filename>recipes-bsp</filename>,
776 <filename>recipes-core</filename>,
777 <filename>recipes-graphics</filename>,
778 <filename>recipes-kernel</filename>, and so forth.
779 Metadata can exist for multiple formfactors, graphics
780 support systems, and so forth.
781 <note>
782 While the figure shows several
783 <filename>recipes-*</filename> directories, not all
784 these directories appear in all BSP layers.
785 </note>
786 </para>
787 </section>
788
789 <section id="software-layer">
790 <title>Software Layer</title>
791
792 <para>
793 The software layer provides the Metadata for additional
794 software packages used during the build.
795 This layer does not include Metadata that is specific to
796 the distribution or the machine, which are found in their
797 respective layers.
798 </para>
799
800 <para>
801 This layer contains any recipes, append files, and
802 patches, that your project needs.
803 </para>
804 </section>
805 </section>
806
807 <section id="sources-dev-environment">
808 <title>Sources</title>
809
810 <para>
811 In order for the OpenEmbedded build system to create an
812 image or any target, it must be able to access source files.
813 The
814 <link linkend='general-workflow-figure'>general workflow figure</link>
815 represents source files using the "Upstream Project Releases",
816 "Local Projects", and "SCMs (optional)" boxes.
817 The figure represents mirrors, which also play a role in
818 locating source files, with the "Source Materials" box.
819 </para>
820
821 <para>
822 The method by which source files are ultimately organized is
823 a function of the project.
824 For example, for released software, projects tend to use
825 tarballs or other archived files that can capture the
826 state of a release guaranteeing that it is statically
827 represented.
828 On the other hand, for a project that is more dynamic or
829 experimental in nature, a project might keep source files in a
830 repository controlled by a Source Control Manager (SCM) such as
831 Git.
832 Pulling source from a repository allows you to control
833 the point in the repository (the revision) from which you
834 want to build software.
835 Finally, a combination of the two might exist, which would
836 give the consumer a choice when deciding where to get
837 source files.
838 </para>
839
840 <para>
841 BitBake uses the
842 <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink>
843 variable to point to source files regardless of their location.
844 Each recipe must have a <filename>SRC_URI</filename> variable
845 that points to the source.
846 </para>
847
848 <para>
849 Another area that plays a significant role in where source
850 files come from is pointed to by the
851 <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink>
852 variable.
853 This area is a cache that can hold previously downloaded
854 source.
855 You can also instruct the OpenEmbedded build system to create
856 tarballs from Git repositories, which is not the default
857 behavior, and store them in the <filename>DL_DIR</filename>
858 by using the
859 <ulink url='&YOCTO_DOCS_REF_URL;#var-BB_GENERATE_MIRROR_TARBALLS'><filename>BB_GENERATE_MIRROR_TARBALLS</filename></ulink>
860 variable.
861 </para>
862
863 <para>
864 Judicious use of a <filename>DL_DIR</filename> directory can
865 save the build system a trip across the Internet when looking
866 for files.
867 A good method for using a download directory is to have
868 <filename>DL_DIR</filename> point to an area outside of your
869 Build Directory.
870 Doing so allows you to safely delete the Build Directory
871 if needed without fear of removing any downloaded source file.
872 </para>
873
874 <para>
875 The remainder of this section provides a deeper look into the
876 source files and the mirrors.
877 Here is a more detailed look at the source file area of the
878 <link linkend='general-workflow-figure'>general workflow figure</link>:
879 </para>
880
881 <para>
882 <imagedata fileref="figures/source-input.png" width="6in" depth="6in" align="center" />
883 </para>
884
885 <section id='upstream-project-releases'>
886 <title>Upstream Project Releases</title>
887
888 <para>
889 Upstream project releases exist anywhere in the form of an
890 archived file (e.g. tarball or zip file).
891 These files correspond to individual recipes.
892 For example, the figure uses specific releases each for
893 BusyBox, Qt, and Dbus.
894 An archive file can be for any released product that can be
895 built using a recipe.
896 </para>
897 </section>
898
899 <section id='local-projects'>
900 <title>Local Projects</title>
901
902 <para>
903 Local projects are custom bits of software the user
904 provides.
905 These bits reside somewhere local to a project - perhaps
906 a directory into which the user checks in items (e.g.
907 a local directory containing a development source tree
908 used by the group).
909 </para>
910
911 <para>
912 The canonical method through which to include a local
913 project is to use the
914 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-externalsrc'><filename>externalsrc</filename></ulink>
915 class to include that local project.
916 You use either the <filename>local.conf</filename> or a
917 recipe's append file to override or set the
918 recipe to point to the local directory on your disk to pull
919 in the whole source tree.
920 </para>
921 </section>
922
923 <section id='scms'>
924 <title>Source Control Managers (Optional)</title>
925
926 <para>
927 Another place the build system can get source files from is
928 through an SCM such as Git or Subversion.
929 In this case, a repository is cloned or checked out.
930 The
931 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-fetch'><filename>do_fetch</filename></ulink>
932 task inside BitBake uses
933 the <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink>
934 variable and the argument's prefix to determine the correct
935 fetcher module.
936 <note>
937 For information on how to have the OpenEmbedded build
938 system generate tarballs for Git repositories and place
939 them in the
940 <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink>
941 directory, see the
942 <ulink url='&YOCTO_DOCS_REF_URL;#var-BB_GENERATE_MIRROR_TARBALLS'><filename>BB_GENERATE_MIRROR_TARBALLS</filename></ulink>
943 variable.
944 </note>
945 </para>
946
947 <para>
948 When fetching a repository, BitBake uses the
949 <ulink url='&YOCTO_DOCS_REF_URL;#var-SRCREV'><filename>SRCREV</filename></ulink>
950 variable to determine the specific revision from which to
951 build.
952 </para>
953 </section>
954
955 <section id='source-mirrors'>
956 <title>Source Mirror(s)</title>
957
958 <para>
959 Two kinds of mirrors exist: pre-mirrors and regular
960 mirrors.
961 The
962 <ulink url='&YOCTO_DOCS_REF_URL;#var-PREMIRRORS'><filename>PREMIRRORS</filename></ulink>
963 and
964 <ulink url='&YOCTO_DOCS_REF_URL;#var-MIRRORS'><filename>MIRRORS</filename></ulink>
965 variables point to these, respectively.
966 BitBake checks pre-mirrors before looking upstream for any
967 source files.
968 Pre-mirrors are appropriate when you have a shared
969 directory that is not a directory defined by the
970 <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink>
971 variable.
972 A Pre-mirror typically points to a shared directory that is
973 local to your organization.
974 </para>
975
976 <para>
977 Regular mirrors can be any site across the Internet
978 that is used as an alternative location for source
979 code should the primary site not be functioning for
980 some reason or another.
981 </para>
982 </section>
983 </section>
984
985 <section id="package-feeds-dev-environment">
986 <title>Package Feeds</title>
987
988 <para>
989 When the OpenEmbedded build system generates an image or an
990 SDK, it gets the packages from a package feed area located
991 in the
992 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>.
993 The
994 <link linkend='general-workflow-figure'>general workflow figure</link>
995 shows this package feeds area in the upper-right corner.
996 </para>
997
998 <para>
999 This section looks a little closer into the package feeds
1000 area used by the build system.
1001 Here is a more detailed look at the area:
1002 <imagedata fileref="figures/package-feeds.png" align="center" width="7in" depth="6in" />
1003 </para>
1004
1005 <para>
1006 Package feeds are an intermediary step in the build process.
1007 The OpenEmbedded build system provides classes to generate
1008 different package types, and you specify which classes to
1009 enable through the
1010 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></ulink>
1011 variable.
1012 Before placing the packages into package feeds,
1013 the build process validates them with generated output quality
1014 assurance checks through the
1015 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-insane'><filename>insane</filename></ulink>
1016 class.
1017 </para>
1018
1019 <para>
1020 The package feed area resides in the Build Directory.
1021 The directory the build system uses to temporarily store
1022 packages is determined by a combination of variables and the
1023 particular package manager in use.
1024 See the "Package Feeds" box in the illustration and note the
1025 information to the right of that area.
1026 In particular, the following defines where package files are
1027 kept:
1028 <itemizedlist>
1029 <listitem><para>
1030 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></ulink>:
1031 Defined as <filename>tmp/deploy</filename> in the Build
1032 Directory.
1033 </para></listitem>
1034 <listitem><para>
1035 <filename>DEPLOY_DIR_*</filename>:
1036 Depending on the package manager used, the package type
1037 sub-folder.
1038 Given RPM, IPK, or DEB packaging and tarball creation,
1039 the
1040 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_RPM'><filename>DEPLOY_DIR_RPM</filename></ulink>,
1041 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_IPK'><filename>DEPLOY_DIR_IPK</filename></ulink>,
1042 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_DEB'><filename>DEPLOY_DIR_DEB</filename></ulink>,
1043 or
1044 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_TAR'><filename>DEPLOY_DIR_TAR</filename></ulink>,
1045 variables are used, respectively.
1046 </para></listitem>
1047 <listitem><para>
1048 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCH'><filename>PACKAGE_ARCH</filename></ulink>:
1049 Defines architecture-specific sub-folders.
1050 For example, packages could exist for the i586 or
1051 qemux86 architectures.
1052 </para></listitem>
1053 </itemizedlist>
1054 </para>
1055
1056 <para>
1057 BitBake uses the
1058 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb'><filename>do_package_write_*</filename></ulink>
1059 tasks to generate packages and place them into the package
1060 holding area (e.g. <filename>do_package_write_ipk</filename>
1061 for IPK packages).
1062 See the
1063 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb'><filename>do_package_write_deb</filename></ulink>",
1064 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_ipk'><filename>do_package_write_ipk</filename></ulink>",
1065 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_rpm'><filename>do_package_write_rpm</filename></ulink>",
1066 and
1067 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_tar'><filename>do_package_write_tar</filename></ulink>"
1068 sections in the Yocto Project Reference Manual
1069 for additional information.
1070 As an example, consider a scenario where an IPK packaging
1071 manager is being used and package architecture support for
1072 both i586 and qemux86 exist.
1073 Packages for the i586 architecture are placed in
1074 <filename>build/tmp/deploy/ipk/i586</filename>, while packages
1075 for the qemux86 architecture are placed in
1076 <filename>build/tmp/deploy/ipk/qemux86</filename>.
1077 </para>
1078 </section>
1079
1080 <section id='bitbake-dev-environment'>
1081 <title>BitBake</title>
1082
1083 <para>
1084 The OpenEmbedded build system uses
1085 <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink>
1086 to produce images and Software Development Kits (SDKs).
1087 You can see from the
1088 <link linkend='general-workflow-figure'>general workflow figure</link>,
1089 the BitBake area consists of several functional areas.
1090 This section takes a closer look at each of those areas.
1091 <note>
1092 Separate documentation exists for the BitBake tool.
1093 See the
1094 <ulink url='&YOCTO_DOCS_BB_URL;#bitbake-user-manual'>BitBake User Manual</ulink>
1095 for reference material on BitBake.
1096 </note>
1097 </para>
1098
1099 <section id='source-fetching-dev-environment'>
1100 <title>Source Fetching</title>
1101
1102 <para>
1103 The first stages of building a recipe are to fetch and
1104 unpack the source code:
1105 <imagedata fileref="figures/source-fetching.png" align="center" width="6.5in" depth="5in" />
1106 </para>
1107
1108 <para>
1109 The
1110 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-fetch'><filename>do_fetch</filename></ulink>
1111 and
1112 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-unpack'><filename>do_unpack</filename></ulink>
1113 tasks fetch the source files and unpack them into the
1114 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>.
1115 <note>
1116 For every local file (e.g. <filename>file://</filename>)
1117 that is part of a recipe's
1118 <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink>
1119 statement, the OpenEmbedded build system takes a
1120 checksum of the file for the recipe and inserts the
1121 checksum into the signature for the
1122 <filename>do_fetch</filename> task.
1123 If any local file has been modified, the
1124 <filename>do_fetch</filename> task and all tasks that
1125 depend on it are re-executed.
1126 </note>
1127 By default, everything is accomplished in the Build
1128 Directory, which has a defined structure.
1129 For additional general information on the Build Directory,
1130 see the
1131 "<ulink url='&YOCTO_DOCS_REF_URL;#structure-core-build'><filename>build/</filename></ulink>"
1132 section in the Yocto Project Reference Manual.
1133 </para>
1134
1135 <para>
1136 Each recipe has an area in the Build Directory where the
1137 unpacked source code resides.
1138 The
1139 <ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink>
1140 variable points to this area for a recipe's unpacked source
1141 code.
1142 The name of that directory for any given recipe is defined
1143 from several different variables.
1144 The preceding figure and the following list describe
1145 the Build Directory's hierarchy:
1146 <itemizedlist>
1147 <listitem><para>
1148 <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink>:
1149 The base directory where the OpenEmbedded build
1150 system performs all its work during the build.
1151 The default base directory is the
1152 <filename>tmp</filename> directory.
1153 </para></listitem>
1154 <listitem><para>
1155 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCH'><filename>PACKAGE_ARCH</filename></ulink>:
1156 The architecture of the built package or packages.
1157 Depending on the eventual destination of the
1158 package or packages (i.e. machine architecture,
1159 <ulink url='&YOCTO_DOCS_REF_URL;#hardware-build-system-term'>build host</ulink>,
1160 SDK, or specific machine),
1161 <filename>PACKAGE_ARCH</filename> varies.
1162 See the variable's description for details.
1163 </para></listitem>
1164 <listitem><para>
1165 <ulink url='&YOCTO_DOCS_REF_URL;#var-TARGET_OS'><filename>TARGET_OS</filename></ulink>:
1166 The operating system of the target device.
1167 A typical value would be "linux" (e.g.
1168 "qemux86-poky-linux").
1169 </para></listitem>
1170 <listitem><para>
1171 <ulink url='&YOCTO_DOCS_REF_URL;#var-PN'><filename>PN</filename></ulink>:
1172 The name of the recipe used to build the package.
1173 This variable can have multiple meanings.
1174 However, when used in the context of input files,
1175 <filename>PN</filename> represents the the name
1176 of the recipe.
1177 </para></listitem>
1178 <listitem><para>
1179 <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>:
1180 The location where the OpenEmbedded build system
1181 builds a recipe (i.e. does the work to create the
1182 package).
1183 <itemizedlist>
1184 <listitem><para>
1185 <ulink url='&YOCTO_DOCS_REF_URL;#var-PV'><filename>PV</filename></ulink>:
1186 The version of the recipe used to build the
1187 package.
1188 </para></listitem>
1189 <listitem><para>
1190 <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink>:
1191 The revision of the recipe used to build the
1192 package.
1193 </para></listitem>
1194 </itemizedlist>
1195 </para></listitem>
1196 <listitem><para>
1197 <ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink>:
1198 Contains the unpacked source files for a given
1199 recipe.
1200 <itemizedlist>
1201 <listitem><para>
1202 <ulink url='&YOCTO_DOCS_REF_URL;#var-BPN'><filename>BPN</filename></ulink>:
1203 The name of the recipe used to build the
1204 package.
1205 The <filename>BPN</filename> variable is
1206 a version of the <filename>PN</filename>
1207 variable but with common prefixes and
1208 suffixes removed.
1209 </para></listitem>
1210 <listitem><para>
1211 <ulink url='&YOCTO_DOCS_REF_URL;#var-PV'><filename>PV</filename></ulink>:
1212 The version of the recipe used to build the
1213 package.
1214 </para></listitem>
1215 </itemizedlist>
1216 </para></listitem>
1217 </itemizedlist>
1218 <note>
1219 In the previous figure, notice that two sample
1220 hierarchies exist: one based on package architecture (i.e.
1221 <filename>PACKAGE_ARCH</filename>) and one based on a
1222 machine (i.e. <filename>MACHINE</filename>).
1223 The underlying structures are identical.
1224 The differentiator being what the OpenEmbedded build
1225 system is using as a build target (e.g. general
1226 architecture, a build host, an SDK, or a specific
1227 machine).
1228 </note>
1229 </para>
1230 </section>
1231
1232 <section id='patching-dev-environment'>
1233 <title>Patching</title>
1234
1235 <para>
1236 Once source code is fetched and unpacked, BitBake locates
1237 patch files and applies them to the source files:
1238 <imagedata fileref="figures/patching.png" align="center" width="7in" depth="6in" />
1239 </para>
1240
1241 <para>
1242 The
1243 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-patch'><filename>do_patch</filename></ulink>
1244 task uses a recipe's
1245 <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink>
1246 statements and the
1247 <ulink url='&YOCTO_DOCS_REF_URL;#var-FILESPATH'><filename>FILESPATH</filename></ulink>
1248 variable to locate applicable patch files.
1249 </para>
1250
1251 <para>
1252 Default processing for patch files assumes the files have
1253 either <filename>*.patch</filename> or
1254 <filename>*.diff</filename> file types.
1255 You can use <filename>SRC_URI</filename> parameters to
1256 change the way the build system recognizes patch files.
1257 See the
1258 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-patch'><filename>do_patch</filename></ulink>
1259 task for more information.
1260 </para>
1261
1262 <para>
1263 BitBake finds and applies multiple patches for a single
1264 recipe in the order in which it locates the patches.
1265 The <filename>FILESPATH</filename> variable defines the
1266 default set of directories that the build system uses to
1267 search for patch files.
1268 Once found, patches are applied to the recipe's source
1269 files, which are located in the
1270 <ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink>
1271 directory.
1272 </para>
1273
1274 <para>
1275 For more information on how the source directories are
1276 created, see the
1277 "<link linkend='source-fetching-dev-environment'>Source Fetching</link>"
1278 section.
1279 For more information on how to create patches and how the
1280 build system processes patches, see the
1281 "<ulink url='&YOCTO_DOCS_DEV_URL;#new-recipe-patching-code'>Patching Code</ulink>"
1282 section in the Yocto Project Development Tasks Manual.
1283 You can also see the
1284 "<ulink url='&YOCTO_DOCS_SDK_URL;#sdk-devtool-use-devtool-modify-to-modify-the-source-of-an-existing-component'>Use <filename>devtool modify</filename> to Modify the Source of an Existing Component</ulink>"
1285 section in the Yocto Project Application Development and
1286 the Extensible Software Development Kit (SDK) manual and
1287 the
1288 "<ulink url='&YOCTO_DOCS_KERNEL_DEV_URL;#using-traditional-kernel-development-to-patch-the-kernel'>Using Traditional Kernel Development to Patch the Kernel</ulink>"
1289 section in the Yocto Project Linux Kernel Development
1290 Manual.
1291 </para>
1292 </section>
1293
1294 <section id='configuration-compilation-and-staging-dev-environment'>
1295 <title>Configuration, Compilation, and Staging</title>
1296
1297 <para>
1298 After source code is patched, BitBake executes tasks that
1299 configure and compile the source code.
1300 Once compilation occurs, the files are copied to a holding
1301 area (staged) in preparation for packaging:
1302 <imagedata fileref="figures/configuration-compile-autoreconf.png" align="center" width="7in" depth="5in" />
1303 </para>
1304
1305 <para>
1306 This step in the build process consists of the following
1307 tasks:
1308 <itemizedlist>
1309 <listitem><para>
1310 <emphasis><ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-prepare_recipe_sysroot'><filename>do_prepare_recipe_sysroot</filename></ulink></emphasis>:
1311 This task sets up the two sysroots in
1312 <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink><filename>}</filename>
1313 (i.e. <filename>recipe-sysroot</filename> and
1314 <filename>recipe-sysroot-native</filename>) so that
1315 during the packaging phase the sysroots can contain
1316 the contents of the
1317 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sysroot'><filename>do_populate_sysroot</filename></ulink>
1318 tasks of the recipes on which the recipe
1319 containing the tasks depends.
1320 A sysroot exists for both the target and for the
1321 native binaries, which run on the host system.
1322 </para></listitem>
1323 <listitem><para>
1324 <emphasis><filename>do_configure</filename></emphasis>:
1325 This task configures the source by enabling and
1326 disabling any build-time and configuration options
1327 for the software being built.
1328 Configurations can come from the recipe itself as
1329 well as from an inherited class.
1330 Additionally, the software itself might configure
1331 itself depending on the target for which it is
1332 being built.</para>
1333
1334 <para>The configurations handled by the
1335 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-configure'><filename>do_configure</filename></ulink>
1336 task are specific to configurations for the source
1337 code being built by the recipe.</para>
1338
1339 <para>If you are using the
1340 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-autotools'><filename>autotools</filename></ulink>
1341 class, you can add additional configuration options
1342 by using the
1343 <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTRA_OECONF'><filename>EXTRA_OECONF</filename></ulink>
1344 or
1345 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGECONFIG_CONFARGS'><filename>PACKAGECONFIG_CONFARGS</filename></ulink>
1346 variables.
1347 For information on how this variable works within
1348 that class, see the
1349 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-autotools'><filename>autotools</filename></ulink>
1350 class
1351 <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta/classes/autotools.bbclass'>here</ulink>.
1352 </para></listitem>
1353 <listitem><para>
1354 <emphasis><filename>do_compile</filename></emphasis>:
1355 Once a configuration task has been satisfied,
1356 BitBake compiles the source using the
1357 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-compile'><filename>do_compile</filename></ulink>
1358 task.
1359 Compilation occurs in the directory pointed to by
1360 the
1361 <ulink url='&YOCTO_DOCS_REF_URL;#var-B'><filename>B</filename></ulink>
1362 variable.
1363 Realize that the <filename>B</filename> directory
1364 is, by default, the same as the
1365 <ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink>
1366 directory.
1367 </para></listitem>
1368 <listitem><para>
1369 <emphasis><filename>do_install</filename></emphasis>:
1370 After compilation completes, BitBake executes the
1371 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink>
1372 task.
1373 This task copies files from the
1374 <filename>B</filename> directory and places them
1375 in a holding area pointed to by the
1376 <ulink url='&YOCTO_DOCS_REF_URL;#var-D'><filename>D</filename></ulink>
1377 variable.
1378 Packaging occurs later using files from this
1379 holding directory.
1380 </para></listitem>
1381 </itemizedlist>
1382 </para>
1383 </section>
1384
1385 <section id='package-splitting-dev-environment'>
1386 <title>Package Splitting</title>
1387
1388 <para>
1389 After source code is configured, compiled, and staged, the
1390 build system analyzes the results and splits the output
1391 into packages:
1392 <imagedata fileref="figures/analysis-for-package-splitting.png" align="center" width="7in" depth="7in" />
1393 </para>
1394
1395 <para>
1396 The
1397 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>
1398 and
1399 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-packagedata'><filename>do_packagedata</filename></ulink>
1400 tasks combine to analyze the files found in the
1401 <ulink url='&YOCTO_DOCS_REF_URL;#var-D'><filename>D</filename></ulink>
1402 directory and split them into subsets based on available
1403 packages and files.
1404 Analysis involves the following as well as other items:
1405 splitting out debugging symbols, looking at shared library
1406 dependencies between packages, and looking at package
1407 relationships.
1408 </para>
1409
1410 <para>
1411 The <filename>do_packagedata</filename> task creates
1412 package metadata based on the analysis such that the
1413 build system can generate the final packages.
1414 The
1415 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sysroot'><filename>do_populate_sysroot</filename></ulink>
1416 task stages (copies) a subset of the files installed by
1417 the
1418 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink>
1419 task into the appropriate sysroot.
1420 Working, staged, and intermediate results of the analysis
1421 and package splitting process use several areas:
1422 <itemizedlist>
1423 <listitem><para>
1424 <ulink url='&YOCTO_DOCS_REF_URL;#var-PKGD'><filename>PKGD</filename></ulink>:
1425 The destination directory
1426 (i.e. <filename>package</filename>) for packages
1427 before they are split into individual packages.
1428 </para></listitem>
1429 <listitem><para>
1430 <ulink url='&YOCTO_DOCS_REF_URL;#var-PKGDESTWORK'><filename>PKGDESTWORK</filename></ulink>:
1431 A temporary work area (i.e.
1432 <filename>pkgdata</filename>) used by the
1433 <filename>do_package</filename> task to save
1434 package metadata.
1435 </para></listitem>
1436 <listitem><para>
1437 <ulink url='&YOCTO_DOCS_REF_URL;#var-PKGDEST'><filename>PKGDEST</filename></ulink>:
1438 The parent directory (i.e.
1439 <filename>packages-split</filename>) for packages
1440 after they have been split.
1441 </para></listitem>
1442 <listitem><para>
1443 <ulink url='&YOCTO_DOCS_REF_URL;#var-PKGDATA_DIR'><filename>PKGDATA_DIR</filename></ulink>:
1444 A shared, global-state directory that holds
1445 packaging metadata generated during the packaging
1446 process.
1447 The packaging process copies metadata from
1448 <filename>PKGDESTWORK</filename> to the
1449 <filename>PKGDATA_DIR</filename> area where it
1450 becomes globally available.
1451 </para></listitem>
1452 <listitem><para>
1453 <ulink url='&YOCTO_DOCS_REF_URL;#var-STAGING_DIR_HOST'><filename>STAGING_DIR_HOST</filename></ulink>:
1454 The path for the sysroot for the system on which
1455 a component is built to run (i.e.
1456 <filename>recipe-sysroot</filename>).
1457 </para></listitem>
1458 <listitem><para>
1459 <ulink url='&YOCTO_DOCS_REF_URL;#var-STAGING_DIR_NATIVE'><filename>STAGING_DIR_NATIVE</filename></ulink>:
1460 The path for the sysroot used when building
1461 components for the build host (i.e.
1462 <filename>recipe-sysroot-native</filename>).
1463 </para></listitem>
1464 <listitem><para>
1465 <ulink url='&YOCTO_DOCS_REF_URL;#var-STAGING_DIR_TARGET'><filename>STAGING_DIR_TARGET</filename></ulink>:
1466 The path for the sysroot used when a component that
1467 is built to execute on a system and it generates
1468 code for yet another machine (e.g. cross-canadian
1469 recipes).
1470 </para></listitem>
1471 </itemizedlist>
1472 The
1473 <ulink url='&YOCTO_DOCS_REF_URL;#var-FILES'><filename>FILES</filename></ulink>
1474 variable defines the files that go into each package in
1475 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGES'><filename>PACKAGES</filename></ulink>.
1476 If you want details on how this is accomplished, you can
1477 look at
1478 <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta/classes/package.bbclass'><filename>package.bbclass</filename></ulink>.
1479 </para>
1480
1481 <para>
1482 Depending on the type of packages being created (RPM, DEB,
1483 or IPK), the
1484 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb'><filename>do_package_write_*</filename></ulink>
1485 task creates the actual packages and places them in the
1486 Package Feed area, which is
1487 <filename>${TMPDIR}/deploy</filename>.
1488 You can see the
1489 "<link linkend='package-feeds-dev-environment'>Package Feeds</link>"
1490 section for more detail on that part of the build process.
1491 <note>
1492 Support for creating feeds directly from the
1493 <filename>deploy/*</filename> directories does not
1494 exist.
1495 Creating such feeds usually requires some kind of feed
1496 maintenance mechanism that would upload the new
1497 packages into an official package feed (e.g. the
1498 Ångström distribution).
1499 This functionality is highly distribution-specific
1500 and thus is not provided out of the box.
1501 </note>
1502 </para>
1503 </section>
1504
1505 <section id='image-generation-dev-environment'>
1506 <title>Image Generation</title>
1507
1508 <para>
1509 Once packages are split and stored in the Package Feeds
1510 area, the build system uses BitBake to generate the root
1511 filesystem image:
1512 <imagedata fileref="figures/image-generation.png" align="center" width="7.5in" depth="7.5in" />
1513 </para>
1514
1515 <para>
1516 The image generation process consists of several stages and
1517 depends on several tasks and variables.
1518 The
1519 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-rootfs'><filename>do_rootfs</filename></ulink>
1520 task creates the root filesystem (file and directory
1521 structure) for an image.
1522 This task uses several key variables to help create the
1523 list of packages to actually install:
1524 <itemizedlist>
1525 <listitem><para>
1526 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_INSTALL'><filename>IMAGE_INSTALL</filename></ulink>:
1527 Lists out the base set of packages from which to
1528 install from the Package Feeds area.
1529 </para></listitem>
1530 <listitem><para>
1531 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_EXCLUDE'><filename>PACKAGE_EXCLUDE</filename></ulink>:
1532 Specifies packages that should not be installed
1533 into the image.
1534 </para></listitem>
1535 <listitem><para>
1536 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FEATURES'><filename>IMAGE_FEATURES</filename></ulink>:
1537 Specifies features to include in the image.
1538 Most of these features map to additional packages
1539 for installation.
1540 </para></listitem>
1541 <listitem><para>
1542 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></ulink>:
1543 Specifies the package backend (e.g. RPM, DEB, or
1544 IPK) to use and consequently helps determine where
1545 to locate packages within the Package Feeds area.
1546 </para></listitem>
1547 <listitem><para>
1548 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_LINGUAS'><filename>IMAGE_LINGUAS</filename></ulink>:
1549 Determines the language(s) for which additional
1550 language support packages are installed.
1551 </para></listitem>
1552 <listitem><para>
1553 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_INSTALL'><filename>PACKAGE_INSTALL</filename></ulink>:
1554 The final list of packages passed to the package
1555 manager for installation into the image.
1556 </para></listitem>
1557 </itemizedlist>
1558 </para>
1559
1560 <para>
1561 With
1562 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_ROOTFS'><filename>IMAGE_ROOTFS</filename></ulink>
1563 pointing to the location of the filesystem under
1564 construction and the <filename>PACKAGE_INSTALL</filename>
1565 variable providing the final list of packages to install,
1566 the root file system is created.
1567 </para>
1568
1569 <para>
1570 Package installation is under control of the package
1571 manager (e.g. dnf/rpm, opkg, or apt/dpkg) regardless of
1572 whether or not package management is enabled for the
1573 target.
1574 At the end of the process, if package management is not
1575 enabled for the target, the package manager's data files
1576 are deleted from the root filesystem.
1577 As part of the final stage of package installation,
1578 post installation scripts that are part of the packages
1579 are run.
1580 Any scripts that fail to run on the build host are run on
1581 the target when the target system is first booted.
1582 If you are using a
1583 <ulink url='&YOCTO_DOCS_DEV_URL;#creating-a-read-only-root-filesystem'>read-only root filesystem</ulink>,
1584 all the post installation scripts must succeed on the
1585 build host during the package installation phase since the
1586 root filesystem on the target is read-only.
1587 </para>
1588
1589 <para>
1590 The final stages of the <filename>do_rootfs</filename> task
1591 handle post processing.
1592 Post processing includes creation of a manifest file and
1593 optimizations.
1594 </para>
1595
1596 <para>
1597 The manifest file (<filename>.manifest</filename>) resides
1598 in the same directory as the root filesystem image.
1599 This file lists out, line-by-line, the installed packages.
1600 The manifest file is useful for the
1601 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-testimage*'><filename>testimage</filename></ulink>
1602 class, for example, to determine whether or not to run
1603 specific tests.
1604 See the
1605 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_MANIFEST'><filename>IMAGE_MANIFEST</filename></ulink>
1606 variable for additional information.
1607 </para>
1608
1609 <para>
1610 Optimizing processes that are run across the image include
1611 <filename>mklibs</filename>, <filename>prelink</filename>,
1612 and any other post-processing commands as defined by the
1613 <ulink url='&YOCTO_DOCS_REF_URL;#var-ROOTFS_POSTPROCESS_COMMAND'><filename>ROOTFS_POSTPROCESS_COMMAND</filename></ulink>
1614 variable.
1615 The <filename>mklibs</filename> process optimizes the size
1616 of the libraries, while the <filename>prelink</filename>
1617 process optimizes the dynamic linking of shared libraries
1618 to reduce start up time of executables.
1619 </para>
1620
1621 <para>
1622 After the root filesystem is built, processing begins on
1623 the image through the
1624 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-image'><filename>do_image</filename></ulink>
1625 task.
1626 The build system runs any pre-processing commands as
1627 defined by the
1628 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_PREPROCESS_COMMAND'><filename>IMAGE_PREPROCESS_COMMAND</filename></ulink>
1629 variable.
1630 This variable specifies a list of functions to call before
1631 the build system creates the final image output files.
1632 </para>
1633
1634 <para>
1635 The build system dynamically creates
1636 <filename>do_image_*</filename> tasks as needed, based
1637 on the image types specified in the
1638 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FSTYPES'><filename>IMAGE_FSTYPES</filename></ulink>
1639 variable.
1640 The process turns everything into an image file or a set of
1641 image files and can compress the root filesystem image to
1642 reduce the overall size of the image.
1643 The formats used for the root filesystem depend on the
1644 <filename>IMAGE_FSTYPES</filename> variable.
1645 Compression depends on whether the formats support
1646 compression.
1647 </para>
1648
1649 <para>
1650 As an example, a dynamically created task when creating a
1651 particular image <replaceable>type</replaceable> would
1652 take the following form:
1653 <literallayout class='monospaced'>
1654 do_image_<replaceable>type</replaceable>
1655 </literallayout>
1656 So, if the <replaceable>type</replaceable> as specified by
1657 the <filename>IMAGE_FSTYPES</filename> were
1658 <filename>ext4</filename>, the dynamically generated task
1659 would be as follows:
1660 <literallayout class='monospaced'>
1661 do_image_ext4
1662 </literallayout>
1663 </para>
1664
1665 <para>
1666 The final task involved in image creation is the
1667 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-image-complete'><filename>do_image_complete</filename></ulink>
1668 task.
1669 This task completes the image by applying any image
1670 post processing as defined through the
1671 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_POSTPROCESS_COMMAND'><filename>IMAGE_POSTPROCESS_COMMAND</filename></ulink>
1672 variable.
1673 The variable specifies a list of functions to call once the
1674 build system has created the final image output files.
1675 <note>
1676 The entire image generation process is run under
1677 <link linkend='fakeroot-and-pseudo'>Pseudo</link>.
1678 Running under Pseudo ensures that the files in the
1679 root filesystem have correct ownership.
1680 </note>
1681 </para>
1682 </section>
1683
1684 <section id='sdk-generation-dev-environment'>
1685 <title>SDK Generation</title>
1686
1687 <para>
1688 The OpenEmbedded build system uses BitBake to generate the
1689 Software Development Kit (SDK) installer scripts for both
1690 the standard SDK and the extensible SDK (eSDK):
1691 </para>
1692
1693 <para>
1694 <imagedata fileref="figures/sdk-generation.png" width="9in" align="center" />
1695 <note>
1696 For more information on the cross-development toolchain
1697 generation, see the
1698 "<link linkend='cross-development-toolchain-generation'>Cross-Development Toolchain Generation</link>"
1699 section.
1700 For information on advantages gained when building a
1701 cross-development toolchain using the
1702 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sdk'><filename>do_populate_sdk</filename></ulink>
1703 task, see the
1704 "<ulink url='&YOCTO_DOCS_SDK_URL;#sdk-building-an-sdk-installer'>Building an SDK Installer</ulink>"
1705 section in the Yocto Project Application Development
1706 and the Extensible Software Development Kit (eSDK)
1707 manual.
1708 </note>
1709 </para>
1710
1711 <para>
1712 Like image generation, the SDK script process consists of
1713 several stages and depends on many variables.
1714 The
1715 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sdk'><filename>do_populate_sdk</filename></ulink>
1716 and
1717 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sdk_ext'><filename>do_populate_sdk_ext</filename></ulink>
1718 tasks use these key variables to help create the list of
1719 packages to actually install.
1720 For information on the variables listed in the figure,
1721 see the
1722 "<link linkend='sdk-dev-environment'>Application Development SDK</link>"
1723 section.
1724 </para>
1725
1726 <para>
1727 The <filename>do_populate_sdk</filename> task helps create
1728 the standard SDK and handles two parts: a target part and a
1729 host part.
1730 The target part is the part built for the target hardware
1731 and includes libraries and headers.
1732 The host part is the part of the SDK that runs on the
1733 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKMACHINE'><filename>SDKMACHINE</filename></ulink>.
1734 </para>
1735
1736 <para>
1737 The <filename>do_populate_sdk_ext</filename> task helps
1738 create the extensible SDK and handles host and target parts
1739 differently than its counter part does for the standard SDK.
1740 For the extensible SDK, the task encapsulates the build
1741 system, which includes everything needed (host and target)
1742 for the SDK.
1743 </para>
1744
1745 <para>
1746 Regardless of the type of SDK being constructed, the
1747 tasks perform some cleanup after which a cross-development
1748 environment setup script and any needed configuration files
1749 are created.
1750 The final output is the Cross-development
1751 toolchain installation script (<filename>.sh</filename>
1752 file), which includes the environment setup script.
1753 </para>
1754 </section>
1755
1756 <section id='stamp-files-and-the-rerunning-of-tasks'>
1757 <title>Stamp Files and the Rerunning of Tasks</title>
1758
1759 <para>
1760 For each task that completes successfully, BitBake writes a
1761 stamp file into the
1762 <ulink url='&YOCTO_DOCS_REF_URL;#var-STAMPS_DIR'><filename>STAMPS_DIR</filename></ulink>
1763 directory.
1764 The beginning of the stamp file's filename is determined
1765 by the
1766 <ulink url='&YOCTO_DOCS_REF_URL;#var-STAMP'><filename>STAMP</filename></ulink>
1767 variable, and the end of the name consists of the task's
1768 name and current
1769 <link linkend='overview-checksums'>input checksum</link>.
1770 <note>
1771 This naming scheme assumes that
1772 <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_SIGNATURE_HANDLER'><filename>BB_SIGNATURE_HANDLER</filename></ulink>
1773 is "OEBasicHash", which is almost always the case in
1774 current OpenEmbedded.
1775 </note>
1776 To determine if a task needs to be rerun, BitBake checks
1777 if a stamp file with a matching input checksum exists
1778 for the task.
1779 If such a stamp file exists, the task's output is
1780 assumed to exist and still be valid.
1781 If the file does not exist, the task is rerun.
1782 <note>
1783 <para>The stamp mechanism is more general than the
1784 shared state (sstate) cache mechanism described in the
1785 "<link linkend='setscene-tasks-and-shared-state'>Setscene Tasks and Shared State</link>"
1786 section.
1787 BitBake avoids rerunning any task that has a valid
1788 stamp file, not just tasks that can be accelerated
1789 through the sstate cache.</para>
1790
1791 <para>However, you should realize that stamp files only
1792 serve as a marker that some work has been done and that
1793 these files do not record task output.
1794 The actual task output would usually be somewhere in
1795 <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink>
1796 (e.g. in some recipe's
1797 <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>.)
1798 What the sstate cache mechanism adds is a way to cache
1799 task output that can then be shared between build
1800 machines.</para>
1801 </note>
1802 Since <filename>STAMPS_DIR</filename> is usually a
1803 subdirectory of <filename>TMPDIR</filename>, removing
1804 <filename>TMPDIR</filename> will also remove
1805 <filename>STAMPS_DIR</filename>, which means tasks will
1806 properly be rerun to repopulate
1807 <filename>TMPDIR</filename>.
1808 </para>
1809
1810 <para>
1811 If you want some task to always be considered "out of
1812 date", you can mark it with the
1813 <ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'><filename>nostamp</filename></ulink>
1814 varflag.
1815 If some other task depends on such a task, then that
1816 task will also always be considered out of date, which
1817 might not be what you want.
1818 </para>
1819
1820 <para>
1821 For details on how to view information about a task's
1822 signature, see the
1823 "<ulink url='&YOCTO_DOCS_DEV_URL;#dev-viewing-task-variable-dependencies'>Viewing Task Variable Dependencies</ulink>"
1824 section in the Yocto Project Development Tasks Manual.
1825 </para>
1826 </section>
1827
1828 <section id='setscene-tasks-and-shared-state'>
1829 <title>Setscene Tasks and Shared State</title>
1830
1831 <para>
1832 The description of tasks so far assumes that BitBake needs
1833 to build everything and no available prebuilt objects
1834 exist.
1835 BitBake does support skipping tasks if prebuilt objects are
1836 available.
1837 These objects are usually made available in the form of a
1838 shared state (sstate) cache.
1839 <note>
1840 For information on variables affecting sstate, see the
1841 <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink>
1842 and
1843 <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_MIRRORS'><filename>SSTATE_MIRRORS</filename></ulink>
1844 variables.
1845 </note>
1846 </para>
1847
1848 <para>
1849 The idea of a setscene task (i.e
1850 <filename>do_</filename><replaceable>taskname</replaceable><filename>_setscene</filename>)
1851 is a version of the task where
1852 instead of building something, BitBake can skip to the end
1853 result and simply place a set of files into specific
1854 locations as needed.
1855 In some cases, it makes sense to have a setscene task
1856 variant (e.g. generating package files in the
1857 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb'><filename>do_package_write_*</filename></ulink>
1858 task).
1859 In other cases, it does not make sense (e.g. a
1860 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-patch'><filename>do_patch</filename></ulink>
1861 task or a
1862 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-unpack'><filename>do_unpack</filename></ulink>
1863 task) since the work involved would be equal to or greater
1864 than the underlying task.
1865 </para>
1866
1867 <para>
1868 In the build system, the common tasks that have setscene
1869 variants are
1870 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>,
1871 <filename>do_package_write_*</filename>,
1872 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-deploy'><filename>do_deploy</filename></ulink>,
1873 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-packagedata'><filename>do_packagedata</filename></ulink>,
1874 and
1875 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sysroot'><filename>do_populate_sysroot</filename></ulink>.
1876 Notice that these tasks represent most of the tasks whose
1877 output is an end result.
1878 </para>
1879
1880 <para>
1881 The build system has knowledge of the relationship between
1882 these tasks and other preceding tasks.
1883 For example, if BitBake runs
1884 <filename>do_populate_sysroot_setscene</filename> for
1885 something, it does not make sense to run any of the
1886 <filename>do_fetch</filename>,
1887 <filename>do_unpack</filename>,
1888 <filename>do_patch</filename>,
1889 <filename>do_configure</filename>,
1890 <filename>do_compile</filename>, and
1891 <filename>do_install</filename> tasks.
1892 However, if <filename>do_package</filename> needs to be
1893 run, BitBake needs to run those other tasks.
1894 </para>
1895
1896 <para>
1897 It becomes more complicated if everything can come
1898 from an sstate cache because some objects are simply
1899 not required at all.
1900 For example, you do not need a compiler or native tools,
1901 such as quilt, if nothing exists to compile or patch.
1902 If the <filename>do_package_write_*</filename> packages
1903 are available from sstate, BitBake does not need the
1904 <filename>do_package</filename> task data.
1905 </para>
1906
1907 <para>
1908 To handle all these complexities, BitBake runs in two
1909 phases.
1910 The first is the "setscene" stage.
1911 During this stage, BitBake first checks the sstate cache
1912 for any targets it is planning to build.
1913 BitBake does a fast check to see if the object exists
1914 rather than a complete download.
1915 If nothing exists, the second phase, which is the setscene
1916 stage, completes and the main build proceeds.
1917 </para>
1918
1919 <para>
1920 If objects are found in the sstate cache, the build system
1921 works backwards from the end targets specified by the user.
1922 For example, if an image is being built, the build system
1923 first looks for the packages needed for that image and the
1924 tools needed to construct an image.
1925 If those are available, the compiler is not needed.
1926 Thus, the compiler is not even downloaded.
1927 If something was found to be unavailable, or the
1928 download or setscene task fails, the build system then
1929 tries to install dependencies, such as the compiler, from
1930 the cache.
1931 </para>
1932
1933 <para>
1934 The availability of objects in the sstate cache is
1935 handled by the function specified by the
1936 <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_HASHCHECK_FUNCTION'><filename>BB_HASHCHECK_FUNCTION</filename></ulink>
1937 variable and returns a list of available objects.
1938 The function specified by the
1939 <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_SETSCENE_DEPVALID'><filename>BB_SETSCENE_DEPVALID</filename></ulink>
1940 variable is the function that determines whether a given
1941 dependency needs to be followed, and whether for any given
1942 relationship the function needs to be passed.
1943 The function returns a True or False value.
1944 </para>
1945 </section>
1946 </section>
1947
1948 <section id='images-dev-environment'>
1949 <title>Images</title>
1950
1951 <para>
1952 The images produced by the build system are compressed forms
1953 of the root filesystem and are ready to boot on a target
1954 device.
1955 You can see from the
1956 <link linkend='general-workflow-figure'>general workflow figure</link>
1957 that BitBake output, in part, consists of images.
1958 This section takes a closer look at this output:
1959 <imagedata fileref="figures/images.png" align="center" width="5.5in" depth="5.5in" />
1960 </para>
1961
1962 <note>
1963 For a list of example images that the Yocto Project provides,
1964 see the
1965 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-images'>Images</ulink>"
1966 chapter in the Yocto Project Reference Manual.
1967 </note>
1968
1969 <para>
1970 The build process writes images out to the
1971 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>
1972 inside the
1973 <filename>tmp/deploy/images/<replaceable>machine</replaceable>/</filename>
1974 folder as shown in the figure.
1975 This folder contains any files expected to be loaded on the
1976 target device.
1977 The
1978 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></ulink>
1979 variable points to the <filename>deploy</filename> directory,
1980 while the
1981 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_IMAGE'><filename>DEPLOY_DIR_IMAGE</filename></ulink>
1982 variable points to the appropriate directory containing images
1983 for the current configuration.
1984 <itemizedlist>
1985 <listitem><para>
1986 <replaceable>kernel-image</replaceable>:
1987 A kernel binary file.
1988 The
1989 <ulink url='&YOCTO_DOCS_REF_URL;#var-KERNEL_IMAGETYPE'><filename>KERNEL_IMAGETYPE</filename></ulink>
1990 variable determines the naming scheme for the
1991 kernel image file.
1992 Depending on this variable, the file could begin with
1993 a variety of naming strings.
1994 The
1995 <filename>deploy/images/</filename><replaceable>machine</replaceable>
1996 directory can contain multiple image files for the
1997 machine.
1998 </para></listitem>
1999 <listitem><para>
2000 <replaceable>root-filesystem-image</replaceable>:
2001 Root filesystems for the target device (e.g.
2002 <filename>*.ext3</filename> or
2003 <filename>*.bz2</filename> files).
2004 The
2005 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FSTYPES'><filename>IMAGE_FSTYPES</filename></ulink>
2006 variable determines the root filesystem image type.
2007 The
2008 <filename>deploy/images/</filename><replaceable>machine</replaceable>
2009 directory can contain multiple root filesystems for the
2010 machine.
2011 </para></listitem>
2012 <listitem><para>
2013 <replaceable>kernel-modules</replaceable>:
2014 Tarballs that contain all the modules built for the
2015 kernel.
2016 Kernel module tarballs exist for legacy purposes and
2017 can be suppressed by setting the
2018 <ulink url='&YOCTO_DOCS_REF_URL;#var-MODULE_TARBALL_DEPLOY'><filename>MODULE_TARBALL_DEPLOY</filename></ulink>
2019 variable to "0".
2020 The
2021 <filename>deploy/images/</filename><replaceable>machine</replaceable>
2022 directory can contain multiple kernel module tarballs
2023 for the machine.
2024 </para></listitem>
2025 <listitem><para>
2026 <replaceable>bootloaders</replaceable>:
2027 If applicable to the target machine, bootloaders
2028 supporting the image.
2029 The <filename>deploy/images/</filename><replaceable>machine</replaceable>
2030 directory can contain multiple bootloaders for the
2031 machine.
2032 </para></listitem>
2033 <listitem><para>
2034 <replaceable>symlinks</replaceable>:
2035 The
2036 <filename>deploy/images/</filename><replaceable>machine</replaceable>
2037 folder contains a symbolic link that points to the
2038 most recently built file for each machine.
2039 These links might be useful for external scripts that
2040 need to obtain the latest version of each file.
2041 </para></listitem>
2042 </itemizedlist>
2043 </para>
2044 </section>
2045
2046 <section id='sdk-dev-environment'>
2047 <title>Application Development SDK</title>
2048
2049 <para>
2050 In the
2051 <link linkend='general-workflow-figure'>general workflow figure</link>,
2052 the output labeled "Application Development SDK" represents an
2053 SDK.
2054 The SDK generation process differs depending on whether you
2055 build an extensible SDK (e.g.
2056 <filename>bitbake -c populate_sdk_ext</filename> <replaceable>imagename</replaceable>)
2057 or a standard SDK (e.g.
2058 <filename>bitbake -c populate_sdk</filename> <replaceable>imagename</replaceable>).
2059 This section takes a closer look at this output:
2060 <imagedata fileref="figures/sdk.png" align="center" width="9in" depth="7.25in" />
2061 </para>
2062
2063 <para>
2064 The specific form of this output is a set of files that
2065 includes a self-extracting SDK installer
2066 (<filename>*.sh</filename>), host and target manifest files,
2067 and files used for SDK testing.
2068 When the SDK installer file is run, it installs the SDK.
2069 The SDK consists of a cross-development toolchain, a set of
2070 libraries and headers, and an SDK environment setup script.
2071 Running this installer essentially sets up your
2072 cross-development environment.
2073 You can think of the cross-toolchain as the "host"
2074 part because it runs on the SDK machine.
2075 You can think of the libraries and headers as the "target"
2076 part because they are built for the target hardware.
2077 The environment setup script is added so that you can
2078 initialize the environment before using the tools.
2079 </para>
2080
2081 <note><title>Notes</title>
2082 <itemizedlist>
2083 <listitem><para>
2084 The Yocto Project supports several methods by which
2085 you can set up this cross-development environment.
2086 These methods include downloading pre-built SDK
2087 installers or building and installing your own SDK
2088 installer.
2089 </para></listitem>
2090 <listitem><para>
2091 For background information on cross-development
2092 toolchains in the Yocto Project development
2093 environment, see the
2094 "<link linkend='cross-development-toolchain-generation'>Cross-Development Toolchain Generation</link>"
2095 section.
2096 </para></listitem>
2097 <listitem><para>
2098 For information on setting up a cross-development
2099 environment, see the
2100 <ulink url='&YOCTO_DOCS_SDK_URL;'>Yocto Project Application Development and the Extensible Software Development Kit (eSDK)</ulink>
2101 manual.
2102 </para></listitem>
2103 </itemizedlist>
2104 </note>
2105
2106 <para>
2107 All the output files for an SDK are written to the
2108 <filename>deploy/sdk</filename> folder inside the
2109 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>
2110 as shown in the previous figure.
2111 Depending on the type of SDK, several variables exist that help
2112 configure these files.
2113 The following list shows the variables associated with an
2114 extensible SDK:
2115 <itemizedlist>
2116 <listitem><para>
2117 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></ulink>:
2118 Points to the <filename>deploy</filename> directory.
2119 </para></listitem>
2120 <listitem><para>
2121 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_EXT_TYPE'><filename>SDK_EXT_TYPE</filename></ulink>:
2122 Controls whether or not shared state artifacts are
2123 copied into the extensible SDK.
2124 By default, all required shared state artifacts are
2125 copied into the SDK.
2126 </para></listitem>
2127 <listitem><para>
2128 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_INCLUDE_PKGDATA'><filename>SDK_INCLUDE_PKGDATA</filename></ulink>:
2129 Specifies whether or not packagedata is included in the
2130 extensible SDK for all recipes in the "world" target.
2131 </para></listitem>
2132 <listitem><para>
2133 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_INCLUDE_TOOLCHAIN'><filename>SDK_INCLUDE_TOOLCHAIN</filename></ulink>:
2134 Specifies whether or not the toolchain is included
2135 when building the extensible SDK.
2136 </para></listitem>
2137 <listitem><para>
2138 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_LOCAL_CONF_WHITELIST'><filename>SDK_LOCAL_CONF_WHITELIST</filename></ulink>:
2139 A list of variables allowed through from the build
2140 system configuration into the extensible SDK
2141 configuration.
2142 </para></listitem>
2143 <listitem><para>
2144 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_LOCAL_CONF_BLACKLIST'><filename>SDK_LOCAL_CONF_BLACKLIST</filename></ulink>:
2145 A list of variables not allowed through from the build
2146 system configuration into the extensible SDK
2147 configuration.
2148 </para></listitem>
2149 <listitem><para>
2150 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_INHERIT_BLACKLIST'><filename>SDK_INHERIT_BLACKLIST</filename></ulink>:
2151 A list of classes to remove from the
2152 <ulink url='&YOCTO_DOCS_REF_URL;#var-INHERIT'><filename>INHERIT</filename></ulink>
2153 value globally within the extensible SDK configuration.
2154 </para></listitem>
2155 </itemizedlist>
2156 This next list, shows the variables associated with a standard
2157 SDK:
2158 <itemizedlist>
2159 <listitem><para>
2160 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></ulink>:
2161 Points to the <filename>deploy</filename> directory.
2162 </para></listitem>
2163 <listitem><para>
2164 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKMACHINE'><filename>SDKMACHINE</filename></ulink>:
2165 Specifies the architecture of the machine on which the
2166 cross-development tools are run to create packages for
2167 the target hardware.
2168 </para></listitem>
2169 <listitem><para>
2170 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKIMAGE_FEATURES'><filename>SDKIMAGE_FEATURES</filename></ulink>:
2171 Lists the features to include in the "target" part
2172 of the SDK.
2173 </para></listitem>
2174 <listitem><para>
2175 <ulink url='&YOCTO_DOCS_REF_URL;#var-TOOLCHAIN_HOST_TASK'><filename>TOOLCHAIN_HOST_TASK</filename></ulink>:
2176 Lists packages that make up the host part of the SDK
2177 (i.e. the part that runs on the
2178 <filename>SDKMACHINE</filename>).
2179 When you use
2180 <filename>bitbake -c populate_sdk <replaceable>imagename</replaceable></filename>
2181 to create the SDK, a set of default packages apply.
2182 This variable allows you to add more packages.
2183 </para></listitem>
2184 <listitem><para>
2185 <ulink url='&YOCTO_DOCS_REF_URL;#var-TOOLCHAIN_TARGET_TASK'><filename>TOOLCHAIN_TARGET_TASK</filename></ulink>:
2186 Lists packages that make up the target part of the SDK
2187 (i.e. the part built for the target hardware).
2188 </para></listitem>
2189 <listitem><para>
2190 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKPATH'><filename>SDKPATH</filename></ulink>:
2191 Defines the default SDK installation path offered by
2192 the installation script.
2193 </para></listitem>
2194 <listitem><para>
2195 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_HOST_MANIFEST'><filename>SDK_HOST_MANIFEST</filename></ulink>:
2196 Lists all the installed packages that make up the host
2197 part of the SDK.
2198 This variable also plays a minor role for extensible
2199 SDK development as well.
2200 However, it is mainly used for the standard SDK.
2201 </para></listitem>
2202 <listitem><para>
2203 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_TARGET_MANIFEST'><filename>SDK_TARGET_MANIFEST</filename></ulink>:
2204 Lists all the installed packages that make up the
2205 target part of the SDK.
2206 This variable also plays a minor role for extensible
2207 SDK development as well.
2208 However, it is mainly used for the standard SDK.
2209 </para></listitem>
2210 </itemizedlist>
2211 </para>
2212 </section>
2213 </section>
2214
2215 <section id="cross-development-toolchain-generation">
2216 <title>Cross-Development Toolchain Generation</title>
2217
2218 <para>
2219 The Yocto Project does most of the work for you when it comes to
2220 creating
2221 <ulink url='&YOCTO_DOCS_REF_URL;#cross-development-toolchain'>cross-development toolchains</ulink>.
2222 This section provides some technical background on how
2223 cross-development toolchains are created and used.
2224 For more information on toolchains, you can also see the
2225 <ulink url='&YOCTO_DOCS_SDK_URL;'>Yocto Project Application Development and the Extensible Software Development Kit (eSDK)</ulink>
2226 manual.
2227 </para>
2228
2229 <para>
2230 In the Yocto Project development environment, cross-development
2231 toolchains are used to build images and applications that run
2232 on the target hardware.
2233 With just a few commands, the OpenEmbedded build system creates
2234 these necessary toolchains for you.
2235 </para>
2236
2237 <para>
2238 The following figure shows a high-level build environment regarding
2239 toolchain construction and use.
2240 </para>
2241
2242 <para>
2243 <imagedata fileref="figures/cross-development-toolchains.png" width="8in" depth="6in" align="center" />
2244 </para>
2245
2246 <para>
2247 Most of the work occurs on the Build Host.
2248 This is the machine used to build images and generally work within
2249 the the Yocto Project environment.
2250 When you run
2251 <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink>
2252 to create an image, the OpenEmbedded build system
2253 uses the host <filename>gcc</filename> compiler to bootstrap a
2254 cross-compiler named <filename>gcc-cross</filename>.
2255 The <filename>gcc-cross</filename> compiler is what BitBake uses to
2256 compile source files when creating the target image.
2257 You can think of <filename>gcc-cross</filename> simply as an
2258 automatically generated cross-compiler that is used internally
2259 within BitBake only.
2260 <note>
2261 The extensible SDK does not use
2262 <filename>gcc-cross-canadian</filename> since this SDK
2263 ships a copy of the OpenEmbedded build system and the sysroot
2264 within it contains <filename>gcc-cross</filename>.
2265 </note>
2266 </para>
2267
2268 <para>
2269 The chain of events that occurs when <filename>gcc-cross</filename> is
2270 bootstrapped is as follows:
2271 <literallayout class='monospaced'>
2272 gcc -> binutils-cross -> gcc-cross-initial -> linux-libc-headers -> glibc-initial -> glibc -> gcc-cross -> gcc-runtime
2273 </literallayout>
2274 <itemizedlist>
2275 <listitem><para>
2276 <filename>gcc</filename>:
2277 The build host's GNU Compiler Collection (GCC).
2278 </para></listitem>
2279 <listitem><para>
2280 <filename>binutils-cross</filename>:
2281 The bare minimum binary utilities needed in order to run
2282 the <filename>gcc-cross-initial</filename> phase of the
2283 bootstrap operation.
2284 </para></listitem>
2285 <listitem><para>
2286 <filename>gcc-cross-initial</filename>:
2287 An early stage of the bootstrap process for creating
2288 the cross-compiler.
2289 This stage builds enough of the <filename>gcc-cross</filename>,
2290 the C library, and other pieces needed to finish building the
2291 final cross-compiler in later stages.
2292 This tool is a "native" package (i.e. it is designed to run on
2293 the build host).
2294 </para></listitem>
2295 <listitem><para>
2296 <filename>linux-libc-headers</filename>:
2297 Headers needed for the cross-compiler.
2298 </para></listitem>
2299 <listitem><para>
2300 <filename>glibc-initial</filename>:
2301 An initial version of the Embedded GNU C Library
2302 (GLIBC) needed to bootstrap <filename>glibc</filename>.
2303 </para></listitem>
2304 <listitem><para>
2305 <filename>glibc</filename>:
2306 The GNU C Library.
2307 </para></listitem>
2308 <listitem><para>
2309 <filename>gcc-cross</filename>:
2310 The final stage of the bootstrap process for the
2311 cross-compiler.
2312 This stage results in the actual cross-compiler that
2313 BitBake uses when it builds an image for a targeted
2314 device.
2315 <note>
2316 If you are replacing this cross compiler toolchain
2317 with a custom version, you must replace
2318 <filename>gcc-cross</filename>.
2319 </note>
2320 This tool is also a "native" package (i.e. it is
2321 designed to run on the build host).
2322 </para></listitem>
2323 <listitem><para>
2324 <filename>gcc-runtime</filename>:
2325 Runtime libraries resulting from the toolchain bootstrapping
2326 process.
2327 This tool produces a binary that consists of the
2328 runtime libraries need for the targeted device.
2329 </para></listitem>
2330 </itemizedlist>
2331 </para>
2332
2333 <para>
2334 You can use the OpenEmbedded build system to build an installer for
2335 the relocatable SDK used to develop applications.
2336 When you run the installer, it installs the toolchain, which
2337 contains the development tools (e.g.,
2338 <filename>gcc-cross-canadian</filename>,
2339 <filename>binutils-cross-canadian</filename>, and other
2340 <filename>nativesdk-*</filename> tools),
2341 which are tools native to the SDK (i.e. native to
2342 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_ARCH'><filename>SDK_ARCH</filename></ulink>),
2343 you need to cross-compile and test your software.
2344 The figure shows the commands you use to easily build out this
2345 toolchain.
2346 This cross-development toolchain is built to execute on the
2347 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKMACHINE'><filename>SDKMACHINE</filename></ulink>,
2348 which might or might not be the same
2349 machine as the Build Host.
2350 <note>
2351 If your target architecture is supported by the Yocto Project,
2352 you can take advantage of pre-built images that ship with the
2353 Yocto Project and already contain cross-development toolchain
2354 installers.
2355 </note>
2356 </para>
2357
2358 <para>
2359 Here is the bootstrap process for the relocatable toolchain:
2360 <literallayout class='monospaced'>
2361 gcc -> binutils-crosssdk -> gcc-crosssdk-initial -> linux-libc-headers ->
2362 glibc-initial -> nativesdk-glibc -> gcc-crosssdk -> gcc-cross-canadian
2363 </literallayout>
2364 <itemizedlist>
2365 <listitem><para>
2366 <filename>gcc</filename>:
2367 The build host's GNU Compiler Collection (GCC).
2368 </para></listitem>
2369 <listitem><para>
2370 <filename>binutils-crosssdk</filename>:
2371 The bare minimum binary utilities needed in order to run
2372 the <filename>gcc-crosssdk-initial</filename> phase of the
2373 bootstrap operation.
2374 </para></listitem>
2375 <listitem><para>
2376 <filename>gcc-crosssdk-initial</filename>:
2377 An early stage of the bootstrap process for creating
2378 the cross-compiler.
2379 This stage builds enough of the
2380 <filename>gcc-crosssdk</filename> and supporting pieces so that
2381 the final stage of the bootstrap process can produce the
2382 finished cross-compiler.
2383 This tool is a "native" binary that runs on the build host.
2384 </para></listitem>
2385 <listitem><para>
2386 <filename>linux-libc-headers</filename>:
2387 Headers needed for the cross-compiler.
2388 </para></listitem>
2389 <listitem><para>
2390 <filename>glibc-initial</filename>:
2391 An initial version of the Embedded GLIBC needed to bootstrap
2392 <filename>nativesdk-glibc</filename>.
2393 </para></listitem>
2394 <listitem><para>
2395 <filename>nativesdk-glibc</filename>:
2396 The Embedded GLIBC needed to bootstrap the
2397 <filename>gcc-crosssdk</filename>.
2398 </para></listitem>
2399 <listitem><para>
2400 <filename>gcc-crosssdk</filename>:
2401 The final stage of the bootstrap process for the
2402 relocatable cross-compiler.
2403 The <filename>gcc-crosssdk</filename> is a transitory
2404 compiler and never leaves the build host.
2405 Its purpose is to help in the bootstrap process to create
2406 the eventual <filename>gcc-cross-canadian</filename>
2407 compiler, which is relocatable.
2408 This tool is also a "native" package (i.e. it is
2409 designed to run on the build host).
2410 </para></listitem>
2411 <listitem><para>
2412 <filename>gcc-cross-canadian</filename>:
2413 The final relocatable cross-compiler.
2414 When run on the
2415 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKMACHINE'><filename>SDKMACHINE</filename></ulink>,
2416 this tool
2417 produces executable code that runs on the target device.
2418 Only one cross-canadian compiler is produced per architecture
2419 since they can be targeted at different processor optimizations
2420 using configurations passed to the compiler through the
2421 compile commands.
2422 This circumvents the need for multiple compilers and thus
2423 reduces the size of the toolchains.
2424 </para></listitem>
2425 </itemizedlist>
2426 </para>
2427
2428 <note>
2429 For information on advantages gained when building a
2430 cross-development toolchain installer, see the
2431 "<ulink url='&YOCTO_DOCS_SDK_URL;#sdk-building-an-sdk-installer'>Building an SDK Installer</ulink>"
2432 appendix in the Yocto Project Application Development and the
2433 Extensible Software Development Kit (eSDK) manual.
2434 </note>
2435 </section>
2436
2437 <section id="shared-state-cache">
2438 <title>Shared State Cache</title>
2439
2440 <para>
2441 By design, the OpenEmbedded build system builds everything from
2442 scratch unless
2443 <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink>
2444 can determine that parts do not need to be rebuilt.
2445 Fundamentally, building from scratch is attractive as it means all
2446 parts are built fresh and no possibility of stale data exists that
2447 can cause problems.
2448 When developers hit problems, they typically default back to
2449 building from scratch so they have a know state from the
2450 start.
2451 </para>
2452
2453 <para>
2454 Building an image from scratch is both an advantage and a
2455 disadvantage to the process.
2456 As mentioned in the previous paragraph, building from scratch
2457 ensures that everything is current and starts from a known state.
2458 However, building from scratch also takes much longer as it
2459 generally means rebuilding things that do not necessarily need
2460 to be rebuilt.
2461 </para>
2462
2463 <para>
2464 The Yocto Project implements shared state code that supports
2465 incremental builds.
2466 The implementation of the shared state code answers the following
2467 questions that were fundamental roadblocks within the OpenEmbedded
2468 incremental build support system:
2469 <itemizedlist>
2470 <listitem><para>
2471 What pieces of the system have changed and what pieces have
2472 not changed?
2473 </para></listitem>
2474 <listitem><para>
2475 How are changed pieces of software removed and replaced?
2476 </para></listitem>
2477 <listitem><para>
2478 How are pre-built components that do not need to be rebuilt
2479 from scratch used when they are available?
2480 </para></listitem>
2481 </itemizedlist>
2482 </para>
2483
2484 <para>
2485 For the first question, the build system detects changes in the
2486 "inputs" to a given task by creating a checksum (or signature) of
2487 the task's inputs.
2488 If the checksum changes, the system assumes the inputs have changed
2489 and the task needs to be rerun.
2490 For the second question, the shared state (sstate) code tracks
2491 which tasks add which output to the build process.
2492 This means the output from a given task can be removed, upgraded
2493 or otherwise manipulated.
2494 The third question is partly addressed by the solution for the
2495 second question assuming the build system can fetch the sstate
2496 objects from remote locations and install them if they are deemed
2497 to be valid.
2498 <note><title>Notes</title>
2499 <itemizedlist>
2500 <listitem><para>
2501 The build system does not maintain
2502 <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink>
2503 information as part of the shared state packages.
2504 Consequently, considerations exist that affect
2505 maintaining shared state feeds.
2506 For information on how the build system works with
2507 packages and can track incrementing
2508 <filename>PR</filename> information, see the
2509 "<ulink url='&YOCTO_DOCS_DEV_URL;#automatically-incrementing-a-binary-package-revision-number'>Automatically Incrementing a Binary Package Revision Number</ulink>"
2510 section in the Yocto Project Development Tasks Manual.
2511 </para></listitem>
2512 <listitem><para>
2513 The code in the build system that supports incremental
2514 builds is not simple code.
2515 For techniques that help you work around issues related
2516 to shared state code, see the
2517 "<ulink url='&YOCTO_DOCS_DEV_URL;#dev-viewing-metadata-used-to-create-the-input-signature-of-a-shared-state-task'>Viewing Metadata Used to Create the Input Signature of a Shared State Task</ulink>"
2518 and
2519 "<ulink url='&YOCTO_DOCS_DEV_URL;#dev-invalidating-shared-state-to-force-a-task-to-run'>Invalidating Shared State to Force a Task to Run</ulink>"
2520 sections both in the Yocto Project Development Tasks
2521 Manual.
2522 </para></listitem>
2523 </itemizedlist>
2524 </note>
2525 </para>
2526
2527 <para>
2528 The rest of this section goes into detail about the overall
2529 incremental build architecture, the checksums (signatures), and
2530 shared state.
2531 </para>
2532
2533 <section id='concepts-overall-architecture'>
2534 <title>Overall Architecture</title>
2535
2536 <para>
2537 When determining what parts of the system need to be built,
2538 BitBake works on a per-task basis rather than a per-recipe
2539 basis.
2540 You might wonder why using a per-task basis is preferred over
2541 a per-recipe basis.
2542 To help explain, consider having the IPK packaging backend
2543 enabled and then switching to DEB.
2544 In this case, the
2545 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink>
2546 and
2547 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>
2548 task outputs are still valid.
2549 However, with a per-recipe approach, the build would not
2550 include the <filename>.deb</filename> files.
2551 Consequently, you would have to invalidate the whole build and
2552 rerun it.
2553 Rerunning everything is not the best solution.
2554 Also, in this case, the core must be "taught" much about
2555 specific tasks.
2556 This methodology does not scale well and does not allow users
2557 to easily add new tasks in layers or as external recipes
2558 without touching the packaged-staging core.
2559 </para>
2560 </section>
2561
2562 <section id='overview-checksums'>
2563 <title>Checksums (Signatures)</title>
2564
2565 <para>
2566 The shared state code uses a checksum, which is a unique
2567 signature of a task's inputs, to determine if a task needs to
2568 be run again.
2569 Because it is a change in a task's inputs that triggers a
2570 rerun, the process needs to detect all the inputs to a given
2571 task.
2572 For shell tasks, this turns out to be fairly easy because
2573 the build process generates a "run" shell script for each task
2574 and it is possible to create a checksum that gives you a good
2575 idea of when the task's data changes.
2576 </para>
2577
2578 <para>
2579 To complicate the problem, there are things that should not be
2580 included in the checksum.
2581 First, there is the actual specific build path of a given
2582 task - the
2583 <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>.
2584 It does not matter if the work directory changes because it
2585 should not affect the output for target packages.
2586 Also, the build process has the objective of making native
2587 or cross packages relocatable.
2588 <note>
2589 Both native and cross packages run on the
2590 <ulink url='&YOCTO_DOCS_REF_URL;#hardware-build-system-term'>build host</ulink>.
2591 However, cross packages generate output for the target
2592 architecture.
2593 </note>
2594 The checksum therefore needs to exclude
2595 <filename>WORKDIR</filename>.
2596 The simplistic approach for excluding the work directory is to
2597 set <filename>WORKDIR</filename> to some fixed value and
2598 create the checksum for the "run" script.
2599 </para>
2600
2601 <para>
2602 Another problem results from the "run" scripts containing
2603 functions that might or might not get called.
2604 The incremental build solution contains code that figures out
2605 dependencies between shell functions.
2606 This code is used to prune the "run" scripts down to the
2607 minimum set, thereby alleviating this problem and making the
2608 "run" scripts much more readable as a bonus.
2609 </para>
2610
2611 <para>
2612 So far, solutions for shell scripts exist.
2613 What about Python tasks?
2614 The same approach applies even though these tasks are more
2615 difficult.
2616 The process needs to figure out what variables a Python
2617 function accesses and what functions it calls.
2618 Again, the incremental build solution contains code that first
2619 figures out the variable and function dependencies, and then
2620 creates a checksum for the data used as the input to the task.
2621 </para>
2622
2623 <para>
2624 Like the <filename>WORKDIR</filename> case, situations exist
2625 where dependencies should be ignored.
2626 For these situations, you can instruct the build process to
2627 ignore a dependency by using a line like the following:
2628 <literallayout class='monospaced'>
2629 PACKAGE_ARCHS[vardepsexclude] = "MACHINE"
2630 </literallayout>
2631 This example ensures that the
2632 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCHS'><filename>PACKAGE_ARCHS</filename></ulink>
2633 variable does not depend on the value of
2634 <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink>,
2635 even if it does reference it.
2636 </para>
2637
2638 <para>
2639 Equally, there are cases where you need to add dependencies
2640 BitBake is not able to find.
2641 You can accomplish this by using a line like the following:
2642 <literallayout class='monospaced'>
2643 PACKAGE_ARCHS[vardeps] = "MACHINE"
2644 </literallayout>
2645 This example explicitly adds the <filename>MACHINE</filename>
2646 variable as a dependency for
2647 <filename>PACKAGE_ARCHS</filename>.
2648 </para>
2649
2650 <para>
2651 As an example, consider a case with in-line Python where
2652 BitBake is not able to figure out dependencies.
2653 When running in debug mode (i.e. using
2654 <filename>-DDD</filename>), BitBake produces output when it
2655 discovers something for which it cannot figure out dependencies.
2656 The Yocto Project team has currently not managed to cover
2657 those dependencies in detail and is aware of the need to fix
2658 this situation.
2659 </para>
2660
2661 <para>
2662 Thus far, this section has limited discussion to the direct
2663 inputs into a task.
2664 Information based on direct inputs is referred to as the
2665 "basehash" in the code.
2666 However, the question of a task's indirect inputs still
2667 exits - items already built and present in the
2668 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>.
2669 The checksum (or signature) for a particular task needs to add
2670 the hashes of all the tasks on which the particular task
2671 depends.
2672 Choosing which dependencies to add is a policy decision.
2673 However, the effect is to generate a master checksum that
2674 combines the basehash and the hashes of the task's
2675 dependencies.
2676 </para>
2677
2678 <para>
2679 At the code level, a variety of ways exist by which both the
2680 basehash and the dependent task hashes can be influenced.
2681 Within the BitBake configuration file, you can give BitBake
2682 some extra information to help it construct the basehash.
2683 The following statement effectively results in a list of
2684 global variable dependency excludes (i.e. variables never
2685 included in any checksum):
2686 <literallayout class='monospaced'>
2687 BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \
2688 SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \
2689 USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \
2690 PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \
2691 CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX"
2692 </literallayout>
2693 The previous example excludes
2694 <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>
2695 since that variable is actually constructed as a path within
2696 <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink>,
2697 which is on the whitelist.
2698 </para>
2699
2700 <para>
2701 The rules for deciding which hashes of dependent tasks to
2702 include through dependency chains are more complex and are
2703 generally accomplished with a Python function.
2704 The code in <filename>meta/lib/oe/sstatesig.py</filename> shows
2705 two examples of this and also illustrates how you can insert
2706 your own policy into the system if so desired.
2707 This file defines the two basic signature generators
2708 <ulink url='&YOCTO_DOCS_REF_URL;#oe-core'>OE-Core</ulink>
2709 uses: "OEBasic" and "OEBasicHash".
2710 By default, a dummy "noop" signature handler is enabled
2711 in BitBake.
2712 This means that behavior is unchanged from previous versions.
2713 OE-Core uses the "OEBasicHash" signature handler by default
2714 through this setting in the <filename>bitbake.conf</filename>
2715 file:
2716 <literallayout class='monospaced'>
2717 BB_SIGNATURE_HANDLER ?= "OEBasicHash"
2718 </literallayout>
2719 The "OEBasicHash" <filename>BB_SIGNATURE_HANDLER</filename>
2720 is the same as the "OEBasic" version but adds the task hash to
2721 the
2722 <link linkend='stamp-files-and-the-rerunning-of-tasks'>stamp files</link>.
2723 This results in any metadata change that changes the task hash,
2724 automatically causing the task to be run again.
2725 This removes the need to bump
2726 <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink>
2727 values, and changes to metadata automatically ripple across
2728 the build.
2729 </para>
2730
2731 <para>
2732 It is also worth noting that the end result of these
2733 signature generators is to make some dependency and hash
2734 information available to the build.
2735 This information includes:
2736 <itemizedlist>
2737 <listitem><para>
2738 <filename>BB_BASEHASH_task-</filename><replaceable>taskname</replaceable>:
2739 The base hashes for each task in the recipe.
2740 </para></listitem>
2741 <listitem><para>
2742 <filename>BB_BASEHASH_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>:
2743 The base hashes for each dependent task.
2744 </para></listitem>
2745 <listitem><para>
2746 <filename>BBHASHDEPS_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>:
2747 The task dependencies for each task.
2748 </para></listitem>
2749 <listitem><para>
2750 <filename>BB_TASKHASH</filename>:
2751 The hash of the currently running task.
2752 </para></listitem>
2753 </itemizedlist>
2754 </para>
2755 </section>
2756
2757 <section id='shared-state'>
2758 <title>Shared State</title>
2759
2760 <para>
2761 Checksums and dependencies, as discussed in the previous
2762 section, solve half the problem of supporting a shared state.
2763 The other half of the problem is being able to use checksum
2764 information during the build and being able to reuse or rebuild
2765 specific components.
2766 </para>
2767
2768 <para>
2769 The
2770 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-sstate'><filename>sstate</filename></ulink>
2771 class is a relatively generic implementation of how to
2772 "capture" a snapshot of a given task.
2773 The idea is that the build process does not care about the
2774 source of a task's output.
2775 Output could be freshly built or it could be downloaded and
2776 unpacked from somewhere.
2777 In other words, the build process does not need to worry about
2778 its origin.
2779 </para>
2780
2781 <para>
2782 Two types of output exist.
2783 One type is just about creating a directory in
2784 <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>.
2785 A good example is the output of either
2786 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink>
2787 or
2788 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>.
2789 The other type of output occurs when a set of data is merged
2790 into a shared directory tree such as the sysroot.
2791 </para>
2792
2793 <para>
2794 The Yocto Project team has tried to keep the details of the
2795 implementation hidden in <filename>sstate</filename> class.
2796 From a user's perspective, adding shared state wrapping to a
2797 task is as simple as this
2798 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-deploy'><filename>do_deploy</filename></ulink>
2799 example taken from the
2800 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-deploy'><filename>deploy</filename></ulink>
2801 class:
2802 <literallayout class='monospaced'>
2803 DEPLOYDIR = "${WORKDIR}/deploy-${PN}"
2804 SSTATETASKS += "do_deploy"
2805 do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"
2806 do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
2807
2808 python do_deploy_setscene () {
2809 sstate_setscene(d)
2810 }
2811 addtask do_deploy_setscene
2812 do_deploy[dirs] = "${DEPLOYDIR} ${B}"
2813 do_deploy[stamp-extra-info] = "${MACHINE_ARCH}"
2814 </literallayout>
2815 The following list explains the previous example:
2816 <itemizedlist>
2817 <listitem><para>
2818 Adding "do_deploy" to <filename>SSTATETASKS</filename>
2819 adds some required sstate-related processing, which is
2820 implemented in the
2821 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-sstate'><filename>sstate</filename></ulink>
2822 class, to before and after the
2823 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-deploy'><filename>do_deploy</filename></ulink>
2824 task.
2825 </para></listitem>
2826 <listitem><para>
2827 The
2828 <filename>do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"</filename>
2829 declares that <filename>do_deploy</filename> places its
2830 output in <filename>${DEPLOYDIR}</filename> when run
2831 normally (i.e. when not using the sstate cache).
2832 This output becomes the input to the shared state cache.
2833 </para></listitem>
2834 <listitem><para>
2835 The
2836 <filename>do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"</filename>
2837 line causes the contents of the shared state cache to be
2838 copied to <filename>${DEPLOY_DIR_IMAGE}</filename>.
2839 <note>
2840 If <filename>do_deploy</filename> is not already in
2841 the shared state cache or if its input checksum
2842 (signature) has changed from when the output was
2843 cached, the task runs to populate the shared
2844 state cache, after which the contents of the shared
2845 state cache is copied to
2846 <filename>${DEPLOY_DIR_IMAGE}</filename>.
2847 If <filename>do_deploy</filename> is in the shared
2848 state cache and its signature indicates that the
2849 cached output is still valid (i.e. if no
2850 relevant task inputs have changed), then the
2851 contents of the shared state cache copies
2852 directly to
2853 <filename>${DEPLOY_DIR_IMAGE}</filename> by the
2854 <filename>do_deploy_setscene</filename> task
2855 instead, skipping the
2856 <filename>do_deploy</filename> task.
2857 </note>
2858 </para></listitem>
2859 <listitem><para>
2860 The following task definition is glue logic needed to
2861 make the previous settings effective:
2862 <literallayout class='monospaced'>
2863 python do_deploy_setscene () {
2864 sstate_setscene(d)
2865 }
2866 addtask do_deploy_setscene
2867 </literallayout>
2868 <filename>sstate_setscene()</filename> takes the flags
2869 above as input and accelerates the
2870 <filename>do_deploy</filename> task through the
2871 shared state cache if possible.
2872 If the task was accelerated,
2873 <filename>sstate_setscene()</filename> returns True.
2874 Otherwise, it returns False, and the normal
2875 <filename>do_deploy</filename> task runs.
2876 For more information, see the
2877 "<ulink url='&YOCTO_DOCS_BB_URL;#setscene'>setscene</ulink>"
2878 section in the BitBake User Manual.
2879 </para></listitem>
2880 <listitem><para>
2881 The <filename>do_deploy[dirs] = "${DEPLOYDIR} ${B}"</filename>
2882 line creates <filename>${DEPLOYDIR}</filename> and
2883 <filename>${B}</filename> before the
2884 <filename>do_deploy</filename> task runs, and also sets
2885 the current working directory of
2886 <filename>do_deploy</filename> to
2887 <filename>${B}</filename>.
2888 For more information, see the
2889 "<ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'>Variable Flags</ulink>"
2890 section in the BitBake User Manual.
2891 <note>
2892 In cases where
2893 <filename>sstate-inputdirs</filename> and
2894 <filename>sstate-outputdirs</filename> would be the
2895 same, you can use
2896 <filename>sstate-plaindirs</filename>.
2897 For example, to preserve the
2898 <filename>${PKGD}</filename> and
2899 <filename>${PKGDEST}</filename> output from the
2900 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>
2901 task, use the following:
2902 <literallayout class='monospaced'>
2903 do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}"
2904 </literallayout>
2905 </note>
2906 </para></listitem>
2907 <listitem><para>
2908 The <filename>do_deploy[stamp-extra-info] = "${MACHINE_ARCH}"</filename>
2909 line appends extra metadata to the
2910 <link linkend='stamp-files-and-the-rerunning-of-tasks'>stamp file</link>.
2911 In this case, the metadata makes the task specific
2912 to a machine's architecture.
2913 See
2914 "<ulink url='&YOCTO_DOCS_BB_URL;#ref-bitbake-tasklist'>The Task List</ulink>"
2915 section in the BitBake User Manual for more
2916 information on the <filename>stamp-extra-info</filename>
2917 flag.
2918 </para></listitem>
2919 <listitem><para>
2920 <filename>sstate-inputdirs</filename> and
2921 <filename>sstate-outputdirs</filename> can also be used
2922 with multiple directories.
2923 For example, the following declares
2924 <filename>PKGDESTWORK</filename> and
2925 <filename>SHLIBWORK</filename> as shared state
2926 input directories, which populates the shared state
2927 cache, and <filename>PKGDATA_DIR</filename> and
2928 <filename>SHLIBSDIR</filename> as the corresponding
2929 shared state output directories:
2930 <literallayout class='monospaced'>
2931 do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}"
2932 do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}"
2933 </literallayout>
2934 </para></listitem>
2935 <listitem><para>
2936 These methods also include the ability to take a
2937 lockfile when manipulating shared state directory
2938 structures, for cases where file additions or removals
2939 are sensitive:
2940 <literallayout class='monospaced'>
2941 do_package[sstate-lockfile] = "${PACKAGELOCK}"
2942 </literallayout>
2943 </para></listitem>
2944 </itemizedlist>
2945 </para>
2946
2947 <para>
2948 Behind the scenes, the shared state code works by looking in
2949 <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink>
2950 and
2951 <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_MIRRORS'><filename>SSTATE_MIRRORS</filename></ulink>
2952 for shared state files.
2953 Here is an example:
2954 <literallayout class='monospaced'>
2955 SSTATE_MIRRORS ?= "\
2956 file://.* http://someserver.tld/share/sstate/PATH;downloadfilename=PATH \n \
2957 file://.* file:///some/local/dir/sstate/PATH"
2958 </literallayout>
2959 <note>
2960 The shared state directory
2961 (<filename>SSTATE_DIR</filename>) is organized into
2962 two-character subdirectories, where the subdirectory
2963 names are based on the first two characters of the hash.
2964 If the shared state directory structure for a mirror has the
2965 same structure as <filename>SSTATE_DIR</filename>, you must
2966 specify "PATH" as part of the URI to enable the build system
2967 to map to the appropriate subdirectory.
2968 </note>
2969 </para>
2970
2971 <para>
2972 The shared state package validity can be detected just by
2973 looking at the filename since the filename contains the task
2974 checksum (or signature) as described earlier in this section.
2975 If a valid shared state package is found, the build process
2976 downloads it and uses it to accelerate the task.
2977 </para>
2978
2979 <para>
2980 The build processes use the <filename>*_setscene</filename>
2981 tasks for the task acceleration phase.
2982 BitBake goes through this phase before the main execution
2983 code and tries to accelerate any tasks for which it can find
2984 shared state packages.
2985 If a shared state package for a task is available, the
2986 shared state package is used.
2987 This means the task and any tasks on which it is dependent
2988 are not executed.
2989 </para>
2990
2991 <para>
2992 As a real world example, the aim is when building an IPK-based
2993 image, only the
2994 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_ipk'><filename>do_package_write_ipk</filename></ulink>
2995 tasks would have their shared state packages fetched and
2996 extracted.
2997 Since the sysroot is not used, it would never get extracted.
2998 This is another reason why a task-based approach is preferred
2999 over a recipe-based approach, which would have to install the
3000 output from every task.
3001 </para>
3002 </section>
3003 </section>
3004
3005 <section id='automatically-added-runtime-dependencies'>
3006 <title>Automatically Added Runtime Dependencies</title>
3007
3008 <para>
3009 The OpenEmbedded build system automatically adds common types of
3010 runtime dependencies between packages, which means that you do not
3011 need to explicitly declare the packages using
3012 <ulink url='&YOCTO_DOCS_REF_URL;#var-RDEPENDS'><filename>RDEPENDS</filename></ulink>.
3013 Three automatic mechanisms exist (<filename>shlibdeps</filename>,
3014 <filename>pcdeps</filename>, and <filename>depchains</filename>)
3015 that handle shared libraries, package configuration (pkg-config)
3016 modules, and <filename>-dev</filename> and
3017 <filename>-dbg</filename> packages, respectively.
3018 For other types of runtime dependencies, you must manually declare
3019 the dependencies.
3020 <itemizedlist>
3021 <listitem><para>
3022 <filename>shlibdeps</filename>:
3023 During the
3024 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>
3025 task of each recipe, all shared libraries installed by the
3026 recipe are located.
3027 For each shared library, the package that contains the
3028 shared library is registered as providing the shared
3029 library.
3030 More specifically, the package is registered as providing
3031 the
3032 <ulink url='https://en.wikipedia.org/wiki/Soname'>soname</ulink>
3033 of the library.
3034 The resulting shared-library-to-package mapping
3035 is saved globally in
3036 <ulink url='&YOCTO_DOCS_REF_URL;#var-PKGDATA_DIR'><filename>PKGDATA_DIR</filename></ulink>
3037 by the
3038 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-packagedata'><filename>do_packagedata</filename></ulink>
3039 task.</para>
3040
3041 <para>Simultaneously, all executables and shared libraries
3042 installed by the recipe are inspected to see what shared
3043 libraries they link against.
3044 For each shared library dependency that is found,
3045 <filename>PKGDATA_DIR</filename> is queried to
3046 see if some package (likely from a different recipe)
3047 contains the shared library.
3048 If such a package is found, a runtime dependency is added
3049 from the package that depends on the shared library to the
3050 package that contains the library.</para>
3051
3052 <para>The automatically added runtime dependency also
3053 includes a version restriction.
3054 This version restriction specifies that at least the
3055 current version of the package that provides the shared
3056 library must be used, as if
3057 "<replaceable>package</replaceable> (>= <replaceable>version</replaceable>)"
3058 had been added to <filename>RDEPENDS</filename>.
3059 This forces an upgrade of the package containing the shared
3060 library when installing the package that depends on the
3061 library, if needed.</para>
3062
3063 <para>If you want to avoid a package being registered as
3064 providing a particular shared library (e.g. because the library
3065 is for internal use only), then add the library to
3066 <ulink url='&YOCTO_DOCS_REF_URL;#var-PRIVATE_LIBS'><filename>PRIVATE_LIBS</filename></ulink>
3067 inside the package's recipe.
3068 </para></listitem>
3069 <listitem><para>
3070 <filename>pcdeps</filename>:
3071 During the <filename>do_package</filename> task of each
3072 recipe, all pkg-config modules
3073 (<filename>*.pc</filename> files) installed by the recipe
3074 are located.
3075 For each module, the package that contains the module is
3076 registered as providing the module.
3077 The resulting module-to-package mapping is saved globally in
3078 <filename>PKGDATA_DIR</filename> by the
3079 <filename>do_packagedata</filename> task.</para>
3080
3081 <para>Simultaneously, all pkg-config modules installed by
3082 the recipe are inspected to see what other pkg-config
3083 modules they depend on.
3084 A module is seen as depending on another module if it
3085 contains a "Requires:" line that specifies the other module.
3086 For each module dependency,
3087 <filename>PKGDATA_DIR</filename> is queried to see if some
3088 package contains the module.
3089 If such a package is found, a runtime dependency is added
3090 from the package that depends on the module to the package
3091 that contains the module.
3092 <note>
3093 The <filename>pcdeps</filename> mechanism most often
3094 infers dependencies between <filename>-dev</filename>
3095 packages.
3096 </note>
3097 </para></listitem>
3098 <listitem><para>
3099 <filename>depchains</filename>:
3100 If a package <filename>foo</filename> depends on a package
3101 <filename>bar</filename>, then <filename>foo-dev</filename>
3102 and <filename>foo-dbg</filename> are also made to depend on
3103 <filename>bar-dev</filename> and
3104 <filename>bar-dbg</filename>, respectively.
3105 Taking the <filename>-dev</filename> packages as an
3106 example, the <filename>bar-dev</filename> package might
3107 provide headers and shared library symlinks needed by
3108 <filename>foo-dev</filename>, which shows the need
3109 for a dependency between the packages.</para>
3110
3111 <para>The dependencies added by
3112 <filename>depchains</filename> are in the form of
3113 <ulink url='&YOCTO_DOCS_REF_URL;#var-RRECOMMENDS'><filename>RRECOMMENDS</filename></ulink>.
3114 <note>
3115 By default, <filename>foo-dev</filename> also has an
3116 <filename>RDEPENDS</filename>-style dependency on
3117 <filename>foo</filename>, because the default value of
3118 <filename>RDEPENDS_${PN}-dev</filename> (set in
3119 <filename>bitbake.conf</filename>) includes
3120 "${PN}".
3121 </note></para>
3122
3123 <para>To ensure that the dependency chain is never broken,
3124 <filename>-dev</filename> and <filename>-dbg</filename>
3125 packages are always generated by default, even if the
3126 packages turn out to be empty.
3127 See the
3128 <ulink url='&YOCTO_DOCS_REF_URL;#var-ALLOW_EMPTY'><filename>ALLOW_EMPTY</filename></ulink>
3129 variable for more information.
3130 </para></listitem>
3131 </itemizedlist>
3132 </para>
3133
3134 <para>
3135 The <filename>do_package</filename> task depends on the
3136 <filename>do_packagedata</filename> task of each recipe in
3137 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPENDS'><filename>DEPENDS</filename></ulink>
3138 through use of a
3139 <filename>[</filename><ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'><filename>deptask</filename></ulink><filename>]</filename>
3140 declaration, which guarantees that the required
3141 shared-library/module-to-package mapping information will be available
3142 when needed as long as <filename>DEPENDS</filename> has been
3143 correctly set.
3144 </para>
3145 </section>
3146
3147 <section id='fakeroot-and-pseudo'>
3148 <title>Fakeroot and Pseudo</title>
3149
3150 <para>
3151 Some tasks are easier to implement when allowed to perform certain
3152 operations that are normally reserved for the root user (e.g.
3153 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink>,
3154 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb'><filename>do_package_write*</filename></ulink>,
3155 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-rootfs'><filename>do_rootfs</filename></ulink>,
3156 and
3157 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-image'><filename>do_image*</filename></ulink>).
3158 For example, the <filename>do_install</filename> task benefits
3159 from being able to set the UID and GID of installed files to
3160 arbitrary values.
3161 </para>
3162
3163 <para>
3164 One approach to allowing tasks to perform root-only operations
3165 would be to require
3166 <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink>
3167 to run as root.
3168 However, this method is cumbersome and has security issues.
3169 The approach that is actually used is to run tasks that benefit
3170 from root privileges in a "fake" root environment.
3171 Within this environment, the task and its child processes believe
3172 that they are running as the root user, and see an internally
3173 consistent view of the filesystem.
3174 As long as generating the final output (e.g. a package or an image)
3175 does not require root privileges, the fact that some earlier
3176 steps ran in a fake root environment does not cause problems.
3177 </para>
3178
3179 <para>
3180 The capability to run tasks in a fake root environment is known as
3181 "<ulink url='http://man.he.net/man1/fakeroot'>fakeroot</ulink>",
3182 which is derived from the BitBake keyword/variable
3183 flag that requests a fake root environment for a task.
3184 </para>
3185
3186 <para>
3187 In the
3188 <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink>,
3189 the program that implements fakeroot is known as Pseudo.
3190 Pseudo overrides system calls by using the environment variable
3191 <filename>LD_PRELOAD</filename>, which results in the illusion
3192 of running as root.
3193 To keep track of "fake" file ownership and permissions resulting
3194 from operations that require root permissions, Pseudo uses
3195 an SQLite 3 database.
3196 This database is stored in
3197 <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink><filename>}/pseudo/files.db</filename>
3198 for individual recipes.
3199 Storing the database in a file as opposed to in memory
3200 gives persistence between tasks and builds, which is not
3201 accomplished using fakeroot.
3202 <note><title>Caution</title>
3203 If you add your own task that manipulates the same files or
3204 directories as a fakeroot task, then that task also needs to
3205 run under fakeroot.
3206 Otherwise, the task cannot run root-only operations, and
3207 cannot see the fake file ownership and permissions set by the
3208 other task.
3209 You need to also add a dependency on
3210 <filename>virtual/fakeroot-native:do_populate_sysroot</filename>,
3211 giving the following:
3212 <literallayout class='monospaced'>
3213 fakeroot do_mytask () {
3214 ...
3215 }
3216 do_mytask[depends] += "virtual/fakeroot-native:do_populate_sysroot"
3217 </literallayout>
3218 </note>
3219 For more information, see the
3220 <ulink url='&YOCTO_DOCS_BB_URL;#var-FAKEROOT'><filename>FAKEROOT*</filename></ulink>
3221 variables in the BitBake User Manual.
3222 You can also reference the
3223 "<ulink url='http://www.ibm.com/developerworks/opensource/library/os-aapseudo1/index.html'>Pseudo</ulink>"
3224 and
3225 "<ulink url='https://github.com/wrpseudo/pseudo/wiki/WhyNotFakeroot'>Why Not Fakeroot?</ulink>"
3226 articles for background information on Pseudo.
3227 </para>
3228 </section>
3229</chapter>
3230<!--
3231vim: expandtab tw=80 ts=4
3232-->