blob: d08031617b1fa1b08502c8df950ce12d97b2c6ae [file] [log] [blame]
Patrick Williamsc124f4f2015-09-15 14:41:29 -05001<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
2"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
3[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
4
5<chapter id='usingpoky'>
6<title>Using the Yocto Project</title>
7
8 <para>
9 This chapter describes common usage for the Yocto Project.
10 The information is introductory in nature as other manuals in the Yocto Project
11 documentation set provide more details on how to use the Yocto Project.
12 </para>
13
14<section id='usingpoky-build'>
15 <title>Running a Build</title>
16
17 <para>
18 This section provides a summary of the build process and provides information
19 for less obvious aspects of the build process.
20 For general information on how to build an image using the OpenEmbedded build
21 system, see the
22 "<ulink url='&YOCTO_DOCS_QS_URL;#qs-building-images'>Building Images</ulink>"
23 section of the Yocto Project Quick Start.
24 </para>
25
26 <section id='build-overview'>
27 <title>Build Overview</title>
28
29 <para>
30 In the development environment you will need to build an image whenever you change hardware
31 support, add or change system libraries, or add or change services that have dependencies.
32 </para>
33
34 <mediaobject>
35 <imageobject>
36 <imagedata fileref="figures/building-an-image.png" format="PNG" align='center' scalefit='1'/>
37 </imageobject>
38 <caption>
39 <para>Building an Image</para>
40 </caption>
41 </mediaobject>
42
43 <para>
44 The first thing you need to do is set up the OpenEmbedded build
45 environment by sourcing an environment setup script
46 (i.e.
47 <link linkend='structure-core-script'><filename>&OE_INIT_FILE;</filename></link>
48 or
49 <link linkend='structure-memres-core-script'><filename>oe-init-build-env-memres</filename></link>).
50 Here is an example:
51 <literallayout class='monospaced'>
52 $ source &OE_INIT_FILE; [<replaceable>build_dir</replaceable>]
53 </literallayout>
54 </para>
55
56 <para>
57 The <replaceable>build_dir</replaceable> argument is optional and specifies the directory the
58 OpenEmbedded build system uses for the build -
59 the <ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink>.
60 If you do not specify a Build Directory, it defaults to a directory
61 named <filename>build</filename> in your current working directory.
62 A common practice is to use a different Build Directory for different targets.
63 For example, <filename>~/build/x86</filename> for a <filename>qemux86</filename>
64 target, and <filename>~/build/arm</filename> for a <filename>qemuarm</filename> target.
65 </para>
66
67 <para>
68 Once the build environment is set up, you can build a target using:
69 <literallayout class='monospaced'>
70 $ bitbake <replaceable>target</replaceable>
71 </literallayout>
Brad Bishop6e60e8b2018-02-01 10:27:11 -050072 <note>
73 <para>
74 If you experience a build error due to resources
75 temporarily being unavailable and it appears you
76 should not be having this issue, it might be due
77 to the combination of a 4.3+ Linux kernel and
78 <filename>systemd</filename> version 228+
79 (i.e. see this
80 <ulink url='http://unix.stackexchange.com/questions/253903/creating-threads-fails-with-resource-temporarily-unavailable-with-4-3-kernel'>link</ulink>
81 for information).
82 </para>
83
84 <para>
85 To work around this issue, you can try either
86 of the following:
87 <itemizedlist>
88 <listitem><para>
89 Try the build again.
90 </para></listitem>
91 <listitem><para>
92 Modify the "DefaultTasksMax"
93 <filename>systemd</filename> parameter
94 by uncommenting it and setting it to
95 "infinity".
96 You can find this parameter in the
97 <filename>system.conf</filename> file
98 located in
99 <filename>/etc/systemd</filename>
100 on most systems.
101 </para></listitem>
102 </itemizedlist>
103 </para>
104 </note>
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500105 </para>
106
107 <para>
108 The <replaceable>target</replaceable> is the name of the recipe you want to build.
109 Common targets are the images in <filename>meta/recipes-core/images</filename>,
110 <filename>meta/recipes-sato/images</filename>, etc. all found in the
111 <ulink url='&YOCTO_DOCS_DEV_URL;#source-directory'>Source Directory</ulink>.
112 Or, the target can be the name of a recipe for a specific piece of software such as
113 BusyBox.
114 For more details about the images the OpenEmbedded build system supports, see the
115 "<link linkend="ref-images">Images</link>" chapter.
116 </para>
117
118 <note>
119 Building an image without GNU General Public License Version
120 3 (GPLv3), or similarly licensed, components is supported for
121 only minimal and base images.
122 See the "<link linkend='ref-images'>Images</link>" chapter for more information.
123 </note>
124 </section>
125
126 <section id='building-an-image-using-gpl-components'>
127 <title>Building an Image Using GPL Components</title>
128
129 <para>
130 When building an image using GPL components, you need to maintain your original
131 settings and not switch back and forth applying different versions of the GNU
132 General Public License.
133 If you rebuild using different versions of GPL, dependency errors might occur
134 due to some components not being rebuilt.
135 </para>
136 </section>
137</section>
138
139<section id='usingpoky-install'>
140 <title>Installing and Using the Result</title>
141
142 <para>
143 Once an image has been built, it often needs to be installed.
144 The images and kernels built by the OpenEmbedded build system are placed in the
145 <ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink> in
146 <filename class="directory">tmp/deploy/images</filename>.
147 For information on how to run pre-built images such as <filename>qemux86</filename>
148 and <filename>qemuarm</filename>, see the
Patrick Williamsd8c66bc2016-06-20 12:57:21 -0500149 <ulink url='&YOCTO_DOCS_SDK_URL;#sdk-manual'>Yocto Project Software Development Kit (SDK) Developer's Guide</ulink>.
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500150 For information about how to install these images, see the documentation for your
151 particular board or machine.
152 </para>
153</section>
154
Patrick Williamsc0f7c042017-02-23 20:41:17 -0600155<section id='usingpoky-debugging-tools-and-techniques'>
156 <title>Debugging Tools and Techniques</title>
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500157
158 <para>
159 The exact method for debugging build failures depends on the nature of
160 the problem and on the system's area from which the bug originates.
161 Standard debugging practices such as comparison against the last
162 known working version with examination of the changes and the
163 re-application of steps to identify the one causing the problem are
164 valid for the Yocto Project just as they are for any other system.
165 Even though it is impossible to detail every possible potential failure,
166 this section provides some general tips to aid in debugging.
167 </para>
168
169 <para>
170 A useful feature for debugging is the error reporting tool.
171 Configuring the Yocto Project to use this tool causes the
172 OpenEmbedded build system to produce error reporting commands as
173 part of the console output.
174 You can enter the commands after the build completes
175 to log error information
176 into a common database, that can help you figure out what might be
177 going wrong.
178 For information on how to enable and use this feature, see the
179 "<ulink url='&YOCTO_DOCS_DEV_URL;#using-the-error-reporting-tool'>Using the Error Reporting Tool</ulink>"
180 section in the Yocto Project Development Manual.
181 </para>
182
183 <para>
184 For discussions on debugging, see the
Patrick Williamsd8c66bc2016-06-20 12:57:21 -0500185 "<ulink url='&YOCTO_DOCS_DEV_URL;#platdev-gdb-remotedebug'>Debugging With the GNU Project Debugger (GDB) Remotely</ulink>" section
186 in the Yocto Project Developer's Manual
187 and the
188 "<ulink url='&YOCTO_DOCS_SDK_URL;#adt-eclipse'>Working within Eclipse</ulink>"
189 section in the Yocto Project Software Development Kit (SDK) Developer's Guide.
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500190 </para>
191
192 <note>
193 The remainder of this section presents many examples of the
194 <filename>bitbake</filename> command.
195 You can learn about BitBake by reading the
196 <ulink url='&YOCTO_DOCS_BB_URL;#bitbake-user-manual'>BitBake User Manual</ulink>.
197 </note>
198
Patrick Williamsc0f7c042017-02-23 20:41:17 -0600199 <section id='usingpoky-debugging-viewing-logs-from-failed-tasks'>
200 <title>Viewing Logs from Failed Tasks</title>
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500201
Patrick Williamsc0f7c042017-02-23 20:41:17 -0600202 <para>
203 You can find the log for a task in the file
204 <filename>${</filename><link linkend='var-WORKDIR'><filename>WORKDIR</filename></link><filename>}/temp/log.do_</filename><replaceable>taskname</replaceable>.
205 For example, the log for the
206 <link linkend='ref-tasks-compile'><filename>do_compile</filename></link>
207 task of the QEMU minimal image for the x86 machine
208 (<filename>qemux86</filename>) might be in
209 <filename>tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/temp/log.do_compile</filename>.
210 To see the commands
211 <ulink url='&YOCTO_DOCS_DEV_URL;#bitbake-term'>BitBake</ulink> ran
212 to generate a log, look at the corresponding
213 <filename>run.do_</filename><replaceable>taskname</replaceable>
214 file in the same directory.
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500215 </para>
216
217 <para>
Patrick Williamsc0f7c042017-02-23 20:41:17 -0600218 <filename>log.do_</filename><replaceable>taskname</replaceable> and
219 <filename>run.do_</filename><replaceable>taskname</replaceable>
220 are actually symbolic links to
221 <filename>log.do_</filename><replaceable>taskname</replaceable><filename>.</filename><replaceable>pid</replaceable>
222 and
223 <filename>log.run_</filename><replaceable>taskname</replaceable><filename>.</filename><replaceable>pid</replaceable>,
224 where <replaceable>pid</replaceable> is the PID the task had when
225 it ran.
226 The symlinks always point to the files corresponding to the most
227 recent run.
228 </para>
229 </section>
230
231 <section id='usingpoky-debugging-viewing-variable-values'>
232 <title>Viewing Variable Values</title>
233 <para>
234 BitBake's <filename>-e</filename> option is used to display
235 variable values after parsing.
236 The following command displays the variable values after the
237 configuration files (i.e. <filename>local.conf</filename>,
238 <filename>bblayers.conf</filename>,
239 <filename>bitbake.conf</filename> and so forth) have been
240 parsed:
241 <literallayout class='monospaced'>
242 $ bitbake -e
243 </literallayout>
244 The following command displays variable values after a specific
245 recipe has been parsed.
246 The variables include those from the configuration as well:
247 <literallayout class='monospaced'>
248 $ bitbake -e recipename
249 </literallayout>
250 <note><para>
251 Each recipe has its own private set of variables (datastore).
252 Internally, after parsing the configuration, a copy of the
253 resulting datastore is made prior to parsing each recipe.
254 This copying implies that variables set in one recipe will
255 not be visible to other recipes.</para>
256
257 <para>Likewise, each task within a recipe gets a private
258 datastore based on the recipe datastore, which means that
259 variables set within one task will not be visible to
260 other tasks.</para>
261 </note>
262 </para>
263
264 <para>
265 In the output of <filename>bitbake -e</filename>, each variable is
266 preceded by a description of how the variable got its value,
267 including temporary values that were later overriden.
268 This description also includes variable flags (varflags) set on
269 the variable.
270 The output can be very helpful during debugging.
271 </para>
272
273 <para>
274 Variables that are exported to the environment are preceded by
275 <filename>export</filename> in the output of
276 <filename>bitbake -e</filename>.
277 See the following example:
278 <literallayout class='monospaced'>
279 export CC="i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/ulf/poky/build/tmp/sysroots/qemux86"
280 </literallayout>
281 </para>
282
283 <para>
284 In addition to variable values, the output of the
285 <filename>bitbake -e</filename> and
286 <filename>bitbake -e</filename>&nbsp;<replaceable>recipe</replaceable>
287 commands includes the following information:
288 <itemizedlist>
289 <listitem><para>
290 The output starts with a tree listing all configuration
291 files and classes included globally, recursively listing
292 the files they include or inherit in turn.
293 Much of the behavior of the OpenEmbedded build system
294 (including the behavior of the
295 <link linkend='normal-recipe-build-tasks'>normal recipe build tasks</link>)
296 is implemented in the
297 <link linkend='ref-classes-base'><filename>base</filename></link>
298 class and the classes it inherits, rather than being built
299 into BitBake itself.
300 </para></listitem>
301 <listitem><para>
302 After the variable values, all functions appear in the
303 output.
304 For shell functions, variables referenced within the
305 function body are expanded.
306 If a function has been modified using overrides or
307 using override-style operators like
308 <filename>_append</filename> and
309 <filename>_prepend</filename>, then the final assembled
310 function body appears in the output.
311 </para></listitem>
312 </itemizedlist>
313 </para>
314 </section>
315
316 <section id='viewing-package-information-with-oe-pkgdata-util'>
317 <title>Viewing Package Information with <filename>oe-pkgdata-util</filename></title>
318
319 <para>
320 You can use the <filename>oe-pkgdata-util</filename> command-line
321 utility to query
322 <link linkend='var-PKGDATA_DIR'><filename>PKGDATA_DIR</filename></link>
323 and display various package-related information.
324 When you use the utility, you must use it to view information
325 on packages that have already been built.
326 </para>
327
328 <para>
329 Following are a few of the available
330 <filename>oe-pkgdata-util</filename> subcommands.
331 <note>
332 You can use the standard * and ? globbing wildcards as part of
333 package names and paths.
334 </note>
335 <itemizedlist>
336 <listitem><para>
337 <filename>oe-pkgdata-util list-pkgs [</filename><replaceable>pattern</replaceable><filename>]</filename>:
338 Lists all packages that have been built, optionally
339 limiting the match to packages that match
340 <replaceable>pattern</replaceable>.
341 </para></listitem>
342 <listitem><para>
343 <filename>oe-pkgdata-util list-pkg-files&nbsp;</filename><replaceable>package</replaceable><filename>&nbsp;...</filename>:
344 Lists the files and directories contained in the given
345 packages.
346 <note>
347 <para>
348 A different way to view the contents of a package is
349 to look at the
350 <filename>${</filename><link linkend='var-WORKDIR'><filename>WORKDIR</filename></link><filename>}/packages-split</filename>
351 directory of the recipe that generates the
352 package.
353 This directory is created by the
354 <link linkend='ref-tasks-package'><filename>do_package</filename></link>
355 task and has one subdirectory for each package the
356 recipe generates, which contains the files stored in
357 that package.</para>
358 <para>
359 If you want to inspect the
360 <filename>${WORKDIR}/packages-split</filename>
361 directory, make sure that
362 <link linkend='ref-classes-rm-work'><filename>rm_work</filename></link>
363 is not enabled when you build the recipe.
364 </para>
365 </note>
366 </para></listitem>
367 <listitem><para>
368 <filename>oe-pkgdata-util find-path&nbsp;</filename><replaceable>path</replaceable><filename>&nbsp;...</filename>:
369 Lists the names of the packages that contain the given
370 paths.
371 For example, the following tells us that
372 <filename>/usr/share/man/man1/make.1</filename>
373 is contained in the <filename>make-doc</filename>
374 package:
375 <literallayout class='monospaced'>
376 $ oe-pkgdata-util find-path /usr/share/man/man1/make.1
377 make-doc: /usr/share/man/man1/make.1
378 </literallayout>
379 </para></listitem>
380 <listitem><para>
381 <filename>oe-pkgdata-util lookup-recipe&nbsp;</filename><replaceable>package</replaceable><filename>&nbsp;...</filename>:
382 Lists the name of the recipes that
383 produce the given packages.
384 </para></listitem>
385 </itemizedlist>
386 </para>
387
388 <para>
389 For more information on the <filename>oe-pkgdata-util</filename>
390 command, use the help facility:
391 <literallayout class='monospaced'>
392 $ oe-pkgdata-util &dash;&dash;help
393 $ oe-pkgdata-util <replaceable>subcommand</replaceable> --help
394 </literallayout>
395 </para>
396 </section>
397
398 <section id='usingpoky-viewing-dependencies-between-recipes-and-tasks'>
399 <title>Viewing Dependencies Between Recipes and Tasks</title>
400
401 <para>
402 Sometimes it can be hard to see why BitBake wants to build other
403 recipes before the one you have specified.
404 Dependency information can help you understand why a recipe is
405 built.
406 </para>
407
408 <para>
409 To generate dependency information for a recipe, run the following
410 command:
411 <literallayout class='monospaced'>
412 $ bitbake -g <replaceable>recipename</replaceable>
413 </literallayout>
414 This command writes the following files in the current directory:
415 <itemizedlist>
416 <listitem><para>
417 <filename>pn-buildlist</filename>: A list of
418 recipes/targets involved in building
419 <replaceable>recipename</replaceable>.
420 "Involved" here means that at least one task from the
421 recipe needs to run when building
422 <replaceable>recipename</replaceable> from scratch.
423 Targets that are in
424 <link linkend='var-ASSUME_PROVIDED'><filename>ASSUME_PROVIDED</filename></link>
425 are not listed.
426 </para></listitem>
427 <listitem><para>
Patrick Williamsc0f7c042017-02-23 20:41:17 -0600428 <filename>task-depends.dot</filename>: A graph showing
429 dependencies between tasks.
430 </para></listitem>
431 </itemizedlist>
432 </para>
433
434 <para>
435 The graphs are in
436 <ulink url='https://en.wikipedia.org/wiki/DOT_%28graph_description_language%29'>DOT</ulink>
437 format and can be converted to images (e.g. using the
438 <filename>dot</filename> tool from
439 <ulink url='http://www.graphviz.org/'>Graphviz</ulink>).
440 <note><title>Notes</title>
441 <itemizedlist>
442 <listitem><para>
443 DOT files use a plain text format.
444 The graphs generated using the
445 <filename>bitbake -g</filename> command are often so
446 large as to be difficult to read without special
447 pruning (e.g. with Bitbake's
448 <filename>-I</filename> option) and processing.
449 Despite the form and size of the graphs, the
450 corresponding <filename>.dot</filename> files can still
451 be possible to read and provide useful information.
452 </para>
453
454 <para>As an example, the
455 <filename>task-depends.dot</filename> file contains
456 lines such as the following:
457 <literallayout class='monospaced'>
458 "libxslt.do_configure" -> "libxml2.do_populate_sysroot"
459 </literallayout>
460 The above example line reveals that the
461 <link linkend='ref-tasks-configure'><filename>do_configure</filename></link>
462 task in <filename>libxslt</filename> depends on the
463 <link linkend='ref-tasks-populate_sysroot'><filename>do_populate_sysroot</filename></link>
464 task in <filename>libxml2</filename>, which is a normal
465 <link linkend='var-DEPENDS'><filename>DEPENDS</filename></link>
466 dependency between the two recipes.
467 </para></listitem>
468 <listitem><para>
469 For an example of how <filename>.dot</filename> files
470 can be processed, see the
471 <filename>scripts/contrib/graph-tool</filename> Python
472 script, which finds and displays paths between graph
473 nodes.
474 </para></listitem>
475 </itemizedlist>
476 </note>
477 </para>
478
479 <para>
480 You can use a different method to view dependency information
481 by using the following command:
482 <literallayout class='monospaced'>
Brad Bishop6e60e8b2018-02-01 10:27:11 -0500483 $ bitbake -g -u taskexp <replaceable>recipename</replaceable>
Patrick Williamsc0f7c042017-02-23 20:41:17 -0600484 </literallayout>
485 This command displays a GUI window from which you can view
486 build-time and runtime dependencies for the recipes involved in
487 building <replaceable>recipename</replaceable>.
488 </para>
489 </section>
490
491 <section id='usingpoky-viewing-task-variable-dependencies'>
492 <title>Viewing Task Variable Dependencies</title>
493
494 <para>
495 As mentioned in the
496 "<ulink url='&YOCTO_DOCS_BB_URL;#checksums'>Checksums (Signatures)</ulink>"
497 section of the BitBake User Manual, BitBake tries to automatically
498 determine what variables a task depends on so that it can rerun
499 the task if any values of the variables change.
500 This determination is usually reliable.
501 However, if you do things like construct variable names at runtime,
502 then you might have to manually declare dependencies on those
503 variables using <filename>vardeps</filename> as described in the
504 "<ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'>Variable Flags</ulink>"
505 section of the BitBake User Manual.
506 </para>
507
508 <para>
509 If you are unsure whether a variable dependency is being picked up
510 automatically for a given task, you can list the variable
511 dependencies BitBake has determined by doing the following:
512 <orderedlist>
513 <listitem><para>
514 Build the recipe containing the task:
515 <literallayout class='monospaced'>
516 $ bitbake <replaceable>recipename</replaceable>
517 </literallayout>
518 </para></listitem>
519 <listitem><para>
520 Inside the
521 <link linkend='var-STAMPS_DIR'><filename>STAMPS_DIR</filename></link>
522 directory, find the signature data
523 (<filename>sigdata</filename>) file that corresponds to the
524 task.
525 The <filename>sigdata</filename> files contain a pickled
526 Python database of all the metadata that went into creating
527 the input checksum for the task.
528 As an example, for the
529 <link linkend='ref-tasks-fetch'><filename>do_fetch</filename></link>
530 task of the <filename>db</filename> recipe, the
531 <filename>sigdata</filename> file might be found in the
532 following location:
533 <literallayout class='monospaced'>
534 ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1
535 </literallayout>
536 For tasks that are accelerated through the shared state
537 (<link linkend='shared-state-cache'>sstate</link>)
538 cache, an additional <filename>siginfo</filename> file is
539 written into
540 <link linkend='var-SSTATE_DIR'><filename>SSTATE_DIR</filename></link>
541 along with the cached task output.
542 The <filename>siginfo</filename> files contain exactly the
543 same information as <filename>sigdata</filename> files.
544 </para></listitem>
545 <listitem><para>
546 Run <filename>bitbake-dumpsig</filename> on the
547 <filename>sigdata</filename> or
548 <filename>siginfo</filename> file.
549 Here is an example:
550 <literallayout class='monospaced'>
551 $ bitbake-dumpsig ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1
552 </literallayout>
553 In the output of the above command, you will find a line
554 like the following, which lists all the (inferred) variable
555 dependencies for the task.
556 This list also includes indirect dependencies from
557 variables depending on other variables, recursively.
558 <literallayout class='monospaced'>
559 Task dependencies: ['PV', 'SRCREV', 'SRC_URI', 'SRC_URI[md5sum]', 'SRC_URI[sha256sum]', 'base_do_fetch']
560 </literallayout>
561 <note>
562 Functions (e.g. <filename>base_do_fetch</filename>)
563 also count as variable dependencies.
564 These functions in turn depend on the variables they
565 reference.
566 </note>
567 The output of <filename>bitbake-dumpsig</filename> also includes
568 the value each variable had, a list of dependencies for each
569 variable, and
570 <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_HASHBASE_WHITELIST'><filename>BB_HASHBASE_WHITELIST</filename></ulink>
571 information.
572 </para></listitem>
573 </orderedlist>
574 </para>
575
576 <para>
577 There is also a <filename>bitbake-diffsigs</filename> command for
578 comparing two <filename>siginfo</filename> or
579 <filename>sigdata</filename> files.
580 This command can be helpful when trying to figure out what changed
581 between two versions of a task.
582 If you call <filename>bitbake-diffsigs</filename> with just one
583 file, the command behaves like
584 <filename>bitbake-dumpsig</filename>.
585 </para>
586
587 <para>
588 You can also use BitBake to dump out the signature construction
589 information without executing tasks by using either of the
590 following BitBake command-line options:
591 <literallayout class='monospaced'>
592 &dash;&dash;dump-signatures=<replaceable>SIGNATURE_HANDLER</replaceable>
593 -S <replaceable>SIGNATURE_HANDLER</replaceable>
594 </literallayout>
595 <note>
596 Two common values for
597 <replaceable>SIGNATURE_HANDLER</replaceable> are "none" and
598 "printdiff", which dump only the signature or compare the
599 dumped signature with the cached one, respectively.
600 </note>
601 Using BitBake with either of these options causes BitBake to dump
602 out <filename>sigdata</filename> files in the
603 <filename>stamps</filename> directory for every task it would have
604 executed instead of building the specified target package.
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500605 </para>
606 </section>
607
608 <section id='usingpoky-debugging-taskrunning'>
609 <title>Running Specific Tasks</title>
610
611 <para>
Patrick Williamsc0f7c042017-02-23 20:41:17 -0600612 Any given recipe consists of a set of tasks.
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500613 The standard BitBake behavior in most cases is:
614 <filename>do_fetch</filename>,
615 <filename>do_unpack</filename>,
616 <filename>do_patch</filename>, <filename>do_configure</filename>,
617 <filename>do_compile</filename>, <filename>do_install</filename>,
618 <filename>do_package</filename>,
619 <filename>do_package_write_*</filename>, and
620 <filename>do_build</filename>.
621 The default task is <filename>do_build</filename> and any tasks
622 on which it depends build first.
623 Some tasks, such as <filename>do_devshell</filename>, are not part
624 of the default build chain.
625 If you wish to run a task that is not part of the default build
626 chain, you can use the <filename>-c</filename> option in BitBake.
627 Here is an example:
628 <literallayout class='monospaced'>
629 $ bitbake matchbox-desktop -c devshell
630 </literallayout>
631 </para>
632
633 <para>
Patrick Williamsc0f7c042017-02-23 20:41:17 -0600634 The <filename>-c</filename> option respects task dependencies,
635 which means that all other tasks (including tasks from other
636 recipes) that the specified task depends on will be run before the
637 task.
638 Even when you manually specify a task to run with
639 <filename>-c</filename>, BitBake will only run the task if it
640 considers it "out of date".
641 See the
642 "<link linkend='stamp-files-and-the-rerunning-of-tasks'>Stamp Files and the Rerunning of Tasks</link>"
643 section for how BitBake determines whether a task is "out of date".
644 </para>
645
646 <para>
647 If you want to force an up-to-date task to be rerun (e.g.
648 because you made manual modifications to the recipe's
649 <link linkend='var-WORKDIR'><filename>WORKDIR</filename></link>
650 that you want to try out), then you can use the
651 <filename>-f</filename> option.
652 <note>
653 The reason <filename>-f</filename> is never required when
654 running the
655 <link linkend='ref-tasks-devshell'><filename>do_devshell</filename></link>
656 task is because the
657 <filename>[</filename><ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'><filename>nostamp</filename></ulink><filename>]</filename>
658 variable flag is already set for the task.
659 </note>
660 The following example shows one way you can use the
661 <filename>-f</filename> option:
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500662 <literallayout class='monospaced'>
663 $ bitbake matchbox-desktop
664 .
665 .
Patrick Williamsc0f7c042017-02-23 20:41:17 -0600666 make some changes to the source code in the work directory
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500667 .
668 .
669 $ bitbake matchbox-desktop -c compile -f
670 $ bitbake matchbox-desktop
671 </literallayout>
672 </para>
673
674 <para>
675 This sequence first builds and then recompiles
676 <filename>matchbox-desktop</filename>.
677 The last command reruns all tasks (basically the packaging tasks)
678 after the compile.
679 BitBake recognizes that the <filename>do_compile</filename>
680 task was rerun and therefore understands that the other tasks
681 also need to be run again.
682 </para>
683
684 <para>
Patrick Williamsc0f7c042017-02-23 20:41:17 -0600685 Another, shorter way to rerun a task and all
686 <link linkend='normal-recipe-build-tasks'>normal recipe build tasks</link>
687 that depend on it is to use the <filename>-C</filename>
688 option.
689 <note>
690 This option is upper-cased and is separate from the
691 <filename>-c</filename> option, which is lower-cased.
692 </note>
693 Using this option invalidates the given task and then runs the
694 <link linkend='ref-tasks-build'><filename>do_build</filename></link>
695 task, which is the default task if no task is given, and the
696 tasks on which it depends.
697 You could replace the final two commands in the previous example
698 with the following single command:
699 <literallayout class='monospaced'>
700 $ bitbake matchbox-desktop -C compile
701 </literallayout>
702 Internally, the <filename>-f</filename> and
703 <filename>-C</filename> options work by tainting (modifying) the
704 input checksum of the specified task.
705 This tainting indirectly causes the task and its
706 dependent tasks to be rerun through the normal task dependency
707 mechanisms.
708 <note>
709 BitBake explicitly keeps track of which tasks have been
710 tainted in this fashion, and will print warnings such as the
711 following for builds involving such tasks:
712 <literallayout class='monospaced'>
713 WARNING: /home/ulf/poky/meta/recipes-sato/matchbox-desktop/matchbox-desktop_2.1.bb.do_compile is tainted from a forced run
714 </literallayout>
715 The purpose of the warning is to let you know that the work
716 directory and build output might not be in the clean state they
717 would be in for a "normal" build, depending on what actions
718 you took.
719 To get rid of such warnings, you can remove the work directory
720 and rebuild the recipe, as follows:
721 <literallayout class='monospaced'>
722 $ bitbake matchbox-desktop -c clean
723 $ bitbake matchbox-desktop
724 </literallayout>
725 </note>
726 </para>
727
728 <para>
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500729 You can view a list of tasks in a given package by running the
730 <filename>do_listtasks</filename> task as follows:
731 <literallayout class='monospaced'>
732 $ bitbake matchbox-desktop -c listtasks
733 </literallayout>
734 The results appear as output to the console and are also in the
735 file <filename>${WORKDIR}/temp/log.do_listtasks</filename>.
736 </para>
737 </section>
738
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500739 <section id='usingpoky-debugging-bitbake'>
740 <title>General BitBake Problems</title>
741
742 <para>
743 You can see debug output from BitBake by using the <filename>-D</filename> option.
744 The debug output gives more information about what BitBake
745 is doing and the reason behind it.
746 Each <filename>-D</filename> option you use increases the logging level.
747 The most common usage is <filename>-DDD</filename>.
748 </para>
749
750 <para>
751 The output from <filename>bitbake -DDD -v</filename> <replaceable>targetname</replaceable> can reveal why
752 BitBake chose a certain version of a package or why BitBake
753 picked a certain provider.
754 This command could also help you in a situation where you think BitBake did something
755 unexpected.
756 </para>
757 </section>
758
759 <section id='development-host-system-issues'>
760 <title>Development Host System Issues</title>
761
762 <para>
763 Sometimes issues on the host development system can cause your
764 build to fail.
765 Following are known, host-specific problems.
766 Be sure to always consult the
767 <ulink url='&YOCTO_RELEASE_NOTES;'>Release Notes</ulink>
768 for a look at all release-related issues.
769 <itemizedlist>
770 <listitem><para><emphasis><filename>glibc-initial</filename> fails to build</emphasis>:
771 If your development host system has the unpatched
772 <filename>GNU Make 3.82</filename>,
773 the
774 <link linkend='ref-tasks-install'><filename>do_install</filename></link>
775 task fails for <filename>glibc-initial</filename> during
776 the build.</para>
777 <para>Typically, every distribution that ships
778 <filename>GNU Make 3.82</filename> as
779 the default already has the patched version.
780 However, some distributions, such as Debian, have
781 <filename>GNU Make 3.82</filename> as an option, which
782 is unpatched.
783 You will see this error on these types of distributions.
784 Switch to <filename>GNU Make 3.81</filename> or patch
785 your <filename>make</filename> to solve the problem.
786 </para></listitem>
787 </itemizedlist>
788 </para>
789 </section>
790
791 <section id='usingpoky-debugging-buildfile'>
792 <title>Building with No Dependencies</title>
793 <para>
794 To build a specific recipe (<filename>.bb</filename> file),
795 you can use the following command form:
796 <literallayout class='monospaced'>
797 $ bitbake -b <replaceable>somepath</replaceable>/<replaceable>somerecipe</replaceable>.bb
798 </literallayout>
799 This command form does not check for dependencies.
800 Consequently, you should use it
801 only when you know existing dependencies have been met.
802 <note>
803 You can also specify fragments of the filename.
804 In this case, BitBake checks for a unique match.
805 </note>
806 </para>
807 </section>
808
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500809 <section id='recipe-logging-mechanisms'>
810 <title>Recipe Logging Mechanisms</title>
811 <para>
Patrick Williamsc0f7c042017-02-23 20:41:17 -0600812 The Yocto Project provides several logging functions for producing
813 debugging output and reporting errors and warnings.
814 For Python functions, the following logging functions exist.
815 All of these functions log to
816 <filename>${T}/log.do_</filename><replaceable>task</replaceable>,
817 and can also log to standard output (stdout) with the right
818 settings:
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500819 <itemizedlist>
Patrick Williamsc0f7c042017-02-23 20:41:17 -0600820 <listitem><para>
821 <filename>bb.plain(</filename><replaceable>msg</replaceable><filename>)</filename>:
822 Writes <replaceable>msg</replaceable> as is to the log while
823 also logging to stdout.
824 </para></listitem>
825 <listitem><para>
826 <filename>bb.note(</filename><replaceable>msg</replaceable><filename>)</filename>:
827 Writes "NOTE: <replaceable>msg</replaceable>" to the log.
828 Also logs to stdout if BitBake is called with "-v".
829 </para></listitem>
830 <listitem><para>
831 <filename>bb.debug(</filename><replaceable>level</replaceable><filename>,&nbsp;</filename><replaceable>msg</replaceable><filename>)</filename>:
832 Writes "DEBUG: <replaceable>msg</replaceable>" to the log.
833 Also logs to stdout if the log level is greater than or
834 equal to <replaceable>level</replaceable>.
835 See the
836 "<ulink url='&YOCTO_DOCS_BB_URL;#usage-and-syntax'>-D</ulink>"
837 option in the BitBake User Manual for more information.
838 </para></listitem>
839 <listitem><para>
840 <filename>bb.warn(</filename><replaceable>msg</replaceable><filename>)</filename>:
841 Writes "WARNING: <replaceable>msg</replaceable>" to the log
842 while also logging to stdout.
843 </para></listitem>
844 <listitem><para>
845 <filename>bb.error(</filename><replaceable>msg</replaceable><filename>)</filename>:
846 Writes "ERROR: <replaceable>msg</replaceable>" to the log
847 while also logging to stdout.
848 <note>
849 Calling this function does not cause the task to fail.
850 </note>
851 </para></listitem>
852 <listitem><para>
853 <filename>bb.fatal(</filename><replaceable>msg</replaceable><filename>)</filename>:
854 This logging function is similar to
855 <filename>bb.error(</filename><replaceable>msg</replaceable><filename>)</filename>
856 but also causes the calling task to fail.
857 <note>
858 <filename>bb.fatal()</filename> raises an exception,
859 which means you do not need to put a "return"
860 statement after the function.
861 </note>
862 </para></listitem>
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500863 </itemizedlist>
864 </para>
865
866 <para>
Patrick Williamsc0f7c042017-02-23 20:41:17 -0600867 The same logging functions are also available in shell functions,
868 under the names
869 <filename>bbplain</filename>, <filename>bbnote</filename>,
870 <filename>bbdebug</filename>, <filename>bbwarn</filename>,
871 <filename>bberror</filename>, and <filename>bbfatal</filename>.
872 The
873 <link linkend='ref-classes-logging'><filename>logging</filename></link>
874 class implements these functions.
875 See that class in the
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500876 <filename>meta/classes</filename> folder of the
Patrick Williamsc0f7c042017-02-23 20:41:17 -0600877 <ulink url='&YOCTO_DOCS_DEV_URL;#source-directory'>Source Directory</ulink>
878 for information.
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500879 </para>
880
881 <section id='logging-with-python'>
882 <title>Logging With Python</title>
883 <para>
884 When creating recipes using Python and inserting code that handles build logs,
885 keep in mind the goal is to have informative logs while keeping the console as
886 "silent" as possible.
887 Also, if you want status messages in the log, use the "debug" loglevel.
888 </para>
889
890 <para>
891 Following is an example written in Python.
892 The code handles logging for a function that determines the
893 number of tasks needed to be run.
894 See the
895 "<link linkend='ref-tasks-listtasks'><filename>do_listtasks</filename></link>"
896 section for additional information:
897 <literallayout class='monospaced'>
898 python do_listtasks() {
899 bb.debug(2, "Starting to figure out the task list")
900 if noteworthy_condition:
901 bb.note("There are 47 tasks to run")
902 bb.debug(2, "Got to point xyz")
903 if warning_trigger:
904 bb.warn("Detected warning_trigger, this might be a problem later.")
905 if recoverable_error:
906 bb.error("Hit recoverable_error, you really need to fix this!")
907 if fatal_error:
908 bb.fatal("fatal_error detected, unable to print the task list")
909 bb.plain("The tasks present are abc")
910 bb.debug(2, "Finished figuring out the tasklist")
911 }
912 </literallayout>
913 </para>
914 </section>
915
916 <section id='logging-with-bash'>
917 <title>Logging With Bash</title>
918 <para>
919 When creating recipes using Bash and inserting code that handles build
920 logs, you have the same goals - informative with minimal console output.
921 The syntax you use for recipes written in Bash is similar to that of
922 recipes written in Python described in the previous section.
923 </para>
924
925 <para>
926 Following is an example written in Bash.
927 The code logs the progress of the <filename>do_my_function</filename> function.
928 <literallayout class='monospaced'>
929 do_my_function() {
930 bbdebug 2 "Running do_my_function"
931 if [ exceptional_condition ]; then
932 bbnote "Hit exceptional_condition"
933 fi
934 bbdebug 2 "Got to point xyz"
935 if [ warning_trigger ]; then
936 bbwarn "Detected warning_trigger, this might cause a problem later."
937 fi
938 if [ recoverable_error ]; then
939 bberror "Hit recoverable_error, correcting"
940 fi
941 if [ fatal_error ]; then
942 bbfatal "fatal_error detected"
943 fi
944 bbdebug 2 "Completed do_my_function"
945 }
946 </literallayout>
947 </para>
948 </section>
949 </section>
950
951 <section id='usingpoky-debugging-others'>
952 <title>Other Tips</title>
953
954 <para>
955 Here are some other tips that you might find useful:
956 <itemizedlist>
Patrick Williamsc0f7c042017-02-23 20:41:17 -0600957 <listitem><para>
958 When adding new packages, it is worth watching for
959 undesirable items making their way into compiler command
960 lines.
961 For example, you do not want references to local system
962 files like
963 <filename>/usr/lib/</filename> or
964 <filename>/usr/include/</filename>.
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500965 </para></listitem>
Patrick Williamsc0f7c042017-02-23 20:41:17 -0600966 <listitem><para>
967 If you want to remove the <filename>psplash</filename>
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500968 boot splashscreen,
Patrick Williamsc0f7c042017-02-23 20:41:17 -0600969 add <filename>psplash=false</filename> to the kernel
970 command line.
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500971 Doing so prevents <filename>psplash</filename> from loading
972 and thus allows you to see the console.
973 It is also possible to switch out of the splashscreen by
Patrick Williamsc0f7c042017-02-23 20:41:17 -0600974 switching the virtual console (e.g. Fn+Left or Fn+Right
975 on a Zaurus).
976 </para></listitem>
977 <listitem><para>
978 Removing
979 <link linkend='var-TMPDIR'><filename>TMPDIR</filename></link>
980 (usually <filename>tmp/</filename>, within the
981 <ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink>)
982 can often fix temporary build issues.
983 Removing <filename>TMPDIR</filename> is usually a
984 relatively cheap operation, because task output will be
985 cached in
986 <link linkend='var-SSTATE_DIR'><filename>SSTATE_DIR</filename></link>
987 (usually <filename>sstate-cache/</filename>, which is
988 also in the Build Directory).
989 <note>
990 Removing <filename>TMPDIR</filename> might be a
991 workaround rather than a fix.
992 Consequently, trying to determine the underlying cause
993 of an issue before removing the directory is a good
994 idea.
995 </note>
996 </para></listitem>
997 <listitem><para>
998 Understanding how a feature is used in practice within
999 existing recipes can be very helpful.
1000 It is recommended that you configure some method that
1001 allows you to quickly search through files.</para>
1002
1003 <para>Using GNU Grep, you can use the following shell
1004 function to recursively search through common
1005 recipe-related files, skipping binary files,
1006 <filename>.git</filename> directories, and the
1007 Build Directory (assuming its name starts with
1008 "build"):
1009 <literallayout class='monospaced'>
1010 g() {
1011 grep -Ir \
1012 --exclude-dir=.git \
1013 --exclude-dir='build*' \
1014 --include='*.bb*' \
1015 --include='*.inc*' \
1016 --include='*.conf*' \
1017 --include='*.py*' \
1018 "$@"
1019 }
1020 </literallayout>
1021 Following are some usage examples:
1022 <literallayout class='monospaced'>
1023 $ g FOO # Search recursively for "FOO"
1024 $ g -i foo # Search recursively for "foo", ignoring case
1025 $ g -w FOO # Search recursively for "FOO" as a word, ignoring e.g. "FOOBAR"
1026 </literallayout>
1027 If figuring out how some feature works requires a lot of
1028 searching, it might indicate that the documentation should
1029 be extended or improved.
1030 In such cases, consider filing a documentation bug using
1031 the Yocto Project implementation of
1032 <ulink url='https://bugzilla.yoctoproject.org/'>Bugzilla</ulink>.
1033 For general information on how to submit a bug against
1034 the Yocto Project, see the
1035 "<ulink url='&YOCTO_DOCS_DEV_URL;#tracking-bugs'>Tracking Bugs</ulink>"
1036 section in the Yocto Project Development Manual.
1037 <note>
1038 The manuals might not be the right place to document
1039 variables that are purely internal and have a limited
1040 scope (e.g. internal variables used to implement a
1041 single <filename>.bbclass</filename> file).
1042 </note>
Patrick Williamsc124f4f2015-09-15 14:41:29 -05001043 </para></listitem>
1044 </itemizedlist>
1045 </para>
1046 </section>
1047</section>
1048
1049<section id='maintaining-build-output-quality'>
1050 <title>Maintaining Build Output Quality</title>
1051
1052 <para>
1053 Many factors can influence the quality of a build.
1054 For example, if you upgrade a recipe to use a new version of an upstream software
1055 package or you experiment with some new configuration options, subtle changes
1056 can occur that you might not detect until later.
1057 Consider the case where your recipe is using a newer version of an upstream package.
1058 In this case, a new version of a piece of software might introduce an optional
1059 dependency on another library, which is auto-detected.
1060 If that library has already been built when the software is building,
1061 the software will link to the built library and that library will be pulled
1062 into your image along with the new software even if you did not want the
1063 library.
1064 </para>
1065
1066 <para>
1067 The
1068 <link linkend='ref-classes-buildhistory'><filename>buildhistory</filename></link>
1069 class exists to help you maintain
1070 the quality of your build output.
1071 You can use the class to highlight unexpected and possibly unwanted
1072 changes in the build output.
1073 When you enable build history, it records information about the contents of
1074 each package and image and then commits that information to a local Git
1075 repository where you can examine the information.
1076 </para>
1077
1078 <para>
1079 The remainder of this section describes the following:
1080 <itemizedlist>
1081 <listitem><para>How you can enable and disable
1082 build history</para></listitem>
1083 <listitem><para>How to understand what the build history contains
1084 </para></listitem>
1085 <listitem><para>How to limit the information used for build history
1086 </para></listitem>
1087 <listitem><para>How to examine the build history from both a
1088 command-line and web interface</para></listitem>
1089 </itemizedlist>
1090 </para>
1091
1092 <section id='enabling-and-disabling-build-history'>
1093 <title>Enabling and Disabling Build History</title>
1094
1095 <para>
1096 Build history is disabled by default.
1097 To enable it, add the following <filename>INHERIT</filename>
1098 statement and set the
1099 <link linkend='var-BUILDHISTORY_COMMIT'><filename>BUILDHISTORY_COMMIT</filename></link>
1100 variable to "1" at the end of your
1101 <filename>conf/local.conf</filename> file found in the
1102 <ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink>:
1103 <literallayout class='monospaced'>
1104 INHERIT += "buildhistory"
1105 BUILDHISTORY_COMMIT = "1"
1106 </literallayout>
1107 Enabling build history as previously described
1108 causes the build process to collect build
1109 output information and commit it to a local
1110 <ulink url='&YOCTO_DOCS_DEV_URL;#git'>Git</ulink> repository.
1111 <note>
1112 Enabling build history increases your build times slightly,
1113 particularly for images, and increases the amount of disk
1114 space used during the build.
1115 </note>
1116 </para>
1117
1118 <para>
1119 You can disable build history by removing the previous statements
1120 from your <filename>conf/local.conf</filename> file.
1121 </para>
1122 </section>
1123
1124 <section id='understanding-what-the-build-history-contains'>
1125 <title>Understanding What the Build History Contains</title>
1126
1127 <para>
1128 Build history information is kept in
1129 <filename>${</filename><link linkend='var-TOPDIR'><filename>TOPDIR</filename></link><filename>}/buildhistory</filename>
1130 in the Build Directory as defined by the
1131 <link linkend='var-BUILDHISTORY_DIR'><filename>BUILDHISTORY_DIR</filename></link>
1132 variable.
1133 The following is an example abbreviated listing:
1134 <imagedata fileref="figures/buildhistory.png" align="center" width="6in" depth="4in" />
1135 </para>
1136
1137 <para>
1138 At the top level, there is a <filename>metadata-revs</filename> file
1139 that lists the revisions of the repositories for the layers enabled
1140 when the build was produced.
1141 The rest of the data splits into separate
1142 <filename>packages</filename>, <filename>images</filename> and
1143 <filename>sdk</filename> directories, the contents of which are
1144 described below.
1145 </para>
1146
1147 <section id='build-history-package-information'>
1148 <title>Build History Package Information</title>
1149
1150 <para>
1151 The history for each package contains a text file that has
1152 name-value pairs with information about the package.
1153 For example, <filename>buildhistory/packages/i586-poky-linux/busybox/busybox/latest</filename>
1154 contains the following:
1155 <literallayout class='monospaced'>
1156 PV = 1.22.1
1157 PR = r32
1158 RPROVIDES =
1159 RDEPENDS = glibc (>= 2.20) update-alternatives-opkg
1160 RRECOMMENDS = busybox-syslog busybox-udhcpc update-rc.d
1161 PKGSIZE = 540168
1162 FILES = /usr/bin/* /usr/sbin/* /usr/lib/busybox/* /usr/lib/lib*.so.* \
1163 /etc /com /var /bin/* /sbin/* /lib/*.so.* /lib/udev/rules.d \
1164 /usr/lib/udev/rules.d /usr/share/busybox /usr/lib/busybox/* \
1165 /usr/share/pixmaps /usr/share/applications /usr/share/idl \
1166 /usr/share/omf /usr/share/sounds /usr/lib/bonobo/servers
1167 FILELIST = /bin/busybox /bin/busybox.nosuid /bin/busybox.suid /bin/sh \
1168 /etc/busybox.links.nosuid /etc/busybox.links.suid
1169 </literallayout>
1170 Most of these name-value pairs correspond to variables used
1171 to produce the package.
1172 The exceptions are <filename>FILELIST</filename>, which is the
1173 actual list of files in the package, and
1174 <filename>PKGSIZE</filename>, which is the total size of files
1175 in the package in bytes.
1176 </para>
1177
1178 <para>
1179 There is also a file corresponding to the recipe from which the
1180 package came (e.g.
1181 <filename>buildhistory/packages/i586-poky-linux/busybox/latest</filename>):
1182 <literallayout class='monospaced'>
1183 PV = 1.22.1
1184 PR = r32
1185 DEPENDS = initscripts kern-tools-native update-rc.d-native \
1186 virtual/i586-poky-linux-compilerlibs virtual/i586-poky-linux-gcc \
1187 virtual/libc virtual/update-alternatives
1188 PACKAGES = busybox-ptest busybox-httpd busybox-udhcpd busybox-udhcpc \
1189 busybox-syslog busybox-mdev busybox-hwclock busybox-dbg \
1190 busybox-staticdev busybox-dev busybox-doc busybox-locale busybox
1191 </literallayout>
1192 </para>
1193
1194 <para>
1195 Finally, for those recipes fetched from a version control
1196 system (e.g., Git), a file exists that lists source revisions
1197 that are specified in the recipe and lists the actual revisions
1198 used during the build.
1199 Listed and actual revisions might differ when
1200 <link linkend='var-SRCREV'><filename>SRCREV</filename></link>
1201 is set to
1202 <filename>${<link linkend='var-AUTOREV'>AUTOREV</link>}</filename>.
1203 Here is an example assuming
1204 <filename>buildhistory/packages/qemux86-poky-linux/linux-yocto/latest_srcrev</filename>):
1205 <literallayout class='monospaced'>
1206 # SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
1207 SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
1208 # SRCREV_meta = "a227f20eff056e511d504b2e490f3774ab260d6f"
1209 SRCREV_meta = "a227f20eff056e511d504b2e490f3774ab260d6f"
1210 </literallayout>
1211 You can use the <filename>buildhistory-collect-srcrevs</filename>
1212 command with the <filename>-a</filename> option to
1213 collect the stored <filename>SRCREV</filename> values
1214 from build history and report them in a format suitable for
1215 use in global configuration (e.g.,
1216 <filename>local.conf</filename> or a distro include file) to
1217 override floating <filename>AUTOREV</filename> values to a
1218 fixed set of revisions.
1219 Here is some example output from this command:
1220 <literallayout class='monospaced'>
1221 $ buildhistory-collect-srcrevs -a
1222 # i586-poky-linux
1223 SRCREV_pn-glibc = "b8079dd0d360648e4e8de48656c5c38972621072"
1224 SRCREV_pn-glibc-initial = "b8079dd0d360648e4e8de48656c5c38972621072"
1225 SRCREV_pn-opkg-utils = "53274f087565fd45d8452c5367997ba6a682a37a"
1226 SRCREV_pn-kmod = "fd56638aed3fe147015bfa10ed4a5f7491303cb4"
1227 # x86_64-linux
1228 SRCREV_pn-gtk-doc-stub-native = "1dea266593edb766d6d898c79451ef193eb17cfa"
1229 SRCREV_pn-dtc-native = "65cc4d2748a2c2e6f27f1cf39e07a5dbabd80ebf"
1230 SRCREV_pn-update-rc.d-native = "eca680ddf28d024954895f59a241a622dd575c11"
1231 SRCREV_glibc_pn-cross-localedef-native = "b8079dd0d360648e4e8de48656c5c38972621072"
1232 SRCREV_localedef_pn-cross-localedef-native = "c833367348d39dad7ba018990bfdaffaec8e9ed3"
1233 SRCREV_pn-prelink-native = "faa069deec99bf61418d0bab831c83d7c1b797ca"
1234 SRCREV_pn-opkg-utils-native = "53274f087565fd45d8452c5367997ba6a682a37a"
1235 SRCREV_pn-kern-tools-native = "23345b8846fe4bd167efdf1bd8a1224b2ba9a5ff"
1236 SRCREV_pn-kmod-native = "fd56638aed3fe147015bfa10ed4a5f7491303cb4"
1237 # qemux86-poky-linux
1238 SRCREV_machine_pn-linux-yocto = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
1239 SRCREV_meta_pn-linux-yocto = "a227f20eff056e511d504b2e490f3774ab260d6f"
1240 # all-poky-linux
1241 SRCREV_pn-update-rc.d = "eca680ddf28d024954895f59a241a622dd575c11"
1242 </literallayout>
1243 <note>
1244 Here are some notes on using the
1245 <filename>buildhistory-collect-srcrevs</filename> command:
1246 <itemizedlist>
1247 <listitem><para>By default, only values where the
1248 <filename>SRCREV</filename> was
1249 not hardcoded (usually when <filename>AUTOREV</filename>
1250 was used) are reported.
1251 Use the <filename>-a</filename> option to see all
1252 <filename>SRCREV</filename> values.
1253 </para></listitem>
1254 <listitem><para>The output statements might not have any effect
1255 if overrides are applied elsewhere in the build system
1256 configuration.
1257 Use the <filename>-f</filename> option to add the
1258 <filename>forcevariable</filename> override to each output line
1259 if you need to work around this restriction.
1260 </para></listitem>
1261 <listitem><para>The script does apply special handling when
1262 building for multiple machines.
1263 However, the script does place a
1264 comment before each set of values that specifies
1265 which triplet to which they belong as shown above
1266 (e.g., <filename>i586-poky-linux</filename>).
1267 </para></listitem>
1268 </itemizedlist>
1269 </note>
1270 </para>
1271 </section>
1272
1273 <section id='build-history-image-information'>
1274 <title>Build History Image Information</title>
1275
1276 <para>
1277 The files produced for each image are as follows:
1278 <itemizedlist>
1279 <listitem><para><filename>image-files:</filename>
1280 A directory containing selected files from the root
1281 filesystem.
1282 The files are defined by
1283 <link linkend='var-BUILDHISTORY_IMAGE_FILES'><filename>BUILDHISTORY_IMAGE_FILES</filename></link>.
1284 </para></listitem>
1285 <listitem><para><filename>build-id.txt:</filename>
1286 Human-readable information about the build configuration
1287 and metadata source revisions.
1288 This file contains the full build header as printed
1289 by BitBake.</para></listitem>
1290 <listitem><para><filename>*.dot:</filename>
1291 Dependency graphs for the image that are
1292 compatible with <filename>graphviz</filename>.
1293 </para></listitem>
1294 <listitem><para><filename>files-in-image.txt:</filename>
1295 A list of files in the image with permissions,
1296 owner, group, size, and symlink information.
1297 </para></listitem>
1298 <listitem><para><filename>image-info.txt:</filename>
1299 A text file containing name-value pairs with information
1300 about the image.
1301 See the following listing example for more information.
1302 </para></listitem>
1303 <listitem><para><filename>installed-package-names.txt:</filename>
1304 A list of installed packages by name only.</para></listitem>
1305 <listitem><para><filename>installed-package-sizes.txt:</filename>
1306 A list of installed packages ordered by size.
1307 </para></listitem>
1308 <listitem><para><filename>installed-packages.txt:</filename>
1309 A list of installed packages with full package
1310 filenames.</para></listitem>
1311 </itemizedlist>
1312 <note>
1313 Installed package information is able to be gathered and
1314 produced even if package management is disabled for the final
1315 image.
1316 </note>
1317 </para>
1318
1319 <para>
1320 Here is an example of <filename>image-info.txt</filename>:
1321 <literallayout class='monospaced'>
1322 DISTRO = poky
1323 DISTRO_VERSION = 1.7
1324 USER_CLASSES = buildstats image-mklibs image-prelink
1325 IMAGE_CLASSES = image_types
1326 IMAGE_FEATURES = debug-tweaks
1327 IMAGE_LINGUAS =
1328 IMAGE_INSTALL = packagegroup-core-boot run-postinsts
1329 BAD_RECOMMENDATIONS =
1330 NO_RECOMMENDATIONS =
1331 PACKAGE_EXCLUDE =
1332 ROOTFS_POSTPROCESS_COMMAND = write_package_manifest; license_create_manifest; \
1333 write_image_manifest ; buildhistory_list_installed_image ; \
1334 buildhistory_get_image_installed ; ssh_allow_empty_password; \
1335 postinst_enable_logging; rootfs_update_timestamp ; ssh_disable_dns_lookup ;
1336 IMAGE_POSTPROCESS_COMMAND = buildhistory_get_imageinfo ;
1337 IMAGESIZE = 6900
1338 </literallayout>
1339 Other than <filename>IMAGESIZE</filename>, which is the
1340 total size of the files in the image in Kbytes, the
1341 name-value pairs are variables that may have influenced the
1342 content of the image.
1343 This information is often useful when you are trying to determine
1344 why a change in the package or file listings has occurred.
1345 </para>
1346 </section>
1347
1348 <section id='using-build-history-to-gather-image-information-only'>
1349 <title>Using Build History to Gather Image Information Only</title>
1350
1351 <para>
1352 As you can see, build history produces image information,
1353 including dependency graphs, so you can see why something
1354 was pulled into the image.
1355 If you are just interested in this information and not
1356 interested in collecting specific package or SDK information,
1357 you can enable writing only image information without
1358 any history by adding the following to your
1359 <filename>conf/local.conf</filename> file found in the
1360 <ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink>:
1361 <literallayout class='monospaced'>
1362 INHERIT += "buildhistory"
1363 BUILDHISTORY_COMMIT = "0"
1364 BUILDHISTORY_FEATURES = "image"
1365 </literallayout>
1366 Here, you set the
1367 <link linkend='var-BUILDHISTORY_FEATURES'><filename>BUILDHISTORY_FEATURES</filename></link>
1368 variable to use the image feature only.
1369 </para>
1370 </section>
1371
1372 <section id='build-history-sdk-information'>
1373 <title>Build History SDK Information</title>
Patrick Williamsd8c66bc2016-06-20 12:57:21 -05001374
Patrick Williamsc124f4f2015-09-15 14:41:29 -05001375 <para>
1376 Build history collects similar information on the contents
Patrick Williamsd8c66bc2016-06-20 12:57:21 -05001377 of SDKs
1378 (e.g. <filename>bitbake -c populate_sdk imagename</filename>)
Patrick Williamsc124f4f2015-09-15 14:41:29 -05001379 as compared to information it collects for images.
Patrick Williamsd8c66bc2016-06-20 12:57:21 -05001380 Furthermore, this information differs depending on whether an
1381 extensible or standard SDK is being produced.
1382 </para>
1383
1384 <para>
1385 The following list shows the files produced for SDKs:
Patrick Williamsc124f4f2015-09-15 14:41:29 -05001386 <itemizedlist>
1387 <listitem><para><filename>files-in-sdk.txt:</filename>
1388 A list of files in the SDK with permissions,
1389 owner, group, size, and symlink information.
1390 This list includes both the host and target parts
1391 of the SDK.
1392 </para></listitem>
1393 <listitem><para><filename>sdk-info.txt:</filename>
1394 A text file containing name-value pairs with information
1395 about the SDK.
1396 See the following listing example for more information.
1397 </para></listitem>
Patrick Williamsd8c66bc2016-06-20 12:57:21 -05001398 <listitem><para><filename>sstate-task-sizes.txt:</filename>
1399 A text file containing name-value pairs with information
1400 about task group sizes
1401 (e.g. <filename>do_populate_sysroot</filename> tasks
1402 have a total size).
1403 The <filename>sstate-task-sizes.txt</filename> file
1404 exists only when an extensible SDK is created.
1405 </para></listitem>
1406 <listitem><para><filename>sstate-package-sizes.txt:</filename>
1407 A text file containing name-value pairs with information
1408 for the shared-state packages and sizes in the SDK.
1409 The <filename>sstate-package-sizes.txt</filename> file
1410 exists only when an extensible SDK is created.
1411 </para></listitem>
1412 <listitem><para><filename>sdk-files:</filename>
1413 A folder that contains copies of the files mentioned in
1414 <filename>BUILDHISTORY_SDK_FILES</filename> if the
1415 files are present in the output.
1416 Additionally, the default value of
1417 <filename>BUILDHISTORY_SDK_FILES</filename> is specific
1418 to the extensible SDK although you can set it
1419 differently if you would like to pull in specific files
1420 from the standard SDK.</para>
1421 <para>The default files are
1422 <filename>conf/local.conf</filename>,
1423 <filename>conf/bblayers.conf</filename>,
1424 <filename>conf/auto.conf</filename>,
1425 <filename>conf/locked-sigs.inc</filename>, and
1426 <filename>conf/devtool.conf</filename>.
1427 Thus, for an extensible SDK, these files get copied
1428 into the <filename>sdk-files</filename> directory.
1429 </para></listitem>
Patrick Williamsc124f4f2015-09-15 14:41:29 -05001430 <listitem><para>The following information appears under
1431 each of the <filename>host</filename>
1432 and <filename>target</filename> directories
1433 for the portions of the SDK that run on the host and
1434 on the target, respectively:
Patrick Williamsd8c66bc2016-06-20 12:57:21 -05001435 <note>
1436 The following files for the most part are empty
1437 when producing an extensible SDK because this
1438 type of SDK is not constructed from packages as is
1439 the standard SDK.
1440 </note>
Patrick Williamsc124f4f2015-09-15 14:41:29 -05001441 <itemizedlist>
1442 <listitem><para><filename>depends.dot:</filename>
1443 Dependency graph for the SDK that is
1444 compatible with <filename>graphviz</filename>.
1445 </para></listitem>
1446 <listitem><para><filename>installed-package-names.txt:</filename>
1447 A list of installed packages by name only.
1448 </para></listitem>
1449 <listitem><para><filename>installed-package-sizes.txt:</filename>
1450 A list of installed packages ordered by size.
1451 </para></listitem>
1452 <listitem><para><filename>installed-packages.txt:</filename>
1453 A list of installed packages with full package
1454 filenames.</para></listitem>
1455 </itemizedlist>
1456 </para></listitem>
1457 </itemizedlist>
1458 </para>
1459
1460 <para>
1461 Here is an example of <filename>sdk-info.txt</filename>:
1462 <literallayout class='monospaced'>
1463 DISTRO = poky
1464 DISTRO_VERSION = 1.3+snapshot-20130327
1465 SDK_NAME = poky-glibc-i686-arm
1466 SDK_VERSION = 1.3+snapshot
1467 SDKMACHINE =
1468 SDKIMAGE_FEATURES = dev-pkgs dbg-pkgs
1469 BAD_RECOMMENDATIONS =
1470 SDKSIZE = 352712
1471 </literallayout>
1472 Other than <filename>SDKSIZE</filename>, which is the
1473 total size of the files in the SDK in Kbytes, the
1474 name-value pairs are variables that might have influenced the
1475 content of the SDK.
1476 This information is often useful when you are trying to
1477 determine why a change in the package or file listings
1478 has occurred.
1479 </para>
1480 </section>
1481
1482 <section id='examining-build-history-information'>
1483 <title>Examining Build History Information</title>
1484
1485 <para>
1486 You can examine build history output from the command line or
1487 from a web interface.
1488 </para>
1489
1490 <para>
1491 To see any changes that have occurred (assuming you have
1492 <link linkend='var-BUILDHISTORY_COMMIT'><filename>BUILDHISTORY_COMMIT = "1"</filename></link>),
1493 you can simply
1494 use any Git command that allows you to view the history of
1495 a repository.
1496 Here is one method:
1497 <literallayout class='monospaced'>
1498 $ git log -p
1499 </literallayout>
1500 You need to realize, however, that this method does show
1501 changes that are not significant (e.g. a package's size
1502 changing by a few bytes).
1503 </para>
1504
1505 <para>
1506 A command-line tool called <filename>buildhistory-diff</filename>
1507 does exist, though, that queries the Git repository and prints just
1508 the differences that might be significant in human-readable form.
1509 Here is an example:
1510 <literallayout class='monospaced'>
1511 $ ~/poky/poky/scripts/buildhistory-diff . HEAD^
1512 Changes to images/qemux86_64/glibc/core-image-minimal (files-in-image.txt):
1513 /etc/anotherpkg.conf was added
1514 /sbin/anotherpkg was added
1515 * (installed-package-names.txt):
1516 * anotherpkg was added
1517 Changes to images/qemux86_64/glibc/core-image-minimal (installed-package-names.txt):
1518 anotherpkg was added
1519 packages/qemux86_64-poky-linux/v86d: PACKAGES: added "v86d-extras"
1520 * PR changed from "r0" to "r1"
1521 * PV changed from "0.1.10" to "0.1.12"
1522 packages/qemux86_64-poky-linux/v86d/v86d: PKGSIZE changed from 110579 to 144381 (+30%)
1523 * PR changed from "r0" to "r1"
1524 * PV changed from "0.1.10" to "0.1.12"
1525 </literallayout>
Patrick Williamsc0f7c042017-02-23 20:41:17 -06001526 <note>
1527 The <filename>buildhistory-diff</filename> tool requires
1528 the <filename>GitPython</filename> package.
1529 Be sure to install it using Pip3 as follows:
1530 <literallayout class='monospaced'>
1531 $ pip3 install GitPython --user
1532 </literallayout>
1533 Alternatively, you can install
1534 <filename>python3-git</filename> using the appropriate
1535 distribution package manager (e.g.
1536 <filename>apt-get</filename>, <filename>dnf</filename>, or
1537 <filename>zipper</filename>).
1538 </note>
Patrick Williamsc124f4f2015-09-15 14:41:29 -05001539 </para>
1540
1541 <para>
1542 To see changes to the build history using a web interface, follow
1543 the instruction in the <filename>README</filename> file here.
1544 <ulink url='http://git.yoctoproject.org/cgit/cgit.cgi/buildhistory-web/'></ulink>.
1545 </para>
1546
1547 <para>
1548 Here is a sample screenshot of the interface:
1549 <imagedata fileref="figures/buildhistory-web.png" align="center" scalefit="1" width="130%" contentdepth="130%" />
1550 </para>
1551 </section>
1552 </section>
1553</section>
1554
1555<section id='speeding-up-the-build'>
1556 <title>Speeding Up the Build</title>
1557
1558 <para>
1559 Build time can be an issue.
1560 By default, the build system uses simple controls to try and maximize
1561 build efficiency.
1562 In general, the default settings for all the following variables
1563 result in the most efficient build times when dealing with single
1564 socket systems (i.e. a single CPU).
1565 If you have multiple CPUs, you might try increasing the default
1566 values to gain more speed.
1567 See the descriptions in the glossary for each variable for more
1568 information:
1569 <itemizedlist>
1570 <listitem><para>
1571 <link linkend='var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename>:</link>
1572 The maximum number of threads BitBake simultaneously executes.
1573 </para></listitem>
1574 <listitem><para>
1575 <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_NUMBER_PARSE_THREADS'><filename>BB_NUMBER_PARSE_THREADS</filename>:</ulink>
1576 The number of threads BitBake uses during parsing.
1577 </para></listitem>
1578 <listitem><para>
1579 <link linkend='var-PARALLEL_MAKE'><filename>PARALLEL_MAKE</filename>:</link>
1580 Extra options passed to the <filename>make</filename> command
1581 during the
1582 <link linkend='ref-tasks-compile'><filename>do_compile</filename></link>
1583 task in order to specify parallel compilation on the
1584 local build host.
1585 </para></listitem>
1586 <listitem><para>
1587 <link linkend='var-PARALLEL_MAKEINST'><filename>PARALLEL_MAKEINST</filename>:</link>
1588 Extra options passed to the <filename>make</filename> command
1589 during the
1590 <link linkend='ref-tasks-install'><filename>do_install</filename></link>
1591 task in order to specify parallel installation on the
1592 local build host.
1593 </para></listitem>
1594 </itemizedlist>
1595 As mentioned, these variables all scale to the number of processor
1596 cores available on the build system.
1597 For single socket systems, this auto-scaling ensures that the build
1598 system fundamentally takes advantage of potential parallel operations
1599 during the build based on the build machine's capabilities.
1600 </para>
1601
1602 <para>
1603 Following are additional factors that can affect build speed:
1604 <itemizedlist>
1605 <listitem><para>
1606 File system type:
1607 The file system type that the build is being performed on can
1608 also influence performance.
1609 Using <filename>ext4</filename> is recommended as compared
1610 to <filename>ext2</filename> and <filename>ext3</filename>
1611 due to <filename>ext4</filename> improved features
1612 such as extents.
1613 </para></listitem>
1614 <listitem><para>
1615 Disabling the updating of access time using
1616 <filename>noatime</filename>:
1617 The <filename>noatime</filename> mount option prevents the
1618 build system from updating file and directory access times.
1619 </para></listitem>
1620 <listitem><para>
1621 Setting a longer commit:
1622 Using the "commit=" mount option increases the interval
1623 in seconds between disk cache writes.
1624 Changing this interval from the five second default to
1625 something longer increases the risk of data loss but decreases
1626 the need to write to the disk, thus increasing the build
1627 performance.
1628 </para></listitem>
1629 <listitem><para>
1630 Choosing the packaging backend:
1631 Of the available packaging backends, IPK is the fastest.
1632 Additionally, selecting a singular packaging backend also
1633 helps.
1634 </para></listitem>
1635 <listitem><para>
1636 Using <filename>tmpfs</filename> for
1637 <link linkend='var-TMPDIR'><filename>TMPDIR</filename></link>
1638 as a temporary file system:
1639 While this can help speed up the build, the benefits are
1640 limited due to the compiler using
1641 <filename>-pipe</filename>.
1642 The build system goes to some lengths to avoid
1643 <filename>sync()</filename> calls into the
1644 file system on the principle that if there was a significant
1645 failure, the
1646 <ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink>
1647 contents could easily be rebuilt.
1648 </para></listitem>
1649 <listitem><para>
1650 Inheriting the
1651 <link linkend='ref-classes-rm-work'><filename>rm_work</filename></link>
1652 class:
1653 Inheriting this class has shown to speed up builds due to
1654 significantly lower amounts of data stored in the data
1655 cache as well as on disk.
1656 Inheriting this class also makes cleanup of
1657 <link linkend='var-TMPDIR'><filename>TMPDIR</filename></link>
1658 faster, at the expense of being easily able to dive into the
1659 source code.
1660 File system maintainers have recommended that the fastest way
1661 to clean up large numbers of files is to reformat partitions
1662 rather than delete files due to the linear nature of partitions.
1663 This, of course, assumes you structure the disk partitions and
1664 file systems in a way that this is practical.
1665 </para></listitem>
1666 </itemizedlist>
1667 Aside from the previous list, you should keep some trade offs in
1668 mind that can help you speed up the build:
1669 <itemizedlist>
1670 <listitem><para>
1671 Remove items from
1672 <link linkend='var-DISTRO_FEATURES'><filename>DISTRO_FEATURES</filename></link>
1673 that you might not need.
1674 </para></listitem>
1675 <listitem><para>
1676 Exclude debug symbols and other debug information:
1677 If you do not need these symbols and other debug information,
1678 disabling the <filename>*-dbg</filename> package generation
1679 can speed up the build.
1680 You can disable this generation by setting the
1681 <link linkend='var-INHIBIT_PACKAGE_DEBUG_SPLIT'><filename>INHIBIT_PACKAGE_DEBUG_SPLIT</filename></link>
1682 variable to "1".
1683 </para></listitem>
1684 <listitem><para>
1685 Disable static library generation for recipes derived from
1686 <filename>autoconf</filename> or <filename>libtool</filename>:
1687 Following is an example showing how to disable static
1688 libraries and still provide an override to handle exceptions:
1689 <literallayout class='monospaced'>
1690 STATICLIBCONF = "--disable-static"
1691 STATICLIBCONF_sqlite3-native = ""
1692 EXTRA_OECONF += "${STATICLIBCONF}"
1693 </literallayout>
1694 <note><title>Notes</title>
1695 <itemizedlist>
1696 <listitem><para>
1697 Some recipes need static libraries in order to work
1698 correctly (e.g. <filename>pseudo-native</filename>
1699 needs <filename>sqlite3-native</filename>).
1700 Overrides, as in the previous example, account for
1701 these kinds of exceptions.
1702 </para></listitem>
1703 <listitem><para>
1704 Some packages have packaging code that assumes the
1705 presence of the static libraries.
1706 If so, you might need to exclude them as well.
1707 </para></listitem>
1708 </itemizedlist>
1709 </note>
1710 </para></listitem>
1711 </itemizedlist>
1712 </para>
1713</section>
1714</chapter>
1715<!--
1716vim: expandtab tw=80 ts=4
1717-->