Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 1 | <!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" |
| 2 | "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd" |
| 3 | [<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] > |
| 4 | |
| 5 | <chapter id='usingpoky'> |
| 6 | <title>Using the Yocto Project</title> |
| 7 | |
| 8 | <para> |
| 9 | This chapter describes common usage for the Yocto Project. |
| 10 | The information is introductory in nature as other manuals in the Yocto Project |
| 11 | documentation set provide more details on how to use the Yocto Project. |
| 12 | </para> |
| 13 | |
| 14 | <section id='usingpoky-build'> |
| 15 | <title>Running a Build</title> |
| 16 | |
| 17 | <para> |
| 18 | This section provides a summary of the build process and provides information |
| 19 | for less obvious aspects of the build process. |
| 20 | For general information on how to build an image using the OpenEmbedded build |
| 21 | system, see the |
| 22 | "<ulink url='&YOCTO_DOCS_QS_URL;#qs-building-images'>Building Images</ulink>" |
| 23 | section of the Yocto Project Quick Start. |
| 24 | </para> |
| 25 | |
| 26 | <section id='build-overview'> |
| 27 | <title>Build Overview</title> |
| 28 | |
| 29 | <para> |
| 30 | In the development environment you will need to build an image whenever you change hardware |
| 31 | support, add or change system libraries, or add or change services that have dependencies. |
| 32 | </para> |
| 33 | |
| 34 | <mediaobject> |
| 35 | <imageobject> |
| 36 | <imagedata fileref="figures/building-an-image.png" format="PNG" align='center' scalefit='1'/> |
| 37 | </imageobject> |
| 38 | <caption> |
| 39 | <para>Building an Image</para> |
| 40 | </caption> |
| 41 | </mediaobject> |
| 42 | |
| 43 | <para> |
| 44 | The first thing you need to do is set up the OpenEmbedded build |
| 45 | environment by sourcing an environment setup script |
| 46 | (i.e. |
| 47 | <link linkend='structure-core-script'><filename>&OE_INIT_FILE;</filename></link> |
| 48 | or |
| 49 | <link linkend='structure-memres-core-script'><filename>oe-init-build-env-memres</filename></link>). |
| 50 | Here is an example: |
| 51 | <literallayout class='monospaced'> |
| 52 | $ source &OE_INIT_FILE; [<replaceable>build_dir</replaceable>] |
| 53 | </literallayout> |
| 54 | </para> |
| 55 | |
| 56 | <para> |
| 57 | The <replaceable>build_dir</replaceable> argument is optional and specifies the directory the |
| 58 | OpenEmbedded build system uses for the build - |
| 59 | the <ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink>. |
| 60 | If you do not specify a Build Directory, it defaults to a directory |
| 61 | named <filename>build</filename> in your current working directory. |
| 62 | A common practice is to use a different Build Directory for different targets. |
| 63 | For example, <filename>~/build/x86</filename> for a <filename>qemux86</filename> |
| 64 | target, and <filename>~/build/arm</filename> for a <filename>qemuarm</filename> target. |
| 65 | </para> |
| 66 | |
| 67 | <para> |
| 68 | Once the build environment is set up, you can build a target using: |
| 69 | <literallayout class='monospaced'> |
| 70 | $ bitbake <replaceable>target</replaceable> |
| 71 | </literallayout> |
| 72 | </para> |
| 73 | |
| 74 | <para> |
| 75 | The <replaceable>target</replaceable> is the name of the recipe you want to build. |
| 76 | Common targets are the images in <filename>meta/recipes-core/images</filename>, |
| 77 | <filename>meta/recipes-sato/images</filename>, etc. all found in the |
| 78 | <ulink url='&YOCTO_DOCS_DEV_URL;#source-directory'>Source Directory</ulink>. |
| 79 | Or, the target can be the name of a recipe for a specific piece of software such as |
| 80 | BusyBox. |
| 81 | For more details about the images the OpenEmbedded build system supports, see the |
| 82 | "<link linkend="ref-images">Images</link>" chapter. |
| 83 | </para> |
| 84 | |
| 85 | <note> |
| 86 | Building an image without GNU General Public License Version |
| 87 | 3 (GPLv3), or similarly licensed, components is supported for |
| 88 | only minimal and base images. |
| 89 | See the "<link linkend='ref-images'>Images</link>" chapter for more information. |
| 90 | </note> |
| 91 | </section> |
| 92 | |
| 93 | <section id='building-an-image-using-gpl-components'> |
| 94 | <title>Building an Image Using GPL Components</title> |
| 95 | |
| 96 | <para> |
| 97 | When building an image using GPL components, you need to maintain your original |
| 98 | settings and not switch back and forth applying different versions of the GNU |
| 99 | General Public License. |
| 100 | If you rebuild using different versions of GPL, dependency errors might occur |
| 101 | due to some components not being rebuilt. |
| 102 | </para> |
| 103 | </section> |
| 104 | </section> |
| 105 | |
| 106 | <section id='usingpoky-install'> |
| 107 | <title>Installing and Using the Result</title> |
| 108 | |
| 109 | <para> |
| 110 | Once an image has been built, it often needs to be installed. |
| 111 | The images and kernels built by the OpenEmbedded build system are placed in the |
| 112 | <ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink> in |
| 113 | <filename class="directory">tmp/deploy/images</filename>. |
| 114 | For information on how to run pre-built images such as <filename>qemux86</filename> |
| 115 | and <filename>qemuarm</filename>, see the |
Patrick Williams | d8c66bc | 2016-06-20 12:57:21 -0500 | [diff] [blame] | 116 | <ulink url='&YOCTO_DOCS_SDK_URL;#sdk-manual'>Yocto Project Software Development Kit (SDK) Developer's Guide</ulink>. |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 117 | For information about how to install these images, see the documentation for your |
| 118 | particular board or machine. |
| 119 | </para> |
| 120 | </section> |
| 121 | |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 122 | <section id='usingpoky-debugging-tools-and-techniques'> |
| 123 | <title>Debugging Tools and Techniques</title> |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 124 | |
| 125 | <para> |
| 126 | The exact method for debugging build failures depends on the nature of |
| 127 | the problem and on the system's area from which the bug originates. |
| 128 | Standard debugging practices such as comparison against the last |
| 129 | known working version with examination of the changes and the |
| 130 | re-application of steps to identify the one causing the problem are |
| 131 | valid for the Yocto Project just as they are for any other system. |
| 132 | Even though it is impossible to detail every possible potential failure, |
| 133 | this section provides some general tips to aid in debugging. |
| 134 | </para> |
| 135 | |
| 136 | <para> |
| 137 | A useful feature for debugging is the error reporting tool. |
| 138 | Configuring the Yocto Project to use this tool causes the |
| 139 | OpenEmbedded build system to produce error reporting commands as |
| 140 | part of the console output. |
| 141 | You can enter the commands after the build completes |
| 142 | to log error information |
| 143 | into a common database, that can help you figure out what might be |
| 144 | going wrong. |
| 145 | For information on how to enable and use this feature, see the |
| 146 | "<ulink url='&YOCTO_DOCS_DEV_URL;#using-the-error-reporting-tool'>Using the Error Reporting Tool</ulink>" |
| 147 | section in the Yocto Project Development Manual. |
| 148 | </para> |
| 149 | |
| 150 | <para> |
| 151 | For discussions on debugging, see the |
Patrick Williams | d8c66bc | 2016-06-20 12:57:21 -0500 | [diff] [blame] | 152 | "<ulink url='&YOCTO_DOCS_DEV_URL;#platdev-gdb-remotedebug'>Debugging With the GNU Project Debugger (GDB) Remotely</ulink>" section |
| 153 | in the Yocto Project Developer's Manual |
| 154 | and the |
| 155 | "<ulink url='&YOCTO_DOCS_SDK_URL;#adt-eclipse'>Working within Eclipse</ulink>" |
| 156 | section in the Yocto Project Software Development Kit (SDK) Developer's Guide. |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 157 | </para> |
| 158 | |
| 159 | <note> |
| 160 | The remainder of this section presents many examples of the |
| 161 | <filename>bitbake</filename> command. |
| 162 | You can learn about BitBake by reading the |
| 163 | <ulink url='&YOCTO_DOCS_BB_URL;#bitbake-user-manual'>BitBake User Manual</ulink>. |
| 164 | </note> |
| 165 | |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 166 | <section id='usingpoky-debugging-viewing-logs-from-failed-tasks'> |
| 167 | <title>Viewing Logs from Failed Tasks</title> |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 168 | |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 169 | <para> |
| 170 | You can find the log for a task in the file |
| 171 | <filename>${</filename><link linkend='var-WORKDIR'><filename>WORKDIR</filename></link><filename>}/temp/log.do_</filename><replaceable>taskname</replaceable>. |
| 172 | For example, the log for the |
| 173 | <link linkend='ref-tasks-compile'><filename>do_compile</filename></link> |
| 174 | task of the QEMU minimal image for the x86 machine |
| 175 | (<filename>qemux86</filename>) might be in |
| 176 | <filename>tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/temp/log.do_compile</filename>. |
| 177 | To see the commands |
| 178 | <ulink url='&YOCTO_DOCS_DEV_URL;#bitbake-term'>BitBake</ulink> ran |
| 179 | to generate a log, look at the corresponding |
| 180 | <filename>run.do_</filename><replaceable>taskname</replaceable> |
| 181 | file in the same directory. |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 182 | </para> |
| 183 | |
| 184 | <para> |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 185 | <filename>log.do_</filename><replaceable>taskname</replaceable> and |
| 186 | <filename>run.do_</filename><replaceable>taskname</replaceable> |
| 187 | are actually symbolic links to |
| 188 | <filename>log.do_</filename><replaceable>taskname</replaceable><filename>.</filename><replaceable>pid</replaceable> |
| 189 | and |
| 190 | <filename>log.run_</filename><replaceable>taskname</replaceable><filename>.</filename><replaceable>pid</replaceable>, |
| 191 | where <replaceable>pid</replaceable> is the PID the task had when |
| 192 | it ran. |
| 193 | The symlinks always point to the files corresponding to the most |
| 194 | recent run. |
| 195 | </para> |
| 196 | </section> |
| 197 | |
| 198 | <section id='usingpoky-debugging-viewing-variable-values'> |
| 199 | <title>Viewing Variable Values</title> |
| 200 | <para> |
| 201 | BitBake's <filename>-e</filename> option is used to display |
| 202 | variable values after parsing. |
| 203 | The following command displays the variable values after the |
| 204 | configuration files (i.e. <filename>local.conf</filename>, |
| 205 | <filename>bblayers.conf</filename>, |
| 206 | <filename>bitbake.conf</filename> and so forth) have been |
| 207 | parsed: |
| 208 | <literallayout class='monospaced'> |
| 209 | $ bitbake -e |
| 210 | </literallayout> |
| 211 | The following command displays variable values after a specific |
| 212 | recipe has been parsed. |
| 213 | The variables include those from the configuration as well: |
| 214 | <literallayout class='monospaced'> |
| 215 | $ bitbake -e recipename |
| 216 | </literallayout> |
| 217 | <note><para> |
| 218 | Each recipe has its own private set of variables (datastore). |
| 219 | Internally, after parsing the configuration, a copy of the |
| 220 | resulting datastore is made prior to parsing each recipe. |
| 221 | This copying implies that variables set in one recipe will |
| 222 | not be visible to other recipes.</para> |
| 223 | |
| 224 | <para>Likewise, each task within a recipe gets a private |
| 225 | datastore based on the recipe datastore, which means that |
| 226 | variables set within one task will not be visible to |
| 227 | other tasks.</para> |
| 228 | </note> |
| 229 | </para> |
| 230 | |
| 231 | <para> |
| 232 | In the output of <filename>bitbake -e</filename>, each variable is |
| 233 | preceded by a description of how the variable got its value, |
| 234 | including temporary values that were later overriden. |
| 235 | This description also includes variable flags (varflags) set on |
| 236 | the variable. |
| 237 | The output can be very helpful during debugging. |
| 238 | </para> |
| 239 | |
| 240 | <para> |
| 241 | Variables that are exported to the environment are preceded by |
| 242 | <filename>export</filename> in the output of |
| 243 | <filename>bitbake -e</filename>. |
| 244 | See the following example: |
| 245 | <literallayout class='monospaced'> |
| 246 | export CC="i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/ulf/poky/build/tmp/sysroots/qemux86" |
| 247 | </literallayout> |
| 248 | </para> |
| 249 | |
| 250 | <para> |
| 251 | In addition to variable values, the output of the |
| 252 | <filename>bitbake -e</filename> and |
| 253 | <filename>bitbake -e</filename> <replaceable>recipe</replaceable> |
| 254 | commands includes the following information: |
| 255 | <itemizedlist> |
| 256 | <listitem><para> |
| 257 | The output starts with a tree listing all configuration |
| 258 | files and classes included globally, recursively listing |
| 259 | the files they include or inherit in turn. |
| 260 | Much of the behavior of the OpenEmbedded build system |
| 261 | (including the behavior of the |
| 262 | <link linkend='normal-recipe-build-tasks'>normal recipe build tasks</link>) |
| 263 | is implemented in the |
| 264 | <link linkend='ref-classes-base'><filename>base</filename></link> |
| 265 | class and the classes it inherits, rather than being built |
| 266 | into BitBake itself. |
| 267 | </para></listitem> |
| 268 | <listitem><para> |
| 269 | After the variable values, all functions appear in the |
| 270 | output. |
| 271 | For shell functions, variables referenced within the |
| 272 | function body are expanded. |
| 273 | If a function has been modified using overrides or |
| 274 | using override-style operators like |
| 275 | <filename>_append</filename> and |
| 276 | <filename>_prepend</filename>, then the final assembled |
| 277 | function body appears in the output. |
| 278 | </para></listitem> |
| 279 | </itemizedlist> |
| 280 | </para> |
| 281 | </section> |
| 282 | |
| 283 | <section id='viewing-package-information-with-oe-pkgdata-util'> |
| 284 | <title>Viewing Package Information with <filename>oe-pkgdata-util</filename></title> |
| 285 | |
| 286 | <para> |
| 287 | You can use the <filename>oe-pkgdata-util</filename> command-line |
| 288 | utility to query |
| 289 | <link linkend='var-PKGDATA_DIR'><filename>PKGDATA_DIR</filename></link> |
| 290 | and display various package-related information. |
| 291 | When you use the utility, you must use it to view information |
| 292 | on packages that have already been built. |
| 293 | </para> |
| 294 | |
| 295 | <para> |
| 296 | Following are a few of the available |
| 297 | <filename>oe-pkgdata-util</filename> subcommands. |
| 298 | <note> |
| 299 | You can use the standard * and ? globbing wildcards as part of |
| 300 | package names and paths. |
| 301 | </note> |
| 302 | <itemizedlist> |
| 303 | <listitem><para> |
| 304 | <filename>oe-pkgdata-util list-pkgs [</filename><replaceable>pattern</replaceable><filename>]</filename>: |
| 305 | Lists all packages that have been built, optionally |
| 306 | limiting the match to packages that match |
| 307 | <replaceable>pattern</replaceable>. |
| 308 | </para></listitem> |
| 309 | <listitem><para> |
| 310 | <filename>oe-pkgdata-util list-pkg-files </filename><replaceable>package</replaceable><filename> ...</filename>: |
| 311 | Lists the files and directories contained in the given |
| 312 | packages. |
| 313 | <note> |
| 314 | <para> |
| 315 | A different way to view the contents of a package is |
| 316 | to look at the |
| 317 | <filename>${</filename><link linkend='var-WORKDIR'><filename>WORKDIR</filename></link><filename>}/packages-split</filename> |
| 318 | directory of the recipe that generates the |
| 319 | package. |
| 320 | This directory is created by the |
| 321 | <link linkend='ref-tasks-package'><filename>do_package</filename></link> |
| 322 | task and has one subdirectory for each package the |
| 323 | recipe generates, which contains the files stored in |
| 324 | that package.</para> |
| 325 | <para> |
| 326 | If you want to inspect the |
| 327 | <filename>${WORKDIR}/packages-split</filename> |
| 328 | directory, make sure that |
| 329 | <link linkend='ref-classes-rm-work'><filename>rm_work</filename></link> |
| 330 | is not enabled when you build the recipe. |
| 331 | </para> |
| 332 | </note> |
| 333 | </para></listitem> |
| 334 | <listitem><para> |
| 335 | <filename>oe-pkgdata-util find-path </filename><replaceable>path</replaceable><filename> ...</filename>: |
| 336 | Lists the names of the packages that contain the given |
| 337 | paths. |
| 338 | For example, the following tells us that |
| 339 | <filename>/usr/share/man/man1/make.1</filename> |
| 340 | is contained in the <filename>make-doc</filename> |
| 341 | package: |
| 342 | <literallayout class='monospaced'> |
| 343 | $ oe-pkgdata-util find-path /usr/share/man/man1/make.1 |
| 344 | make-doc: /usr/share/man/man1/make.1 |
| 345 | </literallayout> |
| 346 | </para></listitem> |
| 347 | <listitem><para> |
| 348 | <filename>oe-pkgdata-util lookup-recipe </filename><replaceable>package</replaceable><filename> ...</filename>: |
| 349 | Lists the name of the recipes that |
| 350 | produce the given packages. |
| 351 | </para></listitem> |
| 352 | </itemizedlist> |
| 353 | </para> |
| 354 | |
| 355 | <para> |
| 356 | For more information on the <filename>oe-pkgdata-util</filename> |
| 357 | command, use the help facility: |
| 358 | <literallayout class='monospaced'> |
| 359 | $ oe-pkgdata-util ‐‐help |
| 360 | $ oe-pkgdata-util <replaceable>subcommand</replaceable> --help |
| 361 | </literallayout> |
| 362 | </para> |
| 363 | </section> |
| 364 | |
| 365 | <section id='usingpoky-viewing-dependencies-between-recipes-and-tasks'> |
| 366 | <title>Viewing Dependencies Between Recipes and Tasks</title> |
| 367 | |
| 368 | <para> |
| 369 | Sometimes it can be hard to see why BitBake wants to build other |
| 370 | recipes before the one you have specified. |
| 371 | Dependency information can help you understand why a recipe is |
| 372 | built. |
| 373 | </para> |
| 374 | |
| 375 | <para> |
| 376 | To generate dependency information for a recipe, run the following |
| 377 | command: |
| 378 | <literallayout class='monospaced'> |
| 379 | $ bitbake -g <replaceable>recipename</replaceable> |
| 380 | </literallayout> |
| 381 | This command writes the following files in the current directory: |
| 382 | <itemizedlist> |
| 383 | <listitem><para> |
| 384 | <filename>pn-buildlist</filename>: A list of |
| 385 | recipes/targets involved in building |
| 386 | <replaceable>recipename</replaceable>. |
| 387 | "Involved" here means that at least one task from the |
| 388 | recipe needs to run when building |
| 389 | <replaceable>recipename</replaceable> from scratch. |
| 390 | Targets that are in |
| 391 | <link linkend='var-ASSUME_PROVIDED'><filename>ASSUME_PROVIDED</filename></link> |
| 392 | are not listed. |
| 393 | </para></listitem> |
| 394 | <listitem><para> |
| 395 | <filename>pn-depends.dot</filename>: A graph showing |
| 396 | dependencies between build-time targets (recipes). |
| 397 | </para></listitem> |
| 398 | <listitem><para> |
| 399 | <filename>package-depends.dot</filename>: A graph showing |
| 400 | known dependencies between runtime targets. |
| 401 | </para></listitem> |
| 402 | <listitem><para> |
| 403 | <filename>task-depends.dot</filename>: A graph showing |
| 404 | dependencies between tasks. |
| 405 | </para></listitem> |
| 406 | </itemizedlist> |
| 407 | </para> |
| 408 | |
| 409 | <para> |
| 410 | The graphs are in |
| 411 | <ulink url='https://en.wikipedia.org/wiki/DOT_%28graph_description_language%29'>DOT</ulink> |
| 412 | format and can be converted to images (e.g. using the |
| 413 | <filename>dot</filename> tool from |
| 414 | <ulink url='http://www.graphviz.org/'>Graphviz</ulink>). |
| 415 | <note><title>Notes</title> |
| 416 | <itemizedlist> |
| 417 | <listitem><para> |
| 418 | DOT files use a plain text format. |
| 419 | The graphs generated using the |
| 420 | <filename>bitbake -g</filename> command are often so |
| 421 | large as to be difficult to read without special |
| 422 | pruning (e.g. with Bitbake's |
| 423 | <filename>-I</filename> option) and processing. |
| 424 | Despite the form and size of the graphs, the |
| 425 | corresponding <filename>.dot</filename> files can still |
| 426 | be possible to read and provide useful information. |
| 427 | </para> |
| 428 | |
| 429 | <para>As an example, the |
| 430 | <filename>task-depends.dot</filename> file contains |
| 431 | lines such as the following: |
| 432 | <literallayout class='monospaced'> |
| 433 | "libxslt.do_configure" -> "libxml2.do_populate_sysroot" |
| 434 | </literallayout> |
| 435 | The above example line reveals that the |
| 436 | <link linkend='ref-tasks-configure'><filename>do_configure</filename></link> |
| 437 | task in <filename>libxslt</filename> depends on the |
| 438 | <link linkend='ref-tasks-populate_sysroot'><filename>do_populate_sysroot</filename></link> |
| 439 | task in <filename>libxml2</filename>, which is a normal |
| 440 | <link linkend='var-DEPENDS'><filename>DEPENDS</filename></link> |
| 441 | dependency between the two recipes. |
| 442 | </para></listitem> |
| 443 | <listitem><para> |
| 444 | For an example of how <filename>.dot</filename> files |
| 445 | can be processed, see the |
| 446 | <filename>scripts/contrib/graph-tool</filename> Python |
| 447 | script, which finds and displays paths between graph |
| 448 | nodes. |
| 449 | </para></listitem> |
| 450 | </itemizedlist> |
| 451 | </note> |
| 452 | </para> |
| 453 | |
| 454 | <para> |
| 455 | You can use a different method to view dependency information |
| 456 | by using the following command: |
| 457 | <literallayout class='monospaced'> |
| 458 | $ bitbake -g -u depexp <replaceable>recipename</replaceable> |
| 459 | </literallayout> |
| 460 | This command displays a GUI window from which you can view |
| 461 | build-time and runtime dependencies for the recipes involved in |
| 462 | building <replaceable>recipename</replaceable>. |
| 463 | </para> |
| 464 | </section> |
| 465 | |
| 466 | <section id='usingpoky-viewing-task-variable-dependencies'> |
| 467 | <title>Viewing Task Variable Dependencies</title> |
| 468 | |
| 469 | <para> |
| 470 | As mentioned in the |
| 471 | "<ulink url='&YOCTO_DOCS_BB_URL;#checksums'>Checksums (Signatures)</ulink>" |
| 472 | section of the BitBake User Manual, BitBake tries to automatically |
| 473 | determine what variables a task depends on so that it can rerun |
| 474 | the task if any values of the variables change. |
| 475 | This determination is usually reliable. |
| 476 | However, if you do things like construct variable names at runtime, |
| 477 | then you might have to manually declare dependencies on those |
| 478 | variables using <filename>vardeps</filename> as described in the |
| 479 | "<ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'>Variable Flags</ulink>" |
| 480 | section of the BitBake User Manual. |
| 481 | </para> |
| 482 | |
| 483 | <para> |
| 484 | If you are unsure whether a variable dependency is being picked up |
| 485 | automatically for a given task, you can list the variable |
| 486 | dependencies BitBake has determined by doing the following: |
| 487 | <orderedlist> |
| 488 | <listitem><para> |
| 489 | Build the recipe containing the task: |
| 490 | <literallayout class='monospaced'> |
| 491 | $ bitbake <replaceable>recipename</replaceable> |
| 492 | </literallayout> |
| 493 | </para></listitem> |
| 494 | <listitem><para> |
| 495 | Inside the |
| 496 | <link linkend='var-STAMPS_DIR'><filename>STAMPS_DIR</filename></link> |
| 497 | directory, find the signature data |
| 498 | (<filename>sigdata</filename>) file that corresponds to the |
| 499 | task. |
| 500 | The <filename>sigdata</filename> files contain a pickled |
| 501 | Python database of all the metadata that went into creating |
| 502 | the input checksum for the task. |
| 503 | As an example, for the |
| 504 | <link linkend='ref-tasks-fetch'><filename>do_fetch</filename></link> |
| 505 | task of the <filename>db</filename> recipe, the |
| 506 | <filename>sigdata</filename> file might be found in the |
| 507 | following location: |
| 508 | <literallayout class='monospaced'> |
| 509 | ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1 |
| 510 | </literallayout> |
| 511 | For tasks that are accelerated through the shared state |
| 512 | (<link linkend='shared-state-cache'>sstate</link>) |
| 513 | cache, an additional <filename>siginfo</filename> file is |
| 514 | written into |
| 515 | <link linkend='var-SSTATE_DIR'><filename>SSTATE_DIR</filename></link> |
| 516 | along with the cached task output. |
| 517 | The <filename>siginfo</filename> files contain exactly the |
| 518 | same information as <filename>sigdata</filename> files. |
| 519 | </para></listitem> |
| 520 | <listitem><para> |
| 521 | Run <filename>bitbake-dumpsig</filename> on the |
| 522 | <filename>sigdata</filename> or |
| 523 | <filename>siginfo</filename> file. |
| 524 | Here is an example: |
| 525 | <literallayout class='monospaced'> |
| 526 | $ bitbake-dumpsig ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1 |
| 527 | </literallayout> |
| 528 | In the output of the above command, you will find a line |
| 529 | like the following, which lists all the (inferred) variable |
| 530 | dependencies for the task. |
| 531 | This list also includes indirect dependencies from |
| 532 | variables depending on other variables, recursively. |
| 533 | <literallayout class='monospaced'> |
| 534 | Task dependencies: ['PV', 'SRCREV', 'SRC_URI', 'SRC_URI[md5sum]', 'SRC_URI[sha256sum]', 'base_do_fetch'] |
| 535 | </literallayout> |
| 536 | <note> |
| 537 | Functions (e.g. <filename>base_do_fetch</filename>) |
| 538 | also count as variable dependencies. |
| 539 | These functions in turn depend on the variables they |
| 540 | reference. |
| 541 | </note> |
| 542 | The output of <filename>bitbake-dumpsig</filename> also includes |
| 543 | the value each variable had, a list of dependencies for each |
| 544 | variable, and |
| 545 | <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_HASHBASE_WHITELIST'><filename>BB_HASHBASE_WHITELIST</filename></ulink> |
| 546 | information. |
| 547 | </para></listitem> |
| 548 | </orderedlist> |
| 549 | </para> |
| 550 | |
| 551 | <para> |
| 552 | There is also a <filename>bitbake-diffsigs</filename> command for |
| 553 | comparing two <filename>siginfo</filename> or |
| 554 | <filename>sigdata</filename> files. |
| 555 | This command can be helpful when trying to figure out what changed |
| 556 | between two versions of a task. |
| 557 | If you call <filename>bitbake-diffsigs</filename> with just one |
| 558 | file, the command behaves like |
| 559 | <filename>bitbake-dumpsig</filename>. |
| 560 | </para> |
| 561 | |
| 562 | <para> |
| 563 | You can also use BitBake to dump out the signature construction |
| 564 | information without executing tasks by using either of the |
| 565 | following BitBake command-line options: |
| 566 | <literallayout class='monospaced'> |
| 567 | ‐‐dump-signatures=<replaceable>SIGNATURE_HANDLER</replaceable> |
| 568 | -S <replaceable>SIGNATURE_HANDLER</replaceable> |
| 569 | </literallayout> |
| 570 | <note> |
| 571 | Two common values for |
| 572 | <replaceable>SIGNATURE_HANDLER</replaceable> are "none" and |
| 573 | "printdiff", which dump only the signature or compare the |
| 574 | dumped signature with the cached one, respectively. |
| 575 | </note> |
| 576 | Using BitBake with either of these options causes BitBake to dump |
| 577 | out <filename>sigdata</filename> files in the |
| 578 | <filename>stamps</filename> directory for every task it would have |
| 579 | executed instead of building the specified target package. |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 580 | </para> |
| 581 | </section> |
| 582 | |
| 583 | <section id='usingpoky-debugging-taskrunning'> |
| 584 | <title>Running Specific Tasks</title> |
| 585 | |
| 586 | <para> |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 587 | Any given recipe consists of a set of tasks. |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 588 | The standard BitBake behavior in most cases is: |
| 589 | <filename>do_fetch</filename>, |
| 590 | <filename>do_unpack</filename>, |
| 591 | <filename>do_patch</filename>, <filename>do_configure</filename>, |
| 592 | <filename>do_compile</filename>, <filename>do_install</filename>, |
| 593 | <filename>do_package</filename>, |
| 594 | <filename>do_package_write_*</filename>, and |
| 595 | <filename>do_build</filename>. |
| 596 | The default task is <filename>do_build</filename> and any tasks |
| 597 | on which it depends build first. |
| 598 | Some tasks, such as <filename>do_devshell</filename>, are not part |
| 599 | of the default build chain. |
| 600 | If you wish to run a task that is not part of the default build |
| 601 | chain, you can use the <filename>-c</filename> option in BitBake. |
| 602 | Here is an example: |
| 603 | <literallayout class='monospaced'> |
| 604 | $ bitbake matchbox-desktop -c devshell |
| 605 | </literallayout> |
| 606 | </para> |
| 607 | |
| 608 | <para> |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 609 | The <filename>-c</filename> option respects task dependencies, |
| 610 | which means that all other tasks (including tasks from other |
| 611 | recipes) that the specified task depends on will be run before the |
| 612 | task. |
| 613 | Even when you manually specify a task to run with |
| 614 | <filename>-c</filename>, BitBake will only run the task if it |
| 615 | considers it "out of date". |
| 616 | See the |
| 617 | "<link linkend='stamp-files-and-the-rerunning-of-tasks'>Stamp Files and the Rerunning of Tasks</link>" |
| 618 | section for how BitBake determines whether a task is "out of date". |
| 619 | </para> |
| 620 | |
| 621 | <para> |
| 622 | If you want to force an up-to-date task to be rerun (e.g. |
| 623 | because you made manual modifications to the recipe's |
| 624 | <link linkend='var-WORKDIR'><filename>WORKDIR</filename></link> |
| 625 | that you want to try out), then you can use the |
| 626 | <filename>-f</filename> option. |
| 627 | <note> |
| 628 | The reason <filename>-f</filename> is never required when |
| 629 | running the |
| 630 | <link linkend='ref-tasks-devshell'><filename>do_devshell</filename></link> |
| 631 | task is because the |
| 632 | <filename>[</filename><ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'><filename>nostamp</filename></ulink><filename>]</filename> |
| 633 | variable flag is already set for the task. |
| 634 | </note> |
| 635 | The following example shows one way you can use the |
| 636 | <filename>-f</filename> option: |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 637 | <literallayout class='monospaced'> |
| 638 | $ bitbake matchbox-desktop |
| 639 | . |
| 640 | . |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 641 | make some changes to the source code in the work directory |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 642 | . |
| 643 | . |
| 644 | $ bitbake matchbox-desktop -c compile -f |
| 645 | $ bitbake matchbox-desktop |
| 646 | </literallayout> |
| 647 | </para> |
| 648 | |
| 649 | <para> |
| 650 | This sequence first builds and then recompiles |
| 651 | <filename>matchbox-desktop</filename>. |
| 652 | The last command reruns all tasks (basically the packaging tasks) |
| 653 | after the compile. |
| 654 | BitBake recognizes that the <filename>do_compile</filename> |
| 655 | task was rerun and therefore understands that the other tasks |
| 656 | also need to be run again. |
| 657 | </para> |
| 658 | |
| 659 | <para> |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 660 | Another, shorter way to rerun a task and all |
| 661 | <link linkend='normal-recipe-build-tasks'>normal recipe build tasks</link> |
| 662 | that depend on it is to use the <filename>-C</filename> |
| 663 | option. |
| 664 | <note> |
| 665 | This option is upper-cased and is separate from the |
| 666 | <filename>-c</filename> option, which is lower-cased. |
| 667 | </note> |
| 668 | Using this option invalidates the given task and then runs the |
| 669 | <link linkend='ref-tasks-build'><filename>do_build</filename></link> |
| 670 | task, which is the default task if no task is given, and the |
| 671 | tasks on which it depends. |
| 672 | You could replace the final two commands in the previous example |
| 673 | with the following single command: |
| 674 | <literallayout class='monospaced'> |
| 675 | $ bitbake matchbox-desktop -C compile |
| 676 | </literallayout> |
| 677 | Internally, the <filename>-f</filename> and |
| 678 | <filename>-C</filename> options work by tainting (modifying) the |
| 679 | input checksum of the specified task. |
| 680 | This tainting indirectly causes the task and its |
| 681 | dependent tasks to be rerun through the normal task dependency |
| 682 | mechanisms. |
| 683 | <note> |
| 684 | BitBake explicitly keeps track of which tasks have been |
| 685 | tainted in this fashion, and will print warnings such as the |
| 686 | following for builds involving such tasks: |
| 687 | <literallayout class='monospaced'> |
| 688 | WARNING: /home/ulf/poky/meta/recipes-sato/matchbox-desktop/matchbox-desktop_2.1.bb.do_compile is tainted from a forced run |
| 689 | </literallayout> |
| 690 | The purpose of the warning is to let you know that the work |
| 691 | directory and build output might not be in the clean state they |
| 692 | would be in for a "normal" build, depending on what actions |
| 693 | you took. |
| 694 | To get rid of such warnings, you can remove the work directory |
| 695 | and rebuild the recipe, as follows: |
| 696 | <literallayout class='monospaced'> |
| 697 | $ bitbake matchbox-desktop -c clean |
| 698 | $ bitbake matchbox-desktop |
| 699 | </literallayout> |
| 700 | </note> |
| 701 | </para> |
| 702 | |
| 703 | <para> |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 704 | You can view a list of tasks in a given package by running the |
| 705 | <filename>do_listtasks</filename> task as follows: |
| 706 | <literallayout class='monospaced'> |
| 707 | $ bitbake matchbox-desktop -c listtasks |
| 708 | </literallayout> |
| 709 | The results appear as output to the console and are also in the |
| 710 | file <filename>${WORKDIR}/temp/log.do_listtasks</filename>. |
| 711 | </para> |
| 712 | </section> |
| 713 | |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 714 | <section id='checking-for-missing-build-time-dependencies'> |
| 715 | <title>Checking for Missing Build-Time Dependencies</title> |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 716 | |
| 717 | <para> |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 718 | A recipe might build successfully even though some of its |
| 719 | build-time dependencies are missing from |
| 720 | <link linkend='var-DEPENDS'><filename>DEPENDS</filename></link>. |
| 721 | Following are the two most common ways in which that can happen: |
| 722 | <itemizedlist> |
| 723 | <listitem><para> |
| 724 | The build-time dependency just happens to already exist in |
| 725 | the staging sysroot |
| 726 | (<link linkend='var-STAGING_DIR_HOST'><filename>STAGING_DIR_HOST</filename></link>) |
| 727 | by the time the recipe is built. |
| 728 | This situation occurs when the build-time dependency is |
| 729 | built earlier during recipe processing. |
| 730 | </para></listitem> |
| 731 | <listitem><para> |
| 732 | The component built by the recipe conditionally enables |
| 733 | functionality depending on whether it can find the |
| 734 | build-time dependency in the staging sysroot. |
| 735 | If the build-time dependency is missing, the corresponding |
| 736 | functionality is disabled. |
| 737 | This condition is known as a "floating dependency". |
| 738 | </para></listitem> |
| 739 | </itemizedlist> |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 740 | </para> |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 741 | |
| 742 | <para> |
| 743 | Because dealing with the second case is more complex, focus will |
| 744 | be on the first case. |
| 745 | The |
| 746 | <link linkend='ref-classes-insane'><filename>build-deps</filename></link> |
| 747 | QA check checks that every library the component linked against is |
| 748 | declared as a build-time dependency. |
| 749 | If that is not the case, then the first situation described in the |
| 750 | previous list exists, and <filename>build-deps</filename> reports |
| 751 | a missing build-time dependency. |
| 752 | </para> |
| 753 | |
| 754 | <para> |
| 755 | Another, more manual, way to check a recipe for missing build-time |
| 756 | dependencies of the first type is to build with an empty staging |
| 757 | sysroot. |
| 758 | This method can also find missing build-time dependencies |
| 759 | that are not in the form of libraries, which the |
| 760 | <filename>build-deps</filename> QA check is unable to find. |
| 761 | </para> |
| 762 | |
| 763 | <para> |
| 764 | An easy way to empty the staging sysroots is to simply remove |
| 765 | <link linkend='var-TMPDIR'><filename>TMPDIR</filename></link>, |
| 766 | which is usually |
| 767 | <filename>${</filename><link linkend='var-BUILDDIR'><filename>BUILDDIR</filename></link><filename>}/tmp</filename>, |
| 768 | as it includes the staging sysroots. |
| 769 | Another, faster method to empty the staging sysroots is to use the |
| 770 | <filename>scripts/wipe-sysroot</filename> script, |
| 771 | which removes just the staging sysroots and keeps everything else |
| 772 | in <filename>TMPDIR</filename>. |
| 773 | <note> |
| 774 | The <filename>scripts/</filename> directory appears in |
| 775 | <filename>PATH</filename> after running the build environment |
| 776 | initialization script (i.e. |
| 777 | <link linkend='structure-core-script'><filename>oe-init-build-env</filename></link> |
| 778 | or |
| 779 | <link linkend='structure-memres-core-script'><filename>oe-init-build-env-memres</filename></link>), |
| 780 | which results in the ability to to run |
| 781 | <filename>wipe-sysroot</filename> immediately. |
| 782 | </note> |
| 783 | </para> |
| 784 | |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 785 | </section> |
| 786 | |
| 787 | <section id='usingpoky-debugging-bitbake'> |
| 788 | <title>General BitBake Problems</title> |
| 789 | |
| 790 | <para> |
| 791 | You can see debug output from BitBake by using the <filename>-D</filename> option. |
| 792 | The debug output gives more information about what BitBake |
| 793 | is doing and the reason behind it. |
| 794 | Each <filename>-D</filename> option you use increases the logging level. |
| 795 | The most common usage is <filename>-DDD</filename>. |
| 796 | </para> |
| 797 | |
| 798 | <para> |
| 799 | The output from <filename>bitbake -DDD -v</filename> <replaceable>targetname</replaceable> can reveal why |
| 800 | BitBake chose a certain version of a package or why BitBake |
| 801 | picked a certain provider. |
| 802 | This command could also help you in a situation where you think BitBake did something |
| 803 | unexpected. |
| 804 | </para> |
| 805 | </section> |
| 806 | |
| 807 | <section id='development-host-system-issues'> |
| 808 | <title>Development Host System Issues</title> |
| 809 | |
| 810 | <para> |
| 811 | Sometimes issues on the host development system can cause your |
| 812 | build to fail. |
| 813 | Following are known, host-specific problems. |
| 814 | Be sure to always consult the |
| 815 | <ulink url='&YOCTO_RELEASE_NOTES;'>Release Notes</ulink> |
| 816 | for a look at all release-related issues. |
| 817 | <itemizedlist> |
| 818 | <listitem><para><emphasis><filename>glibc-initial</filename> fails to build</emphasis>: |
| 819 | If your development host system has the unpatched |
| 820 | <filename>GNU Make 3.82</filename>, |
| 821 | the |
| 822 | <link linkend='ref-tasks-install'><filename>do_install</filename></link> |
| 823 | task fails for <filename>glibc-initial</filename> during |
| 824 | the build.</para> |
| 825 | <para>Typically, every distribution that ships |
| 826 | <filename>GNU Make 3.82</filename> as |
| 827 | the default already has the patched version. |
| 828 | However, some distributions, such as Debian, have |
| 829 | <filename>GNU Make 3.82</filename> as an option, which |
| 830 | is unpatched. |
| 831 | You will see this error on these types of distributions. |
| 832 | Switch to <filename>GNU Make 3.81</filename> or patch |
| 833 | your <filename>make</filename> to solve the problem. |
| 834 | </para></listitem> |
| 835 | </itemizedlist> |
| 836 | </para> |
| 837 | </section> |
| 838 | |
| 839 | <section id='usingpoky-debugging-buildfile'> |
| 840 | <title>Building with No Dependencies</title> |
| 841 | <para> |
| 842 | To build a specific recipe (<filename>.bb</filename> file), |
| 843 | you can use the following command form: |
| 844 | <literallayout class='monospaced'> |
| 845 | $ bitbake -b <replaceable>somepath</replaceable>/<replaceable>somerecipe</replaceable>.bb |
| 846 | </literallayout> |
| 847 | This command form does not check for dependencies. |
| 848 | Consequently, you should use it |
| 849 | only when you know existing dependencies have been met. |
| 850 | <note> |
| 851 | You can also specify fragments of the filename. |
| 852 | In this case, BitBake checks for a unique match. |
| 853 | </note> |
| 854 | </para> |
| 855 | </section> |
| 856 | |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 857 | <section id='recipe-logging-mechanisms'> |
| 858 | <title>Recipe Logging Mechanisms</title> |
| 859 | <para> |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 860 | The Yocto Project provides several logging functions for producing |
| 861 | debugging output and reporting errors and warnings. |
| 862 | For Python functions, the following logging functions exist. |
| 863 | All of these functions log to |
| 864 | <filename>${T}/log.do_</filename><replaceable>task</replaceable>, |
| 865 | and can also log to standard output (stdout) with the right |
| 866 | settings: |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 867 | <itemizedlist> |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 868 | <listitem><para> |
| 869 | <filename>bb.plain(</filename><replaceable>msg</replaceable><filename>)</filename>: |
| 870 | Writes <replaceable>msg</replaceable> as is to the log while |
| 871 | also logging to stdout. |
| 872 | </para></listitem> |
| 873 | <listitem><para> |
| 874 | <filename>bb.note(</filename><replaceable>msg</replaceable><filename>)</filename>: |
| 875 | Writes "NOTE: <replaceable>msg</replaceable>" to the log. |
| 876 | Also logs to stdout if BitBake is called with "-v". |
| 877 | </para></listitem> |
| 878 | <listitem><para> |
| 879 | <filename>bb.debug(</filename><replaceable>level</replaceable><filename>, </filename><replaceable>msg</replaceable><filename>)</filename>: |
| 880 | Writes "DEBUG: <replaceable>msg</replaceable>" to the log. |
| 881 | Also logs to stdout if the log level is greater than or |
| 882 | equal to <replaceable>level</replaceable>. |
| 883 | See the |
| 884 | "<ulink url='&YOCTO_DOCS_BB_URL;#usage-and-syntax'>-D</ulink>" |
| 885 | option in the BitBake User Manual for more information. |
| 886 | </para></listitem> |
| 887 | <listitem><para> |
| 888 | <filename>bb.warn(</filename><replaceable>msg</replaceable><filename>)</filename>: |
| 889 | Writes "WARNING: <replaceable>msg</replaceable>" to the log |
| 890 | while also logging to stdout. |
| 891 | </para></listitem> |
| 892 | <listitem><para> |
| 893 | <filename>bb.error(</filename><replaceable>msg</replaceable><filename>)</filename>: |
| 894 | Writes "ERROR: <replaceable>msg</replaceable>" to the log |
| 895 | while also logging to stdout. |
| 896 | <note> |
| 897 | Calling this function does not cause the task to fail. |
| 898 | </note> |
| 899 | </para></listitem> |
| 900 | <listitem><para> |
| 901 | <filename>bb.fatal(</filename><replaceable>msg</replaceable><filename>)</filename>: |
| 902 | This logging function is similar to |
| 903 | <filename>bb.error(</filename><replaceable>msg</replaceable><filename>)</filename> |
| 904 | but also causes the calling task to fail. |
| 905 | <note> |
| 906 | <filename>bb.fatal()</filename> raises an exception, |
| 907 | which means you do not need to put a "return" |
| 908 | statement after the function. |
| 909 | </note> |
| 910 | </para></listitem> |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 911 | </itemizedlist> |
| 912 | </para> |
| 913 | |
| 914 | <para> |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 915 | The same logging functions are also available in shell functions, |
| 916 | under the names |
| 917 | <filename>bbplain</filename>, <filename>bbnote</filename>, |
| 918 | <filename>bbdebug</filename>, <filename>bbwarn</filename>, |
| 919 | <filename>bberror</filename>, and <filename>bbfatal</filename>. |
| 920 | The |
| 921 | <link linkend='ref-classes-logging'><filename>logging</filename></link> |
| 922 | class implements these functions. |
| 923 | See that class in the |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 924 | <filename>meta/classes</filename> folder of the |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 925 | <ulink url='&YOCTO_DOCS_DEV_URL;#source-directory'>Source Directory</ulink> |
| 926 | for information. |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 927 | </para> |
| 928 | |
| 929 | <section id='logging-with-python'> |
| 930 | <title>Logging With Python</title> |
| 931 | <para> |
| 932 | When creating recipes using Python and inserting code that handles build logs, |
| 933 | keep in mind the goal is to have informative logs while keeping the console as |
| 934 | "silent" as possible. |
| 935 | Also, if you want status messages in the log, use the "debug" loglevel. |
| 936 | </para> |
| 937 | |
| 938 | <para> |
| 939 | Following is an example written in Python. |
| 940 | The code handles logging for a function that determines the |
| 941 | number of tasks needed to be run. |
| 942 | See the |
| 943 | "<link linkend='ref-tasks-listtasks'><filename>do_listtasks</filename></link>" |
| 944 | section for additional information: |
| 945 | <literallayout class='monospaced'> |
| 946 | python do_listtasks() { |
| 947 | bb.debug(2, "Starting to figure out the task list") |
| 948 | if noteworthy_condition: |
| 949 | bb.note("There are 47 tasks to run") |
| 950 | bb.debug(2, "Got to point xyz") |
| 951 | if warning_trigger: |
| 952 | bb.warn("Detected warning_trigger, this might be a problem later.") |
| 953 | if recoverable_error: |
| 954 | bb.error("Hit recoverable_error, you really need to fix this!") |
| 955 | if fatal_error: |
| 956 | bb.fatal("fatal_error detected, unable to print the task list") |
| 957 | bb.plain("The tasks present are abc") |
| 958 | bb.debug(2, "Finished figuring out the tasklist") |
| 959 | } |
| 960 | </literallayout> |
| 961 | </para> |
| 962 | </section> |
| 963 | |
| 964 | <section id='logging-with-bash'> |
| 965 | <title>Logging With Bash</title> |
| 966 | <para> |
| 967 | When creating recipes using Bash and inserting code that handles build |
| 968 | logs, you have the same goals - informative with minimal console output. |
| 969 | The syntax you use for recipes written in Bash is similar to that of |
| 970 | recipes written in Python described in the previous section. |
| 971 | </para> |
| 972 | |
| 973 | <para> |
| 974 | Following is an example written in Bash. |
| 975 | The code logs the progress of the <filename>do_my_function</filename> function. |
| 976 | <literallayout class='monospaced'> |
| 977 | do_my_function() { |
| 978 | bbdebug 2 "Running do_my_function" |
| 979 | if [ exceptional_condition ]; then |
| 980 | bbnote "Hit exceptional_condition" |
| 981 | fi |
| 982 | bbdebug 2 "Got to point xyz" |
| 983 | if [ warning_trigger ]; then |
| 984 | bbwarn "Detected warning_trigger, this might cause a problem later." |
| 985 | fi |
| 986 | if [ recoverable_error ]; then |
| 987 | bberror "Hit recoverable_error, correcting" |
| 988 | fi |
| 989 | if [ fatal_error ]; then |
| 990 | bbfatal "fatal_error detected" |
| 991 | fi |
| 992 | bbdebug 2 "Completed do_my_function" |
| 993 | } |
| 994 | </literallayout> |
| 995 | </para> |
| 996 | </section> |
| 997 | </section> |
| 998 | |
| 999 | <section id='usingpoky-debugging-others'> |
| 1000 | <title>Other Tips</title> |
| 1001 | |
| 1002 | <para> |
| 1003 | Here are some other tips that you might find useful: |
| 1004 | <itemizedlist> |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 1005 | <listitem><para> |
| 1006 | When adding new packages, it is worth watching for |
| 1007 | undesirable items making their way into compiler command |
| 1008 | lines. |
| 1009 | For example, you do not want references to local system |
| 1010 | files like |
| 1011 | <filename>/usr/lib/</filename> or |
| 1012 | <filename>/usr/include/</filename>. |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 1013 | </para></listitem> |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 1014 | <listitem><para> |
| 1015 | If you want to remove the <filename>psplash</filename> |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 1016 | boot splashscreen, |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 1017 | add <filename>psplash=false</filename> to the kernel |
| 1018 | command line. |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 1019 | Doing so prevents <filename>psplash</filename> from loading |
| 1020 | and thus allows you to see the console. |
| 1021 | It is also possible to switch out of the splashscreen by |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 1022 | switching the virtual console (e.g. Fn+Left or Fn+Right |
| 1023 | on a Zaurus). |
| 1024 | </para></listitem> |
| 1025 | <listitem><para> |
| 1026 | Removing |
| 1027 | <link linkend='var-TMPDIR'><filename>TMPDIR</filename></link> |
| 1028 | (usually <filename>tmp/</filename>, within the |
| 1029 | <ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink>) |
| 1030 | can often fix temporary build issues. |
| 1031 | Removing <filename>TMPDIR</filename> is usually a |
| 1032 | relatively cheap operation, because task output will be |
| 1033 | cached in |
| 1034 | <link linkend='var-SSTATE_DIR'><filename>SSTATE_DIR</filename></link> |
| 1035 | (usually <filename>sstate-cache/</filename>, which is |
| 1036 | also in the Build Directory). |
| 1037 | <note> |
| 1038 | Removing <filename>TMPDIR</filename> might be a |
| 1039 | workaround rather than a fix. |
| 1040 | Consequently, trying to determine the underlying cause |
| 1041 | of an issue before removing the directory is a good |
| 1042 | idea. |
| 1043 | </note> |
| 1044 | </para></listitem> |
| 1045 | <listitem><para> |
| 1046 | Understanding how a feature is used in practice within |
| 1047 | existing recipes can be very helpful. |
| 1048 | It is recommended that you configure some method that |
| 1049 | allows you to quickly search through files.</para> |
| 1050 | |
| 1051 | <para>Using GNU Grep, you can use the following shell |
| 1052 | function to recursively search through common |
| 1053 | recipe-related files, skipping binary files, |
| 1054 | <filename>.git</filename> directories, and the |
| 1055 | Build Directory (assuming its name starts with |
| 1056 | "build"): |
| 1057 | <literallayout class='monospaced'> |
| 1058 | g() { |
| 1059 | grep -Ir \ |
| 1060 | --exclude-dir=.git \ |
| 1061 | --exclude-dir='build*' \ |
| 1062 | --include='*.bb*' \ |
| 1063 | --include='*.inc*' \ |
| 1064 | --include='*.conf*' \ |
| 1065 | --include='*.py*' \ |
| 1066 | "$@" |
| 1067 | } |
| 1068 | </literallayout> |
| 1069 | Following are some usage examples: |
| 1070 | <literallayout class='monospaced'> |
| 1071 | $ g FOO # Search recursively for "FOO" |
| 1072 | $ g -i foo # Search recursively for "foo", ignoring case |
| 1073 | $ g -w FOO # Search recursively for "FOO" as a word, ignoring e.g. "FOOBAR" |
| 1074 | </literallayout> |
| 1075 | If figuring out how some feature works requires a lot of |
| 1076 | searching, it might indicate that the documentation should |
| 1077 | be extended or improved. |
| 1078 | In such cases, consider filing a documentation bug using |
| 1079 | the Yocto Project implementation of |
| 1080 | <ulink url='https://bugzilla.yoctoproject.org/'>Bugzilla</ulink>. |
| 1081 | For general information on how to submit a bug against |
| 1082 | the Yocto Project, see the |
| 1083 | "<ulink url='&YOCTO_DOCS_DEV_URL;#tracking-bugs'>Tracking Bugs</ulink>" |
| 1084 | section in the Yocto Project Development Manual. |
| 1085 | <note> |
| 1086 | The manuals might not be the right place to document |
| 1087 | variables that are purely internal and have a limited |
| 1088 | scope (e.g. internal variables used to implement a |
| 1089 | single <filename>.bbclass</filename> file). |
| 1090 | </note> |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 1091 | </para></listitem> |
| 1092 | </itemizedlist> |
| 1093 | </para> |
| 1094 | </section> |
| 1095 | </section> |
| 1096 | |
| 1097 | <section id='maintaining-build-output-quality'> |
| 1098 | <title>Maintaining Build Output Quality</title> |
| 1099 | |
| 1100 | <para> |
| 1101 | Many factors can influence the quality of a build. |
| 1102 | For example, if you upgrade a recipe to use a new version of an upstream software |
| 1103 | package or you experiment with some new configuration options, subtle changes |
| 1104 | can occur that you might not detect until later. |
| 1105 | Consider the case where your recipe is using a newer version of an upstream package. |
| 1106 | In this case, a new version of a piece of software might introduce an optional |
| 1107 | dependency on another library, which is auto-detected. |
| 1108 | If that library has already been built when the software is building, |
| 1109 | the software will link to the built library and that library will be pulled |
| 1110 | into your image along with the new software even if you did not want the |
| 1111 | library. |
| 1112 | </para> |
| 1113 | |
| 1114 | <para> |
| 1115 | The |
| 1116 | <link linkend='ref-classes-buildhistory'><filename>buildhistory</filename></link> |
| 1117 | class exists to help you maintain |
| 1118 | the quality of your build output. |
| 1119 | You can use the class to highlight unexpected and possibly unwanted |
| 1120 | changes in the build output. |
| 1121 | When you enable build history, it records information about the contents of |
| 1122 | each package and image and then commits that information to a local Git |
| 1123 | repository where you can examine the information. |
| 1124 | </para> |
| 1125 | |
| 1126 | <para> |
| 1127 | The remainder of this section describes the following: |
| 1128 | <itemizedlist> |
| 1129 | <listitem><para>How you can enable and disable |
| 1130 | build history</para></listitem> |
| 1131 | <listitem><para>How to understand what the build history contains |
| 1132 | </para></listitem> |
| 1133 | <listitem><para>How to limit the information used for build history |
| 1134 | </para></listitem> |
| 1135 | <listitem><para>How to examine the build history from both a |
| 1136 | command-line and web interface</para></listitem> |
| 1137 | </itemizedlist> |
| 1138 | </para> |
| 1139 | |
| 1140 | <section id='enabling-and-disabling-build-history'> |
| 1141 | <title>Enabling and Disabling Build History</title> |
| 1142 | |
| 1143 | <para> |
| 1144 | Build history is disabled by default. |
| 1145 | To enable it, add the following <filename>INHERIT</filename> |
| 1146 | statement and set the |
| 1147 | <link linkend='var-BUILDHISTORY_COMMIT'><filename>BUILDHISTORY_COMMIT</filename></link> |
| 1148 | variable to "1" at the end of your |
| 1149 | <filename>conf/local.conf</filename> file found in the |
| 1150 | <ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink>: |
| 1151 | <literallayout class='monospaced'> |
| 1152 | INHERIT += "buildhistory" |
| 1153 | BUILDHISTORY_COMMIT = "1" |
| 1154 | </literallayout> |
| 1155 | Enabling build history as previously described |
| 1156 | causes the build process to collect build |
| 1157 | output information and commit it to a local |
| 1158 | <ulink url='&YOCTO_DOCS_DEV_URL;#git'>Git</ulink> repository. |
| 1159 | <note> |
| 1160 | Enabling build history increases your build times slightly, |
| 1161 | particularly for images, and increases the amount of disk |
| 1162 | space used during the build. |
| 1163 | </note> |
| 1164 | </para> |
| 1165 | |
| 1166 | <para> |
| 1167 | You can disable build history by removing the previous statements |
| 1168 | from your <filename>conf/local.conf</filename> file. |
| 1169 | </para> |
| 1170 | </section> |
| 1171 | |
| 1172 | <section id='understanding-what-the-build-history-contains'> |
| 1173 | <title>Understanding What the Build History Contains</title> |
| 1174 | |
| 1175 | <para> |
| 1176 | Build history information is kept in |
| 1177 | <filename>${</filename><link linkend='var-TOPDIR'><filename>TOPDIR</filename></link><filename>}/buildhistory</filename> |
| 1178 | in the Build Directory as defined by the |
| 1179 | <link linkend='var-BUILDHISTORY_DIR'><filename>BUILDHISTORY_DIR</filename></link> |
| 1180 | variable. |
| 1181 | The following is an example abbreviated listing: |
| 1182 | <imagedata fileref="figures/buildhistory.png" align="center" width="6in" depth="4in" /> |
| 1183 | </para> |
| 1184 | |
| 1185 | <para> |
| 1186 | At the top level, there is a <filename>metadata-revs</filename> file |
| 1187 | that lists the revisions of the repositories for the layers enabled |
| 1188 | when the build was produced. |
| 1189 | The rest of the data splits into separate |
| 1190 | <filename>packages</filename>, <filename>images</filename> and |
| 1191 | <filename>sdk</filename> directories, the contents of which are |
| 1192 | described below. |
| 1193 | </para> |
| 1194 | |
| 1195 | <section id='build-history-package-information'> |
| 1196 | <title>Build History Package Information</title> |
| 1197 | |
| 1198 | <para> |
| 1199 | The history for each package contains a text file that has |
| 1200 | name-value pairs with information about the package. |
| 1201 | For example, <filename>buildhistory/packages/i586-poky-linux/busybox/busybox/latest</filename> |
| 1202 | contains the following: |
| 1203 | <literallayout class='monospaced'> |
| 1204 | PV = 1.22.1 |
| 1205 | PR = r32 |
| 1206 | RPROVIDES = |
| 1207 | RDEPENDS = glibc (>= 2.20) update-alternatives-opkg |
| 1208 | RRECOMMENDS = busybox-syslog busybox-udhcpc update-rc.d |
| 1209 | PKGSIZE = 540168 |
| 1210 | FILES = /usr/bin/* /usr/sbin/* /usr/lib/busybox/* /usr/lib/lib*.so.* \ |
| 1211 | /etc /com /var /bin/* /sbin/* /lib/*.so.* /lib/udev/rules.d \ |
| 1212 | /usr/lib/udev/rules.d /usr/share/busybox /usr/lib/busybox/* \ |
| 1213 | /usr/share/pixmaps /usr/share/applications /usr/share/idl \ |
| 1214 | /usr/share/omf /usr/share/sounds /usr/lib/bonobo/servers |
| 1215 | FILELIST = /bin/busybox /bin/busybox.nosuid /bin/busybox.suid /bin/sh \ |
| 1216 | /etc/busybox.links.nosuid /etc/busybox.links.suid |
| 1217 | </literallayout> |
| 1218 | Most of these name-value pairs correspond to variables used |
| 1219 | to produce the package. |
| 1220 | The exceptions are <filename>FILELIST</filename>, which is the |
| 1221 | actual list of files in the package, and |
| 1222 | <filename>PKGSIZE</filename>, which is the total size of files |
| 1223 | in the package in bytes. |
| 1224 | </para> |
| 1225 | |
| 1226 | <para> |
| 1227 | There is also a file corresponding to the recipe from which the |
| 1228 | package came (e.g. |
| 1229 | <filename>buildhistory/packages/i586-poky-linux/busybox/latest</filename>): |
| 1230 | <literallayout class='monospaced'> |
| 1231 | PV = 1.22.1 |
| 1232 | PR = r32 |
| 1233 | DEPENDS = initscripts kern-tools-native update-rc.d-native \ |
| 1234 | virtual/i586-poky-linux-compilerlibs virtual/i586-poky-linux-gcc \ |
| 1235 | virtual/libc virtual/update-alternatives |
| 1236 | PACKAGES = busybox-ptest busybox-httpd busybox-udhcpd busybox-udhcpc \ |
| 1237 | busybox-syslog busybox-mdev busybox-hwclock busybox-dbg \ |
| 1238 | busybox-staticdev busybox-dev busybox-doc busybox-locale busybox |
| 1239 | </literallayout> |
| 1240 | </para> |
| 1241 | |
| 1242 | <para> |
| 1243 | Finally, for those recipes fetched from a version control |
| 1244 | system (e.g., Git), a file exists that lists source revisions |
| 1245 | that are specified in the recipe and lists the actual revisions |
| 1246 | used during the build. |
| 1247 | Listed and actual revisions might differ when |
| 1248 | <link linkend='var-SRCREV'><filename>SRCREV</filename></link> |
| 1249 | is set to |
| 1250 | <filename>${<link linkend='var-AUTOREV'>AUTOREV</link>}</filename>. |
| 1251 | Here is an example assuming |
| 1252 | <filename>buildhistory/packages/qemux86-poky-linux/linux-yocto/latest_srcrev</filename>): |
| 1253 | <literallayout class='monospaced'> |
| 1254 | # SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1" |
| 1255 | SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1" |
| 1256 | # SRCREV_meta = "a227f20eff056e511d504b2e490f3774ab260d6f" |
| 1257 | SRCREV_meta = "a227f20eff056e511d504b2e490f3774ab260d6f" |
| 1258 | </literallayout> |
| 1259 | You can use the <filename>buildhistory-collect-srcrevs</filename> |
| 1260 | command with the <filename>-a</filename> option to |
| 1261 | collect the stored <filename>SRCREV</filename> values |
| 1262 | from build history and report them in a format suitable for |
| 1263 | use in global configuration (e.g., |
| 1264 | <filename>local.conf</filename> or a distro include file) to |
| 1265 | override floating <filename>AUTOREV</filename> values to a |
| 1266 | fixed set of revisions. |
| 1267 | Here is some example output from this command: |
| 1268 | <literallayout class='monospaced'> |
| 1269 | $ buildhistory-collect-srcrevs -a |
| 1270 | # i586-poky-linux |
| 1271 | SRCREV_pn-glibc = "b8079dd0d360648e4e8de48656c5c38972621072" |
| 1272 | SRCREV_pn-glibc-initial = "b8079dd0d360648e4e8de48656c5c38972621072" |
| 1273 | SRCREV_pn-opkg-utils = "53274f087565fd45d8452c5367997ba6a682a37a" |
| 1274 | SRCREV_pn-kmod = "fd56638aed3fe147015bfa10ed4a5f7491303cb4" |
| 1275 | # x86_64-linux |
| 1276 | SRCREV_pn-gtk-doc-stub-native = "1dea266593edb766d6d898c79451ef193eb17cfa" |
| 1277 | SRCREV_pn-dtc-native = "65cc4d2748a2c2e6f27f1cf39e07a5dbabd80ebf" |
| 1278 | SRCREV_pn-update-rc.d-native = "eca680ddf28d024954895f59a241a622dd575c11" |
| 1279 | SRCREV_glibc_pn-cross-localedef-native = "b8079dd0d360648e4e8de48656c5c38972621072" |
| 1280 | SRCREV_localedef_pn-cross-localedef-native = "c833367348d39dad7ba018990bfdaffaec8e9ed3" |
| 1281 | SRCREV_pn-prelink-native = "faa069deec99bf61418d0bab831c83d7c1b797ca" |
| 1282 | SRCREV_pn-opkg-utils-native = "53274f087565fd45d8452c5367997ba6a682a37a" |
| 1283 | SRCREV_pn-kern-tools-native = "23345b8846fe4bd167efdf1bd8a1224b2ba9a5ff" |
| 1284 | SRCREV_pn-kmod-native = "fd56638aed3fe147015bfa10ed4a5f7491303cb4" |
| 1285 | # qemux86-poky-linux |
| 1286 | SRCREV_machine_pn-linux-yocto = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1" |
| 1287 | SRCREV_meta_pn-linux-yocto = "a227f20eff056e511d504b2e490f3774ab260d6f" |
| 1288 | # all-poky-linux |
| 1289 | SRCREV_pn-update-rc.d = "eca680ddf28d024954895f59a241a622dd575c11" |
| 1290 | </literallayout> |
| 1291 | <note> |
| 1292 | Here are some notes on using the |
| 1293 | <filename>buildhistory-collect-srcrevs</filename> command: |
| 1294 | <itemizedlist> |
| 1295 | <listitem><para>By default, only values where the |
| 1296 | <filename>SRCREV</filename> was |
| 1297 | not hardcoded (usually when <filename>AUTOREV</filename> |
| 1298 | was used) are reported. |
| 1299 | Use the <filename>-a</filename> option to see all |
| 1300 | <filename>SRCREV</filename> values. |
| 1301 | </para></listitem> |
| 1302 | <listitem><para>The output statements might not have any effect |
| 1303 | if overrides are applied elsewhere in the build system |
| 1304 | configuration. |
| 1305 | Use the <filename>-f</filename> option to add the |
| 1306 | <filename>forcevariable</filename> override to each output line |
| 1307 | if you need to work around this restriction. |
| 1308 | </para></listitem> |
| 1309 | <listitem><para>The script does apply special handling when |
| 1310 | building for multiple machines. |
| 1311 | However, the script does place a |
| 1312 | comment before each set of values that specifies |
| 1313 | which triplet to which they belong as shown above |
| 1314 | (e.g., <filename>i586-poky-linux</filename>). |
| 1315 | </para></listitem> |
| 1316 | </itemizedlist> |
| 1317 | </note> |
| 1318 | </para> |
| 1319 | </section> |
| 1320 | |
| 1321 | <section id='build-history-image-information'> |
| 1322 | <title>Build History Image Information</title> |
| 1323 | |
| 1324 | <para> |
| 1325 | The files produced for each image are as follows: |
| 1326 | <itemizedlist> |
| 1327 | <listitem><para><filename>image-files:</filename> |
| 1328 | A directory containing selected files from the root |
| 1329 | filesystem. |
| 1330 | The files are defined by |
| 1331 | <link linkend='var-BUILDHISTORY_IMAGE_FILES'><filename>BUILDHISTORY_IMAGE_FILES</filename></link>. |
| 1332 | </para></listitem> |
| 1333 | <listitem><para><filename>build-id.txt:</filename> |
| 1334 | Human-readable information about the build configuration |
| 1335 | and metadata source revisions. |
| 1336 | This file contains the full build header as printed |
| 1337 | by BitBake.</para></listitem> |
| 1338 | <listitem><para><filename>*.dot:</filename> |
| 1339 | Dependency graphs for the image that are |
| 1340 | compatible with <filename>graphviz</filename>. |
| 1341 | </para></listitem> |
| 1342 | <listitem><para><filename>files-in-image.txt:</filename> |
| 1343 | A list of files in the image with permissions, |
| 1344 | owner, group, size, and symlink information. |
| 1345 | </para></listitem> |
| 1346 | <listitem><para><filename>image-info.txt:</filename> |
| 1347 | A text file containing name-value pairs with information |
| 1348 | about the image. |
| 1349 | See the following listing example for more information. |
| 1350 | </para></listitem> |
| 1351 | <listitem><para><filename>installed-package-names.txt:</filename> |
| 1352 | A list of installed packages by name only.</para></listitem> |
| 1353 | <listitem><para><filename>installed-package-sizes.txt:</filename> |
| 1354 | A list of installed packages ordered by size. |
| 1355 | </para></listitem> |
| 1356 | <listitem><para><filename>installed-packages.txt:</filename> |
| 1357 | A list of installed packages with full package |
| 1358 | filenames.</para></listitem> |
| 1359 | </itemizedlist> |
| 1360 | <note> |
| 1361 | Installed package information is able to be gathered and |
| 1362 | produced even if package management is disabled for the final |
| 1363 | image. |
| 1364 | </note> |
| 1365 | </para> |
| 1366 | |
| 1367 | <para> |
| 1368 | Here is an example of <filename>image-info.txt</filename>: |
| 1369 | <literallayout class='monospaced'> |
| 1370 | DISTRO = poky |
| 1371 | DISTRO_VERSION = 1.7 |
| 1372 | USER_CLASSES = buildstats image-mklibs image-prelink |
| 1373 | IMAGE_CLASSES = image_types |
| 1374 | IMAGE_FEATURES = debug-tweaks |
| 1375 | IMAGE_LINGUAS = |
| 1376 | IMAGE_INSTALL = packagegroup-core-boot run-postinsts |
| 1377 | BAD_RECOMMENDATIONS = |
| 1378 | NO_RECOMMENDATIONS = |
| 1379 | PACKAGE_EXCLUDE = |
| 1380 | ROOTFS_POSTPROCESS_COMMAND = write_package_manifest; license_create_manifest; \ |
| 1381 | write_image_manifest ; buildhistory_list_installed_image ; \ |
| 1382 | buildhistory_get_image_installed ; ssh_allow_empty_password; \ |
| 1383 | postinst_enable_logging; rootfs_update_timestamp ; ssh_disable_dns_lookup ; |
| 1384 | IMAGE_POSTPROCESS_COMMAND = buildhistory_get_imageinfo ; |
| 1385 | IMAGESIZE = 6900 |
| 1386 | </literallayout> |
| 1387 | Other than <filename>IMAGESIZE</filename>, which is the |
| 1388 | total size of the files in the image in Kbytes, the |
| 1389 | name-value pairs are variables that may have influenced the |
| 1390 | content of the image. |
| 1391 | This information is often useful when you are trying to determine |
| 1392 | why a change in the package or file listings has occurred. |
| 1393 | </para> |
| 1394 | </section> |
| 1395 | |
| 1396 | <section id='using-build-history-to-gather-image-information-only'> |
| 1397 | <title>Using Build History to Gather Image Information Only</title> |
| 1398 | |
| 1399 | <para> |
| 1400 | As you can see, build history produces image information, |
| 1401 | including dependency graphs, so you can see why something |
| 1402 | was pulled into the image. |
| 1403 | If you are just interested in this information and not |
| 1404 | interested in collecting specific package or SDK information, |
| 1405 | you can enable writing only image information without |
| 1406 | any history by adding the following to your |
| 1407 | <filename>conf/local.conf</filename> file found in the |
| 1408 | <ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink>: |
| 1409 | <literallayout class='monospaced'> |
| 1410 | INHERIT += "buildhistory" |
| 1411 | BUILDHISTORY_COMMIT = "0" |
| 1412 | BUILDHISTORY_FEATURES = "image" |
| 1413 | </literallayout> |
| 1414 | Here, you set the |
| 1415 | <link linkend='var-BUILDHISTORY_FEATURES'><filename>BUILDHISTORY_FEATURES</filename></link> |
| 1416 | variable to use the image feature only. |
| 1417 | </para> |
| 1418 | </section> |
| 1419 | |
| 1420 | <section id='build-history-sdk-information'> |
| 1421 | <title>Build History SDK Information</title> |
Patrick Williams | d8c66bc | 2016-06-20 12:57:21 -0500 | [diff] [blame] | 1422 | |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 1423 | <para> |
| 1424 | Build history collects similar information on the contents |
Patrick Williams | d8c66bc | 2016-06-20 12:57:21 -0500 | [diff] [blame] | 1425 | of SDKs |
| 1426 | (e.g. <filename>bitbake -c populate_sdk imagename</filename>) |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 1427 | as compared to information it collects for images. |
Patrick Williams | d8c66bc | 2016-06-20 12:57:21 -0500 | [diff] [blame] | 1428 | Furthermore, this information differs depending on whether an |
| 1429 | extensible or standard SDK is being produced. |
| 1430 | </para> |
| 1431 | |
| 1432 | <para> |
| 1433 | The following list shows the files produced for SDKs: |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 1434 | <itemizedlist> |
| 1435 | <listitem><para><filename>files-in-sdk.txt:</filename> |
| 1436 | A list of files in the SDK with permissions, |
| 1437 | owner, group, size, and symlink information. |
| 1438 | This list includes both the host and target parts |
| 1439 | of the SDK. |
| 1440 | </para></listitem> |
| 1441 | <listitem><para><filename>sdk-info.txt:</filename> |
| 1442 | A text file containing name-value pairs with information |
| 1443 | about the SDK. |
| 1444 | See the following listing example for more information. |
| 1445 | </para></listitem> |
Patrick Williams | d8c66bc | 2016-06-20 12:57:21 -0500 | [diff] [blame] | 1446 | <listitem><para><filename>sstate-task-sizes.txt:</filename> |
| 1447 | A text file containing name-value pairs with information |
| 1448 | about task group sizes |
| 1449 | (e.g. <filename>do_populate_sysroot</filename> tasks |
| 1450 | have a total size). |
| 1451 | The <filename>sstate-task-sizes.txt</filename> file |
| 1452 | exists only when an extensible SDK is created. |
| 1453 | </para></listitem> |
| 1454 | <listitem><para><filename>sstate-package-sizes.txt:</filename> |
| 1455 | A text file containing name-value pairs with information |
| 1456 | for the shared-state packages and sizes in the SDK. |
| 1457 | The <filename>sstate-package-sizes.txt</filename> file |
| 1458 | exists only when an extensible SDK is created. |
| 1459 | </para></listitem> |
| 1460 | <listitem><para><filename>sdk-files:</filename> |
| 1461 | A folder that contains copies of the files mentioned in |
| 1462 | <filename>BUILDHISTORY_SDK_FILES</filename> if the |
| 1463 | files are present in the output. |
| 1464 | Additionally, the default value of |
| 1465 | <filename>BUILDHISTORY_SDK_FILES</filename> is specific |
| 1466 | to the extensible SDK although you can set it |
| 1467 | differently if you would like to pull in specific files |
| 1468 | from the standard SDK.</para> |
| 1469 | <para>The default files are |
| 1470 | <filename>conf/local.conf</filename>, |
| 1471 | <filename>conf/bblayers.conf</filename>, |
| 1472 | <filename>conf/auto.conf</filename>, |
| 1473 | <filename>conf/locked-sigs.inc</filename>, and |
| 1474 | <filename>conf/devtool.conf</filename>. |
| 1475 | Thus, for an extensible SDK, these files get copied |
| 1476 | into the <filename>sdk-files</filename> directory. |
| 1477 | </para></listitem> |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 1478 | <listitem><para>The following information appears under |
| 1479 | each of the <filename>host</filename> |
| 1480 | and <filename>target</filename> directories |
| 1481 | for the portions of the SDK that run on the host and |
| 1482 | on the target, respectively: |
Patrick Williams | d8c66bc | 2016-06-20 12:57:21 -0500 | [diff] [blame] | 1483 | <note> |
| 1484 | The following files for the most part are empty |
| 1485 | when producing an extensible SDK because this |
| 1486 | type of SDK is not constructed from packages as is |
| 1487 | the standard SDK. |
| 1488 | </note> |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 1489 | <itemizedlist> |
| 1490 | <listitem><para><filename>depends.dot:</filename> |
| 1491 | Dependency graph for the SDK that is |
| 1492 | compatible with <filename>graphviz</filename>. |
| 1493 | </para></listitem> |
| 1494 | <listitem><para><filename>installed-package-names.txt:</filename> |
| 1495 | A list of installed packages by name only. |
| 1496 | </para></listitem> |
| 1497 | <listitem><para><filename>installed-package-sizes.txt:</filename> |
| 1498 | A list of installed packages ordered by size. |
| 1499 | </para></listitem> |
| 1500 | <listitem><para><filename>installed-packages.txt:</filename> |
| 1501 | A list of installed packages with full package |
| 1502 | filenames.</para></listitem> |
| 1503 | </itemizedlist> |
| 1504 | </para></listitem> |
| 1505 | </itemizedlist> |
| 1506 | </para> |
| 1507 | |
| 1508 | <para> |
| 1509 | Here is an example of <filename>sdk-info.txt</filename>: |
| 1510 | <literallayout class='monospaced'> |
| 1511 | DISTRO = poky |
| 1512 | DISTRO_VERSION = 1.3+snapshot-20130327 |
| 1513 | SDK_NAME = poky-glibc-i686-arm |
| 1514 | SDK_VERSION = 1.3+snapshot |
| 1515 | SDKMACHINE = |
| 1516 | SDKIMAGE_FEATURES = dev-pkgs dbg-pkgs |
| 1517 | BAD_RECOMMENDATIONS = |
| 1518 | SDKSIZE = 352712 |
| 1519 | </literallayout> |
| 1520 | Other than <filename>SDKSIZE</filename>, which is the |
| 1521 | total size of the files in the SDK in Kbytes, the |
| 1522 | name-value pairs are variables that might have influenced the |
| 1523 | content of the SDK. |
| 1524 | This information is often useful when you are trying to |
| 1525 | determine why a change in the package or file listings |
| 1526 | has occurred. |
| 1527 | </para> |
| 1528 | </section> |
| 1529 | |
| 1530 | <section id='examining-build-history-information'> |
| 1531 | <title>Examining Build History Information</title> |
| 1532 | |
| 1533 | <para> |
| 1534 | You can examine build history output from the command line or |
| 1535 | from a web interface. |
| 1536 | </para> |
| 1537 | |
| 1538 | <para> |
| 1539 | To see any changes that have occurred (assuming you have |
| 1540 | <link linkend='var-BUILDHISTORY_COMMIT'><filename>BUILDHISTORY_COMMIT = "1"</filename></link>), |
| 1541 | you can simply |
| 1542 | use any Git command that allows you to view the history of |
| 1543 | a repository. |
| 1544 | Here is one method: |
| 1545 | <literallayout class='monospaced'> |
| 1546 | $ git log -p |
| 1547 | </literallayout> |
| 1548 | You need to realize, however, that this method does show |
| 1549 | changes that are not significant (e.g. a package's size |
| 1550 | changing by a few bytes). |
| 1551 | </para> |
| 1552 | |
| 1553 | <para> |
| 1554 | A command-line tool called <filename>buildhistory-diff</filename> |
| 1555 | does exist, though, that queries the Git repository and prints just |
| 1556 | the differences that might be significant in human-readable form. |
| 1557 | Here is an example: |
| 1558 | <literallayout class='monospaced'> |
| 1559 | $ ~/poky/poky/scripts/buildhistory-diff . HEAD^ |
| 1560 | Changes to images/qemux86_64/glibc/core-image-minimal (files-in-image.txt): |
| 1561 | /etc/anotherpkg.conf was added |
| 1562 | /sbin/anotherpkg was added |
| 1563 | * (installed-package-names.txt): |
| 1564 | * anotherpkg was added |
| 1565 | Changes to images/qemux86_64/glibc/core-image-minimal (installed-package-names.txt): |
| 1566 | anotherpkg was added |
| 1567 | packages/qemux86_64-poky-linux/v86d: PACKAGES: added "v86d-extras" |
| 1568 | * PR changed from "r0" to "r1" |
| 1569 | * PV changed from "0.1.10" to "0.1.12" |
| 1570 | packages/qemux86_64-poky-linux/v86d/v86d: PKGSIZE changed from 110579 to 144381 (+30%) |
| 1571 | * PR changed from "r0" to "r1" |
| 1572 | * PV changed from "0.1.10" to "0.1.12" |
| 1573 | </literallayout> |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 1574 | <note> |
| 1575 | The <filename>buildhistory-diff</filename> tool requires |
| 1576 | the <filename>GitPython</filename> package. |
| 1577 | Be sure to install it using Pip3 as follows: |
| 1578 | <literallayout class='monospaced'> |
| 1579 | $ pip3 install GitPython --user |
| 1580 | </literallayout> |
| 1581 | Alternatively, you can install |
| 1582 | <filename>python3-git</filename> using the appropriate |
| 1583 | distribution package manager (e.g. |
| 1584 | <filename>apt-get</filename>, <filename>dnf</filename>, or |
| 1585 | <filename>zipper</filename>). |
| 1586 | </note> |
Patrick Williams | c124f4f | 2015-09-15 14:41:29 -0500 | [diff] [blame] | 1587 | </para> |
| 1588 | |
| 1589 | <para> |
| 1590 | To see changes to the build history using a web interface, follow |
| 1591 | the instruction in the <filename>README</filename> file here. |
| 1592 | <ulink url='http://git.yoctoproject.org/cgit/cgit.cgi/buildhistory-web/'></ulink>. |
| 1593 | </para> |
| 1594 | |
| 1595 | <para> |
| 1596 | Here is a sample screenshot of the interface: |
| 1597 | <imagedata fileref="figures/buildhistory-web.png" align="center" scalefit="1" width="130%" contentdepth="130%" /> |
| 1598 | </para> |
| 1599 | </section> |
| 1600 | </section> |
| 1601 | </section> |
| 1602 | |
| 1603 | <section id='speeding-up-the-build'> |
| 1604 | <title>Speeding Up the Build</title> |
| 1605 | |
| 1606 | <para> |
| 1607 | Build time can be an issue. |
| 1608 | By default, the build system uses simple controls to try and maximize |
| 1609 | build efficiency. |
| 1610 | In general, the default settings for all the following variables |
| 1611 | result in the most efficient build times when dealing with single |
| 1612 | socket systems (i.e. a single CPU). |
| 1613 | If you have multiple CPUs, you might try increasing the default |
| 1614 | values to gain more speed. |
| 1615 | See the descriptions in the glossary for each variable for more |
| 1616 | information: |
| 1617 | <itemizedlist> |
| 1618 | <listitem><para> |
| 1619 | <link linkend='var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename>:</link> |
| 1620 | The maximum number of threads BitBake simultaneously executes. |
| 1621 | </para></listitem> |
| 1622 | <listitem><para> |
| 1623 | <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_NUMBER_PARSE_THREADS'><filename>BB_NUMBER_PARSE_THREADS</filename>:</ulink> |
| 1624 | The number of threads BitBake uses during parsing. |
| 1625 | </para></listitem> |
| 1626 | <listitem><para> |
| 1627 | <link linkend='var-PARALLEL_MAKE'><filename>PARALLEL_MAKE</filename>:</link> |
| 1628 | Extra options passed to the <filename>make</filename> command |
| 1629 | during the |
| 1630 | <link linkend='ref-tasks-compile'><filename>do_compile</filename></link> |
| 1631 | task in order to specify parallel compilation on the |
| 1632 | local build host. |
| 1633 | </para></listitem> |
| 1634 | <listitem><para> |
| 1635 | <link linkend='var-PARALLEL_MAKEINST'><filename>PARALLEL_MAKEINST</filename>:</link> |
| 1636 | Extra options passed to the <filename>make</filename> command |
| 1637 | during the |
| 1638 | <link linkend='ref-tasks-install'><filename>do_install</filename></link> |
| 1639 | task in order to specify parallel installation on the |
| 1640 | local build host. |
| 1641 | </para></listitem> |
| 1642 | </itemizedlist> |
| 1643 | As mentioned, these variables all scale to the number of processor |
| 1644 | cores available on the build system. |
| 1645 | For single socket systems, this auto-scaling ensures that the build |
| 1646 | system fundamentally takes advantage of potential parallel operations |
| 1647 | during the build based on the build machine's capabilities. |
| 1648 | </para> |
| 1649 | |
| 1650 | <para> |
| 1651 | Following are additional factors that can affect build speed: |
| 1652 | <itemizedlist> |
| 1653 | <listitem><para> |
| 1654 | File system type: |
| 1655 | The file system type that the build is being performed on can |
| 1656 | also influence performance. |
| 1657 | Using <filename>ext4</filename> is recommended as compared |
| 1658 | to <filename>ext2</filename> and <filename>ext3</filename> |
| 1659 | due to <filename>ext4</filename> improved features |
| 1660 | such as extents. |
| 1661 | </para></listitem> |
| 1662 | <listitem><para> |
| 1663 | Disabling the updating of access time using |
| 1664 | <filename>noatime</filename>: |
| 1665 | The <filename>noatime</filename> mount option prevents the |
| 1666 | build system from updating file and directory access times. |
| 1667 | </para></listitem> |
| 1668 | <listitem><para> |
| 1669 | Setting a longer commit: |
| 1670 | Using the "commit=" mount option increases the interval |
| 1671 | in seconds between disk cache writes. |
| 1672 | Changing this interval from the five second default to |
| 1673 | something longer increases the risk of data loss but decreases |
| 1674 | the need to write to the disk, thus increasing the build |
| 1675 | performance. |
| 1676 | </para></listitem> |
| 1677 | <listitem><para> |
| 1678 | Choosing the packaging backend: |
| 1679 | Of the available packaging backends, IPK is the fastest. |
| 1680 | Additionally, selecting a singular packaging backend also |
| 1681 | helps. |
| 1682 | </para></listitem> |
| 1683 | <listitem><para> |
| 1684 | Using <filename>tmpfs</filename> for |
| 1685 | <link linkend='var-TMPDIR'><filename>TMPDIR</filename></link> |
| 1686 | as a temporary file system: |
| 1687 | While this can help speed up the build, the benefits are |
| 1688 | limited due to the compiler using |
| 1689 | <filename>-pipe</filename>. |
| 1690 | The build system goes to some lengths to avoid |
| 1691 | <filename>sync()</filename> calls into the |
| 1692 | file system on the principle that if there was a significant |
| 1693 | failure, the |
| 1694 | <ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink> |
| 1695 | contents could easily be rebuilt. |
| 1696 | </para></listitem> |
| 1697 | <listitem><para> |
| 1698 | Inheriting the |
| 1699 | <link linkend='ref-classes-rm-work'><filename>rm_work</filename></link> |
| 1700 | class: |
| 1701 | Inheriting this class has shown to speed up builds due to |
| 1702 | significantly lower amounts of data stored in the data |
| 1703 | cache as well as on disk. |
| 1704 | Inheriting this class also makes cleanup of |
| 1705 | <link linkend='var-TMPDIR'><filename>TMPDIR</filename></link> |
| 1706 | faster, at the expense of being easily able to dive into the |
| 1707 | source code. |
| 1708 | File system maintainers have recommended that the fastest way |
| 1709 | to clean up large numbers of files is to reformat partitions |
| 1710 | rather than delete files due to the linear nature of partitions. |
| 1711 | This, of course, assumes you structure the disk partitions and |
| 1712 | file systems in a way that this is practical. |
| 1713 | </para></listitem> |
| 1714 | </itemizedlist> |
| 1715 | Aside from the previous list, you should keep some trade offs in |
| 1716 | mind that can help you speed up the build: |
| 1717 | <itemizedlist> |
| 1718 | <listitem><para> |
| 1719 | Remove items from |
| 1720 | <link linkend='var-DISTRO_FEATURES'><filename>DISTRO_FEATURES</filename></link> |
| 1721 | that you might not need. |
| 1722 | </para></listitem> |
| 1723 | <listitem><para> |
| 1724 | Exclude debug symbols and other debug information: |
| 1725 | If you do not need these symbols and other debug information, |
| 1726 | disabling the <filename>*-dbg</filename> package generation |
| 1727 | can speed up the build. |
| 1728 | You can disable this generation by setting the |
| 1729 | <link linkend='var-INHIBIT_PACKAGE_DEBUG_SPLIT'><filename>INHIBIT_PACKAGE_DEBUG_SPLIT</filename></link> |
| 1730 | variable to "1". |
| 1731 | </para></listitem> |
| 1732 | <listitem><para> |
| 1733 | Disable static library generation for recipes derived from |
| 1734 | <filename>autoconf</filename> or <filename>libtool</filename>: |
| 1735 | Following is an example showing how to disable static |
| 1736 | libraries and still provide an override to handle exceptions: |
| 1737 | <literallayout class='monospaced'> |
| 1738 | STATICLIBCONF = "--disable-static" |
| 1739 | STATICLIBCONF_sqlite3-native = "" |
| 1740 | EXTRA_OECONF += "${STATICLIBCONF}" |
| 1741 | </literallayout> |
| 1742 | <note><title>Notes</title> |
| 1743 | <itemizedlist> |
| 1744 | <listitem><para> |
| 1745 | Some recipes need static libraries in order to work |
| 1746 | correctly (e.g. <filename>pseudo-native</filename> |
| 1747 | needs <filename>sqlite3-native</filename>). |
| 1748 | Overrides, as in the previous example, account for |
| 1749 | these kinds of exceptions. |
| 1750 | </para></listitem> |
| 1751 | <listitem><para> |
| 1752 | Some packages have packaging code that assumes the |
| 1753 | presence of the static libraries. |
| 1754 | If so, you might need to exclude them as well. |
| 1755 | </para></listitem> |
| 1756 | </itemizedlist> |
| 1757 | </note> |
| 1758 | </para></listitem> |
| 1759 | </itemizedlist> |
| 1760 | </para> |
| 1761 | </section> |
| 1762 | </chapter> |
| 1763 | <!-- |
| 1764 | vim: expandtab tw=80 ts=4 |
| 1765 | --> |