blob: 179979c76377aafec5d693dbec55b034a79bf074 [file] [log] [blame]
Andrew Geissler4873add2020-11-02 18:44:49 -06001.. SPDX-License-Identifier: CC-BY-2.0-UK
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002
3************
4Common Tasks
5************
6
7This chapter describes fundamental procedures such as creating layers,
8adding new software packages, extending or customizing images, porting
9work to new hardware (adding a new machine), and so forth. You will find
10that the procedures documented here occur often in the development cycle
11using the Yocto Project.
12
13Understanding and Creating Layers
14=================================
15
16The OpenEmbedded build system supports organizing
17:term:`Metadata` into multiple layers.
18Layers allow you to isolate different types of customizations from each
19other. For introductory information on the Yocto Project Layer Model,
20see the
21":ref:`overview-manual/overview-manual-yp-intro:the yocto project layer model`"
22section in the Yocto Project Overview and Concepts Manual.
23
24Creating Your Own Layer
25-----------------------
26
27It is very easy to create your own layers to use with the OpenEmbedded
28build system. The Yocto Project ships with tools that speed up creating
29layers. This section describes the steps you perform by hand to create
30layers so that you can better understand them. For information about the
31layer-creation tools, see the
32":ref:`bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script`"
33section in the Yocto Project Board Support Package (BSP) Developer's
34Guide and the ":ref:`dev-manual/dev-manual-common-tasks:creating a general layer using the \`\`bitbake-layers\`\` script`"
35section further down in this manual.
36
37Follow these general steps to create your layer without using tools:
38
391. *Check Existing Layers:* Before creating a new layer, you should be
40 sure someone has not already created a layer containing the Metadata
41 you need. You can see the `OpenEmbedded Metadata
42 Index <http://layers.openembedded.org/layerindex/layers/>`__ for a
43 list of layers from the OpenEmbedded community that can be used in
44 the Yocto Project. You could find a layer that is identical or close
45 to what you need.
46
472. *Create a Directory:* Create the directory for your layer. When you
48 create the layer, be sure to create the directory in an area not
49 associated with the Yocto Project :term:`Source Directory`
50 (e.g. the cloned ``poky`` repository).
51
52 While not strictly required, prepend the name of the directory with
53 the string "meta-". For example:
54 ::
55
56 meta-mylayer
57 meta-GUI_xyz
58 meta-mymachine
59
60 With rare exceptions, a layer's name follows this form:
61 ::
62
63 meta-root_name
64
65 Following this layer naming convention can save
66 you trouble later when tools, components, or variables "assume" your
67 layer name begins with "meta-". A notable example is in configuration
68 files as shown in the following step where layer names without the
69 "meta-" string are appended to several variables used in the
70 configuration.
71
723. *Create a Layer Configuration File:* Inside your new layer folder,
73 you need to create a ``conf/layer.conf`` file. It is easiest to take
74 an existing layer configuration file and copy that to your layer's
75 ``conf`` directory and then modify the file as needed.
76
77 The ``meta-yocto-bsp/conf/layer.conf`` file in the Yocto Project
78 :yocto_git:`Source Repositories </cgit/cgit.cgi/poky/tree/meta-yocto-bsp/conf>`
79 demonstrates the required syntax. For your layer, you need to replace
80 "yoctobsp" with a unique identifier for your layer (e.g. "machinexyz"
81 for a layer named "meta-machinexyz"):
82 ::
83
84 # We have a conf and classes directory, add to BBPATH
85 BBPATH .= ":${LAYERDIR}"
86
87 # We have recipes-\* directories, add to BBFILES
88 BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
89 ${LAYERDIR}/recipes-*/*/*.bbappend"
90
91 BBFILE_COLLECTIONS += "yoctobsp"
92 BBFILE_PATTERN_yoctobsp = "^${LAYERDIR}/"
93 BBFILE_PRIORITY_yoctobsp = "5"
94 LAYERVERSION_yoctobsp = "4"
95 LAYERSERIES_COMPAT_yoctobsp = "dunfell"
96
97 Following is an explanation of the layer configuration file:
98
99 - :term:`BBPATH`: Adds the layer's
100 root directory to BitBake's search path. Through the use of the
101 ``BBPATH`` variable, BitBake locates class files (``.bbclass``),
102 configuration files, and files that are included with ``include``
103 and ``require`` statements. For these cases, BitBake uses the
104 first file that matches the name found in ``BBPATH``. This is
105 similar to the way the ``PATH`` variable is used for binaries. It
106 is recommended, therefore, that you use unique class and
107 configuration filenames in your custom layer.
108
109 - :term:`BBFILES`: Defines the
110 location for all recipes in the layer.
111
112 - :term:`BBFILE_COLLECTIONS`:
113 Establishes the current layer through a unique identifier that is
114 used throughout the OpenEmbedded build system to refer to the
115 layer. In this example, the identifier "yoctobsp" is the
116 representation for the container layer named "meta-yocto-bsp".
117
118 - :term:`BBFILE_PATTERN`:
119 Expands immediately during parsing to provide the directory of the
120 layer.
121
122 - :term:`BBFILE_PRIORITY`:
123 Establishes a priority to use for recipes in the layer when the
124 OpenEmbedded build finds recipes of the same name in different
125 layers.
126
127 - :term:`LAYERVERSION`:
128 Establishes a version number for the layer. You can use this
129 version number to specify this exact version of the layer as a
130 dependency when using the
131 :term:`LAYERDEPENDS`
132 variable.
133
134 - :term:`LAYERDEPENDS`:
135 Lists all layers on which this layer depends (if any).
136
137 - :term:`LAYERSERIES_COMPAT`:
138 Lists the :yocto_wiki:`Yocto Project </wiki/Releases>`
139 releases for which the current version is compatible. This
140 variable is a good way to indicate if your particular layer is
141 current.
142
1434. *Add Content:* Depending on the type of layer, add the content. If
144 the layer adds support for a machine, add the machine configuration
145 in a ``conf/machine/`` file within the layer. If the layer adds
146 distro policy, add the distro configuration in a ``conf/distro/``
147 file within the layer. If the layer introduces new recipes, put the
148 recipes you need in ``recipes-*`` subdirectories within the layer.
149
150 .. note::
151
152 For an explanation of layer hierarchy that is compliant with the
153 Yocto Project, see the "
154 Example Filesystem Layout
155 " section in the Yocto Project Board Support Package (BSP)
156 Developer's Guide.
157
1585. *Optionally Test for Compatibility:* If you want permission to use
159 the Yocto Project Compatibility logo with your layer or application
160 that uses your layer, perform the steps to apply for compatibility.
161 See the "`Making Sure Your Layer is Compatible With Yocto
162 Project <#making-sure-your-layer-is-compatible-with-yocto-project>`__"
163 section for more information.
164
165.. _best-practices-to-follow-when-creating-layers:
166
167Following Best Practices When Creating Layers
168---------------------------------------------
169
170To create layers that are easier to maintain and that will not impact
171builds for other machines, you should consider the information in the
172following list:
173
174- *Avoid "Overlaying" Entire Recipes from Other Layers in Your
175 Configuration:* In other words, do not copy an entire recipe into
176 your layer and then modify it. Rather, use an append file
177 (``.bbappend``) to override only those parts of the original recipe
178 you need to modify.
179
180- *Avoid Duplicating Include Files:* Use append files (``.bbappend``)
181 for each recipe that uses an include file. Or, if you are introducing
182 a new recipe that requires the included file, use the path relative
183 to the original layer directory to refer to the file. For example,
184 use ``require recipes-core/``\ package\ ``/``\ file\ ``.inc`` instead
185 of ``require``\ file\ ``.inc``. If you're finding you have to overlay
186 the include file, it could indicate a deficiency in the include file
187 in the layer to which it originally belongs. If this is the case, you
188 should try to address that deficiency instead of overlaying the
189 include file. For example, you could address this by getting the
190 maintainer of the include file to add a variable or variables to make
191 it easy to override the parts needing to be overridden.
192
193- *Structure Your Layers:* Proper use of overrides within append files
194 and placement of machine-specific files within your layer can ensure
195 that a build is not using the wrong Metadata and negatively impacting
196 a build for a different machine. Following are some examples:
197
198 - *Modify Variables to Support a Different Machine:* Suppose you
199 have a layer named ``meta-one`` that adds support for building
200 machine "one". To do so, you use an append file named
201 ``base-files.bbappend`` and create a dependency on "foo" by
202 altering the :term:`DEPENDS`
203 variable:
204 ::
205
206 DEPENDS = "foo"
207
208 The dependency is created during any
209 build that includes the layer ``meta-one``. However, you might not
210 want this dependency for all machines. For example, suppose you
211 are building for machine "two" but your ``bblayers.conf`` file has
212 the ``meta-one`` layer included. During the build, the
213 ``base-files`` for machine "two" will also have the dependency on
214 ``foo``.
215
216 To make sure your changes apply only when building machine "one",
217 use a machine override with the ``DEPENDS`` statement: DEPENDS_one
218 = "foo" You should follow the same strategy when using ``_append``
219 and ``_prepend`` operations:
220 ::
221
222 DEPENDS_append_one = " foo"
223 DEPENDS_prepend_one = "foo "
224
225 As an actual example, here's a
226 snippet from the generic kernel include file ``linux-yocto.inc``,
227 wherein the kernel compile and link options are adjusted in the
228 case of a subset of the supported architectures:
229 ::
230
231 DEPENDS_append_aarch64 = " libgcc"
232 KERNEL_CC_append_aarch64 = " ${TOOLCHAIN_OPTIONS}"
233 KERNEL_LD_append_aarch64 = " ${TOOLCHAIN_OPTIONS}"
234
235 DEPENDS_append_nios2 = " libgcc"
236 KERNEL_CC_append_nios2 = " ${TOOLCHAIN_OPTIONS}"
237 KERNEL_LD_append_nios2 = " ${TOOLCHAIN_OPTIONS}"
238
239 DEPENDS_append_arc = " libgcc"
240 KERNEL_CC_append_arc = " ${TOOLCHAIN_OPTIONS}"
241 KERNEL_LD_append_arc = " ${TOOLCHAIN_OPTIONS}"
242
243 KERNEL_FEATURES_append_qemuall=" features/debug/printk.scc"
244
245 .. note::
246
247 Avoiding "+=" and "=+" and using machine-specific
248 \_append
249 and
250 \_prepend
251 operations is recommended as well.
252
253 - *Place Machine-Specific Files in Machine-Specific Locations:* When
254 you have a base recipe, such as ``base-files.bb``, that contains a
255 :term:`SRC_URI` statement to a
256 file, you can use an append file to cause the build to use your
257 own version of the file. For example, an append file in your layer
258 at ``meta-one/recipes-core/base-files/base-files.bbappend`` could
259 extend :term:`FILESPATH`
260 using
261 :term:`FILESEXTRAPATHS`
262 as follows: FILESEXTRAPATHS_prepend := "${THISDIR}/${BPN}:" The
263 build for machine "one" will pick up your machine-specific file as
264 long as you have the file in
265 ``meta-one/recipes-core/base-files/base-files/``. However, if you
266 are building for a different machine and the ``bblayers.conf``
267 file includes the ``meta-one`` layer and the location of your
268 machine-specific file is the first location where that file is
269 found according to ``FILESPATH``, builds for all machines will
270 also use that machine-specific file.
271
272 You can make sure that a machine-specific file is used for a
273 particular machine by putting the file in a subdirectory specific
274 to the machine. For example, rather than placing the file in
275 ``meta-one/recipes-core/base-files/base-files/`` as shown above,
276 put it in ``meta-one/recipes-core/base-files/base-files/one/``.
277 Not only does this make sure the file is used only when building
278 for machine "one", but the build process locates the file more
279 quickly.
280
281 In summary, you need to place all files referenced from
282 ``SRC_URI`` in a machine-specific subdirectory within the layer in
283 order to restrict those files to machine-specific builds.
284
285- *Perform Steps to Apply for Yocto Project Compatibility:* If you want
286 permission to use the Yocto Project Compatibility logo with your
287 layer or application that uses your layer, perform the steps to apply
288 for compatibility. See the "`Making Sure Your Layer is Compatible
289 With Yocto
290 Project <#making-sure-your-layer-is-compatible-with-yocto-project>`__"
291 section for more information.
292
293- *Follow the Layer Naming Convention:* Store custom layers in a Git
294 repository that use the ``meta-layer_name`` format.
295
296- *Group Your Layers Locally:* Clone your repository alongside other
297 cloned ``meta`` directories from the :term:`Source Directory`.
298
299Making Sure Your Layer is Compatible With Yocto Project
300-------------------------------------------------------
301
302When you create a layer used with the Yocto Project, it is advantageous
303to make sure that the layer interacts well with existing Yocto Project
304layers (i.e. the layer is compatible with the Yocto Project). Ensuring
305compatibility makes the layer easy to be consumed by others in the Yocto
306Project community and could allow you permission to use the Yocto
307Project Compatible Logo.
308
309.. note::
310
311 Only Yocto Project member organizations are permitted to use the
312 Yocto Project Compatible Logo. The logo is not available for general
313 use. For information on how to become a Yocto Project member
314 organization, see the
315 Yocto Project Website
316 .
317
318The Yocto Project Compatibility Program consists of a layer application
319process that requests permission to use the Yocto Project Compatibility
320Logo for your layer and application. The process consists of two parts:
321
3221. Successfully passing a script (``yocto-check-layer``) that when run
323 against your layer, tests it against constraints based on experiences
324 of how layers have worked in the real world and where pitfalls have
325 been found. Getting a "PASS" result from the script is required for
326 successful compatibility registration.
327
3282. Completion of an application acceptance form, which you can find at
329 https://www.yoctoproject.org/webform/yocto-project-compatible-registration.
330
331To be granted permission to use the logo, you need to satisfy the
332following:
333
334- Be able to check the box indicating that you got a "PASS" when
335 running the script against your layer.
336
337- Answer "Yes" to the questions on the form or have an acceptable
338 explanation for any questions answered "No".
339
340- Be a Yocto Project Member Organization.
341
342The remainder of this section presents information on the registration
343form and on the ``yocto-check-layer`` script.
344
345Yocto Project Compatible Program Application
346~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
347
348Use the form to apply for your layer's approval. Upon successful
349application, you can use the Yocto Project Compatibility Logo with your
350layer and the application that uses your layer.
351
352To access the form, use this link:
353https://www.yoctoproject.org/webform/yocto-project-compatible-registration.
354Follow the instructions on the form to complete your application.
355
356The application consists of the following sections:
357
358- *Contact Information:* Provide your contact information as the fields
359 require. Along with your information, provide the released versions
360 of the Yocto Project for which your layer is compatible.
361
362- *Acceptance Criteria:* Provide "Yes" or "No" answers for each of the
363 items in the checklist. Space exists at the bottom of the form for
364 any explanations for items for which you answered "No".
365
366- *Recommendations:* Provide answers for the questions regarding Linux
367 kernel use and build success.
368
369``yocto-check-layer`` Script
370~~~~~~~~~~~~~~~~~~~~~~~~~~~~
371
372The ``yocto-check-layer`` script provides you a way to assess how
373compatible your layer is with the Yocto Project. You should run this
374script prior to using the form to apply for compatibility as described
375in the previous section. You need to achieve a "PASS" result in order to
376have your application form successfully processed.
377
378The script divides tests into three areas: COMMON, BSP, and DISTRO. For
379example, given a distribution layer (DISTRO), the layer must pass both
380the COMMON and DISTRO related tests. Furthermore, if your layer is a BSP
381layer, the layer must pass the COMMON and BSP set of tests.
382
383To execute the script, enter the following commands from your build
384directory:
385::
386
387 $ source oe-init-build-env
388 $ yocto-check-layer your_layer_directory
389
390Be sure to provide the actual directory for your
391layer as part of the command.
392
393Entering the command causes the script to determine the type of layer
394and then to execute a set of specific tests against the layer. The
395following list overviews the test:
396
397- ``common.test_readme``: Tests if a ``README`` file exists in the
398 layer and the file is not empty.
399
400- ``common.test_parse``: Tests to make sure that BitBake can parse the
401 files without error (i.e. ``bitbake -p``).
402
403- ``common.test_show_environment``: Tests that the global or per-recipe
404 environment is in order without errors (i.e. ``bitbake -e``).
405
406- ``common.test_world``: Verifies that ``bitbake world`` works.
407
408- ``common.test_signatures``: Tests to be sure that BSP and DISTRO
409 layers do not come with recipes that change signatures.
410
411- ``common.test_layerseries_compat``: Verifies layer compatibility is
412 set properly.
413
414- ``bsp.test_bsp_defines_machines``: Tests if a BSP layer has machine
415 configurations.
416
417- ``bsp.test_bsp_no_set_machine``: Tests to ensure a BSP layer does not
418 set the machine when the layer is added.
419
420- ``bsp.test_machine_world``: Verifies that ``bitbake world`` works
421 regardless of which machine is selected.
422
423- ``bsp.test_machine_signatures``: Verifies that building for a
424 particular machine affects only the signature of tasks specific to
425 that machine.
426
427- ``distro.test_distro_defines_distros``: Tests if a DISTRO layer has
428 distro configurations.
429
430- ``distro.test_distro_no_set_distros``: Tests to ensure a DISTRO layer
431 does not set the distribution when the layer is added.
432
433Enabling Your Layer
434-------------------
435
436Before the OpenEmbedded build system can use your new layer, you need to
437enable it. To enable your layer, simply add your layer's path to the
438``BBLAYERS`` variable in your ``conf/bblayers.conf`` file, which is
439found in the :term:`Build Directory`.
440The following example shows how to enable a layer named
441``meta-mylayer``:
442::
443
444 # POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
445 # changes incompatibly
446 POKY_BBLAYERS_CONF_VERSION = "2"
447 BBPATH = "${TOPDIR}"
448 BBFILES ?= ""
449 BBLAYERS ?= " \
450 /home/user/poky/meta \
451 /home/user/poky/meta-poky \
452 /home/user/poky/meta-yocto-bsp \
453 /home/user/poky/meta-mylayer \
454 "
455
456BitBake parses each ``conf/layer.conf`` file from the top down as
457specified in the ``BBLAYERS`` variable within the ``conf/bblayers.conf``
458file. During the processing of each ``conf/layer.conf`` file, BitBake
459adds the recipes, classes and configurations contained within the
460particular layer to the source directory.
461
462.. _using-bbappend-files:
463
464Using .bbappend Files in Your Layer
465-----------------------------------
466
467A recipe that appends Metadata to another recipe is called a BitBake
468append file. A BitBake append file uses the ``.bbappend`` file type
469suffix, while the corresponding recipe to which Metadata is being
470appended uses the ``.bb`` file type suffix.
471
472You can use a ``.bbappend`` file in your layer to make additions or
473changes to the content of another layer's recipe without having to copy
474the other layer's recipe into your layer. Your ``.bbappend`` file
475resides in your layer, while the main ``.bb`` recipe file to which you
476are appending Metadata resides in a different layer.
477
478Being able to append information to an existing recipe not only avoids
479duplication, but also automatically applies recipe changes from a
480different layer into your layer. If you were copying recipes, you would
481have to manually merge changes as they occur.
482
483When you create an append file, you must use the same root name as the
484corresponding recipe file. For example, the append file
485``someapp_DISTRO.bbappend`` must apply to ``someapp_DISTRO.bb``. This
486means the original recipe and append file names are version
487number-specific. If the corresponding recipe is renamed to update to a
488newer version, you must also rename and possibly update the
489corresponding ``.bbappend`` as well. During the build process, BitBake
490displays an error on starting if it detects a ``.bbappend`` file that
491does not have a corresponding recipe with a matching name. See the
492:term:`BB_DANGLINGAPPENDS_WARNONLY`
493variable for information on how to handle this error.
494
495As an example, consider the main formfactor recipe and a corresponding
496formfactor append file both from the :term:`Source Directory`.
497Here is the main
498formfactor recipe, which is named ``formfactor_0.0.bb`` and located in
499the "meta" layer at ``meta/recipes-bsp/formfactor``:
500::
501
502 SUMMARY = "Device formfactor information"
503 SECTION = "base"
504 LICENSE = "MIT"
505 LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
506 PR = "r45"
507
508 SRC_URI = "file://config file://machconfig"
509 S = "${WORKDIR}"
510
511 PACKAGE_ARCH = "${MACHINE_ARCH}"
512 INHIBIT_DEFAULT_DEPS = "1"
513
514 do_install() {
515 # Install file only if it has contents
516 install -d ${D}${sysconfdir}/formfactor/
517 install -m 0644 ${S}/config ${D}${sysconfdir}/formfactor/
518 if [ -s "${S}/machconfig" ]; then
519 install -m 0644 ${S}/machconfig ${D}${sysconfdir}/formfactor/
520 fi
521 }
522
523In the main recipe, note the :term:`SRC_URI`
524variable, which tells the OpenEmbedded build system where to find files
525during the build.
526
527Following is the append file, which is named ``formfactor_0.0.bbappend``
528and is from the Raspberry Pi BSP Layer named ``meta-raspberrypi``. The
529file is in the layer at ``recipes-bsp/formfactor``:
530::
531
532 FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"
533
534By default, the build system uses the
535:term:`FILESPATH` variable to
536locate files. This append file extends the locations by setting the
537:term:`FILESEXTRAPATHS`
538variable. Setting this variable in the ``.bbappend`` file is the most
539reliable and recommended method for adding directories to the search
540path used by the build system to find files.
541
542The statement in this example extends the directories to include
543``${``\ :term:`THISDIR`\ ``}/${``\ :term:`PN`\ ``}``,
544which resolves to a directory named ``formfactor`` in the same directory
545in which the append file resides (i.e.
546``meta-raspberrypi/recipes-bsp/formfactor``. This implies that you must
547have the supporting directory structure set up that will contain any
548files or patches you will be including from the layer.
549
550Using the immediate expansion assignment operator ``:=`` is important
551because of the reference to ``THISDIR``. The trailing colon character is
552important as it ensures that items in the list remain colon-separated.
553
554.. note::
555
556 BitBake automatically defines the ``THISDIR`` variable. You should
557 never set this variable yourself. Using "_prepend" as part of the
558 ``FILESEXTRAPATHS`` ensures your path will be searched prior to other
559 paths in the final list.
560
561 Also, not all append files add extra files. Many append files simply
562 exist to add build options (e.g. ``systemd``). For these cases, your
563 append file would not even use the ``FILESEXTRAPATHS`` statement.
564
565Prioritizing Your Layer
566-----------------------
567
568Each layer is assigned a priority value. Priority values control which
569layer takes precedence if there are recipe files with the same name in
570multiple layers. For these cases, the recipe file from the layer with a
571higher priority number takes precedence. Priority values also affect the
572order in which multiple ``.bbappend`` files for the same recipe are
573applied. You can either specify the priority manually, or allow the
574build system to calculate it based on the layer's dependencies.
575
576To specify the layer's priority manually, use the
577:term:`BBFILE_PRIORITY`
578variable and append the layer's root name:
579::
580
581 BBFILE_PRIORITY_mylayer = "1"
582
583.. note::
584
585 It is possible for a recipe with a lower version number
586 :term:`PV` in a layer that has a higher
587 priority to take precedence.
588
589 Also, the layer priority does not currently affect the precedence
590 order of ``.conf`` or ``.bbclass`` files. Future versions of BitBake
591 might address this.
592
593Managing Layers
594---------------
595
596You can use the BitBake layer management tool ``bitbake-layers`` to
597provide a view into the structure of recipes across a multi-layer
598project. Being able to generate output that reports on configured layers
599with their paths and priorities and on ``.bbappend`` files and their
600applicable recipes can help to reveal potential problems.
601
602For help on the BitBake layer management tool, use the following
603command:
604::
605
606 $ bitbake-layers --help NOTE: Starting bitbake server... usage:
607 NOTE: Starting bitbake server...
608 usage: bitbake-layers [-d] [-q] [-F] [--color COLOR] [-h] <subcommand> ...
609
610 BitBake layers utility
611
612 optional arguments:
613 -d, --debug Enable debug output
614 -q, --quiet Print only errors
615 -F, --force Force add without recipe parse verification
616 --color COLOR Colorize output (where COLOR is auto, always, never)
617 -h, --help show this help message and exit
618
619 subcommands:
620 <subcommand>
621 layerindex-fetch Fetches a layer from a layer index along with its
622 dependent layers, and adds them to conf/bblayers.conf.
623 layerindex-show-depends
624 Find layer dependencies from layer index.
625 add-layer Add one or more layers to bblayers.conf.
626 remove-layer Remove one or more layers from bblayers.conf.
627 flatten flatten layer configuration into a separate output
628 directory.
629 show-layers show current configured layers.
630 show-overlayed list overlayed recipes (where the same recipe exists
631 in another layer)
632 show-recipes list available recipes, showing the layer they are
633 provided by
634 show-appends list bbappend files and recipe files they apply to
635 show-cross-depends Show dependencies between recipes that cross layer
636 boundaries.
637 create-layer Create a basic layer
638
639 Use bitbake-layers <subcommand> --help to get help on a specific command
640
641The following list describes the available commands:
642
643- ``help:`` Displays general help or help on a specified command.
644
645- ``show-layers:`` Shows the current configured layers.
646
647- ``show-overlayed:`` Lists overlayed recipes. A recipe is overlayed
648 when a recipe with the same name exists in another layer that has a
649 higher layer priority.
650
651- ``show-recipes:`` Lists available recipes and the layers that
652 provide them.
653
654- ``show-appends:`` Lists ``.bbappend`` files and the recipe files to
655 which they apply.
656
657- ``show-cross-depends:`` Lists dependency relationships between
658 recipes that cross layer boundaries.
659
660- ``add-layer:`` Adds a layer to ``bblayers.conf``.
661
662- ``remove-layer:`` Removes a layer from ``bblayers.conf``
663
664- ``flatten:`` Flattens the layer configuration into a separate
665 output directory. Flattening your layer configuration builds a
666 "flattened" directory that contains the contents of all layers, with
667 any overlayed recipes removed and any ``.bbappend`` files appended to
668 the corresponding recipes. You might have to perform some manual
669 cleanup of the flattened layer as follows:
670
671 - Non-recipe files (such as patches) are overwritten. The flatten
672 command shows a warning for these files.
673
674 - Anything beyond the normal layer setup has been added to the
675 ``layer.conf`` file. Only the lowest priority layer's
676 ``layer.conf`` is used.
677
678 - Overridden and appended items from ``.bbappend`` files need to be
679 cleaned up. The contents of each ``.bbappend`` end up in the
680 flattened recipe. However, if there are appended or changed
681 variable values, you need to tidy these up yourself. Consider the
682 following example. Here, the ``bitbake-layers`` command adds the
683 line ``#### bbappended ...`` so that you know where the following
684 lines originate:
685 ::
686
687 ...
688 DESCRIPTION = "A useful utility"
689 ...
690 EXTRA_OECONF = "--enable-something"
691 ...
692
693 #### bbappended from meta-anotherlayer ####
694
695 DESCRIPTION = "Customized utility"
696 EXTRA_OECONF += "--enable-somethingelse"
697
698
699 Ideally, you would tidy up these utilities as follows:
700 ::
701
702 ...
703 DESCRIPTION = "Customized utility"
704 ...
705 EXTRA_OECONF = "--enable-something --enable-somethingelse"
706 ...
707
708- ``layerindex-fetch``: Fetches a layer from a layer index, along
709 with its dependent layers, and adds the layers to the
710 ``conf/bblayers.conf`` file.
711
712- ``layerindex-show-depends``: Finds layer dependencies from the
713 layer index.
714
715- ``create-layer``: Creates a basic layer.
716
717Creating a General Layer Using the ``bitbake-layers`` Script
718------------------------------------------------------------
719
720The ``bitbake-layers`` script with the ``create-layer`` subcommand
721simplifies creating a new general layer.
722
723.. note::
724
725 - For information on BSP layers, see the ":ref:`bsp-guide/bsp:bsp layers`"
726 section in the Yocto
727 Project Board Specific (BSP) Developer's Guide.
728
729 - In order to use a layer with the OpenEmbedded build system, you
730 need to add the layer to your ``bblayers.conf`` configuration
731 file. See the ":ref:`dev-manual/dev-manual-common-tasks:adding a layer using the \`\`bitbake-layers\`\` script`"
732 section for more information.
733
734The default mode of the script's operation with this subcommand is to
735create a layer with the following:
736
737- A layer priority of 6.
738
739- A ``conf`` subdirectory that contains a ``layer.conf`` file.
740
741- A ``recipes-example`` subdirectory that contains a further
742 subdirectory named ``example``, which contains an ``example.bb``
743 recipe file.
744
745- A ``COPYING.MIT``, which is the license statement for the layer. The
746 script assumes you want to use the MIT license, which is typical for
747 most layers, for the contents of the layer itself.
748
749- A ``README`` file, which is a file describing the contents of your
750 new layer.
751
752In its simplest form, you can use the following command form to create a
753layer. The command creates a layer whose name corresponds to
754your_layer_name in the current directory: $ bitbake-layers create-layer
755your_layer_name As an example, the following command creates a layer
756named ``meta-scottrif`` in your home directory:
757::
758
759 $ cd /usr/home
760 $ bitbake-layers create-layer meta-scottrif
761 NOTE: Starting bitbake server...
762 Add your new layer with 'bitbake-layers add-layer meta-scottrif'
763
764If you want to set the priority of the layer to other than the default
765value of "6", you can either use the ``DASHDASHpriority`` option or you
766can edit the
767:term:`BBFILE_PRIORITY` value
768in the ``conf/layer.conf`` after the script creates it. Furthermore, if
769you want to give the example recipe file some name other than the
770default, you can use the ``DASHDASHexample-recipe-name`` option.
771
772The easiest way to see how the ``bitbake-layers create-layer`` command
773works is to experiment with the script. You can also read the usage
774information by entering the following:
775::
776
777 $ bitbake-layers create-layer --help
778 NOTE: Starting bitbake server...
779 usage: bitbake-layers create-layer [-h] [--priority PRIORITY]
780 [--example-recipe-name EXAMPLERECIPE]
781 layerdir
782
783 Create a basic layer
784
785 positional arguments:
786 layerdir Layer directory to create
787
788 optional arguments:
789 -h, --help show this help message and exit
790 --priority PRIORITY, -p PRIORITY
791 Layer directory to create
792 --example-recipe-name EXAMPLERECIPE, -e EXAMPLERECIPE
793 Filename of the example recipe
794
795Adding a Layer Using the ``bitbake-layers`` Script
796--------------------------------------------------
797
798Once you create your general layer, you must add it to your
799``bblayers.conf`` file. Adding the layer to this configuration file
800makes the OpenEmbedded build system aware of your layer so that it can
801search it for metadata.
802
803Add your layer by using the ``bitbake-layers add-layer`` command:
804::
805
806 $ bitbake-layers add-layer your_layer_name
807
808Here is an example that adds a
809layer named ``meta-scottrif`` to the configuration file. Following the
810command that adds the layer is another ``bitbake-layers`` command that
811shows the layers that are in your ``bblayers.conf`` file:
812::
813
814 $ bitbake-layers add-layer meta-scottrif
815 NOTE: Starting bitbake server...
816 Parsing recipes: 100% |##########################################################| Time: 0:00:49
817 Parsing of 1441 .bb files complete (0 cached, 1441 parsed). 2055 targets, 56 skipped, 0 masked, 0 errors.
818 $ bitbake-layers show-layers
819 NOTE: Starting bitbake server...
820 layer path priority
821 ==========================================================================
822 meta /home/scottrif/poky/meta 5
823 meta-poky /home/scottrif/poky/meta-poky 5
824 meta-yocto-bsp /home/scottrif/poky/meta-yocto-bsp 5
825 workspace /home/scottrif/poky/build/workspace 99
826 meta-scottrif /home/scottrif/poky/build/meta-scottrif 6
827
828
829Adding the layer to this file
830enables the build system to locate the layer during the build.
831
832.. note::
833
834 During a build, the OpenEmbedded build system looks in the layers
835 from the top of the list down to the bottom in that order.
836
837.. _usingpoky-extend-customimage:
838
839Customizing Images
840==================
841
842You can customize images to satisfy particular requirements. This
843section describes several methods and provides guidelines for each.
844
845.. _usingpoky-extend-customimage-localconf:
846
847Customizing Images Using ``local.conf``
848---------------------------------------
849
850Probably the easiest way to customize an image is to add a package by
851way of the ``local.conf`` configuration file. Because it is limited to
852local use, this method generally only allows you to add packages and is
853not as flexible as creating your own customized image. When you add
854packages using local variables this way, you need to realize that these
855variable changes are in effect for every build and consequently affect
856all images, which might not be what you require.
857
858To add a package to your image using the local configuration file, use
859the ``IMAGE_INSTALL`` variable with the ``_append`` operator:
860::
861
862 IMAGE_INSTALL_append = " strace"
863
864Use of the syntax is important -
865specifically, the space between the quote and the package name, which is
866``strace`` in this example. This space is required since the ``_append``
867operator does not add the space.
868
869Furthermore, you must use ``_append`` instead of the ``+=`` operator if
870you want to avoid ordering issues. The reason for this is because doing
871so unconditionally appends to the variable and avoids ordering problems
872due to the variable being set in image recipes and ``.bbclass`` files
873with operators like ``?=``. Using ``_append`` ensures the operation
874takes affect.
875
876As shown in its simplest use, ``IMAGE_INSTALL_append`` affects all
877images. It is possible to extend the syntax so that the variable applies
878to a specific image only. Here is an example:
879IMAGE_INSTALL_append_pn-core-image-minimal = " strace" This example adds
880``strace`` to the ``core-image-minimal`` image only.
881
882You can add packages using a similar approach through the
883``CORE_IMAGE_EXTRA_INSTALL`` variable. If you use this variable, only
884``core-image-*`` images are affected.
885
886.. _usingpoky-extend-customimage-imagefeatures:
887
888Customizing Images Using Custom ``IMAGE_FEATURES`` and ``EXTRA_IMAGE_FEATURES``
889-------------------------------------------------------------------------------
890
891Another method for customizing your image is to enable or disable
892high-level image features by using the
893:term:`IMAGE_FEATURES` and
894:term:`EXTRA_IMAGE_FEATURES`
895variables. Although the functions for both variables are nearly
896equivalent, best practices dictate using ``IMAGE_FEATURES`` from within
897a recipe and using ``EXTRA_IMAGE_FEATURES`` from within your
898``local.conf`` file, which is found in the
899:term:`Build Directory`.
900
901To understand how these features work, the best reference is
902``meta/classes/core-image.bbclass``. This class lists out the available
903``IMAGE_FEATURES`` of which most map to package groups while some, such
904as ``debug-tweaks`` and ``read-only-rootfs``, resolve as general
905configuration settings.
906
907In summary, the file looks at the contents of the ``IMAGE_FEATURES``
908variable and then maps or configures the feature accordingly. Based on
909this information, the build system automatically adds the appropriate
910packages or configurations to the
911:term:`IMAGE_INSTALL` variable.
912Effectively, you are enabling extra features by extending the class or
913creating a custom class for use with specialized image ``.bb`` files.
914
915Use the ``EXTRA_IMAGE_FEATURES`` variable from within your local
916configuration file. Using a separate area from which to enable features
917with this variable helps you avoid overwriting the features in the image
918recipe that are enabled with ``IMAGE_FEATURES``. The value of
919``EXTRA_IMAGE_FEATURES`` is added to ``IMAGE_FEATURES`` within
920``meta/conf/bitbake.conf``.
921
922To illustrate how you can use these variables to modify your image,
923consider an example that selects the SSH server. The Yocto Project ships
924with two SSH servers you can use with your images: Dropbear and OpenSSH.
925Dropbear is a minimal SSH server appropriate for resource-constrained
926environments, while OpenSSH is a well-known standard SSH server
927implementation. By default, the ``core-image-sato`` image is configured
928to use Dropbear. The ``core-image-full-cmdline`` and ``core-image-lsb``
929images both include OpenSSH. The ``core-image-minimal`` image does not
930contain an SSH server.
931
932You can customize your image and change these defaults. Edit the
933``IMAGE_FEATURES`` variable in your recipe or use the
934``EXTRA_IMAGE_FEATURES`` in your ``local.conf`` file so that it
935configures the image you are working with to include
936``ssh-server-dropbear`` or ``ssh-server-openssh``.
937
938.. note::
939
940 See the "
941 Images
942 " section in the Yocto Project Reference Manual for a complete list
943 of image features that ship with the Yocto Project.
944
945.. _usingpoky-extend-customimage-custombb:
946
947Customizing Images Using Custom .bb Files
948-----------------------------------------
949
950You can also customize an image by creating a custom recipe that defines
951additional software as part of the image. The following example shows
952the form for the two lines you need:
953::
954
955 IMAGE_INSTALL = "packagegroup-core-x11-base package1 package2"
956 inherit core-image
957
958Defining the software using a custom recipe gives you total control over
959the contents of the image. It is important to use the correct names of
960packages in the ``IMAGE_INSTALL`` variable. You must use the
961OpenEmbedded notation and not the Debian notation for the names (e.g.
962``glibc-dev`` instead of ``libc6-dev``).
963
964The other method for creating a custom image is to base it on an
965existing image. For example, if you want to create an image based on
966``core-image-sato`` but add the additional package ``strace`` to the
967image, copy the ``meta/recipes-sato/images/core-image-sato.bb`` to a new
968``.bb`` and add the following line to the end of the copy:
969::
970
971 IMAGE_INSTALL += "strace"
972
973.. _usingpoky-extend-customimage-customtasks:
974
975Customizing Images Using Custom Package Groups
976----------------------------------------------
977
978For complex custom images, the best approach for customizing an image is
979to create a custom package group recipe that is used to build the image
980or images. A good example of a package group recipe is
981``meta/recipes-core/packagegroups/packagegroup-base.bb``.
982
983If you examine that recipe, you see that the ``PACKAGES`` variable lists
984the package group packages to produce. The ``inherit packagegroup``
985statement sets appropriate default values and automatically adds
986``-dev``, ``-dbg``, and ``-ptest`` complementary packages for each
987package specified in the ``PACKAGES`` statement.
988
989.. note::
990
991 The
992 inherit packagegroup
993 line should be located near the top of the recipe, certainly before
994 the
995 PACKAGES
996 statement.
997
998For each package you specify in ``PACKAGES``, you can use ``RDEPENDS``
999and ``RRECOMMENDS`` entries to provide a list of packages the parent
1000task package should contain. You can see examples of these further down
1001in the ``packagegroup-base.bb`` recipe.
1002
1003Here is a short, fabricated example showing the same basic pieces for a
1004hypothetical packagegroup defined in ``packagegroup-custom.bb``, where
1005the variable ``PN`` is the standard way to abbreviate the reference to
1006the full packagegroup name ``packagegroup-custom``:
1007::
1008
1009 DESCRIPTION = "My Custom Package Groups"
1010
1011 inherit packagegroup
1012
1013 PACKAGES = "\
1014 ${PN}-apps \
1015 ${PN}-tools \
1016 "
1017
1018 RDEPENDS_${PN}-apps = "\
1019 dropbear \
1020 portmap \
1021 psplash"
1022
1023 RDEPENDS_${PN}-tools = "\
1024 oprofile \
1025 oprofileui-server \
1026 lttng-tools"
1027
1028 RRECOMMENDS_${PN}-tools = "\
1029 kernel-module-oprofile"
1030
1031In the previous example, two package group packages are created with
1032their dependencies and their recommended package dependencies listed:
1033``packagegroup-custom-apps``, and ``packagegroup-custom-tools``. To
1034build an image using these package group packages, you need to add
1035``packagegroup-custom-apps`` and/or ``packagegroup-custom-tools`` to
1036``IMAGE_INSTALL``. For other forms of image dependencies see the other
1037areas of this section.
1038
1039.. _usingpoky-extend-customimage-image-name:
1040
1041Customizing an Image Hostname
1042-----------------------------
1043
1044By default, the configured hostname (i.e. ``/etc/hostname``) in an image
1045is the same as the machine name. For example, if
1046:term:`MACHINE` equals "qemux86", the
1047configured hostname written to ``/etc/hostname`` is "qemux86".
1048
1049You can customize this name by altering the value of the "hostname"
1050variable in the ``base-files`` recipe using either an append file or a
1051configuration file. Use the following in an append file:
1052::
1053
1054 hostname = "myhostname"
1055
1056Use the following in a configuration file:
1057::
1058
1059 hostname_pn-base-files = "myhostname"
1060
1061Changing the default value of the variable "hostname" can be useful in
1062certain situations. For example, suppose you need to do extensive
1063testing on an image and you would like to easily identify the image
1064under test from existing images with typical default hostnames. In this
1065situation, you could change the default hostname to "testme", which
1066results in all the images using the name "testme". Once testing is
1067complete and you do not need to rebuild the image for test any longer,
1068you can easily reset the default hostname.
1069
1070Another point of interest is that if you unset the variable, the image
1071will have no default hostname in the filesystem. Here is an example that
1072unsets the variable in a configuration file:
1073::
1074
1075 hostname_pn-base-files = ""
1076
1077Having no default hostname in the filesystem is suitable for
1078environments that use dynamic hostnames such as virtual machines.
1079
1080.. _new-recipe-writing-a-new-recipe:
1081
1082Writing a New Recipe
1083====================
1084
1085Recipes (``.bb`` files) are fundamental components in the Yocto Project
1086environment. Each software component built by the OpenEmbedded build
1087system requires a recipe to define the component. This section describes
1088how to create, write, and test a new recipe.
1089
1090.. note::
1091
1092 For information on variables that are useful for recipes and for
1093 information about recipe naming issues, see the "
1094 Required
1095 " section of the Yocto Project Reference Manual.
1096
1097.. _new-recipe-overview:
1098
1099Overview
1100--------
1101
1102The following figure shows the basic process for creating a new recipe.
1103The remainder of the section provides details for the steps.
1104
1105.. image:: figures/recipe-workflow.png
1106 :align: center
1107
1108.. _new-recipe-locate-or-automatically-create-a-base-recipe:
1109
1110Locate or Automatically Create a Base Recipe
1111--------------------------------------------
1112
1113You can always write a recipe from scratch. However, three choices exist
1114that can help you quickly get a start on a new recipe:
1115
1116- ``devtool add``: A command that assists in creating a recipe and an
1117 environment conducive to development.
1118
1119- ``recipetool create``: A command provided by the Yocto Project that
1120 automates creation of a base recipe based on the source files.
1121
1122- *Existing Recipes:* Location and modification of an existing recipe
1123 that is similar in function to the recipe you need.
1124
1125.. note::
1126
1127 For information on recipe syntax, see the "
1128 Recipe Syntax
1129 " section.
1130
1131.. _new-recipe-creating-the-base-recipe-using-devtool:
1132
1133Creating the Base Recipe Using ``devtool add``
1134~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1135
1136The ``devtool add`` command uses the same logic for auto-creating the
1137recipe as ``recipetool create``, which is listed below. Additionally,
1138however, ``devtool add`` sets up an environment that makes it easy for
1139you to patch the source and to make changes to the recipe as is often
1140necessary when adding a recipe to build a new piece of software to be
1141included in a build.
1142
1143You can find a complete description of the ``devtool add`` command in
1144the ":ref:`sdk-a-closer-look-at-devtool-add`" section
1145in the Yocto Project Application Development and the Extensible Software
1146Development Kit (eSDK) manual.
1147
1148.. _new-recipe-creating-the-base-recipe-using-recipetool:
1149
1150Creating the Base Recipe Using ``recipetool create``
1151~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1152
1153``recipetool create`` automates creation of a base recipe given a set of
1154source code files. As long as you can extract or point to the source
1155files, the tool will construct a recipe and automatically configure all
1156pre-build information into the recipe. For example, suppose you have an
1157application that builds using Autotools. Creating the base recipe using
1158``recipetool`` results in a recipe that has the pre-build dependencies,
1159license requirements, and checksums configured.
1160
1161To run the tool, you just need to be in your
1162:term:`Build Directory` and have sourced the
1163build environment setup script (i.e.
1164`:ref:`structure-core-script`).
1165To get help on the tool, use the following command:
1166::
1167
1168 $ recipetool -h
1169 NOTE: Starting bitbake server...
1170 usage: recipetool [-d] [-q] [--color COLOR] [-h] <subcommand> ...
1171
1172 OpenEmbedded recipe tool
1173
1174 options:
1175 -d, --debug Enable debug output
1176 -q, --quiet Print only errors
1177 --color COLOR Colorize output (where COLOR is auto, always, never)
1178 -h, --help show this help message and exit
1179
1180 subcommands:
1181 create Create a new recipe
1182 newappend Create a bbappend for the specified target in the specified
1183 layer
1184 setvar Set a variable within a recipe
1185 appendfile Create/update a bbappend to replace a target file
1186 appendsrcfiles Create/update a bbappend to add or replace source files
1187 appendsrcfile Create/update a bbappend to add or replace a source file
1188 Use recipetool <subcommand> --help to get help on a specific command
1189
1190Running ``recipetool create -o`` OUTFILE creates the base recipe and
1191locates it properly in the layer that contains your source files.
1192Following are some syntax examples:
1193
1194Use this syntax to generate a recipe based on source. Once generated,
1195the recipe resides in the existing source code layer:
1196::
1197
1198 recipetool create -o OUTFILE source
1199
1200Use this syntax to generate a recipe using code that
1201you extract from source. The extracted code is placed in its own layer
1202defined by EXTERNALSRC.
1203::
1204
1205 recipetool create -o OUTFILE -x EXTERNALSRC source
1206
1207Use this syntax to generate a recipe based on source. The options
1208direct ``recipetool`` to generate debugging information. Once generated,
1209the recipe resides in the existing source code layer:
1210::
1211
1212 recipetool create -d -o OUTFILE source
1213
1214.. _new-recipe-locating-and-using-a-similar-recipe:
1215
1216Locating and Using a Similar Recipe
1217~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1218
1219Before writing a recipe from scratch, it is often useful to discover
1220whether someone else has already written one that meets (or comes close
1221to meeting) your needs. The Yocto Project and OpenEmbedded communities
1222maintain many recipes that might be candidates for what you are doing.
1223You can find a good central index of these recipes in the `OpenEmbedded
1224Layer Index <http://layers.openembedded.org>`__.
1225
1226Working from an existing recipe or a skeleton recipe is the best way to
1227get started. Here are some points on both methods:
1228
1229- *Locate and modify a recipe that is close to what you want to do:*
1230 This method works when you are familiar with the current recipe
1231 space. The method does not work so well for those new to the Yocto
1232 Project or writing recipes.
1233
1234 Some risks associated with this method are using a recipe that has
1235 areas totally unrelated to what you are trying to accomplish with
1236 your recipe, not recognizing areas of the recipe that you might have
1237 to add from scratch, and so forth. All these risks stem from
1238 unfamiliarity with the existing recipe space.
1239
1240- *Use and modify the following skeleton recipe:* If for some reason
1241 you do not want to use ``recipetool`` and you cannot find an existing
1242 recipe that is close to meeting your needs, you can use the following
1243 structure to provide the fundamental areas of a new recipe.
1244 ::
1245
1246 DESCRIPTION = ""
1247 HOMEPAGE = ""
1248 LICENSE = ""
1249 SECTION = ""
1250 DEPENDS = ""
1251 LIC_FILES_CHKSUM = ""
1252
1253 SRC_URI = ""
1254
1255.. _new-recipe-storing-and-naming-the-recipe:
1256
1257Storing and Naming the Recipe
1258-----------------------------
1259
1260Once you have your base recipe, you should put it in your own layer and
1261name it appropriately. Locating it correctly ensures that the
1262OpenEmbedded build system can find it when you use BitBake to process
1263the recipe.
1264
1265- *Storing Your Recipe:* The OpenEmbedded build system locates your
1266 recipe through the layer's ``conf/layer.conf`` file and the
1267 :term:`BBFILES` variable. This
1268 variable sets up a path from which the build system can locate
1269 recipes. Here is the typical use:
1270 ::
1271
1272 BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
1273 ${LAYERDIR}/recipes-*/*/*.bbappend"
1274
1275 Consequently, you need to be sure you locate your new recipe inside
1276 your layer such that it can be found.
1277
1278 You can find more information on how layers are structured in the
1279 "`Understanding and Creating
1280 Layers <#understanding-and-creating-layers>`__" section.
1281
1282- *Naming Your Recipe:* When you name your recipe, you need to follow
1283 this naming convention: basename_version.bb Use lower-cased
1284 characters and do not include the reserved suffixes ``-native``,
1285 ``-cross``, ``-initial``, or ``-dev`` casually (i.e. do not use them
1286 as part of your recipe name unless the string applies). Here are some
1287 examples:
1288 ::
1289
1290 cups_1.7.0.bb
1291 gawk_4.0.2.bb
1292 irssi_0.8.16-rc1.bb
1293
1294.. _new-recipe-running-a-build-on-the-recipe:
1295
1296Running a Build on the Recipe
1297-----------------------------
1298
1299Creating a new recipe is usually an iterative process that requires
1300using BitBake to process the recipe multiple times in order to
1301progressively discover and add information to the recipe file.
1302
1303Assuming you have sourced the build environment setup script (i.e.
1304:ref:`structure-core-script`) and you are in
1305the :term:`Build Directory`, use
1306BitBake to process your recipe. All you need to provide is the
1307``basename`` of the recipe as described in the previous section:
1308::
1309
1310 $ bitbake basename
1311
1312During the build, the OpenEmbedded build system creates a temporary work
1313directory for each recipe
1314(``${``\ :term:`WORKDIR`\ ``}``)
1315where it keeps extracted source files, log files, intermediate
1316compilation and packaging files, and so forth.
1317
1318The path to the per-recipe temporary work directory depends on the
1319context in which it is being built. The quickest way to find this path
1320is to have BitBake return it by running the following:
1321::
1322
1323 $ bitbake -e basename \| grep ^WORKDIR=
1324
1325As an example, assume a Source Directory
1326top-level folder named ``poky``, a default Build Directory at
1327``poky/build``, and a ``qemux86-poky-linux`` machine target system.
1328Furthermore, suppose your recipe is named ``foo_1.3.0.bb``. In this
1329case, the work directory the build system uses to build the package
1330would be as follows: poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0
1331Inside this directory you can find sub-directories such as ``image``,
1332``packages-split``, and ``temp``. After the build, you can examine these
1333to determine how well the build went.
1334
1335.. note::
1336
1337 You can find log files for each task in the recipe's
1338 temp
1339 directory (e.g.
1340 poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0/temp
1341 ). Log files are named
1342 log.
1343 taskname
1344 (e.g.
1345 log.do_configure
1346 ,
1347 log.do_fetch
1348 , and
1349 log.do_compile
1350 ).
1351
1352You can find more information about the build process in
1353":doc:`../overview-manual/overview-manual-development-environment`"
1354chapter of the Yocto Project Overview and Concepts Manual.
1355
1356.. _new-recipe-fetching-code:
1357
1358Fetching Code
1359-------------
1360
1361The first thing your recipe must do is specify how to fetch the source
1362files. Fetching is controlled mainly through the
1363:term:`SRC_URI` variable. Your recipe
1364must have a ``SRC_URI`` variable that points to where the source is
1365located. For a graphical representation of source locations, see the
1366":ref:`sources-dev-environment`" section in
1367the Yocto Project Overview and Concepts Manual.
1368
1369The :ref:`ref-tasks-fetch` task uses
1370the prefix of each entry in the ``SRC_URI`` variable value to determine
1371which :ref:`fetcher <bitbake:bb-fetchers>` to use to get your
1372source files. It is the ``SRC_URI`` variable that triggers the fetcher.
1373The :ref:`ref-tasks-patch` task uses
1374the variable after source is fetched to apply patches. The OpenEmbedded
1375build system uses
1376:term:`FILESOVERRIDES` for
1377scanning directory locations for local files in ``SRC_URI``.
1378
1379The ``SRC_URI`` variable in your recipe must define each unique location
1380for your source files. It is good practice to not hard-code version
1381numbers in a URL used in ``SRC_URI``. Rather than hard-code these
1382values, use ``${``\ :term:`PV`\ ``}``,
1383which causes the fetch process to use the version specified in the
1384recipe filename. Specifying the version in this manner means that
1385upgrading the recipe to a future version is as simple as renaming the
1386recipe to match the new version.
1387
1388Here is a simple example from the
1389``meta/recipes-devtools/strace/strace_5.5.bb`` recipe where the source
1390comes from a single tarball. Notice the use of the
1391:term:`PV` variable:
1392::
1393
1394 SRC_URI = "https://strace.io/files/${PV}/strace-${PV}.tar.xz \\
1395
1396Files mentioned in ``SRC_URI`` whose names end in a typical archive
1397extension (e.g. ``.tar``, ``.tar.gz``, ``.tar.bz2``, ``.zip``, and so
1398forth), are automatically extracted during the
1399:ref:`ref-tasks-unpack` task. For
1400another example that specifies these types of files, see the
1401"`Autotooled Package <#new-recipe-autotooled-package>`__" section.
1402
1403Another way of specifying source is from an SCM. For Git repositories,
1404you must specify :term:`SRCREV` and
1405you should specify :term:`PV` to include
1406the revision with :term:`SRCPV`. Here
1407is an example from the recipe
1408``meta/recipes-kernel/blktrace/blktrace_git.bb``:
1409::
1410
1411 SRCREV = "d6918c8832793b4205ed3bfede78c2f915c23385"
1412
1413 PR = "r6"
1414 PV = "1.0.5+git${SRCPV}"
1415
1416 SRC_URI = "git://git.kernel.dk/blktrace.git \
1417 file://ldflags.patch"
1418
1419If your ``SRC_URI`` statement includes URLs pointing to individual files
1420fetched from a remote server other than a version control system,
1421BitBake attempts to verify the files against checksums defined in your
1422recipe to ensure they have not been tampered with or otherwise modified
1423since the recipe was written. Two checksums are used:
1424``SRC_URI[md5sum]`` and ``SRC_URI[sha256sum]``.
1425
1426If your ``SRC_URI`` variable points to more than a single URL (excluding
1427SCM URLs), you need to provide the ``md5`` and ``sha256`` checksums for
1428each URL. For these cases, you provide a name for each URL as part of
1429the ``SRC_URI`` and then reference that name in the subsequent checksum
1430statements. Here is an example combining lines from the files
1431``git.inc`` and ``git_2.24.1.bb``:
1432::
1433
1434 SRC_URI = "${KERNELORG_MIRROR}/software/scm/git/git-${PV}.tar.gz;name=tarball \
1435 ${KERNELORG_MIRROR}/software/scm/git/git-manpages-${PV}.tar.gz;name=manpages"
1436
1437 SRC_URI[tarball.md5sum] = "166bde96adbbc11c8843d4f8f4f9811b"
1438 SRC_URI[tarball.sha256sum] = "ad5334956301c86841eb1e5b1bb20884a6bad89a10a6762c958220c7cf64da02"
1439 SRC_URI[manpages.md5sum] = "31c2272a8979022497ba3d4202df145d"
1440 SRC_URI[manpages.sha256sum] = "9a7ae3a093bea39770eb96ca3e5b40bff7af0b9f6123f089d7821d0e5b8e1230"
1441
1442Proper values for ``md5`` and ``sha256`` checksums might be available
1443with other signatures on the download page for the upstream source (e.g.
1444``md5``, ``sha1``, ``sha256``, ``GPG``, and so forth). Because the
1445OpenEmbedded build system only deals with ``sha256sum`` and ``md5sum``,
1446you should verify all the signatures you find by hand.
1447
1448If no ``SRC_URI`` checksums are specified when you attempt to build the
1449recipe, or you provide an incorrect checksum, the build will produce an
1450error for each missing or incorrect checksum. As part of the error
1451message, the build system provides the checksum string corresponding to
1452the fetched file. Once you have the correct checksums, you can copy and
1453paste them into your recipe and then run the build again to continue.
1454
1455.. note::
1456
1457 As mentioned, if the upstream source provides signatures for
1458 verifying the downloaded source code, you should verify those
1459 manually before setting the checksum values in the recipe and
1460 continuing with the build.
1461
1462This final example is a bit more complicated and is from the
1463``meta/recipes-sato/rxvt-unicode/rxvt-unicode_9.20.bb`` recipe. The
1464example's ``SRC_URI`` statement identifies multiple files as the source
1465files for the recipe: a tarball, a patch file, a desktop file, and an
1466icon.
1467::
1468
1469 SRC_URI = "http://dist.schmorp.de/rxvt-unicode/Attic/rxvt-unicode-${PV}.tar.bz2 \
1470 file://xwc.patch \
1471 file://rxvt.desktop \
1472 file://rxvt.png"
1473
1474When you specify local files using the ``file://`` URI protocol, the
1475build system fetches files from the local machine. The path is relative
1476to the :term:`FILESPATH` variable
1477and searches specific directories in a certain order:
1478``${``\ :term:`BP`\ ``}``,
1479``${``\ :term:`BPN`\ ``}``, and
1480``files``. The directories are assumed to be subdirectories of the
1481directory in which the recipe or append file resides. For another
1482example that specifies these types of files, see the "`Single .c File
1483Package (Hello
1484World!) <#new-recipe-single-c-file-package-hello-world>`__" section.
1485
1486The previous example also specifies a patch file. Patch files are files
1487whose names usually end in ``.patch`` or ``.diff`` but can end with
1488compressed suffixes such as ``diff.gz`` and ``patch.bz2``, for example.
1489The build system automatically applies patches as described in the
1490"`Patching Code <#new-recipe-patching-code>`__" section.
1491
1492.. _new-recipe-unpacking-code:
1493
1494Unpacking Code
1495--------------
1496
1497During the build, the
1498:ref:`ref-tasks-unpack` task unpacks
1499the source with ``${``\ :term:`S`\ ``}``
1500pointing to where it is unpacked.
1501
1502If you are fetching your source files from an upstream source archived
1503tarball and the tarball's internal structure matches the common
1504convention of a top-level subdirectory named
1505``${``\ :term:`BPN`\ ``}-${``\ :term:`PV`\ ``}``,
1506then you do not need to set ``S``. However, if ``SRC_URI`` specifies to
1507fetch source from an archive that does not use this convention, or from
1508an SCM like Git or Subversion, your recipe needs to define ``S``.
1509
1510If processing your recipe using BitBake successfully unpacks the source
1511files, you need to be sure that the directory pointed to by ``${S}``
1512matches the structure of the source.
1513
1514.. _new-recipe-patching-code:
1515
1516Patching Code
1517-------------
1518
1519Sometimes it is necessary to patch code after it has been fetched. Any
1520files mentioned in ``SRC_URI`` whose names end in ``.patch`` or
1521``.diff`` or compressed versions of these suffixes (e.g. ``diff.gz`` are
1522treated as patches. The
1523:ref:`ref-tasks-patch` task
1524automatically applies these patches.
1525
1526The build system should be able to apply patches with the "-p1" option
1527(i.e. one directory level in the path will be stripped off). If your
1528patch needs to have more directory levels stripped off, specify the
1529number of levels using the "striplevel" option in the ``SRC_URI`` entry
1530for the patch. Alternatively, if your patch needs to be applied in a
1531specific subdirectory that is not specified in the patch file, use the
1532"patchdir" option in the entry.
1533
1534As with all local files referenced in
1535:term:`SRC_URI` using ``file://``,
1536you should place patch files in a directory next to the recipe either
1537named the same as the base name of the recipe
1538(:term:`BP` and
1539:term:`BPN`) or "files".
1540
1541.. _new-recipe-licensing:
1542
1543Licensing
1544---------
1545
1546Your recipe needs to have both the
1547:term:`LICENSE` and
1548:term:`LIC_FILES_CHKSUM`
1549variables:
1550
1551- ``LICENSE``: This variable specifies the license for the software.
1552 If you do not know the license under which the software you are
1553 building is distributed, you should go to the source code and look
1554 for that information. Typical files containing this information
1555 include ``COPYING``, ``LICENSE``, and ``README`` files. You could
1556 also find the information near the top of a source file. For example,
1557 given a piece of software licensed under the GNU General Public
1558 License version 2, you would set ``LICENSE`` as follows:
1559 ::
1560
1561 LICENSE = "GPLv2"
1562
1563 The licenses you specify within ``LICENSE`` can have any name as long
1564 as you do not use spaces, since spaces are used as separators between
1565 license names. For standard licenses, use the names of the files in
1566 ``meta/files/common-licenses/`` or the ``SPDXLICENSEMAP`` flag names
1567 defined in ``meta/conf/licenses.conf``.
1568
1569- ``LIC_FILES_CHKSUM``: The OpenEmbedded build system uses this
1570 variable to make sure the license text has not changed. If it has,
1571 the build produces an error and it affords you the chance to figure
1572 it out and correct the problem.
1573
1574 You need to specify all applicable licensing files for the software.
1575 At the end of the configuration step, the build process will compare
1576 the checksums of the files to be sure the text has not changed. Any
1577 differences result in an error with the message containing the
1578 current checksum. For more explanation and examples of how to set the
1579 ``LIC_FILES_CHKSUM`` variable, see the "`Tracking License
1580 Changes <#>`__" section.
1581
1582 To determine the correct checksum string, you can list the
1583 appropriate files in the ``LIC_FILES_CHKSUM`` variable with incorrect
1584 md5 strings, attempt to build the software, and then note the
1585 resulting error messages that will report the correct md5 strings.
1586 See the "`Fetching Code <#new-recipe-fetching-code>`__" section for
1587 additional information.
1588
1589 Here is an example that assumes the software has a ``COPYING`` file:
1590 ::
1591
1592 LIC_FILES_CHKSUM = "file://COPYING;md5=xxx"
1593
1594 When you try to build the
1595 software, the build system will produce an error and give you the
1596 correct string that you can substitute into the recipe file for a
1597 subsequent build.
1598
1599.. _new-dependencies:
1600
1601Dependencies
1602------------
1603
1604Most software packages have a short list of other packages that they
1605require, which are called dependencies. These dependencies fall into two
1606main categories: build-time dependencies, which are required when the
1607software is built; and runtime dependencies, which are required to be
1608installed on the target in order for the software to run.
1609
1610Within a recipe, you specify build-time dependencies using the
1611:term:`DEPENDS` variable. Although
1612nuances exist, items specified in ``DEPENDS`` should be names of other
1613recipes. It is important that you specify all build-time dependencies
1614explicitly. If you do not, due to the parallel nature of BitBake's
1615execution, you can end up with a race condition where the dependency is
1616present for one task of a recipe (e.g.
1617:ref:`ref-tasks-configure`) and
1618then gone when the next task runs (e.g.
1619:ref:`ref-tasks-compile`).
1620
1621Another consideration is that configure scripts might automatically
1622check for optional dependencies and enable corresponding functionality
1623if those dependencies are found. This behavior means that to ensure
1624deterministic results and thus avoid more race conditions, you need to
1625either explicitly specify these dependencies as well, or tell the
1626configure script explicitly to disable the functionality. If you wish to
1627make a recipe that is more generally useful (e.g. publish the recipe in
1628a layer for others to use), instead of hard-disabling the functionality,
1629you can use the
1630:term:`PACKAGECONFIG` variable
1631to allow functionality and the corresponding dependencies to be enabled
1632and disabled easily by other users of the recipe.
1633
1634Similar to build-time dependencies, you specify runtime dependencies
1635through a variable -
1636:term:`RDEPENDS`, which is
1637package-specific. All variables that are package-specific need to have
1638the name of the package added to the end as an override. Since the main
1639package for a recipe has the same name as the recipe, and the recipe's
1640name can be found through the
1641``${``\ :term:`PN`\ ``}`` variable, then
1642you specify the dependencies for the main package by setting
1643``RDEPENDS_${PN}``. If the package were named ``${PN}-tools``, then you
1644would set ``RDEPENDS_${PN}-tools``, and so forth.
1645
1646Some runtime dependencies will be set automatically at packaging time.
1647These dependencies include any shared library dependencies (i.e. if a
1648package "example" contains "libexample" and another package "mypackage"
1649contains a binary that links to "libexample" then the OpenEmbedded build
1650system will automatically add a runtime dependency to "mypackage" on
1651"example"). See the
1652":ref:`overview-manual/overview-manual-concepts:automatically added runtime dependencies`"
1653section in the Yocto Project Overview and Concepts Manual for further
1654details.
1655
1656.. _new-recipe-configuring-the-recipe:
1657
1658Configuring the Recipe
1659----------------------
1660
1661Most software provides some means of setting build-time configuration
1662options before compilation. Typically, setting these options is
1663accomplished by running a configure script with options, or by modifying
1664a build configuration file.
1665
1666.. note::
1667
1668 As of Yocto Project Release 1.7, some of the core recipes that
1669 package binary configuration scripts now disable the scripts due to
1670 the scripts previously requiring error-prone path substitution. The
1671 OpenEmbedded build system uses
1672 pkg-config
1673 now, which is much more robust. You can find a list of the
1674 \*-config
1675 scripts that are disabled list in the "
1676 Binary Configuration Scripts Disabled
1677 " section in the Yocto Project Reference Manual.
1678
1679A major part of build-time configuration is about checking for
1680build-time dependencies and possibly enabling optional functionality as
1681a result. You need to specify any build-time dependencies for the
1682software you are building in your recipe's
1683:term:`DEPENDS` value, in terms of
1684other recipes that satisfy those dependencies. You can often find
1685build-time or runtime dependencies described in the software's
1686documentation.
1687
1688The following list provides configuration items of note based on how
1689your software is built:
1690
1691- *Autotools:* If your source files have a ``configure.ac`` file, then
1692 your software is built using Autotools. If this is the case, you just
1693 need to worry about modifying the configuration.
1694
1695 When using Autotools, your recipe needs to inherit the
1696 :ref:`autotools <ref-classes-autotools>` class
1697 and your recipe does not have to contain a
1698 :ref:`ref-tasks-configure` task.
1699 However, you might still want to make some adjustments. For example,
1700 you can set
1701 :term:`EXTRA_OECONF` or
1702 :term:`PACKAGECONFIG_CONFARGS`
1703 to pass any needed configure options that are specific to the recipe.
1704
1705- *CMake:* If your source files have a ``CMakeLists.txt`` file, then
1706 your software is built using CMake. If this is the case, you just
1707 need to worry about modifying the configuration.
1708
1709 When you use CMake, your recipe needs to inherit the
1710 :ref:`cmake <ref-classes-cmake>` class and your
1711 recipe does not have to contain a
1712 :ref:`ref-tasks-configure` task.
1713 You can make some adjustments by setting
1714 :term:`EXTRA_OECMAKE` to
1715 pass any needed configure options that are specific to the recipe.
1716
1717 .. note::
1718
1719 If you need to install one or more custom CMake toolchain files
1720 that are supplied by the application you are building, install the
1721 files to
1722 ${D}${datadir}/cmake/
1723 Modules during
1724 do_install
1725 .
1726
1727- *Other:* If your source files do not have a ``configure.ac`` or
1728 ``CMakeLists.txt`` file, then your software is built using some
1729 method other than Autotools or CMake. If this is the case, you
1730 normally need to provide a
1731 :ref:`ref-tasks-configure` task
1732 in your recipe unless, of course, there is nothing to configure.
1733
1734 Even if your software is not being built by Autotools or CMake, you
1735 still might not need to deal with any configuration issues. You need
1736 to determine if configuration is even a required step. You might need
1737 to modify a Makefile or some configuration file used for the build to
1738 specify necessary build options. Or, perhaps you might need to run a
1739 provided, custom configure script with the appropriate options.
1740
1741 For the case involving a custom configure script, you would run
1742 ``./configure --help`` and look for the options you need to set.
1743
1744Once configuration succeeds, it is always good practice to look at the
1745``log.do_configure`` file to ensure that the appropriate options have
1746been enabled and no additional build-time dependencies need to be added
1747to ``DEPENDS``. For example, if the configure script reports that it
1748found something not mentioned in ``DEPENDS``, or that it did not find
1749something that it needed for some desired optional functionality, then
1750you would need to add those to ``DEPENDS``. Looking at the log might
1751also reveal items being checked for, enabled, or both that you do not
1752want, or items not being found that are in ``DEPENDS``, in which case
1753you would need to look at passing extra options to the configure script
1754as needed. For reference information on configure options specific to
1755the software you are building, you can consult the output of the
1756``./configure --help`` command within ``${S}`` or consult the software's
1757upstream documentation.
1758
1759.. _new-recipe-using-headers-to-interface-with-devices:
1760
1761Using Headers to Interface with Devices
1762---------------------------------------
1763
1764If your recipe builds an application that needs to communicate with some
1765device or needs an API into a custom kernel, you will need to provide
1766appropriate header files. Under no circumstances should you ever modify
1767the existing
1768``meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc`` file.
1769These headers are used to build ``libc`` and must not be compromised
1770with custom or machine-specific header information. If you customize
1771``libc`` through modified headers all other applications that use
1772``libc`` thus become affected.
1773
1774.. note::
1775
1776 Never copy and customize the
1777 libc
1778 header file (i.e.
1779 meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc
1780 ).
1781
1782The correct way to interface to a device or custom kernel is to use a
1783separate package that provides the additional headers for the driver or
1784other unique interfaces. When doing so, your application also becomes
1785responsible for creating a dependency on that specific provider.
1786
1787Consider the following:
1788
1789- Never modify ``linux-libc-headers.inc``. Consider that file to be
1790 part of the ``libc`` system, and not something you use to access the
1791 kernel directly. You should access ``libc`` through specific ``libc``
1792 calls.
1793
1794- Applications that must talk directly to devices should either provide
1795 necessary headers themselves, or establish a dependency on a special
1796 headers package that is specific to that driver.
1797
1798For example, suppose you want to modify an existing header that adds I/O
1799control or network support. If the modifications are used by a small
1800number programs, providing a unique version of a header is easy and has
1801little impact. When doing so, bear in mind the guidelines in the
1802previous list.
1803
1804.. note::
1805
1806 If for some reason your changes need to modify the behavior of the
1807 libc
1808 , and subsequently all other applications on the system, use a
1809 .bbappend
1810 to modify the
1811 linux-kernel-headers.inc
1812 file. However, take care to not make the changes machine specific.
1813
1814Consider a case where your kernel is older and you need an older
1815``libc`` ABI. The headers installed by your recipe should still be a
1816standard mainline kernel, not your own custom one.
1817
1818When you use custom kernel headers you need to get them from
1819:term:`STAGING_KERNEL_DIR`,
1820which is the directory with kernel headers that are required to build
1821out-of-tree modules. Your recipe will also need the following:
1822::
1823
1824 do_configure[depends] += "virtual/kernel:do_shared_workdir"
1825
1826.. _new-recipe-compilation:
1827
1828Compilation
1829-----------
1830
1831During a build, the ``do_compile`` task happens after source is fetched,
1832unpacked, and configured. If the recipe passes through ``do_compile``
1833successfully, nothing needs to be done.
1834
1835However, if the compile step fails, you need to diagnose the failure.
1836Here are some common issues that cause failures.
1837
1838.. note::
1839
1840 For cases where improper paths are detected for configuration files
1841 or for when libraries/headers cannot be found, be sure you are using
1842 the more robust
1843 pkg-config
1844 . See the note in section "
1845 Configuring the Recipe
1846 " for additional information.
1847
1848- *Parallel build failures:* These failures manifest themselves as
1849 intermittent errors, or errors reporting that a file or directory
1850 that should be created by some other part of the build process could
1851 not be found. This type of failure can occur even if, upon
1852 inspection, the file or directory does exist after the build has
1853 failed, because that part of the build process happened in the wrong
1854 order.
1855
1856 To fix the problem, you need to either satisfy the missing dependency
1857 in the Makefile or whatever script produced the Makefile, or (as a
1858 workaround) set :term:`PARALLEL_MAKE` to an empty string:
1859 ::
1860
1861 PARALLEL_MAKE = ""
1862
1863 For information on parallel Makefile issues, see the "`Debugging
1864 Parallel Make Races <#debugging-parallel-make-races>`__" section.
1865
1866- *Improper host path usage:* This failure applies to recipes building
1867 for the target or ``nativesdk`` only. The failure occurs when the
1868 compilation process uses improper headers, libraries, or other files
1869 from the host system when cross-compiling for the target.
1870
1871 To fix the problem, examine the ``log.do_compile`` file to identify
1872 the host paths being used (e.g. ``/usr/include``, ``/usr/lib``, and
1873 so forth) and then either add configure options, apply a patch, or do
1874 both.
1875
1876- *Failure to find required libraries/headers:* If a build-time
1877 dependency is missing because it has not been declared in
1878 :term:`DEPENDS`, or because the
1879 dependency exists but the path used by the build process to find the
1880 file is incorrect and the configure step did not detect it, the
1881 compilation process could fail. For either of these failures, the
1882 compilation process notes that files could not be found. In these
1883 cases, you need to go back and add additional options to the
1884 configure script as well as possibly add additional build-time
1885 dependencies to ``DEPENDS``.
1886
1887 Occasionally, it is necessary to apply a patch to the source to
1888 ensure the correct paths are used. If you need to specify paths to
1889 find files staged into the sysroot from other recipes, use the
1890 variables that the OpenEmbedded build system provides (e.g.
1891 ``STAGING_BINDIR``, ``STAGING_INCDIR``, ``STAGING_DATADIR``, and so
1892 forth).
1893
1894.. _new-recipe-installing:
1895
1896Installing
1897----------
1898
1899During ``do_install``, the task copies the built files along with their
1900hierarchy to locations that would mirror their locations on the target
1901device. The installation process copies files from the
1902``${``\ :term:`S`\ ``}``,
1903``${``\ :term:`B`\ ``}``, and
1904``${``\ :term:`WORKDIR`\ ``}``
1905directories to the ``${``\ :term:`D`\ ``}``
1906directory to create the structure as it should appear on the target
1907system.
1908
1909How your software is built affects what you must do to be sure your
1910software is installed correctly. The following list describes what you
1911must do for installation depending on the type of build system used by
1912the software being built:
1913
1914- *Autotools and CMake:* If the software your recipe is building uses
1915 Autotools or CMake, the OpenEmbedded build system understands how to
1916 install the software. Consequently, you do not have to have a
1917 ``do_install`` task as part of your recipe. You just need to make
1918 sure the install portion of the build completes with no issues.
1919 However, if you wish to install additional files not already being
1920 installed by ``make install``, you should do this using a
1921 ``do_install_append`` function using the install command as described
1922 in the "Manual" bulleted item later in this list.
1923
1924- Other (using ``make install``): You need to define a ``do_install``
1925 function in your recipe. The function should call
1926 ``oe_runmake install`` and will likely need to pass in the
1927 destination directory as well. How you pass that path is dependent on
1928 how the ``Makefile`` being run is written (e.g. ``DESTDIR=${D}``,
1929 ``PREFIX=${D}``, ``INSTALLROOT=${D}``, and so forth).
1930
1931 For an example recipe using ``make install``, see the
1932 "`Makefile-Based Package <#new-recipe-makefile-based-package>`__"
1933 section.
1934
1935- *Manual:* You need to define a ``do_install`` function in your
1936 recipe. The function must first use ``install -d`` to create the
1937 directories under
1938 ``${``\ :term:`D`\ ``}``. Once the
1939 directories exist, your function can use ``install`` to manually
1940 install the built software into the directories.
1941
1942 You can find more information on ``install`` at
1943 http://www.gnu.org/software/coreutils/manual/html_node/install-invocation.html.
1944
1945For the scenarios that do not use Autotools or CMake, you need to track
1946the installation and diagnose and fix any issues until everything
1947installs correctly. You need to look in the default location of
1948``${D}``, which is ``${WORKDIR}/image``, to be sure your files have been
1949installed correctly.
1950
1951.. note::
1952
1953 - During the installation process, you might need to modify some of
1954 the installed files to suit the target layout. For example, you
1955 might need to replace hard-coded paths in an initscript with
1956 values of variables provided by the build system, such as
1957 replacing ``/usr/bin/`` with ``${bindir}``. If you do perform such
1958 modifications during ``do_install``, be sure to modify the
1959 destination file after copying rather than before copying.
1960 Modifying after copying ensures that the build system can
1961 re-execute ``do_install`` if needed.
1962
1963 - ``oe_runmake install``, which can be run directly or can be run
1964 indirectly by the
1965 :ref:`autotools <ref-classes-autotools>` and
1966 :ref:`cmake <ref-classes-cmake>` classes,
1967 runs ``make install`` in parallel. Sometimes, a Makefile can have
1968 missing dependencies between targets that can result in race
1969 conditions. If you experience intermittent failures during
1970 ``do_install``, you might be able to work around them by disabling
1971 parallel Makefile installs by adding the following to the recipe:
1972 PARALLEL_MAKEINST = "" See
1973 :term:`PARALLEL_MAKEINST`
1974 for additional information.
1975
1976 - If you need to install one or more custom CMake toolchain files
1977 that are supplied by the application you are building, install the
1978 files to ``${D}${datadir}/cmake/`` Modules during
1979 :ref:`ref-tasks-install`.
1980
1981.. _new-recipe-enabling-system-services:
1982
1983Enabling System Services
1984------------------------
1985
1986If you want to install a service, which is a process that usually starts
1987on boot and runs in the background, then you must include some
1988additional definitions in your recipe.
1989
1990If you are adding services and the service initialization script or the
1991service file itself is not installed, you must provide for that
1992installation in your recipe using a ``do_install_append`` function. If
1993your recipe already has a ``do_install`` function, update the function
1994near its end rather than adding an additional ``do_install_append``
1995function.
1996
1997When you create the installation for your services, you need to
1998accomplish what is normally done by ``make install``. In other words,
1999make sure your installation arranges the output similar to how it is
2000arranged on the target system.
2001
2002The OpenEmbedded build system provides support for starting services two
2003different ways:
2004
2005- *SysVinit:* SysVinit is a system and service manager that manages the
2006 init system used to control the very basic functions of your system.
2007 The init program is the first program started by the Linux kernel
2008 when the system boots. Init then controls the startup, running and
2009 shutdown of all other programs.
2010
2011 To enable a service using SysVinit, your recipe needs to inherit the
2012 :ref:`update-rc.d <ref-classes-update-rc.d>`
2013 class. The class helps facilitate safely installing the package on
2014 the target.
2015
2016 You will need to set the
2017 :term:`INITSCRIPT_PACKAGES`,
2018 :term:`INITSCRIPT_NAME`,
2019 and
2020 :term:`INITSCRIPT_PARAMS`
2021 variables within your recipe.
2022
2023- *systemd:* System Management Daemon (systemd) was designed to replace
2024 SysVinit and to provide enhanced management of services. For more
2025 information on systemd, see the systemd homepage at
2026 http://freedesktop.org/wiki/Software/systemd/.
2027
2028 To enable a service using systemd, your recipe needs to inherit the
2029 :ref:`systemd <ref-classes-systemd>` class. See
2030 the ``systemd.bbclass`` file located in your :term:`Source Directory`
2031 section for
2032 more information.
2033
2034.. _new-recipe-packaging:
2035
2036Packaging
2037---------
2038
2039Successful packaging is a combination of automated processes performed
2040by the OpenEmbedded build system and some specific steps you need to
2041take. The following list describes the process:
2042
2043- *Splitting Files*: The ``do_package`` task splits the files produced
2044 by the recipe into logical components. Even software that produces a
2045 single binary might still have debug symbols, documentation, and
2046 other logical components that should be split out. The ``do_package``
2047 task ensures that files are split up and packaged correctly.
2048
2049- *Running QA Checks*: The
2050 :ref:`insane <ref-classes-insane>` class adds a
2051 step to the package generation process so that output quality
2052 assurance checks are generated by the OpenEmbedded build system. This
2053 step performs a range of checks to be sure the build's output is free
2054 of common problems that show up during runtime. For information on
2055 these checks, see the
2056 :ref:`insane <ref-classes-insane>` class and
2057 the ":ref:`ref-manual/ref-qa-checks:qa error and warning messages`"
2058 chapter in the Yocto Project Reference Manual.
2059
2060- *Hand-Checking Your Packages*: After you build your software, you
2061 need to be sure your packages are correct. Examine the
2062 ``${``\ :term:`WORKDIR`\ ``}/packages-split``
2063 directory and make sure files are where you expect them to be. If you
2064 discover problems, you can set
2065 :term:`PACKAGES`,
2066 :term:`FILES`,
2067 ``do_install(_append)``, and so forth as needed.
2068
2069- *Splitting an Application into Multiple Packages*: If you need to
2070 split an application into several packages, see the "`Splitting an
2071 Application into Multiple
2072 Packages <#splitting-an-application-into-multiple-packages>`__"
2073 section for an example.
2074
2075- *Installing a Post-Installation Script*: For an example showing how
2076 to install a post-installation script, see the "`Post-Installation
2077 Scripts <#new-recipe-post-installation-scripts>`__" section.
2078
2079- *Marking Package Architecture*: Depending on what your recipe is
2080 building and how it is configured, it might be important to mark the
2081 packages produced as being specific to a particular machine, or to
2082 mark them as not being specific to a particular machine or
2083 architecture at all.
2084
2085 By default, packages apply to any machine with the same architecture
2086 as the target machine. When a recipe produces packages that are
2087 machine-specific (e.g. the
2088 :term:`MACHINE` value is passed
2089 into the configure script or a patch is applied only for a particular
2090 machine), you should mark them as such by adding the following to the
2091 recipe:
2092 ::
2093
2094 PACKAGE_ARCH = "${MACHINE_ARCH}"
2095
2096 On the other hand, if the recipe produces packages that do not
2097 contain anything specific to the target machine or architecture at
2098 all (e.g. recipes that simply package script files or configuration
2099 files), you should use the
2100 :ref:`allarch <ref-classes-allarch>` class to
2101 do this for you by adding this to your recipe:
2102 ::
2103
2104 inherit allarch
2105
2106 Ensuring that the package architecture is correct is not critical
2107 while you are doing the first few builds of your recipe. However, it
2108 is important in order to ensure that your recipe rebuilds (or does
2109 not rebuild) appropriately in response to changes in configuration,
2110 and to ensure that you get the appropriate packages installed on the
2111 target machine, particularly if you run separate builds for more than
2112 one target machine.
2113
2114.. _new-sharing-files-between-recipes:
2115
2116Sharing Files Between Recipes
2117-----------------------------
2118
2119Recipes often need to use files provided by other recipes on the build
2120host. For example, an application linking to a common library needs
2121access to the library itself and its associated headers. The way this
2122access is accomplished is by populating a sysroot with files. Each
2123recipe has two sysroots in its work directory, one for target files
2124(``recipe-sysroot``) and one for files that are native to the build host
2125(``recipe-sysroot-native``).
2126
2127.. note::
2128
2129 You could find the term "staging" used within the Yocto project
2130 regarding files populating sysroots (e.g. the
2131 STAGING_DIR
2132 variable).
2133
2134Recipes should never populate the sysroot directly (i.e. write files
2135into sysroot). Instead, files should be installed into standard
2136locations during the
2137:ref:`ref-tasks-install` task within
2138the ``${``\ :term:`D`\ ``}`` directory. The
2139reason for this limitation is that almost all files that populate the
2140sysroot are cataloged in manifests in order to ensure the files can be
2141removed later when a recipe is either modified or removed. Thus, the
2142sysroot is able to remain free from stale files.
2143
2144A subset of the files installed by the
2145:ref:`ref-tasks-install` task are
2146used by the
2147:ref:`ref-tasks-populate_sysroot`
2148task as defined by the the
2149:term:`SYSROOT_DIRS` variable to
2150automatically populate the sysroot. It is possible to modify the list of
2151directories that populate the sysroot. The following example shows how
2152you could add the ``/opt`` directory to the list of directories within a
2153recipe:
2154::
2155
2156 SYSROOT_DIRS += "/opt"
2157
2158For a more complete description of the
2159:ref:`ref-tasks-populate_sysroot`
2160task and its associated functions, see the
2161:ref:`staging <ref-classes-staging>` class.
2162
2163.. _metadata-virtual-providers:
2164
2165Using Virtual Providers
2166-----------------------
2167
2168Prior to a build, if you know that several different recipes provide the
2169same functionality, you can use a virtual provider (i.e. ``virtual/*``)
2170as a placeholder for the actual provider. The actual provider is
2171determined at build-time.
2172
2173A common scenario where a virtual provider is used would be for the
2174kernel recipe. Suppose you have three kernel recipes whose
2175:term:`PN` values map to ``kernel-big``,
2176``kernel-mid``, and ``kernel-small``. Furthermore, each of these recipes
2177in some way uses a :term:`PROVIDES`
2178statement that essentially identifies itself as being able to provide
2179``virtual/kernel``. Here is one way through the
2180:ref:`kernel <ref-classes-kernel>` class:
2181::
2182
2183 PROVIDES += "${@ "virtual/kernel" if (d.getVar("KERNEL_PACKAGE_NAME") == "kernel") else "" }"
2184
2185Any recipe that inherits the ``kernel`` class is
2186going to utilize a ``PROVIDES`` statement that identifies that recipe as
2187being able to provide the ``virtual/kernel`` item.
2188
2189Now comes the time to actually build an image and you need a kernel
2190recipe, but which one? You can configure your build to call out the
2191kernel recipe you want by using the
2192:term:`PREFERRED_PROVIDER`
2193variable. As an example, consider the
2194`x86-base.inc <https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/conf/machine/include/x86-base.inc>`_
2195include file, which is a machine (i.e.
2196:term:`MACHINE`) configuration file.
2197This include file is the reason all x86-based machines use the
2198``linux-yocto`` kernel. Here are the relevant lines from the include
2199file:
2200::
2201
2202 PREFERRED_PROVIDER_virtual/kernel ??= "linux-yocto"
2203 PREFERRED_VERSION_linux-yocto ??= "4.15%"
2204
2205When you use a virtual provider, you do not have to "hard code" a recipe
2206name as a build dependency. You can use the
2207:term:`DEPENDS` variable to state the
2208build is dependent on ``virtual/kernel`` for example: DEPENDS =
2209"virtual/kernel" During the build, the OpenEmbedded build system picks
2210the correct recipe needed for the ``virtual/kernel`` dependency based on
2211the ``PREFERRED_PROVIDER`` variable. If you want to use the small kernel
2212mentioned at the beginning of this section, configure your build as
2213follows: PREFERRED_PROVIDER_virtual/kernel ??= "kernel-small"
2214
2215.. note::
2216
2217 Any recipe that
2218 PROVIDES
2219 a
2220 virtual/\*
2221 item that is ultimately not selected through
2222 PREFERRED_PROVIDER
2223 does not get built. Preventing these recipes from building is usually
2224 the desired behavior since this mechanism's purpose is to select
2225 between mutually exclusive alternative providers.
2226
2227The following lists specific examples of virtual providers:
2228
2229- ``virtual/kernel``: Provides the name of the kernel recipe to use
2230 when building a kernel image.
2231
2232- ``virtual/bootloader``: Provides the name of the bootloader to use
2233 when building an image.
2234
2235- ``virtual/libgbm``: Provides ``gbm.pc``.
2236
2237- ``virtual/egl``: Provides ``egl.pc`` and possibly ``wayland-egl.pc``.
2238
2239- ``virtual/libgl``: Provides ``gl.pc`` (i.e. libGL).
2240
2241- ``virtual/libgles1``: Provides ``glesv1_cm.pc`` (i.e. libGLESv1_CM).
2242
2243- ``virtual/libgles2``: Provides ``glesv2.pc`` (i.e. libGLESv2).
2244
2245.. note::
2246
2247 Virtual providers only apply to build time dependencies specified with
2248 :term:`PROVIDES` and :term:`DEPENDS`. They do not apply to runtime
2249 dependencies specified with :term:`RPROVIDES` and :term:`RDEPENDS`.
2250
2251Properly Versioning Pre-Release Recipes
2252---------------------------------------
2253
2254Sometimes the name of a recipe can lead to versioning problems when the
2255recipe is upgraded to a final release. For example, consider the
2256``irssi_0.8.16-rc1.bb`` recipe file in the list of example recipes in
2257the "`Storing and Naming the
2258Recipe <#new-recipe-storing-and-naming-the-recipe>`__" section. This
2259recipe is at a release candidate stage (i.e. "rc1"). When the recipe is
2260released, the recipe filename becomes ``irssi_0.8.16.bb``. The version
2261change from ``0.8.16-rc1`` to ``0.8.16`` is seen as a decrease by the
2262build system and package managers, so the resulting packages will not
2263correctly trigger an upgrade.
2264
2265In order to ensure the versions compare properly, the recommended
2266convention is to set :term:`PV` within the
2267recipe to "previous_version+current_version". You can use an additional
2268variable so that you can use the current version elsewhere. Here is an
2269example:
2270::
2271
2272 REALPV = "0.8.16-rc1"
2273 PV = "0.8.15+${REALPV}"
2274
2275.. _new-recipe-post-installation-scripts:
2276
2277Post-Installation Scripts
2278-------------------------
2279
2280Post-installation scripts run immediately after installing a package on
2281the target or during image creation when a package is included in an
2282image. To add a post-installation script to a package, add a
2283``pkg_postinst_``\ PACKAGENAME\ ``()`` function to the recipe file
2284(``.bb``) and replace PACKAGENAME with the name of the package you want
2285to attach to the ``postinst`` script. To apply the post-installation
2286script to the main package for the recipe, which is usually what is
2287required, specify
2288``${``\ :term:`PN`\ ``}`` in place of
2289PACKAGENAME.
2290
2291A post-installation function has the following structure:
2292pkg_postinst_PACKAGENAME() { # Commands to carry out }
2293
2294The script defined in the post-installation function is called when the
2295root filesystem is created. If the script succeeds, the package is
2296marked as installed.
2297
2298.. note::
2299
2300 Any RPM post-installation script that runs on the target should
2301 return a 0 exit code. RPM does not allow non-zero exit codes for
2302 these scripts, and the RPM package manager will cause the package to
2303 fail installation on the target.
2304
2305Sometimes it is necessary for the execution of a post-installation
2306script to be delayed until the first boot. For example, the script might
2307need to be executed on the device itself. To delay script execution
2308until boot time, you must explicitly mark post installs to defer to the
2309target. You can use ``pkg_postinst_ontarget()`` or call
2310``postinst_intercept delay_to_first_boot`` from ``pkg_postinst()``. Any
2311failure of a ``pkg_postinst()`` script (including exit 1) triggers an
2312error during the
2313:ref:`ref-tasks-rootfs` task.
2314
2315If you have recipes that use ``pkg_postinst`` function and they require
2316the use of non-standard native tools that have dependencies during
2317rootfs construction, you need to use the
2318:term:`PACKAGE_WRITE_DEPS`
2319variable in your recipe to list these tools. If you do not use this
2320variable, the tools might be missing and execution of the
2321post-installation script is deferred until first boot. Deferring the
2322script to first boot is undesirable and for read-only rootfs impossible.
2323
2324.. note::
2325
2326 Equivalent support for pre-install, pre-uninstall, and post-uninstall
2327 scripts exist by way of
2328 pkg_preinst
2329 ,
2330 pkg_prerm
2331 , and
2332 pkg_postrm
2333 , respectively. These scrips work in exactly the same way as does
2334 pkg_postinst
2335 with the exception that they run at different times. Also, because of
2336 when they run, they are not applicable to being run at image creation
2337 time like
2338 pkg_postinst
2339 .
2340
2341.. _new-recipe-testing:
2342
2343Testing
2344-------
2345
2346The final step for completing your recipe is to be sure that the
2347software you built runs correctly. To accomplish runtime testing, add
2348the build's output packages to your image and test them on the target.
2349
2350For information on how to customize your image by adding specific
2351packages, see the "`Customizing
2352Images <#usingpoky-extend-customimage>`__" section.
2353
2354.. _new-recipe-testing-examples:
2355
2356Examples
2357--------
2358
2359To help summarize how to write a recipe, this section provides some
2360examples given various scenarios:
2361
2362- Recipes that use local files
2363
2364- Using an Autotooled package
2365
2366- Using a Makefile-based package
2367
2368- Splitting an application into multiple packages
2369
2370- Adding binaries to an image
2371
2372.. _new-recipe-single-c-file-package-hello-world:
2373
2374Single .c File Package (Hello World!)
2375~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2376
2377Building an application from a single file that is stored locally (e.g.
2378under ``files``) requires a recipe that has the file listed in the
2379``SRC_URI`` variable. Additionally, you need to manually write the
2380``do_compile`` and ``do_install`` tasks. The ``S`` variable defines the
2381directory containing the source code, which is set to
2382:term:`WORKDIR` in this case - the
2383directory BitBake uses for the build.
2384::
2385
2386 SUMMARY = "Simple helloworld application"
2387 SECTION = "examples"
2388 LICENSE = "MIT"
2389 LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
2390
2391 SRC_URI = "file://helloworld.c"
2392
2393 S = "${WORKDIR}"
2394
2395 do_compile() {
2396 ${CC} helloworld.c -o helloworld
2397 }
2398
2399 do_install() {
2400 install -d ${D}${bindir}
2401 install -m 0755 helloworld ${D}${bindir}
2402 }
2403
2404By default, the ``helloworld``, ``helloworld-dbg``, and
2405``helloworld-dev`` packages are built. For information on how to
2406customize the packaging process, see the "`Splitting an Application into
2407Multiple Packages <#splitting-an-application-into-multiple-packages>`__"
2408section.
2409
2410.. _new-recipe-autotooled-package:
2411
2412Autotooled Package
2413~~~~~~~~~~~~~~~~~~
2414
2415Applications that use Autotools such as ``autoconf`` and ``automake``
2416require a recipe that has a source archive listed in ``SRC_URI`` and
2417also inherit the
2418:ref:`autotools <ref-classes-autotools>` class,
2419which contains the definitions of all the steps needed to build an
2420Autotool-based application. The result of the build is automatically
2421packaged. And, if the application uses NLS for localization, packages
2422with local information are generated (one package per language).
2423Following is one example: (``hello_2.3.bb``)
2424::
2425
2426 SUMMARY = "GNU Helloworld application"
2427 SECTION = "examples"
2428 LICENSE = "GPLv2+"
2429 LIC_FILES_CHKSUM = "file://COPYING;md5=751419260aa954499f7abaabaa882bbe"
2430
2431 SRC_URI = "${GNU_MIRROR}/hello/hello-${PV}.tar.gz"
2432
2433 inherit autotools gettext
2434
2435The variable ``LIC_FILES_CHKSUM`` is used to track source license
2436changes as described in the "`Tracking License
2437Changes <#usingpoky-configuring-LIC_FILES_CHKSUM>`__" section in the
2438Yocto Project Overview and Concepts Manual. You can quickly create
2439Autotool-based recipes in a manner similar to the previous example.
2440
2441.. _new-recipe-makefile-based-package:
2442
2443Makefile-Based Package
2444~~~~~~~~~~~~~~~~~~~~~~
2445
2446Applications that use GNU ``make`` also require a recipe that has the
2447source archive listed in ``SRC_URI``. You do not need to add a
2448``do_compile`` step since by default BitBake starts the ``make`` command
2449to compile the application. If you need additional ``make`` options, you
2450should store them in the
2451:term:`EXTRA_OEMAKE` or
2452:term:`PACKAGECONFIG_CONFARGS`
2453variables. BitBake passes these options into the GNU ``make``
2454invocation. Note that a ``do_install`` task is still required.
2455Otherwise, BitBake runs an empty ``do_install`` task by default.
2456
2457Some applications might require extra parameters to be passed to the
2458compiler. For example, the application might need an additional header
2459path. You can accomplish this by adding to the ``CFLAGS`` variable. The
2460following example shows this:
2461::
2462
2463 CFLAGS_prepend = "-I ${S}/include "
2464
2465In the following example, ``mtd-utils`` is a makefile-based package:
2466::
2467
2468 SUMMARY = "Tools for managing memory technology devices"
2469 SECTION = "base"
2470 DEPENDS = "zlib lzo e2fsprogs util-linux"
2471 HOMEPAGE = "http://www.linux-mtd.infradead.org/"
2472 LICENSE = "GPLv2+"
2473 LIC_FILES_CHKSUM = "file://COPYING;md5=0636e73ff0215e8d672dc4c32c317bb3 \
2474 file://include/common.h;beginline=1;endline=17;md5=ba05b07912a44ea2bf81ce409380049c"
2475 # Use the latest version at 26 Oct, 2013
2476 SRCREV = "9f107132a6a073cce37434ca9cda6917dd8d866b"
2477 SRC_URI = "git://git.infradead.org/mtd-utils.git \
2478 file://add-exclusion-to-mkfs-jffs2-git-2.patch \
2479 "
2480 PV = "1.5.1+git${SRCPV}"
2481 S = "${WORKDIR}/git"
2482 EXTRA_OEMAKE = "'CC=${CC}' 'RANLIB=${RANLIB}' 'AR=${AR}' 'CFLAGS=${CFLAGS} -I${S}/include -DWITHOUT_XATTR' 'BUILDDIR=${S}'"
2483 do_install () {
2484 oe_runmake install DESTDIR=${D} SBINDIR=${sbindir} MANDIR=${mandir} INCLUDEDIR=${includedir}
2485 }
2486 PACKAGES =+ "mtd-utils-jffs2 mtd-utils-ubifs mtd-utils-misc"
2487 FILES_mtd-utils-jffs2 = "${sbindir}/mkfs.jffs2 ${sbindir}/jffs2dump ${sbindir}/jffs2reader ${sbindir}/sumtool"
2488 FILES_mtd-utils-ubifs = "${sbindir}/mkfs.ubifs ${sbindir}/ubi*"
2489 FILES_mtd-utils-misc = "${sbindir}/nftl* ${sbindir}/ftl* ${sbindir}/rfd* ${sbindir}/doc* ${sbindir}/serve_image ${sbindir}/recv_image"
2490 PARALLEL_MAKE = ""
2491 BBCLASSEXTEND = "native"
2492
2493Splitting an Application into Multiple Packages
2494~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2495
2496You can use the variables ``PACKAGES`` and ``FILES`` to split an
2497application into multiple packages.
2498
2499Following is an example that uses the ``libxpm`` recipe. By default,
2500this recipe generates a single package that contains the library along
2501with a few binaries. You can modify the recipe to split the binaries
2502into separate packages:
2503::
2504
2505 require xorg-lib-common.inc
2506 SUMMARY = "Xpm: X Pixmap extension library"
2507 LICENSE = "BSD"
2508 LIC_FILES_CHKSUM = "file://COPYING;md5=51f4270b012ecd4ab1a164f5f4ed6cf7"
2509 DEPENDS += "libxext libsm libxt"
2510 PE = "1"
2511 XORG_PN = "libXpm"
2512 PACKAGES =+ "sxpm cxpm"
2513 FILES_cxpm = "${bindir}/cxpm"
2514 FILES_sxpm = "${bindir}/sxpm"
2515
2516In the previous example, we want to ship the ``sxpm`` and ``cxpm``
2517binaries in separate packages. Since ``bindir`` would be packaged into
2518the main ``PN`` package by default, we prepend the ``PACKAGES`` variable
2519so additional package names are added to the start of list. This results
2520in the extra ``FILES_*`` variables then containing information that
2521define which files and directories go into which packages. Files
2522included by earlier packages are skipped by latter packages. Thus, the
2523main ``PN`` package does not include the above listed files.
2524
2525Packaging Externally Produced Binaries
2526~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2527
2528Sometimes, you need to add pre-compiled binaries to an image. For
2529example, suppose that binaries for proprietary code exist, which are
2530created by a particular division of a company. Your part of the company
2531needs to use those binaries as part of an image that you are building
2532using the OpenEmbedded build system. Since you only have the binaries
2533and not the source code, you cannot use a typical recipe that expects to
2534fetch the source specified in
2535:term:`SRC_URI` and then compile it.
2536
2537One method is to package the binaries and then install them as part of
2538the image. Generally, it is not a good idea to package binaries since,
2539among other things, it can hinder the ability to reproduce builds and
2540could lead to compatibility problems with ABI in the future. However,
2541sometimes you have no choice.
2542
2543The easiest solution is to create a recipe that uses the
2544:ref:`bin_package <ref-classes-bin-package>` class
2545and to be sure that you are using default locations for build artifacts.
2546In most cases, the ``bin_package`` class handles "skipping" the
2547configure and compile steps as well as sets things up to grab packages
2548from the appropriate area. In particular, this class sets ``noexec`` on
2549both the :ref:`ref-tasks-configure`
2550and :ref:`ref-tasks-compile` tasks,
2551sets ``FILES_${PN}`` to "/" so that it picks up all files, and sets up a
2552:ref:`ref-tasks-install` task, which
2553effectively copies all files from ``${S}`` to ``${D}``. The
2554``bin_package`` class works well when the files extracted into ``${S}``
2555are already laid out in the way they should be laid out on the target.
2556For more information on these variables, see the
2557:term:`FILES`,
2558:term:`PN`,
2559:term:`S`, and
2560:term:`D` variables in the Yocto Project
2561Reference Manual's variable glossary.
2562
2563.. note::
2564
2565 - Using :term:`DEPENDS` is a good
2566 idea even for components distributed in binary form, and is often
2567 necessary for shared libraries. For a shared library, listing the
2568 library dependencies in ``DEPENDS`` makes sure that the libraries
2569 are available in the staging sysroot when other recipes link
2570 against the library, which might be necessary for successful
2571 linking.
2572
2573 - Using ``DEPENDS`` also allows runtime dependencies between
2574 packages to be added automatically. See the
2575 ":ref:`overview-manual/overview-manual-concepts:automatically added runtime dependencies`"
2576 section in the Yocto Project Overview and Concepts Manual for more
2577 information.
2578
2579If you cannot use the ``bin_package`` class, you need to be sure you are
2580doing the following:
2581
2582- Create a recipe where the
2583 :ref:`ref-tasks-configure` and
2584 :ref:`ref-tasks-compile` tasks do
2585 nothing: It is usually sufficient to just not define these tasks in
2586 the recipe, because the default implementations do nothing unless a
2587 Makefile is found in
2588 ``${``\ :term:`S`\ ``}``.
2589
2590 If ``${S}`` might contain a Makefile, or if you inherit some class
2591 that replaces ``do_configure`` and ``do_compile`` with custom
2592 versions, then you can use the
2593 ``[``\ :ref:`noexec <bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ``]``
2594 flag to turn the tasks into no-ops, as follows:
2595 ::
2596
2597 do_configure[noexec] = "1"
2598 do_compile[noexec] = "1"
2599
2600 Unlike
2601 :ref:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata:deleting a task`,
2602 using the flag preserves the dependency chain from the
2603 :ref:`ref-tasks-fetch`,
2604 :ref:`ref-tasks-unpack`, and
2605 :ref:`ref-tasks-patch` tasks to the
2606 :ref:`ref-tasks-install` task.
2607
2608- Make sure your ``do_install`` task installs the binaries
2609 appropriately.
2610
2611- Ensure that you set up :term:`FILES`
2612 (usually
2613 ``FILES_${``\ :term:`PN`\ ``}``) to
2614 point to the files you have installed, which of course depends on
2615 where you have installed them and whether those files are in
2616 different locations than the defaults.
2617
2618Following Recipe Style Guidelines
2619---------------------------------
2620
2621When writing recipes, it is good to conform to existing style
2622guidelines. The `OpenEmbedded
2623Styleguide <http://www.openembedded.org/wiki/Styleguide>`__ wiki page
2624provides rough guidelines for preferred recipe style.
2625
2626It is common for existing recipes to deviate a bit from this style.
2627However, aiming for at least a consistent style is a good idea. Some
2628practices, such as omitting spaces around ``=`` operators in assignments
2629or ordering recipe components in an erratic way, are widely seen as poor
2630style.
2631
2632Recipe Syntax
2633-------------
2634
2635Understanding recipe file syntax is important for writing recipes. The
2636following list overviews the basic items that make up a BitBake recipe
2637file. For more complete BitBake syntax descriptions, see the
2638":doc:`bitbake-user-manual/bitbake-user-manual-metadata`"
2639chapter of the BitBake User Manual.
2640
2641- *Variable Assignments and Manipulations:* Variable assignments allow
2642 a value to be assigned to a variable. The assignment can be static
2643 text or might include the contents of other variables. In addition to
2644 the assignment, appending and prepending operations are also
2645 supported.
2646
2647 The following example shows some of the ways you can use variables in
2648 recipes:
2649 ::
2650
2651 S = "${WORKDIR}/postfix-${PV}"
2652 CFLAGS += "-DNO_ASM"
2653 SRC_URI_append = " file://fixup.patch"
2654
2655- *Functions:* Functions provide a series of actions to be performed.
2656 You usually use functions to override the default implementation of a
2657 task function or to complement a default function (i.e. append or
2658 prepend to an existing function). Standard functions use ``sh`` shell
2659 syntax, although access to OpenEmbedded variables and internal
2660 methods are also available.
2661
2662 The following is an example function from the ``sed`` recipe:
2663 ::
2664
2665 do_install () {
2666 autotools_do_install
2667 install -d ${D}${base_bindir}
2668 mv ${D}${bindir}/sed ${D}${base_bindir}/sed
2669 rmdir ${D}${bindir}/
2670 }
2671
2672 It is
2673 also possible to implement new functions that are called between
2674 existing tasks as long as the new functions are not replacing or
2675 complementing the default functions. You can implement functions in
2676 Python instead of shell. Both of these options are not seen in the
2677 majority of recipes.
2678
2679- *Keywords:* BitBake recipes use only a few keywords. You use keywords
2680 to include common functions (``inherit``), load parts of a recipe
2681 from other files (``include`` and ``require``) and export variables
2682 to the environment (``export``).
2683
2684 The following example shows the use of some of these keywords:
2685 ::
2686
2687 export POSTCONF = "${STAGING_BINDIR}/postconf"
2688 inherit autoconf
2689 require otherfile.inc
2690
2691- *Comments (#):* Any lines that begin with the hash character (``#``)
2692 are treated as comment lines and are ignored:
2693 ::
2694
2695 # This is a comment
2696
2697This next list summarizes the most important and most commonly used
2698parts of the recipe syntax. For more information on these parts of the
2699syntax, you can reference the
2700:doc:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata` chapter
2701in the BitBake User Manual.
2702
2703- *Line Continuation (\):* Use the backward slash (``\``) character to
2704 split a statement over multiple lines. Place the slash character at
2705 the end of the line that is to be continued on the next line:
2706 ::
2707
2708 VAR = "A really long \
2709 line"
2710
2711 .. note::
2712
2713 You cannot have any characters including spaces or tabs after the
2714 slash character.
2715
2716- *Using Variables (${VARNAME}):* Use the ``${VARNAME}`` syntax to
2717 access the contents of a variable:
2718 ::
2719
2720 SRC_URI = "${SOURCEFORGE_MIRROR}/libpng/zlib-${PV}.tar.gz"
2721
2722 .. note::
2723
2724 It is important to understand that the value of a variable
2725 expressed in this form does not get substituted automatically. The
2726 expansion of these expressions happens on-demand later (e.g.
2727 usually when a function that makes reference to the variable
2728 executes). This behavior ensures that the values are most
2729 appropriate for the context in which they are finally used. On the
2730 rare occasion that you do need the variable expression to be
2731 expanded immediately, you can use the
2732 :=
2733 operator instead of
2734 =
2735 when you make the assignment, but this is not generally needed.
2736
2737- *Quote All Assignments ("value"):* Use double quotes around values in
2738 all variable assignments (e.g. ``"value"``). Following is an example:
2739 ::
2740
2741 VAR1 = "${OTHERVAR}"
2742 VAR2 = "The version is ${PV}"
2743
2744- *Conditional Assignment (?=):* Conditional assignment is used to
2745 assign a value to a variable, but only when the variable is currently
2746 unset. Use the question mark followed by the equal sign (``?=``) to
2747 make a "soft" assignment used for conditional assignment. Typically,
2748 "soft" assignments are used in the ``local.conf`` file for variables
2749 that are allowed to come through from the external environment.
2750
2751 Here is an example where ``VAR1`` is set to "New value" if it is
2752 currently empty. However, if ``VAR1`` has already been set, it
2753 remains unchanged: VAR1 ?= "New value" In this next example, ``VAR1``
2754 is left with the value "Original value":
2755 ::
2756
2757 VAR1 = "Original value"
2758 VAR1 ?= "New value"
2759
2760- *Appending (+=):* Use the plus character followed by the equals sign
2761 (``+=``) to append values to existing variables.
2762
2763 .. note::
2764
2765 This operator adds a space between the existing content of the
2766 variable and the new content.
2767
2768 Here is an example:
2769 ::
2770
2771 SRC_URI += "file://fix-makefile.patch"
2772
2773- *Prepending (=+):* Use the equals sign followed by the plus character
2774 (``=+``) to prepend values to existing variables.
2775
2776 .. note::
2777
2778 This operator adds a space between the new content and the
2779 existing content of the variable.
2780
2781 Here is an example:
2782 ::
2783
2784 VAR =+ "Starts"
2785
2786- *Appending (_append):* Use the ``_append`` operator to append values
2787 to existing variables. This operator does not add any additional
2788 space. Also, the operator is applied after all the ``+=``, and ``=+``
2789 operators have been applied and after all ``=`` assignments have
2790 occurred.
2791
2792 The following example shows the space being explicitly added to the
2793 start to ensure the appended value is not merged with the existing
2794 value:
2795 ::
2796
2797 SRC_URI_append = " file://fix-makefile.patch"
2798
2799 You can also use
2800 the ``_append`` operator with overrides, which results in the actions
2801 only being performed for the specified target or machine:
2802 ::
2803
2804 SRC_URI_append_sh4 = " file://fix-makefile.patch"
2805
2806- *Prepending (_prepend):* Use the ``_prepend`` operator to prepend
2807 values to existing variables. This operator does not add any
2808 additional space. Also, the operator is applied after all the ``+=``,
2809 and ``=+`` operators have been applied and after all ``=``
2810 assignments have occurred.
2811
2812 The following example shows the space being explicitly added to the
2813 end to ensure the prepended value is not merged with the existing
2814 value:
2815 ::
2816
2817 CFLAGS_prepend = "-I${S}/myincludes "
2818
2819 You can also use the
2820 ``_prepend`` operator with overrides, which results in the actions
2821 only being performed for the specified target or machine:
2822 ::
2823
2824 CFLAGS_prepend_sh4 = "-I${S}/myincludes "
2825
2826- *Overrides:* You can use overrides to set a value conditionally,
2827 typically based on how the recipe is being built. For example, to set
2828 the :term:`KBRANCH` variable's
2829 value to "standard/base" for any target
2830 :term:`MACHINE`, except for
2831 qemuarm where it should be set to "standard/arm-versatile-926ejs",
2832 you would do the following:
2833 ::
2834
2835 KBRANCH = "standard/base"
2836 KBRANCH_qemuarm = "standard/arm-versatile-926ejs"
2837
2838 Overrides are also used to separate
2839 alternate values of a variable in other situations. For example, when
2840 setting variables such as
2841 :term:`FILES` and
2842 :term:`RDEPENDS` that are
2843 specific to individual packages produced by a recipe, you should
2844 always use an override that specifies the name of the package.
2845
2846- *Indentation:* Use spaces for indentation rather than than tabs. For
2847 shell functions, both currently work. However, it is a policy
2848 decision of the Yocto Project to use tabs in shell functions. Realize
2849 that some layers have a policy to use spaces for all indentation.
2850
2851- *Using Python for Complex Operations:* For more advanced processing,
2852 it is possible to use Python code during variable assignments (e.g.
2853 search and replacement on a variable).
2854
2855 You indicate Python code using the ``${@python_code}`` syntax for the
2856 variable assignment:
2857 ::
2858
2859 SRC_URI = "ftp://ftp.info-zip.org/pub/infozip/src/zip${@d.getVar('PV',1).replace('.', '')}.tgz
2860
2861- *Shell Function Syntax:* Write shell functions as if you were writing
2862 a shell script when you describe a list of actions to take. You
2863 should ensure that your script works with a generic ``sh`` and that
2864 it does not require any ``bash`` or other shell-specific
2865 functionality. The same considerations apply to various system
2866 utilities (e.g. ``sed``, ``grep``, ``awk``, and so forth) that you
2867 might wish to use. If in doubt, you should check with multiple
2868 implementations - including those from BusyBox.
2869
2870.. _platdev-newmachine:
2871
2872Adding a New Machine
2873====================
2874
2875Adding a new machine to the Yocto Project is a straightforward process.
2876This section describes how to add machines that are similar to those
2877that the Yocto Project already supports.
2878
2879.. note::
2880
2881 Although well within the capabilities of the Yocto Project, adding a
2882 totally new architecture might require changes to
2883 gcc/glibc
2884 and to the site information, which is beyond the scope of this
2885 manual.
2886
2887For a complete example that shows how to add a new machine, see the
2888":ref:`bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script`"
2889section in the Yocto Project Board Support Package (BSP) Developer's
2890Guide.
2891
2892.. _platdev-newmachine-conffile:
2893
2894Adding the Machine Configuration File
2895-------------------------------------
2896
2897To add a new machine, you need to add a new machine configuration file
2898to the layer's ``conf/machine`` directory. This configuration file
2899provides details about the device you are adding.
2900
2901The OpenEmbedded build system uses the root name of the machine
2902configuration file to reference the new machine. For example, given a
2903machine configuration file named ``crownbay.conf``, the build system
2904recognizes the machine as "crownbay".
2905
2906The most important variables you must set in your machine configuration
2907file or include from a lower-level configuration file are as follows:
2908
2909- ``TARGET_ARCH`` (e.g. "arm")
2910
2911- ``PREFERRED_PROVIDER_virtual/kernel``
2912
2913- ``MACHINE_FEATURES`` (e.g. "apm screen wifi")
2914
2915You might also need these variables:
2916
2917- ``SERIAL_CONSOLES`` (e.g. "115200;ttyS0 115200;ttyS1")
2918
2919- ``KERNEL_IMAGETYPE`` (e.g. "zImage")
2920
2921- ``IMAGE_FSTYPES`` (e.g. "tar.gz jffs2")
2922
2923You can find full details on these variables in the reference section.
2924You can leverage existing machine ``.conf`` files from
2925``meta-yocto-bsp/conf/machine/``.
2926
2927.. _platdev-newmachine-kernel:
2928
2929Adding a Kernel for the Machine
2930-------------------------------
2931
2932The OpenEmbedded build system needs to be able to build a kernel for the
2933machine. You need to either create a new kernel recipe for this machine,
2934or extend an existing kernel recipe. You can find several kernel recipe
2935examples in the Source Directory at ``meta/recipes-kernel/linux`` that
2936you can use as references.
2937
2938If you are creating a new kernel recipe, normal recipe-writing rules
2939apply for setting up a ``SRC_URI``. Thus, you need to specify any
2940necessary patches and set ``S`` to point at the source code. You need to
2941create a ``do_configure`` task that configures the unpacked kernel with
2942a ``defconfig`` file. You can do this by using a ``make defconfig``
2943command or, more commonly, by copying in a suitable ``defconfig`` file
2944and then running ``make oldconfig``. By making use of ``inherit kernel``
2945and potentially some of the ``linux-*.inc`` files, most other
2946functionality is centralized and the defaults of the class normally work
2947well.
2948
2949If you are extending an existing kernel recipe, it is usually a matter
2950of adding a suitable ``defconfig`` file. The file needs to be added into
2951a location similar to ``defconfig`` files used for other machines in a
2952given kernel recipe. A possible way to do this is by listing the file in
2953the ``SRC_URI`` and adding the machine to the expression in
2954``COMPATIBLE_MACHINE``:
2955::
2956
2957 COMPATIBLE_MACHINE = '(qemux86|qemumips)'
2958
2959For more information on ``defconfig`` files, see the
2960":ref:`kernel-dev/kernel-dev-common:changing the configuration`"
2961section in the Yocto Project Linux Kernel Development Manual.
2962
2963.. _platdev-newmachine-formfactor:
2964
2965Adding a Formfactor Configuration File
2966--------------------------------------
2967
2968A formfactor configuration file provides information about the target
2969hardware for which the image is being built and information that the
2970build system cannot obtain from other sources such as the kernel. Some
2971examples of information contained in a formfactor configuration file
2972include framebuffer orientation, whether or not the system has a
2973keyboard, the positioning of the keyboard in relation to the screen, and
2974the screen resolution.
2975
2976The build system uses reasonable defaults in most cases. However, if
2977customization is necessary, you need to create a ``machconfig`` file in
2978the ``meta/recipes-bsp/formfactor/files`` directory. This directory
2979contains directories for specific machines such as ``qemuarm`` and
2980``qemux86``. For information about the settings available and the
2981defaults, see the ``meta/recipes-bsp/formfactor/files/config`` file
2982found in the same area.
2983
2984Following is an example for "qemuarm" machine:
2985::
2986
2987 HAVE_TOUCHSCREEN=1
2988 HAVE_KEYBOARD=1
2989 DISPLAY_CAN_ROTATE=0
2990 DISPLAY_ORIENTATION=0
2991 #DISPLAY_WIDTH_PIXELS=640
2992 #DISPLAY_HEIGHT_PIXELS=480
2993 #DISPLAY_BPP=16
2994 DISPLAY_DPI=150
2995 DISPLAY_SUBPIXEL_ORDER=vrgb
2996
2997.. _gs-upgrading-recipes:
2998
2999Upgrading Recipes
3000=================
3001
3002Over time, upstream developers publish new versions for software built
3003by layer recipes. It is recommended to keep recipes up-to-date with
3004upstream version releases.
3005
3006While several methods exist that allow you upgrade a recipe, you might
3007consider checking on the upgrade status of a recipe first. You can do so
3008using the ``devtool check-upgrade-status`` command. See the
3009":ref:`devtool-checking-on-the-upgrade-status-of-a-recipe`"
3010section in the Yocto Project Reference Manual for more information.
3011
3012The remainder of this section describes three ways you can upgrade a
3013recipe. You can use the Automated Upgrade Helper (AUH) to set up
3014automatic version upgrades. Alternatively, you can use
3015``devtool upgrade`` to set up semi-automatic version upgrades. Finally,
3016you can manually upgrade a recipe by editing the recipe itself.
3017
3018.. _gs-using-the-auto-upgrade-helper:
3019
3020Using the Auto Upgrade Helper (AUH)
3021-----------------------------------
3022
3023The AUH utility works in conjunction with the OpenEmbedded build system
3024in order to automatically generate upgrades for recipes based on new
3025versions being published upstream. Use AUH when you want to create a
3026service that performs the upgrades automatically and optionally sends
3027you an email with the results.
3028
3029AUH allows you to update several recipes with a single use. You can also
3030optionally perform build and integration tests using images with the
3031results saved to your hard drive and emails of results optionally sent
3032to recipe maintainers. Finally, AUH creates Git commits with appropriate
3033commit messages in the layer's tree for the changes made to recipes.
3034
3035.. note::
3036
3037 Conditions do exist when you should not use AUH to upgrade recipes
3038 and you should instead use either
3039 devtool upgrade
3040 or upgrade your recipes manually:
3041
3042 - When AUH cannot complete the upgrade sequence. This situation
3043 usually results because custom patches carried by the recipe
3044 cannot be automatically rebased to the new version. In this case,
3045 ``devtool upgrade`` allows you to manually resolve conflicts.
3046
3047 - When for any reason you want fuller control over the upgrade
3048 process. For example, when you want special arrangements for
3049 testing.
3050
3051The following steps describe how to set up the AUH utility:
3052
30531. *Be Sure the Development Host is Set Up:* You need to be sure that
3054 your development host is set up to use the Yocto Project. For
3055 information on how to set up your host, see the "`Preparing the Build
3056 Host <#dev-preparing-the-build-host>`__" section.
3057
30582. *Make Sure Git is Configured:* The AUH utility requires Git to be
3059 configured because AUH uses Git to save upgrades. Thus, you must have
3060 Git user and email configured. The following command shows your
3061 configurations:
3062
3063 $ git config --list
3064
3065 If you do not have the user and
3066 email configured, you can use the following commands to do so:
3067 ::
3068
3069 $ git config --global user.name some_name
3070 $ git config --global user.email username@domain.com
3071
30723. *Clone the AUH Repository:* To use AUH, you must clone the repository
3073 onto your development host. The following command uses Git to create
3074 a local copy of the repository on your system:
3075 ::
3076
3077 $ git clone git://git.yoctoproject.org/auto-upgrade-helper
3078 Cloning into 'auto-upgrade-helper'... remote: Counting objects: 768, done.
3079 remote: Compressing objects: 100% (300/300), done.
3080 remote: Total 768 (delta 499), reused 703 (delta 434)
3081 Receiving objects: 100% (768/768), 191.47 KiB | 98.00 KiB/s, done.
3082 Resolving deltas: 100% (499/499), done.
3083 Checking connectivity... done.
3084
3085 AUH is not part of the :term:`OpenEmbedded-Core (OE-Core)` or
3086 :term:`Poky` repositories.
3087
30884. *Create a Dedicated Build Directory:* Run the
3089 :ref:`structure-core-script`
3090 script to create a fresh build directory that you use exclusively for
3091 running the AUH utility:
3092 ::
3093
3094 $ cd ~/poky
3095 $ source oe-init-build-env
3096
3097 your_AUH_build_directory Re-using an existing build directory and its
3098 configurations is not recommended as existing settings could cause
3099 AUH to fail or behave undesirably.
3100
31015. *Make Configurations in Your Local Configuration File:* Several
3102 settings need to exist in the ``local.conf`` file in the build
3103 directory you just created for AUH. Make these following
3104 configurations:
3105
3106 - If you want to enable :ref:`Build
3107 History <dev-manual/dev-manual-common-tasks:maintaining build output quality>`,
3108 which is optional, you need the following lines in the
3109 ``conf/local.conf`` file:
3110 ::
3111
3112 INHERIT =+ "buildhistory"
3113 BUILDHISTORY_COMMIT = "1"
3114
3115 With this configuration and a successful
3116 upgrade, a build history "diff" file appears in the
3117 ``upgrade-helper/work/recipe/buildhistory-diff.txt`` file found in
3118 your build directory.
3119
3120 - If you want to enable testing through the
3121 :ref:`testimage <ref-classes-testimage*>`
3122 class, which is optional, you need to have the following set in
3123 your ``conf/local.conf`` file: INHERIT += "testimage"
3124
3125 .. note::
3126
3127 If your distro does not enable by default ptest, which Poky
3128 does, you need the following in your
3129 local.conf
3130 file:
3131 ::
3132
3133 DISTRO_FEATURES_append = " ptest"
3134
3135
31366. *Optionally Start a vncserver:* If you are running in a server
3137 without an X11 session, you need to start a vncserver:
3138 ::
3139
3140 $ vncserver :1
3141 $ export DISPLAY=:1
3142
31437. *Create and Edit an AUH Configuration File:* You need to have the
3144 ``upgrade-helper/upgrade-helper.conf`` configuration file in your
3145 build directory. You can find a sample configuration file in the `AUH
3146 source
3147 repository <http://git.yoctoproject.org/cgit/cgit.cgi/auto-upgrade-helper/tree/>`__.
3148
3149 Read through the sample file and make configurations as needed. For
3150 example, if you enabled build history in your ``local.conf`` as
3151 described earlier, you must enable it in ``upgrade-helper.conf``.
3152
3153 Also, if you are using the default ``maintainers.inc`` file supplied
3154 with Poky and located in ``meta-yocto`` and you do not set a
3155 "maintainers_whitelist" or "global_maintainer_override" in the
3156 ``upgrade-helper.conf`` configuration, and you specify "-e all" on
3157 the AUH command-line, the utility automatically sends out emails to
3158 all the default maintainers. Please avoid this.
3159
3160This next set of examples describes how to use the AUH:
3161
3162- *Upgrading a Specific Recipe:* To upgrade a specific recipe, use the
3163 following form: $ upgrade-helper.py recipe_name For example, this
3164 command upgrades the ``xmodmap`` recipe:
3165 ::
3166
3167 $ upgrade-helper.py xmodmap
3168
3169- *Upgrading a Specific Recipe to a Particular Version:* To upgrade a
3170 specific recipe to a particular version, use the following form: $
3171 upgrade-helper.py recipe_name -t version For example, this command
3172 upgrades the ``xmodmap`` recipe to version 1.2.3:
3173 ::
3174
3175 $ upgrade-helper.py xmodmap -t 1.2.3
3176
3177- *Upgrading all Recipes to the Latest Versions and Suppressing Email
3178 Notifications:* To upgrade all recipes to their most recent versions
3179 and suppress the email notifications, use the following command:
3180 ::
3181
3182 $ upgrade-helper.py all
3183
3184- *Upgrading all Recipes to the Latest Versions and Send Email
3185 Notifications:* To upgrade all recipes to their most recent versions
3186 and send email messages to maintainers for each attempted recipe as
3187 well as a status email, use the following command:
3188 ::
3189
3190 $ upgrade-helper.py -e all
3191
3192Once you have run the AUH utility, you can find the results in the AUH
3193build directory:
3194::
3195
3196 ${BUILDDIR}/upgrade-helper/timestamp
3197
3198The AUH utility
3199also creates recipe update commits from successful upgrade attempts in
3200the layer tree.
3201
3202You can easily set up to run the AUH utility on a regular basis by using
3203a cron job. See the
3204`weeklyjob.sh <http://git.yoctoproject.org/cgit/cgit.cgi/auto-upgrade-helper/tree/weeklyjob.sh>`_
3205file distributed with the utility for an example.
3206
3207.. _gs-using-devtool-upgrade:
3208
3209Using ``devtool upgrade``
3210-------------------------
3211
3212As mentioned earlier, an alternative method for upgrading recipes to
3213newer versions is to use
3214:doc:`devtool upgrade <../ref-manual/ref-devtool-reference>`.
3215You can read about ``devtool upgrade`` in general in the
3216":ref:`sdk-devtool-use-devtool-upgrade-to-create-a-version-of-the-recipe-that-supports-a-newer-version-of-the-software`"
3217section in the Yocto Project Application Development and the Extensible
3218Software Development Kit (eSDK) Manual.
3219
3220To see all the command-line options available with ``devtool upgrade``,
3221use the following help command:
3222::
3223
3224 $ devtool upgrade -h
3225
3226If you want to find out what version a recipe is currently at upstream
3227without any attempt to upgrade your local version of the recipe, you can
3228use the following command:
3229::
3230
3231 $ devtool latest-version recipe_name
3232
3233As mentioned in the previous section describing AUH, ``devtool upgrade``
3234works in a less-automated manner than AUH. Specifically,
3235``devtool upgrade`` only works on a single recipe that you name on the
3236command line, cannot perform build and integration testing using images,
3237and does not automatically generate commits for changes in the source
3238tree. Despite all these "limitations", ``devtool upgrade`` updates the
3239recipe file to the new upstream version and attempts to rebase custom
3240patches contained by the recipe as needed.
3241
3242.. note::
3243
3244 AUH uses much of
3245 devtool upgrade
3246 behind the scenes making AUH somewhat of a "wrapper" application for
3247 devtool upgrade
3248 .
3249
3250A typical scenario involves having used Git to clone an upstream
3251repository that you use during build operations. Because you are (or
3252have) built the recipe in the past, the layer is likely added to your
3253configuration already. If for some reason, the layer is not added, you
3254could add it easily using the
3255":ref:`bitbake-layers <bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script>`"
3256script. For example, suppose you use the ``nano.bb`` recipe from the
3257``meta-oe`` layer in the ``meta-openembedded`` repository. For this
3258example, assume that the layer has been cloned into following area:
3259::
3260
3261 /home/scottrif/meta-openembedded
3262
3263The following command from your
3264:term:`Build Directory` adds the layer to
3265your build configuration (i.e. ``${BUILDDIR}/conf/bblayers.conf``):
3266::
3267
3268 $ bitbake-layers add-layer /home/scottrif/meta-openembedded/meta-oe
3269 NOTE: Starting bitbake server...
3270 Parsing recipes: 100% |##########################################| Time: 0:00:55
3271 Parsing of 1431 .bb files complete (0 cached, 1431 parsed). 2040 targets, 56 skipped, 0 masked, 0 errors.
3272 Removing 12 recipes from the x86_64 sysroot: 100% |##############| Time: 0:00:00
3273 Removing 1 recipes from the x86_64_i586 sysroot: 100% |##########| Time: 0:00:00
3274 Removing 5 recipes from the i586 sysroot: 100% |#################| Time: 0:00:00
3275 Removing 5 recipes from the qemux86 sysroot: 100% |##############| Time: 0:00:00
3276
3277For this example, assume that the ``nano.bb`` recipe that
3278is upstream has a 2.9.3 version number. However, the version in the
3279local repository is 2.7.4. The following command from your build
3280directory automatically upgrades the recipe for you:
3281
3282.. note::
3283
3284 Using the
3285 -V
3286 option is not necessary. Omitting the version number causes
3287 devtool upgrade
3288 to upgrade the recipe to the most recent version.
3289
3290::
3291
3292 $ devtool upgrade nano -V 2.9.3
3293 NOTE: Starting bitbake server...
3294 NOTE: Creating workspace layer in /home/scottrif/poky/build/workspace
3295 Parsing recipes: 100% |##########################################| Time: 0:00:46
3296 Parsing of 1431 .bb files complete (0 cached, 1431 parsed). 2040 targets, 56 skipped, 0 masked, 0 errors.
3297 NOTE: Extracting current version source...
3298 NOTE: Resolving any missing task queue dependencies
3299 .
3300 .
3301 .
3302 NOTE: Executing SetScene Tasks
3303 NOTE: Executing RunQueue Tasks
3304 NOTE: Tasks Summary: Attempted 74 tasks of which 72 didn't need to be rerun and all succeeded.
3305 Adding changed files: 100% |#####################################| Time: 0:00:00
3306 NOTE: Upgraded source extracted to /home/scottrif/poky/build/workspace/sources/nano
3307 NOTE: New recipe is /home/scottrif/poky/build/workspace/recipes/nano/nano_2.9.3.bb
3308
3309Continuing with this example, you can use ``devtool build`` to build the
3310newly upgraded recipe:
3311::
3312
3313 $ devtool build nano
3314 NOTE: Starting bitbake server...
3315 Loading cache: 100% |################################################################################################| Time: 0:00:01
3316 Loaded 2040 entries from dependency cache.
3317 Parsing recipes: 100% |##############################################################################################| Time: 0:00:00
3318 Parsing of 1432 .bb files complete (1431 cached, 1 parsed). 2041 targets, 56 skipped, 0 masked, 0 errors.
3319 NOTE: Resolving any missing task queue dependencies
3320 .
3321 .
3322 .
3323 NOTE: Executing SetScene Tasks
3324 NOTE: Executing RunQueue Tasks
3325 NOTE: nano: compiling from external source tree /home/scottrif/poky/build/workspace/sources/nano
3326 NOTE: Tasks Summary: Attempted 520 tasks of which 304 didn't need to be rerun and all succeeded.
3327
3328Within the ``devtool upgrade`` workflow, opportunity
3329exists to deploy and test your rebuilt software. For this example,
3330however, running ``devtool finish`` cleans up the workspace once the
3331source in your workspace is clean. This usually means using Git to stage
3332and submit commits for the changes generated by the upgrade process.
3333
3334Once the tree is clean, you can clean things up in this example with the
3335following command from the ``${BUILDDIR}/workspace/sources/nano``
3336directory:
3337::
3338
3339 $ devtool finish nano meta-oe
3340 NOTE: Starting bitbake server...
3341 Loading cache: 100% |################################################################################################| Time: 0:00:00
3342 Loaded 2040 entries from dependency cache.
3343 Parsing recipes: 100% |##############################################################################################| Time: 0:00:01
3344 Parsing of 1432 .bb files complete (1431 cached, 1 parsed). 2041 targets, 56 skipped, 0 masked, 0 errors.
3345 NOTE: Adding new patch 0001-nano.bb-Stuff-I-changed-when-upgrading-nano.bb.patch
3346 NOTE: Updating recipe nano_2.9.3.bb
3347 NOTE: Removing file /home/scottrif/meta-openembedded/meta-oe/recipes-support/nano/nano_2.7.4.bb
3348 NOTE: Moving recipe file to /home/scottrif/meta-openembedded/meta-oe/recipes-support/nano
3349 NOTE: Leaving source tree /home/scottrif/poky/build/workspace/sources/nano as-is; if you no longer need it then please delete it manually
3350
3351
3352Using the ``devtool finish`` command cleans up the workspace and creates a patch
3353file based on your commits. The tool puts all patch files back into the
3354source directory in a sub-directory named ``nano`` in this case.
3355
3356.. _dev-manually-upgrading-a-recipe:
3357
3358Manually Upgrading a Recipe
3359---------------------------
3360
3361If for some reason you choose not to upgrade recipes using the `Auto
3362Upgrade Helper (AUH) <#gs-using-the-auto-upgrade-helper>`__ or by using
3363```devtool upgrade`` <#gs-using-devtool-upgrade>`__, you can manually
3364edit the recipe files to upgrade the versions.
3365
3366.. note::
3367
3368 Manually updating multiple recipes scales poorly and involves many
3369 steps. The recommendation to upgrade recipe versions is through AUH
3370 or
3371 devtool upgrade
3372 , both of which automate some steps and provide guidance for others
3373 needed for the manual process.
3374
3375To manually upgrade recipe versions, follow these general steps:
3376
33771. *Change the Version:* Rename the recipe such that the version (i.e.
3378 the :term:`PV` part of the recipe name)
3379 changes appropriately. If the version is not part of the recipe name,
3380 change the value as it is set for ``PV`` within the recipe itself.
3381
33822. Update ``SRCREV`` if Needed: If the source code your recipe builds
3383 is fetched from Git or some other version control system, update
3384 :term:`SRCREV` to point to the
3385 commit hash that matches the new version.
3386
33873. *Build the Software:* Try to build the recipe using BitBake. Typical
3388 build failures include the following:
3389
3390 - License statements were updated for the new version. For this
3391 case, you need to review any changes to the license and update the
3392 values of :term:`LICENSE` and
3393 :term:`LIC_FILES_CHKSUM`
3394 as needed.
3395
3396 .. note::
3397
3398 License changes are often inconsequential. For example, the
3399 license text's copyright year might have changed.
3400
3401 - Custom patches carried by the older version of the recipe might
3402 fail to apply to the new version. For these cases, you need to
3403 review the failures. Patches might not be necessary for the new
3404 version of the software if the upgraded version has fixed those
3405 issues. If a patch is necessary and failing, you need to rebase it
3406 into the new version.
3407
34084. *Optionally Attempt to Build for Several Architectures:* Once you
3409 successfully build the new software for a given architecture, you
3410 could test the build for other architectures by changing the
3411 :term:`MACHINE` variable and
3412 rebuilding the software. This optional step is especially important
3413 if the recipe is to be released publicly.
3414
34155. *Check the Upstream Change Log or Release Notes:* Checking both these
3416 reveals if new features exist that could break
3417 backwards-compatibility. If so, you need to take steps to mitigate or
3418 eliminate that situation.
3419
34206. *Optionally Create a Bootable Image and Test:* If you want, you can
3421 test the new software by booting it onto actual hardware.
3422
34237. *Create a Commit with the Change in the Layer Repository:* After all
3424 builds work and any testing is successful, you can create commits for
3425 any changes in the layer holding your upgraded recipe.
3426
3427.. _finding-the-temporary-source-code:
3428
3429Finding Temporary Source Code
3430=============================
3431
3432You might find it helpful during development to modify the temporary
3433source code used by recipes to build packages. For example, suppose you
3434are developing a patch and you need to experiment a bit to figure out
3435your solution. After you have initially built the package, you can
3436iteratively tweak the source code, which is located in the
3437:term:`Build Directory`, and then you can
3438force a re-compile and quickly test your altered code. Once you settle
3439on a solution, you can then preserve your changes in the form of
3440patches.
3441
3442During a build, the unpacked temporary source code used by recipes to
3443build packages is available in the Build Directory as defined by the
3444:term:`S` variable. Below is the default
3445value for the ``S`` variable as defined in the
3446``meta/conf/bitbake.conf`` configuration file in the
3447:term:`Source Directory`:
3448::
3449
3450 S = "${WORKDIR}/${BP}"
3451
3452You should be aware that many recipes override the
3453``S`` variable. For example, recipes that fetch their source from Git
3454usually set ``S`` to ``${WORKDIR}/git``.
3455
3456.. note::
3457
3458 The
3459 BP
3460 represents the base recipe name, which consists of the name and
3461 version:
3462 ::
3463
3464 BP = "${BPN}-${PV}"
3465
3466
3467The path to the work directory for the recipe
3468(:term:`WORKDIR`) is defined as
3469follows:
3470${TMPDIR}/work/${MULTIMACH_TARGET_SYS}/${PN}/${EXTENDPE}${PV}-${PR} The
3471actual directory depends on several things:
3472
3473- :term:`TMPDIR`: The top-level build
3474 output directory.
3475
3476- :term:`MULTIMACH_TARGET_SYS`:
3477 The target system identifier.
3478
3479- :term:`PN`: The recipe name.
3480
3481- :term:`EXTENDPE`: The epoch - (if
3482 :term:`PE` is not specified, which is
3483 usually the case for most recipes, then ``EXTENDPE`` is blank).
3484
3485- :term:`PV`: The recipe version.
3486
3487- :term:`PR`: The recipe revision.
3488
3489As an example, assume a Source Directory top-level folder named
3490``poky``, a default Build Directory at ``poky/build``, and a
3491``qemux86-poky-linux`` machine target system. Furthermore, suppose your
3492recipe is named ``foo_1.3.0.bb``. In this case, the work directory the
3493build system uses to build the package would be as follows:
3494::
3495
3496 poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0
3497
3498.. _using-a-quilt-workflow:
3499
3500Using Quilt in Your Workflow
3501============================
3502
3503`Quilt <http://savannah.nongnu.org/projects/quilt>`__ is a powerful tool
3504that allows you to capture source code changes without having a clean
3505source tree. This section outlines the typical workflow you can use to
3506modify source code, test changes, and then preserve the changes in the
3507form of a patch all using Quilt.
3508
3509.. note::
3510
3511 With regard to preserving changes to source files, if you clean a
3512 recipe or have
3513 rm_work
3514 enabled, the
3515 devtool
3516 workflow
3517 as described in the Yocto Project Application Development and the
3518 Extensible Software Development Kit (eSDK) manual is a safer
3519 development flow than the flow that uses Quilt.
3520
3521Follow these general steps:
3522
35231. *Find the Source Code:* Temporary source code used by the
3524 OpenEmbedded build system is kept in the
3525 :term:`Build Directory`. See the
3526 "`Finding Temporary Source
3527 Code <#finding-the-temporary-source-code>`__" section to learn how to
3528 locate the directory that has the temporary source code for a
3529 particular package.
3530
35312. *Change Your Working Directory:* You need to be in the directory that
3532 has the temporary source code. That directory is defined by the
3533 :term:`S` variable.
3534
35353. *Create a New Patch:* Before modifying source code, you need to
3536 create a new patch. To create a new patch file, use ``quilt new`` as
3537 below:
3538 :;
3539
3540 $ quilt new my_changes.patch
3541
35424. *Notify Quilt and Add Files:* After creating the patch, you need to
3543 notify Quilt about the files you plan to edit. You notify Quilt by
3544 adding the files to the patch you just created:
3545 ::
3546
3547 $ quilt add file1.c file2.c file3.c
3548
35495. *Edit the Files:* Make your changes in the source code to the files
3550 you added to the patch.
3551
35526. *Test Your Changes:* Once you have modified the source code, the
3553 easiest way to test your changes is by calling the ``do_compile``
3554 task as shown in the following example:
3555 ::
3556
3557 $ bitbake -c compile -f package
3558
3559 The ``-f`` or ``--force`` option forces the specified task to
3560 execute. If you find problems with your code, you can just keep
3561 editing and re-testing iteratively until things work as expected.
3562
3563 .. note::
3564
3565 All the modifications you make to the temporary source code
3566 disappear once you run the
3567 do_clean
3568 or
3569 do_cleanall
3570 tasks using BitBake (i.e.
3571 bitbake -c clean
3572 package
3573 and
3574 bitbake -c cleanall
3575 package
3576 ). Modifications will also disappear if you use the
3577 rm_work
3578 feature as described in the "
3579 Conserving Disk Space During Builds
3580 " section.
3581
35827. *Generate the Patch:* Once your changes work as expected, you need to
3583 use Quilt to generate the final patch that contains all your
3584 modifications.
3585 ::
3586
3587 $ quilt refresh
3588
3589 At this point, the
3590 ``my_changes.patch`` file has all your edits made to the ``file1.c``,
3591 ``file2.c``, and ``file3.c`` files.
3592
3593 You can find the resulting patch file in the ``patches/``
3594 subdirectory of the source (``S``) directory.
3595
35968. *Copy the Patch File:* For simplicity, copy the patch file into a
3597 directory named ``files``, which you can create in the same directory
3598 that holds the recipe (``.bb``) file or the append (``.bbappend``)
3599 file. Placing the patch here guarantees that the OpenEmbedded build
3600 system will find the patch. Next, add the patch into the ``SRC_URI``
3601 of the recipe. Here is an example:
3602 ::
3603
3604 SRC_URI += "file://my_changes.patch"
3605
3606.. _platdev-appdev-devshell:
3607
3608Using a Development Shell
3609=========================
3610
3611When debugging certain commands or even when just editing packages,
3612``devshell`` can be a useful tool. When you invoke ``devshell``, all
3613tasks up to and including
3614:ref:`ref-tasks-patch` are run for the
3615specified target. Then, a new terminal is opened and you are placed in
3616``${``\ :term:`S`\ ``}``, the source
3617directory. In the new terminal, all the OpenEmbedded build-related
3618environment variables are still defined so you can use commands such as
3619``configure`` and ``make``. The commands execute just as if the
3620OpenEmbedded build system were executing them. Consequently, working
3621this way can be helpful when debugging a build or preparing software to
3622be used with the OpenEmbedded build system.
3623
3624Following is an example that uses ``devshell`` on a target named
3625``matchbox-desktop``:
3626::
3627
3628 $ bitbake matchbox-desktop -c devshell
3629
3630This command spawns a terminal with a shell prompt within the
3631OpenEmbedded build environment. The
3632:term:`OE_TERMINAL` variable
3633controls what type of shell is opened.
3634
3635For spawned terminals, the following occurs:
3636
3637- The ``PATH`` variable includes the cross-toolchain.
3638
3639- The ``pkgconfig`` variables find the correct ``.pc`` files.
3640
3641- The ``configure`` command finds the Yocto Project site files as well
3642 as any other necessary files.
3643
3644Within this environment, you can run configure or compile commands as if
3645they were being run by the OpenEmbedded build system itself. As noted
3646earlier, the working directory also automatically changes to the Source
3647Directory (:term:`S`).
3648
3649To manually run a specific task using ``devshell``, run the
3650corresponding ``run.*`` script in the
3651``${``\ :term:`WORKDIR`\ ``}/temp``
3652directory (e.g., ``run.do_configure.``\ pid). If a task's script does
3653not exist, which would be the case if the task was skipped by way of the
3654sstate cache, you can create the task by first running it outside of the
3655``devshell``:
3656::
3657
3658 $ bitbake -c task
3659
3660.. note::
3661
3662 - Execution of a task's ``run.*`` script and BitBake's execution of
3663 a task are identical. In other words, running the script re-runs
3664 the task just as it would be run using the ``bitbake -c`` command.
3665
3666 - Any ``run.*`` file that does not have a ``.pid`` extension is a
3667 symbolic link (symlink) to the most recent version of that file.
3668
3669Remember, that the ``devshell`` is a mechanism that allows you to get
3670into the BitBake task execution environment. And as such, all commands
3671must be called just as BitBake would call them. That means you need to
3672provide the appropriate options for cross-compilation and so forth as
3673applicable.
3674
3675When you are finished using ``devshell``, exit the shell or close the
3676terminal window.
3677
3678.. note::
3679
3680 - It is worth remembering that when using ``devshell`` you need to
3681 use the full compiler name such as ``arm-poky-linux-gnueabi-gcc``
3682 instead of just using ``gcc``. The same applies to other
3683 applications such as ``binutils``, ``libtool`` and so forth.
3684 BitBake sets up environment variables such as ``CC`` to assist
3685 applications, such as ``make`` to find the correct tools.
3686
3687 - It is also worth noting that ``devshell`` still works over X11
3688 forwarding and similar situations.
3689
3690.. _platdev-appdev-devpyshell:
3691
3692Using a Development Python Shell
3693================================
3694
3695Similar to working within a development shell as described in the
3696previous section, you can also spawn and work within an interactive
3697Python development shell. When debugging certain commands or even when
3698just editing packages, ``devpyshell`` can be a useful tool. When you
3699invoke ``devpyshell``, all tasks up to and including
3700:ref:`ref-tasks-patch` are run for the
3701specified target. Then a new terminal is opened. Additionally, key
3702Python objects and code are available in the same way they are to
3703BitBake tasks, in particular, the data store 'd'. So, commands such as
3704the following are useful when exploring the data store and running
3705functions:
3706::
3707
3708 pydevshell> d.getVar("STAGING_DIR")
3709 '/media/build1/poky/build/tmp/sysroots'
3710 pydevshell> d.getVar("STAGING_DIR")
3711 '${TMPDIR}/sysroots'
3712 pydevshell> d.setVar("FOO", "bar")
3713 pydevshell> d.getVar("FOO")
3714 'bar'
3715 pydevshell> d.delVar("FOO")
3716 pydevshell> d.getVar("FOO")
3717 pydevshell> bb.build.exec_func("do_unpack", d)
3718 pydevshell>
3719
3720The commands execute just as if the OpenEmbedded build
3721system were executing them. Consequently, working this way can be
3722helpful when debugging a build or preparing software to be used with the
3723OpenEmbedded build system.
3724
3725Following is an example that uses ``devpyshell`` on a target named
3726``matchbox-desktop``:
3727::
3728
3729 $ bitbake matchbox-desktop -c devpyshell
3730
3731This command spawns a terminal and places you in an interactive Python
3732interpreter within the OpenEmbedded build environment. The
3733:term:`OE_TERMINAL` variable
3734controls what type of shell is opened.
3735
3736When you are finished using ``devpyshell``, you can exit the shell
3737either by using Ctrl+d or closing the terminal window.
3738
3739.. _dev-building:
3740
3741Building
3742========
3743
3744This section describes various build procedures. For example, the steps
3745needed for a simple build, a target that uses multiple configurations,
3746building an image for more than one machine, and so forth.
3747
3748.. _dev-building-a-simple-image:
3749
3750Building a Simple Image
3751-----------------------
3752
3753In the development environment, you need to build an image whenever you
3754change hardware support, add or change system libraries, or add or
3755change services that have dependencies. Several methods exist that allow
3756you to build an image within the Yocto Project. This section presents
3757the basic steps you need to build a simple image using BitBake from a
3758build host running Linux.
3759
3760.. note::
3761
3762 - For information on how to build an image using
3763 :term:`Toaster`, see the
3764 :doc:`../toaster-manual/toaster-manual`.
3765
3766 - For information on how to use ``devtool`` to build images, see the
3767 ":ref:`sdk-manual/sdk-extensible:using \`\`devtool\`\` in your sdk workflow`"
3768 section in the Yocto Project Application Development and the
3769 Extensible Software Development Kit (eSDK) manual.
3770
3771 - For a quick example on how to build an image using the
3772 OpenEmbedded build system, see the
3773 :doc:`../brief-yoctoprojectqs/brief-yoctoprojectqs` document.
3774
3775The build process creates an entire Linux distribution from source and
3776places it in your :term:`Build Directory` under
3777``tmp/deploy/images``. For detailed information on the build process
3778using BitBake, see the ":ref:`images-dev-environment`" section in the
3779Yocto Project Overview and Concepts Manual.
3780
3781The following figure and list overviews the build process:
3782
3783.. image:: figures/bitbake-build-flow.png
3784 :align: center
3785
37861. *Set up Your Host Development System to Support Development Using the
3787 Yocto Project*: See the "`Setting Up to Use the Yocto
3788 Project <#dev-manual-start>`__" section for options on how to get a
3789 build host ready to use the Yocto Project.
3790
37912. *Initialize the Build Environment:* Initialize the build environment
3792 by sourcing the build environment script (i.e.
3793 :ref:`structure-core-script`):
3794 ::
3795
3796 $ source oe-init-build-env [build_dir]
3797
3798 When you use the initialization script, the OpenEmbedded build system
3799 uses ``build`` as the default Build Directory in your current work
3800 directory. You can use a build_dir argument with the script to
3801 specify a different build directory.
3802
3803 .. note::
3804
3805 A common practice is to use a different Build Directory for
3806 different targets. For example,
3807 ~/build/x86
3808 for a
3809 qemux86
3810 target, and
3811 ~/build/arm
3812 for a
3813 qemuarm
3814 target.
3815
38163. Make Sure Your ``local.conf`` File is Correct: Ensure the
3817 ``conf/local.conf`` configuration file, which is found in the Build
3818 Directory, is set up how you want it. This file defines many aspects
3819 of the build environment including the target machine architecture
3820 through the ``MACHINE`` variable, the packaging format used during
3821 the build
3822 (:term:`PACKAGE_CLASSES`),
3823 and a centralized tarball download directory through the
3824 :term:`DL_DIR` variable.
3825
38264. *Build the Image:* Build the image using the ``bitbake`` command:
3827 ::
3828
3829 $ bitbake target
3830
3831 .. note::
3832
3833 For information on BitBake, see the
3834 BitBake User Manual
3835 .
3836
3837 The target is the name of the recipe you want to build. Common
3838 targets are the images in ``meta/recipes-core/images``,
3839 ``meta/recipes-sato/images``, and so forth all found in the
3840 :term:`Source Directory`. Or, the target
3841 can be the name of a recipe for a specific piece of software such as
3842 BusyBox. For more details about the images the OpenEmbedded build
3843 system supports, see the
3844 ":ref:`ref-manual/ref-images:Images`" chapter in the Yocto
3845 Project Reference Manual.
3846
3847 As an example, the following command builds the
3848 ``core-image-minimal`` image:
3849 ::
3850
3851 $ bitbake core-image-minimal
3852
3853 Once an
3854 image has been built, it often needs to be installed. The images and
3855 kernels built by the OpenEmbedded build system are placed in the
3856 Build Directory in ``tmp/deploy/images``. For information on how to
3857 run pre-built images such as ``qemux86`` and ``qemuarm``, see the
3858 :doc:`../sdk-manual/sdk-manual` manual. For
3859 information about how to install these images, see the documentation
3860 for your particular board or machine.
3861
3862.. _dev-building-images-for-multiple-targets-using-multiple-configurations:
3863
3864Building Images for Multiple Targets Using Multiple Configurations
3865------------------------------------------------------------------
3866
3867You can use a single ``bitbake`` command to build multiple images or
3868packages for different targets where each image or package requires a
3869different configuration (multiple configuration builds). The builds, in
3870this scenario, are sometimes referred to as "multiconfigs", and this
3871section uses that term throughout.
3872
3873This section describes how to set up for multiple configuration builds
3874and how to account for cross-build dependencies between the
3875multiconfigs.
3876
3877.. _dev-setting-up-and-running-a-multiple-configuration-build:
3878
3879Setting Up and Running a Multiple Configuration Build
3880~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3881
3882To accomplish a multiple configuration build, you must define each
3883target's configuration separately using a parallel configuration file in
3884the :term:`Build Directory`, and you
3885must follow a required file hierarchy. Additionally, you must enable the
3886multiple configuration builds in your ``local.conf`` file.
3887
3888Follow these steps to set up and execute multiple configuration builds:
3889
3890- *Create Separate Configuration Files*: You need to create a single
3891 configuration file for each build target (each multiconfig).
3892 Minimally, each configuration file must define the machine and the
3893 temporary directory BitBake uses for the build. Suggested practice
3894 dictates that you do not overlap the temporary directories used
3895 during the builds. However, it is possible that you can share the
3896 temporary directory
3897 (:term:`TMPDIR`). For example,
3898 consider a scenario with two different multiconfigs for the same
3899 :term:`MACHINE`: "qemux86" built
3900 for two distributions such as "poky" and "poky-lsb". In this case,
3901 you might want to use the same ``TMPDIR``.
3902
3903 Here is an example showing the minimal statements needed in a
3904 configuration file for a "qemux86" target whose temporary build
3905 directory is ``tmpmultix86``:
3906 ::
3907
3908 MACHINE = "qemux86"
3909 TMPDIR = "${TOPDIR}/tmpmultix86"
3910
3911 The location for these multiconfig configuration files is specific.
3912 They must reside in the current build directory in a sub-directory of
3913 ``conf`` named ``multiconfig``. Following is an example that defines
3914 two configuration files for the "x86" and "arm" multiconfigs:
3915
3916 .. image:: figures/multiconfig_files.png
3917 :align: center
3918
3919 The reason for this required file hierarchy is because the ``BBPATH``
3920 variable is not constructed until the layers are parsed.
3921 Consequently, using the configuration file as a pre-configuration
3922 file is not possible unless it is located in the current working
3923 directory.
3924
3925- *Add the BitBake Multi-configuration Variable to the Local
3926 Configuration File*: Use the
3927 :term:`BBMULTICONFIG`
3928 variable in your ``conf/local.conf`` configuration file to specify
3929 each multiconfig. Continuing with the example from the previous
3930 figure, the ``BBMULTICONFIG`` variable needs to enable two
3931 multiconfigs: "x86" and "arm" by specifying each configuration file:
3932 ::
3933
3934 BBMULTICONFIG = "x86 arm"
3935
3936 .. note::
3937
3938 A "default" configuration already exists by definition. This
3939 configuration is named: "" (i.e. empty string) and is defined by
3940 the variables coming from your
3941 local.conf
3942 file. Consequently, the previous example actually adds two
3943 additional configurations to your build: "arm" and "x86" along
3944 with "".
3945
3946- *Launch BitBake*: Use the following BitBake command form to launch
3947 the multiple configuration build:
3948 ::
3949
3950 $ bitbake [mc:multiconfigname:]target [[[mc:multiconfigname:]target] ... ]
3951
3952 For the example in this section, the following command applies:
3953 ::
3954
3955 $ bitbake mc:x86:core-image-minimal mc:arm:core-image-sato mc::core-image-base
3956
3957 The previous BitBake command builds a ``core-image-minimal`` image
3958 that is configured through the ``x86.conf`` configuration file, a
3959 ``core-image-sato`` image that is configured through the ``arm.conf``
3960 configuration file and a ``core-image-base`` that is configured
3961 through your ``local.conf`` configuration file.
3962
3963.. note::
3964
3965 Support for multiple configuration builds in the Yocto Project DISTRO
3966 (DISTRO_NAME) Release does not include Shared State (sstate)
3967 optimizations. Consequently, if a build uses the same object twice
3968 in, for example, two different
3969 TMPDIR
3970 directories, the build either loads from an existing sstate cache for
3971 that build at the start or builds the object fresh.
3972
3973.. _dev-enabling-multiple-configuration-build-dependencies:
3974
3975Enabling Multiple Configuration Build Dependencies
3976~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3977
3978Sometimes dependencies can exist between targets (multiconfigs) in a
3979multiple configuration build. For example, suppose that in order to
3980build a ``core-image-sato`` image for an "x86" multiconfig, the root
3981filesystem of an "arm" multiconfig must exist. This dependency is
3982essentially that the
3983:ref:`ref-tasks-image` task in the
3984``core-image-sato`` recipe depends on the completion of the
3985:ref:`ref-tasks-rootfs` task of the
3986``core-image-minimal`` recipe.
3987
3988To enable dependencies in a multiple configuration build, you must
3989declare the dependencies in the recipe using the following statement
3990form:
3991::
3992
3993 task_or_package[mcdepends] = "mc:from_multiconfig:to_multiconfig:recipe_name:task_on_which_to_depend"
3994
3995To better show how to use this statement, consider the example scenario
3996from the first paragraph of this section. The following statement needs
3997to be added to the recipe that builds the ``core-image-sato`` image:
3998::
3999
4000 do_image[mcdepends] = "mc:x86:arm:core-image-minimal:do_rootfs"
4001
4002In this example, the from_multiconfig is "x86". The to_multiconfig is "arm". The
4003task on which the ``do_image`` task in the recipe depends is the
4004``do_rootfs`` task from the ``core-image-minimal`` recipe associated
4005with the "arm" multiconfig.
4006
4007Once you set up this dependency, you can build the "x86" multiconfig
4008using a BitBake command as follows:
4009::
4010
4011 $ bitbake mc:x86:core-image-sato
4012
4013This command executes all the tasks needed to create the
4014``core-image-sato`` image for the "x86" multiconfig. Because of the
4015dependency, BitBake also executes through the ``do_rootfs`` task for the
4016"arm" multiconfig build.
4017
4018Having a recipe depend on the root filesystem of another build might not
4019seem that useful. Consider this change to the statement in the
4020``core-image-sato`` recipe:
4021::
4022
4023 do_image[mcdepends] = "mc:x86:arm:core-image-minimal:do_image"
4024
4025In this case, BitBake must
4026create the ``core-image-minimal`` image for the "arm" build since the
4027"x86" build depends on it.
4028
4029Because "x86" and "arm" are enabled for multiple configuration builds
4030and have separate configuration files, BitBake places the artifacts for
4031each build in the respective temporary build directories (i.e.
4032:term:`TMPDIR`).
4033
4034.. _building-an-initramfs-image:
4035
4036Building an Initial RAM Filesystem (initramfs) Image
4037----------------------------------------------------
4038
4039An initial RAM filesystem (initramfs) image provides a temporary root
4040filesystem used for early system initialization (e.g. loading of modules
4041needed to locate and mount the "real" root filesystem).
4042
4043.. note::
4044
4045 The initramfs image is the successor of initial RAM disk (initrd). It
4046 is a "copy in and out" (cpio) archive of the initial filesystem that
4047 gets loaded into memory during the Linux startup process. Because
4048 Linux uses the contents of the archive during initialization, the
4049 initramfs image needs to contain all of the device drivers and tools
4050 needed to mount the final root filesystem.
4051
4052Follow these steps to create an initramfs image:
4053
40541. *Create the initramfs Image Recipe:* You can reference the
4055 ``core-image-minimal-initramfs.bb`` recipe found in the
4056 ``meta/recipes-core`` directory of the :term:`Source Directory`
4057 as an example
4058 from which to work.
4059
40602. *Decide if You Need to Bundle the initramfs Image Into the Kernel
4061 Image:* If you want the initramfs image that is built to be bundled
4062 in with the kernel image, set the
4063 :term:`INITRAMFS_IMAGE_BUNDLE`
4064 variable to "1" in your ``local.conf`` configuration file and set the
4065 :term:`INITRAMFS_IMAGE`
4066 variable in the recipe that builds the kernel image.
4067
4068 .. note::
4069
4070 It is recommended that you do bundle the initramfs image with the
4071 kernel image to avoid circular dependencies between the kernel
4072 recipe and the initramfs recipe should the initramfs image include
4073 kernel modules.
4074
4075 Setting the ``INITRAMFS_IMAGE_BUNDLE`` flag causes the initramfs
4076 image to be unpacked into the ``${B}/usr/`` directory. The unpacked
4077 initramfs image is then passed to the kernel's ``Makefile`` using the
4078 :term:`CONFIG_INITRAMFS_SOURCE`
4079 variable, allowing the initramfs image to be built into the kernel
4080 normally.
4081
4082 .. note::
4083
4084 If you choose to not bundle the initramfs image with the kernel
4085 image, you are essentially using an
4086 Initial RAM Disk (initrd)
4087 . Creating an initrd is handled primarily through the
4088 INITRD_IMAGE
4089 ,
4090 INITRD_LIVE
4091 , and
4092 INITRD_IMAGE_LIVE
4093 variables. For more information, see the
4094 image-live.bbclass
4095 file.
4096
40973. *Optionally Add Items to the initramfs Image Through the initramfs
4098 Image Recipe:* If you add items to the initramfs image by way of its
4099 recipe, you should use
4100 :term:`PACKAGE_INSTALL`
4101 rather than
4102 :term:`IMAGE_INSTALL`.
4103 ``PACKAGE_INSTALL`` gives more direct control of what is added to the
4104 image as compared to the defaults you might not necessarily want that
4105 are set by the :ref:`image <ref-classes-image>`
4106 or :ref:`core-image <ref-classes-core-image>`
4107 classes.
4108
41094. *Build the Kernel Image and the initramfs Image:* Build your kernel
4110 image using BitBake. Because the initramfs image recipe is a
4111 dependency of the kernel image, the initramfs image is built as well
4112 and bundled with the kernel image if you used the
4113 :term:`INITRAMFS_IMAGE_BUNDLE`
4114 variable described earlier.
4115
4116Building a Tiny System
4117----------------------
4118
4119Very small distributions have some significant advantages such as
4120requiring less on-die or in-package memory (cheaper), better performance
4121through efficient cache usage, lower power requirements due to less
4122memory, faster boot times, and reduced development overhead. Some
4123real-world examples where a very small distribution gives you distinct
4124advantages are digital cameras, medical devices, and small headless
4125systems.
4126
4127This section presents information that shows you how you can trim your
4128distribution to even smaller sizes than the ``poky-tiny`` distribution,
4129which is around 5 Mbytes, that can be built out-of-the-box using the
4130Yocto Project.
4131
4132.. _tiny-system-overview:
4133
4134Tiny System Overview
4135~~~~~~~~~~~~~~~~~~~~
4136
4137The following list presents the overall steps you need to consider and
4138perform to create distributions with smaller root filesystems, achieve
4139faster boot times, maintain your critical functionality, and avoid
4140initial RAM disks:
4141
4142- `Determine your goals and guiding
4143 principles. <#goals-and-guiding-principles>`__
4144
4145- `Understand what contributes to your image
4146 size. <#understand-what-gives-your-image-size>`__
4147
4148- `Reduce the size of the root
4149 filesystem. <#trim-the-root-filesystem>`__
4150
4151- `Reduce the size of the kernel. <#trim-the-kernel>`__
4152
4153- `Eliminate packaging
4154 requirements. <#remove-package-management-requirements>`__
4155
4156- `Look for other ways to minimize
4157 size. <#look-for-other-ways-to-minimize-size>`__
4158
4159- `Iterate on the process. <#iterate-on-the-process>`__
4160
4161Goals and Guiding Principles
4162~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4163
4164Before you can reach your destination, you need to know where you are
4165going. Here is an example list that you can use as a guide when creating
4166very small distributions:
4167
4168- Determine how much space you need (e.g. a kernel that is 1 Mbyte or
4169 less and a root filesystem that is 3 Mbytes or less).
4170
4171- Find the areas that are currently taking 90% of the space and
4172 concentrate on reducing those areas.
4173
4174- Do not create any difficult "hacks" to achieve your goals.
4175
4176- Leverage the device-specific options.
4177
4178- Work in a separate layer so that you keep changes isolated. For
4179 information on how to create layers, see the "`Understanding and
4180 Creating Layers <#understanding-and-creating-layers>`__" section.
4181
4182.. _understand-what-gives-your-image-size:
4183
4184Understand What Contributes to Your Image Size
4185~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4186
4187It is easiest to have something to start with when creating your own
4188distribution. You can use the Yocto Project out-of-the-box to create the
4189``poky-tiny`` distribution. Ultimately, you will want to make changes in
4190your own distribution that are likely modeled after ``poky-tiny``.
4191
4192.. note::
4193
4194 To use
4195 poky-tiny
4196 in your build, set the
4197 DISTRO
4198 variable in your
4199 local.conf
4200 file to "poky-tiny" as described in the "
4201 Creating Your Own Distribution
4202 " section.
4203
4204Understanding some memory concepts will help you reduce the system size.
4205Memory consists of static, dynamic, and temporary memory. Static memory
4206is the TEXT (code), DATA (initialized data in the code), and BSS
4207(uninitialized data) sections. Dynamic memory represents memory that is
4208allocated at runtime: stacks, hash tables, and so forth. Temporary
4209memory is recovered after the boot process. This memory consists of
4210memory used for decompressing the kernel and for the ``__init__``
4211functions.
4212
4213To help you see where you currently are with kernel and root filesystem
4214sizes, you can use two tools found in the :term:`Source Directory`
4215in the
4216``scripts/tiny/`` directory:
4217
4218- ``ksize.py``: Reports component sizes for the kernel build objects.
4219
4220- ``dirsize.py``: Reports component sizes for the root filesystem.
4221
4222This next tool and command help you organize configuration fragments and
4223view file dependencies in a human-readable form:
4224
4225- ``merge_config.sh``: Helps you manage configuration files and
4226 fragments within the kernel. With this tool, you can merge individual
4227 configuration fragments together. The tool allows you to make
4228 overrides and warns you of any missing configuration options. The
4229 tool is ideal for allowing you to iterate on configurations, create
4230 minimal configurations, and create configuration files for different
4231 machines without having to duplicate your process.
4232
4233 The ``merge_config.sh`` script is part of the Linux Yocto kernel Git
4234 repositories (i.e. ``linux-yocto-3.14``, ``linux-yocto-3.10``,
4235 ``linux-yocto-3.8``, and so forth) in the ``scripts/kconfig``
4236 directory.
4237
4238 For more information on configuration fragments, see the
4239 ":ref:`creating-config-fragments`"
4240 section in the Yocto Project Linux Kernel Development Manual.
4241
4242- ``bitbake -u taskexp -g bitbake_target``: Using the BitBake command
4243 with these options brings up a Dependency Explorer from which you can
4244 view file dependencies. Understanding these dependencies allows you
4245 to make informed decisions when cutting out various pieces of the
4246 kernel and root filesystem.
4247
4248Trim the Root Filesystem
4249~~~~~~~~~~~~~~~~~~~~~~~~
4250
4251The root filesystem is made up of packages for booting, libraries, and
4252applications. To change things, you can configure how the packaging
4253happens, which changes the way you build them. You can also modify the
4254filesystem itself or select a different filesystem.
4255
4256First, find out what is hogging your root filesystem by running the
4257``dirsize.py`` script from your root directory:
4258::
4259
4260 $ cd root-directory-of-image
4261 $ dirsize.py 100000 > dirsize-100k.log
4262 $ cat dirsize-100k.log
4263
4264You can apply a filter to the script to ignore files
4265under a certain size. The previous example filters out any files below
4266100 Kbytes. The sizes reported by the tool are uncompressed, and thus
4267will be smaller by a relatively constant factor in a compressed root
4268filesystem. When you examine your log file, you can focus on areas of
4269the root filesystem that take up large amounts of memory.
4270
4271You need to be sure that what you eliminate does not cripple the
4272functionality you need. One way to see how packages relate to each other
4273is by using the Dependency Explorer UI with the BitBake command:
4274::
4275
4276 $ cd image-directory
4277 $ bitbake -u taskexp -g image
4278
4279Use the interface to
4280select potential packages you wish to eliminate and see their dependency
4281relationships.
4282
4283When deciding how to reduce the size, get rid of packages that result in
4284minimal impact on the feature set. For example, you might not need a VGA
4285display. Or, you might be able to get by with ``devtmpfs`` and ``mdev``
4286instead of ``udev``.
4287
4288Use your ``local.conf`` file to make changes. For example, to eliminate
4289``udev`` and ``glib``, set the following in the local configuration
4290file:
4291::
4292
4293 VIRTUAL-RUNTIME_dev_manager = ""
4294
4295Finally, you should consider exactly the type of root filesystem you
4296need to meet your needs while also reducing its size. For example,
4297consider ``cramfs``, ``squashfs``, ``ubifs``, ``ext2``, or an
4298``initramfs`` using ``initramfs``. Be aware that ``ext3`` requires a 1
4299Mbyte journal. If you are okay with running read-only, you do not need
4300this journal.
4301
4302.. note::
4303
4304 After each round of elimination, you need to rebuild your system and
4305 then use the tools to see the effects of your reductions.
4306
4307Trim the Kernel
4308~~~~~~~~~~~~~~~
4309
4310The kernel is built by including policies for hardware-independent
4311aspects. What subsystems do you enable? For what architecture are you
4312building? Which drivers do you build by default?
4313
4314.. note::
4315
4316 You can modify the kernel source if you want to help with boot time.
4317
4318Run the ``ksize.py`` script from the top-level Linux build directory to
4319get an idea of what is making up the kernel:
4320::
4321
4322 $ cd top-level-linux-build-directory
4323 $ ksize.py > ksize.log
4324 $ cat ksize.log
4325
4326When you examine the log, you will see how much space is taken up with
4327the built-in ``.o`` files for drivers, networking, core kernel files,
4328filesystem, sound, and so forth. The sizes reported by the tool are
4329uncompressed, and thus will be smaller by a relatively constant factor
4330in a compressed kernel image. Look to reduce the areas that are large
4331and taking up around the "90% rule."
4332
4333To examine, or drill down, into any particular area, use the ``-d``
4334option with the script:
4335::
4336
4337 $ ksize.py -d > ksize.log
4338
4339Using this option
4340breaks out the individual file information for each area of the kernel
4341(e.g. drivers, networking, and so forth).
4342
4343Use your log file to see what you can eliminate from the kernel based on
4344features you can let go. For example, if you are not going to need
4345sound, you do not need any drivers that support sound.
4346
4347After figuring out what to eliminate, you need to reconfigure the kernel
4348to reflect those changes during the next build. You could run
4349``menuconfig`` and make all your changes at once. However, that makes it
4350difficult to see the effects of your individual eliminations and also
4351makes it difficult to replicate the changes for perhaps another target
4352device. A better method is to start with no configurations using
4353``allnoconfig``, create configuration fragments for individual changes,
4354and then manage the fragments into a single configuration file using
4355``merge_config.sh``. The tool makes it easy for you to iterate using the
4356configuration change and build cycle.
4357
4358Each time you make configuration changes, you need to rebuild the kernel
4359and check to see what impact your changes had on the overall size.
4360
4361Remove Package Management Requirements
4362~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4363
4364Packaging requirements add size to the image. One way to reduce the size
4365of the image is to remove all the packaging requirements from the image.
4366This reduction includes both removing the package manager and its unique
4367dependencies as well as removing the package management data itself.
4368
4369To eliminate all the packaging requirements for an image, be sure that
4370"package-management" is not part of your
4371:term:`IMAGE_FEATURES`
4372statement for the image. When you remove this feature, you are removing
4373the package manager as well as its dependencies from the root
4374filesystem.
4375
4376Look for Other Ways to Minimize Size
4377~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4378
4379Depending on your particular circumstances, other areas that you can
4380trim likely exist. The key to finding these areas is through tools and
4381methods described here combined with experimentation and iteration. Here
4382are a couple of areas to experiment with:
4383
4384- ``glibc``: In general, follow this process:
4385
4386 1. Remove ``glibc`` features from
4387 :term:`DISTRO_FEATURES`
4388 that you think you do not need.
4389
4390 2. Build your distribution.
4391
4392 3. If the build fails due to missing symbols in a package, determine
4393 if you can reconfigure the package to not need those features. For
4394 example, change the configuration to not support wide character
4395 support as is done for ``ncurses``. Or, if support for those
4396 characters is needed, determine what ``glibc`` features provide
4397 the support and restore the configuration.
4398
4399 4. Rebuild and repeat the process.
4400
4401- ``busybox``: For BusyBox, use a process similar as described for
4402 ``glibc``. A difference is you will need to boot the resulting system
4403 to see if you are able to do everything you expect from the running
4404 system. You need to be sure to integrate configuration fragments into
4405 Busybox because BusyBox handles its own core features and then allows
4406 you to add configuration fragments on top.
4407
4408Iterate on the Process
4409~~~~~~~~~~~~~~~~~~~~~~
4410
4411If you have not reached your goals on system size, you need to iterate
4412on the process. The process is the same. Use the tools and see just what
4413is taking up 90% of the root filesystem and the kernel. Decide what you
4414can eliminate without limiting your device beyond what you need.
4415
4416Depending on your system, a good place to look might be Busybox, which
4417provides a stripped down version of Unix tools in a single, executable
4418file. You might be able to drop virtual terminal services or perhaps
4419ipv6.
4420
4421Building Images for More than One Machine
4422-----------------------------------------
4423
4424A common scenario developers face is creating images for several
4425different machines that use the same software environment. In this
4426situation, it is tempting to set the tunings and optimization flags for
4427each build specifically for the targeted hardware (i.e. "maxing out" the
4428tunings). Doing so can considerably add to build times and package feed
4429maintenance collectively for the machines. For example, selecting tunes
4430that are extremely specific to a CPU core used in a system might enable
4431some micro optimizations in GCC for that particular system but would
4432otherwise not gain you much of a performance difference across the other
4433systems as compared to using a more general tuning across all the builds
4434(e.g. setting :term:`DEFAULTTUNE`
4435specifically for each machine's build). Rather than "max out" each
4436build's tunings, you can take steps that cause the OpenEmbedded build
4437system to reuse software across the various machines where it makes
4438sense.
4439
4440If build speed and package feed maintenance are considerations, you
4441should consider the points in this section that can help you optimize
4442your tunings to best consider build times and package feed maintenance.
4443
4444- *Share the Build Directory:* If at all possible, share the
4445 :term:`TMPDIR` across builds. The
4446 Yocto Project supports switching between different
4447 :term:`MACHINE` values in the same
4448 ``TMPDIR``. This practice is well supported and regularly used by
4449 developers when building for multiple machines. When you use the same
4450 ``TMPDIR`` for multiple machine builds, the OpenEmbedded build system
4451 can reuse the existing native and often cross-recipes for multiple
4452 machines. Thus, build time decreases.
4453
4454 .. note::
4455
4456 If
4457 DISTRO
4458 settings change or fundamental configuration settings such as the
4459 filesystem layout, you need to work with a clean
4460 TMPDIR
4461 . Sharing
4462 TMPDIR
4463 under these circumstances might work but since it is not
4464 guaranteed, you should use a clean
4465 TMPDIR
4466 .
4467
4468- *Enable the Appropriate Package Architecture:* By default, the
4469 OpenEmbedded build system enables three levels of package
4470 architectures: "all", "tune" or "package", and "machine". Any given
4471 recipe usually selects one of these package architectures (types) for
4472 its output. Depending for what a given recipe creates packages,
4473 making sure you enable the appropriate package architecture can
4474 directly impact the build time.
4475
4476 A recipe that just generates scripts can enable "all" architecture
4477 because there are no binaries to build. To specifically enable "all"
4478 architecture, be sure your recipe inherits the
4479 :ref:`allarch <ref-classes-allarch>` class.
4480 This class is useful for "all" architectures because it configures
4481 many variables so packages can be used across multiple architectures.
4482
4483 If your recipe needs to generate packages that are machine-specific
4484 or when one of the build or runtime dependencies is already
4485 machine-architecture dependent, which makes your recipe also
4486 machine-architecture dependent, make sure your recipe enables the
4487 "machine" package architecture through the
4488 :term:`MACHINE_ARCH`
4489 variable:
4490 ::
4491
4492 PACKAGE_ARCH = "${MACHINE_ARCH}"
4493
4494 When you do not
4495 specifically enable a package architecture through the
4496 :term:`PACKAGE_ARCH`, The
4497 OpenEmbedded build system defaults to the
4498 :term:`TUNE_PKGARCH` setting:
4499 ::
4500
4501 PACKAGE_ARCH = "${TUNE_PKGARCH}"
4502
4503- *Choose a Generic Tuning File if Possible:* Some tunes are more
4504 generic and can run on multiple targets (e.g. an ``armv5`` set of
4505 packages could run on ``armv6`` and ``armv7`` processors in most
4506 cases). Similarly, ``i486`` binaries could work on ``i586`` and
4507 higher processors. You should realize, however, that advances on
4508 newer processor versions would not be used.
4509
4510 If you select the same tune for several different machines, the
4511 OpenEmbedded build system reuses software previously built, thus
4512 speeding up the overall build time. Realize that even though a new
4513 sysroot for each machine is generated, the software is not recompiled
4514 and only one package feed exists.
4515
4516- *Manage Granular Level Packaging:* Sometimes cases exist where
4517 injecting another level of package architecture beyond the three
4518 higher levels noted earlier can be useful. For example, consider how
4519 NXP (formerly Freescale) allows for the easy reuse of binary packages
4520 in their layer
4521 :yocto_git:`meta-freescale </cgit/cgit.cgi/meta-freescale/>`.
4522 In this example, the
4523 :yocto_git:`fsl-dynamic-packagearch </cgit/cgit.cgi/meta-freescale/tree/classes/fsl-dynamic-packagearch.bbclass>`
4524 class shares GPU packages for i.MX53 boards because all boards share
4525 the AMD GPU. The i.MX6-based boards can do the same because all
4526 boards share the Vivante GPU. This class inspects the BitBake
4527 datastore to identify if the package provides or depends on one of
4528 the sub-architecture values. If so, the class sets the
4529 :term:`PACKAGE_ARCH` value
4530 based on the ``MACHINE_SUBARCH`` value. If the package does not
4531 provide or depend on one of the sub-architecture values but it
4532 matches a value in the machine-specific filter, it sets
4533 :term:`MACHINE_ARCH`. This
4534 behavior reduces the number of packages built and saves build time by
4535 reusing binaries.
4536
4537- *Use Tools to Debug Issues:* Sometimes you can run into situations
4538 where software is being rebuilt when you think it should not be. For
4539 example, the OpenEmbedded build system might not be using shared
4540 state between machines when you think it should be. These types of
4541 situations are usually due to references to machine-specific
4542 variables such as :term:`MACHINE`,
4543 :term:`SERIAL_CONSOLES`,
4544 :term:`XSERVER`,
4545 :term:`MACHINE_FEATURES`,
4546 and so forth in code that is supposed to only be tune-specific or
4547 when the recipe depends
4548 (:term:`DEPENDS`,
4549 :term:`RDEPENDS`,
4550 :term:`RRECOMMENDS`,
4551 :term:`RSUGGESTS`, and so forth)
4552 on some other recipe that already has
4553 :term:`PACKAGE_ARCH` defined
4554 as "${MACHINE_ARCH}".
4555
4556 .. note::
4557
4558 Patches to fix any issues identified are most welcome as these
4559 issues occasionally do occur.
4560
4561 For such cases, you can use some tools to help you sort out the
4562 situation:
4563
4564 - *sstate-diff-machines.sh:* You can find this tool in the
4565 ``scripts`` directory of the Source Repositories. See the comments
4566 in the script for information on how to use the tool.
4567
4568 - *BitBake's "-S printdiff" Option:* Using this option causes
4569 BitBake to try to establish the closest signature match it can
4570 (e.g. in the shared state cache) and then run ``bitbake-diffsigs``
4571 over the matches to determine the stamps and delta where these two
4572 stamp trees diverge.
4573
4574Building Software from an External Source
4575-----------------------------------------
4576
4577By default, the OpenEmbedded build system uses the
4578:term:`Build Directory` when building source
4579code. The build process involves fetching the source files, unpacking
4580them, and then patching them if necessary before the build takes place.
4581
4582Situations exist where you might want to build software from source
4583files that are external to and thus outside of the OpenEmbedded build
4584system. For example, suppose you have a project that includes a new BSP
4585with a heavily customized kernel. And, you want to minimize exposing the
4586build system to the development team so that they can focus on their
4587project and maintain everyone's workflow as much as possible. In this
4588case, you want a kernel source directory on the development machine
4589where the development occurs. You want the recipe's
4590:term:`SRC_URI` variable to point to
4591the external directory and use it as is, not copy it.
4592
4593To build from software that comes from an external source, all you need
4594to do is inherit the
4595:ref:`externalsrc <ref-classes-externalsrc>` class
4596and then set the
4597:term:`EXTERNALSRC` variable to
4598point to your external source code. Here are the statements to put in
4599your ``local.conf`` file:
4600::
4601
4602 INHERIT += "externalsrc"
4603 EXTERNALSRC_pn-myrecipe = "path-to-your-source-tree"
4604
4605This next example shows how to accomplish the same thing by setting
4606``EXTERNALSRC`` in the recipe itself or in the recipe's append file:
4607::
4608
4609 EXTERNALSRC = "path"
4610 EXTERNALSRC_BUILD = "path"
4611
4612.. note::
4613
4614 In order for these settings to take effect, you must globally or
4615 locally inherit the
4616 externalsrc
4617 class.
4618
4619By default, ``externalsrc.bbclass`` builds the source code in a
4620directory separate from the external source directory as specified by
4621:term:`EXTERNALSRC`. If you need
4622to have the source built in the same directory in which it resides, or
4623some other nominated directory, you can set
4624:term:`EXTERNALSRC_BUILD`
4625to point to that directory:
4626::
4627
4628 EXTERNALSRC_BUILD_pn-myrecipe = "path-to-your-source-tree"
4629
4630Replicating a Build Offline
4631---------------------------
4632
4633It can be useful to take a "snapshot" of upstream sources used in a
4634build and then use that "snapshot" later to replicate the build offline.
4635To do so, you need to first prepare and populate your downloads
4636directory your "snapshot" of files. Once your downloads directory is
4637ready, you can use it at any time and from any machine to replicate your
4638build.
4639
4640Follow these steps to populate your Downloads directory:
4641
46421. *Create a Clean Downloads Directory:* Start with an empty downloads
4643 directory (:term:`DL_DIR`). You
4644 start with an empty downloads directory by either removing the files
4645 in the existing directory or by setting ``DL_DIR`` to point to either
4646 an empty location or one that does not yet exist.
4647
46482. *Generate Tarballs of the Source Git Repositories:* Edit your
4649 ``local.conf`` configuration file as follows:
4650 ::
4651
4652 DL_DIR = "/home/your-download-dir/"
4653 BB_GENERATE_MIRROR_TARBALLS = "1"
4654
4655 During
4656 the fetch process in the next step, BitBake gathers the source files
4657 and creates tarballs in the directory pointed to by ``DL_DIR``. See
4658 the
4659 :term:`BB_GENERATE_MIRROR_TARBALLS`
4660 variable for more information.
4661
46623. *Populate Your Downloads Directory Without Building:* Use BitBake to
4663 fetch your sources but inhibit the build:
4664 ::
4665
4666 $ bitbake target --runonly=fetch
4667
4668 The downloads directory (i.e. ``${DL_DIR}``) now has
4669 a "snapshot" of the source files in the form of tarballs, which can
4670 be used for the build.
4671
46724. *Optionally Remove Any Git or other SCM Subdirectories From the
4673 Downloads Directory:* If you want, you can clean up your downloads
4674 directory by removing any Git or other Source Control Management
4675 (SCM) subdirectories such as ``${DL_DIR}/git2/*``. The tarballs
4676 already contain these subdirectories.
4677
4678Once your downloads directory has everything it needs regarding source
4679files, you can create your "own-mirror" and build your target.
4680Understand that you can use the files to build the target offline from
4681any machine and at any time.
4682
4683Follow these steps to build your target using the files in the downloads
4684directory:
4685
46861. *Using Local Files Only:* Inside your ``local.conf`` file, add the
4687 :term:`SOURCE_MIRROR_URL`
4688 variable, inherit the
4689 :ref:`own-mirrors <ref-classes-own-mirrors>`
4690 class, and use the
4691 :term:`bitbake:BB_NO_NETWORK`
4692 variable to your ``local.conf``.
4693 ::
4694
4695 SOURCE_MIRROR_URL ?= "file:///home/your-download-dir/"
4696 INHERIT += "own-mirrors"
4697 BB_NO_NETWORK = "1"
4698
4699 The ``SOURCE_MIRROR_URL`` and ``own-mirror``
4700 class set up the system to use the downloads directory as your "own
4701 mirror". Using the ``BB_NO_NETWORK`` variable makes sure that
4702 BitBake's fetching process in step 3 stays local, which means files
4703 from your "own-mirror" are used.
4704
47052. *Start With a Clean Build:* You can start with a clean build by
4706 removing the
4707 ``${``\ :term:`TMPDIR`\ ``}``
4708 directory or using a new :term:`Build Directory`.
4709
47103. *Build Your Target:* Use BitBake to build your target:
4711 ::
4712
4713 $ bitbake target
4714
4715 The build completes using the known local "snapshot" of source
4716 files from your mirror. The resulting tarballs for your "snapshot" of
4717 source files are in the downloads directory.
4718
4719 .. note::
4720
4721 The offline build does not work if recipes attempt to find the
4722 latest version of software by setting
4723 :term:`SRCREV` to
4724 ``${``\ :term:`AUTOREV`\ ``}``:
4725 SRCREV = "${AUTOREV}" When a recipe sets ``SRCREV`` to
4726 ``${AUTOREV}``, the build system accesses the network in an
4727 attempt to determine the latest version of software from the SCM.
4728 Typically, recipes that use ``AUTOREV`` are custom or modified
4729 recipes. Recipes that reside in public repositories usually do not
4730 use ``AUTOREV``.
4731
4732 If you do have recipes that use ``AUTOREV``, you can take steps to
4733 still use the recipes in an offline build. Do the following:
4734
4735 1. Use a configuration generated by enabling `build
4736 history <#maintaining-build-output-quality>`__.
4737
4738 2. Use the ``buildhistory-collect-srcrevs`` command to collect the
4739 stored ``SRCREV`` values from the build's history. For more
4740 information on collecting these values, see the "`Build History
4741 Package Information <#build-history-package-information>`__"
4742 section.
4743
4744 3. Once you have the correct source revisions, you can modify
4745 those recipes to to set ``SRCREV`` to specific versions of the
4746 software.
4747
4748Speeding Up a Build
4749===================
4750
4751Build time can be an issue. By default, the build system uses simple
4752controls to try and maximize build efficiency. In general, the default
4753settings for all the following variables result in the most efficient
4754build times when dealing with single socket systems (i.e. a single CPU).
4755If you have multiple CPUs, you might try increasing the default values
4756to gain more speed. See the descriptions in the glossary for each
4757variable for more information:
4758
4759- :term:`BB_NUMBER_THREADS`:
4760 The maximum number of threads BitBake simultaneously executes.
4761
4762- :term:`bitbake:BB_NUMBER_PARSE_THREADS`:
4763 The number of threads BitBake uses during parsing.
4764
4765- :term:`PARALLEL_MAKE`: Extra
4766 options passed to the ``make`` command during the
4767 :ref:`ref-tasks-compile` task in
4768 order to specify parallel compilation on the local build host.
4769
4770- :term:`PARALLEL_MAKEINST`:
4771 Extra options passed to the ``make`` command during the
4772 :ref:`ref-tasks-install` task in
4773 order to specify parallel installation on the local build host.
4774
4775As mentioned, these variables all scale to the number of processor cores
4776available on the build system. For single socket systems, this
4777auto-scaling ensures that the build system fundamentally takes advantage
4778of potential parallel operations during the build based on the build
4779machine's capabilities.
4780
4781Following are additional factors that can affect build speed:
4782
4783- File system type: The file system type that the build is being
4784 performed on can also influence performance. Using ``ext4`` is
4785 recommended as compared to ``ext2`` and ``ext3`` due to ``ext4``
4786 improved features such as extents.
4787
4788- Disabling the updating of access time using ``noatime``: The
4789 ``noatime`` mount option prevents the build system from updating file
4790 and directory access times.
4791
4792- Setting a longer commit: Using the "commit=" mount option increases
4793 the interval in seconds between disk cache writes. Changing this
4794 interval from the five second default to something longer increases
4795 the risk of data loss but decreases the need to write to the disk,
4796 thus increasing the build performance.
4797
4798- Choosing the packaging backend: Of the available packaging backends,
4799 IPK is the fastest. Additionally, selecting a singular packaging
4800 backend also helps.
4801
4802- Using ``tmpfs`` for :term:`TMPDIR`
4803 as a temporary file system: While this can help speed up the build,
4804 the benefits are limited due to the compiler using ``-pipe``. The
4805 build system goes to some lengths to avoid ``sync()`` calls into the
4806 file system on the principle that if there was a significant failure,
4807 the :term:`Build Directory`
4808 contents could easily be rebuilt.
4809
4810- Inheriting the
4811 :ref:`rm_work <ref-classes-rm-work>` class:
4812 Inheriting this class has shown to speed up builds due to
4813 significantly lower amounts of data stored in the data cache as well
4814 as on disk. Inheriting this class also makes cleanup of
4815 :term:`TMPDIR` faster, at the
4816 expense of being easily able to dive into the source code. File
4817 system maintainers have recommended that the fastest way to clean up
4818 large numbers of files is to reformat partitions rather than delete
4819 files due to the linear nature of partitions. This, of course,
4820 assumes you structure the disk partitions and file systems in a way
4821 that this is practical.
4822
4823Aside from the previous list, you should keep some trade offs in mind
4824that can help you speed up the build:
4825
4826- Remove items from
4827 :term:`DISTRO_FEATURES`
4828 that you might not need.
4829
4830- Exclude debug symbols and other debug information: If you do not need
4831 these symbols and other debug information, disabling the ``*-dbg``
4832 package generation can speed up the build. You can disable this
4833 generation by setting the
4834 :term:`INHIBIT_PACKAGE_DEBUG_SPLIT`
4835 variable to "1".
4836
4837- Disable static library generation for recipes derived from
4838 ``autoconf`` or ``libtool``: Following is an example showing how to
4839 disable static libraries and still provide an override to handle
4840 exceptions:
4841 ::
4842
4843 STATICLIBCONF = "--disable-static"
4844 STATICLIBCONF_sqlite3-native = ""
4845 EXTRA_OECONF += "${STATICLIBCONF}"
4846
4847 .. note::
4848
4849 - Some recipes need static libraries in order to work correctly
4850 (e.g. ``pseudo-native`` needs ``sqlite3-native``). Overrides,
4851 as in the previous example, account for these kinds of
4852 exceptions.
4853
4854 - Some packages have packaging code that assumes the presence of
4855 the static libraries. If so, you might need to exclude them as
4856 well.
4857
4858.. _platdev-working-with-libraries:
4859
4860Working With Libraries
4861======================
4862
4863Libraries are an integral part of your system. This section describes
4864some common practices you might find helpful when working with libraries
4865to build your system:
4866
4867- `How to include static library
4868 files <#including-static-library-files>`__
4869
4870- `How to use the Multilib feature to combine multiple versions of
4871 library files into a single
4872 image <#combining-multiple-versions-library-files-into-one-image>`__
4873
4874- `How to install multiple versions of the same library in parallel on
4875 the same
4876 system <#installing-multiple-versions-of-the-same-library>`__
4877
4878Including Static Library Files
4879------------------------------
4880
4881If you are building a library and the library offers static linking, you
4882can control which static library files (``*.a`` files) get included in
4883the built library.
4884
4885The :term:`PACKAGES` and
4886:term:`FILES_* <FILES>` variables in the
4887``meta/conf/bitbake.conf`` configuration file define how files installed
4888by the ``do_install`` task are packaged. By default, the ``PACKAGES``
4889variable includes ``${PN}-staticdev``, which represents all static
4890library files.
4891
4892.. note::
4893
4894 Some previously released versions of the Yocto Project defined the
4895 static library files through
4896 ${PN}-dev
4897 .
4898
4899Following is part of the BitBake configuration file, where you can see
4900how the static library files are defined:
4901::
4902
4903 PACKAGE_BEFORE_PN ?= ""
4904 PACKAGES = "${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc ${PN}-locale ${PACKAGE_BEFORE_PN} ${PN}"
4905 PACKAGES_DYNAMIC = "^${PN}-locale-.*"
4906 FILES = ""
4907
4908 FILES_${PN} = "${bindir}/* ${sbindir}/* ${libexecdir}/* ${libdir}/lib*${SOLIBS} \
4909 ${sysconfdir} ${sharedstatedir} ${localstatedir} \
4910 ${base_bindir}/* ${base_sbindir}/* \
4911 ${base_libdir}/*${SOLIBS} \
4912 ${base_prefix}/lib/udev/rules.d ${prefix}/lib/udev/rules.d \
4913 ${datadir}/${BPN} ${libdir}/${BPN}/* \
4914 ${datadir}/pixmaps ${datadir}/applications \
4915 ${datadir}/idl ${datadir}/omf ${datadir}/sounds \
4916 ${libdir}/bonobo/servers"
4917
4918 FILES_${PN}-bin = "${bindir}/* ${sbindir}/*"
4919
4920 FILES_${PN}-doc = "${docdir} ${mandir} ${infodir} ${datadir}/gtk-doc \
4921 ${datadir}/gnome/help"
4922 SECTION_${PN}-doc = "doc"
4923
4924 FILES_SOLIBSDEV ?= "${base_libdir}/lib*${SOLIBSDEV} ${libdir}/lib*${SOLIBSDEV}"
4925 FILES_${PN}-dev = "${includedir} ${FILES_SOLIBSDEV} ${libdir}/*.la \
4926 ${libdir}/*.o ${libdir}/pkgconfig ${datadir}/pkgconfig \
4927 ${datadir}/aclocal ${base_libdir}/*.o \
4928 ${libdir}/${BPN}/*.la ${base_libdir}/*.la"
4929 SECTION_${PN}-dev = "devel"
4930 ALLOW_EMPTY_${PN}-dev = "1"
4931 RDEPENDS_${PN}-dev = "${PN} (= ${EXTENDPKGV})"
4932
4933 FILES_${PN}-staticdev = "${libdir}/*.a ${base_libdir}/*.a ${libdir}/${BPN}/*.a"
4934 SECTION_${PN}-staticdev = "devel"
4935 RDEPENDS_${PN}-staticdev = "${PN}-dev (= ${EXTENDPKGV})"
4936
4937.. _combining-multiple-versions-library-files-into-one-image:
4938
4939Combining Multiple Versions of Library Files into One Image
4940-----------------------------------------------------------
4941
4942The build system offers the ability to build libraries with different
4943target optimizations or architecture formats and combine these together
4944into one system image. You can link different binaries in the image
4945against the different libraries as needed for specific use cases. This
4946feature is called "Multilib."
4947
4948An example would be where you have most of a system compiled in 32-bit
4949mode using 32-bit libraries, but you have something large, like a
4950database engine, that needs to be a 64-bit application and uses 64-bit
4951libraries. Multilib allows you to get the best of both 32-bit and 64-bit
4952libraries.
4953
4954While the Multilib feature is most commonly used for 32 and 64-bit
4955differences, the approach the build system uses facilitates different
4956target optimizations. You could compile some binaries to use one set of
4957libraries and other binaries to use a different set of libraries. The
4958libraries could differ in architecture, compiler options, or other
4959optimizations.
4960
4961Several examples exist in the ``meta-skeleton`` layer found in the
4962:term:`Source Directory`:
4963
4964- ``conf/multilib-example.conf`` configuration file
4965
4966- ``conf/multilib-example2.conf`` configuration file
4967
4968- ``recipes-multilib/images/core-image-multilib-example.bb`` recipe
4969
4970Preparing to Use Multilib
4971~~~~~~~~~~~~~~~~~~~~~~~~~
4972
4973User-specific requirements drive the Multilib feature. Consequently,
4974there is no one "out-of-the-box" configuration that likely exists to
4975meet your needs.
4976
4977In order to enable Multilib, you first need to ensure your recipe is
4978extended to support multiple libraries. Many standard recipes are
4979already extended and support multiple libraries. You can check in the
4980``meta/conf/multilib.conf`` configuration file in the
4981:term:`Source Directory` to see how this is
4982done using the
4983:term:`BBCLASSEXTEND` variable.
4984Eventually, all recipes will be covered and this list will not be
4985needed.
4986
4987For the most part, the Multilib class extension works automatically to
4988extend the package name from ``${PN}`` to ``${MLPREFIX}${PN}``, where
4989``MLPREFIX`` is the particular multilib (e.g. "lib32-" or "lib64-").
4990Standard variables such as
4991:term:`DEPENDS`,
4992:term:`RDEPENDS`,
4993:term:`RPROVIDES`,
4994:term:`RRECOMMENDS`,
4995:term:`PACKAGES`, and
4996:term:`PACKAGES_DYNAMIC` are
4997automatically extended by the system. If you are extending any manual
4998code in the recipe, you can use the ``${MLPREFIX}`` variable to ensure
4999those names are extended correctly. This automatic extension code
5000resides in ``multilib.bbclass``.
5001
5002Using Multilib
5003~~~~~~~~~~~~~~
5004
5005After you have set up the recipes, you need to define the actual
5006combination of multiple libraries you want to build. You accomplish this
5007through your ``local.conf`` configuration file in the
5008:term:`Build Directory`. An example
5009configuration would be as follows:
5010::
5011
5012 MACHINE = "qemux86-64"
5013 require conf/multilib.conf
5014 MULTILIBS = "multilib:lib32"
5015 DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
5016 IMAGE_INSTALL_append = "lib32-glib-2.0"
5017
5018This example enables an additional library named
5019``lib32`` alongside the normal target packages. When combining these
5020"lib32" alternatives, the example uses "x86" for tuning. For information
5021on this particular tuning, see
5022``meta/conf/machine/include/ia32/arch-ia32.inc``.
5023
5024The example then includes ``lib32-glib-2.0`` in all the images, which
5025illustrates one method of including a multiple library dependency. You
5026can use a normal image build to include this dependency, for example:
5027::
5028
5029 $ bitbake core-image-sato
5030
5031You can also build Multilib packages
5032specifically with a command like this:
5033::
5034
5035 $ bitbake lib32-glib-2.0
5036
5037Additional Implementation Details
5038~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5039
5040Generic implementation details as well as details that are specific to
5041package management systems exist. Following are implementation details
5042that exist regardless of the package management system:
5043
5044- The typical convention used for the class extension code as used by
5045 Multilib assumes that all package names specified in
5046 :term:`PACKAGES` that contain
5047 ``${PN}`` have ``${PN}`` at the start of the name. When that
5048 convention is not followed and ``${PN}`` appears at the middle or the
5049 end of a name, problems occur.
5050
5051- The :term:`TARGET_VENDOR`
5052 value under Multilib will be extended to "-vendormlmultilib" (e.g.
5053 "-pokymllib32" for a "lib32" Multilib with Poky). The reason for this
5054 slightly unwieldy contraction is that any "-" characters in the
5055 vendor string presently break Autoconf's ``config.sub``, and other
5056 separators are problematic for different reasons.
5057
5058For the RPM Package Management System, the following implementation
5059details exist:
5060
5061- A unique architecture is defined for the Multilib packages, along
5062 with creating a unique deploy folder under ``tmp/deploy/rpm`` in the
5063 :term:`Build Directory`. For
5064 example, consider ``lib32`` in a ``qemux86-64`` image. The possible
5065 architectures in the system are "all", "qemux86_64",
5066 "lib32_qemux86_64", and "lib32_x86".
5067
5068- The ``${MLPREFIX}`` variable is stripped from ``${PN}`` during RPM
5069 packaging. The naming for a normal RPM package and a Multilib RPM
5070 package in a ``qemux86-64`` system resolves to something similar to
5071 ``bash-4.1-r2.x86_64.rpm`` and ``bash-4.1.r2.lib32_x86.rpm``,
5072 respectively.
5073
5074- When installing a Multilib image, the RPM backend first installs the
5075 base image and then installs the Multilib libraries.
5076
5077- The build system relies on RPM to resolve the identical files in the
5078 two (or more) Multilib packages.
5079
5080For the IPK Package Management System, the following implementation
5081details exist:
5082
5083- The ``${MLPREFIX}`` is not stripped from ``${PN}`` during IPK
5084 packaging. The naming for a normal RPM package and a Multilib IPK
5085 package in a ``qemux86-64`` system resolves to something like
5086 ``bash_4.1-r2.x86_64.ipk`` and ``lib32-bash_4.1-rw_x86.ipk``,
5087 respectively.
5088
5089- The IPK deploy folder is not modified with ``${MLPREFIX}`` because
5090 packages with and without the Multilib feature can exist in the same
5091 folder due to the ``${PN}`` differences.
5092
5093- IPK defines a sanity check for Multilib installation using certain
5094 rules for file comparison, overridden, etc.
5095
5096Installing Multiple Versions of the Same Library
5097------------------------------------------------
5098
5099Situations can exist where you need to install and use multiple versions
5100of the same library on the same system at the same time. These
5101situations almost always exist when a library API changes and you have
5102multiple pieces of software that depend on the separate versions of the
5103library. To accommodate these situations, you can install multiple
5104versions of the same library in parallel on the same system.
5105
5106The process is straightforward as long as the libraries use proper
5107versioning. With properly versioned libraries, all you need to do to
5108individually specify the libraries is create separate, appropriately
5109named recipes where the :term:`PN` part of
5110the name includes a portion that differentiates each library version
5111(e.g.the major part of the version number). Thus, instead of having a
5112single recipe that loads one version of a library (e.g. ``clutter``),
5113you provide multiple recipes that result in different versions of the
5114libraries you want. As an example, the following two recipes would allow
5115the two separate versions of the ``clutter`` library to co-exist on the
5116same system:
5117::
5118
5119 clutter-1.6_1.6.20.bb
5120 clutter-1.8_1.8.4.bb
5121
5122Additionally, if
5123you have other recipes that depend on a given library, you need to use
5124the :term:`DEPENDS` variable to
5125create the dependency. Continuing with the same example, if you want to
5126have a recipe depend on the 1.8 version of the ``clutter`` library, use
5127the following in your recipe:
5128::
5129
5130 DEPENDS = "clutter-1.8"
5131
5132Using x32 psABI
5133===============
5134
5135x32 processor-specific Application Binary Interface (`x32
5136psABI <https://software.intel.com/en-us/node/628948>`__) is a native
513732-bit processor-specific ABI for Intel 64 (x86-64) architectures. An
5138ABI defines the calling conventions between functions in a processing
5139environment. The interface determines what registers are used and what
5140the sizes are for various C data types.
5141
5142Some processing environments prefer using 32-bit applications even when
5143running on Intel 64-bit platforms. Consider the i386 psABI, which is a
5144very old 32-bit ABI for Intel 64-bit platforms. The i386 psABI does not
5145provide efficient use and access of the Intel 64-bit processor
5146resources, leaving the system underutilized. Now consider the x86_64
5147psABI. This ABI is newer and uses 64-bits for data sizes and program
5148pointers. The extra bits increase the footprint size of the programs,
5149libraries, and also increases the memory and file system size
5150requirements. Executing under the x32 psABI enables user programs to
5151utilize CPU and system resources more efficiently while keeping the
5152memory footprint of the applications low. Extra bits are used for
5153registers but not for addressing mechanisms.
5154
5155The Yocto Project supports the final specifications of x32 psABI as
5156follows:
5157
5158- You can create packages and images in x32 psABI format on x86_64
5159 architecture targets.
5160
5161- You can successfully build recipes with the x32 toolchain.
5162
5163- You can create and boot ``core-image-minimal`` and
5164 ``core-image-sato`` images.
5165
5166- RPM Package Manager (RPM) support exists for x32 binaries.
5167
5168- Support for large images exists.
5169
5170To use the x32 psABI, you need to edit your ``conf/local.conf``
5171configuration file as follows:
5172::
5173
5174 MACHINE = "qemux86-64"
5175 DEFAULTTUNE = "x86-64-x32"
5176 baselib = "${@d.getVar('BASE_LIB_tune-' + (d.getVar('DEFAULTTUNE') \
5177 or 'INVALID')) or 'lib'}"
5178
5179Once you have set
5180up your configuration file, use BitBake to build an image that supports
5181the x32 psABI. Here is an example:
5182::
5183
5184 $ bitbake core-image-sato
5185
5186Enabling GObject Introspection Support
5187======================================
5188
5189`GObject
5190introspection <https://wiki.gnome.org/Projects/GObjectIntrospection>`__
5191is the standard mechanism for accessing GObject-based software from
5192runtime environments. GObject is a feature of the GLib library that
5193provides an object framework for the GNOME desktop and related software.
5194GObject Introspection adds information to GObject that allows objects
5195created within it to be represented across different programming
5196languages. If you want to construct GStreamer pipelines using Python, or
5197control UPnP infrastructure using Javascript and GUPnP, GObject
5198introspection is the only way to do it.
5199
5200This section describes the Yocto Project support for generating and
5201packaging GObject introspection data. GObject introspection data is a
5202description of the API provided by libraries built on top of GLib
5203framework, and, in particular, that framework's GObject mechanism.
5204GObject Introspection Repository (GIR) files go to ``-dev`` packages,
5205``typelib`` files go to main packages as they are packaged together with
5206libraries that are introspected.
5207
5208The data is generated when building such a library, by linking the
5209library with a small executable binary that asks the library to describe
5210itself, and then executing the binary and processing its output.
5211
5212Generating this data in a cross-compilation environment is difficult
5213because the library is produced for the target architecture, but its
5214code needs to be executed on the build host. This problem is solved with
5215the OpenEmbedded build system by running the code through QEMU, which
5216allows precisely that. Unfortunately, QEMU does not always work
5217perfectly as mentioned in the "`Known Issues <#known-issues>`__"
5218section.
5219
5220Enabling the Generation of Introspection Data
5221---------------------------------------------
5222
5223Enabling the generation of introspection data (GIR files) in your
5224library package involves the following:
5225
52261. Inherit the
5227 :ref:`gobject-introspection <ref-classes-gobject-introspection>`
5228 class.
5229
52302. Make sure introspection is not disabled anywhere in the recipe or
5231 from anything the recipe includes. Also, make sure that
5232 "gobject-introspection-data" is not in
5233 :term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`
5234 and that "qemu-usermode" is not in
5235 :term:`MACHINE_FEATURES_BACKFILL_CONSIDERED`.
5236 If either of these conditions exist, nothing will happen.
5237
52383. Try to build the recipe. If you encounter build errors that look like
5239 something is unable to find ``.so`` libraries, check where these
5240 libraries are located in the source tree and add the following to the
5241 recipe:
5242 ::
5243
5244 GIR_EXTRA_LIBS_PATH = "${B}/something/.libs"
5245
5246 .. note::
5247
5248 See recipes in the
5249 oe-core
5250 repository that use that
5251 GIR_EXTRA_LIBS_PATH
5252 variable as an example.
5253
52544. Look for any other errors, which probably mean that introspection
5255 support in a package is not entirely standard, and thus breaks down
5256 in a cross-compilation environment. For such cases, custom-made fixes
5257 are needed. A good place to ask and receive help in these cases is
5258 the :ref:`Yocto Project mailing
5259 lists <resources-mailinglist>`.
5260
5261.. note::
5262
5263 Using a library that no longer builds against the latest Yocto
5264 Project release and prints introspection related errors is a good
5265 candidate for the previous procedure.
5266
5267Disabling the Generation of Introspection Data
5268----------------------------------------------
5269
5270You might find that you do not want to generate introspection data. Or,
5271perhaps QEMU does not work on your build host and target architecture
5272combination. If so, you can use either of the following methods to
5273disable GIR file generations:
5274
5275- Add the following to your distro configuration:
5276 ::
5277
5278 DISTRO_FEATURES_BACKFILL_CONSIDERED = "gobject-introspection-data"
5279
5280 Adding this statement disables generating introspection data using
5281 QEMU but will still enable building introspection tools and libraries
5282 (i.e. building them does not require the use of QEMU).
5283
5284- Add the following to your machine configuration:
5285 ::
5286
5287 MACHINE_FEATURES_BACKFILL_CONSIDERED = "qemu-usermode"
5288
5289 Adding this statement disables the use of QEMU when building packages for your
5290 machine. Currently, this feature is used only by introspection
5291 recipes and has the same effect as the previously described option.
5292
5293 .. note::
5294
5295 Future releases of the Yocto Project might have other features
5296 affected by this option.
5297
5298If you disable introspection data, you can still obtain it through other
5299means such as copying the data from a suitable sysroot, or by generating
5300it on the target hardware. The OpenEmbedded build system does not
5301currently provide specific support for these techniques.
5302
5303Testing that Introspection Works in an Image
5304--------------------------------------------
5305
5306Use the following procedure to test if generating introspection data is
5307working in an image:
5308
53091. Make sure that "gobject-introspection-data" is not in
5310 :term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`
5311 and that "qemu-usermode" is not in
5312 :term:`MACHINE_FEATURES_BACKFILL_CONSIDERED`.
5313
53142. Build ``core-image-sato``.
5315
53163. Launch a Terminal and then start Python in the terminal.
5317
53184. Enter the following in the terminal:
5319 ::
5320
5321 >>> from gi.repository import GLib
5322 >>> GLib.get_host_name()
5323
53245. For something a little more advanced, enter the following see:
5325 http://python-gtk-3-tutorial.readthedocs.org/en/latest/introduction.html
5326
5327Known Issues
5328------------
5329
5330The following know issues exist for GObject Introspection Support:
5331
5332- ``qemu-ppc64`` immediately crashes. Consequently, you cannot build
5333 introspection data on that architecture.
5334
5335- x32 is not supported by QEMU. Consequently, introspection data is
5336 disabled.
5337
5338- musl causes transient GLib binaries to crash on assertion failures.
5339 Consequently, generating introspection data is disabled.
5340
5341- Because QEMU is not able to run the binaries correctly, introspection
5342 is disabled for some specific packages under specific architectures
5343 (e.g. ``gcr``, ``libsecret``, and ``webkit``).
5344
5345- QEMU usermode might not work properly when running 64-bit binaries
5346 under 32-bit host machines. In particular, "qemumips64" is known to
5347 not work under i686.
5348
5349.. _dev-optionally-using-an-external-toolchain:
5350
5351Optionally Using an External Toolchain
5352======================================
5353
5354You might want to use an external toolchain as part of your development.
5355If this is the case, the fundamental steps you need to accomplish are as
5356follows:
5357
5358- Understand where the installed toolchain resides. For cases where you
5359 need to build the external toolchain, you would need to take separate
5360 steps to build and install the toolchain.
5361
5362- Make sure you add the layer that contains the toolchain to your
5363 ``bblayers.conf`` file through the
5364 :term:`BBLAYERS` variable.
5365
5366- Set the ``EXTERNAL_TOOLCHAIN`` variable in your ``local.conf`` file
5367 to the location in which you installed the toolchain.
5368
5369A good example of an external toolchain used with the Yocto Project is
5370Mentor Graphics Sourcery G++ Toolchain. You can see information on how
5371to use that particular layer in the ``README`` file at
5372http://github.com/MentorEmbedded/meta-sourcery/. You can find
5373further information by reading about the
5374:term:`TCMODE` variable in the Yocto
5375Project Reference Manual's variable glossary.
5376
5377Creating Partitioned Images Using Wic
5378=====================================
5379
5380Creating an image for a particular hardware target using the
5381OpenEmbedded build system does not necessarily mean you can boot that
5382image as is on your device. Physical devices accept and boot images in
5383various ways depending on the specifics of the device. Usually,
5384information about the hardware can tell you what image format the device
5385requires. Should your device require multiple partitions on an SD card,
5386flash, or an HDD, you can use the OpenEmbedded Image Creator, Wic, to
5387create the properly partitioned image.
5388
5389The ``wic`` command generates partitioned images from existing
5390OpenEmbedded build artifacts. Image generation is driven by partitioning
5391commands contained in an Openembedded kickstart file (``.wks``)
5392specified either directly on the command line or as one of a selection
5393of canned kickstart files as shown with the ``wic list images`` command
5394in the "`Using an Existing Kickstart
5395File <#using-a-provided-kickstart-file>`__" section. When you apply the
5396command to a given set of build artifacts, the result is an image or set
5397of images that can be directly written onto media and used on a
5398particular system.
5399
5400.. note::
5401
5402 For a kickstart file reference, see the "
5403 OpenEmbedded Kickstart (
5404 .wks
5405 ) Reference
5406 " Chapter in the Yocto Project Reference Manual.
5407
5408The ``wic`` command and the infrastructure it is based on is by
5409definition incomplete. The purpose of the command is to allow the
5410generation of customized images, and as such, was designed to be
5411completely extensible through a plugin interface. See the "`Using the
5412Wic PlugIn Interface <#wic-using-the-wic-plugin-interface>`__" section
5413for information on these plugins.
5414
5415This section provides some background information on Wic, describes what
5416you need to have in place to run the tool, provides instruction on how
5417to use the Wic utility, provides information on using the Wic plugins
5418interface, and provides several examples that show how to use Wic.
5419
5420.. _wic-background:
5421
5422Background
5423----------
5424
5425This section provides some background on the Wic utility. While none of
5426this information is required to use Wic, you might find it interesting.
5427
5428- The name "Wic" is derived from OpenEmbedded Image Creator (oeic). The
5429 "oe" diphthong in "oeic" was promoted to the letter "w", because
5430 "oeic" is both difficult to remember and to pronounce.
5431
5432- Wic is loosely based on the Meego Image Creator (``mic``) framework.
5433 The Wic implementation has been heavily modified to make direct use
5434 of OpenEmbedded build artifacts instead of package installation and
5435 configuration, which are already incorporated within the OpenEmbedded
5436 artifacts.
5437
5438- Wic is a completely independent standalone utility that initially
5439 provides easier-to-use and more flexible replacements for an existing
5440 functionality in OE-Core's
5441 :ref:`image-live <ref-classes-image-live>`
5442 class. The difference between Wic and those examples is that with Wic
5443 the functionality of those scripts is implemented by a
5444 general-purpose partitioning language, which is based on Redhat
5445 kickstart syntax.
5446
5447.. _wic-requirements:
5448
5449Requirements
5450------------
5451
5452In order to use the Wic utility with the OpenEmbedded Build system, your
5453system needs to meet the following requirements:
5454
5455- The Linux distribution on your development host must support the
5456 Yocto Project. See the ":ref:`detailed-supported-distros`"
5457 section in the Yocto Project Reference Manual for the list of
5458 distributions that support the Yocto Project.
5459
5460- The standard system utilities, such as ``cp``, must be installed on
5461 your development host system.
5462
5463- You must have sourced the build environment setup script (i.e.
5464 :ref:`structure-core-script`) found in the
5465 :term:`Build Directory`.
5466
5467- You need to have the build artifacts already available, which
5468 typically means that you must have already created an image using the
5469 Openembedded build system (e.g. ``core-image-minimal``). While it
5470 might seem redundant to generate an image in order to create an image
5471 using Wic, the current version of Wic requires the artifacts in the
5472 form generated by the OpenEmbedded build system.
5473
5474- You must build several native tools, which are built to run on the
5475 build system: $ bitbake parted-native dosfstools-native mtools-native
5476
5477- Include "wic" as part of the
5478 :term:`IMAGE_FSTYPES`
5479 variable.
5480
5481- Include the name of the :ref:`wic kickstart file <openembedded-kickstart-wks-reference>`
5482 as part of the :term:`WKS_FILE` variable
5483
5484.. _wic-getting-help:
5485
5486Getting Help
5487------------
5488
5489You can get general help for the ``wic`` command by entering the ``wic``
5490command by itself or by entering the command with a help argument as
5491follows:
5492::
5493
5494 $ wic -h
5495 $ wic --help
5496 $ wic help
5497
5498Currently, Wic supports seven commands: ``cp``, ``create``, ``help``,
5499``list``, ``ls``, ``rm``, and ``write``. You can get help for all these
5500commands except "help" by using the following form:
5501::
5502
5503 $ wic help command
5504
5505For example, the following command returns help for the ``write``
5506command:
5507::
5508
5509 $ wic help write
5510
5511Wic supports help for three topics: ``overview``, ``plugins``, and
5512``kickstart``. You can get help for any topic using the following form:
5513::
5514
5515 $ wic help topic
5516
5517For example, the following returns overview help for Wic:
5518::
5519
5520 $ wic help overview
5521
5522One additional level of help exists for Wic. You can get help on
5523individual images through the ``list`` command. You can use the ``list``
5524command to return the available Wic images as follows:
5525::
5526
5527 $ wic list images
5528 genericx86 Create an EFI disk image for genericx86*
5529 beaglebone-yocto Create SD card image for Beaglebone
5530 edgerouter Create SD card image for Edgerouter
5531 qemux86-directdisk Create a qemu machine 'pcbios' direct disk image
5532 directdisk-gpt Create a 'pcbios' direct disk image
5533 mkefidisk Create an EFI disk image
5534 directdisk Create a 'pcbios' direct disk image
5535 systemd-bootdisk Create an EFI disk image with systemd-boot
5536 mkhybridiso Create a hybrid ISO image
5537 sdimage-bootpart Create SD card image with a boot partition
5538 directdisk-multi-rootfs Create multi rootfs image using rootfs plugin
5539 directdisk-bootloader-config Create a 'pcbios' direct disk image with custom bootloader config
5540
5541Once you know the list of available
5542Wic images, you can use ``help`` with the command to get help on a
5543particular image. For example, the following command returns help on the
5544"beaglebone-yocto" image:
5545::
5546
5547 $ wic list beaglebone-yocto help
5548
5549 Creates a partitioned SD card image for Beaglebone.
5550 Boot files are located in the first vfat partition.
5551
5552Operational Modes
5553-----------------
5554
5555You can use Wic in two different modes, depending on how much control
5556you need for specifying the Openembedded build artifacts that are used
5557for creating the image: Raw and Cooked:
5558
5559- *Raw Mode:* You explicitly specify build artifacts through Wic
5560 command-line arguments.
5561
5562- *Cooked Mode:* The current
5563 :term:`MACHINE` setting and image
5564 name are used to automatically locate and provide the build
5565 artifacts. You just supply a kickstart file and the name of the image
5566 from which to use artifacts.
5567
5568Regardless of the mode you use, you need to have the build artifacts
5569ready and available.
5570
5571Raw Mode
5572~~~~~~~~
5573
5574Running Wic in raw mode allows you to specify all the partitions through
5575the ``wic`` command line. The primary use for raw mode is if you have
5576built your kernel outside of the Yocto Project
5577:term:`Build Directory`. In other words, you
5578can point to arbitrary kernel, root filesystem locations, and so forth.
5579Contrast this behavior with cooked mode where Wic looks in the Build
5580Directory (e.g. ``tmp/deploy/images/``\ machine).
5581
5582The general form of the ``wic`` command in raw mode is:
5583::
5584
5585 $ wic create wks_file options ...
5586
5587 Where:
5588
5589 wks_file:
5590 An OpenEmbedded kickstart file. You can provide
5591 your own custom file or use a file from a set of
5592 existing files as described by further options.
5593
5594 optional arguments:
5595 -h, --help show this help message and exit
5596 -o OUTDIR, --outdir OUTDIR
5597 name of directory to create image in
5598 -e IMAGE_NAME, --image-name IMAGE_NAME
5599 name of the image to use the artifacts from e.g. core-
5600 image-sato
5601 -r ROOTFS_DIR, --rootfs-dir ROOTFS_DIR
5602 path to the /rootfs dir to use as the .wks rootfs
5603 source
5604 -b BOOTIMG_DIR, --bootimg-dir BOOTIMG_DIR
5605 path to the dir containing the boot artifacts (e.g.
5606 /EFI or /syslinux dirs) to use as the .wks bootimg
5607 source
5608 -k KERNEL_DIR, --kernel-dir KERNEL_DIR
5609 path to the dir containing the kernel to use in the
5610 .wks bootimg
5611 -n NATIVE_SYSROOT, --native-sysroot NATIVE_SYSROOT
5612 path to the native sysroot containing the tools to use
5613 to build the image
5614 -s, --skip-build-check
5615 skip the build check
5616 -f, --build-rootfs build rootfs
5617 -c {gzip,bzip2,xz}, --compress-with {gzip,bzip2,xz}
5618 compress image with specified compressor
5619 -m, --bmap generate .bmap
5620 --no-fstab-update Do not change fstab file.
5621 -v VARS_DIR, --vars VARS_DIR
5622 directory with <image>.env files that store bitbake
5623 variables
5624 -D, --debug output debug information
5625
5626.. note::
5627
5628 You do not need root privileges to run Wic. In fact, you should not
5629 run as root when using the utility.
5630
5631Cooked Mode
5632~~~~~~~~~~~
5633
5634Running Wic in cooked mode leverages off artifacts in the Build
5635Directory. In other words, you do not have to specify kernel or root
5636filesystem locations as part of the command. All you need to provide is
5637a kickstart file and the name of the image from which to use artifacts
5638by using the "-e" option. Wic looks in the Build Directory (e.g.
5639``tmp/deploy/images/``\ machine) for artifacts.
5640
5641The general form of the ``wic`` command using Cooked Mode is as follows:
5642::
5643
5644 $ wic create wks_file -e IMAGE_NAME
5645
5646 Where:
5647
5648 wks_file:
5649 An OpenEmbedded kickstart file. You can provide
5650 your own custom file or use a file from a set of
5651 existing files provided with the Yocto Project
5652 release.
5653
5654 required argument:
5655 -e IMAGE_NAME, --image-name IMAGE_NAME
5656 name of the image to use the artifacts from e.g. core-
5657 image-sato
5658
5659.. _using-a-provided-kickstart-file:
5660
5661Using an Existing Kickstart File
5662--------------------------------
5663
5664If you do not want to create your own kickstart file, you can use an
5665existing file provided by the Wic installation. As shipped, kickstart
5666files can be found in the :ref:`overview-manual/overview-manual-development-environment:yocto project source repositories` in the
5667following two locations:
5668::
5669
5670 poky/meta-yocto-bsp/wic
5671 poky/scripts/lib/wic/canned-wks
5672
5673Use the following command to list the available kickstart files:
5674::
5675
5676 $ wic list images
5677 genericx86 Create an EFI disk image for genericx86*
5678 beaglebone-yocto Create SD card image for Beaglebone
5679 edgerouter Create SD card image for Edgerouter
5680 qemux86-directdisk Create a qemu machine 'pcbios' direct disk image
5681 directdisk-gpt Create a 'pcbios' direct disk image
5682 mkefidisk Create an EFI disk image
5683 directdisk Create a 'pcbios' direct disk image
5684 systemd-bootdisk Create an EFI disk image with systemd-boot
5685 mkhybridiso Create a hybrid ISO image
5686 sdimage-bootpart Create SD card image with a boot partition
5687 directdisk-multi-rootfs Create multi rootfs image using rootfs plugin
5688 directdisk-bootloader-config Create a 'pcbios' direct disk image with custom bootloader config
5689
5690When you use an existing file, you
5691do not have to use the ``.wks`` extension. Here is an example in Raw
5692Mode that uses the ``directdisk`` file:
5693::
5694
5695 $ wic create directdisk -r rootfs_dir -b bootimg_dir \
5696 -k kernel_dir -n native_sysroot
5697
5698Here are the actual partition language commands used in the
5699``genericx86.wks`` file to generate an image:
5700::
5701
5702 # short-description: Create an EFI disk image for genericx86*
5703 # long-description: Creates a partitioned EFI disk image for genericx86* machines
5704 part /boot --source bootimg-efi --sourceparams="loader=grub-efi" --ondisk sda --label msdos --active --align 1024
5705 part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 1024 --use-uuid
5706 part swap --ondisk sda --size 44 --label swap1 --fstype=swap
5707
5708 bootloader --ptable gpt --timeout=5 --append="rootfstype=ext4 console=ttyS0,115200 console=tty0"
5709
5710.. _wic-using-the-wic-plugin-interface:
5711
5712Using the Wic Plugin Interface
5713------------------------------
5714
5715You can extend and specialize Wic functionality by using Wic plugins.
5716This section explains the Wic plugin interface.
5717
5718.. note::
5719
5720 Wic plugins consist of "source" and "imager" plugins. Imager plugins
5721 are beyond the scope of this section.
5722
5723Source plugins provide a mechanism to customize partition content during
5724the Wic image generation process. You can use source plugins to map
5725values that you specify using ``--source`` commands in kickstart files
5726(i.e. ``*.wks``) to a plugin implementation used to populate a given
5727partition.
5728
5729.. note::
5730
5731 If you use plugins that have build-time dependencies (e.g. native
5732 tools, bootloaders, and so forth) when building a Wic image, you need
5733 to specify those dependencies using the
5734 WKS_FILE_DEPENDS
5735 variable.
5736
5737Source plugins are subclasses defined in plugin files. As shipped, the
5738Yocto Project provides several plugin files. You can see the source
5739plugin files that ship with the Yocto Project
5740:yocto_git:`here </cgit/cgit.cgi/poky/tree/scripts/lib/wic/plugins/source>`.
5741Each of these plugin files contains source plugins that are designed to
5742populate a specific Wic image partition.
5743
5744Source plugins are subclasses of the ``SourcePlugin`` class, which is
5745defined in the ``poky/scripts/lib/wic/pluginbase.py`` file. For example,
5746the ``BootimgEFIPlugin`` source plugin found in the ``bootimg-efi.py``
5747file is a subclass of the ``SourcePlugin`` class, which is found in the
5748``pluginbase.py`` file.
5749
5750You can also implement source plugins in a layer outside of the Source
5751Repositories (external layer). To do so, be sure that your plugin files
5752are located in a directory whose path is
5753``scripts/lib/wic/plugins/source/`` within your external layer. When the
5754plugin files are located there, the source plugins they contain are made
5755available to Wic.
5756
5757When the Wic implementation needs to invoke a partition-specific
5758implementation, it looks for the plugin with the same name as the
5759``--source`` parameter used in the kickstart file given to that
5760partition. For example, if the partition is set up using the following
5761command in a kickstart file:
5762::
5763
5764 part /boot --source bootimg-pcbios --ondisk sda --label boot --active --align 1024
5765
5766The methods defined as class
5767members of the matching source plugin (i.e. ``bootimg-pcbios``) in the
5768``bootimg-pcbios.py`` plugin file are used.
5769
5770To be more concrete, here is the corresponding plugin definition from
5771the ``bootimg-pcbios.py`` file for the previous command along with an
5772example method called by the Wic implementation when it needs to prepare
5773a partition using an implementation-specific function:
5774::
5775
5776 .
5777 .
5778 .
5779 class BootimgPcbiosPlugin(SourcePlugin):
5780 """
5781 Create MBR boot partition and install syslinux on it.
5782 """
5783
5784 name = 'bootimg-pcbios'
5785 .
5786 .
5787 .
5788 @classmethod
5789 def do_prepare_partition(cls, part, source_params, creator, cr_workdir,
5790 oe_builddir, bootimg_dir, kernel_dir,
5791 rootfs_dir, native_sysroot):
5792 """
5793 Called to do the actual content population for a partition i.e. it
5794 'prepares' the partition to be incorporated into the image.
5795 In this case, prepare content for legacy bios boot partition.
5796 """
5797 .
5798 .
5799 .
5800
5801If a
5802subclass (plugin) itself does not implement a particular function, Wic
5803locates and uses the default version in the superclass. It is for this
5804reason that all source plugins are derived from the ``SourcePlugin``
5805class.
5806
5807The ``SourcePlugin`` class defined in the ``pluginbase.py`` file defines
5808a set of methods that source plugins can implement or override. Any
5809plugins (subclass of ``SourcePlugin``) that do not implement a
5810particular method inherit the implementation of the method from the
5811``SourcePlugin`` class. For more information, see the ``SourcePlugin``
5812class in the ``pluginbase.py`` file for details:
5813
5814The following list describes the methods implemented in the
5815``SourcePlugin`` class:
5816
5817- ``do_prepare_partition()``: Called to populate a partition with
5818 actual content. In other words, the method prepares the final
5819 partition image that is incorporated into the disk image.
5820
5821- ``do_configure_partition()``: Called before
5822 ``do_prepare_partition()`` to create custom configuration files for a
5823 partition (e.g. syslinux or grub configuration files).
5824
5825- ``do_install_disk()``: Called after all partitions have been
5826 prepared and assembled into a disk image. This method provides a hook
5827 to allow finalization of a disk image (e.g. writing an MBR).
5828
5829- ``do_stage_partition()``: Special content-staging hook called
5830 before ``do_prepare_partition()``. This method is normally empty.
5831
5832 Typically, a partition just uses the passed-in parameters (e.g. the
5833 unmodified value of ``bootimg_dir``). However, in some cases, things
5834 might need to be more tailored. As an example, certain files might
5835 additionally need to be taken from ``bootimg_dir + /boot``. This hook
5836 allows those files to be staged in a customized fashion.
5837
5838 .. note::
5839
5840 get_bitbake_var()
5841 allows you to access non-standard variables that you might want to
5842 use for this behavior.
5843
5844You can extend the source plugin mechanism. To add more hooks, create
5845more source plugin methods within ``SourcePlugin`` and the corresponding
5846derived subclasses. The code that calls the plugin methods uses the
5847``plugin.get_source_plugin_methods()`` function to find the method or
5848methods needed by the call. Retrieval of those methods is accomplished
5849by filling up a dict with keys that contain the method names of
5850interest. On success, these will be filled in with the actual methods.
5851See the Wic implementation for examples and details.
5852
5853.. _wic-usage-examples:
5854
5855Wic Examples
5856------------
5857
5858This section provides several examples that show how to use the Wic
5859utility. All the examples assume the list of requirements in the
5860"`Requirements <#wic-requirements>`__" section have been met. The
5861examples assume the previously generated image is
5862``core-image-minimal``.
5863
5864.. _generate-an-image-using-a-provided-kickstart-file:
5865
5866Generate an Image using an Existing Kickstart File
5867~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5868
5869This example runs in Cooked Mode and uses the ``mkefidisk`` kickstart
5870file:
5871::
5872
5873 $ wic create mkefidisk -e core-image-minimal
5874 INFO: Building wic-tools...
5875 .
5876 .
5877 .
5878 INFO: The new image(s) can be found here:
5879 ./mkefidisk-201804191017-sda.direct
5880
5881 The following build artifacts were used to create the image(s):
5882 ROOTFS_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
5883 BOOTIMG_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
5884 KERNEL_DIR: /home/stephano/build/master/build/tmp-glibc/deploy/images/qemux86
5885 NATIVE_SYSROOT: /home/stephano/build/master/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
5886
5887 INFO: The image(s) were created using OE kickstart file:
5888 /home/stephano/build/master/openembedded-core/scripts/lib/wic/canned-wks/mkefidisk.wks
5889
5890The previous example shows the easiest way to create an image by running
5891in cooked mode and supplying a kickstart file and the "-e" option to
5892point to the existing build artifacts. Your ``local.conf`` file needs to
5893have the :term:`MACHINE` variable set
5894to the machine you are using, which is "qemux86" in this example.
5895
5896Once the image builds, the output provides image location, artifact use,
5897and kickstart file information.
5898
5899.. note::
5900
5901 You should always verify the details provided in the output to make
5902 sure that the image was indeed created exactly as expected.
5903
5904Continuing with the example, you can now write the image from the Build
5905Directory onto a USB stick, or whatever media for which you built your
5906image, and boot from the media. You can write the image by using
5907``bmaptool`` or ``dd``:
5908::
5909
5910 $ oe-run-native bmaptool copy mkefidisk-201804191017-sda.direct /dev/sdX
5911
5912or ::
5913
5914 $ sudo dd if=mkefidisk-201804191017-sda.direct of=/dev/sdX
5915
5916.. note::
5917
5918 For more information on how to use the
5919 bmaptool
5920 to flash a device with an image, see the "
5921 Flashing Images Using
5922 bmaptool
5923 " section.
5924
5925Using a Modified Kickstart File
5926~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5927
5928Because partitioned image creation is driven by the kickstart file, it
5929is easy to affect image creation by changing the parameters in the file.
5930This next example demonstrates that through modification of the
5931``directdisk-gpt`` kickstart file.
5932
5933As mentioned earlier, you can use the command ``wic list images`` to
5934show the list of existing kickstart files. The directory in which the
5935``directdisk-gpt.wks`` file resides is
5936``scripts/lib/image/canned-wks/``, which is located in the
5937:term:`Source Directory` (e.g. ``poky``).
5938Because available files reside in this directory, you can create and add
5939your own custom files to the directory. Subsequent use of the
5940``wic list images`` command would then include your kickstart files.
5941
5942In this example, the existing ``directdisk-gpt`` file already does most
5943of what is needed. However, for the hardware in this example, the image
5944will need to boot from ``sdb`` instead of ``sda``, which is what the
5945``directdisk-gpt`` kickstart file uses.
5946
5947The example begins by making a copy of the ``directdisk-gpt.wks`` file
5948in the ``scripts/lib/image/canned-wks`` directory and then by changing
5949the lines that specify the target disk from which to boot.
5950::
5951
5952 $ cp /home/stephano/poky/scripts/lib/wic/canned-wks/directdisk-gpt.wks \
5953 /home/stephano/poky/scripts/lib/wic/canned-wks/directdisksdb-gpt.wks
5954
5955Next, the example modifies the ``directdisksdb-gpt.wks`` file and
5956changes all instances of "``--ondisk sda``" to "``--ondisk sdb``". The
5957example changes the following two lines and leaves the remaining lines
5958untouched:
5959::
5960
5961 part /boot --source bootimg-pcbios --ondisk sdb --label boot --active --align 1024
5962 part / --source rootfs --ondisk sdb --fstype=ext4 --label platform --align 1024 --use-uuid
5963
5964Once the lines are changed, the
5965example generates the ``directdisksdb-gpt`` image. The command points
5966the process at the ``core-image-minimal`` artifacts for the Next Unit of
5967Computing (nuc) :term:`MACHINE` the
5968``local.conf``.
5969::
5970
5971 $ wic create directdisksdb-gpt -e core-image-minimal
5972 INFO: Building wic-tools...
5973 .
5974 .
5975 .
5976 Initialising tasks: 100% |#######################################| Time: 0:00:01
5977 NOTE: Executing SetScene Tasks
5978 NOTE: Executing RunQueue Tasks
5979 NOTE: Tasks Summary: Attempted 1161 tasks of which 1157 didn't need to be rerun and all succeeded.
5980 INFO: Creating image(s)...
5981
5982 INFO: The new image(s) can be found here:
5983 ./directdisksdb-gpt-201710090938-sdb.direct
5984
5985 The following build artifacts were used to create the image(s):
5986 ROOTFS_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
5987 BOOTIMG_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
5988 KERNEL_DIR: /home/stephano/build/master/build/tmp-glibc/deploy/images/qemux86
5989 NATIVE_SYSROOT: /home/stephano/build/master/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
5990
5991 INFO: The image(s) were created using OE kickstart file:
5992 /home/stephano/poky/scripts/lib/wic/canned-wks/directdisksdb-gpt.wks
5993
5994Continuing with the example, you can now directly ``dd`` the image to a
5995USB stick, or whatever media for which you built your image, and boot
5996the resulting media:
5997::
5998
5999 $ sudo dd if=directdisksdb-gpt-201710090938-sdb.direct of=/dev/sdb
6000 140966+0 records in
6001 140966+0 records out
6002 72174592 bytes (72 MB, 69 MiB) copied, 78.0282 s, 925 kB/s
6003 $ sudo eject /dev/sdb
6004
6005Using a Modified Kickstart File and Running in Raw Mode
6006~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
6007
6008This next example manually specifies each build artifact (runs in Raw
6009Mode) and uses a modified kickstart file. The example also uses the
6010``-o`` option to cause Wic to create the output somewhere other than the
6011default output directory, which is the current directory:
6012::
6013
6014 $ wic create /home/stephano/my_yocto/test.wks -o /home/stephano/testwic \
6015 --rootfs-dir /home/stephano/build/master/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/rootfs \
6016 --bootimg-dir /home/stephano/build/master/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share \
6017 --kernel-dir /home/stephano/build/master/build/tmp/deploy/images/qemux86 \
6018 --native-sysroot /home/stephano/build/master/build/tmp/work/i586-poky-linux/wic-tools/1.0-r0/recipe-sysroot-native
6019
6020 INFO: Creating image(s)...
6021
6022 INFO: The new image(s) can be found here:
6023 /home/stephano/testwic/test-201710091445-sdb.direct
6024
6025 The following build artifacts were used to create the image(s):
6026 ROOTFS_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
6027 BOOTIMG_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
6028 KERNEL_DIR: /home/stephano/build/master/build/tmp-glibc/deploy/images/qemux86
6029 NATIVE_SYSROOT: /home/stephano/build/master/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
6030
6031 INFO: The image(s) were created using OE kickstart file:
6032 /home/stephano/my_yocto/test.wks
6033
6034For this example,
6035:term:`MACHINE` did not have to be
6036specified in the ``local.conf`` file since the artifact is manually
6037specified.
6038
6039Using Wic to Manipulate an Image
6040~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
6041
6042Wic image manipulation allows you to shorten turnaround time during
6043image development. For example, you can use Wic to delete the kernel
6044partition of a Wic image and then insert a newly built kernel. This
6045saves you time from having to rebuild the entire image each time you
6046modify the kernel.
6047
6048.. note::
6049
6050 In order to use Wic to manipulate a Wic image as in this example,
6051 your development machine must have the
6052 mtools
6053 package installed.
6054
6055The following example examines the contents of the Wic image, deletes
6056the existing kernel, and then inserts a new kernel:
6057
60581. *List the Partitions:* Use the ``wic ls`` command to list all the
6059 partitions in the Wic image:
6060 ::
6061
6062 $ wic ls tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic
6063 Num Start End Size Fstype
6064 1 1048576 25041919 23993344 fat16
6065 2 25165824 72157183 46991360 ext4
6066
6067 The previous output shows two partitions in the
6068 ``core-image-minimal-qemux86.wic`` image.
6069
60702. *Examine a Particular Partition:* Use the ``wic ls`` command again
6071 but in a different form to examine a particular partition.
6072
6073 .. note::
6074
6075 You can get command usage on any Wic command using the following
6076 form:
6077 ::
6078
6079 $ wic help command
6080
6081
6082 For example, the following command shows you the various ways to
6083 use the
6084 wic ls
6085 command:
6086 ::
6087
6088 $ wic help ls
6089
6090
6091 The following command shows what is in Partition one:
6092 ::
6093
6094 $ wic ls tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1
6095 Volume in drive : is boot
6096 Volume Serial Number is E894-1809
6097 Directory for ::/
6098
6099 libcom32 c32 186500 2017-10-09 16:06
6100 libutil c32 24148 2017-10-09 16:06
6101 syslinux cfg 220 2017-10-09 16:06
6102 vesamenu c32 27104 2017-10-09 16:06
6103 vmlinuz 6904608 2017-10-09 16:06
6104 5 files 7 142 580 bytes
6105 16 582 656 bytes free
6106
6107 The previous output shows five files, with the
6108 ``vmlinuz`` being the kernel.
6109
6110 .. note::
6111
6112 If you see the following error, you need to update or create a
6113 ~/.mtoolsrc
6114 file and be sure to have the line "mtools_skip_check=1" in the
6115 file. Then, run the Wic command again:
6116 ::
6117
6118 ERROR: _exec_cmd: /usr/bin/mdir -i /tmp/wic-parttfokuwra ::/ returned '1' instead of 0
6119 output: Total number of sectors (47824) not a multiple of sectors per track (32)!
6120 Add mtools_skip_check=1 to your .mtoolsrc file to skip this test
6121
6122
61233. *Remove the Old Kernel:* Use the ``wic rm`` command to remove the
6124 ``vmlinuz`` file (kernel):
6125 ::
6126
6127 $ wic rm tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1/vmlinuz
6128
61294. *Add In the New Kernel:* Use the ``wic cp`` command to add the
6130 updated kernel to the Wic image. Depending on how you built your
6131 kernel, it could be in different places. If you used ``devtool`` and
6132 an SDK to build your kernel, it resides in the ``tmp/work`` directory
6133 of the extensible SDK. If you used ``make`` to build the kernel, the
6134 kernel will be in the ``workspace/sources`` area.
6135
6136 The following example assumes ``devtool`` was used to build the
6137 kernel:
6138 ::
6139
6140 cp ~/poky_sdk/tmp/work/qemux86-poky-linux/linux-yocto/4.12.12+git999-r0/linux-yocto-4.12.12+git999/arch/x86/boot/bzImage \
6141 ~/poky/build/tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1/vmlinuz
6142
6143 Once the new kernel is added back into the image, you can use the
6144 ``dd`` command or ```bmaptool`` <#flashing-images-using-bmaptool>`__
6145 to flash your wic image onto an SD card or USB stick and test your
6146 target.
6147
6148 .. note::
6149
6150 Using
6151 bmaptool
6152 is generally 10 to 20 times faster than using
6153 dd
6154 .
6155
6156Flashing Images Using ``bmaptool``
6157==================================
6158
6159A fast and easy way to flash an image to a bootable device is to use
6160Bmaptool, which is integrated into the OpenEmbedded build system.
6161Bmaptool is a generic tool that creates a file's block map (bmap) and
6162then uses that map to copy the file. As compared to traditional tools
6163such as dd or cp, Bmaptool can copy (or flash) large files like raw
6164system image files much faster.
6165
6166.. note::
6167
6168 - If you are using Ubuntu or Debian distributions, you can install
6169 the ``bmap-tools`` package using the following command and then
6170 use the tool without specifying ``PATH`` even from the root
6171 account: $ sudo apt-get install bmap-tools
6172
6173 - If you are unable to install the ``bmap-tools`` package, you will
6174 need to build Bmaptool before using it. Use the following command:
6175 $ bitbake bmap-tools-native
6176
6177Following, is an example that shows how to flash a Wic image. Realize
6178that while this example uses a Wic image, you can use Bmaptool to flash
6179any type of image. Use these steps to flash an image using Bmaptool:
6180
61811. *Update your local.conf File:* You need to have the following set
6182 in your ``local.conf`` file before building your image:
6183 ::
6184
6185 IMAGE_FSTYPES += "wic wic.bmap"
6186
61872. *Get Your Image:* Either have your image ready (pre-built with the
6188 :term:`IMAGE_FSTYPES`
6189 setting previously mentioned) or take the step to build the image:
6190 ::
6191
6192 $ bitbake image
6193
61943. *Flash the Device:* Flash the device with the image by using Bmaptool
6195 depending on your particular setup. The following commands assume the
6196 image resides in the Build Directory's ``deploy/images/`` area:
6197
6198 - If you have write access to the media, use this command form:
6199 ::
6200
6201 $ oe-run-native bmap-tools-native bmaptool copy build-directory/tmp/deploy/images/machine/image.wic /dev/sdX
6202
6203 - If you do not have write access to the media, set your permissions
6204 first and then use the same command form:
6205 ::
6206
6207 $ sudo chmod 666 /dev/sdX
6208 $ oe-run-native bmap-tools-native bmaptool copy build-directory/tmp/deploy/images/machine/image.wic /dev/sdX
6209
6210For help on the ``bmaptool`` command, use the following command:
6211::
6212
6213 $ bmaptool --help
6214
6215Making Images More Secure
6216=========================
6217
6218Security is of increasing concern for embedded devices. Consider the
6219issues and problems discussed in just this sampling of work found across
6220the Internet:
6221
6222- *"*\ `Security Risks of Embedded
6223 Systems <https://www.schneier.com/blog/archives/2014/01/security_risks_9.html>`__\ *"*
6224 by Bruce Schneier
6225
6226- *"*\ `Internet Census
6227 2012 <http://census2012.sourceforge.net/paper.html>`__\ *"* by Carna
6228 Botnet
6229
6230- *"*\ `Security Issues for Embedded
6231 Devices <http://elinux.org/images/6/6f/Security-issues.pdf>`__\ *"*
6232 by Jake Edge
6233
6234When securing your image is of concern, there are steps, tools, and
6235variables that you can consider to help you reach the security goals you
6236need for your particular device. Not all situations are identical when
6237it comes to making an image secure. Consequently, this section provides
6238some guidance and suggestions for consideration when you want to make
6239your image more secure.
6240
6241.. note::
6242
6243 Because the security requirements and risks are different for every
6244 type of device, this section cannot provide a complete reference on
6245 securing your custom OS. It is strongly recommended that you also
6246 consult other sources of information on embedded Linux system
6247 hardening and on security.
6248
6249General Considerations
6250----------------------
6251
6252General considerations exist that help you create more secure images.
6253You should consider the following suggestions to help make your device
6254more secure:
6255
6256- Scan additional code you are adding to the system (e.g. application
6257 code) by using static analysis tools. Look for buffer overflows and
6258 other potential security problems.
6259
6260- Pay particular attention to the security for any web-based
6261 administration interface.
6262
6263 Web interfaces typically need to perform administrative functions and
6264 tend to need to run with elevated privileges. Thus, the consequences
6265 resulting from the interface's security becoming compromised can be
6266 serious. Look for common web vulnerabilities such as
6267 cross-site-scripting (XSS), unvalidated inputs, and so forth.
6268
6269 As with system passwords, the default credentials for accessing a
6270 web-based interface should not be the same across all devices. This
6271 is particularly true if the interface is enabled by default as it can
6272 be assumed that many end-users will not change the credentials.
6273
6274- Ensure you can update the software on the device to mitigate
6275 vulnerabilities discovered in the future. This consideration
6276 especially applies when your device is network-enabled.
6277
6278- Ensure you remove or disable debugging functionality before producing
6279 the final image. For information on how to do this, see the
6280 "`Considerations Specific to the OpenEmbedded Build
6281 System <#considerations-specific-to-the-openembedded-build-system>`__"
6282 section.
6283
6284- Ensure you have no network services listening that are not needed.
6285
6286- Remove any software from the image that is not needed.
6287
6288- Enable hardware support for secure boot functionality when your
6289 device supports this functionality.
6290
6291Security Flags
6292--------------
6293
6294The Yocto Project has security flags that you can enable that help make
6295your build output more secure. The security flags are in the
6296``meta/conf/distro/include/security_flags.inc`` file in your
6297:term:`Source Directory` (e.g. ``poky``).
6298
6299.. note::
6300
6301 Depending on the recipe, certain security flags are enabled and
6302 disabled by default.
6303
6304Use the following line in your ``local.conf`` file or in your custom
6305distribution configuration file to enable the security compiler and
6306linker flags for your build:
6307::
6308
6309 require conf/distro/include/security_flags.inc
6310
6311Considerations Specific to the OpenEmbedded Build System
6312--------------------------------------------------------
6313
6314You can take some steps that are specific to the OpenEmbedded build
6315system to make your images more secure:
6316
6317- Ensure "debug-tweaks" is not one of your selected
6318 :term:`IMAGE_FEATURES`.
6319 When creating a new project, the default is to provide you with an
6320 initial ``local.conf`` file that enables this feature using the
6321 :term:`EXTRA_IMAGE_FEATURES`
6322 variable with the line:
6323 ::
6324
6325 EXTRA_IMAGE_FEATURES = "debug-tweaks"
6326
6327 To disable that feature, simply comment out that line in your
6328 ``local.conf`` file, or make sure ``IMAGE_FEATURES`` does not contain
6329 "debug-tweaks" before producing your final image. Among other things,
6330 leaving this in place sets the root password as blank, which makes
6331 logging in for debugging or inspection easy during development but
6332 also means anyone can easily log in during production.
6333
6334- It is possible to set a root password for the image and also to set
6335 passwords for any extra users you might add (e.g. administrative or
6336 service type users). When you set up passwords for multiple images or
6337 users, you should not duplicate passwords.
6338
6339 To set up passwords, use the
6340 :ref:`extrausers <ref-classes-extrausers>`
6341 class, which is the preferred method. For an example on how to set up
6342 both root and user passwords, see the
6343 ":ref:`extrausers.bbclass <ref-classes-extrausers>`"
6344 section.
6345
6346 .. note::
6347
6348 When adding extra user accounts or setting a root password, be
6349 cautious about setting the same password on every device. If you
6350 do this, and the password you have set is exposed, then every
6351 device is now potentially compromised. If you need this access but
6352 want to ensure security, consider setting a different, random
6353 password for each device. Typically, you do this as a separate
6354 step after you deploy the image onto the device.
6355
6356- Consider enabling a Mandatory Access Control (MAC) framework such as
6357 SMACK or SELinux and tuning it appropriately for your device's usage.
6358 You can find more information in the
6359 `meta-selinux <http://git.yoctoproject.org/cgit/cgit.cgi/meta-selinux/>`__
6360 layer.
6361
6362Tools for Hardening Your Image
6363------------------------------
6364
6365The Yocto Project provides tools for making your image more secure. You
6366can find these tools in the ``meta-security`` layer of the
6367:yocto_git:`Yocto Project Source Repositories <>`.
6368
6369Creating Your Own Distribution
6370==============================
6371
6372When you build an image using the Yocto Project and do not alter any
6373distribution :term:`Metadata`, you are
6374creating a Poky distribution. If you wish to gain more control over
6375package alternative selections, compile-time options, and other
6376low-level configurations, you can create your own distribution.
6377
6378To create your own distribution, the basic steps consist of creating
6379your own distribution layer, creating your own distribution
6380configuration file, and then adding any needed code and Metadata to the
6381layer. The following steps provide some more detail:
6382
6383- *Create a layer for your new distro:* Create your distribution layer
6384 so that you can keep your Metadata and code for the distribution
6385 separate. It is strongly recommended that you create and use your own
6386 layer for configuration and code. Using your own layer as compared to
6387 just placing configurations in a ``local.conf`` configuration file
6388 makes it easier to reproduce the same build configuration when using
6389 multiple build machines. See the
6390 ":ref:`dev-manual/dev-manual-common-tasks:creating a general layer using the \`\`bitbake-layers\`\` script`"
6391 section for information on how to quickly set up a layer.
6392
6393- *Create the distribution configuration file:* The distribution
6394 configuration file needs to be created in the ``conf/distro``
6395 directory of your layer. You need to name it using your distribution
6396 name (e.g. ``mydistro.conf``).
6397
6398 .. note::
6399
6400 The
6401 DISTRO
6402 variable in your
6403 local.conf
6404 file determines the name of your distribution.
6405
6406 You can split out parts of your configuration file into include files
6407 and then "require" them from within your distribution configuration
6408 file. Be sure to place the include files in the
6409 ``conf/distro/include`` directory of your layer. A common example
6410 usage of include files would be to separate out the selection of
6411 desired version and revisions for individual recipes.
6412
6413 Your configuration file needs to set the following required
6414 variables:
6415
6416 - :term:`DISTRO_NAME`
6417
6418 - :term:`DISTRO_VERSION`
6419
6420 These following variables are optional and you typically set them
6421 from the distribution configuration file:
6422
6423 - :term:`DISTRO_FEATURES`
6424
6425 - :term:`DISTRO_EXTRA_RDEPENDS`
6426
6427 - :term:`DISTRO_EXTRA_RRECOMMENDS`
6428
6429 - :term:`TCLIBC`
6430
6431 .. tip::
6432
6433 If you want to base your distribution configuration file on the
6434 very basic configuration from OE-Core, you can use
6435 conf/distro/defaultsetup.conf
6436 as a reference and just include variables that differ as compared
6437 to
6438 defaultsetup.conf
6439 . Alternatively, you can create a distribution configuration file
6440 from scratch using the
6441 defaultsetup.conf
6442 file or configuration files from other distributions such as Poky
6443 or Angstrom as references.
6444
6445- *Provide miscellaneous variables:* Be sure to define any other
6446 variables for which you want to create a default or enforce as part
6447 of the distribution configuration. You can include nearly any
6448 variable from the ``local.conf`` file. The variables you use are not
6449 limited to the list in the previous bulleted item.
6450
6451- *Point to Your distribution configuration file:* In your
6452 ``local.conf`` file in the :term:`Build Directory`,
6453 set your
6454 :term:`DISTRO` variable to point to
6455 your distribution's configuration file. For example, if your
6456 distribution's configuration file is named ``mydistro.conf``, then
6457 you point to it as follows:
6458 ::
6459
6460 DISTRO = "mydistro"
6461
6462- *Add more to the layer if necessary:* Use your layer to hold other
6463 information needed for the distribution:
6464
6465 - Add recipes for installing distro-specific configuration files
6466 that are not already installed by another recipe. If you have
6467 distro-specific configuration files that are included by an
6468 existing recipe, you should add an append file (``.bbappend``) for
6469 those. For general information and recommendations on how to add
6470 recipes to your layer, see the "`Creating Your Own
6471 Layer <#creating-your-own-layer>`__" and "`Following Best
6472 Practices When Creating
6473 Layers <#best-practices-to-follow-when-creating-layers>`__"
6474 sections.
6475
6476 - Add any image recipes that are specific to your distribution.
6477
6478 - Add a ``psplash`` append file for a branded splash screen. For
6479 information on append files, see the "`Using .bbappend Files in
6480 Your Layer <#using-bbappend-files>`__" section.
6481
6482 - Add any other append files to make custom changes that are
6483 specific to individual recipes.
6484
6485Creating a Custom Template Configuration Directory
6486==================================================
6487
6488If you are producing your own customized version of the build system for
6489use by other users, you might want to customize the message shown by the
6490setup script or you might want to change the template configuration
6491files (i.e. ``local.conf`` and ``bblayers.conf``) that are created in a
6492new build directory.
6493
6494The OpenEmbedded build system uses the environment variable
6495``TEMPLATECONF`` to locate the directory from which it gathers
6496configuration information that ultimately ends up in the
6497:term:`Build Directory` ``conf`` directory.
6498By default, ``TEMPLATECONF`` is set as follows in the ``poky``
6499repository:
6500::
6501
6502 TEMPLATECONF=${TEMPLATECONF:-meta-poky/conf}
6503
6504This is the
6505directory used by the build system to find templates from which to build
6506some key configuration files. If you look at this directory, you will
6507see the ``bblayers.conf.sample``, ``local.conf.sample``, and
6508``conf-notes.txt`` files. The build system uses these files to form the
6509respective ``bblayers.conf`` file, ``local.conf`` file, and display the
6510list of BitBake targets when running the setup script.
6511
6512To override these default configuration files with configurations you
6513want used within every new Build Directory, simply set the
6514``TEMPLATECONF`` variable to your directory. The ``TEMPLATECONF``
6515variable is set in the ``.templateconf`` file, which is in the top-level
6516:term:`Source Directory` folder
6517(e.g. ``poky``). Edit the ``.templateconf`` so that it can locate your
6518directory.
6519
6520Best practices dictate that you should keep your template configuration
6521directory in your custom distribution layer. For example, suppose you
6522have a layer named ``meta-mylayer`` located in your home directory and
6523you want your template configuration directory named ``myconf``.
6524Changing the ``.templateconf`` as follows causes the OpenEmbedded build
6525system to look in your directory and base its configuration files on the
6526``*.sample`` configuration files it finds. The final configuration files
6527(i.e. ``local.conf`` and ``bblayers.conf`` ultimately still end up in
6528your Build Directory, but they are based on your ``*.sample`` files.
6529::
6530
6531 TEMPLATECONF=${TEMPLATECONF:-meta-mylayer/myconf}
6532
6533Aside from the ``*.sample`` configuration files, the ``conf-notes.txt``
6534also resides in the default ``meta-poky/conf`` directory. The script
6535that sets up the build environment (i.e.
6536:ref:`structure-core-script`) uses this file to
6537display BitBake targets as part of the script output. Customizing this
6538``conf-notes.txt`` file is a good way to make sure your list of custom
6539targets appears as part of the script's output.
6540
6541Here is the default list of targets displayed as a result of running
6542either of the setup scripts:
6543::
6544
6545 You can now run 'bitbake <target>'
6546
6547 Common targets are:
6548 core-image-minimal
6549 core-image-sato
6550 meta-toolchain
6551 meta-ide-support
6552
6553Changing the listed common targets is as easy as editing your version of
6554``conf-notes.txt`` in your custom template configuration directory and
6555making sure you have ``TEMPLATECONF`` set to your directory.
6556
6557.. _dev-saving-memory-during-a-build:
6558
6559Conserving Disk Space During Builds
6560===================================
6561
6562To help conserve disk space during builds, you can add the following
6563statement to your project's ``local.conf`` configuration file found in
6564the :term:`Build Directory`:
6565::
6566
6567 INHERIT += "rm_work"
6568
6569Adding this statement deletes the work directory used for
6570building a recipe once the recipe is built. For more information on
6571"rm_work", see the
6572:ref:`rm_work <ref-classes-rm-work>` class in the
6573Yocto Project Reference Manual.
6574
6575Working with Packages
6576=====================
6577
6578This section describes a few tasks that involve packages:
6579
6580- `Excluding packages from an
6581 image <#excluding-packages-from-an-image>`__
6582
6583- `Incrementing a binary package
6584 version <#incrementing-a-binary-package-version>`__
6585
6586- `Handling optional module
6587 packaging <#handling-optional-module-packaging>`__
6588
6589- `Using runtime package
6590 management <#using-runtime-package-management>`__
6591
6592- `Generating and using signed
6593 packages <#generating-and-using-signed-packages>`__
6594
6595- `Setting up and running package test
6596 (ptest) <#testing-packages-with-ptest>`__
6597
6598- `Creating node package manager (NPM)
6599 packages <#creating-node-package-manager-npm-packages>`__
6600
6601- `Adding custom metadata to
6602 packages <#adding-custom-metadata-to-packages>`__
6603
6604Excluding Packages from an Image
6605--------------------------------
6606
6607You might find it necessary to prevent specific packages from being
6608installed into an image. If so, you can use several variables to direct
6609the build system to essentially ignore installing recommended packages
6610or to not install a package at all.
6611
6612The following list introduces variables you can use to prevent packages
6613from being installed into your image. Each of these variables only works
6614with IPK and RPM package types. Support for Debian packages does not
6615exist. Also, you can use these variables from your ``local.conf`` file
6616or attach them to a specific image recipe by using a recipe name
6617override. For more detail on the variables, see the descriptions in the
6618Yocto Project Reference Manual's glossary chapter.
6619
6620- :term:`BAD_RECOMMENDATIONS`:
6621 Use this variable to specify "recommended-only" packages that you do
6622 not want installed.
6623
6624- :term:`NO_RECOMMENDATIONS`:
6625 Use this variable to prevent all "recommended-only" packages from
6626 being installed.
6627
6628- :term:`PACKAGE_EXCLUDE`:
6629 Use this variable to prevent specific packages from being installed
6630 regardless of whether they are "recommended-only" or not. You need to
6631 realize that the build process could fail with an error when you
6632 prevent the installation of a package whose presence is required by
6633 an installed package.
6634
6635.. _incrementing-a-binary-package-version:
6636
6637Incrementing a Package Version
6638------------------------------
6639
6640This section provides some background on how binary package versioning
6641is accomplished and presents some of the services, variables, and
6642terminology involved.
6643
6644In order to understand binary package versioning, you need to consider
6645the following:
6646
6647- Binary Package: The binary package that is eventually built and
6648 installed into an image.
6649
6650- Binary Package Version: The binary package version is composed of two
6651 components - a version and a revision.
6652
6653 .. note::
6654
6655 Technically, a third component, the "epoch" (i.e.
6656 PE
6657 ) is involved but this discussion for the most part ignores
6658 PE
6659 .
6660
6661 The version and revision are taken from the
6662 :term:`PV` and
6663 :term:`PR` variables, respectively.
6664
6665- ``PV``: The recipe version. ``PV`` represents the version of the
6666 software being packaged. Do not confuse ``PV`` with the binary
6667 package version.
6668
6669- ``PR``: The recipe revision.
6670
6671- :term:`SRCPV`: The OpenEmbedded
6672 build system uses this string to help define the value of ``PV`` when
6673 the source code revision needs to be included in it.
6674
6675- :yocto_wiki:`PR Service </wiki/PR_Service>`: A
6676 network-based service that helps automate keeping package feeds
6677 compatible with existing package manager applications such as RPM,
6678 APT, and OPKG.
6679
6680Whenever the binary package content changes, the binary package version
6681must change. Changing the binary package version is accomplished by
6682changing or "bumping" the ``PR`` and/or ``PV`` values. Increasing these
6683values occurs one of two ways:
6684
6685- Automatically using a Package Revision Service (PR Service).
6686
6687- Manually incrementing the ``PR`` and/or ``PV`` variables.
6688
6689Given a primary challenge of any build system and its users is how to
6690maintain a package feed that is compatible with existing package manager
6691applications such as RPM, APT, and OPKG, using an automated system is
6692much preferred over a manual system. In either system, the main
6693requirement is that binary package version numbering increases in a
6694linear fashion and that a number of version components exist that
6695support that linear progression. For information on how to ensure
6696package revisioning remains linear, see the "`Automatically Incrementing
6697a Binary Package Revision
6698Number <#automatically-incrementing-a-binary-package-revision-number>`__"
6699section.
6700
6701The following three sections provide related information on the PR
6702Service, the manual method for "bumping" ``PR`` and/or ``PV``, and on
6703how to ensure binary package revisioning remains linear.
6704
6705Working With a PR Service
6706~~~~~~~~~~~~~~~~~~~~~~~~~
6707
6708As mentioned, attempting to maintain revision numbers in the
6709:term:`Metadata` is error prone, inaccurate,
6710and causes problems for people submitting recipes. Conversely, the PR
6711Service automatically generates increasing numbers, particularly the
6712revision field, which removes the human element.
6713
6714.. note::
6715
6716 For additional information on using a PR Service, you can see the
6717 PR Service
6718 wiki page.
6719
6720The Yocto Project uses variables in order of decreasing priority to
6721facilitate revision numbering (i.e.
6722:term:`PE`,
6723:term:`PV`, and
6724:term:`PR` for epoch, version, and
6725revision, respectively). The values are highly dependent on the policies
6726and procedures of a given distribution and package feed.
6727
6728Because the OpenEmbedded build system uses
6729":ref:`signatures <overview-checksums>`", which are
6730unique to a given build, the build system knows when to rebuild
6731packages. All the inputs into a given task are represented by a
6732signature, which can trigger a rebuild when different. Thus, the build
6733system itself does not rely on the ``PR``, ``PV``, and ``PE`` numbers to
6734trigger a rebuild. The signatures, however, can be used to generate
6735these values.
6736
6737The PR Service works with both ``OEBasic`` and ``OEBasicHash``
6738generators. The value of ``PR`` bumps when the checksum changes and the
6739different generator mechanisms change signatures under different
6740circumstances.
6741
6742As implemented, the build system includes values from the PR Service
6743into the ``PR`` field as an addition using the form "``.x``" so ``r0``
6744becomes ``r0.1``, ``r0.2`` and so forth. This scheme allows existing
6745``PR`` values to be used for whatever reasons, which include manual
6746``PR`` bumps, should it be necessary.
6747
6748By default, the PR Service is not enabled or running. Thus, the packages
6749generated are just "self consistent". The build system adds and removes
6750packages and there are no guarantees about upgrade paths but images will
6751be consistent and correct with the latest changes.
6752
6753The simplest form for a PR Service is for it to exist for a single host
6754development system that builds the package feed (building system). For
6755this scenario, you can enable a local PR Service by setting
6756:term:`PRSERV_HOST` in your
6757``local.conf`` file in the :term:`Build Directory`:
6758::
6759
6760 PRSERV_HOST = "localhost:0"
6761
6762Once the service is started, packages will automatically
6763get increasing ``PR`` values and BitBake takes care of starting and
6764stopping the server.
6765
6766If you have a more complex setup where multiple host development systems
6767work against a common, shared package feed, you have a single PR Service
6768running and it is connected to each building system. For this scenario,
6769you need to start the PR Service using the ``bitbake-prserv`` command:
6770::
6771
6772 bitbake-prserv --host ip --port port --start
6773
6774In addition to
6775hand-starting the service, you need to update the ``local.conf`` file of
6776each building system as described earlier so each system points to the
6777server and port.
6778
6779It is also recommended you use build history, which adds some sanity
6780checks to binary package versions, in conjunction with the server that
6781is running the PR Service. To enable build history, add the following to
6782each building system's ``local.conf`` file:
6783::
6784
6785 # It is recommended to activate "buildhistory" for testing the PR service
6786 INHERIT += "buildhistory"
6787 BUILDHISTORY_COMMIT = "1"
6788
6789For information on build
6790history, see the "`Maintaining Build Output
6791Quality <#maintaining-build-output-quality>`__" section.
6792
6793.. note::
6794
6795 The OpenEmbedded build system does not maintain ``PR`` information as
6796 part of the shared state (sstate) packages. If you maintain an sstate
6797 feed, its expected that either all your building systems that
6798 contribute to the sstate feed use a shared PR Service, or you do not
6799 run a PR Service on any of your building systems. Having some systems
6800 use a PR Service while others do not leads to obvious problems.
6801
6802 For more information on shared state, see the
6803 ":ref:`overview-manual/overview-manual-concepts:shared state cache`"
6804 section in the Yocto Project Overview and Concepts Manual.
6805
6806Manually Bumping PR
6807~~~~~~~~~~~~~~~~~~~
6808
6809The alternative to setting up a PR Service is to manually "bump" the
6810:term:`PR` variable.
6811
6812If a committed change results in changing the package output, then the
6813value of the PR variable needs to be increased (or "bumped") as part of
6814that commit. For new recipes you should add the ``PR`` variable and set
6815its initial value equal to "r0", which is the default. Even though the
6816default value is "r0", the practice of adding it to a new recipe makes
6817it harder to forget to bump the variable when you make changes to the
6818recipe in future.
6819
6820If you are sharing a common ``.inc`` file with multiple recipes, you can
6821also use the ``INC_PR`` variable to ensure that the recipes sharing the
6822``.inc`` file are rebuilt when the ``.inc`` file itself is changed. The
6823``.inc`` file must set ``INC_PR`` (initially to "r0"), and all recipes
6824referring to it should set ``PR`` to "${INC_PR}.0" initially,
6825incrementing the last number when the recipe is changed. If the ``.inc``
6826file is changed then its ``INC_PR`` should be incremented.
6827
6828When upgrading the version of a binary package, assuming the ``PV``
6829changes, the ``PR`` variable should be reset to "r0" (or "${INC_PR}.0"
6830if you are using ``INC_PR``).
6831
6832Usually, version increases occur only to binary packages. However, if
6833for some reason ``PV`` changes but does not increase, you can increase
6834the ``PE`` variable (Package Epoch). The ``PE`` variable defaults to
6835"0".
6836
6837Binary package version numbering strives to follow the `Debian Version
6838Field Policy
6839Guidelines <http://www.debian.org/doc/debian-policy/ch-controlfields.html>`__.
6840These guidelines define how versions are compared and what "increasing"
6841a version means.
6842
6843.. _automatically-incrementing-a-binary-package-revision-number:
6844
6845Automatically Incrementing a Package Version Number
6846~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
6847
6848When fetching a repository, BitBake uses the
6849:term:`SRCREV` variable to determine
6850the specific source code revision from which to build. You set the
6851``SRCREV`` variable to
6852:term:`AUTOREV` to cause the
6853OpenEmbedded build system to automatically use the latest revision of
6854the software:
6855::
6856
6857 SRCREV = "${AUTOREV}"
6858
6859Furthermore, you need to reference ``SRCPV`` in ``PV`` in order to
6860automatically update the version whenever the revision of the source
6861code changes. Here is an example:
6862::
6863
6864 PV = "1.0+git${SRCPV}"
6865
6866The OpenEmbedded build system substitutes ``SRCPV`` with the following:
6867::
6868
6869 AUTOINC+source_code_revision
6870
6871The build system replaces the ``AUTOINC``
6872with a number. The number used depends on the state of the PR Service:
6873
6874- If PR Service is enabled, the build system increments the number,
6875 which is similar to the behavior of
6876 :term:`PR`. This behavior results in
6877 linearly increasing package versions, which is desirable. Here is an
6878 example:
6879 ::
6880
6881 hello-world-git_0.0+git0+b6558dd387-r0.0_armv7a-neon.ipk
6882 hello-world-git_0.0+git1+dd2f5c3565-r0.0_armv7a-neon.ipk
6883
6884- If PR Service is not enabled, the build system replaces the
6885 ``AUTOINC`` placeholder with zero (i.e. "0"). This results in
6886 changing the package version since the source revision is included.
6887 However, package versions are not increased linearly. Here is an
6888 example:
6889 ::
6890
6891 hello-world-git_0.0+git0+b6558dd387-r0.0_armv7a-neon.ipk
6892 hello-world-git_0.0+git0+dd2f5c3565-r0.0_armv7a-neon.ipk
6893
6894In summary, the OpenEmbedded build system does not track the history of
6895binary package versions for this purpose. ``AUTOINC``, in this case, is
6896comparable to ``PR``. If PR server is not enabled, ``AUTOINC`` in the
6897package version is simply replaced by "0". If PR server is enabled, the
6898build system keeps track of the package versions and bumps the number
6899when the package revision changes.
6900
6901Handling Optional Module Packaging
6902----------------------------------
6903
6904Many pieces of software split functionality into optional modules (or
6905plugins) and the plugins that are built might depend on configuration
6906options. To avoid having to duplicate the logic that determines what
6907modules are available in your recipe or to avoid having to package each
6908module by hand, the OpenEmbedded build system provides functionality to
6909handle module packaging dynamically.
6910
6911To handle optional module packaging, you need to do two things:
6912
6913- Ensure the module packaging is actually done.
6914
6915- Ensure that any dependencies on optional modules from other recipes
6916 are satisfied by your recipe.
6917
6918Making Sure the Packaging is Done
6919~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
6920
6921To ensure the module packaging actually gets done, you use the
6922``do_split_packages`` function within the ``populate_packages`` Python
6923function in your recipe. The ``do_split_packages`` function searches for
6924a pattern of files or directories under a specified path and creates a
6925package for each one it finds by appending to the
6926:term:`PACKAGES` variable and
6927setting the appropriate values for ``FILES_packagename``,
6928``RDEPENDS_packagename``, ``DESCRIPTION_packagename``, and so forth.
6929Here is an example from the ``lighttpd`` recipe:
6930::
6931
6932 python populate_packages_prepend () {
6933 lighttpd_libdir = d.expand('${libdir}')
6934 do_split_packages(d, lighttpd_libdir, '^mod_(.*).so$',
6935 'lighttpd-module-%s', 'Lighttpd module for %s',
6936 extra_depends='')
6937 }
6938
6939The previous example specifies a number of things in the call to
6940``do_split_packages``.
6941
6942- A directory within the files installed by your recipe through
6943 ``do_install`` in which to search.
6944
6945- A regular expression used to match module files in that directory. In
6946 the example, note the parentheses () that mark the part of the
6947 expression from which the module name should be derived.
6948
6949- A pattern to use for the package names.
6950
6951- A description for each package.
6952
6953- An empty string for ``extra_depends``, which disables the default
6954 dependency on the main ``lighttpd`` package. Thus, if a file in
6955 ``${libdir}`` called ``mod_alias.so`` is found, a package called
6956 ``lighttpd-module-alias`` is created for it and the
6957 :term:`DESCRIPTION` is set to
6958 "Lighttpd module for alias".
6959
6960Often, packaging modules is as simple as the previous example. However,
6961more advanced options exist that you can use within
6962``do_split_packages`` to modify its behavior. And, if you need to, you
6963can add more logic by specifying a hook function that is called for each
6964package. It is also perfectly acceptable to call ``do_split_packages``
6965multiple times if you have more than one set of modules to package.
6966
6967For more examples that show how to use ``do_split_packages``, see the
6968``connman.inc`` file in the ``meta/recipes-connectivity/connman/``
6969directory of the ``poky`` :ref:`source repository <yocto-project-repositories>`. You can
6970also find examples in ``meta/classes/kernel.bbclass``.
6971
6972Following is a reference that shows ``do_split_packages`` mandatory and
6973optional arguments:
6974::
6975
6976 Mandatory arguments
6977
6978 root
6979 The path in which to search
6980 file_regex
6981 Regular expression to match searched files.
6982 Use parentheses () to mark the part of this
6983 expression that should be used to derive the
6984 module name (to be substituted where %s is
6985 used in other function arguments as noted below)
6986 output_pattern
6987 Pattern to use for the package names. Must
6988 include %s.
6989 description
6990 Description to set for each package. Must
6991 include %s.
6992
6993 Optional arguments
6994
6995 postinst
6996 Postinstall script to use for all packages
6997 (as a string)
6998 recursive
6999 True to perform a recursive search - default
7000 False
7001 hook
7002 A hook function to be called for every match.
7003 The function will be called with the following
7004 arguments (in the order listed):
7005
7006 f
7007 Full path to the file/directory match
7008 pkg
7009 The package name
7010 file_regex
7011 As above
7012 output_pattern
7013 As above
7014 modulename
7015 The module name derived using file_regex
7016 extra_depends
7017 Extra runtime dependencies (RDEPENDS) to be
7018 set for all packages. The default value of None
7019 causes a dependency on the main package
7020 (${PN}) - if you do not want this, pass empty
7021 string '' for this parameter.
7022 aux_files_pattern
7023 Extra item(s) to be added to FILES for each
7024 package. Can be a single string item or a list
7025 of strings for multiple items. Must include %s.
7026 postrm
7027 postrm script to use for all packages (as a
7028 string)
7029 allow_dirs
7030 True to allow directories to be matched -
7031 default False
7032 prepend
7033 If True, prepend created packages to PACKAGES
7034 instead of the default False which appends them
7035 match_path
7036 match file_regex on the whole relative path to
7037 the root rather than just the file name
7038 aux_files_pattern_verbatim
7039 Extra item(s) to be added to FILES for each
7040 package, using the actual derived module name
7041 rather than converting it to something legal
7042 for a package name. Can be a single string item
7043 or a list of strings for multiple items. Must
7044 include %s.
7045 allow_links
7046 True to allow symlinks to be matched - default
7047 False
7048 summary
7049 Summary to set for each package. Must include %s;
7050 defaults to description if not set.
7051
7052
7053
7054Satisfying Dependencies
7055~~~~~~~~~~~~~~~~~~~~~~~
7056
7057The second part for handling optional module packaging is to ensure that
7058any dependencies on optional modules from other recipes are satisfied by
7059your recipe. You can be sure these dependencies are satisfied by using
7060the :term:`PACKAGES_DYNAMIC`
7061variable. Here is an example that continues with the ``lighttpd`` recipe
7062shown earlier:
7063::
7064
7065 PACKAGES_DYNAMIC = "lighttpd-module-.*"
7066
7067The name
7068specified in the regular expression can of course be anything. In this
7069example, it is ``lighttpd-module-`` and is specified as the prefix to
7070ensure that any :term:`RDEPENDS` and
7071:term:`RRECOMMENDS` on a package
7072name starting with the prefix are satisfied during build time. If you
7073are using ``do_split_packages`` as described in the previous section,
7074the value you put in ``PACKAGES_DYNAMIC`` should correspond to the name
7075pattern specified in the call to ``do_split_packages``.
7076
7077Using Runtime Package Management
7078--------------------------------
7079
7080During a build, BitBake always transforms a recipe into one or more
7081packages. For example, BitBake takes the ``bash`` recipe and produces a
7082number of packages (e.g. ``bash``, ``bash-bashbug``,
7083``bash-completion``, ``bash-completion-dbg``, ``bash-completion-dev``,
7084``bash-completion-extra``, ``bash-dbg``, and so forth). Not all
7085generated packages are included in an image.
7086
7087In several situations, you might need to update, add, remove, or query
7088the packages on a target device at runtime (i.e. without having to
7089generate a new image). Examples of such situations include:
7090
7091- You want to provide in-the-field updates to deployed devices (e.g.
7092 security updates).
7093
7094- You want to have a fast turn-around development cycle for one or more
7095 applications that run on your device.
7096
7097- You want to temporarily install the "debug" packages of various
7098 applications on your device so that debugging can be greatly improved
7099 by allowing access to symbols and source debugging.
7100
7101- You want to deploy a more minimal package selection of your device
7102 but allow in-the-field updates to add a larger selection for
7103 customization.
7104
7105In all these situations, you have something similar to a more
7106traditional Linux distribution in that in-field devices are able to
7107receive pre-compiled packages from a server for installation or update.
7108Being able to install these packages on a running, in-field device is
7109what is termed "runtime package management".
7110
7111In order to use runtime package management, you need a host or server
7112machine that serves up the pre-compiled packages plus the required
7113metadata. You also need package manipulation tools on the target. The
7114build machine is a likely candidate to act as the server. However, that
7115machine does not necessarily have to be the package server. The build
7116machine could push its artifacts to another machine that acts as the
7117server (e.g. Internet-facing). In fact, doing so is advantageous for a
7118production environment as getting the packages away from the development
7119system's build directory prevents accidental overwrites.
7120
7121A simple build that targets just one device produces more than one
7122package database. In other words, the packages produced by a build are
7123separated out into a couple of different package groupings based on
7124criteria such as the target's CPU architecture, the target board, or the
7125C library used on the target. For example, a build targeting the
7126``qemux86`` device produces the following three package databases:
7127``noarch``, ``i586``, and ``qemux86``. If you wanted your ``qemux86``
7128device to be aware of all the packages that were available to it, you
7129would need to point it to each of these databases individually. In a
7130similar way, a traditional Linux distribution usually is configured to
7131be aware of a number of software repositories from which it retrieves
7132packages.
7133
7134Using runtime package management is completely optional and not required
7135for a successful build or deployment in any way. But if you want to make
7136use of runtime package management, you need to do a couple things above
7137and beyond the basics. The remainder of this section describes what you
7138need to do.
7139
7140.. _runtime-package-management-build:
7141
7142Build Considerations
7143~~~~~~~~~~~~~~~~~~~~
7144
7145This section describes build considerations of which you need to be
7146aware in order to provide support for runtime package management.
7147
7148When BitBake generates packages, it needs to know what format or formats
7149to use. In your configuration, you use the
7150:term:`PACKAGE_CLASSES`
7151variable to specify the format:
7152
71531. Open the ``local.conf`` file inside your
7154 :term:`Build Directory` (e.g.
7155 ``~/poky/build/conf/local.conf``).
7156
71572. Select the desired package format as follows:
7158 ::
7159
7160 PACKAGE_CLASSES ?= "package_packageformat"
7161
7162 where packageformat can be "ipk", "rpm",
7163 "deb", or "tar" which are the supported package formats.
7164
7165 .. note::
7166
7167 Because the Yocto Project supports four different package formats,
7168 you can set the variable with more than one argument. However, the
7169 OpenEmbedded build system only uses the first argument when
7170 creating an image or Software Development Kit (SDK).
7171
7172If you would like your image to start off with a basic package database
7173containing the packages in your current build as well as to have the
7174relevant tools available on the target for runtime package management,
7175you can include "package-management" in the
7176:term:`IMAGE_FEATURES`
7177variable. Including "package-management" in this configuration variable
7178ensures that when the image is assembled for your target, the image
7179includes the currently-known package databases as well as the
7180target-specific tools required for runtime package management to be
7181performed on the target. However, this is not strictly necessary. You
7182could start your image off without any databases but only include the
7183required on-target package tool(s). As an example, you could include
7184"opkg" in your
7185:term:`IMAGE_INSTALL` variable
7186if you are using the IPK package format. You can then initialize your
7187target's package database(s) later once your image is up and running.
7188
7189Whenever you perform any sort of build step that can potentially
7190generate a package or modify existing package, it is always a good idea
7191to re-generate the package index after the build by using the following
7192command:
7193::
7194
7195 $ bitbake package-index
7196
7197It might be tempting to build the
7198package and the package index at the same time with a command such as
7199the following:
7200::
7201
7202 $ bitbake some-package package-index
7203
7204Do not do this as
7205BitBake does not schedule the package index for after the completion of
7206the package you are building. Consequently, you cannot be sure of the
7207package index including information for the package you just built.
7208Thus, be sure to run the package update step separately after building
7209any packages.
7210
7211You can use the
7212:term:`PACKAGE_FEED_ARCHS`,
7213:term:`PACKAGE_FEED_BASE_PATHS`,
7214and
7215:term:`PACKAGE_FEED_URIS`
7216variables to pre-configure target images to use a package feed. If you
7217do not define these variables, then manual steps as described in the
7218subsequent sections are necessary to configure the target. You should
7219set these variables before building the image in order to produce a
7220correctly configured image.
7221
7222When your build is complete, your packages reside in the
7223``${TMPDIR}/deploy/packageformat`` directory. For example, if
7224``${``\ :term:`TMPDIR`\ ``}`` is
7225``tmp`` and your selected package type is RPM, then your RPM packages
7226are available in ``tmp/deploy/rpm``.
7227
7228.. _runtime-package-management-server:
7229
7230Host or Server Machine Setup
7231~~~~~~~~~~~~~~~~~~~~~~~~~~~~
7232
7233Although other protocols are possible, a server using HTTP typically
7234serves packages. If you want to use HTTP, then set up and configure a
7235web server such as Apache 2, lighttpd, or SimpleHTTPServer on the
7236machine serving the packages.
7237
7238To keep things simple, this section describes how to set up a
7239SimpleHTTPServer web server to share package feeds from the developer's
7240machine. Although this server might not be the best for a production
7241environment, the setup is simple and straight forward. Should you want
7242to use a different server more suited for production (e.g. Apache 2,
7243Lighttpd, or Nginx), take the appropriate steps to do so.
7244
7245From within the build directory where you have built an image based on
7246your packaging choice (i.e. the
7247:term:`PACKAGE_CLASSES`
7248setting), simply start the server. The following example assumes a build
7249directory of ``~/poky/build/tmp/deploy/rpm`` and a ``PACKAGE_CLASSES``
7250setting of "package_rpm":
7251::
7252
7253 $ cd ~/poky/build/tmp/deploy/rpm
7254 $ python -m SimpleHTTPServer
7255
7256.. _runtime-package-management-target:
7257
7258Target Setup
7259~~~~~~~~~~~~
7260
7261Setting up the target differs depending on the package management
7262system. This section provides information for RPM, IPK, and DEB.
7263
7264.. _runtime-package-management-target-rpm:
7265
7266Using RPM
7267^^^^^^^^^
7268
7269The `Dandified Packaging
7270Tool <https://en.wikipedia.org/wiki/DNF_(software)>`__ (DNF) performs
7271runtime package management of RPM packages. In order to use DNF for
7272runtime package management, you must perform an initial setup on the
7273target machine for cases where the ``PACKAGE_FEED_*`` variables were not
7274set as part of the image that is running on the target. This means if
7275you built your image and did not not use these variables as part of the
7276build and your image is now running on the target, you need to perform
7277the steps in this section if you want to use runtime package management.
7278
7279.. note::
7280
7281 For information on the PACKAGE_FEED_* variables, see
7282 PACKAGE_FEED_ARCHS
7283 ,
7284 PACKAGE_FEED_BASE_PATHS
7285 , and
7286 PACKAGE_FEED_URIS
7287 in the Yocto Project Reference Manual variables glossary.
7288
7289On the target, you must inform DNF that package databases are available.
7290You do this by creating a file named
7291``/etc/yum.repos.d/oe-packages.repo`` and defining the ``oe-packages``.
7292
7293As an example, assume the target is able to use the following package
7294databases: ``all``, ``i586``, and ``qemux86`` from a server named
7295``my.server``. The specifics for setting up the web server are up to
7296you. The critical requirement is that the URIs in the target repository
7297configuration point to the correct remote location for the feeds.
7298
7299.. note::
7300
7301 For development purposes, you can point the web server to the build
7302 system's
7303 deploy
7304 directory. However, for production use, it is better to copy the
7305 package directories to a location outside of the build area and use
7306 that location. Doing so avoids situations where the build system
7307 overwrites or changes the
7308 deploy
7309 directory.
7310
7311When telling DNF where to look for the package databases, you must
7312declare individual locations per architecture or a single location used
7313for all architectures. You cannot do both:
7314
7315- *Create an Explicit List of Architectures:* Define individual base
7316 URLs to identify where each package database is located:
7317 ::
7318
7319 [oe-packages]
7320 baseurl=http://my.server/rpm/i586 http://my.server/rpm/qemux86 http://my.server/rpm/all
7321
7322 This example
7323 informs DNF about individual package databases for all three
7324 architectures.
7325
7326- *Create a Single (Full) Package Index:* Define a single base URL that
7327 identifies where a full package database is located:
7328 ::
7329
7330 [oe-packages]
7331 baseurl=http://my.server/rpm
7332
7333 This example informs DNF about a single
7334 package database that contains all the package index information for
7335 all supported architectures.
7336
7337Once you have informed DNF where to find the package databases, you need
7338to fetch them:
7339::
7340
7341 # dnf makecache
7342
7343DNF is now able to find, install, and
7344upgrade packages from the specified repository or repositories.
7345
7346.. note::
7347
7348 See the
7349 DNF documentation
7350 for additional information.
7351
7352.. _runtime-package-management-target-ipk:
7353
7354Using IPK
7355^^^^^^^^^
7356
7357The ``opkg`` application performs runtime package management of IPK
7358packages. You must perform an initial setup for ``opkg`` on the target
7359machine if the
7360:term:`PACKAGE_FEED_ARCHS`,
7361:term:`PACKAGE_FEED_BASE_PATHS`,
7362and
7363:term:`PACKAGE_FEED_URIS`
7364variables have not been set or the target image was built before the
7365variables were set.
7366
7367The ``opkg`` application uses configuration files to find available
7368package databases. Thus, you need to create a configuration file inside
7369the ``/etc/opkg/`` direction, which informs ``opkg`` of any repository
7370you want to use.
7371
7372As an example, suppose you are serving packages from a ``ipk/``
7373directory containing the ``i586``, ``all``, and ``qemux86`` databases
7374through an HTTP server named ``my.server``. On the target, create a
7375configuration file (e.g. ``my_repo.conf``) inside the ``/etc/opkg/``
7376directory containing the following:
7377::
7378
7379 src/gz all http://my.server/ipk/all
7380 src/gz i586 http://my.server/ipk/i586
7381 src/gz qemux86 http://my.server/ipk/qemux86
7382
7383Next, instruct ``opkg`` to fetch the
7384repository information: # opkg update The ``opkg`` application is now
7385able to find, install, and upgrade packages from the specified
7386repository.
7387
7388.. _runtime-package-management-target-deb:
7389
7390Using DEB
7391^^^^^^^^^
7392
7393The ``apt`` application performs runtime package management of DEB
7394packages. This application uses a source list file to find available
7395package databases. You must perform an initial setup for ``apt`` on the
7396target machine if the
7397:term:`PACKAGE_FEED_ARCHS`,
7398:term:`PACKAGE_FEED_BASE_PATHS`,
7399and
7400:term:`PACKAGE_FEED_URIS`
7401variables have not been set or the target image was built before the
7402variables were set.
7403
7404To inform ``apt`` of the repository you want to use, you might create a
7405list file (e.g. ``my_repo.list``) inside the
7406``/etc/apt/sources.list.d/`` directory. As an example, suppose you are
7407serving packages from a ``deb/`` directory containing the ``i586``,
7408``all``, and ``qemux86`` databases through an HTTP server named
7409``my.server``. The list file should contain:
7410::
7411
7412 deb http://my.server/deb/all ./
7413 deb http://my.server/deb/i586 ./
7414 deb http://my.server/deb/qemux86 ./
7415
7416Next, instruct the ``apt`` application
7417to fetch the repository information:
7418::
7419
7420 # apt-get update
7421
7422After this step,
7423``apt`` is able to find, install, and upgrade packages from the
7424specified repository.
7425
7426Generating and Using Signed Packages
7427------------------------------------
7428
7429In order to add security to RPM packages used during a build, you can
7430take steps to securely sign them. Once a signature is verified, the
7431OpenEmbedded build system can use the package in the build. If security
7432fails for a signed package, the build system aborts the build.
7433
7434This section describes how to sign RPM packages during a build and how
7435to use signed package feeds (repositories) when doing a build.
7436
7437Signing RPM Packages
7438~~~~~~~~~~~~~~~~~~~~
7439
7440To enable signing RPM packages, you must set up the following
7441configurations in either your ``local.config`` or ``distro.config``
7442file:
7443::
7444
7445 # Inherit sign_rpm.bbclass to enable signing functionality
7446 INHERIT += " sign_rpm"
7447 # Define the GPG key that will be used for signing.
7448 RPM_GPG_NAME = "key_name"
7449 # Provide passphrase for the key
7450 RPM_GPG_PASSPHRASE = "passphrase"
7451
7452.. note::
7453
7454 Be sure to supply appropriate values for both
7455 key_name
7456 and
7457 passphrase
7458
7459Aside from the ``RPM_GPG_NAME`` and ``RPM_GPG_PASSPHRASE`` variables in
7460the previous example, two optional variables related to signing exist:
7461
7462- *GPG_BIN:* Specifies a ``gpg`` binary/wrapper that is executed
7463 when the package is signed.
7464
7465- *GPG_PATH:* Specifies the ``gpg`` home directory used when the
7466 package is signed.
7467
7468Processing Package Feeds
7469~~~~~~~~~~~~~~~~~~~~~~~~
7470
7471In addition to being able to sign RPM packages, you can also enable
7472signed package feeds for IPK and RPM packages.
7473
7474The steps you need to take to enable signed package feed use are similar
7475to the steps used to sign RPM packages. You must define the following in
7476your ``local.config`` or ``distro.config`` file:
7477::
7478
7479 INHERIT += "sign_package_feed"
7480 PACKAGE_FEED_GPG_NAME = "key_name"
7481 PACKAGE_FEED_GPG_PASSPHRASE_FILE = "path_to_file_containing_passphrase"
7482
7483For signed package feeds, the passphrase must exist in a separate file,
7484which is pointed to by the ``PACKAGE_FEED_GPG_PASSPHRASE_FILE``
7485variable. Regarding security, keeping a plain text passphrase out of the
7486configuration is more secure.
7487
7488Aside from the ``PACKAGE_FEED_GPG_NAME`` and
7489``PACKAGE_FEED_GPG_PASSPHRASE_FILE`` variables, three optional variables
7490related to signed package feeds exist:
7491
7492- *GPG_BIN* Specifies a ``gpg`` binary/wrapper that is executed
7493 when the package is signed.
7494
7495- *GPG_PATH:* Specifies the ``gpg`` home directory used when the
7496 package is signed.
7497
7498- *PACKAGE_FEED_GPG_SIGNATURE_TYPE:* Specifies the type of ``gpg``
7499 signature. This variable applies only to RPM and IPK package feeds.
7500 Allowable values for the ``PACKAGE_FEED_GPG_SIGNATURE_TYPE`` are
7501 "ASC", which is the default and specifies ascii armored, and "BIN",
7502 which specifies binary.
7503
7504Testing Packages With ptest
7505---------------------------
7506
7507A Package Test (ptest) runs tests against packages built by the
7508OpenEmbedded build system on the target machine. A ptest contains at
7509least two items: the actual test, and a shell script (``run-ptest``)
7510that starts the test. The shell script that starts the test must not
7511contain the actual test - the script only starts the test. On the other
7512hand, the test can be anything from a simple shell script that runs a
7513binary and checks the output to an elaborate system of test binaries and
7514data files.
7515
7516The test generates output in the format used by Automake:
7517::
7518
7519 result: testname
7520
7521where the result can be ``PASS``, ``FAIL``, or ``SKIP``, and
7522the testname can be any identifying string.
7523
7524For a list of Yocto Project recipes that are already enabled with ptest,
7525see the :yocto_wiki:`Ptest </wiki/Ptest>` wiki page.
7526
7527.. note::
7528
7529 A recipe is "ptest-enabled" if it inherits the
7530 ptest
7531 class.
7532
7533Adding ptest to Your Build
7534~~~~~~~~~~~~~~~~~~~~~~~~~~
7535
7536To add package testing to your build, add the
7537:term:`DISTRO_FEATURES` and
7538:term:`EXTRA_IMAGE_FEATURES`
7539variables to your ``local.conf`` file, which is found in the
7540:term:`Build Directory`:
7541::
7542
7543 DISTRO_FEATURES_append = " ptest"
7544 EXTRA_IMAGE_FEATURES += "ptest-pkgs"
7545
7546Once your build is complete, the ptest files are installed into the
7547``/usr/lib/package/ptest`` directory within the image, where ``package``
7548is the name of the package.
7549
7550Running ptest
7551~~~~~~~~~~~~~
7552
7553The ``ptest-runner`` package installs a shell script that loops through
7554all installed ptest test suites and runs them in sequence. Consequently,
7555you might want to add this package to your image.
7556
7557Getting Your Package Ready
7558~~~~~~~~~~~~~~~~~~~~~~~~~~
7559
7560In order to enable a recipe to run installed ptests on target hardware,
7561you need to prepare the recipes that build the packages you want to
7562test. Here is what you have to do for each recipe:
7563
7564- *Be sure the recipe inherits
7565 the*\ :ref:`ptest <ref-classes-ptest>`\ *class:*
7566 Include the following line in each recipe:
7567 ::
7568
7569 inherit ptest
7570
7571- *Create run-ptest:* This script starts your test. Locate the
7572 script where you will refer to it using
7573 :term:`SRC_URI`. Here is an
7574 example that starts a test for ``dbus``:
7575 ::
7576
7577 #!/bin/sh
7578 cd test
7579 make -k runtest-TESTS
7580
7581- *Ensure dependencies are met:* If the test adds build or runtime
7582 dependencies that normally do not exist for the package (such as
7583 requiring "make" to run the test suite), use the
7584 :term:`DEPENDS` and
7585 :term:`RDEPENDS` variables in
7586 your recipe in order for the package to meet the dependencies. Here
7587 is an example where the package has a runtime dependency on "make":
7588 ::
7589
7590 RDEPENDS_${PN}-ptest += "make"
7591
7592- *Add a function to build the test suite:* Not many packages support
7593 cross-compilation of their test suites. Consequently, you usually
7594 need to add a cross-compilation function to the package.
7595
7596 Many packages based on Automake compile and run the test suite by
7597 using a single command such as ``make check``. However, the host
7598 ``make check`` builds and runs on the same computer, while
7599 cross-compiling requires that the package is built on the host but
7600 executed for the target architecture (though often, as in the case
7601 for ptest, the execution occurs on the host). The built version of
7602 Automake that ships with the Yocto Project includes a patch that
7603 separates building and execution. Consequently, packages that use the
7604 unaltered, patched version of ``make check`` automatically
7605 cross-compiles.
7606
7607 Regardless, you still must add a ``do_compile_ptest`` function to
7608 build the test suite. Add a function similar to the following to your
7609 recipe:
7610 ::
7611
7612 do_compile_ptest() {
7613 oe_runmake buildtest-TESTS
7614 }
7615
7616- *Ensure special configurations are set:* If the package requires
7617 special configurations prior to compiling the test code, you must
7618 insert a ``do_configure_ptest`` function into the recipe.
7619
7620- *Install the test suite:* The ``ptest`` class automatically copies
7621 the file ``run-ptest`` to the target and then runs make
7622 ``install-ptest`` to run the tests. If this is not enough, you need
7623 to create a ``do_install_ptest`` function and make sure it gets
7624 called after the "make install-ptest" completes.
7625
7626Creating Node Package Manager (NPM) Packages
7627--------------------------------------------
7628
7629`NPM <https://en.wikipedia.org/wiki/Npm_(software)>`__ is a package
7630manager for the JavaScript programming language. The Yocto Project
7631supports the NPM :ref:`fetcher <bitbake:bb-fetchers>`. You can
7632use this fetcher in combination with
7633:doc:```devtool`` <../ref-manual/ref-devtool-reference>` to create
7634recipes that produce NPM packages.
7635
7636Two workflows exist that allow you to create NPM packages using
7637``devtool``: the NPM registry modules method and the NPM project code
7638method.
7639
7640.. note::
7641
7642 While it is possible to create NPM recipes manually, using
7643 devtool
7644 is far simpler.
7645
7646Additionally, some requirements and caveats exist.
7647
7648.. _npm-package-creation-requirements:
7649
7650Requirements and Caveats
7651~~~~~~~~~~~~~~~~~~~~~~~~
7652
7653You need to be aware of the following before using ``devtool`` to create
7654NPM packages:
7655
7656- Of the two methods that you can use ``devtool`` to create NPM
7657 packages, the registry approach is slightly simpler. However, you
7658 might consider the project approach because you do not have to
7659 publish your module in the NPM registry
7660 (`npm-registry <https://docs.npmjs.com/misc/registry>`_), which
7661 is NPM's public registry.
7662
7663- Be familiar with
7664 :doc:```devtool`` <../ref-manual/ref-devtool-reference>`.
7665
7666- The NPM host tools need the native ``nodejs-npm`` package, which is
7667 part of the OpenEmbedded environment. You need to get the package by
7668 cloning the https://github.com/openembedded/meta-openembedded
7669 repository out of GitHub. Be sure to add the path to your local copy
7670 to your ``bblayers.conf`` file.
7671
7672- ``devtool`` cannot detect native libraries in module dependencies.
7673 Consequently, you must manually add packages to your recipe.
7674
7675- While deploying NPM packages, ``devtool`` cannot determine which
7676 dependent packages are missing on the target (e.g. the node runtime
7677 ``nodejs``). Consequently, you need to find out what files are
7678 missing and be sure they are on the target.
7679
7680- Although you might not need NPM to run your node package, it is
7681 useful to have NPM on your target. The NPM package name is
7682 ``nodejs-npm``.
7683
7684.. _npm-using-the-registry-modules-method:
7685
7686Using the Registry Modules Method
7687~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
7688
7689This section presents an example that uses the ``cute-files`` module,
7690which is a file browser web application.
7691
7692.. note::
7693
7694 You must know the
7695 cute-files
7696 module version.
7697
7698The first thing you need to do is use ``devtool`` and the NPM fetcher to
7699create the recipe:
7700::
7701
7702 $ devtool add "npm://registry.npmjs.org;package=cute-files;version=1.0.2"
7703
7704The
7705``devtool add`` command runs ``recipetool create`` and uses the same
7706fetch URI to download each dependency and capture license details where
7707possible. The result is a generated recipe.
7708
7709The recipe file is fairly simple and contains every license that
7710``recipetool`` finds and includes the licenses in the recipe's
7711:term:`LIC_FILES_CHKSUM`
7712variables. You need to examine the variables and look for those with
7713"unknown" in the :term:`LICENSE`
7714field. You need to track down the license information for "unknown"
7715modules and manually add the information to the recipe.
7716
7717``recipetool`` creates a "shrinkwrap" file for your recipe. Shrinkwrap
7718files capture the version of all dependent modules. Many packages do not
7719provide shrinkwrap files. ``recipetool`` create a shrinkwrap file as it
7720runs.
7721
7722.. note::
7723
7724 A package is created for each sub-module. This policy is the only
7725 practical way to have the licenses for all of the dependencies
7726 represented in the license manifest of the image.
7727
7728The ``devtool edit-recipe`` command lets you take a look at the recipe:
7729::
7730
7731 $ devtool edit-recipe cute-files
7732 SUMMARY = "Turn any folder on your computer into a cute file browser, available on the local network."
7733 LICENSE = "MIT & ISC & Unknown"
7734 LIC_FILES_CHKSUM = "file://LICENSE;md5=71d98c0a1db42956787b1909c74a86ca \
7735 file://node_modules/toidentifier/LICENSE;md5=1a261071a044d02eb6f2bb47f51a3502 \
7736 file://node_modules/debug/LICENSE;md5=ddd815a475e7338b0be7a14d8ee35a99 \
7737 ...
7738 SRC_URI = " \
7739 npm://registry.npmjs.org/;package=cute-files;version=${PV} \
7740 npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \
7741 "
7742 S = "${WORKDIR}/npm"
7743 inherit npm LICENSE_${PN} = "MIT"
7744 LICENSE_${PN}-accepts = "MIT"
7745 LICENSE_${PN}-array-flatten = "MIT"
7746 ...
7747 LICENSE_${PN}-vary = "MIT"
7748
7749Three key points exist in the previous example:
7750
7751- :term:`SRC_URI` uses the NPM
7752 scheme so that the NPM fetcher is used.
7753
7754- ``recipetool`` collects all the license information. If a
7755 sub-module's license is unavailable, the sub-module's name appears in
7756 the comments.
7757
7758- The ``inherit npm`` statement causes the
7759 :ref:`npm <ref-classes-npm>` class to package
7760 up all the modules.
7761
7762You can run the following command to build the ``cute-files`` package:
7763::
7764
7765 $ devtool build cute-files
7766
7767Remember that ``nodejs`` must be installed on
7768the target before your package.
7769
7770Assuming 192.168.7.2 for the target's IP address, use the following
7771command to deploy your package:
7772::
7773
7774 $ devtool deploy-target -s cute-files root@192.168.7.2
7775
7776Once the package is installed on the target, you can
7777test the application:
7778
7779.. note::
7780
7781 Because of a know issue, you cannot simply run
7782 cute-files
7783 as you would if you had run
7784 npm install
7785 .
7786
7787::
7788
7789 $ cd /usr/lib/node_modules/cute-files
7790 $ node cute-files.js
7791
7792On a browser,
7793go to ``http://192.168.7.2:3000`` and you see the following:
7794
7795.. image:: figures/cute-files-npm-example.png
7796 :align: center
7797
7798You can find the recipe in ``workspace/recipes/cute-files``. You can use
7799the recipe in any layer you choose.
7800
7801.. _npm-using-the-npm-projects-method:
7802
7803Using the NPM Projects Code Method
7804~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
7805
7806Although it is useful to package modules already in the NPM registry,
7807adding ``node.js`` projects under development is a more common developer
7808use case.
7809
7810This section covers the NPM projects code method, which is very similar
7811to the "registry" approach described in the previous section. In the NPM
7812projects method, you provide ``devtool`` with an URL that points to the
7813source files.
7814
7815Replicating the same example, (i.e. ``cute-files``) use the following
7816command:
7817::
7818
7819 $ devtool add https://github.com/martinaglv/cute-files.git
7820
7821The
7822recipe this command generates is very similar to the recipe created in
7823the previous section. However, the ``SRC_URI`` looks like the following:
7824::
7825
7826 SRC_URI = " \
7827 git://github.com/martinaglv/cute-files.git;protocol=https \
7828 npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \
7829 "
7830
7831In this example,
7832the main module is taken from the Git repository and dependents are
7833taken from the NPM registry. Other than those differences, the recipe is
7834basically the same between the two methods. You can build and deploy the
7835package exactly as described in the previous section that uses the
7836registry modules method.
7837
7838Adding custom metadata to packages
7839----------------------------------
7840
7841The variable
7842:term:`PACKAGE_ADD_METADATA`
7843can be used to add additional metadata to packages. This is reflected in
7844the package control/spec file. To take the ipk format for example, the
7845CONTROL file stored inside would contain the additional metadata as
7846additional lines.
7847
7848The variable can be used in multiple ways, including using suffixes to
7849set it for a specific package type and/or package. Note that the order
7850of precedence is the same as this list:
7851
7852- ``PACKAGE_ADD_METADATA_<PKGTYPE>_<PN>``
7853
7854- ``PACKAGE_ADD_METADATA_<PKGTYPE>``
7855
7856- ``PACKAGE_ADD_METADATA_<PN>``
7857
7858- ``PACKAGE_ADD_METADATA``
7859
7860<PKGTYPE> is a parameter and expected to be a distinct name of specific
7861package type:
7862
7863- IPK for .ipk packages
7864
7865- DEB for .deb packages
7866
7867- RPM for .rpm packages
7868
7869<PN> is a parameter and expected to be a package name.
7870
7871The variable can contain multiple [one-line] metadata fields separated
7872by the literal sequence '\n'. The separator can be redefined using the
7873variable flag ``separator``.
7874
7875The following is an example that adds two custom fields for ipk
7876packages: PACKAGE_ADD_METADATA_IPK = "Vendor: CustomIpk\nGroup:
7877Applications/Spreadsheets"
7878
7879Efficiently Fetching Source Files During a Build
7880================================================
7881
7882The OpenEmbedded build system works with source files located through
7883the :term:`SRC_URI` variable. When
7884you build something using BitBake, a big part of the operation is
7885locating and downloading all the source tarballs. For images,
7886downloading all the source for various packages can take a significant
7887amount of time.
7888
7889This section shows you how you can use mirrors to speed up fetching
7890source files and how you can pre-fetch files all of which leads to more
7891efficient use of resources and time.
7892
7893Setting up Effective Mirrors
7894----------------------------
7895
7896A good deal that goes into a Yocto Project build is simply downloading
7897all of the source tarballs. Maybe you have been working with another
7898build system (OpenEmbedded or Angstrom) for which you have built up a
7899sizable directory of source tarballs. Or, perhaps someone else has such
7900a directory for which you have read access. If so, you can save time by
7901adding statements to your configuration file so that the build process
7902checks local directories first for existing tarballs before checking the
7903Internet.
7904
7905Here is an efficient way to set it up in your ``local.conf`` file:
7906::
7907
7908 SOURCE_MIRROR_URL ?= "file:///home/you/your-download-dir/"
7909 INHERIT += "own-mirrors"
7910 BB_GENERATE_MIRROR_TARBALLS = "1"
7911 # BB_NO_NETWORK = "1"
7912
7913In the previous example, the
7914:term:`BB_GENERATE_MIRROR_TARBALLS`
7915variable causes the OpenEmbedded build system to generate tarballs of
7916the Git repositories and store them in the
7917:term:`DL_DIR` directory. Due to
7918performance reasons, generating and storing these tarballs is not the
7919build system's default behavior.
7920
7921You can also use the
7922:term:`PREMIRRORS` variable. For
7923an example, see the variable's glossary entry in the Yocto Project
7924Reference Manual.
7925
7926Getting Source Files and Suppressing the Build
7927----------------------------------------------
7928
7929Another technique you can use to ready yourself for a successive string
7930of build operations, is to pre-fetch all the source files without
7931actually starting a build. This technique lets you work through any
7932download issues and ultimately gathers all the source files into your
7933download directory :ref:`structure-build-downloads`,
7934which is located with :term:`DL_DIR`.
7935
7936Use the following BitBake command form to fetch all the necessary
7937sources without starting the build:
7938::
7939
7940 $ bitbake target --runall=fetch
7941
7942This
7943variation of the BitBake command guarantees that you have all the
7944sources for that BitBake target should you disconnect from the Internet
7945and want to do the build later offline.
7946
7947Selecting an Initialization Manager
7948===================================
7949
7950By default, the Yocto Project uses SysVinit as the initialization
7951manager. However, support also exists for systemd, which is a full
7952replacement for init with parallel starting of services, reduced shell
7953overhead and other features that are used by many distributions.
7954
7955Within the system, SysVinit treats system components as services. These
7956services are maintained as shell scripts stored in the ``/etc/init.d/``
7957directory. Services organize into different run levels. This
7958organization is maintained by putting links to the services in the
7959``/etc/rcN.d/`` directories, where N/ is one of the following options:
7960"S", "0", "1", "2", "3", "4", "5", or "6".
7961
7962.. note::
7963
7964 Each runlevel has a dependency on the previous runlevel. This
7965 dependency allows the services to work properly.
7966
7967In comparison, systemd treats components as units. Using units is a
7968broader concept as compared to using a service. A unit includes several
7969different types of entities. Service is one of the types of entities.
7970The runlevel concept in SysVinit corresponds to the concept of a target
7971in systemd, where target is also a type of supported unit.
7972
7973In a SysVinit-based system, services load sequentially (i.e. one by one)
7974during and parallelization is not supported. With systemd, services
7975start in parallel. Needless to say, the method can have an impact on
7976system startup performance.
7977
7978If you want to use SysVinit, you do not have to do anything. But, if you
7979want to use systemd, you must take some steps as described in the
7980following sections.
7981
7982Using systemd Exclusively
7983-------------------------
7984
7985Set these variables in your distribution configuration file as follows:
7986::
7987
7988 DISTRO_FEATURES_append = " systemd"
7989 VIRTUAL-RUNTIME_init_manager = "systemd"
7990
7991You can also prevent the SysVinit distribution feature from
7992being automatically enabled as follows:
7993::
7994
7995 DISTRO_FEATURES_BACKFILL_CONSIDERED = "sysvinit"
7996
7997Doing so removes any
7998redundant SysVinit scripts.
7999
8000To remove initscripts from your image altogether, set this variable
8001also:
8002::
8003
8004 VIRTUAL-RUNTIME_initscripts = ""
8005
8006For information on the backfill variable, see
8007:term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`.
8008
8009Using systemd for the Main Image and Using SysVinit for the Rescue Image
8010------------------------------------------------------------------------
8011
8012Set these variables in your distribution configuration file as follows:
8013::
8014
8015 DISTRO_FEATURES_append = " systemd"
8016 VIRTUAL-RUNTIME_init_manager = "systemd"
8017
8018Doing so causes your main image to use the
8019``packagegroup-core-boot.bb`` recipe and systemd. The rescue/minimal
8020image cannot use this package group. However, it can install SysVinit
8021and the appropriate packages will have support for both systemd and
8022SysVinit.
8023
8024.. _selecting-dev-manager:
8025
8026Selecting a Device Manager
8027==========================
8028
8029The Yocto Project provides multiple ways to manage the device manager
8030(``/dev``):
8031
8032- Persistent and Pre-Populated\ ``/dev``: For this case, the ``/dev``
8033 directory is persistent and the required device nodes are created
8034 during the build.
8035
8036- Use ``devtmpfs`` with a Device Manager: For this case, the ``/dev``
8037 directory is provided by the kernel as an in-memory file system and
8038 is automatically populated by the kernel at runtime. Additional
8039 configuration of device nodes is done in user space by a device
8040 manager like ``udev`` or ``busybox-mdev``.
8041
8042.. _static-dev-management:
8043
8044Using Persistent and Pre-Populated\ ``/dev``
8045--------------------------------------------
8046
8047To use the static method for device population, you need to set the
8048:term:`USE_DEVFS` variable to "0"
8049as follows:
8050::
8051
8052 USE_DEVFS = "0"
8053
8054The content of the resulting ``/dev`` directory is defined in a Device
8055Table file. The
8056:term:`IMAGE_DEVICE_TABLES`
8057variable defines the Device Table to use and should be set in the
8058machine or distro configuration file. Alternatively, you can set this
8059variable in your ``local.conf`` configuration file.
8060
8061If you do not define the ``IMAGE_DEVICE_TABLES`` variable, the default
8062``device_table-minimal.txt`` is used:
8063::
8064
8065 IMAGE_DEVICE_TABLES = "device_table-mymachine.txt"
8066
8067The population is handled by the ``makedevs`` utility during image
8068creation:
8069
8070.. _devtmpfs-dev-management:
8071
8072Using ``devtmpfs`` and a Device Manager
8073---------------------------------------
8074
8075To use the dynamic method for device population, you need to use (or be
8076sure to set) the :term:`USE_DEVFS`
8077variable to "1", which is the default:
8078::
8079
8080 USE_DEVFS = "1"
8081
8082With this
8083setting, the resulting ``/dev`` directory is populated by the kernel
8084using ``devtmpfs``. Make sure the corresponding kernel configuration
8085variable ``CONFIG_DEVTMPFS`` is set when building you build a Linux
8086kernel.
8087
8088All devices created by ``devtmpfs`` will be owned by ``root`` and have
8089permissions ``0600``.
8090
8091To have more control over the device nodes, you can use a device manager
8092like ``udev`` or ``busybox-mdev``. You choose the device manager by
8093defining the ``VIRTUAL-RUNTIME_dev_manager`` variable in your machine or
8094distro configuration file. Alternatively, you can set this variable in
8095your ``local.conf`` configuration file:
8096::
8097
8098 VIRTUAL-RUNTIME_dev_manager = "udev"
8099
8100 # Some alternative values
8101 # VIRTUAL-RUNTIME_dev_manager = "busybox-mdev"
8102 # VIRTUAL-RUNTIME_dev_manager = "systemd"
8103
8104.. _platdev-appdev-srcrev:
8105
8106Using an External SCM
8107=====================
8108
8109If you're working on a recipe that pulls from an external Source Code
8110Manager (SCM), it is possible to have the OpenEmbedded build system
8111notice new recipe changes added to the SCM and then build the resulting
8112packages that depend on the new recipes by using the latest versions.
8113This only works for SCMs from which it is possible to get a sensible
8114revision number for changes. Currently, you can do this with Apache
8115Subversion (SVN), Git, and Bazaar (BZR) repositories.
8116
8117To enable this behavior, the :term:`PV` of
8118the recipe needs to reference
8119:term:`SRCPV`. Here is an example:
8120::
8121
8122 PV = "1.2.3+git${SRCPV}"
8123
8124Then, you can add the following to your
8125``local.conf``:
8126::
8127
8128 SRCREV_pn-PN = "${AUTOREV}"
8129
8130:term:`PN` is the name of the recipe for
8131which you want to enable automatic source revision updating.
8132
8133If you do not want to update your local configuration file, you can add
8134the following directly to the recipe to finish enabling the feature:
8135::
8136
8137 SRCREV = "${AUTOREV}"
8138
8139The Yocto Project provides a distribution named ``poky-bleeding``, whose
8140configuration file contains the line:
8141::
8142
8143 require conf/distro/include/poky-floating-revisions.inc
8144
8145This line pulls in the
8146listed include file that contains numerous lines of exactly that form:
8147::
8148
8149 #SRCREV_pn-opkg-native ?= "${AUTOREV}"
8150 #SRCREV_pn-opkg-sdk ?= "${AUTOREV}"
8151 #SRCREV_pn-opkg ?= "${AUTOREV}"
8152 #SRCREV_pn-opkg-utils-native ?= "${AUTOREV}"
8153 #SRCREV_pn-opkg-utils ?= "${AUTOREV}"
8154 SRCREV_pn-gconf-dbus ?= "${AUTOREV}"
8155 SRCREV_pn-matchbox-common ?= "${AUTOREV}"
8156 SRCREV_pn-matchbox-config-gtk ?= "${AUTOREV}"
8157 SRCREV_pn-matchbox-desktop ?= "${AUTOREV}"
8158 SRCREV_pn-matchbox-keyboard ?= "${AUTOREV}"
8159 SRCREV_pn-matchbox-panel-2 ?= "${AUTOREV}"
8160 SRCREV_pn-matchbox-themes-extra ?= "${AUTOREV}"
8161 SRCREV_pn-matchbox-terminal ?= "${AUTOREV}"
8162 SRCREV_pn-matchbox-wm ?= "${AUTOREV}"
8163 SRCREV_pn-settings-daemon ?= "${AUTOREV}"
8164 SRCREV_pn-screenshot ?= "${AUTOREV}"
8165 . . .
8166
8167These lines allow you to
8168experiment with building a distribution that tracks the latest
8169development source for numerous packages.
8170
8171.. note::
8172
8173 The
8174 poky-bleeding
8175 distribution is not tested on a regular basis. Keep this in mind if
8176 you use it.
8177
8178Creating a Read-Only Root Filesystem
8179====================================
8180
8181Suppose, for security reasons, you need to disable your target device's
8182root filesystem's write permissions (i.e. you need a read-only root
8183filesystem). Or, perhaps you are running the device's operating system
8184from a read-only storage device. For either case, you can customize your
8185image for that behavior.
8186
8187.. note::
8188
8189 Supporting a read-only root filesystem requires that the system and
8190 applications do not try to write to the root filesystem. You must
8191 configure all parts of the target system to write elsewhere, or to
8192 gracefully fail in the event of attempting to write to the root
8193 filesystem.
8194
8195Creating the Root Filesystem
8196----------------------------
8197
8198To create the read-only root filesystem, simply add the
8199"read-only-rootfs" feature to your image, normally in one of two ways.
8200The first way is to add the "read-only-rootfs" image feature in the
8201image's recipe file via the ``IMAGE_FEATURES`` variable:
8202::
8203
8204 IMAGE_FEATURES += "read-only-rootfs"
8205
8206As an alternative, you can add the same feature
8207from within your build directory's ``local.conf`` file with the
8208associated ``EXTRA_IMAGE_FEATURES`` variable, as in:
8209::
8210
8211 EXTRA_IMAGE_FEATURES = "read-only-rootfs"
8212
8213For more information on how to use these variables, see the
8214":ref:`usingpoky-extend-customimage-imagefeatures`"
8215section. For information on the variables, see
8216:term:`IMAGE_FEATURES` and
8217:term:`EXTRA_IMAGE_FEATURES`.
8218
8219Post-Installation Scripts and Read-Only Root Filesystem
8220-------------------------------------------------------
8221
8222It is very important that you make sure all post-Installation
8223(``pkg_postinst``) scripts for packages that are installed into the
8224image can be run at the time when the root filesystem is created during
8225the build on the host system. These scripts cannot attempt to run during
8226first-boot on the target device. With the "read-only-rootfs" feature
8227enabled, the build system checks during root filesystem creation to make
8228sure all post-installation scripts succeed. If any of these scripts
8229still need to be run after the root filesystem is created, the build
8230immediately fails. These build-time checks ensure that the build fails
8231rather than the target device fails later during its initial boot
8232operation.
8233
8234Most of the common post-installation scripts generated by the build
8235system for the out-of-the-box Yocto Project are engineered so that they
8236can run during root filesystem creation (e.g. post-installation scripts
8237for caching fonts). However, if you create and add custom scripts, you
8238need to be sure they can be run during this file system creation.
8239
8240Here are some common problems that prevent post-installation scripts
8241from running during root filesystem creation:
8242
8243- *Not using $D in front of absolute paths:* The build system defines
8244 ``$``\ :term:`D` when the root
8245 filesystem is created. Furthermore, ``$D`` is blank when the script
8246 is run on the target device. This implies two purposes for ``$D``:
8247 ensuring paths are valid in both the host and target environments,
8248 and checking to determine which environment is being used as a method
8249 for taking appropriate actions.
8250
8251- *Attempting to run processes that are specific to or dependent on the
8252 target architecture:* You can work around these attempts by using
8253 native tools, which run on the host system, to accomplish the same
8254 tasks, or by alternatively running the processes under QEMU, which
8255 has the ``qemu_run_binary`` function. For more information, see the
8256 :ref:`qemu <ref-classes-qemu>` class.
8257
8258Areas With Write Access
8259-----------------------
8260
8261With the "read-only-rootfs" feature enabled, any attempt by the target
8262to write to the root filesystem at runtime fails. Consequently, you must
8263make sure that you configure processes and applications that attempt
8264these types of writes do so to directories with write access (e.g.
8265``/tmp`` or ``/var/run``).
8266
8267Maintaining Build Output Quality
8268================================
8269
8270Many factors can influence the quality of a build. For example, if you
8271upgrade a recipe to use a new version of an upstream software package or
8272you experiment with some new configuration options, subtle changes can
8273occur that you might not detect until later. Consider the case where
8274your recipe is using a newer version of an upstream package. In this
8275case, a new version of a piece of software might introduce an optional
8276dependency on another library, which is auto-detected. If that library
8277has already been built when the software is building, the software will
8278link to the built library and that library will be pulled into your
8279image along with the new software even if you did not want the library.
8280
8281The :ref:`buildhistory <ref-classes-buildhistory>`
8282class exists to help you maintain the quality of your build output. You
8283can use the class to highlight unexpected and possibly unwanted changes
8284in the build output. When you enable build history, it records
8285information about the contents of each package and image and then
8286commits that information to a local Git repository where you can examine
8287the information.
8288
8289The remainder of this section describes the following:
8290
8291- How you can enable and disable build history
8292
8293- How to understand what the build history contains
8294
8295- How to limit the information used for build history
8296
8297- How to examine the build history from both a command-line and web
8298 interface
8299
8300Enabling and Disabling Build History
8301------------------------------------
8302
8303Build history is disabled by default. To enable it, add the following
8304``INHERIT`` statement and set the
8305:term:`BUILDHISTORY_COMMIT`
8306variable to "1" at the end of your ``conf/local.conf`` file found in the
8307:term:`Build Directory`:
8308::
8309
8310 INHERIT += "buildhistory"
8311 BUILDHISTORY_COMMIT = "1"
8312
8313Enabling build history as
8314previously described causes the OpenEmbedded build system to collect
8315build output information and commit it as a single commit to a local
8316:ref:`overview-manual/overview-manual-development-environment:git` repository.
8317
8318.. note::
8319
8320 Enabling build history increases your build times slightly,
8321 particularly for images, and increases the amount of disk space used
8322 during the build.
8323
8324You can disable build history by removing the previous statements from
8325your ``conf/local.conf`` file.
8326
8327Understanding What the Build History Contains
8328---------------------------------------------
8329
8330Build history information is kept in
8331``${``\ :term:`TOPDIR`\ ``}/buildhistory``
8332in the Build Directory as defined by the
8333:term:`BUILDHISTORY_DIR`
8334variable. The following is an example abbreviated listing:
8335
8336.. image:: figures/buildhistory.png
8337 :align: center
8338
8339At the top level, a ``metadata-revs`` file exists that lists the
8340revisions of the repositories for the enabled layers when the build was
8341produced. The rest of the data splits into separate ``packages``,
8342``images`` and ``sdk`` directories, the contents of which are described
8343as follows.
8344
8345Build History Package Information
8346~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8347
8348The history for each package contains a text file that has name-value
8349pairs with information about the package. For example,
8350``buildhistory/packages/i586-poky-linux/busybox/busybox/latest``
8351contains the following:
8352::
8353
8354 PV = 1.22.1
8355 PR = r32
8356 RPROVIDES =
8357 RDEPENDS = glibc (>= 2.20) update-alternatives-opkg
8358 RRECOMMENDS = busybox-syslog busybox-udhcpc update-rc.d
8359 PKGSIZE = 540168
8360 FILES = /usr/bin/* /usr/sbin/* /usr/lib/busybox/* /usr/lib/lib*.so.* \
8361 /etc /com /var /bin/* /sbin/* /lib/*.so.* /lib/udev/rules.d \
8362 /usr/lib/udev/rules.d /usr/share/busybox /usr/lib/busybox/* \
8363 /usr/share/pixmaps /usr/share/applications /usr/share/idl \
8364 /usr/share/omf /usr/share/sounds /usr/lib/bonobo/servers
8365 FILELIST = /bin/busybox /bin/busybox.nosuid /bin/busybox.suid /bin/sh \
8366 /etc/busybox.links.nosuid /etc/busybox.links.suid
8367
8368Most of these
8369name-value pairs correspond to variables used to produce the package.
8370The exceptions are ``FILELIST``, which is the actual list of files in
8371the package, and ``PKGSIZE``, which is the total size of files in the
8372package in bytes.
8373
8374A file also exists that corresponds to the recipe from which the package
8375came (e.g. ``buildhistory/packages/i586-poky-linux/busybox/latest``):
8376::
8377
8378 PV = 1.22.1
8379 PR = r32
8380 DEPENDS = initscripts kern-tools-native update-rc.d-native \
8381 virtual/i586-poky-linux-compilerlibs virtual/i586-poky-linux-gcc \
8382 virtual/libc virtual/update-alternatives
8383 PACKAGES = busybox-ptest busybox-httpd busybox-udhcpd busybox-udhcpc \
8384 busybox-syslog busybox-mdev busybox-hwclock busybox-dbg \
8385 busybox-staticdev busybox-dev busybox-doc busybox-locale busybox
8386
8387Finally, for those recipes fetched from a version control system (e.g.,
8388Git), a file exists that lists source revisions that are specified in
8389the recipe and lists the actual revisions used during the build. Listed
8390and actual revisions might differ when
8391:term:`SRCREV` is set to
8392${:term:`AUTOREV`}. Here is an
8393example assuming
8394``buildhistory/packages/qemux86-poky-linux/linux-yocto/latest_srcrev``):
8395::
8396
8397 # SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
8398 SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
8399 # SRCREV_meta = "a227f20eff056e511d504b2e490f3774ab260d6f"
8400 SRCREV_meta ="a227f20eff056e511d504b2e490f3774ab260d6f"
8401
8402You can use the
8403``buildhistory-collect-srcrevs`` command with the ``-a`` option to
8404collect the stored ``SRCREV`` values from build history and report them
8405in a format suitable for use in global configuration (e.g.,
8406``local.conf`` or a distro include file) to override floating
8407``AUTOREV`` values to a fixed set of revisions. Here is some example
8408output from this command:
8409::
8410
8411 $ buildhistory-collect-srcrevs -a
8412 # i586-poky-linux
8413 SRCREV_pn-glibc = "b8079dd0d360648e4e8de48656c5c38972621072"
8414 SRCREV_pn-glibc-initial = "b8079dd0d360648e4e8de48656c5c38972621072"
8415 SRCREV_pn-opkg-utils = "53274f087565fd45d8452c5367997ba6a682a37a"
8416 SRCREV_pn-kmod = "fd56638aed3fe147015bfa10ed4a5f7491303cb4"
8417 # x86_64-linux
8418 SRCREV_pn-gtk-doc-stub-native = "1dea266593edb766d6d898c79451ef193eb17cfa"
8419 SRCREV_pn-dtc-native = "65cc4d2748a2c2e6f27f1cf39e07a5dbabd80ebf"
8420 SRCREV_pn-update-rc.d-native = "eca680ddf28d024954895f59a241a622dd575c11"
8421 SRCREV_glibc_pn-cross-localedef-native = "b8079dd0d360648e4e8de48656c5c38972621072"
8422 SRCREV_localedef_pn-cross-localedef-native = "c833367348d39dad7ba018990bfdaffaec8e9ed3"
8423 SRCREV_pn-prelink-native = "faa069deec99bf61418d0bab831c83d7c1b797ca"
8424 SRCREV_pn-opkg-utils-native = "53274f087565fd45d8452c5367997ba6a682a37a"
8425 SRCREV_pn-kern-tools-native = "23345b8846fe4bd167efdf1bd8a1224b2ba9a5ff"
8426 SRCREV_pn-kmod-native = "fd56638aed3fe147015bfa10ed4a5f7491303cb4"
8427 # qemux86-poky-linux
8428 SRCREV_machine_pn-linux-yocto = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
8429 SRCREV_meta_pn-linux-yocto = "a227f20eff056e511d504b2e490f3774ab260d6f"
8430 # all-poky-linux
8431 SRCREV_pn-update-rc.d = "eca680ddf28d024954895f59a241a622dd575c11"
8432
8433.. note::
8434
8435 Here are some notes on using the
8436 buildhistory-collect-srcrevs
8437 command:
8438
8439 - By default, only values where the ``SRCREV`` was not hardcoded
8440 (usually when ``AUTOREV`` is used) are reported. Use the ``-a``
8441 option to see all ``SRCREV`` values.
8442
8443 - The output statements might not have any effect if overrides are
8444 applied elsewhere in the build system configuration. Use the
8445 ``-f`` option to add the ``forcevariable`` override to each output
8446 line if you need to work around this restriction.
8447
8448 - The script does apply special handling when building for multiple
8449 machines. However, the script does place a comment before each set
8450 of values that specifies which triplet to which they belong as
8451 previously shown (e.g., ``i586-poky-linux``).
8452
8453Build History Image Information
8454~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8455
8456The files produced for each image are as follows:
8457
8458- ``image-files:`` A directory containing selected files from the root
8459 filesystem. The files are defined by
8460 :term:`BUILDHISTORY_IMAGE_FILES`.
8461
8462- ``build-id.txt:`` Human-readable information about the build
8463 configuration and metadata source revisions. This file contains the
8464 full build header as printed by BitBake.
8465
8466- ``*.dot:`` Dependency graphs for the image that are compatible with
8467 ``graphviz``.
8468
8469- ``files-in-image.txt:`` A list of files in the image with
8470 permissions, owner, group, size, and symlink information.
8471
8472- ``image-info.txt:`` A text file containing name-value pairs with
8473 information about the image. See the following listing example for
8474 more information.
8475
8476- ``installed-package-names.txt:`` A list of installed packages by name
8477 only.
8478
8479- ``installed-package-sizes.txt:`` A list of installed packages ordered
8480 by size.
8481
8482- ``installed-packages.txt:`` A list of installed packages with full
8483 package filenames.
8484
8485.. note::
8486
8487 Installed package information is able to be gathered and produced
8488 even if package management is disabled for the final image.
8489
8490Here is an example of ``image-info.txt``:
8491::
8492
8493 DISTRO = poky
8494 DISTRO_VERSION = 1.7
8495 USER_CLASSES = buildstats image-mklibs image-prelink
8496 IMAGE_CLASSES = image_types
8497 IMAGE_FEATURES = debug-tweaks
8498 IMAGE_LINGUAS =
8499 IMAGE_INSTALL = packagegroup-core-boot run-postinsts
8500 BAD_RECOMMENDATIONS =
8501 NO_RECOMMENDATIONS =
8502 PACKAGE_EXCLUDE =
8503 ROOTFS_POSTPROCESS_COMMAND = write_package_manifest; license_create_manifest; \
8504 write_image_manifest ; buildhistory_list_installed_image ; \
8505 buildhistory_get_image_installed ; ssh_allow_empty_password; \
8506 postinst_enable_logging; rootfs_update_timestamp ; ssh_disable_dns_lookup ;
8507 IMAGE_POSTPROCESS_COMMAND = buildhistory_get_imageinfo ;
8508 IMAGESIZE = 6900
8509
8510Other than ``IMAGESIZE``,
8511which is the total size of the files in the image in Kbytes, the
8512name-value pairs are variables that may have influenced the content of
8513the image. This information is often useful when you are trying to
8514determine why a change in the package or file listings has occurred.
8515
8516Using Build History to Gather Image Information Only
8517~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8518
8519As you can see, build history produces image information, including
8520dependency graphs, so you can see why something was pulled into the
8521image. If you are just interested in this information and not interested
8522in collecting specific package or SDK information, you can enable
8523writing only image information without any history by adding the
8524following to your ``conf/local.conf`` file found in the
8525:term:`Build Directory`:
8526::
8527
8528 INHERIT += "buildhistory"
8529 BUILDHISTORY_COMMIT = "0"
8530 BUILDHISTORY_FEATURES = "image"
8531
8532Here, you set the
8533:term:`BUILDHISTORY_FEATURES`
8534variable to use the image feature only.
8535
8536Build History SDK Information
8537~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8538
8539Build history collects similar information on the contents of SDKs (e.g.
8540``bitbake -c populate_sdk imagename``) as compared to information it
8541collects for images. Furthermore, this information differs depending on
8542whether an extensible or standard SDK is being produced.
8543
8544The following list shows the files produced for SDKs:
8545
8546- ``files-in-sdk.txt:`` A list of files in the SDK with permissions,
8547 owner, group, size, and symlink information. This list includes both
8548 the host and target parts of the SDK.
8549
8550- ``sdk-info.txt:`` A text file containing name-value pairs with
8551 information about the SDK. See the following listing example for more
8552 information.
8553
8554- ``sstate-task-sizes.txt:`` A text file containing name-value pairs
8555 with information about task group sizes (e.g. ``do_populate_sysroot``
8556 tasks have a total size). The ``sstate-task-sizes.txt`` file exists
8557 only when an extensible SDK is created.
8558
8559- ``sstate-package-sizes.txt:`` A text file containing name-value pairs
8560 with information for the shared-state packages and sizes in the SDK.
8561 The ``sstate-package-sizes.txt`` file exists only when an extensible
8562 SDK is created.
8563
8564- ``sdk-files:`` A folder that contains copies of the files mentioned
8565 in ``BUILDHISTORY_SDK_FILES`` if the files are present in the output.
8566 Additionally, the default value of ``BUILDHISTORY_SDK_FILES`` is
8567 specific to the extensible SDK although you can set it differently if
8568 you would like to pull in specific files from the standard SDK.
8569
8570 The default files are ``conf/local.conf``, ``conf/bblayers.conf``,
8571 ``conf/auto.conf``, ``conf/locked-sigs.inc``, and
8572 ``conf/devtool.conf``. Thus, for an extensible SDK, these files get
8573 copied into the ``sdk-files`` directory.
8574
8575- The following information appears under each of the ``host`` and
8576 ``target`` directories for the portions of the SDK that run on the
8577 host and on the target, respectively:
8578
8579 .. note::
8580
8581 The following files for the most part are empty when producing an
8582 extensible SDK because this type of SDK is not constructed from
8583 packages as is the standard SDK.
8584
8585 - ``depends.dot:`` Dependency graph for the SDK that is compatible
8586 with ``graphviz``.
8587
8588 - ``installed-package-names.txt:`` A list of installed packages by
8589 name only.
8590
8591 - ``installed-package-sizes.txt:`` A list of installed packages
8592 ordered by size.
8593
8594 - ``installed-packages.txt:`` A list of installed packages with full
8595 package filenames.
8596
8597Here is an example of ``sdk-info.txt``:
8598::
8599
8600 DISTRO = poky
8601 DISTRO_VERSION = 1.3+snapshot-20130327
8602 SDK_NAME = poky-glibc-i686-arm
8603 SDK_VERSION = 1.3+snapshot
8604 SDKMACHINE =
8605 SDKIMAGE_FEATURES = dev-pkgs dbg-pkgs
8606 BAD_RECOMMENDATIONS =
8607 SDKSIZE = 352712
8608
8609Other than ``SDKSIZE``, which is
8610the total size of the files in the SDK in Kbytes, the name-value pairs
8611are variables that might have influenced the content of the SDK. This
8612information is often useful when you are trying to determine why a
8613change in the package or file listings has occurred.
8614
8615Examining Build History Information
8616~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8617
8618You can examine build history output from the command line or from a web
8619interface.
8620
8621To see any changes that have occurred (assuming you have
8622:term:`BUILDHISTORY_COMMIT` = "1"),
8623you can simply use any Git command that allows you to view the history
8624of a repository. Here is one method:
8625::
8626
8627 $ git log -p
8628
8629You need to realize,
8630however, that this method does show changes that are not significant
8631(e.g. a package's size changing by a few bytes).
8632
8633A command-line tool called ``buildhistory-diff`` does exist, though,
8634that queries the Git repository and prints just the differences that
8635might be significant in human-readable form. Here is an example:
8636::
8637
8638 $ ~/poky/poky/scripts/buildhistory-diff . HEAD^
8639 Changes to images/qemux86_64/glibc/core-image-minimal (files-in-image.txt):
8640 /etc/anotherpkg.conf was added
8641 /sbin/anotherpkg was added
8642 * (installed-package-names.txt):
8643 * anotherpkg was added
8644 Changes to images/qemux86_64/glibc/core-image-minimal (installed-package-names.txt):
8645 anotherpkg was added
8646 packages/qemux86_64-poky-linux/v86d: PACKAGES: added "v86d-extras"
8647 * PR changed from "r0" to "r1"
8648 * PV changed from "0.1.10" to "0.1.12"
8649 packages/qemux86_64-poky-linux/v86d/v86d: PKGSIZE changed from 110579 to 144381 (+30%)
8650 * PR changed from "r0" to "r1"
8651 * PV changed from "0.1.10" to "0.1.12"
8652
8653.. note::
8654
8655 The
8656 buildhistory-diff
8657 tool requires the
8658 GitPython
8659 package. Be sure to install it using Pip3 as follows:
8660 ::
8661
8662 $ pip3 install GitPython --user
8663
8664
8665 Alternatively, you can install
8666 python3-git
8667 using the appropriate distribution package manager (e.g.
8668 apt-get
8669 ,
8670 dnf
8671 , or
8672 zipper
8673 ).
8674
8675To see changes to the build history using a web interface, follow the
8676instruction in the ``README`` file here.
8677http://git.yoctoproject.org/cgit/cgit.cgi/buildhistory-web/.
8678
8679Here is a sample screenshot of the interface:
8680
8681.. image:: figures/buildhistory-web.png
8682 :align: center
8683
8684Performing Automated Runtime Testing
8685====================================
8686
8687The OpenEmbedded build system makes available a series of automated
8688tests for images to verify runtime functionality. You can run these
8689tests on either QEMU or actual target hardware. Tests are written in
8690Python making use of the ``unittest`` module, and the majority of them
8691run commands on the target system over SSH. This section describes how
8692you set up the environment to use these tests, run available tests, and
8693write and add your own tests.
8694
8695For information on the test and QA infrastructure available within the
8696Yocto Project, see the ":ref:`ref-manual/ref-release-process:testing and quality assurance`"
8697section in the Yocto Project Reference Manual.
8698
8699Enabling Tests
8700--------------
8701
8702Depending on whether you are planning to run tests using QEMU or on the
8703hardware, you have to take different steps to enable the tests. See the
8704following subsections for information on how to enable both types of
8705tests.
8706
8707.. _qemu-image-enabling-tests:
8708
8709Enabling Runtime Tests on QEMU
8710~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8711
8712In order to run tests, you need to do the following:
8713
8714- *Set up to avoid interaction with sudo for networking:* To
8715 accomplish this, you must do one of the following:
8716
8717 - Add ``NOPASSWD`` for your user in ``/etc/sudoers`` either for all
8718 commands or just for ``runqemu-ifup``. You must provide the full
8719 path as that can change if you are using multiple clones of the
8720 source repository.
8721
8722 .. note::
8723
8724 On some distributions, you also need to comment out "Defaults
8725 requiretty" in
8726 /etc/sudoers
8727 .
8728
8729 - Manually configure a tap interface for your system.
8730
8731 - Run as root the script in ``scripts/runqemu-gen-tapdevs``, which
8732 should generate a list of tap devices. This is the option
8733 typically chosen for Autobuilder-type environments.
8734
8735 .. note::
8736
8737 - Be sure to use an absolute path when calling this script
8738 with sudo.
8739
8740 - The package recipe ``qemu-helper-native`` is required to run
8741 this script. Build the package using the following command:
8742 $ bitbake qemu-helper-native
8743
8744- *Set the DISPLAY variable:* You need to set this variable so that
8745 you have an X server available (e.g. start ``vncserver`` for a
8746 headless machine).
8747
8748- *Be sure your host's firewall accepts incoming connections from
8749 192.168.7.0/24:* Some of the tests (in particular DNF tests) start an
8750 HTTP server on a random high number port, which is used to serve
8751 files to the target. The DNF module serves
8752 ``${WORKDIR}/oe-rootfs-repo`` so it can run DNF channel commands.
8753 That means your host's firewall must accept incoming connections from
8754 192.168.7.0/24, which is the default IP range used for tap devices by
8755 ``runqemu``.
8756
8757- *Be sure your host has the correct packages installed:* Depending
8758 your host's distribution, you need to have the following packages
8759 installed:
8760
8761 - Ubuntu and Debian: ``sysstat`` and ``iproute2``
8762
8763 - OpenSUSE: ``sysstat`` and ``iproute2``
8764
8765 - Fedora: ``sysstat`` and ``iproute``
8766
8767 - CentOS: ``sysstat`` and ``iproute``
8768
8769Once you start running the tests, the following happens:
8770
87711. A copy of the root filesystem is written to ``${WORKDIR}/testimage``.
8772
87732. The image is booted under QEMU using the standard ``runqemu`` script.
8774
87753. A default timeout of 500 seconds occurs to allow for the boot process
8776 to reach the login prompt. You can change the timeout period by
8777 setting
8778 :term:`TEST_QEMUBOOT_TIMEOUT`
8779 in the ``local.conf`` file.
8780
87814. Once the boot process is reached and the login prompt appears, the
8782 tests run. The full boot log is written to
8783 ``${WORKDIR}/testimage/qemu_boot_log``.
8784
87855. Each test module loads in the order found in ``TEST_SUITES``. You can
8786 find the full output of the commands run over SSH in
8787 ``${WORKDIR}/testimgage/ssh_target_log``.
8788
87896. If no failures occur, the task running the tests ends successfully.
8790 You can find the output from the ``unittest`` in the task log at
8791 ``${WORKDIR}/temp/log.do_testimage``.
8792
8793.. _hardware-image-enabling-tests:
8794
8795Enabling Runtime Tests on Hardware
8796~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8797
8798The OpenEmbedded build system can run tests on real hardware, and for
8799certain devices it can also deploy the image to be tested onto the
8800device beforehand.
8801
8802For automated deployment, a "master image" is installed onto the
8803hardware once as part of setup. Then, each time tests are to be run, the
8804following occurs:
8805
88061. The master image is booted into and used to write the image to be
8807 tested to a second partition.
8808
88092. The device is then rebooted using an external script that you need to
8810 provide.
8811
88123. The device boots into the image to be tested.
8813
8814When running tests (independent of whether the image has been deployed
8815automatically or not), the device is expected to be connected to a
8816network on a pre-determined IP address. You can either use static IP
8817addresses written into the image, or set the image to use DHCP and have
8818your DHCP server on the test network assign a known IP address based on
8819the MAC address of the device.
8820
8821In order to run tests on hardware, you need to set ``TEST_TARGET`` to an
8822appropriate value. For QEMU, you do not have to change anything, the
8823default value is "qemu". For running tests on hardware, the following
8824options exist:
8825
8826- *"simpleremote":* Choose "simpleremote" if you are going to run tests
8827 on a target system that is already running the image to be tested and
8828 is available on the network. You can use "simpleremote" in
8829 conjunction with either real hardware or an image running within a
8830 separately started QEMU or any other virtual machine manager.
8831
8832- *"SystemdbootTarget":* Choose "SystemdbootTarget" if your hardware is
8833 an EFI-based machine with ``systemd-boot`` as bootloader and
8834 ``core-image-testmaster`` (or something similar) is installed. Also,
8835 your hardware under test must be in a DHCP-enabled network that gives
8836 it the same IP address for each reboot.
8837
8838 If you choose "SystemdbootTarget", there are additional requirements
8839 and considerations. See the "`Selecting
8840 SystemdbootTarget <#selecting-systemdboottarget>`__" section, which
8841 follows, for more information.
8842
8843- *"BeagleBoneTarget":* Choose "BeagleBoneTarget" if you are deploying
8844 images and running tests on the BeagleBone "Black" or original
8845 "White" hardware. For information on how to use these tests, see the
8846 comments at the top of the BeagleBoneTarget
8847 ``meta-yocto-bsp/lib/oeqa/controllers/beaglebonetarget.py`` file.
8848
8849- *"EdgeRouterTarget":* Choose "EdgeRouterTarget" is you are deploying
8850 images and running tests on the Ubiquiti Networks EdgeRouter Lite.
8851 For information on how to use these tests, see the comments at the
8852 top of the EdgeRouterTarget
8853 ``meta-yocto-bsp/lib/oeqa/controllers/edgeroutertarget.py`` file.
8854
8855- *"GrubTarget":* Choose the "supports deploying images and running
8856 tests on any generic PC that boots using GRUB. For information on how
8857 to use these tests, see the comments at the top of the GrubTarget
8858 ``meta-yocto-bsp/lib/oeqa/controllers/grubtarget.py`` file.
8859
8860- *"your-target":* Create your own custom target if you want to run
8861 tests when you are deploying images and running tests on a custom
8862 machine within your BSP layer. To do this, you need to add a Python
8863 unit that defines the target class under ``lib/oeqa/controllers/``
8864 within your layer. You must also provide an empty ``__init__.py``.
8865 For examples, see files in ``meta-yocto-bsp/lib/oeqa/controllers/``.
8866
8867Selecting SystemdbootTarget
8868~~~~~~~~~~~~~~~~~~~~~~~~~~~
8869
8870If you did not set ``TEST_TARGET`` to "SystemdbootTarget", then you do
8871not need any information in this section. You can skip down to the
8872"`Running Tests <#qemu-image-running-tests>`__" section.
8873
8874If you did set ``TEST_TARGET`` to "SystemdbootTarget", you also need to
8875perform a one-time setup of your master image by doing the following:
8876
88771. *Set EFI_PROVIDER:* Be sure that ``EFI_PROVIDER`` is as follows:
8878 ::
8879
8880 EFI_PROVIDER = "systemd-boot"
8881
88822. *Build the master image:* Build the ``core-image-testmaster`` image.
8883 The ``core-image-testmaster`` recipe is provided as an example for a
8884 "master" image and you can customize the image recipe as you would
8885 any other recipe.
8886
8887 Here are the image recipe requirements:
8888
8889 - Inherits ``core-image`` so that kernel modules are installed.
8890
8891 - Installs normal linux utilities not busybox ones (e.g. ``bash``,
8892 ``coreutils``, ``tar``, ``gzip``, and ``kmod``).
8893
8894 - Uses a custom Initial RAM Disk (initramfs) image with a custom
8895 installer. A normal image that you can install usually creates a
8896 single rootfs partition. This image uses another installer that
8897 creates a specific partition layout. Not all Board Support
8898 Packages (BSPs) can use an installer. For such cases, you need to
8899 manually create the following partition layout on the target:
8900
8901 - First partition mounted under ``/boot``, labeled "boot".
8902
8903 - The main rootfs partition where this image gets installed,
8904 which is mounted under ``/``.
8905
8906 - Another partition labeled "testrootfs" where test images get
8907 deployed.
8908
89093. *Install image:* Install the image that you just built on the target
8910 system.
8911
8912The final thing you need to do when setting ``TEST_TARGET`` to
8913"SystemdbootTarget" is to set up the test image:
8914
89151. *Set up your local.conf file:* Make sure you have the following
8916 statements in your ``local.conf`` file:
8917 ::
8918
8919 IMAGE_FSTYPES += "tar.gz"
8920 INHERIT += "testimage"
8921 TEST_TARGET = "SystemdbootTarget"
8922 TEST_TARGET_IP = "192.168.2.3"
8923
89242. *Build your test image:* Use BitBake to build the image:
8925 ::
8926
8927 $ bitbake core-image-sato
8928
8929Power Control
8930~~~~~~~~~~~~~
8931
8932For most hardware targets other than "simpleremote", you can control
8933power:
8934
8935- You can use ``TEST_POWERCONTROL_CMD`` together with
8936 ``TEST_POWERCONTROL_EXTRA_ARGS`` as a command that runs on the host
8937 and does power cycling. The test code passes one argument to that
8938 command: off, on or cycle (off then on). Here is an example that
8939 could appear in your ``local.conf`` file:
8940 ::
8941
8942 TEST_POWERCONTROL_CMD = "powercontrol.exp test 10.11.12.1 nuc1"
8943
8944 In this example, the expect
8945 script does the following:
8946 ::
8947
8948 ssh test@10.11.12.1 "pyctl nuc1 arg"
8949
8950 It then runs a Python script that controls power for a label called
8951 ``nuc1``.
8952
8953 .. note::
8954
8955 You need to customize
8956 TEST_POWERCONTROL_CMD
8957 and
8958 TEST_POWERCONTROL_EXTRA_ARGS
8959 for your own setup. The one requirement is that it accepts "on",
8960 "off", and "cycle" as the last argument.
8961
8962- When no command is defined, it connects to the device over SSH and
8963 uses the classic reboot command to reboot the device. Classic reboot
8964 is fine as long as the machine actually reboots (i.e. the SSH test
8965 has not failed). It is useful for scenarios where you have a simple
8966 setup, typically with a single board, and where some manual
8967 interaction is okay from time to time.
8968
8969If you have no hardware to automatically perform power control but still
8970wish to experiment with automated hardware testing, you can use the
8971dialog-power-control script that shows a dialog prompting you to perform
8972the required power action. This script requires either KDialog or Zenity
8973to be installed. To use this script, set the
8974:term:`TEST_POWERCONTROL_CMD`
8975variable as follows:
8976::
8977
8978 TEST_POWERCONTROL_CMD = "${COREBASE}/scripts/contrib/dialog-power-control"
8979
8980Serial Console Connection
8981~~~~~~~~~~~~~~~~~~~~~~~~~
8982
8983For test target classes requiring a serial console to interact with the
8984bootloader (e.g. BeagleBoneTarget, EdgeRouterTarget, and GrubTarget),
8985you need to specify a command to use to connect to the serial console of
8986the target machine by using the
8987:term:`TEST_SERIALCONTROL_CMD`
8988variable and optionally the
8989:term:`TEST_SERIALCONTROL_EXTRA_ARGS`
8990variable.
8991
8992These cases could be a serial terminal program if the machine is
8993connected to a local serial port, or a ``telnet`` or ``ssh`` command
8994connecting to a remote console server. Regardless of the case, the
8995command simply needs to connect to the serial console and forward that
8996connection to standard input and output as any normal terminal program
8997does. For example, to use the picocom terminal program on serial device
8998``/dev/ttyUSB0`` at 115200bps, you would set the variable as follows:
8999::
9000
9001 TEST_SERIALCONTROL_CMD = "picocom /dev/ttyUSB0 -b 115200"
9002
9003For local
9004devices where the serial port device disappears when the device reboots,
9005an additional "serdevtry" wrapper script is provided. To use this
9006wrapper, simply prefix the terminal command with
9007``${COREBASE}/scripts/contrib/serdevtry``:
9008::
9009
9010 TEST_SERIALCONTROL_CMD = "${COREBASE}/scripts/contrib/serdevtry picocom -b 115200 /dev/ttyUSB0"
9011
9012.. _qemu-image-running-tests:
9013
9014Running Tests
9015-------------
9016
9017You can start the tests automatically or manually:
9018
9019- *Automatically running tests:* To run the tests automatically after
9020 the OpenEmbedded build system successfully creates an image, first
9021 set the
9022 :term:`TESTIMAGE_AUTO`
9023 variable to "1" in your ``local.conf`` file in the
9024 :term:`Build Directory`:
9025 ::
9026
9027 TESTIMAGE_AUTO = "1"
9028
9029 Next, build your image. If the image successfully builds, the
9030 tests run:
9031 ::
9032
9033 bitbake core-image-sato
9034
9035- *Manually running tests:* To manually run the tests, first globally
9036 inherit the
9037 :ref:`testimage <ref-classes-testimage*>` class
9038 by editing your ``local.conf`` file:
9039 ::
9040
9041 INHERIT += "testimage"
9042
9043 Next, use BitBake to run the tests:
9044 ::
9045
9046 bitbake -c testimage image
9047
9048All test files reside in ``meta/lib/oeqa/runtime`` in the
9049:term:`Source Directory`. A test name maps
9050directly to a Python module. Each test module may contain a number of
9051individual tests. Tests are usually grouped together by the area tested
9052(e.g tests for systemd reside in ``meta/lib/oeqa/runtime/systemd.py``).
9053
9054You can add tests to any layer provided you place them in the proper
9055area and you extend :term:`BBPATH` in
9056the ``local.conf`` file as normal. Be sure that tests reside in
9057``layer/lib/oeqa/runtime``.
9058
9059.. note::
9060
9061 Be sure that module names do not collide with module names used in
9062 the default set of test modules in
9063 meta/lib/oeqa/runtime
9064 .
9065
9066You can change the set of tests run by appending or overriding
9067:term:`TEST_SUITES` variable in
9068``local.conf``. Each name in ``TEST_SUITES`` represents a required test
9069for the image. Test modules named within ``TEST_SUITES`` cannot be
9070skipped even if a test is not suitable for an image (e.g. running the
9071RPM tests on an image without ``rpm``). Appending "auto" to
9072``TEST_SUITES`` causes the build system to try to run all tests that are
9073suitable for the image (i.e. each test module may elect to skip itself).
9074
9075The order you list tests in ``TEST_SUITES`` is important and influences
9076test dependencies. Consequently, tests that depend on other tests should
9077be added after the test on which they depend. For example, since the
9078``ssh`` test depends on the ``ping`` test, "ssh" needs to come after
9079"ping" in the list. The test class provides no re-ordering or dependency
9080handling.
9081
9082.. note::
9083
9084 Each module can have multiple classes with multiple test methods.
9085 And, Python
9086 unittest
9087 rules apply.
9088
9089Here are some things to keep in mind when running tests:
9090
9091- The default tests for the image are defined as:
9092 ::
9093
9094 DEFAULT_TEST_SUITES_pn-image = "ping ssh df connman syslog xorg scp vnc date rpm dnf dmesg"
9095
9096- Add your own test to the list of the by using the following:
9097 ::
9098
9099 TEST_SUITES_append = " mytest"
9100
9101- Run a specific list of tests as follows: TEST_SUITES = "test1 test2
9102 test3" Remember, order is important. Be sure to place a test that is
9103 dependent on another test later in the order.
9104
9105Exporting Tests
9106---------------
9107
9108You can export tests so that they can run independently of the build
9109system. Exporting tests is required if you want to be able to hand the
9110test execution off to a scheduler. You can only export tests that are
9111defined in :term:`TEST_SUITES`.
9112
9113If your image is already built, make sure the following are set in your
9114``local.conf`` file:
9115::
9116
9117 INHERIT +="testexport"
9118 TEST_TARGET_IP = "IP-address-for-the-test-target"
9119 TEST_SERVER_IP = "IP-address-for-the-test-server"
9120
9121You can then export the tests with the
9122following BitBake command form:
9123::
9124
9125 $ bitbake image -c testexport
9126
9127Exporting the tests places them in the
9128:term:`Build Directory` in
9129``tmp/testexport/``\ image, which is controlled by the
9130``TEST_EXPORT_DIR`` variable.
9131
9132You can now run the tests outside of the build environment:
9133::
9134
9135 $ cd tmp/testexport/image
9136 $ ./runexported.py testdata.json
9137
9138Here is a complete example that shows IP addresses and uses the
9139``core-image-sato`` image:
9140::
9141
9142 INHERIT +="testexport"
9143 TEST_TARGET_IP = "192.168.7.2"
9144 TEST_SERVER_IP = "192.168.7.1"
9145
9146Use BitBake to export the tests:
9147::
9148
9149 $ bitbake core-image-sato -c testexport
9150
9151Run the tests outside of
9152the build environment using the following:
9153::
9154
9155 $ cd tmp/testexport/core-image-sato
9156 $ ./runexported.py testdata.json
9157
9158.. _qemu-image-writing-new-tests:
9159
9160Writing New Tests
9161-----------------
9162
9163As mentioned previously, all new test files need to be in the proper
9164place for the build system to find them. New tests for additional
9165functionality outside of the core should be added to the layer that adds
9166the functionality, in ``layer/lib/oeqa/runtime`` (as long as
9167:term:`BBPATH` is extended in the
9168layer's ``layer.conf`` file as normal). Just remember the following:
9169
9170- Filenames need to map directly to test (module) names.
9171
9172- Do not use module names that collide with existing core tests.
9173
9174- Minimally, an empty ``__init__.py`` file must exist in the runtime
9175 directory.
9176
9177To create a new test, start by copying an existing module (e.g.
9178``syslog.py`` or ``gcc.py`` are good ones to use). Test modules can use
9179code from ``meta/lib/oeqa/utils``, which are helper classes.
9180
9181.. note::
9182
9183 Structure shell commands such that you rely on them and they return a
9184 single code for success. Be aware that sometimes you will need to
9185 parse the output. See the
9186 df.py
9187 and
9188 date.py
9189 modules for examples.
9190
9191You will notice that all test classes inherit ``oeRuntimeTest``, which
9192is found in ``meta/lib/oetest.py``. This base class offers some helper
9193attributes, which are described in the following sections:
9194
9195.. _qemu-image-writing-tests-class-methods:
9196
9197Class Methods
9198~~~~~~~~~~~~~
9199
9200Class methods are as follows:
9201
9202- *hasPackage(pkg):* Returns "True" if ``pkg`` is in the installed
9203 package list of the image, which is based on the manifest file that
9204 is generated during the ``do_rootfs`` task.
9205
9206- *hasFeature(feature):* Returns "True" if the feature is in
9207 :term:`IMAGE_FEATURES` or
9208 :term:`DISTRO_FEATURES`.
9209
9210.. _qemu-image-writing-tests-class-attributes:
9211
9212Class Attributes
9213~~~~~~~~~~~~~~~~
9214
9215Class attributes are as follows:
9216
9217- *pscmd:* Equals "ps -ef" if ``procps`` is installed in the image.
9218 Otherwise, ``pscmd`` equals "ps" (busybox).
9219
9220- *tc:* The called test context, which gives access to the
9221 following attributes:
9222
9223 - *d:* The BitBake datastore, which allows you to use stuff such
9224 as ``oeRuntimeTest.tc.d.getVar("VIRTUAL-RUNTIME_init_manager")``.
9225
9226 - *testslist and testsrequired:* Used internally. The tests
9227 do not need these.
9228
9229 - *filesdir:* The absolute path to
9230 ``meta/lib/oeqa/runtime/files``, which contains helper files for
9231 tests meant for copying on the target such as small files written
9232 in C for compilation.
9233
9234 - *target:* The target controller object used to deploy and
9235 start an image on a particular target (e.g. Qemu, SimpleRemote,
9236 and SystemdbootTarget). Tests usually use the following:
9237
9238 - *ip:* The target's IP address.
9239
9240 - *server_ip:* The host's IP address, which is usually used
9241 by the DNF test suite.
9242
9243 - *run(cmd, timeout=None):* The single, most used method.
9244 This command is a wrapper for: ``ssh root@host "cmd"``. The
9245 command returns a tuple: (status, output), which are what their
9246 names imply - the return code of "cmd" and whatever output it
9247 produces. The optional timeout argument represents the number
9248 of seconds the test should wait for "cmd" to return. If the
9249 argument is "None", the test uses the default instance's
9250 timeout period, which is 300 seconds. If the argument is "0",
9251 the test runs until the command returns.
9252
9253 - *copy_to(localpath, remotepath):*
9254 ``scp localpath root@ip:remotepath``.
9255
9256 - *copy_from(remotepath, localpath):*
9257 ``scp root@host:remotepath localpath``.
9258
9259.. _qemu-image-writing-tests-instance-attributes:
9260
9261Instance Attributes
9262~~~~~~~~~~~~~~~~~~~
9263
9264A single instance attribute exists, which is ``target``. The ``target``
9265instance attribute is identical to the class attribute of the same name,
9266which is described in the previous section. This attribute exists as
9267both an instance and class attribute so tests can use
9268``self.target.run(cmd)`` in instance methods instead of
9269``oeRuntimeTest.tc.target.run(cmd)``.
9270
9271Installing Packages in the DUT Without the Package Manager
9272----------------------------------------------------------
9273
9274When a test requires a package built by BitBake, it is possible to
9275install that package. Installing the package does not require a package
9276manager be installed in the device under test (DUT). It does, however,
9277require an SSH connection and the target must be using the
9278``sshcontrol`` class.
9279
9280.. note::
9281
9282 This method uses
9283 scp
9284 to copy files from the host to the target, which causes permissions
9285 and special attributes to be lost.
9286
9287A JSON file is used to define the packages needed by a test. This file
9288must be in the same path as the file used to define the tests.
9289Furthermore, the filename must map directly to the test module name with
9290a ``.json`` extension.
9291
9292The JSON file must include an object with the test name as keys of an
9293object or an array. This object (or array of objects) uses the following
9294data:
9295
9296- "pkg" - A mandatory string that is the name of the package to be
9297 installed.
9298
9299- "rm" - An optional boolean, which defaults to "false", that specifies
9300 to remove the package after the test.
9301
9302- "extract" - An optional boolean, which defaults to "false", that
9303 specifies if the package must be extracted from the package format.
9304 When set to "true", the package is not automatically installed into
9305 the DUT.
9306
9307Following is an example JSON file that handles test "foo" installing
9308package "bar" and test "foobar" installing packages "foo" and "bar".
9309Once the test is complete, the packages are removed from the DUT.
9310::
9311
9312 {
9313 "foo": {
9314 "pkg": "bar"
9315 },
9316 "foobar": [
9317 {
9318 "pkg": "foo",
9319 "rm": true
9320 },
9321 {
9322 "pkg": "bar",
9323 "rm": true
9324 }
9325 ]
9326 }
9327
9328.. _usingpoky-debugging-tools-and-techniques:
9329
9330Debugging Tools and Techniques
9331==============================
9332
9333The exact method for debugging build failures depends on the nature of
9334the problem and on the system's area from which the bug originates.
9335Standard debugging practices such as comparison against the last known
9336working version with examination of the changes and the re-application
9337of steps to identify the one causing the problem are valid for the Yocto
9338Project just as they are for any other system. Even though it is
9339impossible to detail every possible potential failure, this section
9340provides some general tips to aid in debugging given a variety of
9341situations.
9342
9343.. note::
9344
9345 A useful feature for debugging is the error reporting tool.
9346 Configuring the Yocto Project to use this tool causes the
9347 OpenEmbedded build system to produce error reporting commands as part
9348 of the console output. You can enter the commands after the build
9349 completes to log error information into a common database, that can
9350 help you figure out what might be going wrong. For information on how
9351 to enable and use this feature, see the "
9352 Using the Error Reporting Tool
9353 " section.
9354
9355The following list shows the debugging topics in the remainder of this
9356section:
9357
9358- "`Viewing Logs from Failed
9359 Tasks <#dev-debugging-viewing-logs-from-failed-tasks>`__" describes
9360 how to find and view logs from tasks that failed during the build
9361 process.
9362
9363- "`Viewing Variable
9364 Values <#dev-debugging-viewing-variable-values>`__" describes how to
9365 use the BitBake ``-e`` option to examine variable values after a
9366 recipe has been parsed.
9367
9368- ":ref:`dev-manual/dev-manual-common-tasks:viewing package information with \`\`oe-pkgdata-util\`\``"
9369 describes how to use the ``oe-pkgdata-util`` utility to query
9370 :term:`PKGDATA_DIR` and
9371 display package-related information for built packages.
9372
9373- "`Viewing Dependencies Between Recipes and
9374 Tasks <#dev-viewing-dependencies-between-recipes-and-tasks>`__"
9375 describes how to use the BitBake ``-g`` option to display recipe
9376 dependency information used during the build.
9377
9378- "`Viewing Task Variable
9379 Dependencies <#dev-viewing-task-variable-dependencies>`__" describes
9380 how to use the ``bitbake-dumpsig`` command in conjunction with key
9381 subdirectories in the
9382 :term:`Build Directory` to determine
9383 variable dependencies.
9384
9385- "`Running Specific Tasks <#dev-debugging-taskrunning>`__" describes
9386 how to use several BitBake options (e.g. ``-c``, ``-C``, and ``-f``)
9387 to run specific tasks in the build chain. It can be useful to run
9388 tasks "out-of-order" when trying isolate build issues.
9389
9390- "`General BitBake Problems <#dev-debugging-bitbake>`__" describes how
9391 to use BitBake's ``-D`` debug output option to reveal more about what
9392 BitBake is doing during the build.
9393
9394- "`Building with No Dependencies <#dev-debugging-buildfile>`__"
9395 describes how to use the BitBake ``-b`` option to build a recipe
9396 while ignoring dependencies.
9397
9398- "`Recipe Logging Mechanisms <#recipe-logging-mechanisms>`__"
9399 describes how to use the many recipe logging functions to produce
9400 debugging output and report errors and warnings.
9401
9402- "`Debugging Parallel Make Races <#debugging-parallel-make-races>`__"
9403 describes how to debug situations where the build consists of several
9404 parts that are run simultaneously and when the output or result of
9405 one part is not ready for use with a different part of the build that
9406 depends on that output.
9407
9408- "`Debugging With the GNU Project Debugger (GDB)
9409 Remotely <#platdev-gdb-remotedebug>`__" describes how to use GDB to
9410 allow you to examine running programs, which can help you fix
9411 problems.
9412
9413- "`Debugging with the GNU Project Debugger (GDB) on the
9414 Target <#debugging-with-the-gnu-project-debugger-gdb-on-the-target>`__"
9415 describes how to use GDB directly on target hardware for debugging.
9416
9417- "`Other Debugging Tips <#dev-other-debugging-others>`__" describes
9418 miscellaneous debugging tips that can be useful.
9419
9420.. _dev-debugging-viewing-logs-from-failed-tasks:
9421
9422Viewing Logs from Failed Tasks
9423------------------------------
9424
9425You can find the log for a task in the file
9426``${``\ :term:`WORKDIR`\ ``}/temp/log.do_``\ taskname.
9427For example, the log for the
9428:ref:`ref-tasks-compile` task of the
9429QEMU minimal image for the x86 machine (``qemux86``) might be in
9430``tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/temp/log.do_compile``.
9431To see the commands :term:`BitBake` ran
9432to generate a log, look at the corresponding ``run.do_``\ taskname file
9433in the same directory.
9434
9435``log.do_``\ taskname and ``run.do_``\ taskname are actually symbolic
9436links to ``log.do_``\ taskname\ ``.``\ pid and
9437``log.run_``\ taskname\ ``.``\ pid, where pid is the PID the task had
9438when it ran. The symlinks always point to the files corresponding to the
9439most recent run.
9440
9441.. _dev-debugging-viewing-variable-values:
9442
9443Viewing Variable Values
9444-----------------------
9445
9446Sometimes you need to know the value of a variable as a result of
9447BitBake's parsing step. This could be because some unexpected behavior
9448occurred in your project. Perhaps an attempt to :ref:`modify a variable
9449<bitbake:bitbake-user-manual/bitbake-user-manual-metadata:modifying existing
9450variables>` did not work out as expected.
9451
9452BitBake's ``-e`` option is used to display variable values after
9453parsing. The following command displays the variable values after the
9454configuration files (i.e. ``local.conf``, ``bblayers.conf``,
9455``bitbake.conf`` and so forth) have been parsed:
9456::
9457
9458 $ bitbake -e
9459
9460The following command displays variable values after a specific recipe has
9461been parsed. The variables include those from the configuration as well:
9462::
9463
9464 $ bitbake -e recipename
9465
9466.. note::
9467
9468 Each recipe has its own private set of variables (datastore).
9469 Internally, after parsing the configuration, a copy of the resulting
9470 datastore is made prior to parsing each recipe. This copying implies
9471 that variables set in one recipe will not be visible to other
9472 recipes.
9473
9474 Likewise, each task within a recipe gets a private datastore based on
9475 the recipe datastore, which means that variables set within one task
9476 will not be visible to other tasks.
9477
9478In the output of ``bitbake -e``, each variable is preceded by a
9479description of how the variable got its value, including temporary
9480values that were later overriden. This description also includes
9481variable flags (varflags) set on the variable. The output can be very
9482helpful during debugging.
9483
9484Variables that are exported to the environment are preceded by
9485``export`` in the output of ``bitbake -e``. See the following example:
9486::
9487
9488 export CC="i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/ulf/poky/build/tmp/sysroots/qemux86"
9489
9490In addition to variable values, the output of the ``bitbake -e`` and
9491``bitbake -e`` recipe commands includes the following information:
9492
9493- The output starts with a tree listing all configuration files and
9494 classes included globally, recursively listing the files they include
9495 or inherit in turn. Much of the behavior of the OpenEmbedded build
9496 system (including the behavior of the :ref:`ref-manual/ref-tasks:normal recipe build tasks`) is
9497 implemented in the
9498 :ref:`base <ref-classes-base>` class and the
9499 classes it inherits, rather than being built into BitBake itself.
9500
9501- After the variable values, all functions appear in the output. For
9502 shell functions, variables referenced within the function body are
9503 expanded. If a function has been modified using overrides or using
9504 override-style operators like ``_append`` and ``_prepend``, then the
9505 final assembled function body appears in the output.
9506
9507Viewing Package Information with ``oe-pkgdata-util``
9508----------------------------------------------------
9509
9510You can use the ``oe-pkgdata-util`` command-line utility to query
9511:term:`PKGDATA_DIR` and display
9512various package-related information. When you use the utility, you must
9513use it to view information on packages that have already been built.
9514
9515Following are a few of the available ``oe-pkgdata-util`` subcommands.
9516
9517.. note::
9518
9519 You can use the standard \* and ? globbing wildcards as part of
9520 package names and paths.
9521
9522- ``oe-pkgdata-util list-pkgs [pattern]``: Lists all packages
9523 that have been built, optionally limiting the match to packages that
9524 match pattern.
9525
9526- ``oe-pkgdata-util list-pkg-files package ...``: Lists the
9527 files and directories contained in the given packages.
9528
9529 .. note::
9530
9531 A different way to view the contents of a package is to look at
9532 the
9533 ``${``\ :term:`WORKDIR`\ ``}/packages-split``
9534 directory of the recipe that generates the package. This directory
9535 is created by the
9536 :ref:`ref-tasks-package` task
9537 and has one subdirectory for each package the recipe generates,
9538 which contains the files stored in that package.
9539
9540 If you want to inspect the ``${WORKDIR}/packages-split``
9541 directory, make sure that
9542 :ref:`rm_work <ref-classes-rm-work>` is not
9543 enabled when you build the recipe.
9544
9545- ``oe-pkgdata-util find-path path ...``: Lists the names of
9546 the packages that contain the given paths. For example, the following
9547 tells us that ``/usr/share/man/man1/make.1`` is contained in the
9548 ``make-doc`` package:
9549 ::
9550
9551 $ oe-pkgdata-util find-path /usr/share/man/man1/make.1 make-doc: /usr/share/man/man1/make.1
9552
9553- ``oe-pkgdata-util lookup-recipe package ...``: Lists the name
9554 of the recipes that produce the given packages.
9555
9556For more information on the ``oe-pkgdata-util`` command, use the help
9557facility:
9558::
9559
9560 $ oe-pkgdata-util DASHDASHhelp
9561 $ oe-pkgdata-util subcommand --help
9562
9563.. _dev-viewing-dependencies-between-recipes-and-tasks:
9564
9565Viewing Dependencies Between Recipes and Tasks
9566----------------------------------------------
9567
9568Sometimes it can be hard to see why BitBake wants to build other recipes
9569before the one you have specified. Dependency information can help you
9570understand why a recipe is built.
9571
9572To generate dependency information for a recipe, run the following
9573command:
9574::
9575
9576 $ bitbake -g recipename
9577
9578This command writes the following files in the current directory:
9579
9580- ``pn-buildlist``: A list of recipes/targets involved in building
9581 recipename. "Involved" here means that at least one task from the
9582 recipe needs to run when building recipename from scratch. Targets
9583 that are in
9584 :term:`ASSUME_PROVIDED`
9585 are not listed.
9586
9587- ``task-depends.dot``: A graph showing dependencies between tasks.
9588
9589The graphs are in
9590`DOT <https://en.wikipedia.org/wiki/DOT_%28graph_description_language%29>`__
9591format and can be converted to images (e.g. using the ``dot`` tool from
9592`Graphviz <http://www.graphviz.org/>`__).
9593
9594.. note::
9595
9596 - DOT files use a plain text format. The graphs generated using the
9597 ``bitbake -g`` command are often so large as to be difficult to
9598 read without special pruning (e.g. with Bitbake's ``-I`` option)
9599 and processing. Despite the form and size of the graphs, the
9600 corresponding ``.dot`` files can still be possible to read and
9601 provide useful information.
9602
9603 As an example, the ``task-depends.dot`` file contains lines such
9604 as the following:
9605 ::
9606
9607 "libxslt.do_configure" -> "libxml2.do_populate_sysroot"
9608
9609 The above example line reveals that the
9610 :ref:`ref-tasks-configure`
9611 task in ``libxslt`` depends on the
9612 :ref:`ref-tasks-populate_sysroot`
9613 task in ``libxml2``, which is a normal
9614 :term:`DEPENDS` dependency
9615 between the two recipes.
9616
9617 - For an example of how ``.dot`` files can be processed, see the
9618 ``scripts/contrib/graph-tool`` Python script, which finds and
9619 displays paths between graph nodes.
9620
9621You can use a different method to view dependency information by using
9622the following command:
9623::
9624
9625 $ bitbake -g -u taskexp recipename
9626
9627This command
9628displays a GUI window from which you can view build-time and runtime
9629dependencies for the recipes involved in building recipename.
9630
9631.. _dev-viewing-task-variable-dependencies:
9632
9633Viewing Task Variable Dependencies
9634----------------------------------
9635
9636As mentioned in the
9637":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-execution:checksums (signatures)`" section of the BitBake
9638User Manual, BitBake tries to automatically determine what variables a
9639task depends on so that it can rerun the task if any values of the
9640variables change. This determination is usually reliable. However, if
9641you do things like construct variable names at runtime, then you might
9642have to manually declare dependencies on those variables using
9643``vardeps`` as described in the
9644":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags`" section of the BitBake
9645User Manual.
9646
9647If you are unsure whether a variable dependency is being picked up
9648automatically for a given task, you can list the variable dependencies
9649BitBake has determined by doing the following:
9650
96511. Build the recipe containing the task:
9652::
9653
9654 $ bitbake recipename
9655
96562. Inside the :term:`STAMPS_DIR`
9657 directory, find the signature data (``sigdata``) file that
9658 corresponds to the task. The ``sigdata`` files contain a pickled
9659 Python database of all the metadata that went into creating the input
9660 checksum for the task. As an example, for the
9661 :ref:`ref-tasks-fetch` task of the
9662 ``db`` recipe, the ``sigdata`` file might be found in the following
9663 location:
9664 ::
9665
9666 ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1
9667
9668 For tasks that are accelerated through the shared state
9669 (:ref:`sstate <overview-manual/overview-manual-concepts:shared state cache>`) cache, an
9670 additional ``siginfo`` file is written into
9671 :term:`SSTATE_DIR` along with
9672 the cached task output. The ``siginfo`` files contain exactly the
9673 same information as ``sigdata`` files.
9674
96753. Run ``bitbake-dumpsig`` on the ``sigdata`` or ``siginfo`` file. Here
9676 is an example:
9677 ::
9678
9679 $ bitbake-dumpsig ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1
9680
9681 In the output of the above command, you will find a line like the
9682 following, which lists all the (inferred) variable dependencies for
9683 the task. This list also includes indirect dependencies from
9684 variables depending on other variables, recursively.
9685 ::
9686
9687 Task dependencies: ['PV', 'SRCREV', 'SRC_URI', 'SRC_URI[md5sum]', 'SRC_URI[sha256sum]', 'base_do_fetch']
9688
9689 .. note::
9690
9691 Functions (e.g.
9692 base_do_fetch
9693 ) also count as variable dependencies. These functions in turn
9694 depend on the variables they reference.
9695
9696 The output of ``bitbake-dumpsig`` also includes the value each
9697 variable had, a list of dependencies for each variable, and
9698 :term:`bitbake:BB_HASHBASE_WHITELIST`
9699 information.
9700
9701There is also a ``bitbake-diffsigs`` command for comparing two
9702``siginfo`` or ``sigdata`` files. This command can be helpful when
9703trying to figure out what changed between two versions of a task. If you
9704call ``bitbake-diffsigs`` with just one file, the command behaves like
9705``bitbake-dumpsig``.
9706
9707You can also use BitBake to dump out the signature construction
9708information without executing tasks by using either of the following
9709BitBake command-line options:
9710::
9711
9712 ‐‐dump-signatures=SIGNATURE_HANDLER
9713 -S SIGNATURE_HANDLER
9714
9715
9716.. note::
9717
9718 Two common values for
9719 SIGNATURE_HANDLER
9720 are "none" and "printdiff", which dump only the signature or compare
9721 the dumped signature with the cached one, respectively.
9722
9723Using BitBake with either of these options causes BitBake to dump out
9724``sigdata`` files in the ``stamps`` directory for every task it would
9725have executed instead of building the specified target package.
9726
9727.. _dev-viewing-metadata-used-to-create-the-input-signature-of-a-shared-state-task:
9728
9729Viewing Metadata Used to Create the Input Signature of a Shared State Task
9730--------------------------------------------------------------------------
9731
9732Seeing what metadata went into creating the input signature of a shared
9733state (sstate) task can be a useful debugging aid. This information is
9734available in signature information (``siginfo``) files in
9735:term:`SSTATE_DIR`. For
9736information on how to view and interpret information in ``siginfo``
9737files, see the "`Viewing Task Variable
9738Dependencies <#dev-viewing-task-variable-dependencies>`__" section.
9739
9740For conceptual information on shared state, see the
9741":ref:`overview-manual/overview-manual-concepts:shared state`"
9742section in the Yocto Project Overview and Concepts Manual.
9743
9744.. _dev-invalidating-shared-state-to-force-a-task-to-run:
9745
9746Invalidating Shared State to Force a Task to Run
9747------------------------------------------------
9748
9749The OpenEmbedded build system uses
9750:ref:`checksums <overview-checksums>` and
9751:ref:`overview-manual/overview-manual-concepts:shared state` cache to avoid unnecessarily
9752rebuilding tasks. Collectively, this scheme is known as "shared state
9753code."
9754
9755As with all schemes, this one has some drawbacks. It is possible that
9756you could make implicit changes to your code that the checksum
9757calculations do not take into account. These implicit changes affect a
9758task's output but do not trigger the shared state code into rebuilding a
9759recipe. Consider an example during which a tool changes its output.
9760Assume that the output of ``rpmdeps`` changes. The result of the change
9761should be that all the ``package`` and ``package_write_rpm`` shared
9762state cache items become invalid. However, because the change to the
9763output is external to the code and therefore implicit, the associated
9764shared state cache items do not become invalidated. In this case, the
9765build process uses the cached items rather than running the task again.
9766Obviously, these types of implicit changes can cause problems.
9767
9768To avoid these problems during the build, you need to understand the
9769effects of any changes you make. Realize that changes you make directly
9770to a function are automatically factored into the checksum calculation.
9771Thus, these explicit changes invalidate the associated area of shared
9772state cache. However, you need to be aware of any implicit changes that
9773are not obvious changes to the code and could affect the output of a
9774given task.
9775
9776When you identify an implicit change, you can easily take steps to
9777invalidate the cache and force the tasks to run. The steps you can take
9778are as simple as changing a function's comments in the source code. For
9779example, to invalidate package shared state files, change the comment
9780statements of
9781:ref:`ref-tasks-package` or the
9782comments of one of the functions it calls. Even though the change is
9783purely cosmetic, it causes the checksum to be recalculated and forces
9784the build system to run the task again.
9785
9786.. note::
9787
9788 For an example of a commit that makes a cosmetic change to invalidate
9789 shared state, see this
9790 commit
9791 .
9792
9793.. _dev-debugging-taskrunning:
9794
9795Running Specific Tasks
9796----------------------
9797
9798Any given recipe consists of a set of tasks. The standard BitBake
9799behavior in most cases is: ``do_fetch``, ``do_unpack``, ``do_patch``,
9800``do_configure``, ``do_compile``, ``do_install``, ``do_package``,
9801``do_package_write_*``, and ``do_build``. The default task is
9802``do_build`` and any tasks on which it depends build first. Some tasks,
9803such as ``do_devshell``, are not part of the default build chain. If you
9804wish to run a task that is not part of the default build chain, you can
9805use the ``-c`` option in BitBake. Here is an example:
9806::
9807
9808 $ bitbake matchbox-desktop -c devshell
9809
9810The ``-c`` option respects task dependencies, which means that all other
9811tasks (including tasks from other recipes) that the specified task
9812depends on will be run before the task. Even when you manually specify a
9813task to run with ``-c``, BitBake will only run the task if it considers
9814it "out of date". See the
9815":ref:`overview-manual/overview-manual-concepts:stamp files and the rerunning of tasks`"
9816section in the Yocto Project Overview and Concepts Manual for how
9817BitBake determines whether a task is "out of date".
9818
9819If you want to force an up-to-date task to be rerun (e.g. because you
9820made manual modifications to the recipe's
9821:term:`WORKDIR` that you want to try
9822out), then you can use the ``-f`` option.
9823
9824.. note::
9825
9826 The reason
9827 -f
9828 is never required when running the
9829 do_devshell
9830 task is because the
9831 [
9832 nostamp
9833 ]
9834 variable flag is already set for the task.
9835
9836The following example shows one way you can use the ``-f`` option:
9837::
9838
9839 $ bitbake matchbox-desktop
9840 .
9841 .
9842 make some changes to the source code in the work directory
9843 .
9844 .
9845 $ bitbake matchbox-desktop -c compile -f
9846 $ bitbake matchbox-desktop
9847
9848This sequence first builds and then recompiles ``matchbox-desktop``. The
9849last command reruns all tasks (basically the packaging tasks) after the
9850compile. BitBake recognizes that the ``do_compile`` task was rerun and
9851therefore understands that the other tasks also need to be run again.
9852
9853Another, shorter way to rerun a task and all
9854:ref:`ref-manual/ref-tasks:normal recipe build tasks`
9855that depend on it is to use the ``-C`` option.
9856
9857.. note::
9858
9859 This option is upper-cased and is separate from the
9860 -c
9861 option, which is lower-cased.
9862
9863Using this option invalidates the given task and then runs the
9864:ref:`ref-tasks-build` task, which is
9865the default task if no task is given, and the tasks on which it depends.
9866You could replace the final two commands in the previous example with
9867the following single command:
9868::
9869
9870 $ bitbake matchbox-desktop -C compile
9871
9872Internally, the ``-f`` and ``-C`` options work by tainting (modifying)
9873the input checksum of the specified task. This tainting indirectly
9874causes the task and its dependent tasks to be rerun through the normal
9875task dependency mechanisms.
9876
9877.. note::
9878
9879 BitBake explicitly keeps track of which tasks have been tainted in
9880 this fashion, and will print warnings such as the following for
9881 builds involving such tasks:
9882 ::
9883
9884 WARNING: /home/ulf/poky/meta/recipes-sato/matchbox-desktop/matchbox-desktop_2.1.bb.do_compile is tainted from a forced run
9885
9886
9887 The purpose of the warning is to let you know that the work directory
9888 and build output might not be in the clean state they would be in for
9889 a "normal" build, depending on what actions you took. To get rid of
9890 such warnings, you can remove the work directory and rebuild the
9891 recipe, as follows:
9892 ::
9893
9894 $ bitbake matchbox-desktop -c clean
9895 $ bitbake matchbox-desktop
9896
9897
9898You can view a list of tasks in a given package by running the
9899``do_listtasks`` task as follows:
9900::
9901
9902 $ bitbake matchbox-desktop -c listtasks
9903
9904The results appear as output to the console and are also in
9905the file ``${WORKDIR}/temp/log.do_listtasks``.
9906
9907.. _dev-debugging-bitbake:
9908
9909General BitBake Problems
9910------------------------
9911
9912You can see debug output from BitBake by using the ``-D`` option. The
9913debug output gives more information about what BitBake is doing and the
9914reason behind it. Each ``-D`` option you use increases the logging
9915level. The most common usage is ``-DDD``.
9916
9917The output from ``bitbake -DDD -v`` targetname can reveal why BitBake
9918chose a certain version of a package or why BitBake picked a certain
9919provider. This command could also help you in a situation where you
9920think BitBake did something unexpected.
9921
9922.. _dev-debugging-buildfile:
9923
9924Building with No Dependencies
9925-----------------------------
9926
9927To build a specific recipe (``.bb`` file), you can use the following
9928command form:
9929::
9930
9931 $ bitbake -b somepath/somerecipe.bb
9932
9933This command form does
9934not check for dependencies. Consequently, you should use it only when
9935you know existing dependencies have been met.
9936
9937.. note::
9938
9939 You can also specify fragments of the filename. In this case, BitBake
9940 checks for a unique match.
9941
9942Recipe Logging Mechanisms
9943-------------------------
9944
9945The Yocto Project provides several logging functions for producing
9946debugging output and reporting errors and warnings. For Python
9947functions, the following logging functions exist. All of these functions
9948log to ``${T}/log.do_``\ task, and can also log to standard output
9949(stdout) with the right settings:
9950
9951- ``bb.plain(msg)``: Writes msg as is to the log while also
9952 logging to stdout.
9953
9954- ``bb.note(msg)``: Writes "NOTE: msg" to the log. Also logs to
9955 stdout if BitBake is called with "-v".
9956
9957- ``bb.debug(level, msg)``: Writes "DEBUG: msg" to the
9958 log. Also logs to stdout if the log level is greater than or equal to
9959 level. See the ":ref:`-D <bitbake:bitbake-user-manual/bitbake-user-manual-intro:usage and syntax>`" option
9960 in the BitBake User Manual for more information.
9961
9962- ``bb.warn(msg)``: Writes "WARNING: msg" to the log while also
9963 logging to stdout.
9964
9965- ``bb.error(msg)``: Writes "ERROR: msg" to the log while also
9966 logging to standard out (stdout).
9967
9968 .. note::
9969
9970 Calling this function does not cause the task to fail.
9971
9972- ``bb.fatal(``\ msg\ ``)``: This logging function is similar to
9973 ``bb.error(``\ msg\ ``)`` but also causes the calling task to fail.
9974
9975 .. note::
9976
9977 bb.fatal()
9978 raises an exception, which means you do not need to put a "return"
9979 statement after the function.
9980
9981The same logging functions are also available in shell functions, under
9982the names ``bbplain``, ``bbnote``, ``bbdebug``, ``bbwarn``, ``bberror``,
9983and ``bbfatal``. The
9984:ref:`logging <ref-classes-logging>` class
9985implements these functions. See that class in the ``meta/classes``
9986folder of the :term:`Source Directory` for information.
9987
9988Logging With Python
9989~~~~~~~~~~~~~~~~~~~
9990
9991When creating recipes using Python and inserting code that handles build
9992logs, keep in mind the goal is to have informative logs while keeping
9993the console as "silent" as possible. Also, if you want status messages
9994in the log, use the "debug" loglevel.
9995
9996Following is an example written in Python. The code handles logging for
9997a function that determines the number of tasks needed to be run. See the
9998":ref:`ref-tasks-listtasks`"
9999section for additional information:
10000::
10001
10002 python do_listtasks() {
10003 bb.debug(2, "Starting to figure out the task list")
10004 if noteworthy_condition:
10005 bb.note("There are 47 tasks to run")
10006 bb.debug(2, "Got to point xyz")
10007 if warning_trigger:
10008 bb.warn("Detected warning_trigger, this might be a problem later.")
10009 if recoverable_error:
10010 bb.error("Hit recoverable_error, you really need to fix this!")
10011 if fatal_error:
10012 bb.fatal("fatal_error detected, unable to print the task list")
10013 bb.plain("The tasks present are abc")
10014 bb.debug(2, "Finished figuring out the tasklist")
10015 }
10016
10017Logging With Bash
10018~~~~~~~~~~~~~~~~~
10019
10020When creating recipes using Bash and inserting code that handles build
10021logs, you have the same goals - informative with minimal console output.
10022The syntax you use for recipes written in Bash is similar to that of
10023recipes written in Python described in the previous section.
10024
10025Following is an example written in Bash. The code logs the progress of
10026the ``do_my_function`` function.
10027::
10028
10029 do_my_function() {
10030 bbdebug 2 "Running do_my_function"
10031 if [ exceptional_condition ]; then
10032 bbnote "Hit exceptional_condition"
10033 fi
10034 bbdebug 2 "Got to point xyz"
10035 if [ warning_trigger ]; then
10036 bbwarn "Detected warning_trigger, this might cause a problem later."
10037 fi
10038 if [ recoverable_error ]; then
10039 bberror "Hit recoverable_error, correcting"
10040 fi
10041 if [ fatal_error ]; then
10042 bbfatal "fatal_error detected"
10043 fi
10044 bbdebug 2 "Completed do_my_function"
10045 }
10046
10047
10048Debugging Parallel Make Races
10049-----------------------------
10050
10051A parallel ``make`` race occurs when the build consists of several parts
10052that are run simultaneously and a situation occurs when the output or
10053result of one part is not ready for use with a different part of the
10054build that depends on that output. Parallel make races are annoying and
10055can sometimes be difficult to reproduce and fix. However, some simple
10056tips and tricks exist that can help you debug and fix them. This section
10057presents a real-world example of an error encountered on the Yocto
10058Project autobuilder and the process used to fix it.
10059
10060.. note::
10061
10062 If you cannot properly fix a
10063 make
10064 race condition, you can work around it by clearing either the
10065 PARALLEL_MAKE
10066 or
10067 PARALLEL_MAKEINST
10068 variables.
10069
10070The Failure
10071~~~~~~~~~~~
10072
10073For this example, assume that you are building an image that depends on
10074the "neard" package. And, during the build, BitBake runs into problems
10075and creates the following output.
10076
10077.. note::
10078
10079 This example log file has longer lines artificially broken to make
10080 the listing easier to read.
10081
10082If you examine the output or the log file, you see the failure during
10083``make``:
10084::
10085
10086 | DEBUG: SITE files ['endian-little', 'bit-32', 'ix86-common', 'common-linux', 'common-glibc', 'i586-linux', 'common']
10087 | DEBUG: Executing shell function do_compile
10088 | NOTE: make -j 16
10089 | make --no-print-directory all-am
10090 | /bin/mkdir -p include/near
10091 | /bin/mkdir -p include/near
10092 | /bin/mkdir -p include/near
10093 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
10094 0.14-r0/neard-0.14/include/types.h include/near/types.h
10095 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
10096 0.14-r0/neard-0.14/include/log.h include/near/log.h
10097 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
10098 0.14-r0/neard-0.14/include/plugin.h include/near/plugin.h
10099 | /bin/mkdir -p include/near
10100 | /bin/mkdir -p include/near
10101 | /bin/mkdir -p include/near
10102 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
10103 0.14-r0/neard-0.14/include/tag.h include/near/tag.h
10104 | /bin/mkdir -p include/near
10105 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
10106 0.14-r0/neard-0.14/include/adapter.h include/near/adapter.h
10107 | /bin/mkdir -p include/near
10108 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
10109 0.14-r0/neard-0.14/include/ndef.h include/near/ndef.h
10110 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
10111 0.14-r0/neard-0.14/include/tlv.h include/near/tlv.h
10112 | /bin/mkdir -p include/near
10113 | /bin/mkdir -p include/near
10114 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
10115 0.14-r0/neard-0.14/include/setting.h include/near/setting.h
10116 | /bin/mkdir -p include/near
10117 | /bin/mkdir -p include/near
10118 | /bin/mkdir -p include/near
10119 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
10120 0.14-r0/neard-0.14/include/device.h include/near/device.h
10121 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
10122 0.14-r0/neard-0.14/include/nfc_copy.h include/near/nfc_copy.h
10123 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
10124 0.14-r0/neard-0.14/include/snep.h include/near/snep.h
10125 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
10126 0.14-r0/neard-0.14/include/version.h include/near/version.h
10127 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
10128 0.14-r0/neard-0.14/include/dbus.h include/near/dbus.h
10129 | ./src/genbuiltin nfctype1 nfctype2 nfctype3 nfctype4 p2p > src/builtin.h
10130 | i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/
10131 build/build/tmp/sysroots/qemux86 -DHAVE_CONFIG_H -I. -I./include -I./src -I./gdbus -I/home/pokybuild/
10132 yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/sysroots/qemux86/usr/include/glib-2.0
10133 -I/home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/sysroots/qemux86/usr/
10134 lib/glib-2.0/include -I/home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/
10135 tmp/sysroots/qemux86/usr/include/dbus-1.0 -I/home/pokybuild/yocto-autobuilder/yocto-slave/
10136 nightly-x86/build/build/tmp/sysroots/qemux86/usr/lib/dbus-1.0/include -I/home/pokybuild/yocto-autobuilder/
10137 yocto-slave/nightly-x86/build/build/tmp/sysroots/qemux86/usr/include/libnl3
10138 -DNEAR_PLUGIN_BUILTIN -DPLUGINDIR=\""/usr/lib/near/plugins"\"
10139 -DCONFIGDIR=\""/etc/neard\"" -O2 -pipe -g -feliminate-unused-debug-types -c
10140 -o tools/snep-send.o tools/snep-send.c
10141 | In file included from tools/snep-send.c:16:0:
10142 | tools/../src/near.h:41:23: fatal error: near/dbus.h: No such file or directory
10143 | #include <near/dbus.h>
10144 | ^
10145 | compilation terminated.
10146 | make[1]: *** [tools/snep-send.o] Error 1
10147 | make[1]: *** Waiting for unfinished jobs....
10148 | make: *** [all] Error 2
10149 | ERROR: oe_runmake failed
10150
10151Reproducing the Error
10152~~~~~~~~~~~~~~~~~~~~~
10153
10154Because race conditions are intermittent, they do not manifest
10155themselves every time you do the build. In fact, most times the build
10156will complete without problems even though the potential race condition
10157exists. Thus, once the error surfaces, you need a way to reproduce it.
10158
10159In this example, compiling the "neard" package is causing the problem.
10160So the first thing to do is build "neard" locally. Before you start the
10161build, set the
10162:term:`PARALLEL_MAKE` variable
10163in your ``local.conf`` file to a high number (e.g. "-j 20"). Using a
10164high value for ``PARALLEL_MAKE`` increases the chances of the race
10165condition showing up:
10166::
10167
10168 $ bitbake neard
10169
10170Once the local build for "neard" completes, start a ``devshell`` build:
10171::
10172
10173 $ bitbake neard -c devshell
10174
10175For information on how to use a
10176``devshell``, see the "`Using a Development
10177Shell <#platdev-appdev-devshell>`__" section.
10178
10179In the ``devshell``, do the following:
10180::
10181
10182 $ make clean
10183 $ make tools/snep-send.o
10184
10185The ``devshell`` commands cause the failure to clearly
10186be visible. In this case, a missing dependency exists for the "neard"
10187Makefile target. Here is some abbreviated, sample output with the
10188missing dependency clearly visible at the end:
10189::
10190
10191 i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/scott-lenovo/......
10192 .
10193 .
10194 .
10195 tools/snep-send.c
10196 In file included from tools/snep-send.c:16:0:
10197 tools/../src/near.h:41:23: fatal error: near/dbus.h: No such file or directory
10198 #include <near/dbus.h>
10199 ^
10200 compilation terminated.
10201 make: *** [tools/snep-send.o] Error 1
10202 $
10203
10204
10205Creating a Patch for the Fix
10206~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10207
10208Because there is a missing dependency for the Makefile target, you need
10209to patch the ``Makefile.am`` file, which is generated from
10210``Makefile.in``. You can use Quilt to create the patch:
10211::
10212
10213 $ quilt new parallelmake.patch
10214 Patch patches/parallelmake.patch is now on top
10215 $ quilt add Makefile.am
10216 File Makefile.am added to patch patches/parallelmake.patch
10217
10218For more information on using Quilt, see the
10219"`Using Quilt in Your Workflow <#using-a-quilt-workflow>`__" section.
10220
10221At this point you need to make the edits to ``Makefile.am`` to add the
10222missing dependency. For our example, you have to add the following line
10223to the file:
10224::
10225
10226 tools/snep-send.$(OBJEXT): include/near/dbus.h
10227
10228Once you have edited the file, use the ``refresh`` command to create the
10229patch:
10230::
10231
10232 $ quilt refresh
10233 Refreshed patch patches/parallelmake.patch
10234
10235Once
10236the patch file exists, you need to add it back to the originating recipe
10237folder. Here is an example assuming a top-level
10238:term:`Source Directory` named ``poky``:
10239::
10240
10241 $ cp patches/parallelmake.patch poky/meta/recipes-connectivity/neard/neard
10242
10243The final thing you need to do to implement the fix in the build is to
10244update the "neard" recipe (i.e. ``neard-0.14.bb``) so that the
10245:term:`SRC_URI` statement includes
10246the patch file. The recipe file is in the folder above the patch. Here
10247is what the edited ``SRC_URI`` statement would look like:
10248::
10249
10250 SRC_URI = "${KERNELORG_MIRROR}/linux/network/nfc/${BPN}-${PV}.tar.xz \
10251 file://neard.in \
10252 file://neard.service.in \
10253 file://parallelmake.patch \
10254 "
10255
10256With the patch complete and moved to the correct folder and the
10257``SRC_URI`` statement updated, you can exit the ``devshell``:
10258::
10259
10260 $ exit
10261
10262Testing the Build
10263~~~~~~~~~~~~~~~~~
10264
10265With everything in place, you can get back to trying the build again
10266locally:
10267::
10268
10269 $ bitbake neard This build should succeed.
10270
10271Now you can open up a ``devshell`` again and repeat the clean and make
10272operations as follows:
10273::
10274
10275 $ bitbake neard -c devshell
10276 $ make clean
10277 $ make tools/snep-send.o
10278
10279The build should work without issue.
10280
10281As with all solved problems, if they originated upstream, you need to
10282submit the fix for the recipe in OE-Core and upstream so that the
10283problem is taken care of at its source. See the "`Submitting a Change to
10284the Yocto Project <#how-to-submit-a-change>`__" section for more
10285information.
10286
10287.. _platdev-gdb-remotedebug:
10288
10289Debugging With the GNU Project Debugger (GDB) Remotely
10290------------------------------------------------------
10291
10292GDB allows you to examine running programs, which in turn helps you to
10293understand and fix problems. It also allows you to perform post-mortem
10294style analysis of program crashes. GDB is available as a package within
10295the Yocto Project and is installed in SDK images by default. See the
10296":ref:`ref-manual/ref-images:Images`" chapter in the Yocto
10297Project Reference Manual for a description of these images. You can find
10298information on GDB at http://sourceware.org/gdb/.
10299
10300.. note::
10301
10302 For best results, install debug (
10303 -dbg
10304 ) packages for the applications you are going to debug. Doing so
10305 makes extra debug symbols available that give you more meaningful
10306 output.
10307
10308Sometimes, due to memory or disk space constraints, it is not possible
10309to use GDB directly on the remote target to debug applications. These
10310constraints arise because GDB needs to load the debugging information
10311and the binaries of the process being debugged. Additionally, GDB needs
10312to perform many computations to locate information such as function
10313names, variable names and values, stack traces and so forth - even
10314before starting the debugging process. These extra computations place
10315more load on the target system and can alter the characteristics of the
10316program being debugged.
10317
10318To help get past the previously mentioned constraints, you can use
10319gdbserver, which runs on the remote target and does not load any
10320debugging information from the debugged process. Instead, a GDB instance
10321processes the debugging information that is run on a remote computer -
10322the host GDB. The host GDB then sends control commands to gdbserver to
10323make it stop or start the debugged program, as well as read or write
10324memory regions of that debugged program. All the debugging information
10325loaded and processed as well as all the heavy debugging is done by the
10326host GDB. Offloading these processes gives the gdbserver running on the
10327target a chance to remain small and fast.
10328
10329Because the host GDB is responsible for loading the debugging
10330information and for doing the necessary processing to make actual
10331debugging happen, you have to make sure the host can access the
10332unstripped binaries complete with their debugging information and also
10333be sure the target is compiled with no optimizations. The host GDB must
10334also have local access to all the libraries used by the debugged
10335program. Because gdbserver does not need any local debugging
10336information, the binaries on the remote target can remain stripped.
10337However, the binaries must also be compiled without optimization so they
10338match the host's binaries.
10339
10340To remain consistent with GDB documentation and terminology, the binary
10341being debugged on the remote target machine is referred to as the
10342"inferior" binary. For documentation on GDB see the `GDB
10343site <http://sourceware.org/gdb/documentation/>`__.
10344
10345The following steps show you how to debug using the GNU project
10346debugger.
10347
103481. *Configure your build system to construct the companion debug
10349 filesystem:*
10350
10351 In your ``local.conf`` file, set the following:
10352 ::
10353
10354 IMAGE_GEN_DEBUGFS = "1"
10355 IMAGE_FSTYPES_DEBUGFS = "tar.bz2"
10356
10357 These options cause the
10358 OpenEmbedded build system to generate a special companion filesystem
10359 fragment, which contains the matching source and debug symbols to
10360 your deployable filesystem. The build system does this by looking at
10361 what is in the deployed filesystem, and pulling the corresponding
10362 ``-dbg`` packages.
10363
10364 The companion debug filesystem is not a complete filesystem, but only
10365 contains the debug fragments. This filesystem must be combined with
10366 the full filesystem for debugging. Subsequent steps in this procedure
10367 show how to combine the partial filesystem with the full filesystem.
10368
103692. *Configure the system to include gdbserver in the target filesystem:*
10370
10371 Make the following addition in either your ``local.conf`` file or in
10372 an image recipe:
10373 ::
10374
10375 IMAGE_INSTALL_append = " gdbserver"
10376
10377 The change makes
10378 sure the ``gdbserver`` package is included.
10379
103803. *Build the environment:*
10381
10382 Use the following command to construct the image and the companion
10383 Debug Filesystem:
10384 ::
10385
10386 $ bitbake image
10387
10388 Build the cross GDB component and
10389 make it available for debugging. Build the SDK that matches the
10390 image. Building the SDK is best for a production build that can be
10391 used later for debugging, especially during long term maintenance:
10392 ::
10393
10394 $ bitbake -c populate_sdk image
10395
10396 Alternatively, you can build the minimal toolchain components that
10397 match the target. Doing so creates a smaller than typical SDK and
10398 only contains a minimal set of components with which to build simple
10399 test applications, as well as run the debugger:
10400 ::
10401
10402 $ bitbake meta-toolchain
10403
10404 A final method is to build Gdb itself within the build system:
10405 ::
10406
10407 $ bitbake gdb-cross-<architecture>
10408
10409 Doing so produces a temporary copy of
10410 ``cross-gdb`` you can use for debugging during development. While
10411 this is the quickest approach, the two previous methods in this step
10412 are better when considering long-term maintenance strategies.
10413
10414 .. note::
10415
10416 If you run
10417 bitbake gdb-cross
10418 , the OpenEmbedded build system suggests the actual image (e.g.
10419 gdb-cross-i586
10420 ). The suggestion is usually the actual name you want to use.
10421
104224. *Set up the* ``debugfs``
10423
10424 Run the following commands to set up the ``debugfs``:
10425 ::
10426
10427 $ mkdir debugfs
10428 $ cd debugfs
10429 $ tar xvfj build-dir/tmp-glibc/deploy/images/machine/image.rootfs.tar.bz2
10430 $ tar xvfj build-dir/tmp-glibc/deploy/images/machine/image-dbg.rootfs.tar.bz2
10431
104325. *Set up GDB*
10433
10434 Install the SDK (if you built one) and then source the correct
10435 environment file. Sourcing the environment file puts the SDK in your
10436 ``PATH`` environment variable.
10437
10438 If you are using the build system, Gdb is located in
10439 build-dir/tmp/sysroots/host/usr/bin/architecture/architecture-gdb
10440
104416. *Boot the target:*
10442
10443 For information on how to run QEMU, see the `QEMU
10444 Documentation <http://wiki.qemu.org/Documentation/GettingStartedDevelopers>`__.
10445
10446 .. note::
10447
10448 Be sure to verify that your host can access the target via TCP.
10449
104507. *Debug a program:*
10451
10452 Debugging a program involves running gdbserver on the target and then
10453 running Gdb on the host. The example in this step debugs ``gzip``:
10454 ::
10455
10456 root@qemux86:~# gdbserver localhost:1234 /bin/gzip —help
10457
10458 For
10459 additional gdbserver options, see the `GDB Server
10460 Documentation <https://www.gnu.org/software/gdb/documentation/>`__.
10461
10462 After running gdbserver on the target, you need to run Gdb on the
10463 host and configure it and connect to the target. Use these commands:
10464 ::
10465
10466 $ cd directory-holding-the-debugfs-directory
10467 $ arch-gdb
10468 (gdb) set sysroot debugfs
10469 (gdb) set substitute-path /usr/src/debug debugfs/usr/src/debug
10470 (gdb) target remote IP-of-target:1234
10471
10472 At this
10473 point, everything should automatically load (i.e. matching binaries,
10474 symbols and headers).
10475
10476 .. note::
10477
10478 The Gdb
10479 set
10480 commands in the previous example can be placed into the users
10481 ~/.gdbinit
10482 file. Upon starting, Gdb automatically runs whatever commands are
10483 in that file.
10484
104858. *Deploying without a full image rebuild:*
10486
10487 In many cases, during development you want a quick method to deploy a
10488 new binary to the target and debug it, without waiting for a full
10489 image build.
10490
10491 One approach to solving this situation is to just build the component
10492 you want to debug. Once you have built the component, copy the
10493 executable directly to both the target and the host ``debugfs``.
10494
10495 If the binary is processed through the debug splitting in
10496 OpenEmbedded, you should also copy the debug items (i.e. ``.debug``
10497 contents and corresponding ``/usr/src/debug`` files) from the work
10498 directory. Here is an example:
10499 ::
10500
10501 $ bitbake bash
10502 $ bitbake -c devshell bash
10503 $ cd ..
10504 $ scp packages-split/bash/bin/bash target:/bin/bash
10505 $ cp -a packages-split/bash-dbg/\* path/debugfs
10506
10507Debugging with the GNU Project Debugger (GDB) on the Target
10508-----------------------------------------------------------
10509
10510The previous section addressed using GDB remotely for debugging
10511purposes, which is the most usual case due to the inherent hardware
10512limitations on many embedded devices. However, debugging in the target
10513hardware itself is also possible with more powerful devices. This
10514section describes what you need to do in order to support using GDB to
10515debug on the target hardware.
10516
10517To support this kind of debugging, you need do the following:
10518
10519- Ensure that GDB is on the target. You can do this by adding "gdb" to
10520 :term:`IMAGE_INSTALL`:
10521 IMAGE_INSTALL_append = " gdb" Alternatively, you can add
10522 "tools-debug" to
10523 :term:`IMAGE_FEATURES`:
10524 ::
10525
10526 IMAGE_FEATURES_append = " tools-debug"
10527
10528- Ensure that debug symbols are present. You can make sure these
10529 symbols are present by installing ``-dbg``:
10530 ::
10531
10532 IMAGE_INSTALL_append = "packagename-dbg"
10533
10534 Alternatively, you can do the following to include
10535 all the debug symbols:
10536 ::
10537
10538 IMAGE_FEATURES_append = " dbg-pkgs"
10539
10540.. note::
10541
10542 To improve the debug information accuracy, you can reduce the level
10543 of optimization used by the compiler. For example, when adding the
10544 following line to your
10545 local.conf
10546 file, you will reduce optimization from
10547 FULL_OPTIMIZATION
10548 of "-O2" to
10549 DEBUG_OPTIMIZATION
10550 of "-O -fno-omit-frame-pointer":
10551 ::
10552
10553 DEBUG_BUILD = "1"
10554
10555
10556 Consider that this will reduce the application's performance and is
10557 recommended only for debugging purposes.
10558
10559.. _dev-other-debugging-others:
10560
10561Other Debugging Tips
10562--------------------
10563
10564Here are some other tips that you might find useful:
10565
10566- When adding new packages, it is worth watching for undesirable items
10567 making their way into compiler command lines. For example, you do not
10568 want references to local system files like ``/usr/lib/`` or
10569 ``/usr/include/``.
10570
10571- If you want to remove the ``psplash`` boot splashscreen, add
10572 ``psplash=false`` to the kernel command line. Doing so prevents
10573 ``psplash`` from loading and thus allows you to see the console. It
10574 is also possible to switch out of the splashscreen by switching the
10575 virtual console (e.g. Fn+Left or Fn+Right on a Zaurus).
10576
10577- Removing :term:`TMPDIR` (usually
10578 ``tmp/``, within the
10579 :term:`Build Directory`) can often fix
10580 temporary build issues. Removing ``TMPDIR`` is usually a relatively
10581 cheap operation, because task output will be cached in
10582 :term:`SSTATE_DIR` (usually
10583 ``sstate-cache/``, which is also in the Build Directory).
10584
10585 .. note::
10586
10587 Removing
10588 TMPDIR
10589 might be a workaround rather than a fix. Consequently, trying to
10590 determine the underlying cause of an issue before removing the
10591 directory is a good idea.
10592
10593- Understanding how a feature is used in practice within existing
10594 recipes can be very helpful. It is recommended that you configure
10595 some method that allows you to quickly search through files.
10596
10597 Using GNU Grep, you can use the following shell function to
10598 recursively search through common recipe-related files, skipping
10599 binary files, ``.git`` directories, and the Build Directory (assuming
10600 its name starts with "build"):
10601 ::
10602
10603 g() {
10604 grep -Ir \
10605 --exclude-dir=.git \
10606 --exclude-dir='build*' \
10607 --include='*.bb*' \
10608 --include='*.inc*' \
10609 --include='*.conf*' \
10610 --include='*.py*' \
10611 "$@"
10612 }
10613
10614 Following are some usage examples:
10615 ::
10616
10617 $ g FOO # Search recursively for "FOO"
10618 $ g -i foo # Search recursively for "foo", ignoring case
10619 $ g -w FOO # Search recursively for "FOO" as a word, ignoring e.g. "FOOBAR"
10620
10621 If figuring
10622 out how some feature works requires a lot of searching, it might
10623 indicate that the documentation should be extended or improved. In
10624 such cases, consider filing a documentation bug using the Yocto
10625 Project implementation of
10626 :yocto_bugs:`Bugzilla <>`. For information on
10627 how to submit a bug against the Yocto Project, see the Yocto Project
10628 Bugzilla :yocto_wiki:`wiki page </wiki/Bugzilla_Configuration_and_Bug_Tracking>`
10629 and the "`Submitting a Defect Against the Yocto
10630 Project <#submitting-a-defect-against-the-yocto-project>`__" section.
10631
10632 .. note::
10633
10634 The manuals might not be the right place to document variables
10635 that are purely internal and have a limited scope (e.g. internal
10636 variables used to implement a single
10637 .bbclass
10638 file).
10639
10640Making Changes to the Yocto Project
10641===================================
10642
10643Because the Yocto Project is an open-source, community-based project,
10644you can effect changes to the project. This section presents procedures
10645that show you how to submit a defect against the project and how to
10646submit a change.
10647
10648Submitting a Defect Against the Yocto Project
10649---------------------------------------------
10650
10651Use the Yocto Project implementation of
10652`Bugzilla <http://www.bugzilla.org/about/>`__ to submit a defect (bug)
10653against the Yocto Project. For additional information on this
10654implementation of Bugzilla see the :ref:"`Yocto Project
10655Bugzilla <resources-bugtracker>`" section in the
10656Yocto Project Reference Manual. For more detail on any of the following
10657steps, see the Yocto Project
10658:yocto_wiki:`Bugzilla wiki page </wiki/Bugzilla_Configuration_and_Bug_Tracking>`.
10659
10660Use the following general steps to submit a bug"
10661
106621. Open the Yocto Project implementation of :yocto_bugs:`Bugzilla <>`.
10663
106642. Click "File a Bug" to enter a new bug.
10665
106663. Choose the appropriate "Classification", "Product", and "Component"
10667 for which the bug was found. Bugs for the Yocto Project fall into
10668 one of several classifications, which in turn break down into
10669 several products and components. For example, for a bug against the
10670 ``meta-intel`` layer, you would choose "Build System, Metadata &
10671 Runtime", "BSPs", and "bsps-meta-intel", respectively.
10672
106734. Choose the "Version" of the Yocto Project for which you found the
10674 bug (e.g. DISTRO).
10675
106765. Determine and select the "Severity" of the bug. The severity
10677 indicates how the bug impacted your work.
10678
106796. Choose the "Hardware" that the bug impacts.
10680
106817. Choose the "Architecture" that the bug impacts.
10682
106838. Choose a "Documentation change" item for the bug. Fixing a bug might
10684 or might not affect the Yocto Project documentation. If you are
10685 unsure of the impact to the documentation, select "Don't Know".
10686
106879. Provide a brief "Summary" of the bug. Try to limit your summary to
10688 just a line or two and be sure to capture the essence of the bug.
10689
1069010. Provide a detailed "Description" of the bug. You should provide as
10691 much detail as you can about the context, behavior, output, and so
10692 forth that surrounds the bug. You can even attach supporting files
10693 for output from logs by using the "Add an attachment" button.
10694
1069511. Click the "Submit Bug" button submit the bug. A new Bugzilla number
10696 is assigned to the bug and the defect is logged in the bug tracking
10697 system.
10698
10699Once you file a bug, the bug is processed by the Yocto Project Bug
10700Triage Team and further details concerning the bug are assigned (e.g.
10701priority and owner). You are the "Submitter" of the bug and any further
10702categorization, progress, or comments on the bug result in Bugzilla
10703sending you an automated email concerning the particular change or
10704progress to the bug.
10705
10706.. _how-to-submit-a-change:
10707
10708Submitting a Change to the Yocto Project
10709----------------------------------------
10710
10711Contributions to the Yocto Project and OpenEmbedded are very welcome.
10712Because the system is extremely configurable and flexible, we recognize
10713that developers will want to extend, configure or optimize it for their
10714specific uses.
10715
10716The Yocto Project uses a mailing list and a patch-based workflow that is
10717similar to the Linux kernel but contains important differences. In
10718general, a mailing list exists through which you can submit patches. You
10719should send patches to the appropriate mailing list so that they can be
10720reviewed and merged by the appropriate maintainer. The specific mailing
10721list you need to use depends on the location of the code you are
10722changing. Each component (e.g. layer) should have a ``README`` file that
10723indicates where to send the changes and which process to follow.
10724
10725You can send the patch to the mailing list using whichever approach you
10726feel comfortable with to generate the patch. Once sent, the patch is
10727usually reviewed by the community at large. If somebody has concerns
10728with the patch, they will usually voice their concern over the mailing
10729list. If a patch does not receive any negative reviews, the maintainer
10730of the affected layer typically takes the patch, tests it, and then
10731based on successful testing, merges the patch.
10732
10733The "poky" repository, which is the Yocto Project's reference build
10734environment, is a hybrid repository that contains several individual
10735pieces (e.g. BitBake, Metadata, documentation, and so forth) built using
10736the combo-layer tool. The upstream location used for submitting changes
10737varies by component:
10738
10739- *Core Metadata:* Send your patch to the
10740 `openembedded-core <http://lists.openembedded.org/mailman/listinfo/openembedded-core>`__
10741 mailing list. For example, a change to anything under the ``meta`` or
10742 ``scripts`` directories should be sent to this mailing list.
10743
10744- *BitBake:* For changes to BitBake (i.e. anything under the
10745 ``bitbake`` directory), send your patch to the
10746 `bitbake-devel <http://lists.openembedded.org/mailman/listinfo/bitbake-devel>`__
10747 mailing list.
10748
Andrew Geisslerc3d88e42020-10-02 09:45:00 -050010749- *"meta-\*" trees:* These trees contain Metadata. Use the
10750 `poky <https://lists.yoctoproject.org/g/poky>`__ mailing list.
10751
10752- *Documentation*: For changes to the Yocto Project documentation, use the `docs
10753 <https://lists.yoctoproject.org/g/docs>`__ mailing list.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010754
10755For changes to other layers hosted in the Yocto Project source
Andrew Geisslerc3d88e42020-10-02 09:45:00 -050010756repositories (i.e. ``yoctoproject.org``) and tools use the `Yocto Project
10757<https://lists.yoctoproject.org/g/yocto/>`__ general mailing list.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010758
10759.. note::
10760
10761 Sometimes a layer's documentation specifies to use a particular
10762 mailing list. If so, use that list.
10763
10764For additional recipes that do not fit into the core Metadata, you
10765should determine which layer the recipe should go into and submit the
10766change in the manner recommended by the documentation (e.g. the
10767``README`` file) supplied with the layer. If in doubt, please ask on the
10768Yocto general mailing list or on the openembedded-devel mailing list.
10769
10770You can also push a change upstream and request a maintainer to pull the
10771change into the component's upstream repository. You do this by pushing
10772to a contribution repository that is upstream. See the ":ref:`gs-git-workflows-and-the-yocto-project`"
10773section in the Yocto Project Overview and Concepts Manual for additional
10774concepts on working in the Yocto Project development environment.
10775
10776Two commonly used testing repositories exist for OpenEmbedded-Core:
10777
10778- *"ross/mut" branch:* The "mut" (master-under-test) tree exists in the
10779 ``poky-contrib`` repository in the
10780 :yocto_git:`Yocto Project source repositories <>`.
10781
10782- *"master-next" branch:* This branch is part of the main "poky"
10783 repository in the Yocto Project source repositories.
10784
10785Maintainers use these branches to test submissions prior to merging
10786patches. Thus, you can get an idea of the status of a patch based on
10787whether the patch has been merged into one of these branches.
10788
10789.. note::
10790
10791 This system is imperfect and changes can sometimes get lost in the
10792 flow. Asking about the status of a patch or change is reasonable if
10793 the change has been idle for a while with no feedback. The Yocto
10794 Project does have plans to use
10795 Patchwork
10796 to track the status of patches and also to automatically preview
10797 patches.
10798
10799The following sections provide procedures for submitting a change.
10800
10801.. _pushing-a-change-upstream:
10802
10803Using Scripts to Push a Change Upstream and Request a Pull
10804~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10805
10806Follow this procedure to push a change to an upstream "contrib" Git
10807repository:
10808
10809.. note::
10810
10811 You can find general Git information on how to push a change upstream
10812 in the
10813 Git Community Book
10814 .
10815
108161. *Make Your Changes Locally:* Make your changes in your local Git
10817 repository. You should make small, controlled, isolated changes.
10818 Keeping changes small and isolated aids review, makes
10819 merging/rebasing easier and keeps the change history clean should
10820 anyone need to refer to it in future.
10821
108222. *Stage Your Changes:* Stage your changes by using the ``git add``
10823 command on each file you changed.
10824
108253. *Commit Your Changes:* Commit the change by using the ``git commit``
10826 command. Make sure your commit information follows standards by
10827 following these accepted conventions:
10828
10829 - Be sure to include a "Signed-off-by:" line in the same style as
10830 required by the Linux kernel. Adding this line signifies that you,
10831 the submitter, have agreed to the Developer's Certificate of
10832 Origin 1.1 as follows:
10833 ::
10834
10835 Developer's Certificate of Origin 1.1
10836
10837 By making a contribution to this project, I certify that:
10838
10839 (a) The contribution was created in whole or in part by me and I
10840 have the right to submit it under the open source license
10841 indicated in the file; or
10842
10843 (b) The contribution is based upon previous work that, to the best
10844 of my knowledge, is covered under an appropriate open source
10845 license and I have the right under that license to submit that
10846 work with modifications, whether created in whole or in part
10847 by me, under the same open source license (unless I am
10848 permitted to submit under a different license), as indicated
10849 in the file; or
10850
10851 (c) The contribution was provided directly to me by some other
10852 person who certified (a), (b) or (c) and I have not modified
10853 it.
10854
10855 (d) I understand and agree that this project and the contribution
10856 are public and that a record of the contribution (including all
10857 personal information I submit with it, including my sign-off) is
10858 maintained indefinitely and may be redistributed consistent with
10859 this project or the open source license(s) involved.
10860
10861 - Provide a single-line summary of the change. and, if more
10862 explanation is needed, provide more detail in the body of the
10863 commit. This summary is typically viewable in the "shortlist" of
10864 changes. Thus, providing something short and descriptive that
10865 gives the reader a summary of the change is useful when viewing a
10866 list of many commits. You should prefix this short description
10867 with the recipe name (if changing a recipe), or else with the
10868 short form path to the file being changed.
10869
10870 - For the body of the commit message, provide detailed information
10871 that describes what you changed, why you made the change, and the
10872 approach you used. It might also be helpful if you mention how you
10873 tested the change. Provide as much detail as you can in the body
10874 of the commit message.
10875
10876 .. note::
10877
10878 You do not need to provide a more detailed explanation of a
10879 change if the change is minor to the point of the single line
10880 summary providing all the information.
10881
10882 - If the change addresses a specific bug or issue that is associated
10883 with a bug-tracking ID, include a reference to that ID in your
10884 detailed description. For example, the Yocto Project uses a
10885 specific convention for bug references - any commit that addresses
10886 a specific bug should use the following form for the detailed
10887 description. Be sure to use the actual bug-tracking ID from
10888 Bugzilla for bug-id:
10889 ::
10890
10891 Fixes [YOCTO #bug-id]
10892
10893 detailed description of change
10894
108954. *Push Your Commits to a "Contrib" Upstream:* If you have arranged for
10896 permissions to push to an upstream contrib repository, push the
10897 change to that repository:
10898 ::
10899
10900 $ git push upstream_remote_repo local_branch_name
10901
10902 For example, suppose you have permissions to push
10903 into the upstream ``meta-intel-contrib`` repository and you are
10904 working in a local branch named your_name\ ``/README``. The following
10905 command pushes your local commits to the ``meta-intel-contrib``
10906 upstream repository and puts the commit in a branch named
10907 your_name\ ``/README``:
10908 ::
10909
10910 $ git push meta-intel-contrib your_name/README
10911
109125. *Determine Who to Notify:* Determine the maintainer or the mailing
10913 list that you need to notify for the change.
10914
10915 Before submitting any change, you need to be sure who the maintainer
10916 is or what mailing list that you need to notify. Use either these
10917 methods to find out:
10918
10919 - *Maintenance File:* Examine the ``maintainers.inc`` file, which is
10920 located in the :term:`Source Directory` at
10921 ``meta/conf/distro/include``, to see who is responsible for code.
10922
10923 - *Search by File:* Using :ref:`overview-manual/overview-manual-development-environment:git`, you can
10924 enter the following command to bring up a short list of all
10925 commits against a specific file:
10926 ::
10927
10928 git shortlog -- filename
10929
10930 Just provide the name of the file for which you are interested. The
10931 information returned is not ordered by history but does include a
10932 list of everyone who has committed grouped by name. From the list,
10933 you can see who is responsible for the bulk of the changes against
10934 the file.
10935
10936 - *Examine the List of Mailing Lists:* For a list of the Yocto
10937 Project and related mailing lists, see the ":ref:`Mailing
10938 lists <resources-mailinglist>`" section in
10939 the Yocto Project Reference Manual.
10940
109416. *Make a Pull Request:* Notify the maintainer or the mailing list that
10942 you have pushed a change by making a pull request.
10943
10944 The Yocto Project provides two scripts that conveniently let you
10945 generate and send pull requests to the Yocto Project. These scripts
10946 are ``create-pull-request`` and ``send-pull-request``. You can find
10947 these scripts in the ``scripts`` directory within the
10948 :term:`Source Directory` (e.g.
10949 ``~/poky/scripts``).
10950
10951 Using these scripts correctly formats the requests without
10952 introducing any whitespace or HTML formatting. The maintainer that
10953 receives your patches either directly or through the mailing list
10954 needs to be able to save and apply them directly from your emails.
10955 Using these scripts is the preferred method for sending patches.
10956
10957 First, create the pull request. For example, the following command
10958 runs the script, specifies the upstream repository in the contrib
10959 directory into which you pushed the change, and provides a subject
10960 line in the created patch files:
10961 ::
10962
10963 $ ~/poky/scripts/create-pull-request -u meta-intel-contrib -s "Updated Manual Section Reference in README"
10964
10965 Running this script forms ``*.patch`` files in a folder named
10966 ``pull-``\ PID in the current directory. One of the patch files is a
10967 cover letter.
10968
10969 Before running the ``send-pull-request`` script, you must edit the
10970 cover letter patch to insert information about your change. After
10971 editing the cover letter, send the pull request. For example, the
10972 following command runs the script and specifies the patch directory
10973 and email address. In this example, the email address is a mailing
10974 list:
10975 ::
10976
10977 $ ~/poky/scripts/send-pull-request -p ~/meta-intel/pull-10565 -t meta-intel@yoctoproject.org
10978
10979 You need to follow the prompts as the script is interactive.
10980
10981 .. note::
10982
10983 For help on using these scripts, simply provide the
10984 -h
10985 argument as follows:
10986 ::
10987
10988 $ poky/scripts/create-pull-request -h
10989 $ poky/scripts/send-pull-request -h
10990
10991
10992.. _submitting-a-patch:
10993
10994Using Email to Submit a Patch
10995~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10996
10997You can submit patches without using the ``create-pull-request`` and
10998``send-pull-request`` scripts described in the previous section.
10999However, keep in mind, the preferred method is to use the scripts.
11000
11001Depending on the components changed, you need to submit the email to a
11002specific mailing list. For some guidance on which mailing list to use,
11003see the `list <#figuring-out-the-mailing-list-to-use>`__ at the
11004beginning of this section. For a description of all the available
11005mailing lists, see the ":ref:`Mailing Lists <resources-mailinglist>`" section in the
11006Yocto Project Reference Manual.
11007
11008Here is the general procedure on how to submit a patch through email
11009without using the scripts:
11010
110111. *Make Your Changes Locally:* Make your changes in your local Git
11012 repository. You should make small, controlled, isolated changes.
11013 Keeping changes small and isolated aids review, makes
11014 merging/rebasing easier and keeps the change history clean should
11015 anyone need to refer to it in future.
11016
110172. *Stage Your Changes:* Stage your changes by using the ``git add``
11018 command on each file you changed.
11019
110203. *Commit Your Changes:* Commit the change by using the
11021 ``git commit --signoff`` command. Using the ``--signoff`` option
11022 identifies you as the person making the change and also satisfies the
11023 Developer's Certificate of Origin (DCO) shown earlier.
11024
11025 When you form a commit, you must follow certain standards established
11026 by the Yocto Project development team. See `Step
11027 3 <#making-sure-you-have-correct-commit-information>`__ in the
11028 previous section for information on how to provide commit information
11029 that meets Yocto Project commit message standards.
11030
110314. *Format the Commit:* Format the commit into an email message. To
11032 format commits, use the ``git format-patch`` command. When you
11033 provide the command, you must include a revision list or a number of
11034 patches as part of the command. For example, either of these two
11035 commands takes your most recent single commit and formats it as an
11036 email message in the current directory:
11037 ::
11038
11039 $ git format-patch -1
11040
11041 or ::
11042
11043 $ git format-patch HEAD~
11044
11045 After the command is run, the current directory contains a numbered
11046 ``.patch`` file for the commit.
11047
11048 If you provide several commits as part of the command, the
11049 ``git format-patch`` command produces a series of numbered files in
11050 the current directory – one for each commit. If you have more than
11051 one patch, you should also use the ``--cover`` option with the
11052 command, which generates a cover letter as the first "patch" in the
11053 series. You can then edit the cover letter to provide a description
11054 for the series of patches. For information on the
11055 ``git format-patch`` command, see ``GIT_FORMAT_PATCH(1)`` displayed
11056 using the ``man git-format-patch`` command.
11057
11058 .. note::
11059
11060 If you are or will be a frequent contributor to the Yocto Project
11061 or to OpenEmbedded, you might consider requesting a contrib area
11062 and the necessary associated rights.
11063
110645. *Import the Files Into Your Mail Client:* Import the files into your
11065 mail client by using the ``git send-email`` command.
11066
11067 .. note::
11068
11069 In order to use
11070 git send-email
11071 , you must have the proper Git packages installed on your host.
11072 For Ubuntu, Debian, and Fedora the package is
11073 git-email
11074 .
11075
11076 The ``git send-email`` command sends email by using a local or remote
11077 Mail Transport Agent (MTA) such as ``msmtp``, ``sendmail``, or
11078 through a direct ``smtp`` configuration in your Git ``~/.gitconfig``
11079 file. If you are submitting patches through email only, it is very
11080 important that you submit them without any whitespace or HTML
11081 formatting that either you or your mailer introduces. The maintainer
11082 that receives your patches needs to be able to save and apply them
11083 directly from your emails. A good way to verify that what you are
11084 sending will be applicable by the maintainer is to do a dry run and
11085 send them to yourself and then save and apply them as the maintainer
11086 would.
11087
11088 The ``git send-email`` command is the preferred method for sending
11089 your patches using email since there is no risk of compromising
11090 whitespace in the body of the message, which can occur when you use
11091 your own mail client. The command also has several options that let
11092 you specify recipients and perform further editing of the email
11093 message. For information on how to use the ``git send-email``
11094 command, see ``GIT-SEND-EMAIL(1)`` displayed using the
11095 ``man git-send-email`` command.
11096
11097Working With Licenses
11098=====================
11099
11100As mentioned in the ":ref:`overview-manual/overview-manual-development-environment:licensing`"
11101section in the Yocto Project Overview and Concepts Manual, open source
11102projects are open to the public and they consequently have different
11103licensing structures in place. This section describes the mechanism by
11104which the :term:`OpenEmbedded Build System`
11105tracks changes to
11106licensing text and covers how to maintain open source license compliance
11107during your project's lifecycle. The section also describes how to
11108enable commercially licensed recipes, which by default are disabled.
11109
11110.. _usingpoky-configuring-LIC_FILES_CHKSUM:
11111
11112Tracking License Changes
11113------------------------
11114
11115The license of an upstream project might change in the future. In order
11116to prevent these changes going unnoticed, the
11117:term:`LIC_FILES_CHKSUM`
11118variable tracks changes to the license text. The checksums are validated
11119at the end of the configure step, and if the checksums do not match, the
11120build will fail.
11121
11122.. _usingpoky-specifying-LIC_FILES_CHKSUM:
11123
11124Specifying the ``LIC_FILES_CHKSUM`` Variable
11125~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
11126
11127The ``LIC_FILES_CHKSUM`` variable contains checksums of the license text
11128in the source code for the recipe. Following is an example of how to
11129specify ``LIC_FILES_CHKSUM``:
11130::
11131
11132 LIC_FILES_CHKSUM = "file://COPYING;md5=xxxx \
11133 file://licfile1.txt;beginline=5;endline=29;md5=yyyy \
11134 file://licfile2.txt;endline=50;md5=zzzz \
11135 ..."
11136
11137.. note::
11138
11139 - When using "beginline" and "endline", realize that line numbering
11140 begins with one and not zero. Also, the included lines are
11141 inclusive (i.e. lines five through and including 29 in the
11142 previous example for ``licfile1.txt``).
11143
11144 - When a license check fails, the selected license text is included
11145 as part of the QA message. Using this output, you can determine
11146 the exact start and finish for the needed license text.
11147
11148The build system uses the :term:`S`
11149variable as the default directory when searching files listed in
11150``LIC_FILES_CHKSUM``. The previous example employs the default
11151directory.
11152
11153Consider this next example:
11154::
11155
11156 LIC_FILES_CHKSUM = "file://src/ls.c;beginline=5;endline=16;\
11157 md5=bb14ed3c4cda583abc85401304b5cd4e"
11158 LIC_FILES_CHKSUM = "file://${WORKDIR}/license.html;md5=5c94767cedb5d6987c902ac850ded2c6"
11159
11160The first line locates a file in ``${S}/src/ls.c`` and isolates lines
11161five through 16 as license text. The second line refers to a file in
11162:term:`WORKDIR`.
11163
11164Note that ``LIC_FILES_CHKSUM`` variable is mandatory for all recipes,
11165unless the ``LICENSE`` variable is set to "CLOSED".
11166
11167.. _usingpoky-LIC_FILES_CHKSUM-explanation-of-syntax:
11168
11169Explanation of Syntax
11170~~~~~~~~~~~~~~~~~~~~~
11171
11172As mentioned in the previous section, the ``LIC_FILES_CHKSUM`` variable
11173lists all the important files that contain the license text for the
11174source code. It is possible to specify a checksum for an entire file, or
11175a specific section of a file (specified by beginning and ending line
11176numbers with the "beginline" and "endline" parameters, respectively).
11177The latter is useful for source files with a license notice header,
11178README documents, and so forth. If you do not use the "beginline"
11179parameter, then it is assumed that the text begins on the first line of
11180the file. Similarly, if you do not use the "endline" parameter, it is
11181assumed that the license text ends with the last line of the file.
11182
11183The "md5" parameter stores the md5 checksum of the license text. If the
11184license text changes in any way as compared to this parameter then a
11185mismatch occurs. This mismatch triggers a build failure and notifies the
11186developer. Notification allows the developer to review and address the
11187license text changes. Also note that if a mismatch occurs during the
11188build, the correct md5 checksum is placed in the build log and can be
11189easily copied to the recipe.
11190
11191There is no limit to how many files you can specify using the
11192``LIC_FILES_CHKSUM`` variable. Generally, however, every project
11193requires a few specifications for license tracking. Many projects have a
11194"COPYING" file that stores the license information for all the source
11195code files. This practice allows you to just track the "COPYING" file as
11196long as it is kept up to date.
11197
11198.. note::
11199
11200 - If you specify an empty or invalid "md5" parameter,
11201 :term:`BitBake` returns an md5
11202 mis-match error and displays the correct "md5" parameter value
11203 during the build. The correct parameter is also captured in the
11204 build log.
11205
11206 - If the whole file contains only license text, you do not need to
11207 use the "beginline" and "endline" parameters.
11208
11209Enabling Commercially Licensed Recipes
11210--------------------------------------
11211
11212By default, the OpenEmbedded build system disables components that have
11213commercial or other special licensing requirements. Such requirements
11214are defined on a recipe-by-recipe basis through the
11215:term:`LICENSE_FLAGS` variable
11216definition in the affected recipe. For instance, the
11217``poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly`` recipe
11218contains the following statement:
11219::
11220
11221 LICENSE_FLAGS = "commercial"
11222
11223Here is a
11224slightly more complicated example that contains both an explicit recipe
11225name and version (after variable expansion):
11226::
11227
11228 LICENSE_FLAGS = "license_${PN}_${PV}"
11229
11230In order for a component restricted by a
11231``LICENSE_FLAGS`` definition to be enabled and included in an image, it
11232needs to have a matching entry in the global
11233:term:`LICENSE_FLAGS_WHITELIST`
11234variable, which is a variable typically defined in your ``local.conf``
11235file. For example, to enable the
11236``poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly`` package, you
11237could add either the string "commercial_gst-plugins-ugly" or the more
11238general string "commercial" to ``LICENSE_FLAGS_WHITELIST``. See the
11239"`License Flag Matching <#license-flag-matching>`__" section for a full
11240explanation of how ``LICENSE_FLAGS`` matching works. Here is the
11241example:
11242::
11243
11244 LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly"
11245
11246Likewise, to additionally enable the package built from the recipe
11247containing ``LICENSE_FLAGS = "license_${PN}_${PV}"``, and assuming that
11248the actual recipe name was ``emgd_1.10.bb``, the following string would
11249enable that package as well as the original ``gst-plugins-ugly``
11250package:
11251::
11252
11253 LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly license_emgd_1.10"
11254
11255As a convenience, you do not need to specify the
11256complete license string in the whitelist for every package. You can use
11257an abbreviated form, which consists of just the first portion or
11258portions of the license string before the initial underscore character
11259or characters. A partial string will match any license that contains the
11260given string as the first portion of its license. For example, the
11261following whitelist string will also match both of the packages
11262previously mentioned as well as any other packages that have licenses
11263starting with "commercial" or "license".
11264::
11265
11266 LICENSE_FLAGS_WHITELIST = "commercial license"
11267
11268License Flag Matching
11269~~~~~~~~~~~~~~~~~~~~~
11270
11271License flag matching allows you to control what recipes the
11272OpenEmbedded build system includes in the build. Fundamentally, the
11273build system attempts to match ``LICENSE_FLAGS`` strings found in
11274recipes against ``LICENSE_FLAGS_WHITELIST`` strings found in the
11275whitelist. A match causes the build system to include a recipe in the
11276build, while failure to find a match causes the build system to exclude
11277a recipe.
11278
11279In general, license flag matching is simple. However, understanding some
11280concepts will help you correctly and effectively use matching.
11281
11282Before a flag defined by a particular recipe is tested against the
11283contents of the whitelist, the expanded string ``_${PN}`` is appended to
11284the flag. This expansion makes each ``LICENSE_FLAGS`` value
11285recipe-specific. After expansion, the string is then matched against the
11286whitelist. Thus, specifying ``LICENSE_FLAGS = "commercial"`` in recipe
11287"foo", for example, results in the string ``"commercial_foo"``. And, to
11288create a match, that string must appear in the whitelist.
11289
11290Judicious use of the ``LICENSE_FLAGS`` strings and the contents of the
11291``LICENSE_FLAGS_WHITELIST`` variable allows you a lot of flexibility for
11292including or excluding recipes based on licensing. For example, you can
11293broaden the matching capabilities by using license flags string subsets
11294in the whitelist.
11295
11296.. note::
11297
11298 When using a string subset, be sure to use the part of the expanded
11299 string that precedes the appended underscore character (e.g.
11300 usethispart_1.3
11301 ,
11302 usethispart_1.4
11303 , and so forth).
11304
11305For example, simply specifying the string "commercial" in the whitelist
11306matches any expanded ``LICENSE_FLAGS`` definition that starts with the
11307string "commercial" such as "commercial_foo" and "commercial_bar", which
11308are the strings the build system automatically generates for
11309hypothetical recipes named "foo" and "bar" assuming those recipes simply
11310specify the following:
11311::
11312
11313 LICENSE_FLAGS = "commercial"
11314
11315Thus, you can choose
11316to exhaustively enumerate each license flag in the whitelist and allow
11317only specific recipes into the image, or you can use a string subset
11318that causes a broader range of matches to allow a range of recipes into
11319the image.
11320
11321This scheme works even if the ``LICENSE_FLAGS`` string already has
11322``_${PN}`` appended. For example, the build system turns the license
11323flag "commercial_1.2_foo" into "commercial_1.2_foo_foo" and would match
11324both the general "commercial" and the specific "commercial_1.2_foo"
11325strings found in the whitelist, as expected.
11326
11327Here are some other scenarios:
11328
11329- You can specify a versioned string in the recipe such as
11330 "commercial_foo_1.2" in a "foo" recipe. The build system expands this
11331 string to "commercial_foo_1.2_foo". Combine this license flag with a
11332 whitelist that has the string "commercial" and you match the flag
11333 along with any other flag that starts with the string "commercial".
11334
11335- Under the same circumstances, you can use "commercial_foo" in the
11336 whitelist and the build system not only matches "commercial_foo_1.2"
11337 but also matches any license flag with the string "commercial_foo",
11338 regardless of the version.
11339
11340- You can be very specific and use both the package and version parts
11341 in the whitelist (e.g. "commercial_foo_1.2") to specifically match a
11342 versioned recipe.
11343
11344Other Variables Related to Commercial Licenses
11345~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
11346
11347Other helpful variables related to commercial license handling exist and
11348are defined in the
11349``poky/meta/conf/distro/include/default-distrovars.inc`` file:
11350::
11351
11352 COMMERCIAL_AUDIO_PLUGINS ?= ""
11353 COMMERCIAL_VIDEO_PLUGINS ?= ""
11354
11355If you
11356want to enable these components, you can do so by making sure you have
11357statements similar to the following in your ``local.conf`` configuration
11358file:
11359::
11360
11361 COMMERCIAL_AUDIO_PLUGINS = "gst-plugins-ugly-mad \
11362 gst-plugins-ugly-mpegaudioparse"
11363 COMMERCIAL_VIDEO_PLUGINS = "gst-plugins-ugly-mpeg2dec \
11364 gst-plugins-ugly-mpegstream gst-plugins-bad-mpegvideoparse"
11365 LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly commercial_gst-plugins-bad commercial_qmmp"
11366
11367
11368Of course, you could also create a matching whitelist for those
11369components using the more general "commercial" in the whitelist, but
11370that would also enable all the other packages with ``LICENSE_FLAGS``
11371containing "commercial", which you may or may not want:
11372::
11373
11374 LICENSE_FLAGS_WHITELIST = "commercial"
11375
11376Specifying audio and video plugins as part of the
11377``COMMERCIAL_AUDIO_PLUGINS`` and ``COMMERCIAL_VIDEO_PLUGINS`` statements
11378(along with the enabling ``LICENSE_FLAGS_WHITELIST``) includes the
11379plugins or components into built images, thus adding support for media
11380formats or components.
11381
11382Maintaining Open Source License Compliance During Your Product's Lifecycle
11383--------------------------------------------------------------------------
11384
11385One of the concerns for a development organization using open source
11386software is how to maintain compliance with various open source
11387licensing during the lifecycle of the product. While this section does
11388not provide legal advice or comprehensively cover all scenarios, it does
11389present methods that you can use to assist you in meeting the compliance
11390requirements during a software release.
11391
11392With hundreds of different open source licenses that the Yocto Project
11393tracks, it is difficult to know the requirements of each and every
11394license. However, the requirements of the major FLOSS licenses can begin
11395to be covered by assuming that three main areas of concern exist:
11396
11397- Source code must be provided.
11398
11399- License text for the software must be provided.
11400
11401- Compilation scripts and modifications to the source code must be
11402 provided.
11403
11404There are other requirements beyond the scope of these three and the
11405methods described in this section (e.g. the mechanism through which
11406source code is distributed).
11407
11408As different organizations have different methods of complying with open
11409source licensing, this section is not meant to imply that there is only
11410one single way to meet your compliance obligations, but rather to
11411describe one method of achieving compliance. The remainder of this
11412section describes methods supported to meet the previously mentioned
11413three requirements. Once you take steps to meet these requirements, and
11414prior to releasing images, sources, and the build system, you should
11415audit all artifacts to ensure completeness.
11416
11417.. note::
11418
11419 The Yocto Project generates a license manifest during image creation
11420 that is located in
11421 ${DEPLOY_DIR}/licenses/
11422 image_name-datestamp
11423 to assist with any audits.
11424
11425Providing the Source Code
11426~~~~~~~~~~~~~~~~~~~~~~~~~
11427
11428Compliance activities should begin before you generate the final image.
11429The first thing you should look at is the requirement that tops the list
11430for most compliance groups - providing the source. The Yocto Project has
11431a few ways of meeting this requirement.
11432
11433One of the easiest ways to meet this requirement is to provide the
11434entire :term:`DL_DIR` used by the
11435build. This method, however, has a few issues. The most obvious is the
11436size of the directory since it includes all sources used in the build
11437and not just the source used in the released image. It will include
11438toolchain source, and other artifacts, which you would not generally
11439release. However, the more serious issue for most companies is
11440accidental release of proprietary software. The Yocto Project provides
11441an :ref:`archiver <ref-classes-archiver>` class to
11442help avoid some of these concerns.
11443
11444Before you employ ``DL_DIR`` or the ``archiver`` class, you need to
11445decide how you choose to provide source. The source ``archiver`` class
11446can generate tarballs and SRPMs and can create them with various levels
11447of compliance in mind.
11448
11449One way of doing this (but certainly not the only way) is to release
11450just the source as a tarball. You can do this by adding the following to
11451the ``local.conf`` file found in the
11452:term:`Build Directory`:
11453::
11454
11455 INHERIT += "archiver"
11456 ARCHIVER_MODE[src] = "original"
11457
11458During the creation of your
11459image, the source from all recipes that deploy packages to the image is
11460placed within subdirectories of ``DEPLOY_DIR/sources`` based on the
11461:term:`LICENSE` for each recipe.
11462Releasing the entire directory enables you to comply with requirements
11463concerning providing the unmodified source. It is important to note that
11464the size of the directory can get large.
11465
11466A way to help mitigate the size issue is to only release tarballs for
11467licenses that require the release of source. Let us assume you are only
11468concerned with GPL code as identified by running the following script:
11469::
11470
11471 # Script to archive a subset of packages matching specific license(s)
11472 # Source and license files are copied into sub folders of package folder
11473 # Must be run from build folder
11474 #!/bin/bash
11475 src_release_dir="source-release"
11476 mkdir -p $src_release_dir
11477 for a in tmp/deploy/sources/*; do
11478 for d in $a/*; do
11479 # Get package name from path
11480 p=`basename $d`
11481 p=${p%-*}
11482 p=${p%-*}
11483 # Only archive GPL packages (update *GPL* regex for your license check)
11484 numfiles=`ls tmp/deploy/licenses/$p/*GPL* 2> /dev/null | wc -l`
11485 if [ $numfiles -gt 1 ]; then
11486 echo Archiving $p
11487 mkdir -p $src_release_dir/$p/source
11488 cp $d/* $src_release_dir/$p/source 2> /dev/null
11489 mkdir -p $src_release_dir/$p/license
11490 cp tmp/deploy/licenses/$p/* $src_release_dir/$p/license 2> /dev/null
11491 fi
11492 done
11493 done
11494
11495At this point, you
11496could create a tarball from the ``gpl_source_release`` directory and
11497provide that to the end user. This method would be a step toward
11498achieving compliance with section 3a of GPLv2 and with section 6 of
11499GPLv3.
11500
11501Providing License Text
11502~~~~~~~~~~~~~~~~~~~~~~
11503
11504One requirement that is often overlooked is inclusion of license text.
11505This requirement also needs to be dealt with prior to generating the
11506final image. Some licenses require the license text to accompany the
11507binary. You can achieve this by adding the following to your
11508``local.conf`` file:
11509::
11510
11511 COPY_LIC_MANIFEST = "1"
11512 COPY_LIC_DIRS = "1"
11513 LICENSE_CREATE_PACKAGE = "1"
11514
11515Adding these statements to the
11516configuration file ensures that the licenses collected during package
11517generation are included on your image.
11518
11519.. note::
11520
11521 Setting all three variables to "1" results in the image having two
11522 copies of the same license file. One copy resides in
11523 ``/usr/share/common-licenses`` and the other resides in
11524 ``/usr/share/license``.
11525
11526 The reason for this behavior is because
11527 :term:`COPY_LIC_DIRS` and
11528 :term:`COPY_LIC_MANIFEST`
11529 add a copy of the license when the image is built but do not offer a
11530 path for adding licenses for newly installed packages to an image.
11531 :term:`LICENSE_CREATE_PACKAGE`
11532 adds a separate package and an upgrade path for adding licenses to an
11533 image.
11534
11535As the source ``archiver`` class has already archived the original
11536unmodified source that contains the license files, you would have
11537already met the requirements for inclusion of the license information
11538with source as defined by the GPL and other open source licenses.
11539
11540Providing Compilation Scripts and Source Code Modifications
11541~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
11542
11543At this point, we have addressed all we need to prior to generating the
11544image. The next two requirements are addressed during the final
11545packaging of the release.
11546
11547By releasing the version of the OpenEmbedded build system and the layers
11548used during the build, you will be providing both compilation scripts
11549and the source code modifications in one step.
11550
11551If the deployment team has a :ref:`overview-manual/overview-manual-concepts:bsp layer`
11552and a distro layer, and those
11553those layers are used to patch, compile, package, or modify (in any way)
11554any open source software included in your released images, you might be
11555required to release those layers under section 3 of GPLv2 or section 1
11556of GPLv3. One way of doing that is with a clean checkout of the version
11557of the Yocto Project and layers used during your build. Here is an
11558example:
11559::
11560
11561 # We built using the dunfell branch of the poky repo
11562 $ git clone -b dunfell git://git.yoctoproject.org/poky
11563 $ cd poky
11564 # We built using the release_branch for our layers
11565 $ git clone -b release_branch git://git.mycompany.com/meta-my-bsp-layer
11566 $ git clone -b release_branch git://git.mycompany.com/meta-my-software-layer
11567 # clean up the .git repos
11568 $ find . -name ".git" -type d -exec rm -rf {} \;
11569
11570One
11571thing a development organization might want to consider for end-user
11572convenience is to modify ``meta-poky/conf/bblayers.conf.sample`` to
11573ensure that when the end user utilizes the released build system to
11574build an image, the development organization's layers are included in
11575the ``bblayers.conf`` file automatically:
11576::
11577
11578 # POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
11579 # changes incompatibly
11580 POKY_BBLAYERS_CONF_VERSION = "2"
11581
11582 BBPATH = "${TOPDIR}"
11583 BBFILES ?= ""
11584
11585 BBLAYERS ?= " \
11586 ##OEROOT##/meta \
11587 ##OEROOT##/meta-poky \
11588 ##OEROOT##/meta-yocto-bsp \
11589 ##OEROOT##/meta-mylayer \
11590 "
11591
11592Creating and
11593providing an archive of the :term:`Metadata`
11594layers (recipes, configuration files, and so forth) enables you to meet
11595your requirements to include the scripts to control compilation as well
11596as any modifications to the original source.
11597
11598Copying Licenses that Do Not Exist
11599----------------------------------
11600
11601Some packages, such as the linux-firmware package, have many licenses
11602that are not in any way common. You can avoid adding a lot of these
11603types of common license files, which are only applicable to a specific
11604package, by using the
11605:term:`NO_GENERIC_LICENSE`
11606variable. Using this variable also avoids QA errors when you use a
11607non-common, non-CLOSED license in a recipe.
11608
11609The following is an example that uses the ``LICENSE.Abilis.txt`` file as
11610the license from the fetched source:
11611::
11612
11613 NO_GENERIC_LICENSE[Firmware-Abilis] = "LICENSE.Abilis.txt"
11614
11615Using the Error Reporting Tool
11616==============================
11617
11618The error reporting tool allows you to submit errors encountered during
11619builds to a central database. Outside of the build environment, you can
11620use a web interface to browse errors, view statistics, and query for
11621errors. The tool works using a client-server system where the client
11622portion is integrated with the installed Yocto Project
11623:term:`Source Directory` (e.g. ``poky``).
11624The server receives the information collected and saves it in a
11625database.
11626
11627A live instance of the error reporting server exists at
11628http://errors.yoctoproject.org. This server exists so that when
11629you want to get help with build failures, you can submit all of the
11630information on the failure easily and then point to the URL in your bug
11631report or send an email to the mailing list.
11632
11633.. note::
11634
11635 If you send error reports to this server, the reports become publicly
11636 visible.
11637
11638Enabling and Using the Tool
11639---------------------------
11640
11641By default, the error reporting tool is disabled. You can enable it by
11642inheriting the
11643:ref:`report-error <ref-classes-report-error>`
11644class by adding the following statement to the end of your
11645``local.conf`` file in your
11646:term:`Build Directory`.
11647::
11648
11649 INHERIT += "report-error"
11650
11651By default, the error reporting feature stores information in
11652``${``\ :term:`LOG_DIR`\ ``}/error-report``.
11653However, you can specify a directory to use by adding the following to
11654your ``local.conf`` file:
11655::
11656
11657 ERR_REPORT_DIR = "path"
11658
11659Enabling error
11660reporting causes the build process to collect the errors and store them
11661in a file as previously described. When the build system encounters an
11662error, it includes a command as part of the console output. You can run
11663the command to send the error file to the server. For example, the
11664following command sends the errors to an upstream server:
11665::
11666
11667 $ send-error-report /home/brandusa/project/poky/build/tmp/log/error-report/error_report_201403141617.txt
11668
11669In the previous example, the errors are sent to a public database
11670available at http://errors.yoctoproject.org, which is used by the
11671entire community. If you specify a particular server, you can send the
11672errors to a different database. Use the following command for more
11673information on available options:
11674::
11675
11676 $ send-error-report --help
11677
11678When sending the error file, you are prompted to review the data being
11679sent as well as to provide a name and optional email address. Once you
11680satisfy these prompts, the command returns a link from the server that
11681corresponds to your entry in the database. For example, here is a
11682typical link: http://errors.yoctoproject.org/Errors/Details/9522/
11683
11684Following the link takes you to a web interface where you can browse,
11685query the errors, and view statistics.
11686
11687Disabling the Tool
11688------------------
11689
11690To disable the error reporting feature, simply remove or comment out the
11691following statement from the end of your ``local.conf`` file in your
11692:term:`Build Directory`.
11693::
11694
11695 INHERIT += "report-error"
11696
11697Setting Up Your Own Error Reporting Server
11698------------------------------------------
11699
11700If you want to set up your own error reporting server, you can obtain
11701the code from the Git repository at
11702http://git.yoctoproject.org/cgit/cgit.cgi/error-report-web/.
11703Instructions on how to set it up are in the README document.
11704
11705.. _dev-using-wayland-and-weston:
11706
11707Using Wayland and Weston
11708========================
11709
11710`Wayland <http://en.wikipedia.org/wiki/Wayland_(display_server_protocol)>`__
11711is a computer display server protocol that provides a method for
11712compositing window managers to communicate directly with applications
11713and video hardware and expects them to communicate with input hardware
11714using other libraries. Using Wayland with supporting targets can result
11715in better control over graphics frame rendering than an application
11716might otherwise achieve.
11717
11718The Yocto Project provides the Wayland protocol libraries and the
11719reference
11720`Weston <http://en.wikipedia.org/wiki/Wayland_(display_server_protocol)#Weston>`__
11721compositor as part of its release. You can find the integrated packages
11722in the ``meta`` layer of the :term:`Source Directory`.
11723Specifically, you
11724can find the recipes that build both Wayland and Weston at
11725``meta/recipes-graphics/wayland``.
11726
11727You can build both the Wayland and Weston packages for use only with
11728targets that accept the `Mesa 3D and Direct Rendering
11729Infrastructure <https://en.wikipedia.org/wiki/Mesa_(computer_graphics)>`__,
11730which is also known as Mesa DRI. This implies that you cannot build and
11731use the packages if your target uses, for example, the Intel Embedded
11732Media and Graphics Driver (Intel EMGD) that overrides Mesa DRI.
11733
11734.. note::
11735
11736 Due to lack of EGL support, Weston 1.0.3 will not run directly on the
11737 emulated QEMU hardware. However, this version of Weston will run
11738 under X emulation without issues.
11739
11740This section describes what you need to do to implement Wayland and use
11741the Weston compositor when building an image for a supporting target.
11742
11743Enabling Wayland in an Image
11744----------------------------
11745
11746To enable Wayland, you need to enable it to be built and enable it to be
11747included (installed) in the image.
11748
11749.. _enable-building:
11750
11751Building Wayland
11752~~~~~~~~~~~~~~~~
11753
11754To cause Mesa to build the ``wayland-egl`` platform and Weston to build
11755Wayland with Kernel Mode Setting
11756(`KMS <https://wiki.archlinux.org/index.php/Kernel_Mode_Setting>`__)
11757support, include the "wayland" flag in the
11758:term:`DISTRO_FEATURES`
11759statement in your ``local.conf`` file:
11760::
11761
11762 DISTRO_FEATURES_append = " wayland"
11763
11764.. note::
11765
11766 If X11 has been enabled elsewhere, Weston will build Wayland with X11
11767 support
11768
11769.. _enable-installation-in-an-image:
11770
11771Installing Wayland and Weston
11772~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
11773
11774To install the Wayland feature into an image, you must include the
11775following
11776:term:`CORE_IMAGE_EXTRA_INSTALL`
11777statement in your ``local.conf`` file:
11778::
11779
11780 CORE_IMAGE_EXTRA_INSTALL += "wayland weston"
11781
11782Running Weston
11783--------------
11784
11785To run Weston inside X11, enabling it as described earlier and building
11786a Sato image is sufficient. If you are running your image under Sato, a
11787Weston Launcher appears in the "Utility" category.
11788
11789Alternatively, you can run Weston through the command-line interpretor
11790(CLI), which is better suited for development work. To run Weston under
11791the CLI, you need to do the following after your image is built:
11792
117931. Run these commands to export ``XDG_RUNTIME_DIR``:
11794 ::
11795
11796 mkdir -p /tmp/$USER-weston
11797 chmod 0700 /tmp/$USER-weston
11798 export XDG_RUNTIME_DIR=/tmp/$USER-weston
11799
118002. Launch Weston in the shell:
11801 ::
11802
11803 weston