blob: 0630040e680c5727a5ec8085f94a2e12808b9f2a [file] [log] [blame]
Andrew Geisslerf0343792020-11-18 10:42:21 -06001.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002
3************
4Common Tasks
5************
6
7This chapter describes fundamental procedures such as creating layers,
8adding new software packages, extending or customizing images, porting
9work to new hardware (adding a new machine), and so forth. You will find
10that the procedures documented here occur often in the development cycle
11using the Yocto Project.
12
13Understanding and Creating Layers
14=================================
15
16The OpenEmbedded build system supports organizing
17:term:`Metadata` into multiple layers.
18Layers allow you to isolate different types of customizations from each
19other. For introductory information on the Yocto Project Layer Model,
20see the
21":ref:`overview-manual/overview-manual-yp-intro:the yocto project layer model`"
22section in the Yocto Project Overview and Concepts Manual.
23
24Creating Your Own Layer
25-----------------------
26
27It is very easy to create your own layers to use with the OpenEmbedded
28build system. The Yocto Project ships with tools that speed up creating
29layers. This section describes the steps you perform by hand to create
30layers so that you can better understand them. For information about the
31layer-creation tools, see the
32":ref:`bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script`"
33section in the Yocto Project Board Support Package (BSP) Developer's
34Guide and the ":ref:`dev-manual/dev-manual-common-tasks:creating a general layer using the \`\`bitbake-layers\`\` script`"
35section further down in this manual.
36
37Follow these general steps to create your layer without using tools:
38
391. *Check Existing Layers:* Before creating a new layer, you should be
40 sure someone has not already created a layer containing the Metadata
41 you need. You can see the `OpenEmbedded Metadata
Andrew Geissler4c19ea12020-10-27 13:52:24 -050042 Index <https://layers.openembedded.org/layerindex/layers/>`__ for a
Andrew Geisslerc9f78652020-09-18 14:11:35 -050043 list of layers from the OpenEmbedded community that can be used in
44 the Yocto Project. You could find a layer that is identical or close
45 to what you need.
46
472. *Create a Directory:* Create the directory for your layer. When you
48 create the layer, be sure to create the directory in an area not
49 associated with the Yocto Project :term:`Source Directory`
50 (e.g. the cloned ``poky`` repository).
51
52 While not strictly required, prepend the name of the directory with
53 the string "meta-". For example:
54 ::
55
56 meta-mylayer
57 meta-GUI_xyz
58 meta-mymachine
59
60 With rare exceptions, a layer's name follows this form:
61 ::
62
63 meta-root_name
64
65 Following this layer naming convention can save
66 you trouble later when tools, components, or variables "assume" your
67 layer name begins with "meta-". A notable example is in configuration
68 files as shown in the following step where layer names without the
69 "meta-" string are appended to several variables used in the
70 configuration.
71
723. *Create a Layer Configuration File:* Inside your new layer folder,
73 you need to create a ``conf/layer.conf`` file. It is easiest to take
74 an existing layer configuration file and copy that to your layer's
75 ``conf`` directory and then modify the file as needed.
76
77 The ``meta-yocto-bsp/conf/layer.conf`` file in the Yocto Project
78 :yocto_git:`Source Repositories </cgit/cgit.cgi/poky/tree/meta-yocto-bsp/conf>`
79 demonstrates the required syntax. For your layer, you need to replace
80 "yoctobsp" with a unique identifier for your layer (e.g. "machinexyz"
81 for a layer named "meta-machinexyz"):
82 ::
83
84 # We have a conf and classes directory, add to BBPATH
85 BBPATH .= ":${LAYERDIR}"
86
Andrew Geissler4c19ea12020-10-27 13:52:24 -050087 # We have recipes-* directories, add to BBFILES
Andrew Geisslerc9f78652020-09-18 14:11:35 -050088 BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
89 ${LAYERDIR}/recipes-*/*/*.bbappend"
90
91 BBFILE_COLLECTIONS += "yoctobsp"
92 BBFILE_PATTERN_yoctobsp = "^${LAYERDIR}/"
93 BBFILE_PRIORITY_yoctobsp = "5"
94 LAYERVERSION_yoctobsp = "4"
95 LAYERSERIES_COMPAT_yoctobsp = "dunfell"
96
97 Following is an explanation of the layer configuration file:
98
99 - :term:`BBPATH`: Adds the layer's
100 root directory to BitBake's search path. Through the use of the
101 ``BBPATH`` variable, BitBake locates class files (``.bbclass``),
102 configuration files, and files that are included with ``include``
103 and ``require`` statements. For these cases, BitBake uses the
104 first file that matches the name found in ``BBPATH``. This is
105 similar to the way the ``PATH`` variable is used for binaries. It
106 is recommended, therefore, that you use unique class and
107 configuration filenames in your custom layer.
108
109 - :term:`BBFILES`: Defines the
110 location for all recipes in the layer.
111
112 - :term:`BBFILE_COLLECTIONS`:
113 Establishes the current layer through a unique identifier that is
114 used throughout the OpenEmbedded build system to refer to the
115 layer. In this example, the identifier "yoctobsp" is the
116 representation for the container layer named "meta-yocto-bsp".
117
118 - :term:`BBFILE_PATTERN`:
119 Expands immediately during parsing to provide the directory of the
120 layer.
121
122 - :term:`BBFILE_PRIORITY`:
123 Establishes a priority to use for recipes in the layer when the
124 OpenEmbedded build finds recipes of the same name in different
125 layers.
126
127 - :term:`LAYERVERSION`:
128 Establishes a version number for the layer. You can use this
129 version number to specify this exact version of the layer as a
130 dependency when using the
131 :term:`LAYERDEPENDS`
132 variable.
133
134 - :term:`LAYERDEPENDS`:
135 Lists all layers on which this layer depends (if any).
136
137 - :term:`LAYERSERIES_COMPAT`:
138 Lists the :yocto_wiki:`Yocto Project </wiki/Releases>`
139 releases for which the current version is compatible. This
140 variable is a good way to indicate if your particular layer is
141 current.
142
1434. *Add Content:* Depending on the type of layer, add the content. If
144 the layer adds support for a machine, add the machine configuration
145 in a ``conf/machine/`` file within the layer. If the layer adds
146 distro policy, add the distro configuration in a ``conf/distro/``
147 file within the layer. If the layer introduces new recipes, put the
148 recipes you need in ``recipes-*`` subdirectories within the layer.
149
150 .. note::
151
152 For an explanation of layer hierarchy that is compliant with the
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500153 Yocto Project, see the ":ref:`bsp-guide/bsp:example filesystem layout`"
154 section in the Yocto Project Board Support Package (BSP) Developer's Guide.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500155
1565. *Optionally Test for Compatibility:* If you want permission to use
157 the Yocto Project Compatibility logo with your layer or application
158 that uses your layer, perform the steps to apply for compatibility.
159 See the "`Making Sure Your Layer is Compatible With Yocto
160 Project <#making-sure-your-layer-is-compatible-with-yocto-project>`__"
161 section for more information.
162
163.. _best-practices-to-follow-when-creating-layers:
164
165Following Best Practices When Creating Layers
166---------------------------------------------
167
168To create layers that are easier to maintain and that will not impact
169builds for other machines, you should consider the information in the
170following list:
171
172- *Avoid "Overlaying" Entire Recipes from Other Layers in Your
173 Configuration:* In other words, do not copy an entire recipe into
174 your layer and then modify it. Rather, use an append file
175 (``.bbappend``) to override only those parts of the original recipe
176 you need to modify.
177
178- *Avoid Duplicating Include Files:* Use append files (``.bbappend``)
179 for each recipe that uses an include file. Or, if you are introducing
180 a new recipe that requires the included file, use the path relative
181 to the original layer directory to refer to the file. For example,
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500182 use ``require recipes-core/``\ `package`\ ``/``\ `file`\ ``.inc`` instead
183 of ``require`` `file`\ ``.inc``. If you're finding you have to overlay
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500184 the include file, it could indicate a deficiency in the include file
185 in the layer to which it originally belongs. If this is the case, you
186 should try to address that deficiency instead of overlaying the
187 include file. For example, you could address this by getting the
188 maintainer of the include file to add a variable or variables to make
189 it easy to override the parts needing to be overridden.
190
191- *Structure Your Layers:* Proper use of overrides within append files
192 and placement of machine-specific files within your layer can ensure
193 that a build is not using the wrong Metadata and negatively impacting
194 a build for a different machine. Following are some examples:
195
196 - *Modify Variables to Support a Different Machine:* Suppose you
197 have a layer named ``meta-one`` that adds support for building
198 machine "one". To do so, you use an append file named
199 ``base-files.bbappend`` and create a dependency on "foo" by
200 altering the :term:`DEPENDS`
201 variable:
202 ::
203
204 DEPENDS = "foo"
205
206 The dependency is created during any
207 build that includes the layer ``meta-one``. However, you might not
208 want this dependency for all machines. For example, suppose you
209 are building for machine "two" but your ``bblayers.conf`` file has
210 the ``meta-one`` layer included. During the build, the
211 ``base-files`` for machine "two" will also have the dependency on
212 ``foo``.
213
214 To make sure your changes apply only when building machine "one",
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500215 use a machine override with the ``DEPENDS`` statement:
216 ::
217
218 DEPENDS_one = "foo"
219
220 You should follow the same strategy when using ``_append``
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500221 and ``_prepend`` operations:
222 ::
223
224 DEPENDS_append_one = " foo"
225 DEPENDS_prepend_one = "foo "
226
227 As an actual example, here's a
228 snippet from the generic kernel include file ``linux-yocto.inc``,
229 wherein the kernel compile and link options are adjusted in the
230 case of a subset of the supported architectures:
231 ::
232
233 DEPENDS_append_aarch64 = " libgcc"
234 KERNEL_CC_append_aarch64 = " ${TOOLCHAIN_OPTIONS}"
235 KERNEL_LD_append_aarch64 = " ${TOOLCHAIN_OPTIONS}"
236
237 DEPENDS_append_nios2 = " libgcc"
238 KERNEL_CC_append_nios2 = " ${TOOLCHAIN_OPTIONS}"
239 KERNEL_LD_append_nios2 = " ${TOOLCHAIN_OPTIONS}"
240
241 DEPENDS_append_arc = " libgcc"
242 KERNEL_CC_append_arc = " ${TOOLCHAIN_OPTIONS}"
243 KERNEL_LD_append_arc = " ${TOOLCHAIN_OPTIONS}"
244
245 KERNEL_FEATURES_append_qemuall=" features/debug/printk.scc"
246
247 .. note::
248
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500249 Avoiding "+=" and "=+" and using machine-specific ``_append``
250 and ``_prepend`` operations is recommended as well.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500251
252 - *Place Machine-Specific Files in Machine-Specific Locations:* When
253 you have a base recipe, such as ``base-files.bb``, that contains a
254 :term:`SRC_URI` statement to a
255 file, you can use an append file to cause the build to use your
256 own version of the file. For example, an append file in your layer
257 at ``meta-one/recipes-core/base-files/base-files.bbappend`` could
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500258 extend :term:`FILESPATH` using :term:`FILESEXTRAPATHS` as follows:
259 ::
260
261 FILESEXTRAPATHS_prepend := "${THISDIR}/${BPN}:"
262
263 The build for machine "one" will pick up your machine-specific file as
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500264 long as you have the file in
265 ``meta-one/recipes-core/base-files/base-files/``. However, if you
266 are building for a different machine and the ``bblayers.conf``
267 file includes the ``meta-one`` layer and the location of your
268 machine-specific file is the first location where that file is
269 found according to ``FILESPATH``, builds for all machines will
270 also use that machine-specific file.
271
272 You can make sure that a machine-specific file is used for a
273 particular machine by putting the file in a subdirectory specific
274 to the machine. For example, rather than placing the file in
275 ``meta-one/recipes-core/base-files/base-files/`` as shown above,
276 put it in ``meta-one/recipes-core/base-files/base-files/one/``.
277 Not only does this make sure the file is used only when building
278 for machine "one", but the build process locates the file more
279 quickly.
280
281 In summary, you need to place all files referenced from
282 ``SRC_URI`` in a machine-specific subdirectory within the layer in
283 order to restrict those files to machine-specific builds.
284
285- *Perform Steps to Apply for Yocto Project Compatibility:* If you want
286 permission to use the Yocto Project Compatibility logo with your
287 layer or application that uses your layer, perform the steps to apply
288 for compatibility. See the "`Making Sure Your Layer is Compatible
289 With Yocto
290 Project <#making-sure-your-layer-is-compatible-with-yocto-project>`__"
291 section for more information.
292
293- *Follow the Layer Naming Convention:* Store custom layers in a Git
294 repository that use the ``meta-layer_name`` format.
295
296- *Group Your Layers Locally:* Clone your repository alongside other
297 cloned ``meta`` directories from the :term:`Source Directory`.
298
299Making Sure Your Layer is Compatible With Yocto Project
300-------------------------------------------------------
301
302When you create a layer used with the Yocto Project, it is advantageous
303to make sure that the layer interacts well with existing Yocto Project
304layers (i.e. the layer is compatible with the Yocto Project). Ensuring
305compatibility makes the layer easy to be consumed by others in the Yocto
306Project community and could allow you permission to use the Yocto
307Project Compatible Logo.
308
309.. note::
310
311 Only Yocto Project member organizations are permitted to use the
312 Yocto Project Compatible Logo. The logo is not available for general
313 use. For information on how to become a Yocto Project member
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500314 organization, see the :yocto_home:`Yocto Project Website <>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500315
316The Yocto Project Compatibility Program consists of a layer application
317process that requests permission to use the Yocto Project Compatibility
318Logo for your layer and application. The process consists of two parts:
319
3201. Successfully passing a script (``yocto-check-layer``) that when run
321 against your layer, tests it against constraints based on experiences
322 of how layers have worked in the real world and where pitfalls have
323 been found. Getting a "PASS" result from the script is required for
324 successful compatibility registration.
325
3262. Completion of an application acceptance form, which you can find at
327 https://www.yoctoproject.org/webform/yocto-project-compatible-registration.
328
329To be granted permission to use the logo, you need to satisfy the
330following:
331
332- Be able to check the box indicating that you got a "PASS" when
333 running the script against your layer.
334
335- Answer "Yes" to the questions on the form or have an acceptable
336 explanation for any questions answered "No".
337
338- Be a Yocto Project Member Organization.
339
340The remainder of this section presents information on the registration
341form and on the ``yocto-check-layer`` script.
342
343Yocto Project Compatible Program Application
344~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
345
346Use the form to apply for your layer's approval. Upon successful
347application, you can use the Yocto Project Compatibility Logo with your
348layer and the application that uses your layer.
349
350To access the form, use this link:
351https://www.yoctoproject.org/webform/yocto-project-compatible-registration.
352Follow the instructions on the form to complete your application.
353
354The application consists of the following sections:
355
356- *Contact Information:* Provide your contact information as the fields
357 require. Along with your information, provide the released versions
358 of the Yocto Project for which your layer is compatible.
359
360- *Acceptance Criteria:* Provide "Yes" or "No" answers for each of the
361 items in the checklist. Space exists at the bottom of the form for
362 any explanations for items for which you answered "No".
363
364- *Recommendations:* Provide answers for the questions regarding Linux
365 kernel use and build success.
366
367``yocto-check-layer`` Script
368~~~~~~~~~~~~~~~~~~~~~~~~~~~~
369
370The ``yocto-check-layer`` script provides you a way to assess how
371compatible your layer is with the Yocto Project. You should run this
372script prior to using the form to apply for compatibility as described
373in the previous section. You need to achieve a "PASS" result in order to
374have your application form successfully processed.
375
376The script divides tests into three areas: COMMON, BSP, and DISTRO. For
377example, given a distribution layer (DISTRO), the layer must pass both
378the COMMON and DISTRO related tests. Furthermore, if your layer is a BSP
379layer, the layer must pass the COMMON and BSP set of tests.
380
381To execute the script, enter the following commands from your build
382directory:
383::
384
385 $ source oe-init-build-env
386 $ yocto-check-layer your_layer_directory
387
388Be sure to provide the actual directory for your
389layer as part of the command.
390
391Entering the command causes the script to determine the type of layer
392and then to execute a set of specific tests against the layer. The
393following list overviews the test:
394
395- ``common.test_readme``: Tests if a ``README`` file exists in the
396 layer and the file is not empty.
397
398- ``common.test_parse``: Tests to make sure that BitBake can parse the
399 files without error (i.e. ``bitbake -p``).
400
401- ``common.test_show_environment``: Tests that the global or per-recipe
402 environment is in order without errors (i.e. ``bitbake -e``).
403
404- ``common.test_world``: Verifies that ``bitbake world`` works.
405
406- ``common.test_signatures``: Tests to be sure that BSP and DISTRO
407 layers do not come with recipes that change signatures.
408
409- ``common.test_layerseries_compat``: Verifies layer compatibility is
410 set properly.
411
412- ``bsp.test_bsp_defines_machines``: Tests if a BSP layer has machine
413 configurations.
414
415- ``bsp.test_bsp_no_set_machine``: Tests to ensure a BSP layer does not
416 set the machine when the layer is added.
417
418- ``bsp.test_machine_world``: Verifies that ``bitbake world`` works
419 regardless of which machine is selected.
420
421- ``bsp.test_machine_signatures``: Verifies that building for a
422 particular machine affects only the signature of tasks specific to
423 that machine.
424
425- ``distro.test_distro_defines_distros``: Tests if a DISTRO layer has
426 distro configurations.
427
428- ``distro.test_distro_no_set_distros``: Tests to ensure a DISTRO layer
429 does not set the distribution when the layer is added.
430
431Enabling Your Layer
432-------------------
433
434Before the OpenEmbedded build system can use your new layer, you need to
435enable it. To enable your layer, simply add your layer's path to the
436``BBLAYERS`` variable in your ``conf/bblayers.conf`` file, which is
437found in the :term:`Build Directory`.
438The following example shows how to enable a layer named
439``meta-mylayer``:
440::
441
442 # POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
443 # changes incompatibly
444 POKY_BBLAYERS_CONF_VERSION = "2"
445 BBPATH = "${TOPDIR}"
446 BBFILES ?= ""
447 BBLAYERS ?= " \
448 /home/user/poky/meta \
449 /home/user/poky/meta-poky \
450 /home/user/poky/meta-yocto-bsp \
451 /home/user/poky/meta-mylayer \
452 "
453
454BitBake parses each ``conf/layer.conf`` file from the top down as
455specified in the ``BBLAYERS`` variable within the ``conf/bblayers.conf``
456file. During the processing of each ``conf/layer.conf`` file, BitBake
457adds the recipes, classes and configurations contained within the
458particular layer to the source directory.
459
460.. _using-bbappend-files:
461
462Using .bbappend Files in Your Layer
463-----------------------------------
464
465A recipe that appends Metadata to another recipe is called a BitBake
466append file. A BitBake append file uses the ``.bbappend`` file type
467suffix, while the corresponding recipe to which Metadata is being
468appended uses the ``.bb`` file type suffix.
469
470You can use a ``.bbappend`` file in your layer to make additions or
471changes to the content of another layer's recipe without having to copy
472the other layer's recipe into your layer. Your ``.bbappend`` file
473resides in your layer, while the main ``.bb`` recipe file to which you
474are appending Metadata resides in a different layer.
475
476Being able to append information to an existing recipe not only avoids
477duplication, but also automatically applies recipe changes from a
478different layer into your layer. If you were copying recipes, you would
479have to manually merge changes as they occur.
480
481When you create an append file, you must use the same root name as the
482corresponding recipe file. For example, the append file
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500483``someapp_3.1.bbappend`` must apply to ``someapp_3.1.bb``. This
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500484means the original recipe and append file names are version
485number-specific. If the corresponding recipe is renamed to update to a
486newer version, you must also rename and possibly update the
487corresponding ``.bbappend`` as well. During the build process, BitBake
488displays an error on starting if it detects a ``.bbappend`` file that
489does not have a corresponding recipe with a matching name. See the
490:term:`BB_DANGLINGAPPENDS_WARNONLY`
491variable for information on how to handle this error.
492
493As an example, consider the main formfactor recipe and a corresponding
494formfactor append file both from the :term:`Source Directory`.
495Here is the main
496formfactor recipe, which is named ``formfactor_0.0.bb`` and located in
497the "meta" layer at ``meta/recipes-bsp/formfactor``:
498::
499
500 SUMMARY = "Device formfactor information"
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500501 DESCRIPTION = "A formfactor configuration file provides information about the \
502 target hardware for which the image is being built and information that the \
503 build system cannot obtain from other sources such as the kernel."
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500504 SECTION = "base"
505 LICENSE = "MIT"
506 LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
507 PR = "r45"
508
509 SRC_URI = "file://config file://machconfig"
510 S = "${WORKDIR}"
511
512 PACKAGE_ARCH = "${MACHINE_ARCH}"
513 INHIBIT_DEFAULT_DEPS = "1"
514
515 do_install() {
516 # Install file only if it has contents
517 install -d ${D}${sysconfdir}/formfactor/
518 install -m 0644 ${S}/config ${D}${sysconfdir}/formfactor/
519 if [ -s "${S}/machconfig" ]; then
520 install -m 0644 ${S}/machconfig ${D}${sysconfdir}/formfactor/
521 fi
522 }
523
524In the main recipe, note the :term:`SRC_URI`
525variable, which tells the OpenEmbedded build system where to find files
526during the build.
527
528Following is the append file, which is named ``formfactor_0.0.bbappend``
529and is from the Raspberry Pi BSP Layer named ``meta-raspberrypi``. The
530file is in the layer at ``recipes-bsp/formfactor``:
531::
532
533 FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"
534
535By default, the build system uses the
536:term:`FILESPATH` variable to
537locate files. This append file extends the locations by setting the
538:term:`FILESEXTRAPATHS`
539variable. Setting this variable in the ``.bbappend`` file is the most
540reliable and recommended method for adding directories to the search
541path used by the build system to find files.
542
543The statement in this example extends the directories to include
544``${``\ :term:`THISDIR`\ ``}/${``\ :term:`PN`\ ``}``,
545which resolves to a directory named ``formfactor`` in the same directory
546in which the append file resides (i.e.
547``meta-raspberrypi/recipes-bsp/formfactor``. This implies that you must
548have the supporting directory structure set up that will contain any
549files or patches you will be including from the layer.
550
551Using the immediate expansion assignment operator ``:=`` is important
552because of the reference to ``THISDIR``. The trailing colon character is
553important as it ensures that items in the list remain colon-separated.
554
555.. note::
556
557 BitBake automatically defines the ``THISDIR`` variable. You should
558 never set this variable yourself. Using "_prepend" as part of the
559 ``FILESEXTRAPATHS`` ensures your path will be searched prior to other
560 paths in the final list.
561
562 Also, not all append files add extra files. Many append files simply
563 exist to add build options (e.g. ``systemd``). For these cases, your
564 append file would not even use the ``FILESEXTRAPATHS`` statement.
565
566Prioritizing Your Layer
567-----------------------
568
569Each layer is assigned a priority value. Priority values control which
570layer takes precedence if there are recipe files with the same name in
571multiple layers. For these cases, the recipe file from the layer with a
572higher priority number takes precedence. Priority values also affect the
573order in which multiple ``.bbappend`` files for the same recipe are
574applied. You can either specify the priority manually, or allow the
575build system to calculate it based on the layer's dependencies.
576
577To specify the layer's priority manually, use the
578:term:`BBFILE_PRIORITY`
579variable and append the layer's root name:
580::
581
582 BBFILE_PRIORITY_mylayer = "1"
583
584.. note::
585
586 It is possible for a recipe with a lower version number
587 :term:`PV` in a layer that has a higher
588 priority to take precedence.
589
590 Also, the layer priority does not currently affect the precedence
591 order of ``.conf`` or ``.bbclass`` files. Future versions of BitBake
592 might address this.
593
594Managing Layers
595---------------
596
597You can use the BitBake layer management tool ``bitbake-layers`` to
598provide a view into the structure of recipes across a multi-layer
599project. Being able to generate output that reports on configured layers
600with their paths and priorities and on ``.bbappend`` files and their
601applicable recipes can help to reveal potential problems.
602
603For help on the BitBake layer management tool, use the following
604command:
605::
606
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500607 $ bitbake-layers --help
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500608 NOTE: Starting bitbake server...
609 usage: bitbake-layers [-d] [-q] [-F] [--color COLOR] [-h] <subcommand> ...
610
611 BitBake layers utility
612
613 optional arguments:
614 -d, --debug Enable debug output
615 -q, --quiet Print only errors
616 -F, --force Force add without recipe parse verification
617 --color COLOR Colorize output (where COLOR is auto, always, never)
618 -h, --help show this help message and exit
619
620 subcommands:
621 <subcommand>
622 layerindex-fetch Fetches a layer from a layer index along with its
623 dependent layers, and adds them to conf/bblayers.conf.
624 layerindex-show-depends
625 Find layer dependencies from layer index.
626 add-layer Add one or more layers to bblayers.conf.
627 remove-layer Remove one or more layers from bblayers.conf.
628 flatten flatten layer configuration into a separate output
629 directory.
630 show-layers show current configured layers.
631 show-overlayed list overlayed recipes (where the same recipe exists
632 in another layer)
633 show-recipes list available recipes, showing the layer they are
634 provided by
635 show-appends list bbappend files and recipe files they apply to
636 show-cross-depends Show dependencies between recipes that cross layer
637 boundaries.
638 create-layer Create a basic layer
639
640 Use bitbake-layers <subcommand> --help to get help on a specific command
641
642The following list describes the available commands:
643
644- ``help:`` Displays general help or help on a specified command.
645
646- ``show-layers:`` Shows the current configured layers.
647
648- ``show-overlayed:`` Lists overlayed recipes. A recipe is overlayed
649 when a recipe with the same name exists in another layer that has a
650 higher layer priority.
651
652- ``show-recipes:`` Lists available recipes and the layers that
653 provide them.
654
655- ``show-appends:`` Lists ``.bbappend`` files and the recipe files to
656 which they apply.
657
658- ``show-cross-depends:`` Lists dependency relationships between
659 recipes that cross layer boundaries.
660
661- ``add-layer:`` Adds a layer to ``bblayers.conf``.
662
663- ``remove-layer:`` Removes a layer from ``bblayers.conf``
664
665- ``flatten:`` Flattens the layer configuration into a separate
666 output directory. Flattening your layer configuration builds a
667 "flattened" directory that contains the contents of all layers, with
668 any overlayed recipes removed and any ``.bbappend`` files appended to
669 the corresponding recipes. You might have to perform some manual
670 cleanup of the flattened layer as follows:
671
672 - Non-recipe files (such as patches) are overwritten. The flatten
673 command shows a warning for these files.
674
675 - Anything beyond the normal layer setup has been added to the
676 ``layer.conf`` file. Only the lowest priority layer's
677 ``layer.conf`` is used.
678
679 - Overridden and appended items from ``.bbappend`` files need to be
680 cleaned up. The contents of each ``.bbappend`` end up in the
681 flattened recipe. However, if there are appended or changed
682 variable values, you need to tidy these up yourself. Consider the
683 following example. Here, the ``bitbake-layers`` command adds the
684 line ``#### bbappended ...`` so that you know where the following
685 lines originate:
686 ::
687
688 ...
689 DESCRIPTION = "A useful utility"
690 ...
691 EXTRA_OECONF = "--enable-something"
692 ...
693
694 #### bbappended from meta-anotherlayer ####
695
696 DESCRIPTION = "Customized utility"
697 EXTRA_OECONF += "--enable-somethingelse"
698
699
700 Ideally, you would tidy up these utilities as follows:
701 ::
702
703 ...
704 DESCRIPTION = "Customized utility"
705 ...
706 EXTRA_OECONF = "--enable-something --enable-somethingelse"
707 ...
708
709- ``layerindex-fetch``: Fetches a layer from a layer index, along
710 with its dependent layers, and adds the layers to the
711 ``conf/bblayers.conf`` file.
712
713- ``layerindex-show-depends``: Finds layer dependencies from the
714 layer index.
715
716- ``create-layer``: Creates a basic layer.
717
718Creating a General Layer Using the ``bitbake-layers`` Script
719------------------------------------------------------------
720
721The ``bitbake-layers`` script with the ``create-layer`` subcommand
722simplifies creating a new general layer.
723
724.. note::
725
726 - For information on BSP layers, see the ":ref:`bsp-guide/bsp:bsp layers`"
727 section in the Yocto
728 Project Board Specific (BSP) Developer's Guide.
729
730 - In order to use a layer with the OpenEmbedded build system, you
731 need to add the layer to your ``bblayers.conf`` configuration
732 file. See the ":ref:`dev-manual/dev-manual-common-tasks:adding a layer using the \`\`bitbake-layers\`\` script`"
733 section for more information.
734
735The default mode of the script's operation with this subcommand is to
736create a layer with the following:
737
738- A layer priority of 6.
739
740- A ``conf`` subdirectory that contains a ``layer.conf`` file.
741
742- A ``recipes-example`` subdirectory that contains a further
743 subdirectory named ``example``, which contains an ``example.bb``
744 recipe file.
745
746- A ``COPYING.MIT``, which is the license statement for the layer. The
747 script assumes you want to use the MIT license, which is typical for
748 most layers, for the contents of the layer itself.
749
750- A ``README`` file, which is a file describing the contents of your
751 new layer.
752
753In its simplest form, you can use the following command form to create a
754layer. The command creates a layer whose name corresponds to
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500755"your_layer_name" in the current directory:
756::
757
758 $ bitbake-layers create-layer your_layer_name
759
760As an example, the following command creates a layer named ``meta-scottrif``
761in your home directory:
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500762::
763
764 $ cd /usr/home
765 $ bitbake-layers create-layer meta-scottrif
766 NOTE: Starting bitbake server...
767 Add your new layer with 'bitbake-layers add-layer meta-scottrif'
768
769If you want to set the priority of the layer to other than the default
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500770value of "6", you can either use the ``--priority`` option or you
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500771can edit the
772:term:`BBFILE_PRIORITY` value
773in the ``conf/layer.conf`` after the script creates it. Furthermore, if
774you want to give the example recipe file some name other than the
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500775default, you can use the ``--example-recipe-name`` option.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500776
777The easiest way to see how the ``bitbake-layers create-layer`` command
778works is to experiment with the script. You can also read the usage
779information by entering the following:
780::
781
782 $ bitbake-layers create-layer --help
783 NOTE: Starting bitbake server...
784 usage: bitbake-layers create-layer [-h] [--priority PRIORITY]
785 [--example-recipe-name EXAMPLERECIPE]
786 layerdir
787
788 Create a basic layer
789
790 positional arguments:
791 layerdir Layer directory to create
792
793 optional arguments:
794 -h, --help show this help message and exit
795 --priority PRIORITY, -p PRIORITY
796 Layer directory to create
797 --example-recipe-name EXAMPLERECIPE, -e EXAMPLERECIPE
798 Filename of the example recipe
799
800Adding a Layer Using the ``bitbake-layers`` Script
801--------------------------------------------------
802
803Once you create your general layer, you must add it to your
804``bblayers.conf`` file. Adding the layer to this configuration file
805makes the OpenEmbedded build system aware of your layer so that it can
806search it for metadata.
807
808Add your layer by using the ``bitbake-layers add-layer`` command:
809::
810
811 $ bitbake-layers add-layer your_layer_name
812
813Here is an example that adds a
814layer named ``meta-scottrif`` to the configuration file. Following the
815command that adds the layer is another ``bitbake-layers`` command that
816shows the layers that are in your ``bblayers.conf`` file:
817::
818
819 $ bitbake-layers add-layer meta-scottrif
820 NOTE: Starting bitbake server...
821 Parsing recipes: 100% |##########################################################| Time: 0:00:49
822 Parsing of 1441 .bb files complete (0 cached, 1441 parsed). 2055 targets, 56 skipped, 0 masked, 0 errors.
823 $ bitbake-layers show-layers
824 NOTE: Starting bitbake server...
825 layer path priority
826 ==========================================================================
827 meta /home/scottrif/poky/meta 5
828 meta-poky /home/scottrif/poky/meta-poky 5
829 meta-yocto-bsp /home/scottrif/poky/meta-yocto-bsp 5
830 workspace /home/scottrif/poky/build/workspace 99
831 meta-scottrif /home/scottrif/poky/build/meta-scottrif 6
832
833
834Adding the layer to this file
835enables the build system to locate the layer during the build.
836
837.. note::
838
839 During a build, the OpenEmbedded build system looks in the layers
840 from the top of the list down to the bottom in that order.
841
842.. _usingpoky-extend-customimage:
843
844Customizing Images
845==================
846
847You can customize images to satisfy particular requirements. This
848section describes several methods and provides guidelines for each.
849
850.. _usingpoky-extend-customimage-localconf:
851
852Customizing Images Using ``local.conf``
853---------------------------------------
854
855Probably the easiest way to customize an image is to add a package by
856way of the ``local.conf`` configuration file. Because it is limited to
857local use, this method generally only allows you to add packages and is
858not as flexible as creating your own customized image. When you add
859packages using local variables this way, you need to realize that these
860variable changes are in effect for every build and consequently affect
861all images, which might not be what you require.
862
863To add a package to your image using the local configuration file, use
864the ``IMAGE_INSTALL`` variable with the ``_append`` operator:
865::
866
867 IMAGE_INSTALL_append = " strace"
868
869Use of the syntax is important -
870specifically, the space between the quote and the package name, which is
871``strace`` in this example. This space is required since the ``_append``
872operator does not add the space.
873
874Furthermore, you must use ``_append`` instead of the ``+=`` operator if
875you want to avoid ordering issues. The reason for this is because doing
876so unconditionally appends to the variable and avoids ordering problems
877due to the variable being set in image recipes and ``.bbclass`` files
878with operators like ``?=``. Using ``_append`` ensures the operation
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500879takes effect.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500880
881As shown in its simplest use, ``IMAGE_INSTALL_append`` affects all
882images. It is possible to extend the syntax so that the variable applies
883to a specific image only. Here is an example:
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500884::
885
886 IMAGE_INSTALL_append_pn-core-image-minimal = " strace"
887
888This example adds ``strace`` to the ``core-image-minimal`` image only.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500889
890You can add packages using a similar approach through the
891``CORE_IMAGE_EXTRA_INSTALL`` variable. If you use this variable, only
892``core-image-*`` images are affected.
893
894.. _usingpoky-extend-customimage-imagefeatures:
895
896Customizing Images Using Custom ``IMAGE_FEATURES`` and ``EXTRA_IMAGE_FEATURES``
897-------------------------------------------------------------------------------
898
899Another method for customizing your image is to enable or disable
900high-level image features by using the
901:term:`IMAGE_FEATURES` and
902:term:`EXTRA_IMAGE_FEATURES`
903variables. Although the functions for both variables are nearly
904equivalent, best practices dictate using ``IMAGE_FEATURES`` from within
905a recipe and using ``EXTRA_IMAGE_FEATURES`` from within your
906``local.conf`` file, which is found in the
907:term:`Build Directory`.
908
909To understand how these features work, the best reference is
910``meta/classes/core-image.bbclass``. This class lists out the available
911``IMAGE_FEATURES`` of which most map to package groups while some, such
912as ``debug-tweaks`` and ``read-only-rootfs``, resolve as general
913configuration settings.
914
915In summary, the file looks at the contents of the ``IMAGE_FEATURES``
916variable and then maps or configures the feature accordingly. Based on
917this information, the build system automatically adds the appropriate
918packages or configurations to the
919:term:`IMAGE_INSTALL` variable.
920Effectively, you are enabling extra features by extending the class or
921creating a custom class for use with specialized image ``.bb`` files.
922
923Use the ``EXTRA_IMAGE_FEATURES`` variable from within your local
924configuration file. Using a separate area from which to enable features
925with this variable helps you avoid overwriting the features in the image
926recipe that are enabled with ``IMAGE_FEATURES``. The value of
927``EXTRA_IMAGE_FEATURES`` is added to ``IMAGE_FEATURES`` within
928``meta/conf/bitbake.conf``.
929
930To illustrate how you can use these variables to modify your image,
931consider an example that selects the SSH server. The Yocto Project ships
932with two SSH servers you can use with your images: Dropbear and OpenSSH.
933Dropbear is a minimal SSH server appropriate for resource-constrained
934environments, while OpenSSH is a well-known standard SSH server
935implementation. By default, the ``core-image-sato`` image is configured
936to use Dropbear. The ``core-image-full-cmdline`` and ``core-image-lsb``
937images both include OpenSSH. The ``core-image-minimal`` image does not
938contain an SSH server.
939
940You can customize your image and change these defaults. Edit the
941``IMAGE_FEATURES`` variable in your recipe or use the
942``EXTRA_IMAGE_FEATURES`` in your ``local.conf`` file so that it
943configures the image you are working with to include
944``ssh-server-dropbear`` or ``ssh-server-openssh``.
945
946.. note::
947
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500948 See the ":ref:`ref-manual/ref-features:image features`" section in the Yocto
949 Project Reference Manual for a complete list of image features that ship
950 with the Yocto Project.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500951
952.. _usingpoky-extend-customimage-custombb:
953
954Customizing Images Using Custom .bb Files
955-----------------------------------------
956
957You can also customize an image by creating a custom recipe that defines
958additional software as part of the image. The following example shows
959the form for the two lines you need:
960::
961
962 IMAGE_INSTALL = "packagegroup-core-x11-base package1 package2"
963 inherit core-image
964
965Defining the software using a custom recipe gives you total control over
966the contents of the image. It is important to use the correct names of
967packages in the ``IMAGE_INSTALL`` variable. You must use the
968OpenEmbedded notation and not the Debian notation for the names (e.g.
969``glibc-dev`` instead of ``libc6-dev``).
970
971The other method for creating a custom image is to base it on an
972existing image. For example, if you want to create an image based on
973``core-image-sato`` but add the additional package ``strace`` to the
974image, copy the ``meta/recipes-sato/images/core-image-sato.bb`` to a new
975``.bb`` and add the following line to the end of the copy:
976::
977
978 IMAGE_INSTALL += "strace"
979
980.. _usingpoky-extend-customimage-customtasks:
981
982Customizing Images Using Custom Package Groups
983----------------------------------------------
984
985For complex custom images, the best approach for customizing an image is
986to create a custom package group recipe that is used to build the image
987or images. A good example of a package group recipe is
988``meta/recipes-core/packagegroups/packagegroup-base.bb``.
989
990If you examine that recipe, you see that the ``PACKAGES`` variable lists
991the package group packages to produce. The ``inherit packagegroup``
992statement sets appropriate default values and automatically adds
993``-dev``, ``-dbg``, and ``-ptest`` complementary packages for each
994package specified in the ``PACKAGES`` statement.
995
996.. note::
997
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500998 The ``inherit packagegroup`` line should be located near the top of the
999 recipe, certainly before the ``PACKAGES`` statement.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001000
1001For each package you specify in ``PACKAGES``, you can use ``RDEPENDS``
1002and ``RRECOMMENDS`` entries to provide a list of packages the parent
1003task package should contain. You can see examples of these further down
1004in the ``packagegroup-base.bb`` recipe.
1005
1006Here is a short, fabricated example showing the same basic pieces for a
1007hypothetical packagegroup defined in ``packagegroup-custom.bb``, where
1008the variable ``PN`` is the standard way to abbreviate the reference to
1009the full packagegroup name ``packagegroup-custom``:
1010::
1011
1012 DESCRIPTION = "My Custom Package Groups"
1013
1014 inherit packagegroup
1015
1016 PACKAGES = "\
1017 ${PN}-apps \
1018 ${PN}-tools \
1019 "
1020
1021 RDEPENDS_${PN}-apps = "\
1022 dropbear \
1023 portmap \
1024 psplash"
1025
1026 RDEPENDS_${PN}-tools = "\
1027 oprofile \
1028 oprofileui-server \
1029 lttng-tools"
1030
1031 RRECOMMENDS_${PN}-tools = "\
1032 kernel-module-oprofile"
1033
1034In the previous example, two package group packages are created with
1035their dependencies and their recommended package dependencies listed:
1036``packagegroup-custom-apps``, and ``packagegroup-custom-tools``. To
1037build an image using these package group packages, you need to add
1038``packagegroup-custom-apps`` and/or ``packagegroup-custom-tools`` to
1039``IMAGE_INSTALL``. For other forms of image dependencies see the other
1040areas of this section.
1041
1042.. _usingpoky-extend-customimage-image-name:
1043
1044Customizing an Image Hostname
1045-----------------------------
1046
1047By default, the configured hostname (i.e. ``/etc/hostname``) in an image
1048is the same as the machine name. For example, if
1049:term:`MACHINE` equals "qemux86", the
1050configured hostname written to ``/etc/hostname`` is "qemux86".
1051
1052You can customize this name by altering the value of the "hostname"
1053variable in the ``base-files`` recipe using either an append file or a
1054configuration file. Use the following in an append file:
1055::
1056
1057 hostname = "myhostname"
1058
1059Use the following in a configuration file:
1060::
1061
1062 hostname_pn-base-files = "myhostname"
1063
1064Changing the default value of the variable "hostname" can be useful in
1065certain situations. For example, suppose you need to do extensive
1066testing on an image and you would like to easily identify the image
1067under test from existing images with typical default hostnames. In this
1068situation, you could change the default hostname to "testme", which
1069results in all the images using the name "testme". Once testing is
1070complete and you do not need to rebuild the image for test any longer,
1071you can easily reset the default hostname.
1072
1073Another point of interest is that if you unset the variable, the image
1074will have no default hostname in the filesystem. Here is an example that
1075unsets the variable in a configuration file:
1076::
1077
1078 hostname_pn-base-files = ""
1079
1080Having no default hostname in the filesystem is suitable for
1081environments that use dynamic hostnames such as virtual machines.
1082
1083.. _new-recipe-writing-a-new-recipe:
1084
1085Writing a New Recipe
1086====================
1087
1088Recipes (``.bb`` files) are fundamental components in the Yocto Project
1089environment. Each software component built by the OpenEmbedded build
1090system requires a recipe to define the component. This section describes
1091how to create, write, and test a new recipe.
1092
1093.. note::
1094
1095 For information on variables that are useful for recipes and for
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001096 information about recipe naming issues, see the
1097 ":ref:`ref-manual/ref-varlocality:recipes`" section of the Yocto Project
1098 Reference Manual.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001099
1100.. _new-recipe-overview:
1101
1102Overview
1103--------
1104
1105The following figure shows the basic process for creating a new recipe.
1106The remainder of the section provides details for the steps.
1107
1108.. image:: figures/recipe-workflow.png
1109 :align: center
1110
1111.. _new-recipe-locate-or-automatically-create-a-base-recipe:
1112
1113Locate or Automatically Create a Base Recipe
1114--------------------------------------------
1115
1116You can always write a recipe from scratch. However, three choices exist
1117that can help you quickly get a start on a new recipe:
1118
1119- ``devtool add``: A command that assists in creating a recipe and an
1120 environment conducive to development.
1121
1122- ``recipetool create``: A command provided by the Yocto Project that
1123 automates creation of a base recipe based on the source files.
1124
1125- *Existing Recipes:* Location and modification of an existing recipe
1126 that is similar in function to the recipe you need.
1127
1128.. note::
1129
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001130 For information on recipe syntax, see the
1131 ":ref:`dev-manual/dev-manual-common-tasks:recipe syntax`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001132
1133.. _new-recipe-creating-the-base-recipe-using-devtool:
1134
1135Creating the Base Recipe Using ``devtool add``
1136~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1137
1138The ``devtool add`` command uses the same logic for auto-creating the
1139recipe as ``recipetool create``, which is listed below. Additionally,
1140however, ``devtool add`` sets up an environment that makes it easy for
1141you to patch the source and to make changes to the recipe as is often
1142necessary when adding a recipe to build a new piece of software to be
1143included in a build.
1144
1145You can find a complete description of the ``devtool add`` command in
1146the ":ref:`sdk-a-closer-look-at-devtool-add`" section
1147in the Yocto Project Application Development and the Extensible Software
1148Development Kit (eSDK) manual.
1149
1150.. _new-recipe-creating-the-base-recipe-using-recipetool:
1151
1152Creating the Base Recipe Using ``recipetool create``
1153~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1154
1155``recipetool create`` automates creation of a base recipe given a set of
1156source code files. As long as you can extract or point to the source
1157files, the tool will construct a recipe and automatically configure all
1158pre-build information into the recipe. For example, suppose you have an
1159application that builds using Autotools. Creating the base recipe using
1160``recipetool`` results in a recipe that has the pre-build dependencies,
1161license requirements, and checksums configured.
1162
1163To run the tool, you just need to be in your
1164:term:`Build Directory` and have sourced the
1165build environment setup script (i.e.
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001166:ref:`structure-core-script`).
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001167To get help on the tool, use the following command:
1168::
1169
1170 $ recipetool -h
1171 NOTE: Starting bitbake server...
1172 usage: recipetool [-d] [-q] [--color COLOR] [-h] <subcommand> ...
1173
1174 OpenEmbedded recipe tool
1175
1176 options:
1177 -d, --debug Enable debug output
1178 -q, --quiet Print only errors
1179 --color COLOR Colorize output (where COLOR is auto, always, never)
1180 -h, --help show this help message and exit
1181
1182 subcommands:
1183 create Create a new recipe
1184 newappend Create a bbappend for the specified target in the specified
1185 layer
1186 setvar Set a variable within a recipe
1187 appendfile Create/update a bbappend to replace a target file
1188 appendsrcfiles Create/update a bbappend to add or replace source files
1189 appendsrcfile Create/update a bbappend to add or replace a source file
1190 Use recipetool <subcommand> --help to get help on a specific command
1191
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001192Running ``recipetool create -o OUTFILE`` creates the base recipe and
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001193locates it properly in the layer that contains your source files.
1194Following are some syntax examples:
1195
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001196 - Use this syntax to generate a recipe based on source. Once generated,
1197 the recipe resides in the existing source code layer:
1198 ::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001199
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001200 recipetool create -o OUTFILE source
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001201
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001202 - Use this syntax to generate a recipe using code that
1203 you extract from source. The extracted code is placed in its own layer
1204 defined by ``EXTERNALSRC``.
1205 ::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001206
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001207 recipetool create -o OUTFILE -x EXTERNALSRC source
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001208
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001209 - Use this syntax to generate a recipe based on source. The options
1210 direct ``recipetool`` to generate debugging information. Once generated,
1211 the recipe resides in the existing source code layer:
1212 ::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001213
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001214 recipetool create -d -o OUTFILE source
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001215
1216.. _new-recipe-locating-and-using-a-similar-recipe:
1217
1218Locating and Using a Similar Recipe
1219~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1220
1221Before writing a recipe from scratch, it is often useful to discover
1222whether someone else has already written one that meets (or comes close
1223to meeting) your needs. The Yocto Project and OpenEmbedded communities
1224maintain many recipes that might be candidates for what you are doing.
1225You can find a good central index of these recipes in the `OpenEmbedded
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001226Layer Index <https://layers.openembedded.org>`__.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001227
1228Working from an existing recipe or a skeleton recipe is the best way to
1229get started. Here are some points on both methods:
1230
1231- *Locate and modify a recipe that is close to what you want to do:*
1232 This method works when you are familiar with the current recipe
1233 space. The method does not work so well for those new to the Yocto
1234 Project or writing recipes.
1235
1236 Some risks associated with this method are using a recipe that has
1237 areas totally unrelated to what you are trying to accomplish with
1238 your recipe, not recognizing areas of the recipe that you might have
1239 to add from scratch, and so forth. All these risks stem from
1240 unfamiliarity with the existing recipe space.
1241
1242- *Use and modify the following skeleton recipe:* If for some reason
1243 you do not want to use ``recipetool`` and you cannot find an existing
1244 recipe that is close to meeting your needs, you can use the following
1245 structure to provide the fundamental areas of a new recipe.
1246 ::
1247
1248 DESCRIPTION = ""
1249 HOMEPAGE = ""
1250 LICENSE = ""
1251 SECTION = ""
1252 DEPENDS = ""
1253 LIC_FILES_CHKSUM = ""
1254
1255 SRC_URI = ""
1256
1257.. _new-recipe-storing-and-naming-the-recipe:
1258
1259Storing and Naming the Recipe
1260-----------------------------
1261
1262Once you have your base recipe, you should put it in your own layer and
1263name it appropriately. Locating it correctly ensures that the
1264OpenEmbedded build system can find it when you use BitBake to process
1265the recipe.
1266
1267- *Storing Your Recipe:* The OpenEmbedded build system locates your
1268 recipe through the layer's ``conf/layer.conf`` file and the
1269 :term:`BBFILES` variable. This
1270 variable sets up a path from which the build system can locate
1271 recipes. Here is the typical use:
1272 ::
1273
1274 BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
1275 ${LAYERDIR}/recipes-*/*/*.bbappend"
1276
1277 Consequently, you need to be sure you locate your new recipe inside
1278 your layer such that it can be found.
1279
1280 You can find more information on how layers are structured in the
1281 "`Understanding and Creating
1282 Layers <#understanding-and-creating-layers>`__" section.
1283
1284- *Naming Your Recipe:* When you name your recipe, you need to follow
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001285 this naming convention:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001286 ::
1287
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001288 basename_version.bb
1289
1290 Use lower-cased characters and do not include the reserved suffixes
1291 ``-native``, ``-cross``, ``-initial``, or ``-dev`` casually (i.e. do not use
1292 them as part of your recipe name unless the string applies). Here are some
1293 examples:
1294
1295 .. code-block:: none
1296
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001297 cups_1.7.0.bb
1298 gawk_4.0.2.bb
1299 irssi_0.8.16-rc1.bb
1300
1301.. _new-recipe-running-a-build-on-the-recipe:
1302
1303Running a Build on the Recipe
1304-----------------------------
1305
1306Creating a new recipe is usually an iterative process that requires
1307using BitBake to process the recipe multiple times in order to
1308progressively discover and add information to the recipe file.
1309
1310Assuming you have sourced the build environment setup script (i.e.
1311:ref:`structure-core-script`) and you are in
1312the :term:`Build Directory`, use
1313BitBake to process your recipe. All you need to provide is the
1314``basename`` of the recipe as described in the previous section:
1315::
1316
1317 $ bitbake basename
1318
1319During the build, the OpenEmbedded build system creates a temporary work
1320directory for each recipe
1321(``${``\ :term:`WORKDIR`\ ``}``)
1322where it keeps extracted source files, log files, intermediate
1323compilation and packaging files, and so forth.
1324
1325The path to the per-recipe temporary work directory depends on the
1326context in which it is being built. The quickest way to find this path
1327is to have BitBake return it by running the following:
1328::
1329
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001330 $ bitbake -e basename | grep ^WORKDIR=
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001331
1332As an example, assume a Source Directory
1333top-level folder named ``poky``, a default Build Directory at
1334``poky/build``, and a ``qemux86-poky-linux`` machine target system.
1335Furthermore, suppose your recipe is named ``foo_1.3.0.bb``. In this
1336case, the work directory the build system uses to build the package
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001337would be as follows:
1338::
1339
1340 poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0
1341
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001342Inside this directory you can find sub-directories such as ``image``,
1343``packages-split``, and ``temp``. After the build, you can examine these
1344to determine how well the build went.
1345
1346.. note::
1347
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001348 You can find log files for each task in the recipe's ``temp``
1349 directory (e.g. ``poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0/temp``).
1350 Log files are named ``log.taskname`` (e.g. ``log.do_configure``,
1351 ``log.do_fetch``, and ``log.do_compile``).
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001352
1353You can find more information about the build process in
1354":doc:`../overview-manual/overview-manual-development-environment`"
1355chapter of the Yocto Project Overview and Concepts Manual.
1356
1357.. _new-recipe-fetching-code:
1358
1359Fetching Code
1360-------------
1361
1362The first thing your recipe must do is specify how to fetch the source
1363files. Fetching is controlled mainly through the
1364:term:`SRC_URI` variable. Your recipe
1365must have a ``SRC_URI`` variable that points to where the source is
1366located. For a graphical representation of source locations, see the
1367":ref:`sources-dev-environment`" section in
1368the Yocto Project Overview and Concepts Manual.
1369
1370The :ref:`ref-tasks-fetch` task uses
1371the prefix of each entry in the ``SRC_URI`` variable value to determine
1372which :ref:`fetcher <bitbake:bb-fetchers>` to use to get your
1373source files. It is the ``SRC_URI`` variable that triggers the fetcher.
1374The :ref:`ref-tasks-patch` task uses
1375the variable after source is fetched to apply patches. The OpenEmbedded
1376build system uses
1377:term:`FILESOVERRIDES` for
1378scanning directory locations for local files in ``SRC_URI``.
1379
1380The ``SRC_URI`` variable in your recipe must define each unique location
1381for your source files. It is good practice to not hard-code version
1382numbers in a URL used in ``SRC_URI``. Rather than hard-code these
1383values, use ``${``\ :term:`PV`\ ``}``,
1384which causes the fetch process to use the version specified in the
1385recipe filename. Specifying the version in this manner means that
1386upgrading the recipe to a future version is as simple as renaming the
1387recipe to match the new version.
1388
1389Here is a simple example from the
1390``meta/recipes-devtools/strace/strace_5.5.bb`` recipe where the source
1391comes from a single tarball. Notice the use of the
1392:term:`PV` variable:
1393::
1394
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001395 SRC_URI = "https://strace.io/files/${PV}/strace-${PV}.tar.xz \
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001396
1397Files mentioned in ``SRC_URI`` whose names end in a typical archive
1398extension (e.g. ``.tar``, ``.tar.gz``, ``.tar.bz2``, ``.zip``, and so
1399forth), are automatically extracted during the
1400:ref:`ref-tasks-unpack` task. For
1401another example that specifies these types of files, see the
1402"`Autotooled Package <#new-recipe-autotooled-package>`__" section.
1403
1404Another way of specifying source is from an SCM. For Git repositories,
1405you must specify :term:`SRCREV` and
1406you should specify :term:`PV` to include
1407the revision with :term:`SRCPV`. Here
1408is an example from the recipe
1409``meta/recipes-kernel/blktrace/blktrace_git.bb``:
1410::
1411
1412 SRCREV = "d6918c8832793b4205ed3bfede78c2f915c23385"
1413
1414 PR = "r6"
1415 PV = "1.0.5+git${SRCPV}"
1416
1417 SRC_URI = "git://git.kernel.dk/blktrace.git \
1418 file://ldflags.patch"
1419
1420If your ``SRC_URI`` statement includes URLs pointing to individual files
1421fetched from a remote server other than a version control system,
1422BitBake attempts to verify the files against checksums defined in your
1423recipe to ensure they have not been tampered with or otherwise modified
1424since the recipe was written. Two checksums are used:
1425``SRC_URI[md5sum]`` and ``SRC_URI[sha256sum]``.
1426
1427If your ``SRC_URI`` variable points to more than a single URL (excluding
1428SCM URLs), you need to provide the ``md5`` and ``sha256`` checksums for
1429each URL. For these cases, you provide a name for each URL as part of
1430the ``SRC_URI`` and then reference that name in the subsequent checksum
1431statements. Here is an example combining lines from the files
1432``git.inc`` and ``git_2.24.1.bb``:
1433::
1434
1435 SRC_URI = "${KERNELORG_MIRROR}/software/scm/git/git-${PV}.tar.gz;name=tarball \
1436 ${KERNELORG_MIRROR}/software/scm/git/git-manpages-${PV}.tar.gz;name=manpages"
1437
1438 SRC_URI[tarball.md5sum] = "166bde96adbbc11c8843d4f8f4f9811b"
1439 SRC_URI[tarball.sha256sum] = "ad5334956301c86841eb1e5b1bb20884a6bad89a10a6762c958220c7cf64da02"
1440 SRC_URI[manpages.md5sum] = "31c2272a8979022497ba3d4202df145d"
1441 SRC_URI[manpages.sha256sum] = "9a7ae3a093bea39770eb96ca3e5b40bff7af0b9f6123f089d7821d0e5b8e1230"
1442
1443Proper values for ``md5`` and ``sha256`` checksums might be available
1444with other signatures on the download page for the upstream source (e.g.
1445``md5``, ``sha1``, ``sha256``, ``GPG``, and so forth). Because the
1446OpenEmbedded build system only deals with ``sha256sum`` and ``md5sum``,
1447you should verify all the signatures you find by hand.
1448
1449If no ``SRC_URI`` checksums are specified when you attempt to build the
1450recipe, or you provide an incorrect checksum, the build will produce an
1451error for each missing or incorrect checksum. As part of the error
1452message, the build system provides the checksum string corresponding to
1453the fetched file. Once you have the correct checksums, you can copy and
1454paste them into your recipe and then run the build again to continue.
1455
1456.. note::
1457
1458 As mentioned, if the upstream source provides signatures for
1459 verifying the downloaded source code, you should verify those
1460 manually before setting the checksum values in the recipe and
1461 continuing with the build.
1462
1463This final example is a bit more complicated and is from the
1464``meta/recipes-sato/rxvt-unicode/rxvt-unicode_9.20.bb`` recipe. The
1465example's ``SRC_URI`` statement identifies multiple files as the source
1466files for the recipe: a tarball, a patch file, a desktop file, and an
1467icon.
1468::
1469
1470 SRC_URI = "http://dist.schmorp.de/rxvt-unicode/Attic/rxvt-unicode-${PV}.tar.bz2 \
1471 file://xwc.patch \
1472 file://rxvt.desktop \
1473 file://rxvt.png"
1474
1475When you specify local files using the ``file://`` URI protocol, the
1476build system fetches files from the local machine. The path is relative
1477to the :term:`FILESPATH` variable
1478and searches specific directories in a certain order:
1479``${``\ :term:`BP`\ ``}``,
1480``${``\ :term:`BPN`\ ``}``, and
1481``files``. The directories are assumed to be subdirectories of the
1482directory in which the recipe or append file resides. For another
1483example that specifies these types of files, see the "`Single .c File
1484Package (Hello
1485World!) <#new-recipe-single-c-file-package-hello-world>`__" section.
1486
1487The previous example also specifies a patch file. Patch files are files
1488whose names usually end in ``.patch`` or ``.diff`` but can end with
1489compressed suffixes such as ``diff.gz`` and ``patch.bz2``, for example.
1490The build system automatically applies patches as described in the
1491"`Patching Code <#new-recipe-patching-code>`__" section.
1492
1493.. _new-recipe-unpacking-code:
1494
1495Unpacking Code
1496--------------
1497
1498During the build, the
1499:ref:`ref-tasks-unpack` task unpacks
1500the source with ``${``\ :term:`S`\ ``}``
1501pointing to where it is unpacked.
1502
1503If you are fetching your source files from an upstream source archived
1504tarball and the tarball's internal structure matches the common
1505convention of a top-level subdirectory named
1506``${``\ :term:`BPN`\ ``}-${``\ :term:`PV`\ ``}``,
1507then you do not need to set ``S``. However, if ``SRC_URI`` specifies to
1508fetch source from an archive that does not use this convention, or from
1509an SCM like Git or Subversion, your recipe needs to define ``S``.
1510
1511If processing your recipe using BitBake successfully unpacks the source
1512files, you need to be sure that the directory pointed to by ``${S}``
1513matches the structure of the source.
1514
1515.. _new-recipe-patching-code:
1516
1517Patching Code
1518-------------
1519
1520Sometimes it is necessary to patch code after it has been fetched. Any
1521files mentioned in ``SRC_URI`` whose names end in ``.patch`` or
1522``.diff`` or compressed versions of these suffixes (e.g. ``diff.gz`` are
1523treated as patches. The
1524:ref:`ref-tasks-patch` task
1525automatically applies these patches.
1526
1527The build system should be able to apply patches with the "-p1" option
1528(i.e. one directory level in the path will be stripped off). If your
1529patch needs to have more directory levels stripped off, specify the
1530number of levels using the "striplevel" option in the ``SRC_URI`` entry
1531for the patch. Alternatively, if your patch needs to be applied in a
1532specific subdirectory that is not specified in the patch file, use the
1533"patchdir" option in the entry.
1534
1535As with all local files referenced in
1536:term:`SRC_URI` using ``file://``,
1537you should place patch files in a directory next to the recipe either
1538named the same as the base name of the recipe
1539(:term:`BP` and
1540:term:`BPN`) or "files".
1541
1542.. _new-recipe-licensing:
1543
1544Licensing
1545---------
1546
1547Your recipe needs to have both the
1548:term:`LICENSE` and
1549:term:`LIC_FILES_CHKSUM`
1550variables:
1551
1552- ``LICENSE``: This variable specifies the license for the software.
1553 If you do not know the license under which the software you are
1554 building is distributed, you should go to the source code and look
1555 for that information. Typical files containing this information
1556 include ``COPYING``, ``LICENSE``, and ``README`` files. You could
1557 also find the information near the top of a source file. For example,
1558 given a piece of software licensed under the GNU General Public
1559 License version 2, you would set ``LICENSE`` as follows:
1560 ::
1561
1562 LICENSE = "GPLv2"
1563
1564 The licenses you specify within ``LICENSE`` can have any name as long
1565 as you do not use spaces, since spaces are used as separators between
1566 license names. For standard licenses, use the names of the files in
1567 ``meta/files/common-licenses/`` or the ``SPDXLICENSEMAP`` flag names
1568 defined in ``meta/conf/licenses.conf``.
1569
1570- ``LIC_FILES_CHKSUM``: The OpenEmbedded build system uses this
1571 variable to make sure the license text has not changed. If it has,
1572 the build produces an error and it affords you the chance to figure
1573 it out and correct the problem.
1574
1575 You need to specify all applicable licensing files for the software.
1576 At the end of the configuration step, the build process will compare
1577 the checksums of the files to be sure the text has not changed. Any
1578 differences result in an error with the message containing the
1579 current checksum. For more explanation and examples of how to set the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001580 ``LIC_FILES_CHKSUM`` variable, see the
1581 ":ref:`dev-manual/dev-manual-common-tasks:tracking license changes`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001582
1583 To determine the correct checksum string, you can list the
1584 appropriate files in the ``LIC_FILES_CHKSUM`` variable with incorrect
1585 md5 strings, attempt to build the software, and then note the
1586 resulting error messages that will report the correct md5 strings.
1587 See the "`Fetching Code <#new-recipe-fetching-code>`__" section for
1588 additional information.
1589
1590 Here is an example that assumes the software has a ``COPYING`` file:
1591 ::
1592
1593 LIC_FILES_CHKSUM = "file://COPYING;md5=xxx"
1594
1595 When you try to build the
1596 software, the build system will produce an error and give you the
1597 correct string that you can substitute into the recipe file for a
1598 subsequent build.
1599
1600.. _new-dependencies:
1601
1602Dependencies
1603------------
1604
1605Most software packages have a short list of other packages that they
1606require, which are called dependencies. These dependencies fall into two
1607main categories: build-time dependencies, which are required when the
1608software is built; and runtime dependencies, which are required to be
1609installed on the target in order for the software to run.
1610
1611Within a recipe, you specify build-time dependencies using the
1612:term:`DEPENDS` variable. Although
1613nuances exist, items specified in ``DEPENDS`` should be names of other
1614recipes. It is important that you specify all build-time dependencies
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001615explicitly.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001616
1617Another consideration is that configure scripts might automatically
1618check for optional dependencies and enable corresponding functionality
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001619if those dependencies are found. If you wish to make a recipe that is
1620more generally useful (e.g. publish the recipe in a layer for others to
1621use), instead of hard-disabling the functionality, you can use the
1622:term:`PACKAGECONFIG` variable to allow functionality and the
1623corresponding dependencies to be enabled and disabled easily by other
1624users of the recipe.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001625
1626Similar to build-time dependencies, you specify runtime dependencies
1627through a variable -
1628:term:`RDEPENDS`, which is
1629package-specific. All variables that are package-specific need to have
1630the name of the package added to the end as an override. Since the main
1631package for a recipe has the same name as the recipe, and the recipe's
1632name can be found through the
1633``${``\ :term:`PN`\ ``}`` variable, then
1634you specify the dependencies for the main package by setting
1635``RDEPENDS_${PN}``. If the package were named ``${PN}-tools``, then you
1636would set ``RDEPENDS_${PN}-tools``, and so forth.
1637
1638Some runtime dependencies will be set automatically at packaging time.
1639These dependencies include any shared library dependencies (i.e. if a
1640package "example" contains "libexample" and another package "mypackage"
1641contains a binary that links to "libexample" then the OpenEmbedded build
1642system will automatically add a runtime dependency to "mypackage" on
1643"example"). See the
1644":ref:`overview-manual/overview-manual-concepts:automatically added runtime dependencies`"
1645section in the Yocto Project Overview and Concepts Manual for further
1646details.
1647
1648.. _new-recipe-configuring-the-recipe:
1649
1650Configuring the Recipe
1651----------------------
1652
1653Most software provides some means of setting build-time configuration
1654options before compilation. Typically, setting these options is
1655accomplished by running a configure script with options, or by modifying
1656a build configuration file.
1657
1658.. note::
1659
1660 As of Yocto Project Release 1.7, some of the core recipes that
1661 package binary configuration scripts now disable the scripts due to
1662 the scripts previously requiring error-prone path substitution. The
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001663 OpenEmbedded build system uses ``pkg-config`` now, which is much more
1664 robust. You can find a list of the ``*-config`` scripts that are disabled
1665 in the ":ref:`migration-1.7-binary-configuration-scripts-disabled`" section
1666 in the Yocto Project Reference Manual.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001667
1668A major part of build-time configuration is about checking for
1669build-time dependencies and possibly enabling optional functionality as
1670a result. You need to specify any build-time dependencies for the
1671software you are building in your recipe's
1672:term:`DEPENDS` value, in terms of
1673other recipes that satisfy those dependencies. You can often find
1674build-time or runtime dependencies described in the software's
1675documentation.
1676
1677The following list provides configuration items of note based on how
1678your software is built:
1679
1680- *Autotools:* If your source files have a ``configure.ac`` file, then
1681 your software is built using Autotools. If this is the case, you just
1682 need to worry about modifying the configuration.
1683
1684 When using Autotools, your recipe needs to inherit the
1685 :ref:`autotools <ref-classes-autotools>` class
1686 and your recipe does not have to contain a
1687 :ref:`ref-tasks-configure` task.
1688 However, you might still want to make some adjustments. For example,
1689 you can set
1690 :term:`EXTRA_OECONF` or
1691 :term:`PACKAGECONFIG_CONFARGS`
1692 to pass any needed configure options that are specific to the recipe.
1693
1694- *CMake:* If your source files have a ``CMakeLists.txt`` file, then
1695 your software is built using CMake. If this is the case, you just
1696 need to worry about modifying the configuration.
1697
1698 When you use CMake, your recipe needs to inherit the
1699 :ref:`cmake <ref-classes-cmake>` class and your
1700 recipe does not have to contain a
1701 :ref:`ref-tasks-configure` task.
1702 You can make some adjustments by setting
1703 :term:`EXTRA_OECMAKE` to
1704 pass any needed configure options that are specific to the recipe.
1705
1706 .. note::
1707
1708 If you need to install one or more custom CMake toolchain files
1709 that are supplied by the application you are building, install the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001710 files to ``${D}${datadir}/cmake/Modules`` during ``do_install``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001711
1712- *Other:* If your source files do not have a ``configure.ac`` or
1713 ``CMakeLists.txt`` file, then your software is built using some
1714 method other than Autotools or CMake. If this is the case, you
1715 normally need to provide a
1716 :ref:`ref-tasks-configure` task
1717 in your recipe unless, of course, there is nothing to configure.
1718
1719 Even if your software is not being built by Autotools or CMake, you
1720 still might not need to deal with any configuration issues. You need
1721 to determine if configuration is even a required step. You might need
1722 to modify a Makefile or some configuration file used for the build to
1723 specify necessary build options. Or, perhaps you might need to run a
1724 provided, custom configure script with the appropriate options.
1725
1726 For the case involving a custom configure script, you would run
1727 ``./configure --help`` and look for the options you need to set.
1728
1729Once configuration succeeds, it is always good practice to look at the
1730``log.do_configure`` file to ensure that the appropriate options have
1731been enabled and no additional build-time dependencies need to be added
1732to ``DEPENDS``. For example, if the configure script reports that it
1733found something not mentioned in ``DEPENDS``, or that it did not find
1734something that it needed for some desired optional functionality, then
1735you would need to add those to ``DEPENDS``. Looking at the log might
1736also reveal items being checked for, enabled, or both that you do not
1737want, or items not being found that are in ``DEPENDS``, in which case
1738you would need to look at passing extra options to the configure script
1739as needed. For reference information on configure options specific to
1740the software you are building, you can consult the output of the
1741``./configure --help`` command within ``${S}`` or consult the software's
1742upstream documentation.
1743
1744.. _new-recipe-using-headers-to-interface-with-devices:
1745
1746Using Headers to Interface with Devices
1747---------------------------------------
1748
1749If your recipe builds an application that needs to communicate with some
1750device or needs an API into a custom kernel, you will need to provide
1751appropriate header files. Under no circumstances should you ever modify
1752the existing
1753``meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc`` file.
1754These headers are used to build ``libc`` and must not be compromised
1755with custom or machine-specific header information. If you customize
1756``libc`` through modified headers all other applications that use
1757``libc`` thus become affected.
1758
1759.. note::
1760
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001761 Never copy and customize the ``libc`` header file (i.e.
1762 ``meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc``).
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001763
1764The correct way to interface to a device or custom kernel is to use a
1765separate package that provides the additional headers for the driver or
1766other unique interfaces. When doing so, your application also becomes
1767responsible for creating a dependency on that specific provider.
1768
1769Consider the following:
1770
1771- Never modify ``linux-libc-headers.inc``. Consider that file to be
1772 part of the ``libc`` system, and not something you use to access the
1773 kernel directly. You should access ``libc`` through specific ``libc``
1774 calls.
1775
1776- Applications that must talk directly to devices should either provide
1777 necessary headers themselves, or establish a dependency on a special
1778 headers package that is specific to that driver.
1779
1780For example, suppose you want to modify an existing header that adds I/O
1781control or network support. If the modifications are used by a small
1782number programs, providing a unique version of a header is easy and has
1783little impact. When doing so, bear in mind the guidelines in the
1784previous list.
1785
1786.. note::
1787
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001788 If for some reason your changes need to modify the behavior of the ``libc``,
1789 and subsequently all other applications on the system, use a ``.bbappend``
1790 to modify the ``linux-kernel-headers.inc`` file. However, take care to not
1791 make the changes machine specific.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001792
1793Consider a case where your kernel is older and you need an older
1794``libc`` ABI. The headers installed by your recipe should still be a
1795standard mainline kernel, not your own custom one.
1796
1797When you use custom kernel headers you need to get them from
1798:term:`STAGING_KERNEL_DIR`,
1799which is the directory with kernel headers that are required to build
1800out-of-tree modules. Your recipe will also need the following:
1801::
1802
1803 do_configure[depends] += "virtual/kernel:do_shared_workdir"
1804
1805.. _new-recipe-compilation:
1806
1807Compilation
1808-----------
1809
1810During a build, the ``do_compile`` task happens after source is fetched,
1811unpacked, and configured. If the recipe passes through ``do_compile``
1812successfully, nothing needs to be done.
1813
1814However, if the compile step fails, you need to diagnose the failure.
1815Here are some common issues that cause failures.
1816
1817.. note::
1818
1819 For cases where improper paths are detected for configuration files
1820 or for when libraries/headers cannot be found, be sure you are using
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001821 the more robust ``pkg-config``. See the note in section
1822 ":ref:`new-recipe-configuring-the-recipe`" for additional information.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001823
1824- *Parallel build failures:* These failures manifest themselves as
1825 intermittent errors, or errors reporting that a file or directory
1826 that should be created by some other part of the build process could
1827 not be found. This type of failure can occur even if, upon
1828 inspection, the file or directory does exist after the build has
1829 failed, because that part of the build process happened in the wrong
1830 order.
1831
1832 To fix the problem, you need to either satisfy the missing dependency
1833 in the Makefile or whatever script produced the Makefile, or (as a
1834 workaround) set :term:`PARALLEL_MAKE` to an empty string:
1835 ::
1836
1837 PARALLEL_MAKE = ""
1838
1839 For information on parallel Makefile issues, see the "`Debugging
1840 Parallel Make Races <#debugging-parallel-make-races>`__" section.
1841
1842- *Improper host path usage:* This failure applies to recipes building
1843 for the target or ``nativesdk`` only. The failure occurs when the
1844 compilation process uses improper headers, libraries, or other files
1845 from the host system when cross-compiling for the target.
1846
1847 To fix the problem, examine the ``log.do_compile`` file to identify
1848 the host paths being used (e.g. ``/usr/include``, ``/usr/lib``, and
1849 so forth) and then either add configure options, apply a patch, or do
1850 both.
1851
1852- *Failure to find required libraries/headers:* If a build-time
1853 dependency is missing because it has not been declared in
1854 :term:`DEPENDS`, or because the
1855 dependency exists but the path used by the build process to find the
1856 file is incorrect and the configure step did not detect it, the
1857 compilation process could fail. For either of these failures, the
1858 compilation process notes that files could not be found. In these
1859 cases, you need to go back and add additional options to the
1860 configure script as well as possibly add additional build-time
1861 dependencies to ``DEPENDS``.
1862
1863 Occasionally, it is necessary to apply a patch to the source to
1864 ensure the correct paths are used. If you need to specify paths to
1865 find files staged into the sysroot from other recipes, use the
1866 variables that the OpenEmbedded build system provides (e.g.
1867 ``STAGING_BINDIR``, ``STAGING_INCDIR``, ``STAGING_DATADIR``, and so
1868 forth).
1869
1870.. _new-recipe-installing:
1871
1872Installing
1873----------
1874
1875During ``do_install``, the task copies the built files along with their
1876hierarchy to locations that would mirror their locations on the target
1877device. The installation process copies files from the
1878``${``\ :term:`S`\ ``}``,
1879``${``\ :term:`B`\ ``}``, and
1880``${``\ :term:`WORKDIR`\ ``}``
1881directories to the ``${``\ :term:`D`\ ``}``
1882directory to create the structure as it should appear on the target
1883system.
1884
1885How your software is built affects what you must do to be sure your
1886software is installed correctly. The following list describes what you
1887must do for installation depending on the type of build system used by
1888the software being built:
1889
1890- *Autotools and CMake:* If the software your recipe is building uses
1891 Autotools or CMake, the OpenEmbedded build system understands how to
1892 install the software. Consequently, you do not have to have a
1893 ``do_install`` task as part of your recipe. You just need to make
1894 sure the install portion of the build completes with no issues.
1895 However, if you wish to install additional files not already being
1896 installed by ``make install``, you should do this using a
1897 ``do_install_append`` function using the install command as described
1898 in the "Manual" bulleted item later in this list.
1899
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001900- *Other (using* ``make install``\ *)*: You need to define a ``do_install``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001901 function in your recipe. The function should call
1902 ``oe_runmake install`` and will likely need to pass in the
1903 destination directory as well. How you pass that path is dependent on
1904 how the ``Makefile`` being run is written (e.g. ``DESTDIR=${D}``,
1905 ``PREFIX=${D}``, ``INSTALLROOT=${D}``, and so forth).
1906
1907 For an example recipe using ``make install``, see the
1908 "`Makefile-Based Package <#new-recipe-makefile-based-package>`__"
1909 section.
1910
1911- *Manual:* You need to define a ``do_install`` function in your
1912 recipe. The function must first use ``install -d`` to create the
1913 directories under
1914 ``${``\ :term:`D`\ ``}``. Once the
1915 directories exist, your function can use ``install`` to manually
1916 install the built software into the directories.
1917
1918 You can find more information on ``install`` at
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001919 https://www.gnu.org/software/coreutils/manual/html_node/install-invocation.html.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001920
1921For the scenarios that do not use Autotools or CMake, you need to track
1922the installation and diagnose and fix any issues until everything
1923installs correctly. You need to look in the default location of
1924``${D}``, which is ``${WORKDIR}/image``, to be sure your files have been
1925installed correctly.
1926
1927.. note::
1928
1929 - During the installation process, you might need to modify some of
1930 the installed files to suit the target layout. For example, you
1931 might need to replace hard-coded paths in an initscript with
1932 values of variables provided by the build system, such as
1933 replacing ``/usr/bin/`` with ``${bindir}``. If you do perform such
1934 modifications during ``do_install``, be sure to modify the
1935 destination file after copying rather than before copying.
1936 Modifying after copying ensures that the build system can
1937 re-execute ``do_install`` if needed.
1938
1939 - ``oe_runmake install``, which can be run directly or can be run
1940 indirectly by the
1941 :ref:`autotools <ref-classes-autotools>` and
1942 :ref:`cmake <ref-classes-cmake>` classes,
1943 runs ``make install`` in parallel. Sometimes, a Makefile can have
1944 missing dependencies between targets that can result in race
1945 conditions. If you experience intermittent failures during
1946 ``do_install``, you might be able to work around them by disabling
1947 parallel Makefile installs by adding the following to the recipe:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001948 ::
1949
1950 PARALLEL_MAKEINST = ""
1951
1952 See :term:`PARALLEL_MAKEINST` for additional information.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001953
1954 - If you need to install one or more custom CMake toolchain files
1955 that are supplied by the application you are building, install the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001956 files to ``${D}${datadir}/cmake/Modules`` during
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001957 :ref:`ref-tasks-install`.
1958
1959.. _new-recipe-enabling-system-services:
1960
1961Enabling System Services
1962------------------------
1963
1964If you want to install a service, which is a process that usually starts
1965on boot and runs in the background, then you must include some
1966additional definitions in your recipe.
1967
1968If you are adding services and the service initialization script or the
1969service file itself is not installed, you must provide for that
1970installation in your recipe using a ``do_install_append`` function. If
1971your recipe already has a ``do_install`` function, update the function
1972near its end rather than adding an additional ``do_install_append``
1973function.
1974
1975When you create the installation for your services, you need to
1976accomplish what is normally done by ``make install``. In other words,
1977make sure your installation arranges the output similar to how it is
1978arranged on the target system.
1979
1980The OpenEmbedded build system provides support for starting services two
1981different ways:
1982
1983- *SysVinit:* SysVinit is a system and service manager that manages the
1984 init system used to control the very basic functions of your system.
1985 The init program is the first program started by the Linux kernel
1986 when the system boots. Init then controls the startup, running and
1987 shutdown of all other programs.
1988
1989 To enable a service using SysVinit, your recipe needs to inherit the
1990 :ref:`update-rc.d <ref-classes-update-rc.d>`
1991 class. The class helps facilitate safely installing the package on
1992 the target.
1993
1994 You will need to set the
1995 :term:`INITSCRIPT_PACKAGES`,
1996 :term:`INITSCRIPT_NAME`,
1997 and
1998 :term:`INITSCRIPT_PARAMS`
1999 variables within your recipe.
2000
2001- *systemd:* System Management Daemon (systemd) was designed to replace
2002 SysVinit and to provide enhanced management of services. For more
2003 information on systemd, see the systemd homepage at
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002004 https://freedesktop.org/wiki/Software/systemd/.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002005
2006 To enable a service using systemd, your recipe needs to inherit the
2007 :ref:`systemd <ref-classes-systemd>` class. See
2008 the ``systemd.bbclass`` file located in your :term:`Source Directory`
2009 section for
2010 more information.
2011
2012.. _new-recipe-packaging:
2013
2014Packaging
2015---------
2016
2017Successful packaging is a combination of automated processes performed
2018by the OpenEmbedded build system and some specific steps you need to
2019take. The following list describes the process:
2020
2021- *Splitting Files*: The ``do_package`` task splits the files produced
2022 by the recipe into logical components. Even software that produces a
2023 single binary might still have debug symbols, documentation, and
2024 other logical components that should be split out. The ``do_package``
2025 task ensures that files are split up and packaged correctly.
2026
2027- *Running QA Checks*: The
2028 :ref:`insane <ref-classes-insane>` class adds a
2029 step to the package generation process so that output quality
2030 assurance checks are generated by the OpenEmbedded build system. This
2031 step performs a range of checks to be sure the build's output is free
2032 of common problems that show up during runtime. For information on
2033 these checks, see the
2034 :ref:`insane <ref-classes-insane>` class and
2035 the ":ref:`ref-manual/ref-qa-checks:qa error and warning messages`"
2036 chapter in the Yocto Project Reference Manual.
2037
2038- *Hand-Checking Your Packages*: After you build your software, you
2039 need to be sure your packages are correct. Examine the
2040 ``${``\ :term:`WORKDIR`\ ``}/packages-split``
2041 directory and make sure files are where you expect them to be. If you
2042 discover problems, you can set
2043 :term:`PACKAGES`,
2044 :term:`FILES`,
2045 ``do_install(_append)``, and so forth as needed.
2046
2047- *Splitting an Application into Multiple Packages*: If you need to
2048 split an application into several packages, see the "`Splitting an
2049 Application into Multiple
2050 Packages <#splitting-an-application-into-multiple-packages>`__"
2051 section for an example.
2052
2053- *Installing a Post-Installation Script*: For an example showing how
2054 to install a post-installation script, see the "`Post-Installation
2055 Scripts <#new-recipe-post-installation-scripts>`__" section.
2056
2057- *Marking Package Architecture*: Depending on what your recipe is
2058 building and how it is configured, it might be important to mark the
2059 packages produced as being specific to a particular machine, or to
2060 mark them as not being specific to a particular machine or
2061 architecture at all.
2062
2063 By default, packages apply to any machine with the same architecture
2064 as the target machine. When a recipe produces packages that are
2065 machine-specific (e.g. the
2066 :term:`MACHINE` value is passed
2067 into the configure script or a patch is applied only for a particular
2068 machine), you should mark them as such by adding the following to the
2069 recipe:
2070 ::
2071
2072 PACKAGE_ARCH = "${MACHINE_ARCH}"
2073
2074 On the other hand, if the recipe produces packages that do not
2075 contain anything specific to the target machine or architecture at
2076 all (e.g. recipes that simply package script files or configuration
2077 files), you should use the
2078 :ref:`allarch <ref-classes-allarch>` class to
2079 do this for you by adding this to your recipe:
2080 ::
2081
2082 inherit allarch
2083
2084 Ensuring that the package architecture is correct is not critical
2085 while you are doing the first few builds of your recipe. However, it
2086 is important in order to ensure that your recipe rebuilds (or does
2087 not rebuild) appropriately in response to changes in configuration,
2088 and to ensure that you get the appropriate packages installed on the
2089 target machine, particularly if you run separate builds for more than
2090 one target machine.
2091
2092.. _new-sharing-files-between-recipes:
2093
2094Sharing Files Between Recipes
2095-----------------------------
2096
2097Recipes often need to use files provided by other recipes on the build
2098host. For example, an application linking to a common library needs
2099access to the library itself and its associated headers. The way this
2100access is accomplished is by populating a sysroot with files. Each
2101recipe has two sysroots in its work directory, one for target files
2102(``recipe-sysroot``) and one for files that are native to the build host
2103(``recipe-sysroot-native``).
2104
2105.. note::
2106
2107 You could find the term "staging" used within the Yocto project
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002108 regarding files populating sysroots (e.g. the :term:`STAGING_DIR`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002109 variable).
2110
2111Recipes should never populate the sysroot directly (i.e. write files
2112into sysroot). Instead, files should be installed into standard
2113locations during the
2114:ref:`ref-tasks-install` task within
2115the ``${``\ :term:`D`\ ``}`` directory. The
2116reason for this limitation is that almost all files that populate the
2117sysroot are cataloged in manifests in order to ensure the files can be
2118removed later when a recipe is either modified or removed. Thus, the
2119sysroot is able to remain free from stale files.
2120
2121A subset of the files installed by the
2122:ref:`ref-tasks-install` task are
2123used by the
2124:ref:`ref-tasks-populate_sysroot`
2125task as defined by the the
2126:term:`SYSROOT_DIRS` variable to
2127automatically populate the sysroot. It is possible to modify the list of
2128directories that populate the sysroot. The following example shows how
2129you could add the ``/opt`` directory to the list of directories within a
2130recipe:
2131::
2132
2133 SYSROOT_DIRS += "/opt"
2134
2135For a more complete description of the
2136:ref:`ref-tasks-populate_sysroot`
2137task and its associated functions, see the
2138:ref:`staging <ref-classes-staging>` class.
2139
2140.. _metadata-virtual-providers:
2141
2142Using Virtual Providers
2143-----------------------
2144
2145Prior to a build, if you know that several different recipes provide the
2146same functionality, you can use a virtual provider (i.e. ``virtual/*``)
2147as a placeholder for the actual provider. The actual provider is
2148determined at build-time.
2149
2150A common scenario where a virtual provider is used would be for the
2151kernel recipe. Suppose you have three kernel recipes whose
2152:term:`PN` values map to ``kernel-big``,
2153``kernel-mid``, and ``kernel-small``. Furthermore, each of these recipes
2154in some way uses a :term:`PROVIDES`
2155statement that essentially identifies itself as being able to provide
2156``virtual/kernel``. Here is one way through the
2157:ref:`kernel <ref-classes-kernel>` class:
2158::
2159
2160 PROVIDES += "${@ "virtual/kernel" if (d.getVar("KERNEL_PACKAGE_NAME") == "kernel") else "" }"
2161
2162Any recipe that inherits the ``kernel`` class is
2163going to utilize a ``PROVIDES`` statement that identifies that recipe as
2164being able to provide the ``virtual/kernel`` item.
2165
2166Now comes the time to actually build an image and you need a kernel
2167recipe, but which one? You can configure your build to call out the
2168kernel recipe you want by using the
2169:term:`PREFERRED_PROVIDER`
2170variable. As an example, consider the
2171`x86-base.inc <https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/conf/machine/include/x86-base.inc>`_
2172include file, which is a machine (i.e.
2173:term:`MACHINE`) configuration file.
2174This include file is the reason all x86-based machines use the
2175``linux-yocto`` kernel. Here are the relevant lines from the include
2176file:
2177::
2178
2179 PREFERRED_PROVIDER_virtual/kernel ??= "linux-yocto"
2180 PREFERRED_VERSION_linux-yocto ??= "4.15%"
2181
2182When you use a virtual provider, you do not have to "hard code" a recipe
2183name as a build dependency. You can use the
2184:term:`DEPENDS` variable to state the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002185build is dependent on ``virtual/kernel`` for example:
2186::
2187
2188 DEPENDS = "virtual/kernel"
2189
2190During the build, the OpenEmbedded build system picks
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002191the correct recipe needed for the ``virtual/kernel`` dependency based on
2192the ``PREFERRED_PROVIDER`` variable. If you want to use the small kernel
2193mentioned at the beginning of this section, configure your build as
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002194follows:
2195::
2196
2197 PREFERRED_PROVIDER_virtual/kernel ??= "kernel-small"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002198
2199.. note::
2200
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002201 Any recipe that ``PROVIDES`` a ``virtual/*`` item that is ultimately not
2202 selected through ``PREFERRED_PROVIDER`` does not get built. Preventing these
2203 recipes from building is usually the desired behavior since this mechanism's
2204 purpose is to select between mutually exclusive alternative providers.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002205
2206The following lists specific examples of virtual providers:
2207
2208- ``virtual/kernel``: Provides the name of the kernel recipe to use
2209 when building a kernel image.
2210
2211- ``virtual/bootloader``: Provides the name of the bootloader to use
2212 when building an image.
2213
2214- ``virtual/libgbm``: Provides ``gbm.pc``.
2215
2216- ``virtual/egl``: Provides ``egl.pc`` and possibly ``wayland-egl.pc``.
2217
2218- ``virtual/libgl``: Provides ``gl.pc`` (i.e. libGL).
2219
2220- ``virtual/libgles1``: Provides ``glesv1_cm.pc`` (i.e. libGLESv1_CM).
2221
2222- ``virtual/libgles2``: Provides ``glesv2.pc`` (i.e. libGLESv2).
2223
2224.. note::
2225
2226 Virtual providers only apply to build time dependencies specified with
2227 :term:`PROVIDES` and :term:`DEPENDS`. They do not apply to runtime
2228 dependencies specified with :term:`RPROVIDES` and :term:`RDEPENDS`.
2229
2230Properly Versioning Pre-Release Recipes
2231---------------------------------------
2232
2233Sometimes the name of a recipe can lead to versioning problems when the
2234recipe is upgraded to a final release. For example, consider the
2235``irssi_0.8.16-rc1.bb`` recipe file in the list of example recipes in
2236the "`Storing and Naming the
2237Recipe <#new-recipe-storing-and-naming-the-recipe>`__" section. This
2238recipe is at a release candidate stage (i.e. "rc1"). When the recipe is
2239released, the recipe filename becomes ``irssi_0.8.16.bb``. The version
2240change from ``0.8.16-rc1`` to ``0.8.16`` is seen as a decrease by the
2241build system and package managers, so the resulting packages will not
2242correctly trigger an upgrade.
2243
2244In order to ensure the versions compare properly, the recommended
2245convention is to set :term:`PV` within the
2246recipe to "previous_version+current_version". You can use an additional
2247variable so that you can use the current version elsewhere. Here is an
2248example:
2249::
2250
2251 REALPV = "0.8.16-rc1"
2252 PV = "0.8.15+${REALPV}"
2253
2254.. _new-recipe-post-installation-scripts:
2255
2256Post-Installation Scripts
2257-------------------------
2258
2259Post-installation scripts run immediately after installing a package on
2260the target or during image creation when a package is included in an
2261image. To add a post-installation script to a package, add a
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002262``pkg_postinst_``\ `PACKAGENAME`\ ``()`` function to the recipe file
2263(``.bb``) and replace `PACKAGENAME` with the name of the package you want
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002264to attach to the ``postinst`` script. To apply the post-installation
2265script to the main package for the recipe, which is usually what is
2266required, specify
2267``${``\ :term:`PN`\ ``}`` in place of
2268PACKAGENAME.
2269
2270A post-installation function has the following structure:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002271::
2272
2273 pkg_postinst_PACKAGENAME() {
2274 # Commands to carry out
2275 }
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002276
2277The script defined in the post-installation function is called when the
2278root filesystem is created. If the script succeeds, the package is
2279marked as installed.
2280
2281.. note::
2282
2283 Any RPM post-installation script that runs on the target should
2284 return a 0 exit code. RPM does not allow non-zero exit codes for
2285 these scripts, and the RPM package manager will cause the package to
2286 fail installation on the target.
2287
2288Sometimes it is necessary for the execution of a post-installation
2289script to be delayed until the first boot. For example, the script might
2290need to be executed on the device itself. To delay script execution
2291until boot time, you must explicitly mark post installs to defer to the
2292target. You can use ``pkg_postinst_ontarget()`` or call
2293``postinst_intercept delay_to_first_boot`` from ``pkg_postinst()``. Any
2294failure of a ``pkg_postinst()`` script (including exit 1) triggers an
2295error during the
2296:ref:`ref-tasks-rootfs` task.
2297
2298If you have recipes that use ``pkg_postinst`` function and they require
2299the use of non-standard native tools that have dependencies during
2300rootfs construction, you need to use the
2301:term:`PACKAGE_WRITE_DEPS`
2302variable in your recipe to list these tools. If you do not use this
2303variable, the tools might be missing and execution of the
2304post-installation script is deferred until first boot. Deferring the
2305script to first boot is undesirable and for read-only rootfs impossible.
2306
2307.. note::
2308
2309 Equivalent support for pre-install, pre-uninstall, and post-uninstall
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002310 scripts exist by way of ``pkg_preinst``, ``pkg_prerm``, and ``pkg_postrm``,
2311 respectively. These scrips work in exactly the same way as does
2312 ``pkg_postinst`` with the exception that they run at different times. Also,
2313 because of when they run, they are not applicable to being run at image
2314 creation time like ``pkg_postinst``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002315
2316.. _new-recipe-testing:
2317
2318Testing
2319-------
2320
2321The final step for completing your recipe is to be sure that the
2322software you built runs correctly. To accomplish runtime testing, add
2323the build's output packages to your image and test them on the target.
2324
2325For information on how to customize your image by adding specific
2326packages, see the "`Customizing
2327Images <#usingpoky-extend-customimage>`__" section.
2328
2329.. _new-recipe-testing-examples:
2330
2331Examples
2332--------
2333
2334To help summarize how to write a recipe, this section provides some
2335examples given various scenarios:
2336
2337- Recipes that use local files
2338
2339- Using an Autotooled package
2340
2341- Using a Makefile-based package
2342
2343- Splitting an application into multiple packages
2344
2345- Adding binaries to an image
2346
2347.. _new-recipe-single-c-file-package-hello-world:
2348
2349Single .c File Package (Hello World!)
2350~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2351
2352Building an application from a single file that is stored locally (e.g.
2353under ``files``) requires a recipe that has the file listed in the
2354``SRC_URI`` variable. Additionally, you need to manually write the
2355``do_compile`` and ``do_install`` tasks. The ``S`` variable defines the
2356directory containing the source code, which is set to
2357:term:`WORKDIR` in this case - the
2358directory BitBake uses for the build.
2359::
2360
2361 SUMMARY = "Simple helloworld application"
2362 SECTION = "examples"
2363 LICENSE = "MIT"
2364 LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
2365
2366 SRC_URI = "file://helloworld.c"
2367
2368 S = "${WORKDIR}"
2369
2370 do_compile() {
2371 ${CC} helloworld.c -o helloworld
2372 }
2373
2374 do_install() {
2375 install -d ${D}${bindir}
2376 install -m 0755 helloworld ${D}${bindir}
2377 }
2378
2379By default, the ``helloworld``, ``helloworld-dbg``, and
2380``helloworld-dev`` packages are built. For information on how to
2381customize the packaging process, see the "`Splitting an Application into
2382Multiple Packages <#splitting-an-application-into-multiple-packages>`__"
2383section.
2384
2385.. _new-recipe-autotooled-package:
2386
2387Autotooled Package
2388~~~~~~~~~~~~~~~~~~
2389
2390Applications that use Autotools such as ``autoconf`` and ``automake``
2391require a recipe that has a source archive listed in ``SRC_URI`` and
2392also inherit the
2393:ref:`autotools <ref-classes-autotools>` class,
2394which contains the definitions of all the steps needed to build an
2395Autotool-based application. The result of the build is automatically
2396packaged. And, if the application uses NLS for localization, packages
2397with local information are generated (one package per language).
2398Following is one example: (``hello_2.3.bb``)
2399::
2400
2401 SUMMARY = "GNU Helloworld application"
2402 SECTION = "examples"
2403 LICENSE = "GPLv2+"
2404 LIC_FILES_CHKSUM = "file://COPYING;md5=751419260aa954499f7abaabaa882bbe"
2405
2406 SRC_URI = "${GNU_MIRROR}/hello/hello-${PV}.tar.gz"
2407
2408 inherit autotools gettext
2409
2410The variable ``LIC_FILES_CHKSUM`` is used to track source license
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002411changes as described in the
2412":ref:`dev-manual/dev-manual-common-tasks:tracking license changes`" section in
2413the Yocto Project Overview and Concepts Manual. You can quickly create
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002414Autotool-based recipes in a manner similar to the previous example.
2415
2416.. _new-recipe-makefile-based-package:
2417
2418Makefile-Based Package
2419~~~~~~~~~~~~~~~~~~~~~~
2420
2421Applications that use GNU ``make`` also require a recipe that has the
2422source archive listed in ``SRC_URI``. You do not need to add a
2423``do_compile`` step since by default BitBake starts the ``make`` command
2424to compile the application. If you need additional ``make`` options, you
2425should store them in the
2426:term:`EXTRA_OEMAKE` or
2427:term:`PACKAGECONFIG_CONFARGS`
2428variables. BitBake passes these options into the GNU ``make``
2429invocation. Note that a ``do_install`` task is still required.
2430Otherwise, BitBake runs an empty ``do_install`` task by default.
2431
2432Some applications might require extra parameters to be passed to the
2433compiler. For example, the application might need an additional header
2434path. You can accomplish this by adding to the ``CFLAGS`` variable. The
2435following example shows this:
2436::
2437
2438 CFLAGS_prepend = "-I ${S}/include "
2439
2440In the following example, ``mtd-utils`` is a makefile-based package:
2441::
2442
2443 SUMMARY = "Tools for managing memory technology devices"
2444 SECTION = "base"
2445 DEPENDS = "zlib lzo e2fsprogs util-linux"
2446 HOMEPAGE = "http://www.linux-mtd.infradead.org/"
2447 LICENSE = "GPLv2+"
2448 LIC_FILES_CHKSUM = "file://COPYING;md5=0636e73ff0215e8d672dc4c32c317bb3 \
2449 file://include/common.h;beginline=1;endline=17;md5=ba05b07912a44ea2bf81ce409380049c"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002450
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002451 # Use the latest version at 26 Oct, 2013
2452 SRCREV = "9f107132a6a073cce37434ca9cda6917dd8d866b"
2453 SRC_URI = "git://git.infradead.org/mtd-utils.git \
2454 file://add-exclusion-to-mkfs-jffs2-git-2.patch \
2455 "
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002456
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002457 PV = "1.5.1+git${SRCPV}"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002458
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002459 S = "${WORKDIR}/git"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002460
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002461 EXTRA_OEMAKE = "'CC=${CC}' 'RANLIB=${RANLIB}' 'AR=${AR}' 'CFLAGS=${CFLAGS} -I${S}/include -DWITHOUT_XATTR' 'BUILDDIR=${S}'"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002462
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002463 do_install () {
2464 oe_runmake install DESTDIR=${D} SBINDIR=${sbindir} MANDIR=${mandir} INCLUDEDIR=${includedir}
2465 }
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002466
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002467 PACKAGES =+ "mtd-utils-jffs2 mtd-utils-ubifs mtd-utils-misc"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002468
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002469 FILES_mtd-utils-jffs2 = "${sbindir}/mkfs.jffs2 ${sbindir}/jffs2dump ${sbindir}/jffs2reader ${sbindir}/sumtool"
2470 FILES_mtd-utils-ubifs = "${sbindir}/mkfs.ubifs ${sbindir}/ubi*"
2471 FILES_mtd-utils-misc = "${sbindir}/nftl* ${sbindir}/ftl* ${sbindir}/rfd* ${sbindir}/doc* ${sbindir}/serve_image ${sbindir}/recv_image"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002472
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002473 PARALLEL_MAKE = ""
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002474
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002475 BBCLASSEXTEND = "native"
2476
2477Splitting an Application into Multiple Packages
2478~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2479
2480You can use the variables ``PACKAGES`` and ``FILES`` to split an
2481application into multiple packages.
2482
2483Following is an example that uses the ``libxpm`` recipe. By default,
2484this recipe generates a single package that contains the library along
2485with a few binaries. You can modify the recipe to split the binaries
2486into separate packages:
2487::
2488
2489 require xorg-lib-common.inc
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002490
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002491 SUMMARY = "Xpm: X Pixmap extension library"
2492 LICENSE = "BSD"
2493 LIC_FILES_CHKSUM = "file://COPYING;md5=51f4270b012ecd4ab1a164f5f4ed6cf7"
2494 DEPENDS += "libxext libsm libxt"
2495 PE = "1"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002496
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002497 XORG_PN = "libXpm"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002498
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002499 PACKAGES =+ "sxpm cxpm"
2500 FILES_cxpm = "${bindir}/cxpm"
2501 FILES_sxpm = "${bindir}/sxpm"
2502
2503In the previous example, we want to ship the ``sxpm`` and ``cxpm``
2504binaries in separate packages. Since ``bindir`` would be packaged into
2505the main ``PN`` package by default, we prepend the ``PACKAGES`` variable
2506so additional package names are added to the start of list. This results
2507in the extra ``FILES_*`` variables then containing information that
2508define which files and directories go into which packages. Files
2509included by earlier packages are skipped by latter packages. Thus, the
2510main ``PN`` package does not include the above listed files.
2511
2512Packaging Externally Produced Binaries
2513~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2514
2515Sometimes, you need to add pre-compiled binaries to an image. For
2516example, suppose that binaries for proprietary code exist, which are
2517created by a particular division of a company. Your part of the company
2518needs to use those binaries as part of an image that you are building
2519using the OpenEmbedded build system. Since you only have the binaries
2520and not the source code, you cannot use a typical recipe that expects to
2521fetch the source specified in
2522:term:`SRC_URI` and then compile it.
2523
2524One method is to package the binaries and then install them as part of
2525the image. Generally, it is not a good idea to package binaries since,
2526among other things, it can hinder the ability to reproduce builds and
2527could lead to compatibility problems with ABI in the future. However,
2528sometimes you have no choice.
2529
2530The easiest solution is to create a recipe that uses the
2531:ref:`bin_package <ref-classes-bin-package>` class
2532and to be sure that you are using default locations for build artifacts.
2533In most cases, the ``bin_package`` class handles "skipping" the
2534configure and compile steps as well as sets things up to grab packages
2535from the appropriate area. In particular, this class sets ``noexec`` on
2536both the :ref:`ref-tasks-configure`
2537and :ref:`ref-tasks-compile` tasks,
2538sets ``FILES_${PN}`` to "/" so that it picks up all files, and sets up a
2539:ref:`ref-tasks-install` task, which
2540effectively copies all files from ``${S}`` to ``${D}``. The
2541``bin_package`` class works well when the files extracted into ``${S}``
2542are already laid out in the way they should be laid out on the target.
2543For more information on these variables, see the
2544:term:`FILES`,
2545:term:`PN`,
2546:term:`S`, and
2547:term:`D` variables in the Yocto Project
2548Reference Manual's variable glossary.
2549
2550.. note::
2551
2552 - Using :term:`DEPENDS` is a good
2553 idea even for components distributed in binary form, and is often
2554 necessary for shared libraries. For a shared library, listing the
2555 library dependencies in ``DEPENDS`` makes sure that the libraries
2556 are available in the staging sysroot when other recipes link
2557 against the library, which might be necessary for successful
2558 linking.
2559
2560 - Using ``DEPENDS`` also allows runtime dependencies between
2561 packages to be added automatically. See the
2562 ":ref:`overview-manual/overview-manual-concepts:automatically added runtime dependencies`"
2563 section in the Yocto Project Overview and Concepts Manual for more
2564 information.
2565
2566If you cannot use the ``bin_package`` class, you need to be sure you are
2567doing the following:
2568
2569- Create a recipe where the
2570 :ref:`ref-tasks-configure` and
2571 :ref:`ref-tasks-compile` tasks do
2572 nothing: It is usually sufficient to just not define these tasks in
2573 the recipe, because the default implementations do nothing unless a
2574 Makefile is found in
2575 ``${``\ :term:`S`\ ``}``.
2576
2577 If ``${S}`` might contain a Makefile, or if you inherit some class
2578 that replaces ``do_configure`` and ``do_compile`` with custom
2579 versions, then you can use the
2580 ``[``\ :ref:`noexec <bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ``]``
2581 flag to turn the tasks into no-ops, as follows:
2582 ::
2583
2584 do_configure[noexec] = "1"
2585 do_compile[noexec] = "1"
2586
2587 Unlike
2588 :ref:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata:deleting a task`,
2589 using the flag preserves the dependency chain from the
2590 :ref:`ref-tasks-fetch`,
2591 :ref:`ref-tasks-unpack`, and
2592 :ref:`ref-tasks-patch` tasks to the
2593 :ref:`ref-tasks-install` task.
2594
2595- Make sure your ``do_install`` task installs the binaries
2596 appropriately.
2597
2598- Ensure that you set up :term:`FILES`
2599 (usually
2600 ``FILES_${``\ :term:`PN`\ ``}``) to
2601 point to the files you have installed, which of course depends on
2602 where you have installed them and whether those files are in
2603 different locations than the defaults.
2604
2605Following Recipe Style Guidelines
2606---------------------------------
2607
2608When writing recipes, it is good to conform to existing style
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002609guidelines. The :oe_home:`OpenEmbedded Styleguide </wiki/Styleguide>` wiki page
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002610provides rough guidelines for preferred recipe style.
2611
2612It is common for existing recipes to deviate a bit from this style.
2613However, aiming for at least a consistent style is a good idea. Some
2614practices, such as omitting spaces around ``=`` operators in assignments
2615or ordering recipe components in an erratic way, are widely seen as poor
2616style.
2617
2618Recipe Syntax
2619-------------
2620
2621Understanding recipe file syntax is important for writing recipes. The
2622following list overviews the basic items that make up a BitBake recipe
2623file. For more complete BitBake syntax descriptions, see the
2624":doc:`bitbake-user-manual/bitbake-user-manual-metadata`"
2625chapter of the BitBake User Manual.
2626
2627- *Variable Assignments and Manipulations:* Variable assignments allow
2628 a value to be assigned to a variable. The assignment can be static
2629 text or might include the contents of other variables. In addition to
2630 the assignment, appending and prepending operations are also
2631 supported.
2632
2633 The following example shows some of the ways you can use variables in
2634 recipes:
2635 ::
2636
2637 S = "${WORKDIR}/postfix-${PV}"
2638 CFLAGS += "-DNO_ASM"
2639 SRC_URI_append = " file://fixup.patch"
2640
2641- *Functions:* Functions provide a series of actions to be performed.
2642 You usually use functions to override the default implementation of a
2643 task function or to complement a default function (i.e. append or
2644 prepend to an existing function). Standard functions use ``sh`` shell
2645 syntax, although access to OpenEmbedded variables and internal
2646 methods are also available.
2647
2648 The following is an example function from the ``sed`` recipe:
2649 ::
2650
2651 do_install () {
2652 autotools_do_install
2653 install -d ${D}${base_bindir}
2654 mv ${D}${bindir}/sed ${D}${base_bindir}/sed
2655 rmdir ${D}${bindir}/
2656 }
2657
2658 It is
2659 also possible to implement new functions that are called between
2660 existing tasks as long as the new functions are not replacing or
2661 complementing the default functions. You can implement functions in
2662 Python instead of shell. Both of these options are not seen in the
2663 majority of recipes.
2664
2665- *Keywords:* BitBake recipes use only a few keywords. You use keywords
2666 to include common functions (``inherit``), load parts of a recipe
2667 from other files (``include`` and ``require``) and export variables
2668 to the environment (``export``).
2669
2670 The following example shows the use of some of these keywords:
2671 ::
2672
2673 export POSTCONF = "${STAGING_BINDIR}/postconf"
2674 inherit autoconf
2675 require otherfile.inc
2676
2677- *Comments (#):* Any lines that begin with the hash character (``#``)
2678 are treated as comment lines and are ignored:
2679 ::
2680
2681 # This is a comment
2682
2683This next list summarizes the most important and most commonly used
2684parts of the recipe syntax. For more information on these parts of the
2685syntax, you can reference the
2686:doc:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata` chapter
2687in the BitBake User Manual.
2688
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002689- *Line Continuation (\\):* Use the backward slash (``\``) character to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002690 split a statement over multiple lines. Place the slash character at
2691 the end of the line that is to be continued on the next line:
2692 ::
2693
2694 VAR = "A really long \
2695 line"
2696
2697 .. note::
2698
2699 You cannot have any characters including spaces or tabs after the
2700 slash character.
2701
2702- *Using Variables (${VARNAME}):* Use the ``${VARNAME}`` syntax to
2703 access the contents of a variable:
2704 ::
2705
2706 SRC_URI = "${SOURCEFORGE_MIRROR}/libpng/zlib-${PV}.tar.gz"
2707
2708 .. note::
2709
2710 It is important to understand that the value of a variable
2711 expressed in this form does not get substituted automatically. The
2712 expansion of these expressions happens on-demand later (e.g.
2713 usually when a function that makes reference to the variable
2714 executes). This behavior ensures that the values are most
2715 appropriate for the context in which they are finally used. On the
2716 rare occasion that you do need the variable expression to be
2717 expanded immediately, you can use the
2718 :=
2719 operator instead of
2720 =
2721 when you make the assignment, but this is not generally needed.
2722
2723- *Quote All Assignments ("value"):* Use double quotes around values in
2724 all variable assignments (e.g. ``"value"``). Following is an example:
2725 ::
2726
2727 VAR1 = "${OTHERVAR}"
2728 VAR2 = "The version is ${PV}"
2729
2730- *Conditional Assignment (?=):* Conditional assignment is used to
2731 assign a value to a variable, but only when the variable is currently
2732 unset. Use the question mark followed by the equal sign (``?=``) to
2733 make a "soft" assignment used for conditional assignment. Typically,
2734 "soft" assignments are used in the ``local.conf`` file for variables
2735 that are allowed to come through from the external environment.
2736
2737 Here is an example where ``VAR1`` is set to "New value" if it is
2738 currently empty. However, if ``VAR1`` has already been set, it
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002739 remains unchanged:
2740 ::
2741
2742 VAR1 ?= "New value"
2743
2744 In this next example, ``VAR1`` is left with the value "Original value":
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002745 ::
2746
2747 VAR1 = "Original value"
2748 VAR1 ?= "New value"
2749
2750- *Appending (+=):* Use the plus character followed by the equals sign
2751 (``+=``) to append values to existing variables.
2752
2753 .. note::
2754
2755 This operator adds a space between the existing content of the
2756 variable and the new content.
2757
2758 Here is an example:
2759 ::
2760
2761 SRC_URI += "file://fix-makefile.patch"
2762
2763- *Prepending (=+):* Use the equals sign followed by the plus character
2764 (``=+``) to prepend values to existing variables.
2765
2766 .. note::
2767
2768 This operator adds a space between the new content and the
2769 existing content of the variable.
2770
2771 Here is an example:
2772 ::
2773
2774 VAR =+ "Starts"
2775
2776- *Appending (_append):* Use the ``_append`` operator to append values
2777 to existing variables. This operator does not add any additional
2778 space. Also, the operator is applied after all the ``+=``, and ``=+``
2779 operators have been applied and after all ``=`` assignments have
2780 occurred.
2781
2782 The following example shows the space being explicitly added to the
2783 start to ensure the appended value is not merged with the existing
2784 value:
2785 ::
2786
2787 SRC_URI_append = " file://fix-makefile.patch"
2788
2789 You can also use
2790 the ``_append`` operator with overrides, which results in the actions
2791 only being performed for the specified target or machine:
2792 ::
2793
2794 SRC_URI_append_sh4 = " file://fix-makefile.patch"
2795
2796- *Prepending (_prepend):* Use the ``_prepend`` operator to prepend
2797 values to existing variables. This operator does not add any
2798 additional space. Also, the operator is applied after all the ``+=``,
2799 and ``=+`` operators have been applied and after all ``=``
2800 assignments have occurred.
2801
2802 The following example shows the space being explicitly added to the
2803 end to ensure the prepended value is not merged with the existing
2804 value:
2805 ::
2806
2807 CFLAGS_prepend = "-I${S}/myincludes "
2808
2809 You can also use the
2810 ``_prepend`` operator with overrides, which results in the actions
2811 only being performed for the specified target or machine:
2812 ::
2813
2814 CFLAGS_prepend_sh4 = "-I${S}/myincludes "
2815
2816- *Overrides:* You can use overrides to set a value conditionally,
2817 typically based on how the recipe is being built. For example, to set
2818 the :term:`KBRANCH` variable's
2819 value to "standard/base" for any target
2820 :term:`MACHINE`, except for
2821 qemuarm where it should be set to "standard/arm-versatile-926ejs",
2822 you would do the following:
2823 ::
2824
2825 KBRANCH = "standard/base"
2826 KBRANCH_qemuarm = "standard/arm-versatile-926ejs"
2827
2828 Overrides are also used to separate
2829 alternate values of a variable in other situations. For example, when
2830 setting variables such as
2831 :term:`FILES` and
2832 :term:`RDEPENDS` that are
2833 specific to individual packages produced by a recipe, you should
2834 always use an override that specifies the name of the package.
2835
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002836- *Indentation:* Use spaces for indentation rather than tabs. For
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002837 shell functions, both currently work. However, it is a policy
2838 decision of the Yocto Project to use tabs in shell functions. Realize
2839 that some layers have a policy to use spaces for all indentation.
2840
2841- *Using Python for Complex Operations:* For more advanced processing,
2842 it is possible to use Python code during variable assignments (e.g.
2843 search and replacement on a variable).
2844
2845 You indicate Python code using the ``${@python_code}`` syntax for the
2846 variable assignment:
2847 ::
2848
2849 SRC_URI = "ftp://ftp.info-zip.org/pub/infozip/src/zip${@d.getVar('PV',1).replace('.', '')}.tgz
2850
2851- *Shell Function Syntax:* Write shell functions as if you were writing
2852 a shell script when you describe a list of actions to take. You
2853 should ensure that your script works with a generic ``sh`` and that
2854 it does not require any ``bash`` or other shell-specific
2855 functionality. The same considerations apply to various system
2856 utilities (e.g. ``sed``, ``grep``, ``awk``, and so forth) that you
2857 might wish to use. If in doubt, you should check with multiple
2858 implementations - including those from BusyBox.
2859
2860.. _platdev-newmachine:
2861
2862Adding a New Machine
2863====================
2864
2865Adding a new machine to the Yocto Project is a straightforward process.
2866This section describes how to add machines that are similar to those
2867that the Yocto Project already supports.
2868
2869.. note::
2870
2871 Although well within the capabilities of the Yocto Project, adding a
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002872 totally new architecture might require changes to ``gcc``/``glibc``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002873 and to the site information, which is beyond the scope of this
2874 manual.
2875
2876For a complete example that shows how to add a new machine, see the
2877":ref:`bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script`"
2878section in the Yocto Project Board Support Package (BSP) Developer's
2879Guide.
2880
2881.. _platdev-newmachine-conffile:
2882
2883Adding the Machine Configuration File
2884-------------------------------------
2885
2886To add a new machine, you need to add a new machine configuration file
2887to the layer's ``conf/machine`` directory. This configuration file
2888provides details about the device you are adding.
2889
2890The OpenEmbedded build system uses the root name of the machine
2891configuration file to reference the new machine. For example, given a
2892machine configuration file named ``crownbay.conf``, the build system
2893recognizes the machine as "crownbay".
2894
2895The most important variables you must set in your machine configuration
2896file or include from a lower-level configuration file are as follows:
2897
2898- ``TARGET_ARCH`` (e.g. "arm")
2899
2900- ``PREFERRED_PROVIDER_virtual/kernel``
2901
2902- ``MACHINE_FEATURES`` (e.g. "apm screen wifi")
2903
2904You might also need these variables:
2905
2906- ``SERIAL_CONSOLES`` (e.g. "115200;ttyS0 115200;ttyS1")
2907
2908- ``KERNEL_IMAGETYPE`` (e.g. "zImage")
2909
2910- ``IMAGE_FSTYPES`` (e.g. "tar.gz jffs2")
2911
2912You can find full details on these variables in the reference section.
2913You can leverage existing machine ``.conf`` files from
2914``meta-yocto-bsp/conf/machine/``.
2915
2916.. _platdev-newmachine-kernel:
2917
2918Adding a Kernel for the Machine
2919-------------------------------
2920
2921The OpenEmbedded build system needs to be able to build a kernel for the
2922machine. You need to either create a new kernel recipe for this machine,
2923or extend an existing kernel recipe. You can find several kernel recipe
2924examples in the Source Directory at ``meta/recipes-kernel/linux`` that
2925you can use as references.
2926
2927If you are creating a new kernel recipe, normal recipe-writing rules
2928apply for setting up a ``SRC_URI``. Thus, you need to specify any
2929necessary patches and set ``S`` to point at the source code. You need to
2930create a ``do_configure`` task that configures the unpacked kernel with
2931a ``defconfig`` file. You can do this by using a ``make defconfig``
2932command or, more commonly, by copying in a suitable ``defconfig`` file
2933and then running ``make oldconfig``. By making use of ``inherit kernel``
2934and potentially some of the ``linux-*.inc`` files, most other
2935functionality is centralized and the defaults of the class normally work
2936well.
2937
2938If you are extending an existing kernel recipe, it is usually a matter
2939of adding a suitable ``defconfig`` file. The file needs to be added into
2940a location similar to ``defconfig`` files used for other machines in a
2941given kernel recipe. A possible way to do this is by listing the file in
2942the ``SRC_URI`` and adding the machine to the expression in
2943``COMPATIBLE_MACHINE``:
2944::
2945
2946 COMPATIBLE_MACHINE = '(qemux86|qemumips)'
2947
2948For more information on ``defconfig`` files, see the
2949":ref:`kernel-dev/kernel-dev-common:changing the configuration`"
2950section in the Yocto Project Linux Kernel Development Manual.
2951
2952.. _platdev-newmachine-formfactor:
2953
2954Adding a Formfactor Configuration File
2955--------------------------------------
2956
2957A formfactor configuration file provides information about the target
2958hardware for which the image is being built and information that the
2959build system cannot obtain from other sources such as the kernel. Some
2960examples of information contained in a formfactor configuration file
2961include framebuffer orientation, whether or not the system has a
2962keyboard, the positioning of the keyboard in relation to the screen, and
2963the screen resolution.
2964
2965The build system uses reasonable defaults in most cases. However, if
2966customization is necessary, you need to create a ``machconfig`` file in
2967the ``meta/recipes-bsp/formfactor/files`` directory. This directory
2968contains directories for specific machines such as ``qemuarm`` and
2969``qemux86``. For information about the settings available and the
2970defaults, see the ``meta/recipes-bsp/formfactor/files/config`` file
2971found in the same area.
2972
2973Following is an example for "qemuarm" machine:
2974::
2975
2976 HAVE_TOUCHSCREEN=1
2977 HAVE_KEYBOARD=1
2978 DISPLAY_CAN_ROTATE=0
2979 DISPLAY_ORIENTATION=0
2980 #DISPLAY_WIDTH_PIXELS=640
2981 #DISPLAY_HEIGHT_PIXELS=480
2982 #DISPLAY_BPP=16
2983 DISPLAY_DPI=150
2984 DISPLAY_SUBPIXEL_ORDER=vrgb
2985
2986.. _gs-upgrading-recipes:
2987
2988Upgrading Recipes
2989=================
2990
2991Over time, upstream developers publish new versions for software built
2992by layer recipes. It is recommended to keep recipes up-to-date with
2993upstream version releases.
2994
2995While several methods exist that allow you upgrade a recipe, you might
2996consider checking on the upgrade status of a recipe first. You can do so
2997using the ``devtool check-upgrade-status`` command. See the
2998":ref:`devtool-checking-on-the-upgrade-status-of-a-recipe`"
2999section in the Yocto Project Reference Manual for more information.
3000
3001The remainder of this section describes three ways you can upgrade a
3002recipe. You can use the Automated Upgrade Helper (AUH) to set up
3003automatic version upgrades. Alternatively, you can use
3004``devtool upgrade`` to set up semi-automatic version upgrades. Finally,
3005you can manually upgrade a recipe by editing the recipe itself.
3006
3007.. _gs-using-the-auto-upgrade-helper:
3008
3009Using the Auto Upgrade Helper (AUH)
3010-----------------------------------
3011
3012The AUH utility works in conjunction with the OpenEmbedded build system
3013in order to automatically generate upgrades for recipes based on new
3014versions being published upstream. Use AUH when you want to create a
3015service that performs the upgrades automatically and optionally sends
3016you an email with the results.
3017
3018AUH allows you to update several recipes with a single use. You can also
3019optionally perform build and integration tests using images with the
3020results saved to your hard drive and emails of results optionally sent
3021to recipe maintainers. Finally, AUH creates Git commits with appropriate
3022commit messages in the layer's tree for the changes made to recipes.
3023
3024.. note::
3025
3026 Conditions do exist when you should not use AUH to upgrade recipes
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003027 and you should instead use either ``devtool upgrade`` or upgrade your
3028 recipes manually:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003029
3030 - When AUH cannot complete the upgrade sequence. This situation
3031 usually results because custom patches carried by the recipe
3032 cannot be automatically rebased to the new version. In this case,
3033 ``devtool upgrade`` allows you to manually resolve conflicts.
3034
3035 - When for any reason you want fuller control over the upgrade
3036 process. For example, when you want special arrangements for
3037 testing.
3038
3039The following steps describe how to set up the AUH utility:
3040
30411. *Be Sure the Development Host is Set Up:* You need to be sure that
3042 your development host is set up to use the Yocto Project. For
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003043 information on how to set up your host, see the
3044 ":ref:`dev-preparing-the-build-host`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003045
30462. *Make Sure Git is Configured:* The AUH utility requires Git to be
3047 configured because AUH uses Git to save upgrades. Thus, you must have
3048 Git user and email configured. The following command shows your
3049 configurations:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003050 ::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003051
3052 $ git config --list
3053
3054 If you do not have the user and
3055 email configured, you can use the following commands to do so:
3056 ::
3057
3058 $ git config --global user.name some_name
3059 $ git config --global user.email username@domain.com
3060
30613. *Clone the AUH Repository:* To use AUH, you must clone the repository
3062 onto your development host. The following command uses Git to create
3063 a local copy of the repository on your system:
3064 ::
3065
3066 $ git clone git://git.yoctoproject.org/auto-upgrade-helper
3067 Cloning into 'auto-upgrade-helper'... remote: Counting objects: 768, done.
3068 remote: Compressing objects: 100% (300/300), done.
3069 remote: Total 768 (delta 499), reused 703 (delta 434)
3070 Receiving objects: 100% (768/768), 191.47 KiB | 98.00 KiB/s, done.
3071 Resolving deltas: 100% (499/499), done.
3072 Checking connectivity... done.
3073
3074 AUH is not part of the :term:`OpenEmbedded-Core (OE-Core)` or
3075 :term:`Poky` repositories.
3076
30774. *Create a Dedicated Build Directory:* Run the
3078 :ref:`structure-core-script`
3079 script to create a fresh build directory that you use exclusively for
3080 running the AUH utility:
3081 ::
3082
3083 $ cd ~/poky
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003084 $ source oe-init-build-env your_AUH_build_directory
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003085
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003086 Re-using an existing build directory and its configurations is not
3087 recommended as existing settings could cause AUH to fail or behave
3088 undesirably.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003089
30905. *Make Configurations in Your Local Configuration File:* Several
3091 settings need to exist in the ``local.conf`` file in the build
3092 directory you just created for AUH. Make these following
3093 configurations:
3094
3095 - If you want to enable :ref:`Build
3096 History <dev-manual/dev-manual-common-tasks:maintaining build output quality>`,
3097 which is optional, you need the following lines in the
3098 ``conf/local.conf`` file:
3099 ::
3100
3101 INHERIT =+ "buildhistory"
3102 BUILDHISTORY_COMMIT = "1"
3103
3104 With this configuration and a successful
3105 upgrade, a build history "diff" file appears in the
3106 ``upgrade-helper/work/recipe/buildhistory-diff.txt`` file found in
3107 your build directory.
3108
3109 - If you want to enable testing through the
3110 :ref:`testimage <ref-classes-testimage*>`
3111 class, which is optional, you need to have the following set in
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003112 your ``conf/local.conf`` file:
3113 ::
3114
3115 INHERIT += "testimage"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003116
3117 .. note::
3118
3119 If your distro does not enable by default ptest, which Poky
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003120 does, you need the following in your ``local.conf`` file:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003121 ::
3122
3123 DISTRO_FEATURES_append = " ptest"
3124
3125
31266. *Optionally Start a vncserver:* If you are running in a server
3127 without an X11 session, you need to start a vncserver:
3128 ::
3129
3130 $ vncserver :1
3131 $ export DISPLAY=:1
3132
31337. *Create and Edit an AUH Configuration File:* You need to have the
3134 ``upgrade-helper/upgrade-helper.conf`` configuration file in your
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003135 build directory. You can find a sample configuration file in the
3136 :yocto_git:`AUH source repository </cgit/cgit.cgi/auto-upgrade-helper/tree/>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003137
3138 Read through the sample file and make configurations as needed. For
3139 example, if you enabled build history in your ``local.conf`` as
3140 described earlier, you must enable it in ``upgrade-helper.conf``.
3141
3142 Also, if you are using the default ``maintainers.inc`` file supplied
3143 with Poky and located in ``meta-yocto`` and you do not set a
3144 "maintainers_whitelist" or "global_maintainer_override" in the
3145 ``upgrade-helper.conf`` configuration, and you specify "-e all" on
3146 the AUH command-line, the utility automatically sends out emails to
3147 all the default maintainers. Please avoid this.
3148
3149This next set of examples describes how to use the AUH:
3150
3151- *Upgrading a Specific Recipe:* To upgrade a specific recipe, use the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003152 following form:
3153 ::
3154
3155 $ upgrade-helper.py recipe_name
3156
3157 For example, this command upgrades the ``xmodmap`` recipe:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003158 ::
3159
3160 $ upgrade-helper.py xmodmap
3161
3162- *Upgrading a Specific Recipe to a Particular Version:* To upgrade a
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003163 specific recipe to a particular version, use the following form:
3164 ::
3165
3166 $ upgrade-helper.py recipe_name -t version
3167
3168 For example, this command upgrades the ``xmodmap`` recipe to version 1.2.3:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003169 ::
3170
3171 $ upgrade-helper.py xmodmap -t 1.2.3
3172
3173- *Upgrading all Recipes to the Latest Versions and Suppressing Email
3174 Notifications:* To upgrade all recipes to their most recent versions
3175 and suppress the email notifications, use the following command:
3176 ::
3177
3178 $ upgrade-helper.py all
3179
3180- *Upgrading all Recipes to the Latest Versions and Send Email
3181 Notifications:* To upgrade all recipes to their most recent versions
3182 and send email messages to maintainers for each attempted recipe as
3183 well as a status email, use the following command:
3184 ::
3185
3186 $ upgrade-helper.py -e all
3187
3188Once you have run the AUH utility, you can find the results in the AUH
3189build directory:
3190::
3191
3192 ${BUILDDIR}/upgrade-helper/timestamp
3193
3194The AUH utility
3195also creates recipe update commits from successful upgrade attempts in
3196the layer tree.
3197
3198You can easily set up to run the AUH utility on a regular basis by using
3199a cron job. See the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003200:yocto_git:`weeklyjob.sh </cgit/cgit.cgi/auto-upgrade-helper/tree/weeklyjob.sh>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003201file distributed with the utility for an example.
3202
3203.. _gs-using-devtool-upgrade:
3204
3205Using ``devtool upgrade``
3206-------------------------
3207
3208As mentioned earlier, an alternative method for upgrading recipes to
3209newer versions is to use
3210:doc:`devtool upgrade <../ref-manual/ref-devtool-reference>`.
3211You can read about ``devtool upgrade`` in general in the
3212":ref:`sdk-devtool-use-devtool-upgrade-to-create-a-version-of-the-recipe-that-supports-a-newer-version-of-the-software`"
3213section in the Yocto Project Application Development and the Extensible
3214Software Development Kit (eSDK) Manual.
3215
3216To see all the command-line options available with ``devtool upgrade``,
3217use the following help command:
3218::
3219
3220 $ devtool upgrade -h
3221
3222If you want to find out what version a recipe is currently at upstream
3223without any attempt to upgrade your local version of the recipe, you can
3224use the following command:
3225::
3226
3227 $ devtool latest-version recipe_name
3228
3229As mentioned in the previous section describing AUH, ``devtool upgrade``
3230works in a less-automated manner than AUH. Specifically,
3231``devtool upgrade`` only works on a single recipe that you name on the
3232command line, cannot perform build and integration testing using images,
3233and does not automatically generate commits for changes in the source
3234tree. Despite all these "limitations", ``devtool upgrade`` updates the
3235recipe file to the new upstream version and attempts to rebase custom
3236patches contained by the recipe as needed.
3237
3238.. note::
3239
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003240 AUH uses much of ``devtool upgrade`` behind the scenes making AUH somewhat
3241 of a "wrapper" application for ``devtool upgrade``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003242
3243A typical scenario involves having used Git to clone an upstream
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003244repository that you use during build operations. Because you have built the
3245recipe in the past, the layer is likely added to your
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003246configuration already. If for some reason, the layer is not added, you
3247could add it easily using the
3248":ref:`bitbake-layers <bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script>`"
3249script. For example, suppose you use the ``nano.bb`` recipe from the
3250``meta-oe`` layer in the ``meta-openembedded`` repository. For this
3251example, assume that the layer has been cloned into following area:
3252::
3253
3254 /home/scottrif/meta-openembedded
3255
3256The following command from your
3257:term:`Build Directory` adds the layer to
3258your build configuration (i.e. ``${BUILDDIR}/conf/bblayers.conf``):
3259::
3260
3261 $ bitbake-layers add-layer /home/scottrif/meta-openembedded/meta-oe
3262 NOTE: Starting bitbake server...
3263 Parsing recipes: 100% |##########################################| Time: 0:00:55
3264 Parsing of 1431 .bb files complete (0 cached, 1431 parsed). 2040 targets, 56 skipped, 0 masked, 0 errors.
3265 Removing 12 recipes from the x86_64 sysroot: 100% |##############| Time: 0:00:00
3266 Removing 1 recipes from the x86_64_i586 sysroot: 100% |##########| Time: 0:00:00
3267 Removing 5 recipes from the i586 sysroot: 100% |#################| Time: 0:00:00
3268 Removing 5 recipes from the qemux86 sysroot: 100% |##############| Time: 0:00:00
3269
3270For this example, assume that the ``nano.bb`` recipe that
3271is upstream has a 2.9.3 version number. However, the version in the
3272local repository is 2.7.4. The following command from your build
3273directory automatically upgrades the recipe for you:
3274
3275.. note::
3276
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003277 Using the ``-V`` option is not necessary. Omitting the version number causes
3278 ``devtool upgrade`` to upgrade the recipe to the most recent version.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003279
3280::
3281
3282 $ devtool upgrade nano -V 2.9.3
3283 NOTE: Starting bitbake server...
3284 NOTE: Creating workspace layer in /home/scottrif/poky/build/workspace
3285 Parsing recipes: 100% |##########################################| Time: 0:00:46
3286 Parsing of 1431 .bb files complete (0 cached, 1431 parsed). 2040 targets, 56 skipped, 0 masked, 0 errors.
3287 NOTE: Extracting current version source...
3288 NOTE: Resolving any missing task queue dependencies
3289 .
3290 .
3291 .
3292 NOTE: Executing SetScene Tasks
3293 NOTE: Executing RunQueue Tasks
3294 NOTE: Tasks Summary: Attempted 74 tasks of which 72 didn't need to be rerun and all succeeded.
3295 Adding changed files: 100% |#####################################| Time: 0:00:00
3296 NOTE: Upgraded source extracted to /home/scottrif/poky/build/workspace/sources/nano
3297 NOTE: New recipe is /home/scottrif/poky/build/workspace/recipes/nano/nano_2.9.3.bb
3298
3299Continuing with this example, you can use ``devtool build`` to build the
3300newly upgraded recipe:
3301::
3302
3303 $ devtool build nano
3304 NOTE: Starting bitbake server...
3305 Loading cache: 100% |################################################################################################| Time: 0:00:01
3306 Loaded 2040 entries from dependency cache.
3307 Parsing recipes: 100% |##############################################################################################| Time: 0:00:00
3308 Parsing of 1432 .bb files complete (1431 cached, 1 parsed). 2041 targets, 56 skipped, 0 masked, 0 errors.
3309 NOTE: Resolving any missing task queue dependencies
3310 .
3311 .
3312 .
3313 NOTE: Executing SetScene Tasks
3314 NOTE: Executing RunQueue Tasks
3315 NOTE: nano: compiling from external source tree /home/scottrif/poky/build/workspace/sources/nano
3316 NOTE: Tasks Summary: Attempted 520 tasks of which 304 didn't need to be rerun and all succeeded.
3317
3318Within the ``devtool upgrade`` workflow, opportunity
3319exists to deploy and test your rebuilt software. For this example,
3320however, running ``devtool finish`` cleans up the workspace once the
3321source in your workspace is clean. This usually means using Git to stage
3322and submit commits for the changes generated by the upgrade process.
3323
3324Once the tree is clean, you can clean things up in this example with the
3325following command from the ``${BUILDDIR}/workspace/sources/nano``
3326directory:
3327::
3328
3329 $ devtool finish nano meta-oe
3330 NOTE: Starting bitbake server...
3331 Loading cache: 100% |################################################################################################| Time: 0:00:00
3332 Loaded 2040 entries from dependency cache.
3333 Parsing recipes: 100% |##############################################################################################| Time: 0:00:01
3334 Parsing of 1432 .bb files complete (1431 cached, 1 parsed). 2041 targets, 56 skipped, 0 masked, 0 errors.
3335 NOTE: Adding new patch 0001-nano.bb-Stuff-I-changed-when-upgrading-nano.bb.patch
3336 NOTE: Updating recipe nano_2.9.3.bb
3337 NOTE: Removing file /home/scottrif/meta-openembedded/meta-oe/recipes-support/nano/nano_2.7.4.bb
3338 NOTE: Moving recipe file to /home/scottrif/meta-openembedded/meta-oe/recipes-support/nano
3339 NOTE: Leaving source tree /home/scottrif/poky/build/workspace/sources/nano as-is; if you no longer need it then please delete it manually
3340
3341
3342Using the ``devtool finish`` command cleans up the workspace and creates a patch
3343file based on your commits. The tool puts all patch files back into the
3344source directory in a sub-directory named ``nano`` in this case.
3345
3346.. _dev-manually-upgrading-a-recipe:
3347
3348Manually Upgrading a Recipe
3349---------------------------
3350
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003351If for some reason you choose not to upgrade recipes using
3352:ref:`gs-using-the-auto-upgrade-helper` or by :ref:`gs-using-devtool-upgrade`,
3353you can manually edit the recipe files to upgrade the versions.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003354
3355.. note::
3356
3357 Manually updating multiple recipes scales poorly and involves many
3358 steps. The recommendation to upgrade recipe versions is through AUH
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003359 or ``devtool upgrade``, both of which automate some steps and provide
3360 guidance for others needed for the manual process.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003361
3362To manually upgrade recipe versions, follow these general steps:
3363
33641. *Change the Version:* Rename the recipe such that the version (i.e.
3365 the :term:`PV` part of the recipe name)
3366 changes appropriately. If the version is not part of the recipe name,
3367 change the value as it is set for ``PV`` within the recipe itself.
3368
Andrew Geissler4c19ea12020-10-27 13:52:24 -050033692. *Update* ``SRCREV`` *if Needed*: If the source code your recipe builds
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003370 is fetched from Git or some other version control system, update
3371 :term:`SRCREV` to point to the
3372 commit hash that matches the new version.
3373
33743. *Build the Software:* Try to build the recipe using BitBake. Typical
3375 build failures include the following:
3376
3377 - License statements were updated for the new version. For this
3378 case, you need to review any changes to the license and update the
3379 values of :term:`LICENSE` and
3380 :term:`LIC_FILES_CHKSUM`
3381 as needed.
3382
3383 .. note::
3384
3385 License changes are often inconsequential. For example, the
3386 license text's copyright year might have changed.
3387
3388 - Custom patches carried by the older version of the recipe might
3389 fail to apply to the new version. For these cases, you need to
3390 review the failures. Patches might not be necessary for the new
3391 version of the software if the upgraded version has fixed those
3392 issues. If a patch is necessary and failing, you need to rebase it
3393 into the new version.
3394
33954. *Optionally Attempt to Build for Several Architectures:* Once you
3396 successfully build the new software for a given architecture, you
3397 could test the build for other architectures by changing the
3398 :term:`MACHINE` variable and
3399 rebuilding the software. This optional step is especially important
3400 if the recipe is to be released publicly.
3401
34025. *Check the Upstream Change Log or Release Notes:* Checking both these
3403 reveals if new features exist that could break
3404 backwards-compatibility. If so, you need to take steps to mitigate or
3405 eliminate that situation.
3406
34076. *Optionally Create a Bootable Image and Test:* If you want, you can
3408 test the new software by booting it onto actual hardware.
3409
34107. *Create a Commit with the Change in the Layer Repository:* After all
3411 builds work and any testing is successful, you can create commits for
3412 any changes in the layer holding your upgraded recipe.
3413
3414.. _finding-the-temporary-source-code:
3415
3416Finding Temporary Source Code
3417=============================
3418
3419You might find it helpful during development to modify the temporary
3420source code used by recipes to build packages. For example, suppose you
3421are developing a patch and you need to experiment a bit to figure out
3422your solution. After you have initially built the package, you can
3423iteratively tweak the source code, which is located in the
3424:term:`Build Directory`, and then you can
3425force a re-compile and quickly test your altered code. Once you settle
3426on a solution, you can then preserve your changes in the form of
3427patches.
3428
3429During a build, the unpacked temporary source code used by recipes to
3430build packages is available in the Build Directory as defined by the
3431:term:`S` variable. Below is the default
3432value for the ``S`` variable as defined in the
3433``meta/conf/bitbake.conf`` configuration file in the
3434:term:`Source Directory`:
3435::
3436
3437 S = "${WORKDIR}/${BP}"
3438
3439You should be aware that many recipes override the
3440``S`` variable. For example, recipes that fetch their source from Git
3441usually set ``S`` to ``${WORKDIR}/git``.
3442
3443.. note::
3444
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003445 The :term:`BP` represents the base recipe name, which consists of the name
3446 and version:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003447 ::
3448
3449 BP = "${BPN}-${PV}"
3450
3451
3452The path to the work directory for the recipe
3453(:term:`WORKDIR`) is defined as
3454follows:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003455::
3456
3457 ${TMPDIR}/work/${MULTIMACH_TARGET_SYS}/${PN}/${EXTENDPE}${PV}-${PR}
3458
3459The actual directory depends on several things:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003460
3461- :term:`TMPDIR`: The top-level build
3462 output directory.
3463
3464- :term:`MULTIMACH_TARGET_SYS`:
3465 The target system identifier.
3466
3467- :term:`PN`: The recipe name.
3468
3469- :term:`EXTENDPE`: The epoch - (if
3470 :term:`PE` is not specified, which is
3471 usually the case for most recipes, then ``EXTENDPE`` is blank).
3472
3473- :term:`PV`: The recipe version.
3474
3475- :term:`PR`: The recipe revision.
3476
3477As an example, assume a Source Directory top-level folder named
3478``poky``, a default Build Directory at ``poky/build``, and a
3479``qemux86-poky-linux`` machine target system. Furthermore, suppose your
3480recipe is named ``foo_1.3.0.bb``. In this case, the work directory the
3481build system uses to build the package would be as follows:
3482::
3483
3484 poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0
3485
3486.. _using-a-quilt-workflow:
3487
3488Using Quilt in Your Workflow
3489============================
3490
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003491`Quilt <https://savannah.nongnu.org/projects/quilt>`__ is a powerful tool
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003492that allows you to capture source code changes without having a clean
3493source tree. This section outlines the typical workflow you can use to
3494modify source code, test changes, and then preserve the changes in the
3495form of a patch all using Quilt.
3496
3497.. note::
3498
3499 With regard to preserving changes to source files, if you clean a
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003500 recipe or have ``rm_work`` enabled, the
3501 :ref:`devtool workflow <sdk-manual/sdk-extensible:using \`\`devtool\`\` in your sdk workflow>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003502 as described in the Yocto Project Application Development and the
3503 Extensible Software Development Kit (eSDK) manual is a safer
3504 development flow than the flow that uses Quilt.
3505
3506Follow these general steps:
3507
35081. *Find the Source Code:* Temporary source code used by the
3509 OpenEmbedded build system is kept in the
3510 :term:`Build Directory`. See the
3511 "`Finding Temporary Source
3512 Code <#finding-the-temporary-source-code>`__" section to learn how to
3513 locate the directory that has the temporary source code for a
3514 particular package.
3515
35162. *Change Your Working Directory:* You need to be in the directory that
3517 has the temporary source code. That directory is defined by the
3518 :term:`S` variable.
3519
35203. *Create a New Patch:* Before modifying source code, you need to
3521 create a new patch. To create a new patch file, use ``quilt new`` as
3522 below:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003523 ::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003524
3525 $ quilt new my_changes.patch
3526
35274. *Notify Quilt and Add Files:* After creating the patch, you need to
3528 notify Quilt about the files you plan to edit. You notify Quilt by
3529 adding the files to the patch you just created:
3530 ::
3531
3532 $ quilt add file1.c file2.c file3.c
3533
35345. *Edit the Files:* Make your changes in the source code to the files
3535 you added to the patch.
3536
35376. *Test Your Changes:* Once you have modified the source code, the
3538 easiest way to test your changes is by calling the ``do_compile``
3539 task as shown in the following example:
3540 ::
3541
3542 $ bitbake -c compile -f package
3543
3544 The ``-f`` or ``--force`` option forces the specified task to
3545 execute. If you find problems with your code, you can just keep
3546 editing and re-testing iteratively until things work as expected.
3547
3548 .. note::
3549
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003550 All the modifications you make to the temporary source code disappear
3551 once you run the ``do_clean`` or ``do_cleanall`` tasks using BitBake
3552 (i.e. ``bitbake -c clean package`` and ``bitbake -c cleanall package``).
3553 Modifications will also disappear if you use the ``rm_work`` feature as
3554 described in the
3555 ":ref:`dev-manual/dev-manual-common-tasks:conserving disk space during builds`"
3556 section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003557
35587. *Generate the Patch:* Once your changes work as expected, you need to
3559 use Quilt to generate the final patch that contains all your
3560 modifications.
3561 ::
3562
3563 $ quilt refresh
3564
3565 At this point, the
3566 ``my_changes.patch`` file has all your edits made to the ``file1.c``,
3567 ``file2.c``, and ``file3.c`` files.
3568
3569 You can find the resulting patch file in the ``patches/``
3570 subdirectory of the source (``S``) directory.
3571
35728. *Copy the Patch File:* For simplicity, copy the patch file into a
3573 directory named ``files``, which you can create in the same directory
3574 that holds the recipe (``.bb``) file or the append (``.bbappend``)
3575 file. Placing the patch here guarantees that the OpenEmbedded build
3576 system will find the patch. Next, add the patch into the ``SRC_URI``
3577 of the recipe. Here is an example:
3578 ::
3579
3580 SRC_URI += "file://my_changes.patch"
3581
3582.. _platdev-appdev-devshell:
3583
3584Using a Development Shell
3585=========================
3586
3587When debugging certain commands or even when just editing packages,
3588``devshell`` can be a useful tool. When you invoke ``devshell``, all
3589tasks up to and including
3590:ref:`ref-tasks-patch` are run for the
3591specified target. Then, a new terminal is opened and you are placed in
3592``${``\ :term:`S`\ ``}``, the source
3593directory. In the new terminal, all the OpenEmbedded build-related
3594environment variables are still defined so you can use commands such as
3595``configure`` and ``make``. The commands execute just as if the
3596OpenEmbedded build system were executing them. Consequently, working
3597this way can be helpful when debugging a build or preparing software to
3598be used with the OpenEmbedded build system.
3599
3600Following is an example that uses ``devshell`` on a target named
3601``matchbox-desktop``:
3602::
3603
3604 $ bitbake matchbox-desktop -c devshell
3605
3606This command spawns a terminal with a shell prompt within the
3607OpenEmbedded build environment. The
3608:term:`OE_TERMINAL` variable
3609controls what type of shell is opened.
3610
3611For spawned terminals, the following occurs:
3612
3613- The ``PATH`` variable includes the cross-toolchain.
3614
3615- The ``pkgconfig`` variables find the correct ``.pc`` files.
3616
3617- The ``configure`` command finds the Yocto Project site files as well
3618 as any other necessary files.
3619
3620Within this environment, you can run configure or compile commands as if
3621they were being run by the OpenEmbedded build system itself. As noted
3622earlier, the working directory also automatically changes to the Source
3623Directory (:term:`S`).
3624
3625To manually run a specific task using ``devshell``, run the
3626corresponding ``run.*`` script in the
3627``${``\ :term:`WORKDIR`\ ``}/temp``
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003628directory (e.g., ``run.do_configure.``\ `pid`). If a task's script does
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003629not exist, which would be the case if the task was skipped by way of the
3630sstate cache, you can create the task by first running it outside of the
3631``devshell``:
3632::
3633
3634 $ bitbake -c task
3635
3636.. note::
3637
3638 - Execution of a task's ``run.*`` script and BitBake's execution of
3639 a task are identical. In other words, running the script re-runs
3640 the task just as it would be run using the ``bitbake -c`` command.
3641
3642 - Any ``run.*`` file that does not have a ``.pid`` extension is a
3643 symbolic link (symlink) to the most recent version of that file.
3644
3645Remember, that the ``devshell`` is a mechanism that allows you to get
3646into the BitBake task execution environment. And as such, all commands
3647must be called just as BitBake would call them. That means you need to
3648provide the appropriate options for cross-compilation and so forth as
3649applicable.
3650
3651When you are finished using ``devshell``, exit the shell or close the
3652terminal window.
3653
3654.. note::
3655
3656 - It is worth remembering that when using ``devshell`` you need to
3657 use the full compiler name such as ``arm-poky-linux-gnueabi-gcc``
3658 instead of just using ``gcc``. The same applies to other
3659 applications such as ``binutils``, ``libtool`` and so forth.
3660 BitBake sets up environment variables such as ``CC`` to assist
3661 applications, such as ``make`` to find the correct tools.
3662
3663 - It is also worth noting that ``devshell`` still works over X11
3664 forwarding and similar situations.
3665
3666.. _platdev-appdev-devpyshell:
3667
3668Using a Development Python Shell
3669================================
3670
3671Similar to working within a development shell as described in the
3672previous section, you can also spawn and work within an interactive
3673Python development shell. When debugging certain commands or even when
3674just editing packages, ``devpyshell`` can be a useful tool. When you
3675invoke ``devpyshell``, all tasks up to and including
3676:ref:`ref-tasks-patch` are run for the
3677specified target. Then a new terminal is opened. Additionally, key
3678Python objects and code are available in the same way they are to
3679BitBake tasks, in particular, the data store 'd'. So, commands such as
3680the following are useful when exploring the data store and running
3681functions:
3682::
3683
3684 pydevshell> d.getVar("STAGING_DIR")
3685 '/media/build1/poky/build/tmp/sysroots'
3686 pydevshell> d.getVar("STAGING_DIR")
3687 '${TMPDIR}/sysroots'
3688 pydevshell> d.setVar("FOO", "bar")
3689 pydevshell> d.getVar("FOO")
3690 'bar'
3691 pydevshell> d.delVar("FOO")
3692 pydevshell> d.getVar("FOO")
3693 pydevshell> bb.build.exec_func("do_unpack", d)
3694 pydevshell>
3695
3696The commands execute just as if the OpenEmbedded build
3697system were executing them. Consequently, working this way can be
3698helpful when debugging a build or preparing software to be used with the
3699OpenEmbedded build system.
3700
3701Following is an example that uses ``devpyshell`` on a target named
3702``matchbox-desktop``:
3703::
3704
3705 $ bitbake matchbox-desktop -c devpyshell
3706
3707This command spawns a terminal and places you in an interactive Python
3708interpreter within the OpenEmbedded build environment. The
3709:term:`OE_TERMINAL` variable
3710controls what type of shell is opened.
3711
3712When you are finished using ``devpyshell``, you can exit the shell
3713either by using Ctrl+d or closing the terminal window.
3714
3715.. _dev-building:
3716
3717Building
3718========
3719
3720This section describes various build procedures. For example, the steps
3721needed for a simple build, a target that uses multiple configurations,
3722building an image for more than one machine, and so forth.
3723
3724.. _dev-building-a-simple-image:
3725
3726Building a Simple Image
3727-----------------------
3728
3729In the development environment, you need to build an image whenever you
3730change hardware support, add or change system libraries, or add or
3731change services that have dependencies. Several methods exist that allow
3732you to build an image within the Yocto Project. This section presents
3733the basic steps you need to build a simple image using BitBake from a
3734build host running Linux.
3735
3736.. note::
3737
3738 - For information on how to build an image using
3739 :term:`Toaster`, see the
3740 :doc:`../toaster-manual/toaster-manual`.
3741
3742 - For information on how to use ``devtool`` to build images, see the
3743 ":ref:`sdk-manual/sdk-extensible:using \`\`devtool\`\` in your sdk workflow`"
3744 section in the Yocto Project Application Development and the
3745 Extensible Software Development Kit (eSDK) manual.
3746
3747 - For a quick example on how to build an image using the
3748 OpenEmbedded build system, see the
3749 :doc:`../brief-yoctoprojectqs/brief-yoctoprojectqs` document.
3750
3751The build process creates an entire Linux distribution from source and
3752places it in your :term:`Build Directory` under
3753``tmp/deploy/images``. For detailed information on the build process
3754using BitBake, see the ":ref:`images-dev-environment`" section in the
3755Yocto Project Overview and Concepts Manual.
3756
3757The following figure and list overviews the build process:
3758
3759.. image:: figures/bitbake-build-flow.png
3760 :align: center
3761
37621. *Set up Your Host Development System to Support Development Using the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003763 Yocto Project*: See the ":doc:`dev-manual-start`" section for options on how to get a
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003764 build host ready to use the Yocto Project.
3765
37662. *Initialize the Build Environment:* Initialize the build environment
3767 by sourcing the build environment script (i.e.
3768 :ref:`structure-core-script`):
3769 ::
3770
3771 $ source oe-init-build-env [build_dir]
3772
3773 When you use the initialization script, the OpenEmbedded build system
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003774 uses ``build`` as the default :term:`Build Directory` in your current work
3775 directory. You can use a `build_dir` argument with the script to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003776 specify a different build directory.
3777
3778 .. note::
3779
3780 A common practice is to use a different Build Directory for
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003781 different targets. For example, ``~/build/x86`` for a ``qemux86``
3782 target, and ``~/build/arm`` for a ``qemuarm`` target.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003783
Andrew Geissler4c19ea12020-10-27 13:52:24 -050037843. *Make Sure Your* ``local.conf`` *File is Correct*: Ensure the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003785 ``conf/local.conf`` configuration file, which is found in the Build
3786 Directory, is set up how you want it. This file defines many aspects
3787 of the build environment including the target machine architecture
3788 through the ``MACHINE`` variable, the packaging format used during
3789 the build
3790 (:term:`PACKAGE_CLASSES`),
3791 and a centralized tarball download directory through the
3792 :term:`DL_DIR` variable.
3793
37944. *Build the Image:* Build the image using the ``bitbake`` command:
3795 ::
3796
3797 $ bitbake target
3798
3799 .. note::
3800
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003801 For information on BitBake, see the :doc:`bitbake:index`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003802
3803 The target is the name of the recipe you want to build. Common
3804 targets are the images in ``meta/recipes-core/images``,
3805 ``meta/recipes-sato/images``, and so forth all found in the
3806 :term:`Source Directory`. Or, the target
3807 can be the name of a recipe for a specific piece of software such as
3808 BusyBox. For more details about the images the OpenEmbedded build
3809 system supports, see the
3810 ":ref:`ref-manual/ref-images:Images`" chapter in the Yocto
3811 Project Reference Manual.
3812
3813 As an example, the following command builds the
3814 ``core-image-minimal`` image:
3815 ::
3816
3817 $ bitbake core-image-minimal
3818
3819 Once an
3820 image has been built, it often needs to be installed. The images and
3821 kernels built by the OpenEmbedded build system are placed in the
3822 Build Directory in ``tmp/deploy/images``. For information on how to
3823 run pre-built images such as ``qemux86`` and ``qemuarm``, see the
3824 :doc:`../sdk-manual/sdk-manual` manual. For
3825 information about how to install these images, see the documentation
3826 for your particular board or machine.
3827
3828.. _dev-building-images-for-multiple-targets-using-multiple-configurations:
3829
3830Building Images for Multiple Targets Using Multiple Configurations
3831------------------------------------------------------------------
3832
3833You can use a single ``bitbake`` command to build multiple images or
3834packages for different targets where each image or package requires a
3835different configuration (multiple configuration builds). The builds, in
3836this scenario, are sometimes referred to as "multiconfigs", and this
3837section uses that term throughout.
3838
3839This section describes how to set up for multiple configuration builds
3840and how to account for cross-build dependencies between the
3841multiconfigs.
3842
3843.. _dev-setting-up-and-running-a-multiple-configuration-build:
3844
3845Setting Up and Running a Multiple Configuration Build
3846~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3847
3848To accomplish a multiple configuration build, you must define each
3849target's configuration separately using a parallel configuration file in
3850the :term:`Build Directory`, and you
3851must follow a required file hierarchy. Additionally, you must enable the
3852multiple configuration builds in your ``local.conf`` file.
3853
3854Follow these steps to set up and execute multiple configuration builds:
3855
3856- *Create Separate Configuration Files*: You need to create a single
3857 configuration file for each build target (each multiconfig).
3858 Minimally, each configuration file must define the machine and the
3859 temporary directory BitBake uses for the build. Suggested practice
3860 dictates that you do not overlap the temporary directories used
3861 during the builds. However, it is possible that you can share the
3862 temporary directory
3863 (:term:`TMPDIR`). For example,
3864 consider a scenario with two different multiconfigs for the same
3865 :term:`MACHINE`: "qemux86" built
3866 for two distributions such as "poky" and "poky-lsb". In this case,
3867 you might want to use the same ``TMPDIR``.
3868
3869 Here is an example showing the minimal statements needed in a
3870 configuration file for a "qemux86" target whose temporary build
3871 directory is ``tmpmultix86``:
3872 ::
3873
3874 MACHINE = "qemux86"
3875 TMPDIR = "${TOPDIR}/tmpmultix86"
3876
3877 The location for these multiconfig configuration files is specific.
3878 They must reside in the current build directory in a sub-directory of
3879 ``conf`` named ``multiconfig``. Following is an example that defines
3880 two configuration files for the "x86" and "arm" multiconfigs:
3881
3882 .. image:: figures/multiconfig_files.png
3883 :align: center
3884
3885 The reason for this required file hierarchy is because the ``BBPATH``
3886 variable is not constructed until the layers are parsed.
3887 Consequently, using the configuration file as a pre-configuration
3888 file is not possible unless it is located in the current working
3889 directory.
3890
3891- *Add the BitBake Multi-configuration Variable to the Local
3892 Configuration File*: Use the
3893 :term:`BBMULTICONFIG`
3894 variable in your ``conf/local.conf`` configuration file to specify
3895 each multiconfig. Continuing with the example from the previous
3896 figure, the ``BBMULTICONFIG`` variable needs to enable two
3897 multiconfigs: "x86" and "arm" by specifying each configuration file:
3898 ::
3899
3900 BBMULTICONFIG = "x86 arm"
3901
3902 .. note::
3903
3904 A "default" configuration already exists by definition. This
3905 configuration is named: "" (i.e. empty string) and is defined by
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003906 the variables coming from your ``local.conf``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003907 file. Consequently, the previous example actually adds two
3908 additional configurations to your build: "arm" and "x86" along
3909 with "".
3910
3911- *Launch BitBake*: Use the following BitBake command form to launch
3912 the multiple configuration build:
3913 ::
3914
3915 $ bitbake [mc:multiconfigname:]target [[[mc:multiconfigname:]target] ... ]
3916
3917 For the example in this section, the following command applies:
3918 ::
3919
3920 $ bitbake mc:x86:core-image-minimal mc:arm:core-image-sato mc::core-image-base
3921
3922 The previous BitBake command builds a ``core-image-minimal`` image
3923 that is configured through the ``x86.conf`` configuration file, a
3924 ``core-image-sato`` image that is configured through the ``arm.conf``
3925 configuration file and a ``core-image-base`` that is configured
3926 through your ``local.conf`` configuration file.
3927
3928.. note::
3929
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003930 Support for multiple configuration builds in the Yocto Project &DISTRO;
3931 (&DISTRO_NAME;) Release does not include Shared State (sstate)
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003932 optimizations. Consequently, if a build uses the same object twice
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003933 in, for example, two different ``TMPDIR``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003934 directories, the build either loads from an existing sstate cache for
3935 that build at the start or builds the object fresh.
3936
3937.. _dev-enabling-multiple-configuration-build-dependencies:
3938
3939Enabling Multiple Configuration Build Dependencies
3940~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3941
3942Sometimes dependencies can exist between targets (multiconfigs) in a
3943multiple configuration build. For example, suppose that in order to
3944build a ``core-image-sato`` image for an "x86" multiconfig, the root
3945filesystem of an "arm" multiconfig must exist. This dependency is
3946essentially that the
3947:ref:`ref-tasks-image` task in the
3948``core-image-sato`` recipe depends on the completion of the
3949:ref:`ref-tasks-rootfs` task of the
3950``core-image-minimal`` recipe.
3951
3952To enable dependencies in a multiple configuration build, you must
3953declare the dependencies in the recipe using the following statement
3954form:
3955::
3956
3957 task_or_package[mcdepends] = "mc:from_multiconfig:to_multiconfig:recipe_name:task_on_which_to_depend"
3958
3959To better show how to use this statement, consider the example scenario
3960from the first paragraph of this section. The following statement needs
3961to be added to the recipe that builds the ``core-image-sato`` image:
3962::
3963
3964 do_image[mcdepends] = "mc:x86:arm:core-image-minimal:do_rootfs"
3965
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003966In this example, the `from_multiconfig` is "x86". The `to_multiconfig` is "arm". The
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003967task on which the ``do_image`` task in the recipe depends is the
3968``do_rootfs`` task from the ``core-image-minimal`` recipe associated
3969with the "arm" multiconfig.
3970
3971Once you set up this dependency, you can build the "x86" multiconfig
3972using a BitBake command as follows:
3973::
3974
3975 $ bitbake mc:x86:core-image-sato
3976
3977This command executes all the tasks needed to create the
3978``core-image-sato`` image for the "x86" multiconfig. Because of the
3979dependency, BitBake also executes through the ``do_rootfs`` task for the
3980"arm" multiconfig build.
3981
3982Having a recipe depend on the root filesystem of another build might not
3983seem that useful. Consider this change to the statement in the
3984``core-image-sato`` recipe:
3985::
3986
3987 do_image[mcdepends] = "mc:x86:arm:core-image-minimal:do_image"
3988
3989In this case, BitBake must
3990create the ``core-image-minimal`` image for the "arm" build since the
3991"x86" build depends on it.
3992
3993Because "x86" and "arm" are enabled for multiple configuration builds
3994and have separate configuration files, BitBake places the artifacts for
3995each build in the respective temporary build directories (i.e.
3996:term:`TMPDIR`).
3997
3998.. _building-an-initramfs-image:
3999
4000Building an Initial RAM Filesystem (initramfs) Image
4001----------------------------------------------------
4002
4003An initial RAM filesystem (initramfs) image provides a temporary root
4004filesystem used for early system initialization (e.g. loading of modules
4005needed to locate and mount the "real" root filesystem).
4006
4007.. note::
4008
4009 The initramfs image is the successor of initial RAM disk (initrd). It
4010 is a "copy in and out" (cpio) archive of the initial filesystem that
4011 gets loaded into memory during the Linux startup process. Because
4012 Linux uses the contents of the archive during initialization, the
4013 initramfs image needs to contain all of the device drivers and tools
4014 needed to mount the final root filesystem.
4015
4016Follow these steps to create an initramfs image:
4017
40181. *Create the initramfs Image Recipe:* You can reference the
4019 ``core-image-minimal-initramfs.bb`` recipe found in the
4020 ``meta/recipes-core`` directory of the :term:`Source Directory`
4021 as an example
4022 from which to work.
4023
40242. *Decide if You Need to Bundle the initramfs Image Into the Kernel
4025 Image:* If you want the initramfs image that is built to be bundled
4026 in with the kernel image, set the
4027 :term:`INITRAMFS_IMAGE_BUNDLE`
4028 variable to "1" in your ``local.conf`` configuration file and set the
4029 :term:`INITRAMFS_IMAGE`
4030 variable in the recipe that builds the kernel image.
4031
4032 .. note::
4033
4034 It is recommended that you do bundle the initramfs image with the
4035 kernel image to avoid circular dependencies between the kernel
4036 recipe and the initramfs recipe should the initramfs image include
4037 kernel modules.
4038
4039 Setting the ``INITRAMFS_IMAGE_BUNDLE`` flag causes the initramfs
4040 image to be unpacked into the ``${B}/usr/`` directory. The unpacked
4041 initramfs image is then passed to the kernel's ``Makefile`` using the
4042 :term:`CONFIG_INITRAMFS_SOURCE`
4043 variable, allowing the initramfs image to be built into the kernel
4044 normally.
4045
4046 .. note::
4047
4048 If you choose to not bundle the initramfs image with the kernel
4049 image, you are essentially using an
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004050 `Initial RAM Disk (initrd) <https://en.wikipedia.org/wiki/Initrd>`__.
4051 Creating an initrd is handled primarily through the :term:`INITRD_IMAGE`,
4052 ``INITRD_LIVE``, and ``INITRD_IMAGE_LIVE`` variables. For more
4053 information, see the :ref:`ref-classes-image-live` file.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004054
40553. *Optionally Add Items to the initramfs Image Through the initramfs
4056 Image Recipe:* If you add items to the initramfs image by way of its
4057 recipe, you should use
4058 :term:`PACKAGE_INSTALL`
4059 rather than
4060 :term:`IMAGE_INSTALL`.
4061 ``PACKAGE_INSTALL`` gives more direct control of what is added to the
4062 image as compared to the defaults you might not necessarily want that
4063 are set by the :ref:`image <ref-classes-image>`
4064 or :ref:`core-image <ref-classes-core-image>`
4065 classes.
4066
40674. *Build the Kernel Image and the initramfs Image:* Build your kernel
4068 image using BitBake. Because the initramfs image recipe is a
4069 dependency of the kernel image, the initramfs image is built as well
4070 and bundled with the kernel image if you used the
4071 :term:`INITRAMFS_IMAGE_BUNDLE`
4072 variable described earlier.
4073
4074Building a Tiny System
4075----------------------
4076
4077Very small distributions have some significant advantages such as
4078requiring less on-die or in-package memory (cheaper), better performance
4079through efficient cache usage, lower power requirements due to less
4080memory, faster boot times, and reduced development overhead. Some
4081real-world examples where a very small distribution gives you distinct
4082advantages are digital cameras, medical devices, and small headless
4083systems.
4084
4085This section presents information that shows you how you can trim your
4086distribution to even smaller sizes than the ``poky-tiny`` distribution,
4087which is around 5 Mbytes, that can be built out-of-the-box using the
4088Yocto Project.
4089
4090.. _tiny-system-overview:
4091
4092Tiny System Overview
4093~~~~~~~~~~~~~~~~~~~~
4094
4095The following list presents the overall steps you need to consider and
4096perform to create distributions with smaller root filesystems, achieve
4097faster boot times, maintain your critical functionality, and avoid
4098initial RAM disks:
4099
4100- `Determine your goals and guiding
4101 principles. <#goals-and-guiding-principles>`__
4102
4103- `Understand what contributes to your image
4104 size. <#understand-what-gives-your-image-size>`__
4105
4106- `Reduce the size of the root
4107 filesystem. <#trim-the-root-filesystem>`__
4108
4109- `Reduce the size of the kernel. <#trim-the-kernel>`__
4110
4111- `Eliminate packaging
4112 requirements. <#remove-package-management-requirements>`__
4113
4114- `Look for other ways to minimize
4115 size. <#look-for-other-ways-to-minimize-size>`__
4116
4117- `Iterate on the process. <#iterate-on-the-process>`__
4118
4119Goals and Guiding Principles
4120~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4121
4122Before you can reach your destination, you need to know where you are
4123going. Here is an example list that you can use as a guide when creating
4124very small distributions:
4125
4126- Determine how much space you need (e.g. a kernel that is 1 Mbyte or
4127 less and a root filesystem that is 3 Mbytes or less).
4128
4129- Find the areas that are currently taking 90% of the space and
4130 concentrate on reducing those areas.
4131
4132- Do not create any difficult "hacks" to achieve your goals.
4133
4134- Leverage the device-specific options.
4135
4136- Work in a separate layer so that you keep changes isolated. For
4137 information on how to create layers, see the "`Understanding and
4138 Creating Layers <#understanding-and-creating-layers>`__" section.
4139
4140.. _understand-what-gives-your-image-size:
4141
4142Understand What Contributes to Your Image Size
4143~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4144
4145It is easiest to have something to start with when creating your own
4146distribution. You can use the Yocto Project out-of-the-box to create the
4147``poky-tiny`` distribution. Ultimately, you will want to make changes in
4148your own distribution that are likely modeled after ``poky-tiny``.
4149
4150.. note::
4151
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004152 To use ``poky-tiny`` in your build, set the ``DISTRO`` variable in your
4153 ``local.conf`` file to "poky-tiny" as described in the
4154 ":ref:`dev-manual/dev-manual-common-tasks:creating your own distribution`"
4155 section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004156
4157Understanding some memory concepts will help you reduce the system size.
4158Memory consists of static, dynamic, and temporary memory. Static memory
4159is the TEXT (code), DATA (initialized data in the code), and BSS
4160(uninitialized data) sections. Dynamic memory represents memory that is
4161allocated at runtime: stacks, hash tables, and so forth. Temporary
4162memory is recovered after the boot process. This memory consists of
4163memory used for decompressing the kernel and for the ``__init__``
4164functions.
4165
4166To help you see where you currently are with kernel and root filesystem
4167sizes, you can use two tools found in the :term:`Source Directory`
4168in the
4169``scripts/tiny/`` directory:
4170
4171- ``ksize.py``: Reports component sizes for the kernel build objects.
4172
4173- ``dirsize.py``: Reports component sizes for the root filesystem.
4174
4175This next tool and command help you organize configuration fragments and
4176view file dependencies in a human-readable form:
4177
4178- ``merge_config.sh``: Helps you manage configuration files and
4179 fragments within the kernel. With this tool, you can merge individual
4180 configuration fragments together. The tool allows you to make
4181 overrides and warns you of any missing configuration options. The
4182 tool is ideal for allowing you to iterate on configurations, create
4183 minimal configurations, and create configuration files for different
4184 machines without having to duplicate your process.
4185
4186 The ``merge_config.sh`` script is part of the Linux Yocto kernel Git
4187 repositories (i.e. ``linux-yocto-3.14``, ``linux-yocto-3.10``,
4188 ``linux-yocto-3.8``, and so forth) in the ``scripts/kconfig``
4189 directory.
4190
4191 For more information on configuration fragments, see the
4192 ":ref:`creating-config-fragments`"
4193 section in the Yocto Project Linux Kernel Development Manual.
4194
4195- ``bitbake -u taskexp -g bitbake_target``: Using the BitBake command
4196 with these options brings up a Dependency Explorer from which you can
4197 view file dependencies. Understanding these dependencies allows you
4198 to make informed decisions when cutting out various pieces of the
4199 kernel and root filesystem.
4200
4201Trim the Root Filesystem
4202~~~~~~~~~~~~~~~~~~~~~~~~
4203
4204The root filesystem is made up of packages for booting, libraries, and
4205applications. To change things, you can configure how the packaging
4206happens, which changes the way you build them. You can also modify the
4207filesystem itself or select a different filesystem.
4208
4209First, find out what is hogging your root filesystem by running the
4210``dirsize.py`` script from your root directory:
4211::
4212
4213 $ cd root-directory-of-image
4214 $ dirsize.py 100000 > dirsize-100k.log
4215 $ cat dirsize-100k.log
4216
4217You can apply a filter to the script to ignore files
4218under a certain size. The previous example filters out any files below
4219100 Kbytes. The sizes reported by the tool are uncompressed, and thus
4220will be smaller by a relatively constant factor in a compressed root
4221filesystem. When you examine your log file, you can focus on areas of
4222the root filesystem that take up large amounts of memory.
4223
4224You need to be sure that what you eliminate does not cripple the
4225functionality you need. One way to see how packages relate to each other
4226is by using the Dependency Explorer UI with the BitBake command:
4227::
4228
4229 $ cd image-directory
4230 $ bitbake -u taskexp -g image
4231
4232Use the interface to
4233select potential packages you wish to eliminate and see their dependency
4234relationships.
4235
4236When deciding how to reduce the size, get rid of packages that result in
4237minimal impact on the feature set. For example, you might not need a VGA
4238display. Or, you might be able to get by with ``devtmpfs`` and ``mdev``
4239instead of ``udev``.
4240
4241Use your ``local.conf`` file to make changes. For example, to eliminate
4242``udev`` and ``glib``, set the following in the local configuration
4243file:
4244::
4245
4246 VIRTUAL-RUNTIME_dev_manager = ""
4247
4248Finally, you should consider exactly the type of root filesystem you
4249need to meet your needs while also reducing its size. For example,
4250consider ``cramfs``, ``squashfs``, ``ubifs``, ``ext2``, or an
4251``initramfs`` using ``initramfs``. Be aware that ``ext3`` requires a 1
4252Mbyte journal. If you are okay with running read-only, you do not need
4253this journal.
4254
4255.. note::
4256
4257 After each round of elimination, you need to rebuild your system and
4258 then use the tools to see the effects of your reductions.
4259
4260Trim the Kernel
4261~~~~~~~~~~~~~~~
4262
4263The kernel is built by including policies for hardware-independent
4264aspects. What subsystems do you enable? For what architecture are you
4265building? Which drivers do you build by default?
4266
4267.. note::
4268
4269 You can modify the kernel source if you want to help with boot time.
4270
4271Run the ``ksize.py`` script from the top-level Linux build directory to
4272get an idea of what is making up the kernel:
4273::
4274
4275 $ cd top-level-linux-build-directory
4276 $ ksize.py > ksize.log
4277 $ cat ksize.log
4278
4279When you examine the log, you will see how much space is taken up with
4280the built-in ``.o`` files for drivers, networking, core kernel files,
4281filesystem, sound, and so forth. The sizes reported by the tool are
4282uncompressed, and thus will be smaller by a relatively constant factor
4283in a compressed kernel image. Look to reduce the areas that are large
4284and taking up around the "90% rule."
4285
4286To examine, or drill down, into any particular area, use the ``-d``
4287option with the script:
4288::
4289
4290 $ ksize.py -d > ksize.log
4291
4292Using this option
4293breaks out the individual file information for each area of the kernel
4294(e.g. drivers, networking, and so forth).
4295
4296Use your log file to see what you can eliminate from the kernel based on
4297features you can let go. For example, if you are not going to need
4298sound, you do not need any drivers that support sound.
4299
4300After figuring out what to eliminate, you need to reconfigure the kernel
4301to reflect those changes during the next build. You could run
4302``menuconfig`` and make all your changes at once. However, that makes it
4303difficult to see the effects of your individual eliminations and also
4304makes it difficult to replicate the changes for perhaps another target
4305device. A better method is to start with no configurations using
4306``allnoconfig``, create configuration fragments for individual changes,
4307and then manage the fragments into a single configuration file using
4308``merge_config.sh``. The tool makes it easy for you to iterate using the
4309configuration change and build cycle.
4310
4311Each time you make configuration changes, you need to rebuild the kernel
4312and check to see what impact your changes had on the overall size.
4313
4314Remove Package Management Requirements
4315~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4316
4317Packaging requirements add size to the image. One way to reduce the size
4318of the image is to remove all the packaging requirements from the image.
4319This reduction includes both removing the package manager and its unique
4320dependencies as well as removing the package management data itself.
4321
4322To eliminate all the packaging requirements for an image, be sure that
4323"package-management" is not part of your
4324:term:`IMAGE_FEATURES`
4325statement for the image. When you remove this feature, you are removing
4326the package manager as well as its dependencies from the root
4327filesystem.
4328
4329Look for Other Ways to Minimize Size
4330~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4331
4332Depending on your particular circumstances, other areas that you can
4333trim likely exist. The key to finding these areas is through tools and
4334methods described here combined with experimentation and iteration. Here
4335are a couple of areas to experiment with:
4336
4337- ``glibc``: In general, follow this process:
4338
4339 1. Remove ``glibc`` features from
4340 :term:`DISTRO_FEATURES`
4341 that you think you do not need.
4342
4343 2. Build your distribution.
4344
4345 3. If the build fails due to missing symbols in a package, determine
4346 if you can reconfigure the package to not need those features. For
4347 example, change the configuration to not support wide character
4348 support as is done for ``ncurses``. Or, if support for those
4349 characters is needed, determine what ``glibc`` features provide
4350 the support and restore the configuration.
4351
4352 4. Rebuild and repeat the process.
4353
4354- ``busybox``: For BusyBox, use a process similar as described for
4355 ``glibc``. A difference is you will need to boot the resulting system
4356 to see if you are able to do everything you expect from the running
4357 system. You need to be sure to integrate configuration fragments into
4358 Busybox because BusyBox handles its own core features and then allows
4359 you to add configuration fragments on top.
4360
4361Iterate on the Process
4362~~~~~~~~~~~~~~~~~~~~~~
4363
4364If you have not reached your goals on system size, you need to iterate
4365on the process. The process is the same. Use the tools and see just what
4366is taking up 90% of the root filesystem and the kernel. Decide what you
4367can eliminate without limiting your device beyond what you need.
4368
4369Depending on your system, a good place to look might be Busybox, which
4370provides a stripped down version of Unix tools in a single, executable
4371file. You might be able to drop virtual terminal services or perhaps
4372ipv6.
4373
4374Building Images for More than One Machine
4375-----------------------------------------
4376
4377A common scenario developers face is creating images for several
4378different machines that use the same software environment. In this
4379situation, it is tempting to set the tunings and optimization flags for
4380each build specifically for the targeted hardware (i.e. "maxing out" the
4381tunings). Doing so can considerably add to build times and package feed
4382maintenance collectively for the machines. For example, selecting tunes
4383that are extremely specific to a CPU core used in a system might enable
4384some micro optimizations in GCC for that particular system but would
4385otherwise not gain you much of a performance difference across the other
4386systems as compared to using a more general tuning across all the builds
4387(e.g. setting :term:`DEFAULTTUNE`
4388specifically for each machine's build). Rather than "max out" each
4389build's tunings, you can take steps that cause the OpenEmbedded build
4390system to reuse software across the various machines where it makes
4391sense.
4392
4393If build speed and package feed maintenance are considerations, you
4394should consider the points in this section that can help you optimize
4395your tunings to best consider build times and package feed maintenance.
4396
4397- *Share the Build Directory:* If at all possible, share the
4398 :term:`TMPDIR` across builds. The
4399 Yocto Project supports switching between different
4400 :term:`MACHINE` values in the same
4401 ``TMPDIR``. This practice is well supported and regularly used by
4402 developers when building for multiple machines. When you use the same
4403 ``TMPDIR`` for multiple machine builds, the OpenEmbedded build system
4404 can reuse the existing native and often cross-recipes for multiple
4405 machines. Thus, build time decreases.
4406
4407 .. note::
4408
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004409 If :term:`DISTRO` settings change or fundamental configuration settings
4410 such as the filesystem layout, you need to work with a clean ``TMPDIR``.
4411 Sharing ``TMPDIR`` under these circumstances might work but since it is
4412 not guaranteed, you should use a clean ``TMPDIR``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004413
4414- *Enable the Appropriate Package Architecture:* By default, the
4415 OpenEmbedded build system enables three levels of package
4416 architectures: "all", "tune" or "package", and "machine". Any given
4417 recipe usually selects one of these package architectures (types) for
4418 its output. Depending for what a given recipe creates packages,
4419 making sure you enable the appropriate package architecture can
4420 directly impact the build time.
4421
4422 A recipe that just generates scripts can enable "all" architecture
4423 because there are no binaries to build. To specifically enable "all"
4424 architecture, be sure your recipe inherits the
4425 :ref:`allarch <ref-classes-allarch>` class.
4426 This class is useful for "all" architectures because it configures
4427 many variables so packages can be used across multiple architectures.
4428
4429 If your recipe needs to generate packages that are machine-specific
4430 or when one of the build or runtime dependencies is already
4431 machine-architecture dependent, which makes your recipe also
4432 machine-architecture dependent, make sure your recipe enables the
4433 "machine" package architecture through the
4434 :term:`MACHINE_ARCH`
4435 variable:
4436 ::
4437
4438 PACKAGE_ARCH = "${MACHINE_ARCH}"
4439
4440 When you do not
4441 specifically enable a package architecture through the
4442 :term:`PACKAGE_ARCH`, The
4443 OpenEmbedded build system defaults to the
4444 :term:`TUNE_PKGARCH` setting:
4445 ::
4446
4447 PACKAGE_ARCH = "${TUNE_PKGARCH}"
4448
4449- *Choose a Generic Tuning File if Possible:* Some tunes are more
4450 generic and can run on multiple targets (e.g. an ``armv5`` set of
4451 packages could run on ``armv6`` and ``armv7`` processors in most
4452 cases). Similarly, ``i486`` binaries could work on ``i586`` and
4453 higher processors. You should realize, however, that advances on
4454 newer processor versions would not be used.
4455
4456 If you select the same tune for several different machines, the
4457 OpenEmbedded build system reuses software previously built, thus
4458 speeding up the overall build time. Realize that even though a new
4459 sysroot for each machine is generated, the software is not recompiled
4460 and only one package feed exists.
4461
4462- *Manage Granular Level Packaging:* Sometimes cases exist where
4463 injecting another level of package architecture beyond the three
4464 higher levels noted earlier can be useful. For example, consider how
4465 NXP (formerly Freescale) allows for the easy reuse of binary packages
4466 in their layer
4467 :yocto_git:`meta-freescale </cgit/cgit.cgi/meta-freescale/>`.
4468 In this example, the
4469 :yocto_git:`fsl-dynamic-packagearch </cgit/cgit.cgi/meta-freescale/tree/classes/fsl-dynamic-packagearch.bbclass>`
4470 class shares GPU packages for i.MX53 boards because all boards share
4471 the AMD GPU. The i.MX6-based boards can do the same because all
4472 boards share the Vivante GPU. This class inspects the BitBake
4473 datastore to identify if the package provides or depends on one of
4474 the sub-architecture values. If so, the class sets the
4475 :term:`PACKAGE_ARCH` value
4476 based on the ``MACHINE_SUBARCH`` value. If the package does not
4477 provide or depend on one of the sub-architecture values but it
4478 matches a value in the machine-specific filter, it sets
4479 :term:`MACHINE_ARCH`. This
4480 behavior reduces the number of packages built and saves build time by
4481 reusing binaries.
4482
4483- *Use Tools to Debug Issues:* Sometimes you can run into situations
4484 where software is being rebuilt when you think it should not be. For
4485 example, the OpenEmbedded build system might not be using shared
4486 state between machines when you think it should be. These types of
4487 situations are usually due to references to machine-specific
4488 variables such as :term:`MACHINE`,
4489 :term:`SERIAL_CONSOLES`,
4490 :term:`XSERVER`,
4491 :term:`MACHINE_FEATURES`,
4492 and so forth in code that is supposed to only be tune-specific or
4493 when the recipe depends
4494 (:term:`DEPENDS`,
4495 :term:`RDEPENDS`,
4496 :term:`RRECOMMENDS`,
4497 :term:`RSUGGESTS`, and so forth)
4498 on some other recipe that already has
4499 :term:`PACKAGE_ARCH` defined
4500 as "${MACHINE_ARCH}".
4501
4502 .. note::
4503
4504 Patches to fix any issues identified are most welcome as these
4505 issues occasionally do occur.
4506
4507 For such cases, you can use some tools to help you sort out the
4508 situation:
4509
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004510 - ``state-diff-machines.sh``*:* You can find this tool in the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004511 ``scripts`` directory of the Source Repositories. See the comments
4512 in the script for information on how to use the tool.
4513
4514 - *BitBake's "-S printdiff" Option:* Using this option causes
4515 BitBake to try to establish the closest signature match it can
4516 (e.g. in the shared state cache) and then run ``bitbake-diffsigs``
4517 over the matches to determine the stamps and delta where these two
4518 stamp trees diverge.
4519
4520Building Software from an External Source
4521-----------------------------------------
4522
4523By default, the OpenEmbedded build system uses the
4524:term:`Build Directory` when building source
4525code. The build process involves fetching the source files, unpacking
4526them, and then patching them if necessary before the build takes place.
4527
4528Situations exist where you might want to build software from source
4529files that are external to and thus outside of the OpenEmbedded build
4530system. For example, suppose you have a project that includes a new BSP
4531with a heavily customized kernel. And, you want to minimize exposing the
4532build system to the development team so that they can focus on their
4533project and maintain everyone's workflow as much as possible. In this
4534case, you want a kernel source directory on the development machine
4535where the development occurs. You want the recipe's
4536:term:`SRC_URI` variable to point to
4537the external directory and use it as is, not copy it.
4538
4539To build from software that comes from an external source, all you need
4540to do is inherit the
4541:ref:`externalsrc <ref-classes-externalsrc>` class
4542and then set the
4543:term:`EXTERNALSRC` variable to
4544point to your external source code. Here are the statements to put in
4545your ``local.conf`` file:
4546::
4547
4548 INHERIT += "externalsrc"
4549 EXTERNALSRC_pn-myrecipe = "path-to-your-source-tree"
4550
4551This next example shows how to accomplish the same thing by setting
4552``EXTERNALSRC`` in the recipe itself or in the recipe's append file:
4553::
4554
4555 EXTERNALSRC = "path"
4556 EXTERNALSRC_BUILD = "path"
4557
4558.. note::
4559
4560 In order for these settings to take effect, you must globally or
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004561 locally inherit the :ref:`externalsrc <ref-classes-externalsrc>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004562 class.
4563
4564By default, ``externalsrc.bbclass`` builds the source code in a
4565directory separate from the external source directory as specified by
4566:term:`EXTERNALSRC`. If you need
4567to have the source built in the same directory in which it resides, or
4568some other nominated directory, you can set
4569:term:`EXTERNALSRC_BUILD`
4570to point to that directory:
4571::
4572
4573 EXTERNALSRC_BUILD_pn-myrecipe = "path-to-your-source-tree"
4574
4575Replicating a Build Offline
4576---------------------------
4577
4578It can be useful to take a "snapshot" of upstream sources used in a
4579build and then use that "snapshot" later to replicate the build offline.
4580To do so, you need to first prepare and populate your downloads
4581directory your "snapshot" of files. Once your downloads directory is
4582ready, you can use it at any time and from any machine to replicate your
4583build.
4584
4585Follow these steps to populate your Downloads directory:
4586
45871. *Create a Clean Downloads Directory:* Start with an empty downloads
4588 directory (:term:`DL_DIR`). You
4589 start with an empty downloads directory by either removing the files
4590 in the existing directory or by setting ``DL_DIR`` to point to either
4591 an empty location or one that does not yet exist.
4592
45932. *Generate Tarballs of the Source Git Repositories:* Edit your
4594 ``local.conf`` configuration file as follows:
4595 ::
4596
4597 DL_DIR = "/home/your-download-dir/"
4598 BB_GENERATE_MIRROR_TARBALLS = "1"
4599
4600 During
4601 the fetch process in the next step, BitBake gathers the source files
4602 and creates tarballs in the directory pointed to by ``DL_DIR``. See
4603 the
4604 :term:`BB_GENERATE_MIRROR_TARBALLS`
4605 variable for more information.
4606
46073. *Populate Your Downloads Directory Without Building:* Use BitBake to
4608 fetch your sources but inhibit the build:
4609 ::
4610
4611 $ bitbake target --runonly=fetch
4612
4613 The downloads directory (i.e. ``${DL_DIR}``) now has
4614 a "snapshot" of the source files in the form of tarballs, which can
4615 be used for the build.
4616
46174. *Optionally Remove Any Git or other SCM Subdirectories From the
4618 Downloads Directory:* If you want, you can clean up your downloads
4619 directory by removing any Git or other Source Control Management
4620 (SCM) subdirectories such as ``${DL_DIR}/git2/*``. The tarballs
4621 already contain these subdirectories.
4622
4623Once your downloads directory has everything it needs regarding source
4624files, you can create your "own-mirror" and build your target.
4625Understand that you can use the files to build the target offline from
4626any machine and at any time.
4627
4628Follow these steps to build your target using the files in the downloads
4629directory:
4630
46311. *Using Local Files Only:* Inside your ``local.conf`` file, add the
4632 :term:`SOURCE_MIRROR_URL`
4633 variable, inherit the
4634 :ref:`own-mirrors <ref-classes-own-mirrors>`
4635 class, and use the
4636 :term:`bitbake:BB_NO_NETWORK`
4637 variable to your ``local.conf``.
4638 ::
4639
4640 SOURCE_MIRROR_URL ?= "file:///home/your-download-dir/"
4641 INHERIT += "own-mirrors"
4642 BB_NO_NETWORK = "1"
4643
4644 The ``SOURCE_MIRROR_URL`` and ``own-mirror``
4645 class set up the system to use the downloads directory as your "own
4646 mirror". Using the ``BB_NO_NETWORK`` variable makes sure that
4647 BitBake's fetching process in step 3 stays local, which means files
4648 from your "own-mirror" are used.
4649
46502. *Start With a Clean Build:* You can start with a clean build by
4651 removing the
4652 ``${``\ :term:`TMPDIR`\ ``}``
4653 directory or using a new :term:`Build Directory`.
4654
46553. *Build Your Target:* Use BitBake to build your target:
4656 ::
4657
4658 $ bitbake target
4659
4660 The build completes using the known local "snapshot" of source
4661 files from your mirror. The resulting tarballs for your "snapshot" of
4662 source files are in the downloads directory.
4663
4664 .. note::
4665
4666 The offline build does not work if recipes attempt to find the
4667 latest version of software by setting
4668 :term:`SRCREV` to
4669 ``${``\ :term:`AUTOREV`\ ``}``:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004670 ::
4671
4672 SRCREV = "${AUTOREV}"
4673
4674 When a recipe sets ``SRCREV`` to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004675 ``${AUTOREV}``, the build system accesses the network in an
4676 attempt to determine the latest version of software from the SCM.
4677 Typically, recipes that use ``AUTOREV`` are custom or modified
4678 recipes. Recipes that reside in public repositories usually do not
4679 use ``AUTOREV``.
4680
4681 If you do have recipes that use ``AUTOREV``, you can take steps to
4682 still use the recipes in an offline build. Do the following:
4683
4684 1. Use a configuration generated by enabling `build
4685 history <#maintaining-build-output-quality>`__.
4686
4687 2. Use the ``buildhistory-collect-srcrevs`` command to collect the
4688 stored ``SRCREV`` values from the build's history. For more
4689 information on collecting these values, see the "`Build History
4690 Package Information <#build-history-package-information>`__"
4691 section.
4692
4693 3. Once you have the correct source revisions, you can modify
4694 those recipes to to set ``SRCREV`` to specific versions of the
4695 software.
4696
4697Speeding Up a Build
4698===================
4699
4700Build time can be an issue. By default, the build system uses simple
4701controls to try and maximize build efficiency. In general, the default
4702settings for all the following variables result in the most efficient
4703build times when dealing with single socket systems (i.e. a single CPU).
4704If you have multiple CPUs, you might try increasing the default values
4705to gain more speed. See the descriptions in the glossary for each
4706variable for more information:
4707
4708- :term:`BB_NUMBER_THREADS`:
4709 The maximum number of threads BitBake simultaneously executes.
4710
4711- :term:`bitbake:BB_NUMBER_PARSE_THREADS`:
4712 The number of threads BitBake uses during parsing.
4713
4714- :term:`PARALLEL_MAKE`: Extra
4715 options passed to the ``make`` command during the
4716 :ref:`ref-tasks-compile` task in
4717 order to specify parallel compilation on the local build host.
4718
4719- :term:`PARALLEL_MAKEINST`:
4720 Extra options passed to the ``make`` command during the
4721 :ref:`ref-tasks-install` task in
4722 order to specify parallel installation on the local build host.
4723
4724As mentioned, these variables all scale to the number of processor cores
4725available on the build system. For single socket systems, this
4726auto-scaling ensures that the build system fundamentally takes advantage
4727of potential parallel operations during the build based on the build
4728machine's capabilities.
4729
4730Following are additional factors that can affect build speed:
4731
4732- File system type: The file system type that the build is being
4733 performed on can also influence performance. Using ``ext4`` is
4734 recommended as compared to ``ext2`` and ``ext3`` due to ``ext4``
4735 improved features such as extents.
4736
4737- Disabling the updating of access time using ``noatime``: The
4738 ``noatime`` mount option prevents the build system from updating file
4739 and directory access times.
4740
4741- Setting a longer commit: Using the "commit=" mount option increases
4742 the interval in seconds between disk cache writes. Changing this
4743 interval from the five second default to something longer increases
4744 the risk of data loss but decreases the need to write to the disk,
4745 thus increasing the build performance.
4746
4747- Choosing the packaging backend: Of the available packaging backends,
4748 IPK is the fastest. Additionally, selecting a singular packaging
4749 backend also helps.
4750
4751- Using ``tmpfs`` for :term:`TMPDIR`
4752 as a temporary file system: While this can help speed up the build,
4753 the benefits are limited due to the compiler using ``-pipe``. The
4754 build system goes to some lengths to avoid ``sync()`` calls into the
4755 file system on the principle that if there was a significant failure,
4756 the :term:`Build Directory`
4757 contents could easily be rebuilt.
4758
4759- Inheriting the
4760 :ref:`rm_work <ref-classes-rm-work>` class:
4761 Inheriting this class has shown to speed up builds due to
4762 significantly lower amounts of data stored in the data cache as well
4763 as on disk. Inheriting this class also makes cleanup of
4764 :term:`TMPDIR` faster, at the
4765 expense of being easily able to dive into the source code. File
4766 system maintainers have recommended that the fastest way to clean up
4767 large numbers of files is to reformat partitions rather than delete
4768 files due to the linear nature of partitions. This, of course,
4769 assumes you structure the disk partitions and file systems in a way
4770 that this is practical.
4771
4772Aside from the previous list, you should keep some trade offs in mind
4773that can help you speed up the build:
4774
4775- Remove items from
4776 :term:`DISTRO_FEATURES`
4777 that you might not need.
4778
4779- Exclude debug symbols and other debug information: If you do not need
4780 these symbols and other debug information, disabling the ``*-dbg``
4781 package generation can speed up the build. You can disable this
4782 generation by setting the
4783 :term:`INHIBIT_PACKAGE_DEBUG_SPLIT`
4784 variable to "1".
4785
4786- Disable static library generation for recipes derived from
4787 ``autoconf`` or ``libtool``: Following is an example showing how to
4788 disable static libraries and still provide an override to handle
4789 exceptions:
4790 ::
4791
4792 STATICLIBCONF = "--disable-static"
4793 STATICLIBCONF_sqlite3-native = ""
4794 EXTRA_OECONF += "${STATICLIBCONF}"
4795
4796 .. note::
4797
4798 - Some recipes need static libraries in order to work correctly
4799 (e.g. ``pseudo-native`` needs ``sqlite3-native``). Overrides,
4800 as in the previous example, account for these kinds of
4801 exceptions.
4802
4803 - Some packages have packaging code that assumes the presence of
4804 the static libraries. If so, you might need to exclude them as
4805 well.
4806
4807.. _platdev-working-with-libraries:
4808
4809Working With Libraries
4810======================
4811
4812Libraries are an integral part of your system. This section describes
4813some common practices you might find helpful when working with libraries
4814to build your system:
4815
4816- `How to include static library
4817 files <#including-static-library-files>`__
4818
4819- `How to use the Multilib feature to combine multiple versions of
4820 library files into a single
4821 image <#combining-multiple-versions-library-files-into-one-image>`__
4822
4823- `How to install multiple versions of the same library in parallel on
4824 the same
4825 system <#installing-multiple-versions-of-the-same-library>`__
4826
4827Including Static Library Files
4828------------------------------
4829
4830If you are building a library and the library offers static linking, you
4831can control which static library files (``*.a`` files) get included in
4832the built library.
4833
4834The :term:`PACKAGES` and
4835:term:`FILES_* <FILES>` variables in the
4836``meta/conf/bitbake.conf`` configuration file define how files installed
4837by the ``do_install`` task are packaged. By default, the ``PACKAGES``
4838variable includes ``${PN}-staticdev``, which represents all static
4839library files.
4840
4841.. note::
4842
4843 Some previously released versions of the Yocto Project defined the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004844 static library files through ``${PN}-dev``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004845
4846Following is part of the BitBake configuration file, where you can see
4847how the static library files are defined:
4848::
4849
4850 PACKAGE_BEFORE_PN ?= ""
4851 PACKAGES = "${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc ${PN}-locale ${PACKAGE_BEFORE_PN} ${PN}"
4852 PACKAGES_DYNAMIC = "^${PN}-locale-.*"
4853 FILES = ""
4854
4855 FILES_${PN} = "${bindir}/* ${sbindir}/* ${libexecdir}/* ${libdir}/lib*${SOLIBS} \
4856 ${sysconfdir} ${sharedstatedir} ${localstatedir} \
4857 ${base_bindir}/* ${base_sbindir}/* \
4858 ${base_libdir}/*${SOLIBS} \
4859 ${base_prefix}/lib/udev/rules.d ${prefix}/lib/udev/rules.d \
4860 ${datadir}/${BPN} ${libdir}/${BPN}/* \
4861 ${datadir}/pixmaps ${datadir}/applications \
4862 ${datadir}/idl ${datadir}/omf ${datadir}/sounds \
4863 ${libdir}/bonobo/servers"
4864
4865 FILES_${PN}-bin = "${bindir}/* ${sbindir}/*"
4866
4867 FILES_${PN}-doc = "${docdir} ${mandir} ${infodir} ${datadir}/gtk-doc \
4868 ${datadir}/gnome/help"
4869 SECTION_${PN}-doc = "doc"
4870
4871 FILES_SOLIBSDEV ?= "${base_libdir}/lib*${SOLIBSDEV} ${libdir}/lib*${SOLIBSDEV}"
4872 FILES_${PN}-dev = "${includedir} ${FILES_SOLIBSDEV} ${libdir}/*.la \
4873 ${libdir}/*.o ${libdir}/pkgconfig ${datadir}/pkgconfig \
4874 ${datadir}/aclocal ${base_libdir}/*.o \
4875 ${libdir}/${BPN}/*.la ${base_libdir}/*.la"
4876 SECTION_${PN}-dev = "devel"
4877 ALLOW_EMPTY_${PN}-dev = "1"
4878 RDEPENDS_${PN}-dev = "${PN} (= ${EXTENDPKGV})"
4879
4880 FILES_${PN}-staticdev = "${libdir}/*.a ${base_libdir}/*.a ${libdir}/${BPN}/*.a"
4881 SECTION_${PN}-staticdev = "devel"
4882 RDEPENDS_${PN}-staticdev = "${PN}-dev (= ${EXTENDPKGV})"
4883
4884.. _combining-multiple-versions-library-files-into-one-image:
4885
4886Combining Multiple Versions of Library Files into One Image
4887-----------------------------------------------------------
4888
4889The build system offers the ability to build libraries with different
4890target optimizations or architecture formats and combine these together
4891into one system image. You can link different binaries in the image
4892against the different libraries as needed for specific use cases. This
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004893feature is called "Multilib".
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004894
4895An example would be where you have most of a system compiled in 32-bit
4896mode using 32-bit libraries, but you have something large, like a
4897database engine, that needs to be a 64-bit application and uses 64-bit
4898libraries. Multilib allows you to get the best of both 32-bit and 64-bit
4899libraries.
4900
4901While the Multilib feature is most commonly used for 32 and 64-bit
4902differences, the approach the build system uses facilitates different
4903target optimizations. You could compile some binaries to use one set of
4904libraries and other binaries to use a different set of libraries. The
4905libraries could differ in architecture, compiler options, or other
4906optimizations.
4907
4908Several examples exist in the ``meta-skeleton`` layer found in the
4909:term:`Source Directory`:
4910
4911- ``conf/multilib-example.conf`` configuration file
4912
4913- ``conf/multilib-example2.conf`` configuration file
4914
4915- ``recipes-multilib/images/core-image-multilib-example.bb`` recipe
4916
4917Preparing to Use Multilib
4918~~~~~~~~~~~~~~~~~~~~~~~~~
4919
4920User-specific requirements drive the Multilib feature. Consequently,
4921there is no one "out-of-the-box" configuration that likely exists to
4922meet your needs.
4923
4924In order to enable Multilib, you first need to ensure your recipe is
4925extended to support multiple libraries. Many standard recipes are
4926already extended and support multiple libraries. You can check in the
4927``meta/conf/multilib.conf`` configuration file in the
4928:term:`Source Directory` to see how this is
4929done using the
4930:term:`BBCLASSEXTEND` variable.
4931Eventually, all recipes will be covered and this list will not be
4932needed.
4933
4934For the most part, the Multilib class extension works automatically to
4935extend the package name from ``${PN}`` to ``${MLPREFIX}${PN}``, where
4936``MLPREFIX`` is the particular multilib (e.g. "lib32-" or "lib64-").
4937Standard variables such as
4938:term:`DEPENDS`,
4939:term:`RDEPENDS`,
4940:term:`RPROVIDES`,
4941:term:`RRECOMMENDS`,
4942:term:`PACKAGES`, and
4943:term:`PACKAGES_DYNAMIC` are
4944automatically extended by the system. If you are extending any manual
4945code in the recipe, you can use the ``${MLPREFIX}`` variable to ensure
4946those names are extended correctly. This automatic extension code
4947resides in ``multilib.bbclass``.
4948
4949Using Multilib
4950~~~~~~~~~~~~~~
4951
4952After you have set up the recipes, you need to define the actual
4953combination of multiple libraries you want to build. You accomplish this
4954through your ``local.conf`` configuration file in the
4955:term:`Build Directory`. An example
4956configuration would be as follows:
4957::
4958
4959 MACHINE = "qemux86-64"
4960 require conf/multilib.conf
4961 MULTILIBS = "multilib:lib32"
4962 DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
4963 IMAGE_INSTALL_append = "lib32-glib-2.0"
4964
4965This example enables an additional library named
4966``lib32`` alongside the normal target packages. When combining these
4967"lib32" alternatives, the example uses "x86" for tuning. For information
4968on this particular tuning, see
4969``meta/conf/machine/include/ia32/arch-ia32.inc``.
4970
4971The example then includes ``lib32-glib-2.0`` in all the images, which
4972illustrates one method of including a multiple library dependency. You
4973can use a normal image build to include this dependency, for example:
4974::
4975
4976 $ bitbake core-image-sato
4977
4978You can also build Multilib packages
4979specifically with a command like this:
4980::
4981
4982 $ bitbake lib32-glib-2.0
4983
4984Additional Implementation Details
4985~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4986
4987Generic implementation details as well as details that are specific to
4988package management systems exist. Following are implementation details
4989that exist regardless of the package management system:
4990
4991- The typical convention used for the class extension code as used by
4992 Multilib assumes that all package names specified in
4993 :term:`PACKAGES` that contain
4994 ``${PN}`` have ``${PN}`` at the start of the name. When that
4995 convention is not followed and ``${PN}`` appears at the middle or the
4996 end of a name, problems occur.
4997
4998- The :term:`TARGET_VENDOR`
4999 value under Multilib will be extended to "-vendormlmultilib" (e.g.
5000 "-pokymllib32" for a "lib32" Multilib with Poky). The reason for this
5001 slightly unwieldy contraction is that any "-" characters in the
5002 vendor string presently break Autoconf's ``config.sub``, and other
5003 separators are problematic for different reasons.
5004
5005For the RPM Package Management System, the following implementation
5006details exist:
5007
5008- A unique architecture is defined for the Multilib packages, along
5009 with creating a unique deploy folder under ``tmp/deploy/rpm`` in the
5010 :term:`Build Directory`. For
5011 example, consider ``lib32`` in a ``qemux86-64`` image. The possible
5012 architectures in the system are "all", "qemux86_64",
5013 "lib32_qemux86_64", and "lib32_x86".
5014
5015- The ``${MLPREFIX}`` variable is stripped from ``${PN}`` during RPM
5016 packaging. The naming for a normal RPM package and a Multilib RPM
5017 package in a ``qemux86-64`` system resolves to something similar to
5018 ``bash-4.1-r2.x86_64.rpm`` and ``bash-4.1.r2.lib32_x86.rpm``,
5019 respectively.
5020
5021- When installing a Multilib image, the RPM backend first installs the
5022 base image and then installs the Multilib libraries.
5023
5024- The build system relies on RPM to resolve the identical files in the
5025 two (or more) Multilib packages.
5026
5027For the IPK Package Management System, the following implementation
5028details exist:
5029
5030- The ``${MLPREFIX}`` is not stripped from ``${PN}`` during IPK
5031 packaging. The naming for a normal RPM package and a Multilib IPK
5032 package in a ``qemux86-64`` system resolves to something like
5033 ``bash_4.1-r2.x86_64.ipk`` and ``lib32-bash_4.1-rw_x86.ipk``,
5034 respectively.
5035
5036- The IPK deploy folder is not modified with ``${MLPREFIX}`` because
5037 packages with and without the Multilib feature can exist in the same
5038 folder due to the ``${PN}`` differences.
5039
5040- IPK defines a sanity check for Multilib installation using certain
5041 rules for file comparison, overridden, etc.
5042
5043Installing Multiple Versions of the Same Library
5044------------------------------------------------
5045
5046Situations can exist where you need to install and use multiple versions
5047of the same library on the same system at the same time. These
5048situations almost always exist when a library API changes and you have
5049multiple pieces of software that depend on the separate versions of the
5050library. To accommodate these situations, you can install multiple
5051versions of the same library in parallel on the same system.
5052
5053The process is straightforward as long as the libraries use proper
5054versioning. With properly versioned libraries, all you need to do to
5055individually specify the libraries is create separate, appropriately
5056named recipes where the :term:`PN` part of
5057the name includes a portion that differentiates each library version
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005058(e.g. the major part of the version number). Thus, instead of having a
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005059single recipe that loads one version of a library (e.g. ``clutter``),
5060you provide multiple recipes that result in different versions of the
5061libraries you want. As an example, the following two recipes would allow
5062the two separate versions of the ``clutter`` library to co-exist on the
5063same system:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005064
5065.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005066
5067 clutter-1.6_1.6.20.bb
5068 clutter-1.8_1.8.4.bb
5069
5070Additionally, if
5071you have other recipes that depend on a given library, you need to use
5072the :term:`DEPENDS` variable to
5073create the dependency. Continuing with the same example, if you want to
5074have a recipe depend on the 1.8 version of the ``clutter`` library, use
5075the following in your recipe:
5076::
5077
5078 DEPENDS = "clutter-1.8"
5079
5080Using x32 psABI
5081===============
5082
5083x32 processor-specific Application Binary Interface (`x32
5084psABI <https://software.intel.com/en-us/node/628948>`__) is a native
508532-bit processor-specific ABI for Intel 64 (x86-64) architectures. An
5086ABI defines the calling conventions between functions in a processing
5087environment. The interface determines what registers are used and what
5088the sizes are for various C data types.
5089
5090Some processing environments prefer using 32-bit applications even when
5091running on Intel 64-bit platforms. Consider the i386 psABI, which is a
5092very old 32-bit ABI for Intel 64-bit platforms. The i386 psABI does not
5093provide efficient use and access of the Intel 64-bit processor
5094resources, leaving the system underutilized. Now consider the x86_64
5095psABI. This ABI is newer and uses 64-bits for data sizes and program
5096pointers. The extra bits increase the footprint size of the programs,
5097libraries, and also increases the memory and file system size
5098requirements. Executing under the x32 psABI enables user programs to
5099utilize CPU and system resources more efficiently while keeping the
5100memory footprint of the applications low. Extra bits are used for
5101registers but not for addressing mechanisms.
5102
5103The Yocto Project supports the final specifications of x32 psABI as
5104follows:
5105
5106- You can create packages and images in x32 psABI format on x86_64
5107 architecture targets.
5108
5109- You can successfully build recipes with the x32 toolchain.
5110
5111- You can create and boot ``core-image-minimal`` and
5112 ``core-image-sato`` images.
5113
5114- RPM Package Manager (RPM) support exists for x32 binaries.
5115
5116- Support for large images exists.
5117
5118To use the x32 psABI, you need to edit your ``conf/local.conf``
5119configuration file as follows:
5120::
5121
5122 MACHINE = "qemux86-64"
5123 DEFAULTTUNE = "x86-64-x32"
5124 baselib = "${@d.getVar('BASE_LIB_tune-' + (d.getVar('DEFAULTTUNE') \
5125 or 'INVALID')) or 'lib'}"
5126
5127Once you have set
5128up your configuration file, use BitBake to build an image that supports
5129the x32 psABI. Here is an example:
5130::
5131
5132 $ bitbake core-image-sato
5133
5134Enabling GObject Introspection Support
5135======================================
5136
5137`GObject
5138introspection <https://wiki.gnome.org/Projects/GObjectIntrospection>`__
5139is the standard mechanism for accessing GObject-based software from
5140runtime environments. GObject is a feature of the GLib library that
5141provides an object framework for the GNOME desktop and related software.
5142GObject Introspection adds information to GObject that allows objects
5143created within it to be represented across different programming
5144languages. If you want to construct GStreamer pipelines using Python, or
5145control UPnP infrastructure using Javascript and GUPnP, GObject
5146introspection is the only way to do it.
5147
5148This section describes the Yocto Project support for generating and
5149packaging GObject introspection data. GObject introspection data is a
5150description of the API provided by libraries built on top of GLib
5151framework, and, in particular, that framework's GObject mechanism.
5152GObject Introspection Repository (GIR) files go to ``-dev`` packages,
5153``typelib`` files go to main packages as they are packaged together with
5154libraries that are introspected.
5155
5156The data is generated when building such a library, by linking the
5157library with a small executable binary that asks the library to describe
5158itself, and then executing the binary and processing its output.
5159
5160Generating this data in a cross-compilation environment is difficult
5161because the library is produced for the target architecture, but its
5162code needs to be executed on the build host. This problem is solved with
5163the OpenEmbedded build system by running the code through QEMU, which
5164allows precisely that. Unfortunately, QEMU does not always work
5165perfectly as mentioned in the "`Known Issues <#known-issues>`__"
5166section.
5167
5168Enabling the Generation of Introspection Data
5169---------------------------------------------
5170
5171Enabling the generation of introspection data (GIR files) in your
5172library package involves the following:
5173
51741. Inherit the
5175 :ref:`gobject-introspection <ref-classes-gobject-introspection>`
5176 class.
5177
51782. Make sure introspection is not disabled anywhere in the recipe or
5179 from anything the recipe includes. Also, make sure that
5180 "gobject-introspection-data" is not in
5181 :term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`
5182 and that "qemu-usermode" is not in
5183 :term:`MACHINE_FEATURES_BACKFILL_CONSIDERED`.
5184 If either of these conditions exist, nothing will happen.
5185
51863. Try to build the recipe. If you encounter build errors that look like
5187 something is unable to find ``.so`` libraries, check where these
5188 libraries are located in the source tree and add the following to the
5189 recipe:
5190 ::
5191
5192 GIR_EXTRA_LIBS_PATH = "${B}/something/.libs"
5193
5194 .. note::
5195
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005196 See recipes in the ``oe-core`` repository that use that
5197 ``GIR_EXTRA_LIBS_PATH`` variable as an example.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005198
51994. Look for any other errors, which probably mean that introspection
5200 support in a package is not entirely standard, and thus breaks down
5201 in a cross-compilation environment. For such cases, custom-made fixes
5202 are needed. A good place to ask and receive help in these cases is
5203 the :ref:`Yocto Project mailing
5204 lists <resources-mailinglist>`.
5205
5206.. note::
5207
5208 Using a library that no longer builds against the latest Yocto
5209 Project release and prints introspection related errors is a good
5210 candidate for the previous procedure.
5211
5212Disabling the Generation of Introspection Data
5213----------------------------------------------
5214
5215You might find that you do not want to generate introspection data. Or,
5216perhaps QEMU does not work on your build host and target architecture
5217combination. If so, you can use either of the following methods to
5218disable GIR file generations:
5219
5220- Add the following to your distro configuration:
5221 ::
5222
5223 DISTRO_FEATURES_BACKFILL_CONSIDERED = "gobject-introspection-data"
5224
5225 Adding this statement disables generating introspection data using
5226 QEMU but will still enable building introspection tools and libraries
5227 (i.e. building them does not require the use of QEMU).
5228
5229- Add the following to your machine configuration:
5230 ::
5231
5232 MACHINE_FEATURES_BACKFILL_CONSIDERED = "qemu-usermode"
5233
5234 Adding this statement disables the use of QEMU when building packages for your
5235 machine. Currently, this feature is used only by introspection
5236 recipes and has the same effect as the previously described option.
5237
5238 .. note::
5239
5240 Future releases of the Yocto Project might have other features
5241 affected by this option.
5242
5243If you disable introspection data, you can still obtain it through other
5244means such as copying the data from a suitable sysroot, or by generating
5245it on the target hardware. The OpenEmbedded build system does not
5246currently provide specific support for these techniques.
5247
5248Testing that Introspection Works in an Image
5249--------------------------------------------
5250
5251Use the following procedure to test if generating introspection data is
5252working in an image:
5253
52541. Make sure that "gobject-introspection-data" is not in
5255 :term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`
5256 and that "qemu-usermode" is not in
5257 :term:`MACHINE_FEATURES_BACKFILL_CONSIDERED`.
5258
52592. Build ``core-image-sato``.
5260
52613. Launch a Terminal and then start Python in the terminal.
5262
52634. Enter the following in the terminal:
5264 ::
5265
5266 >>> from gi.repository import GLib
5267 >>> GLib.get_host_name()
5268
52695. For something a little more advanced, enter the following see:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005270 https://python-gtk-3-tutorial.readthedocs.io/en/latest/introduction.html
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005271
5272Known Issues
5273------------
5274
5275The following know issues exist for GObject Introspection Support:
5276
5277- ``qemu-ppc64`` immediately crashes. Consequently, you cannot build
5278 introspection data on that architecture.
5279
5280- x32 is not supported by QEMU. Consequently, introspection data is
5281 disabled.
5282
5283- musl causes transient GLib binaries to crash on assertion failures.
5284 Consequently, generating introspection data is disabled.
5285
5286- Because QEMU is not able to run the binaries correctly, introspection
5287 is disabled for some specific packages under specific architectures
5288 (e.g. ``gcr``, ``libsecret``, and ``webkit``).
5289
5290- QEMU usermode might not work properly when running 64-bit binaries
5291 under 32-bit host machines. In particular, "qemumips64" is known to
5292 not work under i686.
5293
5294.. _dev-optionally-using-an-external-toolchain:
5295
5296Optionally Using an External Toolchain
5297======================================
5298
5299You might want to use an external toolchain as part of your development.
5300If this is the case, the fundamental steps you need to accomplish are as
5301follows:
5302
5303- Understand where the installed toolchain resides. For cases where you
5304 need to build the external toolchain, you would need to take separate
5305 steps to build and install the toolchain.
5306
5307- Make sure you add the layer that contains the toolchain to your
5308 ``bblayers.conf`` file through the
5309 :term:`BBLAYERS` variable.
5310
5311- Set the ``EXTERNAL_TOOLCHAIN`` variable in your ``local.conf`` file
5312 to the location in which you installed the toolchain.
5313
5314A good example of an external toolchain used with the Yocto Project is
5315Mentor Graphics Sourcery G++ Toolchain. You can see information on how
5316to use that particular layer in the ``README`` file at
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005317https://github.com/MentorEmbedded/meta-sourcery/. You can find
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005318further information by reading about the
5319:term:`TCMODE` variable in the Yocto
5320Project Reference Manual's variable glossary.
5321
5322Creating Partitioned Images Using Wic
5323=====================================
5324
5325Creating an image for a particular hardware target using the
5326OpenEmbedded build system does not necessarily mean you can boot that
5327image as is on your device. Physical devices accept and boot images in
5328various ways depending on the specifics of the device. Usually,
5329information about the hardware can tell you what image format the device
5330requires. Should your device require multiple partitions on an SD card,
5331flash, or an HDD, you can use the OpenEmbedded Image Creator, Wic, to
5332create the properly partitioned image.
5333
5334The ``wic`` command generates partitioned images from existing
5335OpenEmbedded build artifacts. Image generation is driven by partitioning
5336commands contained in an Openembedded kickstart file (``.wks``)
5337specified either directly on the command line or as one of a selection
5338of canned kickstart files as shown with the ``wic list images`` command
5339in the "`Using an Existing Kickstart
5340File <#using-a-provided-kickstart-file>`__" section. When you apply the
5341command to a given set of build artifacts, the result is an image or set
5342of images that can be directly written onto media and used on a
5343particular system.
5344
5345.. note::
5346
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005347 For a kickstart file reference, see the
5348 ":ref:`ref-manual/ref-kickstart:openembedded kickstart (\`\`.wks\`\`) reference`"
5349 Chapter in the Yocto Project Reference Manual.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005350
5351The ``wic`` command and the infrastructure it is based on is by
5352definition incomplete. The purpose of the command is to allow the
5353generation of customized images, and as such, was designed to be
5354completely extensible through a plugin interface. See the "`Using the
5355Wic PlugIn Interface <#wic-using-the-wic-plugin-interface>`__" section
5356for information on these plugins.
5357
5358This section provides some background information on Wic, describes what
5359you need to have in place to run the tool, provides instruction on how
5360to use the Wic utility, provides information on using the Wic plugins
5361interface, and provides several examples that show how to use Wic.
5362
5363.. _wic-background:
5364
5365Background
5366----------
5367
5368This section provides some background on the Wic utility. While none of
5369this information is required to use Wic, you might find it interesting.
5370
5371- The name "Wic" is derived from OpenEmbedded Image Creator (oeic). The
5372 "oe" diphthong in "oeic" was promoted to the letter "w", because
5373 "oeic" is both difficult to remember and to pronounce.
5374
5375- Wic is loosely based on the Meego Image Creator (``mic``) framework.
5376 The Wic implementation has been heavily modified to make direct use
5377 of OpenEmbedded build artifacts instead of package installation and
5378 configuration, which are already incorporated within the OpenEmbedded
5379 artifacts.
5380
5381- Wic is a completely independent standalone utility that initially
5382 provides easier-to-use and more flexible replacements for an existing
5383 functionality in OE-Core's
5384 :ref:`image-live <ref-classes-image-live>`
5385 class. The difference between Wic and those examples is that with Wic
5386 the functionality of those scripts is implemented by a
5387 general-purpose partitioning language, which is based on Redhat
5388 kickstart syntax.
5389
5390.. _wic-requirements:
5391
5392Requirements
5393------------
5394
5395In order to use the Wic utility with the OpenEmbedded Build system, your
5396system needs to meet the following requirements:
5397
5398- The Linux distribution on your development host must support the
5399 Yocto Project. See the ":ref:`detailed-supported-distros`"
5400 section in the Yocto Project Reference Manual for the list of
5401 distributions that support the Yocto Project.
5402
5403- The standard system utilities, such as ``cp``, must be installed on
5404 your development host system.
5405
5406- You must have sourced the build environment setup script (i.e.
5407 :ref:`structure-core-script`) found in the
5408 :term:`Build Directory`.
5409
5410- You need to have the build artifacts already available, which
5411 typically means that you must have already created an image using the
5412 Openembedded build system (e.g. ``core-image-minimal``). While it
5413 might seem redundant to generate an image in order to create an image
5414 using Wic, the current version of Wic requires the artifacts in the
5415 form generated by the OpenEmbedded build system.
5416
5417- You must build several native tools, which are built to run on the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005418 build system:
5419 ::
5420
5421 $ bitbake parted-native dosfstools-native mtools-native
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005422
5423- Include "wic" as part of the
5424 :term:`IMAGE_FSTYPES`
5425 variable.
5426
5427- Include the name of the :ref:`wic kickstart file <openembedded-kickstart-wks-reference>`
5428 as part of the :term:`WKS_FILE` variable
5429
5430.. _wic-getting-help:
5431
5432Getting Help
5433------------
5434
5435You can get general help for the ``wic`` command by entering the ``wic``
5436command by itself or by entering the command with a help argument as
5437follows:
5438::
5439
5440 $ wic -h
5441 $ wic --help
5442 $ wic help
5443
5444Currently, Wic supports seven commands: ``cp``, ``create``, ``help``,
5445``list``, ``ls``, ``rm``, and ``write``. You can get help for all these
5446commands except "help" by using the following form:
5447::
5448
5449 $ wic help command
5450
5451For example, the following command returns help for the ``write``
5452command:
5453::
5454
5455 $ wic help write
5456
5457Wic supports help for three topics: ``overview``, ``plugins``, and
5458``kickstart``. You can get help for any topic using the following form:
5459::
5460
5461 $ wic help topic
5462
5463For example, the following returns overview help for Wic:
5464::
5465
5466 $ wic help overview
5467
5468One additional level of help exists for Wic. You can get help on
5469individual images through the ``list`` command. You can use the ``list``
5470command to return the available Wic images as follows:
5471::
5472
5473 $ wic list images
5474 genericx86 Create an EFI disk image for genericx86*
5475 beaglebone-yocto Create SD card image for Beaglebone
5476 edgerouter Create SD card image for Edgerouter
5477 qemux86-directdisk Create a qemu machine 'pcbios' direct disk image
5478 directdisk-gpt Create a 'pcbios' direct disk image
5479 mkefidisk Create an EFI disk image
5480 directdisk Create a 'pcbios' direct disk image
5481 systemd-bootdisk Create an EFI disk image with systemd-boot
5482 mkhybridiso Create a hybrid ISO image
5483 sdimage-bootpart Create SD card image with a boot partition
5484 directdisk-multi-rootfs Create multi rootfs image using rootfs plugin
5485 directdisk-bootloader-config Create a 'pcbios' direct disk image with custom bootloader config
5486
5487Once you know the list of available
5488Wic images, you can use ``help`` with the command to get help on a
5489particular image. For example, the following command returns help on the
5490"beaglebone-yocto" image:
5491::
5492
5493 $ wic list beaglebone-yocto help
5494
5495 Creates a partitioned SD card image for Beaglebone.
5496 Boot files are located in the first vfat partition.
5497
5498Operational Modes
5499-----------------
5500
5501You can use Wic in two different modes, depending on how much control
5502you need for specifying the Openembedded build artifacts that are used
5503for creating the image: Raw and Cooked:
5504
5505- *Raw Mode:* You explicitly specify build artifacts through Wic
5506 command-line arguments.
5507
5508- *Cooked Mode:* The current
5509 :term:`MACHINE` setting and image
5510 name are used to automatically locate and provide the build
5511 artifacts. You just supply a kickstart file and the name of the image
5512 from which to use artifacts.
5513
5514Regardless of the mode you use, you need to have the build artifacts
5515ready and available.
5516
5517Raw Mode
5518~~~~~~~~
5519
5520Running Wic in raw mode allows you to specify all the partitions through
5521the ``wic`` command line. The primary use for raw mode is if you have
5522built your kernel outside of the Yocto Project
5523:term:`Build Directory`. In other words, you
5524can point to arbitrary kernel, root filesystem locations, and so forth.
5525Contrast this behavior with cooked mode where Wic looks in the Build
5526Directory (e.g. ``tmp/deploy/images/``\ machine).
5527
5528The general form of the ``wic`` command in raw mode is:
5529::
5530
5531 $ wic create wks_file options ...
5532
5533 Where:
5534
5535 wks_file:
5536 An OpenEmbedded kickstart file. You can provide
5537 your own custom file or use a file from a set of
5538 existing files as described by further options.
5539
5540 optional arguments:
5541 -h, --help show this help message and exit
5542 -o OUTDIR, --outdir OUTDIR
5543 name of directory to create image in
5544 -e IMAGE_NAME, --image-name IMAGE_NAME
5545 name of the image to use the artifacts from e.g. core-
5546 image-sato
5547 -r ROOTFS_DIR, --rootfs-dir ROOTFS_DIR
5548 path to the /rootfs dir to use as the .wks rootfs
5549 source
5550 -b BOOTIMG_DIR, --bootimg-dir BOOTIMG_DIR
5551 path to the dir containing the boot artifacts (e.g.
5552 /EFI or /syslinux dirs) to use as the .wks bootimg
5553 source
5554 -k KERNEL_DIR, --kernel-dir KERNEL_DIR
5555 path to the dir containing the kernel to use in the
5556 .wks bootimg
5557 -n NATIVE_SYSROOT, --native-sysroot NATIVE_SYSROOT
5558 path to the native sysroot containing the tools to use
5559 to build the image
5560 -s, --skip-build-check
5561 skip the build check
5562 -f, --build-rootfs build rootfs
5563 -c {gzip,bzip2,xz}, --compress-with {gzip,bzip2,xz}
5564 compress image with specified compressor
5565 -m, --bmap generate .bmap
5566 --no-fstab-update Do not change fstab file.
5567 -v VARS_DIR, --vars VARS_DIR
5568 directory with <image>.env files that store bitbake
5569 variables
5570 -D, --debug output debug information
5571
5572.. note::
5573
5574 You do not need root privileges to run Wic. In fact, you should not
5575 run as root when using the utility.
5576
5577Cooked Mode
5578~~~~~~~~~~~
5579
5580Running Wic in cooked mode leverages off artifacts in the Build
5581Directory. In other words, you do not have to specify kernel or root
5582filesystem locations as part of the command. All you need to provide is
5583a kickstart file and the name of the image from which to use artifacts
5584by using the "-e" option. Wic looks in the Build Directory (e.g.
5585``tmp/deploy/images/``\ machine) for artifacts.
5586
5587The general form of the ``wic`` command using Cooked Mode is as follows:
5588::
5589
5590 $ wic create wks_file -e IMAGE_NAME
5591
5592 Where:
5593
5594 wks_file:
5595 An OpenEmbedded kickstart file. You can provide
5596 your own custom file or use a file from a set of
5597 existing files provided with the Yocto Project
5598 release.
5599
5600 required argument:
5601 -e IMAGE_NAME, --image-name IMAGE_NAME
5602 name of the image to use the artifacts from e.g. core-
5603 image-sato
5604
5605.. _using-a-provided-kickstart-file:
5606
5607Using an Existing Kickstart File
5608--------------------------------
5609
5610If you do not want to create your own kickstart file, you can use an
5611existing file provided by the Wic installation. As shipped, kickstart
5612files can be found in the :ref:`overview-manual/overview-manual-development-environment:yocto project source repositories` in the
5613following two locations:
5614::
5615
5616 poky/meta-yocto-bsp/wic
5617 poky/scripts/lib/wic/canned-wks
5618
5619Use the following command to list the available kickstart files:
5620::
5621
5622 $ wic list images
5623 genericx86 Create an EFI disk image for genericx86*
5624 beaglebone-yocto Create SD card image for Beaglebone
5625 edgerouter Create SD card image for Edgerouter
5626 qemux86-directdisk Create a qemu machine 'pcbios' direct disk image
5627 directdisk-gpt Create a 'pcbios' direct disk image
5628 mkefidisk Create an EFI disk image
5629 directdisk Create a 'pcbios' direct disk image
5630 systemd-bootdisk Create an EFI disk image with systemd-boot
5631 mkhybridiso Create a hybrid ISO image
5632 sdimage-bootpart Create SD card image with a boot partition
5633 directdisk-multi-rootfs Create multi rootfs image using rootfs plugin
5634 directdisk-bootloader-config Create a 'pcbios' direct disk image with custom bootloader config
5635
5636When you use an existing file, you
5637do not have to use the ``.wks`` extension. Here is an example in Raw
5638Mode that uses the ``directdisk`` file:
5639::
5640
5641 $ wic create directdisk -r rootfs_dir -b bootimg_dir \
5642 -k kernel_dir -n native_sysroot
5643
5644Here are the actual partition language commands used in the
5645``genericx86.wks`` file to generate an image:
5646::
5647
5648 # short-description: Create an EFI disk image for genericx86*
5649 # long-description: Creates a partitioned EFI disk image for genericx86* machines
5650 part /boot --source bootimg-efi --sourceparams="loader=grub-efi" --ondisk sda --label msdos --active --align 1024
5651 part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 1024 --use-uuid
5652 part swap --ondisk sda --size 44 --label swap1 --fstype=swap
5653
5654 bootloader --ptable gpt --timeout=5 --append="rootfstype=ext4 console=ttyS0,115200 console=tty0"
5655
5656.. _wic-using-the-wic-plugin-interface:
5657
5658Using the Wic Plugin Interface
5659------------------------------
5660
5661You can extend and specialize Wic functionality by using Wic plugins.
5662This section explains the Wic plugin interface.
5663
5664.. note::
5665
5666 Wic plugins consist of "source" and "imager" plugins. Imager plugins
5667 are beyond the scope of this section.
5668
5669Source plugins provide a mechanism to customize partition content during
5670the Wic image generation process. You can use source plugins to map
5671values that you specify using ``--source`` commands in kickstart files
5672(i.e. ``*.wks``) to a plugin implementation used to populate a given
5673partition.
5674
5675.. note::
5676
5677 If you use plugins that have build-time dependencies (e.g. native
5678 tools, bootloaders, and so forth) when building a Wic image, you need
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005679 to specify those dependencies using the :term:`WKS_FILE_DEPENDS`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005680 variable.
5681
5682Source plugins are subclasses defined in plugin files. As shipped, the
5683Yocto Project provides several plugin files. You can see the source
5684plugin files that ship with the Yocto Project
5685:yocto_git:`here </cgit/cgit.cgi/poky/tree/scripts/lib/wic/plugins/source>`.
5686Each of these plugin files contains source plugins that are designed to
5687populate a specific Wic image partition.
5688
5689Source plugins are subclasses of the ``SourcePlugin`` class, which is
5690defined in the ``poky/scripts/lib/wic/pluginbase.py`` file. For example,
5691the ``BootimgEFIPlugin`` source plugin found in the ``bootimg-efi.py``
5692file is a subclass of the ``SourcePlugin`` class, which is found in the
5693``pluginbase.py`` file.
5694
5695You can also implement source plugins in a layer outside of the Source
5696Repositories (external layer). To do so, be sure that your plugin files
5697are located in a directory whose path is
5698``scripts/lib/wic/plugins/source/`` within your external layer. When the
5699plugin files are located there, the source plugins they contain are made
5700available to Wic.
5701
5702When the Wic implementation needs to invoke a partition-specific
5703implementation, it looks for the plugin with the same name as the
5704``--source`` parameter used in the kickstart file given to that
5705partition. For example, if the partition is set up using the following
5706command in a kickstart file:
5707::
5708
5709 part /boot --source bootimg-pcbios --ondisk sda --label boot --active --align 1024
5710
5711The methods defined as class
5712members of the matching source plugin (i.e. ``bootimg-pcbios``) in the
5713``bootimg-pcbios.py`` plugin file are used.
5714
5715To be more concrete, here is the corresponding plugin definition from
5716the ``bootimg-pcbios.py`` file for the previous command along with an
5717example method called by the Wic implementation when it needs to prepare
5718a partition using an implementation-specific function:
5719::
5720
5721 .
5722 .
5723 .
5724 class BootimgPcbiosPlugin(SourcePlugin):
5725 """
5726 Create MBR boot partition and install syslinux on it.
5727 """
5728
5729 name = 'bootimg-pcbios'
5730 .
5731 .
5732 .
5733 @classmethod
5734 def do_prepare_partition(cls, part, source_params, creator, cr_workdir,
5735 oe_builddir, bootimg_dir, kernel_dir,
5736 rootfs_dir, native_sysroot):
5737 """
5738 Called to do the actual content population for a partition i.e. it
5739 'prepares' the partition to be incorporated into the image.
5740 In this case, prepare content for legacy bios boot partition.
5741 """
5742 .
5743 .
5744 .
5745
5746If a
5747subclass (plugin) itself does not implement a particular function, Wic
5748locates and uses the default version in the superclass. It is for this
5749reason that all source plugins are derived from the ``SourcePlugin``
5750class.
5751
5752The ``SourcePlugin`` class defined in the ``pluginbase.py`` file defines
5753a set of methods that source plugins can implement or override. Any
5754plugins (subclass of ``SourcePlugin``) that do not implement a
5755particular method inherit the implementation of the method from the
5756``SourcePlugin`` class. For more information, see the ``SourcePlugin``
5757class in the ``pluginbase.py`` file for details:
5758
5759The following list describes the methods implemented in the
5760``SourcePlugin`` class:
5761
5762- ``do_prepare_partition()``: Called to populate a partition with
5763 actual content. In other words, the method prepares the final
5764 partition image that is incorporated into the disk image.
5765
5766- ``do_configure_partition()``: Called before
5767 ``do_prepare_partition()`` to create custom configuration files for a
5768 partition (e.g. syslinux or grub configuration files).
5769
5770- ``do_install_disk()``: Called after all partitions have been
5771 prepared and assembled into a disk image. This method provides a hook
5772 to allow finalization of a disk image (e.g. writing an MBR).
5773
5774- ``do_stage_partition()``: Special content-staging hook called
5775 before ``do_prepare_partition()``. This method is normally empty.
5776
5777 Typically, a partition just uses the passed-in parameters (e.g. the
5778 unmodified value of ``bootimg_dir``). However, in some cases, things
5779 might need to be more tailored. As an example, certain files might
5780 additionally need to be taken from ``bootimg_dir + /boot``. This hook
5781 allows those files to be staged in a customized fashion.
5782
5783 .. note::
5784
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005785 ``get_bitbake_var()`` allows you to access non-standard variables that
5786 you might want to use for this behavior.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005787
5788You can extend the source plugin mechanism. To add more hooks, create
5789more source plugin methods within ``SourcePlugin`` and the corresponding
5790derived subclasses. The code that calls the plugin methods uses the
5791``plugin.get_source_plugin_methods()`` function to find the method or
5792methods needed by the call. Retrieval of those methods is accomplished
5793by filling up a dict with keys that contain the method names of
5794interest. On success, these will be filled in with the actual methods.
5795See the Wic implementation for examples and details.
5796
5797.. _wic-usage-examples:
5798
5799Wic Examples
5800------------
5801
5802This section provides several examples that show how to use the Wic
5803utility. All the examples assume the list of requirements in the
5804"`Requirements <#wic-requirements>`__" section have been met. The
5805examples assume the previously generated image is
5806``core-image-minimal``.
5807
5808.. _generate-an-image-using-a-provided-kickstart-file:
5809
5810Generate an Image using an Existing Kickstart File
5811~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5812
5813This example runs in Cooked Mode and uses the ``mkefidisk`` kickstart
5814file:
5815::
5816
5817 $ wic create mkefidisk -e core-image-minimal
5818 INFO: Building wic-tools...
5819 .
5820 .
5821 .
5822 INFO: The new image(s) can be found here:
5823 ./mkefidisk-201804191017-sda.direct
5824
5825 The following build artifacts were used to create the image(s):
5826 ROOTFS_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
5827 BOOTIMG_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
5828 KERNEL_DIR: /home/stephano/build/master/build/tmp-glibc/deploy/images/qemux86
5829 NATIVE_SYSROOT: /home/stephano/build/master/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
5830
5831 INFO: The image(s) were created using OE kickstart file:
5832 /home/stephano/build/master/openembedded-core/scripts/lib/wic/canned-wks/mkefidisk.wks
5833
5834The previous example shows the easiest way to create an image by running
5835in cooked mode and supplying a kickstart file and the "-e" option to
5836point to the existing build artifacts. Your ``local.conf`` file needs to
5837have the :term:`MACHINE` variable set
5838to the machine you are using, which is "qemux86" in this example.
5839
5840Once the image builds, the output provides image location, artifact use,
5841and kickstart file information.
5842
5843.. note::
5844
5845 You should always verify the details provided in the output to make
5846 sure that the image was indeed created exactly as expected.
5847
5848Continuing with the example, you can now write the image from the Build
5849Directory onto a USB stick, or whatever media for which you built your
5850image, and boot from the media. You can write the image by using
5851``bmaptool`` or ``dd``:
5852::
5853
5854 $ oe-run-native bmaptool copy mkefidisk-201804191017-sda.direct /dev/sdX
5855
5856or ::
5857
5858 $ sudo dd if=mkefidisk-201804191017-sda.direct of=/dev/sdX
5859
5860.. note::
5861
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005862 For more information on how to use the ``bmaptool``
5863 to flash a device with an image, see the
5864 ":ref:`dev-manual/dev-manual-common-tasks:flashing images using \`\`bmaptool\`\``"
5865 section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005866
5867Using a Modified Kickstart File
5868~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5869
5870Because partitioned image creation is driven by the kickstart file, it
5871is easy to affect image creation by changing the parameters in the file.
5872This next example demonstrates that through modification of the
5873``directdisk-gpt`` kickstart file.
5874
5875As mentioned earlier, you can use the command ``wic list images`` to
5876show the list of existing kickstart files. The directory in which the
5877``directdisk-gpt.wks`` file resides is
5878``scripts/lib/image/canned-wks/``, which is located in the
5879:term:`Source Directory` (e.g. ``poky``).
5880Because available files reside in this directory, you can create and add
5881your own custom files to the directory. Subsequent use of the
5882``wic list images`` command would then include your kickstart files.
5883
5884In this example, the existing ``directdisk-gpt`` file already does most
5885of what is needed. However, for the hardware in this example, the image
5886will need to boot from ``sdb`` instead of ``sda``, which is what the
5887``directdisk-gpt`` kickstart file uses.
5888
5889The example begins by making a copy of the ``directdisk-gpt.wks`` file
5890in the ``scripts/lib/image/canned-wks`` directory and then by changing
5891the lines that specify the target disk from which to boot.
5892::
5893
5894 $ cp /home/stephano/poky/scripts/lib/wic/canned-wks/directdisk-gpt.wks \
5895 /home/stephano/poky/scripts/lib/wic/canned-wks/directdisksdb-gpt.wks
5896
5897Next, the example modifies the ``directdisksdb-gpt.wks`` file and
5898changes all instances of "``--ondisk sda``" to "``--ondisk sdb``". The
5899example changes the following two lines and leaves the remaining lines
5900untouched:
5901::
5902
5903 part /boot --source bootimg-pcbios --ondisk sdb --label boot --active --align 1024
5904 part / --source rootfs --ondisk sdb --fstype=ext4 --label platform --align 1024 --use-uuid
5905
5906Once the lines are changed, the
5907example generates the ``directdisksdb-gpt`` image. The command points
5908the process at the ``core-image-minimal`` artifacts for the Next Unit of
5909Computing (nuc) :term:`MACHINE` the
5910``local.conf``.
5911::
5912
5913 $ wic create directdisksdb-gpt -e core-image-minimal
5914 INFO: Building wic-tools...
5915 .
5916 .
5917 .
5918 Initialising tasks: 100% |#######################################| Time: 0:00:01
5919 NOTE: Executing SetScene Tasks
5920 NOTE: Executing RunQueue Tasks
5921 NOTE: Tasks Summary: Attempted 1161 tasks of which 1157 didn't need to be rerun and all succeeded.
5922 INFO: Creating image(s)...
5923
5924 INFO: The new image(s) can be found here:
5925 ./directdisksdb-gpt-201710090938-sdb.direct
5926
5927 The following build artifacts were used to create the image(s):
5928 ROOTFS_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
5929 BOOTIMG_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
5930 KERNEL_DIR: /home/stephano/build/master/build/tmp-glibc/deploy/images/qemux86
5931 NATIVE_SYSROOT: /home/stephano/build/master/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
5932
5933 INFO: The image(s) were created using OE kickstart file:
5934 /home/stephano/poky/scripts/lib/wic/canned-wks/directdisksdb-gpt.wks
5935
5936Continuing with the example, you can now directly ``dd`` the image to a
5937USB stick, or whatever media for which you built your image, and boot
5938the resulting media:
5939::
5940
5941 $ sudo dd if=directdisksdb-gpt-201710090938-sdb.direct of=/dev/sdb
5942 140966+0 records in
5943 140966+0 records out
5944 72174592 bytes (72 MB, 69 MiB) copied, 78.0282 s, 925 kB/s
5945 $ sudo eject /dev/sdb
5946
5947Using a Modified Kickstart File and Running in Raw Mode
5948~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5949
5950This next example manually specifies each build artifact (runs in Raw
5951Mode) and uses a modified kickstart file. The example also uses the
5952``-o`` option to cause Wic to create the output somewhere other than the
5953default output directory, which is the current directory:
5954::
5955
5956 $ wic create /home/stephano/my_yocto/test.wks -o /home/stephano/testwic \
5957 --rootfs-dir /home/stephano/build/master/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/rootfs \
5958 --bootimg-dir /home/stephano/build/master/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share \
5959 --kernel-dir /home/stephano/build/master/build/tmp/deploy/images/qemux86 \
5960 --native-sysroot /home/stephano/build/master/build/tmp/work/i586-poky-linux/wic-tools/1.0-r0/recipe-sysroot-native
5961
5962 INFO: Creating image(s)...
5963
5964 INFO: The new image(s) can be found here:
5965 /home/stephano/testwic/test-201710091445-sdb.direct
5966
5967 The following build artifacts were used to create the image(s):
5968 ROOTFS_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
5969 BOOTIMG_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
5970 KERNEL_DIR: /home/stephano/build/master/build/tmp-glibc/deploy/images/qemux86
5971 NATIVE_SYSROOT: /home/stephano/build/master/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
5972
5973 INFO: The image(s) were created using OE kickstart file:
5974 /home/stephano/my_yocto/test.wks
5975
5976For this example,
5977:term:`MACHINE` did not have to be
5978specified in the ``local.conf`` file since the artifact is manually
5979specified.
5980
5981Using Wic to Manipulate an Image
5982~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5983
5984Wic image manipulation allows you to shorten turnaround time during
5985image development. For example, you can use Wic to delete the kernel
5986partition of a Wic image and then insert a newly built kernel. This
5987saves you time from having to rebuild the entire image each time you
5988modify the kernel.
5989
5990.. note::
5991
5992 In order to use Wic to manipulate a Wic image as in this example,
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005993 your development machine must have the ``mtools`` package installed.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005994
5995The following example examines the contents of the Wic image, deletes
5996the existing kernel, and then inserts a new kernel:
5997
59981. *List the Partitions:* Use the ``wic ls`` command to list all the
5999 partitions in the Wic image:
6000 ::
6001
6002 $ wic ls tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic
6003 Num Start End Size Fstype
6004 1 1048576 25041919 23993344 fat16
6005 2 25165824 72157183 46991360 ext4
6006
6007 The previous output shows two partitions in the
6008 ``core-image-minimal-qemux86.wic`` image.
6009
60102. *Examine a Particular Partition:* Use the ``wic ls`` command again
6011 but in a different form to examine a particular partition.
6012
6013 .. note::
6014
6015 You can get command usage on any Wic command using the following
6016 form:
6017 ::
6018
6019 $ wic help command
6020
6021
6022 For example, the following command shows you the various ways to
6023 use the
6024 wic ls
6025 command:
6026 ::
6027
6028 $ wic help ls
6029
6030
6031 The following command shows what is in Partition one:
6032 ::
6033
6034 $ wic ls tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1
6035 Volume in drive : is boot
6036 Volume Serial Number is E894-1809
6037 Directory for ::/
6038
6039 libcom32 c32 186500 2017-10-09 16:06
6040 libutil c32 24148 2017-10-09 16:06
6041 syslinux cfg 220 2017-10-09 16:06
6042 vesamenu c32 27104 2017-10-09 16:06
6043 vmlinuz 6904608 2017-10-09 16:06
6044 5 files 7 142 580 bytes
6045 16 582 656 bytes free
6046
6047 The previous output shows five files, with the
6048 ``vmlinuz`` being the kernel.
6049
6050 .. note::
6051
6052 If you see the following error, you need to update or create a
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006053 ``~/.mtoolsrc`` file and be sure to have the line "mtools_skip_check=1"
6054 in the file. Then, run the Wic command again:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006055 ::
6056
6057 ERROR: _exec_cmd: /usr/bin/mdir -i /tmp/wic-parttfokuwra ::/ returned '1' instead of 0
6058 output: Total number of sectors (47824) not a multiple of sectors per track (32)!
6059 Add mtools_skip_check=1 to your .mtoolsrc file to skip this test
6060
6061
60623. *Remove the Old Kernel:* Use the ``wic rm`` command to remove the
6063 ``vmlinuz`` file (kernel):
6064 ::
6065
6066 $ wic rm tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1/vmlinuz
6067
60684. *Add In the New Kernel:* Use the ``wic cp`` command to add the
6069 updated kernel to the Wic image. Depending on how you built your
6070 kernel, it could be in different places. If you used ``devtool`` and
6071 an SDK to build your kernel, it resides in the ``tmp/work`` directory
6072 of the extensible SDK. If you used ``make`` to build the kernel, the
6073 kernel will be in the ``workspace/sources`` area.
6074
6075 The following example assumes ``devtool`` was used to build the
6076 kernel:
6077 ::
6078
6079 cp ~/poky_sdk/tmp/work/qemux86-poky-linux/linux-yocto/4.12.12+git999-r0/linux-yocto-4.12.12+git999/arch/x86/boot/bzImage \
6080 ~/poky/build/tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1/vmlinuz
6081
6082 Once the new kernel is added back into the image, you can use the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006083 ``dd`` command or :ref:`bmaptool
6084 <dev-manual/dev-manual-common-tasks:flashing images using \`\`bmaptool\`\`>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006085 to flash your wic image onto an SD card or USB stick and test your
6086 target.
6087
6088 .. note::
6089
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006090 Using ``bmaptool`` is generally 10 to 20 times faster than using ``dd``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006091
6092Flashing Images Using ``bmaptool``
6093==================================
6094
6095A fast and easy way to flash an image to a bootable device is to use
6096Bmaptool, which is integrated into the OpenEmbedded build system.
6097Bmaptool is a generic tool that creates a file's block map (bmap) and
6098then uses that map to copy the file. As compared to traditional tools
6099such as dd or cp, Bmaptool can copy (or flash) large files like raw
6100system image files much faster.
6101
6102.. note::
6103
6104 - If you are using Ubuntu or Debian distributions, you can install
6105 the ``bmap-tools`` package using the following command and then
6106 use the tool without specifying ``PATH`` even from the root
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006107 account:
6108 ::
6109
6110 $ sudo apt-get install bmap-tools
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006111
6112 - If you are unable to install the ``bmap-tools`` package, you will
6113 need to build Bmaptool before using it. Use the following command:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006114 ::
6115
6116 $ bitbake bmap-tools-native
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006117
6118Following, is an example that shows how to flash a Wic image. Realize
6119that while this example uses a Wic image, you can use Bmaptool to flash
6120any type of image. Use these steps to flash an image using Bmaptool:
6121
61221. *Update your local.conf File:* You need to have the following set
6123 in your ``local.conf`` file before building your image:
6124 ::
6125
6126 IMAGE_FSTYPES += "wic wic.bmap"
6127
61282. *Get Your Image:* Either have your image ready (pre-built with the
6129 :term:`IMAGE_FSTYPES`
6130 setting previously mentioned) or take the step to build the image:
6131 ::
6132
6133 $ bitbake image
6134
61353. *Flash the Device:* Flash the device with the image by using Bmaptool
6136 depending on your particular setup. The following commands assume the
6137 image resides in the Build Directory's ``deploy/images/`` area:
6138
6139 - If you have write access to the media, use this command form:
6140 ::
6141
6142 $ oe-run-native bmap-tools-native bmaptool copy build-directory/tmp/deploy/images/machine/image.wic /dev/sdX
6143
6144 - If you do not have write access to the media, set your permissions
6145 first and then use the same command form:
6146 ::
6147
6148 $ sudo chmod 666 /dev/sdX
6149 $ oe-run-native bmap-tools-native bmaptool copy build-directory/tmp/deploy/images/machine/image.wic /dev/sdX
6150
6151For help on the ``bmaptool`` command, use the following command:
6152::
6153
6154 $ bmaptool --help
6155
6156Making Images More Secure
6157=========================
6158
6159Security is of increasing concern for embedded devices. Consider the
6160issues and problems discussed in just this sampling of work found across
6161the Internet:
6162
6163- *"*\ `Security Risks of Embedded
6164 Systems <https://www.schneier.com/blog/archives/2014/01/security_risks_9.html>`__\ *"*
6165 by Bruce Schneier
6166
6167- *"*\ `Internet Census
6168 2012 <http://census2012.sourceforge.net/paper.html>`__\ *"* by Carna
6169 Botnet
6170
6171- *"*\ `Security Issues for Embedded
6172 Devices <http://elinux.org/images/6/6f/Security-issues.pdf>`__\ *"*
6173 by Jake Edge
6174
6175When securing your image is of concern, there are steps, tools, and
6176variables that you can consider to help you reach the security goals you
6177need for your particular device. Not all situations are identical when
6178it comes to making an image secure. Consequently, this section provides
6179some guidance and suggestions for consideration when you want to make
6180your image more secure.
6181
6182.. note::
6183
6184 Because the security requirements and risks are different for every
6185 type of device, this section cannot provide a complete reference on
6186 securing your custom OS. It is strongly recommended that you also
6187 consult other sources of information on embedded Linux system
6188 hardening and on security.
6189
6190General Considerations
6191----------------------
6192
6193General considerations exist that help you create more secure images.
6194You should consider the following suggestions to help make your device
6195more secure:
6196
6197- Scan additional code you are adding to the system (e.g. application
6198 code) by using static analysis tools. Look for buffer overflows and
6199 other potential security problems.
6200
6201- Pay particular attention to the security for any web-based
6202 administration interface.
6203
6204 Web interfaces typically need to perform administrative functions and
6205 tend to need to run with elevated privileges. Thus, the consequences
6206 resulting from the interface's security becoming compromised can be
6207 serious. Look for common web vulnerabilities such as
6208 cross-site-scripting (XSS), unvalidated inputs, and so forth.
6209
6210 As with system passwords, the default credentials for accessing a
6211 web-based interface should not be the same across all devices. This
6212 is particularly true if the interface is enabled by default as it can
6213 be assumed that many end-users will not change the credentials.
6214
6215- Ensure you can update the software on the device to mitigate
6216 vulnerabilities discovered in the future. This consideration
6217 especially applies when your device is network-enabled.
6218
6219- Ensure you remove or disable debugging functionality before producing
6220 the final image. For information on how to do this, see the
6221 "`Considerations Specific to the OpenEmbedded Build
6222 System <#considerations-specific-to-the-openembedded-build-system>`__"
6223 section.
6224
6225- Ensure you have no network services listening that are not needed.
6226
6227- Remove any software from the image that is not needed.
6228
6229- Enable hardware support for secure boot functionality when your
6230 device supports this functionality.
6231
6232Security Flags
6233--------------
6234
6235The Yocto Project has security flags that you can enable that help make
6236your build output more secure. The security flags are in the
6237``meta/conf/distro/include/security_flags.inc`` file in your
6238:term:`Source Directory` (e.g. ``poky``).
6239
6240.. note::
6241
6242 Depending on the recipe, certain security flags are enabled and
6243 disabled by default.
6244
6245Use the following line in your ``local.conf`` file or in your custom
6246distribution configuration file to enable the security compiler and
6247linker flags for your build:
6248::
6249
6250 require conf/distro/include/security_flags.inc
6251
6252Considerations Specific to the OpenEmbedded Build System
6253--------------------------------------------------------
6254
6255You can take some steps that are specific to the OpenEmbedded build
6256system to make your images more secure:
6257
6258- Ensure "debug-tweaks" is not one of your selected
6259 :term:`IMAGE_FEATURES`.
6260 When creating a new project, the default is to provide you with an
6261 initial ``local.conf`` file that enables this feature using the
6262 :term:`EXTRA_IMAGE_FEATURES`
6263 variable with the line:
6264 ::
6265
6266 EXTRA_IMAGE_FEATURES = "debug-tweaks"
6267
6268 To disable that feature, simply comment out that line in your
6269 ``local.conf`` file, or make sure ``IMAGE_FEATURES`` does not contain
6270 "debug-tweaks" before producing your final image. Among other things,
6271 leaving this in place sets the root password as blank, which makes
6272 logging in for debugging or inspection easy during development but
6273 also means anyone can easily log in during production.
6274
6275- It is possible to set a root password for the image and also to set
6276 passwords for any extra users you might add (e.g. administrative or
6277 service type users). When you set up passwords for multiple images or
6278 users, you should not duplicate passwords.
6279
6280 To set up passwords, use the
6281 :ref:`extrausers <ref-classes-extrausers>`
6282 class, which is the preferred method. For an example on how to set up
6283 both root and user passwords, see the
6284 ":ref:`extrausers.bbclass <ref-classes-extrausers>`"
6285 section.
6286
6287 .. note::
6288
6289 When adding extra user accounts or setting a root password, be
6290 cautious about setting the same password on every device. If you
6291 do this, and the password you have set is exposed, then every
6292 device is now potentially compromised. If you need this access but
6293 want to ensure security, consider setting a different, random
6294 password for each device. Typically, you do this as a separate
6295 step after you deploy the image onto the device.
6296
6297- Consider enabling a Mandatory Access Control (MAC) framework such as
6298 SMACK or SELinux and tuning it appropriately for your device's usage.
6299 You can find more information in the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006300 :yocto_git:`meta-selinux </cgit/cgit.cgi/meta-selinux/>` layer.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006301
6302Tools for Hardening Your Image
6303------------------------------
6304
6305The Yocto Project provides tools for making your image more secure. You
6306can find these tools in the ``meta-security`` layer of the
6307:yocto_git:`Yocto Project Source Repositories <>`.
6308
6309Creating Your Own Distribution
6310==============================
6311
6312When you build an image using the Yocto Project and do not alter any
6313distribution :term:`Metadata`, you are
6314creating a Poky distribution. If you wish to gain more control over
6315package alternative selections, compile-time options, and other
6316low-level configurations, you can create your own distribution.
6317
6318To create your own distribution, the basic steps consist of creating
6319your own distribution layer, creating your own distribution
6320configuration file, and then adding any needed code and Metadata to the
6321layer. The following steps provide some more detail:
6322
6323- *Create a layer for your new distro:* Create your distribution layer
6324 so that you can keep your Metadata and code for the distribution
6325 separate. It is strongly recommended that you create and use your own
6326 layer for configuration and code. Using your own layer as compared to
6327 just placing configurations in a ``local.conf`` configuration file
6328 makes it easier to reproduce the same build configuration when using
6329 multiple build machines. See the
6330 ":ref:`dev-manual/dev-manual-common-tasks:creating a general layer using the \`\`bitbake-layers\`\` script`"
6331 section for information on how to quickly set up a layer.
6332
6333- *Create the distribution configuration file:* The distribution
6334 configuration file needs to be created in the ``conf/distro``
6335 directory of your layer. You need to name it using your distribution
6336 name (e.g. ``mydistro.conf``).
6337
6338 .. note::
6339
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006340 The :term:`DISTRO` variable in your ``local.conf`` file determines the
6341 name of your distribution.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006342
6343 You can split out parts of your configuration file into include files
6344 and then "require" them from within your distribution configuration
6345 file. Be sure to place the include files in the
6346 ``conf/distro/include`` directory of your layer. A common example
6347 usage of include files would be to separate out the selection of
6348 desired version and revisions for individual recipes.
6349
6350 Your configuration file needs to set the following required
6351 variables:
6352
6353 - :term:`DISTRO_NAME`
6354
6355 - :term:`DISTRO_VERSION`
6356
6357 These following variables are optional and you typically set them
6358 from the distribution configuration file:
6359
6360 - :term:`DISTRO_FEATURES`
6361
6362 - :term:`DISTRO_EXTRA_RDEPENDS`
6363
6364 - :term:`DISTRO_EXTRA_RRECOMMENDS`
6365
6366 - :term:`TCLIBC`
6367
6368 .. tip::
6369
6370 If you want to base your distribution configuration file on the
6371 very basic configuration from OE-Core, you can use
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006372 ``conf/distro/defaultsetup.conf`` as a reference and just include
6373 variables that differ as compared to ``defaultsetup.conf``.
6374 Alternatively, you can create a distribution configuration file
6375 from scratch using the ``defaultsetup.conf`` file or configuration files
6376 from other distributions such as Poky or Angstrom as references.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006377
6378- *Provide miscellaneous variables:* Be sure to define any other
6379 variables for which you want to create a default or enforce as part
6380 of the distribution configuration. You can include nearly any
6381 variable from the ``local.conf`` file. The variables you use are not
6382 limited to the list in the previous bulleted item.
6383
6384- *Point to Your distribution configuration file:* In your
6385 ``local.conf`` file in the :term:`Build Directory`,
6386 set your
6387 :term:`DISTRO` variable to point to
6388 your distribution's configuration file. For example, if your
6389 distribution's configuration file is named ``mydistro.conf``, then
6390 you point to it as follows:
6391 ::
6392
6393 DISTRO = "mydistro"
6394
6395- *Add more to the layer if necessary:* Use your layer to hold other
6396 information needed for the distribution:
6397
6398 - Add recipes for installing distro-specific configuration files
6399 that are not already installed by another recipe. If you have
6400 distro-specific configuration files that are included by an
6401 existing recipe, you should add an append file (``.bbappend``) for
6402 those. For general information and recommendations on how to add
6403 recipes to your layer, see the "`Creating Your Own
6404 Layer <#creating-your-own-layer>`__" and "`Following Best
6405 Practices When Creating
6406 Layers <#best-practices-to-follow-when-creating-layers>`__"
6407 sections.
6408
6409 - Add any image recipes that are specific to your distribution.
6410
6411 - Add a ``psplash`` append file for a branded splash screen. For
6412 information on append files, see the "`Using .bbappend Files in
6413 Your Layer <#using-bbappend-files>`__" section.
6414
6415 - Add any other append files to make custom changes that are
6416 specific to individual recipes.
6417
6418Creating a Custom Template Configuration Directory
6419==================================================
6420
6421If you are producing your own customized version of the build system for
6422use by other users, you might want to customize the message shown by the
6423setup script or you might want to change the template configuration
6424files (i.e. ``local.conf`` and ``bblayers.conf``) that are created in a
6425new build directory.
6426
6427The OpenEmbedded build system uses the environment variable
6428``TEMPLATECONF`` to locate the directory from which it gathers
6429configuration information that ultimately ends up in the
6430:term:`Build Directory` ``conf`` directory.
6431By default, ``TEMPLATECONF`` is set as follows in the ``poky``
6432repository:
6433::
6434
6435 TEMPLATECONF=${TEMPLATECONF:-meta-poky/conf}
6436
6437This is the
6438directory used by the build system to find templates from which to build
6439some key configuration files. If you look at this directory, you will
6440see the ``bblayers.conf.sample``, ``local.conf.sample``, and
6441``conf-notes.txt`` files. The build system uses these files to form the
6442respective ``bblayers.conf`` file, ``local.conf`` file, and display the
6443list of BitBake targets when running the setup script.
6444
6445To override these default configuration files with configurations you
6446want used within every new Build Directory, simply set the
6447``TEMPLATECONF`` variable to your directory. The ``TEMPLATECONF``
6448variable is set in the ``.templateconf`` file, which is in the top-level
6449:term:`Source Directory` folder
6450(e.g. ``poky``). Edit the ``.templateconf`` so that it can locate your
6451directory.
6452
6453Best practices dictate that you should keep your template configuration
6454directory in your custom distribution layer. For example, suppose you
6455have a layer named ``meta-mylayer`` located in your home directory and
6456you want your template configuration directory named ``myconf``.
6457Changing the ``.templateconf`` as follows causes the OpenEmbedded build
6458system to look in your directory and base its configuration files on the
6459``*.sample`` configuration files it finds. The final configuration files
6460(i.e. ``local.conf`` and ``bblayers.conf`` ultimately still end up in
6461your Build Directory, but they are based on your ``*.sample`` files.
6462::
6463
6464 TEMPLATECONF=${TEMPLATECONF:-meta-mylayer/myconf}
6465
6466Aside from the ``*.sample`` configuration files, the ``conf-notes.txt``
6467also resides in the default ``meta-poky/conf`` directory. The script
6468that sets up the build environment (i.e.
6469:ref:`structure-core-script`) uses this file to
6470display BitBake targets as part of the script output. Customizing this
6471``conf-notes.txt`` file is a good way to make sure your list of custom
6472targets appears as part of the script's output.
6473
6474Here is the default list of targets displayed as a result of running
6475either of the setup scripts:
6476::
6477
6478 You can now run 'bitbake <target>'
6479
6480 Common targets are:
6481 core-image-minimal
6482 core-image-sato
6483 meta-toolchain
6484 meta-ide-support
6485
6486Changing the listed common targets is as easy as editing your version of
6487``conf-notes.txt`` in your custom template configuration directory and
6488making sure you have ``TEMPLATECONF`` set to your directory.
6489
6490.. _dev-saving-memory-during-a-build:
6491
6492Conserving Disk Space During Builds
6493===================================
6494
6495To help conserve disk space during builds, you can add the following
6496statement to your project's ``local.conf`` configuration file found in
6497the :term:`Build Directory`:
6498::
6499
6500 INHERIT += "rm_work"
6501
6502Adding this statement deletes the work directory used for
6503building a recipe once the recipe is built. For more information on
6504"rm_work", see the
6505:ref:`rm_work <ref-classes-rm-work>` class in the
6506Yocto Project Reference Manual.
6507
6508Working with Packages
6509=====================
6510
6511This section describes a few tasks that involve packages:
6512
6513- `Excluding packages from an
6514 image <#excluding-packages-from-an-image>`__
6515
6516- `Incrementing a binary package
6517 version <#incrementing-a-binary-package-version>`__
6518
6519- `Handling optional module
6520 packaging <#handling-optional-module-packaging>`__
6521
6522- `Using runtime package
6523 management <#using-runtime-package-management>`__
6524
6525- `Generating and using signed
6526 packages <#generating-and-using-signed-packages>`__
6527
6528- `Setting up and running package test
6529 (ptest) <#testing-packages-with-ptest>`__
6530
6531- `Creating node package manager (NPM)
6532 packages <#creating-node-package-manager-npm-packages>`__
6533
6534- `Adding custom metadata to
6535 packages <#adding-custom-metadata-to-packages>`__
6536
6537Excluding Packages from an Image
6538--------------------------------
6539
6540You might find it necessary to prevent specific packages from being
6541installed into an image. If so, you can use several variables to direct
6542the build system to essentially ignore installing recommended packages
6543or to not install a package at all.
6544
6545The following list introduces variables you can use to prevent packages
6546from being installed into your image. Each of these variables only works
6547with IPK and RPM package types. Support for Debian packages does not
6548exist. Also, you can use these variables from your ``local.conf`` file
6549or attach them to a specific image recipe by using a recipe name
6550override. For more detail on the variables, see the descriptions in the
6551Yocto Project Reference Manual's glossary chapter.
6552
6553- :term:`BAD_RECOMMENDATIONS`:
6554 Use this variable to specify "recommended-only" packages that you do
6555 not want installed.
6556
6557- :term:`NO_RECOMMENDATIONS`:
6558 Use this variable to prevent all "recommended-only" packages from
6559 being installed.
6560
6561- :term:`PACKAGE_EXCLUDE`:
6562 Use this variable to prevent specific packages from being installed
6563 regardless of whether they are "recommended-only" or not. You need to
6564 realize that the build process could fail with an error when you
6565 prevent the installation of a package whose presence is required by
6566 an installed package.
6567
6568.. _incrementing-a-binary-package-version:
6569
6570Incrementing a Package Version
6571------------------------------
6572
6573This section provides some background on how binary package versioning
6574is accomplished and presents some of the services, variables, and
6575terminology involved.
6576
6577In order to understand binary package versioning, you need to consider
6578the following:
6579
6580- Binary Package: The binary package that is eventually built and
6581 installed into an image.
6582
6583- Binary Package Version: The binary package version is composed of two
6584 components - a version and a revision.
6585
6586 .. note::
6587
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006588 Technically, a third component, the "epoch" (i.e. :term:`PE`) is involved
6589 but this discussion for the most part ignores ``PE``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006590
6591 The version and revision are taken from the
6592 :term:`PV` and
6593 :term:`PR` variables, respectively.
6594
6595- ``PV``: The recipe version. ``PV`` represents the version of the
6596 software being packaged. Do not confuse ``PV`` with the binary
6597 package version.
6598
6599- ``PR``: The recipe revision.
6600
6601- :term:`SRCPV`: The OpenEmbedded
6602 build system uses this string to help define the value of ``PV`` when
6603 the source code revision needs to be included in it.
6604
6605- :yocto_wiki:`PR Service </wiki/PR_Service>`: A
6606 network-based service that helps automate keeping package feeds
6607 compatible with existing package manager applications such as RPM,
6608 APT, and OPKG.
6609
6610Whenever the binary package content changes, the binary package version
6611must change. Changing the binary package version is accomplished by
6612changing or "bumping" the ``PR`` and/or ``PV`` values. Increasing these
6613values occurs one of two ways:
6614
6615- Automatically using a Package Revision Service (PR Service).
6616
6617- Manually incrementing the ``PR`` and/or ``PV`` variables.
6618
6619Given a primary challenge of any build system and its users is how to
6620maintain a package feed that is compatible with existing package manager
6621applications such as RPM, APT, and OPKG, using an automated system is
6622much preferred over a manual system. In either system, the main
6623requirement is that binary package version numbering increases in a
6624linear fashion and that a number of version components exist that
6625support that linear progression. For information on how to ensure
6626package revisioning remains linear, see the "`Automatically Incrementing
6627a Binary Package Revision
6628Number <#automatically-incrementing-a-binary-package-revision-number>`__"
6629section.
6630
6631The following three sections provide related information on the PR
6632Service, the manual method for "bumping" ``PR`` and/or ``PV``, and on
6633how to ensure binary package revisioning remains linear.
6634
6635Working With a PR Service
6636~~~~~~~~~~~~~~~~~~~~~~~~~
6637
6638As mentioned, attempting to maintain revision numbers in the
6639:term:`Metadata` is error prone, inaccurate,
6640and causes problems for people submitting recipes. Conversely, the PR
6641Service automatically generates increasing numbers, particularly the
6642revision field, which removes the human element.
6643
6644.. note::
6645
6646 For additional information on using a PR Service, you can see the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006647 :yocto_wiki:`PR Service </wiki/PR_Service>` wiki page.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006648
6649The Yocto Project uses variables in order of decreasing priority to
6650facilitate revision numbering (i.e.
6651:term:`PE`,
6652:term:`PV`, and
6653:term:`PR` for epoch, version, and
6654revision, respectively). The values are highly dependent on the policies
6655and procedures of a given distribution and package feed.
6656
6657Because the OpenEmbedded build system uses
6658":ref:`signatures <overview-checksums>`", which are
6659unique to a given build, the build system knows when to rebuild
6660packages. All the inputs into a given task are represented by a
6661signature, which can trigger a rebuild when different. Thus, the build
6662system itself does not rely on the ``PR``, ``PV``, and ``PE`` numbers to
6663trigger a rebuild. The signatures, however, can be used to generate
6664these values.
6665
6666The PR Service works with both ``OEBasic`` and ``OEBasicHash``
6667generators. The value of ``PR`` bumps when the checksum changes and the
6668different generator mechanisms change signatures under different
6669circumstances.
6670
6671As implemented, the build system includes values from the PR Service
6672into the ``PR`` field as an addition using the form "``.x``" so ``r0``
6673becomes ``r0.1``, ``r0.2`` and so forth. This scheme allows existing
6674``PR`` values to be used for whatever reasons, which include manual
6675``PR`` bumps, should it be necessary.
6676
6677By default, the PR Service is not enabled or running. Thus, the packages
6678generated are just "self consistent". The build system adds and removes
6679packages and there are no guarantees about upgrade paths but images will
6680be consistent and correct with the latest changes.
6681
6682The simplest form for a PR Service is for it to exist for a single host
6683development system that builds the package feed (building system). For
6684this scenario, you can enable a local PR Service by setting
6685:term:`PRSERV_HOST` in your
6686``local.conf`` file in the :term:`Build Directory`:
6687::
6688
6689 PRSERV_HOST = "localhost:0"
6690
6691Once the service is started, packages will automatically
6692get increasing ``PR`` values and BitBake takes care of starting and
6693stopping the server.
6694
6695If you have a more complex setup where multiple host development systems
6696work against a common, shared package feed, you have a single PR Service
6697running and it is connected to each building system. For this scenario,
6698you need to start the PR Service using the ``bitbake-prserv`` command:
6699::
6700
6701 bitbake-prserv --host ip --port port --start
6702
6703In addition to
6704hand-starting the service, you need to update the ``local.conf`` file of
6705each building system as described earlier so each system points to the
6706server and port.
6707
6708It is also recommended you use build history, which adds some sanity
6709checks to binary package versions, in conjunction with the server that
6710is running the PR Service. To enable build history, add the following to
6711each building system's ``local.conf`` file:
6712::
6713
6714 # It is recommended to activate "buildhistory" for testing the PR service
6715 INHERIT += "buildhistory"
6716 BUILDHISTORY_COMMIT = "1"
6717
6718For information on build
6719history, see the "`Maintaining Build Output
6720Quality <#maintaining-build-output-quality>`__" section.
6721
6722.. note::
6723
6724 The OpenEmbedded build system does not maintain ``PR`` information as
6725 part of the shared state (sstate) packages. If you maintain an sstate
6726 feed, its expected that either all your building systems that
6727 contribute to the sstate feed use a shared PR Service, or you do not
6728 run a PR Service on any of your building systems. Having some systems
6729 use a PR Service while others do not leads to obvious problems.
6730
6731 For more information on shared state, see the
6732 ":ref:`overview-manual/overview-manual-concepts:shared state cache`"
6733 section in the Yocto Project Overview and Concepts Manual.
6734
6735Manually Bumping PR
6736~~~~~~~~~~~~~~~~~~~
6737
6738The alternative to setting up a PR Service is to manually "bump" the
6739:term:`PR` variable.
6740
6741If a committed change results in changing the package output, then the
6742value of the PR variable needs to be increased (or "bumped") as part of
6743that commit. For new recipes you should add the ``PR`` variable and set
6744its initial value equal to "r0", which is the default. Even though the
6745default value is "r0", the practice of adding it to a new recipe makes
6746it harder to forget to bump the variable when you make changes to the
6747recipe in future.
6748
6749If you are sharing a common ``.inc`` file with multiple recipes, you can
6750also use the ``INC_PR`` variable to ensure that the recipes sharing the
6751``.inc`` file are rebuilt when the ``.inc`` file itself is changed. The
6752``.inc`` file must set ``INC_PR`` (initially to "r0"), and all recipes
6753referring to it should set ``PR`` to "${INC_PR}.0" initially,
6754incrementing the last number when the recipe is changed. If the ``.inc``
6755file is changed then its ``INC_PR`` should be incremented.
6756
6757When upgrading the version of a binary package, assuming the ``PV``
6758changes, the ``PR`` variable should be reset to "r0" (or "${INC_PR}.0"
6759if you are using ``INC_PR``).
6760
6761Usually, version increases occur only to binary packages. However, if
6762for some reason ``PV`` changes but does not increase, you can increase
6763the ``PE`` variable (Package Epoch). The ``PE`` variable defaults to
6764"0".
6765
6766Binary package version numbering strives to follow the `Debian Version
6767Field Policy
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006768Guidelines <https://www.debian.org/doc/debian-policy/ch-controlfields.html>`__.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006769These guidelines define how versions are compared and what "increasing"
6770a version means.
6771
6772.. _automatically-incrementing-a-binary-package-revision-number:
6773
6774Automatically Incrementing a Package Version Number
6775~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
6776
6777When fetching a repository, BitBake uses the
6778:term:`SRCREV` variable to determine
6779the specific source code revision from which to build. You set the
6780``SRCREV`` variable to
6781:term:`AUTOREV` to cause the
6782OpenEmbedded build system to automatically use the latest revision of
6783the software:
6784::
6785
6786 SRCREV = "${AUTOREV}"
6787
6788Furthermore, you need to reference ``SRCPV`` in ``PV`` in order to
6789automatically update the version whenever the revision of the source
6790code changes. Here is an example:
6791::
6792
6793 PV = "1.0+git${SRCPV}"
6794
6795The OpenEmbedded build system substitutes ``SRCPV`` with the following:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006796
6797.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006798
6799 AUTOINC+source_code_revision
6800
6801The build system replaces the ``AUTOINC``
6802with a number. The number used depends on the state of the PR Service:
6803
6804- If PR Service is enabled, the build system increments the number,
6805 which is similar to the behavior of
6806 :term:`PR`. This behavior results in
6807 linearly increasing package versions, which is desirable. Here is an
6808 example:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006809
6810 .. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006811
6812 hello-world-git_0.0+git0+b6558dd387-r0.0_armv7a-neon.ipk
6813 hello-world-git_0.0+git1+dd2f5c3565-r0.0_armv7a-neon.ipk
6814
6815- If PR Service is not enabled, the build system replaces the
6816 ``AUTOINC`` placeholder with zero (i.e. "0"). This results in
6817 changing the package version since the source revision is included.
6818 However, package versions are not increased linearly. Here is an
6819 example:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006820
6821 .. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006822
6823 hello-world-git_0.0+git0+b6558dd387-r0.0_armv7a-neon.ipk
6824 hello-world-git_0.0+git0+dd2f5c3565-r0.0_armv7a-neon.ipk
6825
6826In summary, the OpenEmbedded build system does not track the history of
6827binary package versions for this purpose. ``AUTOINC``, in this case, is
6828comparable to ``PR``. If PR server is not enabled, ``AUTOINC`` in the
6829package version is simply replaced by "0". If PR server is enabled, the
6830build system keeps track of the package versions and bumps the number
6831when the package revision changes.
6832
6833Handling Optional Module Packaging
6834----------------------------------
6835
6836Many pieces of software split functionality into optional modules (or
6837plugins) and the plugins that are built might depend on configuration
6838options. To avoid having to duplicate the logic that determines what
6839modules are available in your recipe or to avoid having to package each
6840module by hand, the OpenEmbedded build system provides functionality to
6841handle module packaging dynamically.
6842
6843To handle optional module packaging, you need to do two things:
6844
6845- Ensure the module packaging is actually done.
6846
6847- Ensure that any dependencies on optional modules from other recipes
6848 are satisfied by your recipe.
6849
6850Making Sure the Packaging is Done
6851~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
6852
6853To ensure the module packaging actually gets done, you use the
6854``do_split_packages`` function within the ``populate_packages`` Python
6855function in your recipe. The ``do_split_packages`` function searches for
6856a pattern of files or directories under a specified path and creates a
6857package for each one it finds by appending to the
6858:term:`PACKAGES` variable and
6859setting the appropriate values for ``FILES_packagename``,
6860``RDEPENDS_packagename``, ``DESCRIPTION_packagename``, and so forth.
6861Here is an example from the ``lighttpd`` recipe:
6862::
6863
6864 python populate_packages_prepend () {
6865 lighttpd_libdir = d.expand('${libdir}')
6866 do_split_packages(d, lighttpd_libdir, '^mod_(.*).so$',
6867 'lighttpd-module-%s', 'Lighttpd module for %s',
6868 extra_depends='')
6869 }
6870
6871The previous example specifies a number of things in the call to
6872``do_split_packages``.
6873
6874- A directory within the files installed by your recipe through
6875 ``do_install`` in which to search.
6876
6877- A regular expression used to match module files in that directory. In
6878 the example, note the parentheses () that mark the part of the
6879 expression from which the module name should be derived.
6880
6881- A pattern to use for the package names.
6882
6883- A description for each package.
6884
6885- An empty string for ``extra_depends``, which disables the default
6886 dependency on the main ``lighttpd`` package. Thus, if a file in
6887 ``${libdir}`` called ``mod_alias.so`` is found, a package called
6888 ``lighttpd-module-alias`` is created for it and the
6889 :term:`DESCRIPTION` is set to
6890 "Lighttpd module for alias".
6891
6892Often, packaging modules is as simple as the previous example. However,
6893more advanced options exist that you can use within
6894``do_split_packages`` to modify its behavior. And, if you need to, you
6895can add more logic by specifying a hook function that is called for each
6896package. It is also perfectly acceptable to call ``do_split_packages``
6897multiple times if you have more than one set of modules to package.
6898
6899For more examples that show how to use ``do_split_packages``, see the
6900``connman.inc`` file in the ``meta/recipes-connectivity/connman/``
6901directory of the ``poky`` :ref:`source repository <yocto-project-repositories>`. You can
6902also find examples in ``meta/classes/kernel.bbclass``.
6903
6904Following is a reference that shows ``do_split_packages`` mandatory and
6905optional arguments:
6906::
6907
6908 Mandatory arguments
6909
6910 root
6911 The path in which to search
6912 file_regex
6913 Regular expression to match searched files.
6914 Use parentheses () to mark the part of this
6915 expression that should be used to derive the
6916 module name (to be substituted where %s is
6917 used in other function arguments as noted below)
6918 output_pattern
6919 Pattern to use for the package names. Must
6920 include %s.
6921 description
6922 Description to set for each package. Must
6923 include %s.
6924
6925 Optional arguments
6926
6927 postinst
6928 Postinstall script to use for all packages
6929 (as a string)
6930 recursive
6931 True to perform a recursive search - default
6932 False
6933 hook
6934 A hook function to be called for every match.
6935 The function will be called with the following
6936 arguments (in the order listed):
6937
6938 f
6939 Full path to the file/directory match
6940 pkg
6941 The package name
6942 file_regex
6943 As above
6944 output_pattern
6945 As above
6946 modulename
6947 The module name derived using file_regex
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006948 extra_depends
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006949 Extra runtime dependencies (RDEPENDS) to be
6950 set for all packages. The default value of None
6951 causes a dependency on the main package
6952 (${PN}) - if you do not want this, pass empty
6953 string '' for this parameter.
6954 aux_files_pattern
6955 Extra item(s) to be added to FILES for each
6956 package. Can be a single string item or a list
6957 of strings for multiple items. Must include %s.
6958 postrm
6959 postrm script to use for all packages (as a
6960 string)
6961 allow_dirs
6962 True to allow directories to be matched -
6963 default False
6964 prepend
6965 If True, prepend created packages to PACKAGES
6966 instead of the default False which appends them
6967 match_path
6968 match file_regex on the whole relative path to
6969 the root rather than just the file name
6970 aux_files_pattern_verbatim
6971 Extra item(s) to be added to FILES for each
6972 package, using the actual derived module name
6973 rather than converting it to something legal
6974 for a package name. Can be a single string item
6975 or a list of strings for multiple items. Must
6976 include %s.
6977 allow_links
6978 True to allow symlinks to be matched - default
6979 False
6980 summary
6981 Summary to set for each package. Must include %s;
6982 defaults to description if not set.
6983
6984
6985
6986Satisfying Dependencies
6987~~~~~~~~~~~~~~~~~~~~~~~
6988
6989The second part for handling optional module packaging is to ensure that
6990any dependencies on optional modules from other recipes are satisfied by
6991your recipe. You can be sure these dependencies are satisfied by using
6992the :term:`PACKAGES_DYNAMIC`
6993variable. Here is an example that continues with the ``lighttpd`` recipe
6994shown earlier:
6995::
6996
6997 PACKAGES_DYNAMIC = "lighttpd-module-.*"
6998
6999The name
7000specified in the regular expression can of course be anything. In this
7001example, it is ``lighttpd-module-`` and is specified as the prefix to
7002ensure that any :term:`RDEPENDS` and
7003:term:`RRECOMMENDS` on a package
7004name starting with the prefix are satisfied during build time. If you
7005are using ``do_split_packages`` as described in the previous section,
7006the value you put in ``PACKAGES_DYNAMIC`` should correspond to the name
7007pattern specified in the call to ``do_split_packages``.
7008
7009Using Runtime Package Management
7010--------------------------------
7011
7012During a build, BitBake always transforms a recipe into one or more
7013packages. For example, BitBake takes the ``bash`` recipe and produces a
7014number of packages (e.g. ``bash``, ``bash-bashbug``,
7015``bash-completion``, ``bash-completion-dbg``, ``bash-completion-dev``,
7016``bash-completion-extra``, ``bash-dbg``, and so forth). Not all
7017generated packages are included in an image.
7018
7019In several situations, you might need to update, add, remove, or query
7020the packages on a target device at runtime (i.e. without having to
7021generate a new image). Examples of such situations include:
7022
7023- You want to provide in-the-field updates to deployed devices (e.g.
7024 security updates).
7025
7026- You want to have a fast turn-around development cycle for one or more
7027 applications that run on your device.
7028
7029- You want to temporarily install the "debug" packages of various
7030 applications on your device so that debugging can be greatly improved
7031 by allowing access to symbols and source debugging.
7032
7033- You want to deploy a more minimal package selection of your device
7034 but allow in-the-field updates to add a larger selection for
7035 customization.
7036
7037In all these situations, you have something similar to a more
7038traditional Linux distribution in that in-field devices are able to
7039receive pre-compiled packages from a server for installation or update.
7040Being able to install these packages on a running, in-field device is
7041what is termed "runtime package management".
7042
7043In order to use runtime package management, you need a host or server
7044machine that serves up the pre-compiled packages plus the required
7045metadata. You also need package manipulation tools on the target. The
7046build machine is a likely candidate to act as the server. However, that
7047machine does not necessarily have to be the package server. The build
7048machine could push its artifacts to another machine that acts as the
7049server (e.g. Internet-facing). In fact, doing so is advantageous for a
7050production environment as getting the packages away from the development
7051system's build directory prevents accidental overwrites.
7052
7053A simple build that targets just one device produces more than one
7054package database. In other words, the packages produced by a build are
7055separated out into a couple of different package groupings based on
7056criteria such as the target's CPU architecture, the target board, or the
7057C library used on the target. For example, a build targeting the
7058``qemux86`` device produces the following three package databases:
7059``noarch``, ``i586``, and ``qemux86``. If you wanted your ``qemux86``
7060device to be aware of all the packages that were available to it, you
7061would need to point it to each of these databases individually. In a
7062similar way, a traditional Linux distribution usually is configured to
7063be aware of a number of software repositories from which it retrieves
7064packages.
7065
7066Using runtime package management is completely optional and not required
7067for a successful build or deployment in any way. But if you want to make
7068use of runtime package management, you need to do a couple things above
7069and beyond the basics. The remainder of this section describes what you
7070need to do.
7071
7072.. _runtime-package-management-build:
7073
7074Build Considerations
7075~~~~~~~~~~~~~~~~~~~~
7076
7077This section describes build considerations of which you need to be
7078aware in order to provide support for runtime package management.
7079
7080When BitBake generates packages, it needs to know what format or formats
7081to use. In your configuration, you use the
7082:term:`PACKAGE_CLASSES`
7083variable to specify the format:
7084
70851. Open the ``local.conf`` file inside your
7086 :term:`Build Directory` (e.g.
7087 ``~/poky/build/conf/local.conf``).
7088
70892. Select the desired package format as follows:
7090 ::
7091
7092 PACKAGE_CLASSES ?= "package_packageformat"
7093
7094 where packageformat can be "ipk", "rpm",
7095 "deb", or "tar" which are the supported package formats.
7096
7097 .. note::
7098
7099 Because the Yocto Project supports four different package formats,
7100 you can set the variable with more than one argument. However, the
7101 OpenEmbedded build system only uses the first argument when
7102 creating an image or Software Development Kit (SDK).
7103
7104If you would like your image to start off with a basic package database
7105containing the packages in your current build as well as to have the
7106relevant tools available on the target for runtime package management,
7107you can include "package-management" in the
7108:term:`IMAGE_FEATURES`
7109variable. Including "package-management" in this configuration variable
7110ensures that when the image is assembled for your target, the image
7111includes the currently-known package databases as well as the
7112target-specific tools required for runtime package management to be
7113performed on the target. However, this is not strictly necessary. You
7114could start your image off without any databases but only include the
7115required on-target package tool(s). As an example, you could include
7116"opkg" in your
7117:term:`IMAGE_INSTALL` variable
7118if you are using the IPK package format. You can then initialize your
7119target's package database(s) later once your image is up and running.
7120
7121Whenever you perform any sort of build step that can potentially
7122generate a package or modify existing package, it is always a good idea
7123to re-generate the package index after the build by using the following
7124command:
7125::
7126
7127 $ bitbake package-index
7128
7129It might be tempting to build the
7130package and the package index at the same time with a command such as
7131the following:
7132::
7133
7134 $ bitbake some-package package-index
7135
7136Do not do this as
7137BitBake does not schedule the package index for after the completion of
7138the package you are building. Consequently, you cannot be sure of the
7139package index including information for the package you just built.
7140Thus, be sure to run the package update step separately after building
7141any packages.
7142
7143You can use the
7144:term:`PACKAGE_FEED_ARCHS`,
7145:term:`PACKAGE_FEED_BASE_PATHS`,
7146and
7147:term:`PACKAGE_FEED_URIS`
7148variables to pre-configure target images to use a package feed. If you
7149do not define these variables, then manual steps as described in the
7150subsequent sections are necessary to configure the target. You should
7151set these variables before building the image in order to produce a
7152correctly configured image.
7153
7154When your build is complete, your packages reside in the
7155``${TMPDIR}/deploy/packageformat`` directory. For example, if
7156``${``\ :term:`TMPDIR`\ ``}`` is
7157``tmp`` and your selected package type is RPM, then your RPM packages
7158are available in ``tmp/deploy/rpm``.
7159
7160.. _runtime-package-management-server:
7161
7162Host or Server Machine Setup
7163~~~~~~~~~~~~~~~~~~~~~~~~~~~~
7164
7165Although other protocols are possible, a server using HTTP typically
7166serves packages. If you want to use HTTP, then set up and configure a
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007167web server such as Apache 2, lighttpd, or Python web server on the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007168machine serving the packages.
7169
7170To keep things simple, this section describes how to set up a
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007171Python web server to share package feeds from the developer's
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007172machine. Although this server might not be the best for a production
7173environment, the setup is simple and straight forward. Should you want
7174to use a different server more suited for production (e.g. Apache 2,
7175Lighttpd, or Nginx), take the appropriate steps to do so.
7176
7177From within the build directory where you have built an image based on
7178your packaging choice (i.e. the
7179:term:`PACKAGE_CLASSES`
7180setting), simply start the server. The following example assumes a build
7181directory of ``~/poky/build/tmp/deploy/rpm`` and a ``PACKAGE_CLASSES``
7182setting of "package_rpm":
7183::
7184
7185 $ cd ~/poky/build/tmp/deploy/rpm
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007186 $ python3 -m http.server
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007187
7188.. _runtime-package-management-target:
7189
7190Target Setup
7191~~~~~~~~~~~~
7192
7193Setting up the target differs depending on the package management
7194system. This section provides information for RPM, IPK, and DEB.
7195
7196.. _runtime-package-management-target-rpm:
7197
7198Using RPM
7199^^^^^^^^^
7200
7201The `Dandified Packaging
7202Tool <https://en.wikipedia.org/wiki/DNF_(software)>`__ (DNF) performs
7203runtime package management of RPM packages. In order to use DNF for
7204runtime package management, you must perform an initial setup on the
7205target machine for cases where the ``PACKAGE_FEED_*`` variables were not
7206set as part of the image that is running on the target. This means if
7207you built your image and did not not use these variables as part of the
7208build and your image is now running on the target, you need to perform
7209the steps in this section if you want to use runtime package management.
7210
7211.. note::
7212
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007213 For information on the ``PACKAGE_FEED_*`` variables, see
7214 :term:`PACKAGE_FEED_ARCHS`, :term:`PACKAGE_FEED_BASE_PATHS`, and
7215 :term:`PACKAGE_FEED_URIS` in the Yocto Project Reference Manual variables
7216 glossary.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007217
7218On the target, you must inform DNF that package databases are available.
7219You do this by creating a file named
7220``/etc/yum.repos.d/oe-packages.repo`` and defining the ``oe-packages``.
7221
7222As an example, assume the target is able to use the following package
7223databases: ``all``, ``i586``, and ``qemux86`` from a server named
7224``my.server``. The specifics for setting up the web server are up to
7225you. The critical requirement is that the URIs in the target repository
7226configuration point to the correct remote location for the feeds.
7227
7228.. note::
7229
7230 For development purposes, you can point the web server to the build
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007231 system's ``deploy`` directory. However, for production use, it is better to
7232 copy the package directories to a location outside of the build area and use
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007233 that location. Doing so avoids situations where the build system
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007234 overwrites or changes the ``deploy`` directory.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007235
7236When telling DNF where to look for the package databases, you must
7237declare individual locations per architecture or a single location used
7238for all architectures. You cannot do both:
7239
7240- *Create an Explicit List of Architectures:* Define individual base
7241 URLs to identify where each package database is located:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007242
7243 .. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007244
7245 [oe-packages]
7246 baseurl=http://my.server/rpm/i586 http://my.server/rpm/qemux86 http://my.server/rpm/all
7247
7248 This example
7249 informs DNF about individual package databases for all three
7250 architectures.
7251
7252- *Create a Single (Full) Package Index:* Define a single base URL that
7253 identifies where a full package database is located:
7254 ::
7255
7256 [oe-packages]
7257 baseurl=http://my.server/rpm
7258
7259 This example informs DNF about a single
7260 package database that contains all the package index information for
7261 all supported architectures.
7262
7263Once you have informed DNF where to find the package databases, you need
7264to fetch them:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007265
7266.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007267
7268 # dnf makecache
7269
7270DNF is now able to find, install, and
7271upgrade packages from the specified repository or repositories.
7272
7273.. note::
7274
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007275 See the `DNF documentation <https://dnf.readthedocs.io/en/latest/>`__ for
7276 additional information.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007277
7278.. _runtime-package-management-target-ipk:
7279
7280Using IPK
7281^^^^^^^^^
7282
7283The ``opkg`` application performs runtime package management of IPK
7284packages. You must perform an initial setup for ``opkg`` on the target
7285machine if the
7286:term:`PACKAGE_FEED_ARCHS`,
7287:term:`PACKAGE_FEED_BASE_PATHS`,
7288and
7289:term:`PACKAGE_FEED_URIS`
7290variables have not been set or the target image was built before the
7291variables were set.
7292
7293The ``opkg`` application uses configuration files to find available
7294package databases. Thus, you need to create a configuration file inside
7295the ``/etc/opkg/`` direction, which informs ``opkg`` of any repository
7296you want to use.
7297
7298As an example, suppose you are serving packages from a ``ipk/``
7299directory containing the ``i586``, ``all``, and ``qemux86`` databases
7300through an HTTP server named ``my.server``. On the target, create a
7301configuration file (e.g. ``my_repo.conf``) inside the ``/etc/opkg/``
7302directory containing the following:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007303
7304.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007305
7306 src/gz all http://my.server/ipk/all
7307 src/gz i586 http://my.server/ipk/i586
7308 src/gz qemux86 http://my.server/ipk/qemux86
7309
7310Next, instruct ``opkg`` to fetch the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007311repository information:
7312
7313.. code-block:: none
7314
7315 # opkg update
7316
7317The ``opkg`` application is now able to find, install, and upgrade packages
7318from the specified repository.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007319
7320.. _runtime-package-management-target-deb:
7321
7322Using DEB
7323^^^^^^^^^
7324
7325The ``apt`` application performs runtime package management of DEB
7326packages. This application uses a source list file to find available
7327package databases. You must perform an initial setup for ``apt`` on the
7328target machine if the
7329:term:`PACKAGE_FEED_ARCHS`,
7330:term:`PACKAGE_FEED_BASE_PATHS`,
7331and
7332:term:`PACKAGE_FEED_URIS`
7333variables have not been set or the target image was built before the
7334variables were set.
7335
7336To inform ``apt`` of the repository you want to use, you might create a
7337list file (e.g. ``my_repo.list``) inside the
7338``/etc/apt/sources.list.d/`` directory. As an example, suppose you are
7339serving packages from a ``deb/`` directory containing the ``i586``,
7340``all``, and ``qemux86`` databases through an HTTP server named
7341``my.server``. The list file should contain:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007342
7343.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007344
7345 deb http://my.server/deb/all ./
7346 deb http://my.server/deb/i586 ./
7347 deb http://my.server/deb/qemux86 ./
7348
7349Next, instruct the ``apt`` application
7350to fetch the repository information:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007351
7352.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007353
7354 # apt-get update
7355
7356After this step,
7357``apt`` is able to find, install, and upgrade packages from the
7358specified repository.
7359
7360Generating and Using Signed Packages
7361------------------------------------
7362
7363In order to add security to RPM packages used during a build, you can
7364take steps to securely sign them. Once a signature is verified, the
7365OpenEmbedded build system can use the package in the build. If security
7366fails for a signed package, the build system aborts the build.
7367
7368This section describes how to sign RPM packages during a build and how
7369to use signed package feeds (repositories) when doing a build.
7370
7371Signing RPM Packages
7372~~~~~~~~~~~~~~~~~~~~
7373
7374To enable signing RPM packages, you must set up the following
7375configurations in either your ``local.config`` or ``distro.config``
7376file:
7377::
7378
7379 # Inherit sign_rpm.bbclass to enable signing functionality
7380 INHERIT += " sign_rpm"
7381 # Define the GPG key that will be used for signing.
7382 RPM_GPG_NAME = "key_name"
7383 # Provide passphrase for the key
7384 RPM_GPG_PASSPHRASE = "passphrase"
7385
7386.. note::
7387
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007388 Be sure to supply appropriate values for both `key_name` and
7389 `passphrase`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007390
7391Aside from the ``RPM_GPG_NAME`` and ``RPM_GPG_PASSPHRASE`` variables in
7392the previous example, two optional variables related to signing exist:
7393
7394- *GPG_BIN:* Specifies a ``gpg`` binary/wrapper that is executed
7395 when the package is signed.
7396
7397- *GPG_PATH:* Specifies the ``gpg`` home directory used when the
7398 package is signed.
7399
7400Processing Package Feeds
7401~~~~~~~~~~~~~~~~~~~~~~~~
7402
7403In addition to being able to sign RPM packages, you can also enable
7404signed package feeds for IPK and RPM packages.
7405
7406The steps you need to take to enable signed package feed use are similar
7407to the steps used to sign RPM packages. You must define the following in
7408your ``local.config`` or ``distro.config`` file:
7409::
7410
7411 INHERIT += "sign_package_feed"
7412 PACKAGE_FEED_GPG_NAME = "key_name"
7413 PACKAGE_FEED_GPG_PASSPHRASE_FILE = "path_to_file_containing_passphrase"
7414
7415For signed package feeds, the passphrase must exist in a separate file,
7416which is pointed to by the ``PACKAGE_FEED_GPG_PASSPHRASE_FILE``
7417variable. Regarding security, keeping a plain text passphrase out of the
7418configuration is more secure.
7419
7420Aside from the ``PACKAGE_FEED_GPG_NAME`` and
7421``PACKAGE_FEED_GPG_PASSPHRASE_FILE`` variables, three optional variables
7422related to signed package feeds exist:
7423
7424- *GPG_BIN* Specifies a ``gpg`` binary/wrapper that is executed
7425 when the package is signed.
7426
7427- *GPG_PATH:* Specifies the ``gpg`` home directory used when the
7428 package is signed.
7429
7430- *PACKAGE_FEED_GPG_SIGNATURE_TYPE:* Specifies the type of ``gpg``
7431 signature. This variable applies only to RPM and IPK package feeds.
7432 Allowable values for the ``PACKAGE_FEED_GPG_SIGNATURE_TYPE`` are
7433 "ASC", which is the default and specifies ascii armored, and "BIN",
7434 which specifies binary.
7435
7436Testing Packages With ptest
7437---------------------------
7438
7439A Package Test (ptest) runs tests against packages built by the
7440OpenEmbedded build system on the target machine. A ptest contains at
7441least two items: the actual test, and a shell script (``run-ptest``)
7442that starts the test. The shell script that starts the test must not
7443contain the actual test - the script only starts the test. On the other
7444hand, the test can be anything from a simple shell script that runs a
7445binary and checks the output to an elaborate system of test binaries and
7446data files.
7447
7448The test generates output in the format used by Automake:
7449::
7450
7451 result: testname
7452
7453where the result can be ``PASS``, ``FAIL``, or ``SKIP``, and
7454the testname can be any identifying string.
7455
7456For a list of Yocto Project recipes that are already enabled with ptest,
7457see the :yocto_wiki:`Ptest </wiki/Ptest>` wiki page.
7458
7459.. note::
7460
7461 A recipe is "ptest-enabled" if it inherits the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007462 :ref:`ptest <ref-classes-ptest>` class.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007463
7464Adding ptest to Your Build
7465~~~~~~~~~~~~~~~~~~~~~~~~~~
7466
7467To add package testing to your build, add the
7468:term:`DISTRO_FEATURES` and
7469:term:`EXTRA_IMAGE_FEATURES`
7470variables to your ``local.conf`` file, which is found in the
7471:term:`Build Directory`:
7472::
7473
7474 DISTRO_FEATURES_append = " ptest"
7475 EXTRA_IMAGE_FEATURES += "ptest-pkgs"
7476
7477Once your build is complete, the ptest files are installed into the
7478``/usr/lib/package/ptest`` directory within the image, where ``package``
7479is the name of the package.
7480
7481Running ptest
7482~~~~~~~~~~~~~
7483
7484The ``ptest-runner`` package installs a shell script that loops through
7485all installed ptest test suites and runs them in sequence. Consequently,
7486you might want to add this package to your image.
7487
7488Getting Your Package Ready
7489~~~~~~~~~~~~~~~~~~~~~~~~~~
7490
7491In order to enable a recipe to run installed ptests on target hardware,
7492you need to prepare the recipes that build the packages you want to
7493test. Here is what you have to do for each recipe:
7494
7495- *Be sure the recipe inherits
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007496 the* :ref:`ptest <ref-classes-ptest>` *class:*
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007497 Include the following line in each recipe:
7498 ::
7499
7500 inherit ptest
7501
7502- *Create run-ptest:* This script starts your test. Locate the
7503 script where you will refer to it using
7504 :term:`SRC_URI`. Here is an
7505 example that starts a test for ``dbus``:
7506 ::
7507
7508 #!/bin/sh
7509 cd test
7510 make -k runtest-TESTS
7511
7512- *Ensure dependencies are met:* If the test adds build or runtime
7513 dependencies that normally do not exist for the package (such as
7514 requiring "make" to run the test suite), use the
7515 :term:`DEPENDS` and
7516 :term:`RDEPENDS` variables in
7517 your recipe in order for the package to meet the dependencies. Here
7518 is an example where the package has a runtime dependency on "make":
7519 ::
7520
7521 RDEPENDS_${PN}-ptest += "make"
7522
7523- *Add a function to build the test suite:* Not many packages support
7524 cross-compilation of their test suites. Consequently, you usually
7525 need to add a cross-compilation function to the package.
7526
7527 Many packages based on Automake compile and run the test suite by
7528 using a single command such as ``make check``. However, the host
7529 ``make check`` builds and runs on the same computer, while
7530 cross-compiling requires that the package is built on the host but
7531 executed for the target architecture (though often, as in the case
7532 for ptest, the execution occurs on the host). The built version of
7533 Automake that ships with the Yocto Project includes a patch that
7534 separates building and execution. Consequently, packages that use the
7535 unaltered, patched version of ``make check`` automatically
7536 cross-compiles.
7537
7538 Regardless, you still must add a ``do_compile_ptest`` function to
7539 build the test suite. Add a function similar to the following to your
7540 recipe:
7541 ::
7542
7543 do_compile_ptest() {
7544 oe_runmake buildtest-TESTS
7545 }
7546
7547- *Ensure special configurations are set:* If the package requires
7548 special configurations prior to compiling the test code, you must
7549 insert a ``do_configure_ptest`` function into the recipe.
7550
7551- *Install the test suite:* The ``ptest`` class automatically copies
7552 the file ``run-ptest`` to the target and then runs make
7553 ``install-ptest`` to run the tests. If this is not enough, you need
7554 to create a ``do_install_ptest`` function and make sure it gets
7555 called after the "make install-ptest" completes.
7556
7557Creating Node Package Manager (NPM) Packages
7558--------------------------------------------
7559
7560`NPM <https://en.wikipedia.org/wiki/Npm_(software)>`__ is a package
7561manager for the JavaScript programming language. The Yocto Project
7562supports the NPM :ref:`fetcher <bitbake:bb-fetchers>`. You can
7563use this fetcher in combination with
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007564:doc:`devtool <../ref-manual/ref-devtool-reference>` to create
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007565recipes that produce NPM packages.
7566
7567Two workflows exist that allow you to create NPM packages using
7568``devtool``: the NPM registry modules method and the NPM project code
7569method.
7570
7571.. note::
7572
7573 While it is possible to create NPM recipes manually, using
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007574 ``devtool`` is far simpler.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007575
7576Additionally, some requirements and caveats exist.
7577
7578.. _npm-package-creation-requirements:
7579
7580Requirements and Caveats
7581~~~~~~~~~~~~~~~~~~~~~~~~
7582
7583You need to be aware of the following before using ``devtool`` to create
7584NPM packages:
7585
7586- Of the two methods that you can use ``devtool`` to create NPM
7587 packages, the registry approach is slightly simpler. However, you
7588 might consider the project approach because you do not have to
7589 publish your module in the NPM registry
7590 (`npm-registry <https://docs.npmjs.com/misc/registry>`_), which
7591 is NPM's public registry.
7592
7593- Be familiar with
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007594 :doc:`devtool <../ref-manual/ref-devtool-reference>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007595
7596- The NPM host tools need the native ``nodejs-npm`` package, which is
7597 part of the OpenEmbedded environment. You need to get the package by
7598 cloning the https://github.com/openembedded/meta-openembedded
7599 repository out of GitHub. Be sure to add the path to your local copy
7600 to your ``bblayers.conf`` file.
7601
7602- ``devtool`` cannot detect native libraries in module dependencies.
7603 Consequently, you must manually add packages to your recipe.
7604
7605- While deploying NPM packages, ``devtool`` cannot determine which
7606 dependent packages are missing on the target (e.g. the node runtime
7607 ``nodejs``). Consequently, you need to find out what files are
7608 missing and be sure they are on the target.
7609
7610- Although you might not need NPM to run your node package, it is
7611 useful to have NPM on your target. The NPM package name is
7612 ``nodejs-npm``.
7613
7614.. _npm-using-the-registry-modules-method:
7615
7616Using the Registry Modules Method
7617~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
7618
7619This section presents an example that uses the ``cute-files`` module,
7620which is a file browser web application.
7621
7622.. note::
7623
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007624 You must know the ``cute-files`` module version.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007625
7626The first thing you need to do is use ``devtool`` and the NPM fetcher to
7627create the recipe:
7628::
7629
7630 $ devtool add "npm://registry.npmjs.org;package=cute-files;version=1.0.2"
7631
7632The
7633``devtool add`` command runs ``recipetool create`` and uses the same
7634fetch URI to download each dependency and capture license details where
7635possible. The result is a generated recipe.
7636
7637The recipe file is fairly simple and contains every license that
7638``recipetool`` finds and includes the licenses in the recipe's
7639:term:`LIC_FILES_CHKSUM`
7640variables. You need to examine the variables and look for those with
7641"unknown" in the :term:`LICENSE`
7642field. You need to track down the license information for "unknown"
7643modules and manually add the information to the recipe.
7644
7645``recipetool`` creates a "shrinkwrap" file for your recipe. Shrinkwrap
7646files capture the version of all dependent modules. Many packages do not
7647provide shrinkwrap files. ``recipetool`` create a shrinkwrap file as it
7648runs.
7649
7650.. note::
7651
7652 A package is created for each sub-module. This policy is the only
7653 practical way to have the licenses for all of the dependencies
7654 represented in the license manifest of the image.
7655
7656The ``devtool edit-recipe`` command lets you take a look at the recipe:
7657::
7658
7659 $ devtool edit-recipe cute-files
7660 SUMMARY = "Turn any folder on your computer into a cute file browser, available on the local network."
7661 LICENSE = "MIT & ISC & Unknown"
7662 LIC_FILES_CHKSUM = "file://LICENSE;md5=71d98c0a1db42956787b1909c74a86ca \
7663 file://node_modules/toidentifier/LICENSE;md5=1a261071a044d02eb6f2bb47f51a3502 \
7664 file://node_modules/debug/LICENSE;md5=ddd815a475e7338b0be7a14d8ee35a99 \
7665 ...
7666 SRC_URI = " \
7667 npm://registry.npmjs.org/;package=cute-files;version=${PV} \
7668 npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \
7669 "
7670 S = "${WORKDIR}/npm"
7671 inherit npm LICENSE_${PN} = "MIT"
7672 LICENSE_${PN}-accepts = "MIT"
7673 LICENSE_${PN}-array-flatten = "MIT"
7674 ...
7675 LICENSE_${PN}-vary = "MIT"
7676
7677Three key points exist in the previous example:
7678
7679- :term:`SRC_URI` uses the NPM
7680 scheme so that the NPM fetcher is used.
7681
7682- ``recipetool`` collects all the license information. If a
7683 sub-module's license is unavailable, the sub-module's name appears in
7684 the comments.
7685
7686- The ``inherit npm`` statement causes the
7687 :ref:`npm <ref-classes-npm>` class to package
7688 up all the modules.
7689
7690You can run the following command to build the ``cute-files`` package:
7691::
7692
7693 $ devtool build cute-files
7694
7695Remember that ``nodejs`` must be installed on
7696the target before your package.
7697
7698Assuming 192.168.7.2 for the target's IP address, use the following
7699command to deploy your package:
7700::
7701
7702 $ devtool deploy-target -s cute-files root@192.168.7.2
7703
7704Once the package is installed on the target, you can
7705test the application:
7706
7707.. note::
7708
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007709 Because of a known issue, you cannot simply run ``cute-files`` as you would
7710 if you had run ``npm install``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007711
7712::
7713
7714 $ cd /usr/lib/node_modules/cute-files
7715 $ node cute-files.js
7716
7717On a browser,
7718go to ``http://192.168.7.2:3000`` and you see the following:
7719
7720.. image:: figures/cute-files-npm-example.png
7721 :align: center
7722
7723You can find the recipe in ``workspace/recipes/cute-files``. You can use
7724the recipe in any layer you choose.
7725
7726.. _npm-using-the-npm-projects-method:
7727
7728Using the NPM Projects Code Method
7729~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
7730
7731Although it is useful to package modules already in the NPM registry,
7732adding ``node.js`` projects under development is a more common developer
7733use case.
7734
7735This section covers the NPM projects code method, which is very similar
7736to the "registry" approach described in the previous section. In the NPM
7737projects method, you provide ``devtool`` with an URL that points to the
7738source files.
7739
7740Replicating the same example, (i.e. ``cute-files``) use the following
7741command:
7742::
7743
7744 $ devtool add https://github.com/martinaglv/cute-files.git
7745
7746The
7747recipe this command generates is very similar to the recipe created in
7748the previous section. However, the ``SRC_URI`` looks like the following:
7749::
7750
7751 SRC_URI = " \
7752 git://github.com/martinaglv/cute-files.git;protocol=https \
7753 npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \
7754 "
7755
7756In this example,
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007757the main module is taken from the Git repository and dependencies are
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007758taken from the NPM registry. Other than those differences, the recipe is
7759basically the same between the two methods. You can build and deploy the
7760package exactly as described in the previous section that uses the
7761registry modules method.
7762
7763Adding custom metadata to packages
7764----------------------------------
7765
7766The variable
7767:term:`PACKAGE_ADD_METADATA`
7768can be used to add additional metadata to packages. This is reflected in
7769the package control/spec file. To take the ipk format for example, the
7770CONTROL file stored inside would contain the additional metadata as
7771additional lines.
7772
7773The variable can be used in multiple ways, including using suffixes to
7774set it for a specific package type and/or package. Note that the order
7775of precedence is the same as this list:
7776
7777- ``PACKAGE_ADD_METADATA_<PKGTYPE>_<PN>``
7778
7779- ``PACKAGE_ADD_METADATA_<PKGTYPE>``
7780
7781- ``PACKAGE_ADD_METADATA_<PN>``
7782
7783- ``PACKAGE_ADD_METADATA``
7784
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007785`<PKGTYPE>` is a parameter and expected to be a distinct name of specific
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007786package type:
7787
7788- IPK for .ipk packages
7789
7790- DEB for .deb packages
7791
7792- RPM for .rpm packages
7793
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007794`<PN>` is a parameter and expected to be a package name.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007795
7796The variable can contain multiple [one-line] metadata fields separated
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007797by the literal sequence '\\n'. The separator can be redefined using the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007798variable flag ``separator``.
7799
7800The following is an example that adds two custom fields for ipk
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007801packages:
7802::
7803
7804 PACKAGE_ADD_METADATA_IPK = "Vendor: CustomIpk\nGroup:Applications/Spreadsheets"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007805
7806Efficiently Fetching Source Files During a Build
7807================================================
7808
7809The OpenEmbedded build system works with source files located through
7810the :term:`SRC_URI` variable. When
7811you build something using BitBake, a big part of the operation is
7812locating and downloading all the source tarballs. For images,
7813downloading all the source for various packages can take a significant
7814amount of time.
7815
7816This section shows you how you can use mirrors to speed up fetching
7817source files and how you can pre-fetch files all of which leads to more
7818efficient use of resources and time.
7819
7820Setting up Effective Mirrors
7821----------------------------
7822
7823A good deal that goes into a Yocto Project build is simply downloading
7824all of the source tarballs. Maybe you have been working with another
7825build system (OpenEmbedded or Angstrom) for which you have built up a
7826sizable directory of source tarballs. Or, perhaps someone else has such
7827a directory for which you have read access. If so, you can save time by
7828adding statements to your configuration file so that the build process
7829checks local directories first for existing tarballs before checking the
7830Internet.
7831
7832Here is an efficient way to set it up in your ``local.conf`` file:
7833::
7834
7835 SOURCE_MIRROR_URL ?= "file:///home/you/your-download-dir/"
7836 INHERIT += "own-mirrors"
7837 BB_GENERATE_MIRROR_TARBALLS = "1"
7838 # BB_NO_NETWORK = "1"
7839
7840In the previous example, the
7841:term:`BB_GENERATE_MIRROR_TARBALLS`
7842variable causes the OpenEmbedded build system to generate tarballs of
7843the Git repositories and store them in the
7844:term:`DL_DIR` directory. Due to
7845performance reasons, generating and storing these tarballs is not the
7846build system's default behavior.
7847
7848You can also use the
7849:term:`PREMIRRORS` variable. For
7850an example, see the variable's glossary entry in the Yocto Project
7851Reference Manual.
7852
7853Getting Source Files and Suppressing the Build
7854----------------------------------------------
7855
7856Another technique you can use to ready yourself for a successive string
7857of build operations, is to pre-fetch all the source files without
7858actually starting a build. This technique lets you work through any
7859download issues and ultimately gathers all the source files into your
7860download directory :ref:`structure-build-downloads`,
7861which is located with :term:`DL_DIR`.
7862
7863Use the following BitBake command form to fetch all the necessary
7864sources without starting the build:
7865::
7866
7867 $ bitbake target --runall=fetch
7868
7869This
7870variation of the BitBake command guarantees that you have all the
7871sources for that BitBake target should you disconnect from the Internet
7872and want to do the build later offline.
7873
7874Selecting an Initialization Manager
7875===================================
7876
7877By default, the Yocto Project uses SysVinit as the initialization
7878manager. However, support also exists for systemd, which is a full
7879replacement for init with parallel starting of services, reduced shell
7880overhead and other features that are used by many distributions.
7881
7882Within the system, SysVinit treats system components as services. These
7883services are maintained as shell scripts stored in the ``/etc/init.d/``
7884directory. Services organize into different run levels. This
7885organization is maintained by putting links to the services in the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007886``/etc/rcN.d/`` directories, where `N/` is one of the following options:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007887"S", "0", "1", "2", "3", "4", "5", or "6".
7888
7889.. note::
7890
7891 Each runlevel has a dependency on the previous runlevel. This
7892 dependency allows the services to work properly.
7893
7894In comparison, systemd treats components as units. Using units is a
7895broader concept as compared to using a service. A unit includes several
7896different types of entities. Service is one of the types of entities.
7897The runlevel concept in SysVinit corresponds to the concept of a target
7898in systemd, where target is also a type of supported unit.
7899
7900In a SysVinit-based system, services load sequentially (i.e. one by one)
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007901during init and parallelization is not supported. With systemd, services
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007902start in parallel. Needless to say, the method can have an impact on
7903system startup performance.
7904
7905If you want to use SysVinit, you do not have to do anything. But, if you
7906want to use systemd, you must take some steps as described in the
7907following sections.
7908
7909Using systemd Exclusively
7910-------------------------
7911
7912Set these variables in your distribution configuration file as follows:
7913::
7914
7915 DISTRO_FEATURES_append = " systemd"
7916 VIRTUAL-RUNTIME_init_manager = "systemd"
7917
7918You can also prevent the SysVinit distribution feature from
7919being automatically enabled as follows:
7920::
7921
7922 DISTRO_FEATURES_BACKFILL_CONSIDERED = "sysvinit"
7923
7924Doing so removes any
7925redundant SysVinit scripts.
7926
7927To remove initscripts from your image altogether, set this variable
7928also:
7929::
7930
7931 VIRTUAL-RUNTIME_initscripts = ""
7932
7933For information on the backfill variable, see
7934:term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`.
7935
7936Using systemd for the Main Image and Using SysVinit for the Rescue Image
7937------------------------------------------------------------------------
7938
7939Set these variables in your distribution configuration file as follows:
7940::
7941
7942 DISTRO_FEATURES_append = " systemd"
7943 VIRTUAL-RUNTIME_init_manager = "systemd"
7944
7945Doing so causes your main image to use the
7946``packagegroup-core-boot.bb`` recipe and systemd. The rescue/minimal
7947image cannot use this package group. However, it can install SysVinit
7948and the appropriate packages will have support for both systemd and
7949SysVinit.
7950
7951.. _selecting-dev-manager:
7952
7953Selecting a Device Manager
7954==========================
7955
7956The Yocto Project provides multiple ways to manage the device manager
7957(``/dev``):
7958
7959- Persistent and Pre-Populated\ ``/dev``: For this case, the ``/dev``
7960 directory is persistent and the required device nodes are created
7961 during the build.
7962
7963- Use ``devtmpfs`` with a Device Manager: For this case, the ``/dev``
7964 directory is provided by the kernel as an in-memory file system and
7965 is automatically populated by the kernel at runtime. Additional
7966 configuration of device nodes is done in user space by a device
7967 manager like ``udev`` or ``busybox-mdev``.
7968
7969.. _static-dev-management:
7970
7971Using Persistent and Pre-Populated\ ``/dev``
7972--------------------------------------------
7973
7974To use the static method for device population, you need to set the
7975:term:`USE_DEVFS` variable to "0"
7976as follows:
7977::
7978
7979 USE_DEVFS = "0"
7980
7981The content of the resulting ``/dev`` directory is defined in a Device
7982Table file. The
7983:term:`IMAGE_DEVICE_TABLES`
7984variable defines the Device Table to use and should be set in the
7985machine or distro configuration file. Alternatively, you can set this
7986variable in your ``local.conf`` configuration file.
7987
7988If you do not define the ``IMAGE_DEVICE_TABLES`` variable, the default
7989``device_table-minimal.txt`` is used:
7990::
7991
7992 IMAGE_DEVICE_TABLES = "device_table-mymachine.txt"
7993
7994The population is handled by the ``makedevs`` utility during image
7995creation:
7996
7997.. _devtmpfs-dev-management:
7998
7999Using ``devtmpfs`` and a Device Manager
8000---------------------------------------
8001
8002To use the dynamic method for device population, you need to use (or be
8003sure to set) the :term:`USE_DEVFS`
8004variable to "1", which is the default:
8005::
8006
8007 USE_DEVFS = "1"
8008
8009With this
8010setting, the resulting ``/dev`` directory is populated by the kernel
8011using ``devtmpfs``. Make sure the corresponding kernel configuration
8012variable ``CONFIG_DEVTMPFS`` is set when building you build a Linux
8013kernel.
8014
8015All devices created by ``devtmpfs`` will be owned by ``root`` and have
8016permissions ``0600``.
8017
8018To have more control over the device nodes, you can use a device manager
8019like ``udev`` or ``busybox-mdev``. You choose the device manager by
8020defining the ``VIRTUAL-RUNTIME_dev_manager`` variable in your machine or
8021distro configuration file. Alternatively, you can set this variable in
8022your ``local.conf`` configuration file:
8023::
8024
8025 VIRTUAL-RUNTIME_dev_manager = "udev"
8026
8027 # Some alternative values
8028 # VIRTUAL-RUNTIME_dev_manager = "busybox-mdev"
8029 # VIRTUAL-RUNTIME_dev_manager = "systemd"
8030
8031.. _platdev-appdev-srcrev:
8032
8033Using an External SCM
8034=====================
8035
8036If you're working on a recipe that pulls from an external Source Code
8037Manager (SCM), it is possible to have the OpenEmbedded build system
8038notice new recipe changes added to the SCM and then build the resulting
8039packages that depend on the new recipes by using the latest versions.
8040This only works for SCMs from which it is possible to get a sensible
8041revision number for changes. Currently, you can do this with Apache
8042Subversion (SVN), Git, and Bazaar (BZR) repositories.
8043
8044To enable this behavior, the :term:`PV` of
8045the recipe needs to reference
8046:term:`SRCPV`. Here is an example:
8047::
8048
8049 PV = "1.2.3+git${SRCPV}"
8050
8051Then, you can add the following to your
8052``local.conf``:
8053::
8054
8055 SRCREV_pn-PN = "${AUTOREV}"
8056
8057:term:`PN` is the name of the recipe for
8058which you want to enable automatic source revision updating.
8059
8060If you do not want to update your local configuration file, you can add
8061the following directly to the recipe to finish enabling the feature:
8062::
8063
8064 SRCREV = "${AUTOREV}"
8065
8066The Yocto Project provides a distribution named ``poky-bleeding``, whose
8067configuration file contains the line:
8068::
8069
8070 require conf/distro/include/poky-floating-revisions.inc
8071
8072This line pulls in the
8073listed include file that contains numerous lines of exactly that form:
8074::
8075
8076 #SRCREV_pn-opkg-native ?= "${AUTOREV}"
8077 #SRCREV_pn-opkg-sdk ?= "${AUTOREV}"
8078 #SRCREV_pn-opkg ?= "${AUTOREV}"
8079 #SRCREV_pn-opkg-utils-native ?= "${AUTOREV}"
8080 #SRCREV_pn-opkg-utils ?= "${AUTOREV}"
8081 SRCREV_pn-gconf-dbus ?= "${AUTOREV}"
8082 SRCREV_pn-matchbox-common ?= "${AUTOREV}"
8083 SRCREV_pn-matchbox-config-gtk ?= "${AUTOREV}"
8084 SRCREV_pn-matchbox-desktop ?= "${AUTOREV}"
8085 SRCREV_pn-matchbox-keyboard ?= "${AUTOREV}"
8086 SRCREV_pn-matchbox-panel-2 ?= "${AUTOREV}"
8087 SRCREV_pn-matchbox-themes-extra ?= "${AUTOREV}"
8088 SRCREV_pn-matchbox-terminal ?= "${AUTOREV}"
8089 SRCREV_pn-matchbox-wm ?= "${AUTOREV}"
8090 SRCREV_pn-settings-daemon ?= "${AUTOREV}"
8091 SRCREV_pn-screenshot ?= "${AUTOREV}"
8092 . . .
8093
8094These lines allow you to
8095experiment with building a distribution that tracks the latest
8096development source for numerous packages.
8097
8098.. note::
8099
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008100 The ``poky-bleeding`` distribution is not tested on a regular basis. Keep
8101 this in mind if you use it.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008102
8103Creating a Read-Only Root Filesystem
8104====================================
8105
8106Suppose, for security reasons, you need to disable your target device's
8107root filesystem's write permissions (i.e. you need a read-only root
8108filesystem). Or, perhaps you are running the device's operating system
8109from a read-only storage device. For either case, you can customize your
8110image for that behavior.
8111
8112.. note::
8113
8114 Supporting a read-only root filesystem requires that the system and
8115 applications do not try to write to the root filesystem. You must
8116 configure all parts of the target system to write elsewhere, or to
8117 gracefully fail in the event of attempting to write to the root
8118 filesystem.
8119
8120Creating the Root Filesystem
8121----------------------------
8122
8123To create the read-only root filesystem, simply add the
8124"read-only-rootfs" feature to your image, normally in one of two ways.
8125The first way is to add the "read-only-rootfs" image feature in the
8126image's recipe file via the ``IMAGE_FEATURES`` variable:
8127::
8128
8129 IMAGE_FEATURES += "read-only-rootfs"
8130
8131As an alternative, you can add the same feature
8132from within your build directory's ``local.conf`` file with the
8133associated ``EXTRA_IMAGE_FEATURES`` variable, as in:
8134::
8135
8136 EXTRA_IMAGE_FEATURES = "read-only-rootfs"
8137
8138For more information on how to use these variables, see the
8139":ref:`usingpoky-extend-customimage-imagefeatures`"
8140section. For information on the variables, see
8141:term:`IMAGE_FEATURES` and
8142:term:`EXTRA_IMAGE_FEATURES`.
8143
8144Post-Installation Scripts and Read-Only Root Filesystem
8145-------------------------------------------------------
8146
8147It is very important that you make sure all post-Installation
8148(``pkg_postinst``) scripts for packages that are installed into the
8149image can be run at the time when the root filesystem is created during
8150the build on the host system. These scripts cannot attempt to run during
8151first-boot on the target device. With the "read-only-rootfs" feature
8152enabled, the build system checks during root filesystem creation to make
8153sure all post-installation scripts succeed. If any of these scripts
8154still need to be run after the root filesystem is created, the build
8155immediately fails. These build-time checks ensure that the build fails
8156rather than the target device fails later during its initial boot
8157operation.
8158
8159Most of the common post-installation scripts generated by the build
8160system for the out-of-the-box Yocto Project are engineered so that they
8161can run during root filesystem creation (e.g. post-installation scripts
8162for caching fonts). However, if you create and add custom scripts, you
8163need to be sure they can be run during this file system creation.
8164
8165Here are some common problems that prevent post-installation scripts
8166from running during root filesystem creation:
8167
8168- *Not using $D in front of absolute paths:* The build system defines
8169 ``$``\ :term:`D` when the root
8170 filesystem is created. Furthermore, ``$D`` is blank when the script
8171 is run on the target device. This implies two purposes for ``$D``:
8172 ensuring paths are valid in both the host and target environments,
8173 and checking to determine which environment is being used as a method
8174 for taking appropriate actions.
8175
8176- *Attempting to run processes that are specific to or dependent on the
8177 target architecture:* You can work around these attempts by using
8178 native tools, which run on the host system, to accomplish the same
8179 tasks, or by alternatively running the processes under QEMU, which
8180 has the ``qemu_run_binary`` function. For more information, see the
8181 :ref:`qemu <ref-classes-qemu>` class.
8182
8183Areas With Write Access
8184-----------------------
8185
8186With the "read-only-rootfs" feature enabled, any attempt by the target
8187to write to the root filesystem at runtime fails. Consequently, you must
8188make sure that you configure processes and applications that attempt
8189these types of writes do so to directories with write access (e.g.
8190``/tmp`` or ``/var/run``).
8191
8192Maintaining Build Output Quality
8193================================
8194
8195Many factors can influence the quality of a build. For example, if you
8196upgrade a recipe to use a new version of an upstream software package or
8197you experiment with some new configuration options, subtle changes can
8198occur that you might not detect until later. Consider the case where
8199your recipe is using a newer version of an upstream package. In this
8200case, a new version of a piece of software might introduce an optional
8201dependency on another library, which is auto-detected. If that library
8202has already been built when the software is building, the software will
8203link to the built library and that library will be pulled into your
8204image along with the new software even if you did not want the library.
8205
8206The :ref:`buildhistory <ref-classes-buildhistory>`
8207class exists to help you maintain the quality of your build output. You
8208can use the class to highlight unexpected and possibly unwanted changes
8209in the build output. When you enable build history, it records
8210information about the contents of each package and image and then
8211commits that information to a local Git repository where you can examine
8212the information.
8213
8214The remainder of this section describes the following:
8215
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008216- :ref:`How you can enable and disable build history <dev-manual/dev-manual-common-tasks:enabling and disabling build history>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008217
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008218- :ref:`How to understand what the build history contains <dev-manual/dev-manual-common-tasks:understanding what the build history contains>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008219
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008220- :ref:`How to limit the information used for build history <dev-manual/dev-manual-common-tasks:using build history to gather image information only>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008221
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008222- :ref:`How to examine the build history from both a command-line and web interface <dev-manual/dev-manual-common-tasks:examining build history information>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008223
8224Enabling and Disabling Build History
8225------------------------------------
8226
8227Build history is disabled by default. To enable it, add the following
8228``INHERIT`` statement and set the
8229:term:`BUILDHISTORY_COMMIT`
8230variable to "1" at the end of your ``conf/local.conf`` file found in the
8231:term:`Build Directory`:
8232::
8233
8234 INHERIT += "buildhistory"
8235 BUILDHISTORY_COMMIT = "1"
8236
8237Enabling build history as
8238previously described causes the OpenEmbedded build system to collect
8239build output information and commit it as a single commit to a local
8240:ref:`overview-manual/overview-manual-development-environment:git` repository.
8241
8242.. note::
8243
8244 Enabling build history increases your build times slightly,
8245 particularly for images, and increases the amount of disk space used
8246 during the build.
8247
8248You can disable build history by removing the previous statements from
8249your ``conf/local.conf`` file.
8250
8251Understanding What the Build History Contains
8252---------------------------------------------
8253
8254Build history information is kept in
8255``${``\ :term:`TOPDIR`\ ``}/buildhistory``
8256in the Build Directory as defined by the
8257:term:`BUILDHISTORY_DIR`
8258variable. The following is an example abbreviated listing:
8259
8260.. image:: figures/buildhistory.png
8261 :align: center
8262
8263At the top level, a ``metadata-revs`` file exists that lists the
8264revisions of the repositories for the enabled layers when the build was
8265produced. The rest of the data splits into separate ``packages``,
8266``images`` and ``sdk`` directories, the contents of which are described
8267as follows.
8268
8269Build History Package Information
8270~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8271
8272The history for each package contains a text file that has name-value
8273pairs with information about the package. For example,
8274``buildhistory/packages/i586-poky-linux/busybox/busybox/latest``
8275contains the following:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008276
8277.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008278
8279 PV = 1.22.1
8280 PR = r32
8281 RPROVIDES =
8282 RDEPENDS = glibc (>= 2.20) update-alternatives-opkg
8283 RRECOMMENDS = busybox-syslog busybox-udhcpc update-rc.d
8284 PKGSIZE = 540168
8285 FILES = /usr/bin/* /usr/sbin/* /usr/lib/busybox/* /usr/lib/lib*.so.* \
8286 /etc /com /var /bin/* /sbin/* /lib/*.so.* /lib/udev/rules.d \
8287 /usr/lib/udev/rules.d /usr/share/busybox /usr/lib/busybox/* \
8288 /usr/share/pixmaps /usr/share/applications /usr/share/idl \
8289 /usr/share/omf /usr/share/sounds /usr/lib/bonobo/servers
8290 FILELIST = /bin/busybox /bin/busybox.nosuid /bin/busybox.suid /bin/sh \
8291 /etc/busybox.links.nosuid /etc/busybox.links.suid
8292
8293Most of these
8294name-value pairs correspond to variables used to produce the package.
8295The exceptions are ``FILELIST``, which is the actual list of files in
8296the package, and ``PKGSIZE``, which is the total size of files in the
8297package in bytes.
8298
8299A file also exists that corresponds to the recipe from which the package
8300came (e.g. ``buildhistory/packages/i586-poky-linux/busybox/latest``):
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008301
8302.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008303
8304 PV = 1.22.1
8305 PR = r32
8306 DEPENDS = initscripts kern-tools-native update-rc.d-native \
8307 virtual/i586-poky-linux-compilerlibs virtual/i586-poky-linux-gcc \
8308 virtual/libc virtual/update-alternatives
8309 PACKAGES = busybox-ptest busybox-httpd busybox-udhcpd busybox-udhcpc \
8310 busybox-syslog busybox-mdev busybox-hwclock busybox-dbg \
8311 busybox-staticdev busybox-dev busybox-doc busybox-locale busybox
8312
8313Finally, for those recipes fetched from a version control system (e.g.,
8314Git), a file exists that lists source revisions that are specified in
8315the recipe and lists the actual revisions used during the build. Listed
8316and actual revisions might differ when
8317:term:`SRCREV` is set to
8318${:term:`AUTOREV`}. Here is an
8319example assuming
8320``buildhistory/packages/qemux86-poky-linux/linux-yocto/latest_srcrev``):
8321::
8322
8323 # SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
8324 SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
8325 # SRCREV_meta = "a227f20eff056e511d504b2e490f3774ab260d6f"
8326 SRCREV_meta ="a227f20eff056e511d504b2e490f3774ab260d6f"
8327
8328You can use the
8329``buildhistory-collect-srcrevs`` command with the ``-a`` option to
8330collect the stored ``SRCREV`` values from build history and report them
8331in a format suitable for use in global configuration (e.g.,
8332``local.conf`` or a distro include file) to override floating
8333``AUTOREV`` values to a fixed set of revisions. Here is some example
8334output from this command:
8335::
8336
8337 $ buildhistory-collect-srcrevs -a
8338 # i586-poky-linux
8339 SRCREV_pn-glibc = "b8079dd0d360648e4e8de48656c5c38972621072"
8340 SRCREV_pn-glibc-initial = "b8079dd0d360648e4e8de48656c5c38972621072"
8341 SRCREV_pn-opkg-utils = "53274f087565fd45d8452c5367997ba6a682a37a"
8342 SRCREV_pn-kmod = "fd56638aed3fe147015bfa10ed4a5f7491303cb4"
8343 # x86_64-linux
8344 SRCREV_pn-gtk-doc-stub-native = "1dea266593edb766d6d898c79451ef193eb17cfa"
8345 SRCREV_pn-dtc-native = "65cc4d2748a2c2e6f27f1cf39e07a5dbabd80ebf"
8346 SRCREV_pn-update-rc.d-native = "eca680ddf28d024954895f59a241a622dd575c11"
8347 SRCREV_glibc_pn-cross-localedef-native = "b8079dd0d360648e4e8de48656c5c38972621072"
8348 SRCREV_localedef_pn-cross-localedef-native = "c833367348d39dad7ba018990bfdaffaec8e9ed3"
8349 SRCREV_pn-prelink-native = "faa069deec99bf61418d0bab831c83d7c1b797ca"
8350 SRCREV_pn-opkg-utils-native = "53274f087565fd45d8452c5367997ba6a682a37a"
8351 SRCREV_pn-kern-tools-native = "23345b8846fe4bd167efdf1bd8a1224b2ba9a5ff"
8352 SRCREV_pn-kmod-native = "fd56638aed3fe147015bfa10ed4a5f7491303cb4"
8353 # qemux86-poky-linux
8354 SRCREV_machine_pn-linux-yocto = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
8355 SRCREV_meta_pn-linux-yocto = "a227f20eff056e511d504b2e490f3774ab260d6f"
8356 # all-poky-linux
8357 SRCREV_pn-update-rc.d = "eca680ddf28d024954895f59a241a622dd575c11"
8358
8359.. note::
8360
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008361 Here are some notes on using the ``buildhistory-collect-srcrevs`` command:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008362
8363 - By default, only values where the ``SRCREV`` was not hardcoded
8364 (usually when ``AUTOREV`` is used) are reported. Use the ``-a``
8365 option to see all ``SRCREV`` values.
8366
8367 - The output statements might not have any effect if overrides are
8368 applied elsewhere in the build system configuration. Use the
8369 ``-f`` option to add the ``forcevariable`` override to each output
8370 line if you need to work around this restriction.
8371
8372 - The script does apply special handling when building for multiple
8373 machines. However, the script does place a comment before each set
8374 of values that specifies which triplet to which they belong as
8375 previously shown (e.g., ``i586-poky-linux``).
8376
8377Build History Image Information
8378~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8379
8380The files produced for each image are as follows:
8381
8382- ``image-files:`` A directory containing selected files from the root
8383 filesystem. The files are defined by
8384 :term:`BUILDHISTORY_IMAGE_FILES`.
8385
8386- ``build-id.txt:`` Human-readable information about the build
8387 configuration and metadata source revisions. This file contains the
8388 full build header as printed by BitBake.
8389
8390- ``*.dot:`` Dependency graphs for the image that are compatible with
8391 ``graphviz``.
8392
8393- ``files-in-image.txt:`` A list of files in the image with
8394 permissions, owner, group, size, and symlink information.
8395
8396- ``image-info.txt:`` A text file containing name-value pairs with
8397 information about the image. See the following listing example for
8398 more information.
8399
8400- ``installed-package-names.txt:`` A list of installed packages by name
8401 only.
8402
8403- ``installed-package-sizes.txt:`` A list of installed packages ordered
8404 by size.
8405
8406- ``installed-packages.txt:`` A list of installed packages with full
8407 package filenames.
8408
8409.. note::
8410
8411 Installed package information is able to be gathered and produced
8412 even if package management is disabled for the final image.
8413
8414Here is an example of ``image-info.txt``:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008415
8416.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008417
8418 DISTRO = poky
8419 DISTRO_VERSION = 1.7
8420 USER_CLASSES = buildstats image-mklibs image-prelink
8421 IMAGE_CLASSES = image_types
8422 IMAGE_FEATURES = debug-tweaks
8423 IMAGE_LINGUAS =
8424 IMAGE_INSTALL = packagegroup-core-boot run-postinsts
8425 BAD_RECOMMENDATIONS =
8426 NO_RECOMMENDATIONS =
8427 PACKAGE_EXCLUDE =
8428 ROOTFS_POSTPROCESS_COMMAND = write_package_manifest; license_create_manifest; \
8429 write_image_manifest ; buildhistory_list_installed_image ; \
8430 buildhistory_get_image_installed ; ssh_allow_empty_password; \
8431 postinst_enable_logging; rootfs_update_timestamp ; ssh_disable_dns_lookup ;
8432 IMAGE_POSTPROCESS_COMMAND = buildhistory_get_imageinfo ;
8433 IMAGESIZE = 6900
8434
8435Other than ``IMAGESIZE``,
8436which is the total size of the files in the image in Kbytes, the
8437name-value pairs are variables that may have influenced the content of
8438the image. This information is often useful when you are trying to
8439determine why a change in the package or file listings has occurred.
8440
8441Using Build History to Gather Image Information Only
8442~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8443
8444As you can see, build history produces image information, including
8445dependency graphs, so you can see why something was pulled into the
8446image. If you are just interested in this information and not interested
8447in collecting specific package or SDK information, you can enable
8448writing only image information without any history by adding the
8449following to your ``conf/local.conf`` file found in the
8450:term:`Build Directory`:
8451::
8452
8453 INHERIT += "buildhistory"
8454 BUILDHISTORY_COMMIT = "0"
8455 BUILDHISTORY_FEATURES = "image"
8456
8457Here, you set the
8458:term:`BUILDHISTORY_FEATURES`
8459variable to use the image feature only.
8460
8461Build History SDK Information
8462~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8463
8464Build history collects similar information on the contents of SDKs (e.g.
8465``bitbake -c populate_sdk imagename``) as compared to information it
8466collects for images. Furthermore, this information differs depending on
8467whether an extensible or standard SDK is being produced.
8468
8469The following list shows the files produced for SDKs:
8470
8471- ``files-in-sdk.txt:`` A list of files in the SDK with permissions,
8472 owner, group, size, and symlink information. This list includes both
8473 the host and target parts of the SDK.
8474
8475- ``sdk-info.txt:`` A text file containing name-value pairs with
8476 information about the SDK. See the following listing example for more
8477 information.
8478
8479- ``sstate-task-sizes.txt:`` A text file containing name-value pairs
8480 with information about task group sizes (e.g. ``do_populate_sysroot``
8481 tasks have a total size). The ``sstate-task-sizes.txt`` file exists
8482 only when an extensible SDK is created.
8483
8484- ``sstate-package-sizes.txt:`` A text file containing name-value pairs
8485 with information for the shared-state packages and sizes in the SDK.
8486 The ``sstate-package-sizes.txt`` file exists only when an extensible
8487 SDK is created.
8488
8489- ``sdk-files:`` A folder that contains copies of the files mentioned
8490 in ``BUILDHISTORY_SDK_FILES`` if the files are present in the output.
8491 Additionally, the default value of ``BUILDHISTORY_SDK_FILES`` is
8492 specific to the extensible SDK although you can set it differently if
8493 you would like to pull in specific files from the standard SDK.
8494
8495 The default files are ``conf/local.conf``, ``conf/bblayers.conf``,
8496 ``conf/auto.conf``, ``conf/locked-sigs.inc``, and
8497 ``conf/devtool.conf``. Thus, for an extensible SDK, these files get
8498 copied into the ``sdk-files`` directory.
8499
8500- The following information appears under each of the ``host`` and
8501 ``target`` directories for the portions of the SDK that run on the
8502 host and on the target, respectively:
8503
8504 .. note::
8505
8506 The following files for the most part are empty when producing an
8507 extensible SDK because this type of SDK is not constructed from
8508 packages as is the standard SDK.
8509
8510 - ``depends.dot:`` Dependency graph for the SDK that is compatible
8511 with ``graphviz``.
8512
8513 - ``installed-package-names.txt:`` A list of installed packages by
8514 name only.
8515
8516 - ``installed-package-sizes.txt:`` A list of installed packages
8517 ordered by size.
8518
8519 - ``installed-packages.txt:`` A list of installed packages with full
8520 package filenames.
8521
8522Here is an example of ``sdk-info.txt``:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008523
8524.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008525
8526 DISTRO = poky
8527 DISTRO_VERSION = 1.3+snapshot-20130327
8528 SDK_NAME = poky-glibc-i686-arm
8529 SDK_VERSION = 1.3+snapshot
8530 SDKMACHINE =
8531 SDKIMAGE_FEATURES = dev-pkgs dbg-pkgs
8532 BAD_RECOMMENDATIONS =
8533 SDKSIZE = 352712
8534
8535Other than ``SDKSIZE``, which is
8536the total size of the files in the SDK in Kbytes, the name-value pairs
8537are variables that might have influenced the content of the SDK. This
8538information is often useful when you are trying to determine why a
8539change in the package or file listings has occurred.
8540
8541Examining Build History Information
8542~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8543
8544You can examine build history output from the command line or from a web
8545interface.
8546
8547To see any changes that have occurred (assuming you have
8548:term:`BUILDHISTORY_COMMIT` = "1"),
8549you can simply use any Git command that allows you to view the history
8550of a repository. Here is one method:
8551::
8552
8553 $ git log -p
8554
8555You need to realize,
8556however, that this method does show changes that are not significant
8557(e.g. a package's size changing by a few bytes).
8558
8559A command-line tool called ``buildhistory-diff`` does exist, though,
8560that queries the Git repository and prints just the differences that
8561might be significant in human-readable form. Here is an example:
8562::
8563
8564 $ ~/poky/poky/scripts/buildhistory-diff . HEAD^
8565 Changes to images/qemux86_64/glibc/core-image-minimal (files-in-image.txt):
8566 /etc/anotherpkg.conf was added
8567 /sbin/anotherpkg was added
8568 * (installed-package-names.txt):
8569 * anotherpkg was added
8570 Changes to images/qemux86_64/glibc/core-image-minimal (installed-package-names.txt):
8571 anotherpkg was added
8572 packages/qemux86_64-poky-linux/v86d: PACKAGES: added "v86d-extras"
8573 * PR changed from "r0" to "r1"
8574 * PV changed from "0.1.10" to "0.1.12"
8575 packages/qemux86_64-poky-linux/v86d/v86d: PKGSIZE changed from 110579 to 144381 (+30%)
8576 * PR changed from "r0" to "r1"
8577 * PV changed from "0.1.10" to "0.1.12"
8578
8579.. note::
8580
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008581 The ``buildhistory-diff`` tool requires the ``GitPython``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008582 package. Be sure to install it using Pip3 as follows:
8583 ::
8584
8585 $ pip3 install GitPython --user
8586
8587
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008588 Alternatively, you can install ``python3-git`` using the appropriate
8589 distribution package manager (e.g. ``apt-get``, ``dnf``, or ``zipper``).
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008590
8591To see changes to the build history using a web interface, follow the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008592instruction in the ``README`` file
8593:yocto_git:`here </cgit/cgit.cgi/buildhistory-web/>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008594
8595Here is a sample screenshot of the interface:
8596
8597.. image:: figures/buildhistory-web.png
8598 :align: center
8599
8600Performing Automated Runtime Testing
8601====================================
8602
8603The OpenEmbedded build system makes available a series of automated
8604tests for images to verify runtime functionality. You can run these
8605tests on either QEMU or actual target hardware. Tests are written in
8606Python making use of the ``unittest`` module, and the majority of them
8607run commands on the target system over SSH. This section describes how
8608you set up the environment to use these tests, run available tests, and
8609write and add your own tests.
8610
8611For information on the test and QA infrastructure available within the
8612Yocto Project, see the ":ref:`ref-manual/ref-release-process:testing and quality assurance`"
8613section in the Yocto Project Reference Manual.
8614
8615Enabling Tests
8616--------------
8617
8618Depending on whether you are planning to run tests using QEMU or on the
8619hardware, you have to take different steps to enable the tests. See the
8620following subsections for information on how to enable both types of
8621tests.
8622
8623.. _qemu-image-enabling-tests:
8624
8625Enabling Runtime Tests on QEMU
8626~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8627
8628In order to run tests, you need to do the following:
8629
8630- *Set up to avoid interaction with sudo for networking:* To
8631 accomplish this, you must do one of the following:
8632
8633 - Add ``NOPASSWD`` for your user in ``/etc/sudoers`` either for all
8634 commands or just for ``runqemu-ifup``. You must provide the full
8635 path as that can change if you are using multiple clones of the
8636 source repository.
8637
8638 .. note::
8639
8640 On some distributions, you also need to comment out "Defaults
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008641 requiretty" in ``/etc/sudoers``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008642
8643 - Manually configure a tap interface for your system.
8644
8645 - Run as root the script in ``scripts/runqemu-gen-tapdevs``, which
8646 should generate a list of tap devices. This is the option
8647 typically chosen for Autobuilder-type environments.
8648
8649 .. note::
8650
8651 - Be sure to use an absolute path when calling this script
8652 with sudo.
8653
8654 - The package recipe ``qemu-helper-native`` is required to run
8655 this script. Build the package using the following command:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008656 ::
8657
8658 $ bitbake qemu-helper-native
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008659
8660- *Set the DISPLAY variable:* You need to set this variable so that
8661 you have an X server available (e.g. start ``vncserver`` for a
8662 headless machine).
8663
8664- *Be sure your host's firewall accepts incoming connections from
8665 192.168.7.0/24:* Some of the tests (in particular DNF tests) start an
8666 HTTP server on a random high number port, which is used to serve
8667 files to the target. The DNF module serves
8668 ``${WORKDIR}/oe-rootfs-repo`` so it can run DNF channel commands.
8669 That means your host's firewall must accept incoming connections from
8670 192.168.7.0/24, which is the default IP range used for tap devices by
8671 ``runqemu``.
8672
8673- *Be sure your host has the correct packages installed:* Depending
8674 your host's distribution, you need to have the following packages
8675 installed:
8676
8677 - Ubuntu and Debian: ``sysstat`` and ``iproute2``
8678
8679 - OpenSUSE: ``sysstat`` and ``iproute2``
8680
8681 - Fedora: ``sysstat`` and ``iproute``
8682
8683 - CentOS: ``sysstat`` and ``iproute``
8684
8685Once you start running the tests, the following happens:
8686
86871. A copy of the root filesystem is written to ``${WORKDIR}/testimage``.
8688
86892. The image is booted under QEMU using the standard ``runqemu`` script.
8690
86913. A default timeout of 500 seconds occurs to allow for the boot process
8692 to reach the login prompt. You can change the timeout period by
8693 setting
8694 :term:`TEST_QEMUBOOT_TIMEOUT`
8695 in the ``local.conf`` file.
8696
86974. Once the boot process is reached and the login prompt appears, the
8698 tests run. The full boot log is written to
8699 ``${WORKDIR}/testimage/qemu_boot_log``.
8700
87015. Each test module loads in the order found in ``TEST_SUITES``. You can
8702 find the full output of the commands run over SSH in
8703 ``${WORKDIR}/testimgage/ssh_target_log``.
8704
87056. If no failures occur, the task running the tests ends successfully.
8706 You can find the output from the ``unittest`` in the task log at
8707 ``${WORKDIR}/temp/log.do_testimage``.
8708
8709.. _hardware-image-enabling-tests:
8710
8711Enabling Runtime Tests on Hardware
8712~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8713
8714The OpenEmbedded build system can run tests on real hardware, and for
8715certain devices it can also deploy the image to be tested onto the
8716device beforehand.
8717
8718For automated deployment, a "master image" is installed onto the
8719hardware once as part of setup. Then, each time tests are to be run, the
8720following occurs:
8721
87221. The master image is booted into and used to write the image to be
8723 tested to a second partition.
8724
87252. The device is then rebooted using an external script that you need to
8726 provide.
8727
87283. The device boots into the image to be tested.
8729
8730When running tests (independent of whether the image has been deployed
8731automatically or not), the device is expected to be connected to a
8732network on a pre-determined IP address. You can either use static IP
8733addresses written into the image, or set the image to use DHCP and have
8734your DHCP server on the test network assign a known IP address based on
8735the MAC address of the device.
8736
8737In order to run tests on hardware, you need to set ``TEST_TARGET`` to an
8738appropriate value. For QEMU, you do not have to change anything, the
8739default value is "qemu". For running tests on hardware, the following
8740options exist:
8741
8742- *"simpleremote":* Choose "simpleremote" if you are going to run tests
8743 on a target system that is already running the image to be tested and
8744 is available on the network. You can use "simpleremote" in
8745 conjunction with either real hardware or an image running within a
8746 separately started QEMU or any other virtual machine manager.
8747
8748- *"SystemdbootTarget":* Choose "SystemdbootTarget" if your hardware is
8749 an EFI-based machine with ``systemd-boot`` as bootloader and
8750 ``core-image-testmaster`` (or something similar) is installed. Also,
8751 your hardware under test must be in a DHCP-enabled network that gives
8752 it the same IP address for each reboot.
8753
8754 If you choose "SystemdbootTarget", there are additional requirements
8755 and considerations. See the "`Selecting
8756 SystemdbootTarget <#selecting-systemdboottarget>`__" section, which
8757 follows, for more information.
8758
8759- *"BeagleBoneTarget":* Choose "BeagleBoneTarget" if you are deploying
8760 images and running tests on the BeagleBone "Black" or original
8761 "White" hardware. For information on how to use these tests, see the
8762 comments at the top of the BeagleBoneTarget
8763 ``meta-yocto-bsp/lib/oeqa/controllers/beaglebonetarget.py`` file.
8764
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008765- *"EdgeRouterTarget":* Choose "EdgeRouterTarget" if you are deploying
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008766 images and running tests on the Ubiquiti Networks EdgeRouter Lite.
8767 For information on how to use these tests, see the comments at the
8768 top of the EdgeRouterTarget
8769 ``meta-yocto-bsp/lib/oeqa/controllers/edgeroutertarget.py`` file.
8770
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008771- *"GrubTarget":* Choose "GrubTarget" if you are deploying images and running
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008772 tests on any generic PC that boots using GRUB. For information on how
8773 to use these tests, see the comments at the top of the GrubTarget
8774 ``meta-yocto-bsp/lib/oeqa/controllers/grubtarget.py`` file.
8775
8776- *"your-target":* Create your own custom target if you want to run
8777 tests when you are deploying images and running tests on a custom
8778 machine within your BSP layer. To do this, you need to add a Python
8779 unit that defines the target class under ``lib/oeqa/controllers/``
8780 within your layer. You must also provide an empty ``__init__.py``.
8781 For examples, see files in ``meta-yocto-bsp/lib/oeqa/controllers/``.
8782
8783Selecting SystemdbootTarget
8784~~~~~~~~~~~~~~~~~~~~~~~~~~~
8785
8786If you did not set ``TEST_TARGET`` to "SystemdbootTarget", then you do
8787not need any information in this section. You can skip down to the
8788"`Running Tests <#qemu-image-running-tests>`__" section.
8789
8790If you did set ``TEST_TARGET`` to "SystemdbootTarget", you also need to
8791perform a one-time setup of your master image by doing the following:
8792
87931. *Set EFI_PROVIDER:* Be sure that ``EFI_PROVIDER`` is as follows:
8794 ::
8795
8796 EFI_PROVIDER = "systemd-boot"
8797
87982. *Build the master image:* Build the ``core-image-testmaster`` image.
8799 The ``core-image-testmaster`` recipe is provided as an example for a
8800 "master" image and you can customize the image recipe as you would
8801 any other recipe.
8802
8803 Here are the image recipe requirements:
8804
8805 - Inherits ``core-image`` so that kernel modules are installed.
8806
8807 - Installs normal linux utilities not busybox ones (e.g. ``bash``,
8808 ``coreutils``, ``tar``, ``gzip``, and ``kmod``).
8809
8810 - Uses a custom Initial RAM Disk (initramfs) image with a custom
8811 installer. A normal image that you can install usually creates a
8812 single rootfs partition. This image uses another installer that
8813 creates a specific partition layout. Not all Board Support
8814 Packages (BSPs) can use an installer. For such cases, you need to
8815 manually create the following partition layout on the target:
8816
8817 - First partition mounted under ``/boot``, labeled "boot".
8818
8819 - The main rootfs partition where this image gets installed,
8820 which is mounted under ``/``.
8821
8822 - Another partition labeled "testrootfs" where test images get
8823 deployed.
8824
88253. *Install image:* Install the image that you just built on the target
8826 system.
8827
8828The final thing you need to do when setting ``TEST_TARGET`` to
8829"SystemdbootTarget" is to set up the test image:
8830
88311. *Set up your local.conf file:* Make sure you have the following
8832 statements in your ``local.conf`` file:
8833 ::
8834
8835 IMAGE_FSTYPES += "tar.gz"
8836 INHERIT += "testimage"
8837 TEST_TARGET = "SystemdbootTarget"
8838 TEST_TARGET_IP = "192.168.2.3"
8839
88402. *Build your test image:* Use BitBake to build the image:
8841 ::
8842
8843 $ bitbake core-image-sato
8844
8845Power Control
8846~~~~~~~~~~~~~
8847
8848For most hardware targets other than "simpleremote", you can control
8849power:
8850
8851- You can use ``TEST_POWERCONTROL_CMD`` together with
8852 ``TEST_POWERCONTROL_EXTRA_ARGS`` as a command that runs on the host
8853 and does power cycling. The test code passes one argument to that
8854 command: off, on or cycle (off then on). Here is an example that
8855 could appear in your ``local.conf`` file:
8856 ::
8857
8858 TEST_POWERCONTROL_CMD = "powercontrol.exp test 10.11.12.1 nuc1"
8859
8860 In this example, the expect
8861 script does the following:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008862
8863 .. code-block:: shell
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008864
8865 ssh test@10.11.12.1 "pyctl nuc1 arg"
8866
8867 It then runs a Python script that controls power for a label called
8868 ``nuc1``.
8869
8870 .. note::
8871
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008872 You need to customize ``TEST_POWERCONTROL_CMD`` and
8873 ``TEST_POWERCONTROL_EXTRA_ARGS`` for your own setup. The one requirement
8874 is that it accepts "on", "off", and "cycle" as the last argument.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008875
8876- When no command is defined, it connects to the device over SSH and
8877 uses the classic reboot command to reboot the device. Classic reboot
8878 is fine as long as the machine actually reboots (i.e. the SSH test
8879 has not failed). It is useful for scenarios where you have a simple
8880 setup, typically with a single board, and where some manual
8881 interaction is okay from time to time.
8882
8883If you have no hardware to automatically perform power control but still
8884wish to experiment with automated hardware testing, you can use the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008885``dialog-power-control`` script that shows a dialog prompting you to perform
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008886the required power action. This script requires either KDialog or Zenity
8887to be installed. To use this script, set the
8888:term:`TEST_POWERCONTROL_CMD`
8889variable as follows:
8890::
8891
8892 TEST_POWERCONTROL_CMD = "${COREBASE}/scripts/contrib/dialog-power-control"
8893
8894Serial Console Connection
8895~~~~~~~~~~~~~~~~~~~~~~~~~
8896
8897For test target classes requiring a serial console to interact with the
8898bootloader (e.g. BeagleBoneTarget, EdgeRouterTarget, and GrubTarget),
8899you need to specify a command to use to connect to the serial console of
8900the target machine by using the
8901:term:`TEST_SERIALCONTROL_CMD`
8902variable and optionally the
8903:term:`TEST_SERIALCONTROL_EXTRA_ARGS`
8904variable.
8905
8906These cases could be a serial terminal program if the machine is
8907connected to a local serial port, or a ``telnet`` or ``ssh`` command
8908connecting to a remote console server. Regardless of the case, the
8909command simply needs to connect to the serial console and forward that
8910connection to standard input and output as any normal terminal program
8911does. For example, to use the picocom terminal program on serial device
8912``/dev/ttyUSB0`` at 115200bps, you would set the variable as follows:
8913::
8914
8915 TEST_SERIALCONTROL_CMD = "picocom /dev/ttyUSB0 -b 115200"
8916
8917For local
8918devices where the serial port device disappears when the device reboots,
8919an additional "serdevtry" wrapper script is provided. To use this
8920wrapper, simply prefix the terminal command with
8921``${COREBASE}/scripts/contrib/serdevtry``:
8922::
8923
8924 TEST_SERIALCONTROL_CMD = "${COREBASE}/scripts/contrib/serdevtry picocom -b 115200 /dev/ttyUSB0"
8925
8926.. _qemu-image-running-tests:
8927
8928Running Tests
8929-------------
8930
8931You can start the tests automatically or manually:
8932
8933- *Automatically running tests:* To run the tests automatically after
8934 the OpenEmbedded build system successfully creates an image, first
8935 set the
8936 :term:`TESTIMAGE_AUTO`
8937 variable to "1" in your ``local.conf`` file in the
8938 :term:`Build Directory`:
8939 ::
8940
8941 TESTIMAGE_AUTO = "1"
8942
8943 Next, build your image. If the image successfully builds, the
8944 tests run:
8945 ::
8946
8947 bitbake core-image-sato
8948
8949- *Manually running tests:* To manually run the tests, first globally
8950 inherit the
8951 :ref:`testimage <ref-classes-testimage*>` class
8952 by editing your ``local.conf`` file:
8953 ::
8954
8955 INHERIT += "testimage"
8956
8957 Next, use BitBake to run the tests:
8958 ::
8959
8960 bitbake -c testimage image
8961
8962All test files reside in ``meta/lib/oeqa/runtime`` in the
8963:term:`Source Directory`. A test name maps
8964directly to a Python module. Each test module may contain a number of
8965individual tests. Tests are usually grouped together by the area tested
8966(e.g tests for systemd reside in ``meta/lib/oeqa/runtime/systemd.py``).
8967
8968You can add tests to any layer provided you place them in the proper
8969area and you extend :term:`BBPATH` in
8970the ``local.conf`` file as normal. Be sure that tests reside in
8971``layer/lib/oeqa/runtime``.
8972
8973.. note::
8974
8975 Be sure that module names do not collide with module names used in
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008976 the default set of test modules in ``meta/lib/oeqa/runtime``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008977
8978You can change the set of tests run by appending or overriding
8979:term:`TEST_SUITES` variable in
8980``local.conf``. Each name in ``TEST_SUITES`` represents a required test
8981for the image. Test modules named within ``TEST_SUITES`` cannot be
8982skipped even if a test is not suitable for an image (e.g. running the
8983RPM tests on an image without ``rpm``). Appending "auto" to
8984``TEST_SUITES`` causes the build system to try to run all tests that are
8985suitable for the image (i.e. each test module may elect to skip itself).
8986
8987The order you list tests in ``TEST_SUITES`` is important and influences
8988test dependencies. Consequently, tests that depend on other tests should
8989be added after the test on which they depend. For example, since the
8990``ssh`` test depends on the ``ping`` test, "ssh" needs to come after
8991"ping" in the list. The test class provides no re-ordering or dependency
8992handling.
8993
8994.. note::
8995
8996 Each module can have multiple classes with multiple test methods.
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008997 And, Python ``unittest`` rules apply.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008998
8999Here are some things to keep in mind when running tests:
9000
9001- The default tests for the image are defined as:
9002 ::
9003
9004 DEFAULT_TEST_SUITES_pn-image = "ping ssh df connman syslog xorg scp vnc date rpm dnf dmesg"
9005
9006- Add your own test to the list of the by using the following:
9007 ::
9008
9009 TEST_SUITES_append = " mytest"
9010
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009011- Run a specific list of tests as follows:
9012 ::
9013
9014 TEST_SUITES = "test1 test2 test3"
9015
9016 Remember, order is important. Be sure to place a test that is
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009017 dependent on another test later in the order.
9018
9019Exporting Tests
9020---------------
9021
9022You can export tests so that they can run independently of the build
9023system. Exporting tests is required if you want to be able to hand the
9024test execution off to a scheduler. You can only export tests that are
9025defined in :term:`TEST_SUITES`.
9026
9027If your image is already built, make sure the following are set in your
9028``local.conf`` file:
9029::
9030
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009031 INHERIT += "testexport"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009032 TEST_TARGET_IP = "IP-address-for-the-test-target"
9033 TEST_SERVER_IP = "IP-address-for-the-test-server"
9034
9035You can then export the tests with the
9036following BitBake command form:
9037::
9038
9039 $ bitbake image -c testexport
9040
9041Exporting the tests places them in the
9042:term:`Build Directory` in
9043``tmp/testexport/``\ image, which is controlled by the
9044``TEST_EXPORT_DIR`` variable.
9045
9046You can now run the tests outside of the build environment:
9047::
9048
9049 $ cd tmp/testexport/image
9050 $ ./runexported.py testdata.json
9051
9052Here is a complete example that shows IP addresses and uses the
9053``core-image-sato`` image:
9054::
9055
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009056 INHERIT += "testexport"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009057 TEST_TARGET_IP = "192.168.7.2"
9058 TEST_SERVER_IP = "192.168.7.1"
9059
9060Use BitBake to export the tests:
9061::
9062
9063 $ bitbake core-image-sato -c testexport
9064
9065Run the tests outside of
9066the build environment using the following:
9067::
9068
9069 $ cd tmp/testexport/core-image-sato
9070 $ ./runexported.py testdata.json
9071
9072.. _qemu-image-writing-new-tests:
9073
9074Writing New Tests
9075-----------------
9076
9077As mentioned previously, all new test files need to be in the proper
9078place for the build system to find them. New tests for additional
9079functionality outside of the core should be added to the layer that adds
9080the functionality, in ``layer/lib/oeqa/runtime`` (as long as
9081:term:`BBPATH` is extended in the
9082layer's ``layer.conf`` file as normal). Just remember the following:
9083
9084- Filenames need to map directly to test (module) names.
9085
9086- Do not use module names that collide with existing core tests.
9087
9088- Minimally, an empty ``__init__.py`` file must exist in the runtime
9089 directory.
9090
9091To create a new test, start by copying an existing module (e.g.
9092``syslog.py`` or ``gcc.py`` are good ones to use). Test modules can use
9093code from ``meta/lib/oeqa/utils``, which are helper classes.
9094
9095.. note::
9096
9097 Structure shell commands such that you rely on them and they return a
9098 single code for success. Be aware that sometimes you will need to
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009099 parse the output. See the ``df.py`` and ``date.py`` modules for examples.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009100
9101You will notice that all test classes inherit ``oeRuntimeTest``, which
9102is found in ``meta/lib/oetest.py``. This base class offers some helper
9103attributes, which are described in the following sections:
9104
9105.. _qemu-image-writing-tests-class-methods:
9106
9107Class Methods
9108~~~~~~~~~~~~~
9109
9110Class methods are as follows:
9111
9112- *hasPackage(pkg):* Returns "True" if ``pkg`` is in the installed
9113 package list of the image, which is based on the manifest file that
9114 is generated during the ``do_rootfs`` task.
9115
9116- *hasFeature(feature):* Returns "True" if the feature is in
9117 :term:`IMAGE_FEATURES` or
9118 :term:`DISTRO_FEATURES`.
9119
9120.. _qemu-image-writing-tests-class-attributes:
9121
9122Class Attributes
9123~~~~~~~~~~~~~~~~
9124
9125Class attributes are as follows:
9126
9127- *pscmd:* Equals "ps -ef" if ``procps`` is installed in the image.
9128 Otherwise, ``pscmd`` equals "ps" (busybox).
9129
9130- *tc:* The called test context, which gives access to the
9131 following attributes:
9132
9133 - *d:* The BitBake datastore, which allows you to use stuff such
9134 as ``oeRuntimeTest.tc.d.getVar("VIRTUAL-RUNTIME_init_manager")``.
9135
9136 - *testslist and testsrequired:* Used internally. The tests
9137 do not need these.
9138
9139 - *filesdir:* The absolute path to
9140 ``meta/lib/oeqa/runtime/files``, which contains helper files for
9141 tests meant for copying on the target such as small files written
9142 in C for compilation.
9143
9144 - *target:* The target controller object used to deploy and
9145 start an image on a particular target (e.g. Qemu, SimpleRemote,
9146 and SystemdbootTarget). Tests usually use the following:
9147
9148 - *ip:* The target's IP address.
9149
9150 - *server_ip:* The host's IP address, which is usually used
9151 by the DNF test suite.
9152
9153 - *run(cmd, timeout=None):* The single, most used method.
9154 This command is a wrapper for: ``ssh root@host "cmd"``. The
9155 command returns a tuple: (status, output), which are what their
9156 names imply - the return code of "cmd" and whatever output it
9157 produces. The optional timeout argument represents the number
9158 of seconds the test should wait for "cmd" to return. If the
9159 argument is "None", the test uses the default instance's
9160 timeout period, which is 300 seconds. If the argument is "0",
9161 the test runs until the command returns.
9162
9163 - *copy_to(localpath, remotepath):*
9164 ``scp localpath root@ip:remotepath``.
9165
9166 - *copy_from(remotepath, localpath):*
9167 ``scp root@host:remotepath localpath``.
9168
9169.. _qemu-image-writing-tests-instance-attributes:
9170
9171Instance Attributes
9172~~~~~~~~~~~~~~~~~~~
9173
9174A single instance attribute exists, which is ``target``. The ``target``
9175instance attribute is identical to the class attribute of the same name,
9176which is described in the previous section. This attribute exists as
9177both an instance and class attribute so tests can use
9178``self.target.run(cmd)`` in instance methods instead of
9179``oeRuntimeTest.tc.target.run(cmd)``.
9180
9181Installing Packages in the DUT Without the Package Manager
9182----------------------------------------------------------
9183
9184When a test requires a package built by BitBake, it is possible to
9185install that package. Installing the package does not require a package
9186manager be installed in the device under test (DUT). It does, however,
9187require an SSH connection and the target must be using the
9188``sshcontrol`` class.
9189
9190.. note::
9191
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009192 This method uses ``scp`` to copy files from the host to the target, which
9193 causes permissions and special attributes to be lost.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009194
9195A JSON file is used to define the packages needed by a test. This file
9196must be in the same path as the file used to define the tests.
9197Furthermore, the filename must map directly to the test module name with
9198a ``.json`` extension.
9199
9200The JSON file must include an object with the test name as keys of an
9201object or an array. This object (or array of objects) uses the following
9202data:
9203
9204- "pkg" - A mandatory string that is the name of the package to be
9205 installed.
9206
9207- "rm" - An optional boolean, which defaults to "false", that specifies
9208 to remove the package after the test.
9209
9210- "extract" - An optional boolean, which defaults to "false", that
9211 specifies if the package must be extracted from the package format.
9212 When set to "true", the package is not automatically installed into
9213 the DUT.
9214
9215Following is an example JSON file that handles test "foo" installing
9216package "bar" and test "foobar" installing packages "foo" and "bar".
9217Once the test is complete, the packages are removed from the DUT.
9218::
9219
9220 {
9221 "foo": {
9222 "pkg": "bar"
9223 },
9224 "foobar": [
9225 {
9226 "pkg": "foo",
9227 "rm": true
9228 },
9229 {
9230 "pkg": "bar",
9231 "rm": true
9232 }
9233 ]
9234 }
9235
9236.. _usingpoky-debugging-tools-and-techniques:
9237
9238Debugging Tools and Techniques
9239==============================
9240
9241The exact method for debugging build failures depends on the nature of
9242the problem and on the system's area from which the bug originates.
9243Standard debugging practices such as comparison against the last known
9244working version with examination of the changes and the re-application
9245of steps to identify the one causing the problem are valid for the Yocto
9246Project just as they are for any other system. Even though it is
9247impossible to detail every possible potential failure, this section
9248provides some general tips to aid in debugging given a variety of
9249situations.
9250
9251.. note::
9252
9253 A useful feature for debugging is the error reporting tool.
9254 Configuring the Yocto Project to use this tool causes the
9255 OpenEmbedded build system to produce error reporting commands as part
9256 of the console output. You can enter the commands after the build
9257 completes to log error information into a common database, that can
9258 help you figure out what might be going wrong. For information on how
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009259 to enable and use this feature, see the
9260 ":ref:`dev-manual/dev-manual-common-tasks:using the error reporting tool`"
9261 section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009262
9263The following list shows the debugging topics in the remainder of this
9264section:
9265
9266- "`Viewing Logs from Failed
9267 Tasks <#dev-debugging-viewing-logs-from-failed-tasks>`__" describes
9268 how to find and view logs from tasks that failed during the build
9269 process.
9270
9271- "`Viewing Variable
9272 Values <#dev-debugging-viewing-variable-values>`__" describes how to
9273 use the BitBake ``-e`` option to examine variable values after a
9274 recipe has been parsed.
9275
9276- ":ref:`dev-manual/dev-manual-common-tasks:viewing package information with \`\`oe-pkgdata-util\`\``"
9277 describes how to use the ``oe-pkgdata-util`` utility to query
9278 :term:`PKGDATA_DIR` and
9279 display package-related information for built packages.
9280
9281- "`Viewing Dependencies Between Recipes and
9282 Tasks <#dev-viewing-dependencies-between-recipes-and-tasks>`__"
9283 describes how to use the BitBake ``-g`` option to display recipe
9284 dependency information used during the build.
9285
9286- "`Viewing Task Variable
9287 Dependencies <#dev-viewing-task-variable-dependencies>`__" describes
9288 how to use the ``bitbake-dumpsig`` command in conjunction with key
9289 subdirectories in the
9290 :term:`Build Directory` to determine
9291 variable dependencies.
9292
9293- "`Running Specific Tasks <#dev-debugging-taskrunning>`__" describes
9294 how to use several BitBake options (e.g. ``-c``, ``-C``, and ``-f``)
9295 to run specific tasks in the build chain. It can be useful to run
9296 tasks "out-of-order" when trying isolate build issues.
9297
9298- "`General BitBake Problems <#dev-debugging-bitbake>`__" describes how
9299 to use BitBake's ``-D`` debug output option to reveal more about what
9300 BitBake is doing during the build.
9301
9302- "`Building with No Dependencies <#dev-debugging-buildfile>`__"
9303 describes how to use the BitBake ``-b`` option to build a recipe
9304 while ignoring dependencies.
9305
9306- "`Recipe Logging Mechanisms <#recipe-logging-mechanisms>`__"
9307 describes how to use the many recipe logging functions to produce
9308 debugging output and report errors and warnings.
9309
9310- "`Debugging Parallel Make Races <#debugging-parallel-make-races>`__"
9311 describes how to debug situations where the build consists of several
9312 parts that are run simultaneously and when the output or result of
9313 one part is not ready for use with a different part of the build that
9314 depends on that output.
9315
9316- "`Debugging With the GNU Project Debugger (GDB)
9317 Remotely <#platdev-gdb-remotedebug>`__" describes how to use GDB to
9318 allow you to examine running programs, which can help you fix
9319 problems.
9320
9321- "`Debugging with the GNU Project Debugger (GDB) on the
9322 Target <#debugging-with-the-gnu-project-debugger-gdb-on-the-target>`__"
9323 describes how to use GDB directly on target hardware for debugging.
9324
9325- "`Other Debugging Tips <#dev-other-debugging-others>`__" describes
9326 miscellaneous debugging tips that can be useful.
9327
9328.. _dev-debugging-viewing-logs-from-failed-tasks:
9329
9330Viewing Logs from Failed Tasks
9331------------------------------
9332
9333You can find the log for a task in the file
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009334``${``\ :term:`WORKDIR`\ ``}/temp/log.do_``\ `taskname`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009335For example, the log for the
9336:ref:`ref-tasks-compile` task of the
9337QEMU minimal image for the x86 machine (``qemux86``) might be in
9338``tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/temp/log.do_compile``.
9339To see the commands :term:`BitBake` ran
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009340to generate a log, look at the corresponding ``run.do_``\ `taskname` file
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009341in the same directory.
9342
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009343``log.do_``\ `taskname` and ``run.do_``\ `taskname` are actually symbolic
9344links to ``log.do_``\ `taskname`\ ``.``\ `pid` and
9345``log.run_``\ `taskname`\ ``.``\ `pid`, where `pid` is the PID the task had
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009346when it ran. The symlinks always point to the files corresponding to the
9347most recent run.
9348
9349.. _dev-debugging-viewing-variable-values:
9350
9351Viewing Variable Values
9352-----------------------
9353
9354Sometimes you need to know the value of a variable as a result of
9355BitBake's parsing step. This could be because some unexpected behavior
9356occurred in your project. Perhaps an attempt to :ref:`modify a variable
9357<bitbake:bitbake-user-manual/bitbake-user-manual-metadata:modifying existing
9358variables>` did not work out as expected.
9359
9360BitBake's ``-e`` option is used to display variable values after
9361parsing. The following command displays the variable values after the
9362configuration files (i.e. ``local.conf``, ``bblayers.conf``,
9363``bitbake.conf`` and so forth) have been parsed:
9364::
9365
9366 $ bitbake -e
9367
9368The following command displays variable values after a specific recipe has
9369been parsed. The variables include those from the configuration as well:
9370::
9371
9372 $ bitbake -e recipename
9373
9374.. note::
9375
9376 Each recipe has its own private set of variables (datastore).
9377 Internally, after parsing the configuration, a copy of the resulting
9378 datastore is made prior to parsing each recipe. This copying implies
9379 that variables set in one recipe will not be visible to other
9380 recipes.
9381
9382 Likewise, each task within a recipe gets a private datastore based on
9383 the recipe datastore, which means that variables set within one task
9384 will not be visible to other tasks.
9385
9386In the output of ``bitbake -e``, each variable is preceded by a
9387description of how the variable got its value, including temporary
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009388values that were later overridden. This description also includes
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009389variable flags (varflags) set on the variable. The output can be very
9390helpful during debugging.
9391
9392Variables that are exported to the environment are preceded by
9393``export`` in the output of ``bitbake -e``. See the following example:
9394::
9395
9396 export CC="i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/ulf/poky/build/tmp/sysroots/qemux86"
9397
9398In addition to variable values, the output of the ``bitbake -e`` and
9399``bitbake -e`` recipe commands includes the following information:
9400
9401- The output starts with a tree listing all configuration files and
9402 classes included globally, recursively listing the files they include
9403 or inherit in turn. Much of the behavior of the OpenEmbedded build
9404 system (including the behavior of the :ref:`ref-manual/ref-tasks:normal recipe build tasks`) is
9405 implemented in the
9406 :ref:`base <ref-classes-base>` class and the
9407 classes it inherits, rather than being built into BitBake itself.
9408
9409- After the variable values, all functions appear in the output. For
9410 shell functions, variables referenced within the function body are
9411 expanded. If a function has been modified using overrides or using
9412 override-style operators like ``_append`` and ``_prepend``, then the
9413 final assembled function body appears in the output.
9414
9415Viewing Package Information with ``oe-pkgdata-util``
9416----------------------------------------------------
9417
9418You can use the ``oe-pkgdata-util`` command-line utility to query
9419:term:`PKGDATA_DIR` and display
9420various package-related information. When you use the utility, you must
9421use it to view information on packages that have already been built.
9422
9423Following are a few of the available ``oe-pkgdata-util`` subcommands.
9424
9425.. note::
9426
9427 You can use the standard \* and ? globbing wildcards as part of
9428 package names and paths.
9429
9430- ``oe-pkgdata-util list-pkgs [pattern]``: Lists all packages
9431 that have been built, optionally limiting the match to packages that
9432 match pattern.
9433
9434- ``oe-pkgdata-util list-pkg-files package ...``: Lists the
9435 files and directories contained in the given packages.
9436
9437 .. note::
9438
9439 A different way to view the contents of a package is to look at
9440 the
9441 ``${``\ :term:`WORKDIR`\ ``}/packages-split``
9442 directory of the recipe that generates the package. This directory
9443 is created by the
9444 :ref:`ref-tasks-package` task
9445 and has one subdirectory for each package the recipe generates,
9446 which contains the files stored in that package.
9447
9448 If you want to inspect the ``${WORKDIR}/packages-split``
9449 directory, make sure that
9450 :ref:`rm_work <ref-classes-rm-work>` is not
9451 enabled when you build the recipe.
9452
9453- ``oe-pkgdata-util find-path path ...``: Lists the names of
9454 the packages that contain the given paths. For example, the following
9455 tells us that ``/usr/share/man/man1/make.1`` is contained in the
9456 ``make-doc`` package:
9457 ::
9458
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009459 $ oe-pkgdata-util find-path /usr/share/man/man1/make.1
9460 make-doc: /usr/share/man/man1/make.1
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009461
9462- ``oe-pkgdata-util lookup-recipe package ...``: Lists the name
9463 of the recipes that produce the given packages.
9464
9465For more information on the ``oe-pkgdata-util`` command, use the help
9466facility:
9467::
9468
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009469 $ oe-pkgdata-util --help
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009470 $ oe-pkgdata-util subcommand --help
9471
9472.. _dev-viewing-dependencies-between-recipes-and-tasks:
9473
9474Viewing Dependencies Between Recipes and Tasks
9475----------------------------------------------
9476
9477Sometimes it can be hard to see why BitBake wants to build other recipes
9478before the one you have specified. Dependency information can help you
9479understand why a recipe is built.
9480
9481To generate dependency information for a recipe, run the following
9482command:
9483::
9484
9485 $ bitbake -g recipename
9486
9487This command writes the following files in the current directory:
9488
9489- ``pn-buildlist``: A list of recipes/targets involved in building
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009490 `recipename`. "Involved" here means that at least one task from the
9491 recipe needs to run when building `recipename` from scratch. Targets
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009492 that are in
9493 :term:`ASSUME_PROVIDED`
9494 are not listed.
9495
9496- ``task-depends.dot``: A graph showing dependencies between tasks.
9497
9498The graphs are in
9499`DOT <https://en.wikipedia.org/wiki/DOT_%28graph_description_language%29>`__
9500format and can be converted to images (e.g. using the ``dot`` tool from
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009501`Graphviz <https://www.graphviz.org/>`__).
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009502
9503.. note::
9504
9505 - DOT files use a plain text format. The graphs generated using the
9506 ``bitbake -g`` command are often so large as to be difficult to
9507 read without special pruning (e.g. with Bitbake's ``-I`` option)
9508 and processing. Despite the form and size of the graphs, the
9509 corresponding ``.dot`` files can still be possible to read and
9510 provide useful information.
9511
9512 As an example, the ``task-depends.dot`` file contains lines such
9513 as the following:
9514 ::
9515
9516 "libxslt.do_configure" -> "libxml2.do_populate_sysroot"
9517
9518 The above example line reveals that the
9519 :ref:`ref-tasks-configure`
9520 task in ``libxslt`` depends on the
9521 :ref:`ref-tasks-populate_sysroot`
9522 task in ``libxml2``, which is a normal
9523 :term:`DEPENDS` dependency
9524 between the two recipes.
9525
9526 - For an example of how ``.dot`` files can be processed, see the
9527 ``scripts/contrib/graph-tool`` Python script, which finds and
9528 displays paths between graph nodes.
9529
9530You can use a different method to view dependency information by using
9531the following command:
9532::
9533
9534 $ bitbake -g -u taskexp recipename
9535
9536This command
9537displays a GUI window from which you can view build-time and runtime
9538dependencies for the recipes involved in building recipename.
9539
9540.. _dev-viewing-task-variable-dependencies:
9541
9542Viewing Task Variable Dependencies
9543----------------------------------
9544
9545As mentioned in the
9546":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-execution:checksums (signatures)`" section of the BitBake
9547User Manual, BitBake tries to automatically determine what variables a
9548task depends on so that it can rerun the task if any values of the
9549variables change. This determination is usually reliable. However, if
9550you do things like construct variable names at runtime, then you might
9551have to manually declare dependencies on those variables using
9552``vardeps`` as described in the
9553":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags`" section of the BitBake
9554User Manual.
9555
9556If you are unsure whether a variable dependency is being picked up
9557automatically for a given task, you can list the variable dependencies
9558BitBake has determined by doing the following:
9559
95601. Build the recipe containing the task:
9561::
9562
9563 $ bitbake recipename
9564
95652. Inside the :term:`STAMPS_DIR`
9566 directory, find the signature data (``sigdata``) file that
9567 corresponds to the task. The ``sigdata`` files contain a pickled
9568 Python database of all the metadata that went into creating the input
9569 checksum for the task. As an example, for the
9570 :ref:`ref-tasks-fetch` task of the
9571 ``db`` recipe, the ``sigdata`` file might be found in the following
9572 location:
9573 ::
9574
9575 ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1
9576
9577 For tasks that are accelerated through the shared state
9578 (:ref:`sstate <overview-manual/overview-manual-concepts:shared state cache>`) cache, an
9579 additional ``siginfo`` file is written into
9580 :term:`SSTATE_DIR` along with
9581 the cached task output. The ``siginfo`` files contain exactly the
9582 same information as ``sigdata`` files.
9583
95843. Run ``bitbake-dumpsig`` on the ``sigdata`` or ``siginfo`` file. Here
9585 is an example:
9586 ::
9587
9588 $ bitbake-dumpsig ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1
9589
9590 In the output of the above command, you will find a line like the
9591 following, which lists all the (inferred) variable dependencies for
9592 the task. This list also includes indirect dependencies from
9593 variables depending on other variables, recursively.
9594 ::
9595
9596 Task dependencies: ['PV', 'SRCREV', 'SRC_URI', 'SRC_URI[md5sum]', 'SRC_URI[sha256sum]', 'base_do_fetch']
9597
9598 .. note::
9599
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009600 Functions (e.g. ``base_do_fetch``) also count as variable dependencies.
9601 These functions in turn depend on the variables they reference.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009602
9603 The output of ``bitbake-dumpsig`` also includes the value each
9604 variable had, a list of dependencies for each variable, and
9605 :term:`bitbake:BB_HASHBASE_WHITELIST`
9606 information.
9607
9608There is also a ``bitbake-diffsigs`` command for comparing two
9609``siginfo`` or ``sigdata`` files. This command can be helpful when
9610trying to figure out what changed between two versions of a task. If you
9611call ``bitbake-diffsigs`` with just one file, the command behaves like
9612``bitbake-dumpsig``.
9613
9614You can also use BitBake to dump out the signature construction
9615information without executing tasks by using either of the following
9616BitBake command-line options:
9617::
9618
9619 ‐‐dump-signatures=SIGNATURE_HANDLER
9620 -S SIGNATURE_HANDLER
9621
9622
9623.. note::
9624
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009625 Two common values for `SIGNATURE_HANDLER` are "none" and "printdiff", which
9626 dump only the signature or compare the dumped signature with the cached one,
9627 respectively.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009628
9629Using BitBake with either of these options causes BitBake to dump out
9630``sigdata`` files in the ``stamps`` directory for every task it would
9631have executed instead of building the specified target package.
9632
9633.. _dev-viewing-metadata-used-to-create-the-input-signature-of-a-shared-state-task:
9634
9635Viewing Metadata Used to Create the Input Signature of a Shared State Task
9636--------------------------------------------------------------------------
9637
9638Seeing what metadata went into creating the input signature of a shared
9639state (sstate) task can be a useful debugging aid. This information is
9640available in signature information (``siginfo``) files in
9641:term:`SSTATE_DIR`. For
9642information on how to view and interpret information in ``siginfo``
9643files, see the "`Viewing Task Variable
9644Dependencies <#dev-viewing-task-variable-dependencies>`__" section.
9645
9646For conceptual information on shared state, see the
9647":ref:`overview-manual/overview-manual-concepts:shared state`"
9648section in the Yocto Project Overview and Concepts Manual.
9649
9650.. _dev-invalidating-shared-state-to-force-a-task-to-run:
9651
9652Invalidating Shared State to Force a Task to Run
9653------------------------------------------------
9654
9655The OpenEmbedded build system uses
9656:ref:`checksums <overview-checksums>` and
9657:ref:`overview-manual/overview-manual-concepts:shared state` cache to avoid unnecessarily
9658rebuilding tasks. Collectively, this scheme is known as "shared state
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009659code".
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009660
9661As with all schemes, this one has some drawbacks. It is possible that
9662you could make implicit changes to your code that the checksum
9663calculations do not take into account. These implicit changes affect a
9664task's output but do not trigger the shared state code into rebuilding a
9665recipe. Consider an example during which a tool changes its output.
9666Assume that the output of ``rpmdeps`` changes. The result of the change
9667should be that all the ``package`` and ``package_write_rpm`` shared
9668state cache items become invalid. However, because the change to the
9669output is external to the code and therefore implicit, the associated
9670shared state cache items do not become invalidated. In this case, the
9671build process uses the cached items rather than running the task again.
9672Obviously, these types of implicit changes can cause problems.
9673
9674To avoid these problems during the build, you need to understand the
9675effects of any changes you make. Realize that changes you make directly
9676to a function are automatically factored into the checksum calculation.
9677Thus, these explicit changes invalidate the associated area of shared
9678state cache. However, you need to be aware of any implicit changes that
9679are not obvious changes to the code and could affect the output of a
9680given task.
9681
9682When you identify an implicit change, you can easily take steps to
9683invalidate the cache and force the tasks to run. The steps you can take
9684are as simple as changing a function's comments in the source code. For
9685example, to invalidate package shared state files, change the comment
9686statements of
9687:ref:`ref-tasks-package` or the
9688comments of one of the functions it calls. Even though the change is
9689purely cosmetic, it causes the checksum to be recalculated and forces
9690the build system to run the task again.
9691
9692.. note::
9693
9694 For an example of a commit that makes a cosmetic change to invalidate
9695 shared state, see this
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009696 :yocto_git:`commit </cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009697
9698.. _dev-debugging-taskrunning:
9699
9700Running Specific Tasks
9701----------------------
9702
9703Any given recipe consists of a set of tasks. The standard BitBake
9704behavior in most cases is: ``do_fetch``, ``do_unpack``, ``do_patch``,
9705``do_configure``, ``do_compile``, ``do_install``, ``do_package``,
9706``do_package_write_*``, and ``do_build``. The default task is
9707``do_build`` and any tasks on which it depends build first. Some tasks,
9708such as ``do_devshell``, are not part of the default build chain. If you
9709wish to run a task that is not part of the default build chain, you can
9710use the ``-c`` option in BitBake. Here is an example:
9711::
9712
9713 $ bitbake matchbox-desktop -c devshell
9714
9715The ``-c`` option respects task dependencies, which means that all other
9716tasks (including tasks from other recipes) that the specified task
9717depends on will be run before the task. Even when you manually specify a
9718task to run with ``-c``, BitBake will only run the task if it considers
9719it "out of date". See the
9720":ref:`overview-manual/overview-manual-concepts:stamp files and the rerunning of tasks`"
9721section in the Yocto Project Overview and Concepts Manual for how
9722BitBake determines whether a task is "out of date".
9723
9724If you want to force an up-to-date task to be rerun (e.g. because you
9725made manual modifications to the recipe's
9726:term:`WORKDIR` that you want to try
9727out), then you can use the ``-f`` option.
9728
9729.. note::
9730
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009731 The reason ``-f`` is never required when running the
9732 :ref:`ref-tasks-devshell` task is because the
9733 [\ :ref:`nostamp <bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ]
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009734 variable flag is already set for the task.
9735
9736The following example shows one way you can use the ``-f`` option:
9737::
9738
9739 $ bitbake matchbox-desktop
9740 .
9741 .
9742 make some changes to the source code in the work directory
9743 .
9744 .
9745 $ bitbake matchbox-desktop -c compile -f
9746 $ bitbake matchbox-desktop
9747
9748This sequence first builds and then recompiles ``matchbox-desktop``. The
9749last command reruns all tasks (basically the packaging tasks) after the
9750compile. BitBake recognizes that the ``do_compile`` task was rerun and
9751therefore understands that the other tasks also need to be run again.
9752
9753Another, shorter way to rerun a task and all
9754:ref:`ref-manual/ref-tasks:normal recipe build tasks`
9755that depend on it is to use the ``-C`` option.
9756
9757.. note::
9758
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009759 This option is upper-cased and is separate from the ``-c``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009760 option, which is lower-cased.
9761
9762Using this option invalidates the given task and then runs the
9763:ref:`ref-tasks-build` task, which is
9764the default task if no task is given, and the tasks on which it depends.
9765You could replace the final two commands in the previous example with
9766the following single command:
9767::
9768
9769 $ bitbake matchbox-desktop -C compile
9770
9771Internally, the ``-f`` and ``-C`` options work by tainting (modifying)
9772the input checksum of the specified task. This tainting indirectly
9773causes the task and its dependent tasks to be rerun through the normal
9774task dependency mechanisms.
9775
9776.. note::
9777
9778 BitBake explicitly keeps track of which tasks have been tainted in
9779 this fashion, and will print warnings such as the following for
9780 builds involving such tasks:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009781
9782 .. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009783
9784 WARNING: /home/ulf/poky/meta/recipes-sato/matchbox-desktop/matchbox-desktop_2.1.bb.do_compile is tainted from a forced run
9785
9786
9787 The purpose of the warning is to let you know that the work directory
9788 and build output might not be in the clean state they would be in for
9789 a "normal" build, depending on what actions you took. To get rid of
9790 such warnings, you can remove the work directory and rebuild the
9791 recipe, as follows:
9792 ::
9793
9794 $ bitbake matchbox-desktop -c clean
9795 $ bitbake matchbox-desktop
9796
9797
9798You can view a list of tasks in a given package by running the
9799``do_listtasks`` task as follows:
9800::
9801
9802 $ bitbake matchbox-desktop -c listtasks
9803
9804The results appear as output to the console and are also in
9805the file ``${WORKDIR}/temp/log.do_listtasks``.
9806
9807.. _dev-debugging-bitbake:
9808
9809General BitBake Problems
9810------------------------
9811
9812You can see debug output from BitBake by using the ``-D`` option. The
9813debug output gives more information about what BitBake is doing and the
9814reason behind it. Each ``-D`` option you use increases the logging
9815level. The most common usage is ``-DDD``.
9816
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009817The output from ``bitbake -DDD -v targetname`` can reveal why BitBake
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009818chose a certain version of a package or why BitBake picked a certain
9819provider. This command could also help you in a situation where you
9820think BitBake did something unexpected.
9821
9822.. _dev-debugging-buildfile:
9823
9824Building with No Dependencies
9825-----------------------------
9826
9827To build a specific recipe (``.bb`` file), you can use the following
9828command form:
9829::
9830
9831 $ bitbake -b somepath/somerecipe.bb
9832
9833This command form does
9834not check for dependencies. Consequently, you should use it only when
9835you know existing dependencies have been met.
9836
9837.. note::
9838
9839 You can also specify fragments of the filename. In this case, BitBake
9840 checks for a unique match.
9841
9842Recipe Logging Mechanisms
9843-------------------------
9844
9845The Yocto Project provides several logging functions for producing
9846debugging output and reporting errors and warnings. For Python
9847functions, the following logging functions exist. All of these functions
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009848log to ``${T}/log.do_``\ `task`, and can also log to standard output
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009849(stdout) with the right settings:
9850
9851- ``bb.plain(msg)``: Writes msg as is to the log while also
9852 logging to stdout.
9853
9854- ``bb.note(msg)``: Writes "NOTE: msg" to the log. Also logs to
9855 stdout if BitBake is called with "-v".
9856
9857- ``bb.debug(level, msg)``: Writes "DEBUG: msg" to the
9858 log. Also logs to stdout if the log level is greater than or equal to
9859 level. See the ":ref:`-D <bitbake:bitbake-user-manual/bitbake-user-manual-intro:usage and syntax>`" option
9860 in the BitBake User Manual for more information.
9861
9862- ``bb.warn(msg)``: Writes "WARNING: msg" to the log while also
9863 logging to stdout.
9864
9865- ``bb.error(msg)``: Writes "ERROR: msg" to the log while also
9866 logging to standard out (stdout).
9867
9868 .. note::
9869
9870 Calling this function does not cause the task to fail.
9871
9872- ``bb.fatal(``\ msg\ ``)``: This logging function is similar to
9873 ``bb.error(``\ msg\ ``)`` but also causes the calling task to fail.
9874
9875 .. note::
9876
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009877 ``bb.fatal()`` raises an exception, which means you do not need to put a
9878 "return" statement after the function.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009879
9880The same logging functions are also available in shell functions, under
9881the names ``bbplain``, ``bbnote``, ``bbdebug``, ``bbwarn``, ``bberror``,
9882and ``bbfatal``. The
9883:ref:`logging <ref-classes-logging>` class
9884implements these functions. See that class in the ``meta/classes``
9885folder of the :term:`Source Directory` for information.
9886
9887Logging With Python
9888~~~~~~~~~~~~~~~~~~~
9889
9890When creating recipes using Python and inserting code that handles build
9891logs, keep in mind the goal is to have informative logs while keeping
9892the console as "silent" as possible. Also, if you want status messages
9893in the log, use the "debug" loglevel.
9894
9895Following is an example written in Python. The code handles logging for
9896a function that determines the number of tasks needed to be run. See the
9897":ref:`ref-tasks-listtasks`"
9898section for additional information:
9899::
9900
9901 python do_listtasks() {
9902 bb.debug(2, "Starting to figure out the task list")
9903 if noteworthy_condition:
9904 bb.note("There are 47 tasks to run")
9905 bb.debug(2, "Got to point xyz")
9906 if warning_trigger:
9907 bb.warn("Detected warning_trigger, this might be a problem later.")
9908 if recoverable_error:
9909 bb.error("Hit recoverable_error, you really need to fix this!")
9910 if fatal_error:
9911 bb.fatal("fatal_error detected, unable to print the task list")
9912 bb.plain("The tasks present are abc")
9913 bb.debug(2, "Finished figuring out the tasklist")
9914 }
9915
9916Logging With Bash
9917~~~~~~~~~~~~~~~~~
9918
9919When creating recipes using Bash and inserting code that handles build
9920logs, you have the same goals - informative with minimal console output.
9921The syntax you use for recipes written in Bash is similar to that of
9922recipes written in Python described in the previous section.
9923
9924Following is an example written in Bash. The code logs the progress of
9925the ``do_my_function`` function.
9926::
9927
9928 do_my_function() {
9929 bbdebug 2 "Running do_my_function"
9930 if [ exceptional_condition ]; then
9931 bbnote "Hit exceptional_condition"
9932 fi
9933 bbdebug 2 "Got to point xyz"
9934 if [ warning_trigger ]; then
9935 bbwarn "Detected warning_trigger, this might cause a problem later."
9936 fi
9937 if [ recoverable_error ]; then
9938 bberror "Hit recoverable_error, correcting"
9939 fi
9940 if [ fatal_error ]; then
9941 bbfatal "fatal_error detected"
9942 fi
9943 bbdebug 2 "Completed do_my_function"
9944 }
9945
9946
9947Debugging Parallel Make Races
9948-----------------------------
9949
9950A parallel ``make`` race occurs when the build consists of several parts
9951that are run simultaneously and a situation occurs when the output or
9952result of one part is not ready for use with a different part of the
9953build that depends on that output. Parallel make races are annoying and
9954can sometimes be difficult to reproduce and fix. However, some simple
9955tips and tricks exist that can help you debug and fix them. This section
9956presents a real-world example of an error encountered on the Yocto
9957Project autobuilder and the process used to fix it.
9958
9959.. note::
9960
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009961 If you cannot properly fix a ``make`` race condition, you can work around it
9962 by clearing either the :term:`PARALLEL_MAKE` or :term:`PARALLEL_MAKEINST`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009963 variables.
9964
9965The Failure
9966~~~~~~~~~~~
9967
9968For this example, assume that you are building an image that depends on
9969the "neard" package. And, during the build, BitBake runs into problems
9970and creates the following output.
9971
9972.. note::
9973
9974 This example log file has longer lines artificially broken to make
9975 the listing easier to read.
9976
9977If you examine the output or the log file, you see the failure during
9978``make``:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009979
9980.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009981
9982 | DEBUG: SITE files ['endian-little', 'bit-32', 'ix86-common', 'common-linux', 'common-glibc', 'i586-linux', 'common']
9983 | DEBUG: Executing shell function do_compile
9984 | NOTE: make -j 16
9985 | make --no-print-directory all-am
9986 | /bin/mkdir -p include/near
9987 | /bin/mkdir -p include/near
9988 | /bin/mkdir -p include/near
9989 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9990 0.14-r0/neard-0.14/include/types.h include/near/types.h
9991 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9992 0.14-r0/neard-0.14/include/log.h include/near/log.h
9993 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9994 0.14-r0/neard-0.14/include/plugin.h include/near/plugin.h
9995 | /bin/mkdir -p include/near
9996 | /bin/mkdir -p include/near
9997 | /bin/mkdir -p include/near
9998 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9999 0.14-r0/neard-0.14/include/tag.h include/near/tag.h
10000 | /bin/mkdir -p include/near
10001 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
10002 0.14-r0/neard-0.14/include/adapter.h include/near/adapter.h
10003 | /bin/mkdir -p include/near
10004 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
10005 0.14-r0/neard-0.14/include/ndef.h include/near/ndef.h
10006 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
10007 0.14-r0/neard-0.14/include/tlv.h include/near/tlv.h
10008 | /bin/mkdir -p include/near
10009 | /bin/mkdir -p include/near
10010 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
10011 0.14-r0/neard-0.14/include/setting.h include/near/setting.h
10012 | /bin/mkdir -p include/near
10013 | /bin/mkdir -p include/near
10014 | /bin/mkdir -p include/near
10015 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
10016 0.14-r0/neard-0.14/include/device.h include/near/device.h
10017 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
10018 0.14-r0/neard-0.14/include/nfc_copy.h include/near/nfc_copy.h
10019 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
10020 0.14-r0/neard-0.14/include/snep.h include/near/snep.h
10021 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
10022 0.14-r0/neard-0.14/include/version.h include/near/version.h
10023 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
10024 0.14-r0/neard-0.14/include/dbus.h include/near/dbus.h
10025 | ./src/genbuiltin nfctype1 nfctype2 nfctype3 nfctype4 p2p > src/builtin.h
10026 | i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/
10027 build/build/tmp/sysroots/qemux86 -DHAVE_CONFIG_H -I. -I./include -I./src -I./gdbus -I/home/pokybuild/
10028 yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/sysroots/qemux86/usr/include/glib-2.0
10029 -I/home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/sysroots/qemux86/usr/
10030 lib/glib-2.0/include -I/home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/
10031 tmp/sysroots/qemux86/usr/include/dbus-1.0 -I/home/pokybuild/yocto-autobuilder/yocto-slave/
10032 nightly-x86/build/build/tmp/sysroots/qemux86/usr/lib/dbus-1.0/include -I/home/pokybuild/yocto-autobuilder/
10033 yocto-slave/nightly-x86/build/build/tmp/sysroots/qemux86/usr/include/libnl3
10034 -DNEAR_PLUGIN_BUILTIN -DPLUGINDIR=\""/usr/lib/near/plugins"\"
10035 -DCONFIGDIR=\""/etc/neard\"" -O2 -pipe -g -feliminate-unused-debug-types -c
10036 -o tools/snep-send.o tools/snep-send.c
10037 | In file included from tools/snep-send.c:16:0:
10038 | tools/../src/near.h:41:23: fatal error: near/dbus.h: No such file or directory
10039 | #include <near/dbus.h>
10040 | ^
10041 | compilation terminated.
10042 | make[1]: *** [tools/snep-send.o] Error 1
10043 | make[1]: *** Waiting for unfinished jobs....
10044 | make: *** [all] Error 2
10045 | ERROR: oe_runmake failed
10046
10047Reproducing the Error
10048~~~~~~~~~~~~~~~~~~~~~
10049
10050Because race conditions are intermittent, they do not manifest
10051themselves every time you do the build. In fact, most times the build
10052will complete without problems even though the potential race condition
10053exists. Thus, once the error surfaces, you need a way to reproduce it.
10054
10055In this example, compiling the "neard" package is causing the problem.
10056So the first thing to do is build "neard" locally. Before you start the
10057build, set the
10058:term:`PARALLEL_MAKE` variable
10059in your ``local.conf`` file to a high number (e.g. "-j 20"). Using a
10060high value for ``PARALLEL_MAKE`` increases the chances of the race
10061condition showing up:
10062::
10063
10064 $ bitbake neard
10065
10066Once the local build for "neard" completes, start a ``devshell`` build:
10067::
10068
10069 $ bitbake neard -c devshell
10070
10071For information on how to use a
10072``devshell``, see the "`Using a Development
10073Shell <#platdev-appdev-devshell>`__" section.
10074
10075In the ``devshell``, do the following:
10076::
10077
10078 $ make clean
10079 $ make tools/snep-send.o
10080
10081The ``devshell`` commands cause the failure to clearly
10082be visible. In this case, a missing dependency exists for the "neard"
10083Makefile target. Here is some abbreviated, sample output with the
10084missing dependency clearly visible at the end:
10085::
10086
10087 i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/scott-lenovo/......
10088 .
10089 .
10090 .
10091 tools/snep-send.c
10092 In file included from tools/snep-send.c:16:0:
10093 tools/../src/near.h:41:23: fatal error: near/dbus.h: No such file or directory
10094 #include <near/dbus.h>
10095 ^
10096 compilation terminated.
10097 make: *** [tools/snep-send.o] Error 1
10098 $
10099
10100
10101Creating a Patch for the Fix
10102~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10103
10104Because there is a missing dependency for the Makefile target, you need
10105to patch the ``Makefile.am`` file, which is generated from
10106``Makefile.in``. You can use Quilt to create the patch:
10107::
10108
10109 $ quilt new parallelmake.patch
10110 Patch patches/parallelmake.patch is now on top
10111 $ quilt add Makefile.am
10112 File Makefile.am added to patch patches/parallelmake.patch
10113
10114For more information on using Quilt, see the
10115"`Using Quilt in Your Workflow <#using-a-quilt-workflow>`__" section.
10116
10117At this point you need to make the edits to ``Makefile.am`` to add the
10118missing dependency. For our example, you have to add the following line
10119to the file:
10120::
10121
10122 tools/snep-send.$(OBJEXT): include/near/dbus.h
10123
10124Once you have edited the file, use the ``refresh`` command to create the
10125patch:
10126::
10127
10128 $ quilt refresh
10129 Refreshed patch patches/parallelmake.patch
10130
10131Once
10132the patch file exists, you need to add it back to the originating recipe
10133folder. Here is an example assuming a top-level
10134:term:`Source Directory` named ``poky``:
10135::
10136
10137 $ cp patches/parallelmake.patch poky/meta/recipes-connectivity/neard/neard
10138
10139The final thing you need to do to implement the fix in the build is to
10140update the "neard" recipe (i.e. ``neard-0.14.bb``) so that the
10141:term:`SRC_URI` statement includes
10142the patch file. The recipe file is in the folder above the patch. Here
10143is what the edited ``SRC_URI`` statement would look like:
10144::
10145
10146 SRC_URI = "${KERNELORG_MIRROR}/linux/network/nfc/${BPN}-${PV}.tar.xz \
10147 file://neard.in \
10148 file://neard.service.in \
10149 file://parallelmake.patch \
10150 "
10151
10152With the patch complete and moved to the correct folder and the
10153``SRC_URI`` statement updated, you can exit the ``devshell``:
10154::
10155
10156 $ exit
10157
10158Testing the Build
10159~~~~~~~~~~~~~~~~~
10160
10161With everything in place, you can get back to trying the build again
10162locally:
10163::
10164
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010165 $ bitbake neard
10166
10167This build should succeed.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010168
10169Now you can open up a ``devshell`` again and repeat the clean and make
10170operations as follows:
10171::
10172
10173 $ bitbake neard -c devshell
10174 $ make clean
10175 $ make tools/snep-send.o
10176
10177The build should work without issue.
10178
10179As with all solved problems, if they originated upstream, you need to
10180submit the fix for the recipe in OE-Core and upstream so that the
10181problem is taken care of at its source. See the "`Submitting a Change to
10182the Yocto Project <#how-to-submit-a-change>`__" section for more
10183information.
10184
10185.. _platdev-gdb-remotedebug:
10186
10187Debugging With the GNU Project Debugger (GDB) Remotely
10188------------------------------------------------------
10189
10190GDB allows you to examine running programs, which in turn helps you to
10191understand and fix problems. It also allows you to perform post-mortem
10192style analysis of program crashes. GDB is available as a package within
10193the Yocto Project and is installed in SDK images by default. See the
10194":ref:`ref-manual/ref-images:Images`" chapter in the Yocto
10195Project Reference Manual for a description of these images. You can find
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010196information on GDB at https://sourceware.org/gdb/.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010197
10198.. note::
10199
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010200 For best results, install debug (``-dbg``) packages for the applications you
10201 are going to debug. Doing so makes extra debug symbols available that give
10202 you more meaningful output.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010203
10204Sometimes, due to memory or disk space constraints, it is not possible
10205to use GDB directly on the remote target to debug applications. These
10206constraints arise because GDB needs to load the debugging information
10207and the binaries of the process being debugged. Additionally, GDB needs
10208to perform many computations to locate information such as function
10209names, variable names and values, stack traces and so forth - even
10210before starting the debugging process. These extra computations place
10211more load on the target system and can alter the characteristics of the
10212program being debugged.
10213
10214To help get past the previously mentioned constraints, you can use
10215gdbserver, which runs on the remote target and does not load any
10216debugging information from the debugged process. Instead, a GDB instance
10217processes the debugging information that is run on a remote computer -
10218the host GDB. The host GDB then sends control commands to gdbserver to
10219make it stop or start the debugged program, as well as read or write
10220memory regions of that debugged program. All the debugging information
10221loaded and processed as well as all the heavy debugging is done by the
10222host GDB. Offloading these processes gives the gdbserver running on the
10223target a chance to remain small and fast.
10224
10225Because the host GDB is responsible for loading the debugging
10226information and for doing the necessary processing to make actual
10227debugging happen, you have to make sure the host can access the
10228unstripped binaries complete with their debugging information and also
10229be sure the target is compiled with no optimizations. The host GDB must
10230also have local access to all the libraries used by the debugged
10231program. Because gdbserver does not need any local debugging
10232information, the binaries on the remote target can remain stripped.
10233However, the binaries must also be compiled without optimization so they
10234match the host's binaries.
10235
10236To remain consistent with GDB documentation and terminology, the binary
10237being debugged on the remote target machine is referred to as the
10238"inferior" binary. For documentation on GDB see the `GDB
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010239site <https://sourceware.org/gdb/documentation/>`__.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010240
10241The following steps show you how to debug using the GNU project
10242debugger.
10243
102441. *Configure your build system to construct the companion debug
10245 filesystem:*
10246
10247 In your ``local.conf`` file, set the following:
10248 ::
10249
10250 IMAGE_GEN_DEBUGFS = "1"
10251 IMAGE_FSTYPES_DEBUGFS = "tar.bz2"
10252
10253 These options cause the
10254 OpenEmbedded build system to generate a special companion filesystem
10255 fragment, which contains the matching source and debug symbols to
10256 your deployable filesystem. The build system does this by looking at
10257 what is in the deployed filesystem, and pulling the corresponding
10258 ``-dbg`` packages.
10259
10260 The companion debug filesystem is not a complete filesystem, but only
10261 contains the debug fragments. This filesystem must be combined with
10262 the full filesystem for debugging. Subsequent steps in this procedure
10263 show how to combine the partial filesystem with the full filesystem.
10264
102652. *Configure the system to include gdbserver in the target filesystem:*
10266
10267 Make the following addition in either your ``local.conf`` file or in
10268 an image recipe:
10269 ::
10270
10271 IMAGE_INSTALL_append = " gdbserver"
10272
10273 The change makes
10274 sure the ``gdbserver`` package is included.
10275
102763. *Build the environment:*
10277
10278 Use the following command to construct the image and the companion
10279 Debug Filesystem:
10280 ::
10281
10282 $ bitbake image
10283
10284 Build the cross GDB component and
10285 make it available for debugging. Build the SDK that matches the
10286 image. Building the SDK is best for a production build that can be
10287 used later for debugging, especially during long term maintenance:
10288 ::
10289
10290 $ bitbake -c populate_sdk image
10291
10292 Alternatively, you can build the minimal toolchain components that
10293 match the target. Doing so creates a smaller than typical SDK and
10294 only contains a minimal set of components with which to build simple
10295 test applications, as well as run the debugger:
10296 ::
10297
10298 $ bitbake meta-toolchain
10299
10300 A final method is to build Gdb itself within the build system:
10301 ::
10302
10303 $ bitbake gdb-cross-<architecture>
10304
10305 Doing so produces a temporary copy of
10306 ``cross-gdb`` you can use for debugging during development. While
10307 this is the quickest approach, the two previous methods in this step
10308 are better when considering long-term maintenance strategies.
10309
10310 .. note::
10311
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010312 If you run ``bitbake gdb-cross``, the OpenEmbedded build system suggests
10313 the actual image (e.g. ``gdb-cross-i586``). The suggestion is usually the
10314 actual name you want to use.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010315
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500103164. *Set up the* ``debugfs``\ *:*
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010317
10318 Run the following commands to set up the ``debugfs``:
10319 ::
10320
10321 $ mkdir debugfs
10322 $ cd debugfs
10323 $ tar xvfj build-dir/tmp-glibc/deploy/images/machine/image.rootfs.tar.bz2
10324 $ tar xvfj build-dir/tmp-glibc/deploy/images/machine/image-dbg.rootfs.tar.bz2
10325
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500103265. *Set up GDB:*
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010327
10328 Install the SDK (if you built one) and then source the correct
10329 environment file. Sourcing the environment file puts the SDK in your
10330 ``PATH`` environment variable.
10331
10332 If you are using the build system, Gdb is located in
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010333 `build-dir`\ ``/tmp/sysroots/``\ `host`\ ``/usr/bin/``\ `architecture`\ ``/``\ `architecture`\ ``-gdb``
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010334
103356. *Boot the target:*
10336
10337 For information on how to run QEMU, see the `QEMU
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010338 Documentation <https://wiki.qemu.org/Documentation/GettingStartedDevelopers>`__.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010339
10340 .. note::
10341
10342 Be sure to verify that your host can access the target via TCP.
10343
103447. *Debug a program:*
10345
10346 Debugging a program involves running gdbserver on the target and then
10347 running Gdb on the host. The example in this step debugs ``gzip``:
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010348
10349 .. code-block:: shell
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010350
10351 root@qemux86:~# gdbserver localhost:1234 /bin/gzip —help
10352
10353 For
10354 additional gdbserver options, see the `GDB Server
10355 Documentation <https://www.gnu.org/software/gdb/documentation/>`__.
10356
10357 After running gdbserver on the target, you need to run Gdb on the
10358 host and configure it and connect to the target. Use these commands:
10359 ::
10360
10361 $ cd directory-holding-the-debugfs-directory
10362 $ arch-gdb
10363 (gdb) set sysroot debugfs
10364 (gdb) set substitute-path /usr/src/debug debugfs/usr/src/debug
10365 (gdb) target remote IP-of-target:1234
10366
10367 At this
10368 point, everything should automatically load (i.e. matching binaries,
10369 symbols and headers).
10370
10371 .. note::
10372
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010373 The Gdb ``set`` commands in the previous example can be placed into the
10374 users ``~/.gdbinit`` file. Upon starting, Gdb automatically runs whatever
10375 commands are in that file.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010376
103778. *Deploying without a full image rebuild:*
10378
10379 In many cases, during development you want a quick method to deploy a
10380 new binary to the target and debug it, without waiting for a full
10381 image build.
10382
10383 One approach to solving this situation is to just build the component
10384 you want to debug. Once you have built the component, copy the
10385 executable directly to both the target and the host ``debugfs``.
10386
10387 If the binary is processed through the debug splitting in
10388 OpenEmbedded, you should also copy the debug items (i.e. ``.debug``
10389 contents and corresponding ``/usr/src/debug`` files) from the work
10390 directory. Here is an example:
10391 ::
10392
10393 $ bitbake bash
10394 $ bitbake -c devshell bash
10395 $ cd ..
10396 $ scp packages-split/bash/bin/bash target:/bin/bash
10397 $ cp -a packages-split/bash-dbg/\* path/debugfs
10398
10399Debugging with the GNU Project Debugger (GDB) on the Target
10400-----------------------------------------------------------
10401
10402The previous section addressed using GDB remotely for debugging
10403purposes, which is the most usual case due to the inherent hardware
10404limitations on many embedded devices. However, debugging in the target
10405hardware itself is also possible with more powerful devices. This
10406section describes what you need to do in order to support using GDB to
10407debug on the target hardware.
10408
10409To support this kind of debugging, you need do the following:
10410
10411- Ensure that GDB is on the target. You can do this by adding "gdb" to
10412 :term:`IMAGE_INSTALL`:
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010413 ::
10414
10415 IMAGE_INSTALL_append = " gdb"
10416
10417 Alternatively, you can add "tools-debug" to :term:`IMAGE_FEATURES`:
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010418 ::
10419
10420 IMAGE_FEATURES_append = " tools-debug"
10421
10422- Ensure that debug symbols are present. You can make sure these
10423 symbols are present by installing ``-dbg``:
10424 ::
10425
10426 IMAGE_INSTALL_append = "packagename-dbg"
10427
10428 Alternatively, you can do the following to include
10429 all the debug symbols:
10430 ::
10431
10432 IMAGE_FEATURES_append = " dbg-pkgs"
10433
10434.. note::
10435
10436 To improve the debug information accuracy, you can reduce the level
10437 of optimization used by the compiler. For example, when adding the
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010438 following line to your ``local.conf`` file, you will reduce optimization
10439 from :term:`FULL_OPTIMIZATION` of "-O2" to :term:`DEBUG_OPTIMIZATION`
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010440 of "-O -fno-omit-frame-pointer":
10441 ::
10442
10443 DEBUG_BUILD = "1"
10444
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010445 Consider that this will reduce the application's performance and is
10446 recommended only for debugging purposes.
10447
10448.. _dev-other-debugging-others:
10449
10450Other Debugging Tips
10451--------------------
10452
10453Here are some other tips that you might find useful:
10454
10455- When adding new packages, it is worth watching for undesirable items
10456 making their way into compiler command lines. For example, you do not
10457 want references to local system files like ``/usr/lib/`` or
10458 ``/usr/include/``.
10459
10460- If you want to remove the ``psplash`` boot splashscreen, add
10461 ``psplash=false`` to the kernel command line. Doing so prevents
10462 ``psplash`` from loading and thus allows you to see the console. It
10463 is also possible to switch out of the splashscreen by switching the
10464 virtual console (e.g. Fn+Left or Fn+Right on a Zaurus).
10465
10466- Removing :term:`TMPDIR` (usually
10467 ``tmp/``, within the
10468 :term:`Build Directory`) can often fix
10469 temporary build issues. Removing ``TMPDIR`` is usually a relatively
10470 cheap operation, because task output will be cached in
10471 :term:`SSTATE_DIR` (usually
10472 ``sstate-cache/``, which is also in the Build Directory).
10473
10474 .. note::
10475
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010476 Removing ``TMPDIR`` might be a workaround rather than a fix.
10477 Consequently, trying to determine the underlying cause of an issue before
10478 removing the directory is a good idea.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010479
10480- Understanding how a feature is used in practice within existing
10481 recipes can be very helpful. It is recommended that you configure
10482 some method that allows you to quickly search through files.
10483
10484 Using GNU Grep, you can use the following shell function to
10485 recursively search through common recipe-related files, skipping
10486 binary files, ``.git`` directories, and the Build Directory (assuming
10487 its name starts with "build"):
10488 ::
10489
10490 g() {
10491 grep -Ir \
10492 --exclude-dir=.git \
10493 --exclude-dir='build*' \
10494 --include='*.bb*' \
10495 --include='*.inc*' \
10496 --include='*.conf*' \
10497 --include='*.py*' \
10498 "$@"
10499 }
10500
10501 Following are some usage examples:
10502 ::
10503
10504 $ g FOO # Search recursively for "FOO"
10505 $ g -i foo # Search recursively for "foo", ignoring case
10506 $ g -w FOO # Search recursively for "FOO" as a word, ignoring e.g. "FOOBAR"
10507
10508 If figuring
10509 out how some feature works requires a lot of searching, it might
10510 indicate that the documentation should be extended or improved. In
10511 such cases, consider filing a documentation bug using the Yocto
10512 Project implementation of
10513 :yocto_bugs:`Bugzilla <>`. For information on
10514 how to submit a bug against the Yocto Project, see the Yocto Project
10515 Bugzilla :yocto_wiki:`wiki page </wiki/Bugzilla_Configuration_and_Bug_Tracking>`
10516 and the "`Submitting a Defect Against the Yocto
10517 Project <#submitting-a-defect-against-the-yocto-project>`__" section.
10518
10519 .. note::
10520
10521 The manuals might not be the right place to document variables
10522 that are purely internal and have a limited scope (e.g. internal
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010523 variables used to implement a single ``.bbclass`` file).
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010524
10525Making Changes to the Yocto Project
10526===================================
10527
10528Because the Yocto Project is an open-source, community-based project,
10529you can effect changes to the project. This section presents procedures
10530that show you how to submit a defect against the project and how to
10531submit a change.
10532
10533Submitting a Defect Against the Yocto Project
10534---------------------------------------------
10535
10536Use the Yocto Project implementation of
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010537`Bugzilla <https://www.bugzilla.org/about/>`__ to submit a defect (bug)
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010538against the Yocto Project. For additional information on this
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010539implementation of Bugzilla see the ":ref:`Yocto Project
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010540Bugzilla <resources-bugtracker>`" section in the
10541Yocto Project Reference Manual. For more detail on any of the following
10542steps, see the Yocto Project
10543:yocto_wiki:`Bugzilla wiki page </wiki/Bugzilla_Configuration_and_Bug_Tracking>`.
10544
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010545Use the following general steps to submit a bug:
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010546
105471. Open the Yocto Project implementation of :yocto_bugs:`Bugzilla <>`.
10548
105492. Click "File a Bug" to enter a new bug.
10550
105513. Choose the appropriate "Classification", "Product", and "Component"
10552 for which the bug was found. Bugs for the Yocto Project fall into
10553 one of several classifications, which in turn break down into
10554 several products and components. For example, for a bug against the
10555 ``meta-intel`` layer, you would choose "Build System, Metadata &
10556 Runtime", "BSPs", and "bsps-meta-intel", respectively.
10557
105584. Choose the "Version" of the Yocto Project for which you found the
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010559 bug (e.g. &DISTRO;).
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010560
105615. Determine and select the "Severity" of the bug. The severity
10562 indicates how the bug impacted your work.
10563
105646. Choose the "Hardware" that the bug impacts.
10565
105667. Choose the "Architecture" that the bug impacts.
10567
105688. Choose a "Documentation change" item for the bug. Fixing a bug might
10569 or might not affect the Yocto Project documentation. If you are
10570 unsure of the impact to the documentation, select "Don't Know".
10571
105729. Provide a brief "Summary" of the bug. Try to limit your summary to
10573 just a line or two and be sure to capture the essence of the bug.
10574
1057510. Provide a detailed "Description" of the bug. You should provide as
10576 much detail as you can about the context, behavior, output, and so
10577 forth that surrounds the bug. You can even attach supporting files
10578 for output from logs by using the "Add an attachment" button.
10579
1058011. Click the "Submit Bug" button submit the bug. A new Bugzilla number
10581 is assigned to the bug and the defect is logged in the bug tracking
10582 system.
10583
10584Once you file a bug, the bug is processed by the Yocto Project Bug
10585Triage Team and further details concerning the bug are assigned (e.g.
10586priority and owner). You are the "Submitter" of the bug and any further
10587categorization, progress, or comments on the bug result in Bugzilla
10588sending you an automated email concerning the particular change or
10589progress to the bug.
10590
10591.. _how-to-submit-a-change:
10592
10593Submitting a Change to the Yocto Project
10594----------------------------------------
10595
10596Contributions to the Yocto Project and OpenEmbedded are very welcome.
10597Because the system is extremely configurable and flexible, we recognize
10598that developers will want to extend, configure or optimize it for their
10599specific uses.
10600
10601The Yocto Project uses a mailing list and a patch-based workflow that is
10602similar to the Linux kernel but contains important differences. In
10603general, a mailing list exists through which you can submit patches. You
10604should send patches to the appropriate mailing list so that they can be
10605reviewed and merged by the appropriate maintainer. The specific mailing
10606list you need to use depends on the location of the code you are
10607changing. Each component (e.g. layer) should have a ``README`` file that
10608indicates where to send the changes and which process to follow.
10609
10610You can send the patch to the mailing list using whichever approach you
10611feel comfortable with to generate the patch. Once sent, the patch is
10612usually reviewed by the community at large. If somebody has concerns
10613with the patch, they will usually voice their concern over the mailing
10614list. If a patch does not receive any negative reviews, the maintainer
10615of the affected layer typically takes the patch, tests it, and then
10616based on successful testing, merges the patch.
10617
10618The "poky" repository, which is the Yocto Project's reference build
10619environment, is a hybrid repository that contains several individual
10620pieces (e.g. BitBake, Metadata, documentation, and so forth) built using
10621the combo-layer tool. The upstream location used for submitting changes
10622varies by component:
10623
10624- *Core Metadata:* Send your patch to the
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010625 :oe_lists:`openembedded-core </g/openembedded-core>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010626 mailing list. For example, a change to anything under the ``meta`` or
10627 ``scripts`` directories should be sent to this mailing list.
10628
10629- *BitBake:* For changes to BitBake (i.e. anything under the
10630 ``bitbake`` directory), send your patch to the
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010631 :oe_lists:`bitbake-devel </g/bitbake-devel>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010632 mailing list.
10633
Andrew Geisslerc3d88e42020-10-02 09:45:00 -050010634- *"meta-\*" trees:* These trees contain Metadata. Use the
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010635 :yocto_lists:`poky </g/poky>` mailing list.
Andrew Geisslerc3d88e42020-10-02 09:45:00 -050010636
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010637- *Documentation*: For changes to the Yocto Project documentation, use the
10638 :yocto_lists:`docs </g/docs>` mailing list.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010639
10640For changes to other layers hosted in the Yocto Project source
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010641repositories (i.e. ``yoctoproject.org``) and tools use the
10642:yocto_lists:`Yocto Project </g/yocto/>` general mailing list.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010643
10644.. note::
10645
10646 Sometimes a layer's documentation specifies to use a particular
10647 mailing list. If so, use that list.
10648
10649For additional recipes that do not fit into the core Metadata, you
10650should determine which layer the recipe should go into and submit the
10651change in the manner recommended by the documentation (e.g. the
10652``README`` file) supplied with the layer. If in doubt, please ask on the
10653Yocto general mailing list or on the openembedded-devel mailing list.
10654
10655You can also push a change upstream and request a maintainer to pull the
10656change into the component's upstream repository. You do this by pushing
10657to a contribution repository that is upstream. See the ":ref:`gs-git-workflows-and-the-yocto-project`"
10658section in the Yocto Project Overview and Concepts Manual for additional
10659concepts on working in the Yocto Project development environment.
10660
10661Two commonly used testing repositories exist for OpenEmbedded-Core:
10662
10663- *"ross/mut" branch:* The "mut" (master-under-test) tree exists in the
10664 ``poky-contrib`` repository in the
10665 :yocto_git:`Yocto Project source repositories <>`.
10666
10667- *"master-next" branch:* This branch is part of the main "poky"
10668 repository in the Yocto Project source repositories.
10669
10670Maintainers use these branches to test submissions prior to merging
10671patches. Thus, you can get an idea of the status of a patch based on
10672whether the patch has been merged into one of these branches.
10673
10674.. note::
10675
10676 This system is imperfect and changes can sometimes get lost in the
10677 flow. Asking about the status of a patch or change is reasonable if
10678 the change has been idle for a while with no feedback. The Yocto
10679 Project does have plans to use
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010680 `Patchwork <https://en.wikipedia.org/wiki/Patchwork_(software)>`__
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010681 to track the status of patches and also to automatically preview
10682 patches.
10683
10684The following sections provide procedures for submitting a change.
10685
10686.. _pushing-a-change-upstream:
10687
10688Using Scripts to Push a Change Upstream and Request a Pull
10689~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10690
10691Follow this procedure to push a change to an upstream "contrib" Git
10692repository:
10693
10694.. note::
10695
10696 You can find general Git information on how to push a change upstream
10697 in the
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010698 `Git Community Book <https://git-scm.com/book/en/v2/Distributed-Git-Distributed-Workflows>`__.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010699
107001. *Make Your Changes Locally:* Make your changes in your local Git
10701 repository. You should make small, controlled, isolated changes.
10702 Keeping changes small and isolated aids review, makes
10703 merging/rebasing easier and keeps the change history clean should
10704 anyone need to refer to it in future.
10705
107062. *Stage Your Changes:* Stage your changes by using the ``git add``
10707 command on each file you changed.
10708
107093. *Commit Your Changes:* Commit the change by using the ``git commit``
10710 command. Make sure your commit information follows standards by
10711 following these accepted conventions:
10712
10713 - Be sure to include a "Signed-off-by:" line in the same style as
10714 required by the Linux kernel. Adding this line signifies that you,
10715 the submitter, have agreed to the Developer's Certificate of
10716 Origin 1.1 as follows:
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010717
10718 .. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010719
10720 Developer's Certificate of Origin 1.1
10721
10722 By making a contribution to this project, I certify that:
10723
10724 (a) The contribution was created in whole or in part by me and I
10725 have the right to submit it under the open source license
10726 indicated in the file; or
10727
10728 (b) The contribution is based upon previous work that, to the best
10729 of my knowledge, is covered under an appropriate open source
10730 license and I have the right under that license to submit that
10731 work with modifications, whether created in whole or in part
10732 by me, under the same open source license (unless I am
10733 permitted to submit under a different license), as indicated
10734 in the file; or
10735
10736 (c) The contribution was provided directly to me by some other
10737 person who certified (a), (b) or (c) and I have not modified
10738 it.
10739
10740 (d) I understand and agree that this project and the contribution
10741 are public and that a record of the contribution (including all
10742 personal information I submit with it, including my sign-off) is
10743 maintained indefinitely and may be redistributed consistent with
10744 this project or the open source license(s) involved.
10745
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010746 - Provide a single-line summary of the change and, if more
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010747 explanation is needed, provide more detail in the body of the
10748 commit. This summary is typically viewable in the "shortlist" of
10749 changes. Thus, providing something short and descriptive that
10750 gives the reader a summary of the change is useful when viewing a
10751 list of many commits. You should prefix this short description
10752 with the recipe name (if changing a recipe), or else with the
10753 short form path to the file being changed.
10754
10755 - For the body of the commit message, provide detailed information
10756 that describes what you changed, why you made the change, and the
10757 approach you used. It might also be helpful if you mention how you
10758 tested the change. Provide as much detail as you can in the body
10759 of the commit message.
10760
10761 .. note::
10762
10763 You do not need to provide a more detailed explanation of a
10764 change if the change is minor to the point of the single line
10765 summary providing all the information.
10766
10767 - If the change addresses a specific bug or issue that is associated
10768 with a bug-tracking ID, include a reference to that ID in your
10769 detailed description. For example, the Yocto Project uses a
10770 specific convention for bug references - any commit that addresses
10771 a specific bug should use the following form for the detailed
10772 description. Be sure to use the actual bug-tracking ID from
10773 Bugzilla for bug-id:
10774 ::
10775
10776 Fixes [YOCTO #bug-id]
10777
10778 detailed description of change
10779
107804. *Push Your Commits to a "Contrib" Upstream:* If you have arranged for
10781 permissions to push to an upstream contrib repository, push the
10782 change to that repository:
10783 ::
10784
10785 $ git push upstream_remote_repo local_branch_name
10786
10787 For example, suppose you have permissions to push
10788 into the upstream ``meta-intel-contrib`` repository and you are
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010789 working in a local branch named `your_name`\ ``/README``. The following
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010790 command pushes your local commits to the ``meta-intel-contrib``
10791 upstream repository and puts the commit in a branch named
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010792 `your_name`\ ``/README``:
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010793 ::
10794
10795 $ git push meta-intel-contrib your_name/README
10796
107975. *Determine Who to Notify:* Determine the maintainer or the mailing
10798 list that you need to notify for the change.
10799
10800 Before submitting any change, you need to be sure who the maintainer
10801 is or what mailing list that you need to notify. Use either these
10802 methods to find out:
10803
10804 - *Maintenance File:* Examine the ``maintainers.inc`` file, which is
10805 located in the :term:`Source Directory` at
10806 ``meta/conf/distro/include``, to see who is responsible for code.
10807
10808 - *Search by File:* Using :ref:`overview-manual/overview-manual-development-environment:git`, you can
10809 enter the following command to bring up a short list of all
10810 commits against a specific file:
10811 ::
10812
10813 git shortlog -- filename
10814
10815 Just provide the name of the file for which you are interested. The
10816 information returned is not ordered by history but does include a
10817 list of everyone who has committed grouped by name. From the list,
10818 you can see who is responsible for the bulk of the changes against
10819 the file.
10820
10821 - *Examine the List of Mailing Lists:* For a list of the Yocto
10822 Project and related mailing lists, see the ":ref:`Mailing
10823 lists <resources-mailinglist>`" section in
10824 the Yocto Project Reference Manual.
10825
108266. *Make a Pull Request:* Notify the maintainer or the mailing list that
10827 you have pushed a change by making a pull request.
10828
10829 The Yocto Project provides two scripts that conveniently let you
10830 generate and send pull requests to the Yocto Project. These scripts
10831 are ``create-pull-request`` and ``send-pull-request``. You can find
10832 these scripts in the ``scripts`` directory within the
10833 :term:`Source Directory` (e.g.
10834 ``~/poky/scripts``).
10835
10836 Using these scripts correctly formats the requests without
10837 introducing any whitespace or HTML formatting. The maintainer that
10838 receives your patches either directly or through the mailing list
10839 needs to be able to save and apply them directly from your emails.
10840 Using these scripts is the preferred method for sending patches.
10841
10842 First, create the pull request. For example, the following command
10843 runs the script, specifies the upstream repository in the contrib
10844 directory into which you pushed the change, and provides a subject
10845 line in the created patch files:
10846 ::
10847
10848 $ ~/poky/scripts/create-pull-request -u meta-intel-contrib -s "Updated Manual Section Reference in README"
10849
10850 Running this script forms ``*.patch`` files in a folder named
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010851 ``pull-``\ `PID` in the current directory. One of the patch files is a
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010852 cover letter.
10853
10854 Before running the ``send-pull-request`` script, you must edit the
10855 cover letter patch to insert information about your change. After
10856 editing the cover letter, send the pull request. For example, the
10857 following command runs the script and specifies the patch directory
10858 and email address. In this example, the email address is a mailing
10859 list:
10860 ::
10861
10862 $ ~/poky/scripts/send-pull-request -p ~/meta-intel/pull-10565 -t meta-intel@yoctoproject.org
10863
10864 You need to follow the prompts as the script is interactive.
10865
10866 .. note::
10867
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010868 For help on using these scripts, simply provide the ``-h``
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010869 argument as follows:
10870 ::
10871
10872 $ poky/scripts/create-pull-request -h
10873 $ poky/scripts/send-pull-request -h
10874
10875
10876.. _submitting-a-patch:
10877
10878Using Email to Submit a Patch
10879~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10880
10881You can submit patches without using the ``create-pull-request`` and
10882``send-pull-request`` scripts described in the previous section.
10883However, keep in mind, the preferred method is to use the scripts.
10884
10885Depending on the components changed, you need to submit the email to a
10886specific mailing list. For some guidance on which mailing list to use,
10887see the `list <#figuring-out-the-mailing-list-to-use>`__ at the
10888beginning of this section. For a description of all the available
10889mailing lists, see the ":ref:`Mailing Lists <resources-mailinglist>`" section in the
10890Yocto Project Reference Manual.
10891
10892Here is the general procedure on how to submit a patch through email
10893without using the scripts:
10894
108951. *Make Your Changes Locally:* Make your changes in your local Git
10896 repository. You should make small, controlled, isolated changes.
10897 Keeping changes small and isolated aids review, makes
10898 merging/rebasing easier and keeps the change history clean should
10899 anyone need to refer to it in future.
10900
109012. *Stage Your Changes:* Stage your changes by using the ``git add``
10902 command on each file you changed.
10903
109043. *Commit Your Changes:* Commit the change by using the
10905 ``git commit --signoff`` command. Using the ``--signoff`` option
10906 identifies you as the person making the change and also satisfies the
10907 Developer's Certificate of Origin (DCO) shown earlier.
10908
10909 When you form a commit, you must follow certain standards established
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010910 by the Yocto Project development team. See :ref:`Step 3
10911 <dev-manual/dev-manual-common-tasks:using scripts to push a change upstream and request a pull>`
10912 in the previous section for information on how to provide commit information
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010913 that meets Yocto Project commit message standards.
10914
109154. *Format the Commit:* Format the commit into an email message. To
10916 format commits, use the ``git format-patch`` command. When you
10917 provide the command, you must include a revision list or a number of
10918 patches as part of the command. For example, either of these two
10919 commands takes your most recent single commit and formats it as an
10920 email message in the current directory:
10921 ::
10922
10923 $ git format-patch -1
10924
10925 or ::
10926
10927 $ git format-patch HEAD~
10928
10929 After the command is run, the current directory contains a numbered
10930 ``.patch`` file for the commit.
10931
10932 If you provide several commits as part of the command, the
10933 ``git format-patch`` command produces a series of numbered files in
10934 the current directory – one for each commit. If you have more than
10935 one patch, you should also use the ``--cover`` option with the
10936 command, which generates a cover letter as the first "patch" in the
10937 series. You can then edit the cover letter to provide a description
10938 for the series of patches. For information on the
10939 ``git format-patch`` command, see ``GIT_FORMAT_PATCH(1)`` displayed
10940 using the ``man git-format-patch`` command.
10941
10942 .. note::
10943
10944 If you are or will be a frequent contributor to the Yocto Project
10945 or to OpenEmbedded, you might consider requesting a contrib area
10946 and the necessary associated rights.
10947
109485. *Import the Files Into Your Mail Client:* Import the files into your
10949 mail client by using the ``git send-email`` command.
10950
10951 .. note::
10952
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010953 In order to use ``git send-email``, you must have the proper Git packages
10954 installed on your host.
10955 For Ubuntu, Debian, and Fedora the package is ``git-email``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010956
10957 The ``git send-email`` command sends email by using a local or remote
10958 Mail Transport Agent (MTA) such as ``msmtp``, ``sendmail``, or
10959 through a direct ``smtp`` configuration in your Git ``~/.gitconfig``
10960 file. If you are submitting patches through email only, it is very
10961 important that you submit them without any whitespace or HTML
10962 formatting that either you or your mailer introduces. The maintainer
10963 that receives your patches needs to be able to save and apply them
10964 directly from your emails. A good way to verify that what you are
10965 sending will be applicable by the maintainer is to do a dry run and
10966 send them to yourself and then save and apply them as the maintainer
10967 would.
10968
10969 The ``git send-email`` command is the preferred method for sending
10970 your patches using email since there is no risk of compromising
10971 whitespace in the body of the message, which can occur when you use
10972 your own mail client. The command also has several options that let
10973 you specify recipients and perform further editing of the email
10974 message. For information on how to use the ``git send-email``
10975 command, see ``GIT-SEND-EMAIL(1)`` displayed using the
10976 ``man git-send-email`` command.
10977
10978Working With Licenses
10979=====================
10980
10981As mentioned in the ":ref:`overview-manual/overview-manual-development-environment:licensing`"
10982section in the Yocto Project Overview and Concepts Manual, open source
10983projects are open to the public and they consequently have different
10984licensing structures in place. This section describes the mechanism by
10985which the :term:`OpenEmbedded Build System`
10986tracks changes to
10987licensing text and covers how to maintain open source license compliance
10988during your project's lifecycle. The section also describes how to
10989enable commercially licensed recipes, which by default are disabled.
10990
10991.. _usingpoky-configuring-LIC_FILES_CHKSUM:
10992
10993Tracking License Changes
10994------------------------
10995
10996The license of an upstream project might change in the future. In order
10997to prevent these changes going unnoticed, the
10998:term:`LIC_FILES_CHKSUM`
10999variable tracks changes to the license text. The checksums are validated
11000at the end of the configure step, and if the checksums do not match, the
11001build will fail.
11002
11003.. _usingpoky-specifying-LIC_FILES_CHKSUM:
11004
11005Specifying the ``LIC_FILES_CHKSUM`` Variable
11006~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
11007
11008The ``LIC_FILES_CHKSUM`` variable contains checksums of the license text
11009in the source code for the recipe. Following is an example of how to
11010specify ``LIC_FILES_CHKSUM``:
11011::
11012
11013 LIC_FILES_CHKSUM = "file://COPYING;md5=xxxx \
11014 file://licfile1.txt;beginline=5;endline=29;md5=yyyy \
11015 file://licfile2.txt;endline=50;md5=zzzz \
11016 ..."
11017
11018.. note::
11019
11020 - When using "beginline" and "endline", realize that line numbering
11021 begins with one and not zero. Also, the included lines are
11022 inclusive (i.e. lines five through and including 29 in the
11023 previous example for ``licfile1.txt``).
11024
11025 - When a license check fails, the selected license text is included
11026 as part of the QA message. Using this output, you can determine
11027 the exact start and finish for the needed license text.
11028
11029The build system uses the :term:`S`
11030variable as the default directory when searching files listed in
11031``LIC_FILES_CHKSUM``. The previous example employs the default
11032directory.
11033
11034Consider this next example:
11035::
11036
11037 LIC_FILES_CHKSUM = "file://src/ls.c;beginline=5;endline=16;\
11038 md5=bb14ed3c4cda583abc85401304b5cd4e"
11039 LIC_FILES_CHKSUM = "file://${WORKDIR}/license.html;md5=5c94767cedb5d6987c902ac850ded2c6"
11040
11041The first line locates a file in ``${S}/src/ls.c`` and isolates lines
11042five through 16 as license text. The second line refers to a file in
11043:term:`WORKDIR`.
11044
11045Note that ``LIC_FILES_CHKSUM`` variable is mandatory for all recipes,
11046unless the ``LICENSE`` variable is set to "CLOSED".
11047
11048.. _usingpoky-LIC_FILES_CHKSUM-explanation-of-syntax:
11049
11050Explanation of Syntax
11051~~~~~~~~~~~~~~~~~~~~~
11052
11053As mentioned in the previous section, the ``LIC_FILES_CHKSUM`` variable
11054lists all the important files that contain the license text for the
11055source code. It is possible to specify a checksum for an entire file, or
11056a specific section of a file (specified by beginning and ending line
11057numbers with the "beginline" and "endline" parameters, respectively).
11058The latter is useful for source files with a license notice header,
11059README documents, and so forth. If you do not use the "beginline"
11060parameter, then it is assumed that the text begins on the first line of
11061the file. Similarly, if you do not use the "endline" parameter, it is
11062assumed that the license text ends with the last line of the file.
11063
11064The "md5" parameter stores the md5 checksum of the license text. If the
11065license text changes in any way as compared to this parameter then a
11066mismatch occurs. This mismatch triggers a build failure and notifies the
11067developer. Notification allows the developer to review and address the
11068license text changes. Also note that if a mismatch occurs during the
11069build, the correct md5 checksum is placed in the build log and can be
11070easily copied to the recipe.
11071
11072There is no limit to how many files you can specify using the
11073``LIC_FILES_CHKSUM`` variable. Generally, however, every project
11074requires a few specifications for license tracking. Many projects have a
11075"COPYING" file that stores the license information for all the source
11076code files. This practice allows you to just track the "COPYING" file as
11077long as it is kept up to date.
11078
11079.. note::
11080
11081 - If you specify an empty or invalid "md5" parameter,
11082 :term:`BitBake` returns an md5
11083 mis-match error and displays the correct "md5" parameter value
11084 during the build. The correct parameter is also captured in the
11085 build log.
11086
11087 - If the whole file contains only license text, you do not need to
11088 use the "beginline" and "endline" parameters.
11089
11090Enabling Commercially Licensed Recipes
11091--------------------------------------
11092
11093By default, the OpenEmbedded build system disables components that have
11094commercial or other special licensing requirements. Such requirements
11095are defined on a recipe-by-recipe basis through the
11096:term:`LICENSE_FLAGS` variable
11097definition in the affected recipe. For instance, the
11098``poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly`` recipe
11099contains the following statement:
11100::
11101
11102 LICENSE_FLAGS = "commercial"
11103
11104Here is a
11105slightly more complicated example that contains both an explicit recipe
11106name and version (after variable expansion):
11107::
11108
11109 LICENSE_FLAGS = "license_${PN}_${PV}"
11110
11111In order for a component restricted by a
11112``LICENSE_FLAGS`` definition to be enabled and included in an image, it
11113needs to have a matching entry in the global
11114:term:`LICENSE_FLAGS_WHITELIST`
11115variable, which is a variable typically defined in your ``local.conf``
11116file. For example, to enable the
11117``poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly`` package, you
11118could add either the string "commercial_gst-plugins-ugly" or the more
11119general string "commercial" to ``LICENSE_FLAGS_WHITELIST``. See the
11120"`License Flag Matching <#license-flag-matching>`__" section for a full
11121explanation of how ``LICENSE_FLAGS`` matching works. Here is the
11122example:
11123::
11124
11125 LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly"
11126
11127Likewise, to additionally enable the package built from the recipe
11128containing ``LICENSE_FLAGS = "license_${PN}_${PV}"``, and assuming that
11129the actual recipe name was ``emgd_1.10.bb``, the following string would
11130enable that package as well as the original ``gst-plugins-ugly``
11131package:
11132::
11133
11134 LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly license_emgd_1.10"
11135
11136As a convenience, you do not need to specify the
11137complete license string in the whitelist for every package. You can use
11138an abbreviated form, which consists of just the first portion or
11139portions of the license string before the initial underscore character
11140or characters. A partial string will match any license that contains the
11141given string as the first portion of its license. For example, the
11142following whitelist string will also match both of the packages
11143previously mentioned as well as any other packages that have licenses
11144starting with "commercial" or "license".
11145::
11146
11147 LICENSE_FLAGS_WHITELIST = "commercial license"
11148
11149License Flag Matching
11150~~~~~~~~~~~~~~~~~~~~~
11151
11152License flag matching allows you to control what recipes the
11153OpenEmbedded build system includes in the build. Fundamentally, the
11154build system attempts to match ``LICENSE_FLAGS`` strings found in
11155recipes against ``LICENSE_FLAGS_WHITELIST`` strings found in the
11156whitelist. A match causes the build system to include a recipe in the
11157build, while failure to find a match causes the build system to exclude
11158a recipe.
11159
11160In general, license flag matching is simple. However, understanding some
11161concepts will help you correctly and effectively use matching.
11162
11163Before a flag defined by a particular recipe is tested against the
11164contents of the whitelist, the expanded string ``_${PN}`` is appended to
11165the flag. This expansion makes each ``LICENSE_FLAGS`` value
11166recipe-specific. After expansion, the string is then matched against the
11167whitelist. Thus, specifying ``LICENSE_FLAGS = "commercial"`` in recipe
11168"foo", for example, results in the string ``"commercial_foo"``. And, to
11169create a match, that string must appear in the whitelist.
11170
11171Judicious use of the ``LICENSE_FLAGS`` strings and the contents of the
11172``LICENSE_FLAGS_WHITELIST`` variable allows you a lot of flexibility for
11173including or excluding recipes based on licensing. For example, you can
11174broaden the matching capabilities by using license flags string subsets
11175in the whitelist.
11176
11177.. note::
11178
11179 When using a string subset, be sure to use the part of the expanded
11180 string that precedes the appended underscore character (e.g.
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011181 ``usethispart_1.3``, ``usethispart_1.4``, and so forth).
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011182
11183For example, simply specifying the string "commercial" in the whitelist
11184matches any expanded ``LICENSE_FLAGS`` definition that starts with the
11185string "commercial" such as "commercial_foo" and "commercial_bar", which
11186are the strings the build system automatically generates for
11187hypothetical recipes named "foo" and "bar" assuming those recipes simply
11188specify the following:
11189::
11190
11191 LICENSE_FLAGS = "commercial"
11192
11193Thus, you can choose
11194to exhaustively enumerate each license flag in the whitelist and allow
11195only specific recipes into the image, or you can use a string subset
11196that causes a broader range of matches to allow a range of recipes into
11197the image.
11198
11199This scheme works even if the ``LICENSE_FLAGS`` string already has
11200``_${PN}`` appended. For example, the build system turns the license
11201flag "commercial_1.2_foo" into "commercial_1.2_foo_foo" and would match
11202both the general "commercial" and the specific "commercial_1.2_foo"
11203strings found in the whitelist, as expected.
11204
11205Here are some other scenarios:
11206
11207- You can specify a versioned string in the recipe such as
11208 "commercial_foo_1.2" in a "foo" recipe. The build system expands this
11209 string to "commercial_foo_1.2_foo". Combine this license flag with a
11210 whitelist that has the string "commercial" and you match the flag
11211 along with any other flag that starts with the string "commercial".
11212
11213- Under the same circumstances, you can use "commercial_foo" in the
11214 whitelist and the build system not only matches "commercial_foo_1.2"
11215 but also matches any license flag with the string "commercial_foo",
11216 regardless of the version.
11217
11218- You can be very specific and use both the package and version parts
11219 in the whitelist (e.g. "commercial_foo_1.2") to specifically match a
11220 versioned recipe.
11221
11222Other Variables Related to Commercial Licenses
11223~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
11224
11225Other helpful variables related to commercial license handling exist and
11226are defined in the
11227``poky/meta/conf/distro/include/default-distrovars.inc`` file:
11228::
11229
11230 COMMERCIAL_AUDIO_PLUGINS ?= ""
11231 COMMERCIAL_VIDEO_PLUGINS ?= ""
11232
11233If you
11234want to enable these components, you can do so by making sure you have
11235statements similar to the following in your ``local.conf`` configuration
11236file:
11237::
11238
11239 COMMERCIAL_AUDIO_PLUGINS = "gst-plugins-ugly-mad \
11240 gst-plugins-ugly-mpegaudioparse"
11241 COMMERCIAL_VIDEO_PLUGINS = "gst-plugins-ugly-mpeg2dec \
11242 gst-plugins-ugly-mpegstream gst-plugins-bad-mpegvideoparse"
11243 LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly commercial_gst-plugins-bad commercial_qmmp"
11244
11245
11246Of course, you could also create a matching whitelist for those
11247components using the more general "commercial" in the whitelist, but
11248that would also enable all the other packages with ``LICENSE_FLAGS``
11249containing "commercial", which you may or may not want:
11250::
11251
11252 LICENSE_FLAGS_WHITELIST = "commercial"
11253
11254Specifying audio and video plugins as part of the
11255``COMMERCIAL_AUDIO_PLUGINS`` and ``COMMERCIAL_VIDEO_PLUGINS`` statements
11256(along with the enabling ``LICENSE_FLAGS_WHITELIST``) includes the
11257plugins or components into built images, thus adding support for media
11258formats or components.
11259
11260Maintaining Open Source License Compliance During Your Product's Lifecycle
11261--------------------------------------------------------------------------
11262
11263One of the concerns for a development organization using open source
11264software is how to maintain compliance with various open source
11265licensing during the lifecycle of the product. While this section does
11266not provide legal advice or comprehensively cover all scenarios, it does
11267present methods that you can use to assist you in meeting the compliance
11268requirements during a software release.
11269
11270With hundreds of different open source licenses that the Yocto Project
11271tracks, it is difficult to know the requirements of each and every
11272license. However, the requirements of the major FLOSS licenses can begin
11273to be covered by assuming that three main areas of concern exist:
11274
11275- Source code must be provided.
11276
11277- License text for the software must be provided.
11278
11279- Compilation scripts and modifications to the source code must be
11280 provided.
11281
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011282- spdx files can be provided.
11283
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011284There are other requirements beyond the scope of these three and the
11285methods described in this section (e.g. the mechanism through which
11286source code is distributed).
11287
11288As different organizations have different methods of complying with open
11289source licensing, this section is not meant to imply that there is only
11290one single way to meet your compliance obligations, but rather to
11291describe one method of achieving compliance. The remainder of this
11292section describes methods supported to meet the previously mentioned
11293three requirements. Once you take steps to meet these requirements, and
11294prior to releasing images, sources, and the build system, you should
11295audit all artifacts to ensure completeness.
11296
11297.. note::
11298
11299 The Yocto Project generates a license manifest during image creation
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011300 that is located in ``${DEPLOY_DIR}/licenses/``\ `image_name`\ ``-``\ `datestamp`
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011301 to assist with any audits.
11302
11303Providing the Source Code
11304~~~~~~~~~~~~~~~~~~~~~~~~~
11305
11306Compliance activities should begin before you generate the final image.
11307The first thing you should look at is the requirement that tops the list
11308for most compliance groups - providing the source. The Yocto Project has
11309a few ways of meeting this requirement.
11310
11311One of the easiest ways to meet this requirement is to provide the
11312entire :term:`DL_DIR` used by the
11313build. This method, however, has a few issues. The most obvious is the
11314size of the directory since it includes all sources used in the build
11315and not just the source used in the released image. It will include
11316toolchain source, and other artifacts, which you would not generally
11317release. However, the more serious issue for most companies is
11318accidental release of proprietary software. The Yocto Project provides
11319an :ref:`archiver <ref-classes-archiver>` class to
11320help avoid some of these concerns.
11321
11322Before you employ ``DL_DIR`` or the ``archiver`` class, you need to
11323decide how you choose to provide source. The source ``archiver`` class
11324can generate tarballs and SRPMs and can create them with various levels
11325of compliance in mind.
11326
11327One way of doing this (but certainly not the only way) is to release
11328just the source as a tarball. You can do this by adding the following to
11329the ``local.conf`` file found in the
11330:term:`Build Directory`:
11331::
11332
11333 INHERIT += "archiver"
11334 ARCHIVER_MODE[src] = "original"
11335
11336During the creation of your
11337image, the source from all recipes that deploy packages to the image is
11338placed within subdirectories of ``DEPLOY_DIR/sources`` based on the
11339:term:`LICENSE` for each recipe.
11340Releasing the entire directory enables you to comply with requirements
11341concerning providing the unmodified source. It is important to note that
11342the size of the directory can get large.
11343
11344A way to help mitigate the size issue is to only release tarballs for
11345licenses that require the release of source. Let us assume you are only
11346concerned with GPL code as identified by running the following script:
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011347
11348.. code-block:: shell
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011349
11350 # Script to archive a subset of packages matching specific license(s)
11351 # Source and license files are copied into sub folders of package folder
11352 # Must be run from build folder
11353 #!/bin/bash
11354 src_release_dir="source-release"
11355 mkdir -p $src_release_dir
11356 for a in tmp/deploy/sources/*; do
11357 for d in $a/*; do
11358 # Get package name from path
11359 p=`basename $d`
11360 p=${p%-*}
11361 p=${p%-*}
11362 # Only archive GPL packages (update *GPL* regex for your license check)
11363 numfiles=`ls tmp/deploy/licenses/$p/*GPL* 2> /dev/null | wc -l`
11364 if [ $numfiles -gt 1 ]; then
11365 echo Archiving $p
11366 mkdir -p $src_release_dir/$p/source
11367 cp $d/* $src_release_dir/$p/source 2> /dev/null
11368 mkdir -p $src_release_dir/$p/license
11369 cp tmp/deploy/licenses/$p/* $src_release_dir/$p/license 2> /dev/null
11370 fi
11371 done
11372 done
11373
11374At this point, you
11375could create a tarball from the ``gpl_source_release`` directory and
11376provide that to the end user. This method would be a step toward
11377achieving compliance with section 3a of GPLv2 and with section 6 of
11378GPLv3.
11379
11380Providing License Text
11381~~~~~~~~~~~~~~~~~~~~~~
11382
11383One requirement that is often overlooked is inclusion of license text.
11384This requirement also needs to be dealt with prior to generating the
11385final image. Some licenses require the license text to accompany the
11386binary. You can achieve this by adding the following to your
11387``local.conf`` file:
11388::
11389
11390 COPY_LIC_MANIFEST = "1"
11391 COPY_LIC_DIRS = "1"
11392 LICENSE_CREATE_PACKAGE = "1"
11393
11394Adding these statements to the
11395configuration file ensures that the licenses collected during package
11396generation are included on your image.
11397
11398.. note::
11399
11400 Setting all three variables to "1" results in the image having two
11401 copies of the same license file. One copy resides in
11402 ``/usr/share/common-licenses`` and the other resides in
11403 ``/usr/share/license``.
11404
11405 The reason for this behavior is because
11406 :term:`COPY_LIC_DIRS` and
11407 :term:`COPY_LIC_MANIFEST`
11408 add a copy of the license when the image is built but do not offer a
11409 path for adding licenses for newly installed packages to an image.
11410 :term:`LICENSE_CREATE_PACKAGE`
11411 adds a separate package and an upgrade path for adding licenses to an
11412 image.
11413
11414As the source ``archiver`` class has already archived the original
11415unmodified source that contains the license files, you would have
11416already met the requirements for inclusion of the license information
11417with source as defined by the GPL and other open source licenses.
11418
11419Providing Compilation Scripts and Source Code Modifications
11420~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
11421
11422At this point, we have addressed all we need to prior to generating the
11423image. The next two requirements are addressed during the final
11424packaging of the release.
11425
11426By releasing the version of the OpenEmbedded build system and the layers
11427used during the build, you will be providing both compilation scripts
11428and the source code modifications in one step.
11429
11430If the deployment team has a :ref:`overview-manual/overview-manual-concepts:bsp layer`
11431and a distro layer, and those
11432those layers are used to patch, compile, package, or modify (in any way)
11433any open source software included in your released images, you might be
11434required to release those layers under section 3 of GPLv2 or section 1
11435of GPLv3. One way of doing that is with a clean checkout of the version
11436of the Yocto Project and layers used during your build. Here is an
11437example:
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011438
11439.. code-block:: shell
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011440
11441 # We built using the dunfell branch of the poky repo
11442 $ git clone -b dunfell git://git.yoctoproject.org/poky
11443 $ cd poky
11444 # We built using the release_branch for our layers
11445 $ git clone -b release_branch git://git.mycompany.com/meta-my-bsp-layer
11446 $ git clone -b release_branch git://git.mycompany.com/meta-my-software-layer
11447 # clean up the .git repos
11448 $ find . -name ".git" -type d -exec rm -rf {} \;
11449
11450One
11451thing a development organization might want to consider for end-user
11452convenience is to modify ``meta-poky/conf/bblayers.conf.sample`` to
11453ensure that when the end user utilizes the released build system to
11454build an image, the development organization's layers are included in
11455the ``bblayers.conf`` file automatically:
11456::
11457
11458 # POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
11459 # changes incompatibly
11460 POKY_BBLAYERS_CONF_VERSION = "2"
11461
11462 BBPATH = "${TOPDIR}"
11463 BBFILES ?= ""
11464
11465 BBLAYERS ?= " \
11466 ##OEROOT##/meta \
11467 ##OEROOT##/meta-poky \
11468 ##OEROOT##/meta-yocto-bsp \
11469 ##OEROOT##/meta-mylayer \
11470 "
11471
11472Creating and
11473providing an archive of the :term:`Metadata`
11474layers (recipes, configuration files, and so forth) enables you to meet
11475your requirements to include the scripts to control compilation as well
11476as any modifications to the original source.
11477
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011478Providing spdx files
11479~~~~~~~~~~~~~~~~~~~~~~~~~
11480
11481The spdx module has been integrated to a layer named meta-spdxscanner.
11482meta-spdxscanner provides several kinds of scanner. If you want to enable
11483this function, you have to follow the following steps:
11484
114851. Add meta-spdxscanner layer into ``bblayers.conf``.
11486
114872. Refer to the README in meta-spdxscanner to setup the environment (e.g,
11488 setup a fossology server) needed for the scanner.
11489
114903. Meta-spdxscanner provides several methods within the bbclass to create spdx files.
11491 Please choose one that you want to use and enable the spdx task. You have to
11492 add some config options in ``local.conf`` file in your :term:`Build
11493 Directory`. The following is an example showing how to generate spdx files
11494 during bitbake using the fossology-python.bbclass::
11495
11496 # Select fossology-python.bbclass.
11497 INHERIT += "fossology-python"
11498 # For fossology-python.bbclass, TOKEN is necessary, so, after setup a
11499 # Fossology server, you have to create a token.
11500 TOKEN = "eyJ0eXAiO..."
11501 # The fossology server is necessary for fossology-python.bbclass.
11502 FOSSOLOGY_SERVER = "http://xx.xx.xx.xx:8081/repo"
11503 # If you want to upload the source code to a special folder:
11504 FOLDER_NAME = "xxxx" //Optional
11505 # If you don't want to put spdx files in tmp/deploy/spdx, you can enable:
11506 SPDX_DEPLOY_DIR = "${DEPLOY_DIR}" //Optional
11507
11508For more usage information refer to :yocto_git:`the meta-spdxscanner repository
11509</cgit/cgit.cgi/meta-spdxscanner/>`.
11510
11511
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011512Copying Licenses that Do Not Exist
11513----------------------------------
11514
11515Some packages, such as the linux-firmware package, have many licenses
11516that are not in any way common. You can avoid adding a lot of these
11517types of common license files, which are only applicable to a specific
11518package, by using the
11519:term:`NO_GENERIC_LICENSE`
11520variable. Using this variable also avoids QA errors when you use a
11521non-common, non-CLOSED license in a recipe.
11522
11523The following is an example that uses the ``LICENSE.Abilis.txt`` file as
11524the license from the fetched source:
11525::
11526
11527 NO_GENERIC_LICENSE[Firmware-Abilis] = "LICENSE.Abilis.txt"
11528
11529Using the Error Reporting Tool
11530==============================
11531
11532The error reporting tool allows you to submit errors encountered during
11533builds to a central database. Outside of the build environment, you can
11534use a web interface to browse errors, view statistics, and query for
11535errors. The tool works using a client-server system where the client
11536portion is integrated with the installed Yocto Project
11537:term:`Source Directory` (e.g. ``poky``).
11538The server receives the information collected and saves it in a
11539database.
11540
11541A live instance of the error reporting server exists at
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011542https://errors.yoctoproject.org. This server exists so that when
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011543you want to get help with build failures, you can submit all of the
11544information on the failure easily and then point to the URL in your bug
11545report or send an email to the mailing list.
11546
11547.. note::
11548
11549 If you send error reports to this server, the reports become publicly
11550 visible.
11551
11552Enabling and Using the Tool
11553---------------------------
11554
11555By default, the error reporting tool is disabled. You can enable it by
11556inheriting the
11557:ref:`report-error <ref-classes-report-error>`
11558class by adding the following statement to the end of your
11559``local.conf`` file in your
11560:term:`Build Directory`.
11561::
11562
11563 INHERIT += "report-error"
11564
11565By default, the error reporting feature stores information in
11566``${``\ :term:`LOG_DIR`\ ``}/error-report``.
11567However, you can specify a directory to use by adding the following to
11568your ``local.conf`` file:
11569::
11570
11571 ERR_REPORT_DIR = "path"
11572
11573Enabling error
11574reporting causes the build process to collect the errors and store them
11575in a file as previously described. When the build system encounters an
11576error, it includes a command as part of the console output. You can run
11577the command to send the error file to the server. For example, the
11578following command sends the errors to an upstream server:
11579::
11580
11581 $ send-error-report /home/brandusa/project/poky/build/tmp/log/error-report/error_report_201403141617.txt
11582
11583In the previous example, the errors are sent to a public database
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011584available at https://errors.yoctoproject.org, which is used by the
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011585entire community. If you specify a particular server, you can send the
11586errors to a different database. Use the following command for more
11587information on available options:
11588::
11589
11590 $ send-error-report --help
11591
11592When sending the error file, you are prompted to review the data being
11593sent as well as to provide a name and optional email address. Once you
11594satisfy these prompts, the command returns a link from the server that
11595corresponds to your entry in the database. For example, here is a
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011596typical link: https://errors.yoctoproject.org/Errors/Details/9522/
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011597
11598Following the link takes you to a web interface where you can browse,
11599query the errors, and view statistics.
11600
11601Disabling the Tool
11602------------------
11603
11604To disable the error reporting feature, simply remove or comment out the
11605following statement from the end of your ``local.conf`` file in your
11606:term:`Build Directory`.
11607::
11608
11609 INHERIT += "report-error"
11610
11611Setting Up Your Own Error Reporting Server
11612------------------------------------------
11613
11614If you want to set up your own error reporting server, you can obtain
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011615the code from the Git repository at :yocto_git:`/cgit/cgit.cgi/error-report-web/`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011616Instructions on how to set it up are in the README document.
11617
11618.. _dev-using-wayland-and-weston:
11619
11620Using Wayland and Weston
11621========================
11622
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011623`Wayland <https://en.wikipedia.org/wiki/Wayland_(display_server_protocol)>`__
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011624is a computer display server protocol that provides a method for
11625compositing window managers to communicate directly with applications
11626and video hardware and expects them to communicate with input hardware
11627using other libraries. Using Wayland with supporting targets can result
11628in better control over graphics frame rendering than an application
11629might otherwise achieve.
11630
11631The Yocto Project provides the Wayland protocol libraries and the
11632reference
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011633`Weston <https://en.wikipedia.org/wiki/Wayland_(display_server_protocol)#Weston>`__
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011634compositor as part of its release. You can find the integrated packages
11635in the ``meta`` layer of the :term:`Source Directory`.
11636Specifically, you
11637can find the recipes that build both Wayland and Weston at
11638``meta/recipes-graphics/wayland``.
11639
11640You can build both the Wayland and Weston packages for use only with
11641targets that accept the `Mesa 3D and Direct Rendering
11642Infrastructure <https://en.wikipedia.org/wiki/Mesa_(computer_graphics)>`__,
11643which is also known as Mesa DRI. This implies that you cannot build and
11644use the packages if your target uses, for example, the Intel Embedded
11645Media and Graphics Driver (Intel EMGD) that overrides Mesa DRI.
11646
11647.. note::
11648
11649 Due to lack of EGL support, Weston 1.0.3 will not run directly on the
11650 emulated QEMU hardware. However, this version of Weston will run
11651 under X emulation without issues.
11652
11653This section describes what you need to do to implement Wayland and use
11654the Weston compositor when building an image for a supporting target.
11655
11656Enabling Wayland in an Image
11657----------------------------
11658
11659To enable Wayland, you need to enable it to be built and enable it to be
11660included (installed) in the image.
11661
11662.. _enable-building:
11663
11664Building Wayland
11665~~~~~~~~~~~~~~~~
11666
11667To cause Mesa to build the ``wayland-egl`` platform and Weston to build
11668Wayland with Kernel Mode Setting
11669(`KMS <https://wiki.archlinux.org/index.php/Kernel_Mode_Setting>`__)
11670support, include the "wayland" flag in the
11671:term:`DISTRO_FEATURES`
11672statement in your ``local.conf`` file:
11673::
11674
11675 DISTRO_FEATURES_append = " wayland"
11676
11677.. note::
11678
11679 If X11 has been enabled elsewhere, Weston will build Wayland with X11
11680 support
11681
11682.. _enable-installation-in-an-image:
11683
11684Installing Wayland and Weston
11685~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
11686
11687To install the Wayland feature into an image, you must include the
11688following
11689:term:`CORE_IMAGE_EXTRA_INSTALL`
11690statement in your ``local.conf`` file:
11691::
11692
11693 CORE_IMAGE_EXTRA_INSTALL += "wayland weston"
11694
11695Running Weston
11696--------------
11697
11698To run Weston inside X11, enabling it as described earlier and building
11699a Sato image is sufficient. If you are running your image under Sato, a
11700Weston Launcher appears in the "Utility" category.
11701
11702Alternatively, you can run Weston through the command-line interpretor
11703(CLI), which is better suited for development work. To run Weston under
11704the CLI, you need to do the following after your image is built:
11705
117061. Run these commands to export ``XDG_RUNTIME_DIR``:
11707 ::
11708
11709 mkdir -p /tmp/$USER-weston
11710 chmod 0700 /tmp/$USER-weston
11711 export XDG_RUNTIME_DIR=/tmp/$USER-weston
11712
117132. Launch Weston in the shell:
11714 ::
11715
11716 weston