blob: 2af2896171e008213ff192fd36a02f39f02ffb7c [file] [log] [blame]
Andrew Geisslerf0343792020-11-18 10:42:21 -06001.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002
3************
4Common Tasks
5************
6
7This chapter describes fundamental procedures such as creating layers,
8adding new software packages, extending or customizing images, porting
9work to new hardware (adding a new machine), and so forth. You will find
10that the procedures documented here occur often in the development cycle
11using the Yocto Project.
12
13Understanding and Creating Layers
14=================================
15
16The OpenEmbedded build system supports organizing
17:term:`Metadata` into multiple layers.
18Layers allow you to isolate different types of customizations from each
19other. For introductory information on the Yocto Project Layer Model,
20see the
Andrew Geissler09209ee2020-12-13 08:44:15 -060021":ref:`overview-manual/yp-intro:the yocto project layer model`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050022section in the Yocto Project Overview and Concepts Manual.
23
24Creating Your Own Layer
25-----------------------
26
27It is very easy to create your own layers to use with the OpenEmbedded
28build system. The Yocto Project ships with tools that speed up creating
29layers. This section describes the steps you perform by hand to create
30layers so that you can better understand them. For information about the
31layer-creation tools, see the
32":ref:`bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script`"
33section in the Yocto Project Board Support Package (BSP) Developer's
Andrew Geissler09209ee2020-12-13 08:44:15 -060034Guide and the ":ref:`dev-manual/common-tasks:creating a general layer using the \`\`bitbake-layers\`\` script`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050035section further down in this manual.
36
37Follow these general steps to create your layer without using tools:
38
391. *Check Existing Layers:* Before creating a new layer, you should be
40 sure someone has not already created a layer containing the Metadata
Andrew Geisslerd1e89492021-02-12 15:35:20 -060041 you need. You can see the :oe_layerindex:`OpenEmbedded Metadata Index <>`
42 for a list of layers from the OpenEmbedded community that can be used in
Andrew Geisslerc9f78652020-09-18 14:11:35 -050043 the Yocto Project. You could find a layer that is identical or close
44 to what you need.
45
462. *Create a Directory:* Create the directory for your layer. When you
47 create the layer, be sure to create the directory in an area not
48 associated with the Yocto Project :term:`Source Directory`
49 (e.g. the cloned ``poky`` repository).
50
51 While not strictly required, prepend the name of the directory with
Andrew Geisslerc926e172021-05-07 16:11:35 -050052 the string "meta-". For example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050053
54 meta-mylayer
55 meta-GUI_xyz
56 meta-mymachine
57
Andrew Geisslerc926e172021-05-07 16:11:35 -050058 With rare exceptions, a layer's name follows this form::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050059
60 meta-root_name
61
62 Following this layer naming convention can save
63 you trouble later when tools, components, or variables "assume" your
64 layer name begins with "meta-". A notable example is in configuration
65 files as shown in the following step where layer names without the
66 "meta-" string are appended to several variables used in the
67 configuration.
68
693. *Create a Layer Configuration File:* Inside your new layer folder,
70 you need to create a ``conf/layer.conf`` file. It is easiest to take
71 an existing layer configuration file and copy that to your layer's
72 ``conf`` directory and then modify the file as needed.
73
74 The ``meta-yocto-bsp/conf/layer.conf`` file in the Yocto Project
Andrew Geissler09209ee2020-12-13 08:44:15 -060075 :yocto_git:`Source Repositories </poky/tree/meta-yocto-bsp/conf>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -050076 demonstrates the required syntax. For your layer, you need to replace
77 "yoctobsp" with a unique identifier for your layer (e.g. "machinexyz"
Andrew Geisslerc926e172021-05-07 16:11:35 -050078 for a layer named "meta-machinexyz")::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050079
80 # We have a conf and classes directory, add to BBPATH
81 BBPATH .= ":${LAYERDIR}"
82
Andrew Geissler4c19ea12020-10-27 13:52:24 -050083 # We have recipes-* directories, add to BBFILES
Andrew Geisslerc9f78652020-09-18 14:11:35 -050084 BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
85 ${LAYERDIR}/recipes-*/*/*.bbappend"
86
87 BBFILE_COLLECTIONS += "yoctobsp"
88 BBFILE_PATTERN_yoctobsp = "^${LAYERDIR}/"
89 BBFILE_PRIORITY_yoctobsp = "5"
90 LAYERVERSION_yoctobsp = "4"
91 LAYERSERIES_COMPAT_yoctobsp = "dunfell"
92
93 Following is an explanation of the layer configuration file:
94
95 - :term:`BBPATH`: Adds the layer's
96 root directory to BitBake's search path. Through the use of the
Andrew Geissler09036742021-06-25 14:25:14 -050097 :term:`BBPATH` variable, BitBake locates class files (``.bbclass``),
Andrew Geisslerc9f78652020-09-18 14:11:35 -050098 configuration files, and files that are included with ``include``
99 and ``require`` statements. For these cases, BitBake uses the
Andrew Geissler09036742021-06-25 14:25:14 -0500100 first file that matches the name found in :term:`BBPATH`. This is
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500101 similar to the way the ``PATH`` variable is used for binaries. It
102 is recommended, therefore, that you use unique class and
103 configuration filenames in your custom layer.
104
105 - :term:`BBFILES`: Defines the
106 location for all recipes in the layer.
107
108 - :term:`BBFILE_COLLECTIONS`:
109 Establishes the current layer through a unique identifier that is
110 used throughout the OpenEmbedded build system to refer to the
111 layer. In this example, the identifier "yoctobsp" is the
112 representation for the container layer named "meta-yocto-bsp".
113
114 - :term:`BBFILE_PATTERN`:
115 Expands immediately during parsing to provide the directory of the
116 layer.
117
118 - :term:`BBFILE_PRIORITY`:
119 Establishes a priority to use for recipes in the layer when the
120 OpenEmbedded build finds recipes of the same name in different
121 layers.
122
123 - :term:`LAYERVERSION`:
124 Establishes a version number for the layer. You can use this
125 version number to specify this exact version of the layer as a
126 dependency when using the
127 :term:`LAYERDEPENDS`
128 variable.
129
130 - :term:`LAYERDEPENDS`:
131 Lists all layers on which this layer depends (if any).
132
133 - :term:`LAYERSERIES_COMPAT`:
Andrew Geissler09209ee2020-12-13 08:44:15 -0600134 Lists the :yocto_wiki:`Yocto Project </Releases>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500135 releases for which the current version is compatible. This
136 variable is a good way to indicate if your particular layer is
137 current.
138
1394. *Add Content:* Depending on the type of layer, add the content. If
140 the layer adds support for a machine, add the machine configuration
141 in a ``conf/machine/`` file within the layer. If the layer adds
142 distro policy, add the distro configuration in a ``conf/distro/``
143 file within the layer. If the layer introduces new recipes, put the
144 recipes you need in ``recipes-*`` subdirectories within the layer.
145
146 .. note::
147
148 For an explanation of layer hierarchy that is compliant with the
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500149 Yocto Project, see the ":ref:`bsp-guide/bsp:example filesystem layout`"
150 section in the Yocto Project Board Support Package (BSP) Developer's Guide.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500151
1525. *Optionally Test for Compatibility:* If you want permission to use
153 the Yocto Project Compatibility logo with your layer or application
154 that uses your layer, perform the steps to apply for compatibility.
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500155 See the
156 ":ref:`dev-manual/common-tasks:making sure your layer is compatible with yocto project`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500157 section for more information.
158
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500159Following Best Practices When Creating Layers
160---------------------------------------------
161
162To create layers that are easier to maintain and that will not impact
163builds for other machines, you should consider the information in the
164following list:
165
166- *Avoid "Overlaying" Entire Recipes from Other Layers in Your
167 Configuration:* In other words, do not copy an entire recipe into
168 your layer and then modify it. Rather, use an append file
169 (``.bbappend``) to override only those parts of the original recipe
170 you need to modify.
171
172- *Avoid Duplicating Include Files:* Use append files (``.bbappend``)
173 for each recipe that uses an include file. Or, if you are introducing
174 a new recipe that requires the included file, use the path relative
175 to the original layer directory to refer to the file. For example,
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500176 use ``require recipes-core/``\ `package`\ ``/``\ `file`\ ``.inc`` instead
177 of ``require`` `file`\ ``.inc``. If you're finding you have to overlay
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500178 the include file, it could indicate a deficiency in the include file
179 in the layer to which it originally belongs. If this is the case, you
180 should try to address that deficiency instead of overlaying the
181 include file. For example, you could address this by getting the
182 maintainer of the include file to add a variable or variables to make
183 it easy to override the parts needing to be overridden.
184
185- *Structure Your Layers:* Proper use of overrides within append files
186 and placement of machine-specific files within your layer can ensure
187 that a build is not using the wrong Metadata and negatively impacting
188 a build for a different machine. Following are some examples:
189
190 - *Modify Variables to Support a Different Machine:* Suppose you
191 have a layer named ``meta-one`` that adds support for building
192 machine "one". To do so, you use an append file named
193 ``base-files.bbappend`` and create a dependency on "foo" by
194 altering the :term:`DEPENDS`
Andrew Geisslerc926e172021-05-07 16:11:35 -0500195 variable::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500196
197 DEPENDS = "foo"
198
199 The dependency is created during any
200 build that includes the layer ``meta-one``. However, you might not
201 want this dependency for all machines. For example, suppose you
202 are building for machine "two" but your ``bblayers.conf`` file has
203 the ``meta-one`` layer included. During the build, the
204 ``base-files`` for machine "two" will also have the dependency on
205 ``foo``.
206
207 To make sure your changes apply only when building machine "one",
Andrew Geissler09036742021-06-25 14:25:14 -0500208 use a machine override with the :term:`DEPENDS` statement::
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500209
210 DEPENDS_one = "foo"
211
212 You should follow the same strategy when using ``_append``
Andrew Geisslerc926e172021-05-07 16:11:35 -0500213 and ``_prepend`` operations::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500214
215 DEPENDS_append_one = " foo"
216 DEPENDS_prepend_one = "foo "
217
218 As an actual example, here's a
219 snippet from the generic kernel include file ``linux-yocto.inc``,
220 wherein the kernel compile and link options are adjusted in the
Andrew Geisslerc926e172021-05-07 16:11:35 -0500221 case of a subset of the supported architectures::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500222
223 DEPENDS_append_aarch64 = " libgcc"
224 KERNEL_CC_append_aarch64 = " ${TOOLCHAIN_OPTIONS}"
225 KERNEL_LD_append_aarch64 = " ${TOOLCHAIN_OPTIONS}"
226
227 DEPENDS_append_nios2 = " libgcc"
228 KERNEL_CC_append_nios2 = " ${TOOLCHAIN_OPTIONS}"
229 KERNEL_LD_append_nios2 = " ${TOOLCHAIN_OPTIONS}"
230
231 DEPENDS_append_arc = " libgcc"
232 KERNEL_CC_append_arc = " ${TOOLCHAIN_OPTIONS}"
233 KERNEL_LD_append_arc = " ${TOOLCHAIN_OPTIONS}"
234
235 KERNEL_FEATURES_append_qemuall=" features/debug/printk.scc"
236
237 .. note::
238
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500239 Avoiding "+=" and "=+" and using machine-specific ``_append``
240 and ``_prepend`` operations is recommended as well.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500241
242 - *Place Machine-Specific Files in Machine-Specific Locations:* When
243 you have a base recipe, such as ``base-files.bb``, that contains a
244 :term:`SRC_URI` statement to a
245 file, you can use an append file to cause the build to use your
246 own version of the file. For example, an append file in your layer
247 at ``meta-one/recipes-core/base-files/base-files.bbappend`` could
Andrew Geisslerc926e172021-05-07 16:11:35 -0500248 extend :term:`FILESPATH` using :term:`FILESEXTRAPATHS` as follows::
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500249
250 FILESEXTRAPATHS_prepend := "${THISDIR}/${BPN}:"
251
252 The build for machine "one" will pick up your machine-specific file as
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500253 long as you have the file in
254 ``meta-one/recipes-core/base-files/base-files/``. However, if you
255 are building for a different machine and the ``bblayers.conf``
256 file includes the ``meta-one`` layer and the location of your
257 machine-specific file is the first location where that file is
Andrew Geissler09036742021-06-25 14:25:14 -0500258 found according to :term:`FILESPATH`, builds for all machines will
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500259 also use that machine-specific file.
260
261 You can make sure that a machine-specific file is used for a
262 particular machine by putting the file in a subdirectory specific
263 to the machine. For example, rather than placing the file in
264 ``meta-one/recipes-core/base-files/base-files/`` as shown above,
265 put it in ``meta-one/recipes-core/base-files/base-files/one/``.
266 Not only does this make sure the file is used only when building
267 for machine "one", but the build process locates the file more
268 quickly.
269
270 In summary, you need to place all files referenced from
Andrew Geissler5f350902021-07-23 13:09:54 -0400271 :term:`SRC_URI` in a machine-specific subdirectory within the layer in
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500272 order to restrict those files to machine-specific builds.
273
274- *Perform Steps to Apply for Yocto Project Compatibility:* If you want
275 permission to use the Yocto Project Compatibility logo with your
276 layer or application that uses your layer, perform the steps to apply
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500277 for compatibility. See the
278 ":ref:`dev-manual/common-tasks:making sure your layer is compatible with yocto project`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500279 section for more information.
280
281- *Follow the Layer Naming Convention:* Store custom layers in a Git
282 repository that use the ``meta-layer_name`` format.
283
284- *Group Your Layers Locally:* Clone your repository alongside other
285 cloned ``meta`` directories from the :term:`Source Directory`.
286
287Making Sure Your Layer is Compatible With Yocto Project
288-------------------------------------------------------
289
290When you create a layer used with the Yocto Project, it is advantageous
291to make sure that the layer interacts well with existing Yocto Project
292layers (i.e. the layer is compatible with the Yocto Project). Ensuring
293compatibility makes the layer easy to be consumed by others in the Yocto
294Project community and could allow you permission to use the Yocto
295Project Compatible Logo.
296
297.. note::
298
299 Only Yocto Project member organizations are permitted to use the
300 Yocto Project Compatible Logo. The logo is not available for general
301 use. For information on how to become a Yocto Project member
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500302 organization, see the :yocto_home:`Yocto Project Website <>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500303
304The Yocto Project Compatibility Program consists of a layer application
305process that requests permission to use the Yocto Project Compatibility
306Logo for your layer and application. The process consists of two parts:
307
3081. Successfully passing a script (``yocto-check-layer``) that when run
309 against your layer, tests it against constraints based on experiences
310 of how layers have worked in the real world and where pitfalls have
311 been found. Getting a "PASS" result from the script is required for
312 successful compatibility registration.
313
3142. Completion of an application acceptance form, which you can find at
Andrew Geisslerd1e89492021-02-12 15:35:20 -0600315 :yocto_home:`/webform/yocto-project-compatible-registration`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500316
317To be granted permission to use the logo, you need to satisfy the
318following:
319
320- Be able to check the box indicating that you got a "PASS" when
321 running the script against your layer.
322
323- Answer "Yes" to the questions on the form or have an acceptable
324 explanation for any questions answered "No".
325
326- Be a Yocto Project Member Organization.
327
328The remainder of this section presents information on the registration
329form and on the ``yocto-check-layer`` script.
330
331Yocto Project Compatible Program Application
332~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
333
334Use the form to apply for your layer's approval. Upon successful
335application, you can use the Yocto Project Compatibility Logo with your
336layer and the application that uses your layer.
337
338To access the form, use this link:
Andrew Geisslerd1e89492021-02-12 15:35:20 -0600339:yocto_home:`/webform/yocto-project-compatible-registration`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500340Follow the instructions on the form to complete your application.
341
342The application consists of the following sections:
343
344- *Contact Information:* Provide your contact information as the fields
345 require. Along with your information, provide the released versions
346 of the Yocto Project for which your layer is compatible.
347
348- *Acceptance Criteria:* Provide "Yes" or "No" answers for each of the
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700349 items in the checklist. There is space at the bottom of the form for
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500350 any explanations for items for which you answered "No".
351
352- *Recommendations:* Provide answers for the questions regarding Linux
353 kernel use and build success.
354
355``yocto-check-layer`` Script
356~~~~~~~~~~~~~~~~~~~~~~~~~~~~
357
358The ``yocto-check-layer`` script provides you a way to assess how
359compatible your layer is with the Yocto Project. You should run this
360script prior to using the form to apply for compatibility as described
361in the previous section. You need to achieve a "PASS" result in order to
362have your application form successfully processed.
363
364The script divides tests into three areas: COMMON, BSP, and DISTRO. For
365example, given a distribution layer (DISTRO), the layer must pass both
366the COMMON and DISTRO related tests. Furthermore, if your layer is a BSP
367layer, the layer must pass the COMMON and BSP set of tests.
368
369To execute the script, enter the following commands from your build
Andrew Geisslerc926e172021-05-07 16:11:35 -0500370directory::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500371
372 $ source oe-init-build-env
373 $ yocto-check-layer your_layer_directory
374
375Be sure to provide the actual directory for your
376layer as part of the command.
377
378Entering the command causes the script to determine the type of layer
379and then to execute a set of specific tests against the layer. The
380following list overviews the test:
381
382- ``common.test_readme``: Tests if a ``README`` file exists in the
383 layer and the file is not empty.
384
385- ``common.test_parse``: Tests to make sure that BitBake can parse the
386 files without error (i.e. ``bitbake -p``).
387
388- ``common.test_show_environment``: Tests that the global or per-recipe
389 environment is in order without errors (i.e. ``bitbake -e``).
390
391- ``common.test_world``: Verifies that ``bitbake world`` works.
392
393- ``common.test_signatures``: Tests to be sure that BSP and DISTRO
394 layers do not come with recipes that change signatures.
395
396- ``common.test_layerseries_compat``: Verifies layer compatibility is
397 set properly.
398
399- ``bsp.test_bsp_defines_machines``: Tests if a BSP layer has machine
400 configurations.
401
402- ``bsp.test_bsp_no_set_machine``: Tests to ensure a BSP layer does not
403 set the machine when the layer is added.
404
405- ``bsp.test_machine_world``: Verifies that ``bitbake world`` works
406 regardless of which machine is selected.
407
408- ``bsp.test_machine_signatures``: Verifies that building for a
409 particular machine affects only the signature of tasks specific to
410 that machine.
411
412- ``distro.test_distro_defines_distros``: Tests if a DISTRO layer has
413 distro configurations.
414
415- ``distro.test_distro_no_set_distros``: Tests to ensure a DISTRO layer
416 does not set the distribution when the layer is added.
417
418Enabling Your Layer
419-------------------
420
421Before the OpenEmbedded build system can use your new layer, you need to
422enable it. To enable your layer, simply add your layer's path to the
Andrew Geissler09036742021-06-25 14:25:14 -0500423:term:`BBLAYERS` variable in your ``conf/bblayers.conf`` file, which is
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500424found in the :term:`Build Directory`.
425The following example shows how to enable a layer named
Andrew Geisslerc926e172021-05-07 16:11:35 -0500426``meta-mylayer``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500427
428 # POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
429 # changes incompatibly
430 POKY_BBLAYERS_CONF_VERSION = "2"
431 BBPATH = "${TOPDIR}"
432 BBFILES ?= ""
433 BBLAYERS ?= " \
434 /home/user/poky/meta \
435 /home/user/poky/meta-poky \
436 /home/user/poky/meta-yocto-bsp \
437 /home/user/poky/meta-mylayer \
438 "
439
440BitBake parses each ``conf/layer.conf`` file from the top down as
Andrew Geissler09036742021-06-25 14:25:14 -0500441specified in the :term:`BBLAYERS` variable within the ``conf/bblayers.conf``
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500442file. During the processing of each ``conf/layer.conf`` file, BitBake
443adds the recipes, classes and configurations contained within the
444particular layer to the source directory.
445
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500446Using .bbappend Files in Your Layer
447-----------------------------------
448
449A recipe that appends Metadata to another recipe is called a BitBake
450append file. A BitBake append file uses the ``.bbappend`` file type
451suffix, while the corresponding recipe to which Metadata is being
452appended uses the ``.bb`` file type suffix.
453
454You can use a ``.bbappend`` file in your layer to make additions or
455changes to the content of another layer's recipe without having to copy
456the other layer's recipe into your layer. Your ``.bbappend`` file
457resides in your layer, while the main ``.bb`` recipe file to which you
458are appending Metadata resides in a different layer.
459
460Being able to append information to an existing recipe not only avoids
461duplication, but also automatically applies recipe changes from a
462different layer into your layer. If you were copying recipes, you would
463have to manually merge changes as they occur.
464
465When you create an append file, you must use the same root name as the
466corresponding recipe file. For example, the append file
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500467``someapp_3.1.bbappend`` must apply to ``someapp_3.1.bb``. This
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500468means the original recipe and append file names are version
469number-specific. If the corresponding recipe is renamed to update to a
470newer version, you must also rename and possibly update the
471corresponding ``.bbappend`` as well. During the build process, BitBake
472displays an error on starting if it detects a ``.bbappend`` file that
473does not have a corresponding recipe with a matching name. See the
474:term:`BB_DANGLINGAPPENDS_WARNONLY`
475variable for information on how to handle this error.
476
477As an example, consider the main formfactor recipe and a corresponding
478formfactor append file both from the :term:`Source Directory`.
479Here is the main
480formfactor recipe, which is named ``formfactor_0.0.bb`` and located in
Andrew Geisslerc926e172021-05-07 16:11:35 -0500481the "meta" layer at ``meta/recipes-bsp/formfactor``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500482
483 SUMMARY = "Device formfactor information"
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500484 DESCRIPTION = "A formfactor configuration file provides information about the \
485 target hardware for which the image is being built and information that the \
486 build system cannot obtain from other sources such as the kernel."
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500487 SECTION = "base"
488 LICENSE = "MIT"
489 LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
490 PR = "r45"
491
492 SRC_URI = "file://config file://machconfig"
493 S = "${WORKDIR}"
494
495 PACKAGE_ARCH = "${MACHINE_ARCH}"
496 INHIBIT_DEFAULT_DEPS = "1"
497
498 do_install() {
499 # Install file only if it has contents
500 install -d ${D}${sysconfdir}/formfactor/
501 install -m 0644 ${S}/config ${D}${sysconfdir}/formfactor/
502 if [ -s "${S}/machconfig" ]; then
503 install -m 0644 ${S}/machconfig ${D}${sysconfdir}/formfactor/
504 fi
505 }
506
507In the main recipe, note the :term:`SRC_URI`
508variable, which tells the OpenEmbedded build system where to find files
509during the build.
510
511Following is the append file, which is named ``formfactor_0.0.bbappend``
512and is from the Raspberry Pi BSP Layer named ``meta-raspberrypi``. The
Andrew Geisslerc926e172021-05-07 16:11:35 -0500513file is in the layer at ``recipes-bsp/formfactor``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500514
515 FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"
516
517By default, the build system uses the
518:term:`FILESPATH` variable to
519locate files. This append file extends the locations by setting the
520:term:`FILESEXTRAPATHS`
521variable. Setting this variable in the ``.bbappend`` file is the most
522reliable and recommended method for adding directories to the search
523path used by the build system to find files.
524
525The statement in this example extends the directories to include
526``${``\ :term:`THISDIR`\ ``}/${``\ :term:`PN`\ ``}``,
527which resolves to a directory named ``formfactor`` in the same directory
528in which the append file resides (i.e.
529``meta-raspberrypi/recipes-bsp/formfactor``. This implies that you must
530have the supporting directory structure set up that will contain any
531files or patches you will be including from the layer.
532
533Using the immediate expansion assignment operator ``:=`` is important
Andrew Geissler09036742021-06-25 14:25:14 -0500534because of the reference to :term:`THISDIR`. The trailing colon character is
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500535important as it ensures that items in the list remain colon-separated.
536
537.. note::
538
Andrew Geissler09036742021-06-25 14:25:14 -0500539 BitBake automatically defines the :term:`THISDIR` variable. You should
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500540 never set this variable yourself. Using "_prepend" as part of the
Andrew Geissler09036742021-06-25 14:25:14 -0500541 :term:`FILESEXTRAPATHS` ensures your path will be searched prior to other
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500542 paths in the final list.
543
544 Also, not all append files add extra files. Many append files simply
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700545 allow to add build options (e.g. ``systemd``). For these cases, your
Andrew Geissler09036742021-06-25 14:25:14 -0500546 append file would not even use the :term:`FILESEXTRAPATHS` statement.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500547
548Prioritizing Your Layer
549-----------------------
550
551Each layer is assigned a priority value. Priority values control which
552layer takes precedence if there are recipe files with the same name in
553multiple layers. For these cases, the recipe file from the layer with a
554higher priority number takes precedence. Priority values also affect the
555order in which multiple ``.bbappend`` files for the same recipe are
556applied. You can either specify the priority manually, or allow the
557build system to calculate it based on the layer's dependencies.
558
559To specify the layer's priority manually, use the
560:term:`BBFILE_PRIORITY`
Andrew Geisslerc926e172021-05-07 16:11:35 -0500561variable and append the layer's root name::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500562
563 BBFILE_PRIORITY_mylayer = "1"
564
565.. note::
566
567 It is possible for a recipe with a lower version number
568 :term:`PV` in a layer that has a higher
569 priority to take precedence.
570
571 Also, the layer priority does not currently affect the precedence
572 order of ``.conf`` or ``.bbclass`` files. Future versions of BitBake
573 might address this.
574
575Managing Layers
576---------------
577
578You can use the BitBake layer management tool ``bitbake-layers`` to
579provide a view into the structure of recipes across a multi-layer
580project. Being able to generate output that reports on configured layers
581with their paths and priorities and on ``.bbappend`` files and their
582applicable recipes can help to reveal potential problems.
583
584For help on the BitBake layer management tool, use the following
Andrew Geisslerc926e172021-05-07 16:11:35 -0500585command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500586
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500587 $ bitbake-layers --help
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500588 NOTE: Starting bitbake server...
589 usage: bitbake-layers [-d] [-q] [-F] [--color COLOR] [-h] <subcommand> ...
590
591 BitBake layers utility
592
593 optional arguments:
594 -d, --debug Enable debug output
595 -q, --quiet Print only errors
596 -F, --force Force add without recipe parse verification
597 --color COLOR Colorize output (where COLOR is auto, always, never)
598 -h, --help show this help message and exit
599
600 subcommands:
601 <subcommand>
602 layerindex-fetch Fetches a layer from a layer index along with its
603 dependent layers, and adds them to conf/bblayers.conf.
604 layerindex-show-depends
605 Find layer dependencies from layer index.
606 add-layer Add one or more layers to bblayers.conf.
607 remove-layer Remove one or more layers from bblayers.conf.
608 flatten flatten layer configuration into a separate output
609 directory.
610 show-layers show current configured layers.
611 show-overlayed list overlayed recipes (where the same recipe exists
612 in another layer)
613 show-recipes list available recipes, showing the layer they are
614 provided by
615 show-appends list bbappend files and recipe files they apply to
616 show-cross-depends Show dependencies between recipes that cross layer
617 boundaries.
618 create-layer Create a basic layer
619
620 Use bitbake-layers <subcommand> --help to get help on a specific command
621
622The following list describes the available commands:
623
624- ``help:`` Displays general help or help on a specified command.
625
626- ``show-layers:`` Shows the current configured layers.
627
628- ``show-overlayed:`` Lists overlayed recipes. A recipe is overlayed
629 when a recipe with the same name exists in another layer that has a
630 higher layer priority.
631
632- ``show-recipes:`` Lists available recipes and the layers that
633 provide them.
634
635- ``show-appends:`` Lists ``.bbappend`` files and the recipe files to
636 which they apply.
637
638- ``show-cross-depends:`` Lists dependency relationships between
639 recipes that cross layer boundaries.
640
641- ``add-layer:`` Adds a layer to ``bblayers.conf``.
642
643- ``remove-layer:`` Removes a layer from ``bblayers.conf``
644
645- ``flatten:`` Flattens the layer configuration into a separate
646 output directory. Flattening your layer configuration builds a
647 "flattened" directory that contains the contents of all layers, with
648 any overlayed recipes removed and any ``.bbappend`` files appended to
649 the corresponding recipes. You might have to perform some manual
650 cleanup of the flattened layer as follows:
651
652 - Non-recipe files (such as patches) are overwritten. The flatten
653 command shows a warning for these files.
654
655 - Anything beyond the normal layer setup has been added to the
656 ``layer.conf`` file. Only the lowest priority layer's
657 ``layer.conf`` is used.
658
659 - Overridden and appended items from ``.bbappend`` files need to be
660 cleaned up. The contents of each ``.bbappend`` end up in the
661 flattened recipe. However, if there are appended or changed
662 variable values, you need to tidy these up yourself. Consider the
663 following example. Here, the ``bitbake-layers`` command adds the
664 line ``#### bbappended ...`` so that you know where the following
Andrew Geisslerc926e172021-05-07 16:11:35 -0500665 lines originate::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500666
667 ...
668 DESCRIPTION = "A useful utility"
669 ...
670 EXTRA_OECONF = "--enable-something"
671 ...
672
673 #### bbappended from meta-anotherlayer ####
674
675 DESCRIPTION = "Customized utility"
676 EXTRA_OECONF += "--enable-somethingelse"
677
678
Andrew Geisslerc926e172021-05-07 16:11:35 -0500679 Ideally, you would tidy up these utilities as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500680
681 ...
682 DESCRIPTION = "Customized utility"
683 ...
684 EXTRA_OECONF = "--enable-something --enable-somethingelse"
685 ...
686
687- ``layerindex-fetch``: Fetches a layer from a layer index, along
688 with its dependent layers, and adds the layers to the
689 ``conf/bblayers.conf`` file.
690
691- ``layerindex-show-depends``: Finds layer dependencies from the
692 layer index.
693
694- ``create-layer``: Creates a basic layer.
695
696Creating a General Layer Using the ``bitbake-layers`` Script
697------------------------------------------------------------
698
699The ``bitbake-layers`` script with the ``create-layer`` subcommand
700simplifies creating a new general layer.
701
702.. note::
703
704 - For information on BSP layers, see the ":ref:`bsp-guide/bsp:bsp layers`"
705 section in the Yocto
706 Project Board Specific (BSP) Developer's Guide.
707
708 - In order to use a layer with the OpenEmbedded build system, you
709 need to add the layer to your ``bblayers.conf`` configuration
Andrew Geissler09209ee2020-12-13 08:44:15 -0600710 file. See the ":ref:`dev-manual/common-tasks:adding a layer using the \`\`bitbake-layers\`\` script`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500711 section for more information.
712
713The default mode of the script's operation with this subcommand is to
714create a layer with the following:
715
716- A layer priority of 6.
717
718- A ``conf`` subdirectory that contains a ``layer.conf`` file.
719
720- A ``recipes-example`` subdirectory that contains a further
721 subdirectory named ``example``, which contains an ``example.bb``
722 recipe file.
723
724- A ``COPYING.MIT``, which is the license statement for the layer. The
725 script assumes you want to use the MIT license, which is typical for
726 most layers, for the contents of the layer itself.
727
728- A ``README`` file, which is a file describing the contents of your
729 new layer.
730
731In its simplest form, you can use the following command form to create a
732layer. The command creates a layer whose name corresponds to
Andrew Geisslerc926e172021-05-07 16:11:35 -0500733"your_layer_name" in the current directory::
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500734
735 $ bitbake-layers create-layer your_layer_name
736
737As an example, the following command creates a layer named ``meta-scottrif``
Andrew Geisslerc926e172021-05-07 16:11:35 -0500738in your home directory::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500739
740 $ cd /usr/home
741 $ bitbake-layers create-layer meta-scottrif
742 NOTE: Starting bitbake server...
743 Add your new layer with 'bitbake-layers add-layer meta-scottrif'
744
745If you want to set the priority of the layer to other than the default
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500746value of "6", you can either use the ``--priority`` option or you
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500747can edit the
748:term:`BBFILE_PRIORITY` value
749in the ``conf/layer.conf`` after the script creates it. Furthermore, if
750you want to give the example recipe file some name other than the
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500751default, you can use the ``--example-recipe-name`` option.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500752
753The easiest way to see how the ``bitbake-layers create-layer`` command
754works is to experiment with the script. You can also read the usage
Andrew Geisslerc926e172021-05-07 16:11:35 -0500755information by entering the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500756
757 $ bitbake-layers create-layer --help
758 NOTE: Starting bitbake server...
759 usage: bitbake-layers create-layer [-h] [--priority PRIORITY]
760 [--example-recipe-name EXAMPLERECIPE]
761 layerdir
762
763 Create a basic layer
764
765 positional arguments:
766 layerdir Layer directory to create
767
768 optional arguments:
769 -h, --help show this help message and exit
770 --priority PRIORITY, -p PRIORITY
771 Layer directory to create
772 --example-recipe-name EXAMPLERECIPE, -e EXAMPLERECIPE
773 Filename of the example recipe
774
775Adding a Layer Using the ``bitbake-layers`` Script
776--------------------------------------------------
777
778Once you create your general layer, you must add it to your
779``bblayers.conf`` file. Adding the layer to this configuration file
780makes the OpenEmbedded build system aware of your layer so that it can
781search it for metadata.
782
Andrew Geisslerc926e172021-05-07 16:11:35 -0500783Add your layer by using the ``bitbake-layers add-layer`` command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500784
785 $ bitbake-layers add-layer your_layer_name
786
787Here is an example that adds a
788layer named ``meta-scottrif`` to the configuration file. Following the
789command that adds the layer is another ``bitbake-layers`` command that
Andrew Geisslerc926e172021-05-07 16:11:35 -0500790shows the layers that are in your ``bblayers.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500791
792 $ bitbake-layers add-layer meta-scottrif
793 NOTE: Starting bitbake server...
794 Parsing recipes: 100% |##########################################################| Time: 0:00:49
795 Parsing of 1441 .bb files complete (0 cached, 1441 parsed). 2055 targets, 56 skipped, 0 masked, 0 errors.
796 $ bitbake-layers show-layers
797 NOTE: Starting bitbake server...
798 layer path priority
799 ==========================================================================
800 meta /home/scottrif/poky/meta 5
801 meta-poky /home/scottrif/poky/meta-poky 5
802 meta-yocto-bsp /home/scottrif/poky/meta-yocto-bsp 5
803 workspace /home/scottrif/poky/build/workspace 99
804 meta-scottrif /home/scottrif/poky/build/meta-scottrif 6
805
806
807Adding the layer to this file
808enables the build system to locate the layer during the build.
809
810.. note::
811
812 During a build, the OpenEmbedded build system looks in the layers
813 from the top of the list down to the bottom in that order.
814
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500815Customizing Images
816==================
817
818You can customize images to satisfy particular requirements. This
819section describes several methods and provides guidelines for each.
820
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500821Customizing Images Using ``local.conf``
822---------------------------------------
823
824Probably the easiest way to customize an image is to add a package by
825way of the ``local.conf`` configuration file. Because it is limited to
826local use, this method generally only allows you to add packages and is
827not as flexible as creating your own customized image. When you add
828packages using local variables this way, you need to realize that these
829variable changes are in effect for every build and consequently affect
830all images, which might not be what you require.
831
832To add a package to your image using the local configuration file, use
Andrew Geissler09036742021-06-25 14:25:14 -0500833the :term:`IMAGE_INSTALL` variable with the ``_append`` operator::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500834
835 IMAGE_INSTALL_append = " strace"
836
837Use of the syntax is important -
838specifically, the space between the quote and the package name, which is
839``strace`` in this example. This space is required since the ``_append``
840operator does not add the space.
841
842Furthermore, you must use ``_append`` instead of the ``+=`` operator if
843you want to avoid ordering issues. The reason for this is because doing
844so unconditionally appends to the variable and avoids ordering problems
845due to the variable being set in image recipes and ``.bbclass`` files
846with operators like ``?=``. Using ``_append`` ensures the operation
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500847takes effect.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500848
849As shown in its simplest use, ``IMAGE_INSTALL_append`` affects all
850images. It is possible to extend the syntax so that the variable applies
Andrew Geisslerc926e172021-05-07 16:11:35 -0500851to a specific image only. Here is an example::
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500852
853 IMAGE_INSTALL_append_pn-core-image-minimal = " strace"
854
855This example adds ``strace`` to the ``core-image-minimal`` image only.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500856
857You can add packages using a similar approach through the
Andrew Geissler09036742021-06-25 14:25:14 -0500858:term:`CORE_IMAGE_EXTRA_INSTALL` variable. If you use this variable, only
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500859``core-image-*`` images are affected.
860
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500861Customizing Images Using Custom ``IMAGE_FEATURES`` and ``EXTRA_IMAGE_FEATURES``
862-------------------------------------------------------------------------------
863
864Another method for customizing your image is to enable or disable
865high-level image features by using the
866:term:`IMAGE_FEATURES` and
867:term:`EXTRA_IMAGE_FEATURES`
868variables. Although the functions for both variables are nearly
Andrew Geissler09036742021-06-25 14:25:14 -0500869equivalent, best practices dictate using :term:`IMAGE_FEATURES` from within
870a recipe and using :term:`EXTRA_IMAGE_FEATURES` from within your
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500871``local.conf`` file, which is found in the
872:term:`Build Directory`.
873
874To understand how these features work, the best reference is
875``meta/classes/core-image.bbclass``. This class lists out the available
Andrew Geissler09036742021-06-25 14:25:14 -0500876:term:`IMAGE_FEATURES` of which most map to package groups while some, such
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500877as ``debug-tweaks`` and ``read-only-rootfs``, resolve as general
878configuration settings.
879
Andrew Geissler09036742021-06-25 14:25:14 -0500880In summary, the file looks at the contents of the :term:`IMAGE_FEATURES`
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500881variable and then maps or configures the feature accordingly. Based on
882this information, the build system automatically adds the appropriate
883packages or configurations to the
884:term:`IMAGE_INSTALL` variable.
885Effectively, you are enabling extra features by extending the class or
886creating a custom class for use with specialized image ``.bb`` files.
887
Andrew Geissler09036742021-06-25 14:25:14 -0500888Use the :term:`EXTRA_IMAGE_FEATURES` variable from within your local
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500889configuration file. Using a separate area from which to enable features
890with this variable helps you avoid overwriting the features in the image
Andrew Geissler09036742021-06-25 14:25:14 -0500891recipe that are enabled with :term:`IMAGE_FEATURES`. The value of
892:term:`EXTRA_IMAGE_FEATURES` is added to :term:`IMAGE_FEATURES` within
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500893``meta/conf/bitbake.conf``.
894
895To illustrate how you can use these variables to modify your image,
896consider an example that selects the SSH server. The Yocto Project ships
897with two SSH servers you can use with your images: Dropbear and OpenSSH.
898Dropbear is a minimal SSH server appropriate for resource-constrained
899environments, while OpenSSH is a well-known standard SSH server
900implementation. By default, the ``core-image-sato`` image is configured
901to use Dropbear. The ``core-image-full-cmdline`` and ``core-image-lsb``
902images both include OpenSSH. The ``core-image-minimal`` image does not
903contain an SSH server.
904
905You can customize your image and change these defaults. Edit the
Andrew Geissler09036742021-06-25 14:25:14 -0500906:term:`IMAGE_FEATURES` variable in your recipe or use the
907:term:`EXTRA_IMAGE_FEATURES` in your ``local.conf`` file so that it
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500908configures the image you are working with to include
909``ssh-server-dropbear`` or ``ssh-server-openssh``.
910
911.. note::
912
Andrew Geissler09209ee2020-12-13 08:44:15 -0600913 See the ":ref:`ref-manual/features:image features`" section in the Yocto
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500914 Project Reference Manual for a complete list of image features that ship
915 with the Yocto Project.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500916
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500917Customizing Images Using Custom .bb Files
918-----------------------------------------
919
920You can also customize an image by creating a custom recipe that defines
921additional software as part of the image. The following example shows
Andrew Geisslerc926e172021-05-07 16:11:35 -0500922the form for the two lines you need::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500923
924 IMAGE_INSTALL = "packagegroup-core-x11-base package1 package2"
925 inherit core-image
926
927Defining the software using a custom recipe gives you total control over
928the contents of the image. It is important to use the correct names of
Andrew Geissler09036742021-06-25 14:25:14 -0500929packages in the :term:`IMAGE_INSTALL` variable. You must use the
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500930OpenEmbedded notation and not the Debian notation for the names (e.g.
931``glibc-dev`` instead of ``libc6-dev``).
932
933The other method for creating a custom image is to base it on an
934existing image. For example, if you want to create an image based on
935``core-image-sato`` but add the additional package ``strace`` to the
936image, copy the ``meta/recipes-sato/images/core-image-sato.bb`` to a new
Andrew Geisslerc926e172021-05-07 16:11:35 -0500937``.bb`` and add the following line to the end of the copy::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500938
939 IMAGE_INSTALL += "strace"
940
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500941Customizing Images Using Custom Package Groups
942----------------------------------------------
943
944For complex custom images, the best approach for customizing an image is
945to create a custom package group recipe that is used to build the image
946or images. A good example of a package group recipe is
947``meta/recipes-core/packagegroups/packagegroup-base.bb``.
948
Andrew Geissler09036742021-06-25 14:25:14 -0500949If you examine that recipe, you see that the :term:`PACKAGES` variable lists
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500950the package group packages to produce. The ``inherit packagegroup``
951statement sets appropriate default values and automatically adds
952``-dev``, ``-dbg``, and ``-ptest`` complementary packages for each
Andrew Geissler09036742021-06-25 14:25:14 -0500953package specified in the :term:`PACKAGES` statement.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500954
955.. note::
956
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500957 The ``inherit packagegroup`` line should be located near the top of the
Andrew Geissler09036742021-06-25 14:25:14 -0500958 recipe, certainly before the :term:`PACKAGES` statement.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500959
Andrew Geissler09036742021-06-25 14:25:14 -0500960For each package you specify in :term:`PACKAGES`, you can use :term:`RDEPENDS`
961and :term:`RRECOMMENDS` entries to provide a list of packages the parent
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500962task package should contain. You can see examples of these further down
963in the ``packagegroup-base.bb`` recipe.
964
965Here is a short, fabricated example showing the same basic pieces for a
966hypothetical packagegroup defined in ``packagegroup-custom.bb``, where
Andrew Geissler09036742021-06-25 14:25:14 -0500967the variable :term:`PN` is the standard way to abbreviate the reference to
Andrew Geisslerc926e172021-05-07 16:11:35 -0500968the full packagegroup name ``packagegroup-custom``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500969
970 DESCRIPTION = "My Custom Package Groups"
971
972 inherit packagegroup
973
974 PACKAGES = "\
975 ${PN}-apps \
976 ${PN}-tools \
977 "
978
979 RDEPENDS_${PN}-apps = "\
980 dropbear \
981 portmap \
982 psplash"
983
984 RDEPENDS_${PN}-tools = "\
985 oprofile \
986 oprofileui-server \
987 lttng-tools"
988
989 RRECOMMENDS_${PN}-tools = "\
990 kernel-module-oprofile"
991
992In the previous example, two package group packages are created with
993their dependencies and their recommended package dependencies listed:
994``packagegroup-custom-apps``, and ``packagegroup-custom-tools``. To
995build an image using these package group packages, you need to add
996``packagegroup-custom-apps`` and/or ``packagegroup-custom-tools`` to
Andrew Geissler09036742021-06-25 14:25:14 -0500997:term:`IMAGE_INSTALL`. For other forms of image dependencies see the other
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500998areas of this section.
999
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001000Customizing an Image Hostname
1001-----------------------------
1002
1003By default, the configured hostname (i.e. ``/etc/hostname``) in an image
1004is the same as the machine name. For example, if
1005:term:`MACHINE` equals "qemux86", the
1006configured hostname written to ``/etc/hostname`` is "qemux86".
1007
1008You can customize this name by altering the value of the "hostname"
1009variable in the ``base-files`` recipe using either an append file or a
Andrew Geisslerc926e172021-05-07 16:11:35 -05001010configuration file. Use the following in an append file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001011
1012 hostname = "myhostname"
1013
Andrew Geisslerc926e172021-05-07 16:11:35 -05001014Use the following in a configuration file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001015
1016 hostname_pn-base-files = "myhostname"
1017
1018Changing the default value of the variable "hostname" can be useful in
1019certain situations. For example, suppose you need to do extensive
1020testing on an image and you would like to easily identify the image
1021under test from existing images with typical default hostnames. In this
1022situation, you could change the default hostname to "testme", which
1023results in all the images using the name "testme". Once testing is
1024complete and you do not need to rebuild the image for test any longer,
1025you can easily reset the default hostname.
1026
1027Another point of interest is that if you unset the variable, the image
1028will have no default hostname in the filesystem. Here is an example that
Andrew Geisslerc926e172021-05-07 16:11:35 -05001029unsets the variable in a configuration file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001030
1031 hostname_pn-base-files = ""
1032
1033Having no default hostname in the filesystem is suitable for
1034environments that use dynamic hostnames such as virtual machines.
1035
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001036Writing a New Recipe
1037====================
1038
1039Recipes (``.bb`` files) are fundamental components in the Yocto Project
1040environment. Each software component built by the OpenEmbedded build
1041system requires a recipe to define the component. This section describes
1042how to create, write, and test a new recipe.
1043
1044.. note::
1045
1046 For information on variables that are useful for recipes and for
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001047 information about recipe naming issues, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001048 ":ref:`ref-manual/varlocality:recipes`" section of the Yocto Project
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001049 Reference Manual.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001050
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001051Overview
1052--------
1053
1054The following figure shows the basic process for creating a new recipe.
1055The remainder of the section provides details for the steps.
1056
1057.. image:: figures/recipe-workflow.png
1058 :align: center
1059
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001060Locate or Automatically Create a Base Recipe
1061--------------------------------------------
1062
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001063You can always write a recipe from scratch. However, there are three choices
1064that can help you quickly get started with a new recipe:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001065
1066- ``devtool add``: A command that assists in creating a recipe and an
1067 environment conducive to development.
1068
1069- ``recipetool create``: A command provided by the Yocto Project that
1070 automates creation of a base recipe based on the source files.
1071
1072- *Existing Recipes:* Location and modification of an existing recipe
1073 that is similar in function to the recipe you need.
1074
1075.. note::
1076
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001077 For information on recipe syntax, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001078 ":ref:`dev-manual/common-tasks:recipe syntax`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001079
1080Creating the Base Recipe Using ``devtool add``
1081~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1082
1083The ``devtool add`` command uses the same logic for auto-creating the
1084recipe as ``recipetool create``, which is listed below. Additionally,
1085however, ``devtool add`` sets up an environment that makes it easy for
1086you to patch the source and to make changes to the recipe as is often
1087necessary when adding a recipe to build a new piece of software to be
1088included in a build.
1089
1090You can find a complete description of the ``devtool add`` command in
Andrew Geissler09209ee2020-12-13 08:44:15 -06001091the ":ref:`sdk-manual/extensible:a closer look at \`\`devtool add\`\``" section
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001092in the Yocto Project Application Development and the Extensible Software
1093Development Kit (eSDK) manual.
1094
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001095Creating the Base Recipe Using ``recipetool create``
1096~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1097
1098``recipetool create`` automates creation of a base recipe given a set of
1099source code files. As long as you can extract or point to the source
1100files, the tool will construct a recipe and automatically configure all
1101pre-build information into the recipe. For example, suppose you have an
1102application that builds using Autotools. Creating the base recipe using
1103``recipetool`` results in a recipe that has the pre-build dependencies,
1104license requirements, and checksums configured.
1105
1106To run the tool, you just need to be in your
1107:term:`Build Directory` and have sourced the
1108build environment setup script (i.e.
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001109:ref:`structure-core-script`).
Andrew Geisslerc926e172021-05-07 16:11:35 -05001110To get help on the tool, use the following command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001111
1112 $ recipetool -h
1113 NOTE: Starting bitbake server...
1114 usage: recipetool [-d] [-q] [--color COLOR] [-h] <subcommand> ...
1115
1116 OpenEmbedded recipe tool
1117
1118 options:
1119 -d, --debug Enable debug output
1120 -q, --quiet Print only errors
1121 --color COLOR Colorize output (where COLOR is auto, always, never)
1122 -h, --help show this help message and exit
1123
1124 subcommands:
1125 create Create a new recipe
1126 newappend Create a bbappend for the specified target in the specified
1127 layer
1128 setvar Set a variable within a recipe
1129 appendfile Create/update a bbappend to replace a target file
1130 appendsrcfiles Create/update a bbappend to add or replace source files
1131 appendsrcfile Create/update a bbappend to add or replace a source file
1132 Use recipetool <subcommand> --help to get help on a specific command
1133
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001134Running ``recipetool create -o OUTFILE`` creates the base recipe and
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001135locates it properly in the layer that contains your source files.
1136Following are some syntax examples:
1137
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001138 - Use this syntax to generate a recipe based on source. Once generated,
Andrew Geisslerc926e172021-05-07 16:11:35 -05001139 the recipe resides in the existing source code layer::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001140
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001141 recipetool create -o OUTFILE source
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001142
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001143 - Use this syntax to generate a recipe using code that
1144 you extract from source. The extracted code is placed in its own layer
Andrew Geissler09036742021-06-25 14:25:14 -05001145 defined by :term:`EXTERNALSRC`.
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001146 ::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001147
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001148 recipetool create -o OUTFILE -x EXTERNALSRC source
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001149
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001150 - Use this syntax to generate a recipe based on source. The options
1151 direct ``recipetool`` to generate debugging information. Once generated,
Andrew Geisslerc926e172021-05-07 16:11:35 -05001152 the recipe resides in the existing source code layer::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001153
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001154 recipetool create -d -o OUTFILE source
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001155
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001156Locating and Using a Similar Recipe
1157~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1158
1159Before writing a recipe from scratch, it is often useful to discover
1160whether someone else has already written one that meets (or comes close
1161to meeting) your needs. The Yocto Project and OpenEmbedded communities
1162maintain many recipes that might be candidates for what you are doing.
Andrew Geisslerd1e89492021-02-12 15:35:20 -06001163You can find a good central index of these recipes in the
1164:oe_layerindex:`OpenEmbedded Layer Index <>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001165
1166Working from an existing recipe or a skeleton recipe is the best way to
1167get started. Here are some points on both methods:
1168
1169- *Locate and modify a recipe that is close to what you want to do:*
1170 This method works when you are familiar with the current recipe
1171 space. The method does not work so well for those new to the Yocto
1172 Project or writing recipes.
1173
1174 Some risks associated with this method are using a recipe that has
1175 areas totally unrelated to what you are trying to accomplish with
1176 your recipe, not recognizing areas of the recipe that you might have
1177 to add from scratch, and so forth. All these risks stem from
1178 unfamiliarity with the existing recipe space.
1179
1180- *Use and modify the following skeleton recipe:* If for some reason
1181 you do not want to use ``recipetool`` and you cannot find an existing
1182 recipe that is close to meeting your needs, you can use the following
1183 structure to provide the fundamental areas of a new recipe.
1184 ::
1185
1186 DESCRIPTION = ""
1187 HOMEPAGE = ""
1188 LICENSE = ""
1189 SECTION = ""
1190 DEPENDS = ""
1191 LIC_FILES_CHKSUM = ""
1192
1193 SRC_URI = ""
1194
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001195Storing and Naming the Recipe
1196-----------------------------
1197
1198Once you have your base recipe, you should put it in your own layer and
1199name it appropriately. Locating it correctly ensures that the
1200OpenEmbedded build system can find it when you use BitBake to process
1201the recipe.
1202
1203- *Storing Your Recipe:* The OpenEmbedded build system locates your
1204 recipe through the layer's ``conf/layer.conf`` file and the
1205 :term:`BBFILES` variable. This
1206 variable sets up a path from which the build system can locate
Andrew Geisslerc926e172021-05-07 16:11:35 -05001207 recipes. Here is the typical use::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001208
1209 BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
1210 ${LAYERDIR}/recipes-*/*/*.bbappend"
1211
1212 Consequently, you need to be sure you locate your new recipe inside
1213 your layer such that it can be found.
1214
1215 You can find more information on how layers are structured in the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001216 ":ref:`dev-manual/common-tasks:understanding and creating layers`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001217
1218- *Naming Your Recipe:* When you name your recipe, you need to follow
Andrew Geisslerc926e172021-05-07 16:11:35 -05001219 this naming convention::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001220
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001221 basename_version.bb
1222
1223 Use lower-cased characters and do not include the reserved suffixes
1224 ``-native``, ``-cross``, ``-initial``, or ``-dev`` casually (i.e. do not use
1225 them as part of your recipe name unless the string applies). Here are some
1226 examples:
1227
1228 .. code-block:: none
1229
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001230 cups_1.7.0.bb
1231 gawk_4.0.2.bb
1232 irssi_0.8.16-rc1.bb
1233
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001234Running a Build on the Recipe
1235-----------------------------
1236
1237Creating a new recipe is usually an iterative process that requires
1238using BitBake to process the recipe multiple times in order to
1239progressively discover and add information to the recipe file.
1240
1241Assuming you have sourced the build environment setup script (i.e.
1242:ref:`structure-core-script`) and you are in
1243the :term:`Build Directory`, use
1244BitBake to process your recipe. All you need to provide is the
Andrew Geisslerc926e172021-05-07 16:11:35 -05001245``basename`` of the recipe as described in the previous section::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001246
1247 $ bitbake basename
1248
1249During the build, the OpenEmbedded build system creates a temporary work
1250directory for each recipe
1251(``${``\ :term:`WORKDIR`\ ``}``)
1252where it keeps extracted source files, log files, intermediate
1253compilation and packaging files, and so forth.
1254
1255The path to the per-recipe temporary work directory depends on the
1256context in which it is being built. The quickest way to find this path
Andrew Geisslerc926e172021-05-07 16:11:35 -05001257is to have BitBake return it by running the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001258
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001259 $ bitbake -e basename | grep ^WORKDIR=
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001260
1261As an example, assume a Source Directory
1262top-level folder named ``poky``, a default Build Directory at
1263``poky/build``, and a ``qemux86-poky-linux`` machine target system.
1264Furthermore, suppose your recipe is named ``foo_1.3.0.bb``. In this
1265case, the work directory the build system uses to build the package
Andrew Geisslerc926e172021-05-07 16:11:35 -05001266would be as follows::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001267
1268 poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0
1269
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001270Inside this directory you can find sub-directories such as ``image``,
1271``packages-split``, and ``temp``. After the build, you can examine these
1272to determine how well the build went.
1273
1274.. note::
1275
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001276 You can find log files for each task in the recipe's ``temp``
1277 directory (e.g. ``poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0/temp``).
1278 Log files are named ``log.taskname`` (e.g. ``log.do_configure``,
1279 ``log.do_fetch``, and ``log.do_compile``).
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001280
1281You can find more information about the build process in
Andrew Geissler09209ee2020-12-13 08:44:15 -06001282":doc:`/overview-manual/development-environment`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001283chapter of the Yocto Project Overview and Concepts Manual.
1284
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001285Fetching Code
1286-------------
1287
1288The first thing your recipe must do is specify how to fetch the source
1289files. Fetching is controlled mainly through the
1290:term:`SRC_URI` variable. Your recipe
Andrew Geissler09036742021-06-25 14:25:14 -05001291must have a :term:`SRC_URI` variable that points to where the source is
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001292located. For a graphical representation of source locations, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001293":ref:`overview-manual/concepts:sources`" section in
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001294the Yocto Project Overview and Concepts Manual.
1295
1296The :ref:`ref-tasks-fetch` task uses
Andrew Geissler09036742021-06-25 14:25:14 -05001297the prefix of each entry in the :term:`SRC_URI` variable value to determine
Andrew Geissler09209ee2020-12-13 08:44:15 -06001298which :ref:`fetcher <bitbake:bitbake-user-manual/bitbake-user-manual-fetching:fetchers>` to use to get your
Andrew Geissler09036742021-06-25 14:25:14 -05001299source files. It is the :term:`SRC_URI` variable that triggers the fetcher.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001300The :ref:`ref-tasks-patch` task uses
1301the variable after source is fetched to apply patches. The OpenEmbedded
1302build system uses
1303:term:`FILESOVERRIDES` for
Andrew Geissler09036742021-06-25 14:25:14 -05001304scanning directory locations for local files in :term:`SRC_URI`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001305
Andrew Geissler09036742021-06-25 14:25:14 -05001306The :term:`SRC_URI` variable in your recipe must define each unique location
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001307for your source files. It is good practice to not hard-code version
Andrew Geissler5f350902021-07-23 13:09:54 -04001308numbers in a URL used in :term:`SRC_URI`. Rather than hard-code these
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001309values, use ``${``\ :term:`PV`\ ``}``,
1310which causes the fetch process to use the version specified in the
1311recipe filename. Specifying the version in this manner means that
1312upgrading the recipe to a future version is as simple as renaming the
1313recipe to match the new version.
1314
1315Here is a simple example from the
1316``meta/recipes-devtools/strace/strace_5.5.bb`` recipe where the source
1317comes from a single tarball. Notice the use of the
Andrew Geisslerc926e172021-05-07 16:11:35 -05001318:term:`PV` variable::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001319
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001320 SRC_URI = "https://strace.io/files/${PV}/strace-${PV}.tar.xz \
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001321
Andrew Geissler09036742021-06-25 14:25:14 -05001322Files mentioned in :term:`SRC_URI` whose names end in a typical archive
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001323extension (e.g. ``.tar``, ``.tar.gz``, ``.tar.bz2``, ``.zip``, and so
1324forth), are automatically extracted during the
1325:ref:`ref-tasks-unpack` task. For
1326another example that specifies these types of files, see the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001327":ref:`dev-manual/common-tasks:autotooled package`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001328
1329Another way of specifying source is from an SCM. For Git repositories,
1330you must specify :term:`SRCREV` and
1331you should specify :term:`PV` to include
1332the revision with :term:`SRCPV`. Here
1333is an example from the recipe
Andrew Geisslerc926e172021-05-07 16:11:35 -05001334``meta/recipes-kernel/blktrace/blktrace_git.bb``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001335
1336 SRCREV = "d6918c8832793b4205ed3bfede78c2f915c23385"
1337
1338 PR = "r6"
1339 PV = "1.0.5+git${SRCPV}"
1340
1341 SRC_URI = "git://git.kernel.dk/blktrace.git \
1342 file://ldflags.patch"
1343
Andrew Geissler09036742021-06-25 14:25:14 -05001344If your :term:`SRC_URI` statement includes URLs pointing to individual files
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001345fetched from a remote server other than a version control system,
1346BitBake attempts to verify the files against checksums defined in your
1347recipe to ensure they have not been tampered with or otherwise modified
1348since the recipe was written. Two checksums are used:
1349``SRC_URI[md5sum]`` and ``SRC_URI[sha256sum]``.
1350
Andrew Geissler09036742021-06-25 14:25:14 -05001351If your :term:`SRC_URI` variable points to more than a single URL (excluding
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001352SCM URLs), you need to provide the ``md5`` and ``sha256`` checksums for
1353each URL. For these cases, you provide a name for each URL as part of
Andrew Geissler09036742021-06-25 14:25:14 -05001354the :term:`SRC_URI` and then reference that name in the subsequent checksum
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001355statements. Here is an example combining lines from the files
Andrew Geisslerc926e172021-05-07 16:11:35 -05001356``git.inc`` and ``git_2.24.1.bb``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001357
1358 SRC_URI = "${KERNELORG_MIRROR}/software/scm/git/git-${PV}.tar.gz;name=tarball \
1359 ${KERNELORG_MIRROR}/software/scm/git/git-manpages-${PV}.tar.gz;name=manpages"
1360
1361 SRC_URI[tarball.md5sum] = "166bde96adbbc11c8843d4f8f4f9811b"
1362 SRC_URI[tarball.sha256sum] = "ad5334956301c86841eb1e5b1bb20884a6bad89a10a6762c958220c7cf64da02"
1363 SRC_URI[manpages.md5sum] = "31c2272a8979022497ba3d4202df145d"
1364 SRC_URI[manpages.sha256sum] = "9a7ae3a093bea39770eb96ca3e5b40bff7af0b9f6123f089d7821d0e5b8e1230"
1365
1366Proper values for ``md5`` and ``sha256`` checksums might be available
1367with other signatures on the download page for the upstream source (e.g.
1368``md5``, ``sha1``, ``sha256``, ``GPG``, and so forth). Because the
1369OpenEmbedded build system only deals with ``sha256sum`` and ``md5sum``,
1370you should verify all the signatures you find by hand.
1371
Andrew Geissler09036742021-06-25 14:25:14 -05001372If no :term:`SRC_URI` checksums are specified when you attempt to build the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001373recipe, or you provide an incorrect checksum, the build will produce an
1374error for each missing or incorrect checksum. As part of the error
1375message, the build system provides the checksum string corresponding to
1376the fetched file. Once you have the correct checksums, you can copy and
1377paste them into your recipe and then run the build again to continue.
1378
1379.. note::
1380
1381 As mentioned, if the upstream source provides signatures for
1382 verifying the downloaded source code, you should verify those
1383 manually before setting the checksum values in the recipe and
1384 continuing with the build.
1385
1386This final example is a bit more complicated and is from the
1387``meta/recipes-sato/rxvt-unicode/rxvt-unicode_9.20.bb`` recipe. The
Andrew Geissler09036742021-06-25 14:25:14 -05001388example's :term:`SRC_URI` statement identifies multiple files as the source
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001389files for the recipe: a tarball, a patch file, a desktop file, and an
1390icon.
1391::
1392
1393 SRC_URI = "http://dist.schmorp.de/rxvt-unicode/Attic/rxvt-unicode-${PV}.tar.bz2 \
1394 file://xwc.patch \
1395 file://rxvt.desktop \
1396 file://rxvt.png"
1397
1398When you specify local files using the ``file://`` URI protocol, the
1399build system fetches files from the local machine. The path is relative
1400to the :term:`FILESPATH` variable
1401and searches specific directories in a certain order:
1402``${``\ :term:`BP`\ ``}``,
1403``${``\ :term:`BPN`\ ``}``, and
1404``files``. The directories are assumed to be subdirectories of the
1405directory in which the recipe or append file resides. For another
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001406example that specifies these types of files, see the
1407":ref:`dev-manual/common-tasks:single .c file package (hello world!)`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001408
1409The previous example also specifies a patch file. Patch files are files
1410whose names usually end in ``.patch`` or ``.diff`` but can end with
1411compressed suffixes such as ``diff.gz`` and ``patch.bz2``, for example.
1412The build system automatically applies patches as described in the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001413":ref:`dev-manual/common-tasks:patching code`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001414
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001415Unpacking Code
1416--------------
1417
1418During the build, the
1419:ref:`ref-tasks-unpack` task unpacks
1420the source with ``${``\ :term:`S`\ ``}``
1421pointing to where it is unpacked.
1422
1423If you are fetching your source files from an upstream source archived
1424tarball and the tarball's internal structure matches the common
1425convention of a top-level subdirectory named
1426``${``\ :term:`BPN`\ ``}-${``\ :term:`PV`\ ``}``,
Andrew Geissler09036742021-06-25 14:25:14 -05001427then you do not need to set :term:`S`. However, if :term:`SRC_URI` specifies to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001428fetch source from an archive that does not use this convention, or from
Andrew Geissler09036742021-06-25 14:25:14 -05001429an SCM like Git or Subversion, your recipe needs to define :term:`S`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001430
1431If processing your recipe using BitBake successfully unpacks the source
1432files, you need to be sure that the directory pointed to by ``${S}``
1433matches the structure of the source.
1434
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001435Patching Code
1436-------------
1437
1438Sometimes it is necessary to patch code after it has been fetched. Any
Andrew Geissler09036742021-06-25 14:25:14 -05001439files mentioned in :term:`SRC_URI` whose names end in ``.patch`` or
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001440``.diff`` or compressed versions of these suffixes (e.g. ``diff.gz`` are
1441treated as patches. The
1442:ref:`ref-tasks-patch` task
1443automatically applies these patches.
1444
1445The build system should be able to apply patches with the "-p1" option
1446(i.e. one directory level in the path will be stripped off). If your
1447patch needs to have more directory levels stripped off, specify the
Andrew Geissler09036742021-06-25 14:25:14 -05001448number of levels using the "striplevel" option in the :term:`SRC_URI` entry
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001449for the patch. Alternatively, if your patch needs to be applied in a
1450specific subdirectory that is not specified in the patch file, use the
1451"patchdir" option in the entry.
1452
1453As with all local files referenced in
1454:term:`SRC_URI` using ``file://``,
1455you should place patch files in a directory next to the recipe either
1456named the same as the base name of the recipe
1457(:term:`BP` and
1458:term:`BPN`) or "files".
1459
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001460Licensing
1461---------
1462
1463Your recipe needs to have both the
1464:term:`LICENSE` and
1465:term:`LIC_FILES_CHKSUM`
1466variables:
1467
Andrew Geissler09036742021-06-25 14:25:14 -05001468- :term:`LICENSE`: This variable specifies the license for the software.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001469 If you do not know the license under which the software you are
1470 building is distributed, you should go to the source code and look
1471 for that information. Typical files containing this information
Andrew Geissler09036742021-06-25 14:25:14 -05001472 include ``COPYING``, :term:`LICENSE`, and ``README`` files. You could
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001473 also find the information near the top of a source file. For example,
1474 given a piece of software licensed under the GNU General Public
Andrew Geissler09036742021-06-25 14:25:14 -05001475 License version 2, you would set :term:`LICENSE` as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001476
1477 LICENSE = "GPLv2"
1478
Andrew Geissler09036742021-06-25 14:25:14 -05001479 The licenses you specify within :term:`LICENSE` can have any name as long
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001480 as you do not use spaces, since spaces are used as separators between
1481 license names. For standard licenses, use the names of the files in
Andrew Geissler09036742021-06-25 14:25:14 -05001482 ``meta/files/common-licenses/`` or the :term:`SPDXLICENSEMAP` flag names
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001483 defined in ``meta/conf/licenses.conf``.
1484
Andrew Geissler09036742021-06-25 14:25:14 -05001485- :term:`LIC_FILES_CHKSUM`: The OpenEmbedded build system uses this
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001486 variable to make sure the license text has not changed. If it has,
1487 the build produces an error and it affords you the chance to figure
1488 it out and correct the problem.
1489
1490 You need to specify all applicable licensing files for the software.
1491 At the end of the configuration step, the build process will compare
1492 the checksums of the files to be sure the text has not changed. Any
1493 differences result in an error with the message containing the
1494 current checksum. For more explanation and examples of how to set the
Andrew Geissler09036742021-06-25 14:25:14 -05001495 :term:`LIC_FILES_CHKSUM` variable, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001496 ":ref:`dev-manual/common-tasks:tracking license changes`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001497
1498 To determine the correct checksum string, you can list the
Andrew Geissler09036742021-06-25 14:25:14 -05001499 appropriate files in the :term:`LIC_FILES_CHKSUM` variable with incorrect
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001500 md5 strings, attempt to build the software, and then note the
1501 resulting error messages that will report the correct md5 strings.
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001502 See the ":ref:`dev-manual/common-tasks:fetching code`" section for
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001503 additional information.
1504
Andrew Geisslerc926e172021-05-07 16:11:35 -05001505 Here is an example that assumes the software has a ``COPYING`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001506
1507 LIC_FILES_CHKSUM = "file://COPYING;md5=xxx"
1508
1509 When you try to build the
1510 software, the build system will produce an error and give you the
1511 correct string that you can substitute into the recipe file for a
1512 subsequent build.
1513
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001514Dependencies
1515------------
1516
1517Most software packages have a short list of other packages that they
1518require, which are called dependencies. These dependencies fall into two
1519main categories: build-time dependencies, which are required when the
1520software is built; and runtime dependencies, which are required to be
1521installed on the target in order for the software to run.
1522
1523Within a recipe, you specify build-time dependencies using the
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001524:term:`DEPENDS` variable. Although there are nuances,
Andrew Geissler09036742021-06-25 14:25:14 -05001525items specified in :term:`DEPENDS` should be names of other
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001526recipes. It is important that you specify all build-time dependencies
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001527explicitly.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001528
1529Another consideration is that configure scripts might automatically
1530check for optional dependencies and enable corresponding functionality
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001531if those dependencies are found. If you wish to make a recipe that is
1532more generally useful (e.g. publish the recipe in a layer for others to
1533use), instead of hard-disabling the functionality, you can use the
1534:term:`PACKAGECONFIG` variable to allow functionality and the
1535corresponding dependencies to be enabled and disabled easily by other
1536users of the recipe.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001537
1538Similar to build-time dependencies, you specify runtime dependencies
1539through a variable -
1540:term:`RDEPENDS`, which is
1541package-specific. All variables that are package-specific need to have
1542the name of the package added to the end as an override. Since the main
1543package for a recipe has the same name as the recipe, and the recipe's
1544name can be found through the
1545``${``\ :term:`PN`\ ``}`` variable, then
1546you specify the dependencies for the main package by setting
1547``RDEPENDS_${PN}``. If the package were named ``${PN}-tools``, then you
1548would set ``RDEPENDS_${PN}-tools``, and so forth.
1549
1550Some runtime dependencies will be set automatically at packaging time.
1551These dependencies include any shared library dependencies (i.e. if a
1552package "example" contains "libexample" and another package "mypackage"
1553contains a binary that links to "libexample" then the OpenEmbedded build
1554system will automatically add a runtime dependency to "mypackage" on
1555"example"). See the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001556":ref:`overview-manual/concepts:automatically added runtime dependencies`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001557section in the Yocto Project Overview and Concepts Manual for further
1558details.
1559
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001560Configuring the Recipe
1561----------------------
1562
1563Most software provides some means of setting build-time configuration
1564options before compilation. Typically, setting these options is
1565accomplished by running a configure script with options, or by modifying
1566a build configuration file.
1567
1568.. note::
1569
1570 As of Yocto Project Release 1.7, some of the core recipes that
1571 package binary configuration scripts now disable the scripts due to
1572 the scripts previously requiring error-prone path substitution. The
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001573 OpenEmbedded build system uses ``pkg-config`` now, which is much more
1574 robust. You can find a list of the ``*-config`` scripts that are disabled
1575 in the ":ref:`migration-1.7-binary-configuration-scripts-disabled`" section
1576 in the Yocto Project Reference Manual.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001577
1578A major part of build-time configuration is about checking for
1579build-time dependencies and possibly enabling optional functionality as
1580a result. You need to specify any build-time dependencies for the
1581software you are building in your recipe's
1582:term:`DEPENDS` value, in terms of
1583other recipes that satisfy those dependencies. You can often find
1584build-time or runtime dependencies described in the software's
1585documentation.
1586
1587The following list provides configuration items of note based on how
1588your software is built:
1589
1590- *Autotools:* If your source files have a ``configure.ac`` file, then
1591 your software is built using Autotools. If this is the case, you just
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001592 need to modify the configuration.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001593
1594 When using Autotools, your recipe needs to inherit the
1595 :ref:`autotools <ref-classes-autotools>` class
1596 and your recipe does not have to contain a
1597 :ref:`ref-tasks-configure` task.
1598 However, you might still want to make some adjustments. For example,
1599 you can set
1600 :term:`EXTRA_OECONF` or
1601 :term:`PACKAGECONFIG_CONFARGS`
1602 to pass any needed configure options that are specific to the recipe.
1603
1604- *CMake:* If your source files have a ``CMakeLists.txt`` file, then
1605 your software is built using CMake. If this is the case, you just
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001606 need to modify the configuration.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001607
1608 When you use CMake, your recipe needs to inherit the
1609 :ref:`cmake <ref-classes-cmake>` class and your
1610 recipe does not have to contain a
1611 :ref:`ref-tasks-configure` task.
1612 You can make some adjustments by setting
1613 :term:`EXTRA_OECMAKE` to
1614 pass any needed configure options that are specific to the recipe.
1615
1616 .. note::
1617
1618 If you need to install one or more custom CMake toolchain files
1619 that are supplied by the application you are building, install the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001620 files to ``${D}${datadir}/cmake/Modules`` during ``do_install``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001621
1622- *Other:* If your source files do not have a ``configure.ac`` or
1623 ``CMakeLists.txt`` file, then your software is built using some
1624 method other than Autotools or CMake. If this is the case, you
1625 normally need to provide a
1626 :ref:`ref-tasks-configure` task
1627 in your recipe unless, of course, there is nothing to configure.
1628
1629 Even if your software is not being built by Autotools or CMake, you
1630 still might not need to deal with any configuration issues. You need
1631 to determine if configuration is even a required step. You might need
1632 to modify a Makefile or some configuration file used for the build to
1633 specify necessary build options. Or, perhaps you might need to run a
1634 provided, custom configure script with the appropriate options.
1635
1636 For the case involving a custom configure script, you would run
1637 ``./configure --help`` and look for the options you need to set.
1638
1639Once configuration succeeds, it is always good practice to look at the
1640``log.do_configure`` file to ensure that the appropriate options have
1641been enabled and no additional build-time dependencies need to be added
Andrew Geissler09036742021-06-25 14:25:14 -05001642to :term:`DEPENDS`. For example, if the configure script reports that it
1643found something not mentioned in :term:`DEPENDS`, or that it did not find
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001644something that it needed for some desired optional functionality, then
Andrew Geissler09036742021-06-25 14:25:14 -05001645you would need to add those to :term:`DEPENDS`. Looking at the log might
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001646also reveal items being checked for, enabled, or both that you do not
Andrew Geissler09036742021-06-25 14:25:14 -05001647want, or items not being found that are in :term:`DEPENDS`, in which case
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001648you would need to look at passing extra options to the configure script
1649as needed. For reference information on configure options specific to
1650the software you are building, you can consult the output of the
1651``./configure --help`` command within ``${S}`` or consult the software's
1652upstream documentation.
1653
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001654Using Headers to Interface with Devices
1655---------------------------------------
1656
1657If your recipe builds an application that needs to communicate with some
1658device or needs an API into a custom kernel, you will need to provide
1659appropriate header files. Under no circumstances should you ever modify
1660the existing
1661``meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc`` file.
1662These headers are used to build ``libc`` and must not be compromised
1663with custom or machine-specific header information. If you customize
1664``libc`` through modified headers all other applications that use
1665``libc`` thus become affected.
1666
1667.. note::
1668
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001669 Never copy and customize the ``libc`` header file (i.e.
1670 ``meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc``).
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001671
1672The correct way to interface to a device or custom kernel is to use a
1673separate package that provides the additional headers for the driver or
1674other unique interfaces. When doing so, your application also becomes
1675responsible for creating a dependency on that specific provider.
1676
1677Consider the following:
1678
1679- Never modify ``linux-libc-headers.inc``. Consider that file to be
1680 part of the ``libc`` system, and not something you use to access the
1681 kernel directly. You should access ``libc`` through specific ``libc``
1682 calls.
1683
1684- Applications that must talk directly to devices should either provide
1685 necessary headers themselves, or establish a dependency on a special
1686 headers package that is specific to that driver.
1687
1688For example, suppose you want to modify an existing header that adds I/O
1689control or network support. If the modifications are used by a small
1690number programs, providing a unique version of a header is easy and has
1691little impact. When doing so, bear in mind the guidelines in the
1692previous list.
1693
1694.. note::
1695
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001696 If for some reason your changes need to modify the behavior of the ``libc``,
1697 and subsequently all other applications on the system, use a ``.bbappend``
1698 to modify the ``linux-kernel-headers.inc`` file. However, take care to not
1699 make the changes machine specific.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001700
1701Consider a case where your kernel is older and you need an older
1702``libc`` ABI. The headers installed by your recipe should still be a
1703standard mainline kernel, not your own custom one.
1704
1705When you use custom kernel headers you need to get them from
1706:term:`STAGING_KERNEL_DIR`,
1707which is the directory with kernel headers that are required to build
Andrew Geisslerc926e172021-05-07 16:11:35 -05001708out-of-tree modules. Your recipe will also need the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001709
1710 do_configure[depends] += "virtual/kernel:do_shared_workdir"
1711
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001712Compilation
1713-----------
1714
1715During a build, the ``do_compile`` task happens after source is fetched,
1716unpacked, and configured. If the recipe passes through ``do_compile``
1717successfully, nothing needs to be done.
1718
1719However, if the compile step fails, you need to diagnose the failure.
1720Here are some common issues that cause failures.
1721
1722.. note::
1723
1724 For cases where improper paths are detected for configuration files
1725 or for when libraries/headers cannot be found, be sure you are using
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001726 the more robust ``pkg-config``. See the note in section
Andrew Geissler09209ee2020-12-13 08:44:15 -06001727 ":ref:`dev-manual/common-tasks:Configuring the Recipe`" for additional information.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001728
1729- *Parallel build failures:* These failures manifest themselves as
1730 intermittent errors, or errors reporting that a file or directory
1731 that should be created by some other part of the build process could
1732 not be found. This type of failure can occur even if, upon
1733 inspection, the file or directory does exist after the build has
1734 failed, because that part of the build process happened in the wrong
1735 order.
1736
1737 To fix the problem, you need to either satisfy the missing dependency
1738 in the Makefile or whatever script produced the Makefile, or (as a
Andrew Geisslerc926e172021-05-07 16:11:35 -05001739 workaround) set :term:`PARALLEL_MAKE` to an empty string::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001740
1741 PARALLEL_MAKE = ""
1742
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001743 For information on parallel Makefile issues, see the
1744 ":ref:`dev-manual/common-tasks:debugging parallel make races`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001745
1746- *Improper host path usage:* This failure applies to recipes building
1747 for the target or ``nativesdk`` only. The failure occurs when the
1748 compilation process uses improper headers, libraries, or other files
1749 from the host system when cross-compiling for the target.
1750
1751 To fix the problem, examine the ``log.do_compile`` file to identify
1752 the host paths being used (e.g. ``/usr/include``, ``/usr/lib``, and
1753 so forth) and then either add configure options, apply a patch, or do
1754 both.
1755
1756- *Failure to find required libraries/headers:* If a build-time
1757 dependency is missing because it has not been declared in
1758 :term:`DEPENDS`, or because the
1759 dependency exists but the path used by the build process to find the
1760 file is incorrect and the configure step did not detect it, the
1761 compilation process could fail. For either of these failures, the
1762 compilation process notes that files could not be found. In these
1763 cases, you need to go back and add additional options to the
1764 configure script as well as possibly add additional build-time
Andrew Geissler09036742021-06-25 14:25:14 -05001765 dependencies to :term:`DEPENDS`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001766
1767 Occasionally, it is necessary to apply a patch to the source to
1768 ensure the correct paths are used. If you need to specify paths to
1769 find files staged into the sysroot from other recipes, use the
1770 variables that the OpenEmbedded build system provides (e.g.
Andrew Geissler09036742021-06-25 14:25:14 -05001771 :term:`STAGING_BINDIR`, :term:`STAGING_INCDIR`, :term:`STAGING_DATADIR`, and so
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001772 forth).
1773
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001774Installing
1775----------
1776
1777During ``do_install``, the task copies the built files along with their
1778hierarchy to locations that would mirror their locations on the target
1779device. The installation process copies files from the
1780``${``\ :term:`S`\ ``}``,
1781``${``\ :term:`B`\ ``}``, and
1782``${``\ :term:`WORKDIR`\ ``}``
1783directories to the ``${``\ :term:`D`\ ``}``
1784directory to create the structure as it should appear on the target
1785system.
1786
1787How your software is built affects what you must do to be sure your
1788software is installed correctly. The following list describes what you
1789must do for installation depending on the type of build system used by
1790the software being built:
1791
1792- *Autotools and CMake:* If the software your recipe is building uses
1793 Autotools or CMake, the OpenEmbedded build system understands how to
1794 install the software. Consequently, you do not have to have a
1795 ``do_install`` task as part of your recipe. You just need to make
1796 sure the install portion of the build completes with no issues.
1797 However, if you wish to install additional files not already being
1798 installed by ``make install``, you should do this using a
1799 ``do_install_append`` function using the install command as described
1800 in the "Manual" bulleted item later in this list.
1801
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001802- *Other (using* ``make install``\ *)*: You need to define a ``do_install``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001803 function in your recipe. The function should call
1804 ``oe_runmake install`` and will likely need to pass in the
1805 destination directory as well. How you pass that path is dependent on
1806 how the ``Makefile`` being run is written (e.g. ``DESTDIR=${D}``,
1807 ``PREFIX=${D}``, ``INSTALLROOT=${D}``, and so forth).
1808
1809 For an example recipe using ``make install``, see the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001810 ":ref:`dev-manual/common-tasks:makefile-based package`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001811
1812- *Manual:* You need to define a ``do_install`` function in your
1813 recipe. The function must first use ``install -d`` to create the
1814 directories under
1815 ``${``\ :term:`D`\ ``}``. Once the
1816 directories exist, your function can use ``install`` to manually
1817 install the built software into the directories.
1818
1819 You can find more information on ``install`` at
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001820 https://www.gnu.org/software/coreutils/manual/html_node/install-invocation.html.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001821
1822For the scenarios that do not use Autotools or CMake, you need to track
1823the installation and diagnose and fix any issues until everything
1824installs correctly. You need to look in the default location of
1825``${D}``, which is ``${WORKDIR}/image``, to be sure your files have been
1826installed correctly.
1827
1828.. note::
1829
1830 - During the installation process, you might need to modify some of
1831 the installed files to suit the target layout. For example, you
1832 might need to replace hard-coded paths in an initscript with
1833 values of variables provided by the build system, such as
1834 replacing ``/usr/bin/`` with ``${bindir}``. If you do perform such
1835 modifications during ``do_install``, be sure to modify the
1836 destination file after copying rather than before copying.
1837 Modifying after copying ensures that the build system can
1838 re-execute ``do_install`` if needed.
1839
1840 - ``oe_runmake install``, which can be run directly or can be run
1841 indirectly by the
1842 :ref:`autotools <ref-classes-autotools>` and
1843 :ref:`cmake <ref-classes-cmake>` classes,
1844 runs ``make install`` in parallel. Sometimes, a Makefile can have
1845 missing dependencies between targets that can result in race
1846 conditions. If you experience intermittent failures during
1847 ``do_install``, you might be able to work around them by disabling
Andrew Geisslerc926e172021-05-07 16:11:35 -05001848 parallel Makefile installs by adding the following to the recipe::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001849
1850 PARALLEL_MAKEINST = ""
1851
1852 See :term:`PARALLEL_MAKEINST` for additional information.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001853
1854 - If you need to install one or more custom CMake toolchain files
1855 that are supplied by the application you are building, install the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001856 files to ``${D}${datadir}/cmake/Modules`` during
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001857 :ref:`ref-tasks-install`.
1858
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001859Enabling System Services
1860------------------------
1861
1862If you want to install a service, which is a process that usually starts
1863on boot and runs in the background, then you must include some
1864additional definitions in your recipe.
1865
1866If you are adding services and the service initialization script or the
1867service file itself is not installed, you must provide for that
1868installation in your recipe using a ``do_install_append`` function. If
1869your recipe already has a ``do_install`` function, update the function
1870near its end rather than adding an additional ``do_install_append``
1871function.
1872
1873When you create the installation for your services, you need to
1874accomplish what is normally done by ``make install``. In other words,
1875make sure your installation arranges the output similar to how it is
1876arranged on the target system.
1877
1878The OpenEmbedded build system provides support for starting services two
1879different ways:
1880
1881- *SysVinit:* SysVinit is a system and service manager that manages the
1882 init system used to control the very basic functions of your system.
1883 The init program is the first program started by the Linux kernel
1884 when the system boots. Init then controls the startup, running and
1885 shutdown of all other programs.
1886
1887 To enable a service using SysVinit, your recipe needs to inherit the
1888 :ref:`update-rc.d <ref-classes-update-rc.d>`
1889 class. The class helps facilitate safely installing the package on
1890 the target.
1891
1892 You will need to set the
1893 :term:`INITSCRIPT_PACKAGES`,
1894 :term:`INITSCRIPT_NAME`,
1895 and
1896 :term:`INITSCRIPT_PARAMS`
1897 variables within your recipe.
1898
1899- *systemd:* System Management Daemon (systemd) was designed to replace
1900 SysVinit and to provide enhanced management of services. For more
1901 information on systemd, see the systemd homepage at
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001902 https://freedesktop.org/wiki/Software/systemd/.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001903
1904 To enable a service using systemd, your recipe needs to inherit the
1905 :ref:`systemd <ref-classes-systemd>` class. See
1906 the ``systemd.bbclass`` file located in your :term:`Source Directory`
1907 section for
1908 more information.
1909
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001910Packaging
1911---------
1912
1913Successful packaging is a combination of automated processes performed
1914by the OpenEmbedded build system and some specific steps you need to
1915take. The following list describes the process:
1916
1917- *Splitting Files*: The ``do_package`` task splits the files produced
1918 by the recipe into logical components. Even software that produces a
1919 single binary might still have debug symbols, documentation, and
1920 other logical components that should be split out. The ``do_package``
1921 task ensures that files are split up and packaged correctly.
1922
1923- *Running QA Checks*: The
1924 :ref:`insane <ref-classes-insane>` class adds a
1925 step to the package generation process so that output quality
1926 assurance checks are generated by the OpenEmbedded build system. This
1927 step performs a range of checks to be sure the build's output is free
1928 of common problems that show up during runtime. For information on
1929 these checks, see the
1930 :ref:`insane <ref-classes-insane>` class and
Andrew Geissler09209ee2020-12-13 08:44:15 -06001931 the ":ref:`ref-manual/qa-checks:qa error and warning messages`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001932 chapter in the Yocto Project Reference Manual.
1933
1934- *Hand-Checking Your Packages*: After you build your software, you
1935 need to be sure your packages are correct. Examine the
1936 ``${``\ :term:`WORKDIR`\ ``}/packages-split``
1937 directory and make sure files are where you expect them to be. If you
1938 discover problems, you can set
1939 :term:`PACKAGES`,
1940 :term:`FILES`,
1941 ``do_install(_append)``, and so forth as needed.
1942
1943- *Splitting an Application into Multiple Packages*: If you need to
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001944 split an application into several packages, see the
1945 ":ref:`dev-manual/common-tasks:splitting an application into multiple packages`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001946 section for an example.
1947
1948- *Installing a Post-Installation Script*: For an example showing how
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001949 to install a post-installation script, see the
1950 ":ref:`dev-manual/common-tasks:post-installation scripts`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001951
1952- *Marking Package Architecture*: Depending on what your recipe is
1953 building and how it is configured, it might be important to mark the
1954 packages produced as being specific to a particular machine, or to
1955 mark them as not being specific to a particular machine or
1956 architecture at all.
1957
1958 By default, packages apply to any machine with the same architecture
1959 as the target machine. When a recipe produces packages that are
1960 machine-specific (e.g. the
1961 :term:`MACHINE` value is passed
1962 into the configure script or a patch is applied only for a particular
1963 machine), you should mark them as such by adding the following to the
Andrew Geisslerc926e172021-05-07 16:11:35 -05001964 recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001965
1966 PACKAGE_ARCH = "${MACHINE_ARCH}"
1967
1968 On the other hand, if the recipe produces packages that do not
1969 contain anything specific to the target machine or architecture at
1970 all (e.g. recipes that simply package script files or configuration
1971 files), you should use the
1972 :ref:`allarch <ref-classes-allarch>` class to
Andrew Geisslerc926e172021-05-07 16:11:35 -05001973 do this for you by adding this to your recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001974
1975 inherit allarch
1976
1977 Ensuring that the package architecture is correct is not critical
1978 while you are doing the first few builds of your recipe. However, it
1979 is important in order to ensure that your recipe rebuilds (or does
1980 not rebuild) appropriately in response to changes in configuration,
1981 and to ensure that you get the appropriate packages installed on the
1982 target machine, particularly if you run separate builds for more than
1983 one target machine.
1984
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001985Sharing Files Between Recipes
1986-----------------------------
1987
1988Recipes often need to use files provided by other recipes on the build
1989host. For example, an application linking to a common library needs
1990access to the library itself and its associated headers. The way this
1991access is accomplished is by populating a sysroot with files. Each
1992recipe has two sysroots in its work directory, one for target files
1993(``recipe-sysroot``) and one for files that are native to the build host
1994(``recipe-sysroot-native``).
1995
1996.. note::
1997
1998 You could find the term "staging" used within the Yocto project
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001999 regarding files populating sysroots (e.g. the :term:`STAGING_DIR`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002000 variable).
2001
2002Recipes should never populate the sysroot directly (i.e. write files
2003into sysroot). Instead, files should be installed into standard
2004locations during the
2005:ref:`ref-tasks-install` task within
2006the ``${``\ :term:`D`\ ``}`` directory. The
2007reason for this limitation is that almost all files that populate the
2008sysroot are cataloged in manifests in order to ensure the files can be
2009removed later when a recipe is either modified or removed. Thus, the
2010sysroot is able to remain free from stale files.
2011
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05002012A subset of the files installed by the :ref:`ref-tasks-install` task are
2013used by the :ref:`ref-tasks-populate_sysroot` task as defined by the the
2014:term:`SYSROOT_DIRS` variable to automatically populate the sysroot. It
2015is possible to modify the list of directories that populate the sysroot.
2016The following example shows how you could add the ``/opt`` directory to
Andrew Geisslerc926e172021-05-07 16:11:35 -05002017the list of directories within a recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002018
2019 SYSROOT_DIRS += "/opt"
2020
Andrew Geisslerd1e89492021-02-12 15:35:20 -06002021.. note::
2022
2023 The `/sysroot-only` is to be used by recipes that generate artifacts
2024 that are not included in the target filesystem, allowing them to share
Andrew Geissler09036742021-06-25 14:25:14 -05002025 these artifacts without needing to use the :term:`DEPLOY_DIR`.
Andrew Geisslerd1e89492021-02-12 15:35:20 -06002026
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05002027For a more complete description of the :ref:`ref-tasks-populate_sysroot`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002028task and its associated functions, see the
2029:ref:`staging <ref-classes-staging>` class.
2030
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002031Using Virtual Providers
2032-----------------------
2033
2034Prior to a build, if you know that several different recipes provide the
2035same functionality, you can use a virtual provider (i.e. ``virtual/*``)
2036as a placeholder for the actual provider. The actual provider is
2037determined at build-time.
2038
2039A common scenario where a virtual provider is used would be for the
2040kernel recipe. Suppose you have three kernel recipes whose
2041:term:`PN` values map to ``kernel-big``,
2042``kernel-mid``, and ``kernel-small``. Furthermore, each of these recipes
2043in some way uses a :term:`PROVIDES`
2044statement that essentially identifies itself as being able to provide
2045``virtual/kernel``. Here is one way through the
Andrew Geisslerc926e172021-05-07 16:11:35 -05002046:ref:`kernel <ref-classes-kernel>` class::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002047
2048 PROVIDES += "${@ "virtual/kernel" if (d.getVar("KERNEL_PACKAGE_NAME") == "kernel") else "" }"
2049
2050Any recipe that inherits the ``kernel`` class is
Andrew Geissler09036742021-06-25 14:25:14 -05002051going to utilize a :term:`PROVIDES` statement that identifies that recipe as
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002052being able to provide the ``virtual/kernel`` item.
2053
2054Now comes the time to actually build an image and you need a kernel
2055recipe, but which one? You can configure your build to call out the
Andrew Geissler09209ee2020-12-13 08:44:15 -06002056kernel recipe you want by using the :term:`PREFERRED_PROVIDER` variable. As
2057an example, consider the :yocto_git:`x86-base.inc
2058</poky/tree/meta/conf/machine/include/x86-base.inc>` include file, which is a
2059machine (i.e. :term:`MACHINE`) configuration file. This include file is the
2060reason all x86-based machines use the ``linux-yocto`` kernel. Here are the
Andrew Geisslerc926e172021-05-07 16:11:35 -05002061relevant lines from the include file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002062
2063 PREFERRED_PROVIDER_virtual/kernel ??= "linux-yocto"
2064 PREFERRED_VERSION_linux-yocto ??= "4.15%"
2065
2066When you use a virtual provider, you do not have to "hard code" a recipe
2067name as a build dependency. You can use the
2068:term:`DEPENDS` variable to state the
Andrew Geisslerc926e172021-05-07 16:11:35 -05002069build is dependent on ``virtual/kernel`` for example::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002070
2071 DEPENDS = "virtual/kernel"
2072
2073During the build, the OpenEmbedded build system picks
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002074the correct recipe needed for the ``virtual/kernel`` dependency based on
Andrew Geissler09036742021-06-25 14:25:14 -05002075the :term:`PREFERRED_PROVIDER` variable. If you want to use the small kernel
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002076mentioned at the beginning of this section, configure your build as
Andrew Geisslerc926e172021-05-07 16:11:35 -05002077follows::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002078
2079 PREFERRED_PROVIDER_virtual/kernel ??= "kernel-small"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002080
2081.. note::
2082
Andrew Geissler09036742021-06-25 14:25:14 -05002083 Any recipe that :term:`PROVIDES` a ``virtual/*`` item that is ultimately not
2084 selected through :term:`PREFERRED_PROVIDER` does not get built. Preventing these
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002085 recipes from building is usually the desired behavior since this mechanism's
2086 purpose is to select between mutually exclusive alternative providers.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002087
2088The following lists specific examples of virtual providers:
2089
2090- ``virtual/kernel``: Provides the name of the kernel recipe to use
2091 when building a kernel image.
2092
2093- ``virtual/bootloader``: Provides the name of the bootloader to use
2094 when building an image.
2095
2096- ``virtual/libgbm``: Provides ``gbm.pc``.
2097
2098- ``virtual/egl``: Provides ``egl.pc`` and possibly ``wayland-egl.pc``.
2099
2100- ``virtual/libgl``: Provides ``gl.pc`` (i.e. libGL).
2101
2102- ``virtual/libgles1``: Provides ``glesv1_cm.pc`` (i.e. libGLESv1_CM).
2103
2104- ``virtual/libgles2``: Provides ``glesv2.pc`` (i.e. libGLESv2).
2105
2106.. note::
2107
2108 Virtual providers only apply to build time dependencies specified with
2109 :term:`PROVIDES` and :term:`DEPENDS`. They do not apply to runtime
2110 dependencies specified with :term:`RPROVIDES` and :term:`RDEPENDS`.
2111
2112Properly Versioning Pre-Release Recipes
2113---------------------------------------
2114
2115Sometimes the name of a recipe can lead to versioning problems when the
2116recipe is upgraded to a final release. For example, consider the
2117``irssi_0.8.16-rc1.bb`` recipe file in the list of example recipes in
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05002118the ":ref:`dev-manual/common-tasks:storing and naming the recipe`" section.
2119This recipe is at a release candidate stage (i.e. "rc1"). When the recipe is
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002120released, the recipe filename becomes ``irssi_0.8.16.bb``. The version
2121change from ``0.8.16-rc1`` to ``0.8.16`` is seen as a decrease by the
2122build system and package managers, so the resulting packages will not
2123correctly trigger an upgrade.
2124
2125In order to ensure the versions compare properly, the recommended
2126convention is to set :term:`PV` within the
2127recipe to "previous_version+current_version". You can use an additional
2128variable so that you can use the current version elsewhere. Here is an
Andrew Geisslerc926e172021-05-07 16:11:35 -05002129example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002130
2131 REALPV = "0.8.16-rc1"
2132 PV = "0.8.15+${REALPV}"
2133
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002134Post-Installation Scripts
2135-------------------------
2136
2137Post-installation scripts run immediately after installing a package on
2138the target or during image creation when a package is included in an
2139image. To add a post-installation script to a package, add a
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002140``pkg_postinst_``\ `PACKAGENAME`\ ``()`` function to the recipe file
2141(``.bb``) and replace `PACKAGENAME` with the name of the package you want
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002142to attach to the ``postinst`` script. To apply the post-installation
2143script to the main package for the recipe, which is usually what is
2144required, specify
2145``${``\ :term:`PN`\ ``}`` in place of
2146PACKAGENAME.
2147
Andrew Geisslerc926e172021-05-07 16:11:35 -05002148A post-installation function has the following structure::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002149
2150 pkg_postinst_PACKAGENAME() {
2151 # Commands to carry out
2152 }
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002153
2154The script defined in the post-installation function is called when the
2155root filesystem is created. If the script succeeds, the package is
2156marked as installed.
2157
2158.. note::
2159
2160 Any RPM post-installation script that runs on the target should
2161 return a 0 exit code. RPM does not allow non-zero exit codes for
2162 these scripts, and the RPM package manager will cause the package to
2163 fail installation on the target.
2164
2165Sometimes it is necessary for the execution of a post-installation
2166script to be delayed until the first boot. For example, the script might
2167need to be executed on the device itself. To delay script execution
2168until boot time, you must explicitly mark post installs to defer to the
2169target. You can use ``pkg_postinst_ontarget()`` or call
2170``postinst_intercept delay_to_first_boot`` from ``pkg_postinst()``. Any
2171failure of a ``pkg_postinst()`` script (including exit 1) triggers an
2172error during the
2173:ref:`ref-tasks-rootfs` task.
2174
2175If you have recipes that use ``pkg_postinst`` function and they require
2176the use of non-standard native tools that have dependencies during
2177rootfs construction, you need to use the
2178:term:`PACKAGE_WRITE_DEPS`
2179variable in your recipe to list these tools. If you do not use this
2180variable, the tools might be missing and execution of the
2181post-installation script is deferred until first boot. Deferring the
2182script to first boot is undesirable and for read-only rootfs impossible.
2183
2184.. note::
2185
William A. Kennington IIIac69b482021-06-02 12:28:27 -07002186 There is equivalent support for pre-install, pre-uninstall, and post-uninstall
2187 scripts by way of ``pkg_preinst``, ``pkg_prerm``, and ``pkg_postrm``,
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002188 respectively. These scrips work in exactly the same way as does
2189 ``pkg_postinst`` with the exception that they run at different times. Also,
2190 because of when they run, they are not applicable to being run at image
2191 creation time like ``pkg_postinst``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002192
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002193Testing
2194-------
2195
2196The final step for completing your recipe is to be sure that the
2197software you built runs correctly. To accomplish runtime testing, add
2198the build's output packages to your image and test them on the target.
2199
2200For information on how to customize your image by adding specific
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05002201packages, see ":ref:`dev-manual/common-tasks:customizing images`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002202
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002203Examples
2204--------
2205
2206To help summarize how to write a recipe, this section provides some
2207examples given various scenarios:
2208
2209- Recipes that use local files
2210
2211- Using an Autotooled package
2212
2213- Using a Makefile-based package
2214
2215- Splitting an application into multiple packages
2216
2217- Adding binaries to an image
2218
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002219Single .c File Package (Hello World!)
2220~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2221
2222Building an application from a single file that is stored locally (e.g.
2223under ``files``) requires a recipe that has the file listed in the
Andrew Geissler09036742021-06-25 14:25:14 -05002224:term:`SRC_URI` variable. Additionally, you need to manually write the
2225``do_compile`` and ``do_install`` tasks. The :term:`S` variable defines the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002226directory containing the source code, which is set to
2227:term:`WORKDIR` in this case - the
2228directory BitBake uses for the build.
2229::
2230
2231 SUMMARY = "Simple helloworld application"
2232 SECTION = "examples"
2233 LICENSE = "MIT"
2234 LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
2235
2236 SRC_URI = "file://helloworld.c"
2237
2238 S = "${WORKDIR}"
2239
2240 do_compile() {
Andrew Geisslerd1e89492021-02-12 15:35:20 -06002241 ${CC} ${LDFLAGS} helloworld.c -o helloworld
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002242 }
2243
2244 do_install() {
2245 install -d ${D}${bindir}
2246 install -m 0755 helloworld ${D}${bindir}
2247 }
2248
2249By default, the ``helloworld``, ``helloworld-dbg``, and
2250``helloworld-dev`` packages are built. For information on how to
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05002251customize the packaging process, see the
2252":ref:`dev-manual/common-tasks:splitting an application into multiple packages`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002253section.
2254
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002255Autotooled Package
2256~~~~~~~~~~~~~~~~~~
2257
2258Applications that use Autotools such as ``autoconf`` and ``automake``
Andrew Geissler09036742021-06-25 14:25:14 -05002259require a recipe that has a source archive listed in :term:`SRC_URI` and
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002260also inherit the
2261:ref:`autotools <ref-classes-autotools>` class,
2262which contains the definitions of all the steps needed to build an
2263Autotool-based application. The result of the build is automatically
2264packaged. And, if the application uses NLS for localization, packages
2265with local information are generated (one package per language).
2266Following is one example: (``hello_2.3.bb``)
2267::
2268
2269 SUMMARY = "GNU Helloworld application"
2270 SECTION = "examples"
2271 LICENSE = "GPLv2+"
2272 LIC_FILES_CHKSUM = "file://COPYING;md5=751419260aa954499f7abaabaa882bbe"
2273
2274 SRC_URI = "${GNU_MIRROR}/hello/hello-${PV}.tar.gz"
2275
2276 inherit autotools gettext
2277
Andrew Geissler09036742021-06-25 14:25:14 -05002278The variable :term:`LIC_FILES_CHKSUM` is used to track source license
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002279changes as described in the
Andrew Geissler09209ee2020-12-13 08:44:15 -06002280":ref:`dev-manual/common-tasks:tracking license changes`" section in
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002281the Yocto Project Overview and Concepts Manual. You can quickly create
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002282Autotool-based recipes in a manner similar to the previous example.
2283
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002284Makefile-Based Package
2285~~~~~~~~~~~~~~~~~~~~~~
2286
2287Applications that use GNU ``make`` also require a recipe that has the
Andrew Geissler09036742021-06-25 14:25:14 -05002288source archive listed in :term:`SRC_URI`. You do not need to add a
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002289``do_compile`` step since by default BitBake starts the ``make`` command
2290to compile the application. If you need additional ``make`` options, you
2291should store them in the
2292:term:`EXTRA_OEMAKE` or
2293:term:`PACKAGECONFIG_CONFARGS`
2294variables. BitBake passes these options into the GNU ``make``
2295invocation. Note that a ``do_install`` task is still required.
2296Otherwise, BitBake runs an empty ``do_install`` task by default.
2297
2298Some applications might require extra parameters to be passed to the
2299compiler. For example, the application might need an additional header
Andrew Geissler09036742021-06-25 14:25:14 -05002300path. You can accomplish this by adding to the :term:`CFLAGS` variable. The
Andrew Geisslerc926e172021-05-07 16:11:35 -05002301following example shows this::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002302
2303 CFLAGS_prepend = "-I ${S}/include "
2304
Andrew Geisslerc926e172021-05-07 16:11:35 -05002305In the following example, ``mtd-utils`` is a makefile-based package::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002306
2307 SUMMARY = "Tools for managing memory technology devices"
2308 SECTION = "base"
2309 DEPENDS = "zlib lzo e2fsprogs util-linux"
2310 HOMEPAGE = "http://www.linux-mtd.infradead.org/"
2311 LICENSE = "GPLv2+"
2312 LIC_FILES_CHKSUM = "file://COPYING;md5=0636e73ff0215e8d672dc4c32c317bb3 \
2313 file://include/common.h;beginline=1;endline=17;md5=ba05b07912a44ea2bf81ce409380049c"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002314
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002315 # Use the latest version at 26 Oct, 2013
2316 SRCREV = "9f107132a6a073cce37434ca9cda6917dd8d866b"
2317 SRC_URI = "git://git.infradead.org/mtd-utils.git \
2318 file://add-exclusion-to-mkfs-jffs2-git-2.patch \
2319 "
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002320
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002321 PV = "1.5.1+git${SRCPV}"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002322
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002323 S = "${WORKDIR}/git"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002324
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002325 EXTRA_OEMAKE = "'CC=${CC}' 'RANLIB=${RANLIB}' 'AR=${AR}' 'CFLAGS=${CFLAGS} -I${S}/include -DWITHOUT_XATTR' 'BUILDDIR=${S}'"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002326
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002327 do_install () {
2328 oe_runmake install DESTDIR=${D} SBINDIR=${sbindir} MANDIR=${mandir} INCLUDEDIR=${includedir}
2329 }
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002330
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002331 PACKAGES =+ "mtd-utils-jffs2 mtd-utils-ubifs mtd-utils-misc"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002332
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002333 FILES_mtd-utils-jffs2 = "${sbindir}/mkfs.jffs2 ${sbindir}/jffs2dump ${sbindir}/jffs2reader ${sbindir}/sumtool"
2334 FILES_mtd-utils-ubifs = "${sbindir}/mkfs.ubifs ${sbindir}/ubi*"
2335 FILES_mtd-utils-misc = "${sbindir}/nftl* ${sbindir}/ftl* ${sbindir}/rfd* ${sbindir}/doc* ${sbindir}/serve_image ${sbindir}/recv_image"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002336
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002337 PARALLEL_MAKE = ""
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002338
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002339 BBCLASSEXTEND = "native"
2340
2341Splitting an Application into Multiple Packages
2342~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2343
Andrew Geissler09036742021-06-25 14:25:14 -05002344You can use the variables :term:`PACKAGES` and :term:`FILES` to split an
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002345application into multiple packages.
2346
2347Following is an example that uses the ``libxpm`` recipe. By default,
2348this recipe generates a single package that contains the library along
2349with a few binaries. You can modify the recipe to split the binaries
Andrew Geisslerc926e172021-05-07 16:11:35 -05002350into separate packages::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002351
2352 require xorg-lib-common.inc
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002353
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002354 SUMMARY = "Xpm: X Pixmap extension library"
2355 LICENSE = "BSD"
2356 LIC_FILES_CHKSUM = "file://COPYING;md5=51f4270b012ecd4ab1a164f5f4ed6cf7"
2357 DEPENDS += "libxext libsm libxt"
2358 PE = "1"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002359
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002360 XORG_PN = "libXpm"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002361
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002362 PACKAGES =+ "sxpm cxpm"
2363 FILES_cxpm = "${bindir}/cxpm"
2364 FILES_sxpm = "${bindir}/sxpm"
2365
2366In the previous example, we want to ship the ``sxpm`` and ``cxpm``
2367binaries in separate packages. Since ``bindir`` would be packaged into
Andrew Geissler09036742021-06-25 14:25:14 -05002368the main :term:`PN` package by default, we prepend the :term:`PACKAGES` variable
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002369so additional package names are added to the start of list. This results
2370in the extra ``FILES_*`` variables then containing information that
2371define which files and directories go into which packages. Files
2372included by earlier packages are skipped by latter packages. Thus, the
Andrew Geissler09036742021-06-25 14:25:14 -05002373main :term:`PN` package does not include the above listed files.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002374
2375Packaging Externally Produced Binaries
2376~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2377
2378Sometimes, you need to add pre-compiled binaries to an image. For
William A. Kennington IIIac69b482021-06-02 12:28:27 -07002379example, suppose that there are binaries for proprietary code,
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002380created by a particular division of a company. Your part of the company
2381needs to use those binaries as part of an image that you are building
2382using the OpenEmbedded build system. Since you only have the binaries
2383and not the source code, you cannot use a typical recipe that expects to
2384fetch the source specified in
2385:term:`SRC_URI` and then compile it.
2386
2387One method is to package the binaries and then install them as part of
2388the image. Generally, it is not a good idea to package binaries since,
2389among other things, it can hinder the ability to reproduce builds and
2390could lead to compatibility problems with ABI in the future. However,
2391sometimes you have no choice.
2392
2393The easiest solution is to create a recipe that uses the
2394:ref:`bin_package <ref-classes-bin-package>` class
2395and to be sure that you are using default locations for build artifacts.
2396In most cases, the ``bin_package`` class handles "skipping" the
2397configure and compile steps as well as sets things up to grab packages
2398from the appropriate area. In particular, this class sets ``noexec`` on
2399both the :ref:`ref-tasks-configure`
2400and :ref:`ref-tasks-compile` tasks,
2401sets ``FILES_${PN}`` to "/" so that it picks up all files, and sets up a
2402:ref:`ref-tasks-install` task, which
2403effectively copies all files from ``${S}`` to ``${D}``. The
2404``bin_package`` class works well when the files extracted into ``${S}``
2405are already laid out in the way they should be laid out on the target.
2406For more information on these variables, see the
2407:term:`FILES`,
2408:term:`PN`,
2409:term:`S`, and
2410:term:`D` variables in the Yocto Project
2411Reference Manual's variable glossary.
2412
2413.. note::
2414
2415 - Using :term:`DEPENDS` is a good
2416 idea even for components distributed in binary form, and is often
2417 necessary for shared libraries. For a shared library, listing the
Andrew Geissler09036742021-06-25 14:25:14 -05002418 library dependencies in :term:`DEPENDS` makes sure that the libraries
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002419 are available in the staging sysroot when other recipes link
2420 against the library, which might be necessary for successful
2421 linking.
2422
Andrew Geissler09036742021-06-25 14:25:14 -05002423 - Using :term:`DEPENDS` also allows runtime dependencies between
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002424 packages to be added automatically. See the
Andrew Geissler09209ee2020-12-13 08:44:15 -06002425 ":ref:`overview-manual/concepts:automatically added runtime dependencies`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002426 section in the Yocto Project Overview and Concepts Manual for more
2427 information.
2428
2429If you cannot use the ``bin_package`` class, you need to be sure you are
2430doing the following:
2431
2432- Create a recipe where the
2433 :ref:`ref-tasks-configure` and
2434 :ref:`ref-tasks-compile` tasks do
2435 nothing: It is usually sufficient to just not define these tasks in
2436 the recipe, because the default implementations do nothing unless a
2437 Makefile is found in
2438 ``${``\ :term:`S`\ ``}``.
2439
2440 If ``${S}`` might contain a Makefile, or if you inherit some class
2441 that replaces ``do_configure`` and ``do_compile`` with custom
2442 versions, then you can use the
2443 ``[``\ :ref:`noexec <bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ``]``
Andrew Geisslerc926e172021-05-07 16:11:35 -05002444 flag to turn the tasks into no-ops, as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002445
2446 do_configure[noexec] = "1"
2447 do_compile[noexec] = "1"
2448
2449 Unlike
2450 :ref:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata:deleting a task`,
2451 using the flag preserves the dependency chain from the
2452 :ref:`ref-tasks-fetch`,
2453 :ref:`ref-tasks-unpack`, and
2454 :ref:`ref-tasks-patch` tasks to the
2455 :ref:`ref-tasks-install` task.
2456
2457- Make sure your ``do_install`` task installs the binaries
2458 appropriately.
2459
2460- Ensure that you set up :term:`FILES`
2461 (usually
2462 ``FILES_${``\ :term:`PN`\ ``}``) to
2463 point to the files you have installed, which of course depends on
2464 where you have installed them and whether those files are in
2465 different locations than the defaults.
2466
Andrew Geissler6ce62a22020-11-30 19:58:47 -06002467.. note::
2468
2469 If image prelinking is enabled (e.g. "image-prelink" is in :term:`USER_CLASSES`
2470 which it is by default), prelink will change the binaries in the generated images
2471 and this often catches people out. Remove that class to ensure binaries are
2472 preserved exactly if that is necessary.
2473
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002474Following Recipe Style Guidelines
2475---------------------------------
2476
2477When writing recipes, it is good to conform to existing style
Andrew Geisslerd1e89492021-02-12 15:35:20 -06002478guidelines. The :oe_wiki:`OpenEmbedded Styleguide </Styleguide>` wiki page
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002479provides rough guidelines for preferred recipe style.
2480
2481It is common for existing recipes to deviate a bit from this style.
2482However, aiming for at least a consistent style is a good idea. Some
2483practices, such as omitting spaces around ``=`` operators in assignments
2484or ordering recipe components in an erratic way, are widely seen as poor
2485style.
2486
2487Recipe Syntax
2488-------------
2489
2490Understanding recipe file syntax is important for writing recipes. The
2491following list overviews the basic items that make up a BitBake recipe
2492file. For more complete BitBake syntax descriptions, see the
2493":doc:`bitbake-user-manual/bitbake-user-manual-metadata`"
2494chapter of the BitBake User Manual.
2495
2496- *Variable Assignments and Manipulations:* Variable assignments allow
2497 a value to be assigned to a variable. The assignment can be static
2498 text or might include the contents of other variables. In addition to
2499 the assignment, appending and prepending operations are also
2500 supported.
2501
2502 The following example shows some of the ways you can use variables in
Andrew Geisslerc926e172021-05-07 16:11:35 -05002503 recipes::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002504
2505 S = "${WORKDIR}/postfix-${PV}"
2506 CFLAGS += "-DNO_ASM"
2507 SRC_URI_append = " file://fixup.patch"
2508
2509- *Functions:* Functions provide a series of actions to be performed.
2510 You usually use functions to override the default implementation of a
2511 task function or to complement a default function (i.e. append or
2512 prepend to an existing function). Standard functions use ``sh`` shell
2513 syntax, although access to OpenEmbedded variables and internal
2514 methods are also available.
2515
William A. Kennington IIIac69b482021-06-02 12:28:27 -07002516 Here is an example function from the ``sed`` recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002517
2518 do_install () {
2519 autotools_do_install
2520 install -d ${D}${base_bindir}
2521 mv ${D}${bindir}/sed ${D}${base_bindir}/sed
2522 rmdir ${D}${bindir}/
2523 }
2524
2525 It is
2526 also possible to implement new functions that are called between
2527 existing tasks as long as the new functions are not replacing or
2528 complementing the default functions. You can implement functions in
2529 Python instead of shell. Both of these options are not seen in the
2530 majority of recipes.
2531
2532- *Keywords:* BitBake recipes use only a few keywords. You use keywords
2533 to include common functions (``inherit``), load parts of a recipe
2534 from other files (``include`` and ``require``) and export variables
2535 to the environment (``export``).
2536
Andrew Geisslerc926e172021-05-07 16:11:35 -05002537 The following example shows the use of some of these keywords::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002538
2539 export POSTCONF = "${STAGING_BINDIR}/postconf"
2540 inherit autoconf
2541 require otherfile.inc
2542
2543- *Comments (#):* Any lines that begin with the hash character (``#``)
Andrew Geisslerc926e172021-05-07 16:11:35 -05002544 are treated as comment lines and are ignored::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002545
2546 # This is a comment
2547
2548This next list summarizes the most important and most commonly used
2549parts of the recipe syntax. For more information on these parts of the
2550syntax, you can reference the
2551:doc:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata` chapter
2552in the BitBake User Manual.
2553
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002554- *Line Continuation (\\):* Use the backward slash (``\``) character to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002555 split a statement over multiple lines. Place the slash character at
Andrew Geisslerc926e172021-05-07 16:11:35 -05002556 the end of the line that is to be continued on the next line::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002557
2558 VAR = "A really long \
2559 line"
2560
2561 .. note::
2562
2563 You cannot have any characters including spaces or tabs after the
2564 slash character.
2565
2566- *Using Variables (${VARNAME}):* Use the ``${VARNAME}`` syntax to
Andrew Geisslerc926e172021-05-07 16:11:35 -05002567 access the contents of a variable::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002568
2569 SRC_URI = "${SOURCEFORGE_MIRROR}/libpng/zlib-${PV}.tar.gz"
2570
2571 .. note::
2572
2573 It is important to understand that the value of a variable
2574 expressed in this form does not get substituted automatically. The
2575 expansion of these expressions happens on-demand later (e.g.
2576 usually when a function that makes reference to the variable
2577 executes). This behavior ensures that the values are most
2578 appropriate for the context in which they are finally used. On the
2579 rare occasion that you do need the variable expression to be
2580 expanded immediately, you can use the
2581 :=
2582 operator instead of
2583 =
2584 when you make the assignment, but this is not generally needed.
2585
2586- *Quote All Assignments ("value"):* Use double quotes around values in
Andrew Geisslerc926e172021-05-07 16:11:35 -05002587 all variable assignments (e.g. ``"value"``). Following is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002588
2589 VAR1 = "${OTHERVAR}"
2590 VAR2 = "The version is ${PV}"
2591
2592- *Conditional Assignment (?=):* Conditional assignment is used to
2593 assign a value to a variable, but only when the variable is currently
2594 unset. Use the question mark followed by the equal sign (``?=``) to
2595 make a "soft" assignment used for conditional assignment. Typically,
2596 "soft" assignments are used in the ``local.conf`` file for variables
2597 that are allowed to come through from the external environment.
2598
2599 Here is an example where ``VAR1`` is set to "New value" if it is
2600 currently empty. However, if ``VAR1`` has already been set, it
Andrew Geisslerc926e172021-05-07 16:11:35 -05002601 remains unchanged::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002602
2603 VAR1 ?= "New value"
2604
Andrew Geisslerc926e172021-05-07 16:11:35 -05002605 In this next example, ``VAR1`` is left with the value "Original value"::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002606
2607 VAR1 = "Original value"
2608 VAR1 ?= "New value"
2609
2610- *Appending (+=):* Use the plus character followed by the equals sign
2611 (``+=``) to append values to existing variables.
2612
2613 .. note::
2614
2615 This operator adds a space between the existing content of the
2616 variable and the new content.
2617
Andrew Geisslerc926e172021-05-07 16:11:35 -05002618 Here is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002619
2620 SRC_URI += "file://fix-makefile.patch"
2621
2622- *Prepending (=+):* Use the equals sign followed by the plus character
2623 (``=+``) to prepend values to existing variables.
2624
2625 .. note::
2626
2627 This operator adds a space between the new content and the
2628 existing content of the variable.
2629
Andrew Geisslerc926e172021-05-07 16:11:35 -05002630 Here is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002631
2632 VAR =+ "Starts"
2633
2634- *Appending (_append):* Use the ``_append`` operator to append values
2635 to existing variables. This operator does not add any additional
2636 space. Also, the operator is applied after all the ``+=``, and ``=+``
2637 operators have been applied and after all ``=`` assignments have
2638 occurred.
2639
2640 The following example shows the space being explicitly added to the
2641 start to ensure the appended value is not merged with the existing
Andrew Geisslerc926e172021-05-07 16:11:35 -05002642 value::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002643
2644 SRC_URI_append = " file://fix-makefile.patch"
2645
2646 You can also use
2647 the ``_append`` operator with overrides, which results in the actions
Andrew Geisslerc926e172021-05-07 16:11:35 -05002648 only being performed for the specified target or machine::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002649
2650 SRC_URI_append_sh4 = " file://fix-makefile.patch"
2651
2652- *Prepending (_prepend):* Use the ``_prepend`` operator to prepend
2653 values to existing variables. This operator does not add any
2654 additional space. Also, the operator is applied after all the ``+=``,
2655 and ``=+`` operators have been applied and after all ``=``
2656 assignments have occurred.
2657
2658 The following example shows the space being explicitly added to the
2659 end to ensure the prepended value is not merged with the existing
Andrew Geisslerc926e172021-05-07 16:11:35 -05002660 value::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002661
2662 CFLAGS_prepend = "-I${S}/myincludes "
2663
2664 You can also use the
2665 ``_prepend`` operator with overrides, which results in the actions
Andrew Geisslerc926e172021-05-07 16:11:35 -05002666 only being performed for the specified target or machine::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002667
2668 CFLAGS_prepend_sh4 = "-I${S}/myincludes "
2669
2670- *Overrides:* You can use overrides to set a value conditionally,
2671 typically based on how the recipe is being built. For example, to set
2672 the :term:`KBRANCH` variable's
2673 value to "standard/base" for any target
2674 :term:`MACHINE`, except for
2675 qemuarm where it should be set to "standard/arm-versatile-926ejs",
Andrew Geisslerc926e172021-05-07 16:11:35 -05002676 you would do the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002677
2678 KBRANCH = "standard/base"
2679 KBRANCH_qemuarm = "standard/arm-versatile-926ejs"
2680
2681 Overrides are also used to separate
2682 alternate values of a variable in other situations. For example, when
2683 setting variables such as
2684 :term:`FILES` and
2685 :term:`RDEPENDS` that are
2686 specific to individual packages produced by a recipe, you should
2687 always use an override that specifies the name of the package.
2688
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002689- *Indentation:* Use spaces for indentation rather than tabs. For
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002690 shell functions, both currently work. However, it is a policy
2691 decision of the Yocto Project to use tabs in shell functions. Realize
2692 that some layers have a policy to use spaces for all indentation.
2693
2694- *Using Python for Complex Operations:* For more advanced processing,
2695 it is possible to use Python code during variable assignments (e.g.
2696 search and replacement on a variable).
2697
2698 You indicate Python code using the ``${@python_code}`` syntax for the
Andrew Geisslerc926e172021-05-07 16:11:35 -05002699 variable assignment::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002700
2701 SRC_URI = "ftp://ftp.info-zip.org/pub/infozip/src/zip${@d.getVar('PV',1).replace('.', '')}.tgz
2702
2703- *Shell Function Syntax:* Write shell functions as if you were writing
2704 a shell script when you describe a list of actions to take. You
2705 should ensure that your script works with a generic ``sh`` and that
2706 it does not require any ``bash`` or other shell-specific
2707 functionality. The same considerations apply to various system
2708 utilities (e.g. ``sed``, ``grep``, ``awk``, and so forth) that you
2709 might wish to use. If in doubt, you should check with multiple
2710 implementations - including those from BusyBox.
2711
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002712Adding a New Machine
2713====================
2714
2715Adding a new machine to the Yocto Project is a straightforward process.
2716This section describes how to add machines that are similar to those
2717that the Yocto Project already supports.
2718
2719.. note::
2720
2721 Although well within the capabilities of the Yocto Project, adding a
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002722 totally new architecture might require changes to ``gcc``/``glibc``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002723 and to the site information, which is beyond the scope of this
2724 manual.
2725
2726For a complete example that shows how to add a new machine, see the
2727":ref:`bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script`"
2728section in the Yocto Project Board Support Package (BSP) Developer's
2729Guide.
2730
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002731Adding the Machine Configuration File
2732-------------------------------------
2733
2734To add a new machine, you need to add a new machine configuration file
2735to the layer's ``conf/machine`` directory. This configuration file
2736provides details about the device you are adding.
2737
2738The OpenEmbedded build system uses the root name of the machine
2739configuration file to reference the new machine. For example, given a
2740machine configuration file named ``crownbay.conf``, the build system
2741recognizes the machine as "crownbay".
2742
2743The most important variables you must set in your machine configuration
2744file or include from a lower-level configuration file are as follows:
2745
Andrew Geissler5f350902021-07-23 13:09:54 -04002746- :term:`TARGET_ARCH` (e.g. "arm")
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002747
2748- ``PREFERRED_PROVIDER_virtual/kernel``
2749
Andrew Geissler09036742021-06-25 14:25:14 -05002750- :term:`MACHINE_FEATURES` (e.g. "apm screen wifi")
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002751
2752You might also need these variables:
2753
Andrew Geissler5f350902021-07-23 13:09:54 -04002754- :term:`SERIAL_CONSOLES` (e.g. "115200;ttyS0 115200;ttyS1")
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002755
Andrew Geissler5f350902021-07-23 13:09:54 -04002756- :term:`KERNEL_IMAGETYPE` (e.g. "zImage")
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002757
Andrew Geissler09036742021-06-25 14:25:14 -05002758- :term:`IMAGE_FSTYPES` (e.g. "tar.gz jffs2")
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002759
2760You can find full details on these variables in the reference section.
2761You can leverage existing machine ``.conf`` files from
2762``meta-yocto-bsp/conf/machine/``.
2763
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002764Adding a Kernel for the Machine
2765-------------------------------
2766
2767The OpenEmbedded build system needs to be able to build a kernel for the
2768machine. You need to either create a new kernel recipe for this machine,
2769or extend an existing kernel recipe. You can find several kernel recipe
2770examples in the Source Directory at ``meta/recipes-kernel/linux`` that
2771you can use as references.
2772
2773If you are creating a new kernel recipe, normal recipe-writing rules
Andrew Geissler09036742021-06-25 14:25:14 -05002774apply for setting up a :term:`SRC_URI`. Thus, you need to specify any
2775necessary patches and set :term:`S` to point at the source code. You need to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002776create a ``do_configure`` task that configures the unpacked kernel with
2777a ``defconfig`` file. You can do this by using a ``make defconfig``
2778command or, more commonly, by copying in a suitable ``defconfig`` file
2779and then running ``make oldconfig``. By making use of ``inherit kernel``
2780and potentially some of the ``linux-*.inc`` files, most other
2781functionality is centralized and the defaults of the class normally work
2782well.
2783
2784If you are extending an existing kernel recipe, it is usually a matter
2785of adding a suitable ``defconfig`` file. The file needs to be added into
2786a location similar to ``defconfig`` files used for other machines in a
2787given kernel recipe. A possible way to do this is by listing the file in
Andrew Geissler09036742021-06-25 14:25:14 -05002788the :term:`SRC_URI` and adding the machine to the expression in
2789:term:`COMPATIBLE_MACHINE`::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002790
2791 COMPATIBLE_MACHINE = '(qemux86|qemumips)'
2792
2793For more information on ``defconfig`` files, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06002794":ref:`kernel-dev/common:changing the configuration`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002795section in the Yocto Project Linux Kernel Development Manual.
2796
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002797Adding a Formfactor Configuration File
2798--------------------------------------
2799
2800A formfactor configuration file provides information about the target
2801hardware for which the image is being built and information that the
2802build system cannot obtain from other sources such as the kernel. Some
2803examples of information contained in a formfactor configuration file
2804include framebuffer orientation, whether or not the system has a
2805keyboard, the positioning of the keyboard in relation to the screen, and
2806the screen resolution.
2807
2808The build system uses reasonable defaults in most cases. However, if
2809customization is necessary, you need to create a ``machconfig`` file in
2810the ``meta/recipes-bsp/formfactor/files`` directory. This directory
2811contains directories for specific machines such as ``qemuarm`` and
2812``qemux86``. For information about the settings available and the
2813defaults, see the ``meta/recipes-bsp/formfactor/files/config`` file
2814found in the same area.
2815
Andrew Geisslerc926e172021-05-07 16:11:35 -05002816Following is an example for "qemuarm" machine::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002817
2818 HAVE_TOUCHSCREEN=1
2819 HAVE_KEYBOARD=1
2820 DISPLAY_CAN_ROTATE=0
2821 DISPLAY_ORIENTATION=0
2822 #DISPLAY_WIDTH_PIXELS=640
2823 #DISPLAY_HEIGHT_PIXELS=480
2824 #DISPLAY_BPP=16
2825 DISPLAY_DPI=150
2826 DISPLAY_SUBPIXEL_ORDER=vrgb
2827
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002828Upgrading Recipes
2829=================
2830
2831Over time, upstream developers publish new versions for software built
2832by layer recipes. It is recommended to keep recipes up-to-date with
2833upstream version releases.
2834
William A. Kennington IIIac69b482021-06-02 12:28:27 -07002835While there are several methods to upgrade a recipe, you might
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002836consider checking on the upgrade status of a recipe first. You can do so
2837using the ``devtool check-upgrade-status`` command. See the
2838":ref:`devtool-checking-on-the-upgrade-status-of-a-recipe`"
2839section in the Yocto Project Reference Manual for more information.
2840
2841The remainder of this section describes three ways you can upgrade a
2842recipe. You can use the Automated Upgrade Helper (AUH) to set up
2843automatic version upgrades. Alternatively, you can use
2844``devtool upgrade`` to set up semi-automatic version upgrades. Finally,
2845you can manually upgrade a recipe by editing the recipe itself.
2846
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002847Using the Auto Upgrade Helper (AUH)
2848-----------------------------------
2849
2850The AUH utility works in conjunction with the OpenEmbedded build system
2851in order to automatically generate upgrades for recipes based on new
2852versions being published upstream. Use AUH when you want to create a
2853service that performs the upgrades automatically and optionally sends
2854you an email with the results.
2855
2856AUH allows you to update several recipes with a single use. You can also
2857optionally perform build and integration tests using images with the
2858results saved to your hard drive and emails of results optionally sent
2859to recipe maintainers. Finally, AUH creates Git commits with appropriate
2860commit messages in the layer's tree for the changes made to recipes.
2861
2862.. note::
2863
William A. Kennington IIIac69b482021-06-02 12:28:27 -07002864 In some conditions, you should not use AUH to upgrade recipes
2865 and should instead use either ``devtool upgrade`` or upgrade your
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002866 recipes manually:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002867
2868 - When AUH cannot complete the upgrade sequence. This situation
2869 usually results because custom patches carried by the recipe
2870 cannot be automatically rebased to the new version. In this case,
2871 ``devtool upgrade`` allows you to manually resolve conflicts.
2872
2873 - When for any reason you want fuller control over the upgrade
2874 process. For example, when you want special arrangements for
2875 testing.
2876
2877The following steps describe how to set up the AUH utility:
2878
28791. *Be Sure the Development Host is Set Up:* You need to be sure that
2880 your development host is set up to use the Yocto Project. For
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002881 information on how to set up your host, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06002882 ":ref:`dev-manual/start:Preparing the Build Host`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002883
28842. *Make Sure Git is Configured:* The AUH utility requires Git to be
2885 configured because AUH uses Git to save upgrades. Thus, you must have
2886 Git user and email configured. The following command shows your
Andrew Geisslerc926e172021-05-07 16:11:35 -05002887 configurations::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002888
2889 $ git config --list
2890
2891 If you do not have the user and
Andrew Geisslerc926e172021-05-07 16:11:35 -05002892 email configured, you can use the following commands to do so::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002893
2894 $ git config --global user.name some_name
2895 $ git config --global user.email username@domain.com
2896
28973. *Clone the AUH Repository:* To use AUH, you must clone the repository
2898 onto your development host. The following command uses Git to create
Andrew Geisslerc926e172021-05-07 16:11:35 -05002899 a local copy of the repository on your system::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002900
2901 $ git clone git://git.yoctoproject.org/auto-upgrade-helper
2902 Cloning into 'auto-upgrade-helper'... remote: Counting objects: 768, done.
2903 remote: Compressing objects: 100% (300/300), done.
2904 remote: Total 768 (delta 499), reused 703 (delta 434)
2905 Receiving objects: 100% (768/768), 191.47 KiB | 98.00 KiB/s, done.
2906 Resolving deltas: 100% (499/499), done.
2907 Checking connectivity... done.
2908
2909 AUH is not part of the :term:`OpenEmbedded-Core (OE-Core)` or
2910 :term:`Poky` repositories.
2911
29124. *Create a Dedicated Build Directory:* Run the
2913 :ref:`structure-core-script`
2914 script to create a fresh build directory that you use exclusively for
Andrew Geisslerc926e172021-05-07 16:11:35 -05002915 running the AUH utility::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002916
Andrew Geissler95ac1b82021-03-31 14:34:31 -05002917 $ cd poky
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002918 $ source oe-init-build-env your_AUH_build_directory
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002919
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002920 Re-using an existing build directory and its configurations is not
2921 recommended as existing settings could cause AUH to fail or behave
2922 undesirably.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002923
29245. *Make Configurations in Your Local Configuration File:* Several
William A. Kennington IIIac69b482021-06-02 12:28:27 -07002925 settings are needed in the ``local.conf`` file in the build
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002926 directory you just created for AUH. Make these following
2927 configurations:
2928
2929 - If you want to enable :ref:`Build
Andrew Geissler09209ee2020-12-13 08:44:15 -06002930 History <dev-manual/common-tasks:maintaining build output quality>`,
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002931 which is optional, you need the following lines in the
Andrew Geisslerc926e172021-05-07 16:11:35 -05002932 ``conf/local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002933
2934 INHERIT =+ "buildhistory"
2935 BUILDHISTORY_COMMIT = "1"
2936
2937 With this configuration and a successful
2938 upgrade, a build history "diff" file appears in the
2939 ``upgrade-helper/work/recipe/buildhistory-diff.txt`` file found in
2940 your build directory.
2941
2942 - If you want to enable testing through the
2943 :ref:`testimage <ref-classes-testimage*>`
2944 class, which is optional, you need to have the following set in
Andrew Geisslerc926e172021-05-07 16:11:35 -05002945 your ``conf/local.conf`` file::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002946
2947 INHERIT += "testimage"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002948
2949 .. note::
2950
2951 If your distro does not enable by default ptest, which Poky
Andrew Geisslerc926e172021-05-07 16:11:35 -05002952 does, you need the following in your ``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002953
2954 DISTRO_FEATURES_append = " ptest"
2955
2956
29576. *Optionally Start a vncserver:* If you are running in a server
Andrew Geisslerc926e172021-05-07 16:11:35 -05002958 without an X11 session, you need to start a vncserver::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002959
2960 $ vncserver :1
2961 $ export DISPLAY=:1
2962
29637. *Create and Edit an AUH Configuration File:* You need to have the
2964 ``upgrade-helper/upgrade-helper.conf`` configuration file in your
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002965 build directory. You can find a sample configuration file in the
Andrew Geissler09209ee2020-12-13 08:44:15 -06002966 :yocto_git:`AUH source repository </auto-upgrade-helper/tree/>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002967
2968 Read through the sample file and make configurations as needed. For
2969 example, if you enabled build history in your ``local.conf`` as
2970 described earlier, you must enable it in ``upgrade-helper.conf``.
2971
2972 Also, if you are using the default ``maintainers.inc`` file supplied
2973 with Poky and located in ``meta-yocto`` and you do not set a
2974 "maintainers_whitelist" or "global_maintainer_override" in the
2975 ``upgrade-helper.conf`` configuration, and you specify "-e all" on
2976 the AUH command-line, the utility automatically sends out emails to
2977 all the default maintainers. Please avoid this.
2978
2979This next set of examples describes how to use the AUH:
2980
2981- *Upgrading a Specific Recipe:* To upgrade a specific recipe, use the
Andrew Geisslerc926e172021-05-07 16:11:35 -05002982 following form::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002983
2984 $ upgrade-helper.py recipe_name
2985
Andrew Geisslerc926e172021-05-07 16:11:35 -05002986 For example, this command upgrades the ``xmodmap`` recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002987
2988 $ upgrade-helper.py xmodmap
2989
2990- *Upgrading a Specific Recipe to a Particular Version:* To upgrade a
Andrew Geisslerc926e172021-05-07 16:11:35 -05002991 specific recipe to a particular version, use the following form::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002992
2993 $ upgrade-helper.py recipe_name -t version
2994
Andrew Geisslerc926e172021-05-07 16:11:35 -05002995 For example, this command upgrades the ``xmodmap`` recipe to version 1.2.3::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002996
2997 $ upgrade-helper.py xmodmap -t 1.2.3
2998
2999- *Upgrading all Recipes to the Latest Versions and Suppressing Email
3000 Notifications:* To upgrade all recipes to their most recent versions
Andrew Geisslerc926e172021-05-07 16:11:35 -05003001 and suppress the email notifications, use the following command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003002
3003 $ upgrade-helper.py all
3004
3005- *Upgrading all Recipes to the Latest Versions and Send Email
3006 Notifications:* To upgrade all recipes to their most recent versions
3007 and send email messages to maintainers for each attempted recipe as
Andrew Geisslerc926e172021-05-07 16:11:35 -05003008 well as a status email, use the following command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003009
3010 $ upgrade-helper.py -e all
3011
3012Once you have run the AUH utility, you can find the results in the AUH
Andrew Geisslerc926e172021-05-07 16:11:35 -05003013build directory::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003014
3015 ${BUILDDIR}/upgrade-helper/timestamp
3016
3017The AUH utility
3018also creates recipe update commits from successful upgrade attempts in
3019the layer tree.
3020
3021You can easily set up to run the AUH utility on a regular basis by using
3022a cron job. See the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003023:yocto_git:`weeklyjob.sh </auto-upgrade-helper/tree/weeklyjob.sh>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003024file distributed with the utility for an example.
3025
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003026Using ``devtool upgrade``
3027-------------------------
3028
3029As mentioned earlier, an alternative method for upgrading recipes to
3030newer versions is to use
Andrew Geissler09209ee2020-12-13 08:44:15 -06003031:doc:`devtool upgrade </ref-manual/devtool-reference>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003032You can read about ``devtool upgrade`` in general in the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003033":ref:`sdk-manual/extensible:use \`\`devtool upgrade\`\` to create a version of the recipe that supports a newer version of the software`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003034section in the Yocto Project Application Development and the Extensible
3035Software Development Kit (eSDK) Manual.
3036
3037To see all the command-line options available with ``devtool upgrade``,
Andrew Geisslerc926e172021-05-07 16:11:35 -05003038use the following help command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003039
3040 $ devtool upgrade -h
3041
3042If you want to find out what version a recipe is currently at upstream
3043without any attempt to upgrade your local version of the recipe, you can
Andrew Geisslerc926e172021-05-07 16:11:35 -05003044use the following command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003045
3046 $ devtool latest-version recipe_name
3047
3048As mentioned in the previous section describing AUH, ``devtool upgrade``
3049works in a less-automated manner than AUH. Specifically,
3050``devtool upgrade`` only works on a single recipe that you name on the
3051command line, cannot perform build and integration testing using images,
3052and does not automatically generate commits for changes in the source
3053tree. Despite all these "limitations", ``devtool upgrade`` updates the
3054recipe file to the new upstream version and attempts to rebase custom
3055patches contained by the recipe as needed.
3056
3057.. note::
3058
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003059 AUH uses much of ``devtool upgrade`` behind the scenes making AUH somewhat
3060 of a "wrapper" application for ``devtool upgrade``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003061
3062A typical scenario involves having used Git to clone an upstream
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003063repository that you use during build operations. Because you have built the
3064recipe in the past, the layer is likely added to your
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003065configuration already. If for some reason, the layer is not added, you
3066could add it easily using the
3067":ref:`bitbake-layers <bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script>`"
3068script. For example, suppose you use the ``nano.bb`` recipe from the
3069``meta-oe`` layer in the ``meta-openembedded`` repository. For this
Andrew Geisslerc926e172021-05-07 16:11:35 -05003070example, assume that the layer has been cloned into following area::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003071
3072 /home/scottrif/meta-openembedded
3073
3074The following command from your
3075:term:`Build Directory` adds the layer to
Andrew Geisslerc926e172021-05-07 16:11:35 -05003076your build configuration (i.e. ``${BUILDDIR}/conf/bblayers.conf``)::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003077
3078 $ bitbake-layers add-layer /home/scottrif/meta-openembedded/meta-oe
3079 NOTE: Starting bitbake server...
3080 Parsing recipes: 100% |##########################################| Time: 0:00:55
3081 Parsing of 1431 .bb files complete (0 cached, 1431 parsed). 2040 targets, 56 skipped, 0 masked, 0 errors.
3082 Removing 12 recipes from the x86_64 sysroot: 100% |##############| Time: 0:00:00
3083 Removing 1 recipes from the x86_64_i586 sysroot: 100% |##########| Time: 0:00:00
3084 Removing 5 recipes from the i586 sysroot: 100% |#################| Time: 0:00:00
3085 Removing 5 recipes from the qemux86 sysroot: 100% |##############| Time: 0:00:00
3086
3087For this example, assume that the ``nano.bb`` recipe that
3088is upstream has a 2.9.3 version number. However, the version in the
3089local repository is 2.7.4. The following command from your build
3090directory automatically upgrades the recipe for you:
3091
3092.. note::
3093
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003094 Using the ``-V`` option is not necessary. Omitting the version number causes
3095 ``devtool upgrade`` to upgrade the recipe to the most recent version.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003096
3097::
3098
3099 $ devtool upgrade nano -V 2.9.3
3100 NOTE: Starting bitbake server...
3101 NOTE: Creating workspace layer in /home/scottrif/poky/build/workspace
3102 Parsing recipes: 100% |##########################################| Time: 0:00:46
3103 Parsing of 1431 .bb files complete (0 cached, 1431 parsed). 2040 targets, 56 skipped, 0 masked, 0 errors.
3104 NOTE: Extracting current version source...
3105 NOTE: Resolving any missing task queue dependencies
3106 .
3107 .
3108 .
3109 NOTE: Executing SetScene Tasks
3110 NOTE: Executing RunQueue Tasks
3111 NOTE: Tasks Summary: Attempted 74 tasks of which 72 didn't need to be rerun and all succeeded.
3112 Adding changed files: 100% |#####################################| Time: 0:00:00
3113 NOTE: Upgraded source extracted to /home/scottrif/poky/build/workspace/sources/nano
3114 NOTE: New recipe is /home/scottrif/poky/build/workspace/recipes/nano/nano_2.9.3.bb
3115
3116Continuing with this example, you can use ``devtool build`` to build the
Andrew Geisslerc926e172021-05-07 16:11:35 -05003117newly upgraded recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003118
3119 $ devtool build nano
3120 NOTE: Starting bitbake server...
3121 Loading cache: 100% |################################################################################################| Time: 0:00:01
3122 Loaded 2040 entries from dependency cache.
3123 Parsing recipes: 100% |##############################################################################################| Time: 0:00:00
3124 Parsing of 1432 .bb files complete (1431 cached, 1 parsed). 2041 targets, 56 skipped, 0 masked, 0 errors.
3125 NOTE: Resolving any missing task queue dependencies
3126 .
3127 .
3128 .
3129 NOTE: Executing SetScene Tasks
3130 NOTE: Executing RunQueue Tasks
3131 NOTE: nano: compiling from external source tree /home/scottrif/poky/build/workspace/sources/nano
3132 NOTE: Tasks Summary: Attempted 520 tasks of which 304 didn't need to be rerun and all succeeded.
3133
William A. Kennington IIIac69b482021-06-02 12:28:27 -07003134Within the ``devtool upgrade`` workflow, you can
3135deploy and test your rebuilt software. For this example,
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003136however, running ``devtool finish`` cleans up the workspace once the
3137source in your workspace is clean. This usually means using Git to stage
3138and submit commits for the changes generated by the upgrade process.
3139
3140Once the tree is clean, you can clean things up in this example with the
3141following command from the ``${BUILDDIR}/workspace/sources/nano``
Andrew Geisslerc926e172021-05-07 16:11:35 -05003142directory::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003143
3144 $ devtool finish nano meta-oe
3145 NOTE: Starting bitbake server...
3146 Loading cache: 100% |################################################################################################| Time: 0:00:00
3147 Loaded 2040 entries from dependency cache.
3148 Parsing recipes: 100% |##############################################################################################| Time: 0:00:01
3149 Parsing of 1432 .bb files complete (1431 cached, 1 parsed). 2041 targets, 56 skipped, 0 masked, 0 errors.
3150 NOTE: Adding new patch 0001-nano.bb-Stuff-I-changed-when-upgrading-nano.bb.patch
3151 NOTE: Updating recipe nano_2.9.3.bb
3152 NOTE: Removing file /home/scottrif/meta-openembedded/meta-oe/recipes-support/nano/nano_2.7.4.bb
3153 NOTE: Moving recipe file to /home/scottrif/meta-openembedded/meta-oe/recipes-support/nano
3154 NOTE: Leaving source tree /home/scottrif/poky/build/workspace/sources/nano as-is; if you no longer need it then please delete it manually
3155
3156
3157Using the ``devtool finish`` command cleans up the workspace and creates a patch
3158file based on your commits. The tool puts all patch files back into the
3159source directory in a sub-directory named ``nano`` in this case.
3160
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003161Manually Upgrading a Recipe
3162---------------------------
3163
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003164If for some reason you choose not to upgrade recipes using
Andrew Geissler09209ee2020-12-13 08:44:15 -06003165:ref:`dev-manual/common-tasks:Using the Auto Upgrade Helper (AUH)` or
3166by :ref:`dev-manual/common-tasks:Using \`\`devtool upgrade\`\``,
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003167you can manually edit the recipe files to upgrade the versions.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003168
3169.. note::
3170
3171 Manually updating multiple recipes scales poorly and involves many
3172 steps. The recommendation to upgrade recipe versions is through AUH
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003173 or ``devtool upgrade``, both of which automate some steps and provide
3174 guidance for others needed for the manual process.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003175
3176To manually upgrade recipe versions, follow these general steps:
3177
31781. *Change the Version:* Rename the recipe such that the version (i.e.
3179 the :term:`PV` part of the recipe name)
3180 changes appropriately. If the version is not part of the recipe name,
Andrew Geissler09036742021-06-25 14:25:14 -05003181 change the value as it is set for :term:`PV` within the recipe itself.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003182
Andrew Geissler09036742021-06-25 14:25:14 -050031832. *Update* :term:`SRCREV` *if Needed*: If the source code your recipe builds
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003184 is fetched from Git or some other version control system, update
3185 :term:`SRCREV` to point to the
3186 commit hash that matches the new version.
3187
31883. *Build the Software:* Try to build the recipe using BitBake. Typical
3189 build failures include the following:
3190
3191 - License statements were updated for the new version. For this
3192 case, you need to review any changes to the license and update the
3193 values of :term:`LICENSE` and
3194 :term:`LIC_FILES_CHKSUM`
3195 as needed.
3196
3197 .. note::
3198
3199 License changes are often inconsequential. For example, the
3200 license text's copyright year might have changed.
3201
3202 - Custom patches carried by the older version of the recipe might
3203 fail to apply to the new version. For these cases, you need to
3204 review the failures. Patches might not be necessary for the new
3205 version of the software if the upgraded version has fixed those
3206 issues. If a patch is necessary and failing, you need to rebase it
3207 into the new version.
3208
32094. *Optionally Attempt to Build for Several Architectures:* Once you
3210 successfully build the new software for a given architecture, you
3211 could test the build for other architectures by changing the
3212 :term:`MACHINE` variable and
3213 rebuilding the software. This optional step is especially important
3214 if the recipe is to be released publicly.
3215
32165. *Check the Upstream Change Log or Release Notes:* Checking both these
William A. Kennington IIIac69b482021-06-02 12:28:27 -07003217 reveals if there are new features that could break
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003218 backwards-compatibility. If so, you need to take steps to mitigate or
3219 eliminate that situation.
3220
32216. *Optionally Create a Bootable Image and Test:* If you want, you can
3222 test the new software by booting it onto actual hardware.
3223
32247. *Create a Commit with the Change in the Layer Repository:* After all
3225 builds work and any testing is successful, you can create commits for
3226 any changes in the layer holding your upgraded recipe.
3227
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003228Finding Temporary Source Code
3229=============================
3230
3231You might find it helpful during development to modify the temporary
3232source code used by recipes to build packages. For example, suppose you
3233are developing a patch and you need to experiment a bit to figure out
3234your solution. After you have initially built the package, you can
3235iteratively tweak the source code, which is located in the
3236:term:`Build Directory`, and then you can
3237force a re-compile and quickly test your altered code. Once you settle
3238on a solution, you can then preserve your changes in the form of
3239patches.
3240
3241During a build, the unpacked temporary source code used by recipes to
3242build packages is available in the Build Directory as defined by the
3243:term:`S` variable. Below is the default
Andrew Geissler09036742021-06-25 14:25:14 -05003244value for the :term:`S` variable as defined in the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003245``meta/conf/bitbake.conf`` configuration file in the
Andrew Geisslerc926e172021-05-07 16:11:35 -05003246:term:`Source Directory`::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003247
3248 S = "${WORKDIR}/${BP}"
3249
3250You should be aware that many recipes override the
Andrew Geissler09036742021-06-25 14:25:14 -05003251:term:`S` variable. For example, recipes that fetch their source from Git
3252usually set :term:`S` to ``${WORKDIR}/git``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003253
3254.. note::
3255
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003256 The :term:`BP` represents the base recipe name, which consists of the name
Andrew Geisslerc926e172021-05-07 16:11:35 -05003257 and version::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003258
3259 BP = "${BPN}-${PV}"
3260
3261
3262The path to the work directory for the recipe
3263(:term:`WORKDIR`) is defined as
Andrew Geisslerc926e172021-05-07 16:11:35 -05003264follows::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003265
3266 ${TMPDIR}/work/${MULTIMACH_TARGET_SYS}/${PN}/${EXTENDPE}${PV}-${PR}
3267
3268The actual directory depends on several things:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003269
3270- :term:`TMPDIR`: The top-level build
3271 output directory.
3272
3273- :term:`MULTIMACH_TARGET_SYS`:
3274 The target system identifier.
3275
3276- :term:`PN`: The recipe name.
3277
3278- :term:`EXTENDPE`: The epoch - (if
3279 :term:`PE` is not specified, which is
Andrew Geissler5f350902021-07-23 13:09:54 -04003280 usually the case for most recipes, then :term:`EXTENDPE` is blank).
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003281
3282- :term:`PV`: The recipe version.
3283
3284- :term:`PR`: The recipe revision.
3285
3286As an example, assume a Source Directory top-level folder named
3287``poky``, a default Build Directory at ``poky/build``, and a
3288``qemux86-poky-linux`` machine target system. Furthermore, suppose your
3289recipe is named ``foo_1.3.0.bb``. In this case, the work directory the
Andrew Geisslerc926e172021-05-07 16:11:35 -05003290build system uses to build the package would be as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003291
3292 poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0
3293
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003294Using Quilt in Your Workflow
3295============================
3296
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003297`Quilt <https://savannah.nongnu.org/projects/quilt>`__ is a powerful tool
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003298that allows you to capture source code changes without having a clean
3299source tree. This section outlines the typical workflow you can use to
3300modify source code, test changes, and then preserve the changes in the
3301form of a patch all using Quilt.
3302
3303.. note::
3304
3305 With regard to preserving changes to source files, if you clean a
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003306 recipe or have ``rm_work`` enabled, the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003307 :ref:`devtool workflow <sdk-manual/extensible:using \`\`devtool\`\` in your sdk workflow>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003308 as described in the Yocto Project Application Development and the
3309 Extensible Software Development Kit (eSDK) manual is a safer
3310 development flow than the flow that uses Quilt.
3311
3312Follow these general steps:
3313
33141. *Find the Source Code:* Temporary source code used by the
3315 OpenEmbedded build system is kept in the
3316 :term:`Build Directory`. See the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05003317 ":ref:`dev-manual/common-tasks:finding temporary source code`" section to
3318 learn how to locate the directory that has the temporary source code for a
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003319 particular package.
3320
33212. *Change Your Working Directory:* You need to be in the directory that
3322 has the temporary source code. That directory is defined by the
3323 :term:`S` variable.
3324
33253. *Create a New Patch:* Before modifying source code, you need to
3326 create a new patch. To create a new patch file, use ``quilt new`` as
Andrew Geisslerc926e172021-05-07 16:11:35 -05003327 below::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003328
3329 $ quilt new my_changes.patch
3330
33314. *Notify Quilt and Add Files:* After creating the patch, you need to
3332 notify Quilt about the files you plan to edit. You notify Quilt by
Andrew Geisslerc926e172021-05-07 16:11:35 -05003333 adding the files to the patch you just created::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003334
3335 $ quilt add file1.c file2.c file3.c
3336
33375. *Edit the Files:* Make your changes in the source code to the files
3338 you added to the patch.
3339
33406. *Test Your Changes:* Once you have modified the source code, the
3341 easiest way to test your changes is by calling the ``do_compile``
Andrew Geisslerc926e172021-05-07 16:11:35 -05003342 task as shown in the following example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003343
3344 $ bitbake -c compile -f package
3345
3346 The ``-f`` or ``--force`` option forces the specified task to
3347 execute. If you find problems with your code, you can just keep
3348 editing and re-testing iteratively until things work as expected.
3349
3350 .. note::
3351
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003352 All the modifications you make to the temporary source code disappear
3353 once you run the ``do_clean`` or ``do_cleanall`` tasks using BitBake
3354 (i.e. ``bitbake -c clean package`` and ``bitbake -c cleanall package``).
3355 Modifications will also disappear if you use the ``rm_work`` feature as
3356 described in the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003357 ":ref:`dev-manual/common-tasks:conserving disk space during builds`"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003358 section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003359
33607. *Generate the Patch:* Once your changes work as expected, you need to
3361 use Quilt to generate the final patch that contains all your
3362 modifications.
3363 ::
3364
3365 $ quilt refresh
3366
3367 At this point, the
3368 ``my_changes.patch`` file has all your edits made to the ``file1.c``,
3369 ``file2.c``, and ``file3.c`` files.
3370
3371 You can find the resulting patch file in the ``patches/``
Andrew Geissler09036742021-06-25 14:25:14 -05003372 subdirectory of the source (:term:`S`) directory.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003373
33748. *Copy the Patch File:* For simplicity, copy the patch file into a
3375 directory named ``files``, which you can create in the same directory
3376 that holds the recipe (``.bb``) file or the append (``.bbappend``)
3377 file. Placing the patch here guarantees that the OpenEmbedded build
Andrew Geissler09036742021-06-25 14:25:14 -05003378 system will find the patch. Next, add the patch into the :term:`SRC_URI`
Andrew Geisslerc926e172021-05-07 16:11:35 -05003379 of the recipe. Here is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003380
3381 SRC_URI += "file://my_changes.patch"
3382
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003383Using a Development Shell
3384=========================
3385
3386When debugging certain commands or even when just editing packages,
3387``devshell`` can be a useful tool. When you invoke ``devshell``, all
3388tasks up to and including
3389:ref:`ref-tasks-patch` are run for the
3390specified target. Then, a new terminal is opened and you are placed in
3391``${``\ :term:`S`\ ``}``, the source
3392directory. In the new terminal, all the OpenEmbedded build-related
3393environment variables are still defined so you can use commands such as
3394``configure`` and ``make``. The commands execute just as if the
3395OpenEmbedded build system were executing them. Consequently, working
3396this way can be helpful when debugging a build or preparing software to
3397be used with the OpenEmbedded build system.
3398
3399Following is an example that uses ``devshell`` on a target named
Andrew Geisslerc926e172021-05-07 16:11:35 -05003400``matchbox-desktop``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003401
3402 $ bitbake matchbox-desktop -c devshell
3403
3404This command spawns a terminal with a shell prompt within the
3405OpenEmbedded build environment. The
3406:term:`OE_TERMINAL` variable
3407controls what type of shell is opened.
3408
3409For spawned terminals, the following occurs:
3410
3411- The ``PATH`` variable includes the cross-toolchain.
3412
3413- The ``pkgconfig`` variables find the correct ``.pc`` files.
3414
3415- The ``configure`` command finds the Yocto Project site files as well
3416 as any other necessary files.
3417
3418Within this environment, you can run configure or compile commands as if
3419they were being run by the OpenEmbedded build system itself. As noted
3420earlier, the working directory also automatically changes to the Source
3421Directory (:term:`S`).
3422
3423To manually run a specific task using ``devshell``, run the
3424corresponding ``run.*`` script in the
3425``${``\ :term:`WORKDIR`\ ``}/temp``
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003426directory (e.g., ``run.do_configure.``\ `pid`). If a task's script does
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003427not exist, which would be the case if the task was skipped by way of the
3428sstate cache, you can create the task by first running it outside of the
Andrew Geisslerc926e172021-05-07 16:11:35 -05003429``devshell``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003430
3431 $ bitbake -c task
3432
3433.. note::
3434
3435 - Execution of a task's ``run.*`` script and BitBake's execution of
3436 a task are identical. In other words, running the script re-runs
3437 the task just as it would be run using the ``bitbake -c`` command.
3438
3439 - Any ``run.*`` file that does not have a ``.pid`` extension is a
3440 symbolic link (symlink) to the most recent version of that file.
3441
3442Remember, that the ``devshell`` is a mechanism that allows you to get
3443into the BitBake task execution environment. And as such, all commands
3444must be called just as BitBake would call them. That means you need to
3445provide the appropriate options for cross-compilation and so forth as
3446applicable.
3447
3448When you are finished using ``devshell``, exit the shell or close the
3449terminal window.
3450
3451.. note::
3452
3453 - It is worth remembering that when using ``devshell`` you need to
3454 use the full compiler name such as ``arm-poky-linux-gnueabi-gcc``
3455 instead of just using ``gcc``. The same applies to other
3456 applications such as ``binutils``, ``libtool`` and so forth.
Andrew Geissler09036742021-06-25 14:25:14 -05003457 BitBake sets up environment variables such as :term:`CC` to assist
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003458 applications, such as ``make`` to find the correct tools.
3459
3460 - It is also worth noting that ``devshell`` still works over X11
3461 forwarding and similar situations.
3462
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003463Using a Development Python Shell
3464================================
3465
3466Similar to working within a development shell as described in the
3467previous section, you can also spawn and work within an interactive
3468Python development shell. When debugging certain commands or even when
3469just editing packages, ``devpyshell`` can be a useful tool. When you
3470invoke ``devpyshell``, all tasks up to and including
3471:ref:`ref-tasks-patch` are run for the
3472specified target. Then a new terminal is opened. Additionally, key
3473Python objects and code are available in the same way they are to
3474BitBake tasks, in particular, the data store 'd'. So, commands such as
3475the following are useful when exploring the data store and running
Andrew Geisslerc926e172021-05-07 16:11:35 -05003476functions::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003477
3478 pydevshell> d.getVar("STAGING_DIR")
3479 '/media/build1/poky/build/tmp/sysroots'
3480 pydevshell> d.getVar("STAGING_DIR")
3481 '${TMPDIR}/sysroots'
3482 pydevshell> d.setVar("FOO", "bar")
3483 pydevshell> d.getVar("FOO")
3484 'bar'
3485 pydevshell> d.delVar("FOO")
3486 pydevshell> d.getVar("FOO")
3487 pydevshell> bb.build.exec_func("do_unpack", d)
3488 pydevshell>
3489
3490The commands execute just as if the OpenEmbedded build
3491system were executing them. Consequently, working this way can be
3492helpful when debugging a build or preparing software to be used with the
3493OpenEmbedded build system.
3494
3495Following is an example that uses ``devpyshell`` on a target named
Andrew Geisslerc926e172021-05-07 16:11:35 -05003496``matchbox-desktop``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003497
3498 $ bitbake matchbox-desktop -c devpyshell
3499
3500This command spawns a terminal and places you in an interactive Python
3501interpreter within the OpenEmbedded build environment. The
3502:term:`OE_TERMINAL` variable
3503controls what type of shell is opened.
3504
3505When you are finished using ``devpyshell``, you can exit the shell
3506either by using Ctrl+d or closing the terminal window.
3507
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003508Building
3509========
3510
3511This section describes various build procedures. For example, the steps
3512needed for a simple build, a target that uses multiple configurations,
3513building an image for more than one machine, and so forth.
3514
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003515Building a Simple Image
3516-----------------------
3517
3518In the development environment, you need to build an image whenever you
3519change hardware support, add or change system libraries, or add or
William A. Kennington IIIac69b482021-06-02 12:28:27 -07003520change services that have dependencies. There are several methods that allow
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003521you to build an image within the Yocto Project. This section presents
3522the basic steps you need to build a simple image using BitBake from a
3523build host running Linux.
3524
3525.. note::
3526
3527 - For information on how to build an image using
3528 :term:`Toaster`, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003529 :doc:`/toaster-manual/index`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003530
3531 - For information on how to use ``devtool`` to build images, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003532 ":ref:`sdk-manual/extensible:using \`\`devtool\`\` in your sdk workflow`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003533 section in the Yocto Project Application Development and the
3534 Extensible Software Development Kit (eSDK) manual.
3535
3536 - For a quick example on how to build an image using the
3537 OpenEmbedded build system, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003538 :doc:`/brief-yoctoprojectqs/index` document.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003539
3540The build process creates an entire Linux distribution from source and
3541places it in your :term:`Build Directory` under
3542``tmp/deploy/images``. For detailed information on the build process
Andrew Geissler09209ee2020-12-13 08:44:15 -06003543using BitBake, see the ":ref:`overview-manual/concepts:images`" section in the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003544Yocto Project Overview and Concepts Manual.
3545
3546The following figure and list overviews the build process:
3547
3548.. image:: figures/bitbake-build-flow.png
3549 :align: center
3550
35511. *Set up Your Host Development System to Support Development Using the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003552 Yocto Project*: See the ":doc:`start`" section for options on how to get a
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003553 build host ready to use the Yocto Project.
3554
35552. *Initialize the Build Environment:* Initialize the build environment
3556 by sourcing the build environment script (i.e.
Andrew Geisslerc926e172021-05-07 16:11:35 -05003557 :ref:`structure-core-script`)::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003558
3559 $ source oe-init-build-env [build_dir]
3560
3561 When you use the initialization script, the OpenEmbedded build system
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003562 uses ``build`` as the default :term:`Build Directory` in your current work
3563 directory. You can use a `build_dir` argument with the script to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003564 specify a different build directory.
3565
3566 .. note::
3567
3568 A common practice is to use a different Build Directory for
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003569 different targets. For example, ``~/build/x86`` for a ``qemux86``
3570 target, and ``~/build/arm`` for a ``qemuarm`` target.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003571
Andrew Geissler4c19ea12020-10-27 13:52:24 -050035723. *Make Sure Your* ``local.conf`` *File is Correct*: Ensure the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003573 ``conf/local.conf`` configuration file, which is found in the Build
3574 Directory, is set up how you want it. This file defines many aspects
3575 of the build environment including the target machine architecture
Andrew Geissler09036742021-06-25 14:25:14 -05003576 through the :term:`MACHINE` variable, the packaging format used during
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003577 the build
3578 (:term:`PACKAGE_CLASSES`),
3579 and a centralized tarball download directory through the
3580 :term:`DL_DIR` variable.
3581
Andrew Geisslerc926e172021-05-07 16:11:35 -050035824. *Build the Image:* Build the image using the ``bitbake`` command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003583
3584 $ bitbake target
3585
3586 .. note::
3587
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003588 For information on BitBake, see the :doc:`bitbake:index`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003589
3590 The target is the name of the recipe you want to build. Common
3591 targets are the images in ``meta/recipes-core/images``,
3592 ``meta/recipes-sato/images``, and so forth all found in the
3593 :term:`Source Directory`. Or, the target
3594 can be the name of a recipe for a specific piece of software such as
3595 BusyBox. For more details about the images the OpenEmbedded build
3596 system supports, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003597 ":ref:`ref-manual/images:Images`" chapter in the Yocto
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003598 Project Reference Manual.
3599
3600 As an example, the following command builds the
Andrew Geisslerc926e172021-05-07 16:11:35 -05003601 ``core-image-minimal`` image::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003602
3603 $ bitbake core-image-minimal
3604
3605 Once an
3606 image has been built, it often needs to be installed. The images and
3607 kernels built by the OpenEmbedded build system are placed in the
3608 Build Directory in ``tmp/deploy/images``. For information on how to
3609 run pre-built images such as ``qemux86`` and ``qemuarm``, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003610 :doc:`/sdk-manual/index` manual. For
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003611 information about how to install these images, see the documentation
3612 for your particular board or machine.
3613
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003614Building Images for Multiple Targets Using Multiple Configurations
3615------------------------------------------------------------------
3616
3617You can use a single ``bitbake`` command to build multiple images or
3618packages for different targets where each image or package requires a
3619different configuration (multiple configuration builds). The builds, in
3620this scenario, are sometimes referred to as "multiconfigs", and this
3621section uses that term throughout.
3622
3623This section describes how to set up for multiple configuration builds
3624and how to account for cross-build dependencies between the
3625multiconfigs.
3626
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003627Setting Up and Running a Multiple Configuration Build
3628~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3629
3630To accomplish a multiple configuration build, you must define each
3631target's configuration separately using a parallel configuration file in
3632the :term:`Build Directory`, and you
3633must follow a required file hierarchy. Additionally, you must enable the
3634multiple configuration builds in your ``local.conf`` file.
3635
3636Follow these steps to set up and execute multiple configuration builds:
3637
3638- *Create Separate Configuration Files*: You need to create a single
3639 configuration file for each build target (each multiconfig).
3640 Minimally, each configuration file must define the machine and the
3641 temporary directory BitBake uses for the build. Suggested practice
3642 dictates that you do not overlap the temporary directories used
3643 during the builds. However, it is possible that you can share the
3644 temporary directory
3645 (:term:`TMPDIR`). For example,
3646 consider a scenario with two different multiconfigs for the same
3647 :term:`MACHINE`: "qemux86" built
3648 for two distributions such as "poky" and "poky-lsb". In this case,
Andrew Geissler09036742021-06-25 14:25:14 -05003649 you might want to use the same :term:`TMPDIR`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003650
3651 Here is an example showing the minimal statements needed in a
3652 configuration file for a "qemux86" target whose temporary build
Andrew Geisslerc926e172021-05-07 16:11:35 -05003653 directory is ``tmpmultix86``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003654
3655 MACHINE = "qemux86"
3656 TMPDIR = "${TOPDIR}/tmpmultix86"
3657
3658 The location for these multiconfig configuration files is specific.
3659 They must reside in the current build directory in a sub-directory of
3660 ``conf`` named ``multiconfig``. Following is an example that defines
3661 two configuration files for the "x86" and "arm" multiconfigs:
3662
3663 .. image:: figures/multiconfig_files.png
3664 :align: center
3665
Andrew Geissler09036742021-06-25 14:25:14 -05003666 The reason for this required file hierarchy is because the :term:`BBPATH`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003667 variable is not constructed until the layers are parsed.
3668 Consequently, using the configuration file as a pre-configuration
3669 file is not possible unless it is located in the current working
3670 directory.
3671
3672- *Add the BitBake Multi-configuration Variable to the Local
3673 Configuration File*: Use the
3674 :term:`BBMULTICONFIG`
3675 variable in your ``conf/local.conf`` configuration file to specify
3676 each multiconfig. Continuing with the example from the previous
Andrew Geissler09036742021-06-25 14:25:14 -05003677 figure, the :term:`BBMULTICONFIG` variable needs to enable two
Andrew Geisslerc926e172021-05-07 16:11:35 -05003678 multiconfigs: "x86" and "arm" by specifying each configuration file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003679
3680 BBMULTICONFIG = "x86 arm"
3681
3682 .. note::
3683
3684 A "default" configuration already exists by definition. This
3685 configuration is named: "" (i.e. empty string) and is defined by
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003686 the variables coming from your ``local.conf``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003687 file. Consequently, the previous example actually adds two
3688 additional configurations to your build: "arm" and "x86" along
3689 with "".
3690
3691- *Launch BitBake*: Use the following BitBake command form to launch
Andrew Geisslerc926e172021-05-07 16:11:35 -05003692 the multiple configuration build::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003693
3694 $ bitbake [mc:multiconfigname:]target [[[mc:multiconfigname:]target] ... ]
3695
Andrew Geisslerc926e172021-05-07 16:11:35 -05003696 For the example in this section, the following command applies::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003697
3698 $ bitbake mc:x86:core-image-minimal mc:arm:core-image-sato mc::core-image-base
3699
3700 The previous BitBake command builds a ``core-image-minimal`` image
3701 that is configured through the ``x86.conf`` configuration file, a
3702 ``core-image-sato`` image that is configured through the ``arm.conf``
3703 configuration file and a ``core-image-base`` that is configured
3704 through your ``local.conf`` configuration file.
3705
3706.. note::
3707
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003708 Support for multiple configuration builds in the Yocto Project &DISTRO;
3709 (&DISTRO_NAME;) Release does not include Shared State (sstate)
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003710 optimizations. Consequently, if a build uses the same object twice
Andrew Geissler09036742021-06-25 14:25:14 -05003711 in, for example, two different :term:`TMPDIR`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003712 directories, the build either loads from an existing sstate cache for
3713 that build at the start or builds the object fresh.
3714
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003715Enabling Multiple Configuration Build Dependencies
3716~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3717
3718Sometimes dependencies can exist between targets (multiconfigs) in a
3719multiple configuration build. For example, suppose that in order to
3720build a ``core-image-sato`` image for an "x86" multiconfig, the root
3721filesystem of an "arm" multiconfig must exist. This dependency is
3722essentially that the
3723:ref:`ref-tasks-image` task in the
3724``core-image-sato`` recipe depends on the completion of the
3725:ref:`ref-tasks-rootfs` task of the
3726``core-image-minimal`` recipe.
3727
3728To enable dependencies in a multiple configuration build, you must
3729declare the dependencies in the recipe using the following statement
Andrew Geisslerc926e172021-05-07 16:11:35 -05003730form::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003731
3732 task_or_package[mcdepends] = "mc:from_multiconfig:to_multiconfig:recipe_name:task_on_which_to_depend"
3733
3734To better show how to use this statement, consider the example scenario
3735from the first paragraph of this section. The following statement needs
Andrew Geisslerc926e172021-05-07 16:11:35 -05003736to be added to the recipe that builds the ``core-image-sato`` image::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003737
3738 do_image[mcdepends] = "mc:x86:arm:core-image-minimal:do_rootfs"
3739
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003740In this example, the `from_multiconfig` is "x86". The `to_multiconfig` is "arm". The
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003741task on which the ``do_image`` task in the recipe depends is the
3742``do_rootfs`` task from the ``core-image-minimal`` recipe associated
3743with the "arm" multiconfig.
3744
3745Once you set up this dependency, you can build the "x86" multiconfig
Andrew Geisslerc926e172021-05-07 16:11:35 -05003746using a BitBake command as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003747
3748 $ bitbake mc:x86:core-image-sato
3749
3750This command executes all the tasks needed to create the
3751``core-image-sato`` image for the "x86" multiconfig. Because of the
3752dependency, BitBake also executes through the ``do_rootfs`` task for the
3753"arm" multiconfig build.
3754
3755Having a recipe depend on the root filesystem of another build might not
3756seem that useful. Consider this change to the statement in the
Andrew Geisslerc926e172021-05-07 16:11:35 -05003757``core-image-sato`` recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003758
3759 do_image[mcdepends] = "mc:x86:arm:core-image-minimal:do_image"
3760
3761In this case, BitBake must
3762create the ``core-image-minimal`` image for the "arm" build since the
3763"x86" build depends on it.
3764
3765Because "x86" and "arm" are enabled for multiple configuration builds
3766and have separate configuration files, BitBake places the artifacts for
3767each build in the respective temporary build directories (i.e.
3768:term:`TMPDIR`).
3769
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003770Building an Initial RAM Filesystem (initramfs) Image
3771----------------------------------------------------
3772
3773An initial RAM filesystem (initramfs) image provides a temporary root
3774filesystem used for early system initialization (e.g. loading of modules
3775needed to locate and mount the "real" root filesystem).
3776
3777.. note::
3778
3779 The initramfs image is the successor of initial RAM disk (initrd). It
3780 is a "copy in and out" (cpio) archive of the initial filesystem that
3781 gets loaded into memory during the Linux startup process. Because
3782 Linux uses the contents of the archive during initialization, the
3783 initramfs image needs to contain all of the device drivers and tools
3784 needed to mount the final root filesystem.
3785
3786Follow these steps to create an initramfs image:
3787
37881. *Create the initramfs Image Recipe:* You can reference the
3789 ``core-image-minimal-initramfs.bb`` recipe found in the
3790 ``meta/recipes-core`` directory of the :term:`Source Directory`
3791 as an example
3792 from which to work.
3793
37942. *Decide if You Need to Bundle the initramfs Image Into the Kernel
3795 Image:* If you want the initramfs image that is built to be bundled
3796 in with the kernel image, set the
3797 :term:`INITRAMFS_IMAGE_BUNDLE`
3798 variable to "1" in your ``local.conf`` configuration file and set the
3799 :term:`INITRAMFS_IMAGE`
3800 variable in the recipe that builds the kernel image.
3801
3802 .. note::
3803
3804 It is recommended that you do bundle the initramfs image with the
3805 kernel image to avoid circular dependencies between the kernel
3806 recipe and the initramfs recipe should the initramfs image include
3807 kernel modules.
3808
Andrew Geissler09036742021-06-25 14:25:14 -05003809 Setting the :term:`INITRAMFS_IMAGE_BUNDLE` flag causes the initramfs
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003810 image to be unpacked into the ``${B}/usr/`` directory. The unpacked
3811 initramfs image is then passed to the kernel's ``Makefile`` using the
3812 :term:`CONFIG_INITRAMFS_SOURCE`
3813 variable, allowing the initramfs image to be built into the kernel
3814 normally.
3815
3816 .. note::
3817
3818 If you choose to not bundle the initramfs image with the kernel
3819 image, you are essentially using an
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003820 `Initial RAM Disk (initrd) <https://en.wikipedia.org/wiki/Initrd>`__.
3821 Creating an initrd is handled primarily through the :term:`INITRD_IMAGE`,
3822 ``INITRD_LIVE``, and ``INITRD_IMAGE_LIVE`` variables. For more
3823 information, see the :ref:`ref-classes-image-live` file.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003824
38253. *Optionally Add Items to the initramfs Image Through the initramfs
3826 Image Recipe:* If you add items to the initramfs image by way of its
3827 recipe, you should use
3828 :term:`PACKAGE_INSTALL`
3829 rather than
3830 :term:`IMAGE_INSTALL`.
Andrew Geissler09036742021-06-25 14:25:14 -05003831 :term:`PACKAGE_INSTALL` gives more direct control of what is added to the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003832 image as compared to the defaults you might not necessarily want that
3833 are set by the :ref:`image <ref-classes-image>`
3834 or :ref:`core-image <ref-classes-core-image>`
3835 classes.
3836
38374. *Build the Kernel Image and the initramfs Image:* Build your kernel
3838 image using BitBake. Because the initramfs image recipe is a
3839 dependency of the kernel image, the initramfs image is built as well
3840 and bundled with the kernel image if you used the
3841 :term:`INITRAMFS_IMAGE_BUNDLE`
3842 variable described earlier.
3843
3844Building a Tiny System
3845----------------------
3846
3847Very small distributions have some significant advantages such as
3848requiring less on-die or in-package memory (cheaper), better performance
3849through efficient cache usage, lower power requirements due to less
3850memory, faster boot times, and reduced development overhead. Some
3851real-world examples where a very small distribution gives you distinct
3852advantages are digital cameras, medical devices, and small headless
3853systems.
3854
3855This section presents information that shows you how you can trim your
3856distribution to even smaller sizes than the ``poky-tiny`` distribution,
3857which is around 5 Mbytes, that can be built out-of-the-box using the
3858Yocto Project.
3859
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003860Tiny System Overview
3861~~~~~~~~~~~~~~~~~~~~
3862
3863The following list presents the overall steps you need to consider and
3864perform to create distributions with smaller root filesystems, achieve
3865faster boot times, maintain your critical functionality, and avoid
3866initial RAM disks:
3867
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05003868- :ref:`Determine your goals and guiding principles
3869 <dev-manual/common-tasks:goals and guiding principles>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003870
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05003871- :ref:`dev-manual/common-tasks:understand what contributes to your image size`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003872
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05003873- :ref:`Reduce the size of the root filesystem
3874 <dev-manual/common-tasks:trim the root filesystem>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003875
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05003876- :ref:`Reduce the size of the kernel <dev-manual/common-tasks:trim the kernel>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003877
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05003878- :ref:`dev-manual/common-tasks:remove package management requirements`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003879
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05003880- :ref:`dev-manual/common-tasks:look for other ways to minimize size`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003881
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05003882- :ref:`dev-manual/common-tasks:iterate on the process`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003883
3884Goals and Guiding Principles
3885~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3886
3887Before you can reach your destination, you need to know where you are
3888going. Here is an example list that you can use as a guide when creating
3889very small distributions:
3890
3891- Determine how much space you need (e.g. a kernel that is 1 Mbyte or
3892 less and a root filesystem that is 3 Mbytes or less).
3893
3894- Find the areas that are currently taking 90% of the space and
3895 concentrate on reducing those areas.
3896
3897- Do not create any difficult "hacks" to achieve your goals.
3898
3899- Leverage the device-specific options.
3900
3901- Work in a separate layer so that you keep changes isolated. For
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05003902 information on how to create layers, see the
3903 ":ref:`dev-manual/common-tasks:understanding and creating layers`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003904
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003905Understand What Contributes to Your Image Size
3906~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3907
3908It is easiest to have something to start with when creating your own
3909distribution. You can use the Yocto Project out-of-the-box to create the
3910``poky-tiny`` distribution. Ultimately, you will want to make changes in
3911your own distribution that are likely modeled after ``poky-tiny``.
3912
3913.. note::
3914
Andrew Geissler09036742021-06-25 14:25:14 -05003915 To use ``poky-tiny`` in your build, set the :term:`DISTRO` variable in your
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003916 ``local.conf`` file to "poky-tiny" as described in the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003917 ":ref:`dev-manual/common-tasks:creating your own distribution`"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003918 section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003919
3920Understanding some memory concepts will help you reduce the system size.
3921Memory consists of static, dynamic, and temporary memory. Static memory
3922is the TEXT (code), DATA (initialized data in the code), and BSS
3923(uninitialized data) sections. Dynamic memory represents memory that is
3924allocated at runtime: stacks, hash tables, and so forth. Temporary
3925memory is recovered after the boot process. This memory consists of
3926memory used for decompressing the kernel and for the ``__init__``
3927functions.
3928
3929To help you see where you currently are with kernel and root filesystem
3930sizes, you can use two tools found in the :term:`Source Directory`
3931in the
3932``scripts/tiny/`` directory:
3933
3934- ``ksize.py``: Reports component sizes for the kernel build objects.
3935
3936- ``dirsize.py``: Reports component sizes for the root filesystem.
3937
3938This next tool and command help you organize configuration fragments and
3939view file dependencies in a human-readable form:
3940
3941- ``merge_config.sh``: Helps you manage configuration files and
3942 fragments within the kernel. With this tool, you can merge individual
3943 configuration fragments together. The tool allows you to make
3944 overrides and warns you of any missing configuration options. The
3945 tool is ideal for allowing you to iterate on configurations, create
3946 minimal configurations, and create configuration files for different
3947 machines without having to duplicate your process.
3948
3949 The ``merge_config.sh`` script is part of the Linux Yocto kernel Git
3950 repositories (i.e. ``linux-yocto-3.14``, ``linux-yocto-3.10``,
3951 ``linux-yocto-3.8``, and so forth) in the ``scripts/kconfig``
3952 directory.
3953
3954 For more information on configuration fragments, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003955 ":ref:`kernel-dev/common:creating configuration fragments`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003956 section in the Yocto Project Linux Kernel Development Manual.
3957
3958- ``bitbake -u taskexp -g bitbake_target``: Using the BitBake command
3959 with these options brings up a Dependency Explorer from which you can
3960 view file dependencies. Understanding these dependencies allows you
3961 to make informed decisions when cutting out various pieces of the
3962 kernel and root filesystem.
3963
3964Trim the Root Filesystem
3965~~~~~~~~~~~~~~~~~~~~~~~~
3966
3967The root filesystem is made up of packages for booting, libraries, and
3968applications. To change things, you can configure how the packaging
3969happens, which changes the way you build them. You can also modify the
3970filesystem itself or select a different filesystem.
3971
3972First, find out what is hogging your root filesystem by running the
Andrew Geisslerc926e172021-05-07 16:11:35 -05003973``dirsize.py`` script from your root directory::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003974
3975 $ cd root-directory-of-image
3976 $ dirsize.py 100000 > dirsize-100k.log
3977 $ cat dirsize-100k.log
3978
3979You can apply a filter to the script to ignore files
3980under a certain size. The previous example filters out any files below
3981100 Kbytes. The sizes reported by the tool are uncompressed, and thus
3982will be smaller by a relatively constant factor in a compressed root
3983filesystem. When you examine your log file, you can focus on areas of
3984the root filesystem that take up large amounts of memory.
3985
3986You need to be sure that what you eliminate does not cripple the
3987functionality you need. One way to see how packages relate to each other
Andrew Geisslerc926e172021-05-07 16:11:35 -05003988is by using the Dependency Explorer UI with the BitBake command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003989
3990 $ cd image-directory
3991 $ bitbake -u taskexp -g image
3992
3993Use the interface to
3994select potential packages you wish to eliminate and see their dependency
3995relationships.
3996
3997When deciding how to reduce the size, get rid of packages that result in
3998minimal impact on the feature set. For example, you might not need a VGA
3999display. Or, you might be able to get by with ``devtmpfs`` and ``mdev``
4000instead of ``udev``.
4001
4002Use your ``local.conf`` file to make changes. For example, to eliminate
4003``udev`` and ``glib``, set the following in the local configuration
Andrew Geisslerc926e172021-05-07 16:11:35 -05004004file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004005
4006 VIRTUAL-RUNTIME_dev_manager = ""
4007
4008Finally, you should consider exactly the type of root filesystem you
4009need to meet your needs while also reducing its size. For example,
4010consider ``cramfs``, ``squashfs``, ``ubifs``, ``ext2``, or an
4011``initramfs`` using ``initramfs``. Be aware that ``ext3`` requires a 1
4012Mbyte journal. If you are okay with running read-only, you do not need
4013this journal.
4014
4015.. note::
4016
4017 After each round of elimination, you need to rebuild your system and
4018 then use the tools to see the effects of your reductions.
4019
4020Trim the Kernel
4021~~~~~~~~~~~~~~~
4022
4023The kernel is built by including policies for hardware-independent
4024aspects. What subsystems do you enable? For what architecture are you
4025building? Which drivers do you build by default?
4026
4027.. note::
4028
4029 You can modify the kernel source if you want to help with boot time.
4030
4031Run the ``ksize.py`` script from the top-level Linux build directory to
Andrew Geisslerc926e172021-05-07 16:11:35 -05004032get an idea of what is making up the kernel::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004033
4034 $ cd top-level-linux-build-directory
4035 $ ksize.py > ksize.log
4036 $ cat ksize.log
4037
4038When you examine the log, you will see how much space is taken up with
4039the built-in ``.o`` files for drivers, networking, core kernel files,
4040filesystem, sound, and so forth. The sizes reported by the tool are
4041uncompressed, and thus will be smaller by a relatively constant factor
4042in a compressed kernel image. Look to reduce the areas that are large
4043and taking up around the "90% rule."
4044
4045To examine, or drill down, into any particular area, use the ``-d``
Andrew Geisslerc926e172021-05-07 16:11:35 -05004046option with the script::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004047
4048 $ ksize.py -d > ksize.log
4049
4050Using this option
4051breaks out the individual file information for each area of the kernel
4052(e.g. drivers, networking, and so forth).
4053
4054Use your log file to see what you can eliminate from the kernel based on
4055features you can let go. For example, if you are not going to need
4056sound, you do not need any drivers that support sound.
4057
4058After figuring out what to eliminate, you need to reconfigure the kernel
4059to reflect those changes during the next build. You could run
4060``menuconfig`` and make all your changes at once. However, that makes it
4061difficult to see the effects of your individual eliminations and also
4062makes it difficult to replicate the changes for perhaps another target
4063device. A better method is to start with no configurations using
4064``allnoconfig``, create configuration fragments for individual changes,
4065and then manage the fragments into a single configuration file using
4066``merge_config.sh``. The tool makes it easy for you to iterate using the
4067configuration change and build cycle.
4068
4069Each time you make configuration changes, you need to rebuild the kernel
4070and check to see what impact your changes had on the overall size.
4071
4072Remove Package Management Requirements
4073~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4074
4075Packaging requirements add size to the image. One way to reduce the size
4076of the image is to remove all the packaging requirements from the image.
4077This reduction includes both removing the package manager and its unique
4078dependencies as well as removing the package management data itself.
4079
4080To eliminate all the packaging requirements for an image, be sure that
4081"package-management" is not part of your
4082:term:`IMAGE_FEATURES`
4083statement for the image. When you remove this feature, you are removing
4084the package manager as well as its dependencies from the root
4085filesystem.
4086
4087Look for Other Ways to Minimize Size
4088~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4089
4090Depending on your particular circumstances, other areas that you can
4091trim likely exist. The key to finding these areas is through tools and
4092methods described here combined with experimentation and iteration. Here
4093are a couple of areas to experiment with:
4094
4095- ``glibc``: In general, follow this process:
4096
4097 1. Remove ``glibc`` features from
4098 :term:`DISTRO_FEATURES`
4099 that you think you do not need.
4100
4101 2. Build your distribution.
4102
4103 3. If the build fails due to missing symbols in a package, determine
4104 if you can reconfigure the package to not need those features. For
4105 example, change the configuration to not support wide character
4106 support as is done for ``ncurses``. Or, if support for those
4107 characters is needed, determine what ``glibc`` features provide
4108 the support and restore the configuration.
4109
4110 4. Rebuild and repeat the process.
4111
4112- ``busybox``: For BusyBox, use a process similar as described for
4113 ``glibc``. A difference is you will need to boot the resulting system
4114 to see if you are able to do everything you expect from the running
4115 system. You need to be sure to integrate configuration fragments into
4116 Busybox because BusyBox handles its own core features and then allows
4117 you to add configuration fragments on top.
4118
4119Iterate on the Process
4120~~~~~~~~~~~~~~~~~~~~~~
4121
4122If you have not reached your goals on system size, you need to iterate
4123on the process. The process is the same. Use the tools and see just what
4124is taking up 90% of the root filesystem and the kernel. Decide what you
4125can eliminate without limiting your device beyond what you need.
4126
4127Depending on your system, a good place to look might be Busybox, which
4128provides a stripped down version of Unix tools in a single, executable
4129file. You might be able to drop virtual terminal services or perhaps
4130ipv6.
4131
4132Building Images for More than One Machine
4133-----------------------------------------
4134
4135A common scenario developers face is creating images for several
4136different machines that use the same software environment. In this
4137situation, it is tempting to set the tunings and optimization flags for
4138each build specifically for the targeted hardware (i.e. "maxing out" the
4139tunings). Doing so can considerably add to build times and package feed
4140maintenance collectively for the machines. For example, selecting tunes
4141that are extremely specific to a CPU core used in a system might enable
4142some micro optimizations in GCC for that particular system but would
4143otherwise not gain you much of a performance difference across the other
4144systems as compared to using a more general tuning across all the builds
4145(e.g. setting :term:`DEFAULTTUNE`
4146specifically for each machine's build). Rather than "max out" each
4147build's tunings, you can take steps that cause the OpenEmbedded build
4148system to reuse software across the various machines where it makes
4149sense.
4150
4151If build speed and package feed maintenance are considerations, you
4152should consider the points in this section that can help you optimize
4153your tunings to best consider build times and package feed maintenance.
4154
4155- *Share the Build Directory:* If at all possible, share the
4156 :term:`TMPDIR` across builds. The
4157 Yocto Project supports switching between different
4158 :term:`MACHINE` values in the same
Andrew Geissler09036742021-06-25 14:25:14 -05004159 :term:`TMPDIR`. This practice is well supported and regularly used by
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004160 developers when building for multiple machines. When you use the same
Andrew Geissler09036742021-06-25 14:25:14 -05004161 :term:`TMPDIR` for multiple machine builds, the OpenEmbedded build system
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004162 can reuse the existing native and often cross-recipes for multiple
4163 machines. Thus, build time decreases.
4164
4165 .. note::
4166
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004167 If :term:`DISTRO` settings change or fundamental configuration settings
Andrew Geissler09036742021-06-25 14:25:14 -05004168 such as the filesystem layout, you need to work with a clean :term:`TMPDIR`.
4169 Sharing :term:`TMPDIR` under these circumstances might work but since it is
Andrew Geissler5f350902021-07-23 13:09:54 -04004170 not guaranteed, you should use a clean :term:`TMPDIR`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004171
4172- *Enable the Appropriate Package Architecture:* By default, the
4173 OpenEmbedded build system enables three levels of package
4174 architectures: "all", "tune" or "package", and "machine". Any given
4175 recipe usually selects one of these package architectures (types) for
4176 its output. Depending for what a given recipe creates packages,
4177 making sure you enable the appropriate package architecture can
4178 directly impact the build time.
4179
4180 A recipe that just generates scripts can enable "all" architecture
4181 because there are no binaries to build. To specifically enable "all"
4182 architecture, be sure your recipe inherits the
4183 :ref:`allarch <ref-classes-allarch>` class.
4184 This class is useful for "all" architectures because it configures
4185 many variables so packages can be used across multiple architectures.
4186
4187 If your recipe needs to generate packages that are machine-specific
4188 or when one of the build or runtime dependencies is already
4189 machine-architecture dependent, which makes your recipe also
4190 machine-architecture dependent, make sure your recipe enables the
4191 "machine" package architecture through the
4192 :term:`MACHINE_ARCH`
Andrew Geisslerc926e172021-05-07 16:11:35 -05004193 variable::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004194
4195 PACKAGE_ARCH = "${MACHINE_ARCH}"
4196
4197 When you do not
4198 specifically enable a package architecture through the
4199 :term:`PACKAGE_ARCH`, The
4200 OpenEmbedded build system defaults to the
Andrew Geisslerc926e172021-05-07 16:11:35 -05004201 :term:`TUNE_PKGARCH` setting::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004202
4203 PACKAGE_ARCH = "${TUNE_PKGARCH}"
4204
4205- *Choose a Generic Tuning File if Possible:* Some tunes are more
4206 generic and can run on multiple targets (e.g. an ``armv5`` set of
4207 packages could run on ``armv6`` and ``armv7`` processors in most
4208 cases). Similarly, ``i486`` binaries could work on ``i586`` and
4209 higher processors. You should realize, however, that advances on
4210 newer processor versions would not be used.
4211
4212 If you select the same tune for several different machines, the
4213 OpenEmbedded build system reuses software previously built, thus
4214 speeding up the overall build time. Realize that even though a new
4215 sysroot for each machine is generated, the software is not recompiled
4216 and only one package feed exists.
4217
William A. Kennington IIIac69b482021-06-02 12:28:27 -07004218- *Manage Granular Level Packaging:* Sometimes there are cases where
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004219 injecting another level of package architecture beyond the three
4220 higher levels noted earlier can be useful. For example, consider how
4221 NXP (formerly Freescale) allows for the easy reuse of binary packages
4222 in their layer
Andrew Geissler09209ee2020-12-13 08:44:15 -06004223 :yocto_git:`meta-freescale </meta-freescale/>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004224 In this example, the
Andrew Geissler09209ee2020-12-13 08:44:15 -06004225 :yocto_git:`fsl-dynamic-packagearch </meta-freescale/tree/classes/fsl-dynamic-packagearch.bbclass>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004226 class shares GPU packages for i.MX53 boards because all boards share
4227 the AMD GPU. The i.MX6-based boards can do the same because all
4228 boards share the Vivante GPU. This class inspects the BitBake
4229 datastore to identify if the package provides or depends on one of
4230 the sub-architecture values. If so, the class sets the
4231 :term:`PACKAGE_ARCH` value
4232 based on the ``MACHINE_SUBARCH`` value. If the package does not
4233 provide or depend on one of the sub-architecture values but it
4234 matches a value in the machine-specific filter, it sets
4235 :term:`MACHINE_ARCH`. This
4236 behavior reduces the number of packages built and saves build time by
4237 reusing binaries.
4238
4239- *Use Tools to Debug Issues:* Sometimes you can run into situations
4240 where software is being rebuilt when you think it should not be. For
4241 example, the OpenEmbedded build system might not be using shared
4242 state between machines when you think it should be. These types of
4243 situations are usually due to references to machine-specific
4244 variables such as :term:`MACHINE`,
4245 :term:`SERIAL_CONSOLES`,
4246 :term:`XSERVER`,
4247 :term:`MACHINE_FEATURES`,
4248 and so forth in code that is supposed to only be tune-specific or
4249 when the recipe depends
4250 (:term:`DEPENDS`,
4251 :term:`RDEPENDS`,
4252 :term:`RRECOMMENDS`,
4253 :term:`RSUGGESTS`, and so forth)
4254 on some other recipe that already has
4255 :term:`PACKAGE_ARCH` defined
4256 as "${MACHINE_ARCH}".
4257
4258 .. note::
4259
4260 Patches to fix any issues identified are most welcome as these
4261 issues occasionally do occur.
4262
4263 For such cases, you can use some tools to help you sort out the
4264 situation:
4265
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004266 - ``state-diff-machines.sh``*:* You can find this tool in the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004267 ``scripts`` directory of the Source Repositories. See the comments
4268 in the script for information on how to use the tool.
4269
4270 - *BitBake's "-S printdiff" Option:* Using this option causes
4271 BitBake to try to establish the closest signature match it can
4272 (e.g. in the shared state cache) and then run ``bitbake-diffsigs``
4273 over the matches to determine the stamps and delta where these two
4274 stamp trees diverge.
4275
4276Building Software from an External Source
4277-----------------------------------------
4278
4279By default, the OpenEmbedded build system uses the
4280:term:`Build Directory` when building source
4281code. The build process involves fetching the source files, unpacking
4282them, and then patching them if necessary before the build takes place.
4283
William A. Kennington IIIac69b482021-06-02 12:28:27 -07004284There are situations where you might want to build software from source
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004285files that are external to and thus outside of the OpenEmbedded build
4286system. For example, suppose you have a project that includes a new BSP
4287with a heavily customized kernel. And, you want to minimize exposing the
4288build system to the development team so that they can focus on their
4289project and maintain everyone's workflow as much as possible. In this
4290case, you want a kernel source directory on the development machine
4291where the development occurs. You want the recipe's
4292:term:`SRC_URI` variable to point to
4293the external directory and use it as is, not copy it.
4294
4295To build from software that comes from an external source, all you need
4296to do is inherit the
4297:ref:`externalsrc <ref-classes-externalsrc>` class
4298and then set the
4299:term:`EXTERNALSRC` variable to
4300point to your external source code. Here are the statements to put in
Andrew Geisslerc926e172021-05-07 16:11:35 -05004301your ``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004302
4303 INHERIT += "externalsrc"
4304 EXTERNALSRC_pn-myrecipe = "path-to-your-source-tree"
4305
4306This next example shows how to accomplish the same thing by setting
Andrew Geissler09036742021-06-25 14:25:14 -05004307:term:`EXTERNALSRC` in the recipe itself or in the recipe's append file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004308
4309 EXTERNALSRC = "path"
4310 EXTERNALSRC_BUILD = "path"
4311
4312.. note::
4313
4314 In order for these settings to take effect, you must globally or
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004315 locally inherit the :ref:`externalsrc <ref-classes-externalsrc>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004316 class.
4317
4318By default, ``externalsrc.bbclass`` builds the source code in a
4319directory separate from the external source directory as specified by
4320:term:`EXTERNALSRC`. If you need
4321to have the source built in the same directory in which it resides, or
4322some other nominated directory, you can set
4323:term:`EXTERNALSRC_BUILD`
Andrew Geisslerc926e172021-05-07 16:11:35 -05004324to point to that directory::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004325
4326 EXTERNALSRC_BUILD_pn-myrecipe = "path-to-your-source-tree"
4327
4328Replicating a Build Offline
4329---------------------------
4330
4331It can be useful to take a "snapshot" of upstream sources used in a
4332build and then use that "snapshot" later to replicate the build offline.
4333To do so, you need to first prepare and populate your downloads
4334directory your "snapshot" of files. Once your downloads directory is
4335ready, you can use it at any time and from any machine to replicate your
4336build.
4337
4338Follow these steps to populate your Downloads directory:
4339
43401. *Create a Clean Downloads Directory:* Start with an empty downloads
4341 directory (:term:`DL_DIR`). You
4342 start with an empty downloads directory by either removing the files
Andrew Geissler09036742021-06-25 14:25:14 -05004343 in the existing directory or by setting :term:`DL_DIR` to point to either
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004344 an empty location or one that does not yet exist.
4345
43462. *Generate Tarballs of the Source Git Repositories:* Edit your
Andrew Geisslerc926e172021-05-07 16:11:35 -05004347 ``local.conf`` configuration file as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004348
4349 DL_DIR = "/home/your-download-dir/"
4350 BB_GENERATE_MIRROR_TARBALLS = "1"
4351
4352 During
4353 the fetch process in the next step, BitBake gathers the source files
Andrew Geissler09036742021-06-25 14:25:14 -05004354 and creates tarballs in the directory pointed to by :term:`DL_DIR`. See
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004355 the
4356 :term:`BB_GENERATE_MIRROR_TARBALLS`
4357 variable for more information.
4358
43593. *Populate Your Downloads Directory Without Building:* Use BitBake to
Andrew Geisslerc926e172021-05-07 16:11:35 -05004360 fetch your sources but inhibit the build::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004361
4362 $ bitbake target --runonly=fetch
4363
4364 The downloads directory (i.e. ``${DL_DIR}``) now has
4365 a "snapshot" of the source files in the form of tarballs, which can
4366 be used for the build.
4367
43684. *Optionally Remove Any Git or other SCM Subdirectories From the
4369 Downloads Directory:* If you want, you can clean up your downloads
4370 directory by removing any Git or other Source Control Management
4371 (SCM) subdirectories such as ``${DL_DIR}/git2/*``. The tarballs
4372 already contain these subdirectories.
4373
4374Once your downloads directory has everything it needs regarding source
4375files, you can create your "own-mirror" and build your target.
4376Understand that you can use the files to build the target offline from
4377any machine and at any time.
4378
4379Follow these steps to build your target using the files in the downloads
4380directory:
4381
43821. *Using Local Files Only:* Inside your ``local.conf`` file, add the
4383 :term:`SOURCE_MIRROR_URL`
4384 variable, inherit the
4385 :ref:`own-mirrors <ref-classes-own-mirrors>`
4386 class, and use the
4387 :term:`bitbake:BB_NO_NETWORK`
4388 variable to your ``local.conf``.
4389 ::
4390
4391 SOURCE_MIRROR_URL ?= "file:///home/your-download-dir/"
4392 INHERIT += "own-mirrors"
4393 BB_NO_NETWORK = "1"
4394
Andrew Geissler5f350902021-07-23 13:09:54 -04004395 The :term:`SOURCE_MIRROR_URL` and ``own-mirror``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004396 class set up the system to use the downloads directory as your "own
Andrew Geissler09036742021-06-25 14:25:14 -05004397 mirror". Using the :term:`BB_NO_NETWORK` variable makes sure that
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004398 BitBake's fetching process in step 3 stays local, which means files
4399 from your "own-mirror" are used.
4400
44012. *Start With a Clean Build:* You can start with a clean build by
4402 removing the
4403 ``${``\ :term:`TMPDIR`\ ``}``
4404 directory or using a new :term:`Build Directory`.
4405
Andrew Geisslerc926e172021-05-07 16:11:35 -050044063. *Build Your Target:* Use BitBake to build your target::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004407
4408 $ bitbake target
4409
4410 The build completes using the known local "snapshot" of source
4411 files from your mirror. The resulting tarballs for your "snapshot" of
4412 source files are in the downloads directory.
4413
4414 .. note::
4415
4416 The offline build does not work if recipes attempt to find the
4417 latest version of software by setting
4418 :term:`SRCREV` to
Andrew Geisslerc926e172021-05-07 16:11:35 -05004419 ``${``\ :term:`AUTOREV`\ ``}``::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004420
4421 SRCREV = "${AUTOREV}"
4422
Andrew Geissler09036742021-06-25 14:25:14 -05004423 When a recipe sets :term:`SRCREV` to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004424 ``${AUTOREV}``, the build system accesses the network in an
4425 attempt to determine the latest version of software from the SCM.
Andrew Geissler09036742021-06-25 14:25:14 -05004426 Typically, recipes that use :term:`AUTOREV` are custom or modified
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004427 recipes. Recipes that reside in public repositories usually do not
Andrew Geissler09036742021-06-25 14:25:14 -05004428 use :term:`AUTOREV`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004429
Andrew Geissler09036742021-06-25 14:25:14 -05004430 If you do have recipes that use :term:`AUTOREV`, you can take steps to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004431 still use the recipes in an offline build. Do the following:
4432
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05004433 1. Use a configuration generated by enabling :ref:`build
4434 history <dev-manual/common-tasks:maintaining build output quality>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004435
4436 2. Use the ``buildhistory-collect-srcrevs`` command to collect the
Andrew Geissler09036742021-06-25 14:25:14 -05004437 stored :term:`SRCREV` values from the build's history. For more
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05004438 information on collecting these values, see the
4439 ":ref:`dev-manual/common-tasks:build history package information`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004440 section.
4441
4442 3. Once you have the correct source revisions, you can modify
Andrew Geissler09036742021-06-25 14:25:14 -05004443 those recipes to set :term:`SRCREV` to specific versions of the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004444 software.
4445
4446Speeding Up a Build
4447===================
4448
4449Build time can be an issue. By default, the build system uses simple
4450controls to try and maximize build efficiency. In general, the default
4451settings for all the following variables result in the most efficient
4452build times when dealing with single socket systems (i.e. a single CPU).
4453If you have multiple CPUs, you might try increasing the default values
4454to gain more speed. See the descriptions in the glossary for each
4455variable for more information:
4456
4457- :term:`BB_NUMBER_THREADS`:
4458 The maximum number of threads BitBake simultaneously executes.
4459
4460- :term:`bitbake:BB_NUMBER_PARSE_THREADS`:
4461 The number of threads BitBake uses during parsing.
4462
4463- :term:`PARALLEL_MAKE`: Extra
4464 options passed to the ``make`` command during the
4465 :ref:`ref-tasks-compile` task in
4466 order to specify parallel compilation on the local build host.
4467
4468- :term:`PARALLEL_MAKEINST`:
4469 Extra options passed to the ``make`` command during the
4470 :ref:`ref-tasks-install` task in
4471 order to specify parallel installation on the local build host.
4472
4473As mentioned, these variables all scale to the number of processor cores
4474available on the build system. For single socket systems, this
4475auto-scaling ensures that the build system fundamentally takes advantage
4476of potential parallel operations during the build based on the build
4477machine's capabilities.
4478
4479Following are additional factors that can affect build speed:
4480
4481- File system type: The file system type that the build is being
4482 performed on can also influence performance. Using ``ext4`` is
4483 recommended as compared to ``ext2`` and ``ext3`` due to ``ext4``
4484 improved features such as extents.
4485
4486- Disabling the updating of access time using ``noatime``: The
4487 ``noatime`` mount option prevents the build system from updating file
4488 and directory access times.
4489
4490- Setting a longer commit: Using the "commit=" mount option increases
4491 the interval in seconds between disk cache writes. Changing this
4492 interval from the five second default to something longer increases
4493 the risk of data loss but decreases the need to write to the disk,
4494 thus increasing the build performance.
4495
4496- Choosing the packaging backend: Of the available packaging backends,
4497 IPK is the fastest. Additionally, selecting a singular packaging
4498 backend also helps.
4499
4500- Using ``tmpfs`` for :term:`TMPDIR`
4501 as a temporary file system: While this can help speed up the build,
4502 the benefits are limited due to the compiler using ``-pipe``. The
4503 build system goes to some lengths to avoid ``sync()`` calls into the
4504 file system on the principle that if there was a significant failure,
4505 the :term:`Build Directory`
4506 contents could easily be rebuilt.
4507
4508- Inheriting the
4509 :ref:`rm_work <ref-classes-rm-work>` class:
4510 Inheriting this class has shown to speed up builds due to
4511 significantly lower amounts of data stored in the data cache as well
4512 as on disk. Inheriting this class also makes cleanup of
4513 :term:`TMPDIR` faster, at the
4514 expense of being easily able to dive into the source code. File
4515 system maintainers have recommended that the fastest way to clean up
4516 large numbers of files is to reformat partitions rather than delete
4517 files due to the linear nature of partitions. This, of course,
4518 assumes you structure the disk partitions and file systems in a way
4519 that this is practical.
4520
4521Aside from the previous list, you should keep some trade offs in mind
4522that can help you speed up the build:
4523
4524- Remove items from
4525 :term:`DISTRO_FEATURES`
4526 that you might not need.
4527
4528- Exclude debug symbols and other debug information: If you do not need
4529 these symbols and other debug information, disabling the ``*-dbg``
4530 package generation can speed up the build. You can disable this
4531 generation by setting the
4532 :term:`INHIBIT_PACKAGE_DEBUG_SPLIT`
4533 variable to "1".
4534
4535- Disable static library generation for recipes derived from
4536 ``autoconf`` or ``libtool``: Following is an example showing how to
4537 disable static libraries and still provide an override to handle
Andrew Geisslerc926e172021-05-07 16:11:35 -05004538 exceptions::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004539
4540 STATICLIBCONF = "--disable-static"
4541 STATICLIBCONF_sqlite3-native = ""
4542 EXTRA_OECONF += "${STATICLIBCONF}"
4543
4544 .. note::
4545
4546 - Some recipes need static libraries in order to work correctly
4547 (e.g. ``pseudo-native`` needs ``sqlite3-native``). Overrides,
4548 as in the previous example, account for these kinds of
4549 exceptions.
4550
4551 - Some packages have packaging code that assumes the presence of
4552 the static libraries. If so, you might need to exclude them as
4553 well.
4554
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004555Working With Libraries
4556======================
4557
4558Libraries are an integral part of your system. This section describes
4559some common practices you might find helpful when working with libraries
4560to build your system:
4561
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05004562- :ref:`How to include static library files
4563 <dev-manual/common-tasks:including static library files>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004564
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05004565- :ref:`How to use the Multilib feature to combine multiple versions of
4566 library files into a single image
4567 <dev-manual/common-tasks:combining multiple versions of library files into one image>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004568
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05004569- :ref:`How to install multiple versions of the same library in parallel on
4570 the same system
4571 <dev-manual/common-tasks:installing multiple versions of the same library>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004572
4573Including Static Library Files
4574------------------------------
4575
4576If you are building a library and the library offers static linking, you
4577can control which static library files (``*.a`` files) get included in
4578the built library.
4579
4580The :term:`PACKAGES` and
4581:term:`FILES_* <FILES>` variables in the
4582``meta/conf/bitbake.conf`` configuration file define how files installed
Andrew Geissler09036742021-06-25 14:25:14 -05004583by the ``do_install`` task are packaged. By default, the :term:`PACKAGES`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004584variable includes ``${PN}-staticdev``, which represents all static
4585library files.
4586
4587.. note::
4588
4589 Some previously released versions of the Yocto Project defined the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004590 static library files through ``${PN}-dev``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004591
4592Following is part of the BitBake configuration file, where you can see
Andrew Geisslerc926e172021-05-07 16:11:35 -05004593how the static library files are defined::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004594
4595 PACKAGE_BEFORE_PN ?= ""
4596 PACKAGES = "${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc ${PN}-locale ${PACKAGE_BEFORE_PN} ${PN}"
4597 PACKAGES_DYNAMIC = "^${PN}-locale-.*"
4598 FILES = ""
4599
4600 FILES_${PN} = "${bindir}/* ${sbindir}/* ${libexecdir}/* ${libdir}/lib*${SOLIBS} \
4601 ${sysconfdir} ${sharedstatedir} ${localstatedir} \
4602 ${base_bindir}/* ${base_sbindir}/* \
4603 ${base_libdir}/*${SOLIBS} \
4604 ${base_prefix}/lib/udev/rules.d ${prefix}/lib/udev/rules.d \
4605 ${datadir}/${BPN} ${libdir}/${BPN}/* \
4606 ${datadir}/pixmaps ${datadir}/applications \
4607 ${datadir}/idl ${datadir}/omf ${datadir}/sounds \
4608 ${libdir}/bonobo/servers"
4609
4610 FILES_${PN}-bin = "${bindir}/* ${sbindir}/*"
4611
4612 FILES_${PN}-doc = "${docdir} ${mandir} ${infodir} ${datadir}/gtk-doc \
4613 ${datadir}/gnome/help"
4614 SECTION_${PN}-doc = "doc"
4615
4616 FILES_SOLIBSDEV ?= "${base_libdir}/lib*${SOLIBSDEV} ${libdir}/lib*${SOLIBSDEV}"
4617 FILES_${PN}-dev = "${includedir} ${FILES_SOLIBSDEV} ${libdir}/*.la \
4618 ${libdir}/*.o ${libdir}/pkgconfig ${datadir}/pkgconfig \
4619 ${datadir}/aclocal ${base_libdir}/*.o \
4620 ${libdir}/${BPN}/*.la ${base_libdir}/*.la"
4621 SECTION_${PN}-dev = "devel"
4622 ALLOW_EMPTY_${PN}-dev = "1"
4623 RDEPENDS_${PN}-dev = "${PN} (= ${EXTENDPKGV})"
4624
4625 FILES_${PN}-staticdev = "${libdir}/*.a ${base_libdir}/*.a ${libdir}/${BPN}/*.a"
4626 SECTION_${PN}-staticdev = "devel"
4627 RDEPENDS_${PN}-staticdev = "${PN}-dev (= ${EXTENDPKGV})"
4628
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004629Combining Multiple Versions of Library Files into One Image
4630-----------------------------------------------------------
4631
4632The build system offers the ability to build libraries with different
4633target optimizations or architecture formats and combine these together
4634into one system image. You can link different binaries in the image
4635against the different libraries as needed for specific use cases. This
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004636feature is called "Multilib".
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004637
4638An example would be where you have most of a system compiled in 32-bit
4639mode using 32-bit libraries, but you have something large, like a
4640database engine, that needs to be a 64-bit application and uses 64-bit
4641libraries. Multilib allows you to get the best of both 32-bit and 64-bit
4642libraries.
4643
4644While the Multilib feature is most commonly used for 32 and 64-bit
4645differences, the approach the build system uses facilitates different
4646target optimizations. You could compile some binaries to use one set of
4647libraries and other binaries to use a different set of libraries. The
4648libraries could differ in architecture, compiler options, or other
4649optimizations.
4650
William A. Kennington IIIac69b482021-06-02 12:28:27 -07004651There are several examples in the ``meta-skeleton`` layer found in the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004652:term:`Source Directory`:
4653
4654- ``conf/multilib-example.conf`` configuration file
4655
4656- ``conf/multilib-example2.conf`` configuration file
4657
4658- ``recipes-multilib/images/core-image-multilib-example.bb`` recipe
4659
4660Preparing to Use Multilib
4661~~~~~~~~~~~~~~~~~~~~~~~~~
4662
4663User-specific requirements drive the Multilib feature. Consequently,
William A. Kennington IIIac69b482021-06-02 12:28:27 -07004664there is no one "out-of-the-box" configuration that would
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004665meet your needs.
4666
4667In order to enable Multilib, you first need to ensure your recipe is
4668extended to support multiple libraries. Many standard recipes are
4669already extended and support multiple libraries. You can check in the
4670``meta/conf/multilib.conf`` configuration file in the
4671:term:`Source Directory` to see how this is
4672done using the
4673:term:`BBCLASSEXTEND` variable.
4674Eventually, all recipes will be covered and this list will not be
4675needed.
4676
4677For the most part, the Multilib class extension works automatically to
4678extend the package name from ``${PN}`` to ``${MLPREFIX}${PN}``, where
Andrew Geissler5f350902021-07-23 13:09:54 -04004679:term:`MLPREFIX` is the particular multilib (e.g. "lib32-" or "lib64-").
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004680Standard variables such as
4681:term:`DEPENDS`,
4682:term:`RDEPENDS`,
4683:term:`RPROVIDES`,
4684:term:`RRECOMMENDS`,
4685:term:`PACKAGES`, and
4686:term:`PACKAGES_DYNAMIC` are
4687automatically extended by the system. If you are extending any manual
4688code in the recipe, you can use the ``${MLPREFIX}`` variable to ensure
4689those names are extended correctly. This automatic extension code
4690resides in ``multilib.bbclass``.
4691
4692Using Multilib
4693~~~~~~~~~~~~~~
4694
4695After you have set up the recipes, you need to define the actual
4696combination of multiple libraries you want to build. You accomplish this
4697through your ``local.conf`` configuration file in the
4698:term:`Build Directory`. An example
Andrew Geisslerc926e172021-05-07 16:11:35 -05004699configuration would be as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004700
4701 MACHINE = "qemux86-64"
4702 require conf/multilib.conf
4703 MULTILIBS = "multilib:lib32"
4704 DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
4705 IMAGE_INSTALL_append = "lib32-glib-2.0"
4706
4707This example enables an additional library named
4708``lib32`` alongside the normal target packages. When combining these
4709"lib32" alternatives, the example uses "x86" for tuning. For information
4710on this particular tuning, see
4711``meta/conf/machine/include/ia32/arch-ia32.inc``.
4712
4713The example then includes ``lib32-glib-2.0`` in all the images, which
4714illustrates one method of including a multiple library dependency. You
Andrew Geisslerc926e172021-05-07 16:11:35 -05004715can use a normal image build to include this dependency, for example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004716
4717 $ bitbake core-image-sato
4718
4719You can also build Multilib packages
Andrew Geisslerc926e172021-05-07 16:11:35 -05004720specifically with a command like this::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004721
4722 $ bitbake lib32-glib-2.0
4723
4724Additional Implementation Details
4725~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4726
William A. Kennington IIIac69b482021-06-02 12:28:27 -07004727There are generic implementation details as well as details that are specific to
4728package management systems. Following are implementation details
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004729that exist regardless of the package management system:
4730
4731- The typical convention used for the class extension code as used by
4732 Multilib assumes that all package names specified in
4733 :term:`PACKAGES` that contain
4734 ``${PN}`` have ``${PN}`` at the start of the name. When that
4735 convention is not followed and ``${PN}`` appears at the middle or the
4736 end of a name, problems occur.
4737
4738- The :term:`TARGET_VENDOR`
4739 value under Multilib will be extended to "-vendormlmultilib" (e.g.
4740 "-pokymllib32" for a "lib32" Multilib with Poky). The reason for this
4741 slightly unwieldy contraction is that any "-" characters in the
4742 vendor string presently break Autoconf's ``config.sub``, and other
4743 separators are problematic for different reasons.
4744
William A. Kennington IIIac69b482021-06-02 12:28:27 -07004745Here are the implementation details for the RPM Package Management System:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004746
4747- A unique architecture is defined for the Multilib packages, along
4748 with creating a unique deploy folder under ``tmp/deploy/rpm`` in the
4749 :term:`Build Directory`. For
4750 example, consider ``lib32`` in a ``qemux86-64`` image. The possible
4751 architectures in the system are "all", "qemux86_64",
4752 "lib32_qemux86_64", and "lib32_x86".
4753
4754- The ``${MLPREFIX}`` variable is stripped from ``${PN}`` during RPM
4755 packaging. The naming for a normal RPM package and a Multilib RPM
4756 package in a ``qemux86-64`` system resolves to something similar to
4757 ``bash-4.1-r2.x86_64.rpm`` and ``bash-4.1.r2.lib32_x86.rpm``,
4758 respectively.
4759
4760- When installing a Multilib image, the RPM backend first installs the
4761 base image and then installs the Multilib libraries.
4762
4763- The build system relies on RPM to resolve the identical files in the
4764 two (or more) Multilib packages.
4765
William A. Kennington IIIac69b482021-06-02 12:28:27 -07004766Here are the implementation details for the IPK Package Management System:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004767
4768- The ``${MLPREFIX}`` is not stripped from ``${PN}`` during IPK
4769 packaging. The naming for a normal RPM package and a Multilib IPK
4770 package in a ``qemux86-64`` system resolves to something like
4771 ``bash_4.1-r2.x86_64.ipk`` and ``lib32-bash_4.1-rw_x86.ipk``,
4772 respectively.
4773
4774- The IPK deploy folder is not modified with ``${MLPREFIX}`` because
4775 packages with and without the Multilib feature can exist in the same
4776 folder due to the ``${PN}`` differences.
4777
4778- IPK defines a sanity check for Multilib installation using certain
4779 rules for file comparison, overridden, etc.
4780
4781Installing Multiple Versions of the Same Library
4782------------------------------------------------
4783
William A. Kennington IIIac69b482021-06-02 12:28:27 -07004784There are be situations where you need to install and use multiple versions
4785of the same library on the same system at the same time. This
4786almost always happens when a library API changes and you have
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004787multiple pieces of software that depend on the separate versions of the
4788library. To accommodate these situations, you can install multiple
4789versions of the same library in parallel on the same system.
4790
4791The process is straightforward as long as the libraries use proper
4792versioning. With properly versioned libraries, all you need to do to
4793individually specify the libraries is create separate, appropriately
4794named recipes where the :term:`PN` part of
4795the name includes a portion that differentiates each library version
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004796(e.g. the major part of the version number). Thus, instead of having a
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004797single recipe that loads one version of a library (e.g. ``clutter``),
4798you provide multiple recipes that result in different versions of the
4799libraries you want. As an example, the following two recipes would allow
4800the two separate versions of the ``clutter`` library to co-exist on the
4801same system:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004802
4803.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004804
4805 clutter-1.6_1.6.20.bb
4806 clutter-1.8_1.8.4.bb
4807
4808Additionally, if
4809you have other recipes that depend on a given library, you need to use
4810the :term:`DEPENDS` variable to
4811create the dependency. Continuing with the same example, if you want to
4812have a recipe depend on the 1.8 version of the ``clutter`` library, use
Andrew Geisslerc926e172021-05-07 16:11:35 -05004813the following in your recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004814
4815 DEPENDS = "clutter-1.8"
4816
4817Using x32 psABI
4818===============
4819
4820x32 processor-specific Application Binary Interface (`x32
4821psABI <https://software.intel.com/en-us/node/628948>`__) is a native
482232-bit processor-specific ABI for Intel 64 (x86-64) architectures. An
4823ABI defines the calling conventions between functions in a processing
4824environment. The interface determines what registers are used and what
4825the sizes are for various C data types.
4826
4827Some processing environments prefer using 32-bit applications even when
4828running on Intel 64-bit platforms. Consider the i386 psABI, which is a
4829very old 32-bit ABI for Intel 64-bit platforms. The i386 psABI does not
4830provide efficient use and access of the Intel 64-bit processor
4831resources, leaving the system underutilized. Now consider the x86_64
4832psABI. This ABI is newer and uses 64-bits for data sizes and program
4833pointers. The extra bits increase the footprint size of the programs,
4834libraries, and also increases the memory and file system size
4835requirements. Executing under the x32 psABI enables user programs to
4836utilize CPU and system resources more efficiently while keeping the
4837memory footprint of the applications low. Extra bits are used for
4838registers but not for addressing mechanisms.
4839
4840The Yocto Project supports the final specifications of x32 psABI as
4841follows:
4842
4843- You can create packages and images in x32 psABI format on x86_64
4844 architecture targets.
4845
4846- You can successfully build recipes with the x32 toolchain.
4847
4848- You can create and boot ``core-image-minimal`` and
4849 ``core-image-sato`` images.
4850
William A. Kennington IIIac69b482021-06-02 12:28:27 -07004851- There is RPM Package Manager (RPM) support for x32 binaries.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004852
William A. Kennington IIIac69b482021-06-02 12:28:27 -07004853- There is support for large images.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004854
4855To use the x32 psABI, you need to edit your ``conf/local.conf``
Andrew Geisslerc926e172021-05-07 16:11:35 -05004856configuration file as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004857
4858 MACHINE = "qemux86-64"
4859 DEFAULTTUNE = "x86-64-x32"
4860 baselib = "${@d.getVar('BASE_LIB_tune-' + (d.getVar('DEFAULTTUNE') \
4861 or 'INVALID')) or 'lib'}"
4862
4863Once you have set
4864up your configuration file, use BitBake to build an image that supports
Andrew Geisslerc926e172021-05-07 16:11:35 -05004865the x32 psABI. Here is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004866
4867 $ bitbake core-image-sato
4868
4869Enabling GObject Introspection Support
4870======================================
4871
4872`GObject
4873introspection <https://wiki.gnome.org/Projects/GObjectIntrospection>`__
4874is the standard mechanism for accessing GObject-based software from
4875runtime environments. GObject is a feature of the GLib library that
4876provides an object framework for the GNOME desktop and related software.
4877GObject Introspection adds information to GObject that allows objects
4878created within it to be represented across different programming
4879languages. If you want to construct GStreamer pipelines using Python, or
4880control UPnP infrastructure using Javascript and GUPnP, GObject
4881introspection is the only way to do it.
4882
4883This section describes the Yocto Project support for generating and
4884packaging GObject introspection data. GObject introspection data is a
4885description of the API provided by libraries built on top of GLib
4886framework, and, in particular, that framework's GObject mechanism.
4887GObject Introspection Repository (GIR) files go to ``-dev`` packages,
4888``typelib`` files go to main packages as they are packaged together with
4889libraries that are introspected.
4890
4891The data is generated when building such a library, by linking the
4892library with a small executable binary that asks the library to describe
4893itself, and then executing the binary and processing its output.
4894
4895Generating this data in a cross-compilation environment is difficult
4896because the library is produced for the target architecture, but its
4897code needs to be executed on the build host. This problem is solved with
4898the OpenEmbedded build system by running the code through QEMU, which
4899allows precisely that. Unfortunately, QEMU does not always work
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05004900perfectly as mentioned in the ":ref:`dev-manual/common-tasks:known issues`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004901section.
4902
4903Enabling the Generation of Introspection Data
4904---------------------------------------------
4905
4906Enabling the generation of introspection data (GIR files) in your
4907library package involves the following:
4908
49091. Inherit the
4910 :ref:`gobject-introspection <ref-classes-gobject-introspection>`
4911 class.
4912
49132. Make sure introspection is not disabled anywhere in the recipe or
4914 from anything the recipe includes. Also, make sure that
4915 "gobject-introspection-data" is not in
4916 :term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`
4917 and that "qemu-usermode" is not in
4918 :term:`MACHINE_FEATURES_BACKFILL_CONSIDERED`.
William A. Kennington IIIac69b482021-06-02 12:28:27 -07004919 In either of these conditions, nothing will happen.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004920
49213. Try to build the recipe. If you encounter build errors that look like
4922 something is unable to find ``.so`` libraries, check where these
4923 libraries are located in the source tree and add the following to the
Andrew Geisslerc926e172021-05-07 16:11:35 -05004924 recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004925
4926 GIR_EXTRA_LIBS_PATH = "${B}/something/.libs"
4927
4928 .. note::
4929
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004930 See recipes in the ``oe-core`` repository that use that
4931 ``GIR_EXTRA_LIBS_PATH`` variable as an example.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004932
49334. Look for any other errors, which probably mean that introspection
4934 support in a package is not entirely standard, and thus breaks down
4935 in a cross-compilation environment. For such cases, custom-made fixes
4936 are needed. A good place to ask and receive help in these cases is
4937 the :ref:`Yocto Project mailing
4938 lists <resources-mailinglist>`.
4939
4940.. note::
4941
4942 Using a library that no longer builds against the latest Yocto
4943 Project release and prints introspection related errors is a good
4944 candidate for the previous procedure.
4945
4946Disabling the Generation of Introspection Data
4947----------------------------------------------
4948
4949You might find that you do not want to generate introspection data. Or,
4950perhaps QEMU does not work on your build host and target architecture
4951combination. If so, you can use either of the following methods to
4952disable GIR file generations:
4953
Andrew Geisslerc926e172021-05-07 16:11:35 -05004954- Add the following to your distro configuration::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004955
4956 DISTRO_FEATURES_BACKFILL_CONSIDERED = "gobject-introspection-data"
4957
4958 Adding this statement disables generating introspection data using
4959 QEMU but will still enable building introspection tools and libraries
4960 (i.e. building them does not require the use of QEMU).
4961
Andrew Geisslerc926e172021-05-07 16:11:35 -05004962- Add the following to your machine configuration::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004963
4964 MACHINE_FEATURES_BACKFILL_CONSIDERED = "qemu-usermode"
4965
4966 Adding this statement disables the use of QEMU when building packages for your
4967 machine. Currently, this feature is used only by introspection
4968 recipes and has the same effect as the previously described option.
4969
4970 .. note::
4971
4972 Future releases of the Yocto Project might have other features
4973 affected by this option.
4974
4975If you disable introspection data, you can still obtain it through other
4976means such as copying the data from a suitable sysroot, or by generating
4977it on the target hardware. The OpenEmbedded build system does not
4978currently provide specific support for these techniques.
4979
4980Testing that Introspection Works in an Image
4981--------------------------------------------
4982
4983Use the following procedure to test if generating introspection data is
4984working in an image:
4985
49861. Make sure that "gobject-introspection-data" is not in
4987 :term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`
4988 and that "qemu-usermode" is not in
4989 :term:`MACHINE_FEATURES_BACKFILL_CONSIDERED`.
4990
49912. Build ``core-image-sato``.
4992
49933. Launch a Terminal and then start Python in the terminal.
4994
Andrew Geisslerc926e172021-05-07 16:11:35 -050049954. Enter the following in the terminal::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004996
4997 >>> from gi.repository import GLib
4998 >>> GLib.get_host_name()
4999
50005. For something a little more advanced, enter the following see:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005001 https://python-gtk-3-tutorial.readthedocs.io/en/latest/introduction.html
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005002
5003Known Issues
5004------------
5005
William A. Kennington IIIac69b482021-06-02 12:28:27 -07005006Here are know issues in GObject Introspection Support:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005007
5008- ``qemu-ppc64`` immediately crashes. Consequently, you cannot build
5009 introspection data on that architecture.
5010
5011- x32 is not supported by QEMU. Consequently, introspection data is
5012 disabled.
5013
5014- musl causes transient GLib binaries to crash on assertion failures.
5015 Consequently, generating introspection data is disabled.
5016
5017- Because QEMU is not able to run the binaries correctly, introspection
5018 is disabled for some specific packages under specific architectures
5019 (e.g. ``gcr``, ``libsecret``, and ``webkit``).
5020
5021- QEMU usermode might not work properly when running 64-bit binaries
5022 under 32-bit host machines. In particular, "qemumips64" is known to
5023 not work under i686.
5024
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005025Optionally Using an External Toolchain
5026======================================
5027
5028You might want to use an external toolchain as part of your development.
5029If this is the case, the fundamental steps you need to accomplish are as
5030follows:
5031
5032- Understand where the installed toolchain resides. For cases where you
5033 need to build the external toolchain, you would need to take separate
5034 steps to build and install the toolchain.
5035
5036- Make sure you add the layer that contains the toolchain to your
5037 ``bblayers.conf`` file through the
5038 :term:`BBLAYERS` variable.
5039
5040- Set the ``EXTERNAL_TOOLCHAIN`` variable in your ``local.conf`` file
5041 to the location in which you installed the toolchain.
5042
5043A good example of an external toolchain used with the Yocto Project is
5044Mentor Graphics Sourcery G++ Toolchain. You can see information on how
5045to use that particular layer in the ``README`` file at
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005046https://github.com/MentorEmbedded/meta-sourcery/. You can find
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005047further information by reading about the
5048:term:`TCMODE` variable in the Yocto
5049Project Reference Manual's variable glossary.
5050
5051Creating Partitioned Images Using Wic
5052=====================================
5053
5054Creating an image for a particular hardware target using the
5055OpenEmbedded build system does not necessarily mean you can boot that
5056image as is on your device. Physical devices accept and boot images in
5057various ways depending on the specifics of the device. Usually,
5058information about the hardware can tell you what image format the device
5059requires. Should your device require multiple partitions on an SD card,
5060flash, or an HDD, you can use the OpenEmbedded Image Creator, Wic, to
5061create the properly partitioned image.
5062
5063The ``wic`` command generates partitioned images from existing
5064OpenEmbedded build artifacts. Image generation is driven by partitioning
5065commands contained in an Openembedded kickstart file (``.wks``)
5066specified either directly on the command line or as one of a selection
5067of canned kickstart files as shown with the ``wic list images`` command
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05005068in the
5069":ref:`dev-manual/common-tasks:generate an image using an existing kickstart file`"
5070section. When you apply the command to a given set of build artifacts, the
5071result is an image or set of images that can be directly written onto media and
5072used on a particular system.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005073
5074.. note::
5075
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005076 For a kickstart file reference, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06005077 ":ref:`ref-manual/kickstart:openembedded kickstart (\`\`.wks\`\`) reference`"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005078 Chapter in the Yocto Project Reference Manual.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005079
5080The ``wic`` command and the infrastructure it is based on is by
5081definition incomplete. The purpose of the command is to allow the
5082generation of customized images, and as such, was designed to be
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05005083completely extensible through a plugin interface. See the
5084":ref:`dev-manual/common-tasks:using the wic plugin interface`" section
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005085for information on these plugins.
5086
5087This section provides some background information on Wic, describes what
5088you need to have in place to run the tool, provides instruction on how
5089to use the Wic utility, provides information on using the Wic plugins
5090interface, and provides several examples that show how to use Wic.
5091
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005092Background
5093----------
5094
5095This section provides some background on the Wic utility. While none of
5096this information is required to use Wic, you might find it interesting.
5097
5098- The name "Wic" is derived from OpenEmbedded Image Creator (oeic). The
5099 "oe" diphthong in "oeic" was promoted to the letter "w", because
5100 "oeic" is both difficult to remember and to pronounce.
5101
5102- Wic is loosely based on the Meego Image Creator (``mic``) framework.
5103 The Wic implementation has been heavily modified to make direct use
5104 of OpenEmbedded build artifacts instead of package installation and
5105 configuration, which are already incorporated within the OpenEmbedded
5106 artifacts.
5107
5108- Wic is a completely independent standalone utility that initially
5109 provides easier-to-use and more flexible replacements for an existing
5110 functionality in OE-Core's
5111 :ref:`image-live <ref-classes-image-live>`
5112 class. The difference between Wic and those examples is that with Wic
5113 the functionality of those scripts is implemented by a
5114 general-purpose partitioning language, which is based on Redhat
5115 kickstart syntax.
5116
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005117Requirements
5118------------
5119
5120In order to use the Wic utility with the OpenEmbedded Build system, your
5121system needs to meet the following requirements:
5122
5123- The Linux distribution on your development host must support the
5124 Yocto Project. See the ":ref:`detailed-supported-distros`"
5125 section in the Yocto Project Reference Manual for the list of
5126 distributions that support the Yocto Project.
5127
5128- The standard system utilities, such as ``cp``, must be installed on
5129 your development host system.
5130
5131- You must have sourced the build environment setup script (i.e.
5132 :ref:`structure-core-script`) found in the
5133 :term:`Build Directory`.
5134
5135- You need to have the build artifacts already available, which
5136 typically means that you must have already created an image using the
5137 Openembedded build system (e.g. ``core-image-minimal``). While it
5138 might seem redundant to generate an image in order to create an image
5139 using Wic, the current version of Wic requires the artifacts in the
5140 form generated by the OpenEmbedded build system.
5141
5142- You must build several native tools, which are built to run on the
Andrew Geisslerc926e172021-05-07 16:11:35 -05005143 build system::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005144
5145 $ bitbake parted-native dosfstools-native mtools-native
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005146
5147- Include "wic" as part of the
5148 :term:`IMAGE_FSTYPES`
5149 variable.
5150
5151- Include the name of the :ref:`wic kickstart file <openembedded-kickstart-wks-reference>`
5152 as part of the :term:`WKS_FILE` variable
5153
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005154Getting Help
5155------------
5156
5157You can get general help for the ``wic`` command by entering the ``wic``
5158command by itself or by entering the command with a help argument as
Andrew Geisslerc926e172021-05-07 16:11:35 -05005159follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005160
5161 $ wic -h
5162 $ wic --help
5163 $ wic help
5164
5165Currently, Wic supports seven commands: ``cp``, ``create``, ``help``,
5166``list``, ``ls``, ``rm``, and ``write``. You can get help for all these
Andrew Geisslerc926e172021-05-07 16:11:35 -05005167commands except "help" by using the following form::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005168
5169 $ wic help command
5170
5171For example, the following command returns help for the ``write``
Andrew Geisslerc926e172021-05-07 16:11:35 -05005172command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005173
5174 $ wic help write
5175
5176Wic supports help for three topics: ``overview``, ``plugins``, and
Andrew Geisslerc926e172021-05-07 16:11:35 -05005177``kickstart``. You can get help for any topic using the following form::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005178
5179 $ wic help topic
5180
Andrew Geisslerc926e172021-05-07 16:11:35 -05005181For example, the following returns overview help for Wic::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005182
5183 $ wic help overview
5184
William A. Kennington IIIac69b482021-06-02 12:28:27 -07005185There is one additional level of help for Wic. You can get help on
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005186individual images through the ``list`` command. You can use the ``list``
Andrew Geisslerc926e172021-05-07 16:11:35 -05005187command to return the available Wic images as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005188
5189 $ wic list images
5190 genericx86 Create an EFI disk image for genericx86*
5191 beaglebone-yocto Create SD card image for Beaglebone
5192 edgerouter Create SD card image for Edgerouter
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05005193 qemux86-directdisk Create a QEMU machine 'pcbios' direct disk image
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005194 directdisk-gpt Create a 'pcbios' direct disk image
5195 mkefidisk Create an EFI disk image
5196 directdisk Create a 'pcbios' direct disk image
5197 systemd-bootdisk Create an EFI disk image with systemd-boot
5198 mkhybridiso Create a hybrid ISO image
5199 sdimage-bootpart Create SD card image with a boot partition
5200 directdisk-multi-rootfs Create multi rootfs image using rootfs plugin
5201 directdisk-bootloader-config Create a 'pcbios' direct disk image with custom bootloader config
5202
5203Once you know the list of available
5204Wic images, you can use ``help`` with the command to get help on a
5205particular image. For example, the following command returns help on the
Andrew Geisslerc926e172021-05-07 16:11:35 -05005206"beaglebone-yocto" image::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005207
5208 $ wic list beaglebone-yocto help
5209
5210 Creates a partitioned SD card image for Beaglebone.
5211 Boot files are located in the first vfat partition.
5212
5213Operational Modes
5214-----------------
5215
5216You can use Wic in two different modes, depending on how much control
5217you need for specifying the Openembedded build artifacts that are used
5218for creating the image: Raw and Cooked:
5219
5220- *Raw Mode:* You explicitly specify build artifacts through Wic
5221 command-line arguments.
5222
5223- *Cooked Mode:* The current
5224 :term:`MACHINE` setting and image
5225 name are used to automatically locate and provide the build
5226 artifacts. You just supply a kickstart file and the name of the image
5227 from which to use artifacts.
5228
5229Regardless of the mode you use, you need to have the build artifacts
5230ready and available.
5231
5232Raw Mode
5233~~~~~~~~
5234
5235Running Wic in raw mode allows you to specify all the partitions through
5236the ``wic`` command line. The primary use for raw mode is if you have
5237built your kernel outside of the Yocto Project
5238:term:`Build Directory`. In other words, you
5239can point to arbitrary kernel, root filesystem locations, and so forth.
5240Contrast this behavior with cooked mode where Wic looks in the Build
5241Directory (e.g. ``tmp/deploy/images/``\ machine).
5242
Andrew Geisslerc926e172021-05-07 16:11:35 -05005243The general form of the ``wic`` command in raw mode is::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005244
5245 $ wic create wks_file options ...
5246
5247 Where:
5248
5249 wks_file:
5250 An OpenEmbedded kickstart file. You can provide
5251 your own custom file or use a file from a set of
5252 existing files as described by further options.
5253
5254 optional arguments:
5255 -h, --help show this help message and exit
5256 -o OUTDIR, --outdir OUTDIR
5257 name of directory to create image in
5258 -e IMAGE_NAME, --image-name IMAGE_NAME
5259 name of the image to use the artifacts from e.g. core-
5260 image-sato
5261 -r ROOTFS_DIR, --rootfs-dir ROOTFS_DIR
5262 path to the /rootfs dir to use as the .wks rootfs
5263 source
5264 -b BOOTIMG_DIR, --bootimg-dir BOOTIMG_DIR
5265 path to the dir containing the boot artifacts (e.g.
5266 /EFI or /syslinux dirs) to use as the .wks bootimg
5267 source
5268 -k KERNEL_DIR, --kernel-dir KERNEL_DIR
5269 path to the dir containing the kernel to use in the
5270 .wks bootimg
5271 -n NATIVE_SYSROOT, --native-sysroot NATIVE_SYSROOT
5272 path to the native sysroot containing the tools to use
5273 to build the image
5274 -s, --skip-build-check
5275 skip the build check
5276 -f, --build-rootfs build rootfs
5277 -c {gzip,bzip2,xz}, --compress-with {gzip,bzip2,xz}
5278 compress image with specified compressor
5279 -m, --bmap generate .bmap
5280 --no-fstab-update Do not change fstab file.
5281 -v VARS_DIR, --vars VARS_DIR
5282 directory with <image>.env files that store bitbake
5283 variables
5284 -D, --debug output debug information
5285
5286.. note::
5287
5288 You do not need root privileges to run Wic. In fact, you should not
5289 run as root when using the utility.
5290
5291Cooked Mode
5292~~~~~~~~~~~
5293
5294Running Wic in cooked mode leverages off artifacts in the Build
5295Directory. In other words, you do not have to specify kernel or root
5296filesystem locations as part of the command. All you need to provide is
5297a kickstart file and the name of the image from which to use artifacts
5298by using the "-e" option. Wic looks in the Build Directory (e.g.
5299``tmp/deploy/images/``\ machine) for artifacts.
5300
Andrew Geisslerc926e172021-05-07 16:11:35 -05005301The general form of the ``wic`` command using Cooked Mode is as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005302
5303 $ wic create wks_file -e IMAGE_NAME
5304
5305 Where:
5306
5307 wks_file:
5308 An OpenEmbedded kickstart file. You can provide
5309 your own custom file or use a file from a set of
5310 existing files provided with the Yocto Project
5311 release.
5312
5313 required argument:
5314 -e IMAGE_NAME, --image-name IMAGE_NAME
5315 name of the image to use the artifacts from e.g. core-
5316 image-sato
5317
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005318Using an Existing Kickstart File
5319--------------------------------
5320
5321If you do not want to create your own kickstart file, you can use an
5322existing file provided by the Wic installation. As shipped, kickstart
Andrew Geissler09209ee2020-12-13 08:44:15 -06005323files can be found in the :ref:`overview-manual/development-environment:yocto project source repositories` in the
Andrew Geisslerc926e172021-05-07 16:11:35 -05005324following two locations::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005325
5326 poky/meta-yocto-bsp/wic
5327 poky/scripts/lib/wic/canned-wks
5328
Andrew Geisslerc926e172021-05-07 16:11:35 -05005329Use the following command to list the available kickstart files::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005330
5331 $ wic list images
5332 genericx86 Create an EFI disk image for genericx86*
5333 beaglebone-yocto Create SD card image for Beaglebone
5334 edgerouter Create SD card image for Edgerouter
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05005335 qemux86-directdisk Create a QEMU machine 'pcbios' direct disk image
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005336 directdisk-gpt Create a 'pcbios' direct disk image
5337 mkefidisk Create an EFI disk image
5338 directdisk Create a 'pcbios' direct disk image
5339 systemd-bootdisk Create an EFI disk image with systemd-boot
5340 mkhybridiso Create a hybrid ISO image
5341 sdimage-bootpart Create SD card image with a boot partition
5342 directdisk-multi-rootfs Create multi rootfs image using rootfs plugin
5343 directdisk-bootloader-config Create a 'pcbios' direct disk image with custom bootloader config
5344
5345When you use an existing file, you
5346do not have to use the ``.wks`` extension. Here is an example in Raw
Andrew Geisslerc926e172021-05-07 16:11:35 -05005347Mode that uses the ``directdisk`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005348
5349 $ wic create directdisk -r rootfs_dir -b bootimg_dir \
5350 -k kernel_dir -n native_sysroot
5351
5352Here are the actual partition language commands used in the
Andrew Geisslerc926e172021-05-07 16:11:35 -05005353``genericx86.wks`` file to generate an image::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005354
5355 # short-description: Create an EFI disk image for genericx86*
5356 # long-description: Creates a partitioned EFI disk image for genericx86* machines
5357 part /boot --source bootimg-efi --sourceparams="loader=grub-efi" --ondisk sda --label msdos --active --align 1024
5358 part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 1024 --use-uuid
5359 part swap --ondisk sda --size 44 --label swap1 --fstype=swap
5360
5361 bootloader --ptable gpt --timeout=5 --append="rootfstype=ext4 console=ttyS0,115200 console=tty0"
5362
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005363Using the Wic Plugin Interface
5364------------------------------
5365
5366You can extend and specialize Wic functionality by using Wic plugins.
5367This section explains the Wic plugin interface.
5368
5369.. note::
5370
5371 Wic plugins consist of "source" and "imager" plugins. Imager plugins
5372 are beyond the scope of this section.
5373
5374Source plugins provide a mechanism to customize partition content during
5375the Wic image generation process. You can use source plugins to map
5376values that you specify using ``--source`` commands in kickstart files
5377(i.e. ``*.wks``) to a plugin implementation used to populate a given
5378partition.
5379
5380.. note::
5381
5382 If you use plugins that have build-time dependencies (e.g. native
5383 tools, bootloaders, and so forth) when building a Wic image, you need
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005384 to specify those dependencies using the :term:`WKS_FILE_DEPENDS`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005385 variable.
5386
5387Source plugins are subclasses defined in plugin files. As shipped, the
5388Yocto Project provides several plugin files. You can see the source
5389plugin files that ship with the Yocto Project
Andrew Geissler09209ee2020-12-13 08:44:15 -06005390:yocto_git:`here </poky/tree/scripts/lib/wic/plugins/source>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005391Each of these plugin files contains source plugins that are designed to
5392populate a specific Wic image partition.
5393
5394Source plugins are subclasses of the ``SourcePlugin`` class, which is
5395defined in the ``poky/scripts/lib/wic/pluginbase.py`` file. For example,
5396the ``BootimgEFIPlugin`` source plugin found in the ``bootimg-efi.py``
5397file is a subclass of the ``SourcePlugin`` class, which is found in the
5398``pluginbase.py`` file.
5399
5400You can also implement source plugins in a layer outside of the Source
5401Repositories (external layer). To do so, be sure that your plugin files
5402are located in a directory whose path is
5403``scripts/lib/wic/plugins/source/`` within your external layer. When the
5404plugin files are located there, the source plugins they contain are made
5405available to Wic.
5406
5407When the Wic implementation needs to invoke a partition-specific
5408implementation, it looks for the plugin with the same name as the
5409``--source`` parameter used in the kickstart file given to that
5410partition. For example, if the partition is set up using the following
Andrew Geisslerc926e172021-05-07 16:11:35 -05005411command in a kickstart file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005412
5413 part /boot --source bootimg-pcbios --ondisk sda --label boot --active --align 1024
5414
5415The methods defined as class
5416members of the matching source plugin (i.e. ``bootimg-pcbios``) in the
5417``bootimg-pcbios.py`` plugin file are used.
5418
5419To be more concrete, here is the corresponding plugin definition from
5420the ``bootimg-pcbios.py`` file for the previous command along with an
5421example method called by the Wic implementation when it needs to prepare
Andrew Geisslerc926e172021-05-07 16:11:35 -05005422a partition using an implementation-specific function::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005423
5424 .
5425 .
5426 .
5427 class BootimgPcbiosPlugin(SourcePlugin):
5428 """
5429 Create MBR boot partition and install syslinux on it.
5430 """
5431
5432 name = 'bootimg-pcbios'
5433 .
5434 .
5435 .
5436 @classmethod
5437 def do_prepare_partition(cls, part, source_params, creator, cr_workdir,
5438 oe_builddir, bootimg_dir, kernel_dir,
5439 rootfs_dir, native_sysroot):
5440 """
5441 Called to do the actual content population for a partition i.e. it
5442 'prepares' the partition to be incorporated into the image.
5443 In this case, prepare content for legacy bios boot partition.
5444 """
5445 .
5446 .
5447 .
5448
5449If a
5450subclass (plugin) itself does not implement a particular function, Wic
5451locates and uses the default version in the superclass. It is for this
5452reason that all source plugins are derived from the ``SourcePlugin``
5453class.
5454
5455The ``SourcePlugin`` class defined in the ``pluginbase.py`` file defines
5456a set of methods that source plugins can implement or override. Any
5457plugins (subclass of ``SourcePlugin``) that do not implement a
5458particular method inherit the implementation of the method from the
5459``SourcePlugin`` class. For more information, see the ``SourcePlugin``
5460class in the ``pluginbase.py`` file for details:
5461
5462The following list describes the methods implemented in the
5463``SourcePlugin`` class:
5464
5465- ``do_prepare_partition()``: Called to populate a partition with
5466 actual content. In other words, the method prepares the final
5467 partition image that is incorporated into the disk image.
5468
5469- ``do_configure_partition()``: Called before
5470 ``do_prepare_partition()`` to create custom configuration files for a
5471 partition (e.g. syslinux or grub configuration files).
5472
5473- ``do_install_disk()``: Called after all partitions have been
5474 prepared and assembled into a disk image. This method provides a hook
5475 to allow finalization of a disk image (e.g. writing an MBR).
5476
5477- ``do_stage_partition()``: Special content-staging hook called
5478 before ``do_prepare_partition()``. This method is normally empty.
5479
5480 Typically, a partition just uses the passed-in parameters (e.g. the
5481 unmodified value of ``bootimg_dir``). However, in some cases, things
5482 might need to be more tailored. As an example, certain files might
5483 additionally need to be taken from ``bootimg_dir + /boot``. This hook
5484 allows those files to be staged in a customized fashion.
5485
5486 .. note::
5487
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005488 ``get_bitbake_var()`` allows you to access non-standard variables that
5489 you might want to use for this behavior.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005490
5491You can extend the source plugin mechanism. To add more hooks, create
5492more source plugin methods within ``SourcePlugin`` and the corresponding
5493derived subclasses. The code that calls the plugin methods uses the
5494``plugin.get_source_plugin_methods()`` function to find the method or
5495methods needed by the call. Retrieval of those methods is accomplished
5496by filling up a dict with keys that contain the method names of
5497interest. On success, these will be filled in with the actual methods.
5498See the Wic implementation for examples and details.
5499
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005500Wic Examples
5501------------
5502
5503This section provides several examples that show how to use the Wic
5504utility. All the examples assume the list of requirements in the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05005505":ref:`dev-manual/common-tasks:requirements`" section have been met. The
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005506examples assume the previously generated image is
5507``core-image-minimal``.
5508
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005509Generate an Image using an Existing Kickstart File
5510~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5511
5512This example runs in Cooked Mode and uses the ``mkefidisk`` kickstart
Andrew Geisslerc926e172021-05-07 16:11:35 -05005513file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005514
5515 $ wic create mkefidisk -e core-image-minimal
5516 INFO: Building wic-tools...
5517 .
5518 .
5519 .
5520 INFO: The new image(s) can be found here:
5521 ./mkefidisk-201804191017-sda.direct
5522
5523 The following build artifacts were used to create the image(s):
5524 ROOTFS_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
5525 BOOTIMG_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
5526 KERNEL_DIR: /home/stephano/build/master/build/tmp-glibc/deploy/images/qemux86
5527 NATIVE_SYSROOT: /home/stephano/build/master/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
5528
5529 INFO: The image(s) were created using OE kickstart file:
5530 /home/stephano/build/master/openembedded-core/scripts/lib/wic/canned-wks/mkefidisk.wks
5531
5532The previous example shows the easiest way to create an image by running
5533in cooked mode and supplying a kickstart file and the "-e" option to
5534point to the existing build artifacts. Your ``local.conf`` file needs to
5535have the :term:`MACHINE` variable set
5536to the machine you are using, which is "qemux86" in this example.
5537
5538Once the image builds, the output provides image location, artifact use,
5539and kickstart file information.
5540
5541.. note::
5542
5543 You should always verify the details provided in the output to make
5544 sure that the image was indeed created exactly as expected.
5545
5546Continuing with the example, you can now write the image from the Build
5547Directory onto a USB stick, or whatever media for which you built your
5548image, and boot from the media. You can write the image by using
Andrew Geisslerc926e172021-05-07 16:11:35 -05005549``bmaptool`` or ``dd``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005550
5551 $ oe-run-native bmaptool copy mkefidisk-201804191017-sda.direct /dev/sdX
5552
5553or ::
5554
5555 $ sudo dd if=mkefidisk-201804191017-sda.direct of=/dev/sdX
5556
5557.. note::
5558
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005559 For more information on how to use the ``bmaptool``
5560 to flash a device with an image, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06005561 ":ref:`dev-manual/common-tasks:flashing images using \`\`bmaptool\`\``"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005562 section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005563
5564Using a Modified Kickstart File
5565~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5566
5567Because partitioned image creation is driven by the kickstart file, it
5568is easy to affect image creation by changing the parameters in the file.
5569This next example demonstrates that through modification of the
5570``directdisk-gpt`` kickstart file.
5571
5572As mentioned earlier, you can use the command ``wic list images`` to
5573show the list of existing kickstart files. The directory in which the
5574``directdisk-gpt.wks`` file resides is
5575``scripts/lib/image/canned-wks/``, which is located in the
5576:term:`Source Directory` (e.g. ``poky``).
5577Because available files reside in this directory, you can create and add
5578your own custom files to the directory. Subsequent use of the
5579``wic list images`` command would then include your kickstart files.
5580
5581In this example, the existing ``directdisk-gpt`` file already does most
5582of what is needed. However, for the hardware in this example, the image
5583will need to boot from ``sdb`` instead of ``sda``, which is what the
5584``directdisk-gpt`` kickstart file uses.
5585
5586The example begins by making a copy of the ``directdisk-gpt.wks`` file
5587in the ``scripts/lib/image/canned-wks`` directory and then by changing
5588the lines that specify the target disk from which to boot.
5589::
5590
5591 $ cp /home/stephano/poky/scripts/lib/wic/canned-wks/directdisk-gpt.wks \
5592 /home/stephano/poky/scripts/lib/wic/canned-wks/directdisksdb-gpt.wks
5593
5594Next, the example modifies the ``directdisksdb-gpt.wks`` file and
5595changes all instances of "``--ondisk sda``" to "``--ondisk sdb``". The
5596example changes the following two lines and leaves the remaining lines
Andrew Geisslerc926e172021-05-07 16:11:35 -05005597untouched::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005598
5599 part /boot --source bootimg-pcbios --ondisk sdb --label boot --active --align 1024
5600 part / --source rootfs --ondisk sdb --fstype=ext4 --label platform --align 1024 --use-uuid
5601
5602Once the lines are changed, the
5603example generates the ``directdisksdb-gpt`` image. The command points
5604the process at the ``core-image-minimal`` artifacts for the Next Unit of
5605Computing (nuc) :term:`MACHINE` the
5606``local.conf``.
5607::
5608
5609 $ wic create directdisksdb-gpt -e core-image-minimal
5610 INFO: Building wic-tools...
5611 .
5612 .
5613 .
5614 Initialising tasks: 100% |#######################################| Time: 0:00:01
5615 NOTE: Executing SetScene Tasks
5616 NOTE: Executing RunQueue Tasks
5617 NOTE: Tasks Summary: Attempted 1161 tasks of which 1157 didn't need to be rerun and all succeeded.
5618 INFO: Creating image(s)...
5619
5620 INFO: The new image(s) can be found here:
5621 ./directdisksdb-gpt-201710090938-sdb.direct
5622
5623 The following build artifacts were used to create the image(s):
5624 ROOTFS_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
5625 BOOTIMG_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
5626 KERNEL_DIR: /home/stephano/build/master/build/tmp-glibc/deploy/images/qemux86
5627 NATIVE_SYSROOT: /home/stephano/build/master/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
5628
5629 INFO: The image(s) were created using OE kickstart file:
5630 /home/stephano/poky/scripts/lib/wic/canned-wks/directdisksdb-gpt.wks
5631
5632Continuing with the example, you can now directly ``dd`` the image to a
5633USB stick, or whatever media for which you built your image, and boot
Andrew Geisslerc926e172021-05-07 16:11:35 -05005634the resulting media::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005635
5636 $ sudo dd if=directdisksdb-gpt-201710090938-sdb.direct of=/dev/sdb
5637 140966+0 records in
5638 140966+0 records out
5639 72174592 bytes (72 MB, 69 MiB) copied, 78.0282 s, 925 kB/s
5640 $ sudo eject /dev/sdb
5641
5642Using a Modified Kickstart File and Running in Raw Mode
5643~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5644
5645This next example manually specifies each build artifact (runs in Raw
5646Mode) and uses a modified kickstart file. The example also uses the
5647``-o`` option to cause Wic to create the output somewhere other than the
Andrew Geisslerc926e172021-05-07 16:11:35 -05005648default output directory, which is the current directory::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005649
5650 $ wic create /home/stephano/my_yocto/test.wks -o /home/stephano/testwic \
5651 --rootfs-dir /home/stephano/build/master/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/rootfs \
5652 --bootimg-dir /home/stephano/build/master/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share \
5653 --kernel-dir /home/stephano/build/master/build/tmp/deploy/images/qemux86 \
5654 --native-sysroot /home/stephano/build/master/build/tmp/work/i586-poky-linux/wic-tools/1.0-r0/recipe-sysroot-native
5655
5656 INFO: Creating image(s)...
5657
5658 INFO: The new image(s) can be found here:
5659 /home/stephano/testwic/test-201710091445-sdb.direct
5660
5661 The following build artifacts were used to create the image(s):
5662 ROOTFS_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
5663 BOOTIMG_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
5664 KERNEL_DIR: /home/stephano/build/master/build/tmp-glibc/deploy/images/qemux86
5665 NATIVE_SYSROOT: /home/stephano/build/master/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
5666
5667 INFO: The image(s) were created using OE kickstart file:
5668 /home/stephano/my_yocto/test.wks
5669
5670For this example,
5671:term:`MACHINE` did not have to be
5672specified in the ``local.conf`` file since the artifact is manually
5673specified.
5674
5675Using Wic to Manipulate an Image
5676~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5677
5678Wic image manipulation allows you to shorten turnaround time during
5679image development. For example, you can use Wic to delete the kernel
5680partition of a Wic image and then insert a newly built kernel. This
5681saves you time from having to rebuild the entire image each time you
5682modify the kernel.
5683
5684.. note::
5685
5686 In order to use Wic to manipulate a Wic image as in this example,
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005687 your development machine must have the ``mtools`` package installed.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005688
5689The following example examines the contents of the Wic image, deletes
5690the existing kernel, and then inserts a new kernel:
5691
56921. *List the Partitions:* Use the ``wic ls`` command to list all the
Andrew Geisslerc926e172021-05-07 16:11:35 -05005693 partitions in the Wic image::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005694
5695 $ wic ls tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic
5696 Num Start End Size Fstype
5697 1 1048576 25041919 23993344 fat16
5698 2 25165824 72157183 46991360 ext4
5699
5700 The previous output shows two partitions in the
5701 ``core-image-minimal-qemux86.wic`` image.
5702
57032. *Examine a Particular Partition:* Use the ``wic ls`` command again
5704 but in a different form to examine a particular partition.
5705
5706 .. note::
5707
5708 You can get command usage on any Wic command using the following
Andrew Geisslerc926e172021-05-07 16:11:35 -05005709 form::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005710
5711 $ wic help command
5712
5713
5714 For example, the following command shows you the various ways to
5715 use the
5716 wic ls
Andrew Geisslerc926e172021-05-07 16:11:35 -05005717 command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005718
5719 $ wic help ls
5720
5721
Andrew Geisslerc926e172021-05-07 16:11:35 -05005722 The following command shows what is in partition one::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005723
5724 $ wic ls tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1
5725 Volume in drive : is boot
5726 Volume Serial Number is E894-1809
5727 Directory for ::/
5728
5729 libcom32 c32 186500 2017-10-09 16:06
5730 libutil c32 24148 2017-10-09 16:06
5731 syslinux cfg 220 2017-10-09 16:06
5732 vesamenu c32 27104 2017-10-09 16:06
5733 vmlinuz 6904608 2017-10-09 16:06
5734 5 files 7 142 580 bytes
5735 16 582 656 bytes free
5736
5737 The previous output shows five files, with the
5738 ``vmlinuz`` being the kernel.
5739
5740 .. note::
5741
5742 If you see the following error, you need to update or create a
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005743 ``~/.mtoolsrc`` file and be sure to have the line "mtools_skip_check=1"
Andrew Geisslerc926e172021-05-07 16:11:35 -05005744 in the file. Then, run the Wic command again::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005745
5746 ERROR: _exec_cmd: /usr/bin/mdir -i /tmp/wic-parttfokuwra ::/ returned '1' instead of 0
5747 output: Total number of sectors (47824) not a multiple of sectors per track (32)!
5748 Add mtools_skip_check=1 to your .mtoolsrc file to skip this test
5749
5750
57513. *Remove the Old Kernel:* Use the ``wic rm`` command to remove the
Andrew Geisslerc926e172021-05-07 16:11:35 -05005752 ``vmlinuz`` file (kernel)::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005753
5754 $ wic rm tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1/vmlinuz
5755
57564. *Add In the New Kernel:* Use the ``wic cp`` command to add the
5757 updated kernel to the Wic image. Depending on how you built your
5758 kernel, it could be in different places. If you used ``devtool`` and
5759 an SDK to build your kernel, it resides in the ``tmp/work`` directory
5760 of the extensible SDK. If you used ``make`` to build the kernel, the
5761 kernel will be in the ``workspace/sources`` area.
5762
5763 The following example assumes ``devtool`` was used to build the
Andrew Geisslerc926e172021-05-07 16:11:35 -05005764 kernel::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005765
Andrew Geissler95ac1b82021-03-31 14:34:31 -05005766 $ wic cp poky_sdk/tmp/work/qemux86-poky-linux/linux-yocto/4.12.12+git999-r0/linux-yocto-4.12.12+git999/arch/x86/boot/bzImage \
5767 poky/build/tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1/vmlinuz
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005768
5769 Once the new kernel is added back into the image, you can use the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005770 ``dd`` command or :ref:`bmaptool
Andrew Geissler09209ee2020-12-13 08:44:15 -06005771 <dev-manual/common-tasks:flashing images using \`\`bmaptool\`\`>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005772 to flash your wic image onto an SD card or USB stick and test your
5773 target.
5774
5775 .. note::
5776
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005777 Using ``bmaptool`` is generally 10 to 20 times faster than using ``dd``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005778
5779Flashing Images Using ``bmaptool``
5780==================================
5781
5782A fast and easy way to flash an image to a bootable device is to use
5783Bmaptool, which is integrated into the OpenEmbedded build system.
5784Bmaptool is a generic tool that creates a file's block map (bmap) and
5785then uses that map to copy the file. As compared to traditional tools
5786such as dd or cp, Bmaptool can copy (or flash) large files like raw
5787system image files much faster.
5788
5789.. note::
5790
5791 - If you are using Ubuntu or Debian distributions, you can install
5792 the ``bmap-tools`` package using the following command and then
5793 use the tool without specifying ``PATH`` even from the root
Andrew Geisslerc926e172021-05-07 16:11:35 -05005794 account::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005795
5796 $ sudo apt-get install bmap-tools
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005797
5798 - If you are unable to install the ``bmap-tools`` package, you will
Andrew Geisslerc926e172021-05-07 16:11:35 -05005799 need to build Bmaptool before using it. Use the following command::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005800
5801 $ bitbake bmap-tools-native
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005802
5803Following, is an example that shows how to flash a Wic image. Realize
5804that while this example uses a Wic image, you can use Bmaptool to flash
5805any type of image. Use these steps to flash an image using Bmaptool:
5806
58071. *Update your local.conf File:* You need to have the following set
Andrew Geisslerc926e172021-05-07 16:11:35 -05005808 in your ``local.conf`` file before building your image::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005809
5810 IMAGE_FSTYPES += "wic wic.bmap"
5811
58122. *Get Your Image:* Either have your image ready (pre-built with the
5813 :term:`IMAGE_FSTYPES`
Andrew Geisslerc926e172021-05-07 16:11:35 -05005814 setting previously mentioned) or take the step to build the image::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005815
5816 $ bitbake image
5817
58183. *Flash the Device:* Flash the device with the image by using Bmaptool
5819 depending on your particular setup. The following commands assume the
5820 image resides in the Build Directory's ``deploy/images/`` area:
5821
Andrew Geisslerc926e172021-05-07 16:11:35 -05005822 - If you have write access to the media, use this command form::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005823
5824 $ oe-run-native bmap-tools-native bmaptool copy build-directory/tmp/deploy/images/machine/image.wic /dev/sdX
5825
5826 - If you do not have write access to the media, set your permissions
Andrew Geisslerc926e172021-05-07 16:11:35 -05005827 first and then use the same command form::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005828
5829 $ sudo chmod 666 /dev/sdX
5830 $ oe-run-native bmap-tools-native bmaptool copy build-directory/tmp/deploy/images/machine/image.wic /dev/sdX
5831
Andrew Geisslerc926e172021-05-07 16:11:35 -05005832For help on the ``bmaptool`` command, use the following command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005833
5834 $ bmaptool --help
5835
5836Making Images More Secure
5837=========================
5838
5839Security is of increasing concern for embedded devices. Consider the
5840issues and problems discussed in just this sampling of work found across
5841the Internet:
5842
5843- *"*\ `Security Risks of Embedded
5844 Systems <https://www.schneier.com/blog/archives/2014/01/security_risks_9.html>`__\ *"*
5845 by Bruce Schneier
5846
5847- *"*\ `Internet Census
5848 2012 <http://census2012.sourceforge.net/paper.html>`__\ *"* by Carna
5849 Botnet
5850
5851- *"*\ `Security Issues for Embedded
Andrew Geisslerd1e89492021-02-12 15:35:20 -06005852 Devices <https://elinux.org/images/6/6f/Security-issues.pdf>`__\ *"*
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005853 by Jake Edge
5854
5855When securing your image is of concern, there are steps, tools, and
5856variables that you can consider to help you reach the security goals you
5857need for your particular device. Not all situations are identical when
5858it comes to making an image secure. Consequently, this section provides
5859some guidance and suggestions for consideration when you want to make
5860your image more secure.
5861
5862.. note::
5863
5864 Because the security requirements and risks are different for every
5865 type of device, this section cannot provide a complete reference on
5866 securing your custom OS. It is strongly recommended that you also
5867 consult other sources of information on embedded Linux system
5868 hardening and on security.
5869
5870General Considerations
5871----------------------
5872
William A. Kennington IIIac69b482021-06-02 12:28:27 -07005873There are general considerations that help you create more secure images.
5874You should consider the following suggestions to make your device
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005875more secure:
5876
5877- Scan additional code you are adding to the system (e.g. application
5878 code) by using static analysis tools. Look for buffer overflows and
5879 other potential security problems.
5880
5881- Pay particular attention to the security for any web-based
5882 administration interface.
5883
5884 Web interfaces typically need to perform administrative functions and
5885 tend to need to run with elevated privileges. Thus, the consequences
5886 resulting from the interface's security becoming compromised can be
5887 serious. Look for common web vulnerabilities such as
5888 cross-site-scripting (XSS), unvalidated inputs, and so forth.
5889
5890 As with system passwords, the default credentials for accessing a
5891 web-based interface should not be the same across all devices. This
5892 is particularly true if the interface is enabled by default as it can
5893 be assumed that many end-users will not change the credentials.
5894
5895- Ensure you can update the software on the device to mitigate
5896 vulnerabilities discovered in the future. This consideration
5897 especially applies when your device is network-enabled.
5898
5899- Ensure you remove or disable debugging functionality before producing
5900 the final image. For information on how to do this, see the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05005901 ":ref:`dev-manual/common-tasks:considerations specific to the openembedded build system`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005902 section.
5903
5904- Ensure you have no network services listening that are not needed.
5905
5906- Remove any software from the image that is not needed.
5907
5908- Enable hardware support for secure boot functionality when your
5909 device supports this functionality.
5910
5911Security Flags
5912--------------
5913
5914The Yocto Project has security flags that you can enable that help make
5915your build output more secure. The security flags are in the
5916``meta/conf/distro/include/security_flags.inc`` file in your
5917:term:`Source Directory` (e.g. ``poky``).
5918
5919.. note::
5920
5921 Depending on the recipe, certain security flags are enabled and
5922 disabled by default.
5923
5924Use the following line in your ``local.conf`` file or in your custom
5925distribution configuration file to enable the security compiler and
Andrew Geisslerc926e172021-05-07 16:11:35 -05005926linker flags for your build::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005927
5928 require conf/distro/include/security_flags.inc
5929
5930Considerations Specific to the OpenEmbedded Build System
5931--------------------------------------------------------
5932
5933You can take some steps that are specific to the OpenEmbedded build
5934system to make your images more secure:
5935
5936- Ensure "debug-tweaks" is not one of your selected
5937 :term:`IMAGE_FEATURES`.
5938 When creating a new project, the default is to provide you with an
5939 initial ``local.conf`` file that enables this feature using the
5940 :term:`EXTRA_IMAGE_FEATURES`
Andrew Geisslerc926e172021-05-07 16:11:35 -05005941 variable with the line::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005942
5943 EXTRA_IMAGE_FEATURES = "debug-tweaks"
5944
5945 To disable that feature, simply comment out that line in your
Andrew Geissler09036742021-06-25 14:25:14 -05005946 ``local.conf`` file, or make sure :term:`IMAGE_FEATURES` does not contain
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005947 "debug-tweaks" before producing your final image. Among other things,
5948 leaving this in place sets the root password as blank, which makes
5949 logging in for debugging or inspection easy during development but
5950 also means anyone can easily log in during production.
5951
5952- It is possible to set a root password for the image and also to set
5953 passwords for any extra users you might add (e.g. administrative or
5954 service type users). When you set up passwords for multiple images or
5955 users, you should not duplicate passwords.
5956
5957 To set up passwords, use the
5958 :ref:`extrausers <ref-classes-extrausers>`
5959 class, which is the preferred method. For an example on how to set up
5960 both root and user passwords, see the
5961 ":ref:`extrausers.bbclass <ref-classes-extrausers>`"
5962 section.
5963
5964 .. note::
5965
5966 When adding extra user accounts or setting a root password, be
5967 cautious about setting the same password on every device. If you
5968 do this, and the password you have set is exposed, then every
5969 device is now potentially compromised. If you need this access but
5970 want to ensure security, consider setting a different, random
5971 password for each device. Typically, you do this as a separate
5972 step after you deploy the image onto the device.
5973
5974- Consider enabling a Mandatory Access Control (MAC) framework such as
5975 SMACK or SELinux and tuning it appropriately for your device's usage.
5976 You can find more information in the
Andrew Geissler09209ee2020-12-13 08:44:15 -06005977 :yocto_git:`meta-selinux </meta-selinux/>` layer.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005978
5979Tools for Hardening Your Image
5980------------------------------
5981
5982The Yocto Project provides tools for making your image more secure. You
5983can find these tools in the ``meta-security`` layer of the
5984:yocto_git:`Yocto Project Source Repositories <>`.
5985
5986Creating Your Own Distribution
5987==============================
5988
5989When you build an image using the Yocto Project and do not alter any
5990distribution :term:`Metadata`, you are
5991creating a Poky distribution. If you wish to gain more control over
5992package alternative selections, compile-time options, and other
5993low-level configurations, you can create your own distribution.
5994
5995To create your own distribution, the basic steps consist of creating
5996your own distribution layer, creating your own distribution
5997configuration file, and then adding any needed code and Metadata to the
5998layer. The following steps provide some more detail:
5999
6000- *Create a layer for your new distro:* Create your distribution layer
6001 so that you can keep your Metadata and code for the distribution
6002 separate. It is strongly recommended that you create and use your own
6003 layer for configuration and code. Using your own layer as compared to
6004 just placing configurations in a ``local.conf`` configuration file
6005 makes it easier to reproduce the same build configuration when using
6006 multiple build machines. See the
Andrew Geissler09209ee2020-12-13 08:44:15 -06006007 ":ref:`dev-manual/common-tasks:creating a general layer using the \`\`bitbake-layers\`\` script`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006008 section for information on how to quickly set up a layer.
6009
6010- *Create the distribution configuration file:* The distribution
6011 configuration file needs to be created in the ``conf/distro``
6012 directory of your layer. You need to name it using your distribution
6013 name (e.g. ``mydistro.conf``).
6014
6015 .. note::
6016
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006017 The :term:`DISTRO` variable in your ``local.conf`` file determines the
6018 name of your distribution.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006019
6020 You can split out parts of your configuration file into include files
6021 and then "require" them from within your distribution configuration
6022 file. Be sure to place the include files in the
6023 ``conf/distro/include`` directory of your layer. A common example
6024 usage of include files would be to separate out the selection of
6025 desired version and revisions for individual recipes.
6026
6027 Your configuration file needs to set the following required
6028 variables:
6029
6030 - :term:`DISTRO_NAME`
6031
6032 - :term:`DISTRO_VERSION`
6033
6034 These following variables are optional and you typically set them
6035 from the distribution configuration file:
6036
6037 - :term:`DISTRO_FEATURES`
6038
6039 - :term:`DISTRO_EXTRA_RDEPENDS`
6040
6041 - :term:`DISTRO_EXTRA_RRECOMMENDS`
6042
6043 - :term:`TCLIBC`
6044
6045 .. tip::
6046
6047 If you want to base your distribution configuration file on the
6048 very basic configuration from OE-Core, you can use
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006049 ``conf/distro/defaultsetup.conf`` as a reference and just include
6050 variables that differ as compared to ``defaultsetup.conf``.
6051 Alternatively, you can create a distribution configuration file
6052 from scratch using the ``defaultsetup.conf`` file or configuration files
6053 from other distributions such as Poky or Angstrom as references.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006054
6055- *Provide miscellaneous variables:* Be sure to define any other
6056 variables for which you want to create a default or enforce as part
6057 of the distribution configuration. You can include nearly any
6058 variable from the ``local.conf`` file. The variables you use are not
6059 limited to the list in the previous bulleted item.
6060
6061- *Point to Your distribution configuration file:* In your
6062 ``local.conf`` file in the :term:`Build Directory`,
6063 set your
6064 :term:`DISTRO` variable to point to
6065 your distribution's configuration file. For example, if your
6066 distribution's configuration file is named ``mydistro.conf``, then
Andrew Geisslerc926e172021-05-07 16:11:35 -05006067 you point to it as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006068
6069 DISTRO = "mydistro"
6070
6071- *Add more to the layer if necessary:* Use your layer to hold other
6072 information needed for the distribution:
6073
6074 - Add recipes for installing distro-specific configuration files
6075 that are not already installed by another recipe. If you have
6076 distro-specific configuration files that are included by an
6077 existing recipe, you should add an append file (``.bbappend``) for
6078 those. For general information and recommendations on how to add
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006079 recipes to your layer, see the
6080 ":ref:`dev-manual/common-tasks:creating your own layer`" and
6081 ":ref:`dev-manual/common-tasks:following best practices when creating layers`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006082 sections.
6083
6084 - Add any image recipes that are specific to your distribution.
6085
6086 - Add a ``psplash`` append file for a branded splash screen. For
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006087 information on append files, see the
6088 ":ref:`dev-manual/common-tasks:using .bbappend files in your layer`"
6089 section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006090
6091 - Add any other append files to make custom changes that are
6092 specific to individual recipes.
6093
6094Creating a Custom Template Configuration Directory
6095==================================================
6096
6097If you are producing your own customized version of the build system for
6098use by other users, you might want to customize the message shown by the
6099setup script or you might want to change the template configuration
6100files (i.e. ``local.conf`` and ``bblayers.conf``) that are created in a
6101new build directory.
6102
6103The OpenEmbedded build system uses the environment variable
6104``TEMPLATECONF`` to locate the directory from which it gathers
6105configuration information that ultimately ends up in the
6106:term:`Build Directory` ``conf`` directory.
6107By default, ``TEMPLATECONF`` is set as follows in the ``poky``
Andrew Geisslerc926e172021-05-07 16:11:35 -05006108repository::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006109
6110 TEMPLATECONF=${TEMPLATECONF:-meta-poky/conf}
6111
6112This is the
6113directory used by the build system to find templates from which to build
6114some key configuration files. If you look at this directory, you will
6115see the ``bblayers.conf.sample``, ``local.conf.sample``, and
6116``conf-notes.txt`` files. The build system uses these files to form the
6117respective ``bblayers.conf`` file, ``local.conf`` file, and display the
6118list of BitBake targets when running the setup script.
6119
6120To override these default configuration files with configurations you
6121want used within every new Build Directory, simply set the
6122``TEMPLATECONF`` variable to your directory. The ``TEMPLATECONF``
6123variable is set in the ``.templateconf`` file, which is in the top-level
6124:term:`Source Directory` folder
6125(e.g. ``poky``). Edit the ``.templateconf`` so that it can locate your
6126directory.
6127
6128Best practices dictate that you should keep your template configuration
6129directory in your custom distribution layer. For example, suppose you
6130have a layer named ``meta-mylayer`` located in your home directory and
6131you want your template configuration directory named ``myconf``.
6132Changing the ``.templateconf`` as follows causes the OpenEmbedded build
6133system to look in your directory and base its configuration files on the
6134``*.sample`` configuration files it finds. The final configuration files
6135(i.e. ``local.conf`` and ``bblayers.conf`` ultimately still end up in
6136your Build Directory, but they are based on your ``*.sample`` files.
6137::
6138
6139 TEMPLATECONF=${TEMPLATECONF:-meta-mylayer/myconf}
6140
6141Aside from the ``*.sample`` configuration files, the ``conf-notes.txt``
6142also resides in the default ``meta-poky/conf`` directory. The script
6143that sets up the build environment (i.e.
6144:ref:`structure-core-script`) uses this file to
6145display BitBake targets as part of the script output. Customizing this
6146``conf-notes.txt`` file is a good way to make sure your list of custom
6147targets appears as part of the script's output.
6148
6149Here is the default list of targets displayed as a result of running
Andrew Geisslerc926e172021-05-07 16:11:35 -05006150either of the setup scripts::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006151
6152 You can now run 'bitbake <target>'
6153
6154 Common targets are:
6155 core-image-minimal
6156 core-image-sato
6157 meta-toolchain
6158 meta-ide-support
6159
6160Changing the listed common targets is as easy as editing your version of
6161``conf-notes.txt`` in your custom template configuration directory and
6162making sure you have ``TEMPLATECONF`` set to your directory.
6163
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006164Conserving Disk Space During Builds
6165===================================
6166
6167To help conserve disk space during builds, you can add the following
6168statement to your project's ``local.conf`` configuration file found in
Andrew Geisslerc926e172021-05-07 16:11:35 -05006169the :term:`Build Directory`::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006170
6171 INHERIT += "rm_work"
6172
6173Adding this statement deletes the work directory used for
6174building a recipe once the recipe is built. For more information on
6175"rm_work", see the
6176:ref:`rm_work <ref-classes-rm-work>` class in the
6177Yocto Project Reference Manual.
6178
6179Working with Packages
6180=====================
6181
6182This section describes a few tasks that involve packages:
6183
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006184- :ref:`dev-manual/common-tasks:excluding packages from an image`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006185
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006186- :ref:`dev-manual/common-tasks:incrementing a package version`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006187
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006188- :ref:`dev-manual/common-tasks:handling optional module packaging`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006189
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006190- :ref:`dev-manual/common-tasks:using runtime package management`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006191
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006192- :ref:`dev-manual/common-tasks:generating and using signed packages`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006193
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006194- :ref:`Setting up and running package test
6195 (ptest) <dev-manual/common-tasks:testing packages with ptest>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006196
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006197- :ref:`dev-manual/common-tasks:creating node package manager (npm) packages`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006198
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006199- :ref:`dev-manual/common-tasks:adding custom metadata to packages`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006200
6201Excluding Packages from an Image
6202--------------------------------
6203
6204You might find it necessary to prevent specific packages from being
6205installed into an image. If so, you can use several variables to direct
6206the build system to essentially ignore installing recommended packages
6207or to not install a package at all.
6208
6209The following list introduces variables you can use to prevent packages
6210from being installed into your image. Each of these variables only works
William A. Kennington IIIac69b482021-06-02 12:28:27 -07006211with IPK and RPM package types, not for Debian packages.
6212Also, you can use these variables from your ``local.conf`` file
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006213or attach them to a specific image recipe by using a recipe name
6214override. For more detail on the variables, see the descriptions in the
6215Yocto Project Reference Manual's glossary chapter.
6216
6217- :term:`BAD_RECOMMENDATIONS`:
6218 Use this variable to specify "recommended-only" packages that you do
6219 not want installed.
6220
6221- :term:`NO_RECOMMENDATIONS`:
6222 Use this variable to prevent all "recommended-only" packages from
6223 being installed.
6224
6225- :term:`PACKAGE_EXCLUDE`:
6226 Use this variable to prevent specific packages from being installed
6227 regardless of whether they are "recommended-only" or not. You need to
6228 realize that the build process could fail with an error when you
6229 prevent the installation of a package whose presence is required by
6230 an installed package.
6231
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006232Incrementing a Package Version
6233------------------------------
6234
6235This section provides some background on how binary package versioning
6236is accomplished and presents some of the services, variables, and
6237terminology involved.
6238
6239In order to understand binary package versioning, you need to consider
6240the following:
6241
6242- Binary Package: The binary package that is eventually built and
6243 installed into an image.
6244
6245- Binary Package Version: The binary package version is composed of two
6246 components - a version and a revision.
6247
6248 .. note::
6249
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006250 Technically, a third component, the "epoch" (i.e. :term:`PE`) is involved
Andrew Geissler09036742021-06-25 14:25:14 -05006251 but this discussion for the most part ignores :term:`PE`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006252
6253 The version and revision are taken from the
6254 :term:`PV` and
6255 :term:`PR` variables, respectively.
6256
Andrew Geissler09036742021-06-25 14:25:14 -05006257- :term:`PV`: The recipe version. :term:`PV` represents the version of the
6258 software being packaged. Do not confuse :term:`PV` with the binary
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006259 package version.
6260
Andrew Geissler5f350902021-07-23 13:09:54 -04006261- :term:`PR`: The recipe revision.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006262
6263- :term:`SRCPV`: The OpenEmbedded
Andrew Geissler09036742021-06-25 14:25:14 -05006264 build system uses this string to help define the value of :term:`PV` when
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006265 the source code revision needs to be included in it.
6266
Andrew Geissler09209ee2020-12-13 08:44:15 -06006267- :yocto_wiki:`PR Service </PR_Service>`: A
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006268 network-based service that helps automate keeping package feeds
6269 compatible with existing package manager applications such as RPM,
6270 APT, and OPKG.
6271
6272Whenever the binary package content changes, the binary package version
6273must change. Changing the binary package version is accomplished by
Andrew Geissler09036742021-06-25 14:25:14 -05006274changing or "bumping" the :term:`PR` and/or :term:`PV` values. Increasing these
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006275values occurs one of two ways:
6276
6277- Automatically using a Package Revision Service (PR Service).
6278
Andrew Geissler09036742021-06-25 14:25:14 -05006279- Manually incrementing the :term:`PR` and/or :term:`PV` variables.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006280
6281Given a primary challenge of any build system and its users is how to
6282maintain a package feed that is compatible with existing package manager
6283applications such as RPM, APT, and OPKG, using an automated system is
6284much preferred over a manual system. In either system, the main
6285requirement is that binary package version numbering increases in a
William A. Kennington IIIac69b482021-06-02 12:28:27 -07006286linear fashion and that there is a number of version components that
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006287support that linear progression. For information on how to ensure
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006288package revisioning remains linear, see the
6289":ref:`dev-manual/common-tasks:automatically incrementing a package version number`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006290section.
6291
6292The following three sections provide related information on the PR
Andrew Geissler09036742021-06-25 14:25:14 -05006293Service, the manual method for "bumping" :term:`PR` and/or :term:`PV`, and on
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006294how to ensure binary package revisioning remains linear.
6295
6296Working With a PR Service
6297~~~~~~~~~~~~~~~~~~~~~~~~~
6298
6299As mentioned, attempting to maintain revision numbers in the
6300:term:`Metadata` is error prone, inaccurate,
6301and causes problems for people submitting recipes. Conversely, the PR
6302Service automatically generates increasing numbers, particularly the
6303revision field, which removes the human element.
6304
6305.. note::
6306
6307 For additional information on using a PR Service, you can see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06006308 :yocto_wiki:`PR Service </PR_Service>` wiki page.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006309
6310The Yocto Project uses variables in order of decreasing priority to
6311facilitate revision numbering (i.e.
6312:term:`PE`,
6313:term:`PV`, and
6314:term:`PR` for epoch, version, and
6315revision, respectively). The values are highly dependent on the policies
6316and procedures of a given distribution and package feed.
6317
6318Because the OpenEmbedded build system uses
Andrew Geissler09209ee2020-12-13 08:44:15 -06006319":ref:`signatures <overview-manual/concepts:checksums (signatures)>`", which are
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006320unique to a given build, the build system knows when to rebuild
6321packages. All the inputs into a given task are represented by a
6322signature, which can trigger a rebuild when different. Thus, the build
Andrew Geissler09036742021-06-25 14:25:14 -05006323system itself does not rely on the :term:`PR`, :term:`PV`, and :term:`PE` numbers to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006324trigger a rebuild. The signatures, however, can be used to generate
6325these values.
6326
6327The PR Service works with both ``OEBasic`` and ``OEBasicHash``
Andrew Geissler09036742021-06-25 14:25:14 -05006328generators. The value of :term:`PR` bumps when the checksum changes and the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006329different generator mechanisms change signatures under different
6330circumstances.
6331
6332As implemented, the build system includes values from the PR Service
Andrew Geissler09036742021-06-25 14:25:14 -05006333into the :term:`PR` field as an addition using the form "``.x``" so ``r0``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006334becomes ``r0.1``, ``r0.2`` and so forth. This scheme allows existing
Andrew Geissler09036742021-06-25 14:25:14 -05006335:term:`PR` values to be used for whatever reasons, which include manual
6336:term:`PR` bumps, should it be necessary.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006337
6338By default, the PR Service is not enabled or running. Thus, the packages
6339generated are just "self consistent". The build system adds and removes
6340packages and there are no guarantees about upgrade paths but images will
6341be consistent and correct with the latest changes.
6342
William A. Kennington IIIac69b482021-06-02 12:28:27 -07006343The simplest form for a PR Service is for a single host
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006344development system that builds the package feed (building system). For
6345this scenario, you can enable a local PR Service by setting
6346:term:`PRSERV_HOST` in your
Andrew Geisslerc926e172021-05-07 16:11:35 -05006347``local.conf`` file in the :term:`Build Directory`::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006348
6349 PRSERV_HOST = "localhost:0"
6350
6351Once the service is started, packages will automatically
Andrew Geissler09036742021-06-25 14:25:14 -05006352get increasing :term:`PR` values and BitBake takes care of starting and
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006353stopping the server.
6354
6355If you have a more complex setup where multiple host development systems
6356work against a common, shared package feed, you have a single PR Service
6357running and it is connected to each building system. For this scenario,
Andrew Geisslerc926e172021-05-07 16:11:35 -05006358you need to start the PR Service using the ``bitbake-prserv`` command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006359
6360 bitbake-prserv --host ip --port port --start
6361
6362In addition to
6363hand-starting the service, you need to update the ``local.conf`` file of
6364each building system as described earlier so each system points to the
6365server and port.
6366
6367It is also recommended you use build history, which adds some sanity
6368checks to binary package versions, in conjunction with the server that
6369is running the PR Service. To enable build history, add the following to
Andrew Geisslerc926e172021-05-07 16:11:35 -05006370each building system's ``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006371
6372 # It is recommended to activate "buildhistory" for testing the PR service
6373 INHERIT += "buildhistory"
6374 BUILDHISTORY_COMMIT = "1"
6375
6376For information on build
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006377history, see the
6378":ref:`dev-manual/common-tasks:maintaining build output quality`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006379
6380.. note::
6381
Andrew Geissler09036742021-06-25 14:25:14 -05006382 The OpenEmbedded build system does not maintain :term:`PR` information as
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006383 part of the shared state (sstate) packages. If you maintain an sstate
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006384 feed, it's expected that either all your building systems that
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006385 contribute to the sstate feed use a shared PR Service, or you do not
6386 run a PR Service on any of your building systems. Having some systems
6387 use a PR Service while others do not leads to obvious problems.
6388
6389 For more information on shared state, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06006390 ":ref:`overview-manual/concepts:shared state cache`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006391 section in the Yocto Project Overview and Concepts Manual.
6392
6393Manually Bumping PR
6394~~~~~~~~~~~~~~~~~~~
6395
6396The alternative to setting up a PR Service is to manually "bump" the
6397:term:`PR` variable.
6398
6399If a committed change results in changing the package output, then the
6400value of the PR variable needs to be increased (or "bumped") as part of
Andrew Geissler09036742021-06-25 14:25:14 -05006401that commit. For new recipes you should add the :term:`PR` variable and set
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006402its initial value equal to "r0", which is the default. Even though the
6403default value is "r0", the practice of adding it to a new recipe makes
6404it harder to forget to bump the variable when you make changes to the
6405recipe in future.
6406
6407If you are sharing a common ``.inc`` file with multiple recipes, you can
Andrew Geissler09036742021-06-25 14:25:14 -05006408also use the :term:`INC_PR` variable to ensure that the recipes sharing the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006409``.inc`` file are rebuilt when the ``.inc`` file itself is changed. The
Andrew Geissler09036742021-06-25 14:25:14 -05006410``.inc`` file must set :term:`INC_PR` (initially to "r0"), and all recipes
6411referring to it should set :term:`PR` to "${INC_PR}.0" initially,
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006412incrementing the last number when the recipe is changed. If the ``.inc``
Andrew Geissler09036742021-06-25 14:25:14 -05006413file is changed then its :term:`INC_PR` should be incremented.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006414
Andrew Geissler09036742021-06-25 14:25:14 -05006415When upgrading the version of a binary package, assuming the :term:`PV`
6416changes, the :term:`PR` variable should be reset to "r0" (or "${INC_PR}.0"
6417if you are using :term:`INC_PR`).
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006418
6419Usually, version increases occur only to binary packages. However, if
Andrew Geissler09036742021-06-25 14:25:14 -05006420for some reason :term:`PV` changes but does not increase, you can increase
6421the :term:`PE` variable (Package Epoch). The :term:`PE` variable defaults to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006422"0".
6423
6424Binary package version numbering strives to follow the `Debian Version
6425Field Policy
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006426Guidelines <https://www.debian.org/doc/debian-policy/ch-controlfields.html>`__.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006427These guidelines define how versions are compared and what "increasing"
6428a version means.
6429
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006430Automatically Incrementing a Package Version Number
6431~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
6432
6433When fetching a repository, BitBake uses the
6434:term:`SRCREV` variable to determine
6435the specific source code revision from which to build. You set the
Andrew Geissler09036742021-06-25 14:25:14 -05006436:term:`SRCREV` variable to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006437:term:`AUTOREV` to cause the
6438OpenEmbedded build system to automatically use the latest revision of
Andrew Geisslerc926e172021-05-07 16:11:35 -05006439the software::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006440
6441 SRCREV = "${AUTOREV}"
6442
Andrew Geissler09036742021-06-25 14:25:14 -05006443Furthermore, you need to reference :term:`SRCPV` in :term:`PV` in order to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006444automatically update the version whenever the revision of the source
Andrew Geisslerc926e172021-05-07 16:11:35 -05006445code changes. Here is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006446
6447 PV = "1.0+git${SRCPV}"
6448
Andrew Geissler09036742021-06-25 14:25:14 -05006449The OpenEmbedded build system substitutes :term:`SRCPV` with the following:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006450
6451.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006452
6453 AUTOINC+source_code_revision
6454
6455The build system replaces the ``AUTOINC``
6456with a number. The number used depends on the state of the PR Service:
6457
6458- If PR Service is enabled, the build system increments the number,
6459 which is similar to the behavior of
6460 :term:`PR`. This behavior results in
6461 linearly increasing package versions, which is desirable. Here is an
6462 example:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006463
6464 .. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006465
6466 hello-world-git_0.0+git0+b6558dd387-r0.0_armv7a-neon.ipk
6467 hello-world-git_0.0+git1+dd2f5c3565-r0.0_armv7a-neon.ipk
6468
6469- If PR Service is not enabled, the build system replaces the
6470 ``AUTOINC`` placeholder with zero (i.e. "0"). This results in
6471 changing the package version since the source revision is included.
6472 However, package versions are not increased linearly. Here is an
6473 example:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006474
6475 .. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006476
6477 hello-world-git_0.0+git0+b6558dd387-r0.0_armv7a-neon.ipk
6478 hello-world-git_0.0+git0+dd2f5c3565-r0.0_armv7a-neon.ipk
6479
6480In summary, the OpenEmbedded build system does not track the history of
6481binary package versions for this purpose. ``AUTOINC``, in this case, is
Andrew Geissler09036742021-06-25 14:25:14 -05006482comparable to :term:`PR`. If PR server is not enabled, ``AUTOINC`` in the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006483package version is simply replaced by "0". If PR server is enabled, the
6484build system keeps track of the package versions and bumps the number
6485when the package revision changes.
6486
6487Handling Optional Module Packaging
6488----------------------------------
6489
6490Many pieces of software split functionality into optional modules (or
6491plugins) and the plugins that are built might depend on configuration
6492options. To avoid having to duplicate the logic that determines what
6493modules are available in your recipe or to avoid having to package each
6494module by hand, the OpenEmbedded build system provides functionality to
6495handle module packaging dynamically.
6496
6497To handle optional module packaging, you need to do two things:
6498
6499- Ensure the module packaging is actually done.
6500
6501- Ensure that any dependencies on optional modules from other recipes
6502 are satisfied by your recipe.
6503
6504Making Sure the Packaging is Done
6505~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
6506
6507To ensure the module packaging actually gets done, you use the
6508``do_split_packages`` function within the ``populate_packages`` Python
6509function in your recipe. The ``do_split_packages`` function searches for
6510a pattern of files or directories under a specified path and creates a
6511package for each one it finds by appending to the
6512:term:`PACKAGES` variable and
6513setting the appropriate values for ``FILES_packagename``,
6514``RDEPENDS_packagename``, ``DESCRIPTION_packagename``, and so forth.
Andrew Geisslerc926e172021-05-07 16:11:35 -05006515Here is an example from the ``lighttpd`` recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006516
6517 python populate_packages_prepend () {
6518 lighttpd_libdir = d.expand('${libdir}')
6519 do_split_packages(d, lighttpd_libdir, '^mod_(.*).so$',
6520 'lighttpd-module-%s', 'Lighttpd module for %s',
6521 extra_depends='')
6522 }
6523
6524The previous example specifies a number of things in the call to
6525``do_split_packages``.
6526
6527- A directory within the files installed by your recipe through
6528 ``do_install`` in which to search.
6529
6530- A regular expression used to match module files in that directory. In
6531 the example, note the parentheses () that mark the part of the
6532 expression from which the module name should be derived.
6533
6534- A pattern to use for the package names.
6535
6536- A description for each package.
6537
6538- An empty string for ``extra_depends``, which disables the default
6539 dependency on the main ``lighttpd`` package. Thus, if a file in
6540 ``${libdir}`` called ``mod_alias.so`` is found, a package called
6541 ``lighttpd-module-alias`` is created for it and the
6542 :term:`DESCRIPTION` is set to
6543 "Lighttpd module for alias".
6544
6545Often, packaging modules is as simple as the previous example. However,
William A. Kennington IIIac69b482021-06-02 12:28:27 -07006546there are more advanced options that you can use within
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006547``do_split_packages`` to modify its behavior. And, if you need to, you
6548can add more logic by specifying a hook function that is called for each
6549package. It is also perfectly acceptable to call ``do_split_packages``
6550multiple times if you have more than one set of modules to package.
6551
6552For more examples that show how to use ``do_split_packages``, see the
6553``connman.inc`` file in the ``meta/recipes-connectivity/connman/``
Andrew Geissler09209ee2020-12-13 08:44:15 -06006554directory of the ``poky`` :ref:`source repository <overview-manual/development-environment:yocto project source repositories>`. You can
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006555also find examples in ``meta/classes/kernel.bbclass``.
6556
6557Following is a reference that shows ``do_split_packages`` mandatory and
Andrew Geisslerc926e172021-05-07 16:11:35 -05006558optional arguments::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006559
6560 Mandatory arguments
6561
6562 root
6563 The path in which to search
6564 file_regex
6565 Regular expression to match searched files.
6566 Use parentheses () to mark the part of this
6567 expression that should be used to derive the
6568 module name (to be substituted where %s is
6569 used in other function arguments as noted below)
6570 output_pattern
6571 Pattern to use for the package names. Must
6572 include %s.
6573 description
6574 Description to set for each package. Must
6575 include %s.
6576
6577 Optional arguments
6578
6579 postinst
6580 Postinstall script to use for all packages
6581 (as a string)
6582 recursive
6583 True to perform a recursive search - default
6584 False
6585 hook
6586 A hook function to be called for every match.
6587 The function will be called with the following
6588 arguments (in the order listed):
6589
6590 f
6591 Full path to the file/directory match
6592 pkg
6593 The package name
6594 file_regex
6595 As above
6596 output_pattern
6597 As above
6598 modulename
6599 The module name derived using file_regex
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006600 extra_depends
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006601 Extra runtime dependencies (RDEPENDS) to be
6602 set for all packages. The default value of None
6603 causes a dependency on the main package
6604 (${PN}) - if you do not want this, pass empty
6605 string '' for this parameter.
6606 aux_files_pattern
6607 Extra item(s) to be added to FILES for each
6608 package. Can be a single string item or a list
6609 of strings for multiple items. Must include %s.
6610 postrm
6611 postrm script to use for all packages (as a
6612 string)
6613 allow_dirs
6614 True to allow directories to be matched -
6615 default False
6616 prepend
6617 If True, prepend created packages to PACKAGES
6618 instead of the default False which appends them
6619 match_path
6620 match file_regex on the whole relative path to
6621 the root rather than just the file name
6622 aux_files_pattern_verbatim
6623 Extra item(s) to be added to FILES for each
6624 package, using the actual derived module name
6625 rather than converting it to something legal
6626 for a package name. Can be a single string item
6627 or a list of strings for multiple items. Must
6628 include %s.
6629 allow_links
6630 True to allow symlinks to be matched - default
6631 False
6632 summary
6633 Summary to set for each package. Must include %s;
6634 defaults to description if not set.
6635
6636
6637
6638Satisfying Dependencies
6639~~~~~~~~~~~~~~~~~~~~~~~
6640
6641The second part for handling optional module packaging is to ensure that
6642any dependencies on optional modules from other recipes are satisfied by
6643your recipe. You can be sure these dependencies are satisfied by using
6644the :term:`PACKAGES_DYNAMIC`
6645variable. Here is an example that continues with the ``lighttpd`` recipe
Andrew Geisslerc926e172021-05-07 16:11:35 -05006646shown earlier::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006647
6648 PACKAGES_DYNAMIC = "lighttpd-module-.*"
6649
6650The name
6651specified in the regular expression can of course be anything. In this
6652example, it is ``lighttpd-module-`` and is specified as the prefix to
6653ensure that any :term:`RDEPENDS` and
6654:term:`RRECOMMENDS` on a package
6655name starting with the prefix are satisfied during build time. If you
6656are using ``do_split_packages`` as described in the previous section,
Andrew Geissler09036742021-06-25 14:25:14 -05006657the value you put in :term:`PACKAGES_DYNAMIC` should correspond to the name
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006658pattern specified in the call to ``do_split_packages``.
6659
6660Using Runtime Package Management
6661--------------------------------
6662
6663During a build, BitBake always transforms a recipe into one or more
6664packages. For example, BitBake takes the ``bash`` recipe and produces a
6665number of packages (e.g. ``bash``, ``bash-bashbug``,
6666``bash-completion``, ``bash-completion-dbg``, ``bash-completion-dev``,
6667``bash-completion-extra``, ``bash-dbg``, and so forth). Not all
6668generated packages are included in an image.
6669
6670In several situations, you might need to update, add, remove, or query
6671the packages on a target device at runtime (i.e. without having to
6672generate a new image). Examples of such situations include:
6673
6674- You want to provide in-the-field updates to deployed devices (e.g.
6675 security updates).
6676
6677- You want to have a fast turn-around development cycle for one or more
6678 applications that run on your device.
6679
6680- You want to temporarily install the "debug" packages of various
6681 applications on your device so that debugging can be greatly improved
6682 by allowing access to symbols and source debugging.
6683
6684- You want to deploy a more minimal package selection of your device
6685 but allow in-the-field updates to add a larger selection for
6686 customization.
6687
6688In all these situations, you have something similar to a more
6689traditional Linux distribution in that in-field devices are able to
6690receive pre-compiled packages from a server for installation or update.
6691Being able to install these packages on a running, in-field device is
6692what is termed "runtime package management".
6693
6694In order to use runtime package management, you need a host or server
6695machine that serves up the pre-compiled packages plus the required
6696metadata. You also need package manipulation tools on the target. The
6697build machine is a likely candidate to act as the server. However, that
6698machine does not necessarily have to be the package server. The build
6699machine could push its artifacts to another machine that acts as the
6700server (e.g. Internet-facing). In fact, doing so is advantageous for a
6701production environment as getting the packages away from the development
6702system's build directory prevents accidental overwrites.
6703
6704A simple build that targets just one device produces more than one
6705package database. In other words, the packages produced by a build are
6706separated out into a couple of different package groupings based on
6707criteria such as the target's CPU architecture, the target board, or the
6708C library used on the target. For example, a build targeting the
6709``qemux86`` device produces the following three package databases:
6710``noarch``, ``i586``, and ``qemux86``. If you wanted your ``qemux86``
6711device to be aware of all the packages that were available to it, you
6712would need to point it to each of these databases individually. In a
6713similar way, a traditional Linux distribution usually is configured to
6714be aware of a number of software repositories from which it retrieves
6715packages.
6716
6717Using runtime package management is completely optional and not required
6718for a successful build or deployment in any way. But if you want to make
6719use of runtime package management, you need to do a couple things above
6720and beyond the basics. The remainder of this section describes what you
6721need to do.
6722
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006723Build Considerations
6724~~~~~~~~~~~~~~~~~~~~
6725
6726This section describes build considerations of which you need to be
6727aware in order to provide support for runtime package management.
6728
6729When BitBake generates packages, it needs to know what format or formats
6730to use. In your configuration, you use the
6731:term:`PACKAGE_CLASSES`
6732variable to specify the format:
6733
67341. Open the ``local.conf`` file inside your
6735 :term:`Build Directory` (e.g.
Andrew Geissler95ac1b82021-03-31 14:34:31 -05006736 ``poky/build/conf/local.conf``).
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006737
Andrew Geisslerc926e172021-05-07 16:11:35 -050067382. Select the desired package format as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006739
6740 PACKAGE_CLASSES ?= "package_packageformat"
6741
6742 where packageformat can be "ipk", "rpm",
6743 "deb", or "tar" which are the supported package formats.
6744
6745 .. note::
6746
6747 Because the Yocto Project supports four different package formats,
6748 you can set the variable with more than one argument. However, the
6749 OpenEmbedded build system only uses the first argument when
6750 creating an image or Software Development Kit (SDK).
6751
6752If you would like your image to start off with a basic package database
6753containing the packages in your current build as well as to have the
6754relevant tools available on the target for runtime package management,
6755you can include "package-management" in the
6756:term:`IMAGE_FEATURES`
6757variable. Including "package-management" in this configuration variable
6758ensures that when the image is assembled for your target, the image
6759includes the currently-known package databases as well as the
6760target-specific tools required for runtime package management to be
6761performed on the target. However, this is not strictly necessary. You
6762could start your image off without any databases but only include the
6763required on-target package tool(s). As an example, you could include
6764"opkg" in your
6765:term:`IMAGE_INSTALL` variable
6766if you are using the IPK package format. You can then initialize your
6767target's package database(s) later once your image is up and running.
6768
6769Whenever you perform any sort of build step that can potentially
6770generate a package or modify existing package, it is always a good idea
6771to re-generate the package index after the build by using the following
Andrew Geisslerc926e172021-05-07 16:11:35 -05006772command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006773
6774 $ bitbake package-index
6775
6776It might be tempting to build the
6777package and the package index at the same time with a command such as
Andrew Geisslerc926e172021-05-07 16:11:35 -05006778the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006779
6780 $ bitbake some-package package-index
6781
6782Do not do this as
6783BitBake does not schedule the package index for after the completion of
6784the package you are building. Consequently, you cannot be sure of the
6785package index including information for the package you just built.
6786Thus, be sure to run the package update step separately after building
6787any packages.
6788
6789You can use the
6790:term:`PACKAGE_FEED_ARCHS`,
6791:term:`PACKAGE_FEED_BASE_PATHS`,
6792and
6793:term:`PACKAGE_FEED_URIS`
6794variables to pre-configure target images to use a package feed. If you
6795do not define these variables, then manual steps as described in the
6796subsequent sections are necessary to configure the target. You should
6797set these variables before building the image in order to produce a
6798correctly configured image.
6799
6800When your build is complete, your packages reside in the
6801``${TMPDIR}/deploy/packageformat`` directory. For example, if
6802``${``\ :term:`TMPDIR`\ ``}`` is
6803``tmp`` and your selected package type is RPM, then your RPM packages
6804are available in ``tmp/deploy/rpm``.
6805
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006806Host or Server Machine Setup
6807~~~~~~~~~~~~~~~~~~~~~~~~~~~~
6808
6809Although other protocols are possible, a server using HTTP typically
6810serves packages. If you want to use HTTP, then set up and configure a
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006811web server such as Apache 2, lighttpd, or Python web server on the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006812machine serving the packages.
6813
6814To keep things simple, this section describes how to set up a
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006815Python web server to share package feeds from the developer's
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006816machine. Although this server might not be the best for a production
6817environment, the setup is simple and straight forward. Should you want
6818to use a different server more suited for production (e.g. Apache 2,
6819Lighttpd, or Nginx), take the appropriate steps to do so.
6820
6821From within the build directory where you have built an image based on
6822your packaging choice (i.e. the
6823:term:`PACKAGE_CLASSES`
6824setting), simply start the server. The following example assumes a build
Andrew Geissler09036742021-06-25 14:25:14 -05006825directory of ``poky/build/tmp/deploy/rpm`` and a :term:`PACKAGE_CLASSES`
Andrew Geisslerc926e172021-05-07 16:11:35 -05006826setting of "package_rpm"::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006827
Andrew Geissler95ac1b82021-03-31 14:34:31 -05006828 $ cd poky/build/tmp/deploy/rpm
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006829 $ python3 -m http.server
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006830
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006831Target Setup
6832~~~~~~~~~~~~
6833
6834Setting up the target differs depending on the package management
6835system. This section provides information for RPM, IPK, and DEB.
6836
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006837Using RPM
6838^^^^^^^^^
6839
6840The `Dandified Packaging
6841Tool <https://en.wikipedia.org/wiki/DNF_(software)>`__ (DNF) performs
6842runtime package management of RPM packages. In order to use DNF for
6843runtime package management, you must perform an initial setup on the
6844target machine for cases where the ``PACKAGE_FEED_*`` variables were not
6845set as part of the image that is running on the target. This means if
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006846you built your image and did not use these variables as part of the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006847build and your image is now running on the target, you need to perform
6848the steps in this section if you want to use runtime package management.
6849
6850.. note::
6851
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006852 For information on the ``PACKAGE_FEED_*`` variables, see
6853 :term:`PACKAGE_FEED_ARCHS`, :term:`PACKAGE_FEED_BASE_PATHS`, and
6854 :term:`PACKAGE_FEED_URIS` in the Yocto Project Reference Manual variables
6855 glossary.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006856
6857On the target, you must inform DNF that package databases are available.
6858You do this by creating a file named
6859``/etc/yum.repos.d/oe-packages.repo`` and defining the ``oe-packages``.
6860
6861As an example, assume the target is able to use the following package
6862databases: ``all``, ``i586``, and ``qemux86`` from a server named
6863``my.server``. The specifics for setting up the web server are up to
6864you. The critical requirement is that the URIs in the target repository
6865configuration point to the correct remote location for the feeds.
6866
6867.. note::
6868
6869 For development purposes, you can point the web server to the build
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006870 system's ``deploy`` directory. However, for production use, it is better to
6871 copy the package directories to a location outside of the build area and use
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006872 that location. Doing so avoids situations where the build system
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006873 overwrites or changes the ``deploy`` directory.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006874
6875When telling DNF where to look for the package databases, you must
6876declare individual locations per architecture or a single location used
6877for all architectures. You cannot do both:
6878
6879- *Create an Explicit List of Architectures:* Define individual base
6880 URLs to identify where each package database is located:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006881
6882 .. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006883
6884 [oe-packages]
6885 baseurl=http://my.server/rpm/i586 http://my.server/rpm/qemux86 http://my.server/rpm/all
6886
6887 This example
6888 informs DNF about individual package databases for all three
6889 architectures.
6890
6891- *Create a Single (Full) Package Index:* Define a single base URL that
Andrew Geisslerc926e172021-05-07 16:11:35 -05006892 identifies where a full package database is located::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006893
6894 [oe-packages]
6895 baseurl=http://my.server/rpm
6896
6897 This example informs DNF about a single
6898 package database that contains all the package index information for
6899 all supported architectures.
6900
6901Once you have informed DNF where to find the package databases, you need
6902to fetch them:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006903
6904.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006905
6906 # dnf makecache
6907
6908DNF is now able to find, install, and
6909upgrade packages from the specified repository or repositories.
6910
6911.. note::
6912
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006913 See the `DNF documentation <https://dnf.readthedocs.io/en/latest/>`__ for
6914 additional information.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006915
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006916Using IPK
6917^^^^^^^^^
6918
6919The ``opkg`` application performs runtime package management of IPK
6920packages. You must perform an initial setup for ``opkg`` on the target
6921machine if the
6922:term:`PACKAGE_FEED_ARCHS`,
6923:term:`PACKAGE_FEED_BASE_PATHS`,
6924and
6925:term:`PACKAGE_FEED_URIS`
6926variables have not been set or the target image was built before the
6927variables were set.
6928
6929The ``opkg`` application uses configuration files to find available
6930package databases. Thus, you need to create a configuration file inside
6931the ``/etc/opkg/`` direction, which informs ``opkg`` of any repository
6932you want to use.
6933
6934As an example, suppose you are serving packages from a ``ipk/``
6935directory containing the ``i586``, ``all``, and ``qemux86`` databases
6936through an HTTP server named ``my.server``. On the target, create a
6937configuration file (e.g. ``my_repo.conf``) inside the ``/etc/opkg/``
6938directory containing the following:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006939
6940.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006941
6942 src/gz all http://my.server/ipk/all
6943 src/gz i586 http://my.server/ipk/i586
6944 src/gz qemux86 http://my.server/ipk/qemux86
6945
6946Next, instruct ``opkg`` to fetch the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006947repository information:
6948
6949.. code-block:: none
6950
6951 # opkg update
6952
6953The ``opkg`` application is now able to find, install, and upgrade packages
6954from the specified repository.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006955
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006956Using DEB
6957^^^^^^^^^
6958
6959The ``apt`` application performs runtime package management of DEB
6960packages. This application uses a source list file to find available
6961package databases. You must perform an initial setup for ``apt`` on the
6962target machine if the
6963:term:`PACKAGE_FEED_ARCHS`,
6964:term:`PACKAGE_FEED_BASE_PATHS`,
6965and
6966:term:`PACKAGE_FEED_URIS`
6967variables have not been set or the target image was built before the
6968variables were set.
6969
6970To inform ``apt`` of the repository you want to use, you might create a
6971list file (e.g. ``my_repo.list``) inside the
6972``/etc/apt/sources.list.d/`` directory. As an example, suppose you are
6973serving packages from a ``deb/`` directory containing the ``i586``,
6974``all``, and ``qemux86`` databases through an HTTP server named
6975``my.server``. The list file should contain:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006976
6977.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006978
6979 deb http://my.server/deb/all ./
6980 deb http://my.server/deb/i586 ./
6981 deb http://my.server/deb/qemux86 ./
6982
6983Next, instruct the ``apt`` application
6984to fetch the repository information:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006985
6986.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006987
6988 # apt-get update
6989
6990After this step,
6991``apt`` is able to find, install, and upgrade packages from the
6992specified repository.
6993
6994Generating and Using Signed Packages
6995------------------------------------
6996
6997In order to add security to RPM packages used during a build, you can
6998take steps to securely sign them. Once a signature is verified, the
6999OpenEmbedded build system can use the package in the build. If security
7000fails for a signed package, the build system aborts the build.
7001
7002This section describes how to sign RPM packages during a build and how
7003to use signed package feeds (repositories) when doing a build.
7004
7005Signing RPM Packages
7006~~~~~~~~~~~~~~~~~~~~
7007
7008To enable signing RPM packages, you must set up the following
7009configurations in either your ``local.config`` or ``distro.config``
Andrew Geisslerc926e172021-05-07 16:11:35 -05007010file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007011
7012 # Inherit sign_rpm.bbclass to enable signing functionality
7013 INHERIT += " sign_rpm"
7014 # Define the GPG key that will be used for signing.
7015 RPM_GPG_NAME = "key_name"
7016 # Provide passphrase for the key
7017 RPM_GPG_PASSPHRASE = "passphrase"
7018
7019.. note::
7020
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007021 Be sure to supply appropriate values for both `key_name` and
7022 `passphrase`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007023
7024Aside from the ``RPM_GPG_NAME`` and ``RPM_GPG_PASSPHRASE`` variables in
William A. Kennington IIIac69b482021-06-02 12:28:27 -07007025the previous example, two optional variables related to signing are available:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007026
7027- *GPG_BIN:* Specifies a ``gpg`` binary/wrapper that is executed
7028 when the package is signed.
7029
7030- *GPG_PATH:* Specifies the ``gpg`` home directory used when the
7031 package is signed.
7032
7033Processing Package Feeds
7034~~~~~~~~~~~~~~~~~~~~~~~~
7035
7036In addition to being able to sign RPM packages, you can also enable
7037signed package feeds for IPK and RPM packages.
7038
7039The steps you need to take to enable signed package feed use are similar
7040to the steps used to sign RPM packages. You must define the following in
Andrew Geisslerc926e172021-05-07 16:11:35 -05007041your ``local.config`` or ``distro.config`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007042
7043 INHERIT += "sign_package_feed"
7044 PACKAGE_FEED_GPG_NAME = "key_name"
7045 PACKAGE_FEED_GPG_PASSPHRASE_FILE = "path_to_file_containing_passphrase"
7046
William A. Kennington IIIac69b482021-06-02 12:28:27 -07007047For signed package feeds, the passphrase must be specified in a separate file,
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007048which is pointed to by the ``PACKAGE_FEED_GPG_PASSPHRASE_FILE``
7049variable. Regarding security, keeping a plain text passphrase out of the
7050configuration is more secure.
7051
7052Aside from the ``PACKAGE_FEED_GPG_NAME`` and
7053``PACKAGE_FEED_GPG_PASSPHRASE_FILE`` variables, three optional variables
William A. Kennington IIIac69b482021-06-02 12:28:27 -07007054related to signed package feeds are available:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007055
7056- *GPG_BIN* Specifies a ``gpg`` binary/wrapper that is executed
7057 when the package is signed.
7058
7059- *GPG_PATH:* Specifies the ``gpg`` home directory used when the
7060 package is signed.
7061
7062- *PACKAGE_FEED_GPG_SIGNATURE_TYPE:* Specifies the type of ``gpg``
7063 signature. This variable applies only to RPM and IPK package feeds.
7064 Allowable values for the ``PACKAGE_FEED_GPG_SIGNATURE_TYPE`` are
7065 "ASC", which is the default and specifies ascii armored, and "BIN",
7066 which specifies binary.
7067
7068Testing Packages With ptest
7069---------------------------
7070
7071A Package Test (ptest) runs tests against packages built by the
7072OpenEmbedded build system on the target machine. A ptest contains at
7073least two items: the actual test, and a shell script (``run-ptest``)
7074that starts the test. The shell script that starts the test must not
7075contain the actual test - the script only starts the test. On the other
7076hand, the test can be anything from a simple shell script that runs a
7077binary and checks the output to an elaborate system of test binaries and
7078data files.
7079
Andrew Geisslerc926e172021-05-07 16:11:35 -05007080The test generates output in the format used by Automake::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007081
7082 result: testname
7083
7084where the result can be ``PASS``, ``FAIL``, or ``SKIP``, and
7085the testname can be any identifying string.
7086
7087For a list of Yocto Project recipes that are already enabled with ptest,
Andrew Geissler09209ee2020-12-13 08:44:15 -06007088see the :yocto_wiki:`Ptest </Ptest>` wiki page.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007089
7090.. note::
7091
7092 A recipe is "ptest-enabled" if it inherits the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007093 :ref:`ptest <ref-classes-ptest>` class.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007094
7095Adding ptest to Your Build
7096~~~~~~~~~~~~~~~~~~~~~~~~~~
7097
7098To add package testing to your build, add the
7099:term:`DISTRO_FEATURES` and
7100:term:`EXTRA_IMAGE_FEATURES`
7101variables to your ``local.conf`` file, which is found in the
Andrew Geisslerc926e172021-05-07 16:11:35 -05007102:term:`Build Directory`::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007103
7104 DISTRO_FEATURES_append = " ptest"
7105 EXTRA_IMAGE_FEATURES += "ptest-pkgs"
7106
7107Once your build is complete, the ptest files are installed into the
7108``/usr/lib/package/ptest`` directory within the image, where ``package``
7109is the name of the package.
7110
7111Running ptest
7112~~~~~~~~~~~~~
7113
7114The ``ptest-runner`` package installs a shell script that loops through
7115all installed ptest test suites and runs them in sequence. Consequently,
7116you might want to add this package to your image.
7117
7118Getting Your Package Ready
7119~~~~~~~~~~~~~~~~~~~~~~~~~~
7120
7121In order to enable a recipe to run installed ptests on target hardware,
7122you need to prepare the recipes that build the packages you want to
7123test. Here is what you have to do for each recipe:
7124
7125- *Be sure the recipe inherits
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007126 the* :ref:`ptest <ref-classes-ptest>` *class:*
Andrew Geisslerc926e172021-05-07 16:11:35 -05007127 Include the following line in each recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007128
7129 inherit ptest
7130
7131- *Create run-ptest:* This script starts your test. Locate the
7132 script where you will refer to it using
7133 :term:`SRC_URI`. Here is an
Andrew Geisslerc926e172021-05-07 16:11:35 -05007134 example that starts a test for ``dbus``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007135
7136 #!/bin/sh
7137 cd test
7138 make -k runtest-TESTS
7139
7140- *Ensure dependencies are met:* If the test adds build or runtime
7141 dependencies that normally do not exist for the package (such as
7142 requiring "make" to run the test suite), use the
7143 :term:`DEPENDS` and
7144 :term:`RDEPENDS` variables in
7145 your recipe in order for the package to meet the dependencies. Here
Andrew Geisslerc926e172021-05-07 16:11:35 -05007146 is an example where the package has a runtime dependency on "make"::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007147
7148 RDEPENDS_${PN}-ptest += "make"
7149
7150- *Add a function to build the test suite:* Not many packages support
7151 cross-compilation of their test suites. Consequently, you usually
7152 need to add a cross-compilation function to the package.
7153
7154 Many packages based on Automake compile and run the test suite by
7155 using a single command such as ``make check``. However, the host
7156 ``make check`` builds and runs on the same computer, while
7157 cross-compiling requires that the package is built on the host but
7158 executed for the target architecture (though often, as in the case
7159 for ptest, the execution occurs on the host). The built version of
7160 Automake that ships with the Yocto Project includes a patch that
7161 separates building and execution. Consequently, packages that use the
7162 unaltered, patched version of ``make check`` automatically
7163 cross-compiles.
7164
7165 Regardless, you still must add a ``do_compile_ptest`` function to
7166 build the test suite. Add a function similar to the following to your
Andrew Geisslerc926e172021-05-07 16:11:35 -05007167 recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007168
7169 do_compile_ptest() {
7170 oe_runmake buildtest-TESTS
7171 }
7172
7173- *Ensure special configurations are set:* If the package requires
7174 special configurations prior to compiling the test code, you must
7175 insert a ``do_configure_ptest`` function into the recipe.
7176
7177- *Install the test suite:* The ``ptest`` class automatically copies
7178 the file ``run-ptest`` to the target and then runs make
7179 ``install-ptest`` to run the tests. If this is not enough, you need
7180 to create a ``do_install_ptest`` function and make sure it gets
7181 called after the "make install-ptest" completes.
7182
7183Creating Node Package Manager (NPM) Packages
7184--------------------------------------------
7185
7186`NPM <https://en.wikipedia.org/wiki/Npm_(software)>`__ is a package
7187manager for the JavaScript programming language. The Yocto Project
Andrew Geissler09209ee2020-12-13 08:44:15 -06007188supports the NPM :ref:`fetcher <bitbake:bitbake-user-manual/bitbake-user-manual-fetching:fetchers>`. You can
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007189use this fetcher in combination with
Andrew Geissler09209ee2020-12-13 08:44:15 -06007190:doc:`devtool </ref-manual/devtool-reference>` to create
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007191recipes that produce NPM packages.
7192
William A. Kennington IIIac69b482021-06-02 12:28:27 -07007193There are two workflows that allow you to create NPM packages using
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007194``devtool``: the NPM registry modules method and the NPM project code
7195method.
7196
7197.. note::
7198
7199 While it is possible to create NPM recipes manually, using
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007200 ``devtool`` is far simpler.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007201
7202Additionally, some requirements and caveats exist.
7203
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007204Requirements and Caveats
7205~~~~~~~~~~~~~~~~~~~~~~~~
7206
7207You need to be aware of the following before using ``devtool`` to create
7208NPM packages:
7209
7210- Of the two methods that you can use ``devtool`` to create NPM
7211 packages, the registry approach is slightly simpler. However, you
7212 might consider the project approach because you do not have to
7213 publish your module in the NPM registry
7214 (`npm-registry <https://docs.npmjs.com/misc/registry>`_), which
7215 is NPM's public registry.
7216
7217- Be familiar with
Andrew Geissler09209ee2020-12-13 08:44:15 -06007218 :doc:`devtool </ref-manual/devtool-reference>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007219
7220- The NPM host tools need the native ``nodejs-npm`` package, which is
7221 part of the OpenEmbedded environment. You need to get the package by
7222 cloning the https://github.com/openembedded/meta-openembedded
7223 repository out of GitHub. Be sure to add the path to your local copy
7224 to your ``bblayers.conf`` file.
7225
7226- ``devtool`` cannot detect native libraries in module dependencies.
7227 Consequently, you must manually add packages to your recipe.
7228
7229- While deploying NPM packages, ``devtool`` cannot determine which
7230 dependent packages are missing on the target (e.g. the node runtime
7231 ``nodejs``). Consequently, you need to find out what files are
7232 missing and be sure they are on the target.
7233
7234- Although you might not need NPM to run your node package, it is
7235 useful to have NPM on your target. The NPM package name is
7236 ``nodejs-npm``.
7237
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007238Using the Registry Modules Method
7239~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
7240
7241This section presents an example that uses the ``cute-files`` module,
7242which is a file browser web application.
7243
7244.. note::
7245
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007246 You must know the ``cute-files`` module version.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007247
7248The first thing you need to do is use ``devtool`` and the NPM fetcher to
Andrew Geisslerc926e172021-05-07 16:11:35 -05007249create the recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007250
7251 $ devtool add "npm://registry.npmjs.org;package=cute-files;version=1.0.2"
7252
7253The
7254``devtool add`` command runs ``recipetool create`` and uses the same
7255fetch URI to download each dependency and capture license details where
7256possible. The result is a generated recipe.
7257
7258The recipe file is fairly simple and contains every license that
7259``recipetool`` finds and includes the licenses in the recipe's
7260:term:`LIC_FILES_CHKSUM`
7261variables. You need to examine the variables and look for those with
7262"unknown" in the :term:`LICENSE`
7263field. You need to track down the license information for "unknown"
7264modules and manually add the information to the recipe.
7265
7266``recipetool`` creates a "shrinkwrap" file for your recipe. Shrinkwrap
7267files capture the version of all dependent modules. Many packages do not
7268provide shrinkwrap files. ``recipetool`` create a shrinkwrap file as it
7269runs.
7270
7271.. note::
7272
7273 A package is created for each sub-module. This policy is the only
7274 practical way to have the licenses for all of the dependencies
7275 represented in the license manifest of the image.
7276
Andrew Geisslerc926e172021-05-07 16:11:35 -05007277The ``devtool edit-recipe`` command lets you take a look at the recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007278
7279 $ devtool edit-recipe cute-files
7280 SUMMARY = "Turn any folder on your computer into a cute file browser, available on the local network."
7281 LICENSE = "MIT & ISC & Unknown"
7282 LIC_FILES_CHKSUM = "file://LICENSE;md5=71d98c0a1db42956787b1909c74a86ca \
7283 file://node_modules/toidentifier/LICENSE;md5=1a261071a044d02eb6f2bb47f51a3502 \
7284 file://node_modules/debug/LICENSE;md5=ddd815a475e7338b0be7a14d8ee35a99 \
7285 ...
7286 SRC_URI = " \
7287 npm://registry.npmjs.org/;package=cute-files;version=${PV} \
7288 npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \
7289 "
7290 S = "${WORKDIR}/npm"
7291 inherit npm LICENSE_${PN} = "MIT"
7292 LICENSE_${PN}-accepts = "MIT"
7293 LICENSE_${PN}-array-flatten = "MIT"
7294 ...
7295 LICENSE_${PN}-vary = "MIT"
7296
William A. Kennington IIIac69b482021-06-02 12:28:27 -07007297Here are three key points in the previous example:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007298
7299- :term:`SRC_URI` uses the NPM
7300 scheme so that the NPM fetcher is used.
7301
7302- ``recipetool`` collects all the license information. If a
7303 sub-module's license is unavailable, the sub-module's name appears in
7304 the comments.
7305
7306- The ``inherit npm`` statement causes the
7307 :ref:`npm <ref-classes-npm>` class to package
7308 up all the modules.
7309
Andrew Geisslerc926e172021-05-07 16:11:35 -05007310You can run the following command to build the ``cute-files`` package::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007311
7312 $ devtool build cute-files
7313
7314Remember that ``nodejs`` must be installed on
7315the target before your package.
7316
7317Assuming 192.168.7.2 for the target's IP address, use the following
Andrew Geisslerc926e172021-05-07 16:11:35 -05007318command to deploy your package::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007319
7320 $ devtool deploy-target -s cute-files root@192.168.7.2
7321
7322Once the package is installed on the target, you can
7323test the application:
7324
7325.. note::
7326
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007327 Because of a known issue, you cannot simply run ``cute-files`` as you would
7328 if you had run ``npm install``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007329
7330::
7331
7332 $ cd /usr/lib/node_modules/cute-files
7333 $ node cute-files.js
7334
7335On a browser,
7336go to ``http://192.168.7.2:3000`` and you see the following:
7337
7338.. image:: figures/cute-files-npm-example.png
7339 :align: center
7340
7341You can find the recipe in ``workspace/recipes/cute-files``. You can use
7342the recipe in any layer you choose.
7343
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007344Using the NPM Projects Code Method
7345~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
7346
7347Although it is useful to package modules already in the NPM registry,
7348adding ``node.js`` projects under development is a more common developer
7349use case.
7350
7351This section covers the NPM projects code method, which is very similar
7352to the "registry" approach described in the previous section. In the NPM
7353projects method, you provide ``devtool`` with an URL that points to the
7354source files.
7355
7356Replicating the same example, (i.e. ``cute-files``) use the following
Andrew Geisslerc926e172021-05-07 16:11:35 -05007357command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007358
7359 $ devtool add https://github.com/martinaglv/cute-files.git
7360
7361The
7362recipe this command generates is very similar to the recipe created in
Andrew Geissler09036742021-06-25 14:25:14 -05007363the previous section. However, the :term:`SRC_URI` looks like the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007364
7365 SRC_URI = " \
7366 git://github.com/martinaglv/cute-files.git;protocol=https \
7367 npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \
7368 "
7369
7370In this example,
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007371the main module is taken from the Git repository and dependencies are
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007372taken from the NPM registry. Other than those differences, the recipe is
7373basically the same between the two methods. You can build and deploy the
7374package exactly as described in the previous section that uses the
7375registry modules method.
7376
7377Adding custom metadata to packages
7378----------------------------------
7379
7380The variable
7381:term:`PACKAGE_ADD_METADATA`
7382can be used to add additional metadata to packages. This is reflected in
7383the package control/spec file. To take the ipk format for example, the
7384CONTROL file stored inside would contain the additional metadata as
7385additional lines.
7386
7387The variable can be used in multiple ways, including using suffixes to
7388set it for a specific package type and/or package. Note that the order
7389of precedence is the same as this list:
7390
7391- ``PACKAGE_ADD_METADATA_<PKGTYPE>_<PN>``
7392
7393- ``PACKAGE_ADD_METADATA_<PKGTYPE>``
7394
7395- ``PACKAGE_ADD_METADATA_<PN>``
7396
Andrew Geissler09036742021-06-25 14:25:14 -05007397- :term:`PACKAGE_ADD_METADATA`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007398
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007399`<PKGTYPE>` is a parameter and expected to be a distinct name of specific
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007400package type:
7401
7402- IPK for .ipk packages
7403
7404- DEB for .deb packages
7405
7406- RPM for .rpm packages
7407
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007408`<PN>` is a parameter and expected to be a package name.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007409
7410The variable can contain multiple [one-line] metadata fields separated
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007411by the literal sequence '\\n'. The separator can be redefined using the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007412variable flag ``separator``.
7413
William A. Kennington IIIac69b482021-06-02 12:28:27 -07007414Here is an example that adds two custom fields for ipk
Andrew Geisslerc926e172021-05-07 16:11:35 -05007415packages::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007416
7417 PACKAGE_ADD_METADATA_IPK = "Vendor: CustomIpk\nGroup:Applications/Spreadsheets"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007418
7419Efficiently Fetching Source Files During a Build
7420================================================
7421
7422The OpenEmbedded build system works with source files located through
7423the :term:`SRC_URI` variable. When
7424you build something using BitBake, a big part of the operation is
7425locating and downloading all the source tarballs. For images,
7426downloading all the source for various packages can take a significant
7427amount of time.
7428
7429This section shows you how you can use mirrors to speed up fetching
7430source files and how you can pre-fetch files all of which leads to more
7431efficient use of resources and time.
7432
7433Setting up Effective Mirrors
7434----------------------------
7435
7436A good deal that goes into a Yocto Project build is simply downloading
7437all of the source tarballs. Maybe you have been working with another
7438build system (OpenEmbedded or Angstrom) for which you have built up a
7439sizable directory of source tarballs. Or, perhaps someone else has such
7440a directory for which you have read access. If so, you can save time by
7441adding statements to your configuration file so that the build process
7442checks local directories first for existing tarballs before checking the
7443Internet.
7444
Andrew Geisslerc926e172021-05-07 16:11:35 -05007445Here is an efficient way to set it up in your ``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007446
7447 SOURCE_MIRROR_URL ?= "file:///home/you/your-download-dir/"
7448 INHERIT += "own-mirrors"
7449 BB_GENERATE_MIRROR_TARBALLS = "1"
7450 # BB_NO_NETWORK = "1"
7451
7452In the previous example, the
7453:term:`BB_GENERATE_MIRROR_TARBALLS`
7454variable causes the OpenEmbedded build system to generate tarballs of
7455the Git repositories and store them in the
7456:term:`DL_DIR` directory. Due to
7457performance reasons, generating and storing these tarballs is not the
7458build system's default behavior.
7459
7460You can also use the
7461:term:`PREMIRRORS` variable. For
7462an example, see the variable's glossary entry in the Yocto Project
7463Reference Manual.
7464
7465Getting Source Files and Suppressing the Build
7466----------------------------------------------
7467
7468Another technique you can use to ready yourself for a successive string
7469of build operations, is to pre-fetch all the source files without
7470actually starting a build. This technique lets you work through any
7471download issues and ultimately gathers all the source files into your
7472download directory :ref:`structure-build-downloads`,
7473which is located with :term:`DL_DIR`.
7474
7475Use the following BitBake command form to fetch all the necessary
Andrew Geisslerc926e172021-05-07 16:11:35 -05007476sources without starting the build::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007477
7478 $ bitbake target --runall=fetch
7479
7480This
7481variation of the BitBake command guarantees that you have all the
7482sources for that BitBake target should you disconnect from the Internet
7483and want to do the build later offline.
7484
7485Selecting an Initialization Manager
7486===================================
7487
7488By default, the Yocto Project uses SysVinit as the initialization
William A. Kennington IIIac69b482021-06-02 12:28:27 -07007489manager. However, there is also support for systemd, which is a full
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007490replacement for init with parallel starting of services, reduced shell
7491overhead and other features that are used by many distributions.
7492
7493Within the system, SysVinit treats system components as services. These
7494services are maintained as shell scripts stored in the ``/etc/init.d/``
7495directory. Services organize into different run levels. This
7496organization is maintained by putting links to the services in the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007497``/etc/rcN.d/`` directories, where `N/` is one of the following options:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007498"S", "0", "1", "2", "3", "4", "5", or "6".
7499
7500.. note::
7501
7502 Each runlevel has a dependency on the previous runlevel. This
7503 dependency allows the services to work properly.
7504
7505In comparison, systemd treats components as units. Using units is a
7506broader concept as compared to using a service. A unit includes several
7507different types of entities. Service is one of the types of entities.
7508The runlevel concept in SysVinit corresponds to the concept of a target
7509in systemd, where target is also a type of supported unit.
7510
7511In a SysVinit-based system, services load sequentially (i.e. one by one)
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007512during init and parallelization is not supported. With systemd, services
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007513start in parallel. Needless to say, the method can have an impact on
7514system startup performance.
7515
7516If you want to use SysVinit, you do not have to do anything. But, if you
7517want to use systemd, you must take some steps as described in the
7518following sections.
7519
7520Using systemd Exclusively
7521-------------------------
7522
Andrew Geisslerc926e172021-05-07 16:11:35 -05007523Set these variables in your distribution configuration file as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007524
7525 DISTRO_FEATURES_append = " systemd"
7526 VIRTUAL-RUNTIME_init_manager = "systemd"
7527
7528You can also prevent the SysVinit distribution feature from
Andrew Geisslerc926e172021-05-07 16:11:35 -05007529being automatically enabled as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007530
7531 DISTRO_FEATURES_BACKFILL_CONSIDERED = "sysvinit"
7532
7533Doing so removes any
7534redundant SysVinit scripts.
7535
7536To remove initscripts from your image altogether, set this variable
Andrew Geisslerc926e172021-05-07 16:11:35 -05007537also::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007538
7539 VIRTUAL-RUNTIME_initscripts = ""
7540
7541For information on the backfill variable, see
7542:term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`.
7543
7544Using systemd for the Main Image and Using SysVinit for the Rescue Image
7545------------------------------------------------------------------------
7546
Andrew Geisslerc926e172021-05-07 16:11:35 -05007547Set these variables in your distribution configuration file as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007548
7549 DISTRO_FEATURES_append = " systemd"
7550 VIRTUAL-RUNTIME_init_manager = "systemd"
7551
7552Doing so causes your main image to use the
7553``packagegroup-core-boot.bb`` recipe and systemd. The rescue/minimal
7554image cannot use this package group. However, it can install SysVinit
7555and the appropriate packages will have support for both systemd and
7556SysVinit.
7557
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007558Selecting a Device Manager
7559==========================
7560
7561The Yocto Project provides multiple ways to manage the device manager
7562(``/dev``):
7563
7564- Persistent and Pre-Populated\ ``/dev``: For this case, the ``/dev``
7565 directory is persistent and the required device nodes are created
7566 during the build.
7567
7568- Use ``devtmpfs`` with a Device Manager: For this case, the ``/dev``
7569 directory is provided by the kernel as an in-memory file system and
7570 is automatically populated by the kernel at runtime. Additional
7571 configuration of device nodes is done in user space by a device
7572 manager like ``udev`` or ``busybox-mdev``.
7573
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007574Using Persistent and Pre-Populated\ ``/dev``
7575--------------------------------------------
7576
7577To use the static method for device population, you need to set the
7578:term:`USE_DEVFS` variable to "0"
Andrew Geisslerc926e172021-05-07 16:11:35 -05007579as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007580
7581 USE_DEVFS = "0"
7582
7583The content of the resulting ``/dev`` directory is defined in a Device
7584Table file. The
7585:term:`IMAGE_DEVICE_TABLES`
7586variable defines the Device Table to use and should be set in the
7587machine or distro configuration file. Alternatively, you can set this
7588variable in your ``local.conf`` configuration file.
7589
Andrew Geissler09036742021-06-25 14:25:14 -05007590If you do not define the :term:`IMAGE_DEVICE_TABLES` variable, the default
Andrew Geisslerc926e172021-05-07 16:11:35 -05007591``device_table-minimal.txt`` is used::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007592
7593 IMAGE_DEVICE_TABLES = "device_table-mymachine.txt"
7594
7595The population is handled by the ``makedevs`` utility during image
7596creation:
7597
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007598Using ``devtmpfs`` and a Device Manager
7599---------------------------------------
7600
7601To use the dynamic method for device population, you need to use (or be
7602sure to set) the :term:`USE_DEVFS`
Andrew Geisslerc926e172021-05-07 16:11:35 -05007603variable to "1", which is the default::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007604
7605 USE_DEVFS = "1"
7606
7607With this
7608setting, the resulting ``/dev`` directory is populated by the kernel
7609using ``devtmpfs``. Make sure the corresponding kernel configuration
7610variable ``CONFIG_DEVTMPFS`` is set when building you build a Linux
7611kernel.
7612
7613All devices created by ``devtmpfs`` will be owned by ``root`` and have
7614permissions ``0600``.
7615
7616To have more control over the device nodes, you can use a device manager
7617like ``udev`` or ``busybox-mdev``. You choose the device manager by
7618defining the ``VIRTUAL-RUNTIME_dev_manager`` variable in your machine or
7619distro configuration file. Alternatively, you can set this variable in
Andrew Geisslerc926e172021-05-07 16:11:35 -05007620your ``local.conf`` configuration file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007621
7622 VIRTUAL-RUNTIME_dev_manager = "udev"
7623
7624 # Some alternative values
7625 # VIRTUAL-RUNTIME_dev_manager = "busybox-mdev"
7626 # VIRTUAL-RUNTIME_dev_manager = "systemd"
7627
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007628Using an External SCM
7629=====================
7630
7631If you're working on a recipe that pulls from an external Source Code
7632Manager (SCM), it is possible to have the OpenEmbedded build system
7633notice new recipe changes added to the SCM and then build the resulting
7634packages that depend on the new recipes by using the latest versions.
7635This only works for SCMs from which it is possible to get a sensible
7636revision number for changes. Currently, you can do this with Apache
7637Subversion (SVN), Git, and Bazaar (BZR) repositories.
7638
7639To enable this behavior, the :term:`PV` of
7640the recipe needs to reference
Andrew Geisslerc926e172021-05-07 16:11:35 -05007641:term:`SRCPV`. Here is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007642
7643 PV = "1.2.3+git${SRCPV}"
7644
7645Then, you can add the following to your
Andrew Geisslerc926e172021-05-07 16:11:35 -05007646``local.conf``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007647
7648 SRCREV_pn-PN = "${AUTOREV}"
7649
7650:term:`PN` is the name of the recipe for
7651which you want to enable automatic source revision updating.
7652
7653If you do not want to update your local configuration file, you can add
Andrew Geisslerc926e172021-05-07 16:11:35 -05007654the following directly to the recipe to finish enabling the feature::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007655
7656 SRCREV = "${AUTOREV}"
7657
7658The Yocto Project provides a distribution named ``poky-bleeding``, whose
Andrew Geisslerc926e172021-05-07 16:11:35 -05007659configuration file contains the line::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007660
7661 require conf/distro/include/poky-floating-revisions.inc
7662
7663This line pulls in the
Andrew Geisslerc926e172021-05-07 16:11:35 -05007664listed include file that contains numerous lines of exactly that form::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007665
7666 #SRCREV_pn-opkg-native ?= "${AUTOREV}"
7667 #SRCREV_pn-opkg-sdk ?= "${AUTOREV}"
7668 #SRCREV_pn-opkg ?= "${AUTOREV}"
7669 #SRCREV_pn-opkg-utils-native ?= "${AUTOREV}"
7670 #SRCREV_pn-opkg-utils ?= "${AUTOREV}"
7671 SRCREV_pn-gconf-dbus ?= "${AUTOREV}"
7672 SRCREV_pn-matchbox-common ?= "${AUTOREV}"
7673 SRCREV_pn-matchbox-config-gtk ?= "${AUTOREV}"
7674 SRCREV_pn-matchbox-desktop ?= "${AUTOREV}"
7675 SRCREV_pn-matchbox-keyboard ?= "${AUTOREV}"
7676 SRCREV_pn-matchbox-panel-2 ?= "${AUTOREV}"
7677 SRCREV_pn-matchbox-themes-extra ?= "${AUTOREV}"
7678 SRCREV_pn-matchbox-terminal ?= "${AUTOREV}"
7679 SRCREV_pn-matchbox-wm ?= "${AUTOREV}"
7680 SRCREV_pn-settings-daemon ?= "${AUTOREV}"
7681 SRCREV_pn-screenshot ?= "${AUTOREV}"
7682 . . .
7683
7684These lines allow you to
7685experiment with building a distribution that tracks the latest
7686development source for numerous packages.
7687
7688.. note::
7689
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007690 The ``poky-bleeding`` distribution is not tested on a regular basis. Keep
7691 this in mind if you use it.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007692
7693Creating a Read-Only Root Filesystem
7694====================================
7695
7696Suppose, for security reasons, you need to disable your target device's
7697root filesystem's write permissions (i.e. you need a read-only root
7698filesystem). Or, perhaps you are running the device's operating system
7699from a read-only storage device. For either case, you can customize your
7700image for that behavior.
7701
7702.. note::
7703
7704 Supporting a read-only root filesystem requires that the system and
7705 applications do not try to write to the root filesystem. You must
7706 configure all parts of the target system to write elsewhere, or to
7707 gracefully fail in the event of attempting to write to the root
7708 filesystem.
7709
7710Creating the Root Filesystem
7711----------------------------
7712
7713To create the read-only root filesystem, simply add the
7714"read-only-rootfs" feature to your image, normally in one of two ways.
7715The first way is to add the "read-only-rootfs" image feature in the
Andrew Geissler09036742021-06-25 14:25:14 -05007716image's recipe file via the :term:`IMAGE_FEATURES` variable::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007717
7718 IMAGE_FEATURES += "read-only-rootfs"
7719
7720As an alternative, you can add the same feature
7721from within your build directory's ``local.conf`` file with the
Andrew Geissler09036742021-06-25 14:25:14 -05007722associated :term:`EXTRA_IMAGE_FEATURES` variable, as in::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007723
7724 EXTRA_IMAGE_FEATURES = "read-only-rootfs"
7725
7726For more information on how to use these variables, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06007727":ref:`dev-manual/common-tasks:Customizing Images Using Custom \`\`IMAGE_FEATURES\`\` and \`\`EXTRA_IMAGE_FEATURES\`\``"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007728section. For information on the variables, see
7729:term:`IMAGE_FEATURES` and
7730:term:`EXTRA_IMAGE_FEATURES`.
7731
7732Post-Installation Scripts and Read-Only Root Filesystem
7733-------------------------------------------------------
7734
7735It is very important that you make sure all post-Installation
7736(``pkg_postinst``) scripts for packages that are installed into the
7737image can be run at the time when the root filesystem is created during
7738the build on the host system. These scripts cannot attempt to run during
7739first-boot on the target device. With the "read-only-rootfs" feature
7740enabled, the build system checks during root filesystem creation to make
7741sure all post-installation scripts succeed. If any of these scripts
7742still need to be run after the root filesystem is created, the build
7743immediately fails. These build-time checks ensure that the build fails
7744rather than the target device fails later during its initial boot
7745operation.
7746
7747Most of the common post-installation scripts generated by the build
7748system for the out-of-the-box Yocto Project are engineered so that they
7749can run during root filesystem creation (e.g. post-installation scripts
7750for caching fonts). However, if you create and add custom scripts, you
7751need to be sure they can be run during this file system creation.
7752
7753Here are some common problems that prevent post-installation scripts
7754from running during root filesystem creation:
7755
7756- *Not using $D in front of absolute paths:* The build system defines
7757 ``$``\ :term:`D` when the root
7758 filesystem is created. Furthermore, ``$D`` is blank when the script
7759 is run on the target device. This implies two purposes for ``$D``:
7760 ensuring paths are valid in both the host and target environments,
7761 and checking to determine which environment is being used as a method
7762 for taking appropriate actions.
7763
7764- *Attempting to run processes that are specific to or dependent on the
7765 target architecture:* You can work around these attempts by using
7766 native tools, which run on the host system, to accomplish the same
7767 tasks, or by alternatively running the processes under QEMU, which
7768 has the ``qemu_run_binary`` function. For more information, see the
7769 :ref:`qemu <ref-classes-qemu>` class.
7770
7771Areas With Write Access
7772-----------------------
7773
7774With the "read-only-rootfs" feature enabled, any attempt by the target
7775to write to the root filesystem at runtime fails. Consequently, you must
7776make sure that you configure processes and applications that attempt
7777these types of writes do so to directories with write access (e.g.
7778``/tmp`` or ``/var/run``).
7779
7780Maintaining Build Output Quality
7781================================
7782
7783Many factors can influence the quality of a build. For example, if you
7784upgrade a recipe to use a new version of an upstream software package or
7785you experiment with some new configuration options, subtle changes can
7786occur that you might not detect until later. Consider the case where
7787your recipe is using a newer version of an upstream package. In this
7788case, a new version of a piece of software might introduce an optional
7789dependency on another library, which is auto-detected. If that library
7790has already been built when the software is building, the software will
7791link to the built library and that library will be pulled into your
7792image along with the new software even if you did not want the library.
7793
7794The :ref:`buildhistory <ref-classes-buildhistory>`
William A. Kennington IIIac69b482021-06-02 12:28:27 -07007795class helps you maintain the quality of your build output. You
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007796can use the class to highlight unexpected and possibly unwanted changes
7797in the build output. When you enable build history, it records
7798information about the contents of each package and image and then
7799commits that information to a local Git repository where you can examine
7800the information.
7801
7802The remainder of this section describes the following:
7803
Andrew Geissler09209ee2020-12-13 08:44:15 -06007804- :ref:`How you can enable and disable build history <dev-manual/common-tasks:enabling and disabling build history>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007805
Andrew Geissler09209ee2020-12-13 08:44:15 -06007806- :ref:`How to understand what the build history contains <dev-manual/common-tasks:understanding what the build history contains>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007807
Andrew Geissler09209ee2020-12-13 08:44:15 -06007808- :ref:`How to limit the information used for build history <dev-manual/common-tasks:using build history to gather image information only>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007809
Andrew Geissler09209ee2020-12-13 08:44:15 -06007810- :ref:`How to examine the build history from both a command-line and web interface <dev-manual/common-tasks:examining build history information>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007811
7812Enabling and Disabling Build History
7813------------------------------------
7814
7815Build history is disabled by default. To enable it, add the following
Andrew Geissler09036742021-06-25 14:25:14 -05007816:term:`INHERIT` statement and set the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007817:term:`BUILDHISTORY_COMMIT`
7818variable to "1" at the end of your ``conf/local.conf`` file found in the
Andrew Geisslerc926e172021-05-07 16:11:35 -05007819:term:`Build Directory`::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007820
7821 INHERIT += "buildhistory"
7822 BUILDHISTORY_COMMIT = "1"
7823
7824Enabling build history as
7825previously described causes the OpenEmbedded build system to collect
7826build output information and commit it as a single commit to a local
Andrew Geissler09209ee2020-12-13 08:44:15 -06007827:ref:`overview-manual/development-environment:git` repository.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007828
7829.. note::
7830
7831 Enabling build history increases your build times slightly,
7832 particularly for images, and increases the amount of disk space used
7833 during the build.
7834
7835You can disable build history by removing the previous statements from
7836your ``conf/local.conf`` file.
7837
7838Understanding What the Build History Contains
7839---------------------------------------------
7840
7841Build history information is kept in
7842``${``\ :term:`TOPDIR`\ ``}/buildhistory``
7843in the Build Directory as defined by the
7844:term:`BUILDHISTORY_DIR`
William A. Kennington IIIac69b482021-06-02 12:28:27 -07007845variable. Here is an example abbreviated listing:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007846
7847.. image:: figures/buildhistory.png
7848 :align: center
7849
William A. Kennington IIIac69b482021-06-02 12:28:27 -07007850At the top level, there is a ``metadata-revs`` file that lists the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007851revisions of the repositories for the enabled layers when the build was
7852produced. The rest of the data splits into separate ``packages``,
7853``images`` and ``sdk`` directories, the contents of which are described
7854as follows.
7855
7856Build History Package Information
7857~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
7858
7859The history for each package contains a text file that has name-value
7860pairs with information about the package. For example,
7861``buildhistory/packages/i586-poky-linux/busybox/busybox/latest``
7862contains the following:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007863
7864.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007865
7866 PV = 1.22.1
7867 PR = r32
7868 RPROVIDES =
7869 RDEPENDS = glibc (>= 2.20) update-alternatives-opkg
7870 RRECOMMENDS = busybox-syslog busybox-udhcpc update-rc.d
7871 PKGSIZE = 540168
7872 FILES = /usr/bin/* /usr/sbin/* /usr/lib/busybox/* /usr/lib/lib*.so.* \
7873 /etc /com /var /bin/* /sbin/* /lib/*.so.* /lib/udev/rules.d \
7874 /usr/lib/udev/rules.d /usr/share/busybox /usr/lib/busybox/* \
7875 /usr/share/pixmaps /usr/share/applications /usr/share/idl \
7876 /usr/share/omf /usr/share/sounds /usr/lib/bonobo/servers
7877 FILELIST = /bin/busybox /bin/busybox.nosuid /bin/busybox.suid /bin/sh \
7878 /etc/busybox.links.nosuid /etc/busybox.links.suid
7879
7880Most of these
7881name-value pairs correspond to variables used to produce the package.
7882The exceptions are ``FILELIST``, which is the actual list of files in
7883the package, and ``PKGSIZE``, which is the total size of files in the
7884package in bytes.
7885
William A. Kennington IIIac69b482021-06-02 12:28:27 -07007886There is also a file that corresponds to the recipe from which the package
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007887came (e.g. ``buildhistory/packages/i586-poky-linux/busybox/latest``):
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007888
7889.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007890
7891 PV = 1.22.1
7892 PR = r32
7893 DEPENDS = initscripts kern-tools-native update-rc.d-native \
7894 virtual/i586-poky-linux-compilerlibs virtual/i586-poky-linux-gcc \
7895 virtual/libc virtual/update-alternatives
7896 PACKAGES = busybox-ptest busybox-httpd busybox-udhcpd busybox-udhcpc \
7897 busybox-syslog busybox-mdev busybox-hwclock busybox-dbg \
7898 busybox-staticdev busybox-dev busybox-doc busybox-locale busybox
7899
7900Finally, for those recipes fetched from a version control system (e.g.,
William A. Kennington IIIac69b482021-06-02 12:28:27 -07007901Git), there is a file that lists source revisions that are specified in
7902the recipe and the actual revisions used during the build. Listed
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007903and actual revisions might differ when
7904:term:`SRCREV` is set to
7905${:term:`AUTOREV`}. Here is an
7906example assuming
Andrew Geisslerc926e172021-05-07 16:11:35 -05007907``buildhistory/packages/qemux86-poky-linux/linux-yocto/latest_srcrev``)::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007908
7909 # SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
7910 SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
7911 # SRCREV_meta = "a227f20eff056e511d504b2e490f3774ab260d6f"
7912 SRCREV_meta ="a227f20eff056e511d504b2e490f3774ab260d6f"
7913
7914You can use the
7915``buildhistory-collect-srcrevs`` command with the ``-a`` option to
Andrew Geissler09036742021-06-25 14:25:14 -05007916collect the stored :term:`SRCREV` values from build history and report them
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007917in a format suitable for use in global configuration (e.g.,
7918``local.conf`` or a distro include file) to override floating
Andrew Geissler09036742021-06-25 14:25:14 -05007919:term:`AUTOREV` values to a fixed set of revisions. Here is some example
Andrew Geisslerc926e172021-05-07 16:11:35 -05007920output from this command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007921
7922 $ buildhistory-collect-srcrevs -a
7923 # i586-poky-linux
7924 SRCREV_pn-glibc = "b8079dd0d360648e4e8de48656c5c38972621072"
7925 SRCREV_pn-glibc-initial = "b8079dd0d360648e4e8de48656c5c38972621072"
7926 SRCREV_pn-opkg-utils = "53274f087565fd45d8452c5367997ba6a682a37a"
7927 SRCREV_pn-kmod = "fd56638aed3fe147015bfa10ed4a5f7491303cb4"
7928 # x86_64-linux
7929 SRCREV_pn-gtk-doc-stub-native = "1dea266593edb766d6d898c79451ef193eb17cfa"
7930 SRCREV_pn-dtc-native = "65cc4d2748a2c2e6f27f1cf39e07a5dbabd80ebf"
7931 SRCREV_pn-update-rc.d-native = "eca680ddf28d024954895f59a241a622dd575c11"
7932 SRCREV_glibc_pn-cross-localedef-native = "b8079dd0d360648e4e8de48656c5c38972621072"
7933 SRCREV_localedef_pn-cross-localedef-native = "c833367348d39dad7ba018990bfdaffaec8e9ed3"
7934 SRCREV_pn-prelink-native = "faa069deec99bf61418d0bab831c83d7c1b797ca"
7935 SRCREV_pn-opkg-utils-native = "53274f087565fd45d8452c5367997ba6a682a37a"
7936 SRCREV_pn-kern-tools-native = "23345b8846fe4bd167efdf1bd8a1224b2ba9a5ff"
7937 SRCREV_pn-kmod-native = "fd56638aed3fe147015bfa10ed4a5f7491303cb4"
7938 # qemux86-poky-linux
7939 SRCREV_machine_pn-linux-yocto = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
7940 SRCREV_meta_pn-linux-yocto = "a227f20eff056e511d504b2e490f3774ab260d6f"
7941 # all-poky-linux
7942 SRCREV_pn-update-rc.d = "eca680ddf28d024954895f59a241a622dd575c11"
7943
7944.. note::
7945
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007946 Here are some notes on using the ``buildhistory-collect-srcrevs`` command:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007947
Andrew Geissler09036742021-06-25 14:25:14 -05007948 - By default, only values where the :term:`SRCREV` was not hardcoded
Andrew Geissler5f350902021-07-23 13:09:54 -04007949 (usually when :term:`AUTOREV` is used) are reported. Use the ``-a``
7950 option to see all :term:`SRCREV` values.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007951
7952 - The output statements might not have any effect if overrides are
7953 applied elsewhere in the build system configuration. Use the
7954 ``-f`` option to add the ``forcevariable`` override to each output
7955 line if you need to work around this restriction.
7956
7957 - The script does apply special handling when building for multiple
7958 machines. However, the script does place a comment before each set
7959 of values that specifies which triplet to which they belong as
7960 previously shown (e.g., ``i586-poky-linux``).
7961
7962Build History Image Information
7963~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
7964
7965The files produced for each image are as follows:
7966
7967- ``image-files:`` A directory containing selected files from the root
7968 filesystem. The files are defined by
7969 :term:`BUILDHISTORY_IMAGE_FILES`.
7970
7971- ``build-id.txt:`` Human-readable information about the build
7972 configuration and metadata source revisions. This file contains the
7973 full build header as printed by BitBake.
7974
7975- ``*.dot:`` Dependency graphs for the image that are compatible with
7976 ``graphviz``.
7977
7978- ``files-in-image.txt:`` A list of files in the image with
7979 permissions, owner, group, size, and symlink information.
7980
7981- ``image-info.txt:`` A text file containing name-value pairs with
7982 information about the image. See the following listing example for
7983 more information.
7984
7985- ``installed-package-names.txt:`` A list of installed packages by name
7986 only.
7987
7988- ``installed-package-sizes.txt:`` A list of installed packages ordered
7989 by size.
7990
7991- ``installed-packages.txt:`` A list of installed packages with full
7992 package filenames.
7993
7994.. note::
7995
7996 Installed package information is able to be gathered and produced
7997 even if package management is disabled for the final image.
7998
7999Here is an example of ``image-info.txt``:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008000
8001.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008002
8003 DISTRO = poky
8004 DISTRO_VERSION = 1.7
Andrew Geissler5f350902021-07-23 13:09:54 -04008005 USER_CLASSES = buildstats image-prelink
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008006 IMAGE_CLASSES = image_types
8007 IMAGE_FEATURES = debug-tweaks
8008 IMAGE_LINGUAS =
8009 IMAGE_INSTALL = packagegroup-core-boot run-postinsts
8010 BAD_RECOMMENDATIONS =
8011 NO_RECOMMENDATIONS =
8012 PACKAGE_EXCLUDE =
8013 ROOTFS_POSTPROCESS_COMMAND = write_package_manifest; license_create_manifest; \
8014 write_image_manifest ; buildhistory_list_installed_image ; \
8015 buildhistory_get_image_installed ; ssh_allow_empty_password; \
8016 postinst_enable_logging; rootfs_update_timestamp ; ssh_disable_dns_lookup ;
8017 IMAGE_POSTPROCESS_COMMAND = buildhistory_get_imageinfo ;
8018 IMAGESIZE = 6900
8019
8020Other than ``IMAGESIZE``,
8021which is the total size of the files in the image in Kbytes, the
8022name-value pairs are variables that may have influenced the content of
8023the image. This information is often useful when you are trying to
8024determine why a change in the package or file listings has occurred.
8025
8026Using Build History to Gather Image Information Only
8027~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8028
8029As you can see, build history produces image information, including
8030dependency graphs, so you can see why something was pulled into the
8031image. If you are just interested in this information and not interested
8032in collecting specific package or SDK information, you can enable
8033writing only image information without any history by adding the
8034following to your ``conf/local.conf`` file found in the
Andrew Geisslerc926e172021-05-07 16:11:35 -05008035:term:`Build Directory`::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008036
8037 INHERIT += "buildhistory"
8038 BUILDHISTORY_COMMIT = "0"
8039 BUILDHISTORY_FEATURES = "image"
8040
8041Here, you set the
8042:term:`BUILDHISTORY_FEATURES`
8043variable to use the image feature only.
8044
8045Build History SDK Information
8046~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8047
8048Build history collects similar information on the contents of SDKs (e.g.
8049``bitbake -c populate_sdk imagename``) as compared to information it
8050collects for images. Furthermore, this information differs depending on
8051whether an extensible or standard SDK is being produced.
8052
8053The following list shows the files produced for SDKs:
8054
8055- ``files-in-sdk.txt:`` A list of files in the SDK with permissions,
8056 owner, group, size, and symlink information. This list includes both
8057 the host and target parts of the SDK.
8058
8059- ``sdk-info.txt:`` A text file containing name-value pairs with
8060 information about the SDK. See the following listing example for more
8061 information.
8062
8063- ``sstate-task-sizes.txt:`` A text file containing name-value pairs
8064 with information about task group sizes (e.g. ``do_populate_sysroot``
8065 tasks have a total size). The ``sstate-task-sizes.txt`` file exists
8066 only when an extensible SDK is created.
8067
8068- ``sstate-package-sizes.txt:`` A text file containing name-value pairs
8069 with information for the shared-state packages and sizes in the SDK.
8070 The ``sstate-package-sizes.txt`` file exists only when an extensible
8071 SDK is created.
8072
8073- ``sdk-files:`` A folder that contains copies of the files mentioned
8074 in ``BUILDHISTORY_SDK_FILES`` if the files are present in the output.
8075 Additionally, the default value of ``BUILDHISTORY_SDK_FILES`` is
8076 specific to the extensible SDK although you can set it differently if
8077 you would like to pull in specific files from the standard SDK.
8078
8079 The default files are ``conf/local.conf``, ``conf/bblayers.conf``,
8080 ``conf/auto.conf``, ``conf/locked-sigs.inc``, and
8081 ``conf/devtool.conf``. Thus, for an extensible SDK, these files get
8082 copied into the ``sdk-files`` directory.
8083
8084- The following information appears under each of the ``host`` and
8085 ``target`` directories for the portions of the SDK that run on the
8086 host and on the target, respectively:
8087
8088 .. note::
8089
8090 The following files for the most part are empty when producing an
8091 extensible SDK because this type of SDK is not constructed from
8092 packages as is the standard SDK.
8093
8094 - ``depends.dot:`` Dependency graph for the SDK that is compatible
8095 with ``graphviz``.
8096
8097 - ``installed-package-names.txt:`` A list of installed packages by
8098 name only.
8099
8100 - ``installed-package-sizes.txt:`` A list of installed packages
8101 ordered by size.
8102
8103 - ``installed-packages.txt:`` A list of installed packages with full
8104 package filenames.
8105
8106Here is an example of ``sdk-info.txt``:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008107
8108.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008109
8110 DISTRO = poky
8111 DISTRO_VERSION = 1.3+snapshot-20130327
8112 SDK_NAME = poky-glibc-i686-arm
8113 SDK_VERSION = 1.3+snapshot
8114 SDKMACHINE =
8115 SDKIMAGE_FEATURES = dev-pkgs dbg-pkgs
8116 BAD_RECOMMENDATIONS =
8117 SDKSIZE = 352712
8118
8119Other than ``SDKSIZE``, which is
8120the total size of the files in the SDK in Kbytes, the name-value pairs
8121are variables that might have influenced the content of the SDK. This
8122information is often useful when you are trying to determine why a
8123change in the package or file listings has occurred.
8124
8125Examining Build History Information
8126~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8127
8128You can examine build history output from the command line or from a web
8129interface.
8130
8131To see any changes that have occurred (assuming you have
8132:term:`BUILDHISTORY_COMMIT` = "1"),
8133you can simply use any Git command that allows you to view the history
Andrew Geisslerc926e172021-05-07 16:11:35 -05008134of a repository. Here is one method::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008135
8136 $ git log -p
8137
8138You need to realize,
8139however, that this method does show changes that are not significant
8140(e.g. a package's size changing by a few bytes).
8141
William A. Kennington IIIac69b482021-06-02 12:28:27 -07008142There is a command-line tool called ``buildhistory-diff``, though,
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008143that queries the Git repository and prints just the differences that
Andrew Geisslerc926e172021-05-07 16:11:35 -05008144might be significant in human-readable form. Here is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008145
Andrew Geissler95ac1b82021-03-31 14:34:31 -05008146 $ poky/poky/scripts/buildhistory-diff . HEAD^
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008147 Changes to images/qemux86_64/glibc/core-image-minimal (files-in-image.txt):
8148 /etc/anotherpkg.conf was added
8149 /sbin/anotherpkg was added
8150 * (installed-package-names.txt):
8151 * anotherpkg was added
8152 Changes to images/qemux86_64/glibc/core-image-minimal (installed-package-names.txt):
8153 anotherpkg was added
8154 packages/qemux86_64-poky-linux/v86d: PACKAGES: added "v86d-extras"
8155 * PR changed from "r0" to "r1"
8156 * PV changed from "0.1.10" to "0.1.12"
8157 packages/qemux86_64-poky-linux/v86d/v86d: PKGSIZE changed from 110579 to 144381 (+30%)
8158 * PR changed from "r0" to "r1"
8159 * PV changed from "0.1.10" to "0.1.12"
8160
8161.. note::
8162
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008163 The ``buildhistory-diff`` tool requires the ``GitPython``
Andrew Geisslerc926e172021-05-07 16:11:35 -05008164 package. Be sure to install it using Pip3 as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008165
8166 $ pip3 install GitPython --user
8167
8168
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008169 Alternatively, you can install ``python3-git`` using the appropriate
8170 distribution package manager (e.g. ``apt-get``, ``dnf``, or ``zipper``).
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008171
8172To see changes to the build history using a web interface, follow the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008173instruction in the ``README`` file
Andrew Geissler09209ee2020-12-13 08:44:15 -06008174:yocto_git:`here </buildhistory-web/>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008175
8176Here is a sample screenshot of the interface:
8177
8178.. image:: figures/buildhistory-web.png
8179 :align: center
8180
8181Performing Automated Runtime Testing
8182====================================
8183
8184The OpenEmbedded build system makes available a series of automated
8185tests for images to verify runtime functionality. You can run these
8186tests on either QEMU or actual target hardware. Tests are written in
8187Python making use of the ``unittest`` module, and the majority of them
8188run commands on the target system over SSH. This section describes how
8189you set up the environment to use these tests, run available tests, and
8190write and add your own tests.
8191
8192For information on the test and QA infrastructure available within the
Andrew Geissler09209ee2020-12-13 08:44:15 -06008193Yocto Project, see the ":ref:`ref-manual/release-process:testing and quality assurance`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008194section in the Yocto Project Reference Manual.
8195
8196Enabling Tests
8197--------------
8198
8199Depending on whether you are planning to run tests using QEMU or on the
8200hardware, you have to take different steps to enable the tests. See the
8201following subsections for information on how to enable both types of
8202tests.
8203
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008204Enabling Runtime Tests on QEMU
8205~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8206
8207In order to run tests, you need to do the following:
8208
8209- *Set up to avoid interaction with sudo for networking:* To
8210 accomplish this, you must do one of the following:
8211
8212 - Add ``NOPASSWD`` for your user in ``/etc/sudoers`` either for all
8213 commands or just for ``runqemu-ifup``. You must provide the full
8214 path as that can change if you are using multiple clones of the
8215 source repository.
8216
8217 .. note::
8218
8219 On some distributions, you also need to comment out "Defaults
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008220 requiretty" in ``/etc/sudoers``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008221
8222 - Manually configure a tap interface for your system.
8223
8224 - Run as root the script in ``scripts/runqemu-gen-tapdevs``, which
8225 should generate a list of tap devices. This is the option
8226 typically chosen for Autobuilder-type environments.
8227
8228 .. note::
8229
8230 - Be sure to use an absolute path when calling this script
8231 with sudo.
8232
8233 - The package recipe ``qemu-helper-native`` is required to run
Andrew Geisslerc926e172021-05-07 16:11:35 -05008234 this script. Build the package using the following command::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008235
8236 $ bitbake qemu-helper-native
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008237
8238- *Set the DISPLAY variable:* You need to set this variable so that
8239 you have an X server available (e.g. start ``vncserver`` for a
8240 headless machine).
8241
8242- *Be sure your host's firewall accepts incoming connections from
8243 192.168.7.0/24:* Some of the tests (in particular DNF tests) start an
8244 HTTP server on a random high number port, which is used to serve
8245 files to the target. The DNF module serves
8246 ``${WORKDIR}/oe-rootfs-repo`` so it can run DNF channel commands.
8247 That means your host's firewall must accept incoming connections from
8248 192.168.7.0/24, which is the default IP range used for tap devices by
8249 ``runqemu``.
8250
8251- *Be sure your host has the correct packages installed:* Depending
8252 your host's distribution, you need to have the following packages
8253 installed:
8254
8255 - Ubuntu and Debian: ``sysstat`` and ``iproute2``
8256
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05008257 - openSUSE: ``sysstat`` and ``iproute2``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008258
8259 - Fedora: ``sysstat`` and ``iproute``
8260
8261 - CentOS: ``sysstat`` and ``iproute``
8262
8263Once you start running the tests, the following happens:
8264
82651. A copy of the root filesystem is written to ``${WORKDIR}/testimage``.
8266
82672. The image is booted under QEMU using the standard ``runqemu`` script.
8268
82693. A default timeout of 500 seconds occurs to allow for the boot process
8270 to reach the login prompt. You can change the timeout period by
8271 setting
8272 :term:`TEST_QEMUBOOT_TIMEOUT`
8273 in the ``local.conf`` file.
8274
82754. Once the boot process is reached and the login prompt appears, the
8276 tests run. The full boot log is written to
8277 ``${WORKDIR}/testimage/qemu_boot_log``.
8278
Andrew Geissler09036742021-06-25 14:25:14 -050082795. Each test module loads in the order found in :term:`TEST_SUITES`. You can
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008280 find the full output of the commands run over SSH in
8281 ``${WORKDIR}/testimgage/ssh_target_log``.
8282
82836. If no failures occur, the task running the tests ends successfully.
8284 You can find the output from the ``unittest`` in the task log at
8285 ``${WORKDIR}/temp/log.do_testimage``.
8286
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008287Enabling Runtime Tests on Hardware
8288~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8289
8290The OpenEmbedded build system can run tests on real hardware, and for
8291certain devices it can also deploy the image to be tested onto the
8292device beforehand.
8293
8294For automated deployment, a "master image" is installed onto the
8295hardware once as part of setup. Then, each time tests are to be run, the
8296following occurs:
8297
82981. The master image is booted into and used to write the image to be
8299 tested to a second partition.
8300
83012. The device is then rebooted using an external script that you need to
8302 provide.
8303
83043. The device boots into the image to be tested.
8305
8306When running tests (independent of whether the image has been deployed
8307automatically or not), the device is expected to be connected to a
8308network on a pre-determined IP address. You can either use static IP
8309addresses written into the image, or set the image to use DHCP and have
8310your DHCP server on the test network assign a known IP address based on
8311the MAC address of the device.
8312
Andrew Geissler09036742021-06-25 14:25:14 -05008313In order to run tests on hardware, you need to set :term:`TEST_TARGET` to an
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008314appropriate value. For QEMU, you do not have to change anything, the
8315default value is "qemu". For running tests on hardware, the following
William A. Kennington IIIac69b482021-06-02 12:28:27 -07008316options are available:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008317
8318- *"simpleremote":* Choose "simpleremote" if you are going to run tests
8319 on a target system that is already running the image to be tested and
8320 is available on the network. You can use "simpleremote" in
8321 conjunction with either real hardware or an image running within a
8322 separately started QEMU or any other virtual machine manager.
8323
8324- *"SystemdbootTarget":* Choose "SystemdbootTarget" if your hardware is
8325 an EFI-based machine with ``systemd-boot`` as bootloader and
8326 ``core-image-testmaster`` (or something similar) is installed. Also,
8327 your hardware under test must be in a DHCP-enabled network that gives
8328 it the same IP address for each reboot.
8329
8330 If you choose "SystemdbootTarget", there are additional requirements
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05008331 and considerations. See the
8332 ":ref:`dev-manual/common-tasks:selecting systemdboottarget`" section, which
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008333 follows, for more information.
8334
8335- *"BeagleBoneTarget":* Choose "BeagleBoneTarget" if you are deploying
8336 images and running tests on the BeagleBone "Black" or original
8337 "White" hardware. For information on how to use these tests, see the
8338 comments at the top of the BeagleBoneTarget
8339 ``meta-yocto-bsp/lib/oeqa/controllers/beaglebonetarget.py`` file.
8340
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008341- *"EdgeRouterTarget":* Choose "EdgeRouterTarget" if you are deploying
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008342 images and running tests on the Ubiquiti Networks EdgeRouter Lite.
8343 For information on how to use these tests, see the comments at the
8344 top of the EdgeRouterTarget
8345 ``meta-yocto-bsp/lib/oeqa/controllers/edgeroutertarget.py`` file.
8346
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008347- *"GrubTarget":* Choose "GrubTarget" if you are deploying images and running
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008348 tests on any generic PC that boots using GRUB. For information on how
8349 to use these tests, see the comments at the top of the GrubTarget
8350 ``meta-yocto-bsp/lib/oeqa/controllers/grubtarget.py`` file.
8351
8352- *"your-target":* Create your own custom target if you want to run
8353 tests when you are deploying images and running tests on a custom
8354 machine within your BSP layer. To do this, you need to add a Python
8355 unit that defines the target class under ``lib/oeqa/controllers/``
8356 within your layer. You must also provide an empty ``__init__.py``.
8357 For examples, see files in ``meta-yocto-bsp/lib/oeqa/controllers/``.
8358
8359Selecting SystemdbootTarget
8360~~~~~~~~~~~~~~~~~~~~~~~~~~~
8361
Andrew Geissler09036742021-06-25 14:25:14 -05008362If you did not set :term:`TEST_TARGET` to "SystemdbootTarget", then you do
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008363not need any information in this section. You can skip down to the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05008364":ref:`dev-manual/common-tasks:running tests`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008365
Andrew Geissler09036742021-06-25 14:25:14 -05008366If you did set :term:`TEST_TARGET` to "SystemdbootTarget", you also need to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008367perform a one-time setup of your master image by doing the following:
8368
Andrew Geissler09036742021-06-25 14:25:14 -050083691. *Set EFI_PROVIDER:* Be sure that :term:`EFI_PROVIDER` is as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008370
8371 EFI_PROVIDER = "systemd-boot"
8372
83732. *Build the master image:* Build the ``core-image-testmaster`` image.
8374 The ``core-image-testmaster`` recipe is provided as an example for a
8375 "master" image and you can customize the image recipe as you would
8376 any other recipe.
8377
8378 Here are the image recipe requirements:
8379
8380 - Inherits ``core-image`` so that kernel modules are installed.
8381
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05008382 - Installs normal linux utilities not BusyBox ones (e.g. ``bash``,
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008383 ``coreutils``, ``tar``, ``gzip``, and ``kmod``).
8384
8385 - Uses a custom Initial RAM Disk (initramfs) image with a custom
8386 installer. A normal image that you can install usually creates a
8387 single rootfs partition. This image uses another installer that
8388 creates a specific partition layout. Not all Board Support
8389 Packages (BSPs) can use an installer. For such cases, you need to
8390 manually create the following partition layout on the target:
8391
8392 - First partition mounted under ``/boot``, labeled "boot".
8393
8394 - The main rootfs partition where this image gets installed,
8395 which is mounted under ``/``.
8396
8397 - Another partition labeled "testrootfs" where test images get
8398 deployed.
8399
84003. *Install image:* Install the image that you just built on the target
8401 system.
8402
Andrew Geissler09036742021-06-25 14:25:14 -05008403The final thing you need to do when setting :term:`TEST_TARGET` to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008404"SystemdbootTarget" is to set up the test image:
8405
84061. *Set up your local.conf file:* Make sure you have the following
Andrew Geisslerc926e172021-05-07 16:11:35 -05008407 statements in your ``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008408
8409 IMAGE_FSTYPES += "tar.gz"
8410 INHERIT += "testimage"
8411 TEST_TARGET = "SystemdbootTarget"
8412 TEST_TARGET_IP = "192.168.2.3"
8413
Andrew Geisslerc926e172021-05-07 16:11:35 -050084142. *Build your test image:* Use BitBake to build the image::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008415
8416 $ bitbake core-image-sato
8417
8418Power Control
8419~~~~~~~~~~~~~
8420
8421For most hardware targets other than "simpleremote", you can control
8422power:
8423
Andrew Geissler09036742021-06-25 14:25:14 -05008424- You can use :term:`TEST_POWERCONTROL_CMD` together with
8425 :term:`TEST_POWERCONTROL_EXTRA_ARGS` as a command that runs on the host
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008426 and does power cycling. The test code passes one argument to that
8427 command: off, on or cycle (off then on). Here is an example that
Andrew Geisslerc926e172021-05-07 16:11:35 -05008428 could appear in your ``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008429
8430 TEST_POWERCONTROL_CMD = "powercontrol.exp test 10.11.12.1 nuc1"
8431
8432 In this example, the expect
8433 script does the following:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008434
8435 .. code-block:: shell
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008436
8437 ssh test@10.11.12.1 "pyctl nuc1 arg"
8438
8439 It then runs a Python script that controls power for a label called
8440 ``nuc1``.
8441
8442 .. note::
8443
Andrew Geissler09036742021-06-25 14:25:14 -05008444 You need to customize :term:`TEST_POWERCONTROL_CMD` and
8445 :term:`TEST_POWERCONTROL_EXTRA_ARGS` for your own setup. The one requirement
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008446 is that it accepts "on", "off", and "cycle" as the last argument.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008447
8448- When no command is defined, it connects to the device over SSH and
8449 uses the classic reboot command to reboot the device. Classic reboot
8450 is fine as long as the machine actually reboots (i.e. the SSH test
8451 has not failed). It is useful for scenarios where you have a simple
8452 setup, typically with a single board, and where some manual
8453 interaction is okay from time to time.
8454
8455If you have no hardware to automatically perform power control but still
8456wish to experiment with automated hardware testing, you can use the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008457``dialog-power-control`` script that shows a dialog prompting you to perform
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008458the required power action. This script requires either KDialog or Zenity
8459to be installed. To use this script, set the
8460:term:`TEST_POWERCONTROL_CMD`
Andrew Geisslerc926e172021-05-07 16:11:35 -05008461variable as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008462
8463 TEST_POWERCONTROL_CMD = "${COREBASE}/scripts/contrib/dialog-power-control"
8464
8465Serial Console Connection
8466~~~~~~~~~~~~~~~~~~~~~~~~~
8467
8468For test target classes requiring a serial console to interact with the
8469bootloader (e.g. BeagleBoneTarget, EdgeRouterTarget, and GrubTarget),
8470you need to specify a command to use to connect to the serial console of
8471the target machine by using the
8472:term:`TEST_SERIALCONTROL_CMD`
8473variable and optionally the
8474:term:`TEST_SERIALCONTROL_EXTRA_ARGS`
8475variable.
8476
8477These cases could be a serial terminal program if the machine is
8478connected to a local serial port, or a ``telnet`` or ``ssh`` command
8479connecting to a remote console server. Regardless of the case, the
8480command simply needs to connect to the serial console and forward that
8481connection to standard input and output as any normal terminal program
8482does. For example, to use the picocom terminal program on serial device
Andrew Geisslerc926e172021-05-07 16:11:35 -05008483``/dev/ttyUSB0`` at 115200bps, you would set the variable as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008484
8485 TEST_SERIALCONTROL_CMD = "picocom /dev/ttyUSB0 -b 115200"
8486
8487For local
8488devices where the serial port device disappears when the device reboots,
8489an additional "serdevtry" wrapper script is provided. To use this
8490wrapper, simply prefix the terminal command with
Andrew Geisslerc926e172021-05-07 16:11:35 -05008491``${COREBASE}/scripts/contrib/serdevtry``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008492
8493 TEST_SERIALCONTROL_CMD = "${COREBASE}/scripts/contrib/serdevtry picocom -b 115200 /dev/ttyUSB0"
8494
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008495Running Tests
8496-------------
8497
8498You can start the tests automatically or manually:
8499
8500- *Automatically running tests:* To run the tests automatically after
8501 the OpenEmbedded build system successfully creates an image, first
8502 set the
8503 :term:`TESTIMAGE_AUTO`
8504 variable to "1" in your ``local.conf`` file in the
Andrew Geisslerc926e172021-05-07 16:11:35 -05008505 :term:`Build Directory`::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008506
8507 TESTIMAGE_AUTO = "1"
8508
8509 Next, build your image. If the image successfully builds, the
Andrew Geisslerc926e172021-05-07 16:11:35 -05008510 tests run::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008511
8512 bitbake core-image-sato
8513
8514- *Manually running tests:* To manually run the tests, first globally
8515 inherit the
8516 :ref:`testimage <ref-classes-testimage*>` class
Andrew Geisslerc926e172021-05-07 16:11:35 -05008517 by editing your ``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008518
8519 INHERIT += "testimage"
8520
Andrew Geisslerc926e172021-05-07 16:11:35 -05008521 Next, use BitBake to run the tests::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008522
8523 bitbake -c testimage image
8524
8525All test files reside in ``meta/lib/oeqa/runtime`` in the
8526:term:`Source Directory`. A test name maps
8527directly to a Python module. Each test module may contain a number of
8528individual tests. Tests are usually grouped together by the area tested
8529(e.g tests for systemd reside in ``meta/lib/oeqa/runtime/systemd.py``).
8530
8531You can add tests to any layer provided you place them in the proper
8532area and you extend :term:`BBPATH` in
8533the ``local.conf`` file as normal. Be sure that tests reside in
8534``layer/lib/oeqa/runtime``.
8535
8536.. note::
8537
8538 Be sure that module names do not collide with module names used in
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008539 the default set of test modules in ``meta/lib/oeqa/runtime``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008540
8541You can change the set of tests run by appending or overriding
8542:term:`TEST_SUITES` variable in
Andrew Geissler09036742021-06-25 14:25:14 -05008543``local.conf``. Each name in :term:`TEST_SUITES` represents a required test
8544for the image. Test modules named within :term:`TEST_SUITES` cannot be
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008545skipped even if a test is not suitable for an image (e.g. running the
8546RPM tests on an image without ``rpm``). Appending "auto" to
Andrew Geissler09036742021-06-25 14:25:14 -05008547:term:`TEST_SUITES` causes the build system to try to run all tests that are
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008548suitable for the image (i.e. each test module may elect to skip itself).
8549
Andrew Geissler09036742021-06-25 14:25:14 -05008550The order you list tests in :term:`TEST_SUITES` is important and influences
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008551test dependencies. Consequently, tests that depend on other tests should
8552be added after the test on which they depend. For example, since the
8553``ssh`` test depends on the ``ping`` test, "ssh" needs to come after
8554"ping" in the list. The test class provides no re-ordering or dependency
8555handling.
8556
8557.. note::
8558
8559 Each module can have multiple classes with multiple test methods.
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008560 And, Python ``unittest`` rules apply.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008561
8562Here are some things to keep in mind when running tests:
8563
Andrew Geisslerc926e172021-05-07 16:11:35 -05008564- The default tests for the image are defined as::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008565
8566 DEFAULT_TEST_SUITES_pn-image = "ping ssh df connman syslog xorg scp vnc date rpm dnf dmesg"
8567
Andrew Geisslerc926e172021-05-07 16:11:35 -05008568- Add your own test to the list of the by using the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008569
8570 TEST_SUITES_append = " mytest"
8571
Andrew Geisslerc926e172021-05-07 16:11:35 -05008572- Run a specific list of tests as follows::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008573
8574 TEST_SUITES = "test1 test2 test3"
8575
8576 Remember, order is important. Be sure to place a test that is
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008577 dependent on another test later in the order.
8578
8579Exporting Tests
8580---------------
8581
8582You can export tests so that they can run independently of the build
8583system. Exporting tests is required if you want to be able to hand the
8584test execution off to a scheduler. You can only export tests that are
8585defined in :term:`TEST_SUITES`.
8586
8587If your image is already built, make sure the following are set in your
Andrew Geisslerc926e172021-05-07 16:11:35 -05008588``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008589
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008590 INHERIT += "testexport"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008591 TEST_TARGET_IP = "IP-address-for-the-test-target"
8592 TEST_SERVER_IP = "IP-address-for-the-test-server"
8593
8594You can then export the tests with the
Andrew Geisslerc926e172021-05-07 16:11:35 -05008595following BitBake command form::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008596
8597 $ bitbake image -c testexport
8598
8599Exporting the tests places them in the
8600:term:`Build Directory` in
8601``tmp/testexport/``\ image, which is controlled by the
Andrew Geissler09036742021-06-25 14:25:14 -05008602:term:`TEST_EXPORT_DIR` variable.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008603
Andrew Geisslerc926e172021-05-07 16:11:35 -05008604You can now run the tests outside of the build environment::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008605
8606 $ cd tmp/testexport/image
8607 $ ./runexported.py testdata.json
8608
8609Here is a complete example that shows IP addresses and uses the
Andrew Geisslerc926e172021-05-07 16:11:35 -05008610``core-image-sato`` image::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008611
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008612 INHERIT += "testexport"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008613 TEST_TARGET_IP = "192.168.7.2"
8614 TEST_SERVER_IP = "192.168.7.1"
8615
Andrew Geisslerc926e172021-05-07 16:11:35 -05008616Use BitBake to export the tests::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008617
8618 $ bitbake core-image-sato -c testexport
8619
8620Run the tests outside of
Andrew Geisslerc926e172021-05-07 16:11:35 -05008621the build environment using the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008622
8623 $ cd tmp/testexport/core-image-sato
8624 $ ./runexported.py testdata.json
8625
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008626Writing New Tests
8627-----------------
8628
8629As mentioned previously, all new test files need to be in the proper
8630place for the build system to find them. New tests for additional
8631functionality outside of the core should be added to the layer that adds
8632the functionality, in ``layer/lib/oeqa/runtime`` (as long as
8633:term:`BBPATH` is extended in the
8634layer's ``layer.conf`` file as normal). Just remember the following:
8635
8636- Filenames need to map directly to test (module) names.
8637
8638- Do not use module names that collide with existing core tests.
8639
William A. Kennington IIIac69b482021-06-02 12:28:27 -07008640- Minimally, an empty ``__init__.py`` file must be present in the runtime
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008641 directory.
8642
8643To create a new test, start by copying an existing module (e.g.
8644``syslog.py`` or ``gcc.py`` are good ones to use). Test modules can use
8645code from ``meta/lib/oeqa/utils``, which are helper classes.
8646
8647.. note::
8648
8649 Structure shell commands such that you rely on them and they return a
8650 single code for success. Be aware that sometimes you will need to
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008651 parse the output. See the ``df.py`` and ``date.py`` modules for examples.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008652
8653You will notice that all test classes inherit ``oeRuntimeTest``, which
8654is found in ``meta/lib/oetest.py``. This base class offers some helper
8655attributes, which are described in the following sections:
8656
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008657Class Methods
8658~~~~~~~~~~~~~
8659
8660Class methods are as follows:
8661
8662- *hasPackage(pkg):* Returns "True" if ``pkg`` is in the installed
8663 package list of the image, which is based on the manifest file that
8664 is generated during the ``do_rootfs`` task.
8665
8666- *hasFeature(feature):* Returns "True" if the feature is in
8667 :term:`IMAGE_FEATURES` or
8668 :term:`DISTRO_FEATURES`.
8669
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008670Class Attributes
8671~~~~~~~~~~~~~~~~
8672
8673Class attributes are as follows:
8674
8675- *pscmd:* Equals "ps -ef" if ``procps`` is installed in the image.
8676 Otherwise, ``pscmd`` equals "ps" (busybox).
8677
8678- *tc:* The called test context, which gives access to the
8679 following attributes:
8680
8681 - *d:* The BitBake datastore, which allows you to use stuff such
8682 as ``oeRuntimeTest.tc.d.getVar("VIRTUAL-RUNTIME_init_manager")``.
8683
8684 - *testslist and testsrequired:* Used internally. The tests
8685 do not need these.
8686
8687 - *filesdir:* The absolute path to
8688 ``meta/lib/oeqa/runtime/files``, which contains helper files for
8689 tests meant for copying on the target such as small files written
8690 in C for compilation.
8691
8692 - *target:* The target controller object used to deploy and
8693 start an image on a particular target (e.g. Qemu, SimpleRemote,
8694 and SystemdbootTarget). Tests usually use the following:
8695
8696 - *ip:* The target's IP address.
8697
8698 - *server_ip:* The host's IP address, which is usually used
8699 by the DNF test suite.
8700
8701 - *run(cmd, timeout=None):* The single, most used method.
8702 This command is a wrapper for: ``ssh root@host "cmd"``. The
8703 command returns a tuple: (status, output), which are what their
8704 names imply - the return code of "cmd" and whatever output it
8705 produces. The optional timeout argument represents the number
8706 of seconds the test should wait for "cmd" to return. If the
8707 argument is "None", the test uses the default instance's
8708 timeout period, which is 300 seconds. If the argument is "0",
8709 the test runs until the command returns.
8710
8711 - *copy_to(localpath, remotepath):*
8712 ``scp localpath root@ip:remotepath``.
8713
8714 - *copy_from(remotepath, localpath):*
8715 ``scp root@host:remotepath localpath``.
8716
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008717Instance Attributes
8718~~~~~~~~~~~~~~~~~~~
8719
William A. Kennington IIIac69b482021-06-02 12:28:27 -07008720There is a single instance attribute, which is ``target``. The ``target``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008721instance attribute is identical to the class attribute of the same name,
8722which is described in the previous section. This attribute exists as
8723both an instance and class attribute so tests can use
8724``self.target.run(cmd)`` in instance methods instead of
8725``oeRuntimeTest.tc.target.run(cmd)``.
8726
8727Installing Packages in the DUT Without the Package Manager
8728----------------------------------------------------------
8729
8730When a test requires a package built by BitBake, it is possible to
8731install that package. Installing the package does not require a package
8732manager be installed in the device under test (DUT). It does, however,
8733require an SSH connection and the target must be using the
8734``sshcontrol`` class.
8735
8736.. note::
8737
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008738 This method uses ``scp`` to copy files from the host to the target, which
8739 causes permissions and special attributes to be lost.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008740
8741A JSON file is used to define the packages needed by a test. This file
8742must be in the same path as the file used to define the tests.
8743Furthermore, the filename must map directly to the test module name with
8744a ``.json`` extension.
8745
8746The JSON file must include an object with the test name as keys of an
8747object or an array. This object (or array of objects) uses the following
8748data:
8749
8750- "pkg" - A mandatory string that is the name of the package to be
8751 installed.
8752
8753- "rm" - An optional boolean, which defaults to "false", that specifies
8754 to remove the package after the test.
8755
8756- "extract" - An optional boolean, which defaults to "false", that
8757 specifies if the package must be extracted from the package format.
8758 When set to "true", the package is not automatically installed into
8759 the DUT.
8760
8761Following is an example JSON file that handles test "foo" installing
8762package "bar" and test "foobar" installing packages "foo" and "bar".
8763Once the test is complete, the packages are removed from the DUT.
8764::
8765
8766 {
8767 "foo": {
8768 "pkg": "bar"
8769 },
8770 "foobar": [
8771 {
8772 "pkg": "foo",
8773 "rm": true
8774 },
8775 {
8776 "pkg": "bar",
8777 "rm": true
8778 }
8779 ]
8780 }
8781
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008782Debugging Tools and Techniques
8783==============================
8784
8785The exact method for debugging build failures depends on the nature of
8786the problem and on the system's area from which the bug originates.
8787Standard debugging practices such as comparison against the last known
8788working version with examination of the changes and the re-application
8789of steps to identify the one causing the problem are valid for the Yocto
8790Project just as they are for any other system. Even though it is
8791impossible to detail every possible potential failure, this section
8792provides some general tips to aid in debugging given a variety of
8793situations.
8794
8795.. note::
8796
8797 A useful feature for debugging is the error reporting tool.
8798 Configuring the Yocto Project to use this tool causes the
8799 OpenEmbedded build system to produce error reporting commands as part
8800 of the console output. You can enter the commands after the build
8801 completes to log error information into a common database, that can
8802 help you figure out what might be going wrong. For information on how
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008803 to enable and use this feature, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06008804 ":ref:`dev-manual/common-tasks:using the error reporting tool`"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008805 section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008806
8807The following list shows the debugging topics in the remainder of this
8808section:
8809
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05008810- ":ref:`dev-manual/common-tasks:viewing logs from failed tasks`" describes
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008811 how to find and view logs from tasks that failed during the build
8812 process.
8813
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05008814- ":ref:`dev-manual/common-tasks:viewing variable values`" describes how to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008815 use the BitBake ``-e`` option to examine variable values after a
8816 recipe has been parsed.
8817
Andrew Geissler09209ee2020-12-13 08:44:15 -06008818- ":ref:`dev-manual/common-tasks:viewing package information with \`\`oe-pkgdata-util\`\``"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008819 describes how to use the ``oe-pkgdata-util`` utility to query
8820 :term:`PKGDATA_DIR` and
8821 display package-related information for built packages.
8822
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05008823- ":ref:`dev-manual/common-tasks:viewing dependencies between recipes and tasks`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008824 describes how to use the BitBake ``-g`` option to display recipe
8825 dependency information used during the build.
8826
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05008827- ":ref:`dev-manual/common-tasks:viewing task variable dependencies`" describes
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008828 how to use the ``bitbake-dumpsig`` command in conjunction with key
8829 subdirectories in the
8830 :term:`Build Directory` to determine
8831 variable dependencies.
8832
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05008833- ":ref:`dev-manual/common-tasks:running specific tasks`" describes
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008834 how to use several BitBake options (e.g. ``-c``, ``-C``, and ``-f``)
8835 to run specific tasks in the build chain. It can be useful to run
8836 tasks "out-of-order" when trying isolate build issues.
8837
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05008838- ":ref:`dev-manual/common-tasks:general bitbake problems`" describes how
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008839 to use BitBake's ``-D`` debug output option to reveal more about what
8840 BitBake is doing during the build.
8841
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05008842- ":ref:`dev-manual/common-tasks:building with no dependencies`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008843 describes how to use the BitBake ``-b`` option to build a recipe
8844 while ignoring dependencies.
8845
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05008846- ":ref:`dev-manual/common-tasks:recipe logging mechanisms`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008847 describes how to use the many recipe logging functions to produce
8848 debugging output and report errors and warnings.
8849
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05008850- ":ref:`dev-manual/common-tasks:debugging parallel make races`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008851 describes how to debug situations where the build consists of several
8852 parts that are run simultaneously and when the output or result of
8853 one part is not ready for use with a different part of the build that
8854 depends on that output.
8855
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05008856- ":ref:`dev-manual/common-tasks:debugging with the gnu project debugger (gdb) remotely`"
8857 describes how to use GDB to allow you to examine running programs, which can
8858 help you fix problems.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008859
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05008860- ":ref:`dev-manual/common-tasks:debugging with the gnu project debugger (gdb) on the target`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008861 describes how to use GDB directly on target hardware for debugging.
8862
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05008863- ":ref:`dev-manual/common-tasks:other debugging tips`" describes
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008864 miscellaneous debugging tips that can be useful.
8865
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008866Viewing Logs from Failed Tasks
8867------------------------------
8868
8869You can find the log for a task in the file
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008870``${``\ :term:`WORKDIR`\ ``}/temp/log.do_``\ `taskname`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008871For example, the log for the
8872:ref:`ref-tasks-compile` task of the
8873QEMU minimal image for the x86 machine (``qemux86``) might be in
8874``tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/temp/log.do_compile``.
8875To see the commands :term:`BitBake` ran
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008876to generate a log, look at the corresponding ``run.do_``\ `taskname` file
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008877in the same directory.
8878
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008879``log.do_``\ `taskname` and ``run.do_``\ `taskname` are actually symbolic
8880links to ``log.do_``\ `taskname`\ ``.``\ `pid` and
8881``log.run_``\ `taskname`\ ``.``\ `pid`, where `pid` is the PID the task had
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008882when it ran. The symlinks always point to the files corresponding to the
8883most recent run.
8884
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008885Viewing Variable Values
8886-----------------------
8887
8888Sometimes you need to know the value of a variable as a result of
8889BitBake's parsing step. This could be because some unexpected behavior
8890occurred in your project. Perhaps an attempt to :ref:`modify a variable
8891<bitbake:bitbake-user-manual/bitbake-user-manual-metadata:modifying existing
8892variables>` did not work out as expected.
8893
8894BitBake's ``-e`` option is used to display variable values after
8895parsing. The following command displays the variable values after the
8896configuration files (i.e. ``local.conf``, ``bblayers.conf``,
Andrew Geisslerc926e172021-05-07 16:11:35 -05008897``bitbake.conf`` and so forth) have been parsed::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008898
8899 $ bitbake -e
8900
8901The following command displays variable values after a specific recipe has
Andrew Geisslerc926e172021-05-07 16:11:35 -05008902been parsed. The variables include those from the configuration as well::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008903
8904 $ bitbake -e recipename
8905
8906.. note::
8907
8908 Each recipe has its own private set of variables (datastore).
8909 Internally, after parsing the configuration, a copy of the resulting
8910 datastore is made prior to parsing each recipe. This copying implies
8911 that variables set in one recipe will not be visible to other
8912 recipes.
8913
8914 Likewise, each task within a recipe gets a private datastore based on
8915 the recipe datastore, which means that variables set within one task
8916 will not be visible to other tasks.
8917
8918In the output of ``bitbake -e``, each variable is preceded by a
8919description of how the variable got its value, including temporary
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008920values that were later overridden. This description also includes
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008921variable flags (varflags) set on the variable. The output can be very
8922helpful during debugging.
8923
8924Variables that are exported to the environment are preceded by
Andrew Geisslerc926e172021-05-07 16:11:35 -05008925``export`` in the output of ``bitbake -e``. See the following example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008926
8927 export CC="i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/ulf/poky/build/tmp/sysroots/qemux86"
8928
8929In addition to variable values, the output of the ``bitbake -e`` and
8930``bitbake -e`` recipe commands includes the following information:
8931
8932- The output starts with a tree listing all configuration files and
8933 classes included globally, recursively listing the files they include
8934 or inherit in turn. Much of the behavior of the OpenEmbedded build
Andrew Geissler09209ee2020-12-13 08:44:15 -06008935 system (including the behavior of the :ref:`ref-manual/tasks:normal recipe build tasks`) is
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008936 implemented in the
8937 :ref:`base <ref-classes-base>` class and the
8938 classes it inherits, rather than being built into BitBake itself.
8939
8940- After the variable values, all functions appear in the output. For
8941 shell functions, variables referenced within the function body are
8942 expanded. If a function has been modified using overrides or using
8943 override-style operators like ``_append`` and ``_prepend``, then the
8944 final assembled function body appears in the output.
8945
8946Viewing Package Information with ``oe-pkgdata-util``
8947----------------------------------------------------
8948
8949You can use the ``oe-pkgdata-util`` command-line utility to query
8950:term:`PKGDATA_DIR` and display
8951various package-related information. When you use the utility, you must
8952use it to view information on packages that have already been built.
8953
8954Following are a few of the available ``oe-pkgdata-util`` subcommands.
8955
8956.. note::
8957
8958 You can use the standard \* and ? globbing wildcards as part of
8959 package names and paths.
8960
8961- ``oe-pkgdata-util list-pkgs [pattern]``: Lists all packages
8962 that have been built, optionally limiting the match to packages that
8963 match pattern.
8964
8965- ``oe-pkgdata-util list-pkg-files package ...``: Lists the
8966 files and directories contained in the given packages.
8967
8968 .. note::
8969
8970 A different way to view the contents of a package is to look at
8971 the
8972 ``${``\ :term:`WORKDIR`\ ``}/packages-split``
8973 directory of the recipe that generates the package. This directory
8974 is created by the
8975 :ref:`ref-tasks-package` task
8976 and has one subdirectory for each package the recipe generates,
8977 which contains the files stored in that package.
8978
8979 If you want to inspect the ``${WORKDIR}/packages-split``
8980 directory, make sure that
8981 :ref:`rm_work <ref-classes-rm-work>` is not
8982 enabled when you build the recipe.
8983
8984- ``oe-pkgdata-util find-path path ...``: Lists the names of
8985 the packages that contain the given paths. For example, the following
8986 tells us that ``/usr/share/man/man1/make.1`` is contained in the
Andrew Geisslerc926e172021-05-07 16:11:35 -05008987 ``make-doc`` package::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008988
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008989 $ oe-pkgdata-util find-path /usr/share/man/man1/make.1
8990 make-doc: /usr/share/man/man1/make.1
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008991
8992- ``oe-pkgdata-util lookup-recipe package ...``: Lists the name
8993 of the recipes that produce the given packages.
8994
8995For more information on the ``oe-pkgdata-util`` command, use the help
Andrew Geisslerc926e172021-05-07 16:11:35 -05008996facility::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008997
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008998 $ oe-pkgdata-util --help
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008999 $ oe-pkgdata-util subcommand --help
9000
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009001Viewing Dependencies Between Recipes and Tasks
9002----------------------------------------------
9003
9004Sometimes it can be hard to see why BitBake wants to build other recipes
9005before the one you have specified. Dependency information can help you
9006understand why a recipe is built.
9007
9008To generate dependency information for a recipe, run the following
Andrew Geisslerc926e172021-05-07 16:11:35 -05009009command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009010
9011 $ bitbake -g recipename
9012
9013This command writes the following files in the current directory:
9014
9015- ``pn-buildlist``: A list of recipes/targets involved in building
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009016 `recipename`. "Involved" here means that at least one task from the
9017 recipe needs to run when building `recipename` from scratch. Targets
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009018 that are in
9019 :term:`ASSUME_PROVIDED`
9020 are not listed.
9021
9022- ``task-depends.dot``: A graph showing dependencies between tasks.
9023
9024The graphs are in
9025`DOT <https://en.wikipedia.org/wiki/DOT_%28graph_description_language%29>`__
9026format and can be converted to images (e.g. using the ``dot`` tool from
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009027`Graphviz <https://www.graphviz.org/>`__).
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009028
9029.. note::
9030
9031 - DOT files use a plain text format. The graphs generated using the
9032 ``bitbake -g`` command are often so large as to be difficult to
9033 read without special pruning (e.g. with Bitbake's ``-I`` option)
9034 and processing. Despite the form and size of the graphs, the
9035 corresponding ``.dot`` files can still be possible to read and
9036 provide useful information.
9037
9038 As an example, the ``task-depends.dot`` file contains lines such
Andrew Geisslerc926e172021-05-07 16:11:35 -05009039 as the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009040
9041 "libxslt.do_configure" -> "libxml2.do_populate_sysroot"
9042
9043 The above example line reveals that the
9044 :ref:`ref-tasks-configure`
9045 task in ``libxslt`` depends on the
9046 :ref:`ref-tasks-populate_sysroot`
9047 task in ``libxml2``, which is a normal
9048 :term:`DEPENDS` dependency
9049 between the two recipes.
9050
9051 - For an example of how ``.dot`` files can be processed, see the
9052 ``scripts/contrib/graph-tool`` Python script, which finds and
9053 displays paths between graph nodes.
9054
9055You can use a different method to view dependency information by using
Andrew Geisslerc926e172021-05-07 16:11:35 -05009056the following command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009057
9058 $ bitbake -g -u taskexp recipename
9059
9060This command
9061displays a GUI window from which you can view build-time and runtime
9062dependencies for the recipes involved in building recipename.
9063
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009064Viewing Task Variable Dependencies
9065----------------------------------
9066
9067As mentioned in the
9068":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-execution:checksums (signatures)`" section of the BitBake
9069User Manual, BitBake tries to automatically determine what variables a
9070task depends on so that it can rerun the task if any values of the
9071variables change. This determination is usually reliable. However, if
9072you do things like construct variable names at runtime, then you might
9073have to manually declare dependencies on those variables using
9074``vardeps`` as described in the
9075":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags`" section of the BitBake
9076User Manual.
9077
9078If you are unsure whether a variable dependency is being picked up
9079automatically for a given task, you can list the variable dependencies
9080BitBake has determined by doing the following:
9081
Andrew Geisslerc926e172021-05-07 16:11:35 -050090821. Build the recipe containing the task::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009083
9084 $ bitbake recipename
9085
90862. Inside the :term:`STAMPS_DIR`
9087 directory, find the signature data (``sigdata``) file that
9088 corresponds to the task. The ``sigdata`` files contain a pickled
9089 Python database of all the metadata that went into creating the input
9090 checksum for the task. As an example, for the
9091 :ref:`ref-tasks-fetch` task of the
9092 ``db`` recipe, the ``sigdata`` file might be found in the following
Andrew Geisslerc926e172021-05-07 16:11:35 -05009093 location::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009094
9095 ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1
9096
9097 For tasks that are accelerated through the shared state
Andrew Geissler09209ee2020-12-13 08:44:15 -06009098 (:ref:`sstate <overview-manual/concepts:shared state cache>`) cache, an
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009099 additional ``siginfo`` file is written into
9100 :term:`SSTATE_DIR` along with
9101 the cached task output. The ``siginfo`` files contain exactly the
9102 same information as ``sigdata`` files.
9103
91043. Run ``bitbake-dumpsig`` on the ``sigdata`` or ``siginfo`` file. Here
Andrew Geisslerc926e172021-05-07 16:11:35 -05009105 is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009106
9107 $ bitbake-dumpsig ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1
9108
9109 In the output of the above command, you will find a line like the
9110 following, which lists all the (inferred) variable dependencies for
9111 the task. This list also includes indirect dependencies from
9112 variables depending on other variables, recursively.
9113 ::
9114
9115 Task dependencies: ['PV', 'SRCREV', 'SRC_URI', 'SRC_URI[md5sum]', 'SRC_URI[sha256sum]', 'base_do_fetch']
9116
9117 .. note::
9118
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009119 Functions (e.g. ``base_do_fetch``) also count as variable dependencies.
9120 These functions in turn depend on the variables they reference.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009121
9122 The output of ``bitbake-dumpsig`` also includes the value each
9123 variable had, a list of dependencies for each variable, and
9124 :term:`bitbake:BB_HASHBASE_WHITELIST`
9125 information.
9126
9127There is also a ``bitbake-diffsigs`` command for comparing two
9128``siginfo`` or ``sigdata`` files. This command can be helpful when
9129trying to figure out what changed between two versions of a task. If you
9130call ``bitbake-diffsigs`` with just one file, the command behaves like
9131``bitbake-dumpsig``.
9132
9133You can also use BitBake to dump out the signature construction
9134information without executing tasks by using either of the following
Andrew Geisslerc926e172021-05-07 16:11:35 -05009135BitBake command-line options::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009136
9137 ‐‐dump-signatures=SIGNATURE_HANDLER
9138 -S SIGNATURE_HANDLER
9139
9140
9141.. note::
9142
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009143 Two common values for `SIGNATURE_HANDLER` are "none" and "printdiff", which
9144 dump only the signature or compare the dumped signature with the cached one,
9145 respectively.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009146
9147Using BitBake with either of these options causes BitBake to dump out
9148``sigdata`` files in the ``stamps`` directory for every task it would
9149have executed instead of building the specified target package.
9150
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009151Viewing Metadata Used to Create the Input Signature of a Shared State Task
9152--------------------------------------------------------------------------
9153
9154Seeing what metadata went into creating the input signature of a shared
9155state (sstate) task can be a useful debugging aid. This information is
9156available in signature information (``siginfo``) files in
9157:term:`SSTATE_DIR`. For
9158information on how to view and interpret information in ``siginfo``
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05009159files, see the
9160":ref:`dev-manual/common-tasks:viewing task variable dependencies`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009161
9162For conceptual information on shared state, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06009163":ref:`overview-manual/concepts:shared state`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009164section in the Yocto Project Overview and Concepts Manual.
9165
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009166Invalidating Shared State to Force a Task to Run
9167------------------------------------------------
9168
9169The OpenEmbedded build system uses
Andrew Geissler09209ee2020-12-13 08:44:15 -06009170:ref:`checksums <overview-manual/concepts:checksums (signatures)>` and
9171:ref:`overview-manual/concepts:shared state` cache to avoid unnecessarily
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009172rebuilding tasks. Collectively, this scheme is known as "shared state
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009173code".
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009174
9175As with all schemes, this one has some drawbacks. It is possible that
9176you could make implicit changes to your code that the checksum
9177calculations do not take into account. These implicit changes affect a
9178task's output but do not trigger the shared state code into rebuilding a
9179recipe. Consider an example during which a tool changes its output.
9180Assume that the output of ``rpmdeps`` changes. The result of the change
9181should be that all the ``package`` and ``package_write_rpm`` shared
9182state cache items become invalid. However, because the change to the
9183output is external to the code and therefore implicit, the associated
9184shared state cache items do not become invalidated. In this case, the
9185build process uses the cached items rather than running the task again.
9186Obviously, these types of implicit changes can cause problems.
9187
9188To avoid these problems during the build, you need to understand the
9189effects of any changes you make. Realize that changes you make directly
9190to a function are automatically factored into the checksum calculation.
9191Thus, these explicit changes invalidate the associated area of shared
9192state cache. However, you need to be aware of any implicit changes that
9193are not obvious changes to the code and could affect the output of a
9194given task.
9195
9196When you identify an implicit change, you can easily take steps to
9197invalidate the cache and force the tasks to run. The steps you can take
9198are as simple as changing a function's comments in the source code. For
9199example, to invalidate package shared state files, change the comment
9200statements of
9201:ref:`ref-tasks-package` or the
9202comments of one of the functions it calls. Even though the change is
9203purely cosmetic, it causes the checksum to be recalculated and forces
9204the build system to run the task again.
9205
9206.. note::
9207
9208 For an example of a commit that makes a cosmetic change to invalidate
9209 shared state, see this
Andrew Geissler09209ee2020-12-13 08:44:15 -06009210 :yocto_git:`commit </poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009211
9212Running Specific Tasks
9213----------------------
9214
9215Any given recipe consists of a set of tasks. The standard BitBake
9216behavior in most cases is: ``do_fetch``, ``do_unpack``, ``do_patch``,
9217``do_configure``, ``do_compile``, ``do_install``, ``do_package``,
9218``do_package_write_*``, and ``do_build``. The default task is
9219``do_build`` and any tasks on which it depends build first. Some tasks,
9220such as ``do_devshell``, are not part of the default build chain. If you
9221wish to run a task that is not part of the default build chain, you can
Andrew Geisslerc926e172021-05-07 16:11:35 -05009222use the ``-c`` option in BitBake. Here is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009223
9224 $ bitbake matchbox-desktop -c devshell
9225
9226The ``-c`` option respects task dependencies, which means that all other
9227tasks (including tasks from other recipes) that the specified task
9228depends on will be run before the task. Even when you manually specify a
9229task to run with ``-c``, BitBake will only run the task if it considers
9230it "out of date". See the
Andrew Geissler09209ee2020-12-13 08:44:15 -06009231":ref:`overview-manual/concepts:stamp files and the rerunning of tasks`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009232section in the Yocto Project Overview and Concepts Manual for how
9233BitBake determines whether a task is "out of date".
9234
9235If you want to force an up-to-date task to be rerun (e.g. because you
9236made manual modifications to the recipe's
9237:term:`WORKDIR` that you want to try
9238out), then you can use the ``-f`` option.
9239
9240.. note::
9241
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009242 The reason ``-f`` is never required when running the
9243 :ref:`ref-tasks-devshell` task is because the
9244 [\ :ref:`nostamp <bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ]
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009245 variable flag is already set for the task.
9246
Andrew Geisslerc926e172021-05-07 16:11:35 -05009247The following example shows one way you can use the ``-f`` option::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009248
9249 $ bitbake matchbox-desktop
9250 .
9251 .
9252 make some changes to the source code in the work directory
9253 .
9254 .
9255 $ bitbake matchbox-desktop -c compile -f
9256 $ bitbake matchbox-desktop
9257
9258This sequence first builds and then recompiles ``matchbox-desktop``. The
9259last command reruns all tasks (basically the packaging tasks) after the
9260compile. BitBake recognizes that the ``do_compile`` task was rerun and
9261therefore understands that the other tasks also need to be run again.
9262
9263Another, shorter way to rerun a task and all
Andrew Geissler09209ee2020-12-13 08:44:15 -06009264:ref:`ref-manual/tasks:normal recipe build tasks`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009265that depend on it is to use the ``-C`` option.
9266
9267.. note::
9268
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009269 This option is upper-cased and is separate from the ``-c``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009270 option, which is lower-cased.
9271
9272Using this option invalidates the given task and then runs the
9273:ref:`ref-tasks-build` task, which is
9274the default task if no task is given, and the tasks on which it depends.
9275You could replace the final two commands in the previous example with
Andrew Geisslerc926e172021-05-07 16:11:35 -05009276the following single command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009277
9278 $ bitbake matchbox-desktop -C compile
9279
9280Internally, the ``-f`` and ``-C`` options work by tainting (modifying)
9281the input checksum of the specified task. This tainting indirectly
9282causes the task and its dependent tasks to be rerun through the normal
9283task dependency mechanisms.
9284
9285.. note::
9286
9287 BitBake explicitly keeps track of which tasks have been tainted in
9288 this fashion, and will print warnings such as the following for
9289 builds involving such tasks:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009290
9291 .. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009292
9293 WARNING: /home/ulf/poky/meta/recipes-sato/matchbox-desktop/matchbox-desktop_2.1.bb.do_compile is tainted from a forced run
9294
9295
9296 The purpose of the warning is to let you know that the work directory
9297 and build output might not be in the clean state they would be in for
9298 a "normal" build, depending on what actions you took. To get rid of
9299 such warnings, you can remove the work directory and rebuild the
Andrew Geisslerc926e172021-05-07 16:11:35 -05009300 recipe, as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009301
9302 $ bitbake matchbox-desktop -c clean
9303 $ bitbake matchbox-desktop
9304
9305
9306You can view a list of tasks in a given package by running the
Andrew Geisslerc926e172021-05-07 16:11:35 -05009307``do_listtasks`` task as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009308
9309 $ bitbake matchbox-desktop -c listtasks
9310
9311The results appear as output to the console and are also in
9312the file ``${WORKDIR}/temp/log.do_listtasks``.
9313
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009314General BitBake Problems
9315------------------------
9316
9317You can see debug output from BitBake by using the ``-D`` option. The
9318debug output gives more information about what BitBake is doing and the
9319reason behind it. Each ``-D`` option you use increases the logging
9320level. The most common usage is ``-DDD``.
9321
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009322The output from ``bitbake -DDD -v targetname`` can reveal why BitBake
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009323chose a certain version of a package or why BitBake picked a certain
9324provider. This command could also help you in a situation where you
9325think BitBake did something unexpected.
9326
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009327Building with No Dependencies
9328-----------------------------
9329
9330To build a specific recipe (``.bb`` file), you can use the following
Andrew Geisslerc926e172021-05-07 16:11:35 -05009331command form::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009332
9333 $ bitbake -b somepath/somerecipe.bb
9334
9335This command form does
9336not check for dependencies. Consequently, you should use it only when
9337you know existing dependencies have been met.
9338
9339.. note::
9340
9341 You can also specify fragments of the filename. In this case, BitBake
9342 checks for a unique match.
9343
9344Recipe Logging Mechanisms
9345-------------------------
9346
9347The Yocto Project provides several logging functions for producing
9348debugging output and reporting errors and warnings. For Python
William A. Kennington IIIac69b482021-06-02 12:28:27 -07009349functions, the following logging functions are available. All of these functions
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009350log to ``${T}/log.do_``\ `task`, and can also log to standard output
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009351(stdout) with the right settings:
9352
9353- ``bb.plain(msg)``: Writes msg as is to the log while also
9354 logging to stdout.
9355
9356- ``bb.note(msg)``: Writes "NOTE: msg" to the log. Also logs to
9357 stdout if BitBake is called with "-v".
9358
9359- ``bb.debug(level, msg)``: Writes "DEBUG: msg" to the
9360 log. Also logs to stdout if the log level is greater than or equal to
9361 level. See the ":ref:`-D <bitbake:bitbake-user-manual/bitbake-user-manual-intro:usage and syntax>`" option
9362 in the BitBake User Manual for more information.
9363
9364- ``bb.warn(msg)``: Writes "WARNING: msg" to the log while also
9365 logging to stdout.
9366
9367- ``bb.error(msg)``: Writes "ERROR: msg" to the log while also
9368 logging to standard out (stdout).
9369
9370 .. note::
9371
9372 Calling this function does not cause the task to fail.
9373
9374- ``bb.fatal(``\ msg\ ``)``: This logging function is similar to
9375 ``bb.error(``\ msg\ ``)`` but also causes the calling task to fail.
9376
9377 .. note::
9378
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009379 ``bb.fatal()`` raises an exception, which means you do not need to put a
9380 "return" statement after the function.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009381
9382The same logging functions are also available in shell functions, under
9383the names ``bbplain``, ``bbnote``, ``bbdebug``, ``bbwarn``, ``bberror``,
9384and ``bbfatal``. The
9385:ref:`logging <ref-classes-logging>` class
9386implements these functions. See that class in the ``meta/classes``
9387folder of the :term:`Source Directory` for information.
9388
9389Logging With Python
9390~~~~~~~~~~~~~~~~~~~
9391
9392When creating recipes using Python and inserting code that handles build
9393logs, keep in mind the goal is to have informative logs while keeping
9394the console as "silent" as possible. Also, if you want status messages
9395in the log, use the "debug" loglevel.
9396
9397Following is an example written in Python. The code handles logging for
9398a function that determines the number of tasks needed to be run. See the
9399":ref:`ref-tasks-listtasks`"
Andrew Geisslerc926e172021-05-07 16:11:35 -05009400section for additional information::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009401
9402 python do_listtasks() {
9403 bb.debug(2, "Starting to figure out the task list")
9404 if noteworthy_condition:
9405 bb.note("There are 47 tasks to run")
9406 bb.debug(2, "Got to point xyz")
9407 if warning_trigger:
9408 bb.warn("Detected warning_trigger, this might be a problem later.")
9409 if recoverable_error:
9410 bb.error("Hit recoverable_error, you really need to fix this!")
9411 if fatal_error:
9412 bb.fatal("fatal_error detected, unable to print the task list")
9413 bb.plain("The tasks present are abc")
9414 bb.debug(2, "Finished figuring out the tasklist")
9415 }
9416
9417Logging With Bash
9418~~~~~~~~~~~~~~~~~
9419
9420When creating recipes using Bash and inserting code that handles build
9421logs, you have the same goals - informative with minimal console output.
9422The syntax you use for recipes written in Bash is similar to that of
9423recipes written in Python described in the previous section.
9424
9425Following is an example written in Bash. The code logs the progress of
9426the ``do_my_function`` function.
9427::
9428
9429 do_my_function() {
9430 bbdebug 2 "Running do_my_function"
9431 if [ exceptional_condition ]; then
9432 bbnote "Hit exceptional_condition"
9433 fi
9434 bbdebug 2 "Got to point xyz"
9435 if [ warning_trigger ]; then
9436 bbwarn "Detected warning_trigger, this might cause a problem later."
9437 fi
9438 if [ recoverable_error ]; then
9439 bberror "Hit recoverable_error, correcting"
9440 fi
9441 if [ fatal_error ]; then
9442 bbfatal "fatal_error detected"
9443 fi
9444 bbdebug 2 "Completed do_my_function"
9445 }
9446
9447
9448Debugging Parallel Make Races
9449-----------------------------
9450
9451A parallel ``make`` race occurs when the build consists of several parts
9452that are run simultaneously and a situation occurs when the output or
9453result of one part is not ready for use with a different part of the
9454build that depends on that output. Parallel make races are annoying and
William A. Kennington IIIac69b482021-06-02 12:28:27 -07009455can sometimes be difficult to reproduce and fix. However, there are some simple
9456tips and tricks that can help you debug and fix them. This section
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009457presents a real-world example of an error encountered on the Yocto
9458Project autobuilder and the process used to fix it.
9459
9460.. note::
9461
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009462 If you cannot properly fix a ``make`` race condition, you can work around it
9463 by clearing either the :term:`PARALLEL_MAKE` or :term:`PARALLEL_MAKEINST`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009464 variables.
9465
9466The Failure
9467~~~~~~~~~~~
9468
9469For this example, assume that you are building an image that depends on
9470the "neard" package. And, during the build, BitBake runs into problems
9471and creates the following output.
9472
9473.. note::
9474
9475 This example log file has longer lines artificially broken to make
9476 the listing easier to read.
9477
9478If you examine the output or the log file, you see the failure during
9479``make``:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009480
9481.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009482
9483 | DEBUG: SITE files ['endian-little', 'bit-32', 'ix86-common', 'common-linux', 'common-glibc', 'i586-linux', 'common']
9484 | DEBUG: Executing shell function do_compile
9485 | NOTE: make -j 16
9486 | make --no-print-directory all-am
9487 | /bin/mkdir -p include/near
9488 | /bin/mkdir -p include/near
9489 | /bin/mkdir -p include/near
9490 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9491 0.14-r0/neard-0.14/include/types.h include/near/types.h
9492 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9493 0.14-r0/neard-0.14/include/log.h include/near/log.h
9494 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9495 0.14-r0/neard-0.14/include/plugin.h include/near/plugin.h
9496 | /bin/mkdir -p include/near
9497 | /bin/mkdir -p include/near
9498 | /bin/mkdir -p include/near
9499 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9500 0.14-r0/neard-0.14/include/tag.h include/near/tag.h
9501 | /bin/mkdir -p include/near
9502 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9503 0.14-r0/neard-0.14/include/adapter.h include/near/adapter.h
9504 | /bin/mkdir -p include/near
9505 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9506 0.14-r0/neard-0.14/include/ndef.h include/near/ndef.h
9507 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9508 0.14-r0/neard-0.14/include/tlv.h include/near/tlv.h
9509 | /bin/mkdir -p include/near
9510 | /bin/mkdir -p include/near
9511 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9512 0.14-r0/neard-0.14/include/setting.h include/near/setting.h
9513 | /bin/mkdir -p include/near
9514 | /bin/mkdir -p include/near
9515 | /bin/mkdir -p include/near
9516 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9517 0.14-r0/neard-0.14/include/device.h include/near/device.h
9518 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9519 0.14-r0/neard-0.14/include/nfc_copy.h include/near/nfc_copy.h
9520 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9521 0.14-r0/neard-0.14/include/snep.h include/near/snep.h
9522 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9523 0.14-r0/neard-0.14/include/version.h include/near/version.h
9524 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9525 0.14-r0/neard-0.14/include/dbus.h include/near/dbus.h
9526 | ./src/genbuiltin nfctype1 nfctype2 nfctype3 nfctype4 p2p > src/builtin.h
9527 | i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/
9528 build/build/tmp/sysroots/qemux86 -DHAVE_CONFIG_H -I. -I./include -I./src -I./gdbus -I/home/pokybuild/
9529 yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/sysroots/qemux86/usr/include/glib-2.0
9530 -I/home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/sysroots/qemux86/usr/
9531 lib/glib-2.0/include -I/home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/
9532 tmp/sysroots/qemux86/usr/include/dbus-1.0 -I/home/pokybuild/yocto-autobuilder/yocto-slave/
9533 nightly-x86/build/build/tmp/sysroots/qemux86/usr/lib/dbus-1.0/include -I/home/pokybuild/yocto-autobuilder/
9534 yocto-slave/nightly-x86/build/build/tmp/sysroots/qemux86/usr/include/libnl3
9535 -DNEAR_PLUGIN_BUILTIN -DPLUGINDIR=\""/usr/lib/near/plugins"\"
9536 -DCONFIGDIR=\""/etc/neard\"" -O2 -pipe -g -feliminate-unused-debug-types -c
9537 -o tools/snep-send.o tools/snep-send.c
9538 | In file included from tools/snep-send.c:16:0:
9539 | tools/../src/near.h:41:23: fatal error: near/dbus.h: No such file or directory
9540 | #include <near/dbus.h>
9541 | ^
9542 | compilation terminated.
9543 | make[1]: *** [tools/snep-send.o] Error 1
9544 | make[1]: *** Waiting for unfinished jobs....
9545 | make: *** [all] Error 2
9546 | ERROR: oe_runmake failed
9547
9548Reproducing the Error
9549~~~~~~~~~~~~~~~~~~~~~
9550
9551Because race conditions are intermittent, they do not manifest
9552themselves every time you do the build. In fact, most times the build
9553will complete without problems even though the potential race condition
9554exists. Thus, once the error surfaces, you need a way to reproduce it.
9555
9556In this example, compiling the "neard" package is causing the problem.
9557So the first thing to do is build "neard" locally. Before you start the
9558build, set the
9559:term:`PARALLEL_MAKE` variable
9560in your ``local.conf`` file to a high number (e.g. "-j 20"). Using a
Andrew Geissler09036742021-06-25 14:25:14 -05009561high value for :term:`PARALLEL_MAKE` increases the chances of the race
Andrew Geisslerc926e172021-05-07 16:11:35 -05009562condition showing up::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009563
9564 $ bitbake neard
9565
Andrew Geisslerc926e172021-05-07 16:11:35 -05009566Once the local build for "neard" completes, start a ``devshell`` build::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009567
9568 $ bitbake neard -c devshell
9569
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05009570For information on how to use a ``devshell``, see the
9571":ref:`dev-manual/common-tasks:using a development shell`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009572
Andrew Geisslerc926e172021-05-07 16:11:35 -05009573In the ``devshell``, do the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009574
9575 $ make clean
9576 $ make tools/snep-send.o
9577
9578The ``devshell`` commands cause the failure to clearly
William A. Kennington IIIac69b482021-06-02 12:28:27 -07009579be visible. In this case, there is a missing dependency for the ``neard``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009580Makefile target. Here is some abbreviated, sample output with the
Andrew Geisslerc926e172021-05-07 16:11:35 -05009581missing dependency clearly visible at the end::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009582
9583 i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/scott-lenovo/......
9584 .
9585 .
9586 .
9587 tools/snep-send.c
9588 In file included from tools/snep-send.c:16:0:
9589 tools/../src/near.h:41:23: fatal error: near/dbus.h: No such file or directory
9590 #include <near/dbus.h>
9591 ^
9592 compilation terminated.
9593 make: *** [tools/snep-send.o] Error 1
9594 $
9595
9596
9597Creating a Patch for the Fix
9598~~~~~~~~~~~~~~~~~~~~~~~~~~~~
9599
9600Because there is a missing dependency for the Makefile target, you need
9601to patch the ``Makefile.am`` file, which is generated from
Andrew Geisslerc926e172021-05-07 16:11:35 -05009602``Makefile.in``. You can use Quilt to create the patch::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009603
9604 $ quilt new parallelmake.patch
9605 Patch patches/parallelmake.patch is now on top
9606 $ quilt add Makefile.am
9607 File Makefile.am added to patch patches/parallelmake.patch
9608
9609For more information on using Quilt, see the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05009610":ref:`dev-manual/common-tasks:using quilt in your workflow`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009611
9612At this point you need to make the edits to ``Makefile.am`` to add the
9613missing dependency. For our example, you have to add the following line
Andrew Geisslerc926e172021-05-07 16:11:35 -05009614to the file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009615
9616 tools/snep-send.$(OBJEXT): include/near/dbus.h
9617
9618Once you have edited the file, use the ``refresh`` command to create the
Andrew Geisslerc926e172021-05-07 16:11:35 -05009619patch::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009620
9621 $ quilt refresh
9622 Refreshed patch patches/parallelmake.patch
9623
William A. Kennington IIIac69b482021-06-02 12:28:27 -07009624Once the patch file is created, you need to add it back to the originating
9625recipe folder. Here is an example assuming a top-level
Andrew Geisslerc926e172021-05-07 16:11:35 -05009626:term:`Source Directory` named ``poky``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009627
9628 $ cp patches/parallelmake.patch poky/meta/recipes-connectivity/neard/neard
9629
9630The final thing you need to do to implement the fix in the build is to
9631update the "neard" recipe (i.e. ``neard-0.14.bb``) so that the
9632:term:`SRC_URI` statement includes
9633the patch file. The recipe file is in the folder above the patch. Here
Andrew Geissler09036742021-06-25 14:25:14 -05009634is what the edited :term:`SRC_URI` statement would look like::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009635
9636 SRC_URI = "${KERNELORG_MIRROR}/linux/network/nfc/${BPN}-${PV}.tar.xz \
9637 file://neard.in \
9638 file://neard.service.in \
9639 file://parallelmake.patch \
9640 "
9641
9642With the patch complete and moved to the correct folder and the
Andrew Geissler09036742021-06-25 14:25:14 -05009643:term:`SRC_URI` statement updated, you can exit the ``devshell``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009644
9645 $ exit
9646
9647Testing the Build
9648~~~~~~~~~~~~~~~~~
9649
9650With everything in place, you can get back to trying the build again
Andrew Geisslerc926e172021-05-07 16:11:35 -05009651locally::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009652
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009653 $ bitbake neard
9654
9655This build should succeed.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009656
9657Now you can open up a ``devshell`` again and repeat the clean and make
Andrew Geisslerc926e172021-05-07 16:11:35 -05009658operations as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009659
9660 $ bitbake neard -c devshell
9661 $ make clean
9662 $ make tools/snep-send.o
9663
9664The build should work without issue.
9665
9666As with all solved problems, if they originated upstream, you need to
9667submit the fix for the recipe in OE-Core and upstream so that the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05009668problem is taken care of at its source. See the
9669":ref:`dev-manual/common-tasks:submitting a change to the yocto project`"
9670section for more information.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009671
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009672Debugging With the GNU Project Debugger (GDB) Remotely
9673------------------------------------------------------
9674
9675GDB allows you to examine running programs, which in turn helps you to
9676understand and fix problems. It also allows you to perform post-mortem
9677style analysis of program crashes. GDB is available as a package within
9678the Yocto Project and is installed in SDK images by default. See the
Andrew Geissler09209ee2020-12-13 08:44:15 -06009679":ref:`ref-manual/images:Images`" chapter in the Yocto
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009680Project Reference Manual for a description of these images. You can find
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009681information on GDB at https://sourceware.org/gdb/.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009682
9683.. note::
9684
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009685 For best results, install debug (``-dbg``) packages for the applications you
9686 are going to debug. Doing so makes extra debug symbols available that give
9687 you more meaningful output.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009688
9689Sometimes, due to memory or disk space constraints, it is not possible
9690to use GDB directly on the remote target to debug applications. These
9691constraints arise because GDB needs to load the debugging information
9692and the binaries of the process being debugged. Additionally, GDB needs
9693to perform many computations to locate information such as function
9694names, variable names and values, stack traces and so forth - even
9695before starting the debugging process. These extra computations place
9696more load on the target system and can alter the characteristics of the
9697program being debugged.
9698
Andrew Geissler95ac1b82021-03-31 14:34:31 -05009699To help get past the previously mentioned constraints, there are two
9700methods you can use: running a debuginfod server and using gdbserver.
9701
9702Using the debuginfod server method
9703~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
9704
Andrew Geisslerc926e172021-05-07 16:11:35 -05009705``debuginfod`` from ``elfutils`` is a way to distribute ``debuginfo`` files.
9706Running a ``debuginfod`` server makes debug symbols readily available,
Andrew Geissler95ac1b82021-03-31 14:34:31 -05009707which means you don't need to download debugging information
9708and the binaries of the process being debugged. You can just fetch
9709debug symbols from the server.
9710
Andrew Geisslerc926e172021-05-07 16:11:35 -05009711To run a ``debuginfod`` server, you need to do the following:
Andrew Geissler95ac1b82021-03-31 14:34:31 -05009712
Andrew Geisslerc926e172021-05-07 16:11:35 -05009713- Ensure that ``debuginfod`` is present in :term:`DISTRO_FEATURES`
9714 (it already is in ``OpenEmbedded-core`` defaults and ``poky`` reference distribution).
9715 If not, set in your distro config file or in ``local.conf``::
Andrew Geissler95ac1b82021-03-31 14:34:31 -05009716
Andrew Geisslerc926e172021-05-07 16:11:35 -05009717 DISTRO_FEATURES_append = " debuginfod"
Andrew Geissler95ac1b82021-03-31 14:34:31 -05009718
Andrew Geisslerc926e172021-05-07 16:11:35 -05009719 This distro feature enables the server and client library in ``elfutils``,
9720 and enables ``debuginfod`` support in clients (at the moment, ``gdb`` and ``binutils``).
Andrew Geissler95ac1b82021-03-31 14:34:31 -05009721
Andrew Geisslerc926e172021-05-07 16:11:35 -05009722- Run the following commands to launch the ``debuginfod`` server on the host::
Andrew Geissler95ac1b82021-03-31 14:34:31 -05009723
9724 $ oe-debuginfod
9725
Andrew Geisslerc926e172021-05-07 16:11:35 -05009726- To use ``debuginfod`` on the target, you need to know the ip:port where
9727 ``debuginfod`` is listening on the host (port defaults to 8002), and export
9728 that into the shell environment, for example in ``qemu``::
Andrew Geissler95ac1b82021-03-31 14:34:31 -05009729
Andrew Geisslerc926e172021-05-07 16:11:35 -05009730 root@qemux86-64:~# export DEBUGINFOD_URLS="http://192.168.7.1:8002/"
Andrew Geissler95ac1b82021-03-31 14:34:31 -05009731
Andrew Geisslerc926e172021-05-07 16:11:35 -05009732- Then debug info fetching should simply work when running the target ``gdb``,
9733 ``readelf`` or ``objdump``, for example::
Andrew Geissler95ac1b82021-03-31 14:34:31 -05009734
Andrew Geisslerc926e172021-05-07 16:11:35 -05009735 root@qemux86-64:~# gdb /bin/cat
9736 ...
9737 Reading symbols from /bin/cat...
9738 Downloading separate debug info for /bin/cat...
9739 Reading symbols from /home/root/.cache/debuginfod_client/923dc4780cfbc545850c616bffa884b6b5eaf322/debuginfo...
Andrew Geissler95ac1b82021-03-31 14:34:31 -05009740
Andrew Geisslerc926e172021-05-07 16:11:35 -05009741- It's also possible to use ``debuginfod-find`` to just query the server::
Andrew Geissler95ac1b82021-03-31 14:34:31 -05009742
Andrew Geisslerc926e172021-05-07 16:11:35 -05009743 root@qemux86-64:~# debuginfod-find debuginfo /bin/ls
9744 /home/root/.cache/debuginfod_client/356edc585f7f82d46f94fcb87a86a3fe2d2e60bd/debuginfo
Andrew Geissler95ac1b82021-03-31 14:34:31 -05009745
Andrew Geissler95ac1b82021-03-31 14:34:31 -05009746
9747Using the gdbserver method
9748~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
9749
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009750gdbserver, which runs on the remote target and does not load any
9751debugging information from the debugged process. Instead, a GDB instance
9752processes the debugging information that is run on a remote computer -
9753the host GDB. The host GDB then sends control commands to gdbserver to
9754make it stop or start the debugged program, as well as read or write
9755memory regions of that debugged program. All the debugging information
9756loaded and processed as well as all the heavy debugging is done by the
9757host GDB. Offloading these processes gives the gdbserver running on the
9758target a chance to remain small and fast.
9759
9760Because the host GDB is responsible for loading the debugging
9761information and for doing the necessary processing to make actual
9762debugging happen, you have to make sure the host can access the
9763unstripped binaries complete with their debugging information and also
9764be sure the target is compiled with no optimizations. The host GDB must
9765also have local access to all the libraries used by the debugged
9766program. Because gdbserver does not need any local debugging
9767information, the binaries on the remote target can remain stripped.
9768However, the binaries must also be compiled without optimization so they
9769match the host's binaries.
9770
9771To remain consistent with GDB documentation and terminology, the binary
9772being debugged on the remote target machine is referred to as the
9773"inferior" binary. For documentation on GDB see the `GDB
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009774site <https://sourceware.org/gdb/documentation/>`__.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009775
9776The following steps show you how to debug using the GNU project
9777debugger.
9778
97791. *Configure your build system to construct the companion debug
9780 filesystem:*
9781
Andrew Geisslerc926e172021-05-07 16:11:35 -05009782 In your ``local.conf`` file, set the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009783
9784 IMAGE_GEN_DEBUGFS = "1"
9785 IMAGE_FSTYPES_DEBUGFS = "tar.bz2"
9786
9787 These options cause the
9788 OpenEmbedded build system to generate a special companion filesystem
9789 fragment, which contains the matching source and debug symbols to
9790 your deployable filesystem. The build system does this by looking at
9791 what is in the deployed filesystem, and pulling the corresponding
9792 ``-dbg`` packages.
9793
9794 The companion debug filesystem is not a complete filesystem, but only
9795 contains the debug fragments. This filesystem must be combined with
9796 the full filesystem for debugging. Subsequent steps in this procedure
9797 show how to combine the partial filesystem with the full filesystem.
9798
97992. *Configure the system to include gdbserver in the target filesystem:*
9800
9801 Make the following addition in either your ``local.conf`` file or in
Andrew Geisslerc926e172021-05-07 16:11:35 -05009802 an image recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009803
9804 IMAGE_INSTALL_append = " gdbserver"
9805
9806 The change makes
9807 sure the ``gdbserver`` package is included.
9808
98093. *Build the environment:*
9810
9811 Use the following command to construct the image and the companion
Andrew Geisslerc926e172021-05-07 16:11:35 -05009812 Debug Filesystem::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009813
9814 $ bitbake image
9815
9816 Build the cross GDB component and
9817 make it available for debugging. Build the SDK that matches the
9818 image. Building the SDK is best for a production build that can be
Andrew Geisslerc926e172021-05-07 16:11:35 -05009819 used later for debugging, especially during long term maintenance::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009820
9821 $ bitbake -c populate_sdk image
9822
9823 Alternatively, you can build the minimal toolchain components that
9824 match the target. Doing so creates a smaller than typical SDK and
9825 only contains a minimal set of components with which to build simple
Andrew Geisslerc926e172021-05-07 16:11:35 -05009826 test applications, as well as run the debugger::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009827
9828 $ bitbake meta-toolchain
9829
Andrew Geisslerc926e172021-05-07 16:11:35 -05009830 A final method is to build Gdb itself within the build system::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009831
9832 $ bitbake gdb-cross-<architecture>
9833
9834 Doing so produces a temporary copy of
9835 ``cross-gdb`` you can use for debugging during development. While
9836 this is the quickest approach, the two previous methods in this step
9837 are better when considering long-term maintenance strategies.
9838
9839 .. note::
9840
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009841 If you run ``bitbake gdb-cross``, the OpenEmbedded build system suggests
9842 the actual image (e.g. ``gdb-cross-i586``). The suggestion is usually the
9843 actual name you want to use.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009844
Andrew Geissler4c19ea12020-10-27 13:52:24 -050098454. *Set up the* ``debugfs``\ *:*
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009846
Andrew Geisslerc926e172021-05-07 16:11:35 -05009847 Run the following commands to set up the ``debugfs``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009848
9849 $ mkdir debugfs
9850 $ cd debugfs
9851 $ tar xvfj build-dir/tmp-glibc/deploy/images/machine/image.rootfs.tar.bz2
9852 $ tar xvfj build-dir/tmp-glibc/deploy/images/machine/image-dbg.rootfs.tar.bz2
9853
Andrew Geissler4c19ea12020-10-27 13:52:24 -050098545. *Set up GDB:*
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009855
9856 Install the SDK (if you built one) and then source the correct
9857 environment file. Sourcing the environment file puts the SDK in your
9858 ``PATH`` environment variable.
9859
9860 If you are using the build system, Gdb is located in
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009861 `build-dir`\ ``/tmp/sysroots/``\ `host`\ ``/usr/bin/``\ `architecture`\ ``/``\ `architecture`\ ``-gdb``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009862
98636. *Boot the target:*
9864
9865 For information on how to run QEMU, see the `QEMU
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009866 Documentation <https://wiki.qemu.org/Documentation/GettingStartedDevelopers>`__.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009867
9868 .. note::
9869
9870 Be sure to verify that your host can access the target via TCP.
9871
98727. *Debug a program:*
9873
9874 Debugging a program involves running gdbserver on the target and then
9875 running Gdb on the host. The example in this step debugs ``gzip``:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009876
9877 .. code-block:: shell
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009878
9879 root@qemux86:~# gdbserver localhost:1234 /bin/gzip —help
9880
9881 For
9882 additional gdbserver options, see the `GDB Server
9883 Documentation <https://www.gnu.org/software/gdb/documentation/>`__.
9884
9885 After running gdbserver on the target, you need to run Gdb on the
Andrew Geisslerc926e172021-05-07 16:11:35 -05009886 host and configure it and connect to the target. Use these commands::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009887
9888 $ cd directory-holding-the-debugfs-directory
9889 $ arch-gdb
9890 (gdb) set sysroot debugfs
9891 (gdb) set substitute-path /usr/src/debug debugfs/usr/src/debug
9892 (gdb) target remote IP-of-target:1234
9893
9894 At this
9895 point, everything should automatically load (i.e. matching binaries,
9896 symbols and headers).
9897
9898 .. note::
9899
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009900 The Gdb ``set`` commands in the previous example can be placed into the
9901 users ``~/.gdbinit`` file. Upon starting, Gdb automatically runs whatever
9902 commands are in that file.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009903
99048. *Deploying without a full image rebuild:*
9905
9906 In many cases, during development you want a quick method to deploy a
9907 new binary to the target and debug it, without waiting for a full
9908 image build.
9909
9910 One approach to solving this situation is to just build the component
9911 you want to debug. Once you have built the component, copy the
9912 executable directly to both the target and the host ``debugfs``.
9913
9914 If the binary is processed through the debug splitting in
9915 OpenEmbedded, you should also copy the debug items (i.e. ``.debug``
9916 contents and corresponding ``/usr/src/debug`` files) from the work
Andrew Geisslerc926e172021-05-07 16:11:35 -05009917 directory. Here is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009918
9919 $ bitbake bash
9920 $ bitbake -c devshell bash
9921 $ cd ..
9922 $ scp packages-split/bash/bin/bash target:/bin/bash
9923 $ cp -a packages-split/bash-dbg/\* path/debugfs
9924
9925Debugging with the GNU Project Debugger (GDB) on the Target
9926-----------------------------------------------------------
9927
9928The previous section addressed using GDB remotely for debugging
9929purposes, which is the most usual case due to the inherent hardware
9930limitations on many embedded devices. However, debugging in the target
9931hardware itself is also possible with more powerful devices. This
9932section describes what you need to do in order to support using GDB to
9933debug on the target hardware.
9934
9935To support this kind of debugging, you need do the following:
9936
9937- Ensure that GDB is on the target. You can do this by adding "gdb" to
Andrew Geisslerc926e172021-05-07 16:11:35 -05009938 :term:`IMAGE_INSTALL`::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009939
9940 IMAGE_INSTALL_append = " gdb"
9941
Andrew Geisslerc926e172021-05-07 16:11:35 -05009942 Alternatively, you can add "tools-debug" to :term:`IMAGE_FEATURES`::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009943
9944 IMAGE_FEATURES_append = " tools-debug"
9945
9946- Ensure that debug symbols are present. You can make sure these
Andrew Geisslerc926e172021-05-07 16:11:35 -05009947 symbols are present by installing ``-dbg``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009948
9949 IMAGE_INSTALL_append = "packagename-dbg"
9950
9951 Alternatively, you can do the following to include
Andrew Geisslerc926e172021-05-07 16:11:35 -05009952 all the debug symbols::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009953
9954 IMAGE_FEATURES_append = " dbg-pkgs"
9955
9956.. note::
9957
9958 To improve the debug information accuracy, you can reduce the level
9959 of optimization used by the compiler. For example, when adding the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009960 following line to your ``local.conf`` file, you will reduce optimization
9961 from :term:`FULL_OPTIMIZATION` of "-O2" to :term:`DEBUG_OPTIMIZATION`
Andrew Geisslerc926e172021-05-07 16:11:35 -05009962 of "-O -fno-omit-frame-pointer"::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009963
9964 DEBUG_BUILD = "1"
9965
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009966 Consider that this will reduce the application's performance and is
9967 recommended only for debugging purposes.
9968
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009969Other Debugging Tips
9970--------------------
9971
9972Here are some other tips that you might find useful:
9973
9974- When adding new packages, it is worth watching for undesirable items
9975 making their way into compiler command lines. For example, you do not
9976 want references to local system files like ``/usr/lib/`` or
9977 ``/usr/include/``.
9978
9979- If you want to remove the ``psplash`` boot splashscreen, add
9980 ``psplash=false`` to the kernel command line. Doing so prevents
9981 ``psplash`` from loading and thus allows you to see the console. It
9982 is also possible to switch out of the splashscreen by switching the
9983 virtual console (e.g. Fn+Left or Fn+Right on a Zaurus).
9984
9985- Removing :term:`TMPDIR` (usually
9986 ``tmp/``, within the
9987 :term:`Build Directory`) can often fix
Andrew Geissler09036742021-06-25 14:25:14 -05009988 temporary build issues. Removing :term:`TMPDIR` is usually a relatively
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009989 cheap operation, because task output will be cached in
9990 :term:`SSTATE_DIR` (usually
9991 ``sstate-cache/``, which is also in the Build Directory).
9992
9993 .. note::
9994
Andrew Geissler09036742021-06-25 14:25:14 -05009995 Removing :term:`TMPDIR` might be a workaround rather than a fix.
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009996 Consequently, trying to determine the underlying cause of an issue before
9997 removing the directory is a good idea.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009998
9999- Understanding how a feature is used in practice within existing
10000 recipes can be very helpful. It is recommended that you configure
10001 some method that allows you to quickly search through files.
10002
10003 Using GNU Grep, you can use the following shell function to
10004 recursively search through common recipe-related files, skipping
10005 binary files, ``.git`` directories, and the Build Directory (assuming
Andrew Geisslerc926e172021-05-07 16:11:35 -050010006 its name starts with "build")::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010007
10008 g() {
10009 grep -Ir \
10010 --exclude-dir=.git \
10011 --exclude-dir='build*' \
10012 --include='*.bb*' \
10013 --include='*.inc*' \
10014 --include='*.conf*' \
10015 --include='*.py*' \
10016 "$@"
10017 }
10018
Andrew Geisslerc926e172021-05-07 16:11:35 -050010019 Following are some usage examples::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010020
10021 $ g FOO # Search recursively for "FOO"
10022 $ g -i foo # Search recursively for "foo", ignoring case
10023 $ g -w FOO # Search recursively for "FOO" as a word, ignoring e.g. "FOOBAR"
10024
10025 If figuring
10026 out how some feature works requires a lot of searching, it might
10027 indicate that the documentation should be extended or improved. In
10028 such cases, consider filing a documentation bug using the Yocto
10029 Project implementation of
10030 :yocto_bugs:`Bugzilla <>`. For information on
10031 how to submit a bug against the Yocto Project, see the Yocto Project
Andrew Geissler09209ee2020-12-13 08:44:15 -060010032 Bugzilla :yocto_wiki:`wiki page </Bugzilla_Configuration_and_Bug_Tracking>`
Andrew Geissler3b8a17c2021-04-15 15:55:55 -050010033 and the
10034 ":ref:`dev-manual/common-tasks:submitting a defect against the yocto project`"
10035 section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010036
10037 .. note::
10038
10039 The manuals might not be the right place to document variables
10040 that are purely internal and have a limited scope (e.g. internal
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010041 variables used to implement a single ``.bbclass`` file).
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010042
10043Making Changes to the Yocto Project
10044===================================
10045
10046Because the Yocto Project is an open-source, community-based project,
10047you can effect changes to the project. This section presents procedures
10048that show you how to submit a defect against the project and how to
10049submit a change.
10050
10051Submitting a Defect Against the Yocto Project
10052---------------------------------------------
10053
10054Use the Yocto Project implementation of
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010055`Bugzilla <https://www.bugzilla.org/about/>`__ to submit a defect (bug)
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010056against the Yocto Project. For additional information on this
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010057implementation of Bugzilla see the ":ref:`Yocto Project
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010058Bugzilla <resources-bugtracker>`" section in the
10059Yocto Project Reference Manual. For more detail on any of the following
10060steps, see the Yocto Project
Andrew Geissler09209ee2020-12-13 08:44:15 -060010061:yocto_wiki:`Bugzilla wiki page </Bugzilla_Configuration_and_Bug_Tracking>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010062
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010063Use the following general steps to submit a bug:
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010064
100651. Open the Yocto Project implementation of :yocto_bugs:`Bugzilla <>`.
10066
100672. Click "File a Bug" to enter a new bug.
10068
100693. Choose the appropriate "Classification", "Product", and "Component"
10070 for which the bug was found. Bugs for the Yocto Project fall into
10071 one of several classifications, which in turn break down into
10072 several products and components. For example, for a bug against the
10073 ``meta-intel`` layer, you would choose "Build System, Metadata &
10074 Runtime", "BSPs", and "bsps-meta-intel", respectively.
10075
100764. Choose the "Version" of the Yocto Project for which you found the
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010077 bug (e.g. &DISTRO;).
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010078
100795. Determine and select the "Severity" of the bug. The severity
10080 indicates how the bug impacted your work.
10081
100826. Choose the "Hardware" that the bug impacts.
10083
100847. Choose the "Architecture" that the bug impacts.
10085
100868. Choose a "Documentation change" item for the bug. Fixing a bug might
10087 or might not affect the Yocto Project documentation. If you are
10088 unsure of the impact to the documentation, select "Don't Know".
10089
100909. Provide a brief "Summary" of the bug. Try to limit your summary to
10091 just a line or two and be sure to capture the essence of the bug.
10092
1009310. Provide a detailed "Description" of the bug. You should provide as
10094 much detail as you can about the context, behavior, output, and so
10095 forth that surrounds the bug. You can even attach supporting files
10096 for output from logs by using the "Add an attachment" button.
10097
1009811. Click the "Submit Bug" button submit the bug. A new Bugzilla number
10099 is assigned to the bug and the defect is logged in the bug tracking
10100 system.
10101
10102Once you file a bug, the bug is processed by the Yocto Project Bug
10103Triage Team and further details concerning the bug are assigned (e.g.
10104priority and owner). You are the "Submitter" of the bug and any further
10105categorization, progress, or comments on the bug result in Bugzilla
10106sending you an automated email concerning the particular change or
10107progress to the bug.
10108
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010109Submitting a Change to the Yocto Project
10110----------------------------------------
10111
10112Contributions to the Yocto Project and OpenEmbedded are very welcome.
10113Because the system is extremely configurable and flexible, we recognize
10114that developers will want to extend, configure or optimize it for their
10115specific uses.
10116
10117The Yocto Project uses a mailing list and a patch-based workflow that is
10118similar to the Linux kernel but contains important differences. In
William A. Kennington IIIac69b482021-06-02 12:28:27 -070010119general, there is a mailing list through which you can submit patches. You
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010120should send patches to the appropriate mailing list so that they can be
10121reviewed and merged by the appropriate maintainer. The specific mailing
10122list you need to use depends on the location of the code you are
10123changing. Each component (e.g. layer) should have a ``README`` file that
10124indicates where to send the changes and which process to follow.
10125
10126You can send the patch to the mailing list using whichever approach you
10127feel comfortable with to generate the patch. Once sent, the patch is
10128usually reviewed by the community at large. If somebody has concerns
10129with the patch, they will usually voice their concern over the mailing
10130list. If a patch does not receive any negative reviews, the maintainer
10131of the affected layer typically takes the patch, tests it, and then
10132based on successful testing, merges the patch.
10133
10134The "poky" repository, which is the Yocto Project's reference build
10135environment, is a hybrid repository that contains several individual
10136pieces (e.g. BitBake, Metadata, documentation, and so forth) built using
10137the combo-layer tool. The upstream location used for submitting changes
10138varies by component:
10139
10140- *Core Metadata:* Send your patch to the
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010141 :oe_lists:`openembedded-core </g/openembedded-core>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010142 mailing list. For example, a change to anything under the ``meta`` or
10143 ``scripts`` directories should be sent to this mailing list.
10144
10145- *BitBake:* For changes to BitBake (i.e. anything under the
10146 ``bitbake`` directory), send your patch to the
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010147 :oe_lists:`bitbake-devel </g/bitbake-devel>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010148 mailing list.
10149
Andrew Geisslerc3d88e42020-10-02 09:45:00 -050010150- *"meta-\*" trees:* These trees contain Metadata. Use the
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010151 :yocto_lists:`poky </g/poky>` mailing list.
Andrew Geisslerc3d88e42020-10-02 09:45:00 -050010152
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010153- *Documentation*: For changes to the Yocto Project documentation, use the
10154 :yocto_lists:`docs </g/docs>` mailing list.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010155
10156For changes to other layers hosted in the Yocto Project source
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010157repositories (i.e. ``yoctoproject.org``) and tools use the
10158:yocto_lists:`Yocto Project </g/yocto/>` general mailing list.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010159
10160.. note::
10161
10162 Sometimes a layer's documentation specifies to use a particular
10163 mailing list. If so, use that list.
10164
10165For additional recipes that do not fit into the core Metadata, you
10166should determine which layer the recipe should go into and submit the
10167change in the manner recommended by the documentation (e.g. the
10168``README`` file) supplied with the layer. If in doubt, please ask on the
10169Yocto general mailing list or on the openembedded-devel mailing list.
10170
10171You can also push a change upstream and request a maintainer to pull the
10172change into the component's upstream repository. You do this by pushing
Andrew Geissler09209ee2020-12-13 08:44:15 -060010173to a contribution repository that is upstream. See the
10174":ref:`overview-manual/development-environment:git workflows and the yocto project`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010175section in the Yocto Project Overview and Concepts Manual for additional
10176concepts on working in the Yocto Project development environment.
10177
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010178Maintainers commonly use ``-next`` branches to test submissions prior to
10179merging patches. Thus, you can get an idea of the status of a patch based on
10180whether the patch has been merged into one of these branches. The commonly
10181used testing branches for OpenEmbedded-Core are as follows:
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010182
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010183- *openembedded-core "master-next" branch:* This branch is part of the
10184 :oe_git:`openembedded-core </openembedded-core/>` repository and contains
10185 proposed changes to the core metadata.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010186
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010187- *poky "master-next" branch:* This branch is part of the
Andrew Geissler09209ee2020-12-13 08:44:15 -060010188 :yocto_git:`poky </poky/>` repository and combines proposed
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010189 changes to bitbake, the core metadata and the poky distro.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010190
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010191Similarly, stable branches maintained by the project may have corresponding
10192``-next`` branches which collect proposed changes. For example,
10193``&DISTRO_NAME_NO_CAP;-next`` and ``&DISTRO_NAME_NO_CAP_MINUS_ONE;-next``
10194branches in both the "openembdedded-core" and "poky" repositories.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010195
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010196Other layers may have similar testing branches but there is no formal
10197requirement or standard for these so please check the documentation for the
10198layers you are contributing to.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010199
10200The following sections provide procedures for submitting a change.
10201
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010202Preparing Changes for Submission
10203~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010204
102051. *Make Your Changes Locally:* Make your changes in your local Git
10206 repository. You should make small, controlled, isolated changes.
10207 Keeping changes small and isolated aids review, makes
10208 merging/rebasing easier and keeps the change history clean should
10209 anyone need to refer to it in future.
10210
102112. *Stage Your Changes:* Stage your changes by using the ``git add``
10212 command on each file you changed.
10213
102143. *Commit Your Changes:* Commit the change by using the ``git commit``
10215 command. Make sure your commit information follows standards by
10216 following these accepted conventions:
10217
10218 - Be sure to include a "Signed-off-by:" line in the same style as
10219 required by the Linux kernel. Adding this line signifies that you,
10220 the submitter, have agreed to the Developer's Certificate of
10221 Origin 1.1 as follows:
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010222
10223 .. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010224
10225 Developer's Certificate of Origin 1.1
10226
10227 By making a contribution to this project, I certify that:
10228
10229 (a) The contribution was created in whole or in part by me and I
10230 have the right to submit it under the open source license
10231 indicated in the file; or
10232
10233 (b) The contribution is based upon previous work that, to the best
10234 of my knowledge, is covered under an appropriate open source
10235 license and I have the right under that license to submit that
10236 work with modifications, whether created in whole or in part
10237 by me, under the same open source license (unless I am
10238 permitted to submit under a different license), as indicated
10239 in the file; or
10240
10241 (c) The contribution was provided directly to me by some other
10242 person who certified (a), (b) or (c) and I have not modified
10243 it.
10244
10245 (d) I understand and agree that this project and the contribution
10246 are public and that a record of the contribution (including all
10247 personal information I submit with it, including my sign-off) is
10248 maintained indefinitely and may be redistributed consistent with
10249 this project or the open source license(s) involved.
10250
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010251 - Provide a single-line summary of the change and, if more
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010252 explanation is needed, provide more detail in the body of the
10253 commit. This summary is typically viewable in the "shortlist" of
10254 changes. Thus, providing something short and descriptive that
10255 gives the reader a summary of the change is useful when viewing a
10256 list of many commits. You should prefix this short description
10257 with the recipe name (if changing a recipe), or else with the
10258 short form path to the file being changed.
10259
10260 - For the body of the commit message, provide detailed information
10261 that describes what you changed, why you made the change, and the
10262 approach you used. It might also be helpful if you mention how you
10263 tested the change. Provide as much detail as you can in the body
10264 of the commit message.
10265
10266 .. note::
10267
10268 You do not need to provide a more detailed explanation of a
10269 change if the change is minor to the point of the single line
10270 summary providing all the information.
10271
10272 - If the change addresses a specific bug or issue that is associated
10273 with a bug-tracking ID, include a reference to that ID in your
10274 detailed description. For example, the Yocto Project uses a
10275 specific convention for bug references - any commit that addresses
10276 a specific bug should use the following form for the detailed
10277 description. Be sure to use the actual bug-tracking ID from
Andrew Geisslerc926e172021-05-07 16:11:35 -050010278 Bugzilla for bug-id::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010279
10280 Fixes [YOCTO #bug-id]
10281
10282 detailed description of change
10283
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010284Using Email to Submit a Patch
10285~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10286
10287Depending on the components changed, you need to submit the email to a
10288specific mailing list. For some guidance on which mailing list to use,
Andrew Geissler3b8a17c2021-04-15 15:55:55 -050010289see the
10290:ref:`list <dev-manual/common-tasks:submitting a change to the yocto project>`
10291at the beginning of this section. For a description of all the available
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010292mailing lists, see the ":ref:`Mailing Lists <resources-mailinglist>`" section in the
10293Yocto Project Reference Manual.
10294
10295Here is the general procedure on how to submit a patch through email
10296without using the scripts once the steps in
Andrew Geissler09209ee2020-12-13 08:44:15 -060010297:ref:`dev-manual/common-tasks:preparing changes for submission` have been followed:
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010298
102991. *Format the Commit:* Format the commit into an email message. To
10300 format commits, use the ``git format-patch`` command. When you
10301 provide the command, you must include a revision list or a number of
10302 patches as part of the command. For example, either of these two
10303 commands takes your most recent single commit and formats it as an
Andrew Geisslerc926e172021-05-07 16:11:35 -050010304 email message in the current directory::
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010305
10306 $ git format-patch -1
10307
10308 or ::
10309
10310 $ git format-patch HEAD~
10311
10312 After the command is run, the current directory contains a numbered
10313 ``.patch`` file for the commit.
10314
10315 If you provide several commits as part of the command, the
10316 ``git format-patch`` command produces a series of numbered files in
10317 the current directory – one for each commit. If you have more than
10318 one patch, you should also use the ``--cover`` option with the
10319 command, which generates a cover letter as the first "patch" in the
10320 series. You can then edit the cover letter to provide a description
10321 for the series of patches. For information on the
10322 ``git format-patch`` command, see ``GIT_FORMAT_PATCH(1)`` displayed
10323 using the ``man git-format-patch`` command.
10324
10325 .. note::
10326
10327 If you are or will be a frequent contributor to the Yocto Project
10328 or to OpenEmbedded, you might consider requesting a contrib area
10329 and the necessary associated rights.
10330
103312. *Send the patches via email:* Send the patches to the recipients and
10332 relevant mailing lists by using the ``git send-email`` command.
10333
10334 .. note::
10335
10336 In order to use ``git send-email``, you must have the proper Git packages
10337 installed on your host.
10338 For Ubuntu, Debian, and Fedora the package is ``git-email``.
10339
10340 The ``git send-email`` command sends email by using a local or remote
10341 Mail Transport Agent (MTA) such as ``msmtp``, ``sendmail``, or
10342 through a direct ``smtp`` configuration in your Git ``~/.gitconfig``
10343 file. If you are submitting patches through email only, it is very
10344 important that you submit them without any whitespace or HTML
10345 formatting that either you or your mailer introduces. The maintainer
10346 that receives your patches needs to be able to save and apply them
10347 directly from your emails. A good way to verify that what you are
10348 sending will be applicable by the maintainer is to do a dry run and
10349 send them to yourself and then save and apply them as the maintainer
10350 would.
10351
10352 The ``git send-email`` command is the preferred method for sending
10353 your patches using email since there is no risk of compromising
10354 whitespace in the body of the message, which can occur when you use
10355 your own mail client. The command also has several options that let
10356 you specify recipients and perform further editing of the email
10357 message. For information on how to use the ``git send-email``
10358 command, see ``GIT-SEND-EMAIL(1)`` displayed using the
10359 ``man git-send-email`` command.
10360
10361The Yocto Project uses a `Patchwork instance <https://patchwork.openembedded.org/>`__
10362to track the status of patches submitted to the various mailing lists and to
10363support automated patch testing. Each submitted patch is checked for common
10364mistakes and deviations from the expected patch format and submitters are
10365notified by patchtest if such mistakes are found. This process helps to
10366reduce the burden of patch review on maintainers.
10367
10368.. note::
10369
10370 This system is imperfect and changes can sometimes get lost in the flow.
10371 Asking about the status of a patch or change is reasonable if the change
10372 has been idle for a while with no feedback.
10373
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010374Using Scripts to Push a Change Upstream and Request a Pull
10375~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10376
10377For larger patch series it is preferable to send a pull request which not
10378only includes the patch but also a pointer to a branch that can be pulled
10379from. This involves making a local branch for your changes, pushing this
10380branch to an accessible repository and then using the ``create-pull-request``
10381and ``send-pull-request`` scripts from openembedded-core to create and send a
10382patch series with a link to the branch for review.
10383
10384Follow this procedure to push a change to an upstream "contrib" Git
Andrew Geissler09209ee2020-12-13 08:44:15 -060010385repository once the steps in :ref:`dev-manual/common-tasks:preparing changes for submission` have
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010386been followed:
10387
10388.. note::
10389
10390 You can find general Git information on how to push a change upstream
10391 in the
10392 `Git Community Book <https://git-scm.com/book/en/v2/Distributed-Git-Distributed-Workflows>`__.
10393
103941. *Push Your Commits to a "Contrib" Upstream:* If you have arranged for
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010395 permissions to push to an upstream contrib repository, push the
Andrew Geisslerc926e172021-05-07 16:11:35 -050010396 change to that repository::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010397
10398 $ git push upstream_remote_repo local_branch_name
10399
10400 For example, suppose you have permissions to push
10401 into the upstream ``meta-intel-contrib`` repository and you are
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010402 working in a local branch named `your_name`\ ``/README``. The following
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010403 command pushes your local commits to the ``meta-intel-contrib``
10404 upstream repository and puts the commit in a branch named
Andrew Geisslerc926e172021-05-07 16:11:35 -050010405 `your_name`\ ``/README``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010406
10407 $ git push meta-intel-contrib your_name/README
10408
Andrew Geissler6ce62a22020-11-30 19:58:47 -0600104092. *Determine Who to Notify:* Determine the maintainer or the mailing
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010410 list that you need to notify for the change.
10411
10412 Before submitting any change, you need to be sure who the maintainer
10413 is or what mailing list that you need to notify. Use either these
10414 methods to find out:
10415
10416 - *Maintenance File:* Examine the ``maintainers.inc`` file, which is
10417 located in the :term:`Source Directory` at
10418 ``meta/conf/distro/include``, to see who is responsible for code.
10419
Andrew Geissler09209ee2020-12-13 08:44:15 -060010420 - *Search by File:* Using :ref:`overview-manual/development-environment:git`, you can
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010421 enter the following command to bring up a short list of all
Andrew Geisslerc926e172021-05-07 16:11:35 -050010422 commits against a specific file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010423
10424 git shortlog -- filename
10425
10426 Just provide the name of the file for which you are interested. The
10427 information returned is not ordered by history but does include a
10428 list of everyone who has committed grouped by name. From the list,
10429 you can see who is responsible for the bulk of the changes against
10430 the file.
10431
10432 - *Examine the List of Mailing Lists:* For a list of the Yocto
10433 Project and related mailing lists, see the ":ref:`Mailing
10434 lists <resources-mailinglist>`" section in
10435 the Yocto Project Reference Manual.
10436
Andrew Geissler6ce62a22020-11-30 19:58:47 -0600104373. *Make a Pull Request:* Notify the maintainer or the mailing list that
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010438 you have pushed a change by making a pull request.
10439
10440 The Yocto Project provides two scripts that conveniently let you
10441 generate and send pull requests to the Yocto Project. These scripts
10442 are ``create-pull-request`` and ``send-pull-request``. You can find
10443 these scripts in the ``scripts`` directory within the
10444 :term:`Source Directory` (e.g.
Andrew Geissler95ac1b82021-03-31 14:34:31 -050010445 ``poky/scripts``).
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010446
10447 Using these scripts correctly formats the requests without
10448 introducing any whitespace or HTML formatting. The maintainer that
10449 receives your patches either directly or through the mailing list
10450 needs to be able to save and apply them directly from your emails.
10451 Using these scripts is the preferred method for sending patches.
10452
10453 First, create the pull request. For example, the following command
10454 runs the script, specifies the upstream repository in the contrib
10455 directory into which you pushed the change, and provides a subject
Andrew Geisslerc926e172021-05-07 16:11:35 -050010456 line in the created patch files::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010457
Andrew Geissler95ac1b82021-03-31 14:34:31 -050010458 $ poky/scripts/create-pull-request -u meta-intel-contrib -s "Updated Manual Section Reference in README"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010459
10460 Running this script forms ``*.patch`` files in a folder named
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010461 ``pull-``\ `PID` in the current directory. One of the patch files is a
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010462 cover letter.
10463
10464 Before running the ``send-pull-request`` script, you must edit the
10465 cover letter patch to insert information about your change. After
10466 editing the cover letter, send the pull request. For example, the
10467 following command runs the script and specifies the patch directory
10468 and email address. In this example, the email address is a mailing
Andrew Geisslerc926e172021-05-07 16:11:35 -050010469 list::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010470
Andrew Geissler95ac1b82021-03-31 14:34:31 -050010471 $ poky/scripts/send-pull-request -p ~/meta-intel/pull-10565 -t meta-intel@yoctoproject.org
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010472
10473 You need to follow the prompts as the script is interactive.
10474
10475 .. note::
10476
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010477 For help on using these scripts, simply provide the ``-h``
Andrew Geisslerc926e172021-05-07 16:11:35 -050010478 argument as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010479
10480 $ poky/scripts/create-pull-request -h
10481 $ poky/scripts/send-pull-request -h
10482
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010483Responding to Patch Review
10484~~~~~~~~~~~~~~~~~~~~~~~~~~
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010485
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010486You may get feedback on your submitted patches from other community members
10487or from the automated patchtest service. If issues are identified in your
10488patch then it is usually necessary to address these before the patch will be
10489accepted into the project. In this case you should amend the patch according
10490to the feedback and submit an updated version to the relevant mailing list,
10491copying in the reviewers who provided feedback to the previous version of the
10492patch.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010493
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010494The patch should be amended using ``git commit --amend`` or perhaps ``git
10495rebase`` for more expert git users. You should also modify the ``[PATCH]``
10496tag in the email subject line when sending the revised patch to mark the new
10497iteration as ``[PATCH v2]``, ``[PATCH v3]``, etc as appropriate. This can be
10498done by passing the ``-v`` argument to ``git format-patch`` with a version
10499number.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010500
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010501Lastly please ensure that you also test your revised changes. In particular
10502please don't just edit the patch file written out by ``git format-patch`` and
10503resend it.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010504
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010505Submitting Changes to Stable Release Branches
10506~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010507
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010508The process for proposing changes to a Yocto Project stable branch differs
10509from the steps described above. Changes to a stable branch must address
10510identified bugs or CVEs and should be made carefully in order to avoid the
10511risk of introducing new bugs or breaking backwards compatibility. Typically
10512bug fixes must already be accepted into the master branch before they can be
10513backported to a stable branch unless the bug in question does not affect the
10514master branch or the fix on the master branch is unsuitable for backporting.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010515
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010516The list of stable branches along with the status and maintainer for each
10517branch can be obtained from the
Andrew Geissler09209ee2020-12-13 08:44:15 -060010518:yocto_wiki:`Releases wiki page </Releases>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010519
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010520.. note::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010521
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010522 Changes will not typically be accepted for branches which are marked as
10523 End-Of-Life (EOL).
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010524
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010525With this in mind, the steps to submit a change for a stable branch are as
10526follows:
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010527
Andrew Geissler6ce62a22020-11-30 19:58:47 -0600105281. *Identify the bug or CVE to be fixed:* This information should be
10529 collected so that it can be included in your submission.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010530
Andrew Geissler6ce62a22020-11-30 19:58:47 -0600105312. *Check if the fix is already present in the master branch:* This will
10532 result in the most straightforward path into the stable branch for the
10533 fix.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010534
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010535 a. *If the fix is present in the master branch - Submit a backport request
10536 by email:* You should send an email to the relevant stable branch
10537 maintainer and the mailing list with details of the bug or CVE to be
10538 fixed, the commit hash on the master branch that fixes the issue and
10539 the stable branches which you would like this fix to be backported to.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010540
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010541 b. *If the fix is not present in the master branch - Submit the fix to the
10542 master branch first:* This will ensure that the fix passes through the
10543 project's usual patch review and test processes before being accepted.
10544 It will also ensure that bugs are not left unresolved in the master
10545 branch itself. Once the fix is accepted in the master branch a backport
10546 request can be submitted as above.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010547
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010548 c. *If the fix is unsuitable for the master branch - Submit a patch
10549 directly for the stable branch:* This method should be considered as a
10550 last resort. It is typically necessary when the master branch is using
10551 a newer version of the software which includes an upstream fix for the
10552 issue or when the issue has been fixed on the master branch in a way
10553 that introduces backwards incompatible changes. In this case follow the
Andrew Geissler09209ee2020-12-13 08:44:15 -060010554 steps in :ref:`dev-manual/common-tasks:preparing changes for submission` and
10555 :ref:`dev-manual/common-tasks:using email to submit a patch` but modify the subject header of your patch
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010556 email to include the name of the stable branch which you are
10557 targetting. This can be done using the ``--subject-prefix`` argument to
10558 ``git format-patch``, for example to submit a patch to the dunfell
10559 branch use
10560 ``git format-patch --subject-prefix='&DISTRO_NAME_NO_CAP_MINUS_ONE;][PATCH' ...``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010561
10562Working With Licenses
10563=====================
10564
Andrew Geissler09209ee2020-12-13 08:44:15 -060010565As mentioned in the ":ref:`overview-manual/development-environment:licensing`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010566section in the Yocto Project Overview and Concepts Manual, open source
10567projects are open to the public and they consequently have different
10568licensing structures in place. This section describes the mechanism by
10569which the :term:`OpenEmbedded Build System`
10570tracks changes to
10571licensing text and covers how to maintain open source license compliance
10572during your project's lifecycle. The section also describes how to
10573enable commercially licensed recipes, which by default are disabled.
10574
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010575Tracking License Changes
10576------------------------
10577
10578The license of an upstream project might change in the future. In order
10579to prevent these changes going unnoticed, the
10580:term:`LIC_FILES_CHKSUM`
10581variable tracks changes to the license text. The checksums are validated
10582at the end of the configure step, and if the checksums do not match, the
10583build will fail.
10584
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010585Specifying the ``LIC_FILES_CHKSUM`` Variable
10586~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10587
Andrew Geissler09036742021-06-25 14:25:14 -050010588The :term:`LIC_FILES_CHKSUM` variable contains checksums of the license text
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010589in the source code for the recipe. Following is an example of how to
Andrew Geissler09036742021-06-25 14:25:14 -050010590specify :term:`LIC_FILES_CHKSUM`::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010591
10592 LIC_FILES_CHKSUM = "file://COPYING;md5=xxxx \
10593 file://licfile1.txt;beginline=5;endline=29;md5=yyyy \
10594 file://licfile2.txt;endline=50;md5=zzzz \
10595 ..."
10596
10597.. note::
10598
10599 - When using "beginline" and "endline", realize that line numbering
10600 begins with one and not zero. Also, the included lines are
10601 inclusive (i.e. lines five through and including 29 in the
10602 previous example for ``licfile1.txt``).
10603
10604 - When a license check fails, the selected license text is included
10605 as part of the QA message. Using this output, you can determine
10606 the exact start and finish for the needed license text.
10607
10608The build system uses the :term:`S`
10609variable as the default directory when searching files listed in
Andrew Geissler09036742021-06-25 14:25:14 -050010610:term:`LIC_FILES_CHKSUM`. The previous example employs the default
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010611directory.
10612
Andrew Geisslerc926e172021-05-07 16:11:35 -050010613Consider this next example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010614
10615 LIC_FILES_CHKSUM = "file://src/ls.c;beginline=5;endline=16;\
10616 md5=bb14ed3c4cda583abc85401304b5cd4e"
10617 LIC_FILES_CHKSUM = "file://${WORKDIR}/license.html;md5=5c94767cedb5d6987c902ac850ded2c6"
10618
10619The first line locates a file in ``${S}/src/ls.c`` and isolates lines
10620five through 16 as license text. The second line refers to a file in
10621:term:`WORKDIR`.
10622
Andrew Geissler09036742021-06-25 14:25:14 -050010623Note that :term:`LIC_FILES_CHKSUM` variable is mandatory for all recipes,
10624unless the :term:`LICENSE` variable is set to "CLOSED".
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010625
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010626Explanation of Syntax
10627~~~~~~~~~~~~~~~~~~~~~
10628
Andrew Geissler09036742021-06-25 14:25:14 -050010629As mentioned in the previous section, the :term:`LIC_FILES_CHKSUM` variable
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010630lists all the important files that contain the license text for the
10631source code. It is possible to specify a checksum for an entire file, or
10632a specific section of a file (specified by beginning and ending line
10633numbers with the "beginline" and "endline" parameters, respectively).
10634The latter is useful for source files with a license notice header,
10635README documents, and so forth. If you do not use the "beginline"
10636parameter, then it is assumed that the text begins on the first line of
10637the file. Similarly, if you do not use the "endline" parameter, it is
10638assumed that the license text ends with the last line of the file.
10639
10640The "md5" parameter stores the md5 checksum of the license text. If the
10641license text changes in any way as compared to this parameter then a
10642mismatch occurs. This mismatch triggers a build failure and notifies the
10643developer. Notification allows the developer to review and address the
10644license text changes. Also note that if a mismatch occurs during the
10645build, the correct md5 checksum is placed in the build log and can be
10646easily copied to the recipe.
10647
10648There is no limit to how many files you can specify using the
Andrew Geissler09036742021-06-25 14:25:14 -050010649:term:`LIC_FILES_CHKSUM` variable. Generally, however, every project
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010650requires a few specifications for license tracking. Many projects have a
10651"COPYING" file that stores the license information for all the source
10652code files. This practice allows you to just track the "COPYING" file as
10653long as it is kept up to date.
10654
10655.. note::
10656
10657 - If you specify an empty or invalid "md5" parameter,
10658 :term:`BitBake` returns an md5
10659 mis-match error and displays the correct "md5" parameter value
10660 during the build. The correct parameter is also captured in the
10661 build log.
10662
10663 - If the whole file contains only license text, you do not need to
10664 use the "beginline" and "endline" parameters.
10665
10666Enabling Commercially Licensed Recipes
10667--------------------------------------
10668
10669By default, the OpenEmbedded build system disables components that have
10670commercial or other special licensing requirements. Such requirements
10671are defined on a recipe-by-recipe basis through the
10672:term:`LICENSE_FLAGS` variable
10673definition in the affected recipe. For instance, the
10674``poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly`` recipe
Andrew Geisslerc926e172021-05-07 16:11:35 -050010675contains the following statement::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010676
10677 LICENSE_FLAGS = "commercial"
10678
10679Here is a
10680slightly more complicated example that contains both an explicit recipe
Andrew Geisslerc926e172021-05-07 16:11:35 -050010681name and version (after variable expansion)::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010682
10683 LICENSE_FLAGS = "license_${PN}_${PV}"
10684
10685In order for a component restricted by a
Andrew Geissler09036742021-06-25 14:25:14 -050010686:term:`LICENSE_FLAGS` definition to be enabled and included in an image, it
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010687needs to have a matching entry in the global
10688:term:`LICENSE_FLAGS_WHITELIST`
10689variable, which is a variable typically defined in your ``local.conf``
10690file. For example, to enable the
10691``poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly`` package, you
10692could add either the string "commercial_gst-plugins-ugly" or the more
Andrew Geissler09036742021-06-25 14:25:14 -050010693general string "commercial" to :term:`LICENSE_FLAGS_WHITELIST`. See the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -050010694":ref:`dev-manual/common-tasks:license flag matching`" section for a full
Andrew Geissler09036742021-06-25 14:25:14 -050010695explanation of how :term:`LICENSE_FLAGS` matching works. Here is the
Andrew Geisslerc926e172021-05-07 16:11:35 -050010696example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010697
10698 LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly"
10699
10700Likewise, to additionally enable the package built from the recipe
10701containing ``LICENSE_FLAGS = "license_${PN}_${PV}"``, and assuming that
10702the actual recipe name was ``emgd_1.10.bb``, the following string would
10703enable that package as well as the original ``gst-plugins-ugly``
Andrew Geisslerc926e172021-05-07 16:11:35 -050010704package::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010705
10706 LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly license_emgd_1.10"
10707
10708As a convenience, you do not need to specify the
10709complete license string in the whitelist for every package. You can use
10710an abbreviated form, which consists of just the first portion or
10711portions of the license string before the initial underscore character
10712or characters. A partial string will match any license that contains the
10713given string as the first portion of its license. For example, the
10714following whitelist string will also match both of the packages
10715previously mentioned as well as any other packages that have licenses
10716starting with "commercial" or "license".
10717::
10718
10719 LICENSE_FLAGS_WHITELIST = "commercial license"
10720
10721License Flag Matching
10722~~~~~~~~~~~~~~~~~~~~~
10723
10724License flag matching allows you to control what recipes the
10725OpenEmbedded build system includes in the build. Fundamentally, the
Andrew Geissler09036742021-06-25 14:25:14 -050010726build system attempts to match :term:`LICENSE_FLAGS` strings found in
10727recipes against :term:`LICENSE_FLAGS_WHITELIST` strings found in the
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010728whitelist. A match causes the build system to include a recipe in the
10729build, while failure to find a match causes the build system to exclude
10730a recipe.
10731
10732In general, license flag matching is simple. However, understanding some
10733concepts will help you correctly and effectively use matching.
10734
10735Before a flag defined by a particular recipe is tested against the
10736contents of the whitelist, the expanded string ``_${PN}`` is appended to
Andrew Geissler09036742021-06-25 14:25:14 -050010737the flag. This expansion makes each :term:`LICENSE_FLAGS` value
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010738recipe-specific. After expansion, the string is then matched against the
10739whitelist. Thus, specifying ``LICENSE_FLAGS = "commercial"`` in recipe
10740"foo", for example, results in the string ``"commercial_foo"``. And, to
10741create a match, that string must appear in the whitelist.
10742
Andrew Geissler09036742021-06-25 14:25:14 -050010743Judicious use of the :term:`LICENSE_FLAGS` strings and the contents of the
10744:term:`LICENSE_FLAGS_WHITELIST` variable allows you a lot of flexibility for
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010745including or excluding recipes based on licensing. For example, you can
10746broaden the matching capabilities by using license flags string subsets
10747in the whitelist.
10748
10749.. note::
10750
10751 When using a string subset, be sure to use the part of the expanded
10752 string that precedes the appended underscore character (e.g.
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010753 ``usethispart_1.3``, ``usethispart_1.4``, and so forth).
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010754
10755For example, simply specifying the string "commercial" in the whitelist
Andrew Geissler09036742021-06-25 14:25:14 -050010756matches any expanded :term:`LICENSE_FLAGS` definition that starts with the
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010757string "commercial" such as "commercial_foo" and "commercial_bar", which
10758are the strings the build system automatically generates for
10759hypothetical recipes named "foo" and "bar" assuming those recipes simply
Andrew Geisslerc926e172021-05-07 16:11:35 -050010760specify the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010761
10762 LICENSE_FLAGS = "commercial"
10763
10764Thus, you can choose
10765to exhaustively enumerate each license flag in the whitelist and allow
10766only specific recipes into the image, or you can use a string subset
10767that causes a broader range of matches to allow a range of recipes into
10768the image.
10769
Andrew Geissler09036742021-06-25 14:25:14 -050010770This scheme works even if the :term:`LICENSE_FLAGS` string already has
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010771``_${PN}`` appended. For example, the build system turns the license
10772flag "commercial_1.2_foo" into "commercial_1.2_foo_foo" and would match
10773both the general "commercial" and the specific "commercial_1.2_foo"
10774strings found in the whitelist, as expected.
10775
10776Here are some other scenarios:
10777
10778- You can specify a versioned string in the recipe such as
10779 "commercial_foo_1.2" in a "foo" recipe. The build system expands this
10780 string to "commercial_foo_1.2_foo". Combine this license flag with a
10781 whitelist that has the string "commercial" and you match the flag
10782 along with any other flag that starts with the string "commercial".
10783
10784- Under the same circumstances, you can use "commercial_foo" in the
10785 whitelist and the build system not only matches "commercial_foo_1.2"
10786 but also matches any license flag with the string "commercial_foo",
10787 regardless of the version.
10788
10789- You can be very specific and use both the package and version parts
10790 in the whitelist (e.g. "commercial_foo_1.2") to specifically match a
10791 versioned recipe.
10792
10793Other Variables Related to Commercial Licenses
10794~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10795
William A. Kennington IIIac69b482021-06-02 12:28:27 -070010796There are other helpful variables related to commercial license handling,
10797defined in the
Andrew Geisslerc926e172021-05-07 16:11:35 -050010798``poky/meta/conf/distro/include/default-distrovars.inc`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010799
10800 COMMERCIAL_AUDIO_PLUGINS ?= ""
10801 COMMERCIAL_VIDEO_PLUGINS ?= ""
10802
10803If you
10804want to enable these components, you can do so by making sure you have
10805statements similar to the following in your ``local.conf`` configuration
Andrew Geisslerc926e172021-05-07 16:11:35 -050010806file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010807
10808 COMMERCIAL_AUDIO_PLUGINS = "gst-plugins-ugly-mad \
10809 gst-plugins-ugly-mpegaudioparse"
10810 COMMERCIAL_VIDEO_PLUGINS = "gst-plugins-ugly-mpeg2dec \
10811 gst-plugins-ugly-mpegstream gst-plugins-bad-mpegvideoparse"
10812 LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly commercial_gst-plugins-bad commercial_qmmp"
10813
10814
10815Of course, you could also create a matching whitelist for those
10816components using the more general "commercial" in the whitelist, but
Andrew Geissler09036742021-06-25 14:25:14 -050010817that would also enable all the other packages with :term:`LICENSE_FLAGS`
Andrew Geisslerc926e172021-05-07 16:11:35 -050010818containing "commercial", which you may or may not want::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010819
10820 LICENSE_FLAGS_WHITELIST = "commercial"
10821
10822Specifying audio and video plugins as part of the
10823``COMMERCIAL_AUDIO_PLUGINS`` and ``COMMERCIAL_VIDEO_PLUGINS`` statements
Andrew Geissler09036742021-06-25 14:25:14 -050010824(along with the enabling :term:`LICENSE_FLAGS_WHITELIST`) includes the
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010825plugins or components into built images, thus adding support for media
10826formats or components.
10827
10828Maintaining Open Source License Compliance During Your Product's Lifecycle
10829--------------------------------------------------------------------------
10830
10831One of the concerns for a development organization using open source
10832software is how to maintain compliance with various open source
10833licensing during the lifecycle of the product. While this section does
10834not provide legal advice or comprehensively cover all scenarios, it does
10835present methods that you can use to assist you in meeting the compliance
10836requirements during a software release.
10837
10838With hundreds of different open source licenses that the Yocto Project
10839tracks, it is difficult to know the requirements of each and every
10840license. However, the requirements of the major FLOSS licenses can begin
William A. Kennington IIIac69b482021-06-02 12:28:27 -070010841to be covered by assuming that there are three main areas of concern:
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010842
10843- Source code must be provided.
10844
10845- License text for the software must be provided.
10846
10847- Compilation scripts and modifications to the source code must be
10848 provided.
10849
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010850- spdx files can be provided.
10851
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010852There are other requirements beyond the scope of these three and the
10853methods described in this section (e.g. the mechanism through which
10854source code is distributed).
10855
10856As different organizations have different methods of complying with open
10857source licensing, this section is not meant to imply that there is only
10858one single way to meet your compliance obligations, but rather to
10859describe one method of achieving compliance. The remainder of this
10860section describes methods supported to meet the previously mentioned
10861three requirements. Once you take steps to meet these requirements, and
10862prior to releasing images, sources, and the build system, you should
10863audit all artifacts to ensure completeness.
10864
10865.. note::
10866
10867 The Yocto Project generates a license manifest during image creation
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010868 that is located in ``${DEPLOY_DIR}/licenses/``\ `image_name`\ ``-``\ `datestamp`
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010869 to assist with any audits.
10870
10871Providing the Source Code
10872~~~~~~~~~~~~~~~~~~~~~~~~~
10873
10874Compliance activities should begin before you generate the final image.
10875The first thing you should look at is the requirement that tops the list
10876for most compliance groups - providing the source. The Yocto Project has
10877a few ways of meeting this requirement.
10878
10879One of the easiest ways to meet this requirement is to provide the
10880entire :term:`DL_DIR` used by the
10881build. This method, however, has a few issues. The most obvious is the
10882size of the directory since it includes all sources used in the build
10883and not just the source used in the released image. It will include
10884toolchain source, and other artifacts, which you would not generally
10885release. However, the more serious issue for most companies is
10886accidental release of proprietary software. The Yocto Project provides
10887an :ref:`archiver <ref-classes-archiver>` class to
10888help avoid some of these concerns.
10889
Andrew Geissler09036742021-06-25 14:25:14 -050010890Before you employ :term:`DL_DIR` or the ``archiver`` class, you need to
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010891decide how you choose to provide source. The source ``archiver`` class
10892can generate tarballs and SRPMs and can create them with various levels
10893of compliance in mind.
10894
10895One way of doing this (but certainly not the only way) is to release
10896just the source as a tarball. You can do this by adding the following to
10897the ``local.conf`` file found in the
Andrew Geisslerc926e172021-05-07 16:11:35 -050010898:term:`Build Directory`::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010899
10900 INHERIT += "archiver"
10901 ARCHIVER_MODE[src] = "original"
10902
10903During the creation of your
10904image, the source from all recipes that deploy packages to the image is
10905placed within subdirectories of ``DEPLOY_DIR/sources`` based on the
10906:term:`LICENSE` for each recipe.
10907Releasing the entire directory enables you to comply with requirements
10908concerning providing the unmodified source. It is important to note that
10909the size of the directory can get large.
10910
10911A way to help mitigate the size issue is to only release tarballs for
10912licenses that require the release of source. Let us assume you are only
10913concerned with GPL code as identified by running the following script:
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010914
10915.. code-block:: shell
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010916
10917 # Script to archive a subset of packages matching specific license(s)
10918 # Source and license files are copied into sub folders of package folder
10919 # Must be run from build folder
10920 #!/bin/bash
10921 src_release_dir="source-release"
10922 mkdir -p $src_release_dir
10923 for a in tmp/deploy/sources/*; do
10924 for d in $a/*; do
10925 # Get package name from path
10926 p=`basename $d`
10927 p=${p%-*}
10928 p=${p%-*}
10929 # Only archive GPL packages (update *GPL* regex for your license check)
10930 numfiles=`ls tmp/deploy/licenses/$p/*GPL* 2> /dev/null | wc -l`
10931 if [ $numfiles -gt 1 ]; then
10932 echo Archiving $p
10933 mkdir -p $src_release_dir/$p/source
10934 cp $d/* $src_release_dir/$p/source 2> /dev/null
10935 mkdir -p $src_release_dir/$p/license
10936 cp tmp/deploy/licenses/$p/* $src_release_dir/$p/license 2> /dev/null
10937 fi
10938 done
10939 done
10940
10941At this point, you
10942could create a tarball from the ``gpl_source_release`` directory and
10943provide that to the end user. This method would be a step toward
10944achieving compliance with section 3a of GPLv2 and with section 6 of
10945GPLv3.
10946
10947Providing License Text
10948~~~~~~~~~~~~~~~~~~~~~~
10949
10950One requirement that is often overlooked is inclusion of license text.
10951This requirement also needs to be dealt with prior to generating the
10952final image. Some licenses require the license text to accompany the
10953binary. You can achieve this by adding the following to your
Andrew Geisslerc926e172021-05-07 16:11:35 -050010954``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010955
10956 COPY_LIC_MANIFEST = "1"
10957 COPY_LIC_DIRS = "1"
10958 LICENSE_CREATE_PACKAGE = "1"
10959
10960Adding these statements to the
10961configuration file ensures that the licenses collected during package
10962generation are included on your image.
10963
10964.. note::
10965
10966 Setting all three variables to "1" results in the image having two
10967 copies of the same license file. One copy resides in
10968 ``/usr/share/common-licenses`` and the other resides in
10969 ``/usr/share/license``.
10970
10971 The reason for this behavior is because
10972 :term:`COPY_LIC_DIRS` and
10973 :term:`COPY_LIC_MANIFEST`
10974 add a copy of the license when the image is built but do not offer a
10975 path for adding licenses for newly installed packages to an image.
10976 :term:`LICENSE_CREATE_PACKAGE`
10977 adds a separate package and an upgrade path for adding licenses to an
10978 image.
10979
10980As the source ``archiver`` class has already archived the original
10981unmodified source that contains the license files, you would have
10982already met the requirements for inclusion of the license information
10983with source as defined by the GPL and other open source licenses.
10984
10985Providing Compilation Scripts and Source Code Modifications
10986~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10987
10988At this point, we have addressed all we need to prior to generating the
10989image. The next two requirements are addressed during the final
10990packaging of the release.
10991
10992By releasing the version of the OpenEmbedded build system and the layers
10993used during the build, you will be providing both compilation scripts
10994and the source code modifications in one step.
10995
Andrew Geissler09209ee2020-12-13 08:44:15 -060010996If the deployment team has a :ref:`overview-manual/concepts:bsp layer`
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010997and a distro layer, and those
10998those layers are used to patch, compile, package, or modify (in any way)
10999any open source software included in your released images, you might be
11000required to release those layers under section 3 of GPLv2 or section 1
11001of GPLv3. One way of doing that is with a clean checkout of the version
11002of the Yocto Project and layers used during your build. Here is an
11003example:
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011004
11005.. code-block:: shell
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011006
11007 # We built using the dunfell branch of the poky repo
11008 $ git clone -b dunfell git://git.yoctoproject.org/poky
11009 $ cd poky
11010 # We built using the release_branch for our layers
11011 $ git clone -b release_branch git://git.mycompany.com/meta-my-bsp-layer
11012 $ git clone -b release_branch git://git.mycompany.com/meta-my-software-layer
11013 # clean up the .git repos
11014 $ find . -name ".git" -type d -exec rm -rf {} \;
11015
11016One
11017thing a development organization might want to consider for end-user
11018convenience is to modify ``meta-poky/conf/bblayers.conf.sample`` to
11019ensure that when the end user utilizes the released build system to
11020build an image, the development organization's layers are included in
Andrew Geisslerc926e172021-05-07 16:11:35 -050011021the ``bblayers.conf`` file automatically::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011022
11023 # POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
11024 # changes incompatibly
11025 POKY_BBLAYERS_CONF_VERSION = "2"
11026
11027 BBPATH = "${TOPDIR}"
11028 BBFILES ?= ""
11029
11030 BBLAYERS ?= " \
11031 ##OEROOT##/meta \
11032 ##OEROOT##/meta-poky \
11033 ##OEROOT##/meta-yocto-bsp \
11034 ##OEROOT##/meta-mylayer \
11035 "
11036
11037Creating and
11038providing an archive of the :term:`Metadata`
11039layers (recipes, configuration files, and so forth) enables you to meet
11040your requirements to include the scripts to control compilation as well
11041as any modifications to the original source.
11042
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011043Providing spdx files
11044~~~~~~~~~~~~~~~~~~~~~~~~~
11045
11046The spdx module has been integrated to a layer named meta-spdxscanner.
11047meta-spdxscanner provides several kinds of scanner. If you want to enable
11048this function, you have to follow the following steps:
11049
110501. Add meta-spdxscanner layer into ``bblayers.conf``.
11051
110522. Refer to the README in meta-spdxscanner to setup the environment (e.g,
11053 setup a fossology server) needed for the scanner.
11054
110553. Meta-spdxscanner provides several methods within the bbclass to create spdx files.
11056 Please choose one that you want to use and enable the spdx task. You have to
11057 add some config options in ``local.conf`` file in your :term:`Build
William A. Kennington IIIac69b482021-06-02 12:28:27 -070011058 Directory`. Here is an example showing how to generate spdx files
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011059 during bitbake using the fossology-python.bbclass::
11060
11061 # Select fossology-python.bbclass.
11062 INHERIT += "fossology-python"
11063 # For fossology-python.bbclass, TOKEN is necessary, so, after setup a
11064 # Fossology server, you have to create a token.
11065 TOKEN = "eyJ0eXAiO..."
11066 # The fossology server is necessary for fossology-python.bbclass.
11067 FOSSOLOGY_SERVER = "http://xx.xx.xx.xx:8081/repo"
11068 # If you want to upload the source code to a special folder:
11069 FOLDER_NAME = "xxxx" //Optional
11070 # If you don't want to put spdx files in tmp/deploy/spdx, you can enable:
11071 SPDX_DEPLOY_DIR = "${DEPLOY_DIR}" //Optional
11072
11073For more usage information refer to :yocto_git:`the meta-spdxscanner repository
Andrew Geissler09209ee2020-12-13 08:44:15 -060011074</meta-spdxscanner/>`.
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011075
11076
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011077Copying Licenses that Do Not Exist
11078----------------------------------
11079
11080Some packages, such as the linux-firmware package, have many licenses
11081that are not in any way common. You can avoid adding a lot of these
11082types of common license files, which are only applicable to a specific
11083package, by using the
11084:term:`NO_GENERIC_LICENSE`
11085variable. Using this variable also avoids QA errors when you use a
11086non-common, non-CLOSED license in a recipe.
11087
William A. Kennington IIIac69b482021-06-02 12:28:27 -070011088Here is an example that uses the ``LICENSE.Abilis.txt`` file as
Andrew Geisslerc926e172021-05-07 16:11:35 -050011089the license from the fetched source::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011090
11091 NO_GENERIC_LICENSE[Firmware-Abilis] = "LICENSE.Abilis.txt"
11092
11093Using the Error Reporting Tool
11094==============================
11095
11096The error reporting tool allows you to submit errors encountered during
11097builds to a central database. Outside of the build environment, you can
11098use a web interface to browse errors, view statistics, and query for
11099errors. The tool works using a client-server system where the client
11100portion is integrated with the installed Yocto Project
11101:term:`Source Directory` (e.g. ``poky``).
11102The server receives the information collected and saves it in a
11103database.
11104
William A. Kennington IIIac69b482021-06-02 12:28:27 -070011105There is a live instance of the error reporting server at
11106https://errors.yoctoproject.org.
11107When you want to get help with build failures, you can submit all of the
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011108information on the failure easily and then point to the URL in your bug
11109report or send an email to the mailing list.
11110
11111.. note::
11112
11113 If you send error reports to this server, the reports become publicly
11114 visible.
11115
11116Enabling and Using the Tool
11117---------------------------
11118
11119By default, the error reporting tool is disabled. You can enable it by
11120inheriting the
11121:ref:`report-error <ref-classes-report-error>`
11122class by adding the following statement to the end of your
11123``local.conf`` file in your
11124:term:`Build Directory`.
11125::
11126
11127 INHERIT += "report-error"
11128
11129By default, the error reporting feature stores information in
11130``${``\ :term:`LOG_DIR`\ ``}/error-report``.
11131However, you can specify a directory to use by adding the following to
Andrew Geisslerc926e172021-05-07 16:11:35 -050011132your ``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011133
11134 ERR_REPORT_DIR = "path"
11135
11136Enabling error
11137reporting causes the build process to collect the errors and store them
11138in a file as previously described. When the build system encounters an
11139error, it includes a command as part of the console output. You can run
11140the command to send the error file to the server. For example, the
Andrew Geisslerc926e172021-05-07 16:11:35 -050011141following command sends the errors to an upstream server::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011142
11143 $ send-error-report /home/brandusa/project/poky/build/tmp/log/error-report/error_report_201403141617.txt
11144
11145In the previous example, the errors are sent to a public database
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011146available at https://errors.yoctoproject.org, which is used by the
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011147entire community. If you specify a particular server, you can send the
11148errors to a different database. Use the following command for more
Andrew Geisslerc926e172021-05-07 16:11:35 -050011149information on available options::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011150
11151 $ send-error-report --help
11152
11153When sending the error file, you are prompted to review the data being
11154sent as well as to provide a name and optional email address. Once you
11155satisfy these prompts, the command returns a link from the server that
11156corresponds to your entry in the database. For example, here is a
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011157typical link: https://errors.yoctoproject.org/Errors/Details/9522/
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011158
11159Following the link takes you to a web interface where you can browse,
11160query the errors, and view statistics.
11161
11162Disabling the Tool
11163------------------
11164
11165To disable the error reporting feature, simply remove or comment out the
11166following statement from the end of your ``local.conf`` file in your
11167:term:`Build Directory`.
11168::
11169
11170 INHERIT += "report-error"
11171
11172Setting Up Your Own Error Reporting Server
11173------------------------------------------
11174
11175If you want to set up your own error reporting server, you can obtain
Andrew Geissler09209ee2020-12-13 08:44:15 -060011176the code from the Git repository at :yocto_git:`/error-report-web/`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011177Instructions on how to set it up are in the README document.
11178
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011179Using Wayland and Weston
11180========================
11181
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011182`Wayland <https://en.wikipedia.org/wiki/Wayland_(display_server_protocol)>`__
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011183is a computer display server protocol that provides a method for
11184compositing window managers to communicate directly with applications
11185and video hardware and expects them to communicate with input hardware
11186using other libraries. Using Wayland with supporting targets can result
11187in better control over graphics frame rendering than an application
11188might otherwise achieve.
11189
11190The Yocto Project provides the Wayland protocol libraries and the
11191reference
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011192`Weston <https://en.wikipedia.org/wiki/Wayland_(display_server_protocol)#Weston>`__
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011193compositor as part of its release. You can find the integrated packages
11194in the ``meta`` layer of the :term:`Source Directory`.
11195Specifically, you
11196can find the recipes that build both Wayland and Weston at
11197``meta/recipes-graphics/wayland``.
11198
11199You can build both the Wayland and Weston packages for use only with
11200targets that accept the `Mesa 3D and Direct Rendering
11201Infrastructure <https://en.wikipedia.org/wiki/Mesa_(computer_graphics)>`__,
11202which is also known as Mesa DRI. This implies that you cannot build and
11203use the packages if your target uses, for example, the Intel Embedded
11204Media and Graphics Driver (Intel EMGD) that overrides Mesa DRI.
11205
11206.. note::
11207
11208 Due to lack of EGL support, Weston 1.0.3 will not run directly on the
11209 emulated QEMU hardware. However, this version of Weston will run
11210 under X emulation without issues.
11211
11212This section describes what you need to do to implement Wayland and use
11213the Weston compositor when building an image for a supporting target.
11214
11215Enabling Wayland in an Image
11216----------------------------
11217
11218To enable Wayland, you need to enable it to be built and enable it to be
11219included (installed) in the image.
11220
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011221Building Wayland
11222~~~~~~~~~~~~~~~~
11223
11224To cause Mesa to build the ``wayland-egl`` platform and Weston to build
11225Wayland with Kernel Mode Setting
11226(`KMS <https://wiki.archlinux.org/index.php/Kernel_Mode_Setting>`__)
11227support, include the "wayland" flag in the
11228:term:`DISTRO_FEATURES`
Andrew Geisslerc926e172021-05-07 16:11:35 -050011229statement in your ``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011230
11231 DISTRO_FEATURES_append = " wayland"
11232
11233.. note::
11234
11235 If X11 has been enabled elsewhere, Weston will build Wayland with X11
11236 support
11237
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011238Installing Wayland and Weston
11239~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
11240
11241To install the Wayland feature into an image, you must include the
11242following
11243:term:`CORE_IMAGE_EXTRA_INSTALL`
Andrew Geisslerc926e172021-05-07 16:11:35 -050011244statement in your ``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011245
11246 CORE_IMAGE_EXTRA_INSTALL += "wayland weston"
11247
11248Running Weston
11249--------------
11250
11251To run Weston inside X11, enabling it as described earlier and building
11252a Sato image is sufficient. If you are running your image under Sato, a
11253Weston Launcher appears in the "Utility" category.
11254
11255Alternatively, you can run Weston through the command-line interpretor
11256(CLI), which is better suited for development work. To run Weston under
11257the CLI, you need to do the following after your image is built:
11258
Andrew Geisslerc926e172021-05-07 16:11:35 -0500112591. Run these commands to export ``XDG_RUNTIME_DIR``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011260
11261 mkdir -p /tmp/$USER-weston
11262 chmod 0700 /tmp/$USER-weston
11263 export XDG_RUNTIME_DIR=/tmp/$USER-weston
11264
Andrew Geisslerc926e172021-05-07 16:11:35 -0500112652. Launch Weston in the shell::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011266
11267 weston