blob: 8be3c7c0e8eadd5c26a6b9356e658cdf200e845b [file] [log] [blame]
Andrew Geisslerf0343792020-11-18 10:42:21 -06001.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002
3************
4Common Tasks
5************
6
7This chapter describes fundamental procedures such as creating layers,
8adding new software packages, extending or customizing images, porting
9work to new hardware (adding a new machine), and so forth. You will find
10that the procedures documented here occur often in the development cycle
11using the Yocto Project.
12
13Understanding and Creating Layers
14=================================
15
16The OpenEmbedded build system supports organizing
17:term:`Metadata` into multiple layers.
18Layers allow you to isolate different types of customizations from each
19other. For introductory information on the Yocto Project Layer Model,
20see the
Andrew Geissler09209ee2020-12-13 08:44:15 -060021":ref:`overview-manual/yp-intro:the yocto project layer model`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050022section in the Yocto Project Overview and Concepts Manual.
23
24Creating Your Own Layer
25-----------------------
26
Andrew Geissler595f6302022-01-24 19:11:47 +000027.. note::
28
29 It is very easy to create your own layers to use with the OpenEmbedded
30 build system, as the Yocto Project ships with tools that speed up creating
31 layers. This section describes the steps you perform by hand to create
32 layers so that you can better understand them. For information about the
33 layer-creation tools, see the
34 ":ref:`bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script`"
35 section in the Yocto Project Board Support Package (BSP) Developer's
36 Guide and the ":ref:`dev-manual/common-tasks:creating a general layer using the \`\`bitbake-layers\`\` script`"
37 section further down in this manual.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050038
39Follow these general steps to create your layer without using tools:
40
411. *Check Existing Layers:* Before creating a new layer, you should be
42 sure someone has not already created a layer containing the Metadata
Andrew Geisslerd1e89492021-02-12 15:35:20 -060043 you need. You can see the :oe_layerindex:`OpenEmbedded Metadata Index <>`
44 for a list of layers from the OpenEmbedded community that can be used in
Andrew Geisslerc9f78652020-09-18 14:11:35 -050045 the Yocto Project. You could find a layer that is identical or close
46 to what you need.
47
482. *Create a Directory:* Create the directory for your layer. When you
49 create the layer, be sure to create the directory in an area not
50 associated with the Yocto Project :term:`Source Directory`
51 (e.g. the cloned ``poky`` repository).
52
53 While not strictly required, prepend the name of the directory with
Andrew Geisslerc926e172021-05-07 16:11:35 -050054 the string "meta-". For example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050055
56 meta-mylayer
57 meta-GUI_xyz
58 meta-mymachine
59
Andrew Geisslerc926e172021-05-07 16:11:35 -050060 With rare exceptions, a layer's name follows this form::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050061
62 meta-root_name
63
64 Following this layer naming convention can save
65 you trouble later when tools, components, or variables "assume" your
66 layer name begins with "meta-". A notable example is in configuration
67 files as shown in the following step where layer names without the
68 "meta-" string are appended to several variables used in the
69 configuration.
70
713. *Create a Layer Configuration File:* Inside your new layer folder,
72 you need to create a ``conf/layer.conf`` file. It is easiest to take
73 an existing layer configuration file and copy that to your layer's
74 ``conf`` directory and then modify the file as needed.
75
76 The ``meta-yocto-bsp/conf/layer.conf`` file in the Yocto Project
Andrew Geissler09209ee2020-12-13 08:44:15 -060077 :yocto_git:`Source Repositories </poky/tree/meta-yocto-bsp/conf>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -050078 demonstrates the required syntax. For your layer, you need to replace
79 "yoctobsp" with a unique identifier for your layer (e.g. "machinexyz"
Andrew Geisslerc926e172021-05-07 16:11:35 -050080 for a layer named "meta-machinexyz")::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050081
82 # We have a conf and classes directory, add to BBPATH
83 BBPATH .= ":${LAYERDIR}"
84
Andrew Geissler4c19ea12020-10-27 13:52:24 -050085 # We have recipes-* directories, add to BBFILES
Andrew Geisslerc9f78652020-09-18 14:11:35 -050086 BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
87 ${LAYERDIR}/recipes-*/*/*.bbappend"
88
89 BBFILE_COLLECTIONS += "yoctobsp"
90 BBFILE_PATTERN_yoctobsp = "^${LAYERDIR}/"
91 BBFILE_PRIORITY_yoctobsp = "5"
92 LAYERVERSION_yoctobsp = "4"
93 LAYERSERIES_COMPAT_yoctobsp = "dunfell"
94
95 Following is an explanation of the layer configuration file:
96
97 - :term:`BBPATH`: Adds the layer's
98 root directory to BitBake's search path. Through the use of the
Andrew Geissler09036742021-06-25 14:25:14 -050099 :term:`BBPATH` variable, BitBake locates class files (``.bbclass``),
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500100 configuration files, and files that are included with ``include``
101 and ``require`` statements. For these cases, BitBake uses the
Andrew Geissler09036742021-06-25 14:25:14 -0500102 first file that matches the name found in :term:`BBPATH`. This is
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500103 similar to the way the ``PATH`` variable is used for binaries. It
104 is recommended, therefore, that you use unique class and
105 configuration filenames in your custom layer.
106
107 - :term:`BBFILES`: Defines the
108 location for all recipes in the layer.
109
110 - :term:`BBFILE_COLLECTIONS`:
111 Establishes the current layer through a unique identifier that is
112 used throughout the OpenEmbedded build system to refer to the
113 layer. In this example, the identifier "yoctobsp" is the
114 representation for the container layer named "meta-yocto-bsp".
115
116 - :term:`BBFILE_PATTERN`:
117 Expands immediately during parsing to provide the directory of the
118 layer.
119
120 - :term:`BBFILE_PRIORITY`:
121 Establishes a priority to use for recipes in the layer when the
122 OpenEmbedded build finds recipes of the same name in different
123 layers.
124
125 - :term:`LAYERVERSION`:
126 Establishes a version number for the layer. You can use this
127 version number to specify this exact version of the layer as a
128 dependency when using the
129 :term:`LAYERDEPENDS`
130 variable.
131
132 - :term:`LAYERDEPENDS`:
133 Lists all layers on which this layer depends (if any).
134
135 - :term:`LAYERSERIES_COMPAT`:
Andrew Geissler09209ee2020-12-13 08:44:15 -0600136 Lists the :yocto_wiki:`Yocto Project </Releases>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500137 releases for which the current version is compatible. This
138 variable is a good way to indicate if your particular layer is
139 current.
140
1414. *Add Content:* Depending on the type of layer, add the content. If
142 the layer adds support for a machine, add the machine configuration
143 in a ``conf/machine/`` file within the layer. If the layer adds
144 distro policy, add the distro configuration in a ``conf/distro/``
145 file within the layer. If the layer introduces new recipes, put the
146 recipes you need in ``recipes-*`` subdirectories within the layer.
147
148 .. note::
149
150 For an explanation of layer hierarchy that is compliant with the
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500151 Yocto Project, see the ":ref:`bsp-guide/bsp:example filesystem layout`"
152 section in the Yocto Project Board Support Package (BSP) Developer's Guide.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500153
1545. *Optionally Test for Compatibility:* If you want permission to use
155 the Yocto Project Compatibility logo with your layer or application
156 that uses your layer, perform the steps to apply for compatibility.
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500157 See the
158 ":ref:`dev-manual/common-tasks:making sure your layer is compatible with yocto project`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500159 section for more information.
160
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500161Following Best Practices When Creating Layers
162---------------------------------------------
163
164To create layers that are easier to maintain and that will not impact
165builds for other machines, you should consider the information in the
166following list:
167
168- *Avoid "Overlaying" Entire Recipes from Other Layers in Your
169 Configuration:* In other words, do not copy an entire recipe into
170 your layer and then modify it. Rather, use an append file
171 (``.bbappend``) to override only those parts of the original recipe
172 you need to modify.
173
174- *Avoid Duplicating Include Files:* Use append files (``.bbappend``)
175 for each recipe that uses an include file. Or, if you are introducing
176 a new recipe that requires the included file, use the path relative
177 to the original layer directory to refer to the file. For example,
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500178 use ``require recipes-core/``\ `package`\ ``/``\ `file`\ ``.inc`` instead
179 of ``require`` `file`\ ``.inc``. If you're finding you have to overlay
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500180 the include file, it could indicate a deficiency in the include file
181 in the layer to which it originally belongs. If this is the case, you
182 should try to address that deficiency instead of overlaying the
183 include file. For example, you could address this by getting the
184 maintainer of the include file to add a variable or variables to make
185 it easy to override the parts needing to be overridden.
186
187- *Structure Your Layers:* Proper use of overrides within append files
188 and placement of machine-specific files within your layer can ensure
189 that a build is not using the wrong Metadata and negatively impacting
190 a build for a different machine. Following are some examples:
191
192 - *Modify Variables to Support a Different Machine:* Suppose you
193 have a layer named ``meta-one`` that adds support for building
194 machine "one". To do so, you use an append file named
195 ``base-files.bbappend`` and create a dependency on "foo" by
196 altering the :term:`DEPENDS`
Andrew Geisslerc926e172021-05-07 16:11:35 -0500197 variable::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500198
199 DEPENDS = "foo"
200
201 The dependency is created during any
202 build that includes the layer ``meta-one``. However, you might not
203 want this dependency for all machines. For example, suppose you
204 are building for machine "two" but your ``bblayers.conf`` file has
205 the ``meta-one`` layer included. During the build, the
206 ``base-files`` for machine "two" will also have the dependency on
207 ``foo``.
208
209 To make sure your changes apply only when building machine "one",
Andrew Geissler09036742021-06-25 14:25:14 -0500210 use a machine override with the :term:`DEPENDS` statement::
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500211
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500212 DEPENDS:one = "foo"
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500213
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500214 You should follow the same strategy when using ``:append``
215 and ``:prepend`` operations::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500216
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500217 DEPENDS:append:one = " foo"
218 DEPENDS:prepend:one = "foo "
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500219
220 As an actual example, here's a
221 snippet from the generic kernel include file ``linux-yocto.inc``,
222 wherein the kernel compile and link options are adjusted in the
Andrew Geisslerc926e172021-05-07 16:11:35 -0500223 case of a subset of the supported architectures::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500224
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500225 DEPENDS:append:aarch64 = " libgcc"
226 KERNEL_CC:append:aarch64 = " ${TOOLCHAIN_OPTIONS}"
227 KERNEL_LD:append:aarch64 = " ${TOOLCHAIN_OPTIONS}"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500228
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500229 DEPENDS:append:nios2 = " libgcc"
230 KERNEL_CC:append:nios2 = " ${TOOLCHAIN_OPTIONS}"
231 KERNEL_LD:append:nios2 = " ${TOOLCHAIN_OPTIONS}"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500232
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500233 DEPENDS:append:arc = " libgcc"
234 KERNEL_CC:append:arc = " ${TOOLCHAIN_OPTIONS}"
235 KERNEL_LD:append:arc = " ${TOOLCHAIN_OPTIONS}"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500236
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500237 KERNEL_FEATURES:append:qemuall=" features/debug/printk.scc"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500238
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500239 - *Place Machine-Specific Files in Machine-Specific Locations:* When
240 you have a base recipe, such as ``base-files.bb``, that contains a
241 :term:`SRC_URI` statement to a
242 file, you can use an append file to cause the build to use your
243 own version of the file. For example, an append file in your layer
244 at ``meta-one/recipes-core/base-files/base-files.bbappend`` could
Andrew Geisslerc926e172021-05-07 16:11:35 -0500245 extend :term:`FILESPATH` using :term:`FILESEXTRAPATHS` as follows::
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500246
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500247 FILESEXTRAPATHS:prepend := "${THISDIR}/${BPN}:"
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500248
249 The build for machine "one" will pick up your machine-specific file as
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500250 long as you have the file in
251 ``meta-one/recipes-core/base-files/base-files/``. However, if you
252 are building for a different machine and the ``bblayers.conf``
253 file includes the ``meta-one`` layer and the location of your
254 machine-specific file is the first location where that file is
Andrew Geissler09036742021-06-25 14:25:14 -0500255 found according to :term:`FILESPATH`, builds for all machines will
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500256 also use that machine-specific file.
257
258 You can make sure that a machine-specific file is used for a
259 particular machine by putting the file in a subdirectory specific
260 to the machine. For example, rather than placing the file in
261 ``meta-one/recipes-core/base-files/base-files/`` as shown above,
262 put it in ``meta-one/recipes-core/base-files/base-files/one/``.
263 Not only does this make sure the file is used only when building
264 for machine "one", but the build process locates the file more
265 quickly.
266
267 In summary, you need to place all files referenced from
Andrew Geissler5f350902021-07-23 13:09:54 -0400268 :term:`SRC_URI` in a machine-specific subdirectory within the layer in
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500269 order to restrict those files to machine-specific builds.
270
271- *Perform Steps to Apply for Yocto Project Compatibility:* If you want
272 permission to use the Yocto Project Compatibility logo with your
273 layer or application that uses your layer, perform the steps to apply
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500274 for compatibility. See the
275 ":ref:`dev-manual/common-tasks:making sure your layer is compatible with yocto project`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500276 section for more information.
277
278- *Follow the Layer Naming Convention:* Store custom layers in a Git
279 repository that use the ``meta-layer_name`` format.
280
281- *Group Your Layers Locally:* Clone your repository alongside other
282 cloned ``meta`` directories from the :term:`Source Directory`.
283
284Making Sure Your Layer is Compatible With Yocto Project
285-------------------------------------------------------
286
287When you create a layer used with the Yocto Project, it is advantageous
288to make sure that the layer interacts well with existing Yocto Project
289layers (i.e. the layer is compatible with the Yocto Project). Ensuring
290compatibility makes the layer easy to be consumed by others in the Yocto
291Project community and could allow you permission to use the Yocto
292Project Compatible Logo.
293
294.. note::
295
296 Only Yocto Project member organizations are permitted to use the
297 Yocto Project Compatible Logo. The logo is not available for general
298 use. For information on how to become a Yocto Project member
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500299 organization, see the :yocto_home:`Yocto Project Website <>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500300
301The Yocto Project Compatibility Program consists of a layer application
302process that requests permission to use the Yocto Project Compatibility
303Logo for your layer and application. The process consists of two parts:
304
3051. Successfully passing a script (``yocto-check-layer``) that when run
306 against your layer, tests it against constraints based on experiences
307 of how layers have worked in the real world and where pitfalls have
308 been found. Getting a "PASS" result from the script is required for
309 successful compatibility registration.
310
3112. Completion of an application acceptance form, which you can find at
Andrew Geisslerd1e89492021-02-12 15:35:20 -0600312 :yocto_home:`/webform/yocto-project-compatible-registration`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500313
314To be granted permission to use the logo, you need to satisfy the
315following:
316
317- Be able to check the box indicating that you got a "PASS" when
318 running the script against your layer.
319
320- Answer "Yes" to the questions on the form or have an acceptable
321 explanation for any questions answered "No".
322
323- Be a Yocto Project Member Organization.
324
325The remainder of this section presents information on the registration
326form and on the ``yocto-check-layer`` script.
327
328Yocto Project Compatible Program Application
329~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
330
331Use the form to apply for your layer's approval. Upon successful
332application, you can use the Yocto Project Compatibility Logo with your
333layer and the application that uses your layer.
334
335To access the form, use this link:
Andrew Geisslerd1e89492021-02-12 15:35:20 -0600336:yocto_home:`/webform/yocto-project-compatible-registration`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500337Follow the instructions on the form to complete your application.
338
339The application consists of the following sections:
340
341- *Contact Information:* Provide your contact information as the fields
342 require. Along with your information, provide the released versions
343 of the Yocto Project for which your layer is compatible.
344
345- *Acceptance Criteria:* Provide "Yes" or "No" answers for each of the
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700346 items in the checklist. There is space at the bottom of the form for
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500347 any explanations for items for which you answered "No".
348
349- *Recommendations:* Provide answers for the questions regarding Linux
350 kernel use and build success.
351
352``yocto-check-layer`` Script
353~~~~~~~~~~~~~~~~~~~~~~~~~~~~
354
355The ``yocto-check-layer`` script provides you a way to assess how
356compatible your layer is with the Yocto Project. You should run this
357script prior to using the form to apply for compatibility as described
358in the previous section. You need to achieve a "PASS" result in order to
359have your application form successfully processed.
360
361The script divides tests into three areas: COMMON, BSP, and DISTRO. For
362example, given a distribution layer (DISTRO), the layer must pass both
363the COMMON and DISTRO related tests. Furthermore, if your layer is a BSP
364layer, the layer must pass the COMMON and BSP set of tests.
365
366To execute the script, enter the following commands from your build
Andrew Geisslerc926e172021-05-07 16:11:35 -0500367directory::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500368
369 $ source oe-init-build-env
370 $ yocto-check-layer your_layer_directory
371
372Be sure to provide the actual directory for your
373layer as part of the command.
374
375Entering the command causes the script to determine the type of layer
376and then to execute a set of specific tests against the layer. The
377following list overviews the test:
378
379- ``common.test_readme``: Tests if a ``README`` file exists in the
380 layer and the file is not empty.
381
382- ``common.test_parse``: Tests to make sure that BitBake can parse the
383 files without error (i.e. ``bitbake -p``).
384
385- ``common.test_show_environment``: Tests that the global or per-recipe
386 environment is in order without errors (i.e. ``bitbake -e``).
387
388- ``common.test_world``: Verifies that ``bitbake world`` works.
389
390- ``common.test_signatures``: Tests to be sure that BSP and DISTRO
391 layers do not come with recipes that change signatures.
392
393- ``common.test_layerseries_compat``: Verifies layer compatibility is
394 set properly.
395
396- ``bsp.test_bsp_defines_machines``: Tests if a BSP layer has machine
397 configurations.
398
399- ``bsp.test_bsp_no_set_machine``: Tests to ensure a BSP layer does not
400 set the machine when the layer is added.
401
402- ``bsp.test_machine_world``: Verifies that ``bitbake world`` works
403 regardless of which machine is selected.
404
405- ``bsp.test_machine_signatures``: Verifies that building for a
406 particular machine affects only the signature of tasks specific to
407 that machine.
408
409- ``distro.test_distro_defines_distros``: Tests if a DISTRO layer has
410 distro configurations.
411
412- ``distro.test_distro_no_set_distros``: Tests to ensure a DISTRO layer
413 does not set the distribution when the layer is added.
414
415Enabling Your Layer
416-------------------
417
418Before the OpenEmbedded build system can use your new layer, you need to
419enable it. To enable your layer, simply add your layer's path to the
Andrew Geissler09036742021-06-25 14:25:14 -0500420:term:`BBLAYERS` variable in your ``conf/bblayers.conf`` file, which is
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500421found in the :term:`Build Directory`.
Andrew Geissler5199d832021-09-24 16:47:35 -0500422The following example shows how to enable your new
423``meta-mylayer`` layer (note how your new layer exists outside of
424the official ``poky`` repository which you would have checked out earlier)::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500425
426 # POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
427 # changes incompatibly
428 POKY_BBLAYERS_CONF_VERSION = "2"
429 BBPATH = "${TOPDIR}"
430 BBFILES ?= ""
431 BBLAYERS ?= " \
432 /home/user/poky/meta \
433 /home/user/poky/meta-poky \
434 /home/user/poky/meta-yocto-bsp \
Andrew Geissler5199d832021-09-24 16:47:35 -0500435 /home/user/mystuff/meta-mylayer \
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500436 "
437
438BitBake parses each ``conf/layer.conf`` file from the top down as
Andrew Geissler09036742021-06-25 14:25:14 -0500439specified in the :term:`BBLAYERS` variable within the ``conf/bblayers.conf``
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500440file. During the processing of each ``conf/layer.conf`` file, BitBake
441adds the recipes, classes and configurations contained within the
442particular layer to the source directory.
443
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500444Appending Other Layers Metadata With Your Layer
445-----------------------------------------------
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500446
447A recipe that appends Metadata to another recipe is called a BitBake
448append file. A BitBake append file uses the ``.bbappend`` file type
449suffix, while the corresponding recipe to which Metadata is being
450appended uses the ``.bb`` file type suffix.
451
452You can use a ``.bbappend`` file in your layer to make additions or
453changes to the content of another layer's recipe without having to copy
454the other layer's recipe into your layer. Your ``.bbappend`` file
455resides in your layer, while the main ``.bb`` recipe file to which you
456are appending Metadata resides in a different layer.
457
458Being able to append information to an existing recipe not only avoids
459duplication, but also automatically applies recipe changes from a
460different layer into your layer. If you were copying recipes, you would
461have to manually merge changes as they occur.
462
463When you create an append file, you must use the same root name as the
464corresponding recipe file. For example, the append file
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500465``someapp_3.1.bbappend`` must apply to ``someapp_3.1.bb``. This
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500466means the original recipe and append filenames are version
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500467number-specific. If the corresponding recipe is renamed to update to a
468newer version, you must also rename and possibly update the
469corresponding ``.bbappend`` as well. During the build process, BitBake
470displays an error on starting if it detects a ``.bbappend`` file that
471does not have a corresponding recipe with a matching name. See the
472:term:`BB_DANGLINGAPPENDS_WARNONLY`
473variable for information on how to handle this error.
474
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500475Overlaying a File Using Your Layer
476~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
477
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500478As an example, consider the main formfactor recipe and a corresponding
479formfactor append file both from the :term:`Source Directory`.
480Here is the main
481formfactor recipe, which is named ``formfactor_0.0.bb`` and located in
Andrew Geisslerc926e172021-05-07 16:11:35 -0500482the "meta" layer at ``meta/recipes-bsp/formfactor``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500483
484 SUMMARY = "Device formfactor information"
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500485 DESCRIPTION = "A formfactor configuration file provides information about the \
486 target hardware for which the image is being built and information that the \
487 build system cannot obtain from other sources such as the kernel."
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500488 SECTION = "base"
489 LICENSE = "MIT"
490 LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
491 PR = "r45"
492
493 SRC_URI = "file://config file://machconfig"
494 S = "${WORKDIR}"
495
496 PACKAGE_ARCH = "${MACHINE_ARCH}"
497 INHIBIT_DEFAULT_DEPS = "1"
498
499 do_install() {
500 # Install file only if it has contents
501 install -d ${D}${sysconfdir}/formfactor/
502 install -m 0644 ${S}/config ${D}${sysconfdir}/formfactor/
503 if [ -s "${S}/machconfig" ]; then
504 install -m 0644 ${S}/machconfig ${D}${sysconfdir}/formfactor/
505 fi
506 }
507
508In the main recipe, note the :term:`SRC_URI`
509variable, which tells the OpenEmbedded build system where to find files
510during the build.
511
512Following is the append file, which is named ``formfactor_0.0.bbappend``
513and is from the Raspberry Pi BSP Layer named ``meta-raspberrypi``. The
Andrew Geisslerc926e172021-05-07 16:11:35 -0500514file is in the layer at ``recipes-bsp/formfactor``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500515
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500516 FILESEXTRAPATHS:prepend := "${THISDIR}/${PN}:"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500517
518By default, the build system uses the
519:term:`FILESPATH` variable to
520locate files. This append file extends the locations by setting the
521:term:`FILESEXTRAPATHS`
522variable. Setting this variable in the ``.bbappend`` file is the most
523reliable and recommended method for adding directories to the search
524path used by the build system to find files.
525
526The statement in this example extends the directories to include
527``${``\ :term:`THISDIR`\ ``}/${``\ :term:`PN`\ ``}``,
528which resolves to a directory named ``formfactor`` in the same directory
529in which the append file resides (i.e.
530``meta-raspberrypi/recipes-bsp/formfactor``. This implies that you must
531have the supporting directory structure set up that will contain any
532files or patches you will be including from the layer.
533
534Using the immediate expansion assignment operator ``:=`` is important
Andrew Geissler09036742021-06-25 14:25:14 -0500535because of the reference to :term:`THISDIR`. The trailing colon character is
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500536important as it ensures that items in the list remain colon-separated.
537
538.. note::
539
Andrew Geissler09036742021-06-25 14:25:14 -0500540 BitBake automatically defines the :term:`THISDIR` variable. You should
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500541 never set this variable yourself. Using ":prepend" as part of the
Andrew Geissler09036742021-06-25 14:25:14 -0500542 :term:`FILESEXTRAPATHS` ensures your path will be searched prior to other
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500543 paths in the final list.
544
545 Also, not all append files add extra files. Many append files simply
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700546 allow to add build options (e.g. ``systemd``). For these cases, your
Andrew Geissler09036742021-06-25 14:25:14 -0500547 append file would not even use the :term:`FILESEXTRAPATHS` statement.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500548
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500549The end result of this ``.bbappend`` file is that on a Raspberry Pi, where
550``rpi`` will exist in the list of :term:`OVERRIDES`, the file
551``meta-raspberrypi/recipes-bsp/formfactor/formfactor/rpi/machconfig`` will be
552used during :ref:`ref-tasks-fetch` and the test for a non-zero file size in
553:ref:`ref-tasks-install` will return true, and the file will be installed.
554
Andrew Geissler5199d832021-09-24 16:47:35 -0500555Installing Additional Files Using Your Layer
556~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
557
558As another example, consider the main ``xserver-xf86-config`` recipe and a
559corresponding ``xserver-xf86-config`` append file both from the :term:`Source
560Directory`. Here is the main ``xserver-xf86-config`` recipe, which is named
561``xserver-xf86-config_0.1.bb`` and located in the "meta" layer at
562``meta/recipes-graphics/xorg-xserver``::
563
564 SUMMARY = "X.Org X server configuration file"
565 HOMEPAGE = "http://www.x.org"
566 SECTION = "x11/base"
Andrew Geissler9aee5002022-03-30 16:27:02 +0000567 LICENSE = "MIT"
Andrew Geissler5199d832021-09-24 16:47:35 -0500568 LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
569 PR = "r33"
570
571 SRC_URI = "file://xorg.conf"
572
573 S = "${WORKDIR}"
574
575 CONFFILES:${PN} = "${sysconfdir}/X11/xorg.conf"
576
577 PACKAGE_ARCH = "${MACHINE_ARCH}"
578 ALLOW_EMPTY:${PN} = "1"
579
580 do_install () {
581 if test -s ${WORKDIR}/xorg.conf; then
582 install -d ${D}/${sysconfdir}/X11
583 install -m 0644 ${WORKDIR}/xorg.conf ${D}/${sysconfdir}/X11/
584 fi
585 }
586
587Following is the append file, which is named ``xserver-xf86-config_%.bbappend``
588and is from the Raspberry Pi BSP Layer named ``meta-raspberrypi``. The
589file is in the layer at ``recipes-graphics/xorg-xserver``::
590
591 FILESEXTRAPATHS:prepend := "${THISDIR}/${PN}:"
592
593 SRC_URI:append:rpi = " \
594 file://xorg.conf.d/98-pitft.conf \
595 file://xorg.conf.d/99-calibration.conf \
596 "
597 do_install:append:rpi () {
598 PITFT="${@bb.utils.contains("MACHINE_FEATURES", "pitft", "1", "0", d)}"
599 if [ "${PITFT}" = "1" ]; then
600 install -d ${D}/${sysconfdir}/X11/xorg.conf.d/
601 install -m 0644 ${WORKDIR}/xorg.conf.d/98-pitft.conf ${D}/${sysconfdir}/X11/xorg.conf.d/
602 install -m 0644 ${WORKDIR}/xorg.conf.d/99-calibration.conf ${D}/${sysconfdir}/X11/xorg.conf.d/
603 fi
604 }
605
606 FILES:${PN}:append:rpi = " ${sysconfdir}/X11/xorg.conf.d/*"
607
608Building off of the previous example, we once again are setting the
609:term:`FILESEXTRAPATHS` variable. In this case we are also using
610:term:`SRC_URI` to list additional source files to use when ``rpi`` is found in
611the list of :term:`OVERRIDES`. The :ref:`ref-tasks-install` task will then perform a
612check for an additional :term:`MACHINE_FEATURES` that if set will cause these
613additional files to be installed. These additional files are listed in
614:term:`FILES` so that they will be packaged.
615
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500616Prioritizing Your Layer
617-----------------------
618
619Each layer is assigned a priority value. Priority values control which
620layer takes precedence if there are recipe files with the same name in
621multiple layers. For these cases, the recipe file from the layer with a
622higher priority number takes precedence. Priority values also affect the
623order in which multiple ``.bbappend`` files for the same recipe are
624applied. You can either specify the priority manually, or allow the
625build system to calculate it based on the layer's dependencies.
626
627To specify the layer's priority manually, use the
628:term:`BBFILE_PRIORITY`
Andrew Geisslerc926e172021-05-07 16:11:35 -0500629variable and append the layer's root name::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500630
631 BBFILE_PRIORITY_mylayer = "1"
632
633.. note::
634
635 It is possible for a recipe with a lower version number
636 :term:`PV` in a layer that has a higher
637 priority to take precedence.
638
639 Also, the layer priority does not currently affect the precedence
640 order of ``.conf`` or ``.bbclass`` files. Future versions of BitBake
641 might address this.
642
643Managing Layers
644---------------
645
646You can use the BitBake layer management tool ``bitbake-layers`` to
647provide a view into the structure of recipes across a multi-layer
648project. Being able to generate output that reports on configured layers
649with their paths and priorities and on ``.bbappend`` files and their
650applicable recipes can help to reveal potential problems.
651
652For help on the BitBake layer management tool, use the following
Andrew Geisslerc926e172021-05-07 16:11:35 -0500653command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500654
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500655 $ bitbake-layers --help
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500656 NOTE: Starting bitbake server...
657 usage: bitbake-layers [-d] [-q] [-F] [--color COLOR] [-h] <subcommand> ...
658
659 BitBake layers utility
660
661 optional arguments:
662 -d, --debug Enable debug output
663 -q, --quiet Print only errors
664 -F, --force Force add without recipe parse verification
665 --color COLOR Colorize output (where COLOR is auto, always, never)
666 -h, --help show this help message and exit
667
668 subcommands:
669 <subcommand>
670 layerindex-fetch Fetches a layer from a layer index along with its
671 dependent layers, and adds them to conf/bblayers.conf.
672 layerindex-show-depends
673 Find layer dependencies from layer index.
674 add-layer Add one or more layers to bblayers.conf.
675 remove-layer Remove one or more layers from bblayers.conf.
676 flatten flatten layer configuration into a separate output
677 directory.
678 show-layers show current configured layers.
679 show-overlayed list overlayed recipes (where the same recipe exists
680 in another layer)
681 show-recipes list available recipes, showing the layer they are
682 provided by
683 show-appends list bbappend files and recipe files they apply to
684 show-cross-depends Show dependencies between recipes that cross layer
685 boundaries.
686 create-layer Create a basic layer
687
688 Use bitbake-layers <subcommand> --help to get help on a specific command
689
690The following list describes the available commands:
691
692- ``help:`` Displays general help or help on a specified command.
693
694- ``show-layers:`` Shows the current configured layers.
695
696- ``show-overlayed:`` Lists overlayed recipes. A recipe is overlayed
697 when a recipe with the same name exists in another layer that has a
698 higher layer priority.
699
700- ``show-recipes:`` Lists available recipes and the layers that
701 provide them.
702
703- ``show-appends:`` Lists ``.bbappend`` files and the recipe files to
704 which they apply.
705
706- ``show-cross-depends:`` Lists dependency relationships between
707 recipes that cross layer boundaries.
708
709- ``add-layer:`` Adds a layer to ``bblayers.conf``.
710
711- ``remove-layer:`` Removes a layer from ``bblayers.conf``
712
713- ``flatten:`` Flattens the layer configuration into a separate
714 output directory. Flattening your layer configuration builds a
715 "flattened" directory that contains the contents of all layers, with
716 any overlayed recipes removed and any ``.bbappend`` files appended to
717 the corresponding recipes. You might have to perform some manual
718 cleanup of the flattened layer as follows:
719
720 - Non-recipe files (such as patches) are overwritten. The flatten
721 command shows a warning for these files.
722
723 - Anything beyond the normal layer setup has been added to the
724 ``layer.conf`` file. Only the lowest priority layer's
725 ``layer.conf`` is used.
726
727 - Overridden and appended items from ``.bbappend`` files need to be
728 cleaned up. The contents of each ``.bbappend`` end up in the
729 flattened recipe. However, if there are appended or changed
730 variable values, you need to tidy these up yourself. Consider the
731 following example. Here, the ``bitbake-layers`` command adds the
732 line ``#### bbappended ...`` so that you know where the following
Andrew Geisslerc926e172021-05-07 16:11:35 -0500733 lines originate::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500734
735 ...
736 DESCRIPTION = "A useful utility"
737 ...
738 EXTRA_OECONF = "--enable-something"
739 ...
740
741 #### bbappended from meta-anotherlayer ####
742
743 DESCRIPTION = "Customized utility"
744 EXTRA_OECONF += "--enable-somethingelse"
745
746
Andrew Geisslerc926e172021-05-07 16:11:35 -0500747 Ideally, you would tidy up these utilities as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500748
749 ...
750 DESCRIPTION = "Customized utility"
751 ...
752 EXTRA_OECONF = "--enable-something --enable-somethingelse"
753 ...
754
755- ``layerindex-fetch``: Fetches a layer from a layer index, along
756 with its dependent layers, and adds the layers to the
757 ``conf/bblayers.conf`` file.
758
759- ``layerindex-show-depends``: Finds layer dependencies from the
760 layer index.
761
762- ``create-layer``: Creates a basic layer.
763
764Creating a General Layer Using the ``bitbake-layers`` Script
765------------------------------------------------------------
766
767The ``bitbake-layers`` script with the ``create-layer`` subcommand
768simplifies creating a new general layer.
769
770.. note::
771
772 - For information on BSP layers, see the ":ref:`bsp-guide/bsp:bsp layers`"
773 section in the Yocto
774 Project Board Specific (BSP) Developer's Guide.
775
776 - In order to use a layer with the OpenEmbedded build system, you
777 need to add the layer to your ``bblayers.conf`` configuration
Andrew Geissler09209ee2020-12-13 08:44:15 -0600778 file. See the ":ref:`dev-manual/common-tasks:adding a layer using the \`\`bitbake-layers\`\` script`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500779 section for more information.
780
781The default mode of the script's operation with this subcommand is to
782create a layer with the following:
783
784- A layer priority of 6.
785
786- A ``conf`` subdirectory that contains a ``layer.conf`` file.
787
788- A ``recipes-example`` subdirectory that contains a further
789 subdirectory named ``example``, which contains an ``example.bb``
790 recipe file.
791
792- A ``COPYING.MIT``, which is the license statement for the layer. The
793 script assumes you want to use the MIT license, which is typical for
794 most layers, for the contents of the layer itself.
795
796- A ``README`` file, which is a file describing the contents of your
797 new layer.
798
799In its simplest form, you can use the following command form to create a
800layer. The command creates a layer whose name corresponds to
Andrew Geisslerc926e172021-05-07 16:11:35 -0500801"your_layer_name" in the current directory::
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500802
803 $ bitbake-layers create-layer your_layer_name
804
805As an example, the following command creates a layer named ``meta-scottrif``
Andrew Geisslerc926e172021-05-07 16:11:35 -0500806in your home directory::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500807
808 $ cd /usr/home
809 $ bitbake-layers create-layer meta-scottrif
810 NOTE: Starting bitbake server...
811 Add your new layer with 'bitbake-layers add-layer meta-scottrif'
812
813If you want to set the priority of the layer to other than the default
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500814value of "6", you can either use the ``--priority`` option or you
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500815can edit the
816:term:`BBFILE_PRIORITY` value
817in the ``conf/layer.conf`` after the script creates it. Furthermore, if
818you want to give the example recipe file some name other than the
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500819default, you can use the ``--example-recipe-name`` option.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500820
821The easiest way to see how the ``bitbake-layers create-layer`` command
822works is to experiment with the script. You can also read the usage
Andrew Geisslerc926e172021-05-07 16:11:35 -0500823information by entering the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500824
825 $ bitbake-layers create-layer --help
826 NOTE: Starting bitbake server...
827 usage: bitbake-layers create-layer [-h] [--priority PRIORITY]
828 [--example-recipe-name EXAMPLERECIPE]
829 layerdir
830
831 Create a basic layer
832
833 positional arguments:
834 layerdir Layer directory to create
835
836 optional arguments:
837 -h, --help show this help message and exit
838 --priority PRIORITY, -p PRIORITY
839 Layer directory to create
840 --example-recipe-name EXAMPLERECIPE, -e EXAMPLERECIPE
841 Filename of the example recipe
842
843Adding a Layer Using the ``bitbake-layers`` Script
844--------------------------------------------------
845
846Once you create your general layer, you must add it to your
847``bblayers.conf`` file. Adding the layer to this configuration file
848makes the OpenEmbedded build system aware of your layer so that it can
849search it for metadata.
850
Andrew Geisslerc926e172021-05-07 16:11:35 -0500851Add your layer by using the ``bitbake-layers add-layer`` command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500852
853 $ bitbake-layers add-layer your_layer_name
854
855Here is an example that adds a
856layer named ``meta-scottrif`` to the configuration file. Following the
857command that adds the layer is another ``bitbake-layers`` command that
Andrew Geisslerc926e172021-05-07 16:11:35 -0500858shows the layers that are in your ``bblayers.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500859
860 $ bitbake-layers add-layer meta-scottrif
861 NOTE: Starting bitbake server...
862 Parsing recipes: 100% |##########################################################| Time: 0:00:49
863 Parsing of 1441 .bb files complete (0 cached, 1441 parsed). 2055 targets, 56 skipped, 0 masked, 0 errors.
864 $ bitbake-layers show-layers
865 NOTE: Starting bitbake server...
866 layer path priority
867 ==========================================================================
868 meta /home/scottrif/poky/meta 5
869 meta-poky /home/scottrif/poky/meta-poky 5
870 meta-yocto-bsp /home/scottrif/poky/meta-yocto-bsp 5
871 workspace /home/scottrif/poky/build/workspace 99
872 meta-scottrif /home/scottrif/poky/build/meta-scottrif 6
873
874
875Adding the layer to this file
876enables the build system to locate the layer during the build.
877
878.. note::
879
880 During a build, the OpenEmbedded build system looks in the layers
881 from the top of the list down to the bottom in that order.
882
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500883Customizing Images
884==================
885
886You can customize images to satisfy particular requirements. This
887section describes several methods and provides guidelines for each.
888
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500889Customizing Images Using ``local.conf``
890---------------------------------------
891
892Probably the easiest way to customize an image is to add a package by
893way of the ``local.conf`` configuration file. Because it is limited to
894local use, this method generally only allows you to add packages and is
895not as flexible as creating your own customized image. When you add
896packages using local variables this way, you need to realize that these
897variable changes are in effect for every build and consequently affect
898all images, which might not be what you require.
899
900To add a package to your image using the local configuration file, use
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500901the :term:`IMAGE_INSTALL` variable with the ``:append`` operator::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500902
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500903 IMAGE_INSTALL:append = " strace"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500904
Andrew Geissler5199d832021-09-24 16:47:35 -0500905Use of the syntax is important; specifically, the leading space
906after the opening quote and before the package name, which is
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500907``strace`` in this example. This space is required since the ``:append``
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500908operator does not add the space.
909
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500910Furthermore, you must use ``:append`` instead of the ``+=`` operator if
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500911you want to avoid ordering issues. The reason for this is because doing
912so unconditionally appends to the variable and avoids ordering problems
913due to the variable being set in image recipes and ``.bbclass`` files
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500914with operators like ``?=``. Using ``:append`` ensures the operation
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500915takes effect.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500916
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500917As shown in its simplest use, ``IMAGE_INSTALL:append`` affects all
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500918images. It is possible to extend the syntax so that the variable applies
Andrew Geisslerc926e172021-05-07 16:11:35 -0500919to a specific image only. Here is an example::
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500920
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500921 IMAGE_INSTALL:append:pn-core-image-minimal = " strace"
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500922
923This example adds ``strace`` to the ``core-image-minimal`` image only.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500924
925You can add packages using a similar approach through the
Andrew Geissler09036742021-06-25 14:25:14 -0500926:term:`CORE_IMAGE_EXTRA_INSTALL` variable. If you use this variable, only
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500927``core-image-*`` images are affected.
928
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500929Customizing Images Using Custom ``IMAGE_FEATURES`` and ``EXTRA_IMAGE_FEATURES``
930-------------------------------------------------------------------------------
931
932Another method for customizing your image is to enable or disable
933high-level image features by using the
934:term:`IMAGE_FEATURES` and
935:term:`EXTRA_IMAGE_FEATURES`
936variables. Although the functions for both variables are nearly
Andrew Geissler09036742021-06-25 14:25:14 -0500937equivalent, best practices dictate using :term:`IMAGE_FEATURES` from within
938a recipe and using :term:`EXTRA_IMAGE_FEATURES` from within your
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500939``local.conf`` file, which is found in the
940:term:`Build Directory`.
941
942To understand how these features work, the best reference is
Andrew Geissler595f6302022-01-24 19:11:47 +0000943:ref:`meta/classes/image.bbclass <ref-classes-image>`.
944This class lists out the available
Andrew Geissler09036742021-06-25 14:25:14 -0500945:term:`IMAGE_FEATURES` of which most map to package groups while some, such
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500946as ``debug-tweaks`` and ``read-only-rootfs``, resolve as general
947configuration settings.
948
Andrew Geissler09036742021-06-25 14:25:14 -0500949In summary, the file looks at the contents of the :term:`IMAGE_FEATURES`
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500950variable and then maps or configures the feature accordingly. Based on
951this information, the build system automatically adds the appropriate
952packages or configurations to the
953:term:`IMAGE_INSTALL` variable.
954Effectively, you are enabling extra features by extending the class or
955creating a custom class for use with specialized image ``.bb`` files.
956
Andrew Geissler09036742021-06-25 14:25:14 -0500957Use the :term:`EXTRA_IMAGE_FEATURES` variable from within your local
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500958configuration file. Using a separate area from which to enable features
959with this variable helps you avoid overwriting the features in the image
Andrew Geissler09036742021-06-25 14:25:14 -0500960recipe that are enabled with :term:`IMAGE_FEATURES`. The value of
961:term:`EXTRA_IMAGE_FEATURES` is added to :term:`IMAGE_FEATURES` within
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500962``meta/conf/bitbake.conf``.
963
964To illustrate how you can use these variables to modify your image,
965consider an example that selects the SSH server. The Yocto Project ships
966with two SSH servers you can use with your images: Dropbear and OpenSSH.
967Dropbear is a minimal SSH server appropriate for resource-constrained
968environments, while OpenSSH is a well-known standard SSH server
969implementation. By default, the ``core-image-sato`` image is configured
970to use Dropbear. The ``core-image-full-cmdline`` and ``core-image-lsb``
971images both include OpenSSH. The ``core-image-minimal`` image does not
972contain an SSH server.
973
974You can customize your image and change these defaults. Edit the
Andrew Geissler09036742021-06-25 14:25:14 -0500975:term:`IMAGE_FEATURES` variable in your recipe or use the
976:term:`EXTRA_IMAGE_FEATURES` in your ``local.conf`` file so that it
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500977configures the image you are working with to include
978``ssh-server-dropbear`` or ``ssh-server-openssh``.
979
980.. note::
981
Andrew Geissler09209ee2020-12-13 08:44:15 -0600982 See the ":ref:`ref-manual/features:image features`" section in the Yocto
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500983 Project Reference Manual for a complete list of image features that ship
984 with the Yocto Project.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500985
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500986Customizing Images Using Custom .bb Files
987-----------------------------------------
988
989You can also customize an image by creating a custom recipe that defines
990additional software as part of the image. The following example shows
Andrew Geisslerc926e172021-05-07 16:11:35 -0500991the form for the two lines you need::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500992
993 IMAGE_INSTALL = "packagegroup-core-x11-base package1 package2"
994 inherit core-image
995
996Defining the software using a custom recipe gives you total control over
997the contents of the image. It is important to use the correct names of
Andrew Geissler09036742021-06-25 14:25:14 -0500998packages in the :term:`IMAGE_INSTALL` variable. You must use the
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500999OpenEmbedded notation and not the Debian notation for the names (e.g.
1000``glibc-dev`` instead of ``libc6-dev``).
1001
1002The other method for creating a custom image is to base it on an
1003existing image. For example, if you want to create an image based on
1004``core-image-sato`` but add the additional package ``strace`` to the
1005image, copy the ``meta/recipes-sato/images/core-image-sato.bb`` to a new
Andrew Geisslerc926e172021-05-07 16:11:35 -05001006``.bb`` and add the following line to the end of the copy::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001007
1008 IMAGE_INSTALL += "strace"
1009
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001010Customizing Images Using Custom Package Groups
1011----------------------------------------------
1012
1013For complex custom images, the best approach for customizing an image is
1014to create a custom package group recipe that is used to build the image
1015or images. A good example of a package group recipe is
1016``meta/recipes-core/packagegroups/packagegroup-base.bb``.
1017
Andrew Geissler09036742021-06-25 14:25:14 -05001018If you examine that recipe, you see that the :term:`PACKAGES` variable lists
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001019the package group packages to produce. The ``inherit packagegroup``
1020statement sets appropriate default values and automatically adds
1021``-dev``, ``-dbg``, and ``-ptest`` complementary packages for each
Andrew Geissler09036742021-06-25 14:25:14 -05001022package specified in the :term:`PACKAGES` statement.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001023
1024.. note::
1025
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001026 The ``inherit packagegroup`` line should be located near the top of the
Andrew Geissler09036742021-06-25 14:25:14 -05001027 recipe, certainly before the :term:`PACKAGES` statement.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001028
Andrew Geissler09036742021-06-25 14:25:14 -05001029For each package you specify in :term:`PACKAGES`, you can use :term:`RDEPENDS`
1030and :term:`RRECOMMENDS` entries to provide a list of packages the parent
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001031task package should contain. You can see examples of these further down
1032in the ``packagegroup-base.bb`` recipe.
1033
1034Here is a short, fabricated example showing the same basic pieces for a
1035hypothetical packagegroup defined in ``packagegroup-custom.bb``, where
Andrew Geissler09036742021-06-25 14:25:14 -05001036the variable :term:`PN` is the standard way to abbreviate the reference to
Andrew Geisslerc926e172021-05-07 16:11:35 -05001037the full packagegroup name ``packagegroup-custom``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001038
1039 DESCRIPTION = "My Custom Package Groups"
1040
1041 inherit packagegroup
1042
1043 PACKAGES = "\
1044 ${PN}-apps \
1045 ${PN}-tools \
1046 "
1047
Patrick Williams0ca19cc2021-08-16 14:03:13 -05001048 RDEPENDS:${PN}-apps = "\
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001049 dropbear \
1050 portmap \
1051 psplash"
1052
Patrick Williams0ca19cc2021-08-16 14:03:13 -05001053 RDEPENDS:${PN}-tools = "\
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001054 oprofile \
1055 oprofileui-server \
1056 lttng-tools"
1057
Patrick Williams0ca19cc2021-08-16 14:03:13 -05001058 RRECOMMENDS:${PN}-tools = "\
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001059 kernel-module-oprofile"
1060
1061In the previous example, two package group packages are created with
1062their dependencies and their recommended package dependencies listed:
1063``packagegroup-custom-apps``, and ``packagegroup-custom-tools``. To
1064build an image using these package group packages, you need to add
1065``packagegroup-custom-apps`` and/or ``packagegroup-custom-tools`` to
Andrew Geissler09036742021-06-25 14:25:14 -05001066:term:`IMAGE_INSTALL`. For other forms of image dependencies see the other
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001067areas of this section.
1068
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001069Customizing an Image Hostname
1070-----------------------------
1071
1072By default, the configured hostname (i.e. ``/etc/hostname``) in an image
1073is the same as the machine name. For example, if
1074:term:`MACHINE` equals "qemux86", the
1075configured hostname written to ``/etc/hostname`` is "qemux86".
1076
1077You can customize this name by altering the value of the "hostname"
1078variable in the ``base-files`` recipe using either an append file or a
Andrew Geisslerc926e172021-05-07 16:11:35 -05001079configuration file. Use the following in an append file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001080
1081 hostname = "myhostname"
1082
Andrew Geisslerc926e172021-05-07 16:11:35 -05001083Use the following in a configuration file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001084
Patrick Williams0ca19cc2021-08-16 14:03:13 -05001085 hostname:pn-base-files = "myhostname"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001086
1087Changing the default value of the variable "hostname" can be useful in
1088certain situations. For example, suppose you need to do extensive
1089testing on an image and you would like to easily identify the image
1090under test from existing images with typical default hostnames. In this
1091situation, you could change the default hostname to "testme", which
1092results in all the images using the name "testme". Once testing is
1093complete and you do not need to rebuild the image for test any longer,
1094you can easily reset the default hostname.
1095
1096Another point of interest is that if you unset the variable, the image
1097will have no default hostname in the filesystem. Here is an example that
Andrew Geisslerc926e172021-05-07 16:11:35 -05001098unsets the variable in a configuration file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001099
Patrick Williams0ca19cc2021-08-16 14:03:13 -05001100 hostname:pn-base-files = ""
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001101
1102Having no default hostname in the filesystem is suitable for
1103environments that use dynamic hostnames such as virtual machines.
1104
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001105Writing a New Recipe
1106====================
1107
1108Recipes (``.bb`` files) are fundamental components in the Yocto Project
1109environment. Each software component built by the OpenEmbedded build
1110system requires a recipe to define the component. This section describes
1111how to create, write, and test a new recipe.
1112
1113.. note::
1114
1115 For information on variables that are useful for recipes and for
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001116 information about recipe naming issues, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001117 ":ref:`ref-manual/varlocality:recipes`" section of the Yocto Project
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001118 Reference Manual.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001119
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001120Overview
1121--------
1122
1123The following figure shows the basic process for creating a new recipe.
1124The remainder of the section provides details for the steps.
1125
1126.. image:: figures/recipe-workflow.png
1127 :align: center
Andrew Geisslerd5838332022-05-27 11:33:10 -05001128 :width: 50%
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001129
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001130Locate or Automatically Create a Base Recipe
1131--------------------------------------------
1132
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001133You can always write a recipe from scratch. However, there are three choices
1134that can help you quickly get started with a new recipe:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001135
1136- ``devtool add``: A command that assists in creating a recipe and an
1137 environment conducive to development.
1138
1139- ``recipetool create``: A command provided by the Yocto Project that
1140 automates creation of a base recipe based on the source files.
1141
1142- *Existing Recipes:* Location and modification of an existing recipe
1143 that is similar in function to the recipe you need.
1144
1145.. note::
1146
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001147 For information on recipe syntax, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001148 ":ref:`dev-manual/common-tasks:recipe syntax`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001149
1150Creating the Base Recipe Using ``devtool add``
1151~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1152
1153The ``devtool add`` command uses the same logic for auto-creating the
1154recipe as ``recipetool create``, which is listed below. Additionally,
1155however, ``devtool add`` sets up an environment that makes it easy for
1156you to patch the source and to make changes to the recipe as is often
1157necessary when adding a recipe to build a new piece of software to be
1158included in a build.
1159
1160You can find a complete description of the ``devtool add`` command in
Andrew Geissler09209ee2020-12-13 08:44:15 -06001161the ":ref:`sdk-manual/extensible:a closer look at \`\`devtool add\`\``" section
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001162in the Yocto Project Application Development and the Extensible Software
1163Development Kit (eSDK) manual.
1164
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001165Creating the Base Recipe Using ``recipetool create``
1166~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1167
1168``recipetool create`` automates creation of a base recipe given a set of
1169source code files. As long as you can extract or point to the source
1170files, the tool will construct a recipe and automatically configure all
1171pre-build information into the recipe. For example, suppose you have an
1172application that builds using Autotools. Creating the base recipe using
1173``recipetool`` results in a recipe that has the pre-build dependencies,
1174license requirements, and checksums configured.
1175
1176To run the tool, you just need to be in your
1177:term:`Build Directory` and have sourced the
1178build environment setup script (i.e.
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001179:ref:`structure-core-script`).
Andrew Geisslerc926e172021-05-07 16:11:35 -05001180To get help on the tool, use the following command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001181
1182 $ recipetool -h
1183 NOTE: Starting bitbake server...
1184 usage: recipetool [-d] [-q] [--color COLOR] [-h] <subcommand> ...
1185
1186 OpenEmbedded recipe tool
1187
1188 options:
1189 -d, --debug Enable debug output
1190 -q, --quiet Print only errors
1191 --color COLOR Colorize output (where COLOR is auto, always, never)
1192 -h, --help show this help message and exit
1193
1194 subcommands:
1195 create Create a new recipe
1196 newappend Create a bbappend for the specified target in the specified
1197 layer
1198 setvar Set a variable within a recipe
1199 appendfile Create/update a bbappend to replace a target file
1200 appendsrcfiles Create/update a bbappend to add or replace source files
1201 appendsrcfile Create/update a bbappend to add or replace a source file
1202 Use recipetool <subcommand> --help to get help on a specific command
1203
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001204Running ``recipetool create -o OUTFILE`` creates the base recipe and
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001205locates it properly in the layer that contains your source files.
1206Following are some syntax examples:
1207
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001208 - Use this syntax to generate a recipe based on source. Once generated,
Andrew Geisslerc926e172021-05-07 16:11:35 -05001209 the recipe resides in the existing source code layer::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001210
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001211 recipetool create -o OUTFILE source
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001212
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001213 - Use this syntax to generate a recipe using code that
1214 you extract from source. The extracted code is placed in its own layer
Andrew Geissler09036742021-06-25 14:25:14 -05001215 defined by :term:`EXTERNALSRC`.
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001216 ::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001217
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001218 recipetool create -o OUTFILE -x EXTERNALSRC source
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001219
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001220 - Use this syntax to generate a recipe based on source. The options
1221 direct ``recipetool`` to generate debugging information. Once generated,
Andrew Geisslerc926e172021-05-07 16:11:35 -05001222 the recipe resides in the existing source code layer::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001223
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001224 recipetool create -d -o OUTFILE source
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001225
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001226Locating and Using a Similar Recipe
1227~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1228
1229Before writing a recipe from scratch, it is often useful to discover
1230whether someone else has already written one that meets (or comes close
1231to meeting) your needs. The Yocto Project and OpenEmbedded communities
1232maintain many recipes that might be candidates for what you are doing.
Andrew Geisslerd1e89492021-02-12 15:35:20 -06001233You can find a good central index of these recipes in the
1234:oe_layerindex:`OpenEmbedded Layer Index <>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001235
1236Working from an existing recipe or a skeleton recipe is the best way to
1237get started. Here are some points on both methods:
1238
1239- *Locate and modify a recipe that is close to what you want to do:*
1240 This method works when you are familiar with the current recipe
1241 space. The method does not work so well for those new to the Yocto
1242 Project or writing recipes.
1243
1244 Some risks associated with this method are using a recipe that has
1245 areas totally unrelated to what you are trying to accomplish with
1246 your recipe, not recognizing areas of the recipe that you might have
1247 to add from scratch, and so forth. All these risks stem from
1248 unfamiliarity with the existing recipe space.
1249
1250- *Use and modify the following skeleton recipe:* If for some reason
1251 you do not want to use ``recipetool`` and you cannot find an existing
1252 recipe that is close to meeting your needs, you can use the following
1253 structure to provide the fundamental areas of a new recipe.
1254 ::
1255
1256 DESCRIPTION = ""
1257 HOMEPAGE = ""
1258 LICENSE = ""
1259 SECTION = ""
1260 DEPENDS = ""
1261 LIC_FILES_CHKSUM = ""
1262
1263 SRC_URI = ""
1264
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001265Storing and Naming the Recipe
1266-----------------------------
1267
1268Once you have your base recipe, you should put it in your own layer and
1269name it appropriately. Locating it correctly ensures that the
1270OpenEmbedded build system can find it when you use BitBake to process
1271the recipe.
1272
1273- *Storing Your Recipe:* The OpenEmbedded build system locates your
1274 recipe through the layer's ``conf/layer.conf`` file and the
1275 :term:`BBFILES` variable. This
1276 variable sets up a path from which the build system can locate
Andrew Geisslerc926e172021-05-07 16:11:35 -05001277 recipes. Here is the typical use::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001278
1279 BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
1280 ${LAYERDIR}/recipes-*/*/*.bbappend"
1281
1282 Consequently, you need to be sure you locate your new recipe inside
1283 your layer such that it can be found.
1284
1285 You can find more information on how layers are structured in the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001286 ":ref:`dev-manual/common-tasks:understanding and creating layers`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001287
1288- *Naming Your Recipe:* When you name your recipe, you need to follow
Andrew Geisslerc926e172021-05-07 16:11:35 -05001289 this naming convention::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001290
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001291 basename_version.bb
1292
1293 Use lower-cased characters and do not include the reserved suffixes
1294 ``-native``, ``-cross``, ``-initial``, or ``-dev`` casually (i.e. do not use
1295 them as part of your recipe name unless the string applies). Here are some
1296 examples:
1297
1298 .. code-block:: none
1299
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001300 cups_1.7.0.bb
1301 gawk_4.0.2.bb
1302 irssi_0.8.16-rc1.bb
1303
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001304Running a Build on the Recipe
1305-----------------------------
1306
1307Creating a new recipe is usually an iterative process that requires
1308using BitBake to process the recipe multiple times in order to
1309progressively discover and add information to the recipe file.
1310
1311Assuming you have sourced the build environment setup script (i.e.
1312:ref:`structure-core-script`) and you are in
1313the :term:`Build Directory`, use
1314BitBake to process your recipe. All you need to provide is the
Andrew Geisslerc926e172021-05-07 16:11:35 -05001315``basename`` of the recipe as described in the previous section::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001316
1317 $ bitbake basename
1318
1319During the build, the OpenEmbedded build system creates a temporary work
1320directory for each recipe
1321(``${``\ :term:`WORKDIR`\ ``}``)
1322where it keeps extracted source files, log files, intermediate
1323compilation and packaging files, and so forth.
1324
1325The path to the per-recipe temporary work directory depends on the
1326context in which it is being built. The quickest way to find this path
Andrew Geisslerc926e172021-05-07 16:11:35 -05001327is to have BitBake return it by running the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001328
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001329 $ bitbake -e basename | grep ^WORKDIR=
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001330
1331As an example, assume a Source Directory
1332top-level folder named ``poky``, a default Build Directory at
1333``poky/build``, and a ``qemux86-poky-linux`` machine target system.
1334Furthermore, suppose your recipe is named ``foo_1.3.0.bb``. In this
1335case, the work directory the build system uses to build the package
Andrew Geisslerc926e172021-05-07 16:11:35 -05001336would be as follows::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001337
1338 poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0
1339
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001340Inside this directory you can find sub-directories such as ``image``,
1341``packages-split``, and ``temp``. After the build, you can examine these
1342to determine how well the build went.
1343
1344.. note::
1345
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001346 You can find log files for each task in the recipe's ``temp``
1347 directory (e.g. ``poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0/temp``).
1348 Log files are named ``log.taskname`` (e.g. ``log.do_configure``,
1349 ``log.do_fetch``, and ``log.do_compile``).
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001350
1351You can find more information about the build process in
Andrew Geissler09209ee2020-12-13 08:44:15 -06001352":doc:`/overview-manual/development-environment`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001353chapter of the Yocto Project Overview and Concepts Manual.
1354
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001355Fetching Code
1356-------------
1357
1358The first thing your recipe must do is specify how to fetch the source
1359files. Fetching is controlled mainly through the
1360:term:`SRC_URI` variable. Your recipe
Andrew Geissler09036742021-06-25 14:25:14 -05001361must have a :term:`SRC_URI` variable that points to where the source is
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001362located. For a graphical representation of source locations, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001363":ref:`overview-manual/concepts:sources`" section in
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001364the Yocto Project Overview and Concepts Manual.
1365
1366The :ref:`ref-tasks-fetch` task uses
Andrew Geissler09036742021-06-25 14:25:14 -05001367the prefix of each entry in the :term:`SRC_URI` variable value to determine
Andrew Geissler09209ee2020-12-13 08:44:15 -06001368which :ref:`fetcher <bitbake:bitbake-user-manual/bitbake-user-manual-fetching:fetchers>` to use to get your
Andrew Geissler09036742021-06-25 14:25:14 -05001369source files. It is the :term:`SRC_URI` variable that triggers the fetcher.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001370The :ref:`ref-tasks-patch` task uses
1371the variable after source is fetched to apply patches. The OpenEmbedded
1372build system uses
1373:term:`FILESOVERRIDES` for
Andrew Geissler09036742021-06-25 14:25:14 -05001374scanning directory locations for local files in :term:`SRC_URI`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001375
Andrew Geissler09036742021-06-25 14:25:14 -05001376The :term:`SRC_URI` variable in your recipe must define each unique location
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001377for your source files. It is good practice to not hard-code version
Andrew Geissler5f350902021-07-23 13:09:54 -04001378numbers in a URL used in :term:`SRC_URI`. Rather than hard-code these
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001379values, use ``${``\ :term:`PV`\ ``}``,
1380which causes the fetch process to use the version specified in the
1381recipe filename. Specifying the version in this manner means that
1382upgrading the recipe to a future version is as simple as renaming the
1383recipe to match the new version.
1384
1385Here is a simple example from the
1386``meta/recipes-devtools/strace/strace_5.5.bb`` recipe where the source
1387comes from a single tarball. Notice the use of the
Andrew Geisslerc926e172021-05-07 16:11:35 -05001388:term:`PV` variable::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001389
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001390 SRC_URI = "https://strace.io/files/${PV}/strace-${PV}.tar.xz \
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001391
Andrew Geissler09036742021-06-25 14:25:14 -05001392Files mentioned in :term:`SRC_URI` whose names end in a typical archive
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001393extension (e.g. ``.tar``, ``.tar.gz``, ``.tar.bz2``, ``.zip``, and so
1394forth), are automatically extracted during the
1395:ref:`ref-tasks-unpack` task. For
1396another example that specifies these types of files, see the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001397":ref:`dev-manual/common-tasks:autotooled package`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001398
1399Another way of specifying source is from an SCM. For Git repositories,
Andrew Geissler9aee5002022-03-30 16:27:02 +00001400you must specify :term:`SRCREV` and you should specify :term:`PV` to include
1401the revision with :term:`SRCPV`. Here is an example from the recipe
1402``meta/recipes-core/musl/gcompat_git.bb``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001403
Andrew Geissler9aee5002022-03-30 16:27:02 +00001404 SRC_URI = "git://git.adelielinux.org/adelie/gcompat.git;protocol=https;branch=current"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001405
Andrew Geissler9aee5002022-03-30 16:27:02 +00001406 PV = "1.0.0+1.1+git${SRCPV}"
1407 SRCREV = "af5a49e489fdc04b9cf02547650d7aeaccd43793"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001408
Andrew Geissler09036742021-06-25 14:25:14 -05001409If your :term:`SRC_URI` statement includes URLs pointing to individual files
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001410fetched from a remote server other than a version control system,
1411BitBake attempts to verify the files against checksums defined in your
1412recipe to ensure they have not been tampered with or otherwise modified
1413since the recipe was written. Two checksums are used:
1414``SRC_URI[md5sum]`` and ``SRC_URI[sha256sum]``.
1415
Andrew Geissler09036742021-06-25 14:25:14 -05001416If your :term:`SRC_URI` variable points to more than a single URL (excluding
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001417SCM URLs), you need to provide the ``md5`` and ``sha256`` checksums for
1418each URL. For these cases, you provide a name for each URL as part of
Andrew Geissler09036742021-06-25 14:25:14 -05001419the :term:`SRC_URI` and then reference that name in the subsequent checksum
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001420statements. Here is an example combining lines from the files
Andrew Geisslerc926e172021-05-07 16:11:35 -05001421``git.inc`` and ``git_2.24.1.bb``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001422
1423 SRC_URI = "${KERNELORG_MIRROR}/software/scm/git/git-${PV}.tar.gz;name=tarball \
1424 ${KERNELORG_MIRROR}/software/scm/git/git-manpages-${PV}.tar.gz;name=manpages"
1425
1426 SRC_URI[tarball.md5sum] = "166bde96adbbc11c8843d4f8f4f9811b"
1427 SRC_URI[tarball.sha256sum] = "ad5334956301c86841eb1e5b1bb20884a6bad89a10a6762c958220c7cf64da02"
1428 SRC_URI[manpages.md5sum] = "31c2272a8979022497ba3d4202df145d"
1429 SRC_URI[manpages.sha256sum] = "9a7ae3a093bea39770eb96ca3e5b40bff7af0b9f6123f089d7821d0e5b8e1230"
1430
1431Proper values for ``md5`` and ``sha256`` checksums might be available
1432with other signatures on the download page for the upstream source (e.g.
1433``md5``, ``sha1``, ``sha256``, ``GPG``, and so forth). Because the
1434OpenEmbedded build system only deals with ``sha256sum`` and ``md5sum``,
1435you should verify all the signatures you find by hand.
1436
Andrew Geissler09036742021-06-25 14:25:14 -05001437If no :term:`SRC_URI` checksums are specified when you attempt to build the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001438recipe, or you provide an incorrect checksum, the build will produce an
1439error for each missing or incorrect checksum. As part of the error
1440message, the build system provides the checksum string corresponding to
1441the fetched file. Once you have the correct checksums, you can copy and
1442paste them into your recipe and then run the build again to continue.
1443
1444.. note::
1445
1446 As mentioned, if the upstream source provides signatures for
1447 verifying the downloaded source code, you should verify those
1448 manually before setting the checksum values in the recipe and
1449 continuing with the build.
1450
1451This final example is a bit more complicated and is from the
1452``meta/recipes-sato/rxvt-unicode/rxvt-unicode_9.20.bb`` recipe. The
Andrew Geissler09036742021-06-25 14:25:14 -05001453example's :term:`SRC_URI` statement identifies multiple files as the source
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001454files for the recipe: a tarball, a patch file, a desktop file, and an
1455icon.
1456::
1457
1458 SRC_URI = "http://dist.schmorp.de/rxvt-unicode/Attic/rxvt-unicode-${PV}.tar.bz2 \
1459 file://xwc.patch \
1460 file://rxvt.desktop \
1461 file://rxvt.png"
1462
1463When you specify local files using the ``file://`` URI protocol, the
1464build system fetches files from the local machine. The path is relative
1465to the :term:`FILESPATH` variable
1466and searches specific directories in a certain order:
1467``${``\ :term:`BP`\ ``}``,
1468``${``\ :term:`BPN`\ ``}``, and
1469``files``. The directories are assumed to be subdirectories of the
1470directory in which the recipe or append file resides. For another
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001471example that specifies these types of files, see the
1472":ref:`dev-manual/common-tasks:single .c file package (hello world!)`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001473
1474The previous example also specifies a patch file. Patch files are files
1475whose names usually end in ``.patch`` or ``.diff`` but can end with
1476compressed suffixes such as ``diff.gz`` and ``patch.bz2``, for example.
1477The build system automatically applies patches as described in the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001478":ref:`dev-manual/common-tasks:patching code`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001479
Andrew Geissler9aee5002022-03-30 16:27:02 +00001480Fetching Code Through Firewalls
1481~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1482
1483Some users are behind firewalls and need to fetch code through a proxy.
1484See the ":doc:`/ref-manual/faq`" chapter for advice.
1485
1486Limiting the Number of Parallel Connections
1487~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1488
1489Some users are behind firewalls or use servers where the number of parallel
1490connections is limited. In such cases, you can limit the number of fetch
1491tasks being run in parallel by adding the following to your ``local.conf``
1492file::
1493
1494 do_fetch[number_threads] = "4"
1495
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001496Unpacking Code
1497--------------
1498
1499During the build, the
1500:ref:`ref-tasks-unpack` task unpacks
1501the source with ``${``\ :term:`S`\ ``}``
1502pointing to where it is unpacked.
1503
1504If you are fetching your source files from an upstream source archived
1505tarball and the tarball's internal structure matches the common
1506convention of a top-level subdirectory named
1507``${``\ :term:`BPN`\ ``}-${``\ :term:`PV`\ ``}``,
Andrew Geissler09036742021-06-25 14:25:14 -05001508then you do not need to set :term:`S`. However, if :term:`SRC_URI` specifies to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001509fetch source from an archive that does not use this convention, or from
Andrew Geissler09036742021-06-25 14:25:14 -05001510an SCM like Git or Subversion, your recipe needs to define :term:`S`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001511
1512If processing your recipe using BitBake successfully unpacks the source
1513files, you need to be sure that the directory pointed to by ``${S}``
1514matches the structure of the source.
1515
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001516Patching Code
1517-------------
1518
1519Sometimes it is necessary to patch code after it has been fetched. Any
Andrew Geissler09036742021-06-25 14:25:14 -05001520files mentioned in :term:`SRC_URI` whose names end in ``.patch`` or
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001521``.diff`` or compressed versions of these suffixes (e.g. ``diff.gz`` are
1522treated as patches. The
1523:ref:`ref-tasks-patch` task
1524automatically applies these patches.
1525
1526The build system should be able to apply patches with the "-p1" option
1527(i.e. one directory level in the path will be stripped off). If your
1528patch needs to have more directory levels stripped off, specify the
Andrew Geissler09036742021-06-25 14:25:14 -05001529number of levels using the "striplevel" option in the :term:`SRC_URI` entry
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001530for the patch. Alternatively, if your patch needs to be applied in a
1531specific subdirectory that is not specified in the patch file, use the
1532"patchdir" option in the entry.
1533
1534As with all local files referenced in
1535:term:`SRC_URI` using ``file://``,
1536you should place patch files in a directory next to the recipe either
1537named the same as the base name of the recipe
1538(:term:`BP` and
1539:term:`BPN`) or "files".
1540
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001541Licensing
1542---------
1543
1544Your recipe needs to have both the
1545:term:`LICENSE` and
1546:term:`LIC_FILES_CHKSUM`
1547variables:
1548
Andrew Geissler09036742021-06-25 14:25:14 -05001549- :term:`LICENSE`: This variable specifies the license for the software.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001550 If you do not know the license under which the software you are
1551 building is distributed, you should go to the source code and look
1552 for that information. Typical files containing this information
Andrew Geissler09036742021-06-25 14:25:14 -05001553 include ``COPYING``, :term:`LICENSE`, and ``README`` files. You could
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001554 also find the information near the top of a source file. For example,
1555 given a piece of software licensed under the GNU General Public
Andrew Geissler09036742021-06-25 14:25:14 -05001556 License version 2, you would set :term:`LICENSE` as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001557
Andrew Geissler9aee5002022-03-30 16:27:02 +00001558 LICENSE = "GPL-2.0-only"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001559
Andrew Geissler09036742021-06-25 14:25:14 -05001560 The licenses you specify within :term:`LICENSE` can have any name as long
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001561 as you do not use spaces, since spaces are used as separators between
1562 license names. For standard licenses, use the names of the files in
Andrew Geissler09036742021-06-25 14:25:14 -05001563 ``meta/files/common-licenses/`` or the :term:`SPDXLICENSEMAP` flag names
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001564 defined in ``meta/conf/licenses.conf``.
1565
Andrew Geissler09036742021-06-25 14:25:14 -05001566- :term:`LIC_FILES_CHKSUM`: The OpenEmbedded build system uses this
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001567 variable to make sure the license text has not changed. If it has,
1568 the build produces an error and it affords you the chance to figure
1569 it out and correct the problem.
1570
1571 You need to specify all applicable licensing files for the software.
1572 At the end of the configuration step, the build process will compare
1573 the checksums of the files to be sure the text has not changed. Any
1574 differences result in an error with the message containing the
1575 current checksum. For more explanation and examples of how to set the
Andrew Geissler09036742021-06-25 14:25:14 -05001576 :term:`LIC_FILES_CHKSUM` variable, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001577 ":ref:`dev-manual/common-tasks:tracking license changes`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001578
1579 To determine the correct checksum string, you can list the
Andrew Geissler09036742021-06-25 14:25:14 -05001580 appropriate files in the :term:`LIC_FILES_CHKSUM` variable with incorrect
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001581 md5 strings, attempt to build the software, and then note the
1582 resulting error messages that will report the correct md5 strings.
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001583 See the ":ref:`dev-manual/common-tasks:fetching code`" section for
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001584 additional information.
1585
Andrew Geisslerc926e172021-05-07 16:11:35 -05001586 Here is an example that assumes the software has a ``COPYING`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001587
1588 LIC_FILES_CHKSUM = "file://COPYING;md5=xxx"
1589
1590 When you try to build the
1591 software, the build system will produce an error and give you the
1592 correct string that you can substitute into the recipe file for a
1593 subsequent build.
1594
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001595Dependencies
1596------------
1597
1598Most software packages have a short list of other packages that they
1599require, which are called dependencies. These dependencies fall into two
1600main categories: build-time dependencies, which are required when the
1601software is built; and runtime dependencies, which are required to be
1602installed on the target in order for the software to run.
1603
1604Within a recipe, you specify build-time dependencies using the
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001605:term:`DEPENDS` variable. Although there are nuances,
Andrew Geissler09036742021-06-25 14:25:14 -05001606items specified in :term:`DEPENDS` should be names of other
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001607recipes. It is important that you specify all build-time dependencies
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001608explicitly.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001609
1610Another consideration is that configure scripts might automatically
1611check for optional dependencies and enable corresponding functionality
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001612if those dependencies are found. If you wish to make a recipe that is
1613more generally useful (e.g. publish the recipe in a layer for others to
1614use), instead of hard-disabling the functionality, you can use the
1615:term:`PACKAGECONFIG` variable to allow functionality and the
1616corresponding dependencies to be enabled and disabled easily by other
1617users of the recipe.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001618
1619Similar to build-time dependencies, you specify runtime dependencies
1620through a variable -
1621:term:`RDEPENDS`, which is
1622package-specific. All variables that are package-specific need to have
1623the name of the package added to the end as an override. Since the main
1624package for a recipe has the same name as the recipe, and the recipe's
1625name can be found through the
1626``${``\ :term:`PN`\ ``}`` variable, then
1627you specify the dependencies for the main package by setting
Patrick Williams0ca19cc2021-08-16 14:03:13 -05001628``RDEPENDS:${PN}``. If the package were named ``${PN}-tools``, then you
1629would set ``RDEPENDS:${PN}-tools``, and so forth.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001630
1631Some runtime dependencies will be set automatically at packaging time.
1632These dependencies include any shared library dependencies (i.e. if a
1633package "example" contains "libexample" and another package "mypackage"
1634contains a binary that links to "libexample" then the OpenEmbedded build
1635system will automatically add a runtime dependency to "mypackage" on
1636"example"). See the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001637":ref:`overview-manual/concepts:automatically added runtime dependencies`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001638section in the Yocto Project Overview and Concepts Manual for further
1639details.
1640
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001641Configuring the Recipe
1642----------------------
1643
1644Most software provides some means of setting build-time configuration
1645options before compilation. Typically, setting these options is
1646accomplished by running a configure script with options, or by modifying
1647a build configuration file.
1648
1649.. note::
1650
1651 As of Yocto Project Release 1.7, some of the core recipes that
1652 package binary configuration scripts now disable the scripts due to
1653 the scripts previously requiring error-prone path substitution. The
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001654 OpenEmbedded build system uses ``pkg-config`` now, which is much more
1655 robust. You can find a list of the ``*-config`` scripts that are disabled
1656 in the ":ref:`migration-1.7-binary-configuration-scripts-disabled`" section
1657 in the Yocto Project Reference Manual.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001658
1659A major part of build-time configuration is about checking for
1660build-time dependencies and possibly enabling optional functionality as
1661a result. You need to specify any build-time dependencies for the
1662software you are building in your recipe's
1663:term:`DEPENDS` value, in terms of
1664other recipes that satisfy those dependencies. You can often find
1665build-time or runtime dependencies described in the software's
1666documentation.
1667
1668The following list provides configuration items of note based on how
1669your software is built:
1670
1671- *Autotools:* If your source files have a ``configure.ac`` file, then
1672 your software is built using Autotools. If this is the case, you just
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001673 need to modify the configuration.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001674
1675 When using Autotools, your recipe needs to inherit the
1676 :ref:`autotools <ref-classes-autotools>` class
1677 and your recipe does not have to contain a
1678 :ref:`ref-tasks-configure` task.
1679 However, you might still want to make some adjustments. For example,
1680 you can set
1681 :term:`EXTRA_OECONF` or
1682 :term:`PACKAGECONFIG_CONFARGS`
1683 to pass any needed configure options that are specific to the recipe.
1684
1685- *CMake:* If your source files have a ``CMakeLists.txt`` file, then
1686 your software is built using CMake. If this is the case, you just
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001687 need to modify the configuration.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001688
1689 When you use CMake, your recipe needs to inherit the
1690 :ref:`cmake <ref-classes-cmake>` class and your
1691 recipe does not have to contain a
1692 :ref:`ref-tasks-configure` task.
1693 You can make some adjustments by setting
1694 :term:`EXTRA_OECMAKE` to
1695 pass any needed configure options that are specific to the recipe.
1696
1697 .. note::
1698
1699 If you need to install one or more custom CMake toolchain files
1700 that are supplied by the application you are building, install the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001701 files to ``${D}${datadir}/cmake/Modules`` during ``do_install``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001702
1703- *Other:* If your source files do not have a ``configure.ac`` or
1704 ``CMakeLists.txt`` file, then your software is built using some
1705 method other than Autotools or CMake. If this is the case, you
1706 normally need to provide a
1707 :ref:`ref-tasks-configure` task
1708 in your recipe unless, of course, there is nothing to configure.
1709
1710 Even if your software is not being built by Autotools or CMake, you
1711 still might not need to deal with any configuration issues. You need
1712 to determine if configuration is even a required step. You might need
1713 to modify a Makefile or some configuration file used for the build to
1714 specify necessary build options. Or, perhaps you might need to run a
1715 provided, custom configure script with the appropriate options.
1716
1717 For the case involving a custom configure script, you would run
1718 ``./configure --help`` and look for the options you need to set.
1719
1720Once configuration succeeds, it is always good practice to look at the
1721``log.do_configure`` file to ensure that the appropriate options have
1722been enabled and no additional build-time dependencies need to be added
Andrew Geissler09036742021-06-25 14:25:14 -05001723to :term:`DEPENDS`. For example, if the configure script reports that it
1724found something not mentioned in :term:`DEPENDS`, or that it did not find
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001725something that it needed for some desired optional functionality, then
Andrew Geissler09036742021-06-25 14:25:14 -05001726you would need to add those to :term:`DEPENDS`. Looking at the log might
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001727also reveal items being checked for, enabled, or both that you do not
Andrew Geissler09036742021-06-25 14:25:14 -05001728want, or items not being found that are in :term:`DEPENDS`, in which case
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001729you would need to look at passing extra options to the configure script
1730as needed. For reference information on configure options specific to
1731the software you are building, you can consult the output of the
1732``./configure --help`` command within ``${S}`` or consult the software's
1733upstream documentation.
1734
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001735Using Headers to Interface with Devices
1736---------------------------------------
1737
1738If your recipe builds an application that needs to communicate with some
1739device or needs an API into a custom kernel, you will need to provide
1740appropriate header files. Under no circumstances should you ever modify
1741the existing
1742``meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc`` file.
1743These headers are used to build ``libc`` and must not be compromised
1744with custom or machine-specific header information. If you customize
1745``libc`` through modified headers all other applications that use
1746``libc`` thus become affected.
1747
1748.. note::
1749
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001750 Never copy and customize the ``libc`` header file (i.e.
1751 ``meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc``).
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001752
1753The correct way to interface to a device or custom kernel is to use a
1754separate package that provides the additional headers for the driver or
1755other unique interfaces. When doing so, your application also becomes
1756responsible for creating a dependency on that specific provider.
1757
1758Consider the following:
1759
1760- Never modify ``linux-libc-headers.inc``. Consider that file to be
1761 part of the ``libc`` system, and not something you use to access the
1762 kernel directly. You should access ``libc`` through specific ``libc``
1763 calls.
1764
1765- Applications that must talk directly to devices should either provide
1766 necessary headers themselves, or establish a dependency on a special
1767 headers package that is specific to that driver.
1768
1769For example, suppose you want to modify an existing header that adds I/O
1770control or network support. If the modifications are used by a small
1771number programs, providing a unique version of a header is easy and has
1772little impact. When doing so, bear in mind the guidelines in the
1773previous list.
1774
1775.. note::
1776
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001777 If for some reason your changes need to modify the behavior of the ``libc``,
1778 and subsequently all other applications on the system, use a ``.bbappend``
1779 to modify the ``linux-kernel-headers.inc`` file. However, take care to not
1780 make the changes machine specific.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001781
1782Consider a case where your kernel is older and you need an older
1783``libc`` ABI. The headers installed by your recipe should still be a
1784standard mainline kernel, not your own custom one.
1785
1786When you use custom kernel headers you need to get them from
1787:term:`STAGING_KERNEL_DIR`,
1788which is the directory with kernel headers that are required to build
Andrew Geisslerc926e172021-05-07 16:11:35 -05001789out-of-tree modules. Your recipe will also need the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001790
1791 do_configure[depends] += "virtual/kernel:do_shared_workdir"
1792
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001793Compilation
1794-----------
1795
1796During a build, the ``do_compile`` task happens after source is fetched,
1797unpacked, and configured. If the recipe passes through ``do_compile``
1798successfully, nothing needs to be done.
1799
1800However, if the compile step fails, you need to diagnose the failure.
1801Here are some common issues that cause failures.
1802
1803.. note::
1804
1805 For cases where improper paths are detected for configuration files
1806 or for when libraries/headers cannot be found, be sure you are using
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001807 the more robust ``pkg-config``. See the note in section
Andrew Geissler09209ee2020-12-13 08:44:15 -06001808 ":ref:`dev-manual/common-tasks:Configuring the Recipe`" for additional information.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001809
1810- *Parallel build failures:* These failures manifest themselves as
1811 intermittent errors, or errors reporting that a file or directory
1812 that should be created by some other part of the build process could
1813 not be found. This type of failure can occur even if, upon
1814 inspection, the file or directory does exist after the build has
1815 failed, because that part of the build process happened in the wrong
1816 order.
1817
1818 To fix the problem, you need to either satisfy the missing dependency
1819 in the Makefile or whatever script produced the Makefile, or (as a
Andrew Geisslerc926e172021-05-07 16:11:35 -05001820 workaround) set :term:`PARALLEL_MAKE` to an empty string::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001821
1822 PARALLEL_MAKE = ""
1823
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001824 For information on parallel Makefile issues, see the
1825 ":ref:`dev-manual/common-tasks:debugging parallel make races`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001826
1827- *Improper host path usage:* This failure applies to recipes building
1828 for the target or ``nativesdk`` only. The failure occurs when the
1829 compilation process uses improper headers, libraries, or other files
1830 from the host system when cross-compiling for the target.
1831
1832 To fix the problem, examine the ``log.do_compile`` file to identify
1833 the host paths being used (e.g. ``/usr/include``, ``/usr/lib``, and
1834 so forth) and then either add configure options, apply a patch, or do
1835 both.
1836
1837- *Failure to find required libraries/headers:* If a build-time
1838 dependency is missing because it has not been declared in
1839 :term:`DEPENDS`, or because the
1840 dependency exists but the path used by the build process to find the
1841 file is incorrect and the configure step did not detect it, the
1842 compilation process could fail. For either of these failures, the
1843 compilation process notes that files could not be found. In these
1844 cases, you need to go back and add additional options to the
1845 configure script as well as possibly add additional build-time
Andrew Geissler09036742021-06-25 14:25:14 -05001846 dependencies to :term:`DEPENDS`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001847
1848 Occasionally, it is necessary to apply a patch to the source to
1849 ensure the correct paths are used. If you need to specify paths to
1850 find files staged into the sysroot from other recipes, use the
1851 variables that the OpenEmbedded build system provides (e.g.
Andrew Geissler09036742021-06-25 14:25:14 -05001852 :term:`STAGING_BINDIR`, :term:`STAGING_INCDIR`, :term:`STAGING_DATADIR`, and so
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001853 forth).
1854
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001855Installing
1856----------
1857
1858During ``do_install``, the task copies the built files along with their
1859hierarchy to locations that would mirror their locations on the target
1860device. The installation process copies files from the
1861``${``\ :term:`S`\ ``}``,
1862``${``\ :term:`B`\ ``}``, and
1863``${``\ :term:`WORKDIR`\ ``}``
1864directories to the ``${``\ :term:`D`\ ``}``
1865directory to create the structure as it should appear on the target
1866system.
1867
1868How your software is built affects what you must do to be sure your
1869software is installed correctly. The following list describes what you
1870must do for installation depending on the type of build system used by
1871the software being built:
1872
1873- *Autotools and CMake:* If the software your recipe is building uses
1874 Autotools or CMake, the OpenEmbedded build system understands how to
1875 install the software. Consequently, you do not have to have a
1876 ``do_install`` task as part of your recipe. You just need to make
1877 sure the install portion of the build completes with no issues.
1878 However, if you wish to install additional files not already being
1879 installed by ``make install``, you should do this using a
Patrick Williams0ca19cc2021-08-16 14:03:13 -05001880 ``do_install:append`` function using the install command as described
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001881 in the "Manual" bulleted item later in this list.
1882
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001883- *Other (using* ``make install``\ *)*: You need to define a ``do_install``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001884 function in your recipe. The function should call
1885 ``oe_runmake install`` and will likely need to pass in the
1886 destination directory as well. How you pass that path is dependent on
1887 how the ``Makefile`` being run is written (e.g. ``DESTDIR=${D}``,
1888 ``PREFIX=${D}``, ``INSTALLROOT=${D}``, and so forth).
1889
1890 For an example recipe using ``make install``, see the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001891 ":ref:`dev-manual/common-tasks:makefile-based package`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001892
1893- *Manual:* You need to define a ``do_install`` function in your
1894 recipe. The function must first use ``install -d`` to create the
1895 directories under
1896 ``${``\ :term:`D`\ ``}``. Once the
1897 directories exist, your function can use ``install`` to manually
1898 install the built software into the directories.
1899
1900 You can find more information on ``install`` at
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001901 https://www.gnu.org/software/coreutils/manual/html_node/install-invocation.html.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001902
1903For the scenarios that do not use Autotools or CMake, you need to track
1904the installation and diagnose and fix any issues until everything
1905installs correctly. You need to look in the default location of
1906``${D}``, which is ``${WORKDIR}/image``, to be sure your files have been
1907installed correctly.
1908
1909.. note::
1910
1911 - During the installation process, you might need to modify some of
1912 the installed files to suit the target layout. For example, you
1913 might need to replace hard-coded paths in an initscript with
1914 values of variables provided by the build system, such as
1915 replacing ``/usr/bin/`` with ``${bindir}``. If you do perform such
1916 modifications during ``do_install``, be sure to modify the
1917 destination file after copying rather than before copying.
1918 Modifying after copying ensures that the build system can
1919 re-execute ``do_install`` if needed.
1920
1921 - ``oe_runmake install``, which can be run directly or can be run
1922 indirectly by the
1923 :ref:`autotools <ref-classes-autotools>` and
1924 :ref:`cmake <ref-classes-cmake>` classes,
1925 runs ``make install`` in parallel. Sometimes, a Makefile can have
1926 missing dependencies between targets that can result in race
1927 conditions. If you experience intermittent failures during
1928 ``do_install``, you might be able to work around them by disabling
Andrew Geisslerc926e172021-05-07 16:11:35 -05001929 parallel Makefile installs by adding the following to the recipe::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001930
1931 PARALLEL_MAKEINST = ""
1932
1933 See :term:`PARALLEL_MAKEINST` for additional information.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001934
1935 - If you need to install one or more custom CMake toolchain files
1936 that are supplied by the application you are building, install the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001937 files to ``${D}${datadir}/cmake/Modules`` during
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001938 :ref:`ref-tasks-install`.
1939
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001940Enabling System Services
1941------------------------
1942
1943If you want to install a service, which is a process that usually starts
1944on boot and runs in the background, then you must include some
1945additional definitions in your recipe.
1946
1947If you are adding services and the service initialization script or the
1948service file itself is not installed, you must provide for that
Patrick Williams0ca19cc2021-08-16 14:03:13 -05001949installation in your recipe using a ``do_install:append`` function. If
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001950your recipe already has a ``do_install`` function, update the function
Patrick Williams0ca19cc2021-08-16 14:03:13 -05001951near its end rather than adding an additional ``do_install:append``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001952function.
1953
1954When you create the installation for your services, you need to
1955accomplish what is normally done by ``make install``. In other words,
1956make sure your installation arranges the output similar to how it is
1957arranged on the target system.
1958
1959The OpenEmbedded build system provides support for starting services two
1960different ways:
1961
1962- *SysVinit:* SysVinit is a system and service manager that manages the
1963 init system used to control the very basic functions of your system.
1964 The init program is the first program started by the Linux kernel
1965 when the system boots. Init then controls the startup, running and
1966 shutdown of all other programs.
1967
1968 To enable a service using SysVinit, your recipe needs to inherit the
1969 :ref:`update-rc.d <ref-classes-update-rc.d>`
1970 class. The class helps facilitate safely installing the package on
1971 the target.
1972
1973 You will need to set the
1974 :term:`INITSCRIPT_PACKAGES`,
1975 :term:`INITSCRIPT_NAME`,
1976 and
1977 :term:`INITSCRIPT_PARAMS`
1978 variables within your recipe.
1979
1980- *systemd:* System Management Daemon (systemd) was designed to replace
1981 SysVinit and to provide enhanced management of services. For more
1982 information on systemd, see the systemd homepage at
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001983 https://freedesktop.org/wiki/Software/systemd/.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001984
1985 To enable a service using systemd, your recipe needs to inherit the
1986 :ref:`systemd <ref-classes-systemd>` class. See
1987 the ``systemd.bbclass`` file located in your :term:`Source Directory`
1988 section for
1989 more information.
1990
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001991Packaging
1992---------
1993
1994Successful packaging is a combination of automated processes performed
1995by the OpenEmbedded build system and some specific steps you need to
1996take. The following list describes the process:
1997
1998- *Splitting Files*: The ``do_package`` task splits the files produced
1999 by the recipe into logical components. Even software that produces a
2000 single binary might still have debug symbols, documentation, and
2001 other logical components that should be split out. The ``do_package``
2002 task ensures that files are split up and packaged correctly.
2003
2004- *Running QA Checks*: The
2005 :ref:`insane <ref-classes-insane>` class adds a
2006 step to the package generation process so that output quality
2007 assurance checks are generated by the OpenEmbedded build system. This
2008 step performs a range of checks to be sure the build's output is free
2009 of common problems that show up during runtime. For information on
2010 these checks, see the
2011 :ref:`insane <ref-classes-insane>` class and
Andrew Geissler09209ee2020-12-13 08:44:15 -06002012 the ":ref:`ref-manual/qa-checks:qa error and warning messages`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002013 chapter in the Yocto Project Reference Manual.
2014
2015- *Hand-Checking Your Packages*: After you build your software, you
2016 need to be sure your packages are correct. Examine the
2017 ``${``\ :term:`WORKDIR`\ ``}/packages-split``
2018 directory and make sure files are where you expect them to be. If you
2019 discover problems, you can set
2020 :term:`PACKAGES`,
2021 :term:`FILES`,
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002022 ``do_install(:append)``, and so forth as needed.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002023
2024- *Splitting an Application into Multiple Packages*: If you need to
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05002025 split an application into several packages, see the
2026 ":ref:`dev-manual/common-tasks:splitting an application into multiple packages`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002027 section for an example.
2028
2029- *Installing a Post-Installation Script*: For an example showing how
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05002030 to install a post-installation script, see the
2031 ":ref:`dev-manual/common-tasks:post-installation scripts`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002032
2033- *Marking Package Architecture*: Depending on what your recipe is
2034 building and how it is configured, it might be important to mark the
2035 packages produced as being specific to a particular machine, or to
2036 mark them as not being specific to a particular machine or
2037 architecture at all.
2038
2039 By default, packages apply to any machine with the same architecture
2040 as the target machine. When a recipe produces packages that are
2041 machine-specific (e.g. the
2042 :term:`MACHINE` value is passed
2043 into the configure script or a patch is applied only for a particular
2044 machine), you should mark them as such by adding the following to the
Andrew Geisslerc926e172021-05-07 16:11:35 -05002045 recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002046
2047 PACKAGE_ARCH = "${MACHINE_ARCH}"
2048
2049 On the other hand, if the recipe produces packages that do not
2050 contain anything specific to the target machine or architecture at
2051 all (e.g. recipes that simply package script files or configuration
2052 files), you should use the
2053 :ref:`allarch <ref-classes-allarch>` class to
Andrew Geisslerc926e172021-05-07 16:11:35 -05002054 do this for you by adding this to your recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002055
2056 inherit allarch
2057
2058 Ensuring that the package architecture is correct is not critical
2059 while you are doing the first few builds of your recipe. However, it
2060 is important in order to ensure that your recipe rebuilds (or does
2061 not rebuild) appropriately in response to changes in configuration,
2062 and to ensure that you get the appropriate packages installed on the
2063 target machine, particularly if you run separate builds for more than
2064 one target machine.
2065
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002066Sharing Files Between Recipes
2067-----------------------------
2068
2069Recipes often need to use files provided by other recipes on the build
2070host. For example, an application linking to a common library needs
2071access to the library itself and its associated headers. The way this
2072access is accomplished is by populating a sysroot with files. Each
2073recipe has two sysroots in its work directory, one for target files
2074(``recipe-sysroot``) and one for files that are native to the build host
2075(``recipe-sysroot-native``).
2076
2077.. note::
2078
2079 You could find the term "staging" used within the Yocto project
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002080 regarding files populating sysroots (e.g. the :term:`STAGING_DIR`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002081 variable).
2082
2083Recipes should never populate the sysroot directly (i.e. write files
2084into sysroot). Instead, files should be installed into standard
2085locations during the
2086:ref:`ref-tasks-install` task within
2087the ``${``\ :term:`D`\ ``}`` directory. The
2088reason for this limitation is that almost all files that populate the
2089sysroot are cataloged in manifests in order to ensure the files can be
2090removed later when a recipe is either modified or removed. Thus, the
2091sysroot is able to remain free from stale files.
2092
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05002093A subset of the files installed by the :ref:`ref-tasks-install` task are
2094used by the :ref:`ref-tasks-populate_sysroot` task as defined by the the
2095:term:`SYSROOT_DIRS` variable to automatically populate the sysroot. It
2096is possible to modify the list of directories that populate the sysroot.
2097The following example shows how you could add the ``/opt`` directory to
Andrew Geisslerc926e172021-05-07 16:11:35 -05002098the list of directories within a recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002099
2100 SYSROOT_DIRS += "/opt"
2101
Andrew Geisslerd1e89492021-02-12 15:35:20 -06002102.. note::
2103
2104 The `/sysroot-only` is to be used by recipes that generate artifacts
2105 that are not included in the target filesystem, allowing them to share
Andrew Geissler09036742021-06-25 14:25:14 -05002106 these artifacts without needing to use the :term:`DEPLOY_DIR`.
Andrew Geisslerd1e89492021-02-12 15:35:20 -06002107
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05002108For a more complete description of the :ref:`ref-tasks-populate_sysroot`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002109task and its associated functions, see the
2110:ref:`staging <ref-classes-staging>` class.
2111
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002112Using Virtual Providers
2113-----------------------
2114
2115Prior to a build, if you know that several different recipes provide the
2116same functionality, you can use a virtual provider (i.e. ``virtual/*``)
2117as a placeholder for the actual provider. The actual provider is
2118determined at build-time.
2119
2120A common scenario where a virtual provider is used would be for the
2121kernel recipe. Suppose you have three kernel recipes whose
2122:term:`PN` values map to ``kernel-big``,
2123``kernel-mid``, and ``kernel-small``. Furthermore, each of these recipes
2124in some way uses a :term:`PROVIDES`
2125statement that essentially identifies itself as being able to provide
2126``virtual/kernel``. Here is one way through the
Andrew Geisslerc926e172021-05-07 16:11:35 -05002127:ref:`kernel <ref-classes-kernel>` class::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002128
Andrew Geissler7e0e3c02022-02-25 20:34:39 +00002129 PROVIDES += "virtual/kernel"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002130
Andrew Geissler595f6302022-01-24 19:11:47 +00002131Any recipe that inherits the :ref:`kernel <ref-classes-kernel>` class is
Andrew Geissler09036742021-06-25 14:25:14 -05002132going to utilize a :term:`PROVIDES` statement that identifies that recipe as
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002133being able to provide the ``virtual/kernel`` item.
2134
2135Now comes the time to actually build an image and you need a kernel
2136recipe, but which one? You can configure your build to call out the
Andrew Geissler09209ee2020-12-13 08:44:15 -06002137kernel recipe you want by using the :term:`PREFERRED_PROVIDER` variable. As
2138an example, consider the :yocto_git:`x86-base.inc
Andrew Geisslerd159c7f2021-09-02 21:05:58 -05002139</poky/tree/meta/conf/machine/include/x86/x86-base.inc>` include file, which is a
Andrew Geissler09209ee2020-12-13 08:44:15 -06002140machine (i.e. :term:`MACHINE`) configuration file. This include file is the
2141reason all x86-based machines use the ``linux-yocto`` kernel. Here are the
Andrew Geisslerc926e172021-05-07 16:11:35 -05002142relevant lines from the include file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002143
2144 PREFERRED_PROVIDER_virtual/kernel ??= "linux-yocto"
2145 PREFERRED_VERSION_linux-yocto ??= "4.15%"
2146
2147When you use a virtual provider, you do not have to "hard code" a recipe
2148name as a build dependency. You can use the
2149:term:`DEPENDS` variable to state the
Andrew Geisslerc926e172021-05-07 16:11:35 -05002150build is dependent on ``virtual/kernel`` for example::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002151
2152 DEPENDS = "virtual/kernel"
2153
2154During the build, the OpenEmbedded build system picks
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002155the correct recipe needed for the ``virtual/kernel`` dependency based on
Andrew Geissler09036742021-06-25 14:25:14 -05002156the :term:`PREFERRED_PROVIDER` variable. If you want to use the small kernel
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002157mentioned at the beginning of this section, configure your build as
Andrew Geisslerc926e172021-05-07 16:11:35 -05002158follows::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002159
2160 PREFERRED_PROVIDER_virtual/kernel ??= "kernel-small"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002161
2162.. note::
2163
Andrew Geissler09036742021-06-25 14:25:14 -05002164 Any recipe that :term:`PROVIDES` a ``virtual/*`` item that is ultimately not
2165 selected through :term:`PREFERRED_PROVIDER` does not get built. Preventing these
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002166 recipes from building is usually the desired behavior since this mechanism's
2167 purpose is to select between mutually exclusive alternative providers.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002168
2169The following lists specific examples of virtual providers:
2170
2171- ``virtual/kernel``: Provides the name of the kernel recipe to use
2172 when building a kernel image.
2173
2174- ``virtual/bootloader``: Provides the name of the bootloader to use
2175 when building an image.
2176
2177- ``virtual/libgbm``: Provides ``gbm.pc``.
2178
2179- ``virtual/egl``: Provides ``egl.pc`` and possibly ``wayland-egl.pc``.
2180
2181- ``virtual/libgl``: Provides ``gl.pc`` (i.e. libGL).
2182
2183- ``virtual/libgles1``: Provides ``glesv1_cm.pc`` (i.e. libGLESv1_CM).
2184
2185- ``virtual/libgles2``: Provides ``glesv2.pc`` (i.e. libGLESv2).
2186
2187.. note::
2188
2189 Virtual providers only apply to build time dependencies specified with
2190 :term:`PROVIDES` and :term:`DEPENDS`. They do not apply to runtime
2191 dependencies specified with :term:`RPROVIDES` and :term:`RDEPENDS`.
2192
2193Properly Versioning Pre-Release Recipes
2194---------------------------------------
2195
2196Sometimes the name of a recipe can lead to versioning problems when the
2197recipe is upgraded to a final release. For example, consider the
2198``irssi_0.8.16-rc1.bb`` recipe file in the list of example recipes in
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05002199the ":ref:`dev-manual/common-tasks:storing and naming the recipe`" section.
2200This recipe is at a release candidate stage (i.e. "rc1"). When the recipe is
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002201released, the recipe filename becomes ``irssi_0.8.16.bb``. The version
2202change from ``0.8.16-rc1`` to ``0.8.16`` is seen as a decrease by the
2203build system and package managers, so the resulting packages will not
2204correctly trigger an upgrade.
2205
2206In order to ensure the versions compare properly, the recommended
2207convention is to set :term:`PV` within the
2208recipe to "previous_version+current_version". You can use an additional
2209variable so that you can use the current version elsewhere. Here is an
Andrew Geisslerc926e172021-05-07 16:11:35 -05002210example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002211
2212 REALPV = "0.8.16-rc1"
2213 PV = "0.8.15+${REALPV}"
2214
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002215Post-Installation Scripts
2216-------------------------
2217
2218Post-installation scripts run immediately after installing a package on
2219the target or during image creation when a package is included in an
2220image. To add a post-installation script to a package, add a
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002221``pkg_postinst:``\ `PACKAGENAME`\ ``()`` function to the recipe file
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002222(``.bb``) and replace `PACKAGENAME` with the name of the package you want
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002223to attach to the ``postinst`` script. To apply the post-installation
2224script to the main package for the recipe, which is usually what is
2225required, specify
2226``${``\ :term:`PN`\ ``}`` in place of
2227PACKAGENAME.
2228
Andrew Geisslerc926e172021-05-07 16:11:35 -05002229A post-installation function has the following structure::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002230
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002231 pkg_postinst:PACKAGENAME() {
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002232 # Commands to carry out
2233 }
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002234
2235The script defined in the post-installation function is called when the
2236root filesystem is created. If the script succeeds, the package is
2237marked as installed.
2238
2239.. note::
2240
2241 Any RPM post-installation script that runs on the target should
2242 return a 0 exit code. RPM does not allow non-zero exit codes for
2243 these scripts, and the RPM package manager will cause the package to
2244 fail installation on the target.
2245
2246Sometimes it is necessary for the execution of a post-installation
2247script to be delayed until the first boot. For example, the script might
2248need to be executed on the device itself. To delay script execution
2249until boot time, you must explicitly mark post installs to defer to the
2250target. You can use ``pkg_postinst_ontarget()`` or call
2251``postinst_intercept delay_to_first_boot`` from ``pkg_postinst()``. Any
2252failure of a ``pkg_postinst()`` script (including exit 1) triggers an
2253error during the
2254:ref:`ref-tasks-rootfs` task.
2255
2256If you have recipes that use ``pkg_postinst`` function and they require
2257the use of non-standard native tools that have dependencies during
Andrew Geissler595f6302022-01-24 19:11:47 +00002258root filesystem construction, you need to use the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002259:term:`PACKAGE_WRITE_DEPS`
2260variable in your recipe to list these tools. If you do not use this
2261variable, the tools might be missing and execution of the
2262post-installation script is deferred until first boot. Deferring the
Andrew Geissler595f6302022-01-24 19:11:47 +00002263script to the first boot is undesirable and impossible for read-only
2264root filesystems.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002265
2266.. note::
2267
William A. Kennington IIIac69b482021-06-02 12:28:27 -07002268 There is equivalent support for pre-install, pre-uninstall, and post-uninstall
2269 scripts by way of ``pkg_preinst``, ``pkg_prerm``, and ``pkg_postrm``,
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002270 respectively. These scrips work in exactly the same way as does
2271 ``pkg_postinst`` with the exception that they run at different times. Also,
2272 because of when they run, they are not applicable to being run at image
2273 creation time like ``pkg_postinst``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002274
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002275Testing
2276-------
2277
2278The final step for completing your recipe is to be sure that the
2279software you built runs correctly. To accomplish runtime testing, add
2280the build's output packages to your image and test them on the target.
2281
2282For information on how to customize your image by adding specific
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05002283packages, see ":ref:`dev-manual/common-tasks:customizing images`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002284
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002285Examples
2286--------
2287
2288To help summarize how to write a recipe, this section provides some
2289examples given various scenarios:
2290
2291- Recipes that use local files
2292
2293- Using an Autotooled package
2294
2295- Using a Makefile-based package
2296
2297- Splitting an application into multiple packages
2298
2299- Adding binaries to an image
2300
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002301Single .c File Package (Hello World!)
2302~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2303
2304Building an application from a single file that is stored locally (e.g.
2305under ``files``) requires a recipe that has the file listed in the
Andrew Geissler09036742021-06-25 14:25:14 -05002306:term:`SRC_URI` variable. Additionally, you need to manually write the
2307``do_compile`` and ``do_install`` tasks. The :term:`S` variable defines the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002308directory containing the source code, which is set to
Andrew Geissler615f2f12022-07-15 14:00:58 -05002309:term:`WORKDIR` in this case --- the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002310directory BitBake uses for the build.
2311::
2312
2313 SUMMARY = "Simple helloworld application"
2314 SECTION = "examples"
2315 LICENSE = "MIT"
2316 LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
2317
2318 SRC_URI = "file://helloworld.c"
2319
2320 S = "${WORKDIR}"
2321
2322 do_compile() {
Andrew Geisslerd1e89492021-02-12 15:35:20 -06002323 ${CC} ${LDFLAGS} helloworld.c -o helloworld
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002324 }
2325
2326 do_install() {
2327 install -d ${D}${bindir}
2328 install -m 0755 helloworld ${D}${bindir}
2329 }
2330
2331By default, the ``helloworld``, ``helloworld-dbg``, and
2332``helloworld-dev`` packages are built. For information on how to
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05002333customize the packaging process, see the
2334":ref:`dev-manual/common-tasks:splitting an application into multiple packages`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002335section.
2336
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002337Autotooled Package
2338~~~~~~~~~~~~~~~~~~
2339
2340Applications that use Autotools such as ``autoconf`` and ``automake``
Andrew Geissler09036742021-06-25 14:25:14 -05002341require a recipe that has a source archive listed in :term:`SRC_URI` and
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002342also inherit the
2343:ref:`autotools <ref-classes-autotools>` class,
2344which contains the definitions of all the steps needed to build an
2345Autotool-based application. The result of the build is automatically
2346packaged. And, if the application uses NLS for localization, packages
2347with local information are generated (one package per language).
2348Following is one example: (``hello_2.3.bb``)
2349::
2350
2351 SUMMARY = "GNU Helloworld application"
2352 SECTION = "examples"
Andrew Geissler9aee5002022-03-30 16:27:02 +00002353 LICENSE = "GPL-2.0-or-later"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002354 LIC_FILES_CHKSUM = "file://COPYING;md5=751419260aa954499f7abaabaa882bbe"
2355
2356 SRC_URI = "${GNU_MIRROR}/hello/hello-${PV}.tar.gz"
2357
2358 inherit autotools gettext
2359
Andrew Geissler09036742021-06-25 14:25:14 -05002360The variable :term:`LIC_FILES_CHKSUM` is used to track source license
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002361changes as described in the
Andrew Geissler09209ee2020-12-13 08:44:15 -06002362":ref:`dev-manual/common-tasks:tracking license changes`" section in
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002363the Yocto Project Overview and Concepts Manual. You can quickly create
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002364Autotool-based recipes in a manner similar to the previous example.
2365
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002366Makefile-Based Package
2367~~~~~~~~~~~~~~~~~~~~~~
2368
2369Applications that use GNU ``make`` also require a recipe that has the
Andrew Geissler09036742021-06-25 14:25:14 -05002370source archive listed in :term:`SRC_URI`. You do not need to add a
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002371``do_compile`` step since by default BitBake starts the ``make`` command
2372to compile the application. If you need additional ``make`` options, you
2373should store them in the
2374:term:`EXTRA_OEMAKE` or
2375:term:`PACKAGECONFIG_CONFARGS`
2376variables. BitBake passes these options into the GNU ``make``
2377invocation. Note that a ``do_install`` task is still required.
2378Otherwise, BitBake runs an empty ``do_install`` task by default.
2379
2380Some applications might require extra parameters to be passed to the
2381compiler. For example, the application might need an additional header
Andrew Geissler09036742021-06-25 14:25:14 -05002382path. You can accomplish this by adding to the :term:`CFLAGS` variable. The
Andrew Geisslerc926e172021-05-07 16:11:35 -05002383following example shows this::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002384
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002385 CFLAGS:prepend = "-I ${S}/include "
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002386
Andrew Geissler9aee5002022-03-30 16:27:02 +00002387In the following example, ``lz4`` is a makefile-based package::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002388
Andrew Geissler9aee5002022-03-30 16:27:02 +00002389 SUMMARY = "Extremely Fast Compression algorithm"
2390 DESCRIPTION = "LZ4 is a very fast lossless compression algorithm, providing compression speed at 400 MB/s per core, scalable with multi-cores CPU. It also features an extremely fast decoder, with speed in multiple GB/s per core, typically reaching RAM speed limits on multi-core systems."
2391 HOMEPAGE = "https://github.com/lz4/lz4"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002392
Andrew Geissler9aee5002022-03-30 16:27:02 +00002393 LICENSE = "BSD-2-Clause | GPL-2.0-only"
2394 LIC_FILES_CHKSUM = "file://lib/LICENSE;md5=ebc2ea4814a64de7708f1571904b32cc \
2395 file://programs/COPYING;md5=b234ee4d69f5fce4486a80fdaf4a4263 \
2396 file://LICENSE;md5=d57c0d21cb917fb4e0af2454aa48b956 \
2397 "
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002398
Andrew Geissler9aee5002022-03-30 16:27:02 +00002399 PE = "1"
2400
2401 SRCREV = "d44371841a2f1728a3f36839fd4b7e872d0927d3"
2402
2403 SRC_URI = "git://github.com/lz4/lz4.git;branch=release;protocol=https \
2404 file://CVE-2021-3520.patch \
2405 "
2406 UPSTREAM_CHECK_GITTAGREGEX = "v(?P<pver>.*)"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002407
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002408 S = "${WORKDIR}/git"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002409
Andrew Geissler9aee5002022-03-30 16:27:02 +00002410 # Fixed in r118, which is larger than the current version.
2411 CVE_CHECK_IGNORE += "CVE-2014-4715"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002412
Andrew Geissler9aee5002022-03-30 16:27:02 +00002413 EXTRA_OEMAKE = "PREFIX=${prefix} CC='${CC}' CFLAGS='${CFLAGS}' DESTDIR=${D} LIBDIR=${libdir} INCLUDEDIR=${includedir} BUILD_STATIC=no"
2414
2415 do_install() {
2416 oe_runmake install
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002417 }
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002418
Andrew Geissler9aee5002022-03-30 16:27:02 +00002419 BBCLASSEXTEND = "native nativesdk"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002420
2421Splitting an Application into Multiple Packages
2422~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2423
Andrew Geissler09036742021-06-25 14:25:14 -05002424You can use the variables :term:`PACKAGES` and :term:`FILES` to split an
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002425application into multiple packages.
2426
2427Following is an example that uses the ``libxpm`` recipe. By default,
2428this recipe generates a single package that contains the library along
2429with a few binaries. You can modify the recipe to split the binaries
Andrew Geisslerc926e172021-05-07 16:11:35 -05002430into separate packages::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002431
2432 require xorg-lib-common.inc
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002433
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002434 SUMMARY = "Xpm: X Pixmap extension library"
Andrew Geissler5199d832021-09-24 16:47:35 -05002435 LICENSE = "MIT"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002436 LIC_FILES_CHKSUM = "file://COPYING;md5=51f4270b012ecd4ab1a164f5f4ed6cf7"
2437 DEPENDS += "libxext libsm libxt"
2438 PE = "1"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002439
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002440 XORG_PN = "libXpm"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002441
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002442 PACKAGES =+ "sxpm cxpm"
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002443 FILES:cxpm = "${bindir}/cxpm"
2444 FILES:sxpm = "${bindir}/sxpm"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002445
2446In the previous example, we want to ship the ``sxpm`` and ``cxpm``
2447binaries in separate packages. Since ``bindir`` would be packaged into
Andrew Geissler09036742021-06-25 14:25:14 -05002448the main :term:`PN` package by default, we prepend the :term:`PACKAGES` variable
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002449so additional package names are added to the start of list. This results
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002450in the extra ``FILES:*`` variables then containing information that
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002451define which files and directories go into which packages. Files
2452included by earlier packages are skipped by latter packages. Thus, the
Andrew Geissler09036742021-06-25 14:25:14 -05002453main :term:`PN` package does not include the above listed files.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002454
2455Packaging Externally Produced Binaries
2456~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2457
2458Sometimes, you need to add pre-compiled binaries to an image. For
William A. Kennington IIIac69b482021-06-02 12:28:27 -07002459example, suppose that there are binaries for proprietary code,
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002460created by a particular division of a company. Your part of the company
2461needs to use those binaries as part of an image that you are building
2462using the OpenEmbedded build system. Since you only have the binaries
2463and not the source code, you cannot use a typical recipe that expects to
2464fetch the source specified in
2465:term:`SRC_URI` and then compile it.
2466
2467One method is to package the binaries and then install them as part of
2468the image. Generally, it is not a good idea to package binaries since,
2469among other things, it can hinder the ability to reproduce builds and
2470could lead to compatibility problems with ABI in the future. However,
2471sometimes you have no choice.
2472
2473The easiest solution is to create a recipe that uses the
2474:ref:`bin_package <ref-classes-bin-package>` class
2475and to be sure that you are using default locations for build artifacts.
Andrew Geissler595f6302022-01-24 19:11:47 +00002476In most cases, the :ref:`bin_package <ref-classes-bin-package>` class handles "skipping" the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002477configure and compile steps as well as sets things up to grab packages
2478from the appropriate area. In particular, this class sets ``noexec`` on
2479both the :ref:`ref-tasks-configure`
2480and :ref:`ref-tasks-compile` tasks,
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002481sets ``FILES:${PN}`` to "/" so that it picks up all files, and sets up a
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002482:ref:`ref-tasks-install` task, which
2483effectively copies all files from ``${S}`` to ``${D}``. The
Andrew Geissler595f6302022-01-24 19:11:47 +00002484:ref:`bin_package <ref-classes-bin-package>` class works well when the files extracted into ``${S}``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002485are already laid out in the way they should be laid out on the target.
2486For more information on these variables, see the
2487:term:`FILES`,
2488:term:`PN`,
2489:term:`S`, and
2490:term:`D` variables in the Yocto Project
2491Reference Manual's variable glossary.
2492
2493.. note::
2494
2495 - Using :term:`DEPENDS` is a good
2496 idea even for components distributed in binary form, and is often
2497 necessary for shared libraries. For a shared library, listing the
Andrew Geissler09036742021-06-25 14:25:14 -05002498 library dependencies in :term:`DEPENDS` makes sure that the libraries
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002499 are available in the staging sysroot when other recipes link
2500 against the library, which might be necessary for successful
2501 linking.
2502
Andrew Geissler09036742021-06-25 14:25:14 -05002503 - Using :term:`DEPENDS` also allows runtime dependencies between
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002504 packages to be added automatically. See the
Andrew Geissler09209ee2020-12-13 08:44:15 -06002505 ":ref:`overview-manual/concepts:automatically added runtime dependencies`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002506 section in the Yocto Project Overview and Concepts Manual for more
2507 information.
2508
Andrew Geissler595f6302022-01-24 19:11:47 +00002509If you cannot use the :ref:`bin_package <ref-classes-bin-package>` class, you need to be sure you are
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002510doing the following:
2511
2512- Create a recipe where the
2513 :ref:`ref-tasks-configure` and
2514 :ref:`ref-tasks-compile` tasks do
2515 nothing: It is usually sufficient to just not define these tasks in
2516 the recipe, because the default implementations do nothing unless a
2517 Makefile is found in
2518 ``${``\ :term:`S`\ ``}``.
2519
2520 If ``${S}`` might contain a Makefile, or if you inherit some class
2521 that replaces ``do_configure`` and ``do_compile`` with custom
2522 versions, then you can use the
2523 ``[``\ :ref:`noexec <bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ``]``
Andrew Geisslerc926e172021-05-07 16:11:35 -05002524 flag to turn the tasks into no-ops, as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002525
2526 do_configure[noexec] = "1"
2527 do_compile[noexec] = "1"
2528
2529 Unlike
2530 :ref:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata:deleting a task`,
2531 using the flag preserves the dependency chain from the
2532 :ref:`ref-tasks-fetch`,
2533 :ref:`ref-tasks-unpack`, and
2534 :ref:`ref-tasks-patch` tasks to the
2535 :ref:`ref-tasks-install` task.
2536
2537- Make sure your ``do_install`` task installs the binaries
2538 appropriately.
2539
2540- Ensure that you set up :term:`FILES`
2541 (usually
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002542 ``FILES:${``\ :term:`PN`\ ``}``) to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002543 point to the files you have installed, which of course depends on
2544 where you have installed them and whether those files are in
2545 different locations than the defaults.
2546
2547Following Recipe Style Guidelines
2548---------------------------------
2549
2550When writing recipes, it is good to conform to existing style
Andrew Geisslerd1e89492021-02-12 15:35:20 -06002551guidelines. The :oe_wiki:`OpenEmbedded Styleguide </Styleguide>` wiki page
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002552provides rough guidelines for preferred recipe style.
2553
2554It is common for existing recipes to deviate a bit from this style.
2555However, aiming for at least a consistent style is a good idea. Some
2556practices, such as omitting spaces around ``=`` operators in assignments
2557or ordering recipe components in an erratic way, are widely seen as poor
2558style.
2559
2560Recipe Syntax
2561-------------
2562
2563Understanding recipe file syntax is important for writing recipes. The
2564following list overviews the basic items that make up a BitBake recipe
2565file. For more complete BitBake syntax descriptions, see the
Andrew Geissler615f2f12022-07-15 14:00:58 -05002566":doc:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002567chapter of the BitBake User Manual.
2568
2569- *Variable Assignments and Manipulations:* Variable assignments allow
2570 a value to be assigned to a variable. The assignment can be static
2571 text or might include the contents of other variables. In addition to
2572 the assignment, appending and prepending operations are also
2573 supported.
2574
2575 The following example shows some of the ways you can use variables in
Andrew Geisslerc926e172021-05-07 16:11:35 -05002576 recipes::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002577
2578 S = "${WORKDIR}/postfix-${PV}"
2579 CFLAGS += "-DNO_ASM"
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002580 SRC_URI:append = " file://fixup.patch"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002581
2582- *Functions:* Functions provide a series of actions to be performed.
2583 You usually use functions to override the default implementation of a
2584 task function or to complement a default function (i.e. append or
2585 prepend to an existing function). Standard functions use ``sh`` shell
2586 syntax, although access to OpenEmbedded variables and internal
2587 methods are also available.
2588
William A. Kennington IIIac69b482021-06-02 12:28:27 -07002589 Here is an example function from the ``sed`` recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002590
2591 do_install () {
2592 autotools_do_install
2593 install -d ${D}${base_bindir}
2594 mv ${D}${bindir}/sed ${D}${base_bindir}/sed
2595 rmdir ${D}${bindir}/
2596 }
2597
2598 It is
2599 also possible to implement new functions that are called between
2600 existing tasks as long as the new functions are not replacing or
2601 complementing the default functions. You can implement functions in
2602 Python instead of shell. Both of these options are not seen in the
2603 majority of recipes.
2604
2605- *Keywords:* BitBake recipes use only a few keywords. You use keywords
2606 to include common functions (``inherit``), load parts of a recipe
2607 from other files (``include`` and ``require``) and export variables
2608 to the environment (``export``).
2609
Andrew Geisslerc926e172021-05-07 16:11:35 -05002610 The following example shows the use of some of these keywords::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002611
2612 export POSTCONF = "${STAGING_BINDIR}/postconf"
2613 inherit autoconf
2614 require otherfile.inc
2615
2616- *Comments (#):* Any lines that begin with the hash character (``#``)
Andrew Geisslerc926e172021-05-07 16:11:35 -05002617 are treated as comment lines and are ignored::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002618
2619 # This is a comment
2620
2621This next list summarizes the most important and most commonly used
2622parts of the recipe syntax. For more information on these parts of the
2623syntax, you can reference the
Andrew Geissler615f2f12022-07-15 14:00:58 -05002624":doc:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata`" chapter
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002625in the BitBake User Manual.
2626
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002627- *Line Continuation (\\):* Use the backward slash (``\``) character to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002628 split a statement over multiple lines. Place the slash character at
Andrew Geisslerc926e172021-05-07 16:11:35 -05002629 the end of the line that is to be continued on the next line::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002630
2631 VAR = "A really long \
2632 line"
2633
2634 .. note::
2635
2636 You cannot have any characters including spaces or tabs after the
2637 slash character.
2638
2639- *Using Variables (${VARNAME}):* Use the ``${VARNAME}`` syntax to
Andrew Geisslerc926e172021-05-07 16:11:35 -05002640 access the contents of a variable::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002641
2642 SRC_URI = "${SOURCEFORGE_MIRROR}/libpng/zlib-${PV}.tar.gz"
2643
2644 .. note::
2645
2646 It is important to understand that the value of a variable
2647 expressed in this form does not get substituted automatically. The
2648 expansion of these expressions happens on-demand later (e.g.
2649 usually when a function that makes reference to the variable
2650 executes). This behavior ensures that the values are most
2651 appropriate for the context in which they are finally used. On the
2652 rare occasion that you do need the variable expression to be
2653 expanded immediately, you can use the
2654 :=
2655 operator instead of
2656 =
2657 when you make the assignment, but this is not generally needed.
2658
2659- *Quote All Assignments ("value"):* Use double quotes around values in
Andrew Geisslerc926e172021-05-07 16:11:35 -05002660 all variable assignments (e.g. ``"value"``). Following is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002661
2662 VAR1 = "${OTHERVAR}"
2663 VAR2 = "The version is ${PV}"
2664
2665- *Conditional Assignment (?=):* Conditional assignment is used to
2666 assign a value to a variable, but only when the variable is currently
2667 unset. Use the question mark followed by the equal sign (``?=``) to
2668 make a "soft" assignment used for conditional assignment. Typically,
2669 "soft" assignments are used in the ``local.conf`` file for variables
2670 that are allowed to come through from the external environment.
2671
2672 Here is an example where ``VAR1`` is set to "New value" if it is
2673 currently empty. However, if ``VAR1`` has already been set, it
Andrew Geisslerc926e172021-05-07 16:11:35 -05002674 remains unchanged::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002675
2676 VAR1 ?= "New value"
2677
Andrew Geisslerc926e172021-05-07 16:11:35 -05002678 In this next example, ``VAR1`` is left with the value "Original value"::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002679
2680 VAR1 = "Original value"
2681 VAR1 ?= "New value"
2682
2683- *Appending (+=):* Use the plus character followed by the equals sign
2684 (``+=``) to append values to existing variables.
2685
2686 .. note::
2687
2688 This operator adds a space between the existing content of the
2689 variable and the new content.
2690
Andrew Geisslerc926e172021-05-07 16:11:35 -05002691 Here is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002692
2693 SRC_URI += "file://fix-makefile.patch"
2694
2695- *Prepending (=+):* Use the equals sign followed by the plus character
2696 (``=+``) to prepend values to existing variables.
2697
2698 .. note::
2699
2700 This operator adds a space between the new content and the
2701 existing content of the variable.
2702
Andrew Geisslerc926e172021-05-07 16:11:35 -05002703 Here is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002704
2705 VAR =+ "Starts"
2706
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002707- *Appending (:append):* Use the ``:append`` operator to append values
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002708 to existing variables. This operator does not add any additional
2709 space. Also, the operator is applied after all the ``+=``, and ``=+``
2710 operators have been applied and after all ``=`` assignments have
2711 occurred.
2712
2713 The following example shows the space being explicitly added to the
2714 start to ensure the appended value is not merged with the existing
Andrew Geisslerc926e172021-05-07 16:11:35 -05002715 value::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002716
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002717 SRC_URI:append = " file://fix-makefile.patch"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002718
2719 You can also use
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002720 the ``:append`` operator with overrides, which results in the actions
Andrew Geisslerc926e172021-05-07 16:11:35 -05002721 only being performed for the specified target or machine::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002722
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002723 SRC_URI:append:sh4 = " file://fix-makefile.patch"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002724
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002725- *Prepending (:prepend):* Use the ``:prepend`` operator to prepend
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002726 values to existing variables. This operator does not add any
2727 additional space. Also, the operator is applied after all the ``+=``,
2728 and ``=+`` operators have been applied and after all ``=``
2729 assignments have occurred.
2730
2731 The following example shows the space being explicitly added to the
2732 end to ensure the prepended value is not merged with the existing
Andrew Geisslerc926e172021-05-07 16:11:35 -05002733 value::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002734
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002735 CFLAGS:prepend = "-I${S}/myincludes "
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002736
2737 You can also use the
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002738 ``:prepend`` operator with overrides, which results in the actions
Andrew Geisslerc926e172021-05-07 16:11:35 -05002739 only being performed for the specified target or machine::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002740
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002741 CFLAGS:prepend:sh4 = "-I${S}/myincludes "
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002742
2743- *Overrides:* You can use overrides to set a value conditionally,
2744 typically based on how the recipe is being built. For example, to set
2745 the :term:`KBRANCH` variable's
2746 value to "standard/base" for any target
2747 :term:`MACHINE`, except for
2748 qemuarm where it should be set to "standard/arm-versatile-926ejs",
Andrew Geisslerc926e172021-05-07 16:11:35 -05002749 you would do the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002750
2751 KBRANCH = "standard/base"
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002752 KBRANCH:qemuarm = "standard/arm-versatile-926ejs"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002753
2754 Overrides are also used to separate
2755 alternate values of a variable in other situations. For example, when
2756 setting variables such as
2757 :term:`FILES` and
2758 :term:`RDEPENDS` that are
2759 specific to individual packages produced by a recipe, you should
2760 always use an override that specifies the name of the package.
2761
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002762- *Indentation:* Use spaces for indentation rather than tabs. For
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002763 shell functions, both currently work. However, it is a policy
2764 decision of the Yocto Project to use tabs in shell functions. Realize
2765 that some layers have a policy to use spaces for all indentation.
2766
2767- *Using Python for Complex Operations:* For more advanced processing,
2768 it is possible to use Python code during variable assignments (e.g.
2769 search and replacement on a variable).
2770
2771 You indicate Python code using the ``${@python_code}`` syntax for the
Andrew Geisslerc926e172021-05-07 16:11:35 -05002772 variable assignment::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002773
2774 SRC_URI = "ftp://ftp.info-zip.org/pub/infozip/src/zip${@d.getVar('PV',1).replace('.', '')}.tgz
2775
2776- *Shell Function Syntax:* Write shell functions as if you were writing
2777 a shell script when you describe a list of actions to take. You
2778 should ensure that your script works with a generic ``sh`` and that
2779 it does not require any ``bash`` or other shell-specific
2780 functionality. The same considerations apply to various system
2781 utilities (e.g. ``sed``, ``grep``, ``awk``, and so forth) that you
2782 might wish to use. If in doubt, you should check with multiple
Andrew Geissler615f2f12022-07-15 14:00:58 -05002783 implementations --- including those from BusyBox.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002784
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002785Adding a New Machine
2786====================
2787
2788Adding a new machine to the Yocto Project is a straightforward process.
2789This section describes how to add machines that are similar to those
2790that the Yocto Project already supports.
2791
2792.. note::
2793
2794 Although well within the capabilities of the Yocto Project, adding a
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002795 totally new architecture might require changes to ``gcc``/``glibc``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002796 and to the site information, which is beyond the scope of this
2797 manual.
2798
2799For a complete example that shows how to add a new machine, see the
2800":ref:`bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script`"
2801section in the Yocto Project Board Support Package (BSP) Developer's
2802Guide.
2803
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002804Adding the Machine Configuration File
2805-------------------------------------
2806
2807To add a new machine, you need to add a new machine configuration file
2808to the layer's ``conf/machine`` directory. This configuration file
2809provides details about the device you are adding.
2810
2811The OpenEmbedded build system uses the root name of the machine
2812configuration file to reference the new machine. For example, given a
2813machine configuration file named ``crownbay.conf``, the build system
2814recognizes the machine as "crownbay".
2815
2816The most important variables you must set in your machine configuration
2817file or include from a lower-level configuration file are as follows:
2818
Andrew Geissler5f350902021-07-23 13:09:54 -04002819- :term:`TARGET_ARCH` (e.g. "arm")
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002820
2821- ``PREFERRED_PROVIDER_virtual/kernel``
2822
Andrew Geissler09036742021-06-25 14:25:14 -05002823- :term:`MACHINE_FEATURES` (e.g. "apm screen wifi")
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002824
2825You might also need these variables:
2826
Andrew Geissler5f350902021-07-23 13:09:54 -04002827- :term:`SERIAL_CONSOLES` (e.g. "115200;ttyS0 115200;ttyS1")
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002828
Andrew Geissler5f350902021-07-23 13:09:54 -04002829- :term:`KERNEL_IMAGETYPE` (e.g. "zImage")
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002830
Andrew Geissler09036742021-06-25 14:25:14 -05002831- :term:`IMAGE_FSTYPES` (e.g. "tar.gz jffs2")
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002832
2833You can find full details on these variables in the reference section.
2834You can leverage existing machine ``.conf`` files from
2835``meta-yocto-bsp/conf/machine/``.
2836
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002837Adding a Kernel for the Machine
2838-------------------------------
2839
2840The OpenEmbedded build system needs to be able to build a kernel for the
2841machine. You need to either create a new kernel recipe for this machine,
2842or extend an existing kernel recipe. You can find several kernel recipe
2843examples in the Source Directory at ``meta/recipes-kernel/linux`` that
2844you can use as references.
2845
2846If you are creating a new kernel recipe, normal recipe-writing rules
Andrew Geissler09036742021-06-25 14:25:14 -05002847apply for setting up a :term:`SRC_URI`. Thus, you need to specify any
2848necessary patches and set :term:`S` to point at the source code. You need to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002849create a ``do_configure`` task that configures the unpacked kernel with
2850a ``defconfig`` file. You can do this by using a ``make defconfig``
2851command or, more commonly, by copying in a suitable ``defconfig`` file
2852and then running ``make oldconfig``. By making use of ``inherit kernel``
2853and potentially some of the ``linux-*.inc`` files, most other
2854functionality is centralized and the defaults of the class normally work
2855well.
2856
2857If you are extending an existing kernel recipe, it is usually a matter
2858of adding a suitable ``defconfig`` file. The file needs to be added into
2859a location similar to ``defconfig`` files used for other machines in a
2860given kernel recipe. A possible way to do this is by listing the file in
Andrew Geissler09036742021-06-25 14:25:14 -05002861the :term:`SRC_URI` and adding the machine to the expression in
2862:term:`COMPATIBLE_MACHINE`::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002863
2864 COMPATIBLE_MACHINE = '(qemux86|qemumips)'
2865
2866For more information on ``defconfig`` files, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06002867":ref:`kernel-dev/common:changing the configuration`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002868section in the Yocto Project Linux Kernel Development Manual.
2869
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002870Adding a Formfactor Configuration File
2871--------------------------------------
2872
2873A formfactor configuration file provides information about the target
2874hardware for which the image is being built and information that the
2875build system cannot obtain from other sources such as the kernel. Some
2876examples of information contained in a formfactor configuration file
2877include framebuffer orientation, whether or not the system has a
2878keyboard, the positioning of the keyboard in relation to the screen, and
2879the screen resolution.
2880
2881The build system uses reasonable defaults in most cases. However, if
2882customization is necessary, you need to create a ``machconfig`` file in
2883the ``meta/recipes-bsp/formfactor/files`` directory. This directory
2884contains directories for specific machines such as ``qemuarm`` and
2885``qemux86``. For information about the settings available and the
2886defaults, see the ``meta/recipes-bsp/formfactor/files/config`` file
2887found in the same area.
2888
Andrew Geisslerc926e172021-05-07 16:11:35 -05002889Following is an example for "qemuarm" machine::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002890
2891 HAVE_TOUCHSCREEN=1
2892 HAVE_KEYBOARD=1
2893 DISPLAY_CAN_ROTATE=0
2894 DISPLAY_ORIENTATION=0
2895 #DISPLAY_WIDTH_PIXELS=640
2896 #DISPLAY_HEIGHT_PIXELS=480
2897 #DISPLAY_BPP=16
2898 DISPLAY_DPI=150
2899 DISPLAY_SUBPIXEL_ORDER=vrgb
2900
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002901Upgrading Recipes
2902=================
2903
2904Over time, upstream developers publish new versions for software built
2905by layer recipes. It is recommended to keep recipes up-to-date with
2906upstream version releases.
2907
William A. Kennington IIIac69b482021-06-02 12:28:27 -07002908While there are several methods to upgrade a recipe, you might
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002909consider checking on the upgrade status of a recipe first. You can do so
2910using the ``devtool check-upgrade-status`` command. See the
2911":ref:`devtool-checking-on-the-upgrade-status-of-a-recipe`"
2912section in the Yocto Project Reference Manual for more information.
2913
2914The remainder of this section describes three ways you can upgrade a
2915recipe. You can use the Automated Upgrade Helper (AUH) to set up
2916automatic version upgrades. Alternatively, you can use
2917``devtool upgrade`` to set up semi-automatic version upgrades. Finally,
2918you can manually upgrade a recipe by editing the recipe itself.
2919
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002920Using the Auto Upgrade Helper (AUH)
2921-----------------------------------
2922
2923The AUH utility works in conjunction with the OpenEmbedded build system
2924in order to automatically generate upgrades for recipes based on new
2925versions being published upstream. Use AUH when you want to create a
2926service that performs the upgrades automatically and optionally sends
2927you an email with the results.
2928
2929AUH allows you to update several recipes with a single use. You can also
2930optionally perform build and integration tests using images with the
2931results saved to your hard drive and emails of results optionally sent
2932to recipe maintainers. Finally, AUH creates Git commits with appropriate
2933commit messages in the layer's tree for the changes made to recipes.
2934
2935.. note::
2936
William A. Kennington IIIac69b482021-06-02 12:28:27 -07002937 In some conditions, you should not use AUH to upgrade recipes
2938 and should instead use either ``devtool upgrade`` or upgrade your
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002939 recipes manually:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002940
2941 - When AUH cannot complete the upgrade sequence. This situation
2942 usually results because custom patches carried by the recipe
2943 cannot be automatically rebased to the new version. In this case,
2944 ``devtool upgrade`` allows you to manually resolve conflicts.
2945
2946 - When for any reason you want fuller control over the upgrade
2947 process. For example, when you want special arrangements for
2948 testing.
2949
2950The following steps describe how to set up the AUH utility:
2951
29521. *Be Sure the Development Host is Set Up:* You need to be sure that
2953 your development host is set up to use the Yocto Project. For
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002954 information on how to set up your host, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06002955 ":ref:`dev-manual/start:Preparing the Build Host`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002956
29572. *Make Sure Git is Configured:* The AUH utility requires Git to be
2958 configured because AUH uses Git to save upgrades. Thus, you must have
2959 Git user and email configured. The following command shows your
Andrew Geisslerc926e172021-05-07 16:11:35 -05002960 configurations::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002961
2962 $ git config --list
2963
2964 If you do not have the user and
Andrew Geisslerc926e172021-05-07 16:11:35 -05002965 email configured, you can use the following commands to do so::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002966
2967 $ git config --global user.name some_name
2968 $ git config --global user.email username@domain.com
2969
29703. *Clone the AUH Repository:* To use AUH, you must clone the repository
2971 onto your development host. The following command uses Git to create
Andrew Geisslerc926e172021-05-07 16:11:35 -05002972 a local copy of the repository on your system::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002973
2974 $ git clone git://git.yoctoproject.org/auto-upgrade-helper
2975 Cloning into 'auto-upgrade-helper'... remote: Counting objects: 768, done.
2976 remote: Compressing objects: 100% (300/300), done.
2977 remote: Total 768 (delta 499), reused 703 (delta 434)
2978 Receiving objects: 100% (768/768), 191.47 KiB | 98.00 KiB/s, done.
2979 Resolving deltas: 100% (499/499), done.
2980 Checking connectivity... done.
2981
2982 AUH is not part of the :term:`OpenEmbedded-Core (OE-Core)` or
2983 :term:`Poky` repositories.
2984
29854. *Create a Dedicated Build Directory:* Run the
2986 :ref:`structure-core-script`
2987 script to create a fresh build directory that you use exclusively for
Andrew Geisslerc926e172021-05-07 16:11:35 -05002988 running the AUH utility::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002989
Andrew Geissler95ac1b82021-03-31 14:34:31 -05002990 $ cd poky
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002991 $ source oe-init-build-env your_AUH_build_directory
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002992
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002993 Re-using an existing build directory and its configurations is not
2994 recommended as existing settings could cause AUH to fail or behave
2995 undesirably.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002996
29975. *Make Configurations in Your Local Configuration File:* Several
William A. Kennington IIIac69b482021-06-02 12:28:27 -07002998 settings are needed in the ``local.conf`` file in the build
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002999 directory you just created for AUH. Make these following
3000 configurations:
3001
3002 - If you want to enable :ref:`Build
Andrew Geissler09209ee2020-12-13 08:44:15 -06003003 History <dev-manual/common-tasks:maintaining build output quality>`,
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003004 which is optional, you need the following lines in the
Andrew Geisslerc926e172021-05-07 16:11:35 -05003005 ``conf/local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003006
3007 INHERIT =+ "buildhistory"
3008 BUILDHISTORY_COMMIT = "1"
3009
3010 With this configuration and a successful
3011 upgrade, a build history "diff" file appears in the
3012 ``upgrade-helper/work/recipe/buildhistory-diff.txt`` file found in
3013 your build directory.
3014
3015 - If you want to enable testing through the
3016 :ref:`testimage <ref-classes-testimage*>`
3017 class, which is optional, you need to have the following set in
Andrew Geisslerc926e172021-05-07 16:11:35 -05003018 your ``conf/local.conf`` file::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003019
3020 INHERIT += "testimage"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003021
3022 .. note::
3023
3024 If your distro does not enable by default ptest, which Poky
Andrew Geisslerc926e172021-05-07 16:11:35 -05003025 does, you need the following in your ``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003026
Patrick Williams0ca19cc2021-08-16 14:03:13 -05003027 DISTRO_FEATURES:append = " ptest"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003028
3029
30306. *Optionally Start a vncserver:* If you are running in a server
Andrew Geisslerc926e172021-05-07 16:11:35 -05003031 without an X11 session, you need to start a vncserver::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003032
3033 $ vncserver :1
3034 $ export DISPLAY=:1
3035
30367. *Create and Edit an AUH Configuration File:* You need to have the
3037 ``upgrade-helper/upgrade-helper.conf`` configuration file in your
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003038 build directory. You can find a sample configuration file in the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003039 :yocto_git:`AUH source repository </auto-upgrade-helper/tree/>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003040
3041 Read through the sample file and make configurations as needed. For
3042 example, if you enabled build history in your ``local.conf`` as
3043 described earlier, you must enable it in ``upgrade-helper.conf``.
3044
3045 Also, if you are using the default ``maintainers.inc`` file supplied
3046 with Poky and located in ``meta-yocto`` and you do not set a
3047 "maintainers_whitelist" or "global_maintainer_override" in the
3048 ``upgrade-helper.conf`` configuration, and you specify "-e all" on
3049 the AUH command-line, the utility automatically sends out emails to
3050 all the default maintainers. Please avoid this.
3051
3052This next set of examples describes how to use the AUH:
3053
3054- *Upgrading a Specific Recipe:* To upgrade a specific recipe, use the
Andrew Geisslerc926e172021-05-07 16:11:35 -05003055 following form::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003056
3057 $ upgrade-helper.py recipe_name
3058
Andrew Geisslerc926e172021-05-07 16:11:35 -05003059 For example, this command upgrades the ``xmodmap`` recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003060
3061 $ upgrade-helper.py xmodmap
3062
3063- *Upgrading a Specific Recipe to a Particular Version:* To upgrade a
Andrew Geisslerc926e172021-05-07 16:11:35 -05003064 specific recipe to a particular version, use the following form::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003065
3066 $ upgrade-helper.py recipe_name -t version
3067
Andrew Geisslerc926e172021-05-07 16:11:35 -05003068 For example, this command upgrades the ``xmodmap`` recipe to version 1.2.3::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003069
3070 $ upgrade-helper.py xmodmap -t 1.2.3
3071
3072- *Upgrading all Recipes to the Latest Versions and Suppressing Email
3073 Notifications:* To upgrade all recipes to their most recent versions
Andrew Geisslerc926e172021-05-07 16:11:35 -05003074 and suppress the email notifications, use the following command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003075
3076 $ upgrade-helper.py all
3077
3078- *Upgrading all Recipes to the Latest Versions and Send Email
3079 Notifications:* To upgrade all recipes to their most recent versions
3080 and send email messages to maintainers for each attempted recipe as
Andrew Geisslerc926e172021-05-07 16:11:35 -05003081 well as a status email, use the following command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003082
3083 $ upgrade-helper.py -e all
3084
3085Once you have run the AUH utility, you can find the results in the AUH
Andrew Geisslerc926e172021-05-07 16:11:35 -05003086build directory::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003087
3088 ${BUILDDIR}/upgrade-helper/timestamp
3089
3090The AUH utility
3091also creates recipe update commits from successful upgrade attempts in
3092the layer tree.
3093
3094You can easily set up to run the AUH utility on a regular basis by using
3095a cron job. See the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003096:yocto_git:`weeklyjob.sh </auto-upgrade-helper/tree/weeklyjob.sh>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003097file distributed with the utility for an example.
3098
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003099Using ``devtool upgrade``
3100-------------------------
3101
3102As mentioned earlier, an alternative method for upgrading recipes to
3103newer versions is to use
Andrew Geissler09209ee2020-12-13 08:44:15 -06003104:doc:`devtool upgrade </ref-manual/devtool-reference>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003105You can read about ``devtool upgrade`` in general in the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003106":ref:`sdk-manual/extensible:use \`\`devtool upgrade\`\` to create a version of the recipe that supports a newer version of the software`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003107section in the Yocto Project Application Development and the Extensible
3108Software Development Kit (eSDK) Manual.
3109
3110To see all the command-line options available with ``devtool upgrade``,
Andrew Geisslerc926e172021-05-07 16:11:35 -05003111use the following help command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003112
3113 $ devtool upgrade -h
3114
3115If you want to find out what version a recipe is currently at upstream
3116without any attempt to upgrade your local version of the recipe, you can
Andrew Geisslerc926e172021-05-07 16:11:35 -05003117use the following command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003118
3119 $ devtool latest-version recipe_name
3120
3121As mentioned in the previous section describing AUH, ``devtool upgrade``
3122works in a less-automated manner than AUH. Specifically,
3123``devtool upgrade`` only works on a single recipe that you name on the
3124command line, cannot perform build and integration testing using images,
3125and does not automatically generate commits for changes in the source
3126tree. Despite all these "limitations", ``devtool upgrade`` updates the
3127recipe file to the new upstream version and attempts to rebase custom
3128patches contained by the recipe as needed.
3129
3130.. note::
3131
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003132 AUH uses much of ``devtool upgrade`` behind the scenes making AUH somewhat
3133 of a "wrapper" application for ``devtool upgrade``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003134
3135A typical scenario involves having used Git to clone an upstream
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003136repository that you use during build operations. Because you have built the
3137recipe in the past, the layer is likely added to your
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003138configuration already. If for some reason, the layer is not added, you
3139could add it easily using the
3140":ref:`bitbake-layers <bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script>`"
3141script. For example, suppose you use the ``nano.bb`` recipe from the
3142``meta-oe`` layer in the ``meta-openembedded`` repository. For this
Andrew Geisslerc926e172021-05-07 16:11:35 -05003143example, assume that the layer has been cloned into following area::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003144
3145 /home/scottrif/meta-openembedded
3146
3147The following command from your
3148:term:`Build Directory` adds the layer to
Andrew Geisslerc926e172021-05-07 16:11:35 -05003149your build configuration (i.e. ``${BUILDDIR}/conf/bblayers.conf``)::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003150
3151 $ bitbake-layers add-layer /home/scottrif/meta-openembedded/meta-oe
3152 NOTE: Starting bitbake server...
3153 Parsing recipes: 100% |##########################################| Time: 0:00:55
3154 Parsing of 1431 .bb files complete (0 cached, 1431 parsed). 2040 targets, 56 skipped, 0 masked, 0 errors.
3155 Removing 12 recipes from the x86_64 sysroot: 100% |##############| Time: 0:00:00
3156 Removing 1 recipes from the x86_64_i586 sysroot: 100% |##########| Time: 0:00:00
3157 Removing 5 recipes from the i586 sysroot: 100% |#################| Time: 0:00:00
3158 Removing 5 recipes from the qemux86 sysroot: 100% |##############| Time: 0:00:00
3159
3160For this example, assume that the ``nano.bb`` recipe that
3161is upstream has a 2.9.3 version number. However, the version in the
3162local repository is 2.7.4. The following command from your build
3163directory automatically upgrades the recipe for you:
3164
3165.. note::
3166
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003167 Using the ``-V`` option is not necessary. Omitting the version number causes
3168 ``devtool upgrade`` to upgrade the recipe to the most recent version.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003169
3170::
3171
3172 $ devtool upgrade nano -V 2.9.3
3173 NOTE: Starting bitbake server...
3174 NOTE: Creating workspace layer in /home/scottrif/poky/build/workspace
3175 Parsing recipes: 100% |##########################################| Time: 0:00:46
3176 Parsing of 1431 .bb files complete (0 cached, 1431 parsed). 2040 targets, 56 skipped, 0 masked, 0 errors.
3177 NOTE: Extracting current version source...
3178 NOTE: Resolving any missing task queue dependencies
3179 .
3180 .
3181 .
3182 NOTE: Executing SetScene Tasks
3183 NOTE: Executing RunQueue Tasks
3184 NOTE: Tasks Summary: Attempted 74 tasks of which 72 didn't need to be rerun and all succeeded.
3185 Adding changed files: 100% |#####################################| Time: 0:00:00
3186 NOTE: Upgraded source extracted to /home/scottrif/poky/build/workspace/sources/nano
3187 NOTE: New recipe is /home/scottrif/poky/build/workspace/recipes/nano/nano_2.9.3.bb
3188
3189Continuing with this example, you can use ``devtool build`` to build the
Andrew Geisslerc926e172021-05-07 16:11:35 -05003190newly upgraded recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003191
3192 $ devtool build nano
3193 NOTE: Starting bitbake server...
3194 Loading cache: 100% |################################################################################################| Time: 0:00:01
3195 Loaded 2040 entries from dependency cache.
3196 Parsing recipes: 100% |##############################################################################################| Time: 0:00:00
3197 Parsing of 1432 .bb files complete (1431 cached, 1 parsed). 2041 targets, 56 skipped, 0 masked, 0 errors.
3198 NOTE: Resolving any missing task queue dependencies
3199 .
3200 .
3201 .
3202 NOTE: Executing SetScene Tasks
3203 NOTE: Executing RunQueue Tasks
3204 NOTE: nano: compiling from external source tree /home/scottrif/poky/build/workspace/sources/nano
3205 NOTE: Tasks Summary: Attempted 520 tasks of which 304 didn't need to be rerun and all succeeded.
3206
William A. Kennington IIIac69b482021-06-02 12:28:27 -07003207Within the ``devtool upgrade`` workflow, you can
3208deploy and test your rebuilt software. For this example,
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003209however, running ``devtool finish`` cleans up the workspace once the
3210source in your workspace is clean. This usually means using Git to stage
3211and submit commits for the changes generated by the upgrade process.
3212
3213Once the tree is clean, you can clean things up in this example with the
3214following command from the ``${BUILDDIR}/workspace/sources/nano``
Andrew Geisslerc926e172021-05-07 16:11:35 -05003215directory::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003216
3217 $ devtool finish nano meta-oe
3218 NOTE: Starting bitbake server...
3219 Loading cache: 100% |################################################################################################| Time: 0:00:00
3220 Loaded 2040 entries from dependency cache.
3221 Parsing recipes: 100% |##############################################################################################| Time: 0:00:01
3222 Parsing of 1432 .bb files complete (1431 cached, 1 parsed). 2041 targets, 56 skipped, 0 masked, 0 errors.
3223 NOTE: Adding new patch 0001-nano.bb-Stuff-I-changed-when-upgrading-nano.bb.patch
3224 NOTE: Updating recipe nano_2.9.3.bb
3225 NOTE: Removing file /home/scottrif/meta-openembedded/meta-oe/recipes-support/nano/nano_2.7.4.bb
3226 NOTE: Moving recipe file to /home/scottrif/meta-openembedded/meta-oe/recipes-support/nano
3227 NOTE: Leaving source tree /home/scottrif/poky/build/workspace/sources/nano as-is; if you no longer need it then please delete it manually
3228
3229
3230Using the ``devtool finish`` command cleans up the workspace and creates a patch
3231file based on your commits. The tool puts all patch files back into the
3232source directory in a sub-directory named ``nano`` in this case.
3233
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003234Manually Upgrading a Recipe
3235---------------------------
3236
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003237If for some reason you choose not to upgrade recipes using
Andrew Geissler09209ee2020-12-13 08:44:15 -06003238:ref:`dev-manual/common-tasks:Using the Auto Upgrade Helper (AUH)` or
3239by :ref:`dev-manual/common-tasks:Using \`\`devtool upgrade\`\``,
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003240you can manually edit the recipe files to upgrade the versions.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003241
3242.. note::
3243
3244 Manually updating multiple recipes scales poorly and involves many
3245 steps. The recommendation to upgrade recipe versions is through AUH
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003246 or ``devtool upgrade``, both of which automate some steps and provide
3247 guidance for others needed for the manual process.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003248
3249To manually upgrade recipe versions, follow these general steps:
3250
32511. *Change the Version:* Rename the recipe such that the version (i.e.
3252 the :term:`PV` part of the recipe name)
3253 changes appropriately. If the version is not part of the recipe name,
Andrew Geissler09036742021-06-25 14:25:14 -05003254 change the value as it is set for :term:`PV` within the recipe itself.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003255
Andrew Geissler09036742021-06-25 14:25:14 -050032562. *Update* :term:`SRCREV` *if Needed*: If the source code your recipe builds
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003257 is fetched from Git or some other version control system, update
3258 :term:`SRCREV` to point to the
3259 commit hash that matches the new version.
3260
32613. *Build the Software:* Try to build the recipe using BitBake. Typical
3262 build failures include the following:
3263
3264 - License statements were updated for the new version. For this
3265 case, you need to review any changes to the license and update the
3266 values of :term:`LICENSE` and
3267 :term:`LIC_FILES_CHKSUM`
3268 as needed.
3269
3270 .. note::
3271
3272 License changes are often inconsequential. For example, the
3273 license text's copyright year might have changed.
3274
3275 - Custom patches carried by the older version of the recipe might
3276 fail to apply to the new version. For these cases, you need to
3277 review the failures. Patches might not be necessary for the new
3278 version of the software if the upgraded version has fixed those
3279 issues. If a patch is necessary and failing, you need to rebase it
3280 into the new version.
3281
32824. *Optionally Attempt to Build for Several Architectures:* Once you
3283 successfully build the new software for a given architecture, you
3284 could test the build for other architectures by changing the
3285 :term:`MACHINE` variable and
3286 rebuilding the software. This optional step is especially important
3287 if the recipe is to be released publicly.
3288
32895. *Check the Upstream Change Log or Release Notes:* Checking both these
William A. Kennington IIIac69b482021-06-02 12:28:27 -07003290 reveals if there are new features that could break
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003291 backwards-compatibility. If so, you need to take steps to mitigate or
3292 eliminate that situation.
3293
32946. *Optionally Create a Bootable Image and Test:* If you want, you can
3295 test the new software by booting it onto actual hardware.
3296
32977. *Create a Commit with the Change in the Layer Repository:* After all
3298 builds work and any testing is successful, you can create commits for
3299 any changes in the layer holding your upgraded recipe.
3300
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003301Finding Temporary Source Code
3302=============================
3303
3304You might find it helpful during development to modify the temporary
3305source code used by recipes to build packages. For example, suppose you
3306are developing a patch and you need to experiment a bit to figure out
3307your solution. After you have initially built the package, you can
3308iteratively tweak the source code, which is located in the
3309:term:`Build Directory`, and then you can
3310force a re-compile and quickly test your altered code. Once you settle
3311on a solution, you can then preserve your changes in the form of
3312patches.
3313
3314During a build, the unpacked temporary source code used by recipes to
3315build packages is available in the Build Directory as defined by the
3316:term:`S` variable. Below is the default
Andrew Geissler09036742021-06-25 14:25:14 -05003317value for the :term:`S` variable as defined in the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003318``meta/conf/bitbake.conf`` configuration file in the
Andrew Geisslerc926e172021-05-07 16:11:35 -05003319:term:`Source Directory`::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003320
3321 S = "${WORKDIR}/${BP}"
3322
3323You should be aware that many recipes override the
Andrew Geissler09036742021-06-25 14:25:14 -05003324:term:`S` variable. For example, recipes that fetch their source from Git
3325usually set :term:`S` to ``${WORKDIR}/git``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003326
3327.. note::
3328
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003329 The :term:`BP` represents the base recipe name, which consists of the name
Andrew Geisslerc926e172021-05-07 16:11:35 -05003330 and version::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003331
3332 BP = "${BPN}-${PV}"
3333
3334
3335The path to the work directory for the recipe
3336(:term:`WORKDIR`) is defined as
Andrew Geisslerc926e172021-05-07 16:11:35 -05003337follows::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003338
3339 ${TMPDIR}/work/${MULTIMACH_TARGET_SYS}/${PN}/${EXTENDPE}${PV}-${PR}
3340
3341The actual directory depends on several things:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003342
3343- :term:`TMPDIR`: The top-level build
3344 output directory.
3345
3346- :term:`MULTIMACH_TARGET_SYS`:
3347 The target system identifier.
3348
3349- :term:`PN`: The recipe name.
3350
Andrew Geissler615f2f12022-07-15 14:00:58 -05003351- :term:`EXTENDPE`: The epoch --- if
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003352 :term:`PE` is not specified, which is
Andrew Geissler615f2f12022-07-15 14:00:58 -05003353 usually the case for most recipes, then :term:`EXTENDPE` is blank.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003354
3355- :term:`PV`: The recipe version.
3356
3357- :term:`PR`: The recipe revision.
3358
3359As an example, assume a Source Directory top-level folder named
3360``poky``, a default Build Directory at ``poky/build``, and a
3361``qemux86-poky-linux`` machine target system. Furthermore, suppose your
3362recipe is named ``foo_1.3.0.bb``. In this case, the work directory the
Andrew Geisslerc926e172021-05-07 16:11:35 -05003363build system uses to build the package would be as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003364
3365 poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0
3366
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003367Using Quilt in Your Workflow
3368============================
3369
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003370`Quilt <https://savannah.nongnu.org/projects/quilt>`__ is a powerful tool
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003371that allows you to capture source code changes without having a clean
3372source tree. This section outlines the typical workflow you can use to
3373modify source code, test changes, and then preserve the changes in the
3374form of a patch all using Quilt.
3375
3376.. note::
3377
3378 With regard to preserving changes to source files, if you clean a
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003379 recipe or have ``rm_work`` enabled, the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003380 :ref:`devtool workflow <sdk-manual/extensible:using \`\`devtool\`\` in your sdk workflow>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003381 as described in the Yocto Project Application Development and the
3382 Extensible Software Development Kit (eSDK) manual is a safer
3383 development flow than the flow that uses Quilt.
3384
3385Follow these general steps:
3386
33871. *Find the Source Code:* Temporary source code used by the
3388 OpenEmbedded build system is kept in the
3389 :term:`Build Directory`. See the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05003390 ":ref:`dev-manual/common-tasks:finding temporary source code`" section to
3391 learn how to locate the directory that has the temporary source code for a
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003392 particular package.
3393
33942. *Change Your Working Directory:* You need to be in the directory that
3395 has the temporary source code. That directory is defined by the
3396 :term:`S` variable.
3397
33983. *Create a New Patch:* Before modifying source code, you need to
3399 create a new patch. To create a new patch file, use ``quilt new`` as
Andrew Geisslerc926e172021-05-07 16:11:35 -05003400 below::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003401
3402 $ quilt new my_changes.patch
3403
34044. *Notify Quilt and Add Files:* After creating the patch, you need to
3405 notify Quilt about the files you plan to edit. You notify Quilt by
Andrew Geisslerc926e172021-05-07 16:11:35 -05003406 adding the files to the patch you just created::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003407
3408 $ quilt add file1.c file2.c file3.c
3409
34105. *Edit the Files:* Make your changes in the source code to the files
3411 you added to the patch.
3412
34136. *Test Your Changes:* Once you have modified the source code, the
3414 easiest way to test your changes is by calling the ``do_compile``
Andrew Geisslerc926e172021-05-07 16:11:35 -05003415 task as shown in the following example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003416
3417 $ bitbake -c compile -f package
3418
3419 The ``-f`` or ``--force`` option forces the specified task to
3420 execute. If you find problems with your code, you can just keep
3421 editing and re-testing iteratively until things work as expected.
3422
3423 .. note::
3424
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003425 All the modifications you make to the temporary source code disappear
3426 once you run the ``do_clean`` or ``do_cleanall`` tasks using BitBake
3427 (i.e. ``bitbake -c clean package`` and ``bitbake -c cleanall package``).
3428 Modifications will also disappear if you use the ``rm_work`` feature as
3429 described in the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003430 ":ref:`dev-manual/common-tasks:conserving disk space during builds`"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003431 section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003432
34337. *Generate the Patch:* Once your changes work as expected, you need to
3434 use Quilt to generate the final patch that contains all your
3435 modifications.
3436 ::
3437
3438 $ quilt refresh
3439
3440 At this point, the
3441 ``my_changes.patch`` file has all your edits made to the ``file1.c``,
3442 ``file2.c``, and ``file3.c`` files.
3443
3444 You can find the resulting patch file in the ``patches/``
Andrew Geissler09036742021-06-25 14:25:14 -05003445 subdirectory of the source (:term:`S`) directory.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003446
34478. *Copy the Patch File:* For simplicity, copy the patch file into a
3448 directory named ``files``, which you can create in the same directory
3449 that holds the recipe (``.bb``) file or the append (``.bbappend``)
3450 file. Placing the patch here guarantees that the OpenEmbedded build
Andrew Geissler09036742021-06-25 14:25:14 -05003451 system will find the patch. Next, add the patch into the :term:`SRC_URI`
Andrew Geisslerc926e172021-05-07 16:11:35 -05003452 of the recipe. Here is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003453
3454 SRC_URI += "file://my_changes.patch"
3455
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003456Using a Development Shell
3457=========================
3458
3459When debugging certain commands or even when just editing packages,
3460``devshell`` can be a useful tool. When you invoke ``devshell``, all
3461tasks up to and including
3462:ref:`ref-tasks-patch` are run for the
3463specified target. Then, a new terminal is opened and you are placed in
3464``${``\ :term:`S`\ ``}``, the source
3465directory. In the new terminal, all the OpenEmbedded build-related
3466environment variables are still defined so you can use commands such as
3467``configure`` and ``make``. The commands execute just as if the
3468OpenEmbedded build system were executing them. Consequently, working
3469this way can be helpful when debugging a build or preparing software to
3470be used with the OpenEmbedded build system.
3471
3472Following is an example that uses ``devshell`` on a target named
Andrew Geisslerc926e172021-05-07 16:11:35 -05003473``matchbox-desktop``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003474
3475 $ bitbake matchbox-desktop -c devshell
3476
3477This command spawns a terminal with a shell prompt within the
3478OpenEmbedded build environment. The
3479:term:`OE_TERMINAL` variable
3480controls what type of shell is opened.
3481
3482For spawned terminals, the following occurs:
3483
3484- The ``PATH`` variable includes the cross-toolchain.
3485
3486- The ``pkgconfig`` variables find the correct ``.pc`` files.
3487
3488- The ``configure`` command finds the Yocto Project site files as well
3489 as any other necessary files.
3490
3491Within this environment, you can run configure or compile commands as if
3492they were being run by the OpenEmbedded build system itself. As noted
3493earlier, the working directory also automatically changes to the Source
3494Directory (:term:`S`).
3495
3496To manually run a specific task using ``devshell``, run the
3497corresponding ``run.*`` script in the
3498``${``\ :term:`WORKDIR`\ ``}/temp``
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003499directory (e.g., ``run.do_configure.``\ `pid`). If a task's script does
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003500not exist, which would be the case if the task was skipped by way of the
3501sstate cache, you can create the task by first running it outside of the
Andrew Geisslerc926e172021-05-07 16:11:35 -05003502``devshell``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003503
3504 $ bitbake -c task
3505
3506.. note::
3507
3508 - Execution of a task's ``run.*`` script and BitBake's execution of
3509 a task are identical. In other words, running the script re-runs
3510 the task just as it would be run using the ``bitbake -c`` command.
3511
3512 - Any ``run.*`` file that does not have a ``.pid`` extension is a
3513 symbolic link (symlink) to the most recent version of that file.
3514
3515Remember, that the ``devshell`` is a mechanism that allows you to get
3516into the BitBake task execution environment. And as such, all commands
3517must be called just as BitBake would call them. That means you need to
3518provide the appropriate options for cross-compilation and so forth as
3519applicable.
3520
3521When you are finished using ``devshell``, exit the shell or close the
3522terminal window.
3523
3524.. note::
3525
3526 - It is worth remembering that when using ``devshell`` you need to
3527 use the full compiler name such as ``arm-poky-linux-gnueabi-gcc``
3528 instead of just using ``gcc``. The same applies to other
3529 applications such as ``binutils``, ``libtool`` and so forth.
Andrew Geissler09036742021-06-25 14:25:14 -05003530 BitBake sets up environment variables such as :term:`CC` to assist
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003531 applications, such as ``make`` to find the correct tools.
3532
3533 - It is also worth noting that ``devshell`` still works over X11
3534 forwarding and similar situations.
3535
Andrew Geisslereff27472021-10-29 15:35:00 -05003536Using a Python Development Shell
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003537================================
3538
3539Similar to working within a development shell as described in the
3540previous section, you can also spawn and work within an interactive
3541Python development shell. When debugging certain commands or even when
Andrew Geisslereff27472021-10-29 15:35:00 -05003542just editing packages, ``pydevshell`` can be a useful tool. When you
3543invoke the ``pydevshell`` task, all tasks up to and including
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003544:ref:`ref-tasks-patch` are run for the
3545specified target. Then a new terminal is opened. Additionally, key
3546Python objects and code are available in the same way they are to
3547BitBake tasks, in particular, the data store 'd'. So, commands such as
3548the following are useful when exploring the data store and running
Andrew Geisslerc926e172021-05-07 16:11:35 -05003549functions::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003550
3551 pydevshell> d.getVar("STAGING_DIR")
3552 '/media/build1/poky/build/tmp/sysroots'
Andrew Geissler5199d832021-09-24 16:47:35 -05003553 pydevshell> d.getVar("STAGING_DIR", False)
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003554 '${TMPDIR}/sysroots'
3555 pydevshell> d.setVar("FOO", "bar")
3556 pydevshell> d.getVar("FOO")
3557 'bar'
3558 pydevshell> d.delVar("FOO")
3559 pydevshell> d.getVar("FOO")
3560 pydevshell> bb.build.exec_func("do_unpack", d)
3561 pydevshell>
3562
3563The commands execute just as if the OpenEmbedded build
3564system were executing them. Consequently, working this way can be
3565helpful when debugging a build or preparing software to be used with the
3566OpenEmbedded build system.
3567
Andrew Geisslereff27472021-10-29 15:35:00 -05003568Following is an example that uses ``pydevshell`` on a target named
Andrew Geisslerc926e172021-05-07 16:11:35 -05003569``matchbox-desktop``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003570
Andrew Geisslereff27472021-10-29 15:35:00 -05003571 $ bitbake matchbox-desktop -c pydevshell
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003572
3573This command spawns a terminal and places you in an interactive Python
3574interpreter within the OpenEmbedded build environment. The
3575:term:`OE_TERMINAL` variable
3576controls what type of shell is opened.
3577
Andrew Geisslereff27472021-10-29 15:35:00 -05003578When you are finished using ``pydevshell``, you can exit the shell
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003579either by using Ctrl+d or closing the terminal window.
3580
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003581Building
3582========
3583
Andrew Geissler5199d832021-09-24 16:47:35 -05003584This section describes various build procedures, such as the steps
3585needed for a simple build, building a target for multiple configurations,
3586generating an image for more than one machine, and so forth.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003587
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003588Building a Simple Image
3589-----------------------
3590
3591In the development environment, you need to build an image whenever you
3592change hardware support, add or change system libraries, or add or
William A. Kennington IIIac69b482021-06-02 12:28:27 -07003593change services that have dependencies. There are several methods that allow
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003594you to build an image within the Yocto Project. This section presents
3595the basic steps you need to build a simple image using BitBake from a
3596build host running Linux.
3597
3598.. note::
3599
3600 - For information on how to build an image using
3601 :term:`Toaster`, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003602 :doc:`/toaster-manual/index`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003603
3604 - For information on how to use ``devtool`` to build images, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003605 ":ref:`sdk-manual/extensible:using \`\`devtool\`\` in your sdk workflow`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003606 section in the Yocto Project Application Development and the
3607 Extensible Software Development Kit (eSDK) manual.
3608
3609 - For a quick example on how to build an image using the
3610 OpenEmbedded build system, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003611 :doc:`/brief-yoctoprojectqs/index` document.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003612
3613The build process creates an entire Linux distribution from source and
3614places it in your :term:`Build Directory` under
3615``tmp/deploy/images``. For detailed information on the build process
Andrew Geissler09209ee2020-12-13 08:44:15 -06003616using BitBake, see the ":ref:`overview-manual/concepts:images`" section in the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003617Yocto Project Overview and Concepts Manual.
3618
3619The following figure and list overviews the build process:
3620
3621.. image:: figures/bitbake-build-flow.png
Andrew Geisslerd5838332022-05-27 11:33:10 -05003622 :width: 100%
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003623
36241. *Set up Your Host Development System to Support Development Using the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003625 Yocto Project*: See the ":doc:`start`" section for options on how to get a
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003626 build host ready to use the Yocto Project.
3627
36282. *Initialize the Build Environment:* Initialize the build environment
3629 by sourcing the build environment script (i.e.
Andrew Geisslerc926e172021-05-07 16:11:35 -05003630 :ref:`structure-core-script`)::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003631
3632 $ source oe-init-build-env [build_dir]
3633
3634 When you use the initialization script, the OpenEmbedded build system
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003635 uses ``build`` as the default :term:`Build Directory` in your current work
3636 directory. You can use a `build_dir` argument with the script to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003637 specify a different build directory.
3638
3639 .. note::
3640
3641 A common practice is to use a different Build Directory for
Andrew Geissler5199d832021-09-24 16:47:35 -05003642 different targets; for example, ``~/build/x86`` for a ``qemux86``
3643 target, and ``~/build/arm`` for a ``qemuarm`` target. In any
3644 event, it's typically cleaner to locate the build directory
3645 somewhere outside of your source directory.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003646
Andrew Geissler4c19ea12020-10-27 13:52:24 -050036473. *Make Sure Your* ``local.conf`` *File is Correct*: Ensure the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003648 ``conf/local.conf`` configuration file, which is found in the Build
3649 Directory, is set up how you want it. This file defines many aspects
3650 of the build environment including the target machine architecture
Andrew Geissler09036742021-06-25 14:25:14 -05003651 through the :term:`MACHINE` variable, the packaging format used during
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003652 the build
3653 (:term:`PACKAGE_CLASSES`),
3654 and a centralized tarball download directory through the
3655 :term:`DL_DIR` variable.
3656
Andrew Geisslerc926e172021-05-07 16:11:35 -050036574. *Build the Image:* Build the image using the ``bitbake`` command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003658
3659 $ bitbake target
3660
3661 .. note::
3662
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003663 For information on BitBake, see the :doc:`bitbake:index`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003664
3665 The target is the name of the recipe you want to build. Common
3666 targets are the images in ``meta/recipes-core/images``,
3667 ``meta/recipes-sato/images``, and so forth all found in the
Andrew Geissler5199d832021-09-24 16:47:35 -05003668 :term:`Source Directory`. Alternatively, the target
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003669 can be the name of a recipe for a specific piece of software such as
3670 BusyBox. For more details about the images the OpenEmbedded build
3671 system supports, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003672 ":ref:`ref-manual/images:Images`" chapter in the Yocto
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003673 Project Reference Manual.
3674
3675 As an example, the following command builds the
Andrew Geisslerc926e172021-05-07 16:11:35 -05003676 ``core-image-minimal`` image::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003677
3678 $ bitbake core-image-minimal
3679
3680 Once an
3681 image has been built, it often needs to be installed. The images and
3682 kernels built by the OpenEmbedded build system are placed in the
3683 Build Directory in ``tmp/deploy/images``. For information on how to
3684 run pre-built images such as ``qemux86`` and ``qemuarm``, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003685 :doc:`/sdk-manual/index` manual. For
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003686 information about how to install these images, see the documentation
3687 for your particular board or machine.
3688
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003689Building Images for Multiple Targets Using Multiple Configurations
3690------------------------------------------------------------------
3691
3692You can use a single ``bitbake`` command to build multiple images or
3693packages for different targets where each image or package requires a
3694different configuration (multiple configuration builds). The builds, in
3695this scenario, are sometimes referred to as "multiconfigs", and this
3696section uses that term throughout.
3697
3698This section describes how to set up for multiple configuration builds
3699and how to account for cross-build dependencies between the
3700multiconfigs.
3701
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003702Setting Up and Running a Multiple Configuration Build
3703~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3704
3705To accomplish a multiple configuration build, you must define each
3706target's configuration separately using a parallel configuration file in
Andrew Geissler615f2f12022-07-15 14:00:58 -05003707the :term:`Build Directory` or configuration directory within a layer, and you
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003708must follow a required file hierarchy. Additionally, you must enable the
3709multiple configuration builds in your ``local.conf`` file.
3710
3711Follow these steps to set up and execute multiple configuration builds:
3712
3713- *Create Separate Configuration Files*: You need to create a single
3714 configuration file for each build target (each multiconfig).
Andrew Geissler615f2f12022-07-15 14:00:58 -05003715 The configuration definitions are implementation dependent but often
3716 each configuration file will define the machine and the
3717 temporary directory BitBake uses for the build. Whether the same
3718 temporary directory (:term:`TMPDIR`) can be shared will depend on what is
3719 similar and what is different between the configurations. Multiple MACHINE
3720 targets can share the same (:term:`TMPDIR`) as long as the rest of the
3721 configuration is the same, multiple DISTRO settings would need separate
3722 (:term:`TMPDIR`) directories.
3723
3724 For example, consider a scenario with two different multiconfigs for the same
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003725 :term:`MACHINE`: "qemux86" built
3726 for two distributions such as "poky" and "poky-lsb". In this case,
Andrew Geissler615f2f12022-07-15 14:00:58 -05003727 you would need to use the different :term:`TMPDIR`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003728
3729 Here is an example showing the minimal statements needed in a
3730 configuration file for a "qemux86" target whose temporary build
Andrew Geisslerc926e172021-05-07 16:11:35 -05003731 directory is ``tmpmultix86``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003732
3733 MACHINE = "qemux86"
3734 TMPDIR = "${TOPDIR}/tmpmultix86"
3735
3736 The location for these multiconfig configuration files is specific.
3737 They must reside in the current build directory in a sub-directory of
Andrew Geissler615f2f12022-07-15 14:00:58 -05003738 ``conf`` named ``multiconfig`` or within a layer's ``conf`` directory
3739 under a directory named ``multiconfig``. Following is an example that defines
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003740 two configuration files for the "x86" and "arm" multiconfigs:
3741
3742 .. image:: figures/multiconfig_files.png
3743 :align: center
Andrew Geisslerd5838332022-05-27 11:33:10 -05003744 :width: 50%
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003745
Andrew Geissler615f2f12022-07-15 14:00:58 -05003746 The usual :term:`BBPATH` search path is used to locate multiconfig files in
3747 a similar way to other conf files.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003748
3749- *Add the BitBake Multi-configuration Variable to the Local
3750 Configuration File*: Use the
3751 :term:`BBMULTICONFIG`
3752 variable in your ``conf/local.conf`` configuration file to specify
3753 each multiconfig. Continuing with the example from the previous
Andrew Geissler09036742021-06-25 14:25:14 -05003754 figure, the :term:`BBMULTICONFIG` variable needs to enable two
Andrew Geisslerc926e172021-05-07 16:11:35 -05003755 multiconfigs: "x86" and "arm" by specifying each configuration file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003756
3757 BBMULTICONFIG = "x86 arm"
3758
3759 .. note::
3760
3761 A "default" configuration already exists by definition. This
3762 configuration is named: "" (i.e. empty string) and is defined by
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003763 the variables coming from your ``local.conf``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003764 file. Consequently, the previous example actually adds two
3765 additional configurations to your build: "arm" and "x86" along
3766 with "".
3767
3768- *Launch BitBake*: Use the following BitBake command form to launch
Andrew Geisslerc926e172021-05-07 16:11:35 -05003769 the multiple configuration build::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003770
3771 $ bitbake [mc:multiconfigname:]target [[[mc:multiconfigname:]target] ... ]
3772
Andrew Geisslerc926e172021-05-07 16:11:35 -05003773 For the example in this section, the following command applies::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003774
3775 $ bitbake mc:x86:core-image-minimal mc:arm:core-image-sato mc::core-image-base
3776
3777 The previous BitBake command builds a ``core-image-minimal`` image
3778 that is configured through the ``x86.conf`` configuration file, a
3779 ``core-image-sato`` image that is configured through the ``arm.conf``
3780 configuration file and a ``core-image-base`` that is configured
3781 through your ``local.conf`` configuration file.
3782
3783.. note::
3784
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003785 Support for multiple configuration builds in the Yocto Project &DISTRO;
3786 (&DISTRO_NAME;) Release does not include Shared State (sstate)
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003787 optimizations. Consequently, if a build uses the same object twice
Andrew Geissler09036742021-06-25 14:25:14 -05003788 in, for example, two different :term:`TMPDIR`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003789 directories, the build either loads from an existing sstate cache for
3790 that build at the start or builds the object fresh.
3791
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003792Enabling Multiple Configuration Build Dependencies
3793~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3794
3795Sometimes dependencies can exist between targets (multiconfigs) in a
3796multiple configuration build. For example, suppose that in order to
3797build a ``core-image-sato`` image for an "x86" multiconfig, the root
3798filesystem of an "arm" multiconfig must exist. This dependency is
3799essentially that the
3800:ref:`ref-tasks-image` task in the
3801``core-image-sato`` recipe depends on the completion of the
3802:ref:`ref-tasks-rootfs` task of the
3803``core-image-minimal`` recipe.
3804
3805To enable dependencies in a multiple configuration build, you must
3806declare the dependencies in the recipe using the following statement
Andrew Geisslerc926e172021-05-07 16:11:35 -05003807form::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003808
3809 task_or_package[mcdepends] = "mc:from_multiconfig:to_multiconfig:recipe_name:task_on_which_to_depend"
3810
3811To better show how to use this statement, consider the example scenario
3812from the first paragraph of this section. The following statement needs
Andrew Geisslerc926e172021-05-07 16:11:35 -05003813to be added to the recipe that builds the ``core-image-sato`` image::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003814
3815 do_image[mcdepends] = "mc:x86:arm:core-image-minimal:do_rootfs"
3816
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003817In this example, the `from_multiconfig` is "x86". The `to_multiconfig` is "arm". The
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003818task on which the ``do_image`` task in the recipe depends is the
3819``do_rootfs`` task from the ``core-image-minimal`` recipe associated
3820with the "arm" multiconfig.
3821
3822Once you set up this dependency, you can build the "x86" multiconfig
Andrew Geisslerc926e172021-05-07 16:11:35 -05003823using a BitBake command as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003824
3825 $ bitbake mc:x86:core-image-sato
3826
3827This command executes all the tasks needed to create the
3828``core-image-sato`` image for the "x86" multiconfig. Because of the
3829dependency, BitBake also executes through the ``do_rootfs`` task for the
3830"arm" multiconfig build.
3831
3832Having a recipe depend on the root filesystem of another build might not
3833seem that useful. Consider this change to the statement in the
Andrew Geisslerc926e172021-05-07 16:11:35 -05003834``core-image-sato`` recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003835
3836 do_image[mcdepends] = "mc:x86:arm:core-image-minimal:do_image"
3837
3838In this case, BitBake must
3839create the ``core-image-minimal`` image for the "arm" build since the
3840"x86" build depends on it.
3841
3842Because "x86" and "arm" are enabled for multiple configuration builds
3843and have separate configuration files, BitBake places the artifacts for
3844each build in the respective temporary build directories (i.e.
3845:term:`TMPDIR`).
3846
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003847Building an Initial RAM Filesystem (initramfs) Image
3848----------------------------------------------------
3849
3850An initial RAM filesystem (initramfs) image provides a temporary root
3851filesystem used for early system initialization (e.g. loading of modules
3852needed to locate and mount the "real" root filesystem).
3853
3854.. note::
3855
3856 The initramfs image is the successor of initial RAM disk (initrd). It
3857 is a "copy in and out" (cpio) archive of the initial filesystem that
3858 gets loaded into memory during the Linux startup process. Because
3859 Linux uses the contents of the archive during initialization, the
3860 initramfs image needs to contain all of the device drivers and tools
3861 needed to mount the final root filesystem.
3862
3863Follow these steps to create an initramfs image:
3864
38651. *Create the initramfs Image Recipe:* You can reference the
3866 ``core-image-minimal-initramfs.bb`` recipe found in the
3867 ``meta/recipes-core`` directory of the :term:`Source Directory`
3868 as an example
3869 from which to work.
3870
38712. *Decide if You Need to Bundle the initramfs Image Into the Kernel
3872 Image:* If you want the initramfs image that is built to be bundled
3873 in with the kernel image, set the
3874 :term:`INITRAMFS_IMAGE_BUNDLE`
3875 variable to "1" in your ``local.conf`` configuration file and set the
3876 :term:`INITRAMFS_IMAGE`
3877 variable in the recipe that builds the kernel image.
3878
3879 .. note::
3880
Andrew Geissler5199d832021-09-24 16:47:35 -05003881 It is recommended that you bundle the initramfs image with the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003882 kernel image to avoid circular dependencies between the kernel
3883 recipe and the initramfs recipe should the initramfs image include
3884 kernel modules.
3885
Andrew Geissler09036742021-06-25 14:25:14 -05003886 Setting the :term:`INITRAMFS_IMAGE_BUNDLE` flag causes the initramfs
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003887 image to be unpacked into the ``${B}/usr/`` directory. The unpacked
3888 initramfs image is then passed to the kernel's ``Makefile`` using the
3889 :term:`CONFIG_INITRAMFS_SOURCE`
3890 variable, allowing the initramfs image to be built into the kernel
3891 normally.
3892
3893 .. note::
3894
Patrick Williams93c203f2021-10-06 16:15:23 -05003895 Bundling the initramfs with the kernel conflates the code in the initramfs
3896 with the GPLv2 licensed Linux kernel binary. Thus only GPLv2 compatible
3897 software may be part of a bundled initramfs.
3898
3899 .. note::
3900
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003901 If you choose to not bundle the initramfs image with the kernel
3902 image, you are essentially using an
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003903 `Initial RAM Disk (initrd) <https://en.wikipedia.org/wiki/Initrd>`__.
3904 Creating an initrd is handled primarily through the :term:`INITRD_IMAGE`,
3905 ``INITRD_LIVE``, and ``INITRD_IMAGE_LIVE`` variables. For more
3906 information, see the :ref:`ref-classes-image-live` file.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003907
39083. *Optionally Add Items to the initramfs Image Through the initramfs
3909 Image Recipe:* If you add items to the initramfs image by way of its
3910 recipe, you should use
3911 :term:`PACKAGE_INSTALL`
3912 rather than
3913 :term:`IMAGE_INSTALL`.
Andrew Geissler09036742021-06-25 14:25:14 -05003914 :term:`PACKAGE_INSTALL` gives more direct control of what is added to the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003915 image as compared to the defaults you might not necessarily want that
3916 are set by the :ref:`image <ref-classes-image>`
3917 or :ref:`core-image <ref-classes-core-image>`
3918 classes.
3919
39204. *Build the Kernel Image and the initramfs Image:* Build your kernel
3921 image using BitBake. Because the initramfs image recipe is a
3922 dependency of the kernel image, the initramfs image is built as well
3923 and bundled with the kernel image if you used the
3924 :term:`INITRAMFS_IMAGE_BUNDLE`
3925 variable described earlier.
3926
Andrew Geissler7e0e3c02022-02-25 20:34:39 +00003927Bundling an Initramfs Image From a Separate Multiconfig
3928~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3929
3930There may be a case where we want to build an initramfs image which does not
3931inherit the same distro policy as our main image, for example, we may want
3932our main image to use ``TCLIBC="glibc"``, but to use ``TCLIBC="musl"`` in our initramfs
3933image to keep a smaller footprint. However, by performing the steps mentioned
3934above the initramfs image will inherit ``TCLIBC="glibc"`` without allowing us
3935to override it.
3936
3937To achieve this, you need to perform some additional steps:
3938
39391. *Create a multiconfig for your initramfs image:* You can perform the steps
3940 on ":ref:`dev-manual/common-tasks:building images for multiple targets using multiple configurations`" to create a separate multiconfig.
3941 For the sake of simplicity let's assume such multiconfig is called: ``initramfscfg.conf`` and
3942 contains the variables::
3943
3944 TMPDIR="${TOPDIR}/tmp-initramfscfg"
3945 TCLIBC="musl"
3946
39472. *Set additional initramfs variables on your main configuration:*
3948 Additionally, on your main configuration (``local.conf``) you need to set the
3949 variables::
3950
3951 INITRAMFS_MULTICONFIG = "initramfscfg"
3952 INITRAMFS_DEPLOY_DIR_IMAGE = "${TOPDIR}/tmp-initramfscfg/deploy/images/${MACHINE}"
3953
3954 The variables :term:`INITRAMFS_MULTICONFIG` and :term:`INITRAMFS_DEPLOY_DIR_IMAGE`
3955 are used to create a multiconfig dependency from the kernel to the :term:`INITRAMFS_IMAGE`
3956 to be built coming from the ``initramfscfg`` multiconfig, and to let the
3957 buildsystem know where the :term:`INITRAMFS_IMAGE` will be located.
3958
3959 Building a system with such configuration will build the kernel using the
3960 main configuration but the ``do_bundle_initramfs`` task will grab the
3961 selected :term:`INITRAMFS_IMAGE` from :term:`INITRAMFS_DEPLOY_DIR_IMAGE`
3962 instead, resulting in a musl based initramfs image bundled in the kernel
3963 but a glibc based main image.
3964
3965 The same is applicable to avoid inheriting :term:`DISTRO_FEATURES` on :term:`INITRAMFS_IMAGE`
3966 or to build a different :term:`DISTRO` for it such as ``poky-tiny``.
3967
3968
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003969Building a Tiny System
3970----------------------
3971
3972Very small distributions have some significant advantages such as
3973requiring less on-die or in-package memory (cheaper), better performance
3974through efficient cache usage, lower power requirements due to less
3975memory, faster boot times, and reduced development overhead. Some
3976real-world examples where a very small distribution gives you distinct
3977advantages are digital cameras, medical devices, and small headless
3978systems.
3979
3980This section presents information that shows you how you can trim your
3981distribution to even smaller sizes than the ``poky-tiny`` distribution,
3982which is around 5 Mbytes, that can be built out-of-the-box using the
3983Yocto Project.
3984
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003985Tiny System Overview
3986~~~~~~~~~~~~~~~~~~~~
3987
3988The following list presents the overall steps you need to consider and
3989perform to create distributions with smaller root filesystems, achieve
3990faster boot times, maintain your critical functionality, and avoid
3991initial RAM disks:
3992
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05003993- :ref:`Determine your goals and guiding principles
3994 <dev-manual/common-tasks:goals and guiding principles>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003995
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05003996- :ref:`dev-manual/common-tasks:understand what contributes to your image size`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003997
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05003998- :ref:`Reduce the size of the root filesystem
3999 <dev-manual/common-tasks:trim the root filesystem>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004000
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05004001- :ref:`Reduce the size of the kernel <dev-manual/common-tasks:trim the kernel>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004002
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05004003- :ref:`dev-manual/common-tasks:remove package management requirements`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004004
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05004005- :ref:`dev-manual/common-tasks:look for other ways to minimize size`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004006
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05004007- :ref:`dev-manual/common-tasks:iterate on the process`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004008
4009Goals and Guiding Principles
4010~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4011
4012Before you can reach your destination, you need to know where you are
4013going. Here is an example list that you can use as a guide when creating
4014very small distributions:
4015
4016- Determine how much space you need (e.g. a kernel that is 1 Mbyte or
4017 less and a root filesystem that is 3 Mbytes or less).
4018
4019- Find the areas that are currently taking 90% of the space and
4020 concentrate on reducing those areas.
4021
4022- Do not create any difficult "hacks" to achieve your goals.
4023
4024- Leverage the device-specific options.
4025
4026- Work in a separate layer so that you keep changes isolated. For
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05004027 information on how to create layers, see the
4028 ":ref:`dev-manual/common-tasks:understanding and creating layers`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004029
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004030Understand What Contributes to Your Image Size
4031~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4032
4033It is easiest to have something to start with when creating your own
4034distribution. You can use the Yocto Project out-of-the-box to create the
4035``poky-tiny`` distribution. Ultimately, you will want to make changes in
4036your own distribution that are likely modeled after ``poky-tiny``.
4037
4038.. note::
4039
Andrew Geissler09036742021-06-25 14:25:14 -05004040 To use ``poky-tiny`` in your build, set the :term:`DISTRO` variable in your
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004041 ``local.conf`` file to "poky-tiny" as described in the
Andrew Geissler09209ee2020-12-13 08:44:15 -06004042 ":ref:`dev-manual/common-tasks:creating your own distribution`"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004043 section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004044
4045Understanding some memory concepts will help you reduce the system size.
4046Memory consists of static, dynamic, and temporary memory. Static memory
4047is the TEXT (code), DATA (initialized data in the code), and BSS
4048(uninitialized data) sections. Dynamic memory represents memory that is
4049allocated at runtime: stacks, hash tables, and so forth. Temporary
4050memory is recovered after the boot process. This memory consists of
4051memory used for decompressing the kernel and for the ``__init__``
4052functions.
4053
4054To help you see where you currently are with kernel and root filesystem
4055sizes, you can use two tools found in the :term:`Source Directory`
4056in the
4057``scripts/tiny/`` directory:
4058
4059- ``ksize.py``: Reports component sizes for the kernel build objects.
4060
4061- ``dirsize.py``: Reports component sizes for the root filesystem.
4062
4063This next tool and command help you organize configuration fragments and
4064view file dependencies in a human-readable form:
4065
4066- ``merge_config.sh``: Helps you manage configuration files and
4067 fragments within the kernel. With this tool, you can merge individual
4068 configuration fragments together. The tool allows you to make
4069 overrides and warns you of any missing configuration options. The
4070 tool is ideal for allowing you to iterate on configurations, create
4071 minimal configurations, and create configuration files for different
4072 machines without having to duplicate your process.
4073
4074 The ``merge_config.sh`` script is part of the Linux Yocto kernel Git
4075 repositories (i.e. ``linux-yocto-3.14``, ``linux-yocto-3.10``,
4076 ``linux-yocto-3.8``, and so forth) in the ``scripts/kconfig``
4077 directory.
4078
4079 For more information on configuration fragments, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06004080 ":ref:`kernel-dev/common:creating configuration fragments`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004081 section in the Yocto Project Linux Kernel Development Manual.
4082
4083- ``bitbake -u taskexp -g bitbake_target``: Using the BitBake command
4084 with these options brings up a Dependency Explorer from which you can
4085 view file dependencies. Understanding these dependencies allows you
4086 to make informed decisions when cutting out various pieces of the
4087 kernel and root filesystem.
4088
4089Trim the Root Filesystem
4090~~~~~~~~~~~~~~~~~~~~~~~~
4091
4092The root filesystem is made up of packages for booting, libraries, and
4093applications. To change things, you can configure how the packaging
4094happens, which changes the way you build them. You can also modify the
4095filesystem itself or select a different filesystem.
4096
4097First, find out what is hogging your root filesystem by running the
Andrew Geisslerc926e172021-05-07 16:11:35 -05004098``dirsize.py`` script from your root directory::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004099
4100 $ cd root-directory-of-image
4101 $ dirsize.py 100000 > dirsize-100k.log
4102 $ cat dirsize-100k.log
4103
4104You can apply a filter to the script to ignore files
4105under a certain size. The previous example filters out any files below
4106100 Kbytes. The sizes reported by the tool are uncompressed, and thus
4107will be smaller by a relatively constant factor in a compressed root
4108filesystem. When you examine your log file, you can focus on areas of
4109the root filesystem that take up large amounts of memory.
4110
4111You need to be sure that what you eliminate does not cripple the
4112functionality you need. One way to see how packages relate to each other
Andrew Geisslerc926e172021-05-07 16:11:35 -05004113is by using the Dependency Explorer UI with the BitBake command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004114
4115 $ cd image-directory
4116 $ bitbake -u taskexp -g image
4117
4118Use the interface to
4119select potential packages you wish to eliminate and see their dependency
4120relationships.
4121
4122When deciding how to reduce the size, get rid of packages that result in
4123minimal impact on the feature set. For example, you might not need a VGA
4124display. Or, you might be able to get by with ``devtmpfs`` and ``mdev``
4125instead of ``udev``.
4126
4127Use your ``local.conf`` file to make changes. For example, to eliminate
4128``udev`` and ``glib``, set the following in the local configuration
Andrew Geisslerc926e172021-05-07 16:11:35 -05004129file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004130
4131 VIRTUAL-RUNTIME_dev_manager = ""
4132
4133Finally, you should consider exactly the type of root filesystem you
4134need to meet your needs while also reducing its size. For example,
4135consider ``cramfs``, ``squashfs``, ``ubifs``, ``ext2``, or an
4136``initramfs`` using ``initramfs``. Be aware that ``ext3`` requires a 1
4137Mbyte journal. If you are okay with running read-only, you do not need
4138this journal.
4139
4140.. note::
4141
4142 After each round of elimination, you need to rebuild your system and
4143 then use the tools to see the effects of your reductions.
4144
4145Trim the Kernel
4146~~~~~~~~~~~~~~~
4147
4148The kernel is built by including policies for hardware-independent
4149aspects. What subsystems do you enable? For what architecture are you
4150building? Which drivers do you build by default?
4151
4152.. note::
4153
4154 You can modify the kernel source if you want to help with boot time.
4155
4156Run the ``ksize.py`` script from the top-level Linux build directory to
Andrew Geisslerc926e172021-05-07 16:11:35 -05004157get an idea of what is making up the kernel::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004158
4159 $ cd top-level-linux-build-directory
4160 $ ksize.py > ksize.log
4161 $ cat ksize.log
4162
4163When you examine the log, you will see how much space is taken up with
4164the built-in ``.o`` files for drivers, networking, core kernel files,
4165filesystem, sound, and so forth. The sizes reported by the tool are
4166uncompressed, and thus will be smaller by a relatively constant factor
4167in a compressed kernel image. Look to reduce the areas that are large
4168and taking up around the "90% rule."
4169
4170To examine, or drill down, into any particular area, use the ``-d``
Andrew Geisslerc926e172021-05-07 16:11:35 -05004171option with the script::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004172
4173 $ ksize.py -d > ksize.log
4174
4175Using this option
4176breaks out the individual file information for each area of the kernel
4177(e.g. drivers, networking, and so forth).
4178
4179Use your log file to see what you can eliminate from the kernel based on
4180features you can let go. For example, if you are not going to need
4181sound, you do not need any drivers that support sound.
4182
4183After figuring out what to eliminate, you need to reconfigure the kernel
4184to reflect those changes during the next build. You could run
4185``menuconfig`` and make all your changes at once. However, that makes it
4186difficult to see the effects of your individual eliminations and also
4187makes it difficult to replicate the changes for perhaps another target
4188device. A better method is to start with no configurations using
4189``allnoconfig``, create configuration fragments for individual changes,
4190and then manage the fragments into a single configuration file using
4191``merge_config.sh``. The tool makes it easy for you to iterate using the
4192configuration change and build cycle.
4193
4194Each time you make configuration changes, you need to rebuild the kernel
4195and check to see what impact your changes had on the overall size.
4196
4197Remove Package Management Requirements
4198~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4199
4200Packaging requirements add size to the image. One way to reduce the size
4201of the image is to remove all the packaging requirements from the image.
4202This reduction includes both removing the package manager and its unique
4203dependencies as well as removing the package management data itself.
4204
4205To eliminate all the packaging requirements for an image, be sure that
4206"package-management" is not part of your
4207:term:`IMAGE_FEATURES`
4208statement for the image. When you remove this feature, you are removing
4209the package manager as well as its dependencies from the root
4210filesystem.
4211
4212Look for Other Ways to Minimize Size
4213~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4214
4215Depending on your particular circumstances, other areas that you can
4216trim likely exist. The key to finding these areas is through tools and
4217methods described here combined with experimentation and iteration. Here
4218are a couple of areas to experiment with:
4219
4220- ``glibc``: In general, follow this process:
4221
4222 1. Remove ``glibc`` features from
4223 :term:`DISTRO_FEATURES`
4224 that you think you do not need.
4225
4226 2. Build your distribution.
4227
4228 3. If the build fails due to missing symbols in a package, determine
4229 if you can reconfigure the package to not need those features. For
4230 example, change the configuration to not support wide character
4231 support as is done for ``ncurses``. Or, if support for those
4232 characters is needed, determine what ``glibc`` features provide
4233 the support and restore the configuration.
4234
4235 4. Rebuild and repeat the process.
4236
4237- ``busybox``: For BusyBox, use a process similar as described for
4238 ``glibc``. A difference is you will need to boot the resulting system
4239 to see if you are able to do everything you expect from the running
4240 system. You need to be sure to integrate configuration fragments into
4241 Busybox because BusyBox handles its own core features and then allows
4242 you to add configuration fragments on top.
4243
4244Iterate on the Process
4245~~~~~~~~~~~~~~~~~~~~~~
4246
4247If you have not reached your goals on system size, you need to iterate
4248on the process. The process is the same. Use the tools and see just what
4249is taking up 90% of the root filesystem and the kernel. Decide what you
4250can eliminate without limiting your device beyond what you need.
4251
4252Depending on your system, a good place to look might be Busybox, which
4253provides a stripped down version of Unix tools in a single, executable
4254file. You might be able to drop virtual terminal services or perhaps
4255ipv6.
4256
4257Building Images for More than One Machine
4258-----------------------------------------
4259
4260A common scenario developers face is creating images for several
4261different machines that use the same software environment. In this
4262situation, it is tempting to set the tunings and optimization flags for
4263each build specifically for the targeted hardware (i.e. "maxing out" the
4264tunings). Doing so can considerably add to build times and package feed
4265maintenance collectively for the machines. For example, selecting tunes
4266that are extremely specific to a CPU core used in a system might enable
4267some micro optimizations in GCC for that particular system but would
4268otherwise not gain you much of a performance difference across the other
4269systems as compared to using a more general tuning across all the builds
4270(e.g. setting :term:`DEFAULTTUNE`
4271specifically for each machine's build). Rather than "max out" each
4272build's tunings, you can take steps that cause the OpenEmbedded build
4273system to reuse software across the various machines where it makes
4274sense.
4275
4276If build speed and package feed maintenance are considerations, you
4277should consider the points in this section that can help you optimize
4278your tunings to best consider build times and package feed maintenance.
4279
4280- *Share the Build Directory:* If at all possible, share the
4281 :term:`TMPDIR` across builds. The
4282 Yocto Project supports switching between different
4283 :term:`MACHINE` values in the same
Andrew Geissler09036742021-06-25 14:25:14 -05004284 :term:`TMPDIR`. This practice is well supported and regularly used by
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004285 developers when building for multiple machines. When you use the same
Andrew Geissler09036742021-06-25 14:25:14 -05004286 :term:`TMPDIR` for multiple machine builds, the OpenEmbedded build system
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004287 can reuse the existing native and often cross-recipes for multiple
4288 machines. Thus, build time decreases.
4289
4290 .. note::
4291
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004292 If :term:`DISTRO` settings change or fundamental configuration settings
Andrew Geissler09036742021-06-25 14:25:14 -05004293 such as the filesystem layout, you need to work with a clean :term:`TMPDIR`.
4294 Sharing :term:`TMPDIR` under these circumstances might work but since it is
Andrew Geissler5f350902021-07-23 13:09:54 -04004295 not guaranteed, you should use a clean :term:`TMPDIR`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004296
4297- *Enable the Appropriate Package Architecture:* By default, the
4298 OpenEmbedded build system enables three levels of package
4299 architectures: "all", "tune" or "package", and "machine". Any given
4300 recipe usually selects one of these package architectures (types) for
4301 its output. Depending for what a given recipe creates packages,
4302 making sure you enable the appropriate package architecture can
4303 directly impact the build time.
4304
4305 A recipe that just generates scripts can enable "all" architecture
4306 because there are no binaries to build. To specifically enable "all"
4307 architecture, be sure your recipe inherits the
4308 :ref:`allarch <ref-classes-allarch>` class.
4309 This class is useful for "all" architectures because it configures
4310 many variables so packages can be used across multiple architectures.
4311
4312 If your recipe needs to generate packages that are machine-specific
4313 or when one of the build or runtime dependencies is already
4314 machine-architecture dependent, which makes your recipe also
4315 machine-architecture dependent, make sure your recipe enables the
4316 "machine" package architecture through the
4317 :term:`MACHINE_ARCH`
Andrew Geisslerc926e172021-05-07 16:11:35 -05004318 variable::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004319
4320 PACKAGE_ARCH = "${MACHINE_ARCH}"
4321
4322 When you do not
4323 specifically enable a package architecture through the
4324 :term:`PACKAGE_ARCH`, The
4325 OpenEmbedded build system defaults to the
Andrew Geisslerc926e172021-05-07 16:11:35 -05004326 :term:`TUNE_PKGARCH` setting::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004327
4328 PACKAGE_ARCH = "${TUNE_PKGARCH}"
4329
4330- *Choose a Generic Tuning File if Possible:* Some tunes are more
4331 generic and can run on multiple targets (e.g. an ``armv5`` set of
4332 packages could run on ``armv6`` and ``armv7`` processors in most
4333 cases). Similarly, ``i486`` binaries could work on ``i586`` and
4334 higher processors. You should realize, however, that advances on
4335 newer processor versions would not be used.
4336
4337 If you select the same tune for several different machines, the
4338 OpenEmbedded build system reuses software previously built, thus
4339 speeding up the overall build time. Realize that even though a new
4340 sysroot for each machine is generated, the software is not recompiled
4341 and only one package feed exists.
4342
William A. Kennington IIIac69b482021-06-02 12:28:27 -07004343- *Manage Granular Level Packaging:* Sometimes there are cases where
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004344 injecting another level of package architecture beyond the three
4345 higher levels noted earlier can be useful. For example, consider how
4346 NXP (formerly Freescale) allows for the easy reuse of binary packages
4347 in their layer
Andrew Geissler09209ee2020-12-13 08:44:15 -06004348 :yocto_git:`meta-freescale </meta-freescale/>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004349 In this example, the
Andrew Geissler09209ee2020-12-13 08:44:15 -06004350 :yocto_git:`fsl-dynamic-packagearch </meta-freescale/tree/classes/fsl-dynamic-packagearch.bbclass>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004351 class shares GPU packages for i.MX53 boards because all boards share
4352 the AMD GPU. The i.MX6-based boards can do the same because all
4353 boards share the Vivante GPU. This class inspects the BitBake
4354 datastore to identify if the package provides or depends on one of
4355 the sub-architecture values. If so, the class sets the
4356 :term:`PACKAGE_ARCH` value
4357 based on the ``MACHINE_SUBARCH`` value. If the package does not
4358 provide or depend on one of the sub-architecture values but it
4359 matches a value in the machine-specific filter, it sets
4360 :term:`MACHINE_ARCH`. This
4361 behavior reduces the number of packages built and saves build time by
4362 reusing binaries.
4363
4364- *Use Tools to Debug Issues:* Sometimes you can run into situations
4365 where software is being rebuilt when you think it should not be. For
4366 example, the OpenEmbedded build system might not be using shared
4367 state between machines when you think it should be. These types of
4368 situations are usually due to references to machine-specific
4369 variables such as :term:`MACHINE`,
4370 :term:`SERIAL_CONSOLES`,
4371 :term:`XSERVER`,
4372 :term:`MACHINE_FEATURES`,
4373 and so forth in code that is supposed to only be tune-specific or
4374 when the recipe depends
4375 (:term:`DEPENDS`,
4376 :term:`RDEPENDS`,
4377 :term:`RRECOMMENDS`,
4378 :term:`RSUGGESTS`, and so forth)
4379 on some other recipe that already has
4380 :term:`PACKAGE_ARCH` defined
4381 as "${MACHINE_ARCH}".
4382
4383 .. note::
4384
4385 Patches to fix any issues identified are most welcome as these
4386 issues occasionally do occur.
4387
4388 For such cases, you can use some tools to help you sort out the
4389 situation:
4390
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004391 - ``state-diff-machines.sh``*:* You can find this tool in the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004392 ``scripts`` directory of the Source Repositories. See the comments
4393 in the script for information on how to use the tool.
4394
4395 - *BitBake's "-S printdiff" Option:* Using this option causes
4396 BitBake to try to establish the closest signature match it can
4397 (e.g. in the shared state cache) and then run ``bitbake-diffsigs``
4398 over the matches to determine the stamps and delta where these two
4399 stamp trees diverge.
4400
4401Building Software from an External Source
4402-----------------------------------------
4403
4404By default, the OpenEmbedded build system uses the
4405:term:`Build Directory` when building source
4406code. The build process involves fetching the source files, unpacking
4407them, and then patching them if necessary before the build takes place.
4408
William A. Kennington IIIac69b482021-06-02 12:28:27 -07004409There are situations where you might want to build software from source
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004410files that are external to and thus outside of the OpenEmbedded build
4411system. For example, suppose you have a project that includes a new BSP
4412with a heavily customized kernel. And, you want to minimize exposing the
4413build system to the development team so that they can focus on their
4414project and maintain everyone's workflow as much as possible. In this
4415case, you want a kernel source directory on the development machine
4416where the development occurs. You want the recipe's
4417:term:`SRC_URI` variable to point to
4418the external directory and use it as is, not copy it.
4419
4420To build from software that comes from an external source, all you need
4421to do is inherit the
4422:ref:`externalsrc <ref-classes-externalsrc>` class
4423and then set the
4424:term:`EXTERNALSRC` variable to
4425point to your external source code. Here are the statements to put in
Andrew Geisslerc926e172021-05-07 16:11:35 -05004426your ``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004427
4428 INHERIT += "externalsrc"
Patrick Williams0ca19cc2021-08-16 14:03:13 -05004429 EXTERNALSRC:pn-myrecipe = "path-to-your-source-tree"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004430
4431This next example shows how to accomplish the same thing by setting
Andrew Geissler09036742021-06-25 14:25:14 -05004432:term:`EXTERNALSRC` in the recipe itself or in the recipe's append file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004433
4434 EXTERNALSRC = "path"
4435 EXTERNALSRC_BUILD = "path"
4436
4437.. note::
4438
4439 In order for these settings to take effect, you must globally or
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004440 locally inherit the :ref:`externalsrc <ref-classes-externalsrc>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004441 class.
4442
Andrew Geissler7e0e3c02022-02-25 20:34:39 +00004443By default, :ref:`ref-classes-externalsrc` builds the source code in a
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004444directory separate from the external source directory as specified by
4445:term:`EXTERNALSRC`. If you need
4446to have the source built in the same directory in which it resides, or
4447some other nominated directory, you can set
4448:term:`EXTERNALSRC_BUILD`
Andrew Geisslerc926e172021-05-07 16:11:35 -05004449to point to that directory::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004450
Patrick Williams0ca19cc2021-08-16 14:03:13 -05004451 EXTERNALSRC_BUILD:pn-myrecipe = "path-to-your-source-tree"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004452
4453Replicating a Build Offline
4454---------------------------
4455
4456It can be useful to take a "snapshot" of upstream sources used in a
4457build and then use that "snapshot" later to replicate the build offline.
4458To do so, you need to first prepare and populate your downloads
4459directory your "snapshot" of files. Once your downloads directory is
4460ready, you can use it at any time and from any machine to replicate your
4461build.
4462
4463Follow these steps to populate your Downloads directory:
4464
44651. *Create a Clean Downloads Directory:* Start with an empty downloads
4466 directory (:term:`DL_DIR`). You
4467 start with an empty downloads directory by either removing the files
Andrew Geissler09036742021-06-25 14:25:14 -05004468 in the existing directory or by setting :term:`DL_DIR` to point to either
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004469 an empty location or one that does not yet exist.
4470
44712. *Generate Tarballs of the Source Git Repositories:* Edit your
Andrew Geisslerc926e172021-05-07 16:11:35 -05004472 ``local.conf`` configuration file as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004473
4474 DL_DIR = "/home/your-download-dir/"
4475 BB_GENERATE_MIRROR_TARBALLS = "1"
4476
4477 During
4478 the fetch process in the next step, BitBake gathers the source files
Andrew Geissler09036742021-06-25 14:25:14 -05004479 and creates tarballs in the directory pointed to by :term:`DL_DIR`. See
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004480 the
4481 :term:`BB_GENERATE_MIRROR_TARBALLS`
4482 variable for more information.
4483
44843. *Populate Your Downloads Directory Without Building:* Use BitBake to
Andrew Geisslerc926e172021-05-07 16:11:35 -05004485 fetch your sources but inhibit the build::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004486
4487 $ bitbake target --runonly=fetch
4488
4489 The downloads directory (i.e. ``${DL_DIR}``) now has
4490 a "snapshot" of the source files in the form of tarballs, which can
4491 be used for the build.
4492
44934. *Optionally Remove Any Git or other SCM Subdirectories From the
4494 Downloads Directory:* If you want, you can clean up your downloads
4495 directory by removing any Git or other Source Control Management
4496 (SCM) subdirectories such as ``${DL_DIR}/git2/*``. The tarballs
4497 already contain these subdirectories.
4498
4499Once your downloads directory has everything it needs regarding source
4500files, you can create your "own-mirror" and build your target.
4501Understand that you can use the files to build the target offline from
4502any machine and at any time.
4503
4504Follow these steps to build your target using the files in the downloads
4505directory:
4506
45071. *Using Local Files Only:* Inside your ``local.conf`` file, add the
Andrew Geissler595f6302022-01-24 19:11:47 +00004508 :term:`SOURCE_MIRROR_URL` variable, inherit the
4509 :ref:`own-mirrors <ref-classes-own-mirrors>` class, and use the
4510 :term:`BB_NO_NETWORK` variable to your ``local.conf``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004511 ::
4512
4513 SOURCE_MIRROR_URL ?= "file:///home/your-download-dir/"
4514 INHERIT += "own-mirrors"
4515 BB_NO_NETWORK = "1"
4516
Andrew Geissler595f6302022-01-24 19:11:47 +00004517 The :term:`SOURCE_MIRROR_URL` and :ref:`own-mirrors <ref-classes-own-mirrors>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004518 class set up the system to use the downloads directory as your "own
Andrew Geissler09036742021-06-25 14:25:14 -05004519 mirror". Using the :term:`BB_NO_NETWORK` variable makes sure that
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004520 BitBake's fetching process in step 3 stays local, which means files
4521 from your "own-mirror" are used.
4522
45232. *Start With a Clean Build:* You can start with a clean build by
4524 removing the
4525 ``${``\ :term:`TMPDIR`\ ``}``
4526 directory or using a new :term:`Build Directory`.
4527
Andrew Geisslerc926e172021-05-07 16:11:35 -050045283. *Build Your Target:* Use BitBake to build your target::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004529
4530 $ bitbake target
4531
4532 The build completes using the known local "snapshot" of source
4533 files from your mirror. The resulting tarballs for your "snapshot" of
4534 source files are in the downloads directory.
4535
4536 .. note::
4537
4538 The offline build does not work if recipes attempt to find the
4539 latest version of software by setting
4540 :term:`SRCREV` to
Andrew Geisslerc926e172021-05-07 16:11:35 -05004541 ``${``\ :term:`AUTOREV`\ ``}``::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004542
4543 SRCREV = "${AUTOREV}"
4544
Andrew Geissler09036742021-06-25 14:25:14 -05004545 When a recipe sets :term:`SRCREV` to
Andrew Geissler5199d832021-09-24 16:47:35 -05004546 ``${``\ :term:`AUTOREV`\ ``}``, the build system accesses the network in an
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004547 attempt to determine the latest version of software from the SCM.
Andrew Geissler09036742021-06-25 14:25:14 -05004548 Typically, recipes that use :term:`AUTOREV` are custom or modified
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004549 recipes. Recipes that reside in public repositories usually do not
Andrew Geissler09036742021-06-25 14:25:14 -05004550 use :term:`AUTOREV`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004551
Andrew Geissler09036742021-06-25 14:25:14 -05004552 If you do have recipes that use :term:`AUTOREV`, you can take steps to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004553 still use the recipes in an offline build. Do the following:
4554
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05004555 1. Use a configuration generated by enabling :ref:`build
4556 history <dev-manual/common-tasks:maintaining build output quality>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004557
4558 2. Use the ``buildhistory-collect-srcrevs`` command to collect the
Andrew Geissler09036742021-06-25 14:25:14 -05004559 stored :term:`SRCREV` values from the build's history. For more
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05004560 information on collecting these values, see the
4561 ":ref:`dev-manual/common-tasks:build history package information`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004562 section.
4563
4564 3. Once you have the correct source revisions, you can modify
Andrew Geissler09036742021-06-25 14:25:14 -05004565 those recipes to set :term:`SRCREV` to specific versions of the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004566 software.
4567
4568Speeding Up a Build
4569===================
4570
4571Build time can be an issue. By default, the build system uses simple
4572controls to try and maximize build efficiency. In general, the default
4573settings for all the following variables result in the most efficient
4574build times when dealing with single socket systems (i.e. a single CPU).
4575If you have multiple CPUs, you might try increasing the default values
4576to gain more speed. See the descriptions in the glossary for each
4577variable for more information:
4578
4579- :term:`BB_NUMBER_THREADS`:
4580 The maximum number of threads BitBake simultaneously executes.
4581
Patrick Williams213cb262021-08-07 19:21:33 -05004582- :term:`BB_NUMBER_PARSE_THREADS`:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004583 The number of threads BitBake uses during parsing.
4584
4585- :term:`PARALLEL_MAKE`: Extra
4586 options passed to the ``make`` command during the
4587 :ref:`ref-tasks-compile` task in
4588 order to specify parallel compilation on the local build host.
4589
4590- :term:`PARALLEL_MAKEINST`:
4591 Extra options passed to the ``make`` command during the
4592 :ref:`ref-tasks-install` task in
4593 order to specify parallel installation on the local build host.
4594
4595As mentioned, these variables all scale to the number of processor cores
4596available on the build system. For single socket systems, this
4597auto-scaling ensures that the build system fundamentally takes advantage
4598of potential parallel operations during the build based on the build
4599machine's capabilities.
4600
4601Following are additional factors that can affect build speed:
4602
4603- File system type: The file system type that the build is being
4604 performed on can also influence performance. Using ``ext4`` is
4605 recommended as compared to ``ext2`` and ``ext3`` due to ``ext4``
4606 improved features such as extents.
4607
4608- Disabling the updating of access time using ``noatime``: The
4609 ``noatime`` mount option prevents the build system from updating file
4610 and directory access times.
4611
4612- Setting a longer commit: Using the "commit=" mount option increases
4613 the interval in seconds between disk cache writes. Changing this
4614 interval from the five second default to something longer increases
4615 the risk of data loss but decreases the need to write to the disk,
4616 thus increasing the build performance.
4617
4618- Choosing the packaging backend: Of the available packaging backends,
4619 IPK is the fastest. Additionally, selecting a singular packaging
4620 backend also helps.
4621
4622- Using ``tmpfs`` for :term:`TMPDIR`
4623 as a temporary file system: While this can help speed up the build,
4624 the benefits are limited due to the compiler using ``-pipe``. The
4625 build system goes to some lengths to avoid ``sync()`` calls into the
4626 file system on the principle that if there was a significant failure,
4627 the :term:`Build Directory`
4628 contents could easily be rebuilt.
4629
4630- Inheriting the
4631 :ref:`rm_work <ref-classes-rm-work>` class:
4632 Inheriting this class has shown to speed up builds due to
4633 significantly lower amounts of data stored in the data cache as well
4634 as on disk. Inheriting this class also makes cleanup of
4635 :term:`TMPDIR` faster, at the
4636 expense of being easily able to dive into the source code. File
4637 system maintainers have recommended that the fastest way to clean up
4638 large numbers of files is to reformat partitions rather than delete
4639 files due to the linear nature of partitions. This, of course,
4640 assumes you structure the disk partitions and file systems in a way
4641 that this is practical.
4642
4643Aside from the previous list, you should keep some trade offs in mind
4644that can help you speed up the build:
4645
4646- Remove items from
4647 :term:`DISTRO_FEATURES`
4648 that you might not need.
4649
4650- Exclude debug symbols and other debug information: If you do not need
4651 these symbols and other debug information, disabling the ``*-dbg``
4652 package generation can speed up the build. You can disable this
4653 generation by setting the
4654 :term:`INHIBIT_PACKAGE_DEBUG_SPLIT`
4655 variable to "1".
4656
4657- Disable static library generation for recipes derived from
4658 ``autoconf`` or ``libtool``: Following is an example showing how to
4659 disable static libraries and still provide an override to handle
Andrew Geisslerc926e172021-05-07 16:11:35 -05004660 exceptions::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004661
4662 STATICLIBCONF = "--disable-static"
Patrick Williams0ca19cc2021-08-16 14:03:13 -05004663 STATICLIBCONF:sqlite3-native = ""
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004664 EXTRA_OECONF += "${STATICLIBCONF}"
4665
4666 .. note::
4667
4668 - Some recipes need static libraries in order to work correctly
4669 (e.g. ``pseudo-native`` needs ``sqlite3-native``). Overrides,
4670 as in the previous example, account for these kinds of
4671 exceptions.
4672
4673 - Some packages have packaging code that assumes the presence of
4674 the static libraries. If so, you might need to exclude them as
4675 well.
4676
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004677Working With Libraries
4678======================
4679
4680Libraries are an integral part of your system. This section describes
4681some common practices you might find helpful when working with libraries
4682to build your system:
4683
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05004684- :ref:`How to include static library files
4685 <dev-manual/common-tasks:including static library files>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004686
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05004687- :ref:`How to use the Multilib feature to combine multiple versions of
4688 library files into a single image
4689 <dev-manual/common-tasks:combining multiple versions of library files into one image>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004690
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05004691- :ref:`How to install multiple versions of the same library in parallel on
4692 the same system
4693 <dev-manual/common-tasks:installing multiple versions of the same library>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004694
4695Including Static Library Files
4696------------------------------
4697
4698If you are building a library and the library offers static linking, you
4699can control which static library files (``*.a`` files) get included in
4700the built library.
4701
4702The :term:`PACKAGES` and
Patrick Williams0ca19cc2021-08-16 14:03:13 -05004703:term:`FILES:* <FILES>` variables in the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004704``meta/conf/bitbake.conf`` configuration file define how files installed
Andrew Geissler09036742021-06-25 14:25:14 -05004705by the ``do_install`` task are packaged. By default, the :term:`PACKAGES`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004706variable includes ``${PN}-staticdev``, which represents all static
4707library files.
4708
4709.. note::
4710
4711 Some previously released versions of the Yocto Project defined the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004712 static library files through ``${PN}-dev``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004713
4714Following is part of the BitBake configuration file, where you can see
Andrew Geisslerc926e172021-05-07 16:11:35 -05004715how the static library files are defined::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004716
4717 PACKAGE_BEFORE_PN ?= ""
Andrew Geissler595f6302022-01-24 19:11:47 +00004718 PACKAGES = "${PN}-src ${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc ${PN}-locale ${PACKAGE_BEFORE_PN} ${PN}"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004719 PACKAGES_DYNAMIC = "^${PN}-locale-.*"
4720 FILES = ""
4721
Patrick Williams0ca19cc2021-08-16 14:03:13 -05004722 FILES:${PN} = "${bindir}/* ${sbindir}/* ${libexecdir}/* ${libdir}/lib*${SOLIBS} \
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004723 ${sysconfdir} ${sharedstatedir} ${localstatedir} \
4724 ${base_bindir}/* ${base_sbindir}/* \
4725 ${base_libdir}/*${SOLIBS} \
Andrew Geissler595f6302022-01-24 19:11:47 +00004726 ${base_prefix}/lib/udev ${prefix}/lib/udev \
4727 ${base_libdir}/udev ${libdir}/udev \
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004728 ${datadir}/${BPN} ${libdir}/${BPN}/* \
4729 ${datadir}/pixmaps ${datadir}/applications \
4730 ${datadir}/idl ${datadir}/omf ${datadir}/sounds \
4731 ${libdir}/bonobo/servers"
4732
Patrick Williams0ca19cc2021-08-16 14:03:13 -05004733 FILES:${PN}-bin = "${bindir}/* ${sbindir}/*"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004734
Patrick Williams0ca19cc2021-08-16 14:03:13 -05004735 FILES:${PN}-doc = "${docdir} ${mandir} ${infodir} ${datadir}/gtk-doc \
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004736 ${datadir}/gnome/help"
Patrick Williams0ca19cc2021-08-16 14:03:13 -05004737 SECTION:${PN}-doc = "doc"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004738
4739 FILES_SOLIBSDEV ?= "${base_libdir}/lib*${SOLIBSDEV} ${libdir}/lib*${SOLIBSDEV}"
Patrick Williams0ca19cc2021-08-16 14:03:13 -05004740 FILES:${PN}-dev = "${includedir} ${FILES_SOLIBSDEV} ${libdir}/*.la \
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004741 ${libdir}/*.o ${libdir}/pkgconfig ${datadir}/pkgconfig \
4742 ${datadir}/aclocal ${base_libdir}/*.o \
Andrew Geissler595f6302022-01-24 19:11:47 +00004743 ${libdir}/${BPN}/*.la ${base_libdir}/*.la \
4744 ${libdir}/cmake ${datadir}/cmake"
Patrick Williams0ca19cc2021-08-16 14:03:13 -05004745 SECTION:${PN}-dev = "devel"
4746 ALLOW_EMPTY:${PN}-dev = "1"
4747 RDEPENDS:${PN}-dev = "${PN} (= ${EXTENDPKGV})"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004748
Patrick Williams0ca19cc2021-08-16 14:03:13 -05004749 FILES:${PN}-staticdev = "${libdir}/*.a ${base_libdir}/*.a ${libdir}/${BPN}/*.a"
4750 SECTION:${PN}-staticdev = "devel"
4751 RDEPENDS:${PN}-staticdev = "${PN}-dev (= ${EXTENDPKGV})"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004752
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004753Combining Multiple Versions of Library Files into One Image
4754-----------------------------------------------------------
4755
4756The build system offers the ability to build libraries with different
4757target optimizations or architecture formats and combine these together
4758into one system image. You can link different binaries in the image
4759against the different libraries as needed for specific use cases. This
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004760feature is called "Multilib".
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004761
4762An example would be where you have most of a system compiled in 32-bit
4763mode using 32-bit libraries, but you have something large, like a
4764database engine, that needs to be a 64-bit application and uses 64-bit
4765libraries. Multilib allows you to get the best of both 32-bit and 64-bit
4766libraries.
4767
4768While the Multilib feature is most commonly used for 32 and 64-bit
4769differences, the approach the build system uses facilitates different
4770target optimizations. You could compile some binaries to use one set of
4771libraries and other binaries to use a different set of libraries. The
4772libraries could differ in architecture, compiler options, or other
4773optimizations.
4774
William A. Kennington IIIac69b482021-06-02 12:28:27 -07004775There are several examples in the ``meta-skeleton`` layer found in the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004776:term:`Source Directory`:
4777
Andrew Geissler595f6302022-01-24 19:11:47 +00004778- :oe_git:`conf/multilib-example.conf </openembedded-core/tree/meta-skeleton/conf/multilib-example.conf>`
4779 configuration file.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004780
Andrew Geissler595f6302022-01-24 19:11:47 +00004781- :oe_git:`conf/multilib-example2.conf </openembedded-core/tree/meta-skeleton/conf/multilib-example2.conf>`
4782 configuration file.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004783
Andrew Geissler595f6302022-01-24 19:11:47 +00004784- :oe_git:`recipes-multilib/images/core-image-multilib-example.bb </openembedded-core/tree/meta-skeleton/recipes-multilib/images/core-image-multilib-example.bb>`
4785 recipe
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004786
4787Preparing to Use Multilib
4788~~~~~~~~~~~~~~~~~~~~~~~~~
4789
4790User-specific requirements drive the Multilib feature. Consequently,
William A. Kennington IIIac69b482021-06-02 12:28:27 -07004791there is no one "out-of-the-box" configuration that would
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004792meet your needs.
4793
4794In order to enable Multilib, you first need to ensure your recipe is
4795extended to support multiple libraries. Many standard recipes are
4796already extended and support multiple libraries. You can check in the
4797``meta/conf/multilib.conf`` configuration file in the
4798:term:`Source Directory` to see how this is
4799done using the
4800:term:`BBCLASSEXTEND` variable.
4801Eventually, all recipes will be covered and this list will not be
4802needed.
4803
Andrew Geissler595f6302022-01-24 19:11:47 +00004804For the most part, the :ref:`Multilib <ref-classes-multilib*>`
4805class extension works automatically to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004806extend the package name from ``${PN}`` to ``${MLPREFIX}${PN}``, where
Andrew Geissler5f350902021-07-23 13:09:54 -04004807:term:`MLPREFIX` is the particular multilib (e.g. "lib32-" or "lib64-").
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004808Standard variables such as
4809:term:`DEPENDS`,
4810:term:`RDEPENDS`,
4811:term:`RPROVIDES`,
4812:term:`RRECOMMENDS`,
4813:term:`PACKAGES`, and
4814:term:`PACKAGES_DYNAMIC` are
4815automatically extended by the system. If you are extending any manual
4816code in the recipe, you can use the ``${MLPREFIX}`` variable to ensure
Andrew Geissler595f6302022-01-24 19:11:47 +00004817those names are extended correctly.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004818
4819Using Multilib
4820~~~~~~~~~~~~~~
4821
4822After you have set up the recipes, you need to define the actual
4823combination of multiple libraries you want to build. You accomplish this
4824through your ``local.conf`` configuration file in the
4825:term:`Build Directory`. An example
Andrew Geisslerc926e172021-05-07 16:11:35 -05004826configuration would be as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004827
4828 MACHINE = "qemux86-64"
4829 require conf/multilib.conf
4830 MULTILIBS = "multilib:lib32"
Patrick Williams0ca19cc2021-08-16 14:03:13 -05004831 DEFAULTTUNE:virtclass-multilib-lib32 = "x86"
Andrew Geisslerd5838332022-05-27 11:33:10 -05004832 IMAGE_INSTALL:append = " lib32-glib-2.0"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004833
4834This example enables an additional library named
4835``lib32`` alongside the normal target packages. When combining these
4836"lib32" alternatives, the example uses "x86" for tuning. For information
4837on this particular tuning, see
4838``meta/conf/machine/include/ia32/arch-ia32.inc``.
4839
4840The example then includes ``lib32-glib-2.0`` in all the images, which
4841illustrates one method of including a multiple library dependency. You
Andrew Geisslerc926e172021-05-07 16:11:35 -05004842can use a normal image build to include this dependency, for example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004843
4844 $ bitbake core-image-sato
4845
4846You can also build Multilib packages
Andrew Geisslerc926e172021-05-07 16:11:35 -05004847specifically with a command like this::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004848
4849 $ bitbake lib32-glib-2.0
4850
4851Additional Implementation Details
4852~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4853
William A. Kennington IIIac69b482021-06-02 12:28:27 -07004854There are generic implementation details as well as details that are specific to
4855package management systems. Following are implementation details
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004856that exist regardless of the package management system:
4857
4858- The typical convention used for the class extension code as used by
4859 Multilib assumes that all package names specified in
4860 :term:`PACKAGES` that contain
4861 ``${PN}`` have ``${PN}`` at the start of the name. When that
4862 convention is not followed and ``${PN}`` appears at the middle or the
4863 end of a name, problems occur.
4864
4865- The :term:`TARGET_VENDOR`
4866 value under Multilib will be extended to "-vendormlmultilib" (e.g.
4867 "-pokymllib32" for a "lib32" Multilib with Poky). The reason for this
4868 slightly unwieldy contraction is that any "-" characters in the
4869 vendor string presently break Autoconf's ``config.sub``, and other
4870 separators are problematic for different reasons.
4871
William A. Kennington IIIac69b482021-06-02 12:28:27 -07004872Here are the implementation details for the RPM Package Management System:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004873
4874- A unique architecture is defined for the Multilib packages, along
4875 with creating a unique deploy folder under ``tmp/deploy/rpm`` in the
4876 :term:`Build Directory`. For
4877 example, consider ``lib32`` in a ``qemux86-64`` image. The possible
4878 architectures in the system are "all", "qemux86_64",
Patrick Williams0ca19cc2021-08-16 14:03:13 -05004879 "lib32:qemux86_64", and "lib32:x86".
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004880
4881- The ``${MLPREFIX}`` variable is stripped from ``${PN}`` during RPM
4882 packaging. The naming for a normal RPM package and a Multilib RPM
4883 package in a ``qemux86-64`` system resolves to something similar to
4884 ``bash-4.1-r2.x86_64.rpm`` and ``bash-4.1.r2.lib32_x86.rpm``,
4885 respectively.
4886
4887- When installing a Multilib image, the RPM backend first installs the
4888 base image and then installs the Multilib libraries.
4889
4890- The build system relies on RPM to resolve the identical files in the
4891 two (or more) Multilib packages.
4892
William A. Kennington IIIac69b482021-06-02 12:28:27 -07004893Here are the implementation details for the IPK Package Management System:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004894
4895- The ``${MLPREFIX}`` is not stripped from ``${PN}`` during IPK
4896 packaging. The naming for a normal RPM package and a Multilib IPK
4897 package in a ``qemux86-64`` system resolves to something like
Patrick Williams0ca19cc2021-08-16 14:03:13 -05004898 ``bash_4.1-r2.x86_64.ipk`` and ``lib32-bash_4.1-rw:x86.ipk``,
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004899 respectively.
4900
4901- The IPK deploy folder is not modified with ``${MLPREFIX}`` because
4902 packages with and without the Multilib feature can exist in the same
4903 folder due to the ``${PN}`` differences.
4904
4905- IPK defines a sanity check for Multilib installation using certain
4906 rules for file comparison, overridden, etc.
4907
4908Installing Multiple Versions of the Same Library
4909------------------------------------------------
4910
William A. Kennington IIIac69b482021-06-02 12:28:27 -07004911There are be situations where you need to install and use multiple versions
4912of the same library on the same system at the same time. This
4913almost always happens when a library API changes and you have
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004914multiple pieces of software that depend on the separate versions of the
4915library. To accommodate these situations, you can install multiple
4916versions of the same library in parallel on the same system.
4917
4918The process is straightforward as long as the libraries use proper
4919versioning. With properly versioned libraries, all you need to do to
4920individually specify the libraries is create separate, appropriately
4921named recipes where the :term:`PN` part of
4922the name includes a portion that differentiates each library version
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004923(e.g. the major part of the version number). Thus, instead of having a
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004924single recipe that loads one version of a library (e.g. ``clutter``),
4925you provide multiple recipes that result in different versions of the
4926libraries you want. As an example, the following two recipes would allow
4927the two separate versions of the ``clutter`` library to co-exist on the
4928same system:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004929
4930.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004931
4932 clutter-1.6_1.6.20.bb
4933 clutter-1.8_1.8.4.bb
4934
4935Additionally, if
4936you have other recipes that depend on a given library, you need to use
4937the :term:`DEPENDS` variable to
4938create the dependency. Continuing with the same example, if you want to
4939have a recipe depend on the 1.8 version of the ``clutter`` library, use
Andrew Geisslerc926e172021-05-07 16:11:35 -05004940the following in your recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004941
4942 DEPENDS = "clutter-1.8"
4943
Andrew Geissler7e0e3c02022-02-25 20:34:39 +00004944Working with Pre-Built Libraries
4945================================
4946
4947Introduction
4948-------------
4949
4950Some library vendors do not release source code for their software but do
4951release pre-built binaries. When shared libraries are built, they should
4952be versioned (see `this article
4953<https://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html>`__
4954for some background), but sometimes this is not done.
4955
4956To summarize, a versioned library must meet two conditions:
4957
4958#. The filename must have the version appended, for example: ``libfoo.so.1.2.3``.
4959#. The library must have the ELF tag ``SONAME`` set to the major version
4960 of the library, for example: ``libfoo.so.1``. You can check this by
4961 running ``readelf -d filename | grep SONAME``.
4962
4963This section shows how to deal with both versioned and unversioned
4964pre-built libraries.
4965
4966Versioned Libraries
4967-------------------
4968
4969In this example we work with pre-built libraries for the FT4222H USB I/O chip.
4970Libraries are built for several target architecture variants and packaged in
4971an archive as follows::
4972
4973 ├── build-arm-hisiv300
4974 │   └── libft4222.so.1.4.4.44
4975 ├── build-arm-v5-sf
4976 │   └── libft4222.so.1.4.4.44
4977 ├── build-arm-v6-hf
4978 │   └── libft4222.so.1.4.4.44
4979 ├── build-arm-v7-hf
4980 │   └── libft4222.so.1.4.4.44
4981 ├── build-arm-v8
4982 │   └── libft4222.so.1.4.4.44
4983 ├── build-i386
4984 │   └── libft4222.so.1.4.4.44
4985 ├── build-i486
4986 │   └── libft4222.so.1.4.4.44
4987 ├── build-mips-eglibc-hf
4988 │   └── libft4222.so.1.4.4.44
4989 ├── build-pentium
4990 │   └── libft4222.so.1.4.4.44
4991 ├── build-x86_64
4992 │   └── libft4222.so.1.4.4.44
4993 ├── examples
4994 │   ├── get-version.c
4995 │   ├── i2cm.c
4996 │   ├── spim.c
4997 │   └── spis.c
4998 ├── ftd2xx.h
4999 ├── install4222.sh
5000 ├── libft4222.h
5001 ├── ReadMe.txt
5002 └── WinTypes.h
5003
5004To write a recipe to use such a library in your system:
5005
5006- The vendor will probably have a proprietary licence, so set
5007 :term:`LICENSE_FLAGS` in your recipe.
5008- The vendor provides a tarball containing libraries so set :term:`SRC_URI`
5009 appropriately.
5010- Set :term:`COMPATIBLE_HOST` so that the recipe cannot be used with an
5011 unsupported architecture. In the following example, we only support the 32
5012 and 64 bit variants of the ``x86`` architecture.
5013- As the vendor provides versioned libraries, we can use ``oe_soinstall``
5014 from :ref:`ref-classes-utils` to install the shared library and create
5015 symbolic links. If the vendor does not do this, we need to follow the
5016 non-versioned library guidelines in the next section.
5017- As the vendor likely used :term:`LDFLAGS` different from those in your Yocto
5018 Project build, disable the corresponding checks by adding ``ldflags``
5019 to :term:`INSANE_SKIP`.
5020- The vendor will typically ship release builds without debugging symbols.
5021 Avoid errors by preventing the packaging task from stripping out the symbols
5022 and adding them to a separate debug package. This is done by setting the
5023 ``INHIBIT_`` flags shown below.
5024
5025The complete recipe would look like this::
5026
5027 SUMMARY = "FTDI FT4222H Library"
5028 SECTION = "libs"
5029 LICENSE_FLAGS = "ftdi"
5030 LICENSE = "CLOSED"
5031
5032 COMPATIBLE_HOST = "(i.86|x86_64).*-linux"
5033
5034 # Sources available in a .tgz file in .zip archive
5035 # at https://ftdichip.com/wp-content/uploads/2021/01/libft4222-linux-1.4.4.44.zip
5036 # Found on https://ftdichip.com/software-examples/ft4222h-software-examples/
5037 # Since dealing with this particular type of archive is out of topic here,
5038 # we use a local link.
5039 SRC_URI = "file://libft4222-linux-${PV}.tgz"
5040
5041 S = "${WORKDIR}"
5042
5043 ARCH_DIR:x86-64 = "build-x86_64"
5044 ARCH_DIR:i586 = "build-i386"
5045 ARCH_DIR:i686 = "build-i386"
5046
5047 INSANE_SKIP:${PN} = "ldflags"
5048 INHIBIT_PACKAGE_STRIP = "1"
5049 INHIBIT_SYSROOT_STRIP = "1"
5050 INHIBIT_PACKAGE_DEBUG_SPLIT = "1"
5051
5052 do_install () {
5053 install -m 0755 -d ${D}${libdir}
5054 oe_soinstall ${S}/${ARCH_DIR}/libft4222.so.${PV} ${D}${libdir}
5055 install -d ${D}${includedir}
5056 install -m 0755 ${S}/*.h ${D}${includedir}
5057 }
5058
5059If the precompiled binaries are not statically linked and have dependencies on
5060other libraries, then by adding those libraries to :term:`DEPENDS`, the linking
5061can be examined and the appropriate :term:`RDEPENDS` automatically added.
5062
5063Non-Versioned Libraries
5064-----------------------
5065
5066Some Background
5067~~~~~~~~~~~~~~~
5068
5069Libraries in Linux systems are generally versioned so that it is possible
5070to have multiple versions of the same library installed, which eases upgrades
5071and support for older software. For example, suppose that in a versioned
5072library, an actual library is called ``libfoo.so.1.2``, a symbolic link named
5073``libfoo.so.1`` points to ``libfoo.so.1.2``, and a symbolic link named
5074``libfoo.so`` points to ``libfoo.so.1.2``. Given these conditions, when you
5075link a binary against a library, you typically provide the unversioned file
5076name (i.e. ``-lfoo`` to the linker). However, the linker follows the symbolic
5077link and actually links against the versioned filename. The unversioned symbolic
5078link is only used at development time. Consequently, the library is packaged
5079along with the headers in the development package ``${PN}-dev`` along with the
5080actual library and versioned symbolic links in ``${PN}``. Because versioned
5081libraries are far more common than unversioned libraries, the default packaging
5082rules assume versioned libraries.
5083
5084Yocto Library Packaging Overview
5085~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5086
5087It follows that packaging an unversioned library requires a bit of work in the
5088recipe. By default, ``libfoo.so`` gets packaged into ``${PN}-dev``, which
5089triggers a QA warning that a non-symlink library is in a ``-dev`` package,
5090and binaries in the same recipe link to the library in ``${PN}-dev``,
5091which triggers more QA warnings. To solve this problem, you need to package the
5092unversioned library into ``${PN}`` where it belongs. The following are the abridged
5093default :term:`FILES` variables in ``bitbake.conf``::
5094
5095 SOLIBS = ".so.*"
5096 SOLIBSDEV = ".so"
5097 FILES_${PN} = "... ${libdir}/lib*${SOLIBS} ..."
5098 FILES_SOLIBSDEV ?= "... ${libdir}/lib*${SOLIBSDEV} ..."
5099 FILES_${PN}-dev = "... ${FILES_SOLIBSDEV} ..."
5100
5101:term:`SOLIBS` defines a pattern that matches real shared object libraries.
5102:term:`SOLIBSDEV` matches the development form (unversioned symlink). These two
5103variables are then used in ``FILES:${PN}`` and ``FILES:${PN}-dev``, which puts
5104the real libraries into ``${PN}`` and the unversioned symbolic link into ``${PN}-dev``.
5105To package unversioned libraries, you need to modify the variables in the recipe
5106as follows::
5107
5108 SOLIBS = ".so"
5109 FILES_SOLIBSDEV = ""
5110
5111The modifications cause the ``.so`` file to be the real library
5112and unset :term:`FILES_SOLIBSDEV` so that no libraries get packaged into
5113``${PN}-dev``. The changes are required because unless :term:`PACKAGES` is changed,
5114``${PN}-dev`` collects files before `${PN}`. ``${PN}-dev`` must not collect any of
5115the files you want in ``${PN}``.
5116
5117Finally, loadable modules, essentially unversioned libraries that are linked
5118at runtime using ``dlopen()`` instead of at build time, should generally be
5119installed in a private directory. However, if they are installed in ``${libdir}``,
5120then the modules can be treated as unversioned libraries.
5121
5122Example
5123~~~~~~~
5124
5125The example below installs an unversioned x86-64 pre-built library named
5126``libfoo.so``. The :term:`COMPATIBLE_HOST` variable limits recipes to the
5127x86-64 architecture while the :term:`INSANE_SKIP`, :term:`INHIBIT_PACKAGE_STRIP`
5128and :term:`INHIBIT_SYSROOT_STRIP` variables are all set as in the above
5129versioned library example. The "magic" is setting the :term:`SOLIBS` and
5130:term:`FILES_SOLIBSDEV` variables as explained above::
5131
5132 SUMMARY = "libfoo sample recipe"
5133 SECTION = "libs"
5134 LICENSE = "CLOSED"
5135
5136 SRC_URI = "file://libfoo.so"
5137
5138 COMPATIBLE_HOST = "x86_64.*-linux"
5139
5140 INSANE_SKIP:${PN} = "ldflags"
5141 INHIBIT_PACKAGE_STRIP = "1"
5142 INHIBIT_SYSROOT_STRIP = "1"
5143 SOLIBS = ".so"
5144 FILES_SOLIBSDEV = ""
5145
5146 do_install () {
5147 install -d ${D}${libdir}
5148 install -m 0755 ${WORKDIR}/libfoo.so ${D}${libdir}
5149 }
5150
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005151Using x32 psABI
5152===============
5153
5154x32 processor-specific Application Binary Interface (`x32
5155psABI <https://software.intel.com/en-us/node/628948>`__) is a native
515632-bit processor-specific ABI for Intel 64 (x86-64) architectures. An
5157ABI defines the calling conventions between functions in a processing
5158environment. The interface determines what registers are used and what
5159the sizes are for various C data types.
5160
5161Some processing environments prefer using 32-bit applications even when
5162running on Intel 64-bit platforms. Consider the i386 psABI, which is a
5163very old 32-bit ABI for Intel 64-bit platforms. The i386 psABI does not
5164provide efficient use and access of the Intel 64-bit processor
5165resources, leaving the system underutilized. Now consider the x86_64
5166psABI. This ABI is newer and uses 64-bits for data sizes and program
5167pointers. The extra bits increase the footprint size of the programs,
5168libraries, and also increases the memory and file system size
5169requirements. Executing under the x32 psABI enables user programs to
5170utilize CPU and system resources more efficiently while keeping the
5171memory footprint of the applications low. Extra bits are used for
5172registers but not for addressing mechanisms.
5173
5174The Yocto Project supports the final specifications of x32 psABI as
5175follows:
5176
5177- You can create packages and images in x32 psABI format on x86_64
5178 architecture targets.
5179
5180- You can successfully build recipes with the x32 toolchain.
5181
5182- You can create and boot ``core-image-minimal`` and
5183 ``core-image-sato`` images.
5184
William A. Kennington IIIac69b482021-06-02 12:28:27 -07005185- There is RPM Package Manager (RPM) support for x32 binaries.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005186
William A. Kennington IIIac69b482021-06-02 12:28:27 -07005187- There is support for large images.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005188
5189To use the x32 psABI, you need to edit your ``conf/local.conf``
Andrew Geisslerc926e172021-05-07 16:11:35 -05005190configuration file as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005191
5192 MACHINE = "qemux86-64"
5193 DEFAULTTUNE = "x86-64-x32"
Patrick Williams0ca19cc2021-08-16 14:03:13 -05005194 baselib = "${@d.getVar('BASE_LIB:tune-' + (d.getVar('DEFAULTTUNE') \
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005195 or 'INVALID')) or 'lib'}"
5196
5197Once you have set
5198up your configuration file, use BitBake to build an image that supports
Andrew Geisslerc926e172021-05-07 16:11:35 -05005199the x32 psABI. Here is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005200
5201 $ bitbake core-image-sato
5202
5203Enabling GObject Introspection Support
5204======================================
5205
Andrew Geissler595f6302022-01-24 19:11:47 +00005206`GObject introspection <https://gi.readthedocs.io/en/latest/>`__
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005207is the standard mechanism for accessing GObject-based software from
5208runtime environments. GObject is a feature of the GLib library that
5209provides an object framework for the GNOME desktop and related software.
5210GObject Introspection adds information to GObject that allows objects
5211created within it to be represented across different programming
5212languages. If you want to construct GStreamer pipelines using Python, or
5213control UPnP infrastructure using Javascript and GUPnP, GObject
5214introspection is the only way to do it.
5215
5216This section describes the Yocto Project support for generating and
5217packaging GObject introspection data. GObject introspection data is a
Andrew Geissler595f6302022-01-24 19:11:47 +00005218description of the API provided by libraries built on top of the GLib
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005219framework, and, in particular, that framework's GObject mechanism.
5220GObject Introspection Repository (GIR) files go to ``-dev`` packages,
5221``typelib`` files go to main packages as they are packaged together with
5222libraries that are introspected.
5223
5224The data is generated when building such a library, by linking the
5225library with a small executable binary that asks the library to describe
5226itself, and then executing the binary and processing its output.
5227
5228Generating this data in a cross-compilation environment is difficult
5229because the library is produced for the target architecture, but its
5230code needs to be executed on the build host. This problem is solved with
5231the OpenEmbedded build system by running the code through QEMU, which
5232allows precisely that. Unfortunately, QEMU does not always work
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05005233perfectly as mentioned in the ":ref:`dev-manual/common-tasks:known issues`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005234section.
5235
5236Enabling the Generation of Introspection Data
5237---------------------------------------------
5238
5239Enabling the generation of introspection data (GIR files) in your
5240library package involves the following:
5241
52421. Inherit the
5243 :ref:`gobject-introspection <ref-classes-gobject-introspection>`
5244 class.
5245
52462. Make sure introspection is not disabled anywhere in the recipe or
5247 from anything the recipe includes. Also, make sure that
5248 "gobject-introspection-data" is not in
5249 :term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`
5250 and that "qemu-usermode" is not in
5251 :term:`MACHINE_FEATURES_BACKFILL_CONSIDERED`.
William A. Kennington IIIac69b482021-06-02 12:28:27 -07005252 In either of these conditions, nothing will happen.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005253
52543. Try to build the recipe. If you encounter build errors that look like
5255 something is unable to find ``.so`` libraries, check where these
5256 libraries are located in the source tree and add the following to the
Andrew Geisslerc926e172021-05-07 16:11:35 -05005257 recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005258
5259 GIR_EXTRA_LIBS_PATH = "${B}/something/.libs"
5260
5261 .. note::
5262
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005263 See recipes in the ``oe-core`` repository that use that
Andrew Geissler595f6302022-01-24 19:11:47 +00005264 :term:`GIR_EXTRA_LIBS_PATH` variable as an example.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005265
52664. Look for any other errors, which probably mean that introspection
5267 support in a package is not entirely standard, and thus breaks down
5268 in a cross-compilation environment. For such cases, custom-made fixes
5269 are needed. A good place to ask and receive help in these cases is
5270 the :ref:`Yocto Project mailing
5271 lists <resources-mailinglist>`.
5272
5273.. note::
5274
5275 Using a library that no longer builds against the latest Yocto
5276 Project release and prints introspection related errors is a good
5277 candidate for the previous procedure.
5278
5279Disabling the Generation of Introspection Data
5280----------------------------------------------
5281
5282You might find that you do not want to generate introspection data. Or,
5283perhaps QEMU does not work on your build host and target architecture
5284combination. If so, you can use either of the following methods to
5285disable GIR file generations:
5286
Andrew Geisslerc926e172021-05-07 16:11:35 -05005287- Add the following to your distro configuration::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005288
5289 DISTRO_FEATURES_BACKFILL_CONSIDERED = "gobject-introspection-data"
5290
5291 Adding this statement disables generating introspection data using
5292 QEMU but will still enable building introspection tools and libraries
5293 (i.e. building them does not require the use of QEMU).
5294
Andrew Geisslerc926e172021-05-07 16:11:35 -05005295- Add the following to your machine configuration::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005296
5297 MACHINE_FEATURES_BACKFILL_CONSIDERED = "qemu-usermode"
5298
5299 Adding this statement disables the use of QEMU when building packages for your
5300 machine. Currently, this feature is used only by introspection
5301 recipes and has the same effect as the previously described option.
5302
5303 .. note::
5304
5305 Future releases of the Yocto Project might have other features
5306 affected by this option.
5307
5308If you disable introspection data, you can still obtain it through other
5309means such as copying the data from a suitable sysroot, or by generating
5310it on the target hardware. The OpenEmbedded build system does not
5311currently provide specific support for these techniques.
5312
5313Testing that Introspection Works in an Image
5314--------------------------------------------
5315
5316Use the following procedure to test if generating introspection data is
5317working in an image:
5318
53191. Make sure that "gobject-introspection-data" is not in
5320 :term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`
5321 and that "qemu-usermode" is not in
5322 :term:`MACHINE_FEATURES_BACKFILL_CONSIDERED`.
5323
53242. Build ``core-image-sato``.
5325
53263. Launch a Terminal and then start Python in the terminal.
5327
Andrew Geisslerc926e172021-05-07 16:11:35 -050053284. Enter the following in the terminal::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005329
5330 >>> from gi.repository import GLib
5331 >>> GLib.get_host_name()
5332
53335. For something a little more advanced, enter the following see:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005334 https://python-gtk-3-tutorial.readthedocs.io/en/latest/introduction.html
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005335
5336Known Issues
5337------------
5338
William A. Kennington IIIac69b482021-06-02 12:28:27 -07005339Here are know issues in GObject Introspection Support:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005340
5341- ``qemu-ppc64`` immediately crashes. Consequently, you cannot build
5342 introspection data on that architecture.
5343
5344- x32 is not supported by QEMU. Consequently, introspection data is
5345 disabled.
5346
5347- musl causes transient GLib binaries to crash on assertion failures.
5348 Consequently, generating introspection data is disabled.
5349
5350- Because QEMU is not able to run the binaries correctly, introspection
5351 is disabled for some specific packages under specific architectures
5352 (e.g. ``gcr``, ``libsecret``, and ``webkit``).
5353
5354- QEMU usermode might not work properly when running 64-bit binaries
5355 under 32-bit host machines. In particular, "qemumips64" is known to
5356 not work under i686.
5357
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005358Optionally Using an External Toolchain
5359======================================
5360
5361You might want to use an external toolchain as part of your development.
5362If this is the case, the fundamental steps you need to accomplish are as
5363follows:
5364
5365- Understand where the installed toolchain resides. For cases where you
5366 need to build the external toolchain, you would need to take separate
5367 steps to build and install the toolchain.
5368
5369- Make sure you add the layer that contains the toolchain to your
5370 ``bblayers.conf`` file through the
5371 :term:`BBLAYERS` variable.
5372
5373- Set the ``EXTERNAL_TOOLCHAIN`` variable in your ``local.conf`` file
5374 to the location in which you installed the toolchain.
5375
5376A good example of an external toolchain used with the Yocto Project is
5377Mentor Graphics Sourcery G++ Toolchain. You can see information on how
5378to use that particular layer in the ``README`` file at
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005379https://github.com/MentorEmbedded/meta-sourcery/. You can find
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005380further information by reading about the
5381:term:`TCMODE` variable in the Yocto
5382Project Reference Manual's variable glossary.
5383
5384Creating Partitioned Images Using Wic
5385=====================================
5386
5387Creating an image for a particular hardware target using the
5388OpenEmbedded build system does not necessarily mean you can boot that
5389image as is on your device. Physical devices accept and boot images in
5390various ways depending on the specifics of the device. Usually,
5391information about the hardware can tell you what image format the device
5392requires. Should your device require multiple partitions on an SD card,
5393flash, or an HDD, you can use the OpenEmbedded Image Creator, Wic, to
5394create the properly partitioned image.
5395
5396The ``wic`` command generates partitioned images from existing
5397OpenEmbedded build artifacts. Image generation is driven by partitioning
Andrew Geisslerd5838332022-05-27 11:33:10 -05005398commands contained in an OpenEmbedded kickstart file (``.wks``)
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005399specified either directly on the command line or as one of a selection
5400of canned kickstart files as shown with the ``wic list images`` command
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05005401in the
5402":ref:`dev-manual/common-tasks:generate an image using an existing kickstart file`"
5403section. When you apply the command to a given set of build artifacts, the
5404result is an image or set of images that can be directly written onto media and
5405used on a particular system.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005406
5407.. note::
5408
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005409 For a kickstart file reference, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06005410 ":ref:`ref-manual/kickstart:openembedded kickstart (\`\`.wks\`\`) reference`"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005411 Chapter in the Yocto Project Reference Manual.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005412
5413The ``wic`` command and the infrastructure it is based on is by
5414definition incomplete. The purpose of the command is to allow the
5415generation of customized images, and as such, was designed to be
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05005416completely extensible through a plugin interface. See the
5417":ref:`dev-manual/common-tasks:using the wic plugin interface`" section
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005418for information on these plugins.
5419
5420This section provides some background information on Wic, describes what
5421you need to have in place to run the tool, provides instruction on how
5422to use the Wic utility, provides information on using the Wic plugins
5423interface, and provides several examples that show how to use Wic.
5424
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005425Background
5426----------
5427
5428This section provides some background on the Wic utility. While none of
5429this information is required to use Wic, you might find it interesting.
5430
5431- The name "Wic" is derived from OpenEmbedded Image Creator (oeic). The
5432 "oe" diphthong in "oeic" was promoted to the letter "w", because
5433 "oeic" is both difficult to remember and to pronounce.
5434
5435- Wic is loosely based on the Meego Image Creator (``mic``) framework.
5436 The Wic implementation has been heavily modified to make direct use
5437 of OpenEmbedded build artifacts instead of package installation and
5438 configuration, which are already incorporated within the OpenEmbedded
5439 artifacts.
5440
5441- Wic is a completely independent standalone utility that initially
5442 provides easier-to-use and more flexible replacements for an existing
5443 functionality in OE-Core's
5444 :ref:`image-live <ref-classes-image-live>`
5445 class. The difference between Wic and those examples is that with Wic
5446 the functionality of those scripts is implemented by a
5447 general-purpose partitioning language, which is based on Redhat
5448 kickstart syntax.
5449
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005450Requirements
5451------------
5452
5453In order to use the Wic utility with the OpenEmbedded Build system, your
5454system needs to meet the following requirements:
5455
5456- The Linux distribution on your development host must support the
5457 Yocto Project. See the ":ref:`detailed-supported-distros`"
5458 section in the Yocto Project Reference Manual for the list of
5459 distributions that support the Yocto Project.
5460
5461- The standard system utilities, such as ``cp``, must be installed on
5462 your development host system.
5463
5464- You must have sourced the build environment setup script (i.e.
5465 :ref:`structure-core-script`) found in the
5466 :term:`Build Directory`.
5467
5468- You need to have the build artifacts already available, which
5469 typically means that you must have already created an image using the
Andrew Geisslerd5838332022-05-27 11:33:10 -05005470 OpenEmbedded build system (e.g. ``core-image-minimal``). While it
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005471 might seem redundant to generate an image in order to create an image
5472 using Wic, the current version of Wic requires the artifacts in the
5473 form generated by the OpenEmbedded build system.
5474
5475- You must build several native tools, which are built to run on the
Andrew Geisslerc926e172021-05-07 16:11:35 -05005476 build system::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005477
5478 $ bitbake parted-native dosfstools-native mtools-native
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005479
5480- Include "wic" as part of the
5481 :term:`IMAGE_FSTYPES`
5482 variable.
5483
5484- Include the name of the :ref:`wic kickstart file <openembedded-kickstart-wks-reference>`
Andrew Geisslerd5838332022-05-27 11:33:10 -05005485 as part of the :term:`WKS_FILE` variable. If multiple candidate files can
5486 be provided by different layers, specify all the possible names through the
5487 :term:`WKS_FILES` variable instead.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005488
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005489Getting Help
5490------------
5491
5492You can get general help for the ``wic`` command by entering the ``wic``
5493command by itself or by entering the command with a help argument as
Andrew Geisslerc926e172021-05-07 16:11:35 -05005494follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005495
5496 $ wic -h
5497 $ wic --help
5498 $ wic help
5499
5500Currently, Wic supports seven commands: ``cp``, ``create``, ``help``,
5501``list``, ``ls``, ``rm``, and ``write``. You can get help for all these
Andrew Geisslerc926e172021-05-07 16:11:35 -05005502commands except "help" by using the following form::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005503
5504 $ wic help command
5505
5506For example, the following command returns help for the ``write``
Andrew Geisslerc926e172021-05-07 16:11:35 -05005507command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005508
5509 $ wic help write
5510
5511Wic supports help for three topics: ``overview``, ``plugins``, and
Andrew Geisslerc926e172021-05-07 16:11:35 -05005512``kickstart``. You can get help for any topic using the following form::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005513
5514 $ wic help topic
5515
Andrew Geisslerc926e172021-05-07 16:11:35 -05005516For example, the following returns overview help for Wic::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005517
5518 $ wic help overview
5519
William A. Kennington IIIac69b482021-06-02 12:28:27 -07005520There is one additional level of help for Wic. You can get help on
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005521individual images through the ``list`` command. You can use the ``list``
Andrew Geisslerc926e172021-05-07 16:11:35 -05005522command to return the available Wic images as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005523
5524 $ wic list images
5525 genericx86 Create an EFI disk image for genericx86*
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005526 edgerouter Create SD card image for Edgerouter
Andrew Geissler5199d832021-09-24 16:47:35 -05005527 beaglebone-yocto Create SD card image for Beaglebone
5528 qemux86-directdisk Create a qemu machine 'pcbios' direct disk image
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005529 systemd-bootdisk Create an EFI disk image with systemd-boot
5530 mkhybridiso Create a hybrid ISO image
Andrew Geissler5199d832021-09-24 16:47:35 -05005531 mkefidisk Create an EFI disk image
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005532 sdimage-bootpart Create SD card image with a boot partition
5533 directdisk-multi-rootfs Create multi rootfs image using rootfs plugin
Andrew Geissler5199d832021-09-24 16:47:35 -05005534 directdisk Create a 'pcbios' direct disk image
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005535 directdisk-bootloader-config Create a 'pcbios' direct disk image with custom bootloader config
Andrew Geissler5199d832021-09-24 16:47:35 -05005536 qemuriscv Create qcow2 image for RISC-V QEMU machines
5537 directdisk-gpt Create a 'pcbios' direct disk image
5538 efi-bootdisk
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005539
5540Once you know the list of available
5541Wic images, you can use ``help`` with the command to get help on a
5542particular image. For example, the following command returns help on the
Andrew Geisslerc926e172021-05-07 16:11:35 -05005543"beaglebone-yocto" image::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005544
5545 $ wic list beaglebone-yocto help
5546
5547 Creates a partitioned SD card image for Beaglebone.
5548 Boot files are located in the first vfat partition.
5549
5550Operational Modes
5551-----------------
5552
5553You can use Wic in two different modes, depending on how much control
Andrew Geisslerd5838332022-05-27 11:33:10 -05005554you need for specifying the OpenEmbedded build artifacts that are used
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005555for creating the image: Raw and Cooked:
5556
5557- *Raw Mode:* You explicitly specify build artifacts through Wic
5558 command-line arguments.
5559
5560- *Cooked Mode:* The current
5561 :term:`MACHINE` setting and image
5562 name are used to automatically locate and provide the build
5563 artifacts. You just supply a kickstart file and the name of the image
5564 from which to use artifacts.
5565
5566Regardless of the mode you use, you need to have the build artifacts
5567ready and available.
5568
5569Raw Mode
5570~~~~~~~~
5571
5572Running Wic in raw mode allows you to specify all the partitions through
5573the ``wic`` command line. The primary use for raw mode is if you have
5574built your kernel outside of the Yocto Project
5575:term:`Build Directory`. In other words, you
5576can point to arbitrary kernel, root filesystem locations, and so forth.
5577Contrast this behavior with cooked mode where Wic looks in the Build
5578Directory (e.g. ``tmp/deploy/images/``\ machine).
5579
Andrew Geisslerc926e172021-05-07 16:11:35 -05005580The general form of the ``wic`` command in raw mode is::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005581
5582 $ wic create wks_file options ...
5583
5584 Where:
5585
5586 wks_file:
5587 An OpenEmbedded kickstart file. You can provide
5588 your own custom file or use a file from a set of
5589 existing files as described by further options.
5590
5591 optional arguments:
5592 -h, --help show this help message and exit
5593 -o OUTDIR, --outdir OUTDIR
5594 name of directory to create image in
5595 -e IMAGE_NAME, --image-name IMAGE_NAME
5596 name of the image to use the artifacts from e.g. core-
5597 image-sato
5598 -r ROOTFS_DIR, --rootfs-dir ROOTFS_DIR
5599 path to the /rootfs dir to use as the .wks rootfs
5600 source
5601 -b BOOTIMG_DIR, --bootimg-dir BOOTIMG_DIR
5602 path to the dir containing the boot artifacts (e.g.
5603 /EFI or /syslinux dirs) to use as the .wks bootimg
5604 source
5605 -k KERNEL_DIR, --kernel-dir KERNEL_DIR
5606 path to the dir containing the kernel to use in the
5607 .wks bootimg
5608 -n NATIVE_SYSROOT, --native-sysroot NATIVE_SYSROOT
5609 path to the native sysroot containing the tools to use
5610 to build the image
5611 -s, --skip-build-check
5612 skip the build check
5613 -f, --build-rootfs build rootfs
5614 -c {gzip,bzip2,xz}, --compress-with {gzip,bzip2,xz}
5615 compress image with specified compressor
5616 -m, --bmap generate .bmap
5617 --no-fstab-update Do not change fstab file.
5618 -v VARS_DIR, --vars VARS_DIR
5619 directory with <image>.env files that store bitbake
5620 variables
5621 -D, --debug output debug information
5622
5623.. note::
5624
5625 You do not need root privileges to run Wic. In fact, you should not
5626 run as root when using the utility.
5627
5628Cooked Mode
5629~~~~~~~~~~~
5630
5631Running Wic in cooked mode leverages off artifacts in the Build
5632Directory. In other words, you do not have to specify kernel or root
5633filesystem locations as part of the command. All you need to provide is
5634a kickstart file and the name of the image from which to use artifacts
5635by using the "-e" option. Wic looks in the Build Directory (e.g.
5636``tmp/deploy/images/``\ machine) for artifacts.
5637
Andrew Geisslerc926e172021-05-07 16:11:35 -05005638The general form of the ``wic`` command using Cooked Mode is as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005639
5640 $ wic create wks_file -e IMAGE_NAME
5641
5642 Where:
5643
5644 wks_file:
5645 An OpenEmbedded kickstart file. You can provide
5646 your own custom file or use a file from a set of
5647 existing files provided with the Yocto Project
5648 release.
5649
5650 required argument:
5651 -e IMAGE_NAME, --image-name IMAGE_NAME
5652 name of the image to use the artifacts from e.g. core-
5653 image-sato
5654
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005655Using an Existing Kickstart File
5656--------------------------------
5657
5658If you do not want to create your own kickstart file, you can use an
5659existing file provided by the Wic installation. As shipped, kickstart
Andrew Geissler09209ee2020-12-13 08:44:15 -06005660files can be found in the :ref:`overview-manual/development-environment:yocto project source repositories` in the
Andrew Geisslerc926e172021-05-07 16:11:35 -05005661following two locations::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005662
5663 poky/meta-yocto-bsp/wic
5664 poky/scripts/lib/wic/canned-wks
5665
Andrew Geisslerc926e172021-05-07 16:11:35 -05005666Use the following command to list the available kickstart files::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005667
5668 $ wic list images
5669 genericx86 Create an EFI disk image for genericx86*
5670 beaglebone-yocto Create SD card image for Beaglebone
5671 edgerouter Create SD card image for Edgerouter
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05005672 qemux86-directdisk Create a QEMU machine 'pcbios' direct disk image
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005673 directdisk-gpt Create a 'pcbios' direct disk image
5674 mkefidisk Create an EFI disk image
5675 directdisk Create a 'pcbios' direct disk image
5676 systemd-bootdisk Create an EFI disk image with systemd-boot
5677 mkhybridiso Create a hybrid ISO image
5678 sdimage-bootpart Create SD card image with a boot partition
5679 directdisk-multi-rootfs Create multi rootfs image using rootfs plugin
5680 directdisk-bootloader-config Create a 'pcbios' direct disk image with custom bootloader config
5681
5682When you use an existing file, you
5683do not have to use the ``.wks`` extension. Here is an example in Raw
Andrew Geisslerc926e172021-05-07 16:11:35 -05005684Mode that uses the ``directdisk`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005685
5686 $ wic create directdisk -r rootfs_dir -b bootimg_dir \
5687 -k kernel_dir -n native_sysroot
5688
5689Here are the actual partition language commands used in the
Andrew Geisslerc926e172021-05-07 16:11:35 -05005690``genericx86.wks`` file to generate an image::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005691
5692 # short-description: Create an EFI disk image for genericx86*
5693 # long-description: Creates a partitioned EFI disk image for genericx86* machines
5694 part /boot --source bootimg-efi --sourceparams="loader=grub-efi" --ondisk sda --label msdos --active --align 1024
5695 part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 1024 --use-uuid
5696 part swap --ondisk sda --size 44 --label swap1 --fstype=swap
5697
5698 bootloader --ptable gpt --timeout=5 --append="rootfstype=ext4 console=ttyS0,115200 console=tty0"
5699
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005700Using the Wic Plugin Interface
5701------------------------------
5702
5703You can extend and specialize Wic functionality by using Wic plugins.
5704This section explains the Wic plugin interface.
5705
5706.. note::
5707
5708 Wic plugins consist of "source" and "imager" plugins. Imager plugins
5709 are beyond the scope of this section.
5710
5711Source plugins provide a mechanism to customize partition content during
5712the Wic image generation process. You can use source plugins to map
5713values that you specify using ``--source`` commands in kickstart files
5714(i.e. ``*.wks``) to a plugin implementation used to populate a given
5715partition.
5716
5717.. note::
5718
5719 If you use plugins that have build-time dependencies (e.g. native
5720 tools, bootloaders, and so forth) when building a Wic image, you need
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005721 to specify those dependencies using the :term:`WKS_FILE_DEPENDS`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005722 variable.
5723
5724Source plugins are subclasses defined in plugin files. As shipped, the
5725Yocto Project provides several plugin files. You can see the source
5726plugin files that ship with the Yocto Project
Andrew Geissler09209ee2020-12-13 08:44:15 -06005727:yocto_git:`here </poky/tree/scripts/lib/wic/plugins/source>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005728Each of these plugin files contains source plugins that are designed to
5729populate a specific Wic image partition.
5730
5731Source plugins are subclasses of the ``SourcePlugin`` class, which is
5732defined in the ``poky/scripts/lib/wic/pluginbase.py`` file. For example,
5733the ``BootimgEFIPlugin`` source plugin found in the ``bootimg-efi.py``
5734file is a subclass of the ``SourcePlugin`` class, which is found in the
5735``pluginbase.py`` file.
5736
5737You can also implement source plugins in a layer outside of the Source
5738Repositories (external layer). To do so, be sure that your plugin files
5739are located in a directory whose path is
5740``scripts/lib/wic/plugins/source/`` within your external layer. When the
5741plugin files are located there, the source plugins they contain are made
5742available to Wic.
5743
5744When the Wic implementation needs to invoke a partition-specific
5745implementation, it looks for the plugin with the same name as the
5746``--source`` parameter used in the kickstart file given to that
5747partition. For example, if the partition is set up using the following
Andrew Geisslerc926e172021-05-07 16:11:35 -05005748command in a kickstart file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005749
5750 part /boot --source bootimg-pcbios --ondisk sda --label boot --active --align 1024
5751
5752The methods defined as class
5753members of the matching source plugin (i.e. ``bootimg-pcbios``) in the
5754``bootimg-pcbios.py`` plugin file are used.
5755
5756To be more concrete, here is the corresponding plugin definition from
5757the ``bootimg-pcbios.py`` file for the previous command along with an
5758example method called by the Wic implementation when it needs to prepare
Andrew Geisslerc926e172021-05-07 16:11:35 -05005759a partition using an implementation-specific function::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005760
5761 .
5762 .
5763 .
5764 class BootimgPcbiosPlugin(SourcePlugin):
5765 """
5766 Create MBR boot partition and install syslinux on it.
5767 """
5768
5769 name = 'bootimg-pcbios'
5770 .
5771 .
5772 .
5773 @classmethod
5774 def do_prepare_partition(cls, part, source_params, creator, cr_workdir,
5775 oe_builddir, bootimg_dir, kernel_dir,
5776 rootfs_dir, native_sysroot):
5777 """
5778 Called to do the actual content population for a partition i.e. it
5779 'prepares' the partition to be incorporated into the image.
5780 In this case, prepare content for legacy bios boot partition.
5781 """
5782 .
5783 .
5784 .
5785
5786If a
5787subclass (plugin) itself does not implement a particular function, Wic
5788locates and uses the default version in the superclass. It is for this
5789reason that all source plugins are derived from the ``SourcePlugin``
5790class.
5791
5792The ``SourcePlugin`` class defined in the ``pluginbase.py`` file defines
5793a set of methods that source plugins can implement or override. Any
5794plugins (subclass of ``SourcePlugin``) that do not implement a
5795particular method inherit the implementation of the method from the
5796``SourcePlugin`` class. For more information, see the ``SourcePlugin``
5797class in the ``pluginbase.py`` file for details:
5798
5799The following list describes the methods implemented in the
5800``SourcePlugin`` class:
5801
5802- ``do_prepare_partition()``: Called to populate a partition with
5803 actual content. In other words, the method prepares the final
5804 partition image that is incorporated into the disk image.
5805
5806- ``do_configure_partition()``: Called before
5807 ``do_prepare_partition()`` to create custom configuration files for a
5808 partition (e.g. syslinux or grub configuration files).
5809
5810- ``do_install_disk()``: Called after all partitions have been
5811 prepared and assembled into a disk image. This method provides a hook
5812 to allow finalization of a disk image (e.g. writing an MBR).
5813
5814- ``do_stage_partition()``: Special content-staging hook called
5815 before ``do_prepare_partition()``. This method is normally empty.
5816
5817 Typically, a partition just uses the passed-in parameters (e.g. the
5818 unmodified value of ``bootimg_dir``). However, in some cases, things
5819 might need to be more tailored. As an example, certain files might
5820 additionally need to be taken from ``bootimg_dir + /boot``. This hook
5821 allows those files to be staged in a customized fashion.
5822
5823 .. note::
5824
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005825 ``get_bitbake_var()`` allows you to access non-standard variables that
5826 you might want to use for this behavior.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005827
5828You can extend the source plugin mechanism. To add more hooks, create
5829more source plugin methods within ``SourcePlugin`` and the corresponding
5830derived subclasses. The code that calls the plugin methods uses the
5831``plugin.get_source_plugin_methods()`` function to find the method or
5832methods needed by the call. Retrieval of those methods is accomplished
5833by filling up a dict with keys that contain the method names of
5834interest. On success, these will be filled in with the actual methods.
5835See the Wic implementation for examples and details.
5836
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005837Wic Examples
5838------------
5839
5840This section provides several examples that show how to use the Wic
5841utility. All the examples assume the list of requirements in the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05005842":ref:`dev-manual/common-tasks:requirements`" section have been met. The
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005843examples assume the previously generated image is
5844``core-image-minimal``.
5845
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005846Generate an Image using an Existing Kickstart File
5847~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5848
5849This example runs in Cooked Mode and uses the ``mkefidisk`` kickstart
Andrew Geisslerc926e172021-05-07 16:11:35 -05005850file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005851
5852 $ wic create mkefidisk -e core-image-minimal
5853 INFO: Building wic-tools...
5854 .
5855 .
5856 .
5857 INFO: The new image(s) can be found here:
5858 ./mkefidisk-201804191017-sda.direct
5859
5860 The following build artifacts were used to create the image(s):
Andrew Geissler595f6302022-01-24 19:11:47 +00005861 ROOTFS_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
5862 BOOTIMG_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
5863 KERNEL_DIR: /home/stephano/yocto/build/tmp-glibc/deploy/images/qemux86
5864 NATIVE_SYSROOT: /home/stephano/yocto/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005865
5866 INFO: The image(s) were created using OE kickstart file:
Andrew Geissler595f6302022-01-24 19:11:47 +00005867 /home/stephano/yocto/openembedded-core/scripts/lib/wic/canned-wks/mkefidisk.wks
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005868
5869The previous example shows the easiest way to create an image by running
5870in cooked mode and supplying a kickstart file and the "-e" option to
5871point to the existing build artifacts. Your ``local.conf`` file needs to
5872have the :term:`MACHINE` variable set
5873to the machine you are using, which is "qemux86" in this example.
5874
5875Once the image builds, the output provides image location, artifact use,
5876and kickstart file information.
5877
5878.. note::
5879
5880 You should always verify the details provided in the output to make
5881 sure that the image was indeed created exactly as expected.
5882
5883Continuing with the example, you can now write the image from the Build
5884Directory onto a USB stick, or whatever media for which you built your
5885image, and boot from the media. You can write the image by using
Andrew Geisslerc926e172021-05-07 16:11:35 -05005886``bmaptool`` or ``dd``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005887
Andrew Geisslerd5838332022-05-27 11:33:10 -05005888 $ oe-run-native bmap-tools-native bmaptool copy mkefidisk-201804191017-sda.direct /dev/sdX
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005889
5890or ::
5891
5892 $ sudo dd if=mkefidisk-201804191017-sda.direct of=/dev/sdX
5893
5894.. note::
5895
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005896 For more information on how to use the ``bmaptool``
5897 to flash a device with an image, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06005898 ":ref:`dev-manual/common-tasks:flashing images using \`\`bmaptool\`\``"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005899 section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005900
5901Using a Modified Kickstart File
5902~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5903
5904Because partitioned image creation is driven by the kickstart file, it
5905is easy to affect image creation by changing the parameters in the file.
5906This next example demonstrates that through modification of the
5907``directdisk-gpt`` kickstart file.
5908
5909As mentioned earlier, you can use the command ``wic list images`` to
5910show the list of existing kickstart files. The directory in which the
5911``directdisk-gpt.wks`` file resides is
5912``scripts/lib/image/canned-wks/``, which is located in the
5913:term:`Source Directory` (e.g. ``poky``).
5914Because available files reside in this directory, you can create and add
5915your own custom files to the directory. Subsequent use of the
5916``wic list images`` command would then include your kickstart files.
5917
5918In this example, the existing ``directdisk-gpt`` file already does most
5919of what is needed. However, for the hardware in this example, the image
5920will need to boot from ``sdb`` instead of ``sda``, which is what the
5921``directdisk-gpt`` kickstart file uses.
5922
5923The example begins by making a copy of the ``directdisk-gpt.wks`` file
5924in the ``scripts/lib/image/canned-wks`` directory and then by changing
5925the lines that specify the target disk from which to boot.
5926::
5927
Andrew Geissler595f6302022-01-24 19:11:47 +00005928 $ cp /home/stephano/yocto/poky/scripts/lib/wic/canned-wks/directdisk-gpt.wks \
5929 /home/stephano/yocto/poky/scripts/lib/wic/canned-wks/directdisksdb-gpt.wks
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005930
5931Next, the example modifies the ``directdisksdb-gpt.wks`` file and
5932changes all instances of "``--ondisk sda``" to "``--ondisk sdb``". The
5933example changes the following two lines and leaves the remaining lines
Andrew Geisslerc926e172021-05-07 16:11:35 -05005934untouched::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005935
5936 part /boot --source bootimg-pcbios --ondisk sdb --label boot --active --align 1024
5937 part / --source rootfs --ondisk sdb --fstype=ext4 --label platform --align 1024 --use-uuid
5938
5939Once the lines are changed, the
5940example generates the ``directdisksdb-gpt`` image. The command points
5941the process at the ``core-image-minimal`` artifacts for the Next Unit of
5942Computing (nuc) :term:`MACHINE` the
5943``local.conf``.
5944::
5945
5946 $ wic create directdisksdb-gpt -e core-image-minimal
5947 INFO: Building wic-tools...
5948 .
5949 .
5950 .
5951 Initialising tasks: 100% |#######################################| Time: 0:00:01
5952 NOTE: Executing SetScene Tasks
5953 NOTE: Executing RunQueue Tasks
5954 NOTE: Tasks Summary: Attempted 1161 tasks of which 1157 didn't need to be rerun and all succeeded.
5955 INFO: Creating image(s)...
5956
5957 INFO: The new image(s) can be found here:
5958 ./directdisksdb-gpt-201710090938-sdb.direct
5959
5960 The following build artifacts were used to create the image(s):
Andrew Geissler595f6302022-01-24 19:11:47 +00005961 ROOTFS_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
5962 BOOTIMG_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
5963 KERNEL_DIR: /home/stephano/yocto/build/tmp-glibc/deploy/images/qemux86
5964 NATIVE_SYSROOT: /home/stephano/yocto/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005965
5966 INFO: The image(s) were created using OE kickstart file:
Andrew Geissler595f6302022-01-24 19:11:47 +00005967 /home/stephano/yocto/poky/scripts/lib/wic/canned-wks/directdisksdb-gpt.wks
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005968
5969Continuing with the example, you can now directly ``dd`` the image to a
5970USB stick, or whatever media for which you built your image, and boot
Andrew Geisslerc926e172021-05-07 16:11:35 -05005971the resulting media::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005972
5973 $ sudo dd if=directdisksdb-gpt-201710090938-sdb.direct of=/dev/sdb
5974 140966+0 records in
5975 140966+0 records out
5976 72174592 bytes (72 MB, 69 MiB) copied, 78.0282 s, 925 kB/s
5977 $ sudo eject /dev/sdb
5978
5979Using a Modified Kickstart File and Running in Raw Mode
5980~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5981
5982This next example manually specifies each build artifact (runs in Raw
5983Mode) and uses a modified kickstart file. The example also uses the
5984``-o`` option to cause Wic to create the output somewhere other than the
Andrew Geisslerc926e172021-05-07 16:11:35 -05005985default output directory, which is the current directory::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005986
Andrew Geissler595f6302022-01-24 19:11:47 +00005987 $ wic create test.wks -o /home/stephano/testwic \
5988 --rootfs-dir /home/stephano/yocto/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/rootfs \
5989 --bootimg-dir /home/stephano/yocto/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share \
5990 --kernel-dir /home/stephano/yocto/build/tmp/deploy/images/qemux86 \
5991 --native-sysroot /home/stephano/yocto/build/tmp/work/i586-poky-linux/wic-tools/1.0-r0/recipe-sysroot-native
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005992
5993 INFO: Creating image(s)...
5994
5995 INFO: The new image(s) can be found here:
5996 /home/stephano/testwic/test-201710091445-sdb.direct
5997
5998 The following build artifacts were used to create the image(s):
Andrew Geissler595f6302022-01-24 19:11:47 +00005999 ROOTFS_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
6000 BOOTIMG_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
6001 KERNEL_DIR: /home/stephano/yocto/build/tmp-glibc/deploy/images/qemux86
6002 NATIVE_SYSROOT: /home/stephano/yocto/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006003
6004 INFO: The image(s) were created using OE kickstart file:
Andrew Geissler595f6302022-01-24 19:11:47 +00006005 test.wks
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006006
6007For this example,
6008:term:`MACHINE` did not have to be
6009specified in the ``local.conf`` file since the artifact is manually
6010specified.
6011
6012Using Wic to Manipulate an Image
6013~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
6014
6015Wic image manipulation allows you to shorten turnaround time during
6016image development. For example, you can use Wic to delete the kernel
6017partition of a Wic image and then insert a newly built kernel. This
6018saves you time from having to rebuild the entire image each time you
6019modify the kernel.
6020
6021.. note::
6022
6023 In order to use Wic to manipulate a Wic image as in this example,
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006024 your development machine must have the ``mtools`` package installed.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006025
6026The following example examines the contents of the Wic image, deletes
6027the existing kernel, and then inserts a new kernel:
6028
60291. *List the Partitions:* Use the ``wic ls`` command to list all the
Andrew Geisslerc926e172021-05-07 16:11:35 -05006030 partitions in the Wic image::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006031
6032 $ wic ls tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic
6033 Num Start End Size Fstype
6034 1 1048576 25041919 23993344 fat16
6035 2 25165824 72157183 46991360 ext4
6036
6037 The previous output shows two partitions in the
6038 ``core-image-minimal-qemux86.wic`` image.
6039
60402. *Examine a Particular Partition:* Use the ``wic ls`` command again
6041 but in a different form to examine a particular partition.
6042
6043 .. note::
6044
6045 You can get command usage on any Wic command using the following
Andrew Geisslerc926e172021-05-07 16:11:35 -05006046 form::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006047
6048 $ wic help command
6049
6050
6051 For example, the following command shows you the various ways to
6052 use the
6053 wic ls
Andrew Geisslerc926e172021-05-07 16:11:35 -05006054 command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006055
6056 $ wic help ls
6057
6058
Andrew Geisslerc926e172021-05-07 16:11:35 -05006059 The following command shows what is in partition one::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006060
6061 $ wic ls tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1
6062 Volume in drive : is boot
6063 Volume Serial Number is E894-1809
6064 Directory for ::/
6065
6066 libcom32 c32 186500 2017-10-09 16:06
6067 libutil c32 24148 2017-10-09 16:06
6068 syslinux cfg 220 2017-10-09 16:06
6069 vesamenu c32 27104 2017-10-09 16:06
6070 vmlinuz 6904608 2017-10-09 16:06
6071 5 files 7 142 580 bytes
6072 16 582 656 bytes free
6073
6074 The previous output shows five files, with the
6075 ``vmlinuz`` being the kernel.
6076
6077 .. note::
6078
6079 If you see the following error, you need to update or create a
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006080 ``~/.mtoolsrc`` file and be sure to have the line "mtools_skip_check=1"
Andrew Geisslerc926e172021-05-07 16:11:35 -05006081 in the file. Then, run the Wic command again::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006082
6083 ERROR: _exec_cmd: /usr/bin/mdir -i /tmp/wic-parttfokuwra ::/ returned '1' instead of 0
6084 output: Total number of sectors (47824) not a multiple of sectors per track (32)!
6085 Add mtools_skip_check=1 to your .mtoolsrc file to skip this test
6086
6087
60883. *Remove the Old Kernel:* Use the ``wic rm`` command to remove the
Andrew Geisslerc926e172021-05-07 16:11:35 -05006089 ``vmlinuz`` file (kernel)::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006090
6091 $ wic rm tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1/vmlinuz
6092
60934. *Add In the New Kernel:* Use the ``wic cp`` command to add the
6094 updated kernel to the Wic image. Depending on how you built your
6095 kernel, it could be in different places. If you used ``devtool`` and
6096 an SDK to build your kernel, it resides in the ``tmp/work`` directory
6097 of the extensible SDK. If you used ``make`` to build the kernel, the
6098 kernel will be in the ``workspace/sources`` area.
6099
6100 The following example assumes ``devtool`` was used to build the
Andrew Geisslerc926e172021-05-07 16:11:35 -05006101 kernel::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006102
Andrew Geissler95ac1b82021-03-31 14:34:31 -05006103 $ wic cp poky_sdk/tmp/work/qemux86-poky-linux/linux-yocto/4.12.12+git999-r0/linux-yocto-4.12.12+git999/arch/x86/boot/bzImage \
6104 poky/build/tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1/vmlinuz
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006105
6106 Once the new kernel is added back into the image, you can use the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006107 ``dd`` command or :ref:`bmaptool
Andrew Geissler09209ee2020-12-13 08:44:15 -06006108 <dev-manual/common-tasks:flashing images using \`\`bmaptool\`\`>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006109 to flash your wic image onto an SD card or USB stick and test your
6110 target.
6111
6112 .. note::
6113
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006114 Using ``bmaptool`` is generally 10 to 20 times faster than using ``dd``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006115
6116Flashing Images Using ``bmaptool``
6117==================================
6118
6119A fast and easy way to flash an image to a bootable device is to use
6120Bmaptool, which is integrated into the OpenEmbedded build system.
6121Bmaptool is a generic tool that creates a file's block map (bmap) and
6122then uses that map to copy the file. As compared to traditional tools
6123such as dd or cp, Bmaptool can copy (or flash) large files like raw
6124system image files much faster.
6125
6126.. note::
6127
6128 - If you are using Ubuntu or Debian distributions, you can install
6129 the ``bmap-tools`` package using the following command and then
6130 use the tool without specifying ``PATH`` even from the root
Andrew Geisslerc926e172021-05-07 16:11:35 -05006131 account::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006132
Andrew Geisslereff27472021-10-29 15:35:00 -05006133 $ sudo apt install bmap-tools
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006134
6135 - If you are unable to install the ``bmap-tools`` package, you will
Andrew Geisslerc926e172021-05-07 16:11:35 -05006136 need to build Bmaptool before using it. Use the following command::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006137
6138 $ bitbake bmap-tools-native
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006139
6140Following, is an example that shows how to flash a Wic image. Realize
6141that while this example uses a Wic image, you can use Bmaptool to flash
6142any type of image. Use these steps to flash an image using Bmaptool:
6143
61441. *Update your local.conf File:* You need to have the following set
Andrew Geisslerc926e172021-05-07 16:11:35 -05006145 in your ``local.conf`` file before building your image::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006146
6147 IMAGE_FSTYPES += "wic wic.bmap"
6148
61492. *Get Your Image:* Either have your image ready (pre-built with the
6150 :term:`IMAGE_FSTYPES`
Andrew Geisslerc926e172021-05-07 16:11:35 -05006151 setting previously mentioned) or take the step to build the image::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006152
6153 $ bitbake image
6154
61553. *Flash the Device:* Flash the device with the image by using Bmaptool
6156 depending on your particular setup. The following commands assume the
6157 image resides in the Build Directory's ``deploy/images/`` area:
6158
Andrew Geisslerc926e172021-05-07 16:11:35 -05006159 - If you have write access to the media, use this command form::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006160
6161 $ oe-run-native bmap-tools-native bmaptool copy build-directory/tmp/deploy/images/machine/image.wic /dev/sdX
6162
6163 - If you do not have write access to the media, set your permissions
Andrew Geisslerc926e172021-05-07 16:11:35 -05006164 first and then use the same command form::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006165
6166 $ sudo chmod 666 /dev/sdX
6167 $ oe-run-native bmap-tools-native bmaptool copy build-directory/tmp/deploy/images/machine/image.wic /dev/sdX
6168
Andrew Geisslerc926e172021-05-07 16:11:35 -05006169For help on the ``bmaptool`` command, use the following command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006170
6171 $ bmaptool --help
6172
6173Making Images More Secure
6174=========================
6175
6176Security is of increasing concern for embedded devices. Consider the
6177issues and problems discussed in just this sampling of work found across
6178the Internet:
6179
6180- *"*\ `Security Risks of Embedded
6181 Systems <https://www.schneier.com/blog/archives/2014/01/security_risks_9.html>`__\ *"*
6182 by Bruce Schneier
6183
6184- *"*\ `Internet Census
6185 2012 <http://census2012.sourceforge.net/paper.html>`__\ *"* by Carna
6186 Botnet
6187
6188- *"*\ `Security Issues for Embedded
Andrew Geisslerd1e89492021-02-12 15:35:20 -06006189 Devices <https://elinux.org/images/6/6f/Security-issues.pdf>`__\ *"*
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006190 by Jake Edge
6191
6192When securing your image is of concern, there are steps, tools, and
6193variables that you can consider to help you reach the security goals you
6194need for your particular device. Not all situations are identical when
6195it comes to making an image secure. Consequently, this section provides
6196some guidance and suggestions for consideration when you want to make
6197your image more secure.
6198
6199.. note::
6200
6201 Because the security requirements and risks are different for every
6202 type of device, this section cannot provide a complete reference on
6203 securing your custom OS. It is strongly recommended that you also
6204 consult other sources of information on embedded Linux system
6205 hardening and on security.
6206
6207General Considerations
6208----------------------
6209
William A. Kennington IIIac69b482021-06-02 12:28:27 -07006210There are general considerations that help you create more secure images.
6211You should consider the following suggestions to make your device
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006212more secure:
6213
6214- Scan additional code you are adding to the system (e.g. application
6215 code) by using static analysis tools. Look for buffer overflows and
6216 other potential security problems.
6217
6218- Pay particular attention to the security for any web-based
6219 administration interface.
6220
6221 Web interfaces typically need to perform administrative functions and
6222 tend to need to run with elevated privileges. Thus, the consequences
6223 resulting from the interface's security becoming compromised can be
6224 serious. Look for common web vulnerabilities such as
6225 cross-site-scripting (XSS), unvalidated inputs, and so forth.
6226
6227 As with system passwords, the default credentials for accessing a
6228 web-based interface should not be the same across all devices. This
6229 is particularly true if the interface is enabled by default as it can
6230 be assumed that many end-users will not change the credentials.
6231
6232- Ensure you can update the software on the device to mitigate
6233 vulnerabilities discovered in the future. This consideration
6234 especially applies when your device is network-enabled.
6235
6236- Ensure you remove or disable debugging functionality before producing
6237 the final image. For information on how to do this, see the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006238 ":ref:`dev-manual/common-tasks:considerations specific to the openembedded build system`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006239 section.
6240
6241- Ensure you have no network services listening that are not needed.
6242
6243- Remove any software from the image that is not needed.
6244
6245- Enable hardware support for secure boot functionality when your
6246 device supports this functionality.
6247
6248Security Flags
6249--------------
6250
6251The Yocto Project has security flags that you can enable that help make
6252your build output more secure. The security flags are in the
6253``meta/conf/distro/include/security_flags.inc`` file in your
6254:term:`Source Directory` (e.g. ``poky``).
6255
6256.. note::
6257
6258 Depending on the recipe, certain security flags are enabled and
6259 disabled by default.
6260
6261Use the following line in your ``local.conf`` file or in your custom
6262distribution configuration file to enable the security compiler and
Andrew Geisslerc926e172021-05-07 16:11:35 -05006263linker flags for your build::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006264
6265 require conf/distro/include/security_flags.inc
6266
6267Considerations Specific to the OpenEmbedded Build System
6268--------------------------------------------------------
6269
6270You can take some steps that are specific to the OpenEmbedded build
6271system to make your images more secure:
6272
6273- Ensure "debug-tweaks" is not one of your selected
6274 :term:`IMAGE_FEATURES`.
6275 When creating a new project, the default is to provide you with an
6276 initial ``local.conf`` file that enables this feature using the
6277 :term:`EXTRA_IMAGE_FEATURES`
Andrew Geisslerc926e172021-05-07 16:11:35 -05006278 variable with the line::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006279
6280 EXTRA_IMAGE_FEATURES = "debug-tweaks"
6281
6282 To disable that feature, simply comment out that line in your
Andrew Geissler09036742021-06-25 14:25:14 -05006283 ``local.conf`` file, or make sure :term:`IMAGE_FEATURES` does not contain
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006284 "debug-tweaks" before producing your final image. Among other things,
6285 leaving this in place sets the root password as blank, which makes
6286 logging in for debugging or inspection easy during development but
6287 also means anyone can easily log in during production.
6288
6289- It is possible to set a root password for the image and also to set
6290 passwords for any extra users you might add (e.g. administrative or
6291 service type users). When you set up passwords for multiple images or
6292 users, you should not duplicate passwords.
6293
6294 To set up passwords, use the
6295 :ref:`extrausers <ref-classes-extrausers>`
6296 class, which is the preferred method. For an example on how to set up
6297 both root and user passwords, see the
Andrew Geissler7e0e3c02022-02-25 20:34:39 +00006298 ":ref:`ref-classes-extrausers`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006299
6300 .. note::
6301
6302 When adding extra user accounts or setting a root password, be
6303 cautious about setting the same password on every device. If you
6304 do this, and the password you have set is exposed, then every
6305 device is now potentially compromised. If you need this access but
6306 want to ensure security, consider setting a different, random
6307 password for each device. Typically, you do this as a separate
6308 step after you deploy the image onto the device.
6309
6310- Consider enabling a Mandatory Access Control (MAC) framework such as
6311 SMACK or SELinux and tuning it appropriately for your device's usage.
6312 You can find more information in the
Andrew Geissler09209ee2020-12-13 08:44:15 -06006313 :yocto_git:`meta-selinux </meta-selinux/>` layer.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006314
6315Tools for Hardening Your Image
6316------------------------------
6317
6318The Yocto Project provides tools for making your image more secure. You
6319can find these tools in the ``meta-security`` layer of the
6320:yocto_git:`Yocto Project Source Repositories <>`.
6321
6322Creating Your Own Distribution
6323==============================
6324
6325When you build an image using the Yocto Project and do not alter any
6326distribution :term:`Metadata`, you are
6327creating a Poky distribution. If you wish to gain more control over
6328package alternative selections, compile-time options, and other
6329low-level configurations, you can create your own distribution.
6330
6331To create your own distribution, the basic steps consist of creating
6332your own distribution layer, creating your own distribution
6333configuration file, and then adding any needed code and Metadata to the
6334layer. The following steps provide some more detail:
6335
6336- *Create a layer for your new distro:* Create your distribution layer
6337 so that you can keep your Metadata and code for the distribution
6338 separate. It is strongly recommended that you create and use your own
6339 layer for configuration and code. Using your own layer as compared to
6340 just placing configurations in a ``local.conf`` configuration file
6341 makes it easier to reproduce the same build configuration when using
6342 multiple build machines. See the
Andrew Geissler09209ee2020-12-13 08:44:15 -06006343 ":ref:`dev-manual/common-tasks:creating a general layer using the \`\`bitbake-layers\`\` script`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006344 section for information on how to quickly set up a layer.
6345
6346- *Create the distribution configuration file:* The distribution
6347 configuration file needs to be created in the ``conf/distro``
6348 directory of your layer. You need to name it using your distribution
6349 name (e.g. ``mydistro.conf``).
6350
6351 .. note::
6352
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006353 The :term:`DISTRO` variable in your ``local.conf`` file determines the
6354 name of your distribution.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006355
6356 You can split out parts of your configuration file into include files
6357 and then "require" them from within your distribution configuration
6358 file. Be sure to place the include files in the
6359 ``conf/distro/include`` directory of your layer. A common example
6360 usage of include files would be to separate out the selection of
6361 desired version and revisions for individual recipes.
6362
6363 Your configuration file needs to set the following required
6364 variables:
6365
6366 - :term:`DISTRO_NAME`
6367
6368 - :term:`DISTRO_VERSION`
6369
6370 These following variables are optional and you typically set them
6371 from the distribution configuration file:
6372
6373 - :term:`DISTRO_FEATURES`
6374
6375 - :term:`DISTRO_EXTRA_RDEPENDS`
6376
6377 - :term:`DISTRO_EXTRA_RRECOMMENDS`
6378
6379 - :term:`TCLIBC`
6380
6381 .. tip::
6382
6383 If you want to base your distribution configuration file on the
6384 very basic configuration from OE-Core, you can use
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006385 ``conf/distro/defaultsetup.conf`` as a reference and just include
6386 variables that differ as compared to ``defaultsetup.conf``.
6387 Alternatively, you can create a distribution configuration file
6388 from scratch using the ``defaultsetup.conf`` file or configuration files
Andrew Geissler7e0e3c02022-02-25 20:34:39 +00006389 from another distribution such as Poky as a reference.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006390
6391- *Provide miscellaneous variables:* Be sure to define any other
6392 variables for which you want to create a default or enforce as part
6393 of the distribution configuration. You can include nearly any
6394 variable from the ``local.conf`` file. The variables you use are not
6395 limited to the list in the previous bulleted item.
6396
6397- *Point to Your distribution configuration file:* In your
6398 ``local.conf`` file in the :term:`Build Directory`,
6399 set your
6400 :term:`DISTRO` variable to point to
6401 your distribution's configuration file. For example, if your
6402 distribution's configuration file is named ``mydistro.conf``, then
Andrew Geisslerc926e172021-05-07 16:11:35 -05006403 you point to it as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006404
6405 DISTRO = "mydistro"
6406
6407- *Add more to the layer if necessary:* Use your layer to hold other
6408 information needed for the distribution:
6409
6410 - Add recipes for installing distro-specific configuration files
6411 that are not already installed by another recipe. If you have
6412 distro-specific configuration files that are included by an
6413 existing recipe, you should add an append file (``.bbappend``) for
6414 those. For general information and recommendations on how to add
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006415 recipes to your layer, see the
6416 ":ref:`dev-manual/common-tasks:creating your own layer`" and
6417 ":ref:`dev-manual/common-tasks:following best practices when creating layers`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006418 sections.
6419
6420 - Add any image recipes that are specific to your distribution.
6421
6422 - Add a ``psplash`` append file for a branded splash screen. For
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006423 information on append files, see the
Patrick Williams0ca19cc2021-08-16 14:03:13 -05006424 ":ref:`dev-manual/common-tasks:appending other layers metadata with your layer`"
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006425 section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006426
6427 - Add any other append files to make custom changes that are
6428 specific to individual recipes.
6429
6430Creating a Custom Template Configuration Directory
6431==================================================
6432
6433If you are producing your own customized version of the build system for
6434use by other users, you might want to customize the message shown by the
6435setup script or you might want to change the template configuration
6436files (i.e. ``local.conf`` and ``bblayers.conf``) that are created in a
6437new build directory.
6438
6439The OpenEmbedded build system uses the environment variable
Andrew Geisslerd5838332022-05-27 11:33:10 -05006440:term:`TEMPLATECONF` to locate the directory from which it gathers
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006441configuration information that ultimately ends up in the
6442:term:`Build Directory` ``conf`` directory.
Andrew Geisslerd5838332022-05-27 11:33:10 -05006443By default, :term:`TEMPLATECONF` is set as follows in the ``poky``
Andrew Geisslerc926e172021-05-07 16:11:35 -05006444repository::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006445
6446 TEMPLATECONF=${TEMPLATECONF:-meta-poky/conf}
6447
6448This is the
6449directory used by the build system to find templates from which to build
6450some key configuration files. If you look at this directory, you will
6451see the ``bblayers.conf.sample``, ``local.conf.sample``, and
6452``conf-notes.txt`` files. The build system uses these files to form the
6453respective ``bblayers.conf`` file, ``local.conf`` file, and display the
6454list of BitBake targets when running the setup script.
6455
6456To override these default configuration files with configurations you
6457want used within every new Build Directory, simply set the
Andrew Geisslerd5838332022-05-27 11:33:10 -05006458:term:`TEMPLATECONF` variable to your directory. The :term:`TEMPLATECONF`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006459variable is set in the ``.templateconf`` file, which is in the top-level
6460:term:`Source Directory` folder
6461(e.g. ``poky``). Edit the ``.templateconf`` so that it can locate your
6462directory.
6463
6464Best practices dictate that you should keep your template configuration
6465directory in your custom distribution layer. For example, suppose you
6466have a layer named ``meta-mylayer`` located in your home directory and
6467you want your template configuration directory named ``myconf``.
6468Changing the ``.templateconf`` as follows causes the OpenEmbedded build
6469system to look in your directory and base its configuration files on the
6470``*.sample`` configuration files it finds. The final configuration files
6471(i.e. ``local.conf`` and ``bblayers.conf`` ultimately still end up in
6472your Build Directory, but they are based on your ``*.sample`` files.
6473::
6474
6475 TEMPLATECONF=${TEMPLATECONF:-meta-mylayer/myconf}
6476
6477Aside from the ``*.sample`` configuration files, the ``conf-notes.txt``
6478also resides in the default ``meta-poky/conf`` directory. The script
6479that sets up the build environment (i.e.
6480:ref:`structure-core-script`) uses this file to
6481display BitBake targets as part of the script output. Customizing this
6482``conf-notes.txt`` file is a good way to make sure your list of custom
6483targets appears as part of the script's output.
6484
6485Here is the default list of targets displayed as a result of running
Andrew Geisslerc926e172021-05-07 16:11:35 -05006486either of the setup scripts::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006487
6488 You can now run 'bitbake <target>'
6489
6490 Common targets are:
6491 core-image-minimal
6492 core-image-sato
6493 meta-toolchain
6494 meta-ide-support
6495
6496Changing the listed common targets is as easy as editing your version of
6497``conf-notes.txt`` in your custom template configuration directory and
Andrew Geisslerd5838332022-05-27 11:33:10 -05006498making sure you have :term:`TEMPLATECONF` set to your directory.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006499
Andrew Geissler595f6302022-01-24 19:11:47 +00006500Conserving Disk Space
6501=====================
6502
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006503Conserving Disk Space During Builds
Andrew Geissler595f6302022-01-24 19:11:47 +00006504-----------------------------------
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006505
6506To help conserve disk space during builds, you can add the following
6507statement to your project's ``local.conf`` configuration file found in
Andrew Geisslerc926e172021-05-07 16:11:35 -05006508the :term:`Build Directory`::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006509
6510 INHERIT += "rm_work"
6511
6512Adding this statement deletes the work directory used for
6513building a recipe once the recipe is built. For more information on
6514"rm_work", see the
6515:ref:`rm_work <ref-classes-rm-work>` class in the
6516Yocto Project Reference Manual.
6517
Andrew Geissler595f6302022-01-24 19:11:47 +00006518Purging Duplicate Shared State Cache Files
6519-------------------------------------------
6520
6521After multiple build iterations, the Shared State (sstate) cache can contain
6522duplicate cache files for a given package, while only the most recent one
6523is likely to be reusable. The following command purges all but the
6524newest sstate cache file for each package::
6525
6526 sstate-cache-management.sh --remove-duplicated --cache-dir=build/sstate-cache
6527
6528This command will ask you to confirm the deletions it identifies.
6529
6530Note::
6531
6532 The duplicated sstate cache files of one package must have the same
6533 architecture, which means that sstate cache files with multiple
6534 architectures are not considered as duplicate.
6535
6536Run ``sstate-cache-management.sh`` for more details about this script.
6537
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006538Working with Packages
6539=====================
6540
6541This section describes a few tasks that involve packages:
6542
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006543- :ref:`dev-manual/common-tasks:excluding packages from an image`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006544
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006545- :ref:`dev-manual/common-tasks:incrementing a package version`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006546
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006547- :ref:`dev-manual/common-tasks:handling optional module packaging`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006548
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006549- :ref:`dev-manual/common-tasks:using runtime package management`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006550
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006551- :ref:`dev-manual/common-tasks:generating and using signed packages`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006552
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006553- :ref:`Setting up and running package test
6554 (ptest) <dev-manual/common-tasks:testing packages with ptest>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006555
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006556- :ref:`dev-manual/common-tasks:creating node package manager (npm) packages`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006557
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006558- :ref:`dev-manual/common-tasks:adding custom metadata to packages`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006559
6560Excluding Packages from an Image
6561--------------------------------
6562
6563You might find it necessary to prevent specific packages from being
6564installed into an image. If so, you can use several variables to direct
6565the build system to essentially ignore installing recommended packages
6566or to not install a package at all.
6567
6568The following list introduces variables you can use to prevent packages
6569from being installed into your image. Each of these variables only works
William A. Kennington IIIac69b482021-06-02 12:28:27 -07006570with IPK and RPM package types, not for Debian packages.
6571Also, you can use these variables from your ``local.conf`` file
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006572or attach them to a specific image recipe by using a recipe name
6573override. For more detail on the variables, see the descriptions in the
6574Yocto Project Reference Manual's glossary chapter.
6575
6576- :term:`BAD_RECOMMENDATIONS`:
6577 Use this variable to specify "recommended-only" packages that you do
6578 not want installed.
6579
6580- :term:`NO_RECOMMENDATIONS`:
6581 Use this variable to prevent all "recommended-only" packages from
6582 being installed.
6583
6584- :term:`PACKAGE_EXCLUDE`:
6585 Use this variable to prevent specific packages from being installed
6586 regardless of whether they are "recommended-only" or not. You need to
6587 realize that the build process could fail with an error when you
6588 prevent the installation of a package whose presence is required by
6589 an installed package.
6590
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006591Incrementing a Package Version
6592------------------------------
6593
6594This section provides some background on how binary package versioning
6595is accomplished and presents some of the services, variables, and
6596terminology involved.
6597
6598In order to understand binary package versioning, you need to consider
6599the following:
6600
6601- Binary Package: The binary package that is eventually built and
6602 installed into an image.
6603
6604- Binary Package Version: The binary package version is composed of two
Andrew Geissler615f2f12022-07-15 14:00:58 -05006605 components --- a version and a revision.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006606
6607 .. note::
6608
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006609 Technically, a third component, the "epoch" (i.e. :term:`PE`) is involved
Andrew Geissler09036742021-06-25 14:25:14 -05006610 but this discussion for the most part ignores :term:`PE`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006611
6612 The version and revision are taken from the
6613 :term:`PV` and
6614 :term:`PR` variables, respectively.
6615
Andrew Geissler09036742021-06-25 14:25:14 -05006616- :term:`PV`: The recipe version. :term:`PV` represents the version of the
6617 software being packaged. Do not confuse :term:`PV` with the binary
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006618 package version.
6619
Andrew Geissler5f350902021-07-23 13:09:54 -04006620- :term:`PR`: The recipe revision.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006621
6622- :term:`SRCPV`: The OpenEmbedded
Andrew Geissler09036742021-06-25 14:25:14 -05006623 build system uses this string to help define the value of :term:`PV` when
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006624 the source code revision needs to be included in it.
6625
Andrew Geissler09209ee2020-12-13 08:44:15 -06006626- :yocto_wiki:`PR Service </PR_Service>`: A
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006627 network-based service that helps automate keeping package feeds
6628 compatible with existing package manager applications such as RPM,
6629 APT, and OPKG.
6630
6631Whenever the binary package content changes, the binary package version
6632must change. Changing the binary package version is accomplished by
Andrew Geissler09036742021-06-25 14:25:14 -05006633changing or "bumping" the :term:`PR` and/or :term:`PV` values. Increasing these
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006634values occurs one of two ways:
6635
6636- Automatically using a Package Revision Service (PR Service).
6637
Andrew Geissler09036742021-06-25 14:25:14 -05006638- Manually incrementing the :term:`PR` and/or :term:`PV` variables.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006639
6640Given a primary challenge of any build system and its users is how to
6641maintain a package feed that is compatible with existing package manager
6642applications such as RPM, APT, and OPKG, using an automated system is
6643much preferred over a manual system. In either system, the main
6644requirement is that binary package version numbering increases in a
William A. Kennington IIIac69b482021-06-02 12:28:27 -07006645linear fashion and that there is a number of version components that
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006646support that linear progression. For information on how to ensure
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006647package revisioning remains linear, see the
6648":ref:`dev-manual/common-tasks:automatically incrementing a package version number`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006649section.
6650
6651The following three sections provide related information on the PR
Andrew Geissler09036742021-06-25 14:25:14 -05006652Service, the manual method for "bumping" :term:`PR` and/or :term:`PV`, and on
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006653how to ensure binary package revisioning remains linear.
6654
6655Working With a PR Service
6656~~~~~~~~~~~~~~~~~~~~~~~~~
6657
6658As mentioned, attempting to maintain revision numbers in the
6659:term:`Metadata` is error prone, inaccurate,
6660and causes problems for people submitting recipes. Conversely, the PR
6661Service automatically generates increasing numbers, particularly the
6662revision field, which removes the human element.
6663
6664.. note::
6665
6666 For additional information on using a PR Service, you can see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06006667 :yocto_wiki:`PR Service </PR_Service>` wiki page.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006668
6669The Yocto Project uses variables in order of decreasing priority to
6670facilitate revision numbering (i.e.
6671:term:`PE`,
6672:term:`PV`, and
6673:term:`PR` for epoch, version, and
6674revision, respectively). The values are highly dependent on the policies
6675and procedures of a given distribution and package feed.
6676
6677Because the OpenEmbedded build system uses
Andrew Geissler09209ee2020-12-13 08:44:15 -06006678":ref:`signatures <overview-manual/concepts:checksums (signatures)>`", which are
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006679unique to a given build, the build system knows when to rebuild
6680packages. All the inputs into a given task are represented by a
6681signature, which can trigger a rebuild when different. Thus, the build
Andrew Geissler09036742021-06-25 14:25:14 -05006682system itself does not rely on the :term:`PR`, :term:`PV`, and :term:`PE` numbers to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006683trigger a rebuild. The signatures, however, can be used to generate
6684these values.
6685
6686The PR Service works with both ``OEBasic`` and ``OEBasicHash``
Andrew Geissler09036742021-06-25 14:25:14 -05006687generators. The value of :term:`PR` bumps when the checksum changes and the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006688different generator mechanisms change signatures under different
6689circumstances.
6690
6691As implemented, the build system includes values from the PR Service
Andrew Geissler09036742021-06-25 14:25:14 -05006692into the :term:`PR` field as an addition using the form "``.x``" so ``r0``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006693becomes ``r0.1``, ``r0.2`` and so forth. This scheme allows existing
Andrew Geissler09036742021-06-25 14:25:14 -05006694:term:`PR` values to be used for whatever reasons, which include manual
6695:term:`PR` bumps, should it be necessary.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006696
6697By default, the PR Service is not enabled or running. Thus, the packages
6698generated are just "self consistent". The build system adds and removes
6699packages and there are no guarantees about upgrade paths but images will
6700be consistent and correct with the latest changes.
6701
William A. Kennington IIIac69b482021-06-02 12:28:27 -07006702The simplest form for a PR Service is for a single host
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006703development system that builds the package feed (building system). For
6704this scenario, you can enable a local PR Service by setting
6705:term:`PRSERV_HOST` in your
Andrew Geisslerc926e172021-05-07 16:11:35 -05006706``local.conf`` file in the :term:`Build Directory`::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006707
6708 PRSERV_HOST = "localhost:0"
6709
6710Once the service is started, packages will automatically
Andrew Geissler09036742021-06-25 14:25:14 -05006711get increasing :term:`PR` values and BitBake takes care of starting and
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006712stopping the server.
6713
6714If you have a more complex setup where multiple host development systems
6715work against a common, shared package feed, you have a single PR Service
6716running and it is connected to each building system. For this scenario,
Andrew Geisslerc926e172021-05-07 16:11:35 -05006717you need to start the PR Service using the ``bitbake-prserv`` command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006718
6719 bitbake-prserv --host ip --port port --start
6720
6721In addition to
6722hand-starting the service, you need to update the ``local.conf`` file of
6723each building system as described earlier so each system points to the
6724server and port.
6725
6726It is also recommended you use build history, which adds some sanity
6727checks to binary package versions, in conjunction with the server that
6728is running the PR Service. To enable build history, add the following to
Andrew Geisslerc926e172021-05-07 16:11:35 -05006729each building system's ``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006730
6731 # It is recommended to activate "buildhistory" for testing the PR service
6732 INHERIT += "buildhistory"
6733 BUILDHISTORY_COMMIT = "1"
6734
6735For information on build
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006736history, see the
6737":ref:`dev-manual/common-tasks:maintaining build output quality`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006738
6739.. note::
6740
Andrew Geissler09036742021-06-25 14:25:14 -05006741 The OpenEmbedded build system does not maintain :term:`PR` information as
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006742 part of the shared state (sstate) packages. If you maintain an sstate
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006743 feed, it's expected that either all your building systems that
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006744 contribute to the sstate feed use a shared PR Service, or you do not
6745 run a PR Service on any of your building systems. Having some systems
6746 use a PR Service while others do not leads to obvious problems.
6747
6748 For more information on shared state, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06006749 ":ref:`overview-manual/concepts:shared state cache`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006750 section in the Yocto Project Overview and Concepts Manual.
6751
6752Manually Bumping PR
6753~~~~~~~~~~~~~~~~~~~
6754
6755The alternative to setting up a PR Service is to manually "bump" the
6756:term:`PR` variable.
6757
6758If a committed change results in changing the package output, then the
6759value of the PR variable needs to be increased (or "bumped") as part of
Andrew Geissler09036742021-06-25 14:25:14 -05006760that commit. For new recipes you should add the :term:`PR` variable and set
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006761its initial value equal to "r0", which is the default. Even though the
6762default value is "r0", the practice of adding it to a new recipe makes
6763it harder to forget to bump the variable when you make changes to the
6764recipe in future.
6765
6766If you are sharing a common ``.inc`` file with multiple recipes, you can
Andrew Geissler09036742021-06-25 14:25:14 -05006767also use the :term:`INC_PR` variable to ensure that the recipes sharing the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006768``.inc`` file are rebuilt when the ``.inc`` file itself is changed. The
Andrew Geissler09036742021-06-25 14:25:14 -05006769``.inc`` file must set :term:`INC_PR` (initially to "r0"), and all recipes
6770referring to it should set :term:`PR` to "${INC_PR}.0" initially,
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006771incrementing the last number when the recipe is changed. If the ``.inc``
Andrew Geissler09036742021-06-25 14:25:14 -05006772file is changed then its :term:`INC_PR` should be incremented.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006773
Andrew Geissler09036742021-06-25 14:25:14 -05006774When upgrading the version of a binary package, assuming the :term:`PV`
6775changes, the :term:`PR` variable should be reset to "r0" (or "${INC_PR}.0"
6776if you are using :term:`INC_PR`).
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006777
6778Usually, version increases occur only to binary packages. However, if
Andrew Geissler09036742021-06-25 14:25:14 -05006779for some reason :term:`PV` changes but does not increase, you can increase
6780the :term:`PE` variable (Package Epoch). The :term:`PE` variable defaults to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006781"0".
6782
6783Binary package version numbering strives to follow the `Debian Version
6784Field Policy
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006785Guidelines <https://www.debian.org/doc/debian-policy/ch-controlfields.html>`__.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006786These guidelines define how versions are compared and what "increasing"
6787a version means.
6788
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006789Automatically Incrementing a Package Version Number
6790~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
6791
6792When fetching a repository, BitBake uses the
6793:term:`SRCREV` variable to determine
6794the specific source code revision from which to build. You set the
Andrew Geissler09036742021-06-25 14:25:14 -05006795:term:`SRCREV` variable to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006796:term:`AUTOREV` to cause the
6797OpenEmbedded build system to automatically use the latest revision of
Andrew Geisslerc926e172021-05-07 16:11:35 -05006798the software::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006799
6800 SRCREV = "${AUTOREV}"
6801
Andrew Geissler09036742021-06-25 14:25:14 -05006802Furthermore, you need to reference :term:`SRCPV` in :term:`PV` in order to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006803automatically update the version whenever the revision of the source
Andrew Geisslerc926e172021-05-07 16:11:35 -05006804code changes. Here is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006805
6806 PV = "1.0+git${SRCPV}"
6807
Andrew Geissler09036742021-06-25 14:25:14 -05006808The OpenEmbedded build system substitutes :term:`SRCPV` with the following:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006809
6810.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006811
6812 AUTOINC+source_code_revision
6813
6814The build system replaces the ``AUTOINC``
6815with a number. The number used depends on the state of the PR Service:
6816
6817- If PR Service is enabled, the build system increments the number,
6818 which is similar to the behavior of
6819 :term:`PR`. This behavior results in
6820 linearly increasing package versions, which is desirable. Here is an
6821 example:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006822
6823 .. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006824
6825 hello-world-git_0.0+git0+b6558dd387-r0.0_armv7a-neon.ipk
6826 hello-world-git_0.0+git1+dd2f5c3565-r0.0_armv7a-neon.ipk
6827
6828- If PR Service is not enabled, the build system replaces the
6829 ``AUTOINC`` placeholder with zero (i.e. "0"). This results in
6830 changing the package version since the source revision is included.
6831 However, package versions are not increased linearly. Here is an
6832 example:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006833
6834 .. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006835
6836 hello-world-git_0.0+git0+b6558dd387-r0.0_armv7a-neon.ipk
6837 hello-world-git_0.0+git0+dd2f5c3565-r0.0_armv7a-neon.ipk
6838
6839In summary, the OpenEmbedded build system does not track the history of
6840binary package versions for this purpose. ``AUTOINC``, in this case, is
Andrew Geissler09036742021-06-25 14:25:14 -05006841comparable to :term:`PR`. If PR server is not enabled, ``AUTOINC`` in the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006842package version is simply replaced by "0". If PR server is enabled, the
6843build system keeps track of the package versions and bumps the number
6844when the package revision changes.
6845
6846Handling Optional Module Packaging
6847----------------------------------
6848
6849Many pieces of software split functionality into optional modules (or
6850plugins) and the plugins that are built might depend on configuration
6851options. To avoid having to duplicate the logic that determines what
6852modules are available in your recipe or to avoid having to package each
6853module by hand, the OpenEmbedded build system provides functionality to
6854handle module packaging dynamically.
6855
6856To handle optional module packaging, you need to do two things:
6857
6858- Ensure the module packaging is actually done.
6859
6860- Ensure that any dependencies on optional modules from other recipes
6861 are satisfied by your recipe.
6862
6863Making Sure the Packaging is Done
6864~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
6865
6866To ensure the module packaging actually gets done, you use the
6867``do_split_packages`` function within the ``populate_packages`` Python
6868function in your recipe. The ``do_split_packages`` function searches for
6869a pattern of files or directories under a specified path and creates a
6870package for each one it finds by appending to the
6871:term:`PACKAGES` variable and
Patrick Williams0ca19cc2021-08-16 14:03:13 -05006872setting the appropriate values for ``FILES:packagename``,
6873``RDEPENDS:packagename``, ``DESCRIPTION:packagename``, and so forth.
Andrew Geisslerc926e172021-05-07 16:11:35 -05006874Here is an example from the ``lighttpd`` recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006875
Patrick Williams0ca19cc2021-08-16 14:03:13 -05006876 python populate_packages:prepend () {
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006877 lighttpd_libdir = d.expand('${libdir}')
6878 do_split_packages(d, lighttpd_libdir, '^mod_(.*).so$',
6879 'lighttpd-module-%s', 'Lighttpd module for %s',
6880 extra_depends='')
6881 }
6882
6883The previous example specifies a number of things in the call to
6884``do_split_packages``.
6885
6886- A directory within the files installed by your recipe through
6887 ``do_install`` in which to search.
6888
6889- A regular expression used to match module files in that directory. In
6890 the example, note the parentheses () that mark the part of the
6891 expression from which the module name should be derived.
6892
6893- A pattern to use for the package names.
6894
6895- A description for each package.
6896
6897- An empty string for ``extra_depends``, which disables the default
6898 dependency on the main ``lighttpd`` package. Thus, if a file in
6899 ``${libdir}`` called ``mod_alias.so`` is found, a package called
6900 ``lighttpd-module-alias`` is created for it and the
6901 :term:`DESCRIPTION` is set to
6902 "Lighttpd module for alias".
6903
6904Often, packaging modules is as simple as the previous example. However,
William A. Kennington IIIac69b482021-06-02 12:28:27 -07006905there are more advanced options that you can use within
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006906``do_split_packages`` to modify its behavior. And, if you need to, you
6907can add more logic by specifying a hook function that is called for each
6908package. It is also perfectly acceptable to call ``do_split_packages``
6909multiple times if you have more than one set of modules to package.
6910
6911For more examples that show how to use ``do_split_packages``, see the
6912``connman.inc`` file in the ``meta/recipes-connectivity/connman/``
Andrew Geissler09209ee2020-12-13 08:44:15 -06006913directory of the ``poky`` :ref:`source repository <overview-manual/development-environment:yocto project source repositories>`. You can
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006914also find examples in ``meta/classes/kernel.bbclass``.
6915
6916Following is a reference that shows ``do_split_packages`` mandatory and
Andrew Geisslerc926e172021-05-07 16:11:35 -05006917optional arguments::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006918
6919 Mandatory arguments
6920
6921 root
6922 The path in which to search
6923 file_regex
6924 Regular expression to match searched files.
6925 Use parentheses () to mark the part of this
6926 expression that should be used to derive the
6927 module name (to be substituted where %s is
6928 used in other function arguments as noted below)
6929 output_pattern
6930 Pattern to use for the package names. Must
6931 include %s.
6932 description
6933 Description to set for each package. Must
6934 include %s.
6935
6936 Optional arguments
6937
6938 postinst
6939 Postinstall script to use for all packages
6940 (as a string)
6941 recursive
Andrew Geissler615f2f12022-07-15 14:00:58 -05006942 True to perform a recursive search --- default
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006943 False
6944 hook
6945 A hook function to be called for every match.
6946 The function will be called with the following
6947 arguments (in the order listed):
6948
6949 f
6950 Full path to the file/directory match
6951 pkg
6952 The package name
6953 file_regex
6954 As above
6955 output_pattern
6956 As above
6957 modulename
6958 The module name derived using file_regex
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006959 extra_depends
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006960 Extra runtime dependencies (RDEPENDS) to be
6961 set for all packages. The default value of None
6962 causes a dependency on the main package
Andrew Geissler615f2f12022-07-15 14:00:58 -05006963 (${PN}) --- if you do not want this, pass empty
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006964 string '' for this parameter.
6965 aux_files_pattern
6966 Extra item(s) to be added to FILES for each
6967 package. Can be a single string item or a list
6968 of strings for multiple items. Must include %s.
6969 postrm
6970 postrm script to use for all packages (as a
6971 string)
6972 allow_dirs
6973 True to allow directories to be matched -
6974 default False
6975 prepend
6976 If True, prepend created packages to PACKAGES
6977 instead of the default False which appends them
6978 match_path
6979 match file_regex on the whole relative path to
Patrick Williams0ca19cc2021-08-16 14:03:13 -05006980 the root rather than just the filename
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006981 aux_files_pattern_verbatim
6982 Extra item(s) to be added to FILES for each
6983 package, using the actual derived module name
6984 rather than converting it to something legal
6985 for a package name. Can be a single string item
6986 or a list of strings for multiple items. Must
6987 include %s.
6988 allow_links
Andrew Geissler615f2f12022-07-15 14:00:58 -05006989 True to allow symlinks to be matched --- default
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006990 False
6991 summary
6992 Summary to set for each package. Must include %s;
6993 defaults to description if not set.
6994
6995
6996
6997Satisfying Dependencies
6998~~~~~~~~~~~~~~~~~~~~~~~
6999
7000The second part for handling optional module packaging is to ensure that
7001any dependencies on optional modules from other recipes are satisfied by
7002your recipe. You can be sure these dependencies are satisfied by using
7003the :term:`PACKAGES_DYNAMIC`
7004variable. Here is an example that continues with the ``lighttpd`` recipe
Andrew Geisslerc926e172021-05-07 16:11:35 -05007005shown earlier::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007006
7007 PACKAGES_DYNAMIC = "lighttpd-module-.*"
7008
7009The name
7010specified in the regular expression can of course be anything. In this
7011example, it is ``lighttpd-module-`` and is specified as the prefix to
7012ensure that any :term:`RDEPENDS` and
7013:term:`RRECOMMENDS` on a package
7014name starting with the prefix are satisfied during build time. If you
7015are using ``do_split_packages`` as described in the previous section,
Andrew Geissler09036742021-06-25 14:25:14 -05007016the value you put in :term:`PACKAGES_DYNAMIC` should correspond to the name
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007017pattern specified in the call to ``do_split_packages``.
7018
7019Using Runtime Package Management
7020--------------------------------
7021
7022During a build, BitBake always transforms a recipe into one or more
7023packages. For example, BitBake takes the ``bash`` recipe and produces a
7024number of packages (e.g. ``bash``, ``bash-bashbug``,
7025``bash-completion``, ``bash-completion-dbg``, ``bash-completion-dev``,
7026``bash-completion-extra``, ``bash-dbg``, and so forth). Not all
7027generated packages are included in an image.
7028
7029In several situations, you might need to update, add, remove, or query
7030the packages on a target device at runtime (i.e. without having to
7031generate a new image). Examples of such situations include:
7032
7033- You want to provide in-the-field updates to deployed devices (e.g.
7034 security updates).
7035
7036- You want to have a fast turn-around development cycle for one or more
7037 applications that run on your device.
7038
7039- You want to temporarily install the "debug" packages of various
7040 applications on your device so that debugging can be greatly improved
7041 by allowing access to symbols and source debugging.
7042
7043- You want to deploy a more minimal package selection of your device
7044 but allow in-the-field updates to add a larger selection for
7045 customization.
7046
7047In all these situations, you have something similar to a more
7048traditional Linux distribution in that in-field devices are able to
7049receive pre-compiled packages from a server for installation or update.
7050Being able to install these packages on a running, in-field device is
7051what is termed "runtime package management".
7052
7053In order to use runtime package management, you need a host or server
7054machine that serves up the pre-compiled packages plus the required
7055metadata. You also need package manipulation tools on the target. The
7056build machine is a likely candidate to act as the server. However, that
7057machine does not necessarily have to be the package server. The build
7058machine could push its artifacts to another machine that acts as the
7059server (e.g. Internet-facing). In fact, doing so is advantageous for a
7060production environment as getting the packages away from the development
7061system's build directory prevents accidental overwrites.
7062
7063A simple build that targets just one device produces more than one
7064package database. In other words, the packages produced by a build are
7065separated out into a couple of different package groupings based on
7066criteria such as the target's CPU architecture, the target board, or the
7067C library used on the target. For example, a build targeting the
7068``qemux86`` device produces the following three package databases:
7069``noarch``, ``i586``, and ``qemux86``. If you wanted your ``qemux86``
7070device to be aware of all the packages that were available to it, you
7071would need to point it to each of these databases individually. In a
7072similar way, a traditional Linux distribution usually is configured to
7073be aware of a number of software repositories from which it retrieves
7074packages.
7075
7076Using runtime package management is completely optional and not required
7077for a successful build or deployment in any way. But if you want to make
7078use of runtime package management, you need to do a couple things above
7079and beyond the basics. The remainder of this section describes what you
7080need to do.
7081
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007082Build Considerations
7083~~~~~~~~~~~~~~~~~~~~
7084
7085This section describes build considerations of which you need to be
7086aware in order to provide support for runtime package management.
7087
7088When BitBake generates packages, it needs to know what format or formats
7089to use. In your configuration, you use the
7090:term:`PACKAGE_CLASSES`
7091variable to specify the format:
7092
70931. Open the ``local.conf`` file inside your
7094 :term:`Build Directory` (e.g.
Andrew Geissler95ac1b82021-03-31 14:34:31 -05007095 ``poky/build/conf/local.conf``).
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007096
Andrew Geisslerc926e172021-05-07 16:11:35 -050070972. Select the desired package format as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007098
7099 PACKAGE_CLASSES ?= "package_packageformat"
7100
7101 where packageformat can be "ipk", "rpm",
7102 "deb", or "tar" which are the supported package formats.
7103
7104 .. note::
7105
7106 Because the Yocto Project supports four different package formats,
7107 you can set the variable with more than one argument. However, the
7108 OpenEmbedded build system only uses the first argument when
7109 creating an image or Software Development Kit (SDK).
7110
7111If you would like your image to start off with a basic package database
7112containing the packages in your current build as well as to have the
7113relevant tools available on the target for runtime package management,
7114you can include "package-management" in the
7115:term:`IMAGE_FEATURES`
7116variable. Including "package-management" in this configuration variable
7117ensures that when the image is assembled for your target, the image
7118includes the currently-known package databases as well as the
7119target-specific tools required for runtime package management to be
7120performed on the target. However, this is not strictly necessary. You
7121could start your image off without any databases but only include the
7122required on-target package tool(s). As an example, you could include
7123"opkg" in your
7124:term:`IMAGE_INSTALL` variable
7125if you are using the IPK package format. You can then initialize your
7126target's package database(s) later once your image is up and running.
7127
7128Whenever you perform any sort of build step that can potentially
7129generate a package or modify existing package, it is always a good idea
7130to re-generate the package index after the build by using the following
Andrew Geisslerc926e172021-05-07 16:11:35 -05007131command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007132
7133 $ bitbake package-index
7134
7135It might be tempting to build the
7136package and the package index at the same time with a command such as
Andrew Geisslerc926e172021-05-07 16:11:35 -05007137the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007138
7139 $ bitbake some-package package-index
7140
7141Do not do this as
7142BitBake does not schedule the package index for after the completion of
7143the package you are building. Consequently, you cannot be sure of the
7144package index including information for the package you just built.
7145Thus, be sure to run the package update step separately after building
7146any packages.
7147
7148You can use the
7149:term:`PACKAGE_FEED_ARCHS`,
7150:term:`PACKAGE_FEED_BASE_PATHS`,
7151and
7152:term:`PACKAGE_FEED_URIS`
7153variables to pre-configure target images to use a package feed. If you
7154do not define these variables, then manual steps as described in the
7155subsequent sections are necessary to configure the target. You should
7156set these variables before building the image in order to produce a
7157correctly configured image.
7158
7159When your build is complete, your packages reside in the
7160``${TMPDIR}/deploy/packageformat`` directory. For example, if
7161``${``\ :term:`TMPDIR`\ ``}`` is
7162``tmp`` and your selected package type is RPM, then your RPM packages
7163are available in ``tmp/deploy/rpm``.
7164
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007165Host or Server Machine Setup
7166~~~~~~~~~~~~~~~~~~~~~~~~~~~~
7167
7168Although other protocols are possible, a server using HTTP typically
7169serves packages. If you want to use HTTP, then set up and configure a
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007170web server such as Apache 2, lighttpd, or Python web server on the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007171machine serving the packages.
7172
7173To keep things simple, this section describes how to set up a
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007174Python web server to share package feeds from the developer's
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007175machine. Although this server might not be the best for a production
7176environment, the setup is simple and straight forward. Should you want
7177to use a different server more suited for production (e.g. Apache 2,
7178Lighttpd, or Nginx), take the appropriate steps to do so.
7179
7180From within the build directory where you have built an image based on
7181your packaging choice (i.e. the
7182:term:`PACKAGE_CLASSES`
7183setting), simply start the server. The following example assumes a build
Andrew Geissler09036742021-06-25 14:25:14 -05007184directory of ``poky/build/tmp/deploy/rpm`` and a :term:`PACKAGE_CLASSES`
Andrew Geisslerc926e172021-05-07 16:11:35 -05007185setting of "package_rpm"::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007186
Andrew Geissler95ac1b82021-03-31 14:34:31 -05007187 $ cd poky/build/tmp/deploy/rpm
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007188 $ python3 -m http.server
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007189
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007190Target Setup
7191~~~~~~~~~~~~
7192
7193Setting up the target differs depending on the package management
7194system. This section provides information for RPM, IPK, and DEB.
7195
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007196Using RPM
7197^^^^^^^^^
7198
7199The `Dandified Packaging
7200Tool <https://en.wikipedia.org/wiki/DNF_(software)>`__ (DNF) performs
7201runtime package management of RPM packages. In order to use DNF for
7202runtime package management, you must perform an initial setup on the
7203target machine for cases where the ``PACKAGE_FEED_*`` variables were not
7204set as part of the image that is running on the target. This means if
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05007205you built your image and did not use these variables as part of the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007206build and your image is now running on the target, you need to perform
7207the steps in this section if you want to use runtime package management.
7208
7209.. note::
7210
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007211 For information on the ``PACKAGE_FEED_*`` variables, see
7212 :term:`PACKAGE_FEED_ARCHS`, :term:`PACKAGE_FEED_BASE_PATHS`, and
7213 :term:`PACKAGE_FEED_URIS` in the Yocto Project Reference Manual variables
7214 glossary.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007215
7216On the target, you must inform DNF that package databases are available.
7217You do this by creating a file named
7218``/etc/yum.repos.d/oe-packages.repo`` and defining the ``oe-packages``.
7219
7220As an example, assume the target is able to use the following package
7221databases: ``all``, ``i586``, and ``qemux86`` from a server named
7222``my.server``. The specifics for setting up the web server are up to
7223you. The critical requirement is that the URIs in the target repository
7224configuration point to the correct remote location for the feeds.
7225
7226.. note::
7227
7228 For development purposes, you can point the web server to the build
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007229 system's ``deploy`` directory. However, for production use, it is better to
7230 copy the package directories to a location outside of the build area and use
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007231 that location. Doing so avoids situations where the build system
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007232 overwrites or changes the ``deploy`` directory.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007233
7234When telling DNF where to look for the package databases, you must
7235declare individual locations per architecture or a single location used
7236for all architectures. You cannot do both:
7237
7238- *Create an Explicit List of Architectures:* Define individual base
7239 URLs to identify where each package database is located:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007240
7241 .. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007242
7243 [oe-packages]
7244 baseurl=http://my.server/rpm/i586 http://my.server/rpm/qemux86 http://my.server/rpm/all
7245
7246 This example
7247 informs DNF about individual package databases for all three
7248 architectures.
7249
7250- *Create a Single (Full) Package Index:* Define a single base URL that
Andrew Geisslerc926e172021-05-07 16:11:35 -05007251 identifies where a full package database is located::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007252
7253 [oe-packages]
7254 baseurl=http://my.server/rpm
7255
7256 This example informs DNF about a single
7257 package database that contains all the package index information for
7258 all supported architectures.
7259
7260Once you have informed DNF where to find the package databases, you need
7261to fetch them:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007262
7263.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007264
7265 # dnf makecache
7266
7267DNF is now able to find, install, and
7268upgrade packages from the specified repository or repositories.
7269
7270.. note::
7271
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007272 See the `DNF documentation <https://dnf.readthedocs.io/en/latest/>`__ for
7273 additional information.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007274
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007275Using IPK
7276^^^^^^^^^
7277
7278The ``opkg`` application performs runtime package management of IPK
7279packages. You must perform an initial setup for ``opkg`` on the target
7280machine if the
7281:term:`PACKAGE_FEED_ARCHS`,
7282:term:`PACKAGE_FEED_BASE_PATHS`,
7283and
7284:term:`PACKAGE_FEED_URIS`
7285variables have not been set or the target image was built before the
7286variables were set.
7287
7288The ``opkg`` application uses configuration files to find available
7289package databases. Thus, you need to create a configuration file inside
Andrew Geissler7e0e3c02022-02-25 20:34:39 +00007290the ``/etc/opkg/`` directory, which informs ``opkg`` of any repository
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007291you want to use.
7292
7293As an example, suppose you are serving packages from a ``ipk/``
7294directory containing the ``i586``, ``all``, and ``qemux86`` databases
7295through an HTTP server named ``my.server``. On the target, create a
7296configuration file (e.g. ``my_repo.conf``) inside the ``/etc/opkg/``
7297directory containing the following:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007298
7299.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007300
7301 src/gz all http://my.server/ipk/all
7302 src/gz i586 http://my.server/ipk/i586
7303 src/gz qemux86 http://my.server/ipk/qemux86
7304
7305Next, instruct ``opkg`` to fetch the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007306repository information:
7307
7308.. code-block:: none
7309
7310 # opkg update
7311
7312The ``opkg`` application is now able to find, install, and upgrade packages
7313from the specified repository.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007314
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007315Using DEB
7316^^^^^^^^^
7317
7318The ``apt`` application performs runtime package management of DEB
7319packages. This application uses a source list file to find available
7320package databases. You must perform an initial setup for ``apt`` on the
7321target machine if the
7322:term:`PACKAGE_FEED_ARCHS`,
7323:term:`PACKAGE_FEED_BASE_PATHS`,
7324and
7325:term:`PACKAGE_FEED_URIS`
7326variables have not been set or the target image was built before the
7327variables were set.
7328
7329To inform ``apt`` of the repository you want to use, you might create a
7330list file (e.g. ``my_repo.list``) inside the
7331``/etc/apt/sources.list.d/`` directory. As an example, suppose you are
7332serving packages from a ``deb/`` directory containing the ``i586``,
7333``all``, and ``qemux86`` databases through an HTTP server named
7334``my.server``. The list file should contain:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007335
7336.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007337
7338 deb http://my.server/deb/all ./
7339 deb http://my.server/deb/i586 ./
7340 deb http://my.server/deb/qemux86 ./
7341
7342Next, instruct the ``apt`` application
7343to fetch the repository information:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007344
7345.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007346
Andrew Geisslereff27472021-10-29 15:35:00 -05007347 $ sudo apt update
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007348
7349After this step,
7350``apt`` is able to find, install, and upgrade packages from the
7351specified repository.
7352
7353Generating and Using Signed Packages
7354------------------------------------
7355
7356In order to add security to RPM packages used during a build, you can
7357take steps to securely sign them. Once a signature is verified, the
7358OpenEmbedded build system can use the package in the build. If security
Andrew Geissler595f6302022-01-24 19:11:47 +00007359fails for a signed package, the build system stops the build.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007360
7361This section describes how to sign RPM packages during a build and how
7362to use signed package feeds (repositories) when doing a build.
7363
7364Signing RPM Packages
7365~~~~~~~~~~~~~~~~~~~~
7366
7367To enable signing RPM packages, you must set up the following
7368configurations in either your ``local.config`` or ``distro.config``
Andrew Geisslerc926e172021-05-07 16:11:35 -05007369file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007370
7371 # Inherit sign_rpm.bbclass to enable signing functionality
7372 INHERIT += " sign_rpm"
7373 # Define the GPG key that will be used for signing.
7374 RPM_GPG_NAME = "key_name"
7375 # Provide passphrase for the key
7376 RPM_GPG_PASSPHRASE = "passphrase"
7377
7378.. note::
7379
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007380 Be sure to supply appropriate values for both `key_name` and
7381 `passphrase`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007382
7383Aside from the ``RPM_GPG_NAME`` and ``RPM_GPG_PASSPHRASE`` variables in
William A. Kennington IIIac69b482021-06-02 12:28:27 -07007384the previous example, two optional variables related to signing are available:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007385
7386- *GPG_BIN:* Specifies a ``gpg`` binary/wrapper that is executed
7387 when the package is signed.
7388
7389- *GPG_PATH:* Specifies the ``gpg`` home directory used when the
7390 package is signed.
7391
7392Processing Package Feeds
7393~~~~~~~~~~~~~~~~~~~~~~~~
7394
7395In addition to being able to sign RPM packages, you can also enable
7396signed package feeds for IPK and RPM packages.
7397
7398The steps you need to take to enable signed package feed use are similar
7399to the steps used to sign RPM packages. You must define the following in
Andrew Geisslerc926e172021-05-07 16:11:35 -05007400your ``local.config`` or ``distro.config`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007401
7402 INHERIT += "sign_package_feed"
7403 PACKAGE_FEED_GPG_NAME = "key_name"
7404 PACKAGE_FEED_GPG_PASSPHRASE_FILE = "path_to_file_containing_passphrase"
7405
William A. Kennington IIIac69b482021-06-02 12:28:27 -07007406For signed package feeds, the passphrase must be specified in a separate file,
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007407which is pointed to by the ``PACKAGE_FEED_GPG_PASSPHRASE_FILE``
7408variable. Regarding security, keeping a plain text passphrase out of the
7409configuration is more secure.
7410
7411Aside from the ``PACKAGE_FEED_GPG_NAME`` and
7412``PACKAGE_FEED_GPG_PASSPHRASE_FILE`` variables, three optional variables
William A. Kennington IIIac69b482021-06-02 12:28:27 -07007413related to signed package feeds are available:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007414
7415- *GPG_BIN* Specifies a ``gpg`` binary/wrapper that is executed
7416 when the package is signed.
7417
7418- *GPG_PATH:* Specifies the ``gpg`` home directory used when the
7419 package is signed.
7420
7421- *PACKAGE_FEED_GPG_SIGNATURE_TYPE:* Specifies the type of ``gpg``
7422 signature. This variable applies only to RPM and IPK package feeds.
7423 Allowable values for the ``PACKAGE_FEED_GPG_SIGNATURE_TYPE`` are
7424 "ASC", which is the default and specifies ascii armored, and "BIN",
7425 which specifies binary.
7426
7427Testing Packages With ptest
7428---------------------------
7429
7430A Package Test (ptest) runs tests against packages built by the
7431OpenEmbedded build system on the target machine. A ptest contains at
7432least two items: the actual test, and a shell script (``run-ptest``)
7433that starts the test. The shell script that starts the test must not
Andrew Geissler615f2f12022-07-15 14:00:58 -05007434contain the actual test --- the script only starts the test. On the other
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007435hand, the test can be anything from a simple shell script that runs a
7436binary and checks the output to an elaborate system of test binaries and
7437data files.
7438
Andrew Geisslerc926e172021-05-07 16:11:35 -05007439The test generates output in the format used by Automake::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007440
7441 result: testname
7442
7443where the result can be ``PASS``, ``FAIL``, or ``SKIP``, and
7444the testname can be any identifying string.
7445
7446For a list of Yocto Project recipes that are already enabled with ptest,
Andrew Geissler09209ee2020-12-13 08:44:15 -06007447see the :yocto_wiki:`Ptest </Ptest>` wiki page.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007448
7449.. note::
7450
7451 A recipe is "ptest-enabled" if it inherits the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007452 :ref:`ptest <ref-classes-ptest>` class.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007453
7454Adding ptest to Your Build
7455~~~~~~~~~~~~~~~~~~~~~~~~~~
7456
7457To add package testing to your build, add the
7458:term:`DISTRO_FEATURES` and
7459:term:`EXTRA_IMAGE_FEATURES`
7460variables to your ``local.conf`` file, which is found in the
Andrew Geisslerc926e172021-05-07 16:11:35 -05007461:term:`Build Directory`::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007462
Patrick Williams0ca19cc2021-08-16 14:03:13 -05007463 DISTRO_FEATURES:append = " ptest"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007464 EXTRA_IMAGE_FEATURES += "ptest-pkgs"
7465
7466Once your build is complete, the ptest files are installed into the
7467``/usr/lib/package/ptest`` directory within the image, where ``package``
7468is the name of the package.
7469
7470Running ptest
7471~~~~~~~~~~~~~
7472
7473The ``ptest-runner`` package installs a shell script that loops through
7474all installed ptest test suites and runs them in sequence. Consequently,
7475you might want to add this package to your image.
7476
7477Getting Your Package Ready
7478~~~~~~~~~~~~~~~~~~~~~~~~~~
7479
7480In order to enable a recipe to run installed ptests on target hardware,
7481you need to prepare the recipes that build the packages you want to
7482test. Here is what you have to do for each recipe:
7483
7484- *Be sure the recipe inherits
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007485 the* :ref:`ptest <ref-classes-ptest>` *class:*
Andrew Geisslerc926e172021-05-07 16:11:35 -05007486 Include the following line in each recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007487
7488 inherit ptest
7489
7490- *Create run-ptest:* This script starts your test. Locate the
7491 script where you will refer to it using
7492 :term:`SRC_URI`. Here is an
Andrew Geisslerc926e172021-05-07 16:11:35 -05007493 example that starts a test for ``dbus``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007494
7495 #!/bin/sh
7496 cd test
7497 make -k runtest-TESTS
7498
7499- *Ensure dependencies are met:* If the test adds build or runtime
7500 dependencies that normally do not exist for the package (such as
7501 requiring "make" to run the test suite), use the
7502 :term:`DEPENDS` and
7503 :term:`RDEPENDS` variables in
7504 your recipe in order for the package to meet the dependencies. Here
Andrew Geisslerc926e172021-05-07 16:11:35 -05007505 is an example where the package has a runtime dependency on "make"::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007506
Patrick Williams0ca19cc2021-08-16 14:03:13 -05007507 RDEPENDS:${PN}-ptest += "make"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007508
7509- *Add a function to build the test suite:* Not many packages support
7510 cross-compilation of their test suites. Consequently, you usually
7511 need to add a cross-compilation function to the package.
7512
7513 Many packages based on Automake compile and run the test suite by
7514 using a single command such as ``make check``. However, the host
7515 ``make check`` builds and runs on the same computer, while
7516 cross-compiling requires that the package is built on the host but
7517 executed for the target architecture (though often, as in the case
7518 for ptest, the execution occurs on the host). The built version of
7519 Automake that ships with the Yocto Project includes a patch that
7520 separates building and execution. Consequently, packages that use the
7521 unaltered, patched version of ``make check`` automatically
7522 cross-compiles.
7523
7524 Regardless, you still must add a ``do_compile_ptest`` function to
7525 build the test suite. Add a function similar to the following to your
Andrew Geisslerc926e172021-05-07 16:11:35 -05007526 recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007527
7528 do_compile_ptest() {
7529 oe_runmake buildtest-TESTS
7530 }
7531
7532- *Ensure special configurations are set:* If the package requires
7533 special configurations prior to compiling the test code, you must
7534 insert a ``do_configure_ptest`` function into the recipe.
7535
7536- *Install the test suite:* The ``ptest`` class automatically copies
7537 the file ``run-ptest`` to the target and then runs make
7538 ``install-ptest`` to run the tests. If this is not enough, you need
7539 to create a ``do_install_ptest`` function and make sure it gets
7540 called after the "make install-ptest" completes.
7541
7542Creating Node Package Manager (NPM) Packages
7543--------------------------------------------
7544
7545`NPM <https://en.wikipedia.org/wiki/Npm_(software)>`__ is a package
7546manager for the JavaScript programming language. The Yocto Project
Andrew Geissler09209ee2020-12-13 08:44:15 -06007547supports the NPM :ref:`fetcher <bitbake:bitbake-user-manual/bitbake-user-manual-fetching:fetchers>`. You can
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007548use this fetcher in combination with
Andrew Geissler09209ee2020-12-13 08:44:15 -06007549:doc:`devtool </ref-manual/devtool-reference>` to create
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007550recipes that produce NPM packages.
7551
William A. Kennington IIIac69b482021-06-02 12:28:27 -07007552There are two workflows that allow you to create NPM packages using
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007553``devtool``: the NPM registry modules method and the NPM project code
7554method.
7555
7556.. note::
7557
7558 While it is possible to create NPM recipes manually, using
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007559 ``devtool`` is far simpler.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007560
7561Additionally, some requirements and caveats exist.
7562
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007563Requirements and Caveats
7564~~~~~~~~~~~~~~~~~~~~~~~~
7565
7566You need to be aware of the following before using ``devtool`` to create
7567NPM packages:
7568
7569- Of the two methods that you can use ``devtool`` to create NPM
7570 packages, the registry approach is slightly simpler. However, you
7571 might consider the project approach because you do not have to
7572 publish your module in the NPM registry
7573 (`npm-registry <https://docs.npmjs.com/misc/registry>`_), which
7574 is NPM's public registry.
7575
7576- Be familiar with
Andrew Geissler09209ee2020-12-13 08:44:15 -06007577 :doc:`devtool </ref-manual/devtool-reference>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007578
7579- The NPM host tools need the native ``nodejs-npm`` package, which is
7580 part of the OpenEmbedded environment. You need to get the package by
7581 cloning the https://github.com/openembedded/meta-openembedded
7582 repository out of GitHub. Be sure to add the path to your local copy
7583 to your ``bblayers.conf`` file.
7584
7585- ``devtool`` cannot detect native libraries in module dependencies.
7586 Consequently, you must manually add packages to your recipe.
7587
7588- While deploying NPM packages, ``devtool`` cannot determine which
7589 dependent packages are missing on the target (e.g. the node runtime
7590 ``nodejs``). Consequently, you need to find out what files are
7591 missing and be sure they are on the target.
7592
7593- Although you might not need NPM to run your node package, it is
7594 useful to have NPM on your target. The NPM package name is
7595 ``nodejs-npm``.
7596
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007597Using the Registry Modules Method
7598~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
7599
7600This section presents an example that uses the ``cute-files`` module,
7601which is a file browser web application.
7602
7603.. note::
7604
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007605 You must know the ``cute-files`` module version.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007606
7607The first thing you need to do is use ``devtool`` and the NPM fetcher to
Andrew Geisslerc926e172021-05-07 16:11:35 -05007608create the recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007609
7610 $ devtool add "npm://registry.npmjs.org;package=cute-files;version=1.0.2"
7611
7612The
7613``devtool add`` command runs ``recipetool create`` and uses the same
7614fetch URI to download each dependency and capture license details where
7615possible. The result is a generated recipe.
7616
Andrew Geissler615f2f12022-07-15 14:00:58 -05007617After running for quite a long time, in particular building the
7618``nodejs-native`` package, the command should end as follows::
7619
7620 INFO: Recipe /home/.../build/workspace/recipes/cute-files/cute-files_1.0.2.bb has been automatically created; further editing may be required to make it fully functional
7621
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007622The recipe file is fairly simple and contains every license that
7623``recipetool`` finds and includes the licenses in the recipe's
7624:term:`LIC_FILES_CHKSUM`
7625variables. You need to examine the variables and look for those with
7626"unknown" in the :term:`LICENSE`
7627field. You need to track down the license information for "unknown"
7628modules and manually add the information to the recipe.
7629
7630``recipetool`` creates a "shrinkwrap" file for your recipe. Shrinkwrap
7631files capture the version of all dependent modules. Many packages do not
Andrew Geissler615f2f12022-07-15 14:00:58 -05007632provide shrinkwrap files but ``recipetool`` will create a shrinkwrap file as it
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007633runs.
7634
7635.. note::
7636
7637 A package is created for each sub-module. This policy is the only
7638 practical way to have the licenses for all of the dependencies
7639 represented in the license manifest of the image.
7640
Andrew Geisslerc926e172021-05-07 16:11:35 -05007641The ``devtool edit-recipe`` command lets you take a look at the recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007642
7643 $ devtool edit-recipe cute-files
Andrew Geissler615f2f12022-07-15 14:00:58 -05007644 # Recipe created by recipetool
7645 # This is the basis of a recipe and may need further editing in order to be fully functional.
7646 # (Feel free to remove these comments when editing.)
7647
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007648 SUMMARY = "Turn any folder on your computer into a cute file browser, available on the local network."
Andrew Geissler615f2f12022-07-15 14:00:58 -05007649 # WARNING: the following LICENSE and LIC_FILES_CHKSUM values are best guesses - it is
7650 # your responsibility to verify that the values are complete and correct.
7651 #
7652 # NOTE: multiple licenses have been detected; they have been separated with &
7653 # in the LICENSE value for now since it is a reasonable assumption that all
7654 # of the licenses apply. If instead there is a choice between the multiple
7655 # licenses then you should change the value to separate the licenses with |
7656 # instead of &. If there is any doubt, check the accompanying documentation
7657 # to determine which situation is applicable.
7658
7659 SUMMARY = "Turn any folder on your computer into a cute file browser, available on the local network."
7660 LICENSE = "BSD-3-Clause & ISC & MIT"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007661 LIC_FILES_CHKSUM = "file://LICENSE;md5=71d98c0a1db42956787b1909c74a86ca \
Andrew Geissler615f2f12022-07-15 14:00:58 -05007662 file://node_modules/accepts/LICENSE;md5=bf1f9ad1e2e1d507aef4883fff7103de \
7663 file://node_modules/array-flatten/LICENSE;md5=44088ba57cb871a58add36ce51b8de08 \
7664 ...
7665 file://node_modules/cookie-signature/Readme.md;md5=57ae8b42de3dd0c1f22d5f4cf191e15a"
7666
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007667 SRC_URI = " \
7668 npm://registry.npmjs.org/;package=cute-files;version=${PV} \
7669 npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \
7670 "
Andrew Geissler615f2f12022-07-15 14:00:58 -05007671
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007672 S = "${WORKDIR}/npm"
Andrew Geissler615f2f12022-07-15 14:00:58 -05007673
Patrick Williams213cb262021-08-07 19:21:33 -05007674 inherit npm
Andrew Geissler615f2f12022-07-15 14:00:58 -05007675
Patrick Williams0ca19cc2021-08-16 14:03:13 -05007676 LICENSE:${PN} = "MIT"
7677 LICENSE:${PN}-accepts = "MIT"
7678 LICENSE:${PN}-array-flatten = "MIT"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007679 ...
Patrick Williams0ca19cc2021-08-16 14:03:13 -05007680 LICENSE:${PN}-vary = "MIT"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007681
William A. Kennington IIIac69b482021-06-02 12:28:27 -07007682Here are three key points in the previous example:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007683
7684- :term:`SRC_URI` uses the NPM
7685 scheme so that the NPM fetcher is used.
7686
7687- ``recipetool`` collects all the license information. If a
7688 sub-module's license is unavailable, the sub-module's name appears in
7689 the comments.
7690
7691- The ``inherit npm`` statement causes the
7692 :ref:`npm <ref-classes-npm>` class to package
7693 up all the modules.
7694
Andrew Geisslerc926e172021-05-07 16:11:35 -05007695You can run the following command to build the ``cute-files`` package::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007696
7697 $ devtool build cute-files
7698
7699Remember that ``nodejs`` must be installed on
7700the target before your package.
7701
7702Assuming 192.168.7.2 for the target's IP address, use the following
Andrew Geisslerc926e172021-05-07 16:11:35 -05007703command to deploy your package::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007704
7705 $ devtool deploy-target -s cute-files root@192.168.7.2
7706
7707Once the package is installed on the target, you can
Andrew Geissler615f2f12022-07-15 14:00:58 -05007708test the application to show the contents of any directory::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007709
7710 $ cd /usr/lib/node_modules/cute-files
Andrew Geissler615f2f12022-07-15 14:00:58 -05007711 $ cute-files
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007712
7713On a browser,
7714go to ``http://192.168.7.2:3000`` and you see the following:
7715
7716.. image:: figures/cute-files-npm-example.png
Andrew Geisslerd5838332022-05-27 11:33:10 -05007717 :width: 100%
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007718
7719You can find the recipe in ``workspace/recipes/cute-files``. You can use
7720the recipe in any layer you choose.
7721
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007722Using the NPM Projects Code Method
7723~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
7724
7725Although it is useful to package modules already in the NPM registry,
7726adding ``node.js`` projects under development is a more common developer
7727use case.
7728
7729This section covers the NPM projects code method, which is very similar
7730to the "registry" approach described in the previous section. In the NPM
7731projects method, you provide ``devtool`` with an URL that points to the
7732source files.
7733
7734Replicating the same example, (i.e. ``cute-files``) use the following
Andrew Geisslerc926e172021-05-07 16:11:35 -05007735command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007736
7737 $ devtool add https://github.com/martinaglv/cute-files.git
7738
Andrew Geissler615f2f12022-07-15 14:00:58 -05007739The recipe this command generates is very similar to the recipe created in
Andrew Geissler09036742021-06-25 14:25:14 -05007740the previous section. However, the :term:`SRC_URI` looks like the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007741
7742 SRC_URI = " \
Andrew Geissler615f2f12022-07-15 14:00:58 -05007743 git://github.com/martinaglv/cute-files.git;protocol=https;branch=master \
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007744 npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \
7745 "
7746
7747In this example,
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007748the main module is taken from the Git repository and dependencies are
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007749taken from the NPM registry. Other than those differences, the recipe is
7750basically the same between the two methods. You can build and deploy the
7751package exactly as described in the previous section that uses the
7752registry modules method.
7753
7754Adding custom metadata to packages
7755----------------------------------
7756
7757The variable
7758:term:`PACKAGE_ADD_METADATA`
7759can be used to add additional metadata to packages. This is reflected in
7760the package control/spec file. To take the ipk format for example, the
7761CONTROL file stored inside would contain the additional metadata as
7762additional lines.
7763
7764The variable can be used in multiple ways, including using suffixes to
7765set it for a specific package type and/or package. Note that the order
7766of precedence is the same as this list:
7767
Patrick Williams0ca19cc2021-08-16 14:03:13 -05007768- ``PACKAGE_ADD_METADATA_<PKGTYPE>:<PN>``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007769
7770- ``PACKAGE_ADD_METADATA_<PKGTYPE>``
7771
Patrick Williams0ca19cc2021-08-16 14:03:13 -05007772- ``PACKAGE_ADD_METADATA:<PN>``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007773
Andrew Geissler09036742021-06-25 14:25:14 -05007774- :term:`PACKAGE_ADD_METADATA`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007775
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007776`<PKGTYPE>` is a parameter and expected to be a distinct name of specific
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007777package type:
7778
7779- IPK for .ipk packages
7780
7781- DEB for .deb packages
7782
7783- RPM for .rpm packages
7784
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007785`<PN>` is a parameter and expected to be a package name.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007786
7787The variable can contain multiple [one-line] metadata fields separated
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007788by the literal sequence '\\n'. The separator can be redefined using the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007789variable flag ``separator``.
7790
William A. Kennington IIIac69b482021-06-02 12:28:27 -07007791Here is an example that adds two custom fields for ipk
Andrew Geisslerc926e172021-05-07 16:11:35 -05007792packages::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007793
7794 PACKAGE_ADD_METADATA_IPK = "Vendor: CustomIpk\nGroup:Applications/Spreadsheets"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007795
7796Efficiently Fetching Source Files During a Build
7797================================================
7798
7799The OpenEmbedded build system works with source files located through
7800the :term:`SRC_URI` variable. When
7801you build something using BitBake, a big part of the operation is
7802locating and downloading all the source tarballs. For images,
7803downloading all the source for various packages can take a significant
7804amount of time.
7805
7806This section shows you how you can use mirrors to speed up fetching
7807source files and how you can pre-fetch files all of which leads to more
7808efficient use of resources and time.
7809
7810Setting up Effective Mirrors
7811----------------------------
7812
7813A good deal that goes into a Yocto Project build is simply downloading
7814all of the source tarballs. Maybe you have been working with another
Andrew Geissler7e0e3c02022-02-25 20:34:39 +00007815build system for which you have built up a
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007816sizable directory of source tarballs. Or, perhaps someone else has such
7817a directory for which you have read access. If so, you can save time by
7818adding statements to your configuration file so that the build process
7819checks local directories first for existing tarballs before checking the
7820Internet.
7821
Andrew Geisslerc926e172021-05-07 16:11:35 -05007822Here is an efficient way to set it up in your ``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007823
7824 SOURCE_MIRROR_URL ?= "file:///home/you/your-download-dir/"
7825 INHERIT += "own-mirrors"
7826 BB_GENERATE_MIRROR_TARBALLS = "1"
7827 # BB_NO_NETWORK = "1"
7828
7829In the previous example, the
7830:term:`BB_GENERATE_MIRROR_TARBALLS`
7831variable causes the OpenEmbedded build system to generate tarballs of
7832the Git repositories and store them in the
7833:term:`DL_DIR` directory. Due to
7834performance reasons, generating and storing these tarballs is not the
7835build system's default behavior.
7836
7837You can also use the
7838:term:`PREMIRRORS` variable. For
7839an example, see the variable's glossary entry in the Yocto Project
7840Reference Manual.
7841
7842Getting Source Files and Suppressing the Build
7843----------------------------------------------
7844
7845Another technique you can use to ready yourself for a successive string
7846of build operations, is to pre-fetch all the source files without
7847actually starting a build. This technique lets you work through any
7848download issues and ultimately gathers all the source files into your
7849download directory :ref:`structure-build-downloads`,
7850which is located with :term:`DL_DIR`.
7851
7852Use the following BitBake command form to fetch all the necessary
Andrew Geisslerc926e172021-05-07 16:11:35 -05007853sources without starting the build::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007854
7855 $ bitbake target --runall=fetch
7856
7857This
7858variation of the BitBake command guarantees that you have all the
7859sources for that BitBake target should you disconnect from the Internet
7860and want to do the build later offline.
7861
7862Selecting an Initialization Manager
7863===================================
7864
7865By default, the Yocto Project uses SysVinit as the initialization
William A. Kennington IIIac69b482021-06-02 12:28:27 -07007866manager. However, there is also support for systemd, which is a full
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007867replacement for init with parallel starting of services, reduced shell
7868overhead and other features that are used by many distributions.
7869
7870Within the system, SysVinit treats system components as services. These
7871services are maintained as shell scripts stored in the ``/etc/init.d/``
7872directory. Services organize into different run levels. This
7873organization is maintained by putting links to the services in the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007874``/etc/rcN.d/`` directories, where `N/` is one of the following options:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007875"S", "0", "1", "2", "3", "4", "5", or "6".
7876
7877.. note::
7878
7879 Each runlevel has a dependency on the previous runlevel. This
7880 dependency allows the services to work properly.
7881
7882In comparison, systemd treats components as units. Using units is a
7883broader concept as compared to using a service. A unit includes several
7884different types of entities. Service is one of the types of entities.
7885The runlevel concept in SysVinit corresponds to the concept of a target
7886in systemd, where target is also a type of supported unit.
7887
7888In a SysVinit-based system, services load sequentially (i.e. one by one)
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007889during init and parallelization is not supported. With systemd, services
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007890start in parallel. Needless to say, the method can have an impact on
7891system startup performance.
7892
7893If you want to use SysVinit, you do not have to do anything. But, if you
7894want to use systemd, you must take some steps as described in the
7895following sections.
7896
7897Using systemd Exclusively
7898-------------------------
7899
Andrew Geisslerc926e172021-05-07 16:11:35 -05007900Set these variables in your distribution configuration file as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007901
Patrick Williams0ca19cc2021-08-16 14:03:13 -05007902 DISTRO_FEATURES:append = " systemd"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007903 VIRTUAL-RUNTIME_init_manager = "systemd"
7904
7905You can also prevent the SysVinit distribution feature from
Andrew Geisslerc926e172021-05-07 16:11:35 -05007906being automatically enabled as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007907
7908 DISTRO_FEATURES_BACKFILL_CONSIDERED = "sysvinit"
7909
7910Doing so removes any
7911redundant SysVinit scripts.
7912
7913To remove initscripts from your image altogether, set this variable
Andrew Geisslerc926e172021-05-07 16:11:35 -05007914also::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007915
7916 VIRTUAL-RUNTIME_initscripts = ""
7917
7918For information on the backfill variable, see
7919:term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`.
7920
7921Using systemd for the Main Image and Using SysVinit for the Rescue Image
7922------------------------------------------------------------------------
7923
Andrew Geisslerc926e172021-05-07 16:11:35 -05007924Set these variables in your distribution configuration file as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007925
Patrick Williams0ca19cc2021-08-16 14:03:13 -05007926 DISTRO_FEATURES:append = " systemd"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007927 VIRTUAL-RUNTIME_init_manager = "systemd"
7928
7929Doing so causes your main image to use the
7930``packagegroup-core-boot.bb`` recipe and systemd. The rescue/minimal
7931image cannot use this package group. However, it can install SysVinit
7932and the appropriate packages will have support for both systemd and
7933SysVinit.
7934
Andrew Geissler9aee5002022-03-30 16:27:02 +00007935Using systemd-journald without a traditional syslog daemon
7936----------------------------------------------------------
7937
7938Counter-intuitively, ``systemd-journald`` is not a syslog runtime or provider,
7939and the proper way to use systemd-journald as your sole logging mechanism is to
7940effectively disable syslog entirely by setting these variables in your distribution
7941configuration file::
7942
7943 VIRTUAL-RUNTIME_syslog = ""
7944 VIRTUAL-RUNTIME_base-utils-syslog = ""
7945
7946Doing so will prevent ``rsyslog`` / ``busybox-syslog`` from being pulled in by
7947default, leaving only ``journald``.
7948
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007949Selecting a Device Manager
7950==========================
7951
7952The Yocto Project provides multiple ways to manage the device manager
7953(``/dev``):
7954
Andrew Geissler5199d832021-09-24 16:47:35 -05007955- Persistent and Pre-Populated ``/dev``: For this case, the ``/dev``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007956 directory is persistent and the required device nodes are created
7957 during the build.
7958
7959- Use ``devtmpfs`` with a Device Manager: For this case, the ``/dev``
7960 directory is provided by the kernel as an in-memory file system and
7961 is automatically populated by the kernel at runtime. Additional
7962 configuration of device nodes is done in user space by a device
7963 manager like ``udev`` or ``busybox-mdev``.
7964
Andrew Geissler5199d832021-09-24 16:47:35 -05007965Using Persistent and Pre-Populated ``/dev``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007966--------------------------------------------
7967
7968To use the static method for device population, you need to set the
7969:term:`USE_DEVFS` variable to "0"
Andrew Geisslerc926e172021-05-07 16:11:35 -05007970as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007971
7972 USE_DEVFS = "0"
7973
7974The content of the resulting ``/dev`` directory is defined in a Device
7975Table file. The
7976:term:`IMAGE_DEVICE_TABLES`
7977variable defines the Device Table to use and should be set in the
7978machine or distro configuration file. Alternatively, you can set this
7979variable in your ``local.conf`` configuration file.
7980
Andrew Geissler09036742021-06-25 14:25:14 -05007981If you do not define the :term:`IMAGE_DEVICE_TABLES` variable, the default
Andrew Geisslerc926e172021-05-07 16:11:35 -05007982``device_table-minimal.txt`` is used::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007983
7984 IMAGE_DEVICE_TABLES = "device_table-mymachine.txt"
7985
7986The population is handled by the ``makedevs`` utility during image
7987creation:
7988
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007989Using ``devtmpfs`` and a Device Manager
7990---------------------------------------
7991
7992To use the dynamic method for device population, you need to use (or be
7993sure to set) the :term:`USE_DEVFS`
Andrew Geisslerc926e172021-05-07 16:11:35 -05007994variable to "1", which is the default::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007995
7996 USE_DEVFS = "1"
7997
7998With this
7999setting, the resulting ``/dev`` directory is populated by the kernel
8000using ``devtmpfs``. Make sure the corresponding kernel configuration
8001variable ``CONFIG_DEVTMPFS`` is set when building you build a Linux
8002kernel.
8003
8004All devices created by ``devtmpfs`` will be owned by ``root`` and have
8005permissions ``0600``.
8006
8007To have more control over the device nodes, you can use a device manager
8008like ``udev`` or ``busybox-mdev``. You choose the device manager by
8009defining the ``VIRTUAL-RUNTIME_dev_manager`` variable in your machine or
8010distro configuration file. Alternatively, you can set this variable in
Andrew Geisslerc926e172021-05-07 16:11:35 -05008011your ``local.conf`` configuration file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008012
8013 VIRTUAL-RUNTIME_dev_manager = "udev"
8014
8015 # Some alternative values
8016 # VIRTUAL-RUNTIME_dev_manager = "busybox-mdev"
8017 # VIRTUAL-RUNTIME_dev_manager = "systemd"
8018
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008019Using an External SCM
8020=====================
8021
8022If you're working on a recipe that pulls from an external Source Code
8023Manager (SCM), it is possible to have the OpenEmbedded build system
8024notice new recipe changes added to the SCM and then build the resulting
8025packages that depend on the new recipes by using the latest versions.
8026This only works for SCMs from which it is possible to get a sensible
8027revision number for changes. Currently, you can do this with Apache
8028Subversion (SVN), Git, and Bazaar (BZR) repositories.
8029
8030To enable this behavior, the :term:`PV` of
8031the recipe needs to reference
Andrew Geisslerc926e172021-05-07 16:11:35 -05008032:term:`SRCPV`. Here is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008033
8034 PV = "1.2.3+git${SRCPV}"
8035
8036Then, you can add the following to your
Andrew Geisslerc926e172021-05-07 16:11:35 -05008037``local.conf``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008038
Patrick Williams0ca19cc2021-08-16 14:03:13 -05008039 SRCREV:pn-PN = "${AUTOREV}"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008040
8041:term:`PN` is the name of the recipe for
8042which you want to enable automatic source revision updating.
8043
8044If you do not want to update your local configuration file, you can add
Andrew Geisslerc926e172021-05-07 16:11:35 -05008045the following directly to the recipe to finish enabling the feature::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008046
8047 SRCREV = "${AUTOREV}"
8048
8049The Yocto Project provides a distribution named ``poky-bleeding``, whose
Andrew Geisslerc926e172021-05-07 16:11:35 -05008050configuration file contains the line::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008051
8052 require conf/distro/include/poky-floating-revisions.inc
8053
8054This line pulls in the
Andrew Geisslerc926e172021-05-07 16:11:35 -05008055listed include file that contains numerous lines of exactly that form::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008056
Patrick Williams0ca19cc2021-08-16 14:03:13 -05008057 #SRCREV:pn-opkg-native ?= "${AUTOREV}"
8058 #SRCREV:pn-opkg-sdk ?= "${AUTOREV}"
8059 #SRCREV:pn-opkg ?= "${AUTOREV}"
8060 #SRCREV:pn-opkg-utils-native ?= "${AUTOREV}"
8061 #SRCREV:pn-opkg-utils ?= "${AUTOREV}"
8062 SRCREV:pn-gconf-dbus ?= "${AUTOREV}"
8063 SRCREV:pn-matchbox-common ?= "${AUTOREV}"
8064 SRCREV:pn-matchbox-config-gtk ?= "${AUTOREV}"
8065 SRCREV:pn-matchbox-desktop ?= "${AUTOREV}"
8066 SRCREV:pn-matchbox-keyboard ?= "${AUTOREV}"
8067 SRCREV:pn-matchbox-panel-2 ?= "${AUTOREV}"
8068 SRCREV:pn-matchbox-themes-extra ?= "${AUTOREV}"
8069 SRCREV:pn-matchbox-terminal ?= "${AUTOREV}"
8070 SRCREV:pn-matchbox-wm ?= "${AUTOREV}"
8071 SRCREV:pn-settings-daemon ?= "${AUTOREV}"
8072 SRCREV:pn-screenshot ?= "${AUTOREV}"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008073 . . .
8074
8075These lines allow you to
8076experiment with building a distribution that tracks the latest
8077development source for numerous packages.
8078
8079.. note::
8080
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008081 The ``poky-bleeding`` distribution is not tested on a regular basis. Keep
8082 this in mind if you use it.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008083
8084Creating a Read-Only Root Filesystem
8085====================================
8086
8087Suppose, for security reasons, you need to disable your target device's
8088root filesystem's write permissions (i.e. you need a read-only root
8089filesystem). Or, perhaps you are running the device's operating system
8090from a read-only storage device. For either case, you can customize your
8091image for that behavior.
8092
8093.. note::
8094
8095 Supporting a read-only root filesystem requires that the system and
8096 applications do not try to write to the root filesystem. You must
8097 configure all parts of the target system to write elsewhere, or to
8098 gracefully fail in the event of attempting to write to the root
8099 filesystem.
8100
8101Creating the Root Filesystem
8102----------------------------
8103
8104To create the read-only root filesystem, simply add the
8105"read-only-rootfs" feature to your image, normally in one of two ways.
8106The first way is to add the "read-only-rootfs" image feature in the
Andrew Geissler09036742021-06-25 14:25:14 -05008107image's recipe file via the :term:`IMAGE_FEATURES` variable::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008108
8109 IMAGE_FEATURES += "read-only-rootfs"
8110
8111As an alternative, you can add the same feature
8112from within your build directory's ``local.conf`` file with the
Andrew Geissler09036742021-06-25 14:25:14 -05008113associated :term:`EXTRA_IMAGE_FEATURES` variable, as in::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008114
8115 EXTRA_IMAGE_FEATURES = "read-only-rootfs"
8116
8117For more information on how to use these variables, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06008118":ref:`dev-manual/common-tasks:Customizing Images Using Custom \`\`IMAGE_FEATURES\`\` and \`\`EXTRA_IMAGE_FEATURES\`\``"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008119section. For information on the variables, see
8120:term:`IMAGE_FEATURES` and
8121:term:`EXTRA_IMAGE_FEATURES`.
8122
8123Post-Installation Scripts and Read-Only Root Filesystem
8124-------------------------------------------------------
8125
8126It is very important that you make sure all post-Installation
8127(``pkg_postinst``) scripts for packages that are installed into the
8128image can be run at the time when the root filesystem is created during
8129the build on the host system. These scripts cannot attempt to run during
Patrick Williams0ca19cc2021-08-16 14:03:13 -05008130the first boot on the target device. With the "read-only-rootfs" feature
8131enabled, the build system makes sure that all post-installation scripts
8132succeed at file system creation time. If any of these scripts
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008133still need to be run after the root filesystem is created, the build
8134immediately fails. These build-time checks ensure that the build fails
8135rather than the target device fails later during its initial boot
8136operation.
8137
8138Most of the common post-installation scripts generated by the build
8139system for the out-of-the-box Yocto Project are engineered so that they
8140can run during root filesystem creation (e.g. post-installation scripts
8141for caching fonts). However, if you create and add custom scripts, you
8142need to be sure they can be run during this file system creation.
8143
8144Here are some common problems that prevent post-installation scripts
8145from running during root filesystem creation:
8146
8147- *Not using $D in front of absolute paths:* The build system defines
8148 ``$``\ :term:`D` when the root
8149 filesystem is created. Furthermore, ``$D`` is blank when the script
8150 is run on the target device. This implies two purposes for ``$D``:
8151 ensuring paths are valid in both the host and target environments,
8152 and checking to determine which environment is being used as a method
8153 for taking appropriate actions.
8154
8155- *Attempting to run processes that are specific to or dependent on the
8156 target architecture:* You can work around these attempts by using
8157 native tools, which run on the host system, to accomplish the same
8158 tasks, or by alternatively running the processes under QEMU, which
8159 has the ``qemu_run_binary`` function. For more information, see the
8160 :ref:`qemu <ref-classes-qemu>` class.
8161
8162Areas With Write Access
8163-----------------------
8164
8165With the "read-only-rootfs" feature enabled, any attempt by the target
8166to write to the root filesystem at runtime fails. Consequently, you must
8167make sure that you configure processes and applications that attempt
8168these types of writes do so to directories with write access (e.g.
8169``/tmp`` or ``/var/run``).
8170
8171Maintaining Build Output Quality
8172================================
8173
8174Many factors can influence the quality of a build. For example, if you
8175upgrade a recipe to use a new version of an upstream software package or
8176you experiment with some new configuration options, subtle changes can
8177occur that you might not detect until later. Consider the case where
8178your recipe is using a newer version of an upstream package. In this
8179case, a new version of a piece of software might introduce an optional
8180dependency on another library, which is auto-detected. If that library
8181has already been built when the software is building, the software will
8182link to the built library and that library will be pulled into your
8183image along with the new software even if you did not want the library.
8184
8185The :ref:`buildhistory <ref-classes-buildhistory>`
William A. Kennington IIIac69b482021-06-02 12:28:27 -07008186class helps you maintain the quality of your build output. You
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008187can use the class to highlight unexpected and possibly unwanted changes
8188in the build output. When you enable build history, it records
8189information about the contents of each package and image and then
8190commits that information to a local Git repository where you can examine
8191the information.
8192
8193The remainder of this section describes the following:
8194
Andrew Geissler09209ee2020-12-13 08:44:15 -06008195- :ref:`How you can enable and disable build history <dev-manual/common-tasks:enabling and disabling build history>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008196
Andrew Geissler09209ee2020-12-13 08:44:15 -06008197- :ref:`How to understand what the build history contains <dev-manual/common-tasks:understanding what the build history contains>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008198
Andrew Geissler09209ee2020-12-13 08:44:15 -06008199- :ref:`How to limit the information used for build history <dev-manual/common-tasks:using build history to gather image information only>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008200
Andrew Geissler09209ee2020-12-13 08:44:15 -06008201- :ref:`How to examine the build history from both a command-line and web interface <dev-manual/common-tasks:examining build history information>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008202
8203Enabling and Disabling Build History
8204------------------------------------
8205
8206Build history is disabled by default. To enable it, add the following
Andrew Geissler09036742021-06-25 14:25:14 -05008207:term:`INHERIT` statement and set the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008208:term:`BUILDHISTORY_COMMIT`
8209variable to "1" at the end of your ``conf/local.conf`` file found in the
Andrew Geisslerc926e172021-05-07 16:11:35 -05008210:term:`Build Directory`::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008211
8212 INHERIT += "buildhistory"
8213 BUILDHISTORY_COMMIT = "1"
8214
8215Enabling build history as
8216previously described causes the OpenEmbedded build system to collect
8217build output information and commit it as a single commit to a local
Andrew Geissler09209ee2020-12-13 08:44:15 -06008218:ref:`overview-manual/development-environment:git` repository.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008219
8220.. note::
8221
8222 Enabling build history increases your build times slightly,
8223 particularly for images, and increases the amount of disk space used
8224 during the build.
8225
8226You can disable build history by removing the previous statements from
8227your ``conf/local.conf`` file.
8228
8229Understanding What the Build History Contains
8230---------------------------------------------
8231
8232Build history information is kept in
8233``${``\ :term:`TOPDIR`\ ``}/buildhistory``
8234in the Build Directory as defined by the
8235:term:`BUILDHISTORY_DIR`
William A. Kennington IIIac69b482021-06-02 12:28:27 -07008236variable. Here is an example abbreviated listing:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008237
8238.. image:: figures/buildhistory.png
8239 :align: center
Andrew Geisslerd5838332022-05-27 11:33:10 -05008240 :width: 50%
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008241
William A. Kennington IIIac69b482021-06-02 12:28:27 -07008242At the top level, there is a ``metadata-revs`` file that lists the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008243revisions of the repositories for the enabled layers when the build was
8244produced. The rest of the data splits into separate ``packages``,
8245``images`` and ``sdk`` directories, the contents of which are described
8246as follows.
8247
8248Build History Package Information
8249~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8250
8251The history for each package contains a text file that has name-value
8252pairs with information about the package. For example,
8253``buildhistory/packages/i586-poky-linux/busybox/busybox/latest``
8254contains the following:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008255
8256.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008257
8258 PV = 1.22.1
8259 PR = r32
8260 RPROVIDES =
8261 RDEPENDS = glibc (>= 2.20) update-alternatives-opkg
8262 RRECOMMENDS = busybox-syslog busybox-udhcpc update-rc.d
8263 PKGSIZE = 540168
8264 FILES = /usr/bin/* /usr/sbin/* /usr/lib/busybox/* /usr/lib/lib*.so.* \
8265 /etc /com /var /bin/* /sbin/* /lib/*.so.* /lib/udev/rules.d \
8266 /usr/lib/udev/rules.d /usr/share/busybox /usr/lib/busybox/* \
8267 /usr/share/pixmaps /usr/share/applications /usr/share/idl \
8268 /usr/share/omf /usr/share/sounds /usr/lib/bonobo/servers
8269 FILELIST = /bin/busybox /bin/busybox.nosuid /bin/busybox.suid /bin/sh \
8270 /etc/busybox.links.nosuid /etc/busybox.links.suid
8271
8272Most of these
8273name-value pairs correspond to variables used to produce the package.
8274The exceptions are ``FILELIST``, which is the actual list of files in
8275the package, and ``PKGSIZE``, which is the total size of files in the
8276package in bytes.
8277
William A. Kennington IIIac69b482021-06-02 12:28:27 -07008278There is also a file that corresponds to the recipe from which the package
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008279came (e.g. ``buildhistory/packages/i586-poky-linux/busybox/latest``):
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008280
8281.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008282
8283 PV = 1.22.1
8284 PR = r32
8285 DEPENDS = initscripts kern-tools-native update-rc.d-native \
8286 virtual/i586-poky-linux-compilerlibs virtual/i586-poky-linux-gcc \
8287 virtual/libc virtual/update-alternatives
8288 PACKAGES = busybox-ptest busybox-httpd busybox-udhcpd busybox-udhcpc \
8289 busybox-syslog busybox-mdev busybox-hwclock busybox-dbg \
8290 busybox-staticdev busybox-dev busybox-doc busybox-locale busybox
8291
8292Finally, for those recipes fetched from a version control system (e.g.,
William A. Kennington IIIac69b482021-06-02 12:28:27 -07008293Git), there is a file that lists source revisions that are specified in
8294the recipe and the actual revisions used during the build. Listed
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008295and actual revisions might differ when
8296:term:`SRCREV` is set to
8297${:term:`AUTOREV`}. Here is an
8298example assuming
Andrew Geisslerc926e172021-05-07 16:11:35 -05008299``buildhistory/packages/qemux86-poky-linux/linux-yocto/latest_srcrev``)::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008300
8301 # SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
8302 SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
8303 # SRCREV_meta = "a227f20eff056e511d504b2e490f3774ab260d6f"
8304 SRCREV_meta ="a227f20eff056e511d504b2e490f3774ab260d6f"
8305
8306You can use the
8307``buildhistory-collect-srcrevs`` command with the ``-a`` option to
Andrew Geissler09036742021-06-25 14:25:14 -05008308collect the stored :term:`SRCREV` values from build history and report them
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008309in a format suitable for use in global configuration (e.g.,
8310``local.conf`` or a distro include file) to override floating
Andrew Geissler09036742021-06-25 14:25:14 -05008311:term:`AUTOREV` values to a fixed set of revisions. Here is some example
Andrew Geisslerc926e172021-05-07 16:11:35 -05008312output from this command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008313
8314 $ buildhistory-collect-srcrevs -a
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008315 # all-poky-linux
Andrew Geissler9aee5002022-03-30 16:27:02 +00008316 SRCREV:pn-ca-certificates = "07de54fdcc5806bde549e1edf60738c6bccf50e8"
8317 SRCREV:pn-update-rc.d = "8636cf478d426b568c1be11dbd9346f67e03adac"
8318 # core2-64-poky-linux
8319 SRCREV:pn-binutils = "87d4632d36323091e731eb07b8aa65f90293da66"
8320 SRCREV:pn-btrfs-tools = "8ad326b2f28c044cb6ed9016d7c3285e23b673c8"
8321 SRCREV_bzip2-tests:pn-bzip2 = "f9061c030a25de5b6829e1abf373057309c734c0"
8322 SRCREV:pn-e2fsprogs = "02540dedd3ddc52c6ae8aaa8a95ce75c3f8be1c0"
8323 SRCREV:pn-file = "504206e53a89fd6eed71aeaf878aa3512418eab1"
8324 SRCREV_glibc:pn-glibc = "24962427071fa532c3c48c918e9d64d719cc8a6c"
8325 SRCREV:pn-gnome-desktop-testing = "e346cd4ed2e2102c9b195b614f3c642d23f5f6e7"
8326 SRCREV:pn-init-system-helpers = "dbd9197569c0935029acd5c9b02b84c68fd937ee"
8327 SRCREV:pn-kmod = "b6ecfc916a17eab8f93be5b09f4e4f845aabd3d1"
8328 SRCREV:pn-libnsl2 = "82245c0c58add79a8e34ab0917358217a70e5100"
8329 SRCREV:pn-libseccomp = "57357d2741a3b3d3e8425889a6b79a130e0fa2f3"
8330 SRCREV:pn-libxcrypt = "50cf2b6dd4fdf04309445f2eec8de7051d953abf"
8331 SRCREV:pn-ncurses = "51d0fd9cc3edb975f04224f29f777f8f448e8ced"
8332 SRCREV:pn-procps = "19a508ea121c0c4ac6d0224575a036de745eaaf8"
8333 SRCREV:pn-psmisc = "5fab6b7ab385080f1db725d6803136ec1841a15f"
8334 SRCREV:pn-ptest-runner = "bcb82804daa8f725b6add259dcef2067e61a75aa"
8335 SRCREV:pn-shared-mime-info = "18e558fa1c8b90b86757ade09a4ba4d6a6cf8f70"
8336 SRCREV:pn-zstd = "e47e674cd09583ff0503f0f6defd6d23d8b718d3"
8337 # qemux86_64-poky-linux
8338 SRCREV_machine:pn-linux-yocto = "20301aeb1a64164b72bc72af58802b315e025c9c"
8339 SRCREV_meta:pn-linux-yocto = "2d38a472b21ae343707c8bd64ac68a9eaca066a0"
8340 # x86_64-linux
8341 SRCREV:pn-binutils-cross-x86_64 = "87d4632d36323091e731eb07b8aa65f90293da66"
8342 SRCREV_glibc:pn-cross-localedef-native = "24962427071fa532c3c48c918e9d64d719cc8a6c"
8343 SRCREV_localedef:pn-cross-localedef-native = "794da69788cbf9bf57b59a852f9f11307663fa87"
8344 SRCREV:pn-debianutils-native = "de14223e5bffe15e374a441302c528ffc1cbed57"
8345 SRCREV:pn-libmodulemd-native = "ee80309bc766d781a144e6879419b29f444d94eb"
8346 SRCREV:pn-virglrenderer-native = "363915595e05fb252e70d6514be2f0c0b5ca312b"
8347 SRCREV:pn-zstd-native = "e47e674cd09583ff0503f0f6defd6d23d8b718d3"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008348
8349.. note::
8350
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008351 Here are some notes on using the ``buildhistory-collect-srcrevs`` command:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008352
Andrew Geissler09036742021-06-25 14:25:14 -05008353 - By default, only values where the :term:`SRCREV` was not hardcoded
Andrew Geissler5f350902021-07-23 13:09:54 -04008354 (usually when :term:`AUTOREV` is used) are reported. Use the ``-a``
8355 option to see all :term:`SRCREV` values.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008356
8357 - The output statements might not have any effect if overrides are
8358 applied elsewhere in the build system configuration. Use the
8359 ``-f`` option to add the ``forcevariable`` override to each output
8360 line if you need to work around this restriction.
8361
8362 - The script does apply special handling when building for multiple
8363 machines. However, the script does place a comment before each set
8364 of values that specifies which triplet to which they belong as
8365 previously shown (e.g., ``i586-poky-linux``).
8366
8367Build History Image Information
8368~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8369
8370The files produced for each image are as follows:
8371
8372- ``image-files:`` A directory containing selected files from the root
8373 filesystem. The files are defined by
8374 :term:`BUILDHISTORY_IMAGE_FILES`.
8375
8376- ``build-id.txt:`` Human-readable information about the build
8377 configuration and metadata source revisions. This file contains the
8378 full build header as printed by BitBake.
8379
8380- ``*.dot:`` Dependency graphs for the image that are compatible with
8381 ``graphviz``.
8382
8383- ``files-in-image.txt:`` A list of files in the image with
8384 permissions, owner, group, size, and symlink information.
8385
8386- ``image-info.txt:`` A text file containing name-value pairs with
8387 information about the image. See the following listing example for
8388 more information.
8389
8390- ``installed-package-names.txt:`` A list of installed packages by name
8391 only.
8392
8393- ``installed-package-sizes.txt:`` A list of installed packages ordered
8394 by size.
8395
8396- ``installed-packages.txt:`` A list of installed packages with full
8397 package filenames.
8398
8399.. note::
8400
8401 Installed package information is able to be gathered and produced
8402 even if package management is disabled for the final image.
8403
8404Here is an example of ``image-info.txt``:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008405
8406.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008407
8408 DISTRO = poky
Andrew Geissler9aee5002022-03-30 16:27:02 +00008409 DISTRO_VERSION = 3.4+snapshot-a0245d7be08f3d24ea1875e9f8872aa6bbff93be
8410 USER_CLASSES = buildstats
8411 IMAGE_CLASSES = qemuboot qemuboot license_image
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008412 IMAGE_FEATURES = debug-tweaks
8413 IMAGE_LINGUAS =
Andrew Geissler9aee5002022-03-30 16:27:02 +00008414 IMAGE_INSTALL = packagegroup-core-boot speex speexdsp
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008415 BAD_RECOMMENDATIONS =
8416 NO_RECOMMENDATIONS =
8417 PACKAGE_EXCLUDE =
Andrew Geissler9aee5002022-03-30 16:27:02 +00008418 ROOTFS_POSTPROCESS_COMMAND = write_package_manifest; license_create_manifest; cve_check_write_rootfs_manifest; ssh_allow_empty_password; ssh_allow_root_login; postinst_enable_logging; rootfs_update_timestamp; write_image_test_data; empty_var_volatile; sort_passwd; rootfs_reproducible;
8419 IMAGE_POSTPROCESS_COMMAND = buildhistory_get_imageinfo ;
8420 IMAGESIZE = 9265
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008421
8422Other than ``IMAGESIZE``,
8423which is the total size of the files in the image in Kbytes, the
8424name-value pairs are variables that may have influenced the content of
8425the image. This information is often useful when you are trying to
8426determine why a change in the package or file listings has occurred.
8427
8428Using Build History to Gather Image Information Only
8429~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8430
8431As you can see, build history produces image information, including
8432dependency graphs, so you can see why something was pulled into the
8433image. If you are just interested in this information and not interested
8434in collecting specific package or SDK information, you can enable
8435writing only image information without any history by adding the
8436following to your ``conf/local.conf`` file found in the
Andrew Geisslerc926e172021-05-07 16:11:35 -05008437:term:`Build Directory`::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008438
8439 INHERIT += "buildhistory"
8440 BUILDHISTORY_COMMIT = "0"
8441 BUILDHISTORY_FEATURES = "image"
8442
8443Here, you set the
8444:term:`BUILDHISTORY_FEATURES`
8445variable to use the image feature only.
8446
8447Build History SDK Information
8448~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8449
8450Build history collects similar information on the contents of SDKs (e.g.
8451``bitbake -c populate_sdk imagename``) as compared to information it
8452collects for images. Furthermore, this information differs depending on
8453whether an extensible or standard SDK is being produced.
8454
8455The following list shows the files produced for SDKs:
8456
8457- ``files-in-sdk.txt:`` A list of files in the SDK with permissions,
8458 owner, group, size, and symlink information. This list includes both
8459 the host and target parts of the SDK.
8460
8461- ``sdk-info.txt:`` A text file containing name-value pairs with
8462 information about the SDK. See the following listing example for more
8463 information.
8464
8465- ``sstate-task-sizes.txt:`` A text file containing name-value pairs
8466 with information about task group sizes (e.g. ``do_populate_sysroot``
8467 tasks have a total size). The ``sstate-task-sizes.txt`` file exists
8468 only when an extensible SDK is created.
8469
8470- ``sstate-package-sizes.txt:`` A text file containing name-value pairs
8471 with information for the shared-state packages and sizes in the SDK.
8472 The ``sstate-package-sizes.txt`` file exists only when an extensible
8473 SDK is created.
8474
8475- ``sdk-files:`` A folder that contains copies of the files mentioned
8476 in ``BUILDHISTORY_SDK_FILES`` if the files are present in the output.
8477 Additionally, the default value of ``BUILDHISTORY_SDK_FILES`` is
8478 specific to the extensible SDK although you can set it differently if
8479 you would like to pull in specific files from the standard SDK.
8480
8481 The default files are ``conf/local.conf``, ``conf/bblayers.conf``,
8482 ``conf/auto.conf``, ``conf/locked-sigs.inc``, and
8483 ``conf/devtool.conf``. Thus, for an extensible SDK, these files get
8484 copied into the ``sdk-files`` directory.
8485
8486- The following information appears under each of the ``host`` and
8487 ``target`` directories for the portions of the SDK that run on the
8488 host and on the target, respectively:
8489
8490 .. note::
8491
8492 The following files for the most part are empty when producing an
8493 extensible SDK because this type of SDK is not constructed from
8494 packages as is the standard SDK.
8495
8496 - ``depends.dot:`` Dependency graph for the SDK that is compatible
8497 with ``graphviz``.
8498
8499 - ``installed-package-names.txt:`` A list of installed packages by
8500 name only.
8501
8502 - ``installed-package-sizes.txt:`` A list of installed packages
8503 ordered by size.
8504
8505 - ``installed-packages.txt:`` A list of installed packages with full
8506 package filenames.
8507
8508Here is an example of ``sdk-info.txt``:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008509
8510.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008511
8512 DISTRO = poky
8513 DISTRO_VERSION = 1.3+snapshot-20130327
8514 SDK_NAME = poky-glibc-i686-arm
8515 SDK_VERSION = 1.3+snapshot
8516 SDKMACHINE =
8517 SDKIMAGE_FEATURES = dev-pkgs dbg-pkgs
8518 BAD_RECOMMENDATIONS =
8519 SDKSIZE = 352712
8520
8521Other than ``SDKSIZE``, which is
8522the total size of the files in the SDK in Kbytes, the name-value pairs
8523are variables that might have influenced the content of the SDK. This
8524information is often useful when you are trying to determine why a
8525change in the package or file listings has occurred.
8526
8527Examining Build History Information
8528~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8529
8530You can examine build history output from the command line or from a web
8531interface.
8532
8533To see any changes that have occurred (assuming you have
8534:term:`BUILDHISTORY_COMMIT` = "1"),
8535you can simply use any Git command that allows you to view the history
Andrew Geisslerc926e172021-05-07 16:11:35 -05008536of a repository. Here is one method::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008537
8538 $ git log -p
8539
8540You need to realize,
8541however, that this method does show changes that are not significant
8542(e.g. a package's size changing by a few bytes).
8543
William A. Kennington IIIac69b482021-06-02 12:28:27 -07008544There is a command-line tool called ``buildhistory-diff``, though,
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008545that queries the Git repository and prints just the differences that
Andrew Geisslerc926e172021-05-07 16:11:35 -05008546might be significant in human-readable form. Here is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008547
Andrew Geissler95ac1b82021-03-31 14:34:31 -05008548 $ poky/poky/scripts/buildhistory-diff . HEAD^
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008549 Changes to images/qemux86_64/glibc/core-image-minimal (files-in-image.txt):
8550 /etc/anotherpkg.conf was added
8551 /sbin/anotherpkg was added
8552 * (installed-package-names.txt):
8553 * anotherpkg was added
8554 Changes to images/qemux86_64/glibc/core-image-minimal (installed-package-names.txt):
8555 anotherpkg was added
8556 packages/qemux86_64-poky-linux/v86d: PACKAGES: added "v86d-extras"
8557 * PR changed from "r0" to "r1"
8558 * PV changed from "0.1.10" to "0.1.12"
8559 packages/qemux86_64-poky-linux/v86d/v86d: PKGSIZE changed from 110579 to 144381 (+30%)
8560 * PR changed from "r0" to "r1"
8561 * PV changed from "0.1.10" to "0.1.12"
8562
8563.. note::
8564
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008565 The ``buildhistory-diff`` tool requires the ``GitPython``
Andrew Geisslerc926e172021-05-07 16:11:35 -05008566 package. Be sure to install it using Pip3 as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008567
8568 $ pip3 install GitPython --user
8569
8570
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008571 Alternatively, you can install ``python3-git`` using the appropriate
Andrew Geisslereff27472021-10-29 15:35:00 -05008572 distribution package manager (e.g. ``apt``, ``dnf``, or ``zipper``).
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008573
8574To see changes to the build history using a web interface, follow the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008575instruction in the ``README`` file
Andrew Geissler09209ee2020-12-13 08:44:15 -06008576:yocto_git:`here </buildhistory-web/>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008577
8578Here is a sample screenshot of the interface:
8579
8580.. image:: figures/buildhistory-web.png
Andrew Geisslerd5838332022-05-27 11:33:10 -05008581 :width: 100%
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008582
8583Performing Automated Runtime Testing
8584====================================
8585
8586The OpenEmbedded build system makes available a series of automated
8587tests for images to verify runtime functionality. You can run these
8588tests on either QEMU or actual target hardware. Tests are written in
8589Python making use of the ``unittest`` module, and the majority of them
8590run commands on the target system over SSH. This section describes how
8591you set up the environment to use these tests, run available tests, and
8592write and add your own tests.
8593
8594For information on the test and QA infrastructure available within the
Andrew Geissler09209ee2020-12-13 08:44:15 -06008595Yocto Project, see the ":ref:`ref-manual/release-process:testing and quality assurance`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008596section in the Yocto Project Reference Manual.
8597
8598Enabling Tests
8599--------------
8600
8601Depending on whether you are planning to run tests using QEMU or on the
8602hardware, you have to take different steps to enable the tests. See the
8603following subsections for information on how to enable both types of
8604tests.
8605
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008606Enabling Runtime Tests on QEMU
8607~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8608
8609In order to run tests, you need to do the following:
8610
8611- *Set up to avoid interaction with sudo for networking:* To
8612 accomplish this, you must do one of the following:
8613
8614 - Add ``NOPASSWD`` for your user in ``/etc/sudoers`` either for all
8615 commands or just for ``runqemu-ifup``. You must provide the full
8616 path as that can change if you are using multiple clones of the
8617 source repository.
8618
8619 .. note::
8620
8621 On some distributions, you also need to comment out "Defaults
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008622 requiretty" in ``/etc/sudoers``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008623
8624 - Manually configure a tap interface for your system.
8625
8626 - Run as root the script in ``scripts/runqemu-gen-tapdevs``, which
8627 should generate a list of tap devices. This is the option
8628 typically chosen for Autobuilder-type environments.
8629
8630 .. note::
8631
8632 - Be sure to use an absolute path when calling this script
8633 with sudo.
8634
8635 - The package recipe ``qemu-helper-native`` is required to run
Andrew Geisslerc926e172021-05-07 16:11:35 -05008636 this script. Build the package using the following command::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008637
8638 $ bitbake qemu-helper-native
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008639
8640- *Set the DISPLAY variable:* You need to set this variable so that
8641 you have an X server available (e.g. start ``vncserver`` for a
8642 headless machine).
8643
8644- *Be sure your host's firewall accepts incoming connections from
8645 192.168.7.0/24:* Some of the tests (in particular DNF tests) start an
8646 HTTP server on a random high number port, which is used to serve
8647 files to the target. The DNF module serves
8648 ``${WORKDIR}/oe-rootfs-repo`` so it can run DNF channel commands.
8649 That means your host's firewall must accept incoming connections from
8650 192.168.7.0/24, which is the default IP range used for tap devices by
8651 ``runqemu``.
8652
8653- *Be sure your host has the correct packages installed:* Depending
8654 your host's distribution, you need to have the following packages
8655 installed:
8656
8657 - Ubuntu and Debian: ``sysstat`` and ``iproute2``
8658
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05008659 - openSUSE: ``sysstat`` and ``iproute2``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008660
8661 - Fedora: ``sysstat`` and ``iproute``
8662
8663 - CentOS: ``sysstat`` and ``iproute``
8664
8665Once you start running the tests, the following happens:
8666
86671. A copy of the root filesystem is written to ``${WORKDIR}/testimage``.
8668
86692. The image is booted under QEMU using the standard ``runqemu`` script.
8670
86713. A default timeout of 500 seconds occurs to allow for the boot process
8672 to reach the login prompt. You can change the timeout period by
8673 setting
8674 :term:`TEST_QEMUBOOT_TIMEOUT`
8675 in the ``local.conf`` file.
8676
86774. Once the boot process is reached and the login prompt appears, the
8678 tests run. The full boot log is written to
8679 ``${WORKDIR}/testimage/qemu_boot_log``.
8680
Andrew Geissler09036742021-06-25 14:25:14 -050086815. Each test module loads in the order found in :term:`TEST_SUITES`. You can
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008682 find the full output of the commands run over SSH in
8683 ``${WORKDIR}/testimgage/ssh_target_log``.
8684
86856. If no failures occur, the task running the tests ends successfully.
8686 You can find the output from the ``unittest`` in the task log at
8687 ``${WORKDIR}/temp/log.do_testimage``.
8688
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008689Enabling Runtime Tests on Hardware
8690~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8691
8692The OpenEmbedded build system can run tests on real hardware, and for
8693certain devices it can also deploy the image to be tested onto the
8694device beforehand.
8695
Andrew Geissler595f6302022-01-24 19:11:47 +00008696For automated deployment, a "controller image" is installed onto the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008697hardware once as part of setup. Then, each time tests are to be run, the
8698following occurs:
8699
Andrew Geissler595f6302022-01-24 19:11:47 +000087001. The controller image is booted into and used to write the image to be
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008701 tested to a second partition.
8702
87032. The device is then rebooted using an external script that you need to
8704 provide.
8705
87063. The device boots into the image to be tested.
8707
8708When running tests (independent of whether the image has been deployed
8709automatically or not), the device is expected to be connected to a
8710network on a pre-determined IP address. You can either use static IP
8711addresses written into the image, or set the image to use DHCP and have
8712your DHCP server on the test network assign a known IP address based on
8713the MAC address of the device.
8714
Andrew Geissler09036742021-06-25 14:25:14 -05008715In order to run tests on hardware, you need to set :term:`TEST_TARGET` to an
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008716appropriate value. For QEMU, you do not have to change anything, the
8717default value is "qemu". For running tests on hardware, the following
William A. Kennington IIIac69b482021-06-02 12:28:27 -07008718options are available:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008719
8720- *"simpleremote":* Choose "simpleremote" if you are going to run tests
8721 on a target system that is already running the image to be tested and
8722 is available on the network. You can use "simpleremote" in
8723 conjunction with either real hardware or an image running within a
8724 separately started QEMU or any other virtual machine manager.
8725
8726- *"SystemdbootTarget":* Choose "SystemdbootTarget" if your hardware is
8727 an EFI-based machine with ``systemd-boot`` as bootloader and
8728 ``core-image-testmaster`` (or something similar) is installed. Also,
8729 your hardware under test must be in a DHCP-enabled network that gives
8730 it the same IP address for each reboot.
8731
8732 If you choose "SystemdbootTarget", there are additional requirements
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05008733 and considerations. See the
8734 ":ref:`dev-manual/common-tasks:selecting systemdboottarget`" section, which
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008735 follows, for more information.
8736
8737- *"BeagleBoneTarget":* Choose "BeagleBoneTarget" if you are deploying
8738 images and running tests on the BeagleBone "Black" or original
8739 "White" hardware. For information on how to use these tests, see the
8740 comments at the top of the BeagleBoneTarget
8741 ``meta-yocto-bsp/lib/oeqa/controllers/beaglebonetarget.py`` file.
8742
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008743- *"EdgeRouterTarget":* Choose "EdgeRouterTarget" if you are deploying
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008744 images and running tests on the Ubiquiti Networks EdgeRouter Lite.
8745 For information on how to use these tests, see the comments at the
8746 top of the EdgeRouterTarget
8747 ``meta-yocto-bsp/lib/oeqa/controllers/edgeroutertarget.py`` file.
8748
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008749- *"GrubTarget":* Choose "GrubTarget" if you are deploying images and running
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008750 tests on any generic PC that boots using GRUB. For information on how
8751 to use these tests, see the comments at the top of the GrubTarget
8752 ``meta-yocto-bsp/lib/oeqa/controllers/grubtarget.py`` file.
8753
8754- *"your-target":* Create your own custom target if you want to run
8755 tests when you are deploying images and running tests on a custom
8756 machine within your BSP layer. To do this, you need to add a Python
8757 unit that defines the target class under ``lib/oeqa/controllers/``
8758 within your layer. You must also provide an empty ``__init__.py``.
8759 For examples, see files in ``meta-yocto-bsp/lib/oeqa/controllers/``.
8760
8761Selecting SystemdbootTarget
8762~~~~~~~~~~~~~~~~~~~~~~~~~~~
8763
Andrew Geissler09036742021-06-25 14:25:14 -05008764If you did not set :term:`TEST_TARGET` to "SystemdbootTarget", then you do
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008765not need any information in this section. You can skip down to the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05008766":ref:`dev-manual/common-tasks:running tests`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008767
Andrew Geissler09036742021-06-25 14:25:14 -05008768If you did set :term:`TEST_TARGET` to "SystemdbootTarget", you also need to
Andrew Geissler595f6302022-01-24 19:11:47 +00008769perform a one-time setup of your controller image by doing the following:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008770
Andrew Geissler09036742021-06-25 14:25:14 -050087711. *Set EFI_PROVIDER:* Be sure that :term:`EFI_PROVIDER` is as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008772
8773 EFI_PROVIDER = "systemd-boot"
8774
Andrew Geissler595f6302022-01-24 19:11:47 +000087752. *Build the controller image:* Build the ``core-image-testmaster`` image.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008776 The ``core-image-testmaster`` recipe is provided as an example for a
Andrew Geissler595f6302022-01-24 19:11:47 +00008777 "controller" image and you can customize the image recipe as you would
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008778 any other recipe.
8779
8780 Here are the image recipe requirements:
8781
8782 - Inherits ``core-image`` so that kernel modules are installed.
8783
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05008784 - Installs normal linux utilities not BusyBox ones (e.g. ``bash``,
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008785 ``coreutils``, ``tar``, ``gzip``, and ``kmod``).
8786
8787 - Uses a custom Initial RAM Disk (initramfs) image with a custom
8788 installer. A normal image that you can install usually creates a
Andrew Geissler595f6302022-01-24 19:11:47 +00008789 single root filesystem partition. This image uses another installer that
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008790 creates a specific partition layout. Not all Board Support
8791 Packages (BSPs) can use an installer. For such cases, you need to
8792 manually create the following partition layout on the target:
8793
8794 - First partition mounted under ``/boot``, labeled "boot".
8795
Andrew Geissler595f6302022-01-24 19:11:47 +00008796 - The main root filesystem partition where this image gets installed,
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008797 which is mounted under ``/``.
8798
8799 - Another partition labeled "testrootfs" where test images get
8800 deployed.
8801
88023. *Install image:* Install the image that you just built on the target
8803 system.
8804
Andrew Geissler09036742021-06-25 14:25:14 -05008805The final thing you need to do when setting :term:`TEST_TARGET` to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008806"SystemdbootTarget" is to set up the test image:
8807
88081. *Set up your local.conf file:* Make sure you have the following
Andrew Geisslerc926e172021-05-07 16:11:35 -05008809 statements in your ``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008810
8811 IMAGE_FSTYPES += "tar.gz"
8812 INHERIT += "testimage"
8813 TEST_TARGET = "SystemdbootTarget"
8814 TEST_TARGET_IP = "192.168.2.3"
8815
Andrew Geisslerc926e172021-05-07 16:11:35 -050088162. *Build your test image:* Use BitBake to build the image::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008817
8818 $ bitbake core-image-sato
8819
8820Power Control
8821~~~~~~~~~~~~~
8822
8823For most hardware targets other than "simpleremote", you can control
8824power:
8825
Andrew Geissler09036742021-06-25 14:25:14 -05008826- You can use :term:`TEST_POWERCONTROL_CMD` together with
8827 :term:`TEST_POWERCONTROL_EXTRA_ARGS` as a command that runs on the host
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008828 and does power cycling. The test code passes one argument to that
8829 command: off, on or cycle (off then on). Here is an example that
Andrew Geisslerc926e172021-05-07 16:11:35 -05008830 could appear in your ``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008831
8832 TEST_POWERCONTROL_CMD = "powercontrol.exp test 10.11.12.1 nuc1"
8833
8834 In this example, the expect
8835 script does the following:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008836
8837 .. code-block:: shell
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008838
8839 ssh test@10.11.12.1 "pyctl nuc1 arg"
8840
8841 It then runs a Python script that controls power for a label called
8842 ``nuc1``.
8843
8844 .. note::
8845
Andrew Geissler09036742021-06-25 14:25:14 -05008846 You need to customize :term:`TEST_POWERCONTROL_CMD` and
8847 :term:`TEST_POWERCONTROL_EXTRA_ARGS` for your own setup. The one requirement
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008848 is that it accepts "on", "off", and "cycle" as the last argument.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008849
8850- When no command is defined, it connects to the device over SSH and
8851 uses the classic reboot command to reboot the device. Classic reboot
8852 is fine as long as the machine actually reboots (i.e. the SSH test
8853 has not failed). It is useful for scenarios where you have a simple
8854 setup, typically with a single board, and where some manual
8855 interaction is okay from time to time.
8856
8857If you have no hardware to automatically perform power control but still
8858wish to experiment with automated hardware testing, you can use the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008859``dialog-power-control`` script that shows a dialog prompting you to perform
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008860the required power action. This script requires either KDialog or Zenity
8861to be installed. To use this script, set the
8862:term:`TEST_POWERCONTROL_CMD`
Andrew Geisslerc926e172021-05-07 16:11:35 -05008863variable as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008864
8865 TEST_POWERCONTROL_CMD = "${COREBASE}/scripts/contrib/dialog-power-control"
8866
8867Serial Console Connection
8868~~~~~~~~~~~~~~~~~~~~~~~~~
8869
8870For test target classes requiring a serial console to interact with the
8871bootloader (e.g. BeagleBoneTarget, EdgeRouterTarget, and GrubTarget),
8872you need to specify a command to use to connect to the serial console of
8873the target machine by using the
8874:term:`TEST_SERIALCONTROL_CMD`
8875variable and optionally the
8876:term:`TEST_SERIALCONTROL_EXTRA_ARGS`
8877variable.
8878
8879These cases could be a serial terminal program if the machine is
8880connected to a local serial port, or a ``telnet`` or ``ssh`` command
8881connecting to a remote console server. Regardless of the case, the
8882command simply needs to connect to the serial console and forward that
8883connection to standard input and output as any normal terminal program
8884does. For example, to use the picocom terminal program on serial device
Andrew Geisslerc926e172021-05-07 16:11:35 -05008885``/dev/ttyUSB0`` at 115200bps, you would set the variable as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008886
8887 TEST_SERIALCONTROL_CMD = "picocom /dev/ttyUSB0 -b 115200"
8888
8889For local
8890devices where the serial port device disappears when the device reboots,
8891an additional "serdevtry" wrapper script is provided. To use this
8892wrapper, simply prefix the terminal command with
Andrew Geisslerc926e172021-05-07 16:11:35 -05008893``${COREBASE}/scripts/contrib/serdevtry``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008894
8895 TEST_SERIALCONTROL_CMD = "${COREBASE}/scripts/contrib/serdevtry picocom -b 115200 /dev/ttyUSB0"
8896
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008897Running Tests
8898-------------
8899
8900You can start the tests automatically or manually:
8901
8902- *Automatically running tests:* To run the tests automatically after
8903 the OpenEmbedded build system successfully creates an image, first
8904 set the
8905 :term:`TESTIMAGE_AUTO`
8906 variable to "1" in your ``local.conf`` file in the
Andrew Geisslerc926e172021-05-07 16:11:35 -05008907 :term:`Build Directory`::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008908
8909 TESTIMAGE_AUTO = "1"
8910
8911 Next, build your image. If the image successfully builds, the
Andrew Geisslerc926e172021-05-07 16:11:35 -05008912 tests run::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008913
8914 bitbake core-image-sato
8915
8916- *Manually running tests:* To manually run the tests, first globally
8917 inherit the
8918 :ref:`testimage <ref-classes-testimage*>` class
Andrew Geisslerc926e172021-05-07 16:11:35 -05008919 by editing your ``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008920
8921 INHERIT += "testimage"
8922
Andrew Geisslerc926e172021-05-07 16:11:35 -05008923 Next, use BitBake to run the tests::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008924
8925 bitbake -c testimage image
8926
8927All test files reside in ``meta/lib/oeqa/runtime`` in the
8928:term:`Source Directory`. A test name maps
8929directly to a Python module. Each test module may contain a number of
8930individual tests. Tests are usually grouped together by the area tested
8931(e.g tests for systemd reside in ``meta/lib/oeqa/runtime/systemd.py``).
8932
8933You can add tests to any layer provided you place them in the proper
8934area and you extend :term:`BBPATH` in
8935the ``local.conf`` file as normal. Be sure that tests reside in
8936``layer/lib/oeqa/runtime``.
8937
8938.. note::
8939
8940 Be sure that module names do not collide with module names used in
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008941 the default set of test modules in ``meta/lib/oeqa/runtime``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008942
8943You can change the set of tests run by appending or overriding
8944:term:`TEST_SUITES` variable in
Andrew Geissler09036742021-06-25 14:25:14 -05008945``local.conf``. Each name in :term:`TEST_SUITES` represents a required test
8946for the image. Test modules named within :term:`TEST_SUITES` cannot be
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008947skipped even if a test is not suitable for an image (e.g. running the
8948RPM tests on an image without ``rpm``). Appending "auto" to
Andrew Geissler09036742021-06-25 14:25:14 -05008949:term:`TEST_SUITES` causes the build system to try to run all tests that are
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008950suitable for the image (i.e. each test module may elect to skip itself).
8951
Andrew Geissler09036742021-06-25 14:25:14 -05008952The order you list tests in :term:`TEST_SUITES` is important and influences
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008953test dependencies. Consequently, tests that depend on other tests should
8954be added after the test on which they depend. For example, since the
8955``ssh`` test depends on the ``ping`` test, "ssh" needs to come after
8956"ping" in the list. The test class provides no re-ordering or dependency
8957handling.
8958
8959.. note::
8960
8961 Each module can have multiple classes with multiple test methods.
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008962 And, Python ``unittest`` rules apply.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008963
8964Here are some things to keep in mind when running tests:
8965
Andrew Geisslerc926e172021-05-07 16:11:35 -05008966- The default tests for the image are defined as::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008967
Patrick Williams0ca19cc2021-08-16 14:03:13 -05008968 DEFAULT_TEST_SUITES:pn-image = "ping ssh df connman syslog xorg scp vnc date rpm dnf dmesg"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008969
Andrew Geisslerc926e172021-05-07 16:11:35 -05008970- Add your own test to the list of the by using the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008971
Patrick Williams0ca19cc2021-08-16 14:03:13 -05008972 TEST_SUITES:append = " mytest"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008973
Andrew Geisslerc926e172021-05-07 16:11:35 -05008974- Run a specific list of tests as follows::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008975
8976 TEST_SUITES = "test1 test2 test3"
8977
8978 Remember, order is important. Be sure to place a test that is
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008979 dependent on another test later in the order.
8980
8981Exporting Tests
8982---------------
8983
8984You can export tests so that they can run independently of the build
8985system. Exporting tests is required if you want to be able to hand the
8986test execution off to a scheduler. You can only export tests that are
8987defined in :term:`TEST_SUITES`.
8988
8989If your image is already built, make sure the following are set in your
Andrew Geisslerc926e172021-05-07 16:11:35 -05008990``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008991
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008992 INHERIT += "testexport"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008993 TEST_TARGET_IP = "IP-address-for-the-test-target"
8994 TEST_SERVER_IP = "IP-address-for-the-test-server"
8995
8996You can then export the tests with the
Andrew Geisslerc926e172021-05-07 16:11:35 -05008997following BitBake command form::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008998
8999 $ bitbake image -c testexport
9000
9001Exporting the tests places them in the
9002:term:`Build Directory` in
9003``tmp/testexport/``\ image, which is controlled by the
Andrew Geissler09036742021-06-25 14:25:14 -05009004:term:`TEST_EXPORT_DIR` variable.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009005
Andrew Geisslerc926e172021-05-07 16:11:35 -05009006You can now run the tests outside of the build environment::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009007
9008 $ cd tmp/testexport/image
9009 $ ./runexported.py testdata.json
9010
9011Here is a complete example that shows IP addresses and uses the
Andrew Geisslerc926e172021-05-07 16:11:35 -05009012``core-image-sato`` image::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009013
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009014 INHERIT += "testexport"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009015 TEST_TARGET_IP = "192.168.7.2"
9016 TEST_SERVER_IP = "192.168.7.1"
9017
Andrew Geisslerc926e172021-05-07 16:11:35 -05009018Use BitBake to export the tests::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009019
9020 $ bitbake core-image-sato -c testexport
9021
9022Run the tests outside of
Andrew Geisslerc926e172021-05-07 16:11:35 -05009023the build environment using the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009024
9025 $ cd tmp/testexport/core-image-sato
9026 $ ./runexported.py testdata.json
9027
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009028Writing New Tests
9029-----------------
9030
9031As mentioned previously, all new test files need to be in the proper
9032place for the build system to find them. New tests for additional
9033functionality outside of the core should be added to the layer that adds
9034the functionality, in ``layer/lib/oeqa/runtime`` (as long as
9035:term:`BBPATH` is extended in the
9036layer's ``layer.conf`` file as normal). Just remember the following:
9037
9038- Filenames need to map directly to test (module) names.
9039
9040- Do not use module names that collide with existing core tests.
9041
William A. Kennington IIIac69b482021-06-02 12:28:27 -07009042- Minimally, an empty ``__init__.py`` file must be present in the runtime
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009043 directory.
9044
9045To create a new test, start by copying an existing module (e.g.
9046``syslog.py`` or ``gcc.py`` are good ones to use). Test modules can use
9047code from ``meta/lib/oeqa/utils``, which are helper classes.
9048
9049.. note::
9050
9051 Structure shell commands such that you rely on them and they return a
9052 single code for success. Be aware that sometimes you will need to
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009053 parse the output. See the ``df.py`` and ``date.py`` modules for examples.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009054
9055You will notice that all test classes inherit ``oeRuntimeTest``, which
9056is found in ``meta/lib/oetest.py``. This base class offers some helper
9057attributes, which are described in the following sections:
9058
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009059Class Methods
9060~~~~~~~~~~~~~
9061
9062Class methods are as follows:
9063
9064- *hasPackage(pkg):* Returns "True" if ``pkg`` is in the installed
9065 package list of the image, which is based on the manifest file that
9066 is generated during the ``do_rootfs`` task.
9067
9068- *hasFeature(feature):* Returns "True" if the feature is in
9069 :term:`IMAGE_FEATURES` or
9070 :term:`DISTRO_FEATURES`.
9071
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009072Class Attributes
9073~~~~~~~~~~~~~~~~
9074
9075Class attributes are as follows:
9076
9077- *pscmd:* Equals "ps -ef" if ``procps`` is installed in the image.
9078 Otherwise, ``pscmd`` equals "ps" (busybox).
9079
9080- *tc:* The called test context, which gives access to the
9081 following attributes:
9082
9083 - *d:* The BitBake datastore, which allows you to use stuff such
9084 as ``oeRuntimeTest.tc.d.getVar("VIRTUAL-RUNTIME_init_manager")``.
9085
9086 - *testslist and testsrequired:* Used internally. The tests
9087 do not need these.
9088
9089 - *filesdir:* The absolute path to
9090 ``meta/lib/oeqa/runtime/files``, which contains helper files for
9091 tests meant for copying on the target such as small files written
9092 in C for compilation.
9093
9094 - *target:* The target controller object used to deploy and
9095 start an image on a particular target (e.g. Qemu, SimpleRemote,
9096 and SystemdbootTarget). Tests usually use the following:
9097
9098 - *ip:* The target's IP address.
9099
9100 - *server_ip:* The host's IP address, which is usually used
9101 by the DNF test suite.
9102
9103 - *run(cmd, timeout=None):* The single, most used method.
9104 This command is a wrapper for: ``ssh root@host "cmd"``. The
9105 command returns a tuple: (status, output), which are what their
9106 names imply - the return code of "cmd" and whatever output it
9107 produces. The optional timeout argument represents the number
9108 of seconds the test should wait for "cmd" to return. If the
9109 argument is "None", the test uses the default instance's
9110 timeout period, which is 300 seconds. If the argument is "0",
9111 the test runs until the command returns.
9112
9113 - *copy_to(localpath, remotepath):*
9114 ``scp localpath root@ip:remotepath``.
9115
9116 - *copy_from(remotepath, localpath):*
9117 ``scp root@host:remotepath localpath``.
9118
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009119Instance Attributes
9120~~~~~~~~~~~~~~~~~~~
9121
William A. Kennington IIIac69b482021-06-02 12:28:27 -07009122There is a single instance attribute, which is ``target``. The ``target``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009123instance attribute is identical to the class attribute of the same name,
9124which is described in the previous section. This attribute exists as
9125both an instance and class attribute so tests can use
9126``self.target.run(cmd)`` in instance methods instead of
9127``oeRuntimeTest.tc.target.run(cmd)``.
9128
9129Installing Packages in the DUT Without the Package Manager
9130----------------------------------------------------------
9131
9132When a test requires a package built by BitBake, it is possible to
9133install that package. Installing the package does not require a package
9134manager be installed in the device under test (DUT). It does, however,
9135require an SSH connection and the target must be using the
9136``sshcontrol`` class.
9137
9138.. note::
9139
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009140 This method uses ``scp`` to copy files from the host to the target, which
9141 causes permissions and special attributes to be lost.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009142
9143A JSON file is used to define the packages needed by a test. This file
9144must be in the same path as the file used to define the tests.
9145Furthermore, the filename must map directly to the test module name with
9146a ``.json`` extension.
9147
9148The JSON file must include an object with the test name as keys of an
9149object or an array. This object (or array of objects) uses the following
9150data:
9151
Andrew Geissler615f2f12022-07-15 14:00:58 -05009152- "pkg" --- a mandatory string that is the name of the package to be
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009153 installed.
9154
Andrew Geissler615f2f12022-07-15 14:00:58 -05009155- "rm" --- an optional boolean, which defaults to "false", that specifies
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009156 to remove the package after the test.
9157
Andrew Geissler615f2f12022-07-15 14:00:58 -05009158- "extract" --- an optional boolean, which defaults to "false", that
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009159 specifies if the package must be extracted from the package format.
9160 When set to "true", the package is not automatically installed into
9161 the DUT.
9162
9163Following is an example JSON file that handles test "foo" installing
9164package "bar" and test "foobar" installing packages "foo" and "bar".
9165Once the test is complete, the packages are removed from the DUT.
9166::
9167
9168 {
9169 "foo": {
9170 "pkg": "bar"
9171 },
9172 "foobar": [
9173 {
9174 "pkg": "foo",
9175 "rm": true
9176 },
9177 {
9178 "pkg": "bar",
9179 "rm": true
9180 }
9181 ]
9182 }
9183
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009184Debugging Tools and Techniques
9185==============================
9186
9187The exact method for debugging build failures depends on the nature of
9188the problem and on the system's area from which the bug originates.
9189Standard debugging practices such as comparison against the last known
9190working version with examination of the changes and the re-application
9191of steps to identify the one causing the problem are valid for the Yocto
9192Project just as they are for any other system. Even though it is
9193impossible to detail every possible potential failure, this section
9194provides some general tips to aid in debugging given a variety of
9195situations.
9196
9197.. note::
9198
9199 A useful feature for debugging is the error reporting tool.
9200 Configuring the Yocto Project to use this tool causes the
9201 OpenEmbedded build system to produce error reporting commands as part
9202 of the console output. You can enter the commands after the build
9203 completes to log error information into a common database, that can
9204 help you figure out what might be going wrong. For information on how
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009205 to enable and use this feature, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06009206 ":ref:`dev-manual/common-tasks:using the error reporting tool`"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009207 section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009208
9209The following list shows the debugging topics in the remainder of this
9210section:
9211
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05009212- ":ref:`dev-manual/common-tasks:viewing logs from failed tasks`" describes
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009213 how to find and view logs from tasks that failed during the build
9214 process.
9215
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05009216- ":ref:`dev-manual/common-tasks:viewing variable values`" describes how to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009217 use the BitBake ``-e`` option to examine variable values after a
9218 recipe has been parsed.
9219
Andrew Geissler09209ee2020-12-13 08:44:15 -06009220- ":ref:`dev-manual/common-tasks:viewing package information with \`\`oe-pkgdata-util\`\``"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009221 describes how to use the ``oe-pkgdata-util`` utility to query
9222 :term:`PKGDATA_DIR` and
9223 display package-related information for built packages.
9224
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05009225- ":ref:`dev-manual/common-tasks:viewing dependencies between recipes and tasks`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009226 describes how to use the BitBake ``-g`` option to display recipe
9227 dependency information used during the build.
9228
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05009229- ":ref:`dev-manual/common-tasks:viewing task variable dependencies`" describes
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009230 how to use the ``bitbake-dumpsig`` command in conjunction with key
9231 subdirectories in the
9232 :term:`Build Directory` to determine
9233 variable dependencies.
9234
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05009235- ":ref:`dev-manual/common-tasks:running specific tasks`" describes
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009236 how to use several BitBake options (e.g. ``-c``, ``-C``, and ``-f``)
9237 to run specific tasks in the build chain. It can be useful to run
9238 tasks "out-of-order" when trying isolate build issues.
9239
Andrew Geisslerd5838332022-05-27 11:33:10 -05009240- ":ref:`dev-manual/common-tasks:general BitBake problems`" describes how
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009241 to use BitBake's ``-D`` debug output option to reveal more about what
9242 BitBake is doing during the build.
9243
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05009244- ":ref:`dev-manual/common-tasks:building with no dependencies`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009245 describes how to use the BitBake ``-b`` option to build a recipe
9246 while ignoring dependencies.
9247
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05009248- ":ref:`dev-manual/common-tasks:recipe logging mechanisms`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009249 describes how to use the many recipe logging functions to produce
9250 debugging output and report errors and warnings.
9251
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05009252- ":ref:`dev-manual/common-tasks:debugging parallel make races`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009253 describes how to debug situations where the build consists of several
9254 parts that are run simultaneously and when the output or result of
9255 one part is not ready for use with a different part of the build that
9256 depends on that output.
9257
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05009258- ":ref:`dev-manual/common-tasks:debugging with the gnu project debugger (gdb) remotely`"
9259 describes how to use GDB to allow you to examine running programs, which can
9260 help you fix problems.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009261
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05009262- ":ref:`dev-manual/common-tasks:debugging with the gnu project debugger (gdb) on the target`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009263 describes how to use GDB directly on target hardware for debugging.
9264
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05009265- ":ref:`dev-manual/common-tasks:other debugging tips`" describes
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009266 miscellaneous debugging tips that can be useful.
9267
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009268Viewing Logs from Failed Tasks
9269------------------------------
9270
9271You can find the log for a task in the file
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009272``${``\ :term:`WORKDIR`\ ``}/temp/log.do_``\ `taskname`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009273For example, the log for the
9274:ref:`ref-tasks-compile` task of the
9275QEMU minimal image for the x86 machine (``qemux86``) might be in
9276``tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/temp/log.do_compile``.
9277To see the commands :term:`BitBake` ran
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009278to generate a log, look at the corresponding ``run.do_``\ `taskname` file
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009279in the same directory.
9280
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009281``log.do_``\ `taskname` and ``run.do_``\ `taskname` are actually symbolic
9282links to ``log.do_``\ `taskname`\ ``.``\ `pid` and
9283``log.run_``\ `taskname`\ ``.``\ `pid`, where `pid` is the PID the task had
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009284when it ran. The symlinks always point to the files corresponding to the
9285most recent run.
9286
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009287Viewing Variable Values
9288-----------------------
9289
9290Sometimes you need to know the value of a variable as a result of
9291BitBake's parsing step. This could be because some unexpected behavior
9292occurred in your project. Perhaps an attempt to :ref:`modify a variable
9293<bitbake:bitbake-user-manual/bitbake-user-manual-metadata:modifying existing
9294variables>` did not work out as expected.
9295
9296BitBake's ``-e`` option is used to display variable values after
9297parsing. The following command displays the variable values after the
9298configuration files (i.e. ``local.conf``, ``bblayers.conf``,
Andrew Geisslerc926e172021-05-07 16:11:35 -05009299``bitbake.conf`` and so forth) have been parsed::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009300
9301 $ bitbake -e
9302
9303The following command displays variable values after a specific recipe has
Andrew Geisslerc926e172021-05-07 16:11:35 -05009304been parsed. The variables include those from the configuration as well::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009305
9306 $ bitbake -e recipename
9307
9308.. note::
9309
9310 Each recipe has its own private set of variables (datastore).
9311 Internally, after parsing the configuration, a copy of the resulting
9312 datastore is made prior to parsing each recipe. This copying implies
9313 that variables set in one recipe will not be visible to other
9314 recipes.
9315
9316 Likewise, each task within a recipe gets a private datastore based on
9317 the recipe datastore, which means that variables set within one task
9318 will not be visible to other tasks.
9319
9320In the output of ``bitbake -e``, each variable is preceded by a
9321description of how the variable got its value, including temporary
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009322values that were later overridden. This description also includes
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009323variable flags (varflags) set on the variable. The output can be very
9324helpful during debugging.
9325
9326Variables that are exported to the environment are preceded by
Andrew Geisslerc926e172021-05-07 16:11:35 -05009327``export`` in the output of ``bitbake -e``. See the following example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009328
9329 export CC="i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/ulf/poky/build/tmp/sysroots/qemux86"
9330
9331In addition to variable values, the output of the ``bitbake -e`` and
9332``bitbake -e`` recipe commands includes the following information:
9333
9334- The output starts with a tree listing all configuration files and
9335 classes included globally, recursively listing the files they include
9336 or inherit in turn. Much of the behavior of the OpenEmbedded build
Andrew Geissler09209ee2020-12-13 08:44:15 -06009337 system (including the behavior of the :ref:`ref-manual/tasks:normal recipe build tasks`) is
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009338 implemented in the
9339 :ref:`base <ref-classes-base>` class and the
9340 classes it inherits, rather than being built into BitBake itself.
9341
9342- After the variable values, all functions appear in the output. For
9343 shell functions, variables referenced within the function body are
9344 expanded. If a function has been modified using overrides or using
Patrick Williams0ca19cc2021-08-16 14:03:13 -05009345 override-style operators like ``:append`` and ``:prepend``, then the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009346 final assembled function body appears in the output.
9347
9348Viewing Package Information with ``oe-pkgdata-util``
9349----------------------------------------------------
9350
9351You can use the ``oe-pkgdata-util`` command-line utility to query
9352:term:`PKGDATA_DIR` and display
9353various package-related information. When you use the utility, you must
9354use it to view information on packages that have already been built.
9355
9356Following are a few of the available ``oe-pkgdata-util`` subcommands.
9357
9358.. note::
9359
9360 You can use the standard \* and ? globbing wildcards as part of
9361 package names and paths.
9362
9363- ``oe-pkgdata-util list-pkgs [pattern]``: Lists all packages
9364 that have been built, optionally limiting the match to packages that
9365 match pattern.
9366
9367- ``oe-pkgdata-util list-pkg-files package ...``: Lists the
9368 files and directories contained in the given packages.
9369
9370 .. note::
9371
9372 A different way to view the contents of a package is to look at
9373 the
9374 ``${``\ :term:`WORKDIR`\ ``}/packages-split``
9375 directory of the recipe that generates the package. This directory
9376 is created by the
9377 :ref:`ref-tasks-package` task
9378 and has one subdirectory for each package the recipe generates,
9379 which contains the files stored in that package.
9380
9381 If you want to inspect the ``${WORKDIR}/packages-split``
9382 directory, make sure that
9383 :ref:`rm_work <ref-classes-rm-work>` is not
9384 enabled when you build the recipe.
9385
9386- ``oe-pkgdata-util find-path path ...``: Lists the names of
9387 the packages that contain the given paths. For example, the following
9388 tells us that ``/usr/share/man/man1/make.1`` is contained in the
Andrew Geisslerc926e172021-05-07 16:11:35 -05009389 ``make-doc`` package::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009390
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009391 $ oe-pkgdata-util find-path /usr/share/man/man1/make.1
9392 make-doc: /usr/share/man/man1/make.1
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009393
9394- ``oe-pkgdata-util lookup-recipe package ...``: Lists the name
9395 of the recipes that produce the given packages.
9396
9397For more information on the ``oe-pkgdata-util`` command, use the help
Andrew Geisslerc926e172021-05-07 16:11:35 -05009398facility::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009399
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009400 $ oe-pkgdata-util --help
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009401 $ oe-pkgdata-util subcommand --help
9402
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009403Viewing Dependencies Between Recipes and Tasks
9404----------------------------------------------
9405
9406Sometimes it can be hard to see why BitBake wants to build other recipes
9407before the one you have specified. Dependency information can help you
9408understand why a recipe is built.
9409
9410To generate dependency information for a recipe, run the following
Andrew Geisslerc926e172021-05-07 16:11:35 -05009411command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009412
9413 $ bitbake -g recipename
9414
9415This command writes the following files in the current directory:
9416
9417- ``pn-buildlist``: A list of recipes/targets involved in building
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009418 `recipename`. "Involved" here means that at least one task from the
9419 recipe needs to run when building `recipename` from scratch. Targets
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009420 that are in
9421 :term:`ASSUME_PROVIDED`
9422 are not listed.
9423
9424- ``task-depends.dot``: A graph showing dependencies between tasks.
9425
9426The graphs are in
9427`DOT <https://en.wikipedia.org/wiki/DOT_%28graph_description_language%29>`__
9428format and can be converted to images (e.g. using the ``dot`` tool from
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009429`Graphviz <https://www.graphviz.org/>`__).
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009430
9431.. note::
9432
9433 - DOT files use a plain text format. The graphs generated using the
9434 ``bitbake -g`` command are often so large as to be difficult to
Andrew Geisslerd5838332022-05-27 11:33:10 -05009435 read without special pruning (e.g. with BitBake's ``-I`` option)
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009436 and processing. Despite the form and size of the graphs, the
9437 corresponding ``.dot`` files can still be possible to read and
9438 provide useful information.
9439
9440 As an example, the ``task-depends.dot`` file contains lines such
Andrew Geisslerc926e172021-05-07 16:11:35 -05009441 as the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009442
9443 "libxslt.do_configure" -> "libxml2.do_populate_sysroot"
9444
9445 The above example line reveals that the
9446 :ref:`ref-tasks-configure`
9447 task in ``libxslt`` depends on the
9448 :ref:`ref-tasks-populate_sysroot`
9449 task in ``libxml2``, which is a normal
9450 :term:`DEPENDS` dependency
9451 between the two recipes.
9452
9453 - For an example of how ``.dot`` files can be processed, see the
9454 ``scripts/contrib/graph-tool`` Python script, which finds and
9455 displays paths between graph nodes.
9456
9457You can use a different method to view dependency information by using
Andrew Geisslerc926e172021-05-07 16:11:35 -05009458the following command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009459
9460 $ bitbake -g -u taskexp recipename
9461
9462This command
9463displays a GUI window from which you can view build-time and runtime
9464dependencies for the recipes involved in building recipename.
9465
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009466Viewing Task Variable Dependencies
9467----------------------------------
9468
9469As mentioned in the
9470":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-execution:checksums (signatures)`" section of the BitBake
9471User Manual, BitBake tries to automatically determine what variables a
9472task depends on so that it can rerun the task if any values of the
9473variables change. This determination is usually reliable. However, if
9474you do things like construct variable names at runtime, then you might
9475have to manually declare dependencies on those variables using
9476``vardeps`` as described in the
9477":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags`" section of the BitBake
9478User Manual.
9479
9480If you are unsure whether a variable dependency is being picked up
9481automatically for a given task, you can list the variable dependencies
9482BitBake has determined by doing the following:
9483
Andrew Geisslerc926e172021-05-07 16:11:35 -050094841. Build the recipe containing the task::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009485
9486 $ bitbake recipename
9487
94882. Inside the :term:`STAMPS_DIR`
9489 directory, find the signature data (``sigdata``) file that
9490 corresponds to the task. The ``sigdata`` files contain a pickled
9491 Python database of all the metadata that went into creating the input
9492 checksum for the task. As an example, for the
9493 :ref:`ref-tasks-fetch` task of the
9494 ``db`` recipe, the ``sigdata`` file might be found in the following
Andrew Geisslerc926e172021-05-07 16:11:35 -05009495 location::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009496
9497 ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1
9498
9499 For tasks that are accelerated through the shared state
Andrew Geissler09209ee2020-12-13 08:44:15 -06009500 (:ref:`sstate <overview-manual/concepts:shared state cache>`) cache, an
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009501 additional ``siginfo`` file is written into
9502 :term:`SSTATE_DIR` along with
9503 the cached task output. The ``siginfo`` files contain exactly the
9504 same information as ``sigdata`` files.
9505
95063. Run ``bitbake-dumpsig`` on the ``sigdata`` or ``siginfo`` file. Here
Andrew Geisslerc926e172021-05-07 16:11:35 -05009507 is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009508
9509 $ bitbake-dumpsig ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1
9510
9511 In the output of the above command, you will find a line like the
9512 following, which lists all the (inferred) variable dependencies for
9513 the task. This list also includes indirect dependencies from
9514 variables depending on other variables, recursively.
9515 ::
9516
9517 Task dependencies: ['PV', 'SRCREV', 'SRC_URI', 'SRC_URI[md5sum]', 'SRC_URI[sha256sum]', 'base_do_fetch']
9518
9519 .. note::
9520
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009521 Functions (e.g. ``base_do_fetch``) also count as variable dependencies.
9522 These functions in turn depend on the variables they reference.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009523
9524 The output of ``bitbake-dumpsig`` also includes the value each
9525 variable had, a list of dependencies for each variable, and
Andrew Geissler7e0e3c02022-02-25 20:34:39 +00009526 :term:`BB_BASEHASH_IGNORE_VARS`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009527 information.
9528
9529There is also a ``bitbake-diffsigs`` command for comparing two
9530``siginfo`` or ``sigdata`` files. This command can be helpful when
9531trying to figure out what changed between two versions of a task. If you
9532call ``bitbake-diffsigs`` with just one file, the command behaves like
9533``bitbake-dumpsig``.
9534
9535You can also use BitBake to dump out the signature construction
9536information without executing tasks by using either of the following
Andrew Geisslerc926e172021-05-07 16:11:35 -05009537BitBake command-line options::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009538
9539 ‐‐dump-signatures=SIGNATURE_HANDLER
9540 -S SIGNATURE_HANDLER
9541
9542
9543.. note::
9544
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009545 Two common values for `SIGNATURE_HANDLER` are "none" and "printdiff", which
9546 dump only the signature or compare the dumped signature with the cached one,
9547 respectively.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009548
9549Using BitBake with either of these options causes BitBake to dump out
9550``sigdata`` files in the ``stamps`` directory for every task it would
9551have executed instead of building the specified target package.
9552
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009553Viewing Metadata Used to Create the Input Signature of a Shared State Task
9554--------------------------------------------------------------------------
9555
9556Seeing what metadata went into creating the input signature of a shared
9557state (sstate) task can be a useful debugging aid. This information is
9558available in signature information (``siginfo``) files in
9559:term:`SSTATE_DIR`. For
9560information on how to view and interpret information in ``siginfo``
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05009561files, see the
9562":ref:`dev-manual/common-tasks:viewing task variable dependencies`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009563
9564For conceptual information on shared state, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06009565":ref:`overview-manual/concepts:shared state`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009566section in the Yocto Project Overview and Concepts Manual.
9567
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009568Invalidating Shared State to Force a Task to Run
9569------------------------------------------------
9570
9571The OpenEmbedded build system uses
Andrew Geissler09209ee2020-12-13 08:44:15 -06009572:ref:`checksums <overview-manual/concepts:checksums (signatures)>` and
9573:ref:`overview-manual/concepts:shared state` cache to avoid unnecessarily
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009574rebuilding tasks. Collectively, this scheme is known as "shared state
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009575code".
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009576
9577As with all schemes, this one has some drawbacks. It is possible that
9578you could make implicit changes to your code that the checksum
9579calculations do not take into account. These implicit changes affect a
9580task's output but do not trigger the shared state code into rebuilding a
9581recipe. Consider an example during which a tool changes its output.
9582Assume that the output of ``rpmdeps`` changes. The result of the change
9583should be that all the ``package`` and ``package_write_rpm`` shared
9584state cache items become invalid. However, because the change to the
9585output is external to the code and therefore implicit, the associated
9586shared state cache items do not become invalidated. In this case, the
9587build process uses the cached items rather than running the task again.
9588Obviously, these types of implicit changes can cause problems.
9589
9590To avoid these problems during the build, you need to understand the
9591effects of any changes you make. Realize that changes you make directly
9592to a function are automatically factored into the checksum calculation.
9593Thus, these explicit changes invalidate the associated area of shared
9594state cache. However, you need to be aware of any implicit changes that
9595are not obvious changes to the code and could affect the output of a
9596given task.
9597
9598When you identify an implicit change, you can easily take steps to
9599invalidate the cache and force the tasks to run. The steps you can take
9600are as simple as changing a function's comments in the source code. For
9601example, to invalidate package shared state files, change the comment
9602statements of
9603:ref:`ref-tasks-package` or the
9604comments of one of the functions it calls. Even though the change is
9605purely cosmetic, it causes the checksum to be recalculated and forces
9606the build system to run the task again.
9607
9608.. note::
9609
9610 For an example of a commit that makes a cosmetic change to invalidate
9611 shared state, see this
Andrew Geissler09209ee2020-12-13 08:44:15 -06009612 :yocto_git:`commit </poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009613
9614Running Specific Tasks
9615----------------------
9616
9617Any given recipe consists of a set of tasks. The standard BitBake
9618behavior in most cases is: ``do_fetch``, ``do_unpack``, ``do_patch``,
9619``do_configure``, ``do_compile``, ``do_install``, ``do_package``,
9620``do_package_write_*``, and ``do_build``. The default task is
9621``do_build`` and any tasks on which it depends build first. Some tasks,
9622such as ``do_devshell``, are not part of the default build chain. If you
9623wish to run a task that is not part of the default build chain, you can
Andrew Geisslerc926e172021-05-07 16:11:35 -05009624use the ``-c`` option in BitBake. Here is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009625
9626 $ bitbake matchbox-desktop -c devshell
9627
9628The ``-c`` option respects task dependencies, which means that all other
9629tasks (including tasks from other recipes) that the specified task
9630depends on will be run before the task. Even when you manually specify a
9631task to run with ``-c``, BitBake will only run the task if it considers
9632it "out of date". See the
Andrew Geissler09209ee2020-12-13 08:44:15 -06009633":ref:`overview-manual/concepts:stamp files and the rerunning of tasks`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009634section in the Yocto Project Overview and Concepts Manual for how
9635BitBake determines whether a task is "out of date".
9636
9637If you want to force an up-to-date task to be rerun (e.g. because you
9638made manual modifications to the recipe's
9639:term:`WORKDIR` that you want to try
9640out), then you can use the ``-f`` option.
9641
9642.. note::
9643
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009644 The reason ``-f`` is never required when running the
9645 :ref:`ref-tasks-devshell` task is because the
9646 [\ :ref:`nostamp <bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ]
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009647 variable flag is already set for the task.
9648
Andrew Geisslerc926e172021-05-07 16:11:35 -05009649The following example shows one way you can use the ``-f`` option::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009650
9651 $ bitbake matchbox-desktop
9652 .
9653 .
9654 make some changes to the source code in the work directory
9655 .
9656 .
9657 $ bitbake matchbox-desktop -c compile -f
9658 $ bitbake matchbox-desktop
9659
9660This sequence first builds and then recompiles ``matchbox-desktop``. The
9661last command reruns all tasks (basically the packaging tasks) after the
9662compile. BitBake recognizes that the ``do_compile`` task was rerun and
9663therefore understands that the other tasks also need to be run again.
9664
9665Another, shorter way to rerun a task and all
Andrew Geissler09209ee2020-12-13 08:44:15 -06009666:ref:`ref-manual/tasks:normal recipe build tasks`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009667that depend on it is to use the ``-C`` option.
9668
9669.. note::
9670
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009671 This option is upper-cased and is separate from the ``-c``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009672 option, which is lower-cased.
9673
9674Using this option invalidates the given task and then runs the
9675:ref:`ref-tasks-build` task, which is
9676the default task if no task is given, and the tasks on which it depends.
9677You could replace the final two commands in the previous example with
Andrew Geisslerc926e172021-05-07 16:11:35 -05009678the following single command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009679
9680 $ bitbake matchbox-desktop -C compile
9681
9682Internally, the ``-f`` and ``-C`` options work by tainting (modifying)
9683the input checksum of the specified task. This tainting indirectly
9684causes the task and its dependent tasks to be rerun through the normal
9685task dependency mechanisms.
9686
9687.. note::
9688
9689 BitBake explicitly keeps track of which tasks have been tainted in
9690 this fashion, and will print warnings such as the following for
9691 builds involving such tasks:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009692
9693 .. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009694
9695 WARNING: /home/ulf/poky/meta/recipes-sato/matchbox-desktop/matchbox-desktop_2.1.bb.do_compile is tainted from a forced run
9696
9697
9698 The purpose of the warning is to let you know that the work directory
9699 and build output might not be in the clean state they would be in for
9700 a "normal" build, depending on what actions you took. To get rid of
9701 such warnings, you can remove the work directory and rebuild the
Andrew Geisslerc926e172021-05-07 16:11:35 -05009702 recipe, as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009703
9704 $ bitbake matchbox-desktop -c clean
9705 $ bitbake matchbox-desktop
9706
9707
9708You can view a list of tasks in a given package by running the
Andrew Geisslerc926e172021-05-07 16:11:35 -05009709``do_listtasks`` task as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009710
9711 $ bitbake matchbox-desktop -c listtasks
9712
9713The results appear as output to the console and are also in
9714the file ``${WORKDIR}/temp/log.do_listtasks``.
9715
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009716General BitBake Problems
9717------------------------
9718
9719You can see debug output from BitBake by using the ``-D`` option. The
9720debug output gives more information about what BitBake is doing and the
9721reason behind it. Each ``-D`` option you use increases the logging
9722level. The most common usage is ``-DDD``.
9723
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009724The output from ``bitbake -DDD -v targetname`` can reveal why BitBake
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009725chose a certain version of a package or why BitBake picked a certain
9726provider. This command could also help you in a situation where you
9727think BitBake did something unexpected.
9728
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009729Building with No Dependencies
9730-----------------------------
9731
9732To build a specific recipe (``.bb`` file), you can use the following
Andrew Geisslerc926e172021-05-07 16:11:35 -05009733command form::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009734
9735 $ bitbake -b somepath/somerecipe.bb
9736
9737This command form does
9738not check for dependencies. Consequently, you should use it only when
9739you know existing dependencies have been met.
9740
9741.. note::
9742
9743 You can also specify fragments of the filename. In this case, BitBake
9744 checks for a unique match.
9745
9746Recipe Logging Mechanisms
9747-------------------------
9748
9749The Yocto Project provides several logging functions for producing
9750debugging output and reporting errors and warnings. For Python
William A. Kennington IIIac69b482021-06-02 12:28:27 -07009751functions, the following logging functions are available. All of these functions
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009752log to ``${T}/log.do_``\ `task`, and can also log to standard output
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009753(stdout) with the right settings:
9754
9755- ``bb.plain(msg)``: Writes msg as is to the log while also
9756 logging to stdout.
9757
9758- ``bb.note(msg)``: Writes "NOTE: msg" to the log. Also logs to
9759 stdout if BitBake is called with "-v".
9760
9761- ``bb.debug(level, msg)``: Writes "DEBUG: msg" to the
9762 log. Also logs to stdout if the log level is greater than or equal to
Patrick Williams213cb262021-08-07 19:21:33 -05009763 level. See the ":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-intro:usage and syntax`" option
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009764 in the BitBake User Manual for more information.
9765
9766- ``bb.warn(msg)``: Writes "WARNING: msg" to the log while also
9767 logging to stdout.
9768
9769- ``bb.error(msg)``: Writes "ERROR: msg" to the log while also
9770 logging to standard out (stdout).
9771
9772 .. note::
9773
9774 Calling this function does not cause the task to fail.
9775
Andrew Geisslereff27472021-10-29 15:35:00 -05009776- ``bb.fatal(msg)``: This logging function is similar to
9777 ``bb.error(msg)`` but also causes the calling task to fail.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009778
9779 .. note::
9780
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009781 ``bb.fatal()`` raises an exception, which means you do not need to put a
9782 "return" statement after the function.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009783
9784The same logging functions are also available in shell functions, under
9785the names ``bbplain``, ``bbnote``, ``bbdebug``, ``bbwarn``, ``bberror``,
9786and ``bbfatal``. The
9787:ref:`logging <ref-classes-logging>` class
9788implements these functions. See that class in the ``meta/classes``
9789folder of the :term:`Source Directory` for information.
9790
9791Logging With Python
9792~~~~~~~~~~~~~~~~~~~
9793
9794When creating recipes using Python and inserting code that handles build
9795logs, keep in mind the goal is to have informative logs while keeping
9796the console as "silent" as possible. Also, if you want status messages
9797in the log, use the "debug" loglevel.
9798
9799Following is an example written in Python. The code handles logging for
9800a function that determines the number of tasks needed to be run. See the
9801":ref:`ref-tasks-listtasks`"
Andrew Geisslerc926e172021-05-07 16:11:35 -05009802section for additional information::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009803
9804 python do_listtasks() {
9805 bb.debug(2, "Starting to figure out the task list")
9806 if noteworthy_condition:
9807 bb.note("There are 47 tasks to run")
9808 bb.debug(2, "Got to point xyz")
9809 if warning_trigger:
9810 bb.warn("Detected warning_trigger, this might be a problem later.")
9811 if recoverable_error:
9812 bb.error("Hit recoverable_error, you really need to fix this!")
9813 if fatal_error:
9814 bb.fatal("fatal_error detected, unable to print the task list")
9815 bb.plain("The tasks present are abc")
9816 bb.debug(2, "Finished figuring out the tasklist")
9817 }
9818
9819Logging With Bash
9820~~~~~~~~~~~~~~~~~
9821
9822When creating recipes using Bash and inserting code that handles build
Andrew Geissler615f2f12022-07-15 14:00:58 -05009823logs, you have the same goals --- informative with minimal console output.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009824The syntax you use for recipes written in Bash is similar to that of
9825recipes written in Python described in the previous section.
9826
9827Following is an example written in Bash. The code logs the progress of
9828the ``do_my_function`` function.
9829::
9830
9831 do_my_function() {
9832 bbdebug 2 "Running do_my_function"
9833 if [ exceptional_condition ]; then
9834 bbnote "Hit exceptional_condition"
9835 fi
9836 bbdebug 2 "Got to point xyz"
9837 if [ warning_trigger ]; then
9838 bbwarn "Detected warning_trigger, this might cause a problem later."
9839 fi
9840 if [ recoverable_error ]; then
9841 bberror "Hit recoverable_error, correcting"
9842 fi
9843 if [ fatal_error ]; then
9844 bbfatal "fatal_error detected"
9845 fi
9846 bbdebug 2 "Completed do_my_function"
9847 }
9848
9849
9850Debugging Parallel Make Races
9851-----------------------------
9852
9853A parallel ``make`` race occurs when the build consists of several parts
9854that are run simultaneously and a situation occurs when the output or
9855result of one part is not ready for use with a different part of the
9856build that depends on that output. Parallel make races are annoying and
William A. Kennington IIIac69b482021-06-02 12:28:27 -07009857can sometimes be difficult to reproduce and fix. However, there are some simple
9858tips and tricks that can help you debug and fix them. This section
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009859presents a real-world example of an error encountered on the Yocto
9860Project autobuilder and the process used to fix it.
9861
9862.. note::
9863
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009864 If you cannot properly fix a ``make`` race condition, you can work around it
9865 by clearing either the :term:`PARALLEL_MAKE` or :term:`PARALLEL_MAKEINST`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009866 variables.
9867
9868The Failure
9869~~~~~~~~~~~
9870
9871For this example, assume that you are building an image that depends on
9872the "neard" package. And, during the build, BitBake runs into problems
9873and creates the following output.
9874
9875.. note::
9876
9877 This example log file has longer lines artificially broken to make
9878 the listing easier to read.
9879
9880If you examine the output or the log file, you see the failure during
9881``make``:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009882
9883.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009884
9885 | DEBUG: SITE files ['endian-little', 'bit-32', 'ix86-common', 'common-linux', 'common-glibc', 'i586-linux', 'common']
9886 | DEBUG: Executing shell function do_compile
9887 | NOTE: make -j 16
9888 | make --no-print-directory all-am
9889 | /bin/mkdir -p include/near
9890 | /bin/mkdir -p include/near
9891 | /bin/mkdir -p include/near
Andrew Geissler595f6302022-01-24 19:11:47 +00009892 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009893 0.14-r0/neard-0.14/include/types.h include/near/types.h
Andrew Geissler595f6302022-01-24 19:11:47 +00009894 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009895 0.14-r0/neard-0.14/include/log.h include/near/log.h
Andrew Geissler595f6302022-01-24 19:11:47 +00009896 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009897 0.14-r0/neard-0.14/include/plugin.h include/near/plugin.h
9898 | /bin/mkdir -p include/near
9899 | /bin/mkdir -p include/near
9900 | /bin/mkdir -p include/near
Andrew Geissler595f6302022-01-24 19:11:47 +00009901 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009902 0.14-r0/neard-0.14/include/tag.h include/near/tag.h
9903 | /bin/mkdir -p include/near
Andrew Geissler595f6302022-01-24 19:11:47 +00009904 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009905 0.14-r0/neard-0.14/include/adapter.h include/near/adapter.h
9906 | /bin/mkdir -p include/near
Andrew Geissler595f6302022-01-24 19:11:47 +00009907 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009908 0.14-r0/neard-0.14/include/ndef.h include/near/ndef.h
Andrew Geissler595f6302022-01-24 19:11:47 +00009909 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009910 0.14-r0/neard-0.14/include/tlv.h include/near/tlv.h
9911 | /bin/mkdir -p include/near
9912 | /bin/mkdir -p include/near
Andrew Geissler595f6302022-01-24 19:11:47 +00009913 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009914 0.14-r0/neard-0.14/include/setting.h include/near/setting.h
9915 | /bin/mkdir -p include/near
9916 | /bin/mkdir -p include/near
9917 | /bin/mkdir -p include/near
Andrew Geissler595f6302022-01-24 19:11:47 +00009918 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009919 0.14-r0/neard-0.14/include/device.h include/near/device.h
Andrew Geissler595f6302022-01-24 19:11:47 +00009920 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009921 0.14-r0/neard-0.14/include/nfc_copy.h include/near/nfc_copy.h
Andrew Geissler595f6302022-01-24 19:11:47 +00009922 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009923 0.14-r0/neard-0.14/include/snep.h include/near/snep.h
Andrew Geissler595f6302022-01-24 19:11:47 +00009924 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009925 0.14-r0/neard-0.14/include/version.h include/near/version.h
Andrew Geissler595f6302022-01-24 19:11:47 +00009926 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009927 0.14-r0/neard-0.14/include/dbus.h include/near/dbus.h
9928 | ./src/genbuiltin nfctype1 nfctype2 nfctype3 nfctype4 p2p > src/builtin.h
Andrew Geissler595f6302022-01-24 19:11:47 +00009929 | i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/pokybuild/yocto-autobuilder/nightly-x86/
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009930 build/build/tmp/sysroots/qemux86 -DHAVE_CONFIG_H -I. -I./include -I./src -I./gdbus -I/home/pokybuild/
Andrew Geissler595f6302022-01-24 19:11:47 +00009931 yocto-autobuilder/nightly-x86/build/build/tmp/sysroots/qemux86/usr/include/glib-2.0
9932 -I/home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/sysroots/qemux86/usr/
9933 lib/glib-2.0/include -I/home/pokybuild/yocto-autobuilder/nightly-x86/build/build/
9934 tmp/sysroots/qemux86/usr/include/dbus-1.0 -I/home/pokybuild/yocto-autobuilder/
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009935 nightly-x86/build/build/tmp/sysroots/qemux86/usr/lib/dbus-1.0/include -I/home/pokybuild/yocto-autobuilder/
Andrew Geissler595f6302022-01-24 19:11:47 +00009936 nightly-x86/build/build/tmp/sysroots/qemux86/usr/include/libnl3
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009937 -DNEAR_PLUGIN_BUILTIN -DPLUGINDIR=\""/usr/lib/near/plugins"\"
9938 -DCONFIGDIR=\""/etc/neard\"" -O2 -pipe -g -feliminate-unused-debug-types -c
9939 -o tools/snep-send.o tools/snep-send.c
9940 | In file included from tools/snep-send.c:16:0:
9941 | tools/../src/near.h:41:23: fatal error: near/dbus.h: No such file or directory
9942 | #include <near/dbus.h>
9943 | ^
9944 | compilation terminated.
9945 | make[1]: *** [tools/snep-send.o] Error 1
9946 | make[1]: *** Waiting for unfinished jobs....
9947 | make: *** [all] Error 2
9948 | ERROR: oe_runmake failed
9949
9950Reproducing the Error
9951~~~~~~~~~~~~~~~~~~~~~
9952
9953Because race conditions are intermittent, they do not manifest
9954themselves every time you do the build. In fact, most times the build
9955will complete without problems even though the potential race condition
9956exists. Thus, once the error surfaces, you need a way to reproduce it.
9957
9958In this example, compiling the "neard" package is causing the problem.
9959So the first thing to do is build "neard" locally. Before you start the
9960build, set the
9961:term:`PARALLEL_MAKE` variable
9962in your ``local.conf`` file to a high number (e.g. "-j 20"). Using a
Andrew Geissler09036742021-06-25 14:25:14 -05009963high value for :term:`PARALLEL_MAKE` increases the chances of the race
Andrew Geisslerc926e172021-05-07 16:11:35 -05009964condition showing up::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009965
9966 $ bitbake neard
9967
Andrew Geisslerc926e172021-05-07 16:11:35 -05009968Once the local build for "neard" completes, start a ``devshell`` build::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009969
9970 $ bitbake neard -c devshell
9971
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05009972For information on how to use a ``devshell``, see the
9973":ref:`dev-manual/common-tasks:using a development shell`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009974
Andrew Geisslerc926e172021-05-07 16:11:35 -05009975In the ``devshell``, do the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009976
9977 $ make clean
9978 $ make tools/snep-send.o
9979
9980The ``devshell`` commands cause the failure to clearly
William A. Kennington IIIac69b482021-06-02 12:28:27 -07009981be visible. In this case, there is a missing dependency for the ``neard``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009982Makefile target. Here is some abbreviated, sample output with the
Andrew Geisslerc926e172021-05-07 16:11:35 -05009983missing dependency clearly visible at the end::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009984
9985 i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/scott-lenovo/......
9986 .
9987 .
9988 .
9989 tools/snep-send.c
9990 In file included from tools/snep-send.c:16:0:
9991 tools/../src/near.h:41:23: fatal error: near/dbus.h: No such file or directory
9992 #include <near/dbus.h>
9993 ^
9994 compilation terminated.
9995 make: *** [tools/snep-send.o] Error 1
9996 $
9997
9998
9999Creating a Patch for the Fix
10000~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10001
10002Because there is a missing dependency for the Makefile target, you need
10003to patch the ``Makefile.am`` file, which is generated from
Andrew Geisslerc926e172021-05-07 16:11:35 -050010004``Makefile.in``. You can use Quilt to create the patch::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010005
10006 $ quilt new parallelmake.patch
10007 Patch patches/parallelmake.patch is now on top
10008 $ quilt add Makefile.am
10009 File Makefile.am added to patch patches/parallelmake.patch
10010
10011For more information on using Quilt, see the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -050010012":ref:`dev-manual/common-tasks:using quilt in your workflow`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010013
10014At this point you need to make the edits to ``Makefile.am`` to add the
10015missing dependency. For our example, you have to add the following line
Andrew Geisslerc926e172021-05-07 16:11:35 -050010016to the file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010017
10018 tools/snep-send.$(OBJEXT): include/near/dbus.h
10019
10020Once you have edited the file, use the ``refresh`` command to create the
Andrew Geisslerc926e172021-05-07 16:11:35 -050010021patch::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010022
10023 $ quilt refresh
10024 Refreshed patch patches/parallelmake.patch
10025
William A. Kennington IIIac69b482021-06-02 12:28:27 -070010026Once the patch file is created, you need to add it back to the originating
10027recipe folder. Here is an example assuming a top-level
Andrew Geisslerc926e172021-05-07 16:11:35 -050010028:term:`Source Directory` named ``poky``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010029
10030 $ cp patches/parallelmake.patch poky/meta/recipes-connectivity/neard/neard
10031
10032The final thing you need to do to implement the fix in the build is to
10033update the "neard" recipe (i.e. ``neard-0.14.bb``) so that the
10034:term:`SRC_URI` statement includes
10035the patch file. The recipe file is in the folder above the patch. Here
Andrew Geissler09036742021-06-25 14:25:14 -050010036is what the edited :term:`SRC_URI` statement would look like::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010037
10038 SRC_URI = "${KERNELORG_MIRROR}/linux/network/nfc/${BPN}-${PV}.tar.xz \
10039 file://neard.in \
10040 file://neard.service.in \
10041 file://parallelmake.patch \
10042 "
10043
10044With the patch complete and moved to the correct folder and the
Andrew Geissler09036742021-06-25 14:25:14 -050010045:term:`SRC_URI` statement updated, you can exit the ``devshell``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010046
10047 $ exit
10048
10049Testing the Build
10050~~~~~~~~~~~~~~~~~
10051
10052With everything in place, you can get back to trying the build again
Andrew Geisslerc926e172021-05-07 16:11:35 -050010053locally::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010054
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010055 $ bitbake neard
10056
10057This build should succeed.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010058
10059Now you can open up a ``devshell`` again and repeat the clean and make
Andrew Geisslerc926e172021-05-07 16:11:35 -050010060operations as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010061
10062 $ bitbake neard -c devshell
10063 $ make clean
10064 $ make tools/snep-send.o
10065
10066The build should work without issue.
10067
10068As with all solved problems, if they originated upstream, you need to
10069submit the fix for the recipe in OE-Core and upstream so that the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -050010070problem is taken care of at its source. See the
10071":ref:`dev-manual/common-tasks:submitting a change to the yocto project`"
10072section for more information.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010073
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010074Debugging With the GNU Project Debugger (GDB) Remotely
10075------------------------------------------------------
10076
10077GDB allows you to examine running programs, which in turn helps you to
10078understand and fix problems. It also allows you to perform post-mortem
10079style analysis of program crashes. GDB is available as a package within
10080the Yocto Project and is installed in SDK images by default. See the
Andrew Geissler09209ee2020-12-13 08:44:15 -060010081":ref:`ref-manual/images:Images`" chapter in the Yocto
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010082Project Reference Manual for a description of these images. You can find
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010083information on GDB at https://sourceware.org/gdb/.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010084
10085.. note::
10086
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010087 For best results, install debug (``-dbg``) packages for the applications you
10088 are going to debug. Doing so makes extra debug symbols available that give
10089 you more meaningful output.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010090
10091Sometimes, due to memory or disk space constraints, it is not possible
10092to use GDB directly on the remote target to debug applications. These
10093constraints arise because GDB needs to load the debugging information
10094and the binaries of the process being debugged. Additionally, GDB needs
10095to perform many computations to locate information such as function
Andrew Geissler615f2f12022-07-15 14:00:58 -050010096names, variable names and values, stack traces and so forth --- even
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010097before starting the debugging process. These extra computations place
10098more load on the target system and can alter the characteristics of the
10099program being debugged.
10100
Andrew Geissler95ac1b82021-03-31 14:34:31 -050010101To help get past the previously mentioned constraints, there are two
10102methods you can use: running a debuginfod server and using gdbserver.
10103
10104Using the debuginfod server method
10105~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10106
Andrew Geisslerc926e172021-05-07 16:11:35 -050010107``debuginfod`` from ``elfutils`` is a way to distribute ``debuginfo`` files.
10108Running a ``debuginfod`` server makes debug symbols readily available,
Andrew Geissler95ac1b82021-03-31 14:34:31 -050010109which means you don't need to download debugging information
10110and the binaries of the process being debugged. You can just fetch
10111debug symbols from the server.
10112
Andrew Geisslerc926e172021-05-07 16:11:35 -050010113To run a ``debuginfod`` server, you need to do the following:
Andrew Geissler95ac1b82021-03-31 14:34:31 -050010114
Andrew Geisslerc926e172021-05-07 16:11:35 -050010115- Ensure that ``debuginfod`` is present in :term:`DISTRO_FEATURES`
10116 (it already is in ``OpenEmbedded-core`` defaults and ``poky`` reference distribution).
10117 If not, set in your distro config file or in ``local.conf``::
Andrew Geissler95ac1b82021-03-31 14:34:31 -050010118
Patrick Williams0ca19cc2021-08-16 14:03:13 -050010119 DISTRO_FEATURES:append = " debuginfod"
Andrew Geissler95ac1b82021-03-31 14:34:31 -050010120
Andrew Geisslerc926e172021-05-07 16:11:35 -050010121 This distro feature enables the server and client library in ``elfutils``,
10122 and enables ``debuginfod`` support in clients (at the moment, ``gdb`` and ``binutils``).
Andrew Geissler95ac1b82021-03-31 14:34:31 -050010123
Andrew Geisslerc926e172021-05-07 16:11:35 -050010124- Run the following commands to launch the ``debuginfod`` server on the host::
Andrew Geissler95ac1b82021-03-31 14:34:31 -050010125
10126 $ oe-debuginfod
10127
Andrew Geisslerc926e172021-05-07 16:11:35 -050010128- To use ``debuginfod`` on the target, you need to know the ip:port where
10129 ``debuginfod`` is listening on the host (port defaults to 8002), and export
10130 that into the shell environment, for example in ``qemu``::
Andrew Geissler95ac1b82021-03-31 14:34:31 -050010131
Andrew Geisslerc926e172021-05-07 16:11:35 -050010132 root@qemux86-64:~# export DEBUGINFOD_URLS="http://192.168.7.1:8002/"
Andrew Geissler95ac1b82021-03-31 14:34:31 -050010133
Andrew Geisslerc926e172021-05-07 16:11:35 -050010134- Then debug info fetching should simply work when running the target ``gdb``,
10135 ``readelf`` or ``objdump``, for example::
Andrew Geissler95ac1b82021-03-31 14:34:31 -050010136
Andrew Geisslerc926e172021-05-07 16:11:35 -050010137 root@qemux86-64:~# gdb /bin/cat
10138 ...
10139 Reading symbols from /bin/cat...
10140 Downloading separate debug info for /bin/cat...
10141 Reading symbols from /home/root/.cache/debuginfod_client/923dc4780cfbc545850c616bffa884b6b5eaf322/debuginfo...
Andrew Geissler95ac1b82021-03-31 14:34:31 -050010142
Andrew Geisslerc926e172021-05-07 16:11:35 -050010143- It's also possible to use ``debuginfod-find`` to just query the server::
Andrew Geissler95ac1b82021-03-31 14:34:31 -050010144
Andrew Geisslerc926e172021-05-07 16:11:35 -050010145 root@qemux86-64:~# debuginfod-find debuginfo /bin/ls
10146 /home/root/.cache/debuginfod_client/356edc585f7f82d46f94fcb87a86a3fe2d2e60bd/debuginfo
Andrew Geissler95ac1b82021-03-31 14:34:31 -050010147
Andrew Geissler95ac1b82021-03-31 14:34:31 -050010148
10149Using the gdbserver method
10150~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10151
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010152gdbserver, which runs on the remote target and does not load any
10153debugging information from the debugged process. Instead, a GDB instance
10154processes the debugging information that is run on a remote computer -
10155the host GDB. The host GDB then sends control commands to gdbserver to
10156make it stop or start the debugged program, as well as read or write
10157memory regions of that debugged program. All the debugging information
10158loaded and processed as well as all the heavy debugging is done by the
10159host GDB. Offloading these processes gives the gdbserver running on the
10160target a chance to remain small and fast.
10161
10162Because the host GDB is responsible for loading the debugging
10163information and for doing the necessary processing to make actual
10164debugging happen, you have to make sure the host can access the
10165unstripped binaries complete with their debugging information and also
10166be sure the target is compiled with no optimizations. The host GDB must
10167also have local access to all the libraries used by the debugged
10168program. Because gdbserver does not need any local debugging
10169information, the binaries on the remote target can remain stripped.
10170However, the binaries must also be compiled without optimization so they
10171match the host's binaries.
10172
10173To remain consistent with GDB documentation and terminology, the binary
10174being debugged on the remote target machine is referred to as the
10175"inferior" binary. For documentation on GDB see the `GDB
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010176site <https://sourceware.org/gdb/documentation/>`__.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010177
10178The following steps show you how to debug using the GNU project
10179debugger.
10180
101811. *Configure your build system to construct the companion debug
10182 filesystem:*
10183
Andrew Geisslerc926e172021-05-07 16:11:35 -050010184 In your ``local.conf`` file, set the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010185
10186 IMAGE_GEN_DEBUGFS = "1"
10187 IMAGE_FSTYPES_DEBUGFS = "tar.bz2"
10188
10189 These options cause the
10190 OpenEmbedded build system to generate a special companion filesystem
10191 fragment, which contains the matching source and debug symbols to
10192 your deployable filesystem. The build system does this by looking at
10193 what is in the deployed filesystem, and pulling the corresponding
10194 ``-dbg`` packages.
10195
10196 The companion debug filesystem is not a complete filesystem, but only
10197 contains the debug fragments. This filesystem must be combined with
10198 the full filesystem for debugging. Subsequent steps in this procedure
10199 show how to combine the partial filesystem with the full filesystem.
10200
102012. *Configure the system to include gdbserver in the target filesystem:*
10202
Andrew Geisslerd5838332022-05-27 11:33:10 -050010203 Make the following addition in your ``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010204
Andrew Geisslerd5838332022-05-27 11:33:10 -050010205 EXTRA_IMAGE_FEATURES:append = " tools-debug"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010206
10207 The change makes
10208 sure the ``gdbserver`` package is included.
10209
102103. *Build the environment:*
10211
10212 Use the following command to construct the image and the companion
Andrew Geisslerc926e172021-05-07 16:11:35 -050010213 Debug Filesystem::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010214
10215 $ bitbake image
10216
10217 Build the cross GDB component and
10218 make it available for debugging. Build the SDK that matches the
10219 image. Building the SDK is best for a production build that can be
Andrew Geisslerc926e172021-05-07 16:11:35 -050010220 used later for debugging, especially during long term maintenance::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010221
10222 $ bitbake -c populate_sdk image
10223
10224 Alternatively, you can build the minimal toolchain components that
10225 match the target. Doing so creates a smaller than typical SDK and
10226 only contains a minimal set of components with which to build simple
Andrew Geisslerc926e172021-05-07 16:11:35 -050010227 test applications, as well as run the debugger::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010228
10229 $ bitbake meta-toolchain
10230
Andrew Geisslerc926e172021-05-07 16:11:35 -050010231 A final method is to build Gdb itself within the build system::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010232
10233 $ bitbake gdb-cross-<architecture>
10234
10235 Doing so produces a temporary copy of
10236 ``cross-gdb`` you can use for debugging during development. While
10237 this is the quickest approach, the two previous methods in this step
10238 are better when considering long-term maintenance strategies.
10239
10240 .. note::
10241
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010242 If you run ``bitbake gdb-cross``, the OpenEmbedded build system suggests
10243 the actual image (e.g. ``gdb-cross-i586``). The suggestion is usually the
10244 actual name you want to use.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010245
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500102464. *Set up the* ``debugfs``\ *:*
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010247
Andrew Geisslerc926e172021-05-07 16:11:35 -050010248 Run the following commands to set up the ``debugfs``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010249
10250 $ mkdir debugfs
10251 $ cd debugfs
Andrew Geisslerd5838332022-05-27 11:33:10 -050010252 $ tar xvfj build-dir/tmp/deploy/images/machine/image.rootfs.tar.bz2
10253 $ tar xvfj build-dir/tmp/deploy/images/machine/image-dbg.rootfs.tar.bz2
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010254
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500102555. *Set up GDB:*
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010256
10257 Install the SDK (if you built one) and then source the correct
10258 environment file. Sourcing the environment file puts the SDK in your
Andrew Geisslerd5838332022-05-27 11:33:10 -050010259 ``PATH`` environment variable and sets ``$GDB`` to the SDK's debugger.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010260
10261 If you are using the build system, Gdb is located in
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010262 `build-dir`\ ``/tmp/sysroots/``\ `host`\ ``/usr/bin/``\ `architecture`\ ``/``\ `architecture`\ ``-gdb``
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010263
102646. *Boot the target:*
10265
10266 For information on how to run QEMU, see the `QEMU
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010267 Documentation <https://wiki.qemu.org/Documentation/GettingStartedDevelopers>`__.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010268
10269 .. note::
10270
10271 Be sure to verify that your host can access the target via TCP.
10272
102737. *Debug a program:*
10274
10275 Debugging a program involves running gdbserver on the target and then
10276 running Gdb on the host. The example in this step debugs ``gzip``:
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010277
10278 .. code-block:: shell
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010279
10280 root@qemux86:~# gdbserver localhost:1234 /bin/gzip —help
10281
10282 For
10283 additional gdbserver options, see the `GDB Server
10284 Documentation <https://www.gnu.org/software/gdb/documentation/>`__.
10285
10286 After running gdbserver on the target, you need to run Gdb on the
Andrew Geisslerc926e172021-05-07 16:11:35 -050010287 host and configure it and connect to the target. Use these commands::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010288
10289 $ cd directory-holding-the-debugfs-directory
10290 $ arch-gdb
10291 (gdb) set sysroot debugfs
10292 (gdb) set substitute-path /usr/src/debug debugfs/usr/src/debug
10293 (gdb) target remote IP-of-target:1234
10294
10295 At this
10296 point, everything should automatically load (i.e. matching binaries,
10297 symbols and headers).
10298
10299 .. note::
10300
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010301 The Gdb ``set`` commands in the previous example can be placed into the
10302 users ``~/.gdbinit`` file. Upon starting, Gdb automatically runs whatever
10303 commands are in that file.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010304
103058. *Deploying without a full image rebuild:*
10306
10307 In many cases, during development you want a quick method to deploy a
10308 new binary to the target and debug it, without waiting for a full
10309 image build.
10310
10311 One approach to solving this situation is to just build the component
10312 you want to debug. Once you have built the component, copy the
10313 executable directly to both the target and the host ``debugfs``.
10314
10315 If the binary is processed through the debug splitting in
10316 OpenEmbedded, you should also copy the debug items (i.e. ``.debug``
10317 contents and corresponding ``/usr/src/debug`` files) from the work
Andrew Geisslerc926e172021-05-07 16:11:35 -050010318 directory. Here is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010319
10320 $ bitbake bash
10321 $ bitbake -c devshell bash
10322 $ cd ..
10323 $ scp packages-split/bash/bin/bash target:/bin/bash
10324 $ cp -a packages-split/bash-dbg/\* path/debugfs
10325
10326Debugging with the GNU Project Debugger (GDB) on the Target
10327-----------------------------------------------------------
10328
10329The previous section addressed using GDB remotely for debugging
10330purposes, which is the most usual case due to the inherent hardware
10331limitations on many embedded devices. However, debugging in the target
10332hardware itself is also possible with more powerful devices. This
10333section describes what you need to do in order to support using GDB to
10334debug on the target hardware.
10335
10336To support this kind of debugging, you need do the following:
10337
Andrew Geisslerd5838332022-05-27 11:33:10 -050010338- Ensure that GDB is on the target. You can do this by making
10339 the following addition to your ``local.conf`` file::
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010340
Andrew Geisslerd5838332022-05-27 11:33:10 -050010341 EXTRA_IMAGE_FEATURES:append = " tools-debug"
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010342
Andrew Geisslerd5838332022-05-27 11:33:10 -050010343- Ensure that debug symbols are present. You can do so by adding the
10344 corresponding ``-dbg`` package to :term:`IMAGE_INSTALL`::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010345
Andrew Geisslerd5838332022-05-27 11:33:10 -050010346 IMAGE_INSTALL:append = " packagename-dbg"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010347
Andrew Geisslerd5838332022-05-27 11:33:10 -050010348 Alternatively, you can add the following to ``local.conf`` to include
Andrew Geisslerc926e172021-05-07 16:11:35 -050010349 all the debug symbols::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010350
Andrew Geisslerd5838332022-05-27 11:33:10 -050010351 EXTRA_IMAGE_FEATURES:append = " dbg-pkgs"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010352
10353.. note::
10354
10355 To improve the debug information accuracy, you can reduce the level
10356 of optimization used by the compiler. For example, when adding the
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010357 following line to your ``local.conf`` file, you will reduce optimization
10358 from :term:`FULL_OPTIMIZATION` of "-O2" to :term:`DEBUG_OPTIMIZATION`
Andrew Geisslerc926e172021-05-07 16:11:35 -050010359 of "-O -fno-omit-frame-pointer"::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010360
10361 DEBUG_BUILD = "1"
10362
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010363 Consider that this will reduce the application's performance and is
10364 recommended only for debugging purposes.
10365
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010366Other Debugging Tips
10367--------------------
10368
10369Here are some other tips that you might find useful:
10370
10371- When adding new packages, it is worth watching for undesirable items
10372 making their way into compiler command lines. For example, you do not
10373 want references to local system files like ``/usr/lib/`` or
10374 ``/usr/include/``.
10375
10376- If you want to remove the ``psplash`` boot splashscreen, add
10377 ``psplash=false`` to the kernel command line. Doing so prevents
10378 ``psplash`` from loading and thus allows you to see the console. It
10379 is also possible to switch out of the splashscreen by switching the
10380 virtual console (e.g. Fn+Left or Fn+Right on a Zaurus).
10381
10382- Removing :term:`TMPDIR` (usually
10383 ``tmp/``, within the
10384 :term:`Build Directory`) can often fix
Andrew Geissler09036742021-06-25 14:25:14 -050010385 temporary build issues. Removing :term:`TMPDIR` is usually a relatively
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010386 cheap operation, because task output will be cached in
10387 :term:`SSTATE_DIR` (usually
10388 ``sstate-cache/``, which is also in the Build Directory).
10389
10390 .. note::
10391
Andrew Geissler09036742021-06-25 14:25:14 -050010392 Removing :term:`TMPDIR` might be a workaround rather than a fix.
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010393 Consequently, trying to determine the underlying cause of an issue before
10394 removing the directory is a good idea.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010395
10396- Understanding how a feature is used in practice within existing
10397 recipes can be very helpful. It is recommended that you configure
10398 some method that allows you to quickly search through files.
10399
10400 Using GNU Grep, you can use the following shell function to
10401 recursively search through common recipe-related files, skipping
10402 binary files, ``.git`` directories, and the Build Directory (assuming
Andrew Geisslerc926e172021-05-07 16:11:35 -050010403 its name starts with "build")::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010404
10405 g() {
10406 grep -Ir \
10407 --exclude-dir=.git \
10408 --exclude-dir='build*' \
10409 --include='*.bb*' \
10410 --include='*.inc*' \
10411 --include='*.conf*' \
10412 --include='*.py*' \
10413 "$@"
10414 }
10415
Andrew Geisslerc926e172021-05-07 16:11:35 -050010416 Following are some usage examples::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010417
10418 $ g FOO # Search recursively for "FOO"
10419 $ g -i foo # Search recursively for "foo", ignoring case
10420 $ g -w FOO # Search recursively for "FOO" as a word, ignoring e.g. "FOOBAR"
10421
10422 If figuring
10423 out how some feature works requires a lot of searching, it might
10424 indicate that the documentation should be extended or improved. In
10425 such cases, consider filing a documentation bug using the Yocto
10426 Project implementation of
10427 :yocto_bugs:`Bugzilla <>`. For information on
10428 how to submit a bug against the Yocto Project, see the Yocto Project
Andrew Geissler09209ee2020-12-13 08:44:15 -060010429 Bugzilla :yocto_wiki:`wiki page </Bugzilla_Configuration_and_Bug_Tracking>`
Andrew Geissler3b8a17c2021-04-15 15:55:55 -050010430 and the
10431 ":ref:`dev-manual/common-tasks:submitting a defect against the yocto project`"
10432 section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010433
10434 .. note::
10435
10436 The manuals might not be the right place to document variables
10437 that are purely internal and have a limited scope (e.g. internal
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010438 variables used to implement a single ``.bbclass`` file).
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010439
10440Making Changes to the Yocto Project
10441===================================
10442
10443Because the Yocto Project is an open-source, community-based project,
10444you can effect changes to the project. This section presents procedures
10445that show you how to submit a defect against the project and how to
10446submit a change.
10447
10448Submitting a Defect Against the Yocto Project
10449---------------------------------------------
10450
10451Use the Yocto Project implementation of
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010452`Bugzilla <https://www.bugzilla.org/about/>`__ to submit a defect (bug)
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010453against the Yocto Project. For additional information on this
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010454implementation of Bugzilla see the ":ref:`Yocto Project
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010455Bugzilla <resources-bugtracker>`" section in the
10456Yocto Project Reference Manual. For more detail on any of the following
10457steps, see the Yocto Project
Andrew Geissler09209ee2020-12-13 08:44:15 -060010458:yocto_wiki:`Bugzilla wiki page </Bugzilla_Configuration_and_Bug_Tracking>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010459
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010460Use the following general steps to submit a bug:
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010461
104621. Open the Yocto Project implementation of :yocto_bugs:`Bugzilla <>`.
10463
104642. Click "File a Bug" to enter a new bug.
10465
104663. Choose the appropriate "Classification", "Product", and "Component"
10467 for which the bug was found. Bugs for the Yocto Project fall into
10468 one of several classifications, which in turn break down into
10469 several products and components. For example, for a bug against the
10470 ``meta-intel`` layer, you would choose "Build System, Metadata &
10471 Runtime", "BSPs", and "bsps-meta-intel", respectively.
10472
104734. Choose the "Version" of the Yocto Project for which you found the
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010474 bug (e.g. &DISTRO;).
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010475
104765. Determine and select the "Severity" of the bug. The severity
10477 indicates how the bug impacted your work.
10478
104796. Choose the "Hardware" that the bug impacts.
10480
104817. Choose the "Architecture" that the bug impacts.
10482
104838. Choose a "Documentation change" item for the bug. Fixing a bug might
10484 or might not affect the Yocto Project documentation. If you are
10485 unsure of the impact to the documentation, select "Don't Know".
10486
104879. Provide a brief "Summary" of the bug. Try to limit your summary to
10488 just a line or two and be sure to capture the essence of the bug.
10489
1049010. Provide a detailed "Description" of the bug. You should provide as
10491 much detail as you can about the context, behavior, output, and so
10492 forth that surrounds the bug. You can even attach supporting files
10493 for output from logs by using the "Add an attachment" button.
10494
1049511. Click the "Submit Bug" button submit the bug. A new Bugzilla number
10496 is assigned to the bug and the defect is logged in the bug tracking
10497 system.
10498
10499Once you file a bug, the bug is processed by the Yocto Project Bug
10500Triage Team and further details concerning the bug are assigned (e.g.
10501priority and owner). You are the "Submitter" of the bug and any further
10502categorization, progress, or comments on the bug result in Bugzilla
10503sending you an automated email concerning the particular change or
10504progress to the bug.
10505
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010506Submitting a Change to the Yocto Project
10507----------------------------------------
10508
10509Contributions to the Yocto Project and OpenEmbedded are very welcome.
10510Because the system is extremely configurable and flexible, we recognize
10511that developers will want to extend, configure or optimize it for their
10512specific uses.
10513
10514The Yocto Project uses a mailing list and a patch-based workflow that is
10515similar to the Linux kernel but contains important differences. In
William A. Kennington IIIac69b482021-06-02 12:28:27 -070010516general, there is a mailing list through which you can submit patches. You
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010517should send patches to the appropriate mailing list so that they can be
10518reviewed and merged by the appropriate maintainer. The specific mailing
10519list you need to use depends on the location of the code you are
10520changing. Each component (e.g. layer) should have a ``README`` file that
10521indicates where to send the changes and which process to follow.
10522
10523You can send the patch to the mailing list using whichever approach you
10524feel comfortable with to generate the patch. Once sent, the patch is
10525usually reviewed by the community at large. If somebody has concerns
10526with the patch, they will usually voice their concern over the mailing
10527list. If a patch does not receive any negative reviews, the maintainer
10528of the affected layer typically takes the patch, tests it, and then
10529based on successful testing, merges the patch.
10530
10531The "poky" repository, which is the Yocto Project's reference build
10532environment, is a hybrid repository that contains several individual
10533pieces (e.g. BitBake, Metadata, documentation, and so forth) built using
10534the combo-layer tool. The upstream location used for submitting changes
10535varies by component:
10536
10537- *Core Metadata:* Send your patch to the
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010538 :oe_lists:`openembedded-core </g/openembedded-core>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010539 mailing list. For example, a change to anything under the ``meta`` or
10540 ``scripts`` directories should be sent to this mailing list.
10541
10542- *BitBake:* For changes to BitBake (i.e. anything under the
10543 ``bitbake`` directory), send your patch to the
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010544 :oe_lists:`bitbake-devel </g/bitbake-devel>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010545 mailing list.
10546
Andrew Geisslerc3d88e42020-10-02 09:45:00 -050010547- *"meta-\*" trees:* These trees contain Metadata. Use the
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010548 :yocto_lists:`poky </g/poky>` mailing list.
Andrew Geisslerc3d88e42020-10-02 09:45:00 -050010549
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010550- *Documentation*: For changes to the Yocto Project documentation, use the
10551 :yocto_lists:`docs </g/docs>` mailing list.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010552
10553For changes to other layers hosted in the Yocto Project source
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010554repositories (i.e. ``yoctoproject.org``) and tools use the
10555:yocto_lists:`Yocto Project </g/yocto/>` general mailing list.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010556
10557.. note::
10558
10559 Sometimes a layer's documentation specifies to use a particular
10560 mailing list. If so, use that list.
10561
10562For additional recipes that do not fit into the core Metadata, you
10563should determine which layer the recipe should go into and submit the
10564change in the manner recommended by the documentation (e.g. the
10565``README`` file) supplied with the layer. If in doubt, please ask on the
10566Yocto general mailing list or on the openembedded-devel mailing list.
10567
10568You can also push a change upstream and request a maintainer to pull the
10569change into the component's upstream repository. You do this by pushing
Andrew Geissler09209ee2020-12-13 08:44:15 -060010570to a contribution repository that is upstream. See the
10571":ref:`overview-manual/development-environment:git workflows and the yocto project`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010572section in the Yocto Project Overview and Concepts Manual for additional
10573concepts on working in the Yocto Project development environment.
10574
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010575Maintainers commonly use ``-next`` branches to test submissions prior to
10576merging patches. Thus, you can get an idea of the status of a patch based on
10577whether the patch has been merged into one of these branches. The commonly
10578used testing branches for OpenEmbedded-Core are as follows:
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010579
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010580- *openembedded-core "master-next" branch:* This branch is part of the
10581 :oe_git:`openembedded-core </openembedded-core/>` repository and contains
10582 proposed changes to the core metadata.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010583
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010584- *poky "master-next" branch:* This branch is part of the
Andrew Geissler09209ee2020-12-13 08:44:15 -060010585 :yocto_git:`poky </poky/>` repository and combines proposed
Andrew Geisslerd5838332022-05-27 11:33:10 -050010586 changes to BitBake, the core metadata and the poky distro.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010587
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010588Similarly, stable branches maintained by the project may have corresponding
10589``-next`` branches which collect proposed changes. For example,
10590``&DISTRO_NAME_NO_CAP;-next`` and ``&DISTRO_NAME_NO_CAP_MINUS_ONE;-next``
10591branches in both the "openembdedded-core" and "poky" repositories.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010592
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010593Other layers may have similar testing branches but there is no formal
10594requirement or standard for these so please check the documentation for the
10595layers you are contributing to.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010596
10597The following sections provide procedures for submitting a change.
10598
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010599Preparing Changes for Submission
10600~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010601
106021. *Make Your Changes Locally:* Make your changes in your local Git
10603 repository. You should make small, controlled, isolated changes.
10604 Keeping changes small and isolated aids review, makes
10605 merging/rebasing easier and keeps the change history clean should
10606 anyone need to refer to it in future.
10607
106082. *Stage Your Changes:* Stage your changes by using the ``git add``
10609 command on each file you changed.
10610
106113. *Commit Your Changes:* Commit the change by using the ``git commit``
10612 command. Make sure your commit information follows standards by
10613 following these accepted conventions:
10614
10615 - Be sure to include a "Signed-off-by:" line in the same style as
Patrick Williams03907ee2022-05-01 06:28:52 -050010616 required by the Linux kernel. This can be done by using the
10617 ``git commit -s`` command. Adding this line signifies that you,
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010618 the submitter, have agreed to the Developer's Certificate of
10619 Origin 1.1 as follows:
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010620
10621 .. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010622
10623 Developer's Certificate of Origin 1.1
10624
10625 By making a contribution to this project, I certify that:
10626
10627 (a) The contribution was created in whole or in part by me and I
10628 have the right to submit it under the open source license
10629 indicated in the file; or
10630
10631 (b) The contribution is based upon previous work that, to the best
10632 of my knowledge, is covered under an appropriate open source
10633 license and I have the right under that license to submit that
10634 work with modifications, whether created in whole or in part
10635 by me, under the same open source license (unless I am
10636 permitted to submit under a different license), as indicated
10637 in the file; or
10638
10639 (c) The contribution was provided directly to me by some other
10640 person who certified (a), (b) or (c) and I have not modified
10641 it.
10642
10643 (d) I understand and agree that this project and the contribution
10644 are public and that a record of the contribution (including all
10645 personal information I submit with it, including my sign-off) is
10646 maintained indefinitely and may be redistributed consistent with
10647 this project or the open source license(s) involved.
10648
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010649 - Provide a single-line summary of the change and, if more
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010650 explanation is needed, provide more detail in the body of the
10651 commit. This summary is typically viewable in the "shortlist" of
10652 changes. Thus, providing something short and descriptive that
10653 gives the reader a summary of the change is useful when viewing a
10654 list of many commits. You should prefix this short description
10655 with the recipe name (if changing a recipe), or else with the
10656 short form path to the file being changed.
10657
10658 - For the body of the commit message, provide detailed information
10659 that describes what you changed, why you made the change, and the
10660 approach you used. It might also be helpful if you mention how you
10661 tested the change. Provide as much detail as you can in the body
10662 of the commit message.
10663
10664 .. note::
10665
10666 You do not need to provide a more detailed explanation of a
10667 change if the change is minor to the point of the single line
10668 summary providing all the information.
10669
10670 - If the change addresses a specific bug or issue that is associated
10671 with a bug-tracking ID, include a reference to that ID in your
10672 detailed description. For example, the Yocto Project uses a
Andrew Geissler615f2f12022-07-15 14:00:58 -050010673 specific convention for bug references --- any commit that addresses
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010674 a specific bug should use the following form for the detailed
10675 description. Be sure to use the actual bug-tracking ID from
Andrew Geisslerc926e172021-05-07 16:11:35 -050010676 Bugzilla for bug-id::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010677
10678 Fixes [YOCTO #bug-id]
10679
10680 detailed description of change
10681
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010682Using Email to Submit a Patch
10683~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10684
10685Depending on the components changed, you need to submit the email to a
10686specific mailing list. For some guidance on which mailing list to use,
Andrew Geissler3b8a17c2021-04-15 15:55:55 -050010687see the
10688:ref:`list <dev-manual/common-tasks:submitting a change to the yocto project>`
10689at the beginning of this section. For a description of all the available
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010690mailing lists, see the ":ref:`Mailing Lists <resources-mailinglist>`" section in the
10691Yocto Project Reference Manual.
10692
10693Here is the general procedure on how to submit a patch through email
10694without using the scripts once the steps in
Andrew Geissler09209ee2020-12-13 08:44:15 -060010695:ref:`dev-manual/common-tasks:preparing changes for submission` have been followed:
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010696
106971. *Format the Commit:* Format the commit into an email message. To
10698 format commits, use the ``git format-patch`` command. When you
10699 provide the command, you must include a revision list or a number of
10700 patches as part of the command. For example, either of these two
10701 commands takes your most recent single commit and formats it as an
Andrew Geisslerc926e172021-05-07 16:11:35 -050010702 email message in the current directory::
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010703
10704 $ git format-patch -1
10705
10706 or ::
10707
10708 $ git format-patch HEAD~
10709
10710 After the command is run, the current directory contains a numbered
10711 ``.patch`` file for the commit.
10712
10713 If you provide several commits as part of the command, the
10714 ``git format-patch`` command produces a series of numbered files in
10715 the current directory – one for each commit. If you have more than
10716 one patch, you should also use the ``--cover`` option with the
10717 command, which generates a cover letter as the first "patch" in the
10718 series. You can then edit the cover letter to provide a description
10719 for the series of patches. For information on the
10720 ``git format-patch`` command, see ``GIT_FORMAT_PATCH(1)`` displayed
10721 using the ``man git-format-patch`` command.
10722
10723 .. note::
10724
10725 If you are or will be a frequent contributor to the Yocto Project
10726 or to OpenEmbedded, you might consider requesting a contrib area
10727 and the necessary associated rights.
10728
107292. *Send the patches via email:* Send the patches to the recipients and
10730 relevant mailing lists by using the ``git send-email`` command.
10731
10732 .. note::
10733
10734 In order to use ``git send-email``, you must have the proper Git packages
10735 installed on your host.
10736 For Ubuntu, Debian, and Fedora the package is ``git-email``.
10737
10738 The ``git send-email`` command sends email by using a local or remote
10739 Mail Transport Agent (MTA) such as ``msmtp``, ``sendmail``, or
10740 through a direct ``smtp`` configuration in your Git ``~/.gitconfig``
10741 file. If you are submitting patches through email only, it is very
10742 important that you submit them without any whitespace or HTML
10743 formatting that either you or your mailer introduces. The maintainer
10744 that receives your patches needs to be able to save and apply them
10745 directly from your emails. A good way to verify that what you are
10746 sending will be applicable by the maintainer is to do a dry run and
10747 send them to yourself and then save and apply them as the maintainer
10748 would.
10749
10750 The ``git send-email`` command is the preferred method for sending
10751 your patches using email since there is no risk of compromising
10752 whitespace in the body of the message, which can occur when you use
10753 your own mail client. The command also has several options that let
10754 you specify recipients and perform further editing of the email
10755 message. For information on how to use the ``git send-email``
10756 command, see ``GIT-SEND-EMAIL(1)`` displayed using the
10757 ``man git-send-email`` command.
10758
10759The Yocto Project uses a `Patchwork instance <https://patchwork.openembedded.org/>`__
10760to track the status of patches submitted to the various mailing lists and to
10761support automated patch testing. Each submitted patch is checked for common
10762mistakes and deviations from the expected patch format and submitters are
10763notified by patchtest if such mistakes are found. This process helps to
10764reduce the burden of patch review on maintainers.
10765
10766.. note::
10767
10768 This system is imperfect and changes can sometimes get lost in the flow.
10769 Asking about the status of a patch or change is reasonable if the change
10770 has been idle for a while with no feedback.
10771
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010772Using Scripts to Push a Change Upstream and Request a Pull
10773~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10774
10775For larger patch series it is preferable to send a pull request which not
10776only includes the patch but also a pointer to a branch that can be pulled
10777from. This involves making a local branch for your changes, pushing this
10778branch to an accessible repository and then using the ``create-pull-request``
10779and ``send-pull-request`` scripts from openembedded-core to create and send a
10780patch series with a link to the branch for review.
10781
10782Follow this procedure to push a change to an upstream "contrib" Git
Andrew Geissler09209ee2020-12-13 08:44:15 -060010783repository once the steps in :ref:`dev-manual/common-tasks:preparing changes for submission` have
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010784been followed:
10785
10786.. note::
10787
10788 You can find general Git information on how to push a change upstream
10789 in the
10790 `Git Community Book <https://git-scm.com/book/en/v2/Distributed-Git-Distributed-Workflows>`__.
10791
107921. *Push Your Commits to a "Contrib" Upstream:* If you have arranged for
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010793 permissions to push to an upstream contrib repository, push the
Andrew Geisslerc926e172021-05-07 16:11:35 -050010794 change to that repository::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010795
10796 $ git push upstream_remote_repo local_branch_name
10797
10798 For example, suppose you have permissions to push
10799 into the upstream ``meta-intel-contrib`` repository and you are
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010800 working in a local branch named `your_name`\ ``/README``. The following
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010801 command pushes your local commits to the ``meta-intel-contrib``
10802 upstream repository and puts the commit in a branch named
Andrew Geisslerc926e172021-05-07 16:11:35 -050010803 `your_name`\ ``/README``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010804
10805 $ git push meta-intel-contrib your_name/README
10806
Andrew Geissler6ce62a22020-11-30 19:58:47 -0600108072. *Determine Who to Notify:* Determine the maintainer or the mailing
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010808 list that you need to notify for the change.
10809
10810 Before submitting any change, you need to be sure who the maintainer
10811 is or what mailing list that you need to notify. Use either these
10812 methods to find out:
10813
10814 - *Maintenance File:* Examine the ``maintainers.inc`` file, which is
10815 located in the :term:`Source Directory` at
10816 ``meta/conf/distro/include``, to see who is responsible for code.
10817
Andrew Geissler09209ee2020-12-13 08:44:15 -060010818 - *Search by File:* Using :ref:`overview-manual/development-environment:git`, you can
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010819 enter the following command to bring up a short list of all
Andrew Geisslerc926e172021-05-07 16:11:35 -050010820 commits against a specific file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010821
10822 git shortlog -- filename
10823
10824 Just provide the name of the file for which you are interested. The
10825 information returned is not ordered by history but does include a
10826 list of everyone who has committed grouped by name. From the list,
10827 you can see who is responsible for the bulk of the changes against
10828 the file.
10829
10830 - *Examine the List of Mailing Lists:* For a list of the Yocto
10831 Project and related mailing lists, see the ":ref:`Mailing
10832 lists <resources-mailinglist>`" section in
10833 the Yocto Project Reference Manual.
10834
Andrew Geissler6ce62a22020-11-30 19:58:47 -0600108353. *Make a Pull Request:* Notify the maintainer or the mailing list that
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010836 you have pushed a change by making a pull request.
10837
10838 The Yocto Project provides two scripts that conveniently let you
10839 generate and send pull requests to the Yocto Project. These scripts
10840 are ``create-pull-request`` and ``send-pull-request``. You can find
10841 these scripts in the ``scripts`` directory within the
10842 :term:`Source Directory` (e.g.
Andrew Geissler95ac1b82021-03-31 14:34:31 -050010843 ``poky/scripts``).
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010844
10845 Using these scripts correctly formats the requests without
10846 introducing any whitespace or HTML formatting. The maintainer that
10847 receives your patches either directly or through the mailing list
10848 needs to be able to save and apply them directly from your emails.
10849 Using these scripts is the preferred method for sending patches.
10850
10851 First, create the pull request. For example, the following command
10852 runs the script, specifies the upstream repository in the contrib
10853 directory into which you pushed the change, and provides a subject
Andrew Geisslerc926e172021-05-07 16:11:35 -050010854 line in the created patch files::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010855
Andrew Geissler95ac1b82021-03-31 14:34:31 -050010856 $ poky/scripts/create-pull-request -u meta-intel-contrib -s "Updated Manual Section Reference in README"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010857
10858 Running this script forms ``*.patch`` files in a folder named
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010859 ``pull-``\ `PID` in the current directory. One of the patch files is a
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010860 cover letter.
10861
10862 Before running the ``send-pull-request`` script, you must edit the
10863 cover letter patch to insert information about your change. After
10864 editing the cover letter, send the pull request. For example, the
10865 following command runs the script and specifies the patch directory
10866 and email address. In this example, the email address is a mailing
Andrew Geisslerc926e172021-05-07 16:11:35 -050010867 list::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010868
Andrew Geissler5199d832021-09-24 16:47:35 -050010869 $ poky/scripts/send-pull-request -p ~/meta-intel/pull-10565 -t meta-intel@lists.yoctoproject.org
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010870
10871 You need to follow the prompts as the script is interactive.
10872
10873 .. note::
10874
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010875 For help on using these scripts, simply provide the ``-h``
Andrew Geisslerc926e172021-05-07 16:11:35 -050010876 argument as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010877
10878 $ poky/scripts/create-pull-request -h
10879 $ poky/scripts/send-pull-request -h
10880
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010881Responding to Patch Review
10882~~~~~~~~~~~~~~~~~~~~~~~~~~
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010883
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010884You may get feedback on your submitted patches from other community members
10885or from the automated patchtest service. If issues are identified in your
10886patch then it is usually necessary to address these before the patch will be
10887accepted into the project. In this case you should amend the patch according
10888to the feedback and submit an updated version to the relevant mailing list,
10889copying in the reviewers who provided feedback to the previous version of the
10890patch.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010891
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010892The patch should be amended using ``git commit --amend`` or perhaps ``git
10893rebase`` for more expert git users. You should also modify the ``[PATCH]``
10894tag in the email subject line when sending the revised patch to mark the new
10895iteration as ``[PATCH v2]``, ``[PATCH v3]``, etc as appropriate. This can be
10896done by passing the ``-v`` argument to ``git format-patch`` with a version
10897number.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010898
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010899Lastly please ensure that you also test your revised changes. In particular
10900please don't just edit the patch file written out by ``git format-patch`` and
10901resend it.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010902
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010903Submitting Changes to Stable Release Branches
10904~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010905
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010906The process for proposing changes to a Yocto Project stable branch differs
10907from the steps described above. Changes to a stable branch must address
10908identified bugs or CVEs and should be made carefully in order to avoid the
10909risk of introducing new bugs or breaking backwards compatibility. Typically
10910bug fixes must already be accepted into the master branch before they can be
10911backported to a stable branch unless the bug in question does not affect the
10912master branch or the fix on the master branch is unsuitable for backporting.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010913
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010914The list of stable branches along with the status and maintainer for each
10915branch can be obtained from the
Andrew Geissler09209ee2020-12-13 08:44:15 -060010916:yocto_wiki:`Releases wiki page </Releases>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010917
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010918.. note::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010919
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010920 Changes will not typically be accepted for branches which are marked as
10921 End-Of-Life (EOL).
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010922
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010923With this in mind, the steps to submit a change for a stable branch are as
10924follows:
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010925
Andrew Geissler6ce62a22020-11-30 19:58:47 -0600109261. *Identify the bug or CVE to be fixed:* This information should be
10927 collected so that it can be included in your submission.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010928
Patrick Williams213cb262021-08-07 19:21:33 -050010929 See :ref:`dev-manual/common-tasks:checking for vulnerabilities`
10930 for details about CVE tracking.
10931
Andrew Geissler6ce62a22020-11-30 19:58:47 -0600109322. *Check if the fix is already present in the master branch:* This will
10933 result in the most straightforward path into the stable branch for the
10934 fix.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010935
Andrew Geissler615f2f12022-07-15 14:00:58 -050010936 a. *If the fix is present in the master branch --- submit a backport request
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010937 by email:* You should send an email to the relevant stable branch
10938 maintainer and the mailing list with details of the bug or CVE to be
10939 fixed, the commit hash on the master branch that fixes the issue and
10940 the stable branches which you would like this fix to be backported to.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010941
Andrew Geissler615f2f12022-07-15 14:00:58 -050010942 b. *If the fix is not present in the master branch --- submit the fix to the
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010943 master branch first:* This will ensure that the fix passes through the
10944 project's usual patch review and test processes before being accepted.
10945 It will also ensure that bugs are not left unresolved in the master
10946 branch itself. Once the fix is accepted in the master branch a backport
10947 request can be submitted as above.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010948
Andrew Geissler615f2f12022-07-15 14:00:58 -050010949 c. *If the fix is unsuitable for the master branch --- submit a patch
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010950 directly for the stable branch:* This method should be considered as a
10951 last resort. It is typically necessary when the master branch is using
10952 a newer version of the software which includes an upstream fix for the
10953 issue or when the issue has been fixed on the master branch in a way
10954 that introduces backwards incompatible changes. In this case follow the
Andrew Geissler09209ee2020-12-13 08:44:15 -060010955 steps in :ref:`dev-manual/common-tasks:preparing changes for submission` and
10956 :ref:`dev-manual/common-tasks:using email to submit a patch` but modify the subject header of your patch
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010957 email to include the name of the stable branch which you are
10958 targetting. This can be done using the ``--subject-prefix`` argument to
10959 ``git format-patch``, for example to submit a patch to the dunfell
10960 branch use
10961 ``git format-patch --subject-prefix='&DISTRO_NAME_NO_CAP_MINUS_ONE;][PATCH' ...``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010962
10963Working With Licenses
10964=====================
10965
Andrew Geissler09209ee2020-12-13 08:44:15 -060010966As mentioned in the ":ref:`overview-manual/development-environment:licensing`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010967section in the Yocto Project Overview and Concepts Manual, open source
10968projects are open to the public and they consequently have different
10969licensing structures in place. This section describes the mechanism by
10970which the :term:`OpenEmbedded Build System`
10971tracks changes to
10972licensing text and covers how to maintain open source license compliance
10973during your project's lifecycle. The section also describes how to
10974enable commercially licensed recipes, which by default are disabled.
10975
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010976Tracking License Changes
10977------------------------
10978
10979The license of an upstream project might change in the future. In order
10980to prevent these changes going unnoticed, the
10981:term:`LIC_FILES_CHKSUM`
10982variable tracks changes to the license text. The checksums are validated
10983at the end of the configure step, and if the checksums do not match, the
10984build will fail.
10985
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010986Specifying the ``LIC_FILES_CHKSUM`` Variable
10987~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10988
Andrew Geissler09036742021-06-25 14:25:14 -050010989The :term:`LIC_FILES_CHKSUM` variable contains checksums of the license text
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010990in the source code for the recipe. Following is an example of how to
Andrew Geissler09036742021-06-25 14:25:14 -050010991specify :term:`LIC_FILES_CHKSUM`::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010992
10993 LIC_FILES_CHKSUM = "file://COPYING;md5=xxxx \
10994 file://licfile1.txt;beginline=5;endline=29;md5=yyyy \
10995 file://licfile2.txt;endline=50;md5=zzzz \
10996 ..."
10997
10998.. note::
10999
11000 - When using "beginline" and "endline", realize that line numbering
11001 begins with one and not zero. Also, the included lines are
11002 inclusive (i.e. lines five through and including 29 in the
11003 previous example for ``licfile1.txt``).
11004
11005 - When a license check fails, the selected license text is included
11006 as part of the QA message. Using this output, you can determine
11007 the exact start and finish for the needed license text.
11008
11009The build system uses the :term:`S`
11010variable as the default directory when searching files listed in
Andrew Geissler09036742021-06-25 14:25:14 -050011011:term:`LIC_FILES_CHKSUM`. The previous example employs the default
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011012directory.
11013
Andrew Geisslerc926e172021-05-07 16:11:35 -050011014Consider this next example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011015
11016 LIC_FILES_CHKSUM = "file://src/ls.c;beginline=5;endline=16;\
11017 md5=bb14ed3c4cda583abc85401304b5cd4e"
11018 LIC_FILES_CHKSUM = "file://${WORKDIR}/license.html;md5=5c94767cedb5d6987c902ac850ded2c6"
11019
11020The first line locates a file in ``${S}/src/ls.c`` and isolates lines
11021five through 16 as license text. The second line refers to a file in
11022:term:`WORKDIR`.
11023
Andrew Geissler09036742021-06-25 14:25:14 -050011024Note that :term:`LIC_FILES_CHKSUM` variable is mandatory for all recipes,
11025unless the :term:`LICENSE` variable is set to "CLOSED".
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011026
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011027Explanation of Syntax
11028~~~~~~~~~~~~~~~~~~~~~
11029
Andrew Geissler09036742021-06-25 14:25:14 -050011030As mentioned in the previous section, the :term:`LIC_FILES_CHKSUM` variable
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011031lists all the important files that contain the license text for the
11032source code. It is possible to specify a checksum for an entire file, or
11033a specific section of a file (specified by beginning and ending line
11034numbers with the "beginline" and "endline" parameters, respectively).
11035The latter is useful for source files with a license notice header,
11036README documents, and so forth. If you do not use the "beginline"
11037parameter, then it is assumed that the text begins on the first line of
11038the file. Similarly, if you do not use the "endline" parameter, it is
11039assumed that the license text ends with the last line of the file.
11040
11041The "md5" parameter stores the md5 checksum of the license text. If the
11042license text changes in any way as compared to this parameter then a
11043mismatch occurs. This mismatch triggers a build failure and notifies the
11044developer. Notification allows the developer to review and address the
11045license text changes. Also note that if a mismatch occurs during the
11046build, the correct md5 checksum is placed in the build log and can be
11047easily copied to the recipe.
11048
11049There is no limit to how many files you can specify using the
Andrew Geissler09036742021-06-25 14:25:14 -050011050:term:`LIC_FILES_CHKSUM` variable. Generally, however, every project
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011051requires a few specifications for license tracking. Many projects have a
11052"COPYING" file that stores the license information for all the source
11053code files. This practice allows you to just track the "COPYING" file as
11054long as it is kept up to date.
11055
11056.. note::
11057
11058 - If you specify an empty or invalid "md5" parameter,
11059 :term:`BitBake` returns an md5
11060 mis-match error and displays the correct "md5" parameter value
11061 during the build. The correct parameter is also captured in the
11062 build log.
11063
11064 - If the whole file contains only license text, you do not need to
11065 use the "beginline" and "endline" parameters.
11066
11067Enabling Commercially Licensed Recipes
11068--------------------------------------
11069
11070By default, the OpenEmbedded build system disables components that have
11071commercial or other special licensing requirements. Such requirements
11072are defined on a recipe-by-recipe basis through the
11073:term:`LICENSE_FLAGS` variable
11074definition in the affected recipe. For instance, the
11075``poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly`` recipe
Andrew Geisslerc926e172021-05-07 16:11:35 -050011076contains the following statement::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011077
11078 LICENSE_FLAGS = "commercial"
11079
11080Here is a
11081slightly more complicated example that contains both an explicit recipe
Andrew Geisslerc926e172021-05-07 16:11:35 -050011082name and version (after variable expansion)::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011083
11084 LICENSE_FLAGS = "license_${PN}_${PV}"
11085
11086In order for a component restricted by a
Andrew Geissler09036742021-06-25 14:25:14 -050011087:term:`LICENSE_FLAGS` definition to be enabled and included in an image, it
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011088needs to have a matching entry in the global
Andrew Geissler9aee5002022-03-30 16:27:02 +000011089:term:`LICENSE_FLAGS_ACCEPTED`
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011090variable, which is a variable typically defined in your ``local.conf``
11091file. For example, to enable the
11092``poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly`` package, you
11093could add either the string "commercial_gst-plugins-ugly" or the more
Andrew Geissler9aee5002022-03-30 16:27:02 +000011094general string "commercial" to :term:`LICENSE_FLAGS_ACCEPTED`. See the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -050011095":ref:`dev-manual/common-tasks:license flag matching`" section for a full
Andrew Geissler09036742021-06-25 14:25:14 -050011096explanation of how :term:`LICENSE_FLAGS` matching works. Here is the
Andrew Geisslerc926e172021-05-07 16:11:35 -050011097example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011098
Andrew Geissler9aee5002022-03-30 16:27:02 +000011099 LICENSE_FLAGS_ACCEPTED = "commercial_gst-plugins-ugly"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011100
11101Likewise, to additionally enable the package built from the recipe
11102containing ``LICENSE_FLAGS = "license_${PN}_${PV}"``, and assuming that
11103the actual recipe name was ``emgd_1.10.bb``, the following string would
11104enable that package as well as the original ``gst-plugins-ugly``
Andrew Geisslerc926e172021-05-07 16:11:35 -050011105package::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011106
Andrew Geissler9aee5002022-03-30 16:27:02 +000011107 LICENSE_FLAGS_ACCEPTED = "commercial_gst-plugins-ugly license_emgd_1.10"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011108
11109As a convenience, you do not need to specify the
Andrew Geissler595f6302022-01-24 19:11:47 +000011110complete license string for every package. You can use
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011111an abbreviated form, which consists of just the first portion or
11112portions of the license string before the initial underscore character
11113or characters. A partial string will match any license that contains the
11114given string as the first portion of its license. For example, the
Andrew Geissler595f6302022-01-24 19:11:47 +000011115following value will also match both of the packages
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011116previously mentioned as well as any other packages that have licenses
11117starting with "commercial" or "license".
11118::
11119
Andrew Geissler9aee5002022-03-30 16:27:02 +000011120 LICENSE_FLAGS_ACCEPTED = "commercial license"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011121
11122License Flag Matching
11123~~~~~~~~~~~~~~~~~~~~~
11124
11125License flag matching allows you to control what recipes the
11126OpenEmbedded build system includes in the build. Fundamentally, the
Andrew Geissler09036742021-06-25 14:25:14 -050011127build system attempts to match :term:`LICENSE_FLAGS` strings found in
Andrew Geissler9aee5002022-03-30 16:27:02 +000011128recipes against strings found in :term:`LICENSE_FLAGS_ACCEPTED`.
Andrew Geissler595f6302022-01-24 19:11:47 +000011129A match causes the build system to include a recipe in the
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011130build, while failure to find a match causes the build system to exclude
11131a recipe.
11132
11133In general, license flag matching is simple. However, understanding some
11134concepts will help you correctly and effectively use matching.
11135
11136Before a flag defined by a particular recipe is tested against the
Andrew Geissler9aee5002022-03-30 16:27:02 +000011137entries of :term:`LICENSE_FLAGS_ACCEPTED`, the expanded
Andrew Geissler595f6302022-01-24 19:11:47 +000011138string ``_${PN}`` is appended to the flag. This expansion makes each
11139:term:`LICENSE_FLAGS` value recipe-specific. After expansion, the
11140string is then matched against the entries. Thus, specifying
11141``LICENSE_FLAGS = "commercial"`` in recipe "foo", for example, results
11142in the string ``"commercial_foo"``. And, to create a match, that string
Andrew Geissler9aee5002022-03-30 16:27:02 +000011143must appear among the entries of :term:`LICENSE_FLAGS_ACCEPTED`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011144
Andrew Geissler09036742021-06-25 14:25:14 -050011145Judicious use of the :term:`LICENSE_FLAGS` strings and the contents of the
Andrew Geissler9aee5002022-03-30 16:27:02 +000011146:term:`LICENSE_FLAGS_ACCEPTED` variable allows you a lot of flexibility for
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011147including or excluding recipes based on licensing. For example, you can
11148broaden the matching capabilities by using license flags string subsets
Andrew Geissler9aee5002022-03-30 16:27:02 +000011149in :term:`LICENSE_FLAGS_ACCEPTED`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011150
11151.. note::
11152
11153 When using a string subset, be sure to use the part of the expanded
11154 string that precedes the appended underscore character (e.g.
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011155 ``usethispart_1.3``, ``usethispart_1.4``, and so forth).
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011156
Andrew Geissler595f6302022-01-24 19:11:47 +000011157For example, simply specifying the string "commercial" in the
Andrew Geissler9aee5002022-03-30 16:27:02 +000011158:term:`LICENSE_FLAGS_ACCEPTED` variable matches any expanded
Andrew Geissler595f6302022-01-24 19:11:47 +000011159:term:`LICENSE_FLAGS` definition that starts with the string
11160"commercial" such as "commercial_foo" and "commercial_bar", which
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011161are the strings the build system automatically generates for
11162hypothetical recipes named "foo" and "bar" assuming those recipes simply
Andrew Geisslerc926e172021-05-07 16:11:35 -050011163specify the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011164
11165 LICENSE_FLAGS = "commercial"
11166
Andrew Geissler595f6302022-01-24 19:11:47 +000011167Thus, you can choose to exhaustively enumerate each license flag in the
11168list and allow only specific recipes into the image, or you can use a
11169string subset that causes a broader range of matches to allow a range of
11170recipes into the image.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011171
Andrew Geissler09036742021-06-25 14:25:14 -050011172This scheme works even if the :term:`LICENSE_FLAGS` string already has
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011173``_${PN}`` appended. For example, the build system turns the license
11174flag "commercial_1.2_foo" into "commercial_1.2_foo_foo" and would match
11175both the general "commercial" and the specific "commercial_1.2_foo"
Andrew Geissler9aee5002022-03-30 16:27:02 +000011176strings found in the :term:`LICENSE_FLAGS_ACCEPTED` variable, as expected.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011177
11178Here are some other scenarios:
11179
11180- You can specify a versioned string in the recipe such as
11181 "commercial_foo_1.2" in a "foo" recipe. The build system expands this
11182 string to "commercial_foo_1.2_foo". Combine this license flag with a
Andrew Geissler9aee5002022-03-30 16:27:02 +000011183 :term:`LICENSE_FLAGS_ACCEPTED` variable that has the string
Andrew Geissler595f6302022-01-24 19:11:47 +000011184 "commercial" and you match the flag along with any other flag that
11185 starts with the string "commercial".
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011186
Andrew Geissler595f6302022-01-24 19:11:47 +000011187- Under the same circumstances, you can add "commercial_foo" in the
Andrew Geissler9aee5002022-03-30 16:27:02 +000011188 :term:`LICENSE_FLAGS_ACCEPTED` variable and the build system not only
Andrew Geissler595f6302022-01-24 19:11:47 +000011189 matches "commercial_foo_1.2" but also matches any license flag with
11190 the string "commercial_foo", regardless of the version.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011191
11192- You can be very specific and use both the package and version parts
Andrew Geissler9aee5002022-03-30 16:27:02 +000011193 in the :term:`LICENSE_FLAGS_ACCEPTED` list (e.g.
Andrew Geissler595f6302022-01-24 19:11:47 +000011194 "commercial_foo_1.2") to specifically match a versioned recipe.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011195
11196Other Variables Related to Commercial Licenses
11197~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
11198
William A. Kennington IIIac69b482021-06-02 12:28:27 -070011199There are other helpful variables related to commercial license handling,
11200defined in the
Andrew Geisslerc926e172021-05-07 16:11:35 -050011201``poky/meta/conf/distro/include/default-distrovars.inc`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011202
11203 COMMERCIAL_AUDIO_PLUGINS ?= ""
11204 COMMERCIAL_VIDEO_PLUGINS ?= ""
11205
11206If you
11207want to enable these components, you can do so by making sure you have
11208statements similar to the following in your ``local.conf`` configuration
Andrew Geisslerc926e172021-05-07 16:11:35 -050011209file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011210
11211 COMMERCIAL_AUDIO_PLUGINS = "gst-plugins-ugly-mad \
11212 gst-plugins-ugly-mpegaudioparse"
11213 COMMERCIAL_VIDEO_PLUGINS = "gst-plugins-ugly-mpeg2dec \
11214 gst-plugins-ugly-mpegstream gst-plugins-bad-mpegvideoparse"
Andrew Geissler9aee5002022-03-30 16:27:02 +000011215 LICENSE_FLAGS_ACCEPTED = "commercial_gst-plugins-ugly commercial_gst-plugins-bad commercial_qmmp"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011216
11217
Andrew Geissler595f6302022-01-24 19:11:47 +000011218Of course, you could also create a matching list for those
11219components using the more general "commercial" in the
Andrew Geissler9aee5002022-03-30 16:27:02 +000011220:term:`LICENSE_FLAGS_ACCEPTED` variable, but that would also enable all
Andrew Geissler595f6302022-01-24 19:11:47 +000011221the other packages with :term:`LICENSE_FLAGS`
Andrew Geisslerc926e172021-05-07 16:11:35 -050011222containing "commercial", which you may or may not want::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011223
Andrew Geissler9aee5002022-03-30 16:27:02 +000011224 LICENSE_FLAGS_ACCEPTED = "commercial"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011225
11226Specifying audio and video plugins as part of the
11227``COMMERCIAL_AUDIO_PLUGINS`` and ``COMMERCIAL_VIDEO_PLUGINS`` statements
Andrew Geissler9aee5002022-03-30 16:27:02 +000011228(along with the enabling :term:`LICENSE_FLAGS_ACCEPTED`) includes the
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011229plugins or components into built images, thus adding support for media
11230formats or components.
11231
11232Maintaining Open Source License Compliance During Your Product's Lifecycle
11233--------------------------------------------------------------------------
11234
11235One of the concerns for a development organization using open source
11236software is how to maintain compliance with various open source
11237licensing during the lifecycle of the product. While this section does
11238not provide legal advice or comprehensively cover all scenarios, it does
11239present methods that you can use to assist you in meeting the compliance
11240requirements during a software release.
11241
11242With hundreds of different open source licenses that the Yocto Project
11243tracks, it is difficult to know the requirements of each and every
11244license. However, the requirements of the major FLOSS licenses can begin
William A. Kennington IIIac69b482021-06-02 12:28:27 -070011245to be covered by assuming that there are three main areas of concern:
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011246
11247- Source code must be provided.
11248
11249- License text for the software must be provided.
11250
11251- Compilation scripts and modifications to the source code must be
11252 provided.
11253
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011254- spdx files can be provided.
11255
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011256There are other requirements beyond the scope of these three and the
11257methods described in this section (e.g. the mechanism through which
11258source code is distributed).
11259
11260As different organizations have different methods of complying with open
11261source licensing, this section is not meant to imply that there is only
11262one single way to meet your compliance obligations, but rather to
11263describe one method of achieving compliance. The remainder of this
11264section describes methods supported to meet the previously mentioned
11265three requirements. Once you take steps to meet these requirements, and
11266prior to releasing images, sources, and the build system, you should
11267audit all artifacts to ensure completeness.
11268
11269.. note::
11270
11271 The Yocto Project generates a license manifest during image creation
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011272 that is located in ``${DEPLOY_DIR}/licenses/``\ `image_name`\ ``-``\ `datestamp`
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011273 to assist with any audits.
11274
11275Providing the Source Code
11276~~~~~~~~~~~~~~~~~~~~~~~~~
11277
11278Compliance activities should begin before you generate the final image.
11279The first thing you should look at is the requirement that tops the list
Andrew Geissler615f2f12022-07-15 14:00:58 -050011280for most compliance groups --- providing the source. The Yocto Project has
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011281a few ways of meeting this requirement.
11282
11283One of the easiest ways to meet this requirement is to provide the
11284entire :term:`DL_DIR` used by the
11285build. This method, however, has a few issues. The most obvious is the
11286size of the directory since it includes all sources used in the build
11287and not just the source used in the released image. It will include
11288toolchain source, and other artifacts, which you would not generally
11289release. However, the more serious issue for most companies is
11290accidental release of proprietary software. The Yocto Project provides
11291an :ref:`archiver <ref-classes-archiver>` class to
11292help avoid some of these concerns.
11293
Andrew Geissler595f6302022-01-24 19:11:47 +000011294Before you employ :term:`DL_DIR` or the :ref:`archiver <ref-classes-archiver>` class, you need to
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011295decide how you choose to provide source. The source ``archiver`` class
11296can generate tarballs and SRPMs and can create them with various levels
11297of compliance in mind.
11298
11299One way of doing this (but certainly not the only way) is to release
11300just the source as a tarball. You can do this by adding the following to
11301the ``local.conf`` file found in the
Andrew Geisslerc926e172021-05-07 16:11:35 -050011302:term:`Build Directory`::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011303
11304 INHERIT += "archiver"
11305 ARCHIVER_MODE[src] = "original"
11306
11307During the creation of your
11308image, the source from all recipes that deploy packages to the image is
11309placed within subdirectories of ``DEPLOY_DIR/sources`` based on the
11310:term:`LICENSE` for each recipe.
11311Releasing the entire directory enables you to comply with requirements
11312concerning providing the unmodified source. It is important to note that
11313the size of the directory can get large.
11314
11315A way to help mitigate the size issue is to only release tarballs for
11316licenses that require the release of source. Let us assume you are only
11317concerned with GPL code as identified by running the following script:
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011318
11319.. code-block:: shell
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011320
11321 # Script to archive a subset of packages matching specific license(s)
11322 # Source and license files are copied into sub folders of package folder
11323 # Must be run from build folder
11324 #!/bin/bash
11325 src_release_dir="source-release"
11326 mkdir -p $src_release_dir
11327 for a in tmp/deploy/sources/*; do
11328 for d in $a/*; do
11329 # Get package name from path
11330 p=`basename $d`
11331 p=${p%-*}
11332 p=${p%-*}
11333 # Only archive GPL packages (update *GPL* regex for your license check)
11334 numfiles=`ls tmp/deploy/licenses/$p/*GPL* 2> /dev/null | wc -l`
Patrick Williams213cb262021-08-07 19:21:33 -050011335 if [ $numfiles -ge 1 ]; then
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011336 echo Archiving $p
11337 mkdir -p $src_release_dir/$p/source
11338 cp $d/* $src_release_dir/$p/source 2> /dev/null
11339 mkdir -p $src_release_dir/$p/license
11340 cp tmp/deploy/licenses/$p/* $src_release_dir/$p/license 2> /dev/null
11341 fi
11342 done
11343 done
11344
11345At this point, you
11346could create a tarball from the ``gpl_source_release`` directory and
11347provide that to the end user. This method would be a step toward
11348achieving compliance with section 3a of GPLv2 and with section 6 of
11349GPLv3.
11350
11351Providing License Text
11352~~~~~~~~~~~~~~~~~~~~~~
11353
11354One requirement that is often overlooked is inclusion of license text.
11355This requirement also needs to be dealt with prior to generating the
11356final image. Some licenses require the license text to accompany the
11357binary. You can achieve this by adding the following to your
Andrew Geisslerc926e172021-05-07 16:11:35 -050011358``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011359
11360 COPY_LIC_MANIFEST = "1"
11361 COPY_LIC_DIRS = "1"
11362 LICENSE_CREATE_PACKAGE = "1"
11363
11364Adding these statements to the
11365configuration file ensures that the licenses collected during package
11366generation are included on your image.
11367
11368.. note::
11369
11370 Setting all three variables to "1" results in the image having two
11371 copies of the same license file. One copy resides in
11372 ``/usr/share/common-licenses`` and the other resides in
11373 ``/usr/share/license``.
11374
11375 The reason for this behavior is because
11376 :term:`COPY_LIC_DIRS` and
11377 :term:`COPY_LIC_MANIFEST`
11378 add a copy of the license when the image is built but do not offer a
11379 path for adding licenses for newly installed packages to an image.
11380 :term:`LICENSE_CREATE_PACKAGE`
11381 adds a separate package and an upgrade path for adding licenses to an
11382 image.
11383
11384As the source ``archiver`` class has already archived the original
11385unmodified source that contains the license files, you would have
11386already met the requirements for inclusion of the license information
11387with source as defined by the GPL and other open source licenses.
11388
11389Providing Compilation Scripts and Source Code Modifications
11390~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
11391
11392At this point, we have addressed all we need to prior to generating the
11393image. The next two requirements are addressed during the final
11394packaging of the release.
11395
11396By releasing the version of the OpenEmbedded build system and the layers
11397used during the build, you will be providing both compilation scripts
11398and the source code modifications in one step.
11399
Andrew Geissler09209ee2020-12-13 08:44:15 -060011400If the deployment team has a :ref:`overview-manual/concepts:bsp layer`
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011401and a distro layer, and those
11402those layers are used to patch, compile, package, or modify (in any way)
11403any open source software included in your released images, you might be
11404required to release those layers under section 3 of GPLv2 or section 1
11405of GPLv3. One way of doing that is with a clean checkout of the version
11406of the Yocto Project and layers used during your build. Here is an
11407example:
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011408
11409.. code-block:: shell
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011410
11411 # We built using the dunfell branch of the poky repo
11412 $ git clone -b dunfell git://git.yoctoproject.org/poky
11413 $ cd poky
11414 # We built using the release_branch for our layers
11415 $ git clone -b release_branch git://git.mycompany.com/meta-my-bsp-layer
11416 $ git clone -b release_branch git://git.mycompany.com/meta-my-software-layer
11417 # clean up the .git repos
11418 $ find . -name ".git" -type d -exec rm -rf {} \;
11419
11420One
11421thing a development organization might want to consider for end-user
11422convenience is to modify ``meta-poky/conf/bblayers.conf.sample`` to
11423ensure that when the end user utilizes the released build system to
11424build an image, the development organization's layers are included in
Andrew Geisslerc926e172021-05-07 16:11:35 -050011425the ``bblayers.conf`` file automatically::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011426
11427 # POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
11428 # changes incompatibly
11429 POKY_BBLAYERS_CONF_VERSION = "2"
11430
11431 BBPATH = "${TOPDIR}"
11432 BBFILES ?= ""
11433
11434 BBLAYERS ?= " \
11435 ##OEROOT##/meta \
11436 ##OEROOT##/meta-poky \
11437 ##OEROOT##/meta-yocto-bsp \
11438 ##OEROOT##/meta-mylayer \
11439 "
11440
11441Creating and
11442providing an archive of the :term:`Metadata`
11443layers (recipes, configuration files, and so forth) enables you to meet
11444your requirements to include the scripts to control compilation as well
11445as any modifications to the original source.
11446
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011447Providing spdx files
11448~~~~~~~~~~~~~~~~~~~~~~~~~
11449
11450The spdx module has been integrated to a layer named meta-spdxscanner.
11451meta-spdxscanner provides several kinds of scanner. If you want to enable
11452this function, you have to follow the following steps:
11453
114541. Add meta-spdxscanner layer into ``bblayers.conf``.
11455
114562. Refer to the README in meta-spdxscanner to setup the environment (e.g,
11457 setup a fossology server) needed for the scanner.
11458
114593. Meta-spdxscanner provides several methods within the bbclass to create spdx files.
11460 Please choose one that you want to use and enable the spdx task. You have to
11461 add some config options in ``local.conf`` file in your :term:`Build
William A. Kennington IIIac69b482021-06-02 12:28:27 -070011462 Directory`. Here is an example showing how to generate spdx files
Andrew Geisslerd5838332022-05-27 11:33:10 -050011463 during BitBake using the fossology-python.bbclass::
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011464
11465 # Select fossology-python.bbclass.
11466 INHERIT += "fossology-python"
11467 # For fossology-python.bbclass, TOKEN is necessary, so, after setup a
11468 # Fossology server, you have to create a token.
11469 TOKEN = "eyJ0eXAiO..."
11470 # The fossology server is necessary for fossology-python.bbclass.
11471 FOSSOLOGY_SERVER = "http://xx.xx.xx.xx:8081/repo"
11472 # If you want to upload the source code to a special folder:
11473 FOLDER_NAME = "xxxx" //Optional
11474 # If you don't want to put spdx files in tmp/deploy/spdx, you can enable:
11475 SPDX_DEPLOY_DIR = "${DEPLOY_DIR}" //Optional
11476
11477For more usage information refer to :yocto_git:`the meta-spdxscanner repository
Andrew Geissler09209ee2020-12-13 08:44:15 -060011478</meta-spdxscanner/>`.
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011479
Andrew Geisslereff27472021-10-29 15:35:00 -050011480Compliance Limitations with Executables Built from Static Libraries
11481~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011482
Andrew Geisslereff27472021-10-29 15:35:00 -050011483When package A is added to an image via the :term:`RDEPENDS` or :term:`RRECOMMENDS`
11484mechanisms as well as explicitly included in the image recipe with
11485:term:`IMAGE_INSTALL`, and depends on a static linked library recipe B
11486(``DEPENDS += "B"``), package B will neither appear in the generated license
11487manifest nor in the generated source tarballs. This occurs as the
11488:ref:`license <ref-classes-license>` and :ref:`archiver <ref-classes-archiver>`
11489classes assume that only packages included via :term:`RDEPENDS` or :term:`RRECOMMENDS`
11490end up in the image.
11491
11492As a result, potential obligations regarding license compliance for package B
11493may not be met.
11494
11495The Yocto Project doesn't enable static libraries by default, in part because
11496of this issue. Before a solution to this limitation is found, you need to
11497keep in mind that if your root filesystem is built from static libraries,
11498you will need to manually ensure that your deliveries are compliant
11499with the licenses of these libraries.
11500
11501Copying Non Standard Licenses
11502-----------------------------
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011503
11504Some packages, such as the linux-firmware package, have many licenses
11505that are not in any way common. You can avoid adding a lot of these
11506types of common license files, which are only applicable to a specific
11507package, by using the
11508:term:`NO_GENERIC_LICENSE`
11509variable. Using this variable also avoids QA errors when you use a
11510non-common, non-CLOSED license in a recipe.
11511
William A. Kennington IIIac69b482021-06-02 12:28:27 -070011512Here is an example that uses the ``LICENSE.Abilis.txt`` file as
Andrew Geisslerc926e172021-05-07 16:11:35 -050011513the license from the fetched source::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011514
11515 NO_GENERIC_LICENSE[Firmware-Abilis] = "LICENSE.Abilis.txt"
11516
Patrick Williams213cb262021-08-07 19:21:33 -050011517Checking for Vulnerabilities
11518============================
11519
11520Vulnerabilities in images
11521-------------------------
11522
11523The Yocto Project has an infrastructure to track and address unfixed
11524known security vulnerabilities, as tracked by the public
11525`Common Vulnerabilities and Exposures (CVE) <https://en.wikipedia.org/wiki/Common_Vulnerabilities_and_Exposures>`__
11526database.
11527
Andrew Geissler615f2f12022-07-15 14:00:58 -050011528The Yocto Project maintains a `list of known vulnerabilities
11529<https://autobuilder.yocto.io/pub/non-release/patchmetrics/>`__
11530for packages in Poky and OE-Core, tracking the evolution of the number of
11531unpatched CVEs and the status of patches. Such information is available for
11532the current development version and for each supported release.
11533
11534To know which packages are vulnerable to known security vulnerabilities
11535in the specific image you are building, add the following setting to your
11536configuration::
Patrick Williams213cb262021-08-07 19:21:33 -050011537
11538 INHERIT += "cve-check"
11539
11540This way, at build time, BitBake will warn you about known CVEs
11541as in the example below::
11542
11543 WARNING: flex-2.6.4-r0 do_cve_check: Found unpatched CVE (CVE-2019-6293), for more information check /poky/build/tmp/work/core2-64-poky-linux/flex/2.6.4-r0/temp/cve.log
11544 WARNING: libarchive-3.5.1-r0 do_cve_check: Found unpatched CVE (CVE-2021-36976), for more information check /poky/build/tmp/work/core2-64-poky-linux/libarchive/3.5.1-r0/temp/cve.log
11545
11546It is also possible to check the CVE status of individual packages as follows::
11547
11548 bitbake -c cve_check flex libarchive
11549
11550Note that OpenEmbedded-Core keeps a list of known unfixed CVE issues which can
11551be ignored. You can pass this list to the check as follows::
11552
11553 bitbake -c cve_check libarchive -R conf/distro/include/cve-extra-exclusions.inc
11554
11555Enabling vulnerabily tracking in recipes
11556----------------------------------------
11557
11558The :term:`CVE_PRODUCT` variable defines the name used to match the recipe name
11559against the name in the upstream `NIST CVE database <https://nvd.nist.gov/>`__.
11560
Patrick Williams0ca19cc2021-08-16 14:03:13 -050011561Editing recipes to fix vulnerabilities
11562--------------------------------------
11563
11564To fix a given known vulnerability, you need to add a patch file to your recipe. Here's
11565an example from the :oe_layerindex:`ffmpeg recipe</layerindex/recipe/47350>`::
11566
11567 SRC_URI = "https://www.ffmpeg.org/releases/${BP}.tar.xz \
11568 file://0001-libavutil-include-assembly-with-full-path-from-sourc.patch \
11569 file://fix-CVE-2020-20446.patch \
11570 file://fix-CVE-2020-20453.patch \
11571 file://fix-CVE-2020-22015.patch \
11572 file://fix-CVE-2020-22021.patch \
11573 file://fix-CVE-2020-22033-CVE-2020-22019.patch \
11574 file://fix-CVE-2021-33815.patch \
11575
11576The :ref:`cve-check <ref-classes-cve-check>` class defines two ways of
11577supplying a patch for a given CVE. The first
11578way is to use a patch filename that matches the below pattern::
11579
11580 cve_file_name_match = re.compile(".*([Cc][Vv][Ee]\-\d{4}\-\d+)")
11581
11582As shown in the example above, multiple CVE IDs can appear in a patch filename,
11583but the :ref:`cve-check <ref-classes-cve-check>` class will only consider
11584the last CVE ID in the filename as patched.
11585
11586The second way to recognize a patched CVE ID is when a line matching the
11587below pattern is found in any patch file provided by the recipe::
11588
11589 cve_match = re.compile("CVE:( CVE\-\d{4}\-\d+)+")
11590
11591This allows a single patch file to address multiple CVE IDs at the same time.
11592
11593Of course, another way to fix vulnerabilities is to upgrade to a version
11594of the package which is not impacted, typically a more recent one.
11595The NIST database knows which versions are vulnerable and which ones
11596are not.
11597
11598Last but not least, you can choose to ignore vulnerabilities through
Andrew Geissler9aee5002022-03-30 16:27:02 +000011599the :term:`CVE_CHECK_SKIP_RECIPE` and :term:`CVE_CHECK_IGNORE`
Patrick Williams0ca19cc2021-08-16 14:03:13 -050011600variables.
11601
11602Implementation details
11603----------------------
11604
11605Here's what the :ref:`cve-check <ref-classes-cve-check>` class does to
11606find unpatched CVE IDs.
11607
11608First the code goes through each patch file provided by a recipe. If a valid CVE ID
11609is found in the name of the file, the corresponding CVE is considered as patched.
11610Don't forget that if multiple CVE IDs are found in the filename, only the last
11611one is considered. Then, the code looks for ``CVE: CVE-ID`` lines in the patch
11612file. The found CVE IDs are also considered as patched.
11613
11614Then, the code looks up all the CVE IDs in the NIST database for all the
11615products defined in :term:`CVE_PRODUCT`. Then, for each found CVE:
11616
11617 - If the package name (:term:`PN`) is part of
Andrew Geissler9aee5002022-03-30 16:27:02 +000011618 :term:`CVE_CHECK_SKIP_RECIPE`, it is considered as patched.
Patrick Williams0ca19cc2021-08-16 14:03:13 -050011619
Andrew Geissler9aee5002022-03-30 16:27:02 +000011620 - If the CVE ID is part of :term:`CVE_CHECK_IGNORE`, it is
Patrick Williams0ca19cc2021-08-16 14:03:13 -050011621 considered as patched too.
11622
11623 - If the CVE ID is part of the patched CVE for the recipe, it is
11624 already considered as patched.
11625
11626 - Otherwise, the code checks whether the recipe version (:term:`PV`)
11627 is within the range of versions impacted by the CVE. If so, the CVE
11628 is considered as unpatched.
11629
Patrick Williams213cb262021-08-07 19:21:33 -050011630The CVE database is stored in :term:`DL_DIR` and can be inspected using
11631``sqlite3`` command as follows::
11632
11633 sqlite3 downloads/CVE_CHECK/nvdcve_1.1.db .dump | grep CVE-2021-37462
11634
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011635Using the Error Reporting Tool
11636==============================
11637
11638The error reporting tool allows you to submit errors encountered during
11639builds to a central database. Outside of the build environment, you can
11640use a web interface to browse errors, view statistics, and query for
11641errors. The tool works using a client-server system where the client
11642portion is integrated with the installed Yocto Project
11643:term:`Source Directory` (e.g. ``poky``).
11644The server receives the information collected and saves it in a
11645database.
11646
William A. Kennington IIIac69b482021-06-02 12:28:27 -070011647There is a live instance of the error reporting server at
11648https://errors.yoctoproject.org.
11649When you want to get help with build failures, you can submit all of the
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011650information on the failure easily and then point to the URL in your bug
11651report or send an email to the mailing list.
11652
11653.. note::
11654
11655 If you send error reports to this server, the reports become publicly
11656 visible.
11657
11658Enabling and Using the Tool
11659---------------------------
11660
11661By default, the error reporting tool is disabled. You can enable it by
11662inheriting the
11663:ref:`report-error <ref-classes-report-error>`
11664class by adding the following statement to the end of your
11665``local.conf`` file in your
11666:term:`Build Directory`.
11667::
11668
11669 INHERIT += "report-error"
11670
11671By default, the error reporting feature stores information in
11672``${``\ :term:`LOG_DIR`\ ``}/error-report``.
11673However, you can specify a directory to use by adding the following to
Andrew Geisslerc926e172021-05-07 16:11:35 -050011674your ``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011675
11676 ERR_REPORT_DIR = "path"
11677
11678Enabling error
11679reporting causes the build process to collect the errors and store them
11680in a file as previously described. When the build system encounters an
11681error, it includes a command as part of the console output. You can run
11682the command to send the error file to the server. For example, the
Andrew Geisslerc926e172021-05-07 16:11:35 -050011683following command sends the errors to an upstream server::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011684
11685 $ send-error-report /home/brandusa/project/poky/build/tmp/log/error-report/error_report_201403141617.txt
11686
11687In the previous example, the errors are sent to a public database
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011688available at https://errors.yoctoproject.org, which is used by the
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011689entire community. If you specify a particular server, you can send the
11690errors to a different database. Use the following command for more
Andrew Geisslerc926e172021-05-07 16:11:35 -050011691information on available options::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011692
11693 $ send-error-report --help
11694
11695When sending the error file, you are prompted to review the data being
11696sent as well as to provide a name and optional email address. Once you
11697satisfy these prompts, the command returns a link from the server that
11698corresponds to your entry in the database. For example, here is a
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011699typical link: https://errors.yoctoproject.org/Errors/Details/9522/
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011700
11701Following the link takes you to a web interface where you can browse,
11702query the errors, and view statistics.
11703
11704Disabling the Tool
11705------------------
11706
11707To disable the error reporting feature, simply remove or comment out the
11708following statement from the end of your ``local.conf`` file in your
11709:term:`Build Directory`.
11710::
11711
11712 INHERIT += "report-error"
11713
11714Setting Up Your Own Error Reporting Server
11715------------------------------------------
11716
11717If you want to set up your own error reporting server, you can obtain
Andrew Geissler09209ee2020-12-13 08:44:15 -060011718the code from the Git repository at :yocto_git:`/error-report-web/`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011719Instructions on how to set it up are in the README document.
11720
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011721Using Wayland and Weston
11722========================
11723
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011724`Wayland <https://en.wikipedia.org/wiki/Wayland_(display_server_protocol)>`__
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011725is a computer display server protocol that provides a method for
11726compositing window managers to communicate directly with applications
11727and video hardware and expects them to communicate with input hardware
11728using other libraries. Using Wayland with supporting targets can result
11729in better control over graphics frame rendering than an application
11730might otherwise achieve.
11731
11732The Yocto Project provides the Wayland protocol libraries and the
11733reference
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011734`Weston <https://en.wikipedia.org/wiki/Wayland_(display_server_protocol)#Weston>`__
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011735compositor as part of its release. You can find the integrated packages
11736in the ``meta`` layer of the :term:`Source Directory`.
11737Specifically, you
11738can find the recipes that build both Wayland and Weston at
11739``meta/recipes-graphics/wayland``.
11740
11741You can build both the Wayland and Weston packages for use only with
11742targets that accept the `Mesa 3D and Direct Rendering
11743Infrastructure <https://en.wikipedia.org/wiki/Mesa_(computer_graphics)>`__,
11744which is also known as Mesa DRI. This implies that you cannot build and
11745use the packages if your target uses, for example, the Intel Embedded
11746Media and Graphics Driver (Intel EMGD) that overrides Mesa DRI.
11747
11748.. note::
11749
11750 Due to lack of EGL support, Weston 1.0.3 will not run directly on the
11751 emulated QEMU hardware. However, this version of Weston will run
11752 under X emulation without issues.
11753
11754This section describes what you need to do to implement Wayland and use
11755the Weston compositor when building an image for a supporting target.
11756
11757Enabling Wayland in an Image
11758----------------------------
11759
11760To enable Wayland, you need to enable it to be built and enable it to be
11761included (installed) in the image.
11762
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011763Building Wayland
11764~~~~~~~~~~~~~~~~
11765
11766To cause Mesa to build the ``wayland-egl`` platform and Weston to build
11767Wayland with Kernel Mode Setting
11768(`KMS <https://wiki.archlinux.org/index.php/Kernel_Mode_Setting>`__)
11769support, include the "wayland" flag in the
11770:term:`DISTRO_FEATURES`
Andrew Geisslerc926e172021-05-07 16:11:35 -050011771statement in your ``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011772
Patrick Williams0ca19cc2021-08-16 14:03:13 -050011773 DISTRO_FEATURES:append = " wayland"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011774
11775.. note::
11776
11777 If X11 has been enabled elsewhere, Weston will build Wayland with X11
11778 support
11779
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011780Installing Wayland and Weston
11781~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
11782
11783To install the Wayland feature into an image, you must include the
11784following
11785:term:`CORE_IMAGE_EXTRA_INSTALL`
Andrew Geisslerc926e172021-05-07 16:11:35 -050011786statement in your ``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011787
11788 CORE_IMAGE_EXTRA_INSTALL += "wayland weston"
11789
11790Running Weston
11791--------------
11792
11793To run Weston inside X11, enabling it as described earlier and building
11794a Sato image is sufficient. If you are running your image under Sato, a
11795Weston Launcher appears in the "Utility" category.
11796
11797Alternatively, you can run Weston through the command-line interpretor
11798(CLI), which is better suited for development work. To run Weston under
11799the CLI, you need to do the following after your image is built:
11800
Andrew Geisslerc926e172021-05-07 16:11:35 -0500118011. Run these commands to export ``XDG_RUNTIME_DIR``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011802
11803 mkdir -p /tmp/$USER-weston
11804 chmod 0700 /tmp/$USER-weston
11805 export XDG_RUNTIME_DIR=/tmp/$USER-weston
11806
Andrew Geisslerc926e172021-05-07 16:11:35 -0500118072. Launch Weston in the shell::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011808
11809 weston