blob: 53e7686633c432632162dc4b496f96c80bcab329 [file] [log] [blame]
Andrew Geisslerf0343792020-11-18 10:42:21 -06001.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002
3************
4Common Tasks
5************
6
7This chapter describes fundamental procedures such as creating layers,
8adding new software packages, extending or customizing images, porting
9work to new hardware (adding a new machine), and so forth. You will find
10that the procedures documented here occur often in the development cycle
11using the Yocto Project.
12
13Understanding and Creating Layers
14=================================
15
16The OpenEmbedded build system supports organizing
17:term:`Metadata` into multiple layers.
18Layers allow you to isolate different types of customizations from each
19other. For introductory information on the Yocto Project Layer Model,
20see the
Andrew Geissler09209ee2020-12-13 08:44:15 -060021":ref:`overview-manual/yp-intro:the yocto project layer model`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050022section in the Yocto Project Overview and Concepts Manual.
23
24Creating Your Own Layer
25-----------------------
26
Andrew Geissler595f6302022-01-24 19:11:47 +000027.. note::
28
29 It is very easy to create your own layers to use with the OpenEmbedded
30 build system, as the Yocto Project ships with tools that speed up creating
31 layers. This section describes the steps you perform by hand to create
32 layers so that you can better understand them. For information about the
33 layer-creation tools, see the
34 ":ref:`bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script`"
35 section in the Yocto Project Board Support Package (BSP) Developer's
36 Guide and the ":ref:`dev-manual/common-tasks:creating a general layer using the \`\`bitbake-layers\`\` script`"
37 section further down in this manual.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050038
39Follow these general steps to create your layer without using tools:
40
411. *Check Existing Layers:* Before creating a new layer, you should be
42 sure someone has not already created a layer containing the Metadata
Andrew Geisslerd1e89492021-02-12 15:35:20 -060043 you need. You can see the :oe_layerindex:`OpenEmbedded Metadata Index <>`
44 for a list of layers from the OpenEmbedded community that can be used in
Andrew Geisslerc9f78652020-09-18 14:11:35 -050045 the Yocto Project. You could find a layer that is identical or close
46 to what you need.
47
482. *Create a Directory:* Create the directory for your layer. When you
49 create the layer, be sure to create the directory in an area not
50 associated with the Yocto Project :term:`Source Directory`
51 (e.g. the cloned ``poky`` repository).
52
53 While not strictly required, prepend the name of the directory with
Andrew Geisslerc926e172021-05-07 16:11:35 -050054 the string "meta-". For example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050055
56 meta-mylayer
57 meta-GUI_xyz
58 meta-mymachine
59
Andrew Geisslerc926e172021-05-07 16:11:35 -050060 With rare exceptions, a layer's name follows this form::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050061
62 meta-root_name
63
64 Following this layer naming convention can save
65 you trouble later when tools, components, or variables "assume" your
66 layer name begins with "meta-". A notable example is in configuration
67 files as shown in the following step where layer names without the
68 "meta-" string are appended to several variables used in the
69 configuration.
70
713. *Create a Layer Configuration File:* Inside your new layer folder,
72 you need to create a ``conf/layer.conf`` file. It is easiest to take
73 an existing layer configuration file and copy that to your layer's
74 ``conf`` directory and then modify the file as needed.
75
76 The ``meta-yocto-bsp/conf/layer.conf`` file in the Yocto Project
Andrew Geissler09209ee2020-12-13 08:44:15 -060077 :yocto_git:`Source Repositories </poky/tree/meta-yocto-bsp/conf>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -050078 demonstrates the required syntax. For your layer, you need to replace
79 "yoctobsp" with a unique identifier for your layer (e.g. "machinexyz"
Andrew Geisslerc926e172021-05-07 16:11:35 -050080 for a layer named "meta-machinexyz")::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050081
82 # We have a conf and classes directory, add to BBPATH
83 BBPATH .= ":${LAYERDIR}"
84
Andrew Geissler4c19ea12020-10-27 13:52:24 -050085 # We have recipes-* directories, add to BBFILES
Andrew Geisslerc9f78652020-09-18 14:11:35 -050086 BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
87 ${LAYERDIR}/recipes-*/*/*.bbappend"
88
89 BBFILE_COLLECTIONS += "yoctobsp"
90 BBFILE_PATTERN_yoctobsp = "^${LAYERDIR}/"
91 BBFILE_PRIORITY_yoctobsp = "5"
92 LAYERVERSION_yoctobsp = "4"
93 LAYERSERIES_COMPAT_yoctobsp = "dunfell"
94
95 Following is an explanation of the layer configuration file:
96
97 - :term:`BBPATH`: Adds the layer's
98 root directory to BitBake's search path. Through the use of the
Andrew Geissler09036742021-06-25 14:25:14 -050099 :term:`BBPATH` variable, BitBake locates class files (``.bbclass``),
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500100 configuration files, and files that are included with ``include``
101 and ``require`` statements. For these cases, BitBake uses the
Andrew Geissler09036742021-06-25 14:25:14 -0500102 first file that matches the name found in :term:`BBPATH`. This is
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500103 similar to the way the ``PATH`` variable is used for binaries. It
104 is recommended, therefore, that you use unique class and
105 configuration filenames in your custom layer.
106
107 - :term:`BBFILES`: Defines the
108 location for all recipes in the layer.
109
110 - :term:`BBFILE_COLLECTIONS`:
111 Establishes the current layer through a unique identifier that is
112 used throughout the OpenEmbedded build system to refer to the
113 layer. In this example, the identifier "yoctobsp" is the
114 representation for the container layer named "meta-yocto-bsp".
115
116 - :term:`BBFILE_PATTERN`:
117 Expands immediately during parsing to provide the directory of the
118 layer.
119
120 - :term:`BBFILE_PRIORITY`:
121 Establishes a priority to use for recipes in the layer when the
122 OpenEmbedded build finds recipes of the same name in different
123 layers.
124
125 - :term:`LAYERVERSION`:
126 Establishes a version number for the layer. You can use this
127 version number to specify this exact version of the layer as a
128 dependency when using the
129 :term:`LAYERDEPENDS`
130 variable.
131
132 - :term:`LAYERDEPENDS`:
133 Lists all layers on which this layer depends (if any).
134
135 - :term:`LAYERSERIES_COMPAT`:
Andrew Geissler09209ee2020-12-13 08:44:15 -0600136 Lists the :yocto_wiki:`Yocto Project </Releases>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500137 releases for which the current version is compatible. This
138 variable is a good way to indicate if your particular layer is
139 current.
140
1414. *Add Content:* Depending on the type of layer, add the content. If
142 the layer adds support for a machine, add the machine configuration
143 in a ``conf/machine/`` file within the layer. If the layer adds
144 distro policy, add the distro configuration in a ``conf/distro/``
145 file within the layer. If the layer introduces new recipes, put the
146 recipes you need in ``recipes-*`` subdirectories within the layer.
147
148 .. note::
149
150 For an explanation of layer hierarchy that is compliant with the
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500151 Yocto Project, see the ":ref:`bsp-guide/bsp:example filesystem layout`"
152 section in the Yocto Project Board Support Package (BSP) Developer's Guide.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500153
1545. *Optionally Test for Compatibility:* If you want permission to use
155 the Yocto Project Compatibility logo with your layer or application
156 that uses your layer, perform the steps to apply for compatibility.
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500157 See the
158 ":ref:`dev-manual/common-tasks:making sure your layer is compatible with yocto project`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500159 section for more information.
160
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500161Following Best Practices When Creating Layers
162---------------------------------------------
163
164To create layers that are easier to maintain and that will not impact
165builds for other machines, you should consider the information in the
166following list:
167
168- *Avoid "Overlaying" Entire Recipes from Other Layers in Your
169 Configuration:* In other words, do not copy an entire recipe into
170 your layer and then modify it. Rather, use an append file
171 (``.bbappend``) to override only those parts of the original recipe
172 you need to modify.
173
174- *Avoid Duplicating Include Files:* Use append files (``.bbappend``)
175 for each recipe that uses an include file. Or, if you are introducing
176 a new recipe that requires the included file, use the path relative
177 to the original layer directory to refer to the file. For example,
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500178 use ``require recipes-core/``\ `package`\ ``/``\ `file`\ ``.inc`` instead
179 of ``require`` `file`\ ``.inc``. If you're finding you have to overlay
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500180 the include file, it could indicate a deficiency in the include file
181 in the layer to which it originally belongs. If this is the case, you
182 should try to address that deficiency instead of overlaying the
183 include file. For example, you could address this by getting the
184 maintainer of the include file to add a variable or variables to make
185 it easy to override the parts needing to be overridden.
186
187- *Structure Your Layers:* Proper use of overrides within append files
188 and placement of machine-specific files within your layer can ensure
189 that a build is not using the wrong Metadata and negatively impacting
190 a build for a different machine. Following are some examples:
191
192 - *Modify Variables to Support a Different Machine:* Suppose you
193 have a layer named ``meta-one`` that adds support for building
194 machine "one". To do so, you use an append file named
195 ``base-files.bbappend`` and create a dependency on "foo" by
196 altering the :term:`DEPENDS`
Andrew Geisslerc926e172021-05-07 16:11:35 -0500197 variable::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500198
199 DEPENDS = "foo"
200
201 The dependency is created during any
202 build that includes the layer ``meta-one``. However, you might not
203 want this dependency for all machines. For example, suppose you
204 are building for machine "two" but your ``bblayers.conf`` file has
205 the ``meta-one`` layer included. During the build, the
206 ``base-files`` for machine "two" will also have the dependency on
207 ``foo``.
208
209 To make sure your changes apply only when building machine "one",
Andrew Geissler09036742021-06-25 14:25:14 -0500210 use a machine override with the :term:`DEPENDS` statement::
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500211
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500212 DEPENDS:one = "foo"
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500213
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500214 You should follow the same strategy when using ``:append``
215 and ``:prepend`` operations::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500216
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500217 DEPENDS:append:one = " foo"
218 DEPENDS:prepend:one = "foo "
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500219
220 As an actual example, here's a
221 snippet from the generic kernel include file ``linux-yocto.inc``,
222 wherein the kernel compile and link options are adjusted in the
Andrew Geisslerc926e172021-05-07 16:11:35 -0500223 case of a subset of the supported architectures::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500224
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500225 DEPENDS:append:aarch64 = " libgcc"
226 KERNEL_CC:append:aarch64 = " ${TOOLCHAIN_OPTIONS}"
227 KERNEL_LD:append:aarch64 = " ${TOOLCHAIN_OPTIONS}"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500228
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500229 DEPENDS:append:nios2 = " libgcc"
230 KERNEL_CC:append:nios2 = " ${TOOLCHAIN_OPTIONS}"
231 KERNEL_LD:append:nios2 = " ${TOOLCHAIN_OPTIONS}"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500232
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500233 DEPENDS:append:arc = " libgcc"
234 KERNEL_CC:append:arc = " ${TOOLCHAIN_OPTIONS}"
235 KERNEL_LD:append:arc = " ${TOOLCHAIN_OPTIONS}"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500236
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500237 KERNEL_FEATURES:append:qemuall=" features/debug/printk.scc"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500238
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500239 - *Place Machine-Specific Files in Machine-Specific Locations:* When
240 you have a base recipe, such as ``base-files.bb``, that contains a
241 :term:`SRC_URI` statement to a
242 file, you can use an append file to cause the build to use your
243 own version of the file. For example, an append file in your layer
244 at ``meta-one/recipes-core/base-files/base-files.bbappend`` could
Andrew Geisslerc926e172021-05-07 16:11:35 -0500245 extend :term:`FILESPATH` using :term:`FILESEXTRAPATHS` as follows::
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500246
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500247 FILESEXTRAPATHS:prepend := "${THISDIR}/${BPN}:"
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500248
249 The build for machine "one" will pick up your machine-specific file as
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500250 long as you have the file in
251 ``meta-one/recipes-core/base-files/base-files/``. However, if you
252 are building for a different machine and the ``bblayers.conf``
253 file includes the ``meta-one`` layer and the location of your
254 machine-specific file is the first location where that file is
Andrew Geissler09036742021-06-25 14:25:14 -0500255 found according to :term:`FILESPATH`, builds for all machines will
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500256 also use that machine-specific file.
257
258 You can make sure that a machine-specific file is used for a
259 particular machine by putting the file in a subdirectory specific
260 to the machine. For example, rather than placing the file in
261 ``meta-one/recipes-core/base-files/base-files/`` as shown above,
262 put it in ``meta-one/recipes-core/base-files/base-files/one/``.
263 Not only does this make sure the file is used only when building
264 for machine "one", but the build process locates the file more
265 quickly.
266
267 In summary, you need to place all files referenced from
Andrew Geissler5f350902021-07-23 13:09:54 -0400268 :term:`SRC_URI` in a machine-specific subdirectory within the layer in
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500269 order to restrict those files to machine-specific builds.
270
271- *Perform Steps to Apply for Yocto Project Compatibility:* If you want
272 permission to use the Yocto Project Compatibility logo with your
273 layer or application that uses your layer, perform the steps to apply
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500274 for compatibility. See the
275 ":ref:`dev-manual/common-tasks:making sure your layer is compatible with yocto project`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500276 section for more information.
277
278- *Follow the Layer Naming Convention:* Store custom layers in a Git
279 repository that use the ``meta-layer_name`` format.
280
281- *Group Your Layers Locally:* Clone your repository alongside other
282 cloned ``meta`` directories from the :term:`Source Directory`.
283
284Making Sure Your Layer is Compatible With Yocto Project
285-------------------------------------------------------
286
287When you create a layer used with the Yocto Project, it is advantageous
288to make sure that the layer interacts well with existing Yocto Project
289layers (i.e. the layer is compatible with the Yocto Project). Ensuring
290compatibility makes the layer easy to be consumed by others in the Yocto
291Project community and could allow you permission to use the Yocto
292Project Compatible Logo.
293
294.. note::
295
296 Only Yocto Project member organizations are permitted to use the
297 Yocto Project Compatible Logo. The logo is not available for general
298 use. For information on how to become a Yocto Project member
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500299 organization, see the :yocto_home:`Yocto Project Website <>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500300
301The Yocto Project Compatibility Program consists of a layer application
302process that requests permission to use the Yocto Project Compatibility
303Logo for your layer and application. The process consists of two parts:
304
3051. Successfully passing a script (``yocto-check-layer``) that when run
306 against your layer, tests it against constraints based on experiences
307 of how layers have worked in the real world and where pitfalls have
308 been found. Getting a "PASS" result from the script is required for
309 successful compatibility registration.
310
3112. Completion of an application acceptance form, which you can find at
Andrew Geisslerd1e89492021-02-12 15:35:20 -0600312 :yocto_home:`/webform/yocto-project-compatible-registration`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500313
314To be granted permission to use the logo, you need to satisfy the
315following:
316
317- Be able to check the box indicating that you got a "PASS" when
318 running the script against your layer.
319
320- Answer "Yes" to the questions on the form or have an acceptable
321 explanation for any questions answered "No".
322
323- Be a Yocto Project Member Organization.
324
325The remainder of this section presents information on the registration
326form and on the ``yocto-check-layer`` script.
327
328Yocto Project Compatible Program Application
329~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
330
331Use the form to apply for your layer's approval. Upon successful
332application, you can use the Yocto Project Compatibility Logo with your
333layer and the application that uses your layer.
334
335To access the form, use this link:
Andrew Geisslerd1e89492021-02-12 15:35:20 -0600336:yocto_home:`/webform/yocto-project-compatible-registration`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500337Follow the instructions on the form to complete your application.
338
339The application consists of the following sections:
340
341- *Contact Information:* Provide your contact information as the fields
342 require. Along with your information, provide the released versions
343 of the Yocto Project for which your layer is compatible.
344
345- *Acceptance Criteria:* Provide "Yes" or "No" answers for each of the
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700346 items in the checklist. There is space at the bottom of the form for
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500347 any explanations for items for which you answered "No".
348
349- *Recommendations:* Provide answers for the questions regarding Linux
350 kernel use and build success.
351
352``yocto-check-layer`` Script
353~~~~~~~~~~~~~~~~~~~~~~~~~~~~
354
355The ``yocto-check-layer`` script provides you a way to assess how
356compatible your layer is with the Yocto Project. You should run this
357script prior to using the form to apply for compatibility as described
358in the previous section. You need to achieve a "PASS" result in order to
359have your application form successfully processed.
360
361The script divides tests into three areas: COMMON, BSP, and DISTRO. For
362example, given a distribution layer (DISTRO), the layer must pass both
363the COMMON and DISTRO related tests. Furthermore, if your layer is a BSP
364layer, the layer must pass the COMMON and BSP set of tests.
365
366To execute the script, enter the following commands from your build
Andrew Geisslerc926e172021-05-07 16:11:35 -0500367directory::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500368
369 $ source oe-init-build-env
370 $ yocto-check-layer your_layer_directory
371
372Be sure to provide the actual directory for your
373layer as part of the command.
374
375Entering the command causes the script to determine the type of layer
376and then to execute a set of specific tests against the layer. The
377following list overviews the test:
378
379- ``common.test_readme``: Tests if a ``README`` file exists in the
380 layer and the file is not empty.
381
382- ``common.test_parse``: Tests to make sure that BitBake can parse the
383 files without error (i.e. ``bitbake -p``).
384
385- ``common.test_show_environment``: Tests that the global or per-recipe
386 environment is in order without errors (i.e. ``bitbake -e``).
387
388- ``common.test_world``: Verifies that ``bitbake world`` works.
389
390- ``common.test_signatures``: Tests to be sure that BSP and DISTRO
391 layers do not come with recipes that change signatures.
392
393- ``common.test_layerseries_compat``: Verifies layer compatibility is
394 set properly.
395
396- ``bsp.test_bsp_defines_machines``: Tests if a BSP layer has machine
397 configurations.
398
399- ``bsp.test_bsp_no_set_machine``: Tests to ensure a BSP layer does not
400 set the machine when the layer is added.
401
402- ``bsp.test_machine_world``: Verifies that ``bitbake world`` works
403 regardless of which machine is selected.
404
405- ``bsp.test_machine_signatures``: Verifies that building for a
406 particular machine affects only the signature of tasks specific to
407 that machine.
408
409- ``distro.test_distro_defines_distros``: Tests if a DISTRO layer has
410 distro configurations.
411
412- ``distro.test_distro_no_set_distros``: Tests to ensure a DISTRO layer
413 does not set the distribution when the layer is added.
414
415Enabling Your Layer
416-------------------
417
418Before the OpenEmbedded build system can use your new layer, you need to
419enable it. To enable your layer, simply add your layer's path to the
Andrew Geissler09036742021-06-25 14:25:14 -0500420:term:`BBLAYERS` variable in your ``conf/bblayers.conf`` file, which is
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500421found in the :term:`Build Directory`.
Andrew Geissler5199d832021-09-24 16:47:35 -0500422The following example shows how to enable your new
423``meta-mylayer`` layer (note how your new layer exists outside of
424the official ``poky`` repository which you would have checked out earlier)::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500425
426 # POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
427 # changes incompatibly
428 POKY_BBLAYERS_CONF_VERSION = "2"
429 BBPATH = "${TOPDIR}"
430 BBFILES ?= ""
431 BBLAYERS ?= " \
432 /home/user/poky/meta \
433 /home/user/poky/meta-poky \
434 /home/user/poky/meta-yocto-bsp \
Andrew Geissler5199d832021-09-24 16:47:35 -0500435 /home/user/mystuff/meta-mylayer \
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500436 "
437
438BitBake parses each ``conf/layer.conf`` file from the top down as
Andrew Geissler09036742021-06-25 14:25:14 -0500439specified in the :term:`BBLAYERS` variable within the ``conf/bblayers.conf``
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500440file. During the processing of each ``conf/layer.conf`` file, BitBake
441adds the recipes, classes and configurations contained within the
442particular layer to the source directory.
443
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500444Appending Other Layers Metadata With Your Layer
445-----------------------------------------------
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500446
447A recipe that appends Metadata to another recipe is called a BitBake
448append file. A BitBake append file uses the ``.bbappend`` file type
449suffix, while the corresponding recipe to which Metadata is being
450appended uses the ``.bb`` file type suffix.
451
452You can use a ``.bbappend`` file in your layer to make additions or
453changes to the content of another layer's recipe without having to copy
454the other layer's recipe into your layer. Your ``.bbappend`` file
455resides in your layer, while the main ``.bb`` recipe file to which you
456are appending Metadata resides in a different layer.
457
458Being able to append information to an existing recipe not only avoids
459duplication, but also automatically applies recipe changes from a
460different layer into your layer. If you were copying recipes, you would
461have to manually merge changes as they occur.
462
463When you create an append file, you must use the same root name as the
464corresponding recipe file. For example, the append file
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500465``someapp_3.1.bbappend`` must apply to ``someapp_3.1.bb``. This
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500466means the original recipe and append filenames are version
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500467number-specific. If the corresponding recipe is renamed to update to a
468newer version, you must also rename and possibly update the
469corresponding ``.bbappend`` as well. During the build process, BitBake
470displays an error on starting if it detects a ``.bbappend`` file that
471does not have a corresponding recipe with a matching name. See the
472:term:`BB_DANGLINGAPPENDS_WARNONLY`
473variable for information on how to handle this error.
474
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500475Overlaying a File Using Your Layer
476~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
477
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500478As an example, consider the main formfactor recipe and a corresponding
479formfactor append file both from the :term:`Source Directory`.
480Here is the main
481formfactor recipe, which is named ``formfactor_0.0.bb`` and located in
Andrew Geisslerc926e172021-05-07 16:11:35 -0500482the "meta" layer at ``meta/recipes-bsp/formfactor``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500483
484 SUMMARY = "Device formfactor information"
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500485 DESCRIPTION = "A formfactor configuration file provides information about the \
486 target hardware for which the image is being built and information that the \
487 build system cannot obtain from other sources such as the kernel."
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500488 SECTION = "base"
489 LICENSE = "MIT"
490 LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
491 PR = "r45"
492
493 SRC_URI = "file://config file://machconfig"
494 S = "${WORKDIR}"
495
496 PACKAGE_ARCH = "${MACHINE_ARCH}"
497 INHIBIT_DEFAULT_DEPS = "1"
498
499 do_install() {
500 # Install file only if it has contents
501 install -d ${D}${sysconfdir}/formfactor/
502 install -m 0644 ${S}/config ${D}${sysconfdir}/formfactor/
503 if [ -s "${S}/machconfig" ]; then
504 install -m 0644 ${S}/machconfig ${D}${sysconfdir}/formfactor/
505 fi
506 }
507
508In the main recipe, note the :term:`SRC_URI`
509variable, which tells the OpenEmbedded build system where to find files
510during the build.
511
512Following is the append file, which is named ``formfactor_0.0.bbappend``
513and is from the Raspberry Pi BSP Layer named ``meta-raspberrypi``. The
Andrew Geisslerc926e172021-05-07 16:11:35 -0500514file is in the layer at ``recipes-bsp/formfactor``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500515
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500516 FILESEXTRAPATHS:prepend := "${THISDIR}/${PN}:"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500517
518By default, the build system uses the
519:term:`FILESPATH` variable to
520locate files. This append file extends the locations by setting the
521:term:`FILESEXTRAPATHS`
522variable. Setting this variable in the ``.bbappend`` file is the most
523reliable and recommended method for adding directories to the search
524path used by the build system to find files.
525
526The statement in this example extends the directories to include
527``${``\ :term:`THISDIR`\ ``}/${``\ :term:`PN`\ ``}``,
528which resolves to a directory named ``formfactor`` in the same directory
529in which the append file resides (i.e.
530``meta-raspberrypi/recipes-bsp/formfactor``. This implies that you must
531have the supporting directory structure set up that will contain any
532files or patches you will be including from the layer.
533
534Using the immediate expansion assignment operator ``:=`` is important
Andrew Geissler09036742021-06-25 14:25:14 -0500535because of the reference to :term:`THISDIR`. The trailing colon character is
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500536important as it ensures that items in the list remain colon-separated.
537
538.. note::
539
Andrew Geissler09036742021-06-25 14:25:14 -0500540 BitBake automatically defines the :term:`THISDIR` variable. You should
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500541 never set this variable yourself. Using ":prepend" as part of the
Andrew Geissler09036742021-06-25 14:25:14 -0500542 :term:`FILESEXTRAPATHS` ensures your path will be searched prior to other
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500543 paths in the final list.
544
545 Also, not all append files add extra files. Many append files simply
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700546 allow to add build options (e.g. ``systemd``). For these cases, your
Andrew Geissler09036742021-06-25 14:25:14 -0500547 append file would not even use the :term:`FILESEXTRAPATHS` statement.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500548
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500549The end result of this ``.bbappend`` file is that on a Raspberry Pi, where
550``rpi`` will exist in the list of :term:`OVERRIDES`, the file
551``meta-raspberrypi/recipes-bsp/formfactor/formfactor/rpi/machconfig`` will be
552used during :ref:`ref-tasks-fetch` and the test for a non-zero file size in
553:ref:`ref-tasks-install` will return true, and the file will be installed.
554
Andrew Geissler5199d832021-09-24 16:47:35 -0500555Installing Additional Files Using Your Layer
556~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
557
558As another example, consider the main ``xserver-xf86-config`` recipe and a
559corresponding ``xserver-xf86-config`` append file both from the :term:`Source
560Directory`. Here is the main ``xserver-xf86-config`` recipe, which is named
561``xserver-xf86-config_0.1.bb`` and located in the "meta" layer at
562``meta/recipes-graphics/xorg-xserver``::
563
564 SUMMARY = "X.Org X server configuration file"
565 HOMEPAGE = "http://www.x.org"
566 SECTION = "x11/base"
Andrew Geissler9aee5002022-03-30 16:27:02 +0000567 LICENSE = "MIT"
Andrew Geissler5199d832021-09-24 16:47:35 -0500568 LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
569 PR = "r33"
570
571 SRC_URI = "file://xorg.conf"
572
573 S = "${WORKDIR}"
574
575 CONFFILES:${PN} = "${sysconfdir}/X11/xorg.conf"
576
577 PACKAGE_ARCH = "${MACHINE_ARCH}"
578 ALLOW_EMPTY:${PN} = "1"
579
580 do_install () {
581 if test -s ${WORKDIR}/xorg.conf; then
582 install -d ${D}/${sysconfdir}/X11
583 install -m 0644 ${WORKDIR}/xorg.conf ${D}/${sysconfdir}/X11/
584 fi
585 }
586
587Following is the append file, which is named ``xserver-xf86-config_%.bbappend``
588and is from the Raspberry Pi BSP Layer named ``meta-raspberrypi``. The
589file is in the layer at ``recipes-graphics/xorg-xserver``::
590
591 FILESEXTRAPATHS:prepend := "${THISDIR}/${PN}:"
592
593 SRC_URI:append:rpi = " \
594 file://xorg.conf.d/98-pitft.conf \
595 file://xorg.conf.d/99-calibration.conf \
596 "
597 do_install:append:rpi () {
598 PITFT="${@bb.utils.contains("MACHINE_FEATURES", "pitft", "1", "0", d)}"
599 if [ "${PITFT}" = "1" ]; then
600 install -d ${D}/${sysconfdir}/X11/xorg.conf.d/
601 install -m 0644 ${WORKDIR}/xorg.conf.d/98-pitft.conf ${D}/${sysconfdir}/X11/xorg.conf.d/
602 install -m 0644 ${WORKDIR}/xorg.conf.d/99-calibration.conf ${D}/${sysconfdir}/X11/xorg.conf.d/
603 fi
604 }
605
606 FILES:${PN}:append:rpi = " ${sysconfdir}/X11/xorg.conf.d/*"
607
608Building off of the previous example, we once again are setting the
609:term:`FILESEXTRAPATHS` variable. In this case we are also using
610:term:`SRC_URI` to list additional source files to use when ``rpi`` is found in
611the list of :term:`OVERRIDES`. The :ref:`ref-tasks-install` task will then perform a
612check for an additional :term:`MACHINE_FEATURES` that if set will cause these
613additional files to be installed. These additional files are listed in
614:term:`FILES` so that they will be packaged.
615
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500616Prioritizing Your Layer
617-----------------------
618
619Each layer is assigned a priority value. Priority values control which
620layer takes precedence if there are recipe files with the same name in
621multiple layers. For these cases, the recipe file from the layer with a
622higher priority number takes precedence. Priority values also affect the
623order in which multiple ``.bbappend`` files for the same recipe are
624applied. You can either specify the priority manually, or allow the
625build system to calculate it based on the layer's dependencies.
626
627To specify the layer's priority manually, use the
628:term:`BBFILE_PRIORITY`
Andrew Geisslerc926e172021-05-07 16:11:35 -0500629variable and append the layer's root name::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500630
631 BBFILE_PRIORITY_mylayer = "1"
632
633.. note::
634
635 It is possible for a recipe with a lower version number
636 :term:`PV` in a layer that has a higher
637 priority to take precedence.
638
639 Also, the layer priority does not currently affect the precedence
640 order of ``.conf`` or ``.bbclass`` files. Future versions of BitBake
641 might address this.
642
643Managing Layers
644---------------
645
646You can use the BitBake layer management tool ``bitbake-layers`` to
647provide a view into the structure of recipes across a multi-layer
648project. Being able to generate output that reports on configured layers
649with their paths and priorities and on ``.bbappend`` files and their
650applicable recipes can help to reveal potential problems.
651
652For help on the BitBake layer management tool, use the following
Andrew Geisslerc926e172021-05-07 16:11:35 -0500653command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500654
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500655 $ bitbake-layers --help
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500656
657The following list describes the available commands:
658
659- ``help:`` Displays general help or help on a specified command.
660
661- ``show-layers:`` Shows the current configured layers.
662
663- ``show-overlayed:`` Lists overlayed recipes. A recipe is overlayed
664 when a recipe with the same name exists in another layer that has a
665 higher layer priority.
666
667- ``show-recipes:`` Lists available recipes and the layers that
668 provide them.
669
670- ``show-appends:`` Lists ``.bbappend`` files and the recipe files to
671 which they apply.
672
673- ``show-cross-depends:`` Lists dependency relationships between
674 recipes that cross layer boundaries.
675
676- ``add-layer:`` Adds a layer to ``bblayers.conf``.
677
678- ``remove-layer:`` Removes a layer from ``bblayers.conf``
679
680- ``flatten:`` Flattens the layer configuration into a separate
681 output directory. Flattening your layer configuration builds a
682 "flattened" directory that contains the contents of all layers, with
683 any overlayed recipes removed and any ``.bbappend`` files appended to
684 the corresponding recipes. You might have to perform some manual
685 cleanup of the flattened layer as follows:
686
687 - Non-recipe files (such as patches) are overwritten. The flatten
688 command shows a warning for these files.
689
690 - Anything beyond the normal layer setup has been added to the
691 ``layer.conf`` file. Only the lowest priority layer's
692 ``layer.conf`` is used.
693
694 - Overridden and appended items from ``.bbappend`` files need to be
695 cleaned up. The contents of each ``.bbappend`` end up in the
696 flattened recipe. However, if there are appended or changed
697 variable values, you need to tidy these up yourself. Consider the
698 following example. Here, the ``bitbake-layers`` command adds the
699 line ``#### bbappended ...`` so that you know where the following
Andrew Geisslerc926e172021-05-07 16:11:35 -0500700 lines originate::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500701
702 ...
703 DESCRIPTION = "A useful utility"
704 ...
705 EXTRA_OECONF = "--enable-something"
706 ...
707
708 #### bbappended from meta-anotherlayer ####
709
710 DESCRIPTION = "Customized utility"
711 EXTRA_OECONF += "--enable-somethingelse"
712
713
Andrew Geisslerc926e172021-05-07 16:11:35 -0500714 Ideally, you would tidy up these utilities as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500715
716 ...
717 DESCRIPTION = "Customized utility"
718 ...
719 EXTRA_OECONF = "--enable-something --enable-somethingelse"
720 ...
721
722- ``layerindex-fetch``: Fetches a layer from a layer index, along
723 with its dependent layers, and adds the layers to the
724 ``conf/bblayers.conf`` file.
725
726- ``layerindex-show-depends``: Finds layer dependencies from the
727 layer index.
728
Andrew Geissler87f5cff2022-09-30 13:13:31 -0500729- ``save-build-conf``: Saves the currently active build configuration
730 (``conf/local.conf``, ``conf/bblayers.conf``) as a template into a layer.
731 This template can later be used for setting up builds via :term:``TEMPLATECONF``.
732 For information about saving and using configuration templates, see
733 ":ref:`dev-manual/common-tasks:creating a custom template configuration directory`".
734
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500735- ``create-layer``: Creates a basic layer.
736
Andrew Geissler87f5cff2022-09-30 13:13:31 -0500737- ``create-layers-setup``: Writes out a configuration file and/or a script that
738 can replicate the directory structure and revisions of the layers in a current build.
739 For more information, see ":ref:`dev-manual/common-tasks:saving and restoring the layers setup`".
740
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500741Creating a General Layer Using the ``bitbake-layers`` Script
742------------------------------------------------------------
743
744The ``bitbake-layers`` script with the ``create-layer`` subcommand
745simplifies creating a new general layer.
746
747.. note::
748
749 - For information on BSP layers, see the ":ref:`bsp-guide/bsp:bsp layers`"
750 section in the Yocto
751 Project Board Specific (BSP) Developer's Guide.
752
753 - In order to use a layer with the OpenEmbedded build system, you
754 need to add the layer to your ``bblayers.conf`` configuration
Andrew Geissler09209ee2020-12-13 08:44:15 -0600755 file. See the ":ref:`dev-manual/common-tasks:adding a layer using the \`\`bitbake-layers\`\` script`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500756 section for more information.
757
758The default mode of the script's operation with this subcommand is to
759create a layer with the following:
760
761- A layer priority of 6.
762
763- A ``conf`` subdirectory that contains a ``layer.conf`` file.
764
765- A ``recipes-example`` subdirectory that contains a further
766 subdirectory named ``example``, which contains an ``example.bb``
767 recipe file.
768
769- A ``COPYING.MIT``, which is the license statement for the layer. The
770 script assumes you want to use the MIT license, which is typical for
771 most layers, for the contents of the layer itself.
772
773- A ``README`` file, which is a file describing the contents of your
774 new layer.
775
776In its simplest form, you can use the following command form to create a
777layer. The command creates a layer whose name corresponds to
Andrew Geisslerc926e172021-05-07 16:11:35 -0500778"your_layer_name" in the current directory::
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500779
780 $ bitbake-layers create-layer your_layer_name
781
782As an example, the following command creates a layer named ``meta-scottrif``
Andrew Geisslerc926e172021-05-07 16:11:35 -0500783in your home directory::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500784
785 $ cd /usr/home
786 $ bitbake-layers create-layer meta-scottrif
787 NOTE: Starting bitbake server...
788 Add your new layer with 'bitbake-layers add-layer meta-scottrif'
789
790If you want to set the priority of the layer to other than the default
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500791value of "6", you can either use the ``--priority`` option or you
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500792can edit the
793:term:`BBFILE_PRIORITY` value
794in the ``conf/layer.conf`` after the script creates it. Furthermore, if
795you want to give the example recipe file some name other than the
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500796default, you can use the ``--example-recipe-name`` option.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500797
798The easiest way to see how the ``bitbake-layers create-layer`` command
799works is to experiment with the script. You can also read the usage
Andrew Geisslerc926e172021-05-07 16:11:35 -0500800information by entering the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500801
802 $ bitbake-layers create-layer --help
803 NOTE: Starting bitbake server...
804 usage: bitbake-layers create-layer [-h] [--priority PRIORITY]
805 [--example-recipe-name EXAMPLERECIPE]
806 layerdir
807
808 Create a basic layer
809
810 positional arguments:
811 layerdir Layer directory to create
812
813 optional arguments:
814 -h, --help show this help message and exit
815 --priority PRIORITY, -p PRIORITY
816 Layer directory to create
817 --example-recipe-name EXAMPLERECIPE, -e EXAMPLERECIPE
818 Filename of the example recipe
819
820Adding a Layer Using the ``bitbake-layers`` Script
821--------------------------------------------------
822
823Once you create your general layer, you must add it to your
824``bblayers.conf`` file. Adding the layer to this configuration file
825makes the OpenEmbedded build system aware of your layer so that it can
826search it for metadata.
827
Andrew Geisslerc926e172021-05-07 16:11:35 -0500828Add your layer by using the ``bitbake-layers add-layer`` command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500829
830 $ bitbake-layers add-layer your_layer_name
831
832Here is an example that adds a
833layer named ``meta-scottrif`` to the configuration file. Following the
834command that adds the layer is another ``bitbake-layers`` command that
Andrew Geisslerc926e172021-05-07 16:11:35 -0500835shows the layers that are in your ``bblayers.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500836
837 $ bitbake-layers add-layer meta-scottrif
838 NOTE: Starting bitbake server...
839 Parsing recipes: 100% |##########################################################| Time: 0:00:49
840 Parsing of 1441 .bb files complete (0 cached, 1441 parsed). 2055 targets, 56 skipped, 0 masked, 0 errors.
841 $ bitbake-layers show-layers
842 NOTE: Starting bitbake server...
843 layer path priority
844 ==========================================================================
845 meta /home/scottrif/poky/meta 5
846 meta-poky /home/scottrif/poky/meta-poky 5
847 meta-yocto-bsp /home/scottrif/poky/meta-yocto-bsp 5
848 workspace /home/scottrif/poky/build/workspace 99
849 meta-scottrif /home/scottrif/poky/build/meta-scottrif 6
850
851
852Adding the layer to this file
853enables the build system to locate the layer during the build.
854
855.. note::
856
857 During a build, the OpenEmbedded build system looks in the layers
858 from the top of the list down to the bottom in that order.
859
Andrew Geissler87f5cff2022-09-30 13:13:31 -0500860Saving and restoring the layers setup
861-------------------------------------
862
863Once you have a working build with the correct set of layers, it is beneficial
864to capture the layer setup --- what they are, which repositories they come from
865and which SCM revisions they're at --- into a configuration file, so that this
866setup can be easily replicated later, perhaps on a different machine. Here's
867how to do this::
868
869 $ bitbake-layers create-layers-setup /srv/work/alex/meta-alex/
870 NOTE: Starting bitbake server...
871 NOTE: Created /srv/work/alex/meta-alex/setup-layers.json
872 NOTE: Created /srv/work/alex/meta-alex/setup-layers
873
874The tool needs a single argument which tells where to place the output, consisting
875of a json formatted layer configuration, and a ``setup-layers`` script that can use that configuration
876to restore the layers in a different location, or on a different host machine. The argument
877can point to a custom layer (which is then deemed a "bootstrap" layer that needs to be
878checked out first), or into a completely independent location.
879
880The replication of the layers is performed by running the ``setup-layers`` script provided
881above:
882
8831. Clone the bootstrap layer or some other repository to obtain
884 the json config and the setup script that can use it.
885
8862. Run the script directly with no options::
887
888 alex@Zen2:/srv/work/alex/my-build$ meta-alex/setup-layers
889 Note: not checking out source meta-alex, use --force-bootstraplayer-checkout to override.
890
891 Setting up source meta-intel, revision 15.0-hardknott-3.3-310-g0a96edae, branch master
892 Running 'git init -q /srv/work/alex/my-build/meta-intel'
893 Running 'git remote remove origin > /dev/null 2>&1; git remote add origin git://git.yoctoproject.org/meta-intel' in /srv/work/alex/my-build/meta-intel
894 Running 'git fetch -q origin || true' in /srv/work/alex/my-build/meta-intel
895 Running 'git checkout -q 0a96edae609a3f48befac36af82cf1eed6786b4a' in /srv/work/alex/my-build/meta-intel
896
897 Setting up source poky, revision 4.1_M1-372-g55483d28f2, branch akanavin/setup-layers
898 Running 'git init -q /srv/work/alex/my-build/poky'
899 Running 'git remote remove origin > /dev/null 2>&1; git remote add origin git://git.yoctoproject.org/poky' in /srv/work/alex/my-build/poky
900 Running 'git fetch -q origin || true' in /srv/work/alex/my-build/poky
901 Running 'git remote remove poky-contrib > /dev/null 2>&1; git remote add poky-contrib ssh://git@push.yoctoproject.org/poky-contrib' in /srv/work/alex/my-build/poky
902 Running 'git fetch -q poky-contrib || true' in /srv/work/alex/my-build/poky
903 Running 'git checkout -q 11db0390b02acac1324e0f827beb0e2e3d0d1d63' in /srv/work/alex/my-build/poky
904
905.. note::
906 This will work to update an existing checkout as well.
907
908.. note::
909 The script is self-sufficient and requires only python3
910 and git on the build machine.
911
912.. note::
913 Both the ``create-layers-setup`` and the ``setup-layers`` provided several additional options
914 that customize their behavior - you are welcome to study them via ``--help`` command line parameter.
915
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500916Customizing Images
917==================
918
919You can customize images to satisfy particular requirements. This
920section describes several methods and provides guidelines for each.
921
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500922Customizing Images Using ``local.conf``
923---------------------------------------
924
925Probably the easiest way to customize an image is to add a package by
926way of the ``local.conf`` configuration file. Because it is limited to
927local use, this method generally only allows you to add packages and is
928not as flexible as creating your own customized image. When you add
929packages using local variables this way, you need to realize that these
930variable changes are in effect for every build and consequently affect
931all images, which might not be what you require.
932
933To add a package to your image using the local configuration file, use
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500934the :term:`IMAGE_INSTALL` variable with the ``:append`` operator::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500935
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500936 IMAGE_INSTALL:append = " strace"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500937
Andrew Geissler5199d832021-09-24 16:47:35 -0500938Use of the syntax is important; specifically, the leading space
939after the opening quote and before the package name, which is
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500940``strace`` in this example. This space is required since the ``:append``
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500941operator does not add the space.
942
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500943Furthermore, you must use ``:append`` instead of the ``+=`` operator if
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500944you want to avoid ordering issues. The reason for this is because doing
945so unconditionally appends to the variable and avoids ordering problems
946due to the variable being set in image recipes and ``.bbclass`` files
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500947with operators like ``?=``. Using ``:append`` ensures the operation
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500948takes effect.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500949
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500950As shown in its simplest use, ``IMAGE_INSTALL:append`` affects all
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500951images. It is possible to extend the syntax so that the variable applies
Andrew Geisslerc926e172021-05-07 16:11:35 -0500952to a specific image only. Here is an example::
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500953
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500954 IMAGE_INSTALL:append:pn-core-image-minimal = " strace"
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500955
956This example adds ``strace`` to the ``core-image-minimal`` image only.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500957
958You can add packages using a similar approach through the
Andrew Geissler09036742021-06-25 14:25:14 -0500959:term:`CORE_IMAGE_EXTRA_INSTALL` variable. If you use this variable, only
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500960``core-image-*`` images are affected.
961
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500962Customizing Images Using Custom ``IMAGE_FEATURES`` and ``EXTRA_IMAGE_FEATURES``
963-------------------------------------------------------------------------------
964
965Another method for customizing your image is to enable or disable
966high-level image features by using the
967:term:`IMAGE_FEATURES` and
968:term:`EXTRA_IMAGE_FEATURES`
969variables. Although the functions for both variables are nearly
Andrew Geissler09036742021-06-25 14:25:14 -0500970equivalent, best practices dictate using :term:`IMAGE_FEATURES` from within
971a recipe and using :term:`EXTRA_IMAGE_FEATURES` from within your
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500972``local.conf`` file, which is found in the
973:term:`Build Directory`.
974
975To understand how these features work, the best reference is
Patrick Williams975a06f2022-10-21 14:42:47 -0500976:ref:`meta/classes-recipe/image.bbclass <ref-classes-image>`.
Andrew Geissler595f6302022-01-24 19:11:47 +0000977This class lists out the available
Andrew Geissler09036742021-06-25 14:25:14 -0500978:term:`IMAGE_FEATURES` of which most map to package groups while some, such
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500979as ``debug-tweaks`` and ``read-only-rootfs``, resolve as general
980configuration settings.
981
Andrew Geissler09036742021-06-25 14:25:14 -0500982In summary, the file looks at the contents of the :term:`IMAGE_FEATURES`
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500983variable and then maps or configures the feature accordingly. Based on
984this information, the build system automatically adds the appropriate
985packages or configurations to the
986:term:`IMAGE_INSTALL` variable.
987Effectively, you are enabling extra features by extending the class or
988creating a custom class for use with specialized image ``.bb`` files.
989
Andrew Geissler09036742021-06-25 14:25:14 -0500990Use the :term:`EXTRA_IMAGE_FEATURES` variable from within your local
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500991configuration file. Using a separate area from which to enable features
992with this variable helps you avoid overwriting the features in the image
Andrew Geissler09036742021-06-25 14:25:14 -0500993recipe that are enabled with :term:`IMAGE_FEATURES`. The value of
994:term:`EXTRA_IMAGE_FEATURES` is added to :term:`IMAGE_FEATURES` within
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500995``meta/conf/bitbake.conf``.
996
997To illustrate how you can use these variables to modify your image,
998consider an example that selects the SSH server. The Yocto Project ships
999with two SSH servers you can use with your images: Dropbear and OpenSSH.
1000Dropbear is a minimal SSH server appropriate for resource-constrained
1001environments, while OpenSSH is a well-known standard SSH server
1002implementation. By default, the ``core-image-sato`` image is configured
1003to use Dropbear. The ``core-image-full-cmdline`` and ``core-image-lsb``
1004images both include OpenSSH. The ``core-image-minimal`` image does not
1005contain an SSH server.
1006
1007You can customize your image and change these defaults. Edit the
Andrew Geissler09036742021-06-25 14:25:14 -05001008:term:`IMAGE_FEATURES` variable in your recipe or use the
1009:term:`EXTRA_IMAGE_FEATURES` in your ``local.conf`` file so that it
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001010configures the image you are working with to include
1011``ssh-server-dropbear`` or ``ssh-server-openssh``.
1012
1013.. note::
1014
Andrew Geissler09209ee2020-12-13 08:44:15 -06001015 See the ":ref:`ref-manual/features:image features`" section in the Yocto
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001016 Project Reference Manual for a complete list of image features that ship
1017 with the Yocto Project.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001018
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001019Customizing Images Using Custom .bb Files
1020-----------------------------------------
1021
1022You can also customize an image by creating a custom recipe that defines
1023additional software as part of the image. The following example shows
Andrew Geisslerc926e172021-05-07 16:11:35 -05001024the form for the two lines you need::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001025
1026 IMAGE_INSTALL = "packagegroup-core-x11-base package1 package2"
1027 inherit core-image
1028
1029Defining the software using a custom recipe gives you total control over
1030the contents of the image. It is important to use the correct names of
Andrew Geissler09036742021-06-25 14:25:14 -05001031packages in the :term:`IMAGE_INSTALL` variable. You must use the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001032OpenEmbedded notation and not the Debian notation for the names (e.g.
1033``glibc-dev`` instead of ``libc6-dev``).
1034
1035The other method for creating a custom image is to base it on an
1036existing image. For example, if you want to create an image based on
1037``core-image-sato`` but add the additional package ``strace`` to the
1038image, copy the ``meta/recipes-sato/images/core-image-sato.bb`` to a new
Andrew Geisslerc926e172021-05-07 16:11:35 -05001039``.bb`` and add the following line to the end of the copy::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001040
1041 IMAGE_INSTALL += "strace"
1042
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001043Customizing Images Using Custom Package Groups
1044----------------------------------------------
1045
1046For complex custom images, the best approach for customizing an image is
1047to create a custom package group recipe that is used to build the image
1048or images. A good example of a package group recipe is
1049``meta/recipes-core/packagegroups/packagegroup-base.bb``.
1050
Andrew Geissler09036742021-06-25 14:25:14 -05001051If you examine that recipe, you see that the :term:`PACKAGES` variable lists
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001052the package group packages to produce. The ``inherit packagegroup``
1053statement sets appropriate default values and automatically adds
1054``-dev``, ``-dbg``, and ``-ptest`` complementary packages for each
Andrew Geissler09036742021-06-25 14:25:14 -05001055package specified in the :term:`PACKAGES` statement.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001056
1057.. note::
1058
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001059 The ``inherit packagegroup`` line should be located near the top of the
Andrew Geissler09036742021-06-25 14:25:14 -05001060 recipe, certainly before the :term:`PACKAGES` statement.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001061
Andrew Geissler09036742021-06-25 14:25:14 -05001062For each package you specify in :term:`PACKAGES`, you can use :term:`RDEPENDS`
1063and :term:`RRECOMMENDS` entries to provide a list of packages the parent
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001064task package should contain. You can see examples of these further down
1065in the ``packagegroup-base.bb`` recipe.
1066
1067Here is a short, fabricated example showing the same basic pieces for a
1068hypothetical packagegroup defined in ``packagegroup-custom.bb``, where
Andrew Geissler09036742021-06-25 14:25:14 -05001069the variable :term:`PN` is the standard way to abbreviate the reference to
Andrew Geisslerc926e172021-05-07 16:11:35 -05001070the full packagegroup name ``packagegroup-custom``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001071
1072 DESCRIPTION = "My Custom Package Groups"
1073
1074 inherit packagegroup
1075
1076 PACKAGES = "\
1077 ${PN}-apps \
1078 ${PN}-tools \
1079 "
1080
Patrick Williams0ca19cc2021-08-16 14:03:13 -05001081 RDEPENDS:${PN}-apps = "\
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001082 dropbear \
1083 portmap \
1084 psplash"
1085
Patrick Williams0ca19cc2021-08-16 14:03:13 -05001086 RDEPENDS:${PN}-tools = "\
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001087 oprofile \
1088 oprofileui-server \
1089 lttng-tools"
1090
Patrick Williams0ca19cc2021-08-16 14:03:13 -05001091 RRECOMMENDS:${PN}-tools = "\
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001092 kernel-module-oprofile"
1093
1094In the previous example, two package group packages are created with
1095their dependencies and their recommended package dependencies listed:
1096``packagegroup-custom-apps``, and ``packagegroup-custom-tools``. To
1097build an image using these package group packages, you need to add
1098``packagegroup-custom-apps`` and/or ``packagegroup-custom-tools`` to
Andrew Geissler09036742021-06-25 14:25:14 -05001099:term:`IMAGE_INSTALL`. For other forms of image dependencies see the other
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001100areas of this section.
1101
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001102Customizing an Image Hostname
1103-----------------------------
1104
1105By default, the configured hostname (i.e. ``/etc/hostname``) in an image
1106is the same as the machine name. For example, if
1107:term:`MACHINE` equals "qemux86", the
1108configured hostname written to ``/etc/hostname`` is "qemux86".
1109
1110You can customize this name by altering the value of the "hostname"
1111variable in the ``base-files`` recipe using either an append file or a
Andrew Geisslerc926e172021-05-07 16:11:35 -05001112configuration file. Use the following in an append file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001113
1114 hostname = "myhostname"
1115
Andrew Geisslerc926e172021-05-07 16:11:35 -05001116Use the following in a configuration file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001117
Patrick Williams0ca19cc2021-08-16 14:03:13 -05001118 hostname:pn-base-files = "myhostname"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001119
1120Changing the default value of the variable "hostname" can be useful in
1121certain situations. For example, suppose you need to do extensive
1122testing on an image and you would like to easily identify the image
1123under test from existing images with typical default hostnames. In this
1124situation, you could change the default hostname to "testme", which
1125results in all the images using the name "testme". Once testing is
1126complete and you do not need to rebuild the image for test any longer,
1127you can easily reset the default hostname.
1128
1129Another point of interest is that if you unset the variable, the image
1130will have no default hostname in the filesystem. Here is an example that
Andrew Geisslerc926e172021-05-07 16:11:35 -05001131unsets the variable in a configuration file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001132
Patrick Williams0ca19cc2021-08-16 14:03:13 -05001133 hostname:pn-base-files = ""
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001134
1135Having no default hostname in the filesystem is suitable for
1136environments that use dynamic hostnames such as virtual machines.
1137
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001138Writing a New Recipe
1139====================
1140
1141Recipes (``.bb`` files) are fundamental components in the Yocto Project
1142environment. Each software component built by the OpenEmbedded build
1143system requires a recipe to define the component. This section describes
1144how to create, write, and test a new recipe.
1145
1146.. note::
1147
1148 For information on variables that are useful for recipes and for
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001149 information about recipe naming issues, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001150 ":ref:`ref-manual/varlocality:recipes`" section of the Yocto Project
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001151 Reference Manual.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001152
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001153Overview
1154--------
1155
1156The following figure shows the basic process for creating a new recipe.
1157The remainder of the section provides details for the steps.
1158
1159.. image:: figures/recipe-workflow.png
1160 :align: center
Andrew Geisslerd5838332022-05-27 11:33:10 -05001161 :width: 50%
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001162
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001163Locate or Automatically Create a Base Recipe
1164--------------------------------------------
1165
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001166You can always write a recipe from scratch. However, there are three choices
1167that can help you quickly get started with a new recipe:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001168
1169- ``devtool add``: A command that assists in creating a recipe and an
1170 environment conducive to development.
1171
1172- ``recipetool create``: A command provided by the Yocto Project that
1173 automates creation of a base recipe based on the source files.
1174
1175- *Existing Recipes:* Location and modification of an existing recipe
1176 that is similar in function to the recipe you need.
1177
1178.. note::
1179
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001180 For information on recipe syntax, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001181 ":ref:`dev-manual/common-tasks:recipe syntax`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001182
1183Creating the Base Recipe Using ``devtool add``
1184~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1185
1186The ``devtool add`` command uses the same logic for auto-creating the
1187recipe as ``recipetool create``, which is listed below. Additionally,
1188however, ``devtool add`` sets up an environment that makes it easy for
1189you to patch the source and to make changes to the recipe as is often
1190necessary when adding a recipe to build a new piece of software to be
1191included in a build.
1192
1193You can find a complete description of the ``devtool add`` command in
Andrew Geissler09209ee2020-12-13 08:44:15 -06001194the ":ref:`sdk-manual/extensible:a closer look at \`\`devtool add\`\``" section
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001195in the Yocto Project Application Development and the Extensible Software
1196Development Kit (eSDK) manual.
1197
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001198Creating the Base Recipe Using ``recipetool create``
1199~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1200
1201``recipetool create`` automates creation of a base recipe given a set of
1202source code files. As long as you can extract or point to the source
1203files, the tool will construct a recipe and automatically configure all
1204pre-build information into the recipe. For example, suppose you have an
1205application that builds using Autotools. Creating the base recipe using
1206``recipetool`` results in a recipe that has the pre-build dependencies,
1207license requirements, and checksums configured.
1208
1209To run the tool, you just need to be in your
1210:term:`Build Directory` and have sourced the
1211build environment setup script (i.e.
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001212:ref:`structure-core-script`).
Andrew Geisslerc926e172021-05-07 16:11:35 -05001213To get help on the tool, use the following command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001214
1215 $ recipetool -h
1216 NOTE: Starting bitbake server...
1217 usage: recipetool [-d] [-q] [--color COLOR] [-h] <subcommand> ...
1218
1219 OpenEmbedded recipe tool
1220
1221 options:
1222 -d, --debug Enable debug output
1223 -q, --quiet Print only errors
1224 --color COLOR Colorize output (where COLOR is auto, always, never)
1225 -h, --help show this help message and exit
1226
1227 subcommands:
1228 create Create a new recipe
1229 newappend Create a bbappend for the specified target in the specified
1230 layer
1231 setvar Set a variable within a recipe
1232 appendfile Create/update a bbappend to replace a target file
1233 appendsrcfiles Create/update a bbappend to add or replace source files
1234 appendsrcfile Create/update a bbappend to add or replace a source file
1235 Use recipetool <subcommand> --help to get help on a specific command
1236
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001237Running ``recipetool create -o OUTFILE`` creates the base recipe and
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001238locates it properly in the layer that contains your source files.
1239Following are some syntax examples:
1240
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001241 - Use this syntax to generate a recipe based on source. Once generated,
Andrew Geisslerc926e172021-05-07 16:11:35 -05001242 the recipe resides in the existing source code layer::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001243
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001244 recipetool create -o OUTFILE source
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001245
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001246 - Use this syntax to generate a recipe using code that
1247 you extract from source. The extracted code is placed in its own layer
Andrew Geissler09036742021-06-25 14:25:14 -05001248 defined by :term:`EXTERNALSRC`.
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001249 ::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001250
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001251 recipetool create -o OUTFILE -x EXTERNALSRC source
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001252
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001253 - Use this syntax to generate a recipe based on source. The options
1254 direct ``recipetool`` to generate debugging information. Once generated,
Andrew Geisslerc926e172021-05-07 16:11:35 -05001255 the recipe resides in the existing source code layer::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001256
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001257 recipetool create -d -o OUTFILE source
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001258
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001259Locating and Using a Similar Recipe
1260~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1261
1262Before writing a recipe from scratch, it is often useful to discover
1263whether someone else has already written one that meets (or comes close
1264to meeting) your needs. The Yocto Project and OpenEmbedded communities
1265maintain many recipes that might be candidates for what you are doing.
Andrew Geisslerd1e89492021-02-12 15:35:20 -06001266You can find a good central index of these recipes in the
1267:oe_layerindex:`OpenEmbedded Layer Index <>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001268
1269Working from an existing recipe or a skeleton recipe is the best way to
1270get started. Here are some points on both methods:
1271
1272- *Locate and modify a recipe that is close to what you want to do:*
1273 This method works when you are familiar with the current recipe
1274 space. The method does not work so well for those new to the Yocto
1275 Project or writing recipes.
1276
1277 Some risks associated with this method are using a recipe that has
1278 areas totally unrelated to what you are trying to accomplish with
1279 your recipe, not recognizing areas of the recipe that you might have
1280 to add from scratch, and so forth. All these risks stem from
1281 unfamiliarity with the existing recipe space.
1282
1283- *Use and modify the following skeleton recipe:* If for some reason
1284 you do not want to use ``recipetool`` and you cannot find an existing
1285 recipe that is close to meeting your needs, you can use the following
1286 structure to provide the fundamental areas of a new recipe.
1287 ::
1288
1289 DESCRIPTION = ""
1290 HOMEPAGE = ""
1291 LICENSE = ""
1292 SECTION = ""
1293 DEPENDS = ""
1294 LIC_FILES_CHKSUM = ""
1295
1296 SRC_URI = ""
1297
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001298Storing and Naming the Recipe
1299-----------------------------
1300
1301Once you have your base recipe, you should put it in your own layer and
1302name it appropriately. Locating it correctly ensures that the
1303OpenEmbedded build system can find it when you use BitBake to process
1304the recipe.
1305
1306- *Storing Your Recipe:* The OpenEmbedded build system locates your
1307 recipe through the layer's ``conf/layer.conf`` file and the
1308 :term:`BBFILES` variable. This
1309 variable sets up a path from which the build system can locate
Andrew Geisslerc926e172021-05-07 16:11:35 -05001310 recipes. Here is the typical use::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001311
1312 BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
1313 ${LAYERDIR}/recipes-*/*/*.bbappend"
1314
1315 Consequently, you need to be sure you locate your new recipe inside
1316 your layer such that it can be found.
1317
1318 You can find more information on how layers are structured in the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001319 ":ref:`dev-manual/common-tasks:understanding and creating layers`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001320
1321- *Naming Your Recipe:* When you name your recipe, you need to follow
Andrew Geisslerc926e172021-05-07 16:11:35 -05001322 this naming convention::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001323
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001324 basename_version.bb
1325
1326 Use lower-cased characters and do not include the reserved suffixes
1327 ``-native``, ``-cross``, ``-initial``, or ``-dev`` casually (i.e. do not use
1328 them as part of your recipe name unless the string applies). Here are some
1329 examples:
1330
1331 .. code-block:: none
1332
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001333 cups_1.7.0.bb
1334 gawk_4.0.2.bb
1335 irssi_0.8.16-rc1.bb
1336
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001337Running a Build on the Recipe
1338-----------------------------
1339
1340Creating a new recipe is usually an iterative process that requires
1341using BitBake to process the recipe multiple times in order to
1342progressively discover and add information to the recipe file.
1343
1344Assuming you have sourced the build environment setup script (i.e.
1345:ref:`structure-core-script`) and you are in
1346the :term:`Build Directory`, use
1347BitBake to process your recipe. All you need to provide is the
Andrew Geisslerc926e172021-05-07 16:11:35 -05001348``basename`` of the recipe as described in the previous section::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001349
1350 $ bitbake basename
1351
1352During the build, the OpenEmbedded build system creates a temporary work
1353directory for each recipe
1354(``${``\ :term:`WORKDIR`\ ``}``)
1355where it keeps extracted source files, log files, intermediate
1356compilation and packaging files, and so forth.
1357
1358The path to the per-recipe temporary work directory depends on the
1359context in which it is being built. The quickest way to find this path
Andrew Geisslerc926e172021-05-07 16:11:35 -05001360is to have BitBake return it by running the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001361
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001362 $ bitbake -e basename | grep ^WORKDIR=
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001363
1364As an example, assume a Source Directory
1365top-level folder named ``poky``, a default Build Directory at
1366``poky/build``, and a ``qemux86-poky-linux`` machine target system.
1367Furthermore, suppose your recipe is named ``foo_1.3.0.bb``. In this
1368case, the work directory the build system uses to build the package
Andrew Geisslerc926e172021-05-07 16:11:35 -05001369would be as follows::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001370
1371 poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0
1372
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001373Inside this directory you can find sub-directories such as ``image``,
1374``packages-split``, and ``temp``. After the build, you can examine these
1375to determine how well the build went.
1376
1377.. note::
1378
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001379 You can find log files for each task in the recipe's ``temp``
1380 directory (e.g. ``poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0/temp``).
1381 Log files are named ``log.taskname`` (e.g. ``log.do_configure``,
1382 ``log.do_fetch``, and ``log.do_compile``).
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001383
1384You can find more information about the build process in
Andrew Geissler09209ee2020-12-13 08:44:15 -06001385":doc:`/overview-manual/development-environment`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001386chapter of the Yocto Project Overview and Concepts Manual.
1387
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001388Fetching Code
1389-------------
1390
1391The first thing your recipe must do is specify how to fetch the source
1392files. Fetching is controlled mainly through the
1393:term:`SRC_URI` variable. Your recipe
Andrew Geissler09036742021-06-25 14:25:14 -05001394must have a :term:`SRC_URI` variable that points to where the source is
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001395located. For a graphical representation of source locations, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001396":ref:`overview-manual/concepts:sources`" section in
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001397the Yocto Project Overview and Concepts Manual.
1398
1399The :ref:`ref-tasks-fetch` task uses
Andrew Geissler09036742021-06-25 14:25:14 -05001400the prefix of each entry in the :term:`SRC_URI` variable value to determine
Andrew Geissler09209ee2020-12-13 08:44:15 -06001401which :ref:`fetcher <bitbake:bitbake-user-manual/bitbake-user-manual-fetching:fetchers>` to use to get your
Andrew Geissler09036742021-06-25 14:25:14 -05001402source files. It is the :term:`SRC_URI` variable that triggers the fetcher.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001403The :ref:`ref-tasks-patch` task uses
1404the variable after source is fetched to apply patches. The OpenEmbedded
1405build system uses
1406:term:`FILESOVERRIDES` for
Andrew Geissler09036742021-06-25 14:25:14 -05001407scanning directory locations for local files in :term:`SRC_URI`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001408
Andrew Geissler09036742021-06-25 14:25:14 -05001409The :term:`SRC_URI` variable in your recipe must define each unique location
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001410for your source files. It is good practice to not hard-code version
Andrew Geissler5f350902021-07-23 13:09:54 -04001411numbers in a URL used in :term:`SRC_URI`. Rather than hard-code these
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001412values, use ``${``\ :term:`PV`\ ``}``,
1413which causes the fetch process to use the version specified in the
1414recipe filename. Specifying the version in this manner means that
1415upgrading the recipe to a future version is as simple as renaming the
1416recipe to match the new version.
1417
1418Here is a simple example from the
1419``meta/recipes-devtools/strace/strace_5.5.bb`` recipe where the source
1420comes from a single tarball. Notice the use of the
Andrew Geisslerc926e172021-05-07 16:11:35 -05001421:term:`PV` variable::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001422
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001423 SRC_URI = "https://strace.io/files/${PV}/strace-${PV}.tar.xz \
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001424
Andrew Geissler09036742021-06-25 14:25:14 -05001425Files mentioned in :term:`SRC_URI` whose names end in a typical archive
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001426extension (e.g. ``.tar``, ``.tar.gz``, ``.tar.bz2``, ``.zip``, and so
1427forth), are automatically extracted during the
1428:ref:`ref-tasks-unpack` task. For
1429another example that specifies these types of files, see the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001430":ref:`dev-manual/common-tasks:autotooled package`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001431
1432Another way of specifying source is from an SCM. For Git repositories,
Andrew Geissler9aee5002022-03-30 16:27:02 +00001433you must specify :term:`SRCREV` and you should specify :term:`PV` to include
1434the revision with :term:`SRCPV`. Here is an example from the recipe
1435``meta/recipes-core/musl/gcompat_git.bb``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001436
Andrew Geissler9aee5002022-03-30 16:27:02 +00001437 SRC_URI = "git://git.adelielinux.org/adelie/gcompat.git;protocol=https;branch=current"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001438
Andrew Geissler9aee5002022-03-30 16:27:02 +00001439 PV = "1.0.0+1.1+git${SRCPV}"
1440 SRCREV = "af5a49e489fdc04b9cf02547650d7aeaccd43793"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001441
Andrew Geissler09036742021-06-25 14:25:14 -05001442If your :term:`SRC_URI` statement includes URLs pointing to individual files
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001443fetched from a remote server other than a version control system,
1444BitBake attempts to verify the files against checksums defined in your
1445recipe to ensure they have not been tampered with or otherwise modified
1446since the recipe was written. Two checksums are used:
1447``SRC_URI[md5sum]`` and ``SRC_URI[sha256sum]``.
1448
Andrew Geissler09036742021-06-25 14:25:14 -05001449If your :term:`SRC_URI` variable points to more than a single URL (excluding
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001450SCM URLs), you need to provide the ``md5`` and ``sha256`` checksums for
1451each URL. For these cases, you provide a name for each URL as part of
Andrew Geissler09036742021-06-25 14:25:14 -05001452the :term:`SRC_URI` and then reference that name in the subsequent checksum
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001453statements. Here is an example combining lines from the files
Andrew Geisslerc926e172021-05-07 16:11:35 -05001454``git.inc`` and ``git_2.24.1.bb``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001455
1456 SRC_URI = "${KERNELORG_MIRROR}/software/scm/git/git-${PV}.tar.gz;name=tarball \
1457 ${KERNELORG_MIRROR}/software/scm/git/git-manpages-${PV}.tar.gz;name=manpages"
1458
1459 SRC_URI[tarball.md5sum] = "166bde96adbbc11c8843d4f8f4f9811b"
1460 SRC_URI[tarball.sha256sum] = "ad5334956301c86841eb1e5b1bb20884a6bad89a10a6762c958220c7cf64da02"
1461 SRC_URI[manpages.md5sum] = "31c2272a8979022497ba3d4202df145d"
1462 SRC_URI[manpages.sha256sum] = "9a7ae3a093bea39770eb96ca3e5b40bff7af0b9f6123f089d7821d0e5b8e1230"
1463
1464Proper values for ``md5`` and ``sha256`` checksums might be available
1465with other signatures on the download page for the upstream source (e.g.
1466``md5``, ``sha1``, ``sha256``, ``GPG``, and so forth). Because the
1467OpenEmbedded build system only deals with ``sha256sum`` and ``md5sum``,
1468you should verify all the signatures you find by hand.
1469
Andrew Geissler09036742021-06-25 14:25:14 -05001470If no :term:`SRC_URI` checksums are specified when you attempt to build the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001471recipe, or you provide an incorrect checksum, the build will produce an
1472error for each missing or incorrect checksum. As part of the error
1473message, the build system provides the checksum string corresponding to
1474the fetched file. Once you have the correct checksums, you can copy and
1475paste them into your recipe and then run the build again to continue.
1476
1477.. note::
1478
1479 As mentioned, if the upstream source provides signatures for
1480 verifying the downloaded source code, you should verify those
1481 manually before setting the checksum values in the recipe and
1482 continuing with the build.
1483
1484This final example is a bit more complicated and is from the
1485``meta/recipes-sato/rxvt-unicode/rxvt-unicode_9.20.bb`` recipe. The
Andrew Geissler09036742021-06-25 14:25:14 -05001486example's :term:`SRC_URI` statement identifies multiple files as the source
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001487files for the recipe: a tarball, a patch file, a desktop file, and an
1488icon.
1489::
1490
1491 SRC_URI = "http://dist.schmorp.de/rxvt-unicode/Attic/rxvt-unicode-${PV}.tar.bz2 \
1492 file://xwc.patch \
1493 file://rxvt.desktop \
1494 file://rxvt.png"
1495
1496When you specify local files using the ``file://`` URI protocol, the
1497build system fetches files from the local machine. The path is relative
1498to the :term:`FILESPATH` variable
1499and searches specific directories in a certain order:
1500``${``\ :term:`BP`\ ``}``,
1501``${``\ :term:`BPN`\ ``}``, and
1502``files``. The directories are assumed to be subdirectories of the
1503directory in which the recipe or append file resides. For another
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001504example that specifies these types of files, see the
1505":ref:`dev-manual/common-tasks:single .c file package (hello world!)`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001506
1507The previous example also specifies a patch file. Patch files are files
1508whose names usually end in ``.patch`` or ``.diff`` but can end with
1509compressed suffixes such as ``diff.gz`` and ``patch.bz2``, for example.
1510The build system automatically applies patches as described in the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001511":ref:`dev-manual/common-tasks:patching code`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001512
Andrew Geissler9aee5002022-03-30 16:27:02 +00001513Fetching Code Through Firewalls
1514~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1515
1516Some users are behind firewalls and need to fetch code through a proxy.
1517See the ":doc:`/ref-manual/faq`" chapter for advice.
1518
1519Limiting the Number of Parallel Connections
1520~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1521
1522Some users are behind firewalls or use servers where the number of parallel
1523connections is limited. In such cases, you can limit the number of fetch
1524tasks being run in parallel by adding the following to your ``local.conf``
1525file::
1526
1527 do_fetch[number_threads] = "4"
1528
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001529Unpacking Code
1530--------------
1531
1532During the build, the
1533:ref:`ref-tasks-unpack` task unpacks
1534the source with ``${``\ :term:`S`\ ``}``
1535pointing to where it is unpacked.
1536
1537If you are fetching your source files from an upstream source archived
1538tarball and the tarball's internal structure matches the common
1539convention of a top-level subdirectory named
1540``${``\ :term:`BPN`\ ``}-${``\ :term:`PV`\ ``}``,
Andrew Geissler09036742021-06-25 14:25:14 -05001541then you do not need to set :term:`S`. However, if :term:`SRC_URI` specifies to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001542fetch source from an archive that does not use this convention, or from
Andrew Geissler09036742021-06-25 14:25:14 -05001543an SCM like Git or Subversion, your recipe needs to define :term:`S`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001544
1545If processing your recipe using BitBake successfully unpacks the source
1546files, you need to be sure that the directory pointed to by ``${S}``
1547matches the structure of the source.
1548
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001549Patching Code
1550-------------
1551
1552Sometimes it is necessary to patch code after it has been fetched. Any
Andrew Geissler09036742021-06-25 14:25:14 -05001553files mentioned in :term:`SRC_URI` whose names end in ``.patch`` or
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001554``.diff`` or compressed versions of these suffixes (e.g. ``diff.gz`` are
1555treated as patches. The
1556:ref:`ref-tasks-patch` task
1557automatically applies these patches.
1558
1559The build system should be able to apply patches with the "-p1" option
1560(i.e. one directory level in the path will be stripped off). If your
1561patch needs to have more directory levels stripped off, specify the
Andrew Geissler09036742021-06-25 14:25:14 -05001562number of levels using the "striplevel" option in the :term:`SRC_URI` entry
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001563for the patch. Alternatively, if your patch needs to be applied in a
1564specific subdirectory that is not specified in the patch file, use the
1565"patchdir" option in the entry.
1566
1567As with all local files referenced in
1568:term:`SRC_URI` using ``file://``,
1569you should place patch files in a directory next to the recipe either
1570named the same as the base name of the recipe
1571(:term:`BP` and
1572:term:`BPN`) or "files".
1573
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001574Licensing
1575---------
1576
1577Your recipe needs to have both the
1578:term:`LICENSE` and
1579:term:`LIC_FILES_CHKSUM`
1580variables:
1581
Andrew Geissler09036742021-06-25 14:25:14 -05001582- :term:`LICENSE`: This variable specifies the license for the software.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001583 If you do not know the license under which the software you are
1584 building is distributed, you should go to the source code and look
1585 for that information. Typical files containing this information
Andrew Geissler09036742021-06-25 14:25:14 -05001586 include ``COPYING``, :term:`LICENSE`, and ``README`` files. You could
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001587 also find the information near the top of a source file. For example,
1588 given a piece of software licensed under the GNU General Public
Andrew Geissler09036742021-06-25 14:25:14 -05001589 License version 2, you would set :term:`LICENSE` as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001590
Andrew Geissler9aee5002022-03-30 16:27:02 +00001591 LICENSE = "GPL-2.0-only"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001592
Andrew Geissler09036742021-06-25 14:25:14 -05001593 The licenses you specify within :term:`LICENSE` can have any name as long
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001594 as you do not use spaces, since spaces are used as separators between
1595 license names. For standard licenses, use the names of the files in
Andrew Geissler09036742021-06-25 14:25:14 -05001596 ``meta/files/common-licenses/`` or the :term:`SPDXLICENSEMAP` flag names
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001597 defined in ``meta/conf/licenses.conf``.
1598
Andrew Geissler09036742021-06-25 14:25:14 -05001599- :term:`LIC_FILES_CHKSUM`: The OpenEmbedded build system uses this
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001600 variable to make sure the license text has not changed. If it has,
1601 the build produces an error and it affords you the chance to figure
1602 it out and correct the problem.
1603
1604 You need to specify all applicable licensing files for the software.
1605 At the end of the configuration step, the build process will compare
1606 the checksums of the files to be sure the text has not changed. Any
1607 differences result in an error with the message containing the
1608 current checksum. For more explanation and examples of how to set the
Andrew Geissler09036742021-06-25 14:25:14 -05001609 :term:`LIC_FILES_CHKSUM` variable, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001610 ":ref:`dev-manual/common-tasks:tracking license changes`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001611
1612 To determine the correct checksum string, you can list the
Andrew Geissler09036742021-06-25 14:25:14 -05001613 appropriate files in the :term:`LIC_FILES_CHKSUM` variable with incorrect
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001614 md5 strings, attempt to build the software, and then note the
1615 resulting error messages that will report the correct md5 strings.
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001616 See the ":ref:`dev-manual/common-tasks:fetching code`" section for
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001617 additional information.
1618
Andrew Geisslerc926e172021-05-07 16:11:35 -05001619 Here is an example that assumes the software has a ``COPYING`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001620
1621 LIC_FILES_CHKSUM = "file://COPYING;md5=xxx"
1622
1623 When you try to build the
1624 software, the build system will produce an error and give you the
1625 correct string that you can substitute into the recipe file for a
1626 subsequent build.
1627
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001628Dependencies
1629------------
1630
1631Most software packages have a short list of other packages that they
1632require, which are called dependencies. These dependencies fall into two
1633main categories: build-time dependencies, which are required when the
1634software is built; and runtime dependencies, which are required to be
1635installed on the target in order for the software to run.
1636
1637Within a recipe, you specify build-time dependencies using the
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001638:term:`DEPENDS` variable. Although there are nuances,
Andrew Geissler09036742021-06-25 14:25:14 -05001639items specified in :term:`DEPENDS` should be names of other
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001640recipes. It is important that you specify all build-time dependencies
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001641explicitly.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001642
1643Another consideration is that configure scripts might automatically
1644check for optional dependencies and enable corresponding functionality
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001645if those dependencies are found. If you wish to make a recipe that is
1646more generally useful (e.g. publish the recipe in a layer for others to
1647use), instead of hard-disabling the functionality, you can use the
1648:term:`PACKAGECONFIG` variable to allow functionality and the
1649corresponding dependencies to be enabled and disabled easily by other
1650users of the recipe.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001651
1652Similar to build-time dependencies, you specify runtime dependencies
1653through a variable -
1654:term:`RDEPENDS`, which is
1655package-specific. All variables that are package-specific need to have
1656the name of the package added to the end as an override. Since the main
1657package for a recipe has the same name as the recipe, and the recipe's
1658name can be found through the
1659``${``\ :term:`PN`\ ``}`` variable, then
1660you specify the dependencies for the main package by setting
Patrick Williams0ca19cc2021-08-16 14:03:13 -05001661``RDEPENDS:${PN}``. If the package were named ``${PN}-tools``, then you
1662would set ``RDEPENDS:${PN}-tools``, and so forth.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001663
1664Some runtime dependencies will be set automatically at packaging time.
1665These dependencies include any shared library dependencies (i.e. if a
1666package "example" contains "libexample" and another package "mypackage"
1667contains a binary that links to "libexample" then the OpenEmbedded build
1668system will automatically add a runtime dependency to "mypackage" on
1669"example"). See the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001670":ref:`overview-manual/concepts:automatically added runtime dependencies`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001671section in the Yocto Project Overview and Concepts Manual for further
1672details.
1673
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001674Configuring the Recipe
1675----------------------
1676
1677Most software provides some means of setting build-time configuration
1678options before compilation. Typically, setting these options is
1679accomplished by running a configure script with options, or by modifying
1680a build configuration file.
1681
1682.. note::
1683
1684 As of Yocto Project Release 1.7, some of the core recipes that
1685 package binary configuration scripts now disable the scripts due to
1686 the scripts previously requiring error-prone path substitution. The
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001687 OpenEmbedded build system uses ``pkg-config`` now, which is much more
1688 robust. You can find a list of the ``*-config`` scripts that are disabled
1689 in the ":ref:`migration-1.7-binary-configuration-scripts-disabled`" section
1690 in the Yocto Project Reference Manual.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001691
1692A major part of build-time configuration is about checking for
1693build-time dependencies and possibly enabling optional functionality as
1694a result. You need to specify any build-time dependencies for the
1695software you are building in your recipe's
1696:term:`DEPENDS` value, in terms of
1697other recipes that satisfy those dependencies. You can often find
1698build-time or runtime dependencies described in the software's
1699documentation.
1700
1701The following list provides configuration items of note based on how
1702your software is built:
1703
1704- *Autotools:* If your source files have a ``configure.ac`` file, then
1705 your software is built using Autotools. If this is the case, you just
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001706 need to modify the configuration.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001707
1708 When using Autotools, your recipe needs to inherit the
1709 :ref:`autotools <ref-classes-autotools>` class
1710 and your recipe does not have to contain a
1711 :ref:`ref-tasks-configure` task.
1712 However, you might still want to make some adjustments. For example,
1713 you can set
1714 :term:`EXTRA_OECONF` or
1715 :term:`PACKAGECONFIG_CONFARGS`
1716 to pass any needed configure options that are specific to the recipe.
1717
1718- *CMake:* If your source files have a ``CMakeLists.txt`` file, then
1719 your software is built using CMake. If this is the case, you just
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001720 need to modify the configuration.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001721
1722 When you use CMake, your recipe needs to inherit the
1723 :ref:`cmake <ref-classes-cmake>` class and your
1724 recipe does not have to contain a
1725 :ref:`ref-tasks-configure` task.
1726 You can make some adjustments by setting
1727 :term:`EXTRA_OECMAKE` to
1728 pass any needed configure options that are specific to the recipe.
1729
1730 .. note::
1731
1732 If you need to install one or more custom CMake toolchain files
1733 that are supplied by the application you are building, install the
Patrick Williams2194f502022-10-16 14:26:09 -05001734 files to ``${D}${datadir}/cmake/Modules`` during :ref:`ref-tasks-install`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001735
1736- *Other:* If your source files do not have a ``configure.ac`` or
1737 ``CMakeLists.txt`` file, then your software is built using some
1738 method other than Autotools or CMake. If this is the case, you
1739 normally need to provide a
1740 :ref:`ref-tasks-configure` task
1741 in your recipe unless, of course, there is nothing to configure.
1742
1743 Even if your software is not being built by Autotools or CMake, you
1744 still might not need to deal with any configuration issues. You need
1745 to determine if configuration is even a required step. You might need
1746 to modify a Makefile or some configuration file used for the build to
1747 specify necessary build options. Or, perhaps you might need to run a
1748 provided, custom configure script with the appropriate options.
1749
1750 For the case involving a custom configure script, you would run
1751 ``./configure --help`` and look for the options you need to set.
1752
1753Once configuration succeeds, it is always good practice to look at the
1754``log.do_configure`` file to ensure that the appropriate options have
1755been enabled and no additional build-time dependencies need to be added
Andrew Geissler09036742021-06-25 14:25:14 -05001756to :term:`DEPENDS`. For example, if the configure script reports that it
1757found something not mentioned in :term:`DEPENDS`, or that it did not find
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001758something that it needed for some desired optional functionality, then
Andrew Geissler09036742021-06-25 14:25:14 -05001759you would need to add those to :term:`DEPENDS`. Looking at the log might
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001760also reveal items being checked for, enabled, or both that you do not
Andrew Geissler09036742021-06-25 14:25:14 -05001761want, or items not being found that are in :term:`DEPENDS`, in which case
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001762you would need to look at passing extra options to the configure script
1763as needed. For reference information on configure options specific to
1764the software you are building, you can consult the output of the
1765``./configure --help`` command within ``${S}`` or consult the software's
1766upstream documentation.
1767
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001768Using Headers to Interface with Devices
1769---------------------------------------
1770
1771If your recipe builds an application that needs to communicate with some
1772device or needs an API into a custom kernel, you will need to provide
1773appropriate header files. Under no circumstances should you ever modify
1774the existing
1775``meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc`` file.
1776These headers are used to build ``libc`` and must not be compromised
1777with custom or machine-specific header information. If you customize
1778``libc`` through modified headers all other applications that use
1779``libc`` thus become affected.
1780
1781.. note::
1782
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001783 Never copy and customize the ``libc`` header file (i.e.
1784 ``meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc``).
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001785
1786The correct way to interface to a device or custom kernel is to use a
1787separate package that provides the additional headers for the driver or
1788other unique interfaces. When doing so, your application also becomes
1789responsible for creating a dependency on that specific provider.
1790
1791Consider the following:
1792
1793- Never modify ``linux-libc-headers.inc``. Consider that file to be
1794 part of the ``libc`` system, and not something you use to access the
1795 kernel directly. You should access ``libc`` through specific ``libc``
1796 calls.
1797
1798- Applications that must talk directly to devices should either provide
1799 necessary headers themselves, or establish a dependency on a special
1800 headers package that is specific to that driver.
1801
1802For example, suppose you want to modify an existing header that adds I/O
1803control or network support. If the modifications are used by a small
1804number programs, providing a unique version of a header is easy and has
1805little impact. When doing so, bear in mind the guidelines in the
1806previous list.
1807
1808.. note::
1809
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001810 If for some reason your changes need to modify the behavior of the ``libc``,
1811 and subsequently all other applications on the system, use a ``.bbappend``
1812 to modify the ``linux-kernel-headers.inc`` file. However, take care to not
1813 make the changes machine specific.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001814
1815Consider a case where your kernel is older and you need an older
1816``libc`` ABI. The headers installed by your recipe should still be a
1817standard mainline kernel, not your own custom one.
1818
1819When you use custom kernel headers you need to get them from
1820:term:`STAGING_KERNEL_DIR`,
1821which is the directory with kernel headers that are required to build
Andrew Geisslerc926e172021-05-07 16:11:35 -05001822out-of-tree modules. Your recipe will also need the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001823
1824 do_configure[depends] += "virtual/kernel:do_shared_workdir"
1825
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001826Compilation
1827-----------
1828
Patrick Williams2194f502022-10-16 14:26:09 -05001829During a build, the :ref:`ref-tasks-compile` task happens after source is fetched,
1830unpacked, and configured. If the recipe passes through :ref:`ref-tasks-compile`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001831successfully, nothing needs to be done.
1832
1833However, if the compile step fails, you need to diagnose the failure.
1834Here are some common issues that cause failures.
1835
1836.. note::
1837
1838 For cases where improper paths are detected for configuration files
1839 or for when libraries/headers cannot be found, be sure you are using
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001840 the more robust ``pkg-config``. See the note in section
Andrew Geissler09209ee2020-12-13 08:44:15 -06001841 ":ref:`dev-manual/common-tasks:Configuring the Recipe`" for additional information.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001842
1843- *Parallel build failures:* These failures manifest themselves as
1844 intermittent errors, or errors reporting that a file or directory
1845 that should be created by some other part of the build process could
1846 not be found. This type of failure can occur even if, upon
1847 inspection, the file or directory does exist after the build has
1848 failed, because that part of the build process happened in the wrong
1849 order.
1850
1851 To fix the problem, you need to either satisfy the missing dependency
1852 in the Makefile or whatever script produced the Makefile, or (as a
Andrew Geisslerc926e172021-05-07 16:11:35 -05001853 workaround) set :term:`PARALLEL_MAKE` to an empty string::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001854
1855 PARALLEL_MAKE = ""
1856
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001857 For information on parallel Makefile issues, see the
1858 ":ref:`dev-manual/common-tasks:debugging parallel make races`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001859
1860- *Improper host path usage:* This failure applies to recipes building
1861 for the target or ``nativesdk`` only. The failure occurs when the
1862 compilation process uses improper headers, libraries, or other files
1863 from the host system when cross-compiling for the target.
1864
1865 To fix the problem, examine the ``log.do_compile`` file to identify
1866 the host paths being used (e.g. ``/usr/include``, ``/usr/lib``, and
1867 so forth) and then either add configure options, apply a patch, or do
1868 both.
1869
1870- *Failure to find required libraries/headers:* If a build-time
1871 dependency is missing because it has not been declared in
1872 :term:`DEPENDS`, or because the
1873 dependency exists but the path used by the build process to find the
1874 file is incorrect and the configure step did not detect it, the
1875 compilation process could fail. For either of these failures, the
1876 compilation process notes that files could not be found. In these
1877 cases, you need to go back and add additional options to the
1878 configure script as well as possibly add additional build-time
Andrew Geissler09036742021-06-25 14:25:14 -05001879 dependencies to :term:`DEPENDS`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001880
1881 Occasionally, it is necessary to apply a patch to the source to
1882 ensure the correct paths are used. If you need to specify paths to
1883 find files staged into the sysroot from other recipes, use the
1884 variables that the OpenEmbedded build system provides (e.g.
Andrew Geissler09036742021-06-25 14:25:14 -05001885 :term:`STAGING_BINDIR`, :term:`STAGING_INCDIR`, :term:`STAGING_DATADIR`, and so
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001886 forth).
1887
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001888Installing
1889----------
1890
Patrick Williams2194f502022-10-16 14:26:09 -05001891During :ref:`ref-tasks-install`, the task copies the built files along with their
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001892hierarchy to locations that would mirror their locations on the target
1893device. The installation process copies files from the
1894``${``\ :term:`S`\ ``}``,
1895``${``\ :term:`B`\ ``}``, and
1896``${``\ :term:`WORKDIR`\ ``}``
1897directories to the ``${``\ :term:`D`\ ``}``
1898directory to create the structure as it should appear on the target
1899system.
1900
1901How your software is built affects what you must do to be sure your
1902software is installed correctly. The following list describes what you
1903must do for installation depending on the type of build system used by
1904the software being built:
1905
1906- *Autotools and CMake:* If the software your recipe is building uses
1907 Autotools or CMake, the OpenEmbedded build system understands how to
1908 install the software. Consequently, you do not have to have a
Patrick Williams2194f502022-10-16 14:26:09 -05001909 :ref:`ref-tasks-install` task as part of your recipe. You just need to make
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001910 sure the install portion of the build completes with no issues.
1911 However, if you wish to install additional files not already being
1912 installed by ``make install``, you should do this using a
Patrick Williams0ca19cc2021-08-16 14:03:13 -05001913 ``do_install:append`` function using the install command as described
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001914 in the "Manual" bulleted item later in this list.
1915
Patrick Williams2194f502022-10-16 14:26:09 -05001916- *Other (using* ``make install``\ *)*: You need to define a :ref:`ref-tasks-install`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001917 function in your recipe. The function should call
1918 ``oe_runmake install`` and will likely need to pass in the
1919 destination directory as well. How you pass that path is dependent on
1920 how the ``Makefile`` being run is written (e.g. ``DESTDIR=${D}``,
1921 ``PREFIX=${D}``, ``INSTALLROOT=${D}``, and so forth).
1922
1923 For an example recipe using ``make install``, see the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001924 ":ref:`dev-manual/common-tasks:makefile-based package`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001925
Patrick Williams2194f502022-10-16 14:26:09 -05001926- *Manual:* You need to define a :ref:`ref-tasks-install` function in your
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001927 recipe. The function must first use ``install -d`` to create the
1928 directories under
1929 ``${``\ :term:`D`\ ``}``. Once the
1930 directories exist, your function can use ``install`` to manually
1931 install the built software into the directories.
1932
1933 You can find more information on ``install`` at
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001934 https://www.gnu.org/software/coreutils/manual/html_node/install-invocation.html.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001935
1936For the scenarios that do not use Autotools or CMake, you need to track
1937the installation and diagnose and fix any issues until everything
1938installs correctly. You need to look in the default location of
1939``${D}``, which is ``${WORKDIR}/image``, to be sure your files have been
1940installed correctly.
1941
1942.. note::
1943
1944 - During the installation process, you might need to modify some of
1945 the installed files to suit the target layout. For example, you
1946 might need to replace hard-coded paths in an initscript with
1947 values of variables provided by the build system, such as
1948 replacing ``/usr/bin/`` with ``${bindir}``. If you do perform such
Patrick Williams2194f502022-10-16 14:26:09 -05001949 modifications during :ref:`ref-tasks-install`, be sure to modify the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001950 destination file after copying rather than before copying.
1951 Modifying after copying ensures that the build system can
Patrick Williams2194f502022-10-16 14:26:09 -05001952 re-execute :ref:`ref-tasks-install` if needed.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001953
1954 - ``oe_runmake install``, which can be run directly or can be run
1955 indirectly by the
1956 :ref:`autotools <ref-classes-autotools>` and
1957 :ref:`cmake <ref-classes-cmake>` classes,
1958 runs ``make install`` in parallel. Sometimes, a Makefile can have
1959 missing dependencies between targets that can result in race
1960 conditions. If you experience intermittent failures during
Patrick Williams2194f502022-10-16 14:26:09 -05001961 :ref:`ref-tasks-install`, you might be able to work around them by disabling
Andrew Geisslerc926e172021-05-07 16:11:35 -05001962 parallel Makefile installs by adding the following to the recipe::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001963
1964 PARALLEL_MAKEINST = ""
1965
1966 See :term:`PARALLEL_MAKEINST` for additional information.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001967
1968 - If you need to install one or more custom CMake toolchain files
1969 that are supplied by the application you are building, install the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001970 files to ``${D}${datadir}/cmake/Modules`` during
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001971 :ref:`ref-tasks-install`.
1972
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001973Enabling System Services
1974------------------------
1975
1976If you want to install a service, which is a process that usually starts
1977on boot and runs in the background, then you must include some
1978additional definitions in your recipe.
1979
1980If you are adding services and the service initialization script or the
1981service file itself is not installed, you must provide for that
Patrick Williams0ca19cc2021-08-16 14:03:13 -05001982installation in your recipe using a ``do_install:append`` function. If
Patrick Williams2194f502022-10-16 14:26:09 -05001983your recipe already has a :ref:`ref-tasks-install` function, update the function
Patrick Williams0ca19cc2021-08-16 14:03:13 -05001984near its end rather than adding an additional ``do_install:append``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001985function.
1986
1987When you create the installation for your services, you need to
1988accomplish what is normally done by ``make install``. In other words,
1989make sure your installation arranges the output similar to how it is
1990arranged on the target system.
1991
1992The OpenEmbedded build system provides support for starting services two
1993different ways:
1994
1995- *SysVinit:* SysVinit is a system and service manager that manages the
1996 init system used to control the very basic functions of your system.
1997 The init program is the first program started by the Linux kernel
1998 when the system boots. Init then controls the startup, running and
1999 shutdown of all other programs.
2000
2001 To enable a service using SysVinit, your recipe needs to inherit the
2002 :ref:`update-rc.d <ref-classes-update-rc.d>`
2003 class. The class helps facilitate safely installing the package on
2004 the target.
2005
2006 You will need to set the
2007 :term:`INITSCRIPT_PACKAGES`,
2008 :term:`INITSCRIPT_NAME`,
2009 and
2010 :term:`INITSCRIPT_PARAMS`
2011 variables within your recipe.
2012
2013- *systemd:* System Management Daemon (systemd) was designed to replace
2014 SysVinit and to provide enhanced management of services. For more
2015 information on systemd, see the systemd homepage at
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002016 https://freedesktop.org/wiki/Software/systemd/.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002017
2018 To enable a service using systemd, your recipe needs to inherit the
2019 :ref:`systemd <ref-classes-systemd>` class. See
2020 the ``systemd.bbclass`` file located in your :term:`Source Directory`
2021 section for
2022 more information.
2023
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002024Packaging
2025---------
2026
2027Successful packaging is a combination of automated processes performed
2028by the OpenEmbedded build system and some specific steps you need to
2029take. The following list describes the process:
2030
Patrick Williams2194f502022-10-16 14:26:09 -05002031- *Splitting Files*: The :ref:`ref-tasks-package` task splits the files produced
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002032 by the recipe into logical components. Even software that produces a
2033 single binary might still have debug symbols, documentation, and
Patrick Williams2194f502022-10-16 14:26:09 -05002034 other logical components that should be split out. The :ref:`ref-tasks-package`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002035 task ensures that files are split up and packaged correctly.
2036
2037- *Running QA Checks*: The
2038 :ref:`insane <ref-classes-insane>` class adds a
2039 step to the package generation process so that output quality
2040 assurance checks are generated by the OpenEmbedded build system. This
2041 step performs a range of checks to be sure the build's output is free
2042 of common problems that show up during runtime. For information on
2043 these checks, see the
2044 :ref:`insane <ref-classes-insane>` class and
Andrew Geissler09209ee2020-12-13 08:44:15 -06002045 the ":ref:`ref-manual/qa-checks:qa error and warning messages`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002046 chapter in the Yocto Project Reference Manual.
2047
2048- *Hand-Checking Your Packages*: After you build your software, you
2049 need to be sure your packages are correct. Examine the
2050 ``${``\ :term:`WORKDIR`\ ``}/packages-split``
2051 directory and make sure files are where you expect them to be. If you
2052 discover problems, you can set
2053 :term:`PACKAGES`,
2054 :term:`FILES`,
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002055 ``do_install(:append)``, and so forth as needed.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002056
2057- *Splitting an Application into Multiple Packages*: If you need to
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05002058 split an application into several packages, see the
2059 ":ref:`dev-manual/common-tasks:splitting an application into multiple packages`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002060 section for an example.
2061
2062- *Installing a Post-Installation Script*: For an example showing how
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05002063 to install a post-installation script, see the
2064 ":ref:`dev-manual/common-tasks:post-installation scripts`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002065
2066- *Marking Package Architecture*: Depending on what your recipe is
2067 building and how it is configured, it might be important to mark the
2068 packages produced as being specific to a particular machine, or to
2069 mark them as not being specific to a particular machine or
2070 architecture at all.
2071
2072 By default, packages apply to any machine with the same architecture
2073 as the target machine. When a recipe produces packages that are
2074 machine-specific (e.g. the
2075 :term:`MACHINE` value is passed
2076 into the configure script or a patch is applied only for a particular
2077 machine), you should mark them as such by adding the following to the
Andrew Geisslerc926e172021-05-07 16:11:35 -05002078 recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002079
2080 PACKAGE_ARCH = "${MACHINE_ARCH}"
2081
2082 On the other hand, if the recipe produces packages that do not
2083 contain anything specific to the target machine or architecture at
2084 all (e.g. recipes that simply package script files or configuration
2085 files), you should use the
2086 :ref:`allarch <ref-classes-allarch>` class to
Andrew Geisslerc926e172021-05-07 16:11:35 -05002087 do this for you by adding this to your recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002088
2089 inherit allarch
2090
2091 Ensuring that the package architecture is correct is not critical
2092 while you are doing the first few builds of your recipe. However, it
2093 is important in order to ensure that your recipe rebuilds (or does
2094 not rebuild) appropriately in response to changes in configuration,
2095 and to ensure that you get the appropriate packages installed on the
2096 target machine, particularly if you run separate builds for more than
2097 one target machine.
2098
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002099Sharing Files Between Recipes
2100-----------------------------
2101
2102Recipes often need to use files provided by other recipes on the build
2103host. For example, an application linking to a common library needs
2104access to the library itself and its associated headers. The way this
2105access is accomplished is by populating a sysroot with files. Each
2106recipe has two sysroots in its work directory, one for target files
2107(``recipe-sysroot``) and one for files that are native to the build host
2108(``recipe-sysroot-native``).
2109
2110.. note::
2111
2112 You could find the term "staging" used within the Yocto project
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002113 regarding files populating sysroots (e.g. the :term:`STAGING_DIR`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002114 variable).
2115
2116Recipes should never populate the sysroot directly (i.e. write files
2117into sysroot). Instead, files should be installed into standard
2118locations during the
2119:ref:`ref-tasks-install` task within
2120the ``${``\ :term:`D`\ ``}`` directory. The
2121reason for this limitation is that almost all files that populate the
2122sysroot are cataloged in manifests in order to ensure the files can be
2123removed later when a recipe is either modified or removed. Thus, the
2124sysroot is able to remain free from stale files.
2125
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05002126A subset of the files installed by the :ref:`ref-tasks-install` task are
Patrick Williams975a06f2022-10-21 14:42:47 -05002127used by the :ref:`ref-tasks-populate_sysroot` task as defined by the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05002128:term:`SYSROOT_DIRS` variable to automatically populate the sysroot. It
2129is possible to modify the list of directories that populate the sysroot.
2130The following example shows how you could add the ``/opt`` directory to
Andrew Geisslerc926e172021-05-07 16:11:35 -05002131the list of directories within a recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002132
2133 SYSROOT_DIRS += "/opt"
2134
Andrew Geisslerd1e89492021-02-12 15:35:20 -06002135.. note::
2136
2137 The `/sysroot-only` is to be used by recipes that generate artifacts
2138 that are not included in the target filesystem, allowing them to share
Andrew Geissler09036742021-06-25 14:25:14 -05002139 these artifacts without needing to use the :term:`DEPLOY_DIR`.
Andrew Geisslerd1e89492021-02-12 15:35:20 -06002140
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05002141For a more complete description of the :ref:`ref-tasks-populate_sysroot`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002142task and its associated functions, see the
2143:ref:`staging <ref-classes-staging>` class.
2144
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002145Using Virtual Providers
2146-----------------------
2147
2148Prior to a build, if you know that several different recipes provide the
2149same functionality, you can use a virtual provider (i.e. ``virtual/*``)
2150as a placeholder for the actual provider. The actual provider is
2151determined at build-time.
2152
2153A common scenario where a virtual provider is used would be for the
2154kernel recipe. Suppose you have three kernel recipes whose
2155:term:`PN` values map to ``kernel-big``,
2156``kernel-mid``, and ``kernel-small``. Furthermore, each of these recipes
2157in some way uses a :term:`PROVIDES`
2158statement that essentially identifies itself as being able to provide
2159``virtual/kernel``. Here is one way through the
Andrew Geisslerc926e172021-05-07 16:11:35 -05002160:ref:`kernel <ref-classes-kernel>` class::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002161
Andrew Geissler7e0e3c02022-02-25 20:34:39 +00002162 PROVIDES += "virtual/kernel"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002163
Andrew Geissler595f6302022-01-24 19:11:47 +00002164Any recipe that inherits the :ref:`kernel <ref-classes-kernel>` class is
Andrew Geissler09036742021-06-25 14:25:14 -05002165going to utilize a :term:`PROVIDES` statement that identifies that recipe as
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002166being able to provide the ``virtual/kernel`` item.
2167
2168Now comes the time to actually build an image and you need a kernel
2169recipe, but which one? You can configure your build to call out the
Andrew Geissler09209ee2020-12-13 08:44:15 -06002170kernel recipe you want by using the :term:`PREFERRED_PROVIDER` variable. As
2171an example, consider the :yocto_git:`x86-base.inc
Andrew Geisslerd159c7f2021-09-02 21:05:58 -05002172</poky/tree/meta/conf/machine/include/x86/x86-base.inc>` include file, which is a
Andrew Geissler09209ee2020-12-13 08:44:15 -06002173machine (i.e. :term:`MACHINE`) configuration file. This include file is the
2174reason all x86-based machines use the ``linux-yocto`` kernel. Here are the
Andrew Geisslerc926e172021-05-07 16:11:35 -05002175relevant lines from the include file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002176
2177 PREFERRED_PROVIDER_virtual/kernel ??= "linux-yocto"
2178 PREFERRED_VERSION_linux-yocto ??= "4.15%"
2179
2180When you use a virtual provider, you do not have to "hard code" a recipe
2181name as a build dependency. You can use the
2182:term:`DEPENDS` variable to state the
Andrew Geisslerc926e172021-05-07 16:11:35 -05002183build is dependent on ``virtual/kernel`` for example::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002184
2185 DEPENDS = "virtual/kernel"
2186
2187During the build, the OpenEmbedded build system picks
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002188the correct recipe needed for the ``virtual/kernel`` dependency based on
Andrew Geissler09036742021-06-25 14:25:14 -05002189the :term:`PREFERRED_PROVIDER` variable. If you want to use the small kernel
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002190mentioned at the beginning of this section, configure your build as
Andrew Geisslerc926e172021-05-07 16:11:35 -05002191follows::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002192
2193 PREFERRED_PROVIDER_virtual/kernel ??= "kernel-small"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002194
2195.. note::
2196
Andrew Geissler09036742021-06-25 14:25:14 -05002197 Any recipe that :term:`PROVIDES` a ``virtual/*`` item that is ultimately not
2198 selected through :term:`PREFERRED_PROVIDER` does not get built. Preventing these
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002199 recipes from building is usually the desired behavior since this mechanism's
2200 purpose is to select between mutually exclusive alternative providers.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002201
2202The following lists specific examples of virtual providers:
2203
2204- ``virtual/kernel``: Provides the name of the kernel recipe to use
2205 when building a kernel image.
2206
2207- ``virtual/bootloader``: Provides the name of the bootloader to use
2208 when building an image.
2209
2210- ``virtual/libgbm``: Provides ``gbm.pc``.
2211
2212- ``virtual/egl``: Provides ``egl.pc`` and possibly ``wayland-egl.pc``.
2213
2214- ``virtual/libgl``: Provides ``gl.pc`` (i.e. libGL).
2215
2216- ``virtual/libgles1``: Provides ``glesv1_cm.pc`` (i.e. libGLESv1_CM).
2217
2218- ``virtual/libgles2``: Provides ``glesv2.pc`` (i.e. libGLESv2).
2219
2220.. note::
2221
2222 Virtual providers only apply to build time dependencies specified with
2223 :term:`PROVIDES` and :term:`DEPENDS`. They do not apply to runtime
2224 dependencies specified with :term:`RPROVIDES` and :term:`RDEPENDS`.
2225
2226Properly Versioning Pre-Release Recipes
2227---------------------------------------
2228
2229Sometimes the name of a recipe can lead to versioning problems when the
2230recipe is upgraded to a final release. For example, consider the
2231``irssi_0.8.16-rc1.bb`` recipe file in the list of example recipes in
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05002232the ":ref:`dev-manual/common-tasks:storing and naming the recipe`" section.
2233This recipe is at a release candidate stage (i.e. "rc1"). When the recipe is
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002234released, the recipe filename becomes ``irssi_0.8.16.bb``. The version
2235change from ``0.8.16-rc1`` to ``0.8.16`` is seen as a decrease by the
2236build system and package managers, so the resulting packages will not
2237correctly trigger an upgrade.
2238
2239In order to ensure the versions compare properly, the recommended
2240convention is to set :term:`PV` within the
2241recipe to "previous_version+current_version". You can use an additional
2242variable so that you can use the current version elsewhere. Here is an
Andrew Geisslerc926e172021-05-07 16:11:35 -05002243example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002244
2245 REALPV = "0.8.16-rc1"
2246 PV = "0.8.15+${REALPV}"
2247
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002248Post-Installation Scripts
2249-------------------------
2250
2251Post-installation scripts run immediately after installing a package on
2252the target or during image creation when a package is included in an
2253image. To add a post-installation script to a package, add a
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002254``pkg_postinst:``\ `PACKAGENAME`\ ``()`` function to the recipe file
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002255(``.bb``) and replace `PACKAGENAME` with the name of the package you want
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002256to attach to the ``postinst`` script. To apply the post-installation
2257script to the main package for the recipe, which is usually what is
2258required, specify
2259``${``\ :term:`PN`\ ``}`` in place of
2260PACKAGENAME.
2261
Andrew Geisslerc926e172021-05-07 16:11:35 -05002262A post-installation function has the following structure::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002263
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002264 pkg_postinst:PACKAGENAME() {
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002265 # Commands to carry out
2266 }
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002267
2268The script defined in the post-installation function is called when the
2269root filesystem is created. If the script succeeds, the package is
2270marked as installed.
2271
2272.. note::
2273
2274 Any RPM post-installation script that runs on the target should
2275 return a 0 exit code. RPM does not allow non-zero exit codes for
2276 these scripts, and the RPM package manager will cause the package to
2277 fail installation on the target.
2278
2279Sometimes it is necessary for the execution of a post-installation
2280script to be delayed until the first boot. For example, the script might
2281need to be executed on the device itself. To delay script execution
2282until boot time, you must explicitly mark post installs to defer to the
2283target. You can use ``pkg_postinst_ontarget()`` or call
2284``postinst_intercept delay_to_first_boot`` from ``pkg_postinst()``. Any
2285failure of a ``pkg_postinst()`` script (including exit 1) triggers an
2286error during the
2287:ref:`ref-tasks-rootfs` task.
2288
2289If you have recipes that use ``pkg_postinst`` function and they require
2290the use of non-standard native tools that have dependencies during
Andrew Geissler595f6302022-01-24 19:11:47 +00002291root filesystem construction, you need to use the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002292:term:`PACKAGE_WRITE_DEPS`
2293variable in your recipe to list these tools. If you do not use this
2294variable, the tools might be missing and execution of the
2295post-installation script is deferred until first boot. Deferring the
Andrew Geissler595f6302022-01-24 19:11:47 +00002296script to the first boot is undesirable and impossible for read-only
2297root filesystems.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002298
2299.. note::
2300
William A. Kennington IIIac69b482021-06-02 12:28:27 -07002301 There is equivalent support for pre-install, pre-uninstall, and post-uninstall
2302 scripts by way of ``pkg_preinst``, ``pkg_prerm``, and ``pkg_postrm``,
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002303 respectively. These scrips work in exactly the same way as does
2304 ``pkg_postinst`` with the exception that they run at different times. Also,
2305 because of when they run, they are not applicable to being run at image
2306 creation time like ``pkg_postinst``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002307
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002308Testing
2309-------
2310
2311The final step for completing your recipe is to be sure that the
2312software you built runs correctly. To accomplish runtime testing, add
2313the build's output packages to your image and test them on the target.
2314
2315For information on how to customize your image by adding specific
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05002316packages, see ":ref:`dev-manual/common-tasks:customizing images`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002317
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002318Examples
2319--------
2320
2321To help summarize how to write a recipe, this section provides some
2322examples given various scenarios:
2323
2324- Recipes that use local files
2325
2326- Using an Autotooled package
2327
2328- Using a Makefile-based package
2329
2330- Splitting an application into multiple packages
2331
2332- Adding binaries to an image
2333
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002334Single .c File Package (Hello World!)
2335~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2336
2337Building an application from a single file that is stored locally (e.g.
2338under ``files``) requires a recipe that has the file listed in the
Andrew Geissler09036742021-06-25 14:25:14 -05002339:term:`SRC_URI` variable. Additionally, you need to manually write the
Patrick Williams2194f502022-10-16 14:26:09 -05002340:ref:`ref-tasks-compile` and :ref:`ref-tasks-install` tasks. The :term:`S` variable defines the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002341directory containing the source code, which is set to
Andrew Geissler615f2f12022-07-15 14:00:58 -05002342:term:`WORKDIR` in this case --- the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002343directory BitBake uses for the build.
2344::
2345
2346 SUMMARY = "Simple helloworld application"
2347 SECTION = "examples"
2348 LICENSE = "MIT"
2349 LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
2350
2351 SRC_URI = "file://helloworld.c"
2352
2353 S = "${WORKDIR}"
2354
2355 do_compile() {
Andrew Geisslerd1e89492021-02-12 15:35:20 -06002356 ${CC} ${LDFLAGS} helloworld.c -o helloworld
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002357 }
2358
2359 do_install() {
2360 install -d ${D}${bindir}
2361 install -m 0755 helloworld ${D}${bindir}
2362 }
2363
2364By default, the ``helloworld``, ``helloworld-dbg``, and
2365``helloworld-dev`` packages are built. For information on how to
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05002366customize the packaging process, see the
2367":ref:`dev-manual/common-tasks:splitting an application into multiple packages`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002368section.
2369
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002370Autotooled Package
2371~~~~~~~~~~~~~~~~~~
2372
2373Applications that use Autotools such as ``autoconf`` and ``automake``
Andrew Geissler09036742021-06-25 14:25:14 -05002374require a recipe that has a source archive listed in :term:`SRC_URI` and
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002375also inherit the
2376:ref:`autotools <ref-classes-autotools>` class,
2377which contains the definitions of all the steps needed to build an
2378Autotool-based application. The result of the build is automatically
2379packaged. And, if the application uses NLS for localization, packages
2380with local information are generated (one package per language).
2381Following is one example: (``hello_2.3.bb``)
2382::
2383
2384 SUMMARY = "GNU Helloworld application"
2385 SECTION = "examples"
Andrew Geissler9aee5002022-03-30 16:27:02 +00002386 LICENSE = "GPL-2.0-or-later"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002387 LIC_FILES_CHKSUM = "file://COPYING;md5=751419260aa954499f7abaabaa882bbe"
2388
2389 SRC_URI = "${GNU_MIRROR}/hello/hello-${PV}.tar.gz"
2390
2391 inherit autotools gettext
2392
Andrew Geissler09036742021-06-25 14:25:14 -05002393The variable :term:`LIC_FILES_CHKSUM` is used to track source license
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002394changes as described in the
Andrew Geissler09209ee2020-12-13 08:44:15 -06002395":ref:`dev-manual/common-tasks:tracking license changes`" section in
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002396the Yocto Project Overview and Concepts Manual. You can quickly create
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002397Autotool-based recipes in a manner similar to the previous example.
2398
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002399Makefile-Based Package
2400~~~~~~~~~~~~~~~~~~~~~~
2401
2402Applications that use GNU ``make`` also require a recipe that has the
Andrew Geissler09036742021-06-25 14:25:14 -05002403source archive listed in :term:`SRC_URI`. You do not need to add a
Patrick Williams2194f502022-10-16 14:26:09 -05002404:ref:`ref-tasks-compile` step since by default BitBake starts the ``make`` command
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002405to compile the application. If you need additional ``make`` options, you
2406should store them in the
2407:term:`EXTRA_OEMAKE` or
2408:term:`PACKAGECONFIG_CONFARGS`
2409variables. BitBake passes these options into the GNU ``make``
Patrick Williams2194f502022-10-16 14:26:09 -05002410invocation. Note that a :ref:`ref-tasks-install` task is still required.
2411Otherwise, BitBake runs an empty :ref:`ref-tasks-install` task by default.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002412
2413Some applications might require extra parameters to be passed to the
2414compiler. For example, the application might need an additional header
Andrew Geissler09036742021-06-25 14:25:14 -05002415path. You can accomplish this by adding to the :term:`CFLAGS` variable. The
Andrew Geisslerc926e172021-05-07 16:11:35 -05002416following example shows this::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002417
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002418 CFLAGS:prepend = "-I ${S}/include "
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002419
Andrew Geissler9aee5002022-03-30 16:27:02 +00002420In the following example, ``lz4`` is a makefile-based package::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002421
Andrew Geissler9aee5002022-03-30 16:27:02 +00002422 SUMMARY = "Extremely Fast Compression algorithm"
2423 DESCRIPTION = "LZ4 is a very fast lossless compression algorithm, providing compression speed at 400 MB/s per core, scalable with multi-cores CPU. It also features an extremely fast decoder, with speed in multiple GB/s per core, typically reaching RAM speed limits on multi-core systems."
2424 HOMEPAGE = "https://github.com/lz4/lz4"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002425
Andrew Geissler9aee5002022-03-30 16:27:02 +00002426 LICENSE = "BSD-2-Clause | GPL-2.0-only"
2427 LIC_FILES_CHKSUM = "file://lib/LICENSE;md5=ebc2ea4814a64de7708f1571904b32cc \
2428 file://programs/COPYING;md5=b234ee4d69f5fce4486a80fdaf4a4263 \
2429 file://LICENSE;md5=d57c0d21cb917fb4e0af2454aa48b956 \
2430 "
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002431
Andrew Geissler9aee5002022-03-30 16:27:02 +00002432 PE = "1"
2433
2434 SRCREV = "d44371841a2f1728a3f36839fd4b7e872d0927d3"
2435
2436 SRC_URI = "git://github.com/lz4/lz4.git;branch=release;protocol=https \
2437 file://CVE-2021-3520.patch \
2438 "
2439 UPSTREAM_CHECK_GITTAGREGEX = "v(?P<pver>.*)"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002440
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002441 S = "${WORKDIR}/git"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002442
Andrew Geissler9aee5002022-03-30 16:27:02 +00002443 # Fixed in r118, which is larger than the current version.
2444 CVE_CHECK_IGNORE += "CVE-2014-4715"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002445
Andrew Geissler9aee5002022-03-30 16:27:02 +00002446 EXTRA_OEMAKE = "PREFIX=${prefix} CC='${CC}' CFLAGS='${CFLAGS}' DESTDIR=${D} LIBDIR=${libdir} INCLUDEDIR=${includedir} BUILD_STATIC=no"
2447
2448 do_install() {
2449 oe_runmake install
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002450 }
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002451
Andrew Geissler9aee5002022-03-30 16:27:02 +00002452 BBCLASSEXTEND = "native nativesdk"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002453
2454Splitting an Application into Multiple Packages
2455~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2456
Andrew Geissler09036742021-06-25 14:25:14 -05002457You can use the variables :term:`PACKAGES` and :term:`FILES` to split an
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002458application into multiple packages.
2459
2460Following is an example that uses the ``libxpm`` recipe. By default,
2461this recipe generates a single package that contains the library along
2462with a few binaries. You can modify the recipe to split the binaries
Andrew Geisslerc926e172021-05-07 16:11:35 -05002463into separate packages::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002464
2465 require xorg-lib-common.inc
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002466
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002467 SUMMARY = "Xpm: X Pixmap extension library"
Andrew Geissler5199d832021-09-24 16:47:35 -05002468 LICENSE = "MIT"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002469 LIC_FILES_CHKSUM = "file://COPYING;md5=51f4270b012ecd4ab1a164f5f4ed6cf7"
2470 DEPENDS += "libxext libsm libxt"
2471 PE = "1"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002472
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002473 XORG_PN = "libXpm"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002474
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002475 PACKAGES =+ "sxpm cxpm"
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002476 FILES:cxpm = "${bindir}/cxpm"
2477 FILES:sxpm = "${bindir}/sxpm"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002478
2479In the previous example, we want to ship the ``sxpm`` and ``cxpm``
2480binaries in separate packages. Since ``bindir`` would be packaged into
Andrew Geissler09036742021-06-25 14:25:14 -05002481the main :term:`PN` package by default, we prepend the :term:`PACKAGES` variable
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002482so additional package names are added to the start of list. This results
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002483in the extra ``FILES:*`` variables then containing information that
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002484define which files and directories go into which packages. Files
2485included by earlier packages are skipped by latter packages. Thus, the
Andrew Geissler09036742021-06-25 14:25:14 -05002486main :term:`PN` package does not include the above listed files.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002487
2488Packaging Externally Produced Binaries
2489~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2490
2491Sometimes, you need to add pre-compiled binaries to an image. For
William A. Kennington IIIac69b482021-06-02 12:28:27 -07002492example, suppose that there are binaries for proprietary code,
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002493created by a particular division of a company. Your part of the company
2494needs to use those binaries as part of an image that you are building
2495using the OpenEmbedded build system. Since you only have the binaries
2496and not the source code, you cannot use a typical recipe that expects to
2497fetch the source specified in
2498:term:`SRC_URI` and then compile it.
2499
2500One method is to package the binaries and then install them as part of
2501the image. Generally, it is not a good idea to package binaries since,
2502among other things, it can hinder the ability to reproduce builds and
2503could lead to compatibility problems with ABI in the future. However,
2504sometimes you have no choice.
2505
2506The easiest solution is to create a recipe that uses the
2507:ref:`bin_package <ref-classes-bin-package>` class
2508and to be sure that you are using default locations for build artifacts.
Andrew Geissler595f6302022-01-24 19:11:47 +00002509In most cases, the :ref:`bin_package <ref-classes-bin-package>` class handles "skipping" the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002510configure and compile steps as well as sets things up to grab packages
2511from the appropriate area. In particular, this class sets ``noexec`` on
2512both the :ref:`ref-tasks-configure`
2513and :ref:`ref-tasks-compile` tasks,
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002514sets ``FILES:${PN}`` to "/" so that it picks up all files, and sets up a
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002515:ref:`ref-tasks-install` task, which
2516effectively copies all files from ``${S}`` to ``${D}``. The
Andrew Geissler595f6302022-01-24 19:11:47 +00002517:ref:`bin_package <ref-classes-bin-package>` class works well when the files extracted into ``${S}``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002518are already laid out in the way they should be laid out on the target.
2519For more information on these variables, see the
2520:term:`FILES`,
2521:term:`PN`,
2522:term:`S`, and
2523:term:`D` variables in the Yocto Project
2524Reference Manual's variable glossary.
2525
2526.. note::
2527
2528 - Using :term:`DEPENDS` is a good
2529 idea even for components distributed in binary form, and is often
2530 necessary for shared libraries. For a shared library, listing the
Andrew Geissler09036742021-06-25 14:25:14 -05002531 library dependencies in :term:`DEPENDS` makes sure that the libraries
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002532 are available in the staging sysroot when other recipes link
2533 against the library, which might be necessary for successful
2534 linking.
2535
Andrew Geissler09036742021-06-25 14:25:14 -05002536 - Using :term:`DEPENDS` also allows runtime dependencies between
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002537 packages to be added automatically. See the
Andrew Geissler09209ee2020-12-13 08:44:15 -06002538 ":ref:`overview-manual/concepts:automatically added runtime dependencies`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002539 section in the Yocto Project Overview and Concepts Manual for more
2540 information.
2541
Andrew Geissler595f6302022-01-24 19:11:47 +00002542If you cannot use the :ref:`bin_package <ref-classes-bin-package>` class, you need to be sure you are
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002543doing the following:
2544
2545- Create a recipe where the
2546 :ref:`ref-tasks-configure` and
2547 :ref:`ref-tasks-compile` tasks do
2548 nothing: It is usually sufficient to just not define these tasks in
2549 the recipe, because the default implementations do nothing unless a
2550 Makefile is found in
2551 ``${``\ :term:`S`\ ``}``.
2552
2553 If ``${S}`` might contain a Makefile, or if you inherit some class
Patrick Williams2194f502022-10-16 14:26:09 -05002554 that replaces :ref:`ref-tasks-configure` and :ref:`ref-tasks-compile` with custom
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002555 versions, then you can use the
2556 ``[``\ :ref:`noexec <bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ``]``
Andrew Geisslerc926e172021-05-07 16:11:35 -05002557 flag to turn the tasks into no-ops, as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002558
2559 do_configure[noexec] = "1"
2560 do_compile[noexec] = "1"
2561
2562 Unlike
2563 :ref:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata:deleting a task`,
2564 using the flag preserves the dependency chain from the
2565 :ref:`ref-tasks-fetch`,
2566 :ref:`ref-tasks-unpack`, and
2567 :ref:`ref-tasks-patch` tasks to the
2568 :ref:`ref-tasks-install` task.
2569
Patrick Williams2194f502022-10-16 14:26:09 -05002570- Make sure your :ref:`ref-tasks-install` task installs the binaries
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002571 appropriately.
2572
2573- Ensure that you set up :term:`FILES`
2574 (usually
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002575 ``FILES:${``\ :term:`PN`\ ``}``) to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002576 point to the files you have installed, which of course depends on
2577 where you have installed them and whether those files are in
2578 different locations than the defaults.
2579
2580Following Recipe Style Guidelines
2581---------------------------------
2582
2583When writing recipes, it is good to conform to existing style
Andrew Geisslerd1e89492021-02-12 15:35:20 -06002584guidelines. The :oe_wiki:`OpenEmbedded Styleguide </Styleguide>` wiki page
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002585provides rough guidelines for preferred recipe style.
2586
2587It is common for existing recipes to deviate a bit from this style.
2588However, aiming for at least a consistent style is a good idea. Some
2589practices, such as omitting spaces around ``=`` operators in assignments
2590or ordering recipe components in an erratic way, are widely seen as poor
2591style.
2592
2593Recipe Syntax
2594-------------
2595
2596Understanding recipe file syntax is important for writing recipes. The
2597following list overviews the basic items that make up a BitBake recipe
2598file. For more complete BitBake syntax descriptions, see the
Andrew Geissler615f2f12022-07-15 14:00:58 -05002599":doc:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002600chapter of the BitBake User Manual.
2601
2602- *Variable Assignments and Manipulations:* Variable assignments allow
2603 a value to be assigned to a variable. The assignment can be static
2604 text or might include the contents of other variables. In addition to
2605 the assignment, appending and prepending operations are also
2606 supported.
2607
2608 The following example shows some of the ways you can use variables in
Andrew Geisslerc926e172021-05-07 16:11:35 -05002609 recipes::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002610
2611 S = "${WORKDIR}/postfix-${PV}"
2612 CFLAGS += "-DNO_ASM"
Andrew Geissler87f5cff2022-09-30 13:13:31 -05002613 CFLAGS:append = " --enable-important-feature"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002614
2615- *Functions:* Functions provide a series of actions to be performed.
2616 You usually use functions to override the default implementation of a
2617 task function or to complement a default function (i.e. append or
2618 prepend to an existing function). Standard functions use ``sh`` shell
2619 syntax, although access to OpenEmbedded variables and internal
2620 methods are also available.
2621
William A. Kennington IIIac69b482021-06-02 12:28:27 -07002622 Here is an example function from the ``sed`` recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002623
2624 do_install () {
2625 autotools_do_install
2626 install -d ${D}${base_bindir}
2627 mv ${D}${bindir}/sed ${D}${base_bindir}/sed
2628 rmdir ${D}${bindir}/
2629 }
2630
2631 It is
2632 also possible to implement new functions that are called between
2633 existing tasks as long as the new functions are not replacing or
2634 complementing the default functions. You can implement functions in
2635 Python instead of shell. Both of these options are not seen in the
2636 majority of recipes.
2637
2638- *Keywords:* BitBake recipes use only a few keywords. You use keywords
2639 to include common functions (``inherit``), load parts of a recipe
2640 from other files (``include`` and ``require``) and export variables
2641 to the environment (``export``).
2642
Andrew Geisslerc926e172021-05-07 16:11:35 -05002643 The following example shows the use of some of these keywords::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002644
2645 export POSTCONF = "${STAGING_BINDIR}/postconf"
2646 inherit autoconf
2647 require otherfile.inc
2648
2649- *Comments (#):* Any lines that begin with the hash character (``#``)
Andrew Geisslerc926e172021-05-07 16:11:35 -05002650 are treated as comment lines and are ignored::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002651
2652 # This is a comment
2653
2654This next list summarizes the most important and most commonly used
2655parts of the recipe syntax. For more information on these parts of the
2656syntax, you can reference the
Andrew Geissler615f2f12022-07-15 14:00:58 -05002657":doc:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata`" chapter
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002658in the BitBake User Manual.
2659
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002660- *Line Continuation (\\):* Use the backward slash (``\``) character to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002661 split a statement over multiple lines. Place the slash character at
Andrew Geisslerc926e172021-05-07 16:11:35 -05002662 the end of the line that is to be continued on the next line::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002663
2664 VAR = "A really long \
2665 line"
2666
2667 .. note::
2668
2669 You cannot have any characters including spaces or tabs after the
2670 slash character.
2671
2672- *Using Variables (${VARNAME}):* Use the ``${VARNAME}`` syntax to
Andrew Geisslerc926e172021-05-07 16:11:35 -05002673 access the contents of a variable::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002674
2675 SRC_URI = "${SOURCEFORGE_MIRROR}/libpng/zlib-${PV}.tar.gz"
2676
2677 .. note::
2678
2679 It is important to understand that the value of a variable
2680 expressed in this form does not get substituted automatically. The
2681 expansion of these expressions happens on-demand later (e.g.
2682 usually when a function that makes reference to the variable
2683 executes). This behavior ensures that the values are most
2684 appropriate for the context in which they are finally used. On the
2685 rare occasion that you do need the variable expression to be
2686 expanded immediately, you can use the
2687 :=
2688 operator instead of
2689 =
2690 when you make the assignment, but this is not generally needed.
2691
2692- *Quote All Assignments ("value"):* Use double quotes around values in
Andrew Geisslerc926e172021-05-07 16:11:35 -05002693 all variable assignments (e.g. ``"value"``). Following is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002694
2695 VAR1 = "${OTHERVAR}"
2696 VAR2 = "The version is ${PV}"
2697
2698- *Conditional Assignment (?=):* Conditional assignment is used to
2699 assign a value to a variable, but only when the variable is currently
2700 unset. Use the question mark followed by the equal sign (``?=``) to
2701 make a "soft" assignment used for conditional assignment. Typically,
2702 "soft" assignments are used in the ``local.conf`` file for variables
2703 that are allowed to come through from the external environment.
2704
2705 Here is an example where ``VAR1`` is set to "New value" if it is
2706 currently empty. However, if ``VAR1`` has already been set, it
Andrew Geisslerc926e172021-05-07 16:11:35 -05002707 remains unchanged::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002708
2709 VAR1 ?= "New value"
2710
Andrew Geisslerc926e172021-05-07 16:11:35 -05002711 In this next example, ``VAR1`` is left with the value "Original value"::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002712
2713 VAR1 = "Original value"
2714 VAR1 ?= "New value"
2715
2716- *Appending (+=):* Use the plus character followed by the equals sign
2717 (``+=``) to append values to existing variables.
2718
2719 .. note::
2720
2721 This operator adds a space between the existing content of the
2722 variable and the new content.
2723
Andrew Geisslerc926e172021-05-07 16:11:35 -05002724 Here is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002725
2726 SRC_URI += "file://fix-makefile.patch"
2727
2728- *Prepending (=+):* Use the equals sign followed by the plus character
2729 (``=+``) to prepend values to existing variables.
2730
2731 .. note::
2732
2733 This operator adds a space between the new content and the
2734 existing content of the variable.
2735
Andrew Geisslerc926e172021-05-07 16:11:35 -05002736 Here is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002737
2738 VAR =+ "Starts"
2739
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002740- *Appending (:append):* Use the ``:append`` operator to append values
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002741 to existing variables. This operator does not add any additional
2742 space. Also, the operator is applied after all the ``+=``, and ``=+``
2743 operators have been applied and after all ``=`` assignments have
Andrew Geissler87f5cff2022-09-30 13:13:31 -05002744 occurred. This means that if ``:append`` is used in a recipe, it can
2745 only be overridden by another layer using the special ``:remove``
2746 operator, which in turn will prevent further layers from adding it back.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002747
2748 The following example shows the space being explicitly added to the
2749 start to ensure the appended value is not merged with the existing
Andrew Geisslerc926e172021-05-07 16:11:35 -05002750 value::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002751
Andrew Geissler87f5cff2022-09-30 13:13:31 -05002752 CFLAGS:append = " --enable-important-feature"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002753
2754 You can also use
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002755 the ``:append`` operator with overrides, which results in the actions
Andrew Geisslerc926e172021-05-07 16:11:35 -05002756 only being performed for the specified target or machine::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002757
Andrew Geissler87f5cff2022-09-30 13:13:31 -05002758 CFLAGS:append:sh4 = " --enable-important-sh4-specific-feature"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002759
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002760- *Prepending (:prepend):* Use the ``:prepend`` operator to prepend
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002761 values to existing variables. This operator does not add any
2762 additional space. Also, the operator is applied after all the ``+=``,
2763 and ``=+`` operators have been applied and after all ``=``
2764 assignments have occurred.
2765
2766 The following example shows the space being explicitly added to the
2767 end to ensure the prepended value is not merged with the existing
Andrew Geisslerc926e172021-05-07 16:11:35 -05002768 value::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002769
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002770 CFLAGS:prepend = "-I${S}/myincludes "
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002771
2772 You can also use the
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002773 ``:prepend`` operator with overrides, which results in the actions
Andrew Geisslerc926e172021-05-07 16:11:35 -05002774 only being performed for the specified target or machine::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002775
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002776 CFLAGS:prepend:sh4 = "-I${S}/myincludes "
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002777
2778- *Overrides:* You can use overrides to set a value conditionally,
2779 typically based on how the recipe is being built. For example, to set
2780 the :term:`KBRANCH` variable's
2781 value to "standard/base" for any target
2782 :term:`MACHINE`, except for
2783 qemuarm where it should be set to "standard/arm-versatile-926ejs",
Andrew Geisslerc926e172021-05-07 16:11:35 -05002784 you would do the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002785
2786 KBRANCH = "standard/base"
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002787 KBRANCH:qemuarm = "standard/arm-versatile-926ejs"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002788
2789 Overrides are also used to separate
2790 alternate values of a variable in other situations. For example, when
2791 setting variables such as
2792 :term:`FILES` and
2793 :term:`RDEPENDS` that are
2794 specific to individual packages produced by a recipe, you should
2795 always use an override that specifies the name of the package.
2796
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002797- *Indentation:* Use spaces for indentation rather than tabs. For
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002798 shell functions, both currently work. However, it is a policy
2799 decision of the Yocto Project to use tabs in shell functions. Realize
2800 that some layers have a policy to use spaces for all indentation.
2801
2802- *Using Python for Complex Operations:* For more advanced processing,
2803 it is possible to use Python code during variable assignments (e.g.
2804 search and replacement on a variable).
2805
2806 You indicate Python code using the ``${@python_code}`` syntax for the
Andrew Geisslerc926e172021-05-07 16:11:35 -05002807 variable assignment::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002808
2809 SRC_URI = "ftp://ftp.info-zip.org/pub/infozip/src/zip${@d.getVar('PV',1).replace('.', '')}.tgz
2810
2811- *Shell Function Syntax:* Write shell functions as if you were writing
2812 a shell script when you describe a list of actions to take. You
2813 should ensure that your script works with a generic ``sh`` and that
2814 it does not require any ``bash`` or other shell-specific
2815 functionality. The same considerations apply to various system
2816 utilities (e.g. ``sed``, ``grep``, ``awk``, and so forth) that you
2817 might wish to use. If in doubt, you should check with multiple
Andrew Geissler615f2f12022-07-15 14:00:58 -05002818 implementations --- including those from BusyBox.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002819
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002820Adding a New Machine
2821====================
2822
2823Adding a new machine to the Yocto Project is a straightforward process.
2824This section describes how to add machines that are similar to those
2825that the Yocto Project already supports.
2826
2827.. note::
2828
2829 Although well within the capabilities of the Yocto Project, adding a
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002830 totally new architecture might require changes to ``gcc``/``glibc``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002831 and to the site information, which is beyond the scope of this
2832 manual.
2833
2834For a complete example that shows how to add a new machine, see the
2835":ref:`bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script`"
2836section in the Yocto Project Board Support Package (BSP) Developer's
2837Guide.
2838
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002839Adding the Machine Configuration File
2840-------------------------------------
2841
2842To add a new machine, you need to add a new machine configuration file
2843to the layer's ``conf/machine`` directory. This configuration file
2844provides details about the device you are adding.
2845
2846The OpenEmbedded build system uses the root name of the machine
2847configuration file to reference the new machine. For example, given a
2848machine configuration file named ``crownbay.conf``, the build system
2849recognizes the machine as "crownbay".
2850
2851The most important variables you must set in your machine configuration
2852file or include from a lower-level configuration file are as follows:
2853
Andrew Geissler5f350902021-07-23 13:09:54 -04002854- :term:`TARGET_ARCH` (e.g. "arm")
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002855
2856- ``PREFERRED_PROVIDER_virtual/kernel``
2857
Andrew Geissler09036742021-06-25 14:25:14 -05002858- :term:`MACHINE_FEATURES` (e.g. "apm screen wifi")
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002859
2860You might also need these variables:
2861
Andrew Geissler5f350902021-07-23 13:09:54 -04002862- :term:`SERIAL_CONSOLES` (e.g. "115200;ttyS0 115200;ttyS1")
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002863
Andrew Geissler5f350902021-07-23 13:09:54 -04002864- :term:`KERNEL_IMAGETYPE` (e.g. "zImage")
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002865
Andrew Geissler09036742021-06-25 14:25:14 -05002866- :term:`IMAGE_FSTYPES` (e.g. "tar.gz jffs2")
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002867
2868You can find full details on these variables in the reference section.
2869You can leverage existing machine ``.conf`` files from
2870``meta-yocto-bsp/conf/machine/``.
2871
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002872Adding a Kernel for the Machine
2873-------------------------------
2874
2875The OpenEmbedded build system needs to be able to build a kernel for the
2876machine. You need to either create a new kernel recipe for this machine,
2877or extend an existing kernel recipe. You can find several kernel recipe
2878examples in the Source Directory at ``meta/recipes-kernel/linux`` that
2879you can use as references.
2880
2881If you are creating a new kernel recipe, normal recipe-writing rules
Andrew Geissler09036742021-06-25 14:25:14 -05002882apply for setting up a :term:`SRC_URI`. Thus, you need to specify any
2883necessary patches and set :term:`S` to point at the source code. You need to
Patrick Williams2194f502022-10-16 14:26:09 -05002884create a :ref:`ref-tasks-configure` task that configures the unpacked kernel with
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002885a ``defconfig`` file. You can do this by using a ``make defconfig``
2886command or, more commonly, by copying in a suitable ``defconfig`` file
2887and then running ``make oldconfig``. By making use of ``inherit kernel``
2888and potentially some of the ``linux-*.inc`` files, most other
2889functionality is centralized and the defaults of the class normally work
2890well.
2891
2892If you are extending an existing kernel recipe, it is usually a matter
2893of adding a suitable ``defconfig`` file. The file needs to be added into
2894a location similar to ``defconfig`` files used for other machines in a
2895given kernel recipe. A possible way to do this is by listing the file in
Andrew Geissler09036742021-06-25 14:25:14 -05002896the :term:`SRC_URI` and adding the machine to the expression in
2897:term:`COMPATIBLE_MACHINE`::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002898
2899 COMPATIBLE_MACHINE = '(qemux86|qemumips)'
2900
2901For more information on ``defconfig`` files, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06002902":ref:`kernel-dev/common:changing the configuration`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002903section in the Yocto Project Linux Kernel Development Manual.
2904
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002905Adding a Formfactor Configuration File
2906--------------------------------------
2907
2908A formfactor configuration file provides information about the target
2909hardware for which the image is being built and information that the
2910build system cannot obtain from other sources such as the kernel. Some
2911examples of information contained in a formfactor configuration file
2912include framebuffer orientation, whether or not the system has a
2913keyboard, the positioning of the keyboard in relation to the screen, and
2914the screen resolution.
2915
2916The build system uses reasonable defaults in most cases. However, if
2917customization is necessary, you need to create a ``machconfig`` file in
2918the ``meta/recipes-bsp/formfactor/files`` directory. This directory
2919contains directories for specific machines such as ``qemuarm`` and
2920``qemux86``. For information about the settings available and the
2921defaults, see the ``meta/recipes-bsp/formfactor/files/config`` file
2922found in the same area.
2923
Andrew Geisslerc926e172021-05-07 16:11:35 -05002924Following is an example for "qemuarm" machine::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002925
2926 HAVE_TOUCHSCREEN=1
2927 HAVE_KEYBOARD=1
2928 DISPLAY_CAN_ROTATE=0
2929 DISPLAY_ORIENTATION=0
2930 #DISPLAY_WIDTH_PIXELS=640
2931 #DISPLAY_HEIGHT_PIXELS=480
2932 #DISPLAY_BPP=16
2933 DISPLAY_DPI=150
2934 DISPLAY_SUBPIXEL_ORDER=vrgb
2935
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002936Upgrading Recipes
2937=================
2938
2939Over time, upstream developers publish new versions for software built
2940by layer recipes. It is recommended to keep recipes up-to-date with
2941upstream version releases.
2942
William A. Kennington IIIac69b482021-06-02 12:28:27 -07002943While there are several methods to upgrade a recipe, you might
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002944consider checking on the upgrade status of a recipe first. You can do so
2945using the ``devtool check-upgrade-status`` command. See the
2946":ref:`devtool-checking-on-the-upgrade-status-of-a-recipe`"
2947section in the Yocto Project Reference Manual for more information.
2948
2949The remainder of this section describes three ways you can upgrade a
2950recipe. You can use the Automated Upgrade Helper (AUH) to set up
2951automatic version upgrades. Alternatively, you can use
2952``devtool upgrade`` to set up semi-automatic version upgrades. Finally,
2953you can manually upgrade a recipe by editing the recipe itself.
2954
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002955Using the Auto Upgrade Helper (AUH)
2956-----------------------------------
2957
2958The AUH utility works in conjunction with the OpenEmbedded build system
2959in order to automatically generate upgrades for recipes based on new
2960versions being published upstream. Use AUH when you want to create a
2961service that performs the upgrades automatically and optionally sends
2962you an email with the results.
2963
2964AUH allows you to update several recipes with a single use. You can also
2965optionally perform build and integration tests using images with the
2966results saved to your hard drive and emails of results optionally sent
2967to recipe maintainers. Finally, AUH creates Git commits with appropriate
2968commit messages in the layer's tree for the changes made to recipes.
2969
2970.. note::
2971
William A. Kennington IIIac69b482021-06-02 12:28:27 -07002972 In some conditions, you should not use AUH to upgrade recipes
2973 and should instead use either ``devtool upgrade`` or upgrade your
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002974 recipes manually:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002975
2976 - When AUH cannot complete the upgrade sequence. This situation
2977 usually results because custom patches carried by the recipe
2978 cannot be automatically rebased to the new version. In this case,
2979 ``devtool upgrade`` allows you to manually resolve conflicts.
2980
2981 - When for any reason you want fuller control over the upgrade
2982 process. For example, when you want special arrangements for
2983 testing.
2984
2985The following steps describe how to set up the AUH utility:
2986
29871. *Be Sure the Development Host is Set Up:* You need to be sure that
2988 your development host is set up to use the Yocto Project. For
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002989 information on how to set up your host, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06002990 ":ref:`dev-manual/start:Preparing the Build Host`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002991
29922. *Make Sure Git is Configured:* The AUH utility requires Git to be
2993 configured because AUH uses Git to save upgrades. Thus, you must have
2994 Git user and email configured. The following command shows your
Andrew Geisslerc926e172021-05-07 16:11:35 -05002995 configurations::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002996
2997 $ git config --list
2998
2999 If you do not have the user and
Andrew Geisslerc926e172021-05-07 16:11:35 -05003000 email configured, you can use the following commands to do so::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003001
3002 $ git config --global user.name some_name
3003 $ git config --global user.email username@domain.com
3004
30053. *Clone the AUH Repository:* To use AUH, you must clone the repository
3006 onto your development host. The following command uses Git to create
Andrew Geisslerc926e172021-05-07 16:11:35 -05003007 a local copy of the repository on your system::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003008
3009 $ git clone git://git.yoctoproject.org/auto-upgrade-helper
3010 Cloning into 'auto-upgrade-helper'... remote: Counting objects: 768, done.
3011 remote: Compressing objects: 100% (300/300), done.
3012 remote: Total 768 (delta 499), reused 703 (delta 434)
3013 Receiving objects: 100% (768/768), 191.47 KiB | 98.00 KiB/s, done.
3014 Resolving deltas: 100% (499/499), done.
3015 Checking connectivity... done.
3016
3017 AUH is not part of the :term:`OpenEmbedded-Core (OE-Core)` or
3018 :term:`Poky` repositories.
3019
30204. *Create a Dedicated Build Directory:* Run the
3021 :ref:`structure-core-script`
3022 script to create a fresh build directory that you use exclusively for
Andrew Geisslerc926e172021-05-07 16:11:35 -05003023 running the AUH utility::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003024
Andrew Geissler95ac1b82021-03-31 14:34:31 -05003025 $ cd poky
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003026 $ source oe-init-build-env your_AUH_build_directory
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003027
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003028 Re-using an existing build directory and its configurations is not
3029 recommended as existing settings could cause AUH to fail or behave
3030 undesirably.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003031
30325. *Make Configurations in Your Local Configuration File:* Several
William A. Kennington IIIac69b482021-06-02 12:28:27 -07003033 settings are needed in the ``local.conf`` file in the build
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003034 directory you just created for AUH. Make these following
3035 configurations:
3036
3037 - If you want to enable :ref:`Build
Andrew Geissler09209ee2020-12-13 08:44:15 -06003038 History <dev-manual/common-tasks:maintaining build output quality>`,
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003039 which is optional, you need the following lines in the
Andrew Geisslerc926e172021-05-07 16:11:35 -05003040 ``conf/local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003041
3042 INHERIT =+ "buildhistory"
3043 BUILDHISTORY_COMMIT = "1"
3044
3045 With this configuration and a successful
3046 upgrade, a build history "diff" file appears in the
3047 ``upgrade-helper/work/recipe/buildhistory-diff.txt`` file found in
3048 your build directory.
3049
3050 - If you want to enable testing through the
Patrick Williams975a06f2022-10-21 14:42:47 -05003051 :ref:`testimage <ref-classes-testimage>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003052 class, which is optional, you need to have the following set in
Andrew Geisslerc926e172021-05-07 16:11:35 -05003053 your ``conf/local.conf`` file::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003054
3055 INHERIT += "testimage"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003056
3057 .. note::
3058
3059 If your distro does not enable by default ptest, which Poky
Andrew Geisslerc926e172021-05-07 16:11:35 -05003060 does, you need the following in your ``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003061
Patrick Williams0ca19cc2021-08-16 14:03:13 -05003062 DISTRO_FEATURES:append = " ptest"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003063
3064
30656. *Optionally Start a vncserver:* If you are running in a server
Andrew Geisslerc926e172021-05-07 16:11:35 -05003066 without an X11 session, you need to start a vncserver::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003067
3068 $ vncserver :1
3069 $ export DISPLAY=:1
3070
30717. *Create and Edit an AUH Configuration File:* You need to have the
3072 ``upgrade-helper/upgrade-helper.conf`` configuration file in your
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003073 build directory. You can find a sample configuration file in the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003074 :yocto_git:`AUH source repository </auto-upgrade-helper/tree/>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003075
3076 Read through the sample file and make configurations as needed. For
3077 example, if you enabled build history in your ``local.conf`` as
3078 described earlier, you must enable it in ``upgrade-helper.conf``.
3079
3080 Also, if you are using the default ``maintainers.inc`` file supplied
3081 with Poky and located in ``meta-yocto`` and you do not set a
3082 "maintainers_whitelist" or "global_maintainer_override" in the
3083 ``upgrade-helper.conf`` configuration, and you specify "-e all" on
3084 the AUH command-line, the utility automatically sends out emails to
3085 all the default maintainers. Please avoid this.
3086
3087This next set of examples describes how to use the AUH:
3088
3089- *Upgrading a Specific Recipe:* To upgrade a specific recipe, use the
Andrew Geisslerc926e172021-05-07 16:11:35 -05003090 following form::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003091
3092 $ upgrade-helper.py recipe_name
3093
Andrew Geisslerc926e172021-05-07 16:11:35 -05003094 For example, this command upgrades the ``xmodmap`` recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003095
3096 $ upgrade-helper.py xmodmap
3097
3098- *Upgrading a Specific Recipe to a Particular Version:* To upgrade a
Andrew Geisslerc926e172021-05-07 16:11:35 -05003099 specific recipe to a particular version, use the following form::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003100
3101 $ upgrade-helper.py recipe_name -t version
3102
Andrew Geisslerc926e172021-05-07 16:11:35 -05003103 For example, this command upgrades the ``xmodmap`` recipe to version 1.2.3::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003104
3105 $ upgrade-helper.py xmodmap -t 1.2.3
3106
3107- *Upgrading all Recipes to the Latest Versions and Suppressing Email
3108 Notifications:* To upgrade all recipes to their most recent versions
Andrew Geisslerc926e172021-05-07 16:11:35 -05003109 and suppress the email notifications, use the following command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003110
3111 $ upgrade-helper.py all
3112
3113- *Upgrading all Recipes to the Latest Versions and Send Email
3114 Notifications:* To upgrade all recipes to their most recent versions
3115 and send email messages to maintainers for each attempted recipe as
Andrew Geisslerc926e172021-05-07 16:11:35 -05003116 well as a status email, use the following command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003117
3118 $ upgrade-helper.py -e all
3119
3120Once you have run the AUH utility, you can find the results in the AUH
Andrew Geisslerc926e172021-05-07 16:11:35 -05003121build directory::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003122
3123 ${BUILDDIR}/upgrade-helper/timestamp
3124
3125The AUH utility
3126also creates recipe update commits from successful upgrade attempts in
3127the layer tree.
3128
3129You can easily set up to run the AUH utility on a regular basis by using
3130a cron job. See the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003131:yocto_git:`weeklyjob.sh </auto-upgrade-helper/tree/weeklyjob.sh>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003132file distributed with the utility for an example.
3133
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003134Using ``devtool upgrade``
3135-------------------------
3136
3137As mentioned earlier, an alternative method for upgrading recipes to
3138newer versions is to use
Andrew Geissler09209ee2020-12-13 08:44:15 -06003139:doc:`devtool upgrade </ref-manual/devtool-reference>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003140You can read about ``devtool upgrade`` in general in the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003141":ref:`sdk-manual/extensible:use \`\`devtool upgrade\`\` to create a version of the recipe that supports a newer version of the software`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003142section in the Yocto Project Application Development and the Extensible
3143Software Development Kit (eSDK) Manual.
3144
3145To see all the command-line options available with ``devtool upgrade``,
Andrew Geisslerc926e172021-05-07 16:11:35 -05003146use the following help command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003147
3148 $ devtool upgrade -h
3149
3150If you want to find out what version a recipe is currently at upstream
3151without any attempt to upgrade your local version of the recipe, you can
Andrew Geisslerc926e172021-05-07 16:11:35 -05003152use the following command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003153
3154 $ devtool latest-version recipe_name
3155
3156As mentioned in the previous section describing AUH, ``devtool upgrade``
3157works in a less-automated manner than AUH. Specifically,
3158``devtool upgrade`` only works on a single recipe that you name on the
3159command line, cannot perform build and integration testing using images,
3160and does not automatically generate commits for changes in the source
3161tree. Despite all these "limitations", ``devtool upgrade`` updates the
3162recipe file to the new upstream version and attempts to rebase custom
3163patches contained by the recipe as needed.
3164
3165.. note::
3166
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003167 AUH uses much of ``devtool upgrade`` behind the scenes making AUH somewhat
3168 of a "wrapper" application for ``devtool upgrade``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003169
3170A typical scenario involves having used Git to clone an upstream
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003171repository that you use during build operations. Because you have built the
3172recipe in the past, the layer is likely added to your
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003173configuration already. If for some reason, the layer is not added, you
3174could add it easily using the
3175":ref:`bitbake-layers <bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script>`"
3176script. For example, suppose you use the ``nano.bb`` recipe from the
3177``meta-oe`` layer in the ``meta-openembedded`` repository. For this
Andrew Geisslerc926e172021-05-07 16:11:35 -05003178example, assume that the layer has been cloned into following area::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003179
3180 /home/scottrif/meta-openembedded
3181
3182The following command from your
3183:term:`Build Directory` adds the layer to
Andrew Geisslerc926e172021-05-07 16:11:35 -05003184your build configuration (i.e. ``${BUILDDIR}/conf/bblayers.conf``)::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003185
3186 $ bitbake-layers add-layer /home/scottrif/meta-openembedded/meta-oe
3187 NOTE: Starting bitbake server...
3188 Parsing recipes: 100% |##########################################| Time: 0:00:55
3189 Parsing of 1431 .bb files complete (0 cached, 1431 parsed). 2040 targets, 56 skipped, 0 masked, 0 errors.
3190 Removing 12 recipes from the x86_64 sysroot: 100% |##############| Time: 0:00:00
3191 Removing 1 recipes from the x86_64_i586 sysroot: 100% |##########| Time: 0:00:00
3192 Removing 5 recipes from the i586 sysroot: 100% |#################| Time: 0:00:00
3193 Removing 5 recipes from the qemux86 sysroot: 100% |##############| Time: 0:00:00
3194
3195For this example, assume that the ``nano.bb`` recipe that
3196is upstream has a 2.9.3 version number. However, the version in the
3197local repository is 2.7.4. The following command from your build
3198directory automatically upgrades the recipe for you:
3199
3200.. note::
3201
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003202 Using the ``-V`` option is not necessary. Omitting the version number causes
3203 ``devtool upgrade`` to upgrade the recipe to the most recent version.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003204
3205::
3206
3207 $ devtool upgrade nano -V 2.9.3
3208 NOTE: Starting bitbake server...
3209 NOTE: Creating workspace layer in /home/scottrif/poky/build/workspace
3210 Parsing recipes: 100% |##########################################| Time: 0:00:46
3211 Parsing of 1431 .bb files complete (0 cached, 1431 parsed). 2040 targets, 56 skipped, 0 masked, 0 errors.
3212 NOTE: Extracting current version source...
3213 NOTE: Resolving any missing task queue dependencies
3214 .
3215 .
3216 .
3217 NOTE: Executing SetScene Tasks
3218 NOTE: Executing RunQueue Tasks
3219 NOTE: Tasks Summary: Attempted 74 tasks of which 72 didn't need to be rerun and all succeeded.
3220 Adding changed files: 100% |#####################################| Time: 0:00:00
3221 NOTE: Upgraded source extracted to /home/scottrif/poky/build/workspace/sources/nano
3222 NOTE: New recipe is /home/scottrif/poky/build/workspace/recipes/nano/nano_2.9.3.bb
3223
3224Continuing with this example, you can use ``devtool build`` to build the
Andrew Geisslerc926e172021-05-07 16:11:35 -05003225newly upgraded recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003226
3227 $ devtool build nano
3228 NOTE: Starting bitbake server...
3229 Loading cache: 100% |################################################################################################| Time: 0:00:01
3230 Loaded 2040 entries from dependency cache.
3231 Parsing recipes: 100% |##############################################################################################| Time: 0:00:00
3232 Parsing of 1432 .bb files complete (1431 cached, 1 parsed). 2041 targets, 56 skipped, 0 masked, 0 errors.
3233 NOTE: Resolving any missing task queue dependencies
3234 .
3235 .
3236 .
3237 NOTE: Executing SetScene Tasks
3238 NOTE: Executing RunQueue Tasks
3239 NOTE: nano: compiling from external source tree /home/scottrif/poky/build/workspace/sources/nano
3240 NOTE: Tasks Summary: Attempted 520 tasks of which 304 didn't need to be rerun and all succeeded.
3241
William A. Kennington IIIac69b482021-06-02 12:28:27 -07003242Within the ``devtool upgrade`` workflow, you can
3243deploy and test your rebuilt software. For this example,
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003244however, running ``devtool finish`` cleans up the workspace once the
3245source in your workspace is clean. This usually means using Git to stage
3246and submit commits for the changes generated by the upgrade process.
3247
3248Once the tree is clean, you can clean things up in this example with the
3249following command from the ``${BUILDDIR}/workspace/sources/nano``
Andrew Geisslerc926e172021-05-07 16:11:35 -05003250directory::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003251
3252 $ devtool finish nano meta-oe
3253 NOTE: Starting bitbake server...
3254 Loading cache: 100% |################################################################################################| Time: 0:00:00
3255 Loaded 2040 entries from dependency cache.
3256 Parsing recipes: 100% |##############################################################################################| Time: 0:00:01
3257 Parsing of 1432 .bb files complete (1431 cached, 1 parsed). 2041 targets, 56 skipped, 0 masked, 0 errors.
3258 NOTE: Adding new patch 0001-nano.bb-Stuff-I-changed-when-upgrading-nano.bb.patch
3259 NOTE: Updating recipe nano_2.9.3.bb
3260 NOTE: Removing file /home/scottrif/meta-openembedded/meta-oe/recipes-support/nano/nano_2.7.4.bb
3261 NOTE: Moving recipe file to /home/scottrif/meta-openembedded/meta-oe/recipes-support/nano
3262 NOTE: Leaving source tree /home/scottrif/poky/build/workspace/sources/nano as-is; if you no longer need it then please delete it manually
3263
3264
3265Using the ``devtool finish`` command cleans up the workspace and creates a patch
3266file based on your commits. The tool puts all patch files back into the
3267source directory in a sub-directory named ``nano`` in this case.
3268
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003269Manually Upgrading a Recipe
3270---------------------------
3271
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003272If for some reason you choose not to upgrade recipes using
Andrew Geissler09209ee2020-12-13 08:44:15 -06003273:ref:`dev-manual/common-tasks:Using the Auto Upgrade Helper (AUH)` or
3274by :ref:`dev-manual/common-tasks:Using \`\`devtool upgrade\`\``,
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003275you can manually edit the recipe files to upgrade the versions.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003276
3277.. note::
3278
3279 Manually updating multiple recipes scales poorly and involves many
3280 steps. The recommendation to upgrade recipe versions is through AUH
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003281 or ``devtool upgrade``, both of which automate some steps and provide
3282 guidance for others needed for the manual process.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003283
3284To manually upgrade recipe versions, follow these general steps:
3285
32861. *Change the Version:* Rename the recipe such that the version (i.e.
3287 the :term:`PV` part of the recipe name)
3288 changes appropriately. If the version is not part of the recipe name,
Andrew Geissler09036742021-06-25 14:25:14 -05003289 change the value as it is set for :term:`PV` within the recipe itself.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003290
Andrew Geissler09036742021-06-25 14:25:14 -050032912. *Update* :term:`SRCREV` *if Needed*: If the source code your recipe builds
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003292 is fetched from Git or some other version control system, update
3293 :term:`SRCREV` to point to the
3294 commit hash that matches the new version.
3295
32963. *Build the Software:* Try to build the recipe using BitBake. Typical
3297 build failures include the following:
3298
3299 - License statements were updated for the new version. For this
3300 case, you need to review any changes to the license and update the
3301 values of :term:`LICENSE` and
3302 :term:`LIC_FILES_CHKSUM`
3303 as needed.
3304
3305 .. note::
3306
3307 License changes are often inconsequential. For example, the
3308 license text's copyright year might have changed.
3309
3310 - Custom patches carried by the older version of the recipe might
3311 fail to apply to the new version. For these cases, you need to
3312 review the failures. Patches might not be necessary for the new
3313 version of the software if the upgraded version has fixed those
3314 issues. If a patch is necessary and failing, you need to rebase it
3315 into the new version.
3316
33174. *Optionally Attempt to Build for Several Architectures:* Once you
3318 successfully build the new software for a given architecture, you
3319 could test the build for other architectures by changing the
3320 :term:`MACHINE` variable and
3321 rebuilding the software. This optional step is especially important
3322 if the recipe is to be released publicly.
3323
33245. *Check the Upstream Change Log or Release Notes:* Checking both these
William A. Kennington IIIac69b482021-06-02 12:28:27 -07003325 reveals if there are new features that could break
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003326 backwards-compatibility. If so, you need to take steps to mitigate or
3327 eliminate that situation.
3328
33296. *Optionally Create a Bootable Image and Test:* If you want, you can
3330 test the new software by booting it onto actual hardware.
3331
33327. *Create a Commit with the Change in the Layer Repository:* After all
3333 builds work and any testing is successful, you can create commits for
3334 any changes in the layer holding your upgraded recipe.
3335
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003336Finding Temporary Source Code
3337=============================
3338
3339You might find it helpful during development to modify the temporary
3340source code used by recipes to build packages. For example, suppose you
3341are developing a patch and you need to experiment a bit to figure out
3342your solution. After you have initially built the package, you can
3343iteratively tweak the source code, which is located in the
3344:term:`Build Directory`, and then you can
3345force a re-compile and quickly test your altered code. Once you settle
3346on a solution, you can then preserve your changes in the form of
3347patches.
3348
3349During a build, the unpacked temporary source code used by recipes to
3350build packages is available in the Build Directory as defined by the
3351:term:`S` variable. Below is the default
Andrew Geissler09036742021-06-25 14:25:14 -05003352value for the :term:`S` variable as defined in the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003353``meta/conf/bitbake.conf`` configuration file in the
Andrew Geisslerc926e172021-05-07 16:11:35 -05003354:term:`Source Directory`::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003355
3356 S = "${WORKDIR}/${BP}"
3357
3358You should be aware that many recipes override the
Andrew Geissler09036742021-06-25 14:25:14 -05003359:term:`S` variable. For example, recipes that fetch their source from Git
3360usually set :term:`S` to ``${WORKDIR}/git``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003361
3362.. note::
3363
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003364 The :term:`BP` represents the base recipe name, which consists of the name
Andrew Geisslerc926e172021-05-07 16:11:35 -05003365 and version::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003366
3367 BP = "${BPN}-${PV}"
3368
3369
3370The path to the work directory for the recipe
3371(:term:`WORKDIR`) is defined as
Andrew Geisslerc926e172021-05-07 16:11:35 -05003372follows::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003373
3374 ${TMPDIR}/work/${MULTIMACH_TARGET_SYS}/${PN}/${EXTENDPE}${PV}-${PR}
3375
3376The actual directory depends on several things:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003377
3378- :term:`TMPDIR`: The top-level build
3379 output directory.
3380
3381- :term:`MULTIMACH_TARGET_SYS`:
3382 The target system identifier.
3383
3384- :term:`PN`: The recipe name.
3385
Andrew Geissler615f2f12022-07-15 14:00:58 -05003386- :term:`EXTENDPE`: The epoch --- if
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003387 :term:`PE` is not specified, which is
Andrew Geissler615f2f12022-07-15 14:00:58 -05003388 usually the case for most recipes, then :term:`EXTENDPE` is blank.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003389
3390- :term:`PV`: The recipe version.
3391
3392- :term:`PR`: The recipe revision.
3393
3394As an example, assume a Source Directory top-level folder named
3395``poky``, a default Build Directory at ``poky/build``, and a
3396``qemux86-poky-linux`` machine target system. Furthermore, suppose your
3397recipe is named ``foo_1.3.0.bb``. In this case, the work directory the
Andrew Geisslerc926e172021-05-07 16:11:35 -05003398build system uses to build the package would be as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003399
3400 poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0
3401
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003402Using Quilt in Your Workflow
3403============================
3404
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003405`Quilt <https://savannah.nongnu.org/projects/quilt>`__ is a powerful tool
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003406that allows you to capture source code changes without having a clean
3407source tree. This section outlines the typical workflow you can use to
3408modify source code, test changes, and then preserve the changes in the
3409form of a patch all using Quilt.
3410
3411.. note::
3412
3413 With regard to preserving changes to source files, if you clean a
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003414 recipe or have ``rm_work`` enabled, the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003415 :ref:`devtool workflow <sdk-manual/extensible:using \`\`devtool\`\` in your sdk workflow>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003416 as described in the Yocto Project Application Development and the
3417 Extensible Software Development Kit (eSDK) manual is a safer
3418 development flow than the flow that uses Quilt.
3419
3420Follow these general steps:
3421
34221. *Find the Source Code:* Temporary source code used by the
3423 OpenEmbedded build system is kept in the
3424 :term:`Build Directory`. See the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05003425 ":ref:`dev-manual/common-tasks:finding temporary source code`" section to
3426 learn how to locate the directory that has the temporary source code for a
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003427 particular package.
3428
34292. *Change Your Working Directory:* You need to be in the directory that
3430 has the temporary source code. That directory is defined by the
3431 :term:`S` variable.
3432
34333. *Create a New Patch:* Before modifying source code, you need to
3434 create a new patch. To create a new patch file, use ``quilt new`` as
Andrew Geisslerc926e172021-05-07 16:11:35 -05003435 below::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003436
3437 $ quilt new my_changes.patch
3438
34394. *Notify Quilt and Add Files:* After creating the patch, you need to
3440 notify Quilt about the files you plan to edit. You notify Quilt by
Andrew Geisslerc926e172021-05-07 16:11:35 -05003441 adding the files to the patch you just created::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003442
3443 $ quilt add file1.c file2.c file3.c
3444
34455. *Edit the Files:* Make your changes in the source code to the files
3446 you added to the patch.
3447
34486. *Test Your Changes:* Once you have modified the source code, the
Patrick Williams2194f502022-10-16 14:26:09 -05003449 easiest way to test your changes is by calling the :ref:`ref-tasks-compile`
Andrew Geisslerc926e172021-05-07 16:11:35 -05003450 task as shown in the following example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003451
3452 $ bitbake -c compile -f package
3453
3454 The ``-f`` or ``--force`` option forces the specified task to
3455 execute. If you find problems with your code, you can just keep
3456 editing and re-testing iteratively until things work as expected.
3457
3458 .. note::
3459
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003460 All the modifications you make to the temporary source code disappear
Patrick Williams2194f502022-10-16 14:26:09 -05003461 once you run the :ref:`ref-tasks-clean` or :ref:`ref-tasks-cleanall` tasks using BitBake
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003462 (i.e. ``bitbake -c clean package`` and ``bitbake -c cleanall package``).
3463 Modifications will also disappear if you use the ``rm_work`` feature as
3464 described in the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003465 ":ref:`dev-manual/common-tasks:conserving disk space during builds`"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003466 section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003467
34687. *Generate the Patch:* Once your changes work as expected, you need to
3469 use Quilt to generate the final patch that contains all your
3470 modifications.
3471 ::
3472
3473 $ quilt refresh
3474
3475 At this point, the
3476 ``my_changes.patch`` file has all your edits made to the ``file1.c``,
3477 ``file2.c``, and ``file3.c`` files.
3478
3479 You can find the resulting patch file in the ``patches/``
Andrew Geissler09036742021-06-25 14:25:14 -05003480 subdirectory of the source (:term:`S`) directory.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003481
34828. *Copy the Patch File:* For simplicity, copy the patch file into a
3483 directory named ``files``, which you can create in the same directory
3484 that holds the recipe (``.bb``) file or the append (``.bbappend``)
3485 file. Placing the patch here guarantees that the OpenEmbedded build
Andrew Geissler09036742021-06-25 14:25:14 -05003486 system will find the patch. Next, add the patch into the :term:`SRC_URI`
Andrew Geisslerc926e172021-05-07 16:11:35 -05003487 of the recipe. Here is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003488
3489 SRC_URI += "file://my_changes.patch"
3490
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003491Using a Development Shell
3492=========================
3493
3494When debugging certain commands or even when just editing packages,
3495``devshell`` can be a useful tool. When you invoke ``devshell``, all
3496tasks up to and including
3497:ref:`ref-tasks-patch` are run for the
3498specified target. Then, a new terminal is opened and you are placed in
3499``${``\ :term:`S`\ ``}``, the source
3500directory. In the new terminal, all the OpenEmbedded build-related
3501environment variables are still defined so you can use commands such as
3502``configure`` and ``make``. The commands execute just as if the
3503OpenEmbedded build system were executing them. Consequently, working
3504this way can be helpful when debugging a build or preparing software to
3505be used with the OpenEmbedded build system.
3506
3507Following is an example that uses ``devshell`` on a target named
Andrew Geisslerc926e172021-05-07 16:11:35 -05003508``matchbox-desktop``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003509
3510 $ bitbake matchbox-desktop -c devshell
3511
3512This command spawns a terminal with a shell prompt within the
3513OpenEmbedded build environment. The
3514:term:`OE_TERMINAL` variable
3515controls what type of shell is opened.
3516
3517For spawned terminals, the following occurs:
3518
3519- The ``PATH`` variable includes the cross-toolchain.
3520
3521- The ``pkgconfig`` variables find the correct ``.pc`` files.
3522
3523- The ``configure`` command finds the Yocto Project site files as well
3524 as any other necessary files.
3525
3526Within this environment, you can run configure or compile commands as if
3527they were being run by the OpenEmbedded build system itself. As noted
3528earlier, the working directory also automatically changes to the Source
3529Directory (:term:`S`).
3530
3531To manually run a specific task using ``devshell``, run the
3532corresponding ``run.*`` script in the
3533``${``\ :term:`WORKDIR`\ ``}/temp``
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003534directory (e.g., ``run.do_configure.``\ `pid`). If a task's script does
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003535not exist, which would be the case if the task was skipped by way of the
3536sstate cache, you can create the task by first running it outside of the
Andrew Geisslerc926e172021-05-07 16:11:35 -05003537``devshell``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003538
3539 $ bitbake -c task
3540
3541.. note::
3542
3543 - Execution of a task's ``run.*`` script and BitBake's execution of
3544 a task are identical. In other words, running the script re-runs
3545 the task just as it would be run using the ``bitbake -c`` command.
3546
3547 - Any ``run.*`` file that does not have a ``.pid`` extension is a
3548 symbolic link (symlink) to the most recent version of that file.
3549
3550Remember, that the ``devshell`` is a mechanism that allows you to get
3551into the BitBake task execution environment. And as such, all commands
3552must be called just as BitBake would call them. That means you need to
3553provide the appropriate options for cross-compilation and so forth as
3554applicable.
3555
3556When you are finished using ``devshell``, exit the shell or close the
3557terminal window.
3558
3559.. note::
3560
3561 - It is worth remembering that when using ``devshell`` you need to
3562 use the full compiler name such as ``arm-poky-linux-gnueabi-gcc``
3563 instead of just using ``gcc``. The same applies to other
3564 applications such as ``binutils``, ``libtool`` and so forth.
Andrew Geissler09036742021-06-25 14:25:14 -05003565 BitBake sets up environment variables such as :term:`CC` to assist
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003566 applications, such as ``make`` to find the correct tools.
3567
3568 - It is also worth noting that ``devshell`` still works over X11
3569 forwarding and similar situations.
3570
Andrew Geisslereff27472021-10-29 15:35:00 -05003571Using a Python Development Shell
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003572================================
3573
3574Similar to working within a development shell as described in the
3575previous section, you can also spawn and work within an interactive
3576Python development shell. When debugging certain commands or even when
Andrew Geisslereff27472021-10-29 15:35:00 -05003577just editing packages, ``pydevshell`` can be a useful tool. When you
3578invoke the ``pydevshell`` task, all tasks up to and including
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003579:ref:`ref-tasks-patch` are run for the
3580specified target. Then a new terminal is opened. Additionally, key
3581Python objects and code are available in the same way they are to
3582BitBake tasks, in particular, the data store 'd'. So, commands such as
3583the following are useful when exploring the data store and running
Andrew Geisslerc926e172021-05-07 16:11:35 -05003584functions::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003585
3586 pydevshell> d.getVar("STAGING_DIR")
3587 '/media/build1/poky/build/tmp/sysroots'
Andrew Geissler5199d832021-09-24 16:47:35 -05003588 pydevshell> d.getVar("STAGING_DIR", False)
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003589 '${TMPDIR}/sysroots'
3590 pydevshell> d.setVar("FOO", "bar")
3591 pydevshell> d.getVar("FOO")
3592 'bar'
3593 pydevshell> d.delVar("FOO")
3594 pydevshell> d.getVar("FOO")
3595 pydevshell> bb.build.exec_func("do_unpack", d)
3596 pydevshell>
3597
Andrew Geissler87f5cff2022-09-30 13:13:31 -05003598See the ":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata:functions you can call from within python`"
3599section in the BitBake User Manual for details about available functions.
3600
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003601The commands execute just as if the OpenEmbedded build
3602system were executing them. Consequently, working this way can be
3603helpful when debugging a build or preparing software to be used with the
3604OpenEmbedded build system.
3605
Andrew Geisslereff27472021-10-29 15:35:00 -05003606Following is an example that uses ``pydevshell`` on a target named
Andrew Geisslerc926e172021-05-07 16:11:35 -05003607``matchbox-desktop``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003608
Andrew Geisslereff27472021-10-29 15:35:00 -05003609 $ bitbake matchbox-desktop -c pydevshell
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003610
3611This command spawns a terminal and places you in an interactive Python
3612interpreter within the OpenEmbedded build environment. The
3613:term:`OE_TERMINAL` variable
3614controls what type of shell is opened.
3615
Andrew Geisslereff27472021-10-29 15:35:00 -05003616When you are finished using ``pydevshell``, you can exit the shell
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003617either by using Ctrl+d or closing the terminal window.
3618
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003619Building
3620========
3621
Andrew Geissler5199d832021-09-24 16:47:35 -05003622This section describes various build procedures, such as the steps
3623needed for a simple build, building a target for multiple configurations,
3624generating an image for more than one machine, and so forth.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003625
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003626Building a Simple Image
3627-----------------------
3628
3629In the development environment, you need to build an image whenever you
3630change hardware support, add or change system libraries, or add or
William A. Kennington IIIac69b482021-06-02 12:28:27 -07003631change services that have dependencies. There are several methods that allow
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003632you to build an image within the Yocto Project. This section presents
3633the basic steps you need to build a simple image using BitBake from a
3634build host running Linux.
3635
3636.. note::
3637
3638 - For information on how to build an image using
3639 :term:`Toaster`, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003640 :doc:`/toaster-manual/index`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003641
3642 - For information on how to use ``devtool`` to build images, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003643 ":ref:`sdk-manual/extensible:using \`\`devtool\`\` in your sdk workflow`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003644 section in the Yocto Project Application Development and the
3645 Extensible Software Development Kit (eSDK) manual.
3646
3647 - For a quick example on how to build an image using the
3648 OpenEmbedded build system, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003649 :doc:`/brief-yoctoprojectqs/index` document.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003650
3651The build process creates an entire Linux distribution from source and
3652places it in your :term:`Build Directory` under
3653``tmp/deploy/images``. For detailed information on the build process
Andrew Geissler09209ee2020-12-13 08:44:15 -06003654using BitBake, see the ":ref:`overview-manual/concepts:images`" section in the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003655Yocto Project Overview and Concepts Manual.
3656
3657The following figure and list overviews the build process:
3658
3659.. image:: figures/bitbake-build-flow.png
Andrew Geisslerd5838332022-05-27 11:33:10 -05003660 :width: 100%
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003661
36621. *Set up Your Host Development System to Support Development Using the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003663 Yocto Project*: See the ":doc:`start`" section for options on how to get a
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003664 build host ready to use the Yocto Project.
3665
36662. *Initialize the Build Environment:* Initialize the build environment
3667 by sourcing the build environment script (i.e.
Andrew Geisslerc926e172021-05-07 16:11:35 -05003668 :ref:`structure-core-script`)::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003669
3670 $ source oe-init-build-env [build_dir]
3671
3672 When you use the initialization script, the OpenEmbedded build system
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003673 uses ``build`` as the default :term:`Build Directory` in your current work
3674 directory. You can use a `build_dir` argument with the script to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003675 specify a different build directory.
3676
3677 .. note::
3678
3679 A common practice is to use a different Build Directory for
Andrew Geissler5199d832021-09-24 16:47:35 -05003680 different targets; for example, ``~/build/x86`` for a ``qemux86``
3681 target, and ``~/build/arm`` for a ``qemuarm`` target. In any
3682 event, it's typically cleaner to locate the build directory
3683 somewhere outside of your source directory.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003684
Andrew Geissler4c19ea12020-10-27 13:52:24 -050036853. *Make Sure Your* ``local.conf`` *File is Correct*: Ensure the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003686 ``conf/local.conf`` configuration file, which is found in the Build
3687 Directory, is set up how you want it. This file defines many aspects
3688 of the build environment including the target machine architecture
Andrew Geissler09036742021-06-25 14:25:14 -05003689 through the :term:`MACHINE` variable, the packaging format used during
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003690 the build
3691 (:term:`PACKAGE_CLASSES`),
3692 and a centralized tarball download directory through the
3693 :term:`DL_DIR` variable.
3694
Andrew Geisslerc926e172021-05-07 16:11:35 -050036954. *Build the Image:* Build the image using the ``bitbake`` command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003696
3697 $ bitbake target
3698
3699 .. note::
3700
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003701 For information on BitBake, see the :doc:`bitbake:index`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003702
3703 The target is the name of the recipe you want to build. Common
3704 targets are the images in ``meta/recipes-core/images``,
3705 ``meta/recipes-sato/images``, and so forth all found in the
Andrew Geissler5199d832021-09-24 16:47:35 -05003706 :term:`Source Directory`. Alternatively, the target
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003707 can be the name of a recipe for a specific piece of software such as
3708 BusyBox. For more details about the images the OpenEmbedded build
3709 system supports, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003710 ":ref:`ref-manual/images:Images`" chapter in the Yocto
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003711 Project Reference Manual.
3712
3713 As an example, the following command builds the
Andrew Geisslerc926e172021-05-07 16:11:35 -05003714 ``core-image-minimal`` image::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003715
3716 $ bitbake core-image-minimal
3717
3718 Once an
3719 image has been built, it often needs to be installed. The images and
3720 kernels built by the OpenEmbedded build system are placed in the
3721 Build Directory in ``tmp/deploy/images``. For information on how to
3722 run pre-built images such as ``qemux86`` and ``qemuarm``, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003723 :doc:`/sdk-manual/index` manual. For
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003724 information about how to install these images, see the documentation
3725 for your particular board or machine.
3726
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003727Building Images for Multiple Targets Using Multiple Configurations
3728------------------------------------------------------------------
3729
3730You can use a single ``bitbake`` command to build multiple images or
3731packages for different targets where each image or package requires a
3732different configuration (multiple configuration builds). The builds, in
3733this scenario, are sometimes referred to as "multiconfigs", and this
3734section uses that term throughout.
3735
3736This section describes how to set up for multiple configuration builds
3737and how to account for cross-build dependencies between the
3738multiconfigs.
3739
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003740Setting Up and Running a Multiple Configuration Build
3741~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3742
3743To accomplish a multiple configuration build, you must define each
3744target's configuration separately using a parallel configuration file in
Andrew Geissler615f2f12022-07-15 14:00:58 -05003745the :term:`Build Directory` or configuration directory within a layer, and you
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003746must follow a required file hierarchy. Additionally, you must enable the
3747multiple configuration builds in your ``local.conf`` file.
3748
3749Follow these steps to set up and execute multiple configuration builds:
3750
3751- *Create Separate Configuration Files*: You need to create a single
3752 configuration file for each build target (each multiconfig).
Andrew Geissler615f2f12022-07-15 14:00:58 -05003753 The configuration definitions are implementation dependent but often
3754 each configuration file will define the machine and the
3755 temporary directory BitBake uses for the build. Whether the same
3756 temporary directory (:term:`TMPDIR`) can be shared will depend on what is
3757 similar and what is different between the configurations. Multiple MACHINE
3758 targets can share the same (:term:`TMPDIR`) as long as the rest of the
3759 configuration is the same, multiple DISTRO settings would need separate
3760 (:term:`TMPDIR`) directories.
3761
3762 For example, consider a scenario with two different multiconfigs for the same
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003763 :term:`MACHINE`: "qemux86" built
3764 for two distributions such as "poky" and "poky-lsb". In this case,
Andrew Geissler615f2f12022-07-15 14:00:58 -05003765 you would need to use the different :term:`TMPDIR`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003766
3767 Here is an example showing the minimal statements needed in a
3768 configuration file for a "qemux86" target whose temporary build
Andrew Geisslerc926e172021-05-07 16:11:35 -05003769 directory is ``tmpmultix86``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003770
3771 MACHINE = "qemux86"
3772 TMPDIR = "${TOPDIR}/tmpmultix86"
3773
3774 The location for these multiconfig configuration files is specific.
3775 They must reside in the current build directory in a sub-directory of
Andrew Geissler615f2f12022-07-15 14:00:58 -05003776 ``conf`` named ``multiconfig`` or within a layer's ``conf`` directory
3777 under a directory named ``multiconfig``. Following is an example that defines
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003778 two configuration files for the "x86" and "arm" multiconfigs:
3779
3780 .. image:: figures/multiconfig_files.png
3781 :align: center
Andrew Geisslerd5838332022-05-27 11:33:10 -05003782 :width: 50%
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003783
Andrew Geissler615f2f12022-07-15 14:00:58 -05003784 The usual :term:`BBPATH` search path is used to locate multiconfig files in
3785 a similar way to other conf files.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003786
3787- *Add the BitBake Multi-configuration Variable to the Local
3788 Configuration File*: Use the
3789 :term:`BBMULTICONFIG`
3790 variable in your ``conf/local.conf`` configuration file to specify
3791 each multiconfig. Continuing with the example from the previous
Andrew Geissler09036742021-06-25 14:25:14 -05003792 figure, the :term:`BBMULTICONFIG` variable needs to enable two
Andrew Geisslerc926e172021-05-07 16:11:35 -05003793 multiconfigs: "x86" and "arm" by specifying each configuration file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003794
3795 BBMULTICONFIG = "x86 arm"
3796
3797 .. note::
3798
3799 A "default" configuration already exists by definition. This
3800 configuration is named: "" (i.e. empty string) and is defined by
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003801 the variables coming from your ``local.conf``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003802 file. Consequently, the previous example actually adds two
3803 additional configurations to your build: "arm" and "x86" along
3804 with "".
3805
3806- *Launch BitBake*: Use the following BitBake command form to launch
Andrew Geisslerc926e172021-05-07 16:11:35 -05003807 the multiple configuration build::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003808
3809 $ bitbake [mc:multiconfigname:]target [[[mc:multiconfigname:]target] ... ]
3810
Andrew Geisslerc926e172021-05-07 16:11:35 -05003811 For the example in this section, the following command applies::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003812
3813 $ bitbake mc:x86:core-image-minimal mc:arm:core-image-sato mc::core-image-base
3814
3815 The previous BitBake command builds a ``core-image-minimal`` image
3816 that is configured through the ``x86.conf`` configuration file, a
3817 ``core-image-sato`` image that is configured through the ``arm.conf``
3818 configuration file and a ``core-image-base`` that is configured
3819 through your ``local.conf`` configuration file.
3820
3821.. note::
3822
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003823 Support for multiple configuration builds in the Yocto Project &DISTRO;
3824 (&DISTRO_NAME;) Release does not include Shared State (sstate)
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003825 optimizations. Consequently, if a build uses the same object twice
Andrew Geissler09036742021-06-25 14:25:14 -05003826 in, for example, two different :term:`TMPDIR`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003827 directories, the build either loads from an existing sstate cache for
3828 that build at the start or builds the object fresh.
3829
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003830Enabling Multiple Configuration Build Dependencies
3831~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3832
3833Sometimes dependencies can exist between targets (multiconfigs) in a
3834multiple configuration build. For example, suppose that in order to
3835build a ``core-image-sato`` image for an "x86" multiconfig, the root
3836filesystem of an "arm" multiconfig must exist. This dependency is
3837essentially that the
3838:ref:`ref-tasks-image` task in the
3839``core-image-sato`` recipe depends on the completion of the
3840:ref:`ref-tasks-rootfs` task of the
3841``core-image-minimal`` recipe.
3842
3843To enable dependencies in a multiple configuration build, you must
3844declare the dependencies in the recipe using the following statement
Andrew Geisslerc926e172021-05-07 16:11:35 -05003845form::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003846
3847 task_or_package[mcdepends] = "mc:from_multiconfig:to_multiconfig:recipe_name:task_on_which_to_depend"
3848
3849To better show how to use this statement, consider the example scenario
3850from the first paragraph of this section. The following statement needs
Andrew Geisslerc926e172021-05-07 16:11:35 -05003851to be added to the recipe that builds the ``core-image-sato`` image::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003852
3853 do_image[mcdepends] = "mc:x86:arm:core-image-minimal:do_rootfs"
3854
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003855In this example, the `from_multiconfig` is "x86". The `to_multiconfig` is "arm". The
Patrick Williams2194f502022-10-16 14:26:09 -05003856task on which the :ref:`ref-tasks-image` task in the recipe depends is the
3857:ref:`ref-tasks-rootfs` task from the ``core-image-minimal`` recipe associated
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003858with the "arm" multiconfig.
3859
3860Once you set up this dependency, you can build the "x86" multiconfig
Andrew Geisslerc926e172021-05-07 16:11:35 -05003861using a BitBake command as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003862
3863 $ bitbake mc:x86:core-image-sato
3864
3865This command executes all the tasks needed to create the
3866``core-image-sato`` image for the "x86" multiconfig. Because of the
Patrick Williams2194f502022-10-16 14:26:09 -05003867dependency, BitBake also executes through the :ref:`ref-tasks-rootfs` task for the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003868"arm" multiconfig build.
3869
3870Having a recipe depend on the root filesystem of another build might not
3871seem that useful. Consider this change to the statement in the
Andrew Geisslerc926e172021-05-07 16:11:35 -05003872``core-image-sato`` recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003873
3874 do_image[mcdepends] = "mc:x86:arm:core-image-minimal:do_image"
3875
3876In this case, BitBake must
3877create the ``core-image-minimal`` image for the "arm" build since the
3878"x86" build depends on it.
3879
3880Because "x86" and "arm" are enabled for multiple configuration builds
3881and have separate configuration files, BitBake places the artifacts for
3882each build in the respective temporary build directories (i.e.
3883:term:`TMPDIR`).
3884
Patrick Williams2194f502022-10-16 14:26:09 -05003885Building an Initial RAM Filesystem (Initramfs) Image
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003886----------------------------------------------------
3887
Patrick Williams2194f502022-10-16 14:26:09 -05003888An initial RAM filesystem (:term:`Initramfs`) image provides a temporary root
3889filesystem used for early system initialization, typically providing tools and
3890loading modules needed to locate and mount the final root filesystem.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003891
Patrick Williams2194f502022-10-16 14:26:09 -05003892Follow these steps to create an :term:`Initramfs` image:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003893
Patrick Williams2194f502022-10-16 14:26:09 -050038941. *Create the Initramfs Image Recipe:* You can reference the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003895 ``core-image-minimal-initramfs.bb`` recipe found in the
3896 ``meta/recipes-core`` directory of the :term:`Source Directory`
Patrick Williams2194f502022-10-16 14:26:09 -05003897 as an example from which to work.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003898
Patrick Williams2194f502022-10-16 14:26:09 -050038992. *Decide if You Need to Bundle the Initramfs Image Into the Kernel
3900 Image:* If you want the :term:`Initramfs` image that is built to be bundled
3901 in with the kernel image, set the :term:`INITRAMFS_IMAGE_BUNDLE`
3902 variable to ``"1"`` in your ``local.conf`` configuration file and set the
3903 :term:`INITRAMFS_IMAGE` variable in the recipe that builds the kernel image.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003904
Patrick Williams2194f502022-10-16 14:26:09 -05003905 Setting the :term:`INITRAMFS_IMAGE_BUNDLE` flag causes the :term:`Initramfs`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003906 image to be unpacked into the ``${B}/usr/`` directory. The unpacked
Patrick Williams2194f502022-10-16 14:26:09 -05003907 :term:`Initramfs` image is then passed to the kernel's ``Makefile`` using the
3908 :term:`CONFIG_INITRAMFS_SOURCE` variable, allowing the :term:`Initramfs`
3909 image to be built into the kernel normally.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003910
Patrick Williams2194f502022-10-16 14:26:09 -050039113. *Optionally Add Items to the Initramfs Image Through the Initramfs
3912 Image Recipe:* If you add items to the :term:`Initramfs` image by way of its
3913 recipe, you should use :term:`PACKAGE_INSTALL` rather than
3914 :term:`IMAGE_INSTALL`. :term:`PACKAGE_INSTALL` gives more direct control of
3915 what is added to the image as compared to the defaults you might not
3916 necessarily want that are set by the :ref:`image <ref-classes-image>`
3917 or :ref:`core-image <ref-classes-core-image>` classes.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003918
Patrick Williams2194f502022-10-16 14:26:09 -050039194. *Build the Kernel Image and the Initramfs Image:* Build your kernel
3920 image using BitBake. Because the :term:`Initramfs` image recipe is a
3921 dependency of the kernel image, the :term:`Initramfs` image is built as well
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003922 and bundled with the kernel image if you used the
Patrick Williams2194f502022-10-16 14:26:09 -05003923 :term:`INITRAMFS_IMAGE_BUNDLE` variable described earlier.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003924
Andrew Geissler7e0e3c02022-02-25 20:34:39 +00003925Bundling an Initramfs Image From a Separate Multiconfig
3926~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3927
Patrick Williams2194f502022-10-16 14:26:09 -05003928There may be a case where we want to build an :term:`Initramfs` image which does not
Andrew Geissler7e0e3c02022-02-25 20:34:39 +00003929inherit the same distro policy as our main image, for example, we may want
Patrick Williams2194f502022-10-16 14:26:09 -05003930our main image to use ``TCLIBC="glibc"``, but to use ``TCLIBC="musl"`` in our :term:`Initramfs`
Andrew Geissler7e0e3c02022-02-25 20:34:39 +00003931image to keep a smaller footprint. However, by performing the steps mentioned
Patrick Williams2194f502022-10-16 14:26:09 -05003932above the :term:`Initramfs` image will inherit ``TCLIBC="glibc"`` without allowing us
Andrew Geissler7e0e3c02022-02-25 20:34:39 +00003933to override it.
3934
3935To achieve this, you need to perform some additional steps:
3936
Patrick Williams2194f502022-10-16 14:26:09 -050039371. *Create a multiconfig for your Initramfs image:* You can perform the steps
Andrew Geissler7e0e3c02022-02-25 20:34:39 +00003938 on ":ref:`dev-manual/common-tasks:building images for multiple targets using multiple configurations`" to create a separate multiconfig.
3939 For the sake of simplicity let's assume such multiconfig is called: ``initramfscfg.conf`` and
3940 contains the variables::
3941
3942 TMPDIR="${TOPDIR}/tmp-initramfscfg"
3943 TCLIBC="musl"
3944
Patrick Williams2194f502022-10-16 14:26:09 -050039452. *Set additional Initramfs variables on your main configuration:*
Andrew Geissler7e0e3c02022-02-25 20:34:39 +00003946 Additionally, on your main configuration (``local.conf``) you need to set the
3947 variables::
3948
3949 INITRAMFS_MULTICONFIG = "initramfscfg"
3950 INITRAMFS_DEPLOY_DIR_IMAGE = "${TOPDIR}/tmp-initramfscfg/deploy/images/${MACHINE}"
3951
3952 The variables :term:`INITRAMFS_MULTICONFIG` and :term:`INITRAMFS_DEPLOY_DIR_IMAGE`
3953 are used to create a multiconfig dependency from the kernel to the :term:`INITRAMFS_IMAGE`
3954 to be built coming from the ``initramfscfg`` multiconfig, and to let the
3955 buildsystem know where the :term:`INITRAMFS_IMAGE` will be located.
3956
3957 Building a system with such configuration will build the kernel using the
Patrick Williams2194f502022-10-16 14:26:09 -05003958 main configuration but the :ref:`ref-tasks-bundle_initramfs` task will grab the
Andrew Geissler7e0e3c02022-02-25 20:34:39 +00003959 selected :term:`INITRAMFS_IMAGE` from :term:`INITRAMFS_DEPLOY_DIR_IMAGE`
Patrick Williams2194f502022-10-16 14:26:09 -05003960 instead, resulting in a musl based :term:`Initramfs` image bundled in the kernel
Andrew Geissler7e0e3c02022-02-25 20:34:39 +00003961 but a glibc based main image.
3962
3963 The same is applicable to avoid inheriting :term:`DISTRO_FEATURES` on :term:`INITRAMFS_IMAGE`
3964 or to build a different :term:`DISTRO` for it such as ``poky-tiny``.
3965
3966
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003967Building a Tiny System
3968----------------------
3969
3970Very small distributions have some significant advantages such as
3971requiring less on-die or in-package memory (cheaper), better performance
3972through efficient cache usage, lower power requirements due to less
3973memory, faster boot times, and reduced development overhead. Some
3974real-world examples where a very small distribution gives you distinct
3975advantages are digital cameras, medical devices, and small headless
3976systems.
3977
3978This section presents information that shows you how you can trim your
3979distribution to even smaller sizes than the ``poky-tiny`` distribution,
3980which is around 5 Mbytes, that can be built out-of-the-box using the
3981Yocto Project.
3982
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003983Tiny System Overview
3984~~~~~~~~~~~~~~~~~~~~
3985
3986The following list presents the overall steps you need to consider and
3987perform to create distributions with smaller root filesystems, achieve
3988faster boot times, maintain your critical functionality, and avoid
3989initial RAM disks:
3990
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05003991- :ref:`Determine your goals and guiding principles
3992 <dev-manual/common-tasks:goals and guiding principles>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003993
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05003994- :ref:`dev-manual/common-tasks:understand what contributes to your image size`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003995
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05003996- :ref:`Reduce the size of the root filesystem
3997 <dev-manual/common-tasks:trim the root filesystem>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003998
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05003999- :ref:`Reduce the size of the kernel <dev-manual/common-tasks:trim the kernel>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004000
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05004001- :ref:`dev-manual/common-tasks:remove package management requirements`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004002
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05004003- :ref:`dev-manual/common-tasks:look for other ways to minimize size`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004004
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05004005- :ref:`dev-manual/common-tasks:iterate on the process`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004006
4007Goals and Guiding Principles
4008~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4009
4010Before you can reach your destination, you need to know where you are
4011going. Here is an example list that you can use as a guide when creating
4012very small distributions:
4013
4014- Determine how much space you need (e.g. a kernel that is 1 Mbyte or
4015 less and a root filesystem that is 3 Mbytes or less).
4016
4017- Find the areas that are currently taking 90% of the space and
4018 concentrate on reducing those areas.
4019
4020- Do not create any difficult "hacks" to achieve your goals.
4021
4022- Leverage the device-specific options.
4023
4024- Work in a separate layer so that you keep changes isolated. For
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05004025 information on how to create layers, see the
4026 ":ref:`dev-manual/common-tasks:understanding and creating layers`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004027
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004028Understand What Contributes to Your Image Size
4029~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4030
4031It is easiest to have something to start with when creating your own
4032distribution. You can use the Yocto Project out-of-the-box to create the
4033``poky-tiny`` distribution. Ultimately, you will want to make changes in
4034your own distribution that are likely modeled after ``poky-tiny``.
4035
4036.. note::
4037
Andrew Geissler09036742021-06-25 14:25:14 -05004038 To use ``poky-tiny`` in your build, set the :term:`DISTRO` variable in your
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004039 ``local.conf`` file to "poky-tiny" as described in the
Andrew Geissler09209ee2020-12-13 08:44:15 -06004040 ":ref:`dev-manual/common-tasks:creating your own distribution`"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004041 section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004042
4043Understanding some memory concepts will help you reduce the system size.
4044Memory consists of static, dynamic, and temporary memory. Static memory
4045is the TEXT (code), DATA (initialized data in the code), and BSS
4046(uninitialized data) sections. Dynamic memory represents memory that is
4047allocated at runtime: stacks, hash tables, and so forth. Temporary
4048memory is recovered after the boot process. This memory consists of
4049memory used for decompressing the kernel and for the ``__init__``
4050functions.
4051
4052To help you see where you currently are with kernel and root filesystem
4053sizes, you can use two tools found in the :term:`Source Directory`
4054in the
4055``scripts/tiny/`` directory:
4056
4057- ``ksize.py``: Reports component sizes for the kernel build objects.
4058
4059- ``dirsize.py``: Reports component sizes for the root filesystem.
4060
4061This next tool and command help you organize configuration fragments and
4062view file dependencies in a human-readable form:
4063
4064- ``merge_config.sh``: Helps you manage configuration files and
4065 fragments within the kernel. With this tool, you can merge individual
4066 configuration fragments together. The tool allows you to make
4067 overrides and warns you of any missing configuration options. The
4068 tool is ideal for allowing you to iterate on configurations, create
4069 minimal configurations, and create configuration files for different
4070 machines without having to duplicate your process.
4071
4072 The ``merge_config.sh`` script is part of the Linux Yocto kernel Git
4073 repositories (i.e. ``linux-yocto-3.14``, ``linux-yocto-3.10``,
4074 ``linux-yocto-3.8``, and so forth) in the ``scripts/kconfig``
4075 directory.
4076
4077 For more information on configuration fragments, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06004078 ":ref:`kernel-dev/common:creating configuration fragments`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004079 section in the Yocto Project Linux Kernel Development Manual.
4080
4081- ``bitbake -u taskexp -g bitbake_target``: Using the BitBake command
4082 with these options brings up a Dependency Explorer from which you can
4083 view file dependencies. Understanding these dependencies allows you
4084 to make informed decisions when cutting out various pieces of the
4085 kernel and root filesystem.
4086
4087Trim the Root Filesystem
4088~~~~~~~~~~~~~~~~~~~~~~~~
4089
4090The root filesystem is made up of packages for booting, libraries, and
4091applications. To change things, you can configure how the packaging
4092happens, which changes the way you build them. You can also modify the
4093filesystem itself or select a different filesystem.
4094
4095First, find out what is hogging your root filesystem by running the
Andrew Geisslerc926e172021-05-07 16:11:35 -05004096``dirsize.py`` script from your root directory::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004097
4098 $ cd root-directory-of-image
4099 $ dirsize.py 100000 > dirsize-100k.log
4100 $ cat dirsize-100k.log
4101
4102You can apply a filter to the script to ignore files
4103under a certain size. The previous example filters out any files below
4104100 Kbytes. The sizes reported by the tool are uncompressed, and thus
4105will be smaller by a relatively constant factor in a compressed root
4106filesystem. When you examine your log file, you can focus on areas of
4107the root filesystem that take up large amounts of memory.
4108
4109You need to be sure that what you eliminate does not cripple the
4110functionality you need. One way to see how packages relate to each other
Andrew Geisslerc926e172021-05-07 16:11:35 -05004111is by using the Dependency Explorer UI with the BitBake command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004112
4113 $ cd image-directory
4114 $ bitbake -u taskexp -g image
4115
4116Use the interface to
4117select potential packages you wish to eliminate and see their dependency
4118relationships.
4119
4120When deciding how to reduce the size, get rid of packages that result in
4121minimal impact on the feature set. For example, you might not need a VGA
4122display. Or, you might be able to get by with ``devtmpfs`` and ``mdev``
4123instead of ``udev``.
4124
4125Use your ``local.conf`` file to make changes. For example, to eliminate
4126``udev`` and ``glib``, set the following in the local configuration
Andrew Geisslerc926e172021-05-07 16:11:35 -05004127file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004128
4129 VIRTUAL-RUNTIME_dev_manager = ""
4130
4131Finally, you should consider exactly the type of root filesystem you
4132need to meet your needs while also reducing its size. For example,
4133consider ``cramfs``, ``squashfs``, ``ubifs``, ``ext2``, or an
Patrick Williams2194f502022-10-16 14:26:09 -05004134:term:`Initramfs` using ``initramfs``. Be aware that ``ext3`` requires a 1
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004135Mbyte journal. If you are okay with running read-only, you do not need
4136this journal.
4137
4138.. note::
4139
4140 After each round of elimination, you need to rebuild your system and
4141 then use the tools to see the effects of your reductions.
4142
4143Trim the Kernel
4144~~~~~~~~~~~~~~~
4145
4146The kernel is built by including policies for hardware-independent
4147aspects. What subsystems do you enable? For what architecture are you
4148building? Which drivers do you build by default?
4149
4150.. note::
4151
4152 You can modify the kernel source if you want to help with boot time.
4153
4154Run the ``ksize.py`` script from the top-level Linux build directory to
Andrew Geisslerc926e172021-05-07 16:11:35 -05004155get an idea of what is making up the kernel::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004156
4157 $ cd top-level-linux-build-directory
4158 $ ksize.py > ksize.log
4159 $ cat ksize.log
4160
4161When you examine the log, you will see how much space is taken up with
4162the built-in ``.o`` files for drivers, networking, core kernel files,
4163filesystem, sound, and so forth. The sizes reported by the tool are
4164uncompressed, and thus will be smaller by a relatively constant factor
4165in a compressed kernel image. Look to reduce the areas that are large
4166and taking up around the "90% rule."
4167
4168To examine, or drill down, into any particular area, use the ``-d``
Andrew Geisslerc926e172021-05-07 16:11:35 -05004169option with the script::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004170
4171 $ ksize.py -d > ksize.log
4172
4173Using this option
4174breaks out the individual file information for each area of the kernel
4175(e.g. drivers, networking, and so forth).
4176
4177Use your log file to see what you can eliminate from the kernel based on
4178features you can let go. For example, if you are not going to need
4179sound, you do not need any drivers that support sound.
4180
4181After figuring out what to eliminate, you need to reconfigure the kernel
4182to reflect those changes during the next build. You could run
4183``menuconfig`` and make all your changes at once. However, that makes it
4184difficult to see the effects of your individual eliminations and also
4185makes it difficult to replicate the changes for perhaps another target
4186device. A better method is to start with no configurations using
4187``allnoconfig``, create configuration fragments for individual changes,
4188and then manage the fragments into a single configuration file using
4189``merge_config.sh``. The tool makes it easy for you to iterate using the
4190configuration change and build cycle.
4191
4192Each time you make configuration changes, you need to rebuild the kernel
4193and check to see what impact your changes had on the overall size.
4194
4195Remove Package Management Requirements
4196~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4197
4198Packaging requirements add size to the image. One way to reduce the size
4199of the image is to remove all the packaging requirements from the image.
4200This reduction includes both removing the package manager and its unique
4201dependencies as well as removing the package management data itself.
4202
4203To eliminate all the packaging requirements for an image, be sure that
4204"package-management" is not part of your
4205:term:`IMAGE_FEATURES`
4206statement for the image. When you remove this feature, you are removing
4207the package manager as well as its dependencies from the root
4208filesystem.
4209
4210Look for Other Ways to Minimize Size
4211~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4212
4213Depending on your particular circumstances, other areas that you can
4214trim likely exist. The key to finding these areas is through tools and
4215methods described here combined with experimentation and iteration. Here
4216are a couple of areas to experiment with:
4217
4218- ``glibc``: In general, follow this process:
4219
4220 1. Remove ``glibc`` features from
4221 :term:`DISTRO_FEATURES`
4222 that you think you do not need.
4223
4224 2. Build your distribution.
4225
4226 3. If the build fails due to missing symbols in a package, determine
4227 if you can reconfigure the package to not need those features. For
4228 example, change the configuration to not support wide character
4229 support as is done for ``ncurses``. Or, if support for those
4230 characters is needed, determine what ``glibc`` features provide
4231 the support and restore the configuration.
4232
4233 4. Rebuild and repeat the process.
4234
4235- ``busybox``: For BusyBox, use a process similar as described for
4236 ``glibc``. A difference is you will need to boot the resulting system
4237 to see if you are able to do everything you expect from the running
4238 system. You need to be sure to integrate configuration fragments into
4239 Busybox because BusyBox handles its own core features and then allows
4240 you to add configuration fragments on top.
4241
4242Iterate on the Process
4243~~~~~~~~~~~~~~~~~~~~~~
4244
4245If you have not reached your goals on system size, you need to iterate
4246on the process. The process is the same. Use the tools and see just what
4247is taking up 90% of the root filesystem and the kernel. Decide what you
4248can eliminate without limiting your device beyond what you need.
4249
4250Depending on your system, a good place to look might be Busybox, which
4251provides a stripped down version of Unix tools in a single, executable
4252file. You might be able to drop virtual terminal services or perhaps
4253ipv6.
4254
4255Building Images for More than One Machine
4256-----------------------------------------
4257
4258A common scenario developers face is creating images for several
4259different machines that use the same software environment. In this
4260situation, it is tempting to set the tunings and optimization flags for
4261each build specifically for the targeted hardware (i.e. "maxing out" the
4262tunings). Doing so can considerably add to build times and package feed
4263maintenance collectively for the machines. For example, selecting tunes
4264that are extremely specific to a CPU core used in a system might enable
4265some micro optimizations in GCC for that particular system but would
4266otherwise not gain you much of a performance difference across the other
4267systems as compared to using a more general tuning across all the builds
4268(e.g. setting :term:`DEFAULTTUNE`
4269specifically for each machine's build). Rather than "max out" each
4270build's tunings, you can take steps that cause the OpenEmbedded build
4271system to reuse software across the various machines where it makes
4272sense.
4273
4274If build speed and package feed maintenance are considerations, you
4275should consider the points in this section that can help you optimize
4276your tunings to best consider build times and package feed maintenance.
4277
4278- *Share the Build Directory:* If at all possible, share the
4279 :term:`TMPDIR` across builds. The
4280 Yocto Project supports switching between different
4281 :term:`MACHINE` values in the same
Andrew Geissler09036742021-06-25 14:25:14 -05004282 :term:`TMPDIR`. This practice is well supported and regularly used by
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004283 developers when building for multiple machines. When you use the same
Andrew Geissler09036742021-06-25 14:25:14 -05004284 :term:`TMPDIR` for multiple machine builds, the OpenEmbedded build system
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004285 can reuse the existing native and often cross-recipes for multiple
4286 machines. Thus, build time decreases.
4287
4288 .. note::
4289
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004290 If :term:`DISTRO` settings change or fundamental configuration settings
Andrew Geissler09036742021-06-25 14:25:14 -05004291 such as the filesystem layout, you need to work with a clean :term:`TMPDIR`.
4292 Sharing :term:`TMPDIR` under these circumstances might work but since it is
Andrew Geissler5f350902021-07-23 13:09:54 -04004293 not guaranteed, you should use a clean :term:`TMPDIR`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004294
4295- *Enable the Appropriate Package Architecture:* By default, the
4296 OpenEmbedded build system enables three levels of package
4297 architectures: "all", "tune" or "package", and "machine". Any given
4298 recipe usually selects one of these package architectures (types) for
4299 its output. Depending for what a given recipe creates packages,
4300 making sure you enable the appropriate package architecture can
4301 directly impact the build time.
4302
4303 A recipe that just generates scripts can enable "all" architecture
4304 because there are no binaries to build. To specifically enable "all"
4305 architecture, be sure your recipe inherits the
4306 :ref:`allarch <ref-classes-allarch>` class.
4307 This class is useful for "all" architectures because it configures
4308 many variables so packages can be used across multiple architectures.
4309
4310 If your recipe needs to generate packages that are machine-specific
4311 or when one of the build or runtime dependencies is already
4312 machine-architecture dependent, which makes your recipe also
4313 machine-architecture dependent, make sure your recipe enables the
4314 "machine" package architecture through the
4315 :term:`MACHINE_ARCH`
Andrew Geisslerc926e172021-05-07 16:11:35 -05004316 variable::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004317
4318 PACKAGE_ARCH = "${MACHINE_ARCH}"
4319
4320 When you do not
4321 specifically enable a package architecture through the
4322 :term:`PACKAGE_ARCH`, The
4323 OpenEmbedded build system defaults to the
Andrew Geisslerc926e172021-05-07 16:11:35 -05004324 :term:`TUNE_PKGARCH` setting::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004325
4326 PACKAGE_ARCH = "${TUNE_PKGARCH}"
4327
4328- *Choose a Generic Tuning File if Possible:* Some tunes are more
4329 generic and can run on multiple targets (e.g. an ``armv5`` set of
4330 packages could run on ``armv6`` and ``armv7`` processors in most
4331 cases). Similarly, ``i486`` binaries could work on ``i586`` and
4332 higher processors. You should realize, however, that advances on
4333 newer processor versions would not be used.
4334
4335 If you select the same tune for several different machines, the
4336 OpenEmbedded build system reuses software previously built, thus
4337 speeding up the overall build time. Realize that even though a new
4338 sysroot for each machine is generated, the software is not recompiled
4339 and only one package feed exists.
4340
William A. Kennington IIIac69b482021-06-02 12:28:27 -07004341- *Manage Granular Level Packaging:* Sometimes there are cases where
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004342 injecting another level of package architecture beyond the three
4343 higher levels noted earlier can be useful. For example, consider how
4344 NXP (formerly Freescale) allows for the easy reuse of binary packages
4345 in their layer
Andrew Geissler09209ee2020-12-13 08:44:15 -06004346 :yocto_git:`meta-freescale </meta-freescale/>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004347 In this example, the
Andrew Geissler09209ee2020-12-13 08:44:15 -06004348 :yocto_git:`fsl-dynamic-packagearch </meta-freescale/tree/classes/fsl-dynamic-packagearch.bbclass>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004349 class shares GPU packages for i.MX53 boards because all boards share
4350 the AMD GPU. The i.MX6-based boards can do the same because all
4351 boards share the Vivante GPU. This class inspects the BitBake
4352 datastore to identify if the package provides or depends on one of
4353 the sub-architecture values. If so, the class sets the
4354 :term:`PACKAGE_ARCH` value
4355 based on the ``MACHINE_SUBARCH`` value. If the package does not
4356 provide or depend on one of the sub-architecture values but it
4357 matches a value in the machine-specific filter, it sets
4358 :term:`MACHINE_ARCH`. This
4359 behavior reduces the number of packages built and saves build time by
4360 reusing binaries.
4361
4362- *Use Tools to Debug Issues:* Sometimes you can run into situations
4363 where software is being rebuilt when you think it should not be. For
4364 example, the OpenEmbedded build system might not be using shared
4365 state between machines when you think it should be. These types of
4366 situations are usually due to references to machine-specific
4367 variables such as :term:`MACHINE`,
4368 :term:`SERIAL_CONSOLES`,
4369 :term:`XSERVER`,
4370 :term:`MACHINE_FEATURES`,
4371 and so forth in code that is supposed to only be tune-specific or
4372 when the recipe depends
4373 (:term:`DEPENDS`,
4374 :term:`RDEPENDS`,
4375 :term:`RRECOMMENDS`,
4376 :term:`RSUGGESTS`, and so forth)
4377 on some other recipe that already has
4378 :term:`PACKAGE_ARCH` defined
4379 as "${MACHINE_ARCH}".
4380
4381 .. note::
4382
4383 Patches to fix any issues identified are most welcome as these
4384 issues occasionally do occur.
4385
4386 For such cases, you can use some tools to help you sort out the
4387 situation:
4388
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004389 - ``state-diff-machines.sh``*:* You can find this tool in the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004390 ``scripts`` directory of the Source Repositories. See the comments
4391 in the script for information on how to use the tool.
4392
4393 - *BitBake's "-S printdiff" Option:* Using this option causes
4394 BitBake to try to establish the closest signature match it can
4395 (e.g. in the shared state cache) and then run ``bitbake-diffsigs``
4396 over the matches to determine the stamps and delta where these two
4397 stamp trees diverge.
4398
4399Building Software from an External Source
4400-----------------------------------------
4401
4402By default, the OpenEmbedded build system uses the
4403:term:`Build Directory` when building source
4404code. The build process involves fetching the source files, unpacking
4405them, and then patching them if necessary before the build takes place.
4406
William A. Kennington IIIac69b482021-06-02 12:28:27 -07004407There are situations where you might want to build software from source
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004408files that are external to and thus outside of the OpenEmbedded build
4409system. For example, suppose you have a project that includes a new BSP
4410with a heavily customized kernel. And, you want to minimize exposing the
4411build system to the development team so that they can focus on their
4412project and maintain everyone's workflow as much as possible. In this
4413case, you want a kernel source directory on the development machine
4414where the development occurs. You want the recipe's
4415:term:`SRC_URI` variable to point to
4416the external directory and use it as is, not copy it.
4417
4418To build from software that comes from an external source, all you need
4419to do is inherit the
4420:ref:`externalsrc <ref-classes-externalsrc>` class
4421and then set the
4422:term:`EXTERNALSRC` variable to
4423point to your external source code. Here are the statements to put in
Andrew Geisslerc926e172021-05-07 16:11:35 -05004424your ``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004425
4426 INHERIT += "externalsrc"
Patrick Williams0ca19cc2021-08-16 14:03:13 -05004427 EXTERNALSRC:pn-myrecipe = "path-to-your-source-tree"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004428
4429This next example shows how to accomplish the same thing by setting
Andrew Geissler09036742021-06-25 14:25:14 -05004430:term:`EXTERNALSRC` in the recipe itself or in the recipe's append file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004431
4432 EXTERNALSRC = "path"
4433 EXTERNALSRC_BUILD = "path"
4434
4435.. note::
4436
4437 In order for these settings to take effect, you must globally or
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004438 locally inherit the :ref:`externalsrc <ref-classes-externalsrc>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004439 class.
4440
Andrew Geissler7e0e3c02022-02-25 20:34:39 +00004441By default, :ref:`ref-classes-externalsrc` builds the source code in a
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004442directory separate from the external source directory as specified by
4443:term:`EXTERNALSRC`. If you need
4444to have the source built in the same directory in which it resides, or
4445some other nominated directory, you can set
4446:term:`EXTERNALSRC_BUILD`
Andrew Geisslerc926e172021-05-07 16:11:35 -05004447to point to that directory::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004448
Patrick Williams0ca19cc2021-08-16 14:03:13 -05004449 EXTERNALSRC_BUILD:pn-myrecipe = "path-to-your-source-tree"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004450
4451Replicating a Build Offline
4452---------------------------
4453
4454It can be useful to take a "snapshot" of upstream sources used in a
4455build and then use that "snapshot" later to replicate the build offline.
4456To do so, you need to first prepare and populate your downloads
4457directory your "snapshot" of files. Once your downloads directory is
4458ready, you can use it at any time and from any machine to replicate your
4459build.
4460
4461Follow these steps to populate your Downloads directory:
4462
44631. *Create a Clean Downloads Directory:* Start with an empty downloads
4464 directory (:term:`DL_DIR`). You
4465 start with an empty downloads directory by either removing the files
Andrew Geissler09036742021-06-25 14:25:14 -05004466 in the existing directory or by setting :term:`DL_DIR` to point to either
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004467 an empty location or one that does not yet exist.
4468
44692. *Generate Tarballs of the Source Git Repositories:* Edit your
Andrew Geisslerc926e172021-05-07 16:11:35 -05004470 ``local.conf`` configuration file as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004471
4472 DL_DIR = "/home/your-download-dir/"
4473 BB_GENERATE_MIRROR_TARBALLS = "1"
4474
4475 During
4476 the fetch process in the next step, BitBake gathers the source files
Andrew Geissler09036742021-06-25 14:25:14 -05004477 and creates tarballs in the directory pointed to by :term:`DL_DIR`. See
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004478 the
4479 :term:`BB_GENERATE_MIRROR_TARBALLS`
4480 variable for more information.
4481
44823. *Populate Your Downloads Directory Without Building:* Use BitBake to
Andrew Geisslerc926e172021-05-07 16:11:35 -05004483 fetch your sources but inhibit the build::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004484
4485 $ bitbake target --runonly=fetch
4486
4487 The downloads directory (i.e. ``${DL_DIR}``) now has
4488 a "snapshot" of the source files in the form of tarballs, which can
4489 be used for the build.
4490
44914. *Optionally Remove Any Git or other SCM Subdirectories From the
4492 Downloads Directory:* If you want, you can clean up your downloads
4493 directory by removing any Git or other Source Control Management
4494 (SCM) subdirectories such as ``${DL_DIR}/git2/*``. The tarballs
4495 already contain these subdirectories.
4496
4497Once your downloads directory has everything it needs regarding source
4498files, you can create your "own-mirror" and build your target.
4499Understand that you can use the files to build the target offline from
4500any machine and at any time.
4501
4502Follow these steps to build your target using the files in the downloads
4503directory:
4504
45051. *Using Local Files Only:* Inside your ``local.conf`` file, add the
Andrew Geissler595f6302022-01-24 19:11:47 +00004506 :term:`SOURCE_MIRROR_URL` variable, inherit the
4507 :ref:`own-mirrors <ref-classes-own-mirrors>` class, and use the
4508 :term:`BB_NO_NETWORK` variable to your ``local.conf``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004509 ::
4510
4511 SOURCE_MIRROR_URL ?= "file:///home/your-download-dir/"
4512 INHERIT += "own-mirrors"
4513 BB_NO_NETWORK = "1"
4514
Andrew Geissler595f6302022-01-24 19:11:47 +00004515 The :term:`SOURCE_MIRROR_URL` and :ref:`own-mirrors <ref-classes-own-mirrors>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004516 class set up the system to use the downloads directory as your "own
Andrew Geissler09036742021-06-25 14:25:14 -05004517 mirror". Using the :term:`BB_NO_NETWORK` variable makes sure that
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004518 BitBake's fetching process in step 3 stays local, which means files
4519 from your "own-mirror" are used.
4520
45212. *Start With a Clean Build:* You can start with a clean build by
4522 removing the
4523 ``${``\ :term:`TMPDIR`\ ``}``
4524 directory or using a new :term:`Build Directory`.
4525
Andrew Geisslerc926e172021-05-07 16:11:35 -050045263. *Build Your Target:* Use BitBake to build your target::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004527
4528 $ bitbake target
4529
4530 The build completes using the known local "snapshot" of source
4531 files from your mirror. The resulting tarballs for your "snapshot" of
4532 source files are in the downloads directory.
4533
4534 .. note::
4535
4536 The offline build does not work if recipes attempt to find the
4537 latest version of software by setting
4538 :term:`SRCREV` to
Andrew Geisslerc926e172021-05-07 16:11:35 -05004539 ``${``\ :term:`AUTOREV`\ ``}``::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004540
4541 SRCREV = "${AUTOREV}"
4542
Andrew Geissler09036742021-06-25 14:25:14 -05004543 When a recipe sets :term:`SRCREV` to
Andrew Geissler5199d832021-09-24 16:47:35 -05004544 ``${``\ :term:`AUTOREV`\ ``}``, the build system accesses the network in an
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004545 attempt to determine the latest version of software from the SCM.
Andrew Geissler09036742021-06-25 14:25:14 -05004546 Typically, recipes that use :term:`AUTOREV` are custom or modified
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004547 recipes. Recipes that reside in public repositories usually do not
Andrew Geissler09036742021-06-25 14:25:14 -05004548 use :term:`AUTOREV`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004549
Andrew Geissler09036742021-06-25 14:25:14 -05004550 If you do have recipes that use :term:`AUTOREV`, you can take steps to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004551 still use the recipes in an offline build. Do the following:
4552
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05004553 1. Use a configuration generated by enabling :ref:`build
4554 history <dev-manual/common-tasks:maintaining build output quality>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004555
4556 2. Use the ``buildhistory-collect-srcrevs`` command to collect the
Andrew Geissler09036742021-06-25 14:25:14 -05004557 stored :term:`SRCREV` values from the build's history. For more
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05004558 information on collecting these values, see the
4559 ":ref:`dev-manual/common-tasks:build history package information`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004560 section.
4561
4562 3. Once you have the correct source revisions, you can modify
Andrew Geissler09036742021-06-25 14:25:14 -05004563 those recipes to set :term:`SRCREV` to specific versions of the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004564 software.
4565
4566Speeding Up a Build
4567===================
4568
4569Build time can be an issue. By default, the build system uses simple
4570controls to try and maximize build efficiency. In general, the default
4571settings for all the following variables result in the most efficient
4572build times when dealing with single socket systems (i.e. a single CPU).
4573If you have multiple CPUs, you might try increasing the default values
4574to gain more speed. See the descriptions in the glossary for each
4575variable for more information:
4576
4577- :term:`BB_NUMBER_THREADS`:
4578 The maximum number of threads BitBake simultaneously executes.
4579
Patrick Williams213cb262021-08-07 19:21:33 -05004580- :term:`BB_NUMBER_PARSE_THREADS`:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004581 The number of threads BitBake uses during parsing.
4582
4583- :term:`PARALLEL_MAKE`: Extra
4584 options passed to the ``make`` command during the
4585 :ref:`ref-tasks-compile` task in
4586 order to specify parallel compilation on the local build host.
4587
4588- :term:`PARALLEL_MAKEINST`:
4589 Extra options passed to the ``make`` command during the
4590 :ref:`ref-tasks-install` task in
4591 order to specify parallel installation on the local build host.
4592
4593As mentioned, these variables all scale to the number of processor cores
4594available on the build system. For single socket systems, this
4595auto-scaling ensures that the build system fundamentally takes advantage
4596of potential parallel operations during the build based on the build
4597machine's capabilities.
4598
4599Following are additional factors that can affect build speed:
4600
4601- File system type: The file system type that the build is being
4602 performed on can also influence performance. Using ``ext4`` is
4603 recommended as compared to ``ext2`` and ``ext3`` due to ``ext4``
4604 improved features such as extents.
4605
4606- Disabling the updating of access time using ``noatime``: The
4607 ``noatime`` mount option prevents the build system from updating file
4608 and directory access times.
4609
4610- Setting a longer commit: Using the "commit=" mount option increases
4611 the interval in seconds between disk cache writes. Changing this
4612 interval from the five second default to something longer increases
4613 the risk of data loss but decreases the need to write to the disk,
4614 thus increasing the build performance.
4615
4616- Choosing the packaging backend: Of the available packaging backends,
4617 IPK is the fastest. Additionally, selecting a singular packaging
4618 backend also helps.
4619
4620- Using ``tmpfs`` for :term:`TMPDIR`
4621 as a temporary file system: While this can help speed up the build,
4622 the benefits are limited due to the compiler using ``-pipe``. The
4623 build system goes to some lengths to avoid ``sync()`` calls into the
4624 file system on the principle that if there was a significant failure,
4625 the :term:`Build Directory`
4626 contents could easily be rebuilt.
4627
4628- Inheriting the
4629 :ref:`rm_work <ref-classes-rm-work>` class:
4630 Inheriting this class has shown to speed up builds due to
4631 significantly lower amounts of data stored in the data cache as well
4632 as on disk. Inheriting this class also makes cleanup of
4633 :term:`TMPDIR` faster, at the
4634 expense of being easily able to dive into the source code. File
4635 system maintainers have recommended that the fastest way to clean up
4636 large numbers of files is to reformat partitions rather than delete
4637 files due to the linear nature of partitions. This, of course,
4638 assumes you structure the disk partitions and file systems in a way
4639 that this is practical.
4640
4641Aside from the previous list, you should keep some trade offs in mind
4642that can help you speed up the build:
4643
4644- Remove items from
4645 :term:`DISTRO_FEATURES`
4646 that you might not need.
4647
4648- Exclude debug symbols and other debug information: If you do not need
4649 these symbols and other debug information, disabling the ``*-dbg``
4650 package generation can speed up the build. You can disable this
4651 generation by setting the
4652 :term:`INHIBIT_PACKAGE_DEBUG_SPLIT`
4653 variable to "1".
4654
4655- Disable static library generation for recipes derived from
4656 ``autoconf`` or ``libtool``: Following is an example showing how to
4657 disable static libraries and still provide an override to handle
Andrew Geisslerc926e172021-05-07 16:11:35 -05004658 exceptions::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004659
4660 STATICLIBCONF = "--disable-static"
Patrick Williams0ca19cc2021-08-16 14:03:13 -05004661 STATICLIBCONF:sqlite3-native = ""
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004662 EXTRA_OECONF += "${STATICLIBCONF}"
4663
4664 .. note::
4665
4666 - Some recipes need static libraries in order to work correctly
4667 (e.g. ``pseudo-native`` needs ``sqlite3-native``). Overrides,
4668 as in the previous example, account for these kinds of
4669 exceptions.
4670
4671 - Some packages have packaging code that assumes the presence of
4672 the static libraries. If so, you might need to exclude them as
4673 well.
4674
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004675Working With Libraries
4676======================
4677
4678Libraries are an integral part of your system. This section describes
4679some common practices you might find helpful when working with libraries
4680to build your system:
4681
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05004682- :ref:`How to include static library files
4683 <dev-manual/common-tasks:including static library files>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004684
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05004685- :ref:`How to use the Multilib feature to combine multiple versions of
4686 library files into a single image
4687 <dev-manual/common-tasks:combining multiple versions of library files into one image>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004688
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05004689- :ref:`How to install multiple versions of the same library in parallel on
4690 the same system
4691 <dev-manual/common-tasks:installing multiple versions of the same library>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004692
4693Including Static Library Files
4694------------------------------
4695
4696If you are building a library and the library offers static linking, you
4697can control which static library files (``*.a`` files) get included in
4698the built library.
4699
4700The :term:`PACKAGES` and
Patrick Williams0ca19cc2021-08-16 14:03:13 -05004701:term:`FILES:* <FILES>` variables in the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004702``meta/conf/bitbake.conf`` configuration file define how files installed
Patrick Williams2194f502022-10-16 14:26:09 -05004703by the :ref:`ref-tasks-install` task are packaged. By default, the :term:`PACKAGES`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004704variable includes ``${PN}-staticdev``, which represents all static
4705library files.
4706
4707.. note::
4708
4709 Some previously released versions of the Yocto Project defined the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004710 static library files through ``${PN}-dev``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004711
4712Following is part of the BitBake configuration file, where you can see
Andrew Geisslerc926e172021-05-07 16:11:35 -05004713how the static library files are defined::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004714
4715 PACKAGE_BEFORE_PN ?= ""
Andrew Geissler595f6302022-01-24 19:11:47 +00004716 PACKAGES = "${PN}-src ${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc ${PN}-locale ${PACKAGE_BEFORE_PN} ${PN}"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004717 PACKAGES_DYNAMIC = "^${PN}-locale-.*"
4718 FILES = ""
4719
Patrick Williams0ca19cc2021-08-16 14:03:13 -05004720 FILES:${PN} = "${bindir}/* ${sbindir}/* ${libexecdir}/* ${libdir}/lib*${SOLIBS} \
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004721 ${sysconfdir} ${sharedstatedir} ${localstatedir} \
4722 ${base_bindir}/* ${base_sbindir}/* \
4723 ${base_libdir}/*${SOLIBS} \
Andrew Geissler595f6302022-01-24 19:11:47 +00004724 ${base_prefix}/lib/udev ${prefix}/lib/udev \
4725 ${base_libdir}/udev ${libdir}/udev \
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004726 ${datadir}/${BPN} ${libdir}/${BPN}/* \
4727 ${datadir}/pixmaps ${datadir}/applications \
4728 ${datadir}/idl ${datadir}/omf ${datadir}/sounds \
4729 ${libdir}/bonobo/servers"
4730
Patrick Williams0ca19cc2021-08-16 14:03:13 -05004731 FILES:${PN}-bin = "${bindir}/* ${sbindir}/*"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004732
Patrick Williams0ca19cc2021-08-16 14:03:13 -05004733 FILES:${PN}-doc = "${docdir} ${mandir} ${infodir} ${datadir}/gtk-doc \
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004734 ${datadir}/gnome/help"
Patrick Williams0ca19cc2021-08-16 14:03:13 -05004735 SECTION:${PN}-doc = "doc"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004736
4737 FILES_SOLIBSDEV ?= "${base_libdir}/lib*${SOLIBSDEV} ${libdir}/lib*${SOLIBSDEV}"
Patrick Williams0ca19cc2021-08-16 14:03:13 -05004738 FILES:${PN}-dev = "${includedir} ${FILES_SOLIBSDEV} ${libdir}/*.la \
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004739 ${libdir}/*.o ${libdir}/pkgconfig ${datadir}/pkgconfig \
4740 ${datadir}/aclocal ${base_libdir}/*.o \
Andrew Geissler595f6302022-01-24 19:11:47 +00004741 ${libdir}/${BPN}/*.la ${base_libdir}/*.la \
4742 ${libdir}/cmake ${datadir}/cmake"
Patrick Williams0ca19cc2021-08-16 14:03:13 -05004743 SECTION:${PN}-dev = "devel"
4744 ALLOW_EMPTY:${PN}-dev = "1"
4745 RDEPENDS:${PN}-dev = "${PN} (= ${EXTENDPKGV})"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004746
Patrick Williams0ca19cc2021-08-16 14:03:13 -05004747 FILES:${PN}-staticdev = "${libdir}/*.a ${base_libdir}/*.a ${libdir}/${BPN}/*.a"
4748 SECTION:${PN}-staticdev = "devel"
4749 RDEPENDS:${PN}-staticdev = "${PN}-dev (= ${EXTENDPKGV})"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004750
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004751Combining Multiple Versions of Library Files into One Image
4752-----------------------------------------------------------
4753
4754The build system offers the ability to build libraries with different
4755target optimizations or architecture formats and combine these together
4756into one system image. You can link different binaries in the image
4757against the different libraries as needed for specific use cases. This
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004758feature is called "Multilib".
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004759
4760An example would be where you have most of a system compiled in 32-bit
4761mode using 32-bit libraries, but you have something large, like a
4762database engine, that needs to be a 64-bit application and uses 64-bit
4763libraries. Multilib allows you to get the best of both 32-bit and 64-bit
4764libraries.
4765
4766While the Multilib feature is most commonly used for 32 and 64-bit
4767differences, the approach the build system uses facilitates different
4768target optimizations. You could compile some binaries to use one set of
4769libraries and other binaries to use a different set of libraries. The
4770libraries could differ in architecture, compiler options, or other
4771optimizations.
4772
William A. Kennington IIIac69b482021-06-02 12:28:27 -07004773There are several examples in the ``meta-skeleton`` layer found in the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004774:term:`Source Directory`:
4775
Andrew Geissler595f6302022-01-24 19:11:47 +00004776- :oe_git:`conf/multilib-example.conf </openembedded-core/tree/meta-skeleton/conf/multilib-example.conf>`
4777 configuration file.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004778
Andrew Geissler595f6302022-01-24 19:11:47 +00004779- :oe_git:`conf/multilib-example2.conf </openembedded-core/tree/meta-skeleton/conf/multilib-example2.conf>`
4780 configuration file.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004781
Andrew Geissler595f6302022-01-24 19:11:47 +00004782- :oe_git:`recipes-multilib/images/core-image-multilib-example.bb </openembedded-core/tree/meta-skeleton/recipes-multilib/images/core-image-multilib-example.bb>`
4783 recipe
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004784
4785Preparing to Use Multilib
4786~~~~~~~~~~~~~~~~~~~~~~~~~
4787
4788User-specific requirements drive the Multilib feature. Consequently,
William A. Kennington IIIac69b482021-06-02 12:28:27 -07004789there is no one "out-of-the-box" configuration that would
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004790meet your needs.
4791
4792In order to enable Multilib, you first need to ensure your recipe is
4793extended to support multiple libraries. Many standard recipes are
4794already extended and support multiple libraries. You can check in the
4795``meta/conf/multilib.conf`` configuration file in the
4796:term:`Source Directory` to see how this is
4797done using the
4798:term:`BBCLASSEXTEND` variable.
4799Eventually, all recipes will be covered and this list will not be
4800needed.
4801
Andrew Geissler595f6302022-01-24 19:11:47 +00004802For the most part, the :ref:`Multilib <ref-classes-multilib*>`
4803class extension works automatically to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004804extend the package name from ``${PN}`` to ``${MLPREFIX}${PN}``, where
Andrew Geissler5f350902021-07-23 13:09:54 -04004805:term:`MLPREFIX` is the particular multilib (e.g. "lib32-" or "lib64-").
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004806Standard variables such as
4807:term:`DEPENDS`,
4808:term:`RDEPENDS`,
4809:term:`RPROVIDES`,
4810:term:`RRECOMMENDS`,
4811:term:`PACKAGES`, and
4812:term:`PACKAGES_DYNAMIC` are
4813automatically extended by the system. If you are extending any manual
4814code in the recipe, you can use the ``${MLPREFIX}`` variable to ensure
Andrew Geissler595f6302022-01-24 19:11:47 +00004815those names are extended correctly.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004816
4817Using Multilib
4818~~~~~~~~~~~~~~
4819
4820After you have set up the recipes, you need to define the actual
4821combination of multiple libraries you want to build. You accomplish this
4822through your ``local.conf`` configuration file in the
4823:term:`Build Directory`. An example
Andrew Geisslerc926e172021-05-07 16:11:35 -05004824configuration would be as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004825
4826 MACHINE = "qemux86-64"
4827 require conf/multilib.conf
4828 MULTILIBS = "multilib:lib32"
Patrick Williams0ca19cc2021-08-16 14:03:13 -05004829 DEFAULTTUNE:virtclass-multilib-lib32 = "x86"
Andrew Geisslerd5838332022-05-27 11:33:10 -05004830 IMAGE_INSTALL:append = " lib32-glib-2.0"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004831
4832This example enables an additional library named
4833``lib32`` alongside the normal target packages. When combining these
4834"lib32" alternatives, the example uses "x86" for tuning. For information
4835on this particular tuning, see
4836``meta/conf/machine/include/ia32/arch-ia32.inc``.
4837
4838The example then includes ``lib32-glib-2.0`` in all the images, which
4839illustrates one method of including a multiple library dependency. You
Andrew Geisslerc926e172021-05-07 16:11:35 -05004840can use a normal image build to include this dependency, for example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004841
4842 $ bitbake core-image-sato
4843
4844You can also build Multilib packages
Andrew Geisslerc926e172021-05-07 16:11:35 -05004845specifically with a command like this::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004846
4847 $ bitbake lib32-glib-2.0
4848
4849Additional Implementation Details
4850~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4851
William A. Kennington IIIac69b482021-06-02 12:28:27 -07004852There are generic implementation details as well as details that are specific to
4853package management systems. Following are implementation details
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004854that exist regardless of the package management system:
4855
4856- The typical convention used for the class extension code as used by
4857 Multilib assumes that all package names specified in
4858 :term:`PACKAGES` that contain
4859 ``${PN}`` have ``${PN}`` at the start of the name. When that
4860 convention is not followed and ``${PN}`` appears at the middle or the
4861 end of a name, problems occur.
4862
4863- The :term:`TARGET_VENDOR`
4864 value under Multilib will be extended to "-vendormlmultilib" (e.g.
4865 "-pokymllib32" for a "lib32" Multilib with Poky). The reason for this
4866 slightly unwieldy contraction is that any "-" characters in the
4867 vendor string presently break Autoconf's ``config.sub``, and other
4868 separators are problematic for different reasons.
4869
William A. Kennington IIIac69b482021-06-02 12:28:27 -07004870Here are the implementation details for the RPM Package Management System:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004871
4872- A unique architecture is defined for the Multilib packages, along
4873 with creating a unique deploy folder under ``tmp/deploy/rpm`` in the
4874 :term:`Build Directory`. For
4875 example, consider ``lib32`` in a ``qemux86-64`` image. The possible
4876 architectures in the system are "all", "qemux86_64",
Patrick Williams0ca19cc2021-08-16 14:03:13 -05004877 "lib32:qemux86_64", and "lib32:x86".
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004878
4879- The ``${MLPREFIX}`` variable is stripped from ``${PN}`` during RPM
4880 packaging. The naming for a normal RPM package and a Multilib RPM
4881 package in a ``qemux86-64`` system resolves to something similar to
4882 ``bash-4.1-r2.x86_64.rpm`` and ``bash-4.1.r2.lib32_x86.rpm``,
4883 respectively.
4884
4885- When installing a Multilib image, the RPM backend first installs the
4886 base image and then installs the Multilib libraries.
4887
4888- The build system relies on RPM to resolve the identical files in the
4889 two (or more) Multilib packages.
4890
William A. Kennington IIIac69b482021-06-02 12:28:27 -07004891Here are the implementation details for the IPK Package Management System:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004892
4893- The ``${MLPREFIX}`` is not stripped from ``${PN}`` during IPK
4894 packaging. The naming for a normal RPM package and a Multilib IPK
4895 package in a ``qemux86-64`` system resolves to something like
Patrick Williams0ca19cc2021-08-16 14:03:13 -05004896 ``bash_4.1-r2.x86_64.ipk`` and ``lib32-bash_4.1-rw:x86.ipk``,
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004897 respectively.
4898
4899- The IPK deploy folder is not modified with ``${MLPREFIX}`` because
4900 packages with and without the Multilib feature can exist in the same
4901 folder due to the ``${PN}`` differences.
4902
4903- IPK defines a sanity check for Multilib installation using certain
4904 rules for file comparison, overridden, etc.
4905
4906Installing Multiple Versions of the Same Library
4907------------------------------------------------
4908
William A. Kennington IIIac69b482021-06-02 12:28:27 -07004909There are be situations where you need to install and use multiple versions
4910of the same library on the same system at the same time. This
4911almost always happens when a library API changes and you have
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004912multiple pieces of software that depend on the separate versions of the
4913library. To accommodate these situations, you can install multiple
4914versions of the same library in parallel on the same system.
4915
4916The process is straightforward as long as the libraries use proper
4917versioning. With properly versioned libraries, all you need to do to
4918individually specify the libraries is create separate, appropriately
4919named recipes where the :term:`PN` part of
4920the name includes a portion that differentiates each library version
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004921(e.g. the major part of the version number). Thus, instead of having a
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004922single recipe that loads one version of a library (e.g. ``clutter``),
4923you provide multiple recipes that result in different versions of the
4924libraries you want. As an example, the following two recipes would allow
4925the two separate versions of the ``clutter`` library to co-exist on the
4926same system:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004927
4928.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004929
4930 clutter-1.6_1.6.20.bb
4931 clutter-1.8_1.8.4.bb
4932
4933Additionally, if
4934you have other recipes that depend on a given library, you need to use
4935the :term:`DEPENDS` variable to
4936create the dependency. Continuing with the same example, if you want to
4937have a recipe depend on the 1.8 version of the ``clutter`` library, use
Andrew Geisslerc926e172021-05-07 16:11:35 -05004938the following in your recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004939
4940 DEPENDS = "clutter-1.8"
4941
Andrew Geissler7e0e3c02022-02-25 20:34:39 +00004942Working with Pre-Built Libraries
4943================================
4944
4945Introduction
4946-------------
4947
4948Some library vendors do not release source code for their software but do
4949release pre-built binaries. When shared libraries are built, they should
4950be versioned (see `this article
4951<https://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html>`__
4952for some background), but sometimes this is not done.
4953
4954To summarize, a versioned library must meet two conditions:
4955
4956#. The filename must have the version appended, for example: ``libfoo.so.1.2.3``.
4957#. The library must have the ELF tag ``SONAME`` set to the major version
4958 of the library, for example: ``libfoo.so.1``. You can check this by
4959 running ``readelf -d filename | grep SONAME``.
4960
4961This section shows how to deal with both versioned and unversioned
4962pre-built libraries.
4963
4964Versioned Libraries
4965-------------------
4966
4967In this example we work with pre-built libraries for the FT4222H USB I/O chip.
4968Libraries are built for several target architecture variants and packaged in
4969an archive as follows::
4970
4971 ├── build-arm-hisiv300
4972 │   └── libft4222.so.1.4.4.44
4973 ├── build-arm-v5-sf
4974 │   └── libft4222.so.1.4.4.44
4975 ├── build-arm-v6-hf
4976 │   └── libft4222.so.1.4.4.44
4977 ├── build-arm-v7-hf
4978 │   └── libft4222.so.1.4.4.44
4979 ├── build-arm-v8
4980 │   └── libft4222.so.1.4.4.44
4981 ├── build-i386
4982 │   └── libft4222.so.1.4.4.44
4983 ├── build-i486
4984 │   └── libft4222.so.1.4.4.44
4985 ├── build-mips-eglibc-hf
4986 │   └── libft4222.so.1.4.4.44
4987 ├── build-pentium
4988 │   └── libft4222.so.1.4.4.44
4989 ├── build-x86_64
4990 │   └── libft4222.so.1.4.4.44
4991 ├── examples
4992 │   ├── get-version.c
4993 │   ├── i2cm.c
4994 │   ├── spim.c
4995 │   └── spis.c
4996 ├── ftd2xx.h
4997 ├── install4222.sh
4998 ├── libft4222.h
4999 ├── ReadMe.txt
5000 └── WinTypes.h
5001
5002To write a recipe to use such a library in your system:
5003
5004- The vendor will probably have a proprietary licence, so set
5005 :term:`LICENSE_FLAGS` in your recipe.
5006- The vendor provides a tarball containing libraries so set :term:`SRC_URI`
5007 appropriately.
5008- Set :term:`COMPATIBLE_HOST` so that the recipe cannot be used with an
5009 unsupported architecture. In the following example, we only support the 32
5010 and 64 bit variants of the ``x86`` architecture.
5011- As the vendor provides versioned libraries, we can use ``oe_soinstall``
5012 from :ref:`ref-classes-utils` to install the shared library and create
5013 symbolic links. If the vendor does not do this, we need to follow the
5014 non-versioned library guidelines in the next section.
5015- As the vendor likely used :term:`LDFLAGS` different from those in your Yocto
5016 Project build, disable the corresponding checks by adding ``ldflags``
5017 to :term:`INSANE_SKIP`.
5018- The vendor will typically ship release builds without debugging symbols.
5019 Avoid errors by preventing the packaging task from stripping out the symbols
5020 and adding them to a separate debug package. This is done by setting the
5021 ``INHIBIT_`` flags shown below.
5022
5023The complete recipe would look like this::
5024
5025 SUMMARY = "FTDI FT4222H Library"
5026 SECTION = "libs"
5027 LICENSE_FLAGS = "ftdi"
5028 LICENSE = "CLOSED"
5029
5030 COMPATIBLE_HOST = "(i.86|x86_64).*-linux"
5031
5032 # Sources available in a .tgz file in .zip archive
5033 # at https://ftdichip.com/wp-content/uploads/2021/01/libft4222-linux-1.4.4.44.zip
5034 # Found on https://ftdichip.com/software-examples/ft4222h-software-examples/
5035 # Since dealing with this particular type of archive is out of topic here,
5036 # we use a local link.
5037 SRC_URI = "file://libft4222-linux-${PV}.tgz"
5038
5039 S = "${WORKDIR}"
5040
5041 ARCH_DIR:x86-64 = "build-x86_64"
5042 ARCH_DIR:i586 = "build-i386"
5043 ARCH_DIR:i686 = "build-i386"
5044
5045 INSANE_SKIP:${PN} = "ldflags"
5046 INHIBIT_PACKAGE_STRIP = "1"
5047 INHIBIT_SYSROOT_STRIP = "1"
5048 INHIBIT_PACKAGE_DEBUG_SPLIT = "1"
5049
5050 do_install () {
5051 install -m 0755 -d ${D}${libdir}
5052 oe_soinstall ${S}/${ARCH_DIR}/libft4222.so.${PV} ${D}${libdir}
5053 install -d ${D}${includedir}
5054 install -m 0755 ${S}/*.h ${D}${includedir}
5055 }
5056
5057If the precompiled binaries are not statically linked and have dependencies on
5058other libraries, then by adding those libraries to :term:`DEPENDS`, the linking
5059can be examined and the appropriate :term:`RDEPENDS` automatically added.
5060
5061Non-Versioned Libraries
5062-----------------------
5063
5064Some Background
5065~~~~~~~~~~~~~~~
5066
5067Libraries in Linux systems are generally versioned so that it is possible
5068to have multiple versions of the same library installed, which eases upgrades
5069and support for older software. For example, suppose that in a versioned
5070library, an actual library is called ``libfoo.so.1.2``, a symbolic link named
5071``libfoo.so.1`` points to ``libfoo.so.1.2``, and a symbolic link named
5072``libfoo.so`` points to ``libfoo.so.1.2``. Given these conditions, when you
5073link a binary against a library, you typically provide the unversioned file
5074name (i.e. ``-lfoo`` to the linker). However, the linker follows the symbolic
5075link and actually links against the versioned filename. The unversioned symbolic
5076link is only used at development time. Consequently, the library is packaged
5077along with the headers in the development package ``${PN}-dev`` along with the
5078actual library and versioned symbolic links in ``${PN}``. Because versioned
5079libraries are far more common than unversioned libraries, the default packaging
5080rules assume versioned libraries.
5081
5082Yocto Library Packaging Overview
5083~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5084
5085It follows that packaging an unversioned library requires a bit of work in the
5086recipe. By default, ``libfoo.so`` gets packaged into ``${PN}-dev``, which
5087triggers a QA warning that a non-symlink library is in a ``-dev`` package,
5088and binaries in the same recipe link to the library in ``${PN}-dev``,
5089which triggers more QA warnings. To solve this problem, you need to package the
5090unversioned library into ``${PN}`` where it belongs. The following are the abridged
5091default :term:`FILES` variables in ``bitbake.conf``::
5092
5093 SOLIBS = ".so.*"
5094 SOLIBSDEV = ".so"
5095 FILES_${PN} = "... ${libdir}/lib*${SOLIBS} ..."
5096 FILES_SOLIBSDEV ?= "... ${libdir}/lib*${SOLIBSDEV} ..."
5097 FILES_${PN}-dev = "... ${FILES_SOLIBSDEV} ..."
5098
5099:term:`SOLIBS` defines a pattern that matches real shared object libraries.
5100:term:`SOLIBSDEV` matches the development form (unversioned symlink). These two
5101variables are then used in ``FILES:${PN}`` and ``FILES:${PN}-dev``, which puts
5102the real libraries into ``${PN}`` and the unversioned symbolic link into ``${PN}-dev``.
5103To package unversioned libraries, you need to modify the variables in the recipe
5104as follows::
5105
5106 SOLIBS = ".so"
5107 FILES_SOLIBSDEV = ""
5108
5109The modifications cause the ``.so`` file to be the real library
5110and unset :term:`FILES_SOLIBSDEV` so that no libraries get packaged into
5111``${PN}-dev``. The changes are required because unless :term:`PACKAGES` is changed,
5112``${PN}-dev`` collects files before `${PN}`. ``${PN}-dev`` must not collect any of
5113the files you want in ``${PN}``.
5114
5115Finally, loadable modules, essentially unversioned libraries that are linked
5116at runtime using ``dlopen()`` instead of at build time, should generally be
5117installed in a private directory. However, if they are installed in ``${libdir}``,
5118then the modules can be treated as unversioned libraries.
5119
5120Example
5121~~~~~~~
5122
5123The example below installs an unversioned x86-64 pre-built library named
5124``libfoo.so``. The :term:`COMPATIBLE_HOST` variable limits recipes to the
5125x86-64 architecture while the :term:`INSANE_SKIP`, :term:`INHIBIT_PACKAGE_STRIP`
5126and :term:`INHIBIT_SYSROOT_STRIP` variables are all set as in the above
5127versioned library example. The "magic" is setting the :term:`SOLIBS` and
5128:term:`FILES_SOLIBSDEV` variables as explained above::
5129
5130 SUMMARY = "libfoo sample recipe"
5131 SECTION = "libs"
5132 LICENSE = "CLOSED"
5133
5134 SRC_URI = "file://libfoo.so"
5135
5136 COMPATIBLE_HOST = "x86_64.*-linux"
5137
5138 INSANE_SKIP:${PN} = "ldflags"
5139 INHIBIT_PACKAGE_STRIP = "1"
5140 INHIBIT_SYSROOT_STRIP = "1"
5141 SOLIBS = ".so"
5142 FILES_SOLIBSDEV = ""
5143
5144 do_install () {
5145 install -d ${D}${libdir}
5146 install -m 0755 ${WORKDIR}/libfoo.so ${D}${libdir}
5147 }
5148
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005149Using x32 psABI
5150===============
5151
5152x32 processor-specific Application Binary Interface (`x32
5153psABI <https://software.intel.com/en-us/node/628948>`__) is a native
515432-bit processor-specific ABI for Intel 64 (x86-64) architectures. An
5155ABI defines the calling conventions between functions in a processing
5156environment. The interface determines what registers are used and what
5157the sizes are for various C data types.
5158
5159Some processing environments prefer using 32-bit applications even when
5160running on Intel 64-bit platforms. Consider the i386 psABI, which is a
5161very old 32-bit ABI for Intel 64-bit platforms. The i386 psABI does not
5162provide efficient use and access of the Intel 64-bit processor
5163resources, leaving the system underutilized. Now consider the x86_64
5164psABI. This ABI is newer and uses 64-bits for data sizes and program
5165pointers. The extra bits increase the footprint size of the programs,
5166libraries, and also increases the memory and file system size
5167requirements. Executing under the x32 psABI enables user programs to
5168utilize CPU and system resources more efficiently while keeping the
5169memory footprint of the applications low. Extra bits are used for
5170registers but not for addressing mechanisms.
5171
5172The Yocto Project supports the final specifications of x32 psABI as
5173follows:
5174
5175- You can create packages and images in x32 psABI format on x86_64
5176 architecture targets.
5177
5178- You can successfully build recipes with the x32 toolchain.
5179
5180- You can create and boot ``core-image-minimal`` and
5181 ``core-image-sato`` images.
5182
William A. Kennington IIIac69b482021-06-02 12:28:27 -07005183- There is RPM Package Manager (RPM) support for x32 binaries.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005184
William A. Kennington IIIac69b482021-06-02 12:28:27 -07005185- There is support for large images.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005186
5187To use the x32 psABI, you need to edit your ``conf/local.conf``
Andrew Geisslerc926e172021-05-07 16:11:35 -05005188configuration file as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005189
5190 MACHINE = "qemux86-64"
5191 DEFAULTTUNE = "x86-64-x32"
Patrick Williams0ca19cc2021-08-16 14:03:13 -05005192 baselib = "${@d.getVar('BASE_LIB:tune-' + (d.getVar('DEFAULTTUNE') \
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005193 or 'INVALID')) or 'lib'}"
5194
5195Once you have set
5196up your configuration file, use BitBake to build an image that supports
Andrew Geisslerc926e172021-05-07 16:11:35 -05005197the x32 psABI. Here is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005198
5199 $ bitbake core-image-sato
5200
5201Enabling GObject Introspection Support
5202======================================
5203
Andrew Geissler595f6302022-01-24 19:11:47 +00005204`GObject introspection <https://gi.readthedocs.io/en/latest/>`__
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005205is the standard mechanism for accessing GObject-based software from
5206runtime environments. GObject is a feature of the GLib library that
5207provides an object framework for the GNOME desktop and related software.
5208GObject Introspection adds information to GObject that allows objects
5209created within it to be represented across different programming
5210languages. If you want to construct GStreamer pipelines using Python, or
5211control UPnP infrastructure using Javascript and GUPnP, GObject
5212introspection is the only way to do it.
5213
5214This section describes the Yocto Project support for generating and
5215packaging GObject introspection data. GObject introspection data is a
Andrew Geissler595f6302022-01-24 19:11:47 +00005216description of the API provided by libraries built on top of the GLib
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005217framework, and, in particular, that framework's GObject mechanism.
5218GObject Introspection Repository (GIR) files go to ``-dev`` packages,
5219``typelib`` files go to main packages as they are packaged together with
5220libraries that are introspected.
5221
5222The data is generated when building such a library, by linking the
5223library with a small executable binary that asks the library to describe
5224itself, and then executing the binary and processing its output.
5225
5226Generating this data in a cross-compilation environment is difficult
5227because the library is produced for the target architecture, but its
5228code needs to be executed on the build host. This problem is solved with
5229the OpenEmbedded build system by running the code through QEMU, which
5230allows precisely that. Unfortunately, QEMU does not always work
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05005231perfectly as mentioned in the ":ref:`dev-manual/common-tasks:known issues`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005232section.
5233
5234Enabling the Generation of Introspection Data
5235---------------------------------------------
5236
5237Enabling the generation of introspection data (GIR files) in your
5238library package involves the following:
5239
52401. Inherit the
5241 :ref:`gobject-introspection <ref-classes-gobject-introspection>`
5242 class.
5243
52442. Make sure introspection is not disabled anywhere in the recipe or
5245 from anything the recipe includes. Also, make sure that
5246 "gobject-introspection-data" is not in
5247 :term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`
5248 and that "qemu-usermode" is not in
5249 :term:`MACHINE_FEATURES_BACKFILL_CONSIDERED`.
William A. Kennington IIIac69b482021-06-02 12:28:27 -07005250 In either of these conditions, nothing will happen.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005251
52523. Try to build the recipe. If you encounter build errors that look like
5253 something is unable to find ``.so`` libraries, check where these
5254 libraries are located in the source tree and add the following to the
Andrew Geisslerc926e172021-05-07 16:11:35 -05005255 recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005256
5257 GIR_EXTRA_LIBS_PATH = "${B}/something/.libs"
5258
5259 .. note::
5260
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005261 See recipes in the ``oe-core`` repository that use that
Andrew Geissler595f6302022-01-24 19:11:47 +00005262 :term:`GIR_EXTRA_LIBS_PATH` variable as an example.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005263
52644. Look for any other errors, which probably mean that introspection
5265 support in a package is not entirely standard, and thus breaks down
5266 in a cross-compilation environment. For such cases, custom-made fixes
5267 are needed. A good place to ask and receive help in these cases is
5268 the :ref:`Yocto Project mailing
5269 lists <resources-mailinglist>`.
5270
5271.. note::
5272
5273 Using a library that no longer builds against the latest Yocto
5274 Project release and prints introspection related errors is a good
5275 candidate for the previous procedure.
5276
5277Disabling the Generation of Introspection Data
5278----------------------------------------------
5279
5280You might find that you do not want to generate introspection data. Or,
5281perhaps QEMU does not work on your build host and target architecture
5282combination. If so, you can use either of the following methods to
5283disable GIR file generations:
5284
Andrew Geisslerc926e172021-05-07 16:11:35 -05005285- Add the following to your distro configuration::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005286
5287 DISTRO_FEATURES_BACKFILL_CONSIDERED = "gobject-introspection-data"
5288
5289 Adding this statement disables generating introspection data using
5290 QEMU but will still enable building introspection tools and libraries
5291 (i.e. building them does not require the use of QEMU).
5292
Andrew Geisslerc926e172021-05-07 16:11:35 -05005293- Add the following to your machine configuration::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005294
5295 MACHINE_FEATURES_BACKFILL_CONSIDERED = "qemu-usermode"
5296
5297 Adding this statement disables the use of QEMU when building packages for your
5298 machine. Currently, this feature is used only by introspection
5299 recipes and has the same effect as the previously described option.
5300
5301 .. note::
5302
5303 Future releases of the Yocto Project might have other features
5304 affected by this option.
5305
5306If you disable introspection data, you can still obtain it through other
5307means such as copying the data from a suitable sysroot, or by generating
5308it on the target hardware. The OpenEmbedded build system does not
5309currently provide specific support for these techniques.
5310
5311Testing that Introspection Works in an Image
5312--------------------------------------------
5313
5314Use the following procedure to test if generating introspection data is
5315working in an image:
5316
53171. Make sure that "gobject-introspection-data" is not in
5318 :term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`
5319 and that "qemu-usermode" is not in
5320 :term:`MACHINE_FEATURES_BACKFILL_CONSIDERED`.
5321
53222. Build ``core-image-sato``.
5323
53243. Launch a Terminal and then start Python in the terminal.
5325
Andrew Geisslerc926e172021-05-07 16:11:35 -050053264. Enter the following in the terminal::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005327
5328 >>> from gi.repository import GLib
5329 >>> GLib.get_host_name()
5330
53315. For something a little more advanced, enter the following see:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005332 https://python-gtk-3-tutorial.readthedocs.io/en/latest/introduction.html
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005333
5334Known Issues
5335------------
5336
William A. Kennington IIIac69b482021-06-02 12:28:27 -07005337Here are know issues in GObject Introspection Support:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005338
5339- ``qemu-ppc64`` immediately crashes. Consequently, you cannot build
5340 introspection data on that architecture.
5341
5342- x32 is not supported by QEMU. Consequently, introspection data is
5343 disabled.
5344
5345- musl causes transient GLib binaries to crash on assertion failures.
5346 Consequently, generating introspection data is disabled.
5347
5348- Because QEMU is not able to run the binaries correctly, introspection
5349 is disabled for some specific packages under specific architectures
5350 (e.g. ``gcr``, ``libsecret``, and ``webkit``).
5351
5352- QEMU usermode might not work properly when running 64-bit binaries
5353 under 32-bit host machines. In particular, "qemumips64" is known to
5354 not work under i686.
5355
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005356Optionally Using an External Toolchain
5357======================================
5358
5359You might want to use an external toolchain as part of your development.
5360If this is the case, the fundamental steps you need to accomplish are as
5361follows:
5362
5363- Understand where the installed toolchain resides. For cases where you
5364 need to build the external toolchain, you would need to take separate
5365 steps to build and install the toolchain.
5366
5367- Make sure you add the layer that contains the toolchain to your
5368 ``bblayers.conf`` file through the
5369 :term:`BBLAYERS` variable.
5370
5371- Set the ``EXTERNAL_TOOLCHAIN`` variable in your ``local.conf`` file
5372 to the location in which you installed the toolchain.
5373
5374A good example of an external toolchain used with the Yocto Project is
5375Mentor Graphics Sourcery G++ Toolchain. You can see information on how
5376to use that particular layer in the ``README`` file at
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005377https://github.com/MentorEmbedded/meta-sourcery/. You can find
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005378further information by reading about the
5379:term:`TCMODE` variable in the Yocto
5380Project Reference Manual's variable glossary.
5381
5382Creating Partitioned Images Using Wic
5383=====================================
5384
5385Creating an image for a particular hardware target using the
5386OpenEmbedded build system does not necessarily mean you can boot that
5387image as is on your device. Physical devices accept and boot images in
5388various ways depending on the specifics of the device. Usually,
5389information about the hardware can tell you what image format the device
5390requires. Should your device require multiple partitions on an SD card,
5391flash, or an HDD, you can use the OpenEmbedded Image Creator, Wic, to
5392create the properly partitioned image.
5393
5394The ``wic`` command generates partitioned images from existing
5395OpenEmbedded build artifacts. Image generation is driven by partitioning
Andrew Geisslerd5838332022-05-27 11:33:10 -05005396commands contained in an OpenEmbedded kickstart file (``.wks``)
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005397specified either directly on the command line or as one of a selection
5398of canned kickstart files as shown with the ``wic list images`` command
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05005399in the
5400":ref:`dev-manual/common-tasks:generate an image using an existing kickstart file`"
5401section. When you apply the command to a given set of build artifacts, the
5402result is an image or set of images that can be directly written onto media and
5403used on a particular system.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005404
5405.. note::
5406
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005407 For a kickstart file reference, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06005408 ":ref:`ref-manual/kickstart:openembedded kickstart (\`\`.wks\`\`) reference`"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005409 Chapter in the Yocto Project Reference Manual.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005410
5411The ``wic`` command and the infrastructure it is based on is by
5412definition incomplete. The purpose of the command is to allow the
5413generation of customized images, and as such, was designed to be
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05005414completely extensible through a plugin interface. See the
5415":ref:`dev-manual/common-tasks:using the wic plugin interface`" section
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005416for information on these plugins.
5417
5418This section provides some background information on Wic, describes what
5419you need to have in place to run the tool, provides instruction on how
5420to use the Wic utility, provides information on using the Wic plugins
5421interface, and provides several examples that show how to use Wic.
5422
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005423Background
5424----------
5425
5426This section provides some background on the Wic utility. While none of
5427this information is required to use Wic, you might find it interesting.
5428
5429- The name "Wic" is derived from OpenEmbedded Image Creator (oeic). The
5430 "oe" diphthong in "oeic" was promoted to the letter "w", because
5431 "oeic" is both difficult to remember and to pronounce.
5432
5433- Wic is loosely based on the Meego Image Creator (``mic``) framework.
5434 The Wic implementation has been heavily modified to make direct use
5435 of OpenEmbedded build artifacts instead of package installation and
5436 configuration, which are already incorporated within the OpenEmbedded
5437 artifacts.
5438
5439- Wic is a completely independent standalone utility that initially
5440 provides easier-to-use and more flexible replacements for an existing
5441 functionality in OE-Core's
5442 :ref:`image-live <ref-classes-image-live>`
5443 class. The difference between Wic and those examples is that with Wic
5444 the functionality of those scripts is implemented by a
5445 general-purpose partitioning language, which is based on Redhat
5446 kickstart syntax.
5447
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005448Requirements
5449------------
5450
5451In order to use the Wic utility with the OpenEmbedded Build system, your
5452system needs to meet the following requirements:
5453
5454- The Linux distribution on your development host must support the
5455 Yocto Project. See the ":ref:`detailed-supported-distros`"
5456 section in the Yocto Project Reference Manual for the list of
5457 distributions that support the Yocto Project.
5458
5459- The standard system utilities, such as ``cp``, must be installed on
5460 your development host system.
5461
5462- You must have sourced the build environment setup script (i.e.
5463 :ref:`structure-core-script`) found in the
5464 :term:`Build Directory`.
5465
5466- You need to have the build artifacts already available, which
5467 typically means that you must have already created an image using the
Andrew Geisslerd5838332022-05-27 11:33:10 -05005468 OpenEmbedded build system (e.g. ``core-image-minimal``). While it
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005469 might seem redundant to generate an image in order to create an image
5470 using Wic, the current version of Wic requires the artifacts in the
5471 form generated by the OpenEmbedded build system.
5472
5473- You must build several native tools, which are built to run on the
Andrew Geisslerc926e172021-05-07 16:11:35 -05005474 build system::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005475
5476 $ bitbake parted-native dosfstools-native mtools-native
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005477
5478- Include "wic" as part of the
5479 :term:`IMAGE_FSTYPES`
5480 variable.
5481
5482- Include the name of the :ref:`wic kickstart file <openembedded-kickstart-wks-reference>`
Andrew Geisslerd5838332022-05-27 11:33:10 -05005483 as part of the :term:`WKS_FILE` variable. If multiple candidate files can
5484 be provided by different layers, specify all the possible names through the
5485 :term:`WKS_FILES` variable instead.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005486
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005487Getting Help
5488------------
5489
5490You can get general help for the ``wic`` command by entering the ``wic``
5491command by itself or by entering the command with a help argument as
Andrew Geisslerc926e172021-05-07 16:11:35 -05005492follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005493
5494 $ wic -h
5495 $ wic --help
5496 $ wic help
5497
5498Currently, Wic supports seven commands: ``cp``, ``create``, ``help``,
5499``list``, ``ls``, ``rm``, and ``write``. You can get help for all these
Andrew Geisslerc926e172021-05-07 16:11:35 -05005500commands except "help" by using the following form::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005501
5502 $ wic help command
5503
5504For example, the following command returns help for the ``write``
Andrew Geisslerc926e172021-05-07 16:11:35 -05005505command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005506
5507 $ wic help write
5508
5509Wic supports help for three topics: ``overview``, ``plugins``, and
Andrew Geisslerc926e172021-05-07 16:11:35 -05005510``kickstart``. You can get help for any topic using the following form::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005511
5512 $ wic help topic
5513
Andrew Geisslerc926e172021-05-07 16:11:35 -05005514For example, the following returns overview help for Wic::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005515
5516 $ wic help overview
5517
William A. Kennington IIIac69b482021-06-02 12:28:27 -07005518There is one additional level of help for Wic. You can get help on
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005519individual images through the ``list`` command. You can use the ``list``
Andrew Geisslerc926e172021-05-07 16:11:35 -05005520command to return the available Wic images as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005521
5522 $ wic list images
5523 genericx86 Create an EFI disk image for genericx86*
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005524 edgerouter Create SD card image for Edgerouter
Andrew Geissler5199d832021-09-24 16:47:35 -05005525 beaglebone-yocto Create SD card image for Beaglebone
5526 qemux86-directdisk Create a qemu machine 'pcbios' direct disk image
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005527 systemd-bootdisk Create an EFI disk image with systemd-boot
5528 mkhybridiso Create a hybrid ISO image
Andrew Geissler5199d832021-09-24 16:47:35 -05005529 mkefidisk Create an EFI disk image
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005530 sdimage-bootpart Create SD card image with a boot partition
5531 directdisk-multi-rootfs Create multi rootfs image using rootfs plugin
Andrew Geissler5199d832021-09-24 16:47:35 -05005532 directdisk Create a 'pcbios' direct disk image
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005533 directdisk-bootloader-config Create a 'pcbios' direct disk image with custom bootloader config
Andrew Geissler5199d832021-09-24 16:47:35 -05005534 qemuriscv Create qcow2 image for RISC-V QEMU machines
5535 directdisk-gpt Create a 'pcbios' direct disk image
5536 efi-bootdisk
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005537
5538Once you know the list of available
5539Wic images, you can use ``help`` with the command to get help on a
5540particular image. For example, the following command returns help on the
Andrew Geisslerc926e172021-05-07 16:11:35 -05005541"beaglebone-yocto" image::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005542
5543 $ wic list beaglebone-yocto help
5544
5545 Creates a partitioned SD card image for Beaglebone.
5546 Boot files are located in the first vfat partition.
5547
5548Operational Modes
5549-----------------
5550
5551You can use Wic in two different modes, depending on how much control
Andrew Geisslerd5838332022-05-27 11:33:10 -05005552you need for specifying the OpenEmbedded build artifacts that are used
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005553for creating the image: Raw and Cooked:
5554
5555- *Raw Mode:* You explicitly specify build artifacts through Wic
5556 command-line arguments.
5557
5558- *Cooked Mode:* The current
5559 :term:`MACHINE` setting and image
5560 name are used to automatically locate and provide the build
5561 artifacts. You just supply a kickstart file and the name of the image
5562 from which to use artifacts.
5563
5564Regardless of the mode you use, you need to have the build artifacts
5565ready and available.
5566
5567Raw Mode
5568~~~~~~~~
5569
5570Running Wic in raw mode allows you to specify all the partitions through
5571the ``wic`` command line. The primary use for raw mode is if you have
5572built your kernel outside of the Yocto Project
5573:term:`Build Directory`. In other words, you
5574can point to arbitrary kernel, root filesystem locations, and so forth.
5575Contrast this behavior with cooked mode where Wic looks in the Build
5576Directory (e.g. ``tmp/deploy/images/``\ machine).
5577
Andrew Geisslerc926e172021-05-07 16:11:35 -05005578The general form of the ``wic`` command in raw mode is::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005579
5580 $ wic create wks_file options ...
5581
5582 Where:
5583
5584 wks_file:
5585 An OpenEmbedded kickstart file. You can provide
5586 your own custom file or use a file from a set of
5587 existing files as described by further options.
5588
5589 optional arguments:
5590 -h, --help show this help message and exit
5591 -o OUTDIR, --outdir OUTDIR
5592 name of directory to create image in
5593 -e IMAGE_NAME, --image-name IMAGE_NAME
5594 name of the image to use the artifacts from e.g. core-
5595 image-sato
5596 -r ROOTFS_DIR, --rootfs-dir ROOTFS_DIR
5597 path to the /rootfs dir to use as the .wks rootfs
5598 source
5599 -b BOOTIMG_DIR, --bootimg-dir BOOTIMG_DIR
5600 path to the dir containing the boot artifacts (e.g.
5601 /EFI or /syslinux dirs) to use as the .wks bootimg
5602 source
5603 -k KERNEL_DIR, --kernel-dir KERNEL_DIR
5604 path to the dir containing the kernel to use in the
5605 .wks bootimg
5606 -n NATIVE_SYSROOT, --native-sysroot NATIVE_SYSROOT
5607 path to the native sysroot containing the tools to use
5608 to build the image
5609 -s, --skip-build-check
5610 skip the build check
5611 -f, --build-rootfs build rootfs
5612 -c {gzip,bzip2,xz}, --compress-with {gzip,bzip2,xz}
5613 compress image with specified compressor
5614 -m, --bmap generate .bmap
5615 --no-fstab-update Do not change fstab file.
5616 -v VARS_DIR, --vars VARS_DIR
5617 directory with <image>.env files that store bitbake
5618 variables
5619 -D, --debug output debug information
5620
5621.. note::
5622
5623 You do not need root privileges to run Wic. In fact, you should not
5624 run as root when using the utility.
5625
5626Cooked Mode
5627~~~~~~~~~~~
5628
5629Running Wic in cooked mode leverages off artifacts in the Build
5630Directory. In other words, you do not have to specify kernel or root
5631filesystem locations as part of the command. All you need to provide is
5632a kickstart file and the name of the image from which to use artifacts
5633by using the "-e" option. Wic looks in the Build Directory (e.g.
5634``tmp/deploy/images/``\ machine) for artifacts.
5635
Andrew Geisslerc926e172021-05-07 16:11:35 -05005636The general form of the ``wic`` command using Cooked Mode is as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005637
5638 $ wic create wks_file -e IMAGE_NAME
5639
5640 Where:
5641
5642 wks_file:
5643 An OpenEmbedded kickstart file. You can provide
5644 your own custom file or use a file from a set of
5645 existing files provided with the Yocto Project
5646 release.
5647
5648 required argument:
5649 -e IMAGE_NAME, --image-name IMAGE_NAME
5650 name of the image to use the artifacts from e.g. core-
5651 image-sato
5652
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005653Using an Existing Kickstart File
5654--------------------------------
5655
5656If you do not want to create your own kickstart file, you can use an
5657existing file provided by the Wic installation. As shipped, kickstart
Andrew Geissler09209ee2020-12-13 08:44:15 -06005658files can be found in the :ref:`overview-manual/development-environment:yocto project source repositories` in the
Andrew Geisslerc926e172021-05-07 16:11:35 -05005659following two locations::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005660
5661 poky/meta-yocto-bsp/wic
5662 poky/scripts/lib/wic/canned-wks
5663
Andrew Geisslerc926e172021-05-07 16:11:35 -05005664Use the following command to list the available kickstart files::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005665
5666 $ wic list images
5667 genericx86 Create an EFI disk image for genericx86*
5668 beaglebone-yocto Create SD card image for Beaglebone
5669 edgerouter Create SD card image for Edgerouter
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05005670 qemux86-directdisk Create a QEMU machine 'pcbios' direct disk image
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005671 directdisk-gpt Create a 'pcbios' direct disk image
5672 mkefidisk Create an EFI disk image
5673 directdisk Create a 'pcbios' direct disk image
5674 systemd-bootdisk Create an EFI disk image with systemd-boot
5675 mkhybridiso Create a hybrid ISO image
5676 sdimage-bootpart Create SD card image with a boot partition
5677 directdisk-multi-rootfs Create multi rootfs image using rootfs plugin
5678 directdisk-bootloader-config Create a 'pcbios' direct disk image with custom bootloader config
5679
5680When you use an existing file, you
5681do not have to use the ``.wks`` extension. Here is an example in Raw
Andrew Geisslerc926e172021-05-07 16:11:35 -05005682Mode that uses the ``directdisk`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005683
5684 $ wic create directdisk -r rootfs_dir -b bootimg_dir \
5685 -k kernel_dir -n native_sysroot
5686
5687Here are the actual partition language commands used in the
Andrew Geisslerc926e172021-05-07 16:11:35 -05005688``genericx86.wks`` file to generate an image::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005689
5690 # short-description: Create an EFI disk image for genericx86*
5691 # long-description: Creates a partitioned EFI disk image for genericx86* machines
5692 part /boot --source bootimg-efi --sourceparams="loader=grub-efi" --ondisk sda --label msdos --active --align 1024
5693 part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 1024 --use-uuid
5694 part swap --ondisk sda --size 44 --label swap1 --fstype=swap
5695
5696 bootloader --ptable gpt --timeout=5 --append="rootfstype=ext4 console=ttyS0,115200 console=tty0"
5697
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005698Using the Wic Plugin Interface
5699------------------------------
5700
5701You can extend and specialize Wic functionality by using Wic plugins.
5702This section explains the Wic plugin interface.
5703
5704.. note::
5705
5706 Wic plugins consist of "source" and "imager" plugins. Imager plugins
5707 are beyond the scope of this section.
5708
5709Source plugins provide a mechanism to customize partition content during
5710the Wic image generation process. You can use source plugins to map
5711values that you specify using ``--source`` commands in kickstart files
5712(i.e. ``*.wks``) to a plugin implementation used to populate a given
5713partition.
5714
5715.. note::
5716
5717 If you use plugins that have build-time dependencies (e.g. native
5718 tools, bootloaders, and so forth) when building a Wic image, you need
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005719 to specify those dependencies using the :term:`WKS_FILE_DEPENDS`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005720 variable.
5721
5722Source plugins are subclasses defined in plugin files. As shipped, the
5723Yocto Project provides several plugin files. You can see the source
5724plugin files that ship with the Yocto Project
Andrew Geissler09209ee2020-12-13 08:44:15 -06005725:yocto_git:`here </poky/tree/scripts/lib/wic/plugins/source>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005726Each of these plugin files contains source plugins that are designed to
5727populate a specific Wic image partition.
5728
5729Source plugins are subclasses of the ``SourcePlugin`` class, which is
5730defined in the ``poky/scripts/lib/wic/pluginbase.py`` file. For example,
5731the ``BootimgEFIPlugin`` source plugin found in the ``bootimg-efi.py``
5732file is a subclass of the ``SourcePlugin`` class, which is found in the
5733``pluginbase.py`` file.
5734
5735You can also implement source plugins in a layer outside of the Source
5736Repositories (external layer). To do so, be sure that your plugin files
5737are located in a directory whose path is
5738``scripts/lib/wic/plugins/source/`` within your external layer. When the
5739plugin files are located there, the source plugins they contain are made
5740available to Wic.
5741
5742When the Wic implementation needs to invoke a partition-specific
5743implementation, it looks for the plugin with the same name as the
5744``--source`` parameter used in the kickstart file given to that
5745partition. For example, if the partition is set up using the following
Andrew Geisslerc926e172021-05-07 16:11:35 -05005746command in a kickstart file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005747
5748 part /boot --source bootimg-pcbios --ondisk sda --label boot --active --align 1024
5749
5750The methods defined as class
5751members of the matching source plugin (i.e. ``bootimg-pcbios``) in the
5752``bootimg-pcbios.py`` plugin file are used.
5753
5754To be more concrete, here is the corresponding plugin definition from
5755the ``bootimg-pcbios.py`` file for the previous command along with an
5756example method called by the Wic implementation when it needs to prepare
Andrew Geisslerc926e172021-05-07 16:11:35 -05005757a partition using an implementation-specific function::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005758
5759 .
5760 .
5761 .
5762 class BootimgPcbiosPlugin(SourcePlugin):
5763 """
5764 Create MBR boot partition and install syslinux on it.
5765 """
5766
5767 name = 'bootimg-pcbios'
5768 .
5769 .
5770 .
5771 @classmethod
5772 def do_prepare_partition(cls, part, source_params, creator, cr_workdir,
5773 oe_builddir, bootimg_dir, kernel_dir,
5774 rootfs_dir, native_sysroot):
5775 """
5776 Called to do the actual content population for a partition i.e. it
5777 'prepares' the partition to be incorporated into the image.
5778 In this case, prepare content for legacy bios boot partition.
5779 """
5780 .
5781 .
5782 .
5783
5784If a
5785subclass (plugin) itself does not implement a particular function, Wic
5786locates and uses the default version in the superclass. It is for this
5787reason that all source plugins are derived from the ``SourcePlugin``
5788class.
5789
5790The ``SourcePlugin`` class defined in the ``pluginbase.py`` file defines
5791a set of methods that source plugins can implement or override. Any
5792plugins (subclass of ``SourcePlugin``) that do not implement a
5793particular method inherit the implementation of the method from the
5794``SourcePlugin`` class. For more information, see the ``SourcePlugin``
5795class in the ``pluginbase.py`` file for details:
5796
5797The following list describes the methods implemented in the
5798``SourcePlugin`` class:
5799
5800- ``do_prepare_partition()``: Called to populate a partition with
5801 actual content. In other words, the method prepares the final
5802 partition image that is incorporated into the disk image.
5803
5804- ``do_configure_partition()``: Called before
5805 ``do_prepare_partition()`` to create custom configuration files for a
5806 partition (e.g. syslinux or grub configuration files).
5807
5808- ``do_install_disk()``: Called after all partitions have been
5809 prepared and assembled into a disk image. This method provides a hook
5810 to allow finalization of a disk image (e.g. writing an MBR).
5811
5812- ``do_stage_partition()``: Special content-staging hook called
5813 before ``do_prepare_partition()``. This method is normally empty.
5814
5815 Typically, a partition just uses the passed-in parameters (e.g. the
5816 unmodified value of ``bootimg_dir``). However, in some cases, things
5817 might need to be more tailored. As an example, certain files might
5818 additionally need to be taken from ``bootimg_dir + /boot``. This hook
5819 allows those files to be staged in a customized fashion.
5820
5821 .. note::
5822
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005823 ``get_bitbake_var()`` allows you to access non-standard variables that
5824 you might want to use for this behavior.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005825
5826You can extend the source plugin mechanism. To add more hooks, create
5827more source plugin methods within ``SourcePlugin`` and the corresponding
5828derived subclasses. The code that calls the plugin methods uses the
5829``plugin.get_source_plugin_methods()`` function to find the method or
5830methods needed by the call. Retrieval of those methods is accomplished
5831by filling up a dict with keys that contain the method names of
5832interest. On success, these will be filled in with the actual methods.
5833See the Wic implementation for examples and details.
5834
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005835Wic Examples
5836------------
5837
5838This section provides several examples that show how to use the Wic
5839utility. All the examples assume the list of requirements in the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05005840":ref:`dev-manual/common-tasks:requirements`" section have been met. The
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005841examples assume the previously generated image is
5842``core-image-minimal``.
5843
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005844Generate an Image using an Existing Kickstart File
5845~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5846
5847This example runs in Cooked Mode and uses the ``mkefidisk`` kickstart
Andrew Geisslerc926e172021-05-07 16:11:35 -05005848file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005849
5850 $ wic create mkefidisk -e core-image-minimal
5851 INFO: Building wic-tools...
5852 .
5853 .
5854 .
5855 INFO: The new image(s) can be found here:
5856 ./mkefidisk-201804191017-sda.direct
5857
5858 The following build artifacts were used to create the image(s):
Andrew Geissler595f6302022-01-24 19:11:47 +00005859 ROOTFS_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
5860 BOOTIMG_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
5861 KERNEL_DIR: /home/stephano/yocto/build/tmp-glibc/deploy/images/qemux86
5862 NATIVE_SYSROOT: /home/stephano/yocto/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005863
5864 INFO: The image(s) were created using OE kickstart file:
Andrew Geissler595f6302022-01-24 19:11:47 +00005865 /home/stephano/yocto/openembedded-core/scripts/lib/wic/canned-wks/mkefidisk.wks
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005866
5867The previous example shows the easiest way to create an image by running
5868in cooked mode and supplying a kickstart file and the "-e" option to
5869point to the existing build artifacts. Your ``local.conf`` file needs to
5870have the :term:`MACHINE` variable set
5871to the machine you are using, which is "qemux86" in this example.
5872
5873Once the image builds, the output provides image location, artifact use,
5874and kickstart file information.
5875
5876.. note::
5877
5878 You should always verify the details provided in the output to make
5879 sure that the image was indeed created exactly as expected.
5880
5881Continuing with the example, you can now write the image from the Build
5882Directory onto a USB stick, or whatever media for which you built your
5883image, and boot from the media. You can write the image by using
Andrew Geisslerc926e172021-05-07 16:11:35 -05005884``bmaptool`` or ``dd``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005885
Andrew Geisslerd5838332022-05-27 11:33:10 -05005886 $ oe-run-native bmap-tools-native bmaptool copy mkefidisk-201804191017-sda.direct /dev/sdX
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005887
5888or ::
5889
5890 $ sudo dd if=mkefidisk-201804191017-sda.direct of=/dev/sdX
5891
5892.. note::
5893
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005894 For more information on how to use the ``bmaptool``
5895 to flash a device with an image, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06005896 ":ref:`dev-manual/common-tasks:flashing images using \`\`bmaptool\`\``"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005897 section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005898
5899Using a Modified Kickstart File
5900~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5901
5902Because partitioned image creation is driven by the kickstart file, it
5903is easy to affect image creation by changing the parameters in the file.
5904This next example demonstrates that through modification of the
5905``directdisk-gpt`` kickstart file.
5906
5907As mentioned earlier, you can use the command ``wic list images`` to
5908show the list of existing kickstart files. The directory in which the
5909``directdisk-gpt.wks`` file resides is
5910``scripts/lib/image/canned-wks/``, which is located in the
5911:term:`Source Directory` (e.g. ``poky``).
5912Because available files reside in this directory, you can create and add
5913your own custom files to the directory. Subsequent use of the
5914``wic list images`` command would then include your kickstart files.
5915
5916In this example, the existing ``directdisk-gpt`` file already does most
5917of what is needed. However, for the hardware in this example, the image
5918will need to boot from ``sdb`` instead of ``sda``, which is what the
5919``directdisk-gpt`` kickstart file uses.
5920
5921The example begins by making a copy of the ``directdisk-gpt.wks`` file
5922in the ``scripts/lib/image/canned-wks`` directory and then by changing
5923the lines that specify the target disk from which to boot.
5924::
5925
Andrew Geissler595f6302022-01-24 19:11:47 +00005926 $ cp /home/stephano/yocto/poky/scripts/lib/wic/canned-wks/directdisk-gpt.wks \
5927 /home/stephano/yocto/poky/scripts/lib/wic/canned-wks/directdisksdb-gpt.wks
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005928
5929Next, the example modifies the ``directdisksdb-gpt.wks`` file and
5930changes all instances of "``--ondisk sda``" to "``--ondisk sdb``". The
5931example changes the following two lines and leaves the remaining lines
Andrew Geisslerc926e172021-05-07 16:11:35 -05005932untouched::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005933
5934 part /boot --source bootimg-pcbios --ondisk sdb --label boot --active --align 1024
5935 part / --source rootfs --ondisk sdb --fstype=ext4 --label platform --align 1024 --use-uuid
5936
5937Once the lines are changed, the
5938example generates the ``directdisksdb-gpt`` image. The command points
5939the process at the ``core-image-minimal`` artifacts for the Next Unit of
5940Computing (nuc) :term:`MACHINE` the
5941``local.conf``.
5942::
5943
5944 $ wic create directdisksdb-gpt -e core-image-minimal
5945 INFO: Building wic-tools...
5946 .
5947 .
5948 .
5949 Initialising tasks: 100% |#######################################| Time: 0:00:01
5950 NOTE: Executing SetScene Tasks
5951 NOTE: Executing RunQueue Tasks
5952 NOTE: Tasks Summary: Attempted 1161 tasks of which 1157 didn't need to be rerun and all succeeded.
5953 INFO: Creating image(s)...
5954
5955 INFO: The new image(s) can be found here:
5956 ./directdisksdb-gpt-201710090938-sdb.direct
5957
5958 The following build artifacts were used to create the image(s):
Andrew Geissler595f6302022-01-24 19:11:47 +00005959 ROOTFS_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
5960 BOOTIMG_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
5961 KERNEL_DIR: /home/stephano/yocto/build/tmp-glibc/deploy/images/qemux86
5962 NATIVE_SYSROOT: /home/stephano/yocto/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005963
5964 INFO: The image(s) were created using OE kickstart file:
Andrew Geissler595f6302022-01-24 19:11:47 +00005965 /home/stephano/yocto/poky/scripts/lib/wic/canned-wks/directdisksdb-gpt.wks
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005966
5967Continuing with the example, you can now directly ``dd`` the image to a
5968USB stick, or whatever media for which you built your image, and boot
Andrew Geisslerc926e172021-05-07 16:11:35 -05005969the resulting media::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005970
5971 $ sudo dd if=directdisksdb-gpt-201710090938-sdb.direct of=/dev/sdb
5972 140966+0 records in
5973 140966+0 records out
5974 72174592 bytes (72 MB, 69 MiB) copied, 78.0282 s, 925 kB/s
5975 $ sudo eject /dev/sdb
5976
5977Using a Modified Kickstart File and Running in Raw Mode
5978~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5979
5980This next example manually specifies each build artifact (runs in Raw
5981Mode) and uses a modified kickstart file. The example also uses the
5982``-o`` option to cause Wic to create the output somewhere other than the
Andrew Geisslerc926e172021-05-07 16:11:35 -05005983default output directory, which is the current directory::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005984
Andrew Geissler595f6302022-01-24 19:11:47 +00005985 $ wic create test.wks -o /home/stephano/testwic \
5986 --rootfs-dir /home/stephano/yocto/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/rootfs \
5987 --bootimg-dir /home/stephano/yocto/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share \
5988 --kernel-dir /home/stephano/yocto/build/tmp/deploy/images/qemux86 \
5989 --native-sysroot /home/stephano/yocto/build/tmp/work/i586-poky-linux/wic-tools/1.0-r0/recipe-sysroot-native
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005990
5991 INFO: Creating image(s)...
5992
5993 INFO: The new image(s) can be found here:
5994 /home/stephano/testwic/test-201710091445-sdb.direct
5995
5996 The following build artifacts were used to create the image(s):
Andrew Geissler595f6302022-01-24 19:11:47 +00005997 ROOTFS_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
5998 BOOTIMG_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
5999 KERNEL_DIR: /home/stephano/yocto/build/tmp-glibc/deploy/images/qemux86
6000 NATIVE_SYSROOT: /home/stephano/yocto/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006001
6002 INFO: The image(s) were created using OE kickstart file:
Andrew Geissler595f6302022-01-24 19:11:47 +00006003 test.wks
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006004
6005For this example,
6006:term:`MACHINE` did not have to be
6007specified in the ``local.conf`` file since the artifact is manually
6008specified.
6009
6010Using Wic to Manipulate an Image
6011~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
6012
6013Wic image manipulation allows you to shorten turnaround time during
6014image development. For example, you can use Wic to delete the kernel
6015partition of a Wic image and then insert a newly built kernel. This
6016saves you time from having to rebuild the entire image each time you
6017modify the kernel.
6018
6019.. note::
6020
6021 In order to use Wic to manipulate a Wic image as in this example,
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006022 your development machine must have the ``mtools`` package installed.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006023
6024The following example examines the contents of the Wic image, deletes
6025the existing kernel, and then inserts a new kernel:
6026
60271. *List the Partitions:* Use the ``wic ls`` command to list all the
Andrew Geisslerc926e172021-05-07 16:11:35 -05006028 partitions in the Wic image::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006029
6030 $ wic ls tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic
6031 Num Start End Size Fstype
6032 1 1048576 25041919 23993344 fat16
6033 2 25165824 72157183 46991360 ext4
6034
6035 The previous output shows two partitions in the
6036 ``core-image-minimal-qemux86.wic`` image.
6037
60382. *Examine a Particular Partition:* Use the ``wic ls`` command again
6039 but in a different form to examine a particular partition.
6040
6041 .. note::
6042
6043 You can get command usage on any Wic command using the following
Andrew Geisslerc926e172021-05-07 16:11:35 -05006044 form::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006045
6046 $ wic help command
6047
6048
6049 For example, the following command shows you the various ways to
6050 use the
6051 wic ls
Andrew Geisslerc926e172021-05-07 16:11:35 -05006052 command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006053
6054 $ wic help ls
6055
6056
Andrew Geisslerc926e172021-05-07 16:11:35 -05006057 The following command shows what is in partition one::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006058
6059 $ wic ls tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1
6060 Volume in drive : is boot
6061 Volume Serial Number is E894-1809
6062 Directory for ::/
6063
6064 libcom32 c32 186500 2017-10-09 16:06
6065 libutil c32 24148 2017-10-09 16:06
6066 syslinux cfg 220 2017-10-09 16:06
6067 vesamenu c32 27104 2017-10-09 16:06
6068 vmlinuz 6904608 2017-10-09 16:06
6069 5 files 7 142 580 bytes
6070 16 582 656 bytes free
6071
6072 The previous output shows five files, with the
6073 ``vmlinuz`` being the kernel.
6074
6075 .. note::
6076
6077 If you see the following error, you need to update or create a
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006078 ``~/.mtoolsrc`` file and be sure to have the line "mtools_skip_check=1"
Andrew Geisslerc926e172021-05-07 16:11:35 -05006079 in the file. Then, run the Wic command again::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006080
6081 ERROR: _exec_cmd: /usr/bin/mdir -i /tmp/wic-parttfokuwra ::/ returned '1' instead of 0
6082 output: Total number of sectors (47824) not a multiple of sectors per track (32)!
6083 Add mtools_skip_check=1 to your .mtoolsrc file to skip this test
6084
6085
60863. *Remove the Old Kernel:* Use the ``wic rm`` command to remove the
Andrew Geisslerc926e172021-05-07 16:11:35 -05006087 ``vmlinuz`` file (kernel)::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006088
6089 $ wic rm tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1/vmlinuz
6090
60914. *Add In the New Kernel:* Use the ``wic cp`` command to add the
6092 updated kernel to the Wic image. Depending on how you built your
6093 kernel, it could be in different places. If you used ``devtool`` and
6094 an SDK to build your kernel, it resides in the ``tmp/work`` directory
6095 of the extensible SDK. If you used ``make`` to build the kernel, the
6096 kernel will be in the ``workspace/sources`` area.
6097
6098 The following example assumes ``devtool`` was used to build the
Andrew Geisslerc926e172021-05-07 16:11:35 -05006099 kernel::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006100
Andrew Geissler95ac1b82021-03-31 14:34:31 -05006101 $ wic cp poky_sdk/tmp/work/qemux86-poky-linux/linux-yocto/4.12.12+git999-r0/linux-yocto-4.12.12+git999/arch/x86/boot/bzImage \
6102 poky/build/tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1/vmlinuz
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006103
6104 Once the new kernel is added back into the image, you can use the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006105 ``dd`` command or :ref:`bmaptool
Andrew Geissler09209ee2020-12-13 08:44:15 -06006106 <dev-manual/common-tasks:flashing images using \`\`bmaptool\`\`>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006107 to flash your wic image onto an SD card or USB stick and test your
6108 target.
6109
6110 .. note::
6111
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006112 Using ``bmaptool`` is generally 10 to 20 times faster than using ``dd``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006113
6114Flashing Images Using ``bmaptool``
6115==================================
6116
6117A fast and easy way to flash an image to a bootable device is to use
6118Bmaptool, which is integrated into the OpenEmbedded build system.
6119Bmaptool is a generic tool that creates a file's block map (bmap) and
6120then uses that map to copy the file. As compared to traditional tools
6121such as dd or cp, Bmaptool can copy (or flash) large files like raw
6122system image files much faster.
6123
6124.. note::
6125
6126 - If you are using Ubuntu or Debian distributions, you can install
6127 the ``bmap-tools`` package using the following command and then
6128 use the tool without specifying ``PATH`` even from the root
Andrew Geisslerc926e172021-05-07 16:11:35 -05006129 account::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006130
Andrew Geisslereff27472021-10-29 15:35:00 -05006131 $ sudo apt install bmap-tools
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006132
6133 - If you are unable to install the ``bmap-tools`` package, you will
Andrew Geisslerc926e172021-05-07 16:11:35 -05006134 need to build Bmaptool before using it. Use the following command::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006135
6136 $ bitbake bmap-tools-native
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006137
6138Following, is an example that shows how to flash a Wic image. Realize
6139that while this example uses a Wic image, you can use Bmaptool to flash
6140any type of image. Use these steps to flash an image using Bmaptool:
6141
61421. *Update your local.conf File:* You need to have the following set
Andrew Geisslerc926e172021-05-07 16:11:35 -05006143 in your ``local.conf`` file before building your image::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006144
6145 IMAGE_FSTYPES += "wic wic.bmap"
6146
61472. *Get Your Image:* Either have your image ready (pre-built with the
6148 :term:`IMAGE_FSTYPES`
Andrew Geisslerc926e172021-05-07 16:11:35 -05006149 setting previously mentioned) or take the step to build the image::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006150
6151 $ bitbake image
6152
61533. *Flash the Device:* Flash the device with the image by using Bmaptool
6154 depending on your particular setup. The following commands assume the
6155 image resides in the Build Directory's ``deploy/images/`` area:
6156
Andrew Geisslerc926e172021-05-07 16:11:35 -05006157 - If you have write access to the media, use this command form::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006158
6159 $ oe-run-native bmap-tools-native bmaptool copy build-directory/tmp/deploy/images/machine/image.wic /dev/sdX
6160
6161 - If you do not have write access to the media, set your permissions
Andrew Geisslerc926e172021-05-07 16:11:35 -05006162 first and then use the same command form::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006163
6164 $ sudo chmod 666 /dev/sdX
6165 $ oe-run-native bmap-tools-native bmaptool copy build-directory/tmp/deploy/images/machine/image.wic /dev/sdX
6166
Andrew Geisslerc926e172021-05-07 16:11:35 -05006167For help on the ``bmaptool`` command, use the following command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006168
6169 $ bmaptool --help
6170
6171Making Images More Secure
6172=========================
6173
6174Security is of increasing concern for embedded devices. Consider the
6175issues and problems discussed in just this sampling of work found across
6176the Internet:
6177
6178- *"*\ `Security Risks of Embedded
6179 Systems <https://www.schneier.com/blog/archives/2014/01/security_risks_9.html>`__\ *"*
6180 by Bruce Schneier
6181
6182- *"*\ `Internet Census
6183 2012 <http://census2012.sourceforge.net/paper.html>`__\ *"* by Carna
6184 Botnet
6185
6186- *"*\ `Security Issues for Embedded
Andrew Geisslerd1e89492021-02-12 15:35:20 -06006187 Devices <https://elinux.org/images/6/6f/Security-issues.pdf>`__\ *"*
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006188 by Jake Edge
6189
6190When securing your image is of concern, there are steps, tools, and
6191variables that you can consider to help you reach the security goals you
6192need for your particular device. Not all situations are identical when
6193it comes to making an image secure. Consequently, this section provides
6194some guidance and suggestions for consideration when you want to make
6195your image more secure.
6196
6197.. note::
6198
6199 Because the security requirements and risks are different for every
6200 type of device, this section cannot provide a complete reference on
6201 securing your custom OS. It is strongly recommended that you also
6202 consult other sources of information on embedded Linux system
6203 hardening and on security.
6204
6205General Considerations
6206----------------------
6207
William A. Kennington IIIac69b482021-06-02 12:28:27 -07006208There are general considerations that help you create more secure images.
6209You should consider the following suggestions to make your device
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006210more secure:
6211
6212- Scan additional code you are adding to the system (e.g. application
6213 code) by using static analysis tools. Look for buffer overflows and
6214 other potential security problems.
6215
6216- Pay particular attention to the security for any web-based
6217 administration interface.
6218
6219 Web interfaces typically need to perform administrative functions and
6220 tend to need to run with elevated privileges. Thus, the consequences
6221 resulting from the interface's security becoming compromised can be
6222 serious. Look for common web vulnerabilities such as
6223 cross-site-scripting (XSS), unvalidated inputs, and so forth.
6224
6225 As with system passwords, the default credentials for accessing a
6226 web-based interface should not be the same across all devices. This
6227 is particularly true if the interface is enabled by default as it can
6228 be assumed that many end-users will not change the credentials.
6229
6230- Ensure you can update the software on the device to mitigate
6231 vulnerabilities discovered in the future. This consideration
6232 especially applies when your device is network-enabled.
6233
6234- Ensure you remove or disable debugging functionality before producing
6235 the final image. For information on how to do this, see the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006236 ":ref:`dev-manual/common-tasks:considerations specific to the openembedded build system`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006237 section.
6238
6239- Ensure you have no network services listening that are not needed.
6240
6241- Remove any software from the image that is not needed.
6242
6243- Enable hardware support for secure boot functionality when your
6244 device supports this functionality.
6245
6246Security Flags
6247--------------
6248
6249The Yocto Project has security flags that you can enable that help make
6250your build output more secure. The security flags are in the
6251``meta/conf/distro/include/security_flags.inc`` file in your
6252:term:`Source Directory` (e.g. ``poky``).
6253
6254.. note::
6255
6256 Depending on the recipe, certain security flags are enabled and
6257 disabled by default.
6258
6259Use the following line in your ``local.conf`` file or in your custom
6260distribution configuration file to enable the security compiler and
Andrew Geisslerc926e172021-05-07 16:11:35 -05006261linker flags for your build::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006262
6263 require conf/distro/include/security_flags.inc
6264
6265Considerations Specific to the OpenEmbedded Build System
6266--------------------------------------------------------
6267
6268You can take some steps that are specific to the OpenEmbedded build
6269system to make your images more secure:
6270
6271- Ensure "debug-tweaks" is not one of your selected
6272 :term:`IMAGE_FEATURES`.
6273 When creating a new project, the default is to provide you with an
6274 initial ``local.conf`` file that enables this feature using the
6275 :term:`EXTRA_IMAGE_FEATURES`
Andrew Geisslerc926e172021-05-07 16:11:35 -05006276 variable with the line::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006277
6278 EXTRA_IMAGE_FEATURES = "debug-tweaks"
6279
6280 To disable that feature, simply comment out that line in your
Andrew Geissler09036742021-06-25 14:25:14 -05006281 ``local.conf`` file, or make sure :term:`IMAGE_FEATURES` does not contain
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006282 "debug-tweaks" before producing your final image. Among other things,
6283 leaving this in place sets the root password as blank, which makes
6284 logging in for debugging or inspection easy during development but
6285 also means anyone can easily log in during production.
6286
6287- It is possible to set a root password for the image and also to set
6288 passwords for any extra users you might add (e.g. administrative or
6289 service type users). When you set up passwords for multiple images or
6290 users, you should not duplicate passwords.
6291
6292 To set up passwords, use the
6293 :ref:`extrausers <ref-classes-extrausers>`
6294 class, which is the preferred method. For an example on how to set up
6295 both root and user passwords, see the
Andrew Geissler7e0e3c02022-02-25 20:34:39 +00006296 ":ref:`ref-classes-extrausers`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006297
6298 .. note::
6299
6300 When adding extra user accounts or setting a root password, be
6301 cautious about setting the same password on every device. If you
6302 do this, and the password you have set is exposed, then every
6303 device is now potentially compromised. If you need this access but
6304 want to ensure security, consider setting a different, random
6305 password for each device. Typically, you do this as a separate
6306 step after you deploy the image onto the device.
6307
6308- Consider enabling a Mandatory Access Control (MAC) framework such as
6309 SMACK or SELinux and tuning it appropriately for your device's usage.
6310 You can find more information in the
Andrew Geissler09209ee2020-12-13 08:44:15 -06006311 :yocto_git:`meta-selinux </meta-selinux/>` layer.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006312
6313Tools for Hardening Your Image
6314------------------------------
6315
6316The Yocto Project provides tools for making your image more secure. You
6317can find these tools in the ``meta-security`` layer of the
6318:yocto_git:`Yocto Project Source Repositories <>`.
6319
6320Creating Your Own Distribution
6321==============================
6322
6323When you build an image using the Yocto Project and do not alter any
6324distribution :term:`Metadata`, you are
6325creating a Poky distribution. If you wish to gain more control over
6326package alternative selections, compile-time options, and other
6327low-level configurations, you can create your own distribution.
6328
6329To create your own distribution, the basic steps consist of creating
6330your own distribution layer, creating your own distribution
6331configuration file, and then adding any needed code and Metadata to the
6332layer. The following steps provide some more detail:
6333
6334- *Create a layer for your new distro:* Create your distribution layer
6335 so that you can keep your Metadata and code for the distribution
6336 separate. It is strongly recommended that you create and use your own
6337 layer for configuration and code. Using your own layer as compared to
6338 just placing configurations in a ``local.conf`` configuration file
6339 makes it easier to reproduce the same build configuration when using
6340 multiple build machines. See the
Andrew Geissler09209ee2020-12-13 08:44:15 -06006341 ":ref:`dev-manual/common-tasks:creating a general layer using the \`\`bitbake-layers\`\` script`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006342 section for information on how to quickly set up a layer.
6343
6344- *Create the distribution configuration file:* The distribution
6345 configuration file needs to be created in the ``conf/distro``
6346 directory of your layer. You need to name it using your distribution
6347 name (e.g. ``mydistro.conf``).
6348
6349 .. note::
6350
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006351 The :term:`DISTRO` variable in your ``local.conf`` file determines the
6352 name of your distribution.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006353
6354 You can split out parts of your configuration file into include files
6355 and then "require" them from within your distribution configuration
6356 file. Be sure to place the include files in the
6357 ``conf/distro/include`` directory of your layer. A common example
6358 usage of include files would be to separate out the selection of
6359 desired version and revisions for individual recipes.
6360
6361 Your configuration file needs to set the following required
6362 variables:
6363
6364 - :term:`DISTRO_NAME`
6365
6366 - :term:`DISTRO_VERSION`
6367
6368 These following variables are optional and you typically set them
6369 from the distribution configuration file:
6370
6371 - :term:`DISTRO_FEATURES`
6372
6373 - :term:`DISTRO_EXTRA_RDEPENDS`
6374
6375 - :term:`DISTRO_EXTRA_RRECOMMENDS`
6376
6377 - :term:`TCLIBC`
6378
6379 .. tip::
6380
6381 If you want to base your distribution configuration file on the
6382 very basic configuration from OE-Core, you can use
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006383 ``conf/distro/defaultsetup.conf`` as a reference and just include
6384 variables that differ as compared to ``defaultsetup.conf``.
6385 Alternatively, you can create a distribution configuration file
6386 from scratch using the ``defaultsetup.conf`` file or configuration files
Andrew Geissler7e0e3c02022-02-25 20:34:39 +00006387 from another distribution such as Poky as a reference.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006388
6389- *Provide miscellaneous variables:* Be sure to define any other
6390 variables for which you want to create a default or enforce as part
6391 of the distribution configuration. You can include nearly any
6392 variable from the ``local.conf`` file. The variables you use are not
6393 limited to the list in the previous bulleted item.
6394
6395- *Point to Your distribution configuration file:* In your
6396 ``local.conf`` file in the :term:`Build Directory`,
6397 set your
6398 :term:`DISTRO` variable to point to
6399 your distribution's configuration file. For example, if your
6400 distribution's configuration file is named ``mydistro.conf``, then
Andrew Geisslerc926e172021-05-07 16:11:35 -05006401 you point to it as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006402
6403 DISTRO = "mydistro"
6404
6405- *Add more to the layer if necessary:* Use your layer to hold other
6406 information needed for the distribution:
6407
6408 - Add recipes for installing distro-specific configuration files
6409 that are not already installed by another recipe. If you have
6410 distro-specific configuration files that are included by an
6411 existing recipe, you should add an append file (``.bbappend``) for
6412 those. For general information and recommendations on how to add
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006413 recipes to your layer, see the
6414 ":ref:`dev-manual/common-tasks:creating your own layer`" and
6415 ":ref:`dev-manual/common-tasks:following best practices when creating layers`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006416 sections.
6417
6418 - Add any image recipes that are specific to your distribution.
6419
6420 - Add a ``psplash`` append file for a branded splash screen. For
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006421 information on append files, see the
Patrick Williams0ca19cc2021-08-16 14:03:13 -05006422 ":ref:`dev-manual/common-tasks:appending other layers metadata with your layer`"
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006423 section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006424
6425 - Add any other append files to make custom changes that are
6426 specific to individual recipes.
6427
6428Creating a Custom Template Configuration Directory
6429==================================================
6430
6431If you are producing your own customized version of the build system for
Andrew Geissler87f5cff2022-09-30 13:13:31 -05006432use by other users, you might want to provide a custom build configuration
6433that includes all the necessary settings and layers (i.e. ``local.conf`` and
6434``bblayers.conf`` that are created in a new build directory) and a custom
6435message that is shown when setting up the build. This can be done by
6436creating one or more template configuration directories in your
6437custom distribution layer.
6438
6439This can be done by using ``bitbake-layers save-build-conf``::
6440
6441 $ bitbake-layers save-build-conf ../../meta-alex/ test-1
6442 NOTE: Starting bitbake server...
6443 NOTE: Configuration template placed into /srv/work/alex/meta-alex/conf/templates/test-1
6444 Please review the files in there, and particularly provide a configuration description in /srv/work/alex/meta-alex/conf/templates/test-1/conf-notes.txt
6445 You can try out the configuration with
6446 TEMPLATECONF=/srv/work/alex/meta-alex/conf/templates/test-1 . /srv/work/alex/poky/oe-init-build-env build-try-test-1
6447
6448The above command takes the config files from the currently active build directory under ``conf``,
6449replaces site-specific paths in ``bblayers.conf`` with ``##OECORE##``-relative paths, and copies
6450the config files into a specified layer under a specified template name.
6451
6452To use those saved templates as a starting point for a build, users should point
6453to one of them with :term:`TEMPLATECONF` environment variable::
6454
6455 TEMPLATECONF=/srv/work/alex/meta-alex/conf/templates/test-1 . /srv/work/alex/poky/oe-init-build-env build-try-test-1
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006456
6457The OpenEmbedded build system uses the environment variable
Andrew Geisslerd5838332022-05-27 11:33:10 -05006458:term:`TEMPLATECONF` to locate the directory from which it gathers
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006459configuration information that ultimately ends up in the
6460:term:`Build Directory` ``conf`` directory.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006461
Andrew Geissler87f5cff2022-09-30 13:13:31 -05006462If :term:`TEMPLATECONF` is not set, the default value is obtained
6463from ``.templateconf`` file that is read from the same directory as
6464``oe-init-build-env`` script. For the Poky reference distribution this
6465would be::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006466
Andrew Geissler87f5cff2022-09-30 13:13:31 -05006467 TEMPLATECONF=${TEMPLATECONF:-meta-poky/conf/templates/default}
6468
6469If you look at a configuration template directory, you will
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006470see the ``bblayers.conf.sample``, ``local.conf.sample``, and
6471``conf-notes.txt`` files. The build system uses these files to form the
Andrew Geissler87f5cff2022-09-30 13:13:31 -05006472respective ``bblayers.conf`` file, ``local.conf`` file, and show
6473users a note about the build they're setting up
6474when running the ``oe-init-build-env`` setup script. These can be
6475edited further if needed to improve or change the build configurations
6476available to the users.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006477
Andrew Geissler595f6302022-01-24 19:11:47 +00006478Conserving Disk Space
6479=====================
6480
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006481Conserving Disk Space During Builds
Andrew Geissler595f6302022-01-24 19:11:47 +00006482-----------------------------------
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006483
6484To help conserve disk space during builds, you can add the following
6485statement to your project's ``local.conf`` configuration file found in
Andrew Geisslerc926e172021-05-07 16:11:35 -05006486the :term:`Build Directory`::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006487
6488 INHERIT += "rm_work"
6489
6490Adding this statement deletes the work directory used for
6491building a recipe once the recipe is built. For more information on
6492"rm_work", see the
6493:ref:`rm_work <ref-classes-rm-work>` class in the
6494Yocto Project Reference Manual.
6495
Andrew Geissler595f6302022-01-24 19:11:47 +00006496Purging Duplicate Shared State Cache Files
6497-------------------------------------------
6498
6499After multiple build iterations, the Shared State (sstate) cache can contain
6500duplicate cache files for a given package, while only the most recent one
6501is likely to be reusable. The following command purges all but the
6502newest sstate cache file for each package::
6503
6504 sstate-cache-management.sh --remove-duplicated --cache-dir=build/sstate-cache
6505
6506This command will ask you to confirm the deletions it identifies.
6507
Patrick Williams92b42cb2022-09-03 06:53:57 -05006508.. note::
Andrew Geissler595f6302022-01-24 19:11:47 +00006509
6510 The duplicated sstate cache files of one package must have the same
6511 architecture, which means that sstate cache files with multiple
6512 architectures are not considered as duplicate.
6513
6514Run ``sstate-cache-management.sh`` for more details about this script.
6515
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006516Working with Packages
6517=====================
6518
6519This section describes a few tasks that involve packages:
6520
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006521- :ref:`dev-manual/common-tasks:excluding packages from an image`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006522
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006523- :ref:`dev-manual/common-tasks:incrementing a package version`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006524
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006525- :ref:`dev-manual/common-tasks:handling optional module packaging`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006526
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006527- :ref:`dev-manual/common-tasks:using runtime package management`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006528
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006529- :ref:`dev-manual/common-tasks:generating and using signed packages`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006530
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006531- :ref:`Setting up and running package test
6532 (ptest) <dev-manual/common-tasks:testing packages with ptest>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006533
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006534- :ref:`dev-manual/common-tasks:creating node package manager (npm) packages`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006535
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006536- :ref:`dev-manual/common-tasks:adding custom metadata to packages`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006537
6538Excluding Packages from an Image
6539--------------------------------
6540
6541You might find it necessary to prevent specific packages from being
6542installed into an image. If so, you can use several variables to direct
6543the build system to essentially ignore installing recommended packages
6544or to not install a package at all.
6545
6546The following list introduces variables you can use to prevent packages
6547from being installed into your image. Each of these variables only works
William A. Kennington IIIac69b482021-06-02 12:28:27 -07006548with IPK and RPM package types, not for Debian packages.
6549Also, you can use these variables from your ``local.conf`` file
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006550or attach them to a specific image recipe by using a recipe name
6551override. For more detail on the variables, see the descriptions in the
6552Yocto Project Reference Manual's glossary chapter.
6553
6554- :term:`BAD_RECOMMENDATIONS`:
6555 Use this variable to specify "recommended-only" packages that you do
6556 not want installed.
6557
6558- :term:`NO_RECOMMENDATIONS`:
6559 Use this variable to prevent all "recommended-only" packages from
6560 being installed.
6561
6562- :term:`PACKAGE_EXCLUDE`:
6563 Use this variable to prevent specific packages from being installed
6564 regardless of whether they are "recommended-only" or not. You need to
6565 realize that the build process could fail with an error when you
6566 prevent the installation of a package whose presence is required by
6567 an installed package.
6568
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006569Incrementing a Package Version
6570------------------------------
6571
6572This section provides some background on how binary package versioning
6573is accomplished and presents some of the services, variables, and
6574terminology involved.
6575
6576In order to understand binary package versioning, you need to consider
6577the following:
6578
6579- Binary Package: The binary package that is eventually built and
6580 installed into an image.
6581
6582- Binary Package Version: The binary package version is composed of two
Andrew Geissler615f2f12022-07-15 14:00:58 -05006583 components --- a version and a revision.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006584
6585 .. note::
6586
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006587 Technically, a third component, the "epoch" (i.e. :term:`PE`) is involved
Andrew Geissler09036742021-06-25 14:25:14 -05006588 but this discussion for the most part ignores :term:`PE`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006589
6590 The version and revision are taken from the
6591 :term:`PV` and
6592 :term:`PR` variables, respectively.
6593
Andrew Geissler09036742021-06-25 14:25:14 -05006594- :term:`PV`: The recipe version. :term:`PV` represents the version of the
6595 software being packaged. Do not confuse :term:`PV` with the binary
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006596 package version.
6597
Andrew Geissler5f350902021-07-23 13:09:54 -04006598- :term:`PR`: The recipe revision.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006599
6600- :term:`SRCPV`: The OpenEmbedded
Andrew Geissler09036742021-06-25 14:25:14 -05006601 build system uses this string to help define the value of :term:`PV` when
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006602 the source code revision needs to be included in it.
6603
Andrew Geissler09209ee2020-12-13 08:44:15 -06006604- :yocto_wiki:`PR Service </PR_Service>`: A
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006605 network-based service that helps automate keeping package feeds
6606 compatible with existing package manager applications such as RPM,
6607 APT, and OPKG.
6608
6609Whenever the binary package content changes, the binary package version
6610must change. Changing the binary package version is accomplished by
Andrew Geissler09036742021-06-25 14:25:14 -05006611changing or "bumping" the :term:`PR` and/or :term:`PV` values. Increasing these
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006612values occurs one of two ways:
6613
6614- Automatically using a Package Revision Service (PR Service).
6615
Andrew Geissler09036742021-06-25 14:25:14 -05006616- Manually incrementing the :term:`PR` and/or :term:`PV` variables.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006617
6618Given a primary challenge of any build system and its users is how to
6619maintain a package feed that is compatible with existing package manager
6620applications such as RPM, APT, and OPKG, using an automated system is
6621much preferred over a manual system. In either system, the main
6622requirement is that binary package version numbering increases in a
William A. Kennington IIIac69b482021-06-02 12:28:27 -07006623linear fashion and that there is a number of version components that
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006624support that linear progression. For information on how to ensure
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006625package revisioning remains linear, see the
6626":ref:`dev-manual/common-tasks:automatically incrementing a package version number`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006627section.
6628
6629The following three sections provide related information on the PR
Andrew Geissler09036742021-06-25 14:25:14 -05006630Service, the manual method for "bumping" :term:`PR` and/or :term:`PV`, and on
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006631how to ensure binary package revisioning remains linear.
6632
6633Working With a PR Service
6634~~~~~~~~~~~~~~~~~~~~~~~~~
6635
6636As mentioned, attempting to maintain revision numbers in the
6637:term:`Metadata` is error prone, inaccurate,
6638and causes problems for people submitting recipes. Conversely, the PR
6639Service automatically generates increasing numbers, particularly the
6640revision field, which removes the human element.
6641
6642.. note::
6643
6644 For additional information on using a PR Service, you can see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06006645 :yocto_wiki:`PR Service </PR_Service>` wiki page.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006646
6647The Yocto Project uses variables in order of decreasing priority to
6648facilitate revision numbering (i.e.
6649:term:`PE`,
6650:term:`PV`, and
6651:term:`PR` for epoch, version, and
6652revision, respectively). The values are highly dependent on the policies
6653and procedures of a given distribution and package feed.
6654
6655Because the OpenEmbedded build system uses
Andrew Geissler09209ee2020-12-13 08:44:15 -06006656":ref:`signatures <overview-manual/concepts:checksums (signatures)>`", which are
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006657unique to a given build, the build system knows when to rebuild
6658packages. All the inputs into a given task are represented by a
6659signature, which can trigger a rebuild when different. Thus, the build
Andrew Geissler09036742021-06-25 14:25:14 -05006660system itself does not rely on the :term:`PR`, :term:`PV`, and :term:`PE` numbers to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006661trigger a rebuild. The signatures, however, can be used to generate
6662these values.
6663
6664The PR Service works with both ``OEBasic`` and ``OEBasicHash``
Andrew Geissler09036742021-06-25 14:25:14 -05006665generators. The value of :term:`PR` bumps when the checksum changes and the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006666different generator mechanisms change signatures under different
6667circumstances.
6668
6669As implemented, the build system includes values from the PR Service
Andrew Geissler09036742021-06-25 14:25:14 -05006670into the :term:`PR` field as an addition using the form "``.x``" so ``r0``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006671becomes ``r0.1``, ``r0.2`` and so forth. This scheme allows existing
Andrew Geissler09036742021-06-25 14:25:14 -05006672:term:`PR` values to be used for whatever reasons, which include manual
6673:term:`PR` bumps, should it be necessary.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006674
6675By default, the PR Service is not enabled or running. Thus, the packages
6676generated are just "self consistent". The build system adds and removes
6677packages and there are no guarantees about upgrade paths but images will
6678be consistent and correct with the latest changes.
6679
William A. Kennington IIIac69b482021-06-02 12:28:27 -07006680The simplest form for a PR Service is for a single host
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006681development system that builds the package feed (building system). For
6682this scenario, you can enable a local PR Service by setting
6683:term:`PRSERV_HOST` in your
Andrew Geisslerc926e172021-05-07 16:11:35 -05006684``local.conf`` file in the :term:`Build Directory`::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006685
6686 PRSERV_HOST = "localhost:0"
6687
6688Once the service is started, packages will automatically
Andrew Geissler09036742021-06-25 14:25:14 -05006689get increasing :term:`PR` values and BitBake takes care of starting and
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006690stopping the server.
6691
6692If you have a more complex setup where multiple host development systems
6693work against a common, shared package feed, you have a single PR Service
6694running and it is connected to each building system. For this scenario,
Andrew Geisslerc926e172021-05-07 16:11:35 -05006695you need to start the PR Service using the ``bitbake-prserv`` command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006696
6697 bitbake-prserv --host ip --port port --start
6698
6699In addition to
6700hand-starting the service, you need to update the ``local.conf`` file of
6701each building system as described earlier so each system points to the
6702server and port.
6703
6704It is also recommended you use build history, which adds some sanity
6705checks to binary package versions, in conjunction with the server that
6706is running the PR Service. To enable build history, add the following to
Andrew Geisslerc926e172021-05-07 16:11:35 -05006707each building system's ``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006708
6709 # It is recommended to activate "buildhistory" for testing the PR service
6710 INHERIT += "buildhistory"
6711 BUILDHISTORY_COMMIT = "1"
6712
6713For information on build
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006714history, see the
6715":ref:`dev-manual/common-tasks:maintaining build output quality`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006716
6717.. note::
6718
Andrew Geissler09036742021-06-25 14:25:14 -05006719 The OpenEmbedded build system does not maintain :term:`PR` information as
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006720 part of the shared state (sstate) packages. If you maintain an sstate
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006721 feed, it's expected that either all your building systems that
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006722 contribute to the sstate feed use a shared PR Service, or you do not
6723 run a PR Service on any of your building systems. Having some systems
6724 use a PR Service while others do not leads to obvious problems.
6725
6726 For more information on shared state, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06006727 ":ref:`overview-manual/concepts:shared state cache`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006728 section in the Yocto Project Overview and Concepts Manual.
6729
6730Manually Bumping PR
6731~~~~~~~~~~~~~~~~~~~
6732
6733The alternative to setting up a PR Service is to manually "bump" the
6734:term:`PR` variable.
6735
6736If a committed change results in changing the package output, then the
6737value of the PR variable needs to be increased (or "bumped") as part of
Andrew Geissler09036742021-06-25 14:25:14 -05006738that commit. For new recipes you should add the :term:`PR` variable and set
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006739its initial value equal to "r0", which is the default. Even though the
6740default value is "r0", the practice of adding it to a new recipe makes
6741it harder to forget to bump the variable when you make changes to the
6742recipe in future.
6743
6744If you are sharing a common ``.inc`` file with multiple recipes, you can
Andrew Geissler09036742021-06-25 14:25:14 -05006745also use the :term:`INC_PR` variable to ensure that the recipes sharing the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006746``.inc`` file are rebuilt when the ``.inc`` file itself is changed. The
Andrew Geissler09036742021-06-25 14:25:14 -05006747``.inc`` file must set :term:`INC_PR` (initially to "r0"), and all recipes
6748referring to it should set :term:`PR` to "${INC_PR}.0" initially,
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006749incrementing the last number when the recipe is changed. If the ``.inc``
Andrew Geissler09036742021-06-25 14:25:14 -05006750file is changed then its :term:`INC_PR` should be incremented.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006751
Andrew Geissler09036742021-06-25 14:25:14 -05006752When upgrading the version of a binary package, assuming the :term:`PV`
6753changes, the :term:`PR` variable should be reset to "r0" (or "${INC_PR}.0"
6754if you are using :term:`INC_PR`).
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006755
6756Usually, version increases occur only to binary packages. However, if
Andrew Geissler09036742021-06-25 14:25:14 -05006757for some reason :term:`PV` changes but does not increase, you can increase
6758the :term:`PE` variable (Package Epoch). The :term:`PE` variable defaults to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006759"0".
6760
6761Binary package version numbering strives to follow the `Debian Version
6762Field Policy
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006763Guidelines <https://www.debian.org/doc/debian-policy/ch-controlfields.html>`__.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006764These guidelines define how versions are compared and what "increasing"
6765a version means.
6766
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006767Automatically Incrementing a Package Version Number
6768~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
6769
6770When fetching a repository, BitBake uses the
6771:term:`SRCREV` variable to determine
6772the specific source code revision from which to build. You set the
Andrew Geissler09036742021-06-25 14:25:14 -05006773:term:`SRCREV` variable to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006774:term:`AUTOREV` to cause the
6775OpenEmbedded build system to automatically use the latest revision of
Andrew Geisslerc926e172021-05-07 16:11:35 -05006776the software::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006777
6778 SRCREV = "${AUTOREV}"
6779
Andrew Geissler09036742021-06-25 14:25:14 -05006780Furthermore, you need to reference :term:`SRCPV` in :term:`PV` in order to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006781automatically update the version whenever the revision of the source
Andrew Geisslerc926e172021-05-07 16:11:35 -05006782code changes. Here is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006783
6784 PV = "1.0+git${SRCPV}"
6785
Andrew Geissler09036742021-06-25 14:25:14 -05006786The OpenEmbedded build system substitutes :term:`SRCPV` with the following:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006787
6788.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006789
6790 AUTOINC+source_code_revision
6791
6792The build system replaces the ``AUTOINC``
6793with a number. The number used depends on the state of the PR Service:
6794
6795- If PR Service is enabled, the build system increments the number,
6796 which is similar to the behavior of
6797 :term:`PR`. This behavior results in
6798 linearly increasing package versions, which is desirable. Here is an
6799 example:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006800
6801 .. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006802
6803 hello-world-git_0.0+git0+b6558dd387-r0.0_armv7a-neon.ipk
6804 hello-world-git_0.0+git1+dd2f5c3565-r0.0_armv7a-neon.ipk
6805
6806- If PR Service is not enabled, the build system replaces the
6807 ``AUTOINC`` placeholder with zero (i.e. "0"). This results in
6808 changing the package version since the source revision is included.
6809 However, package versions are not increased linearly. Here is an
6810 example:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006811
6812 .. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006813
6814 hello-world-git_0.0+git0+b6558dd387-r0.0_armv7a-neon.ipk
6815 hello-world-git_0.0+git0+dd2f5c3565-r0.0_armv7a-neon.ipk
6816
6817In summary, the OpenEmbedded build system does not track the history of
6818binary package versions for this purpose. ``AUTOINC``, in this case, is
Andrew Geissler09036742021-06-25 14:25:14 -05006819comparable to :term:`PR`. If PR server is not enabled, ``AUTOINC`` in the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006820package version is simply replaced by "0". If PR server is enabled, the
6821build system keeps track of the package versions and bumps the number
6822when the package revision changes.
6823
6824Handling Optional Module Packaging
6825----------------------------------
6826
6827Many pieces of software split functionality into optional modules (or
6828plugins) and the plugins that are built might depend on configuration
6829options. To avoid having to duplicate the logic that determines what
6830modules are available in your recipe or to avoid having to package each
6831module by hand, the OpenEmbedded build system provides functionality to
6832handle module packaging dynamically.
6833
6834To handle optional module packaging, you need to do two things:
6835
6836- Ensure the module packaging is actually done.
6837
6838- Ensure that any dependencies on optional modules from other recipes
6839 are satisfied by your recipe.
6840
6841Making Sure the Packaging is Done
6842~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
6843
6844To ensure the module packaging actually gets done, you use the
6845``do_split_packages`` function within the ``populate_packages`` Python
6846function in your recipe. The ``do_split_packages`` function searches for
6847a pattern of files or directories under a specified path and creates a
6848package for each one it finds by appending to the
6849:term:`PACKAGES` variable and
Patrick Williams0ca19cc2021-08-16 14:03:13 -05006850setting the appropriate values for ``FILES:packagename``,
6851``RDEPENDS:packagename``, ``DESCRIPTION:packagename``, and so forth.
Andrew Geisslerc926e172021-05-07 16:11:35 -05006852Here is an example from the ``lighttpd`` recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006853
Patrick Williams0ca19cc2021-08-16 14:03:13 -05006854 python populate_packages:prepend () {
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006855 lighttpd_libdir = d.expand('${libdir}')
6856 do_split_packages(d, lighttpd_libdir, '^mod_(.*).so$',
6857 'lighttpd-module-%s', 'Lighttpd module for %s',
6858 extra_depends='')
6859 }
6860
6861The previous example specifies a number of things in the call to
6862``do_split_packages``.
6863
6864- A directory within the files installed by your recipe through
Patrick Williams2194f502022-10-16 14:26:09 -05006865 :ref:`ref-tasks-install` in which to search.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006866
6867- A regular expression used to match module files in that directory. In
6868 the example, note the parentheses () that mark the part of the
6869 expression from which the module name should be derived.
6870
6871- A pattern to use for the package names.
6872
6873- A description for each package.
6874
6875- An empty string for ``extra_depends``, which disables the default
6876 dependency on the main ``lighttpd`` package. Thus, if a file in
6877 ``${libdir}`` called ``mod_alias.so`` is found, a package called
6878 ``lighttpd-module-alias`` is created for it and the
6879 :term:`DESCRIPTION` is set to
6880 "Lighttpd module for alias".
6881
6882Often, packaging modules is as simple as the previous example. However,
William A. Kennington IIIac69b482021-06-02 12:28:27 -07006883there are more advanced options that you can use within
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006884``do_split_packages`` to modify its behavior. And, if you need to, you
6885can add more logic by specifying a hook function that is called for each
6886package. It is also perfectly acceptable to call ``do_split_packages``
6887multiple times if you have more than one set of modules to package.
6888
6889For more examples that show how to use ``do_split_packages``, see the
6890``connman.inc`` file in the ``meta/recipes-connectivity/connman/``
Andrew Geissler09209ee2020-12-13 08:44:15 -06006891directory of the ``poky`` :ref:`source repository <overview-manual/development-environment:yocto project source repositories>`. You can
Patrick Williams975a06f2022-10-21 14:42:47 -05006892also find examples in ``meta/classes-recipe/kernel.bbclass``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006893
6894Following is a reference that shows ``do_split_packages`` mandatory and
Andrew Geisslerc926e172021-05-07 16:11:35 -05006895optional arguments::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006896
6897 Mandatory arguments
6898
6899 root
6900 The path in which to search
6901 file_regex
6902 Regular expression to match searched files.
6903 Use parentheses () to mark the part of this
6904 expression that should be used to derive the
6905 module name (to be substituted where %s is
6906 used in other function arguments as noted below)
6907 output_pattern
6908 Pattern to use for the package names. Must
6909 include %s.
6910 description
6911 Description to set for each package. Must
6912 include %s.
6913
6914 Optional arguments
6915
6916 postinst
6917 Postinstall script to use for all packages
6918 (as a string)
6919 recursive
Andrew Geissler615f2f12022-07-15 14:00:58 -05006920 True to perform a recursive search --- default
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006921 False
6922 hook
6923 A hook function to be called for every match.
6924 The function will be called with the following
6925 arguments (in the order listed):
6926
6927 f
6928 Full path to the file/directory match
6929 pkg
6930 The package name
6931 file_regex
6932 As above
6933 output_pattern
6934 As above
6935 modulename
6936 The module name derived using file_regex
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006937 extra_depends
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006938 Extra runtime dependencies (RDEPENDS) to be
6939 set for all packages. The default value of None
6940 causes a dependency on the main package
Andrew Geissler615f2f12022-07-15 14:00:58 -05006941 (${PN}) --- if you do not want this, pass empty
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006942 string '' for this parameter.
6943 aux_files_pattern
6944 Extra item(s) to be added to FILES for each
6945 package. Can be a single string item or a list
6946 of strings for multiple items. Must include %s.
6947 postrm
6948 postrm script to use for all packages (as a
6949 string)
6950 allow_dirs
6951 True to allow directories to be matched -
6952 default False
6953 prepend
6954 If True, prepend created packages to PACKAGES
6955 instead of the default False which appends them
6956 match_path
6957 match file_regex on the whole relative path to
Patrick Williams0ca19cc2021-08-16 14:03:13 -05006958 the root rather than just the filename
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006959 aux_files_pattern_verbatim
6960 Extra item(s) to be added to FILES for each
6961 package, using the actual derived module name
6962 rather than converting it to something legal
6963 for a package name. Can be a single string item
6964 or a list of strings for multiple items. Must
6965 include %s.
6966 allow_links
Andrew Geissler615f2f12022-07-15 14:00:58 -05006967 True to allow symlinks to be matched --- default
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006968 False
6969 summary
6970 Summary to set for each package. Must include %s;
6971 defaults to description if not set.
6972
6973
6974
6975Satisfying Dependencies
6976~~~~~~~~~~~~~~~~~~~~~~~
6977
6978The second part for handling optional module packaging is to ensure that
6979any dependencies on optional modules from other recipes are satisfied by
6980your recipe. You can be sure these dependencies are satisfied by using
6981the :term:`PACKAGES_DYNAMIC`
6982variable. Here is an example that continues with the ``lighttpd`` recipe
Andrew Geisslerc926e172021-05-07 16:11:35 -05006983shown earlier::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006984
6985 PACKAGES_DYNAMIC = "lighttpd-module-.*"
6986
6987The name
6988specified in the regular expression can of course be anything. In this
6989example, it is ``lighttpd-module-`` and is specified as the prefix to
6990ensure that any :term:`RDEPENDS` and
6991:term:`RRECOMMENDS` on a package
6992name starting with the prefix are satisfied during build time. If you
6993are using ``do_split_packages`` as described in the previous section,
Andrew Geissler09036742021-06-25 14:25:14 -05006994the value you put in :term:`PACKAGES_DYNAMIC` should correspond to the name
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006995pattern specified in the call to ``do_split_packages``.
6996
6997Using Runtime Package Management
6998--------------------------------
6999
7000During a build, BitBake always transforms a recipe into one or more
7001packages. For example, BitBake takes the ``bash`` recipe and produces a
7002number of packages (e.g. ``bash``, ``bash-bashbug``,
7003``bash-completion``, ``bash-completion-dbg``, ``bash-completion-dev``,
7004``bash-completion-extra``, ``bash-dbg``, and so forth). Not all
7005generated packages are included in an image.
7006
7007In several situations, you might need to update, add, remove, or query
7008the packages on a target device at runtime (i.e. without having to
7009generate a new image). Examples of such situations include:
7010
7011- You want to provide in-the-field updates to deployed devices (e.g.
7012 security updates).
7013
7014- You want to have a fast turn-around development cycle for one or more
7015 applications that run on your device.
7016
7017- You want to temporarily install the "debug" packages of various
7018 applications on your device so that debugging can be greatly improved
7019 by allowing access to symbols and source debugging.
7020
7021- You want to deploy a more minimal package selection of your device
7022 but allow in-the-field updates to add a larger selection for
7023 customization.
7024
7025In all these situations, you have something similar to a more
7026traditional Linux distribution in that in-field devices are able to
7027receive pre-compiled packages from a server for installation or update.
7028Being able to install these packages on a running, in-field device is
7029what is termed "runtime package management".
7030
7031In order to use runtime package management, you need a host or server
7032machine that serves up the pre-compiled packages plus the required
7033metadata. You also need package manipulation tools on the target. The
7034build machine is a likely candidate to act as the server. However, that
7035machine does not necessarily have to be the package server. The build
7036machine could push its artifacts to another machine that acts as the
7037server (e.g. Internet-facing). In fact, doing so is advantageous for a
7038production environment as getting the packages away from the development
7039system's build directory prevents accidental overwrites.
7040
7041A simple build that targets just one device produces more than one
7042package database. In other words, the packages produced by a build are
7043separated out into a couple of different package groupings based on
7044criteria such as the target's CPU architecture, the target board, or the
7045C library used on the target. For example, a build targeting the
7046``qemux86`` device produces the following three package databases:
7047``noarch``, ``i586``, and ``qemux86``. If you wanted your ``qemux86``
7048device to be aware of all the packages that were available to it, you
7049would need to point it to each of these databases individually. In a
7050similar way, a traditional Linux distribution usually is configured to
7051be aware of a number of software repositories from which it retrieves
7052packages.
7053
7054Using runtime package management is completely optional and not required
7055for a successful build or deployment in any way. But if you want to make
7056use of runtime package management, you need to do a couple things above
7057and beyond the basics. The remainder of this section describes what you
7058need to do.
7059
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007060Build Considerations
7061~~~~~~~~~~~~~~~~~~~~
7062
7063This section describes build considerations of which you need to be
7064aware in order to provide support for runtime package management.
7065
7066When BitBake generates packages, it needs to know what format or formats
7067to use. In your configuration, you use the
7068:term:`PACKAGE_CLASSES`
7069variable to specify the format:
7070
70711. Open the ``local.conf`` file inside your
7072 :term:`Build Directory` (e.g.
Andrew Geissler95ac1b82021-03-31 14:34:31 -05007073 ``poky/build/conf/local.conf``).
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007074
Andrew Geisslerc926e172021-05-07 16:11:35 -050070752. Select the desired package format as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007076
7077 PACKAGE_CLASSES ?= "package_packageformat"
7078
7079 where packageformat can be "ipk", "rpm",
7080 "deb", or "tar" which are the supported package formats.
7081
7082 .. note::
7083
7084 Because the Yocto Project supports four different package formats,
7085 you can set the variable with more than one argument. However, the
7086 OpenEmbedded build system only uses the first argument when
7087 creating an image or Software Development Kit (SDK).
7088
7089If you would like your image to start off with a basic package database
7090containing the packages in your current build as well as to have the
7091relevant tools available on the target for runtime package management,
7092you can include "package-management" in the
7093:term:`IMAGE_FEATURES`
7094variable. Including "package-management" in this configuration variable
7095ensures that when the image is assembled for your target, the image
7096includes the currently-known package databases as well as the
7097target-specific tools required for runtime package management to be
7098performed on the target. However, this is not strictly necessary. You
7099could start your image off without any databases but only include the
7100required on-target package tool(s). As an example, you could include
7101"opkg" in your
7102:term:`IMAGE_INSTALL` variable
7103if you are using the IPK package format. You can then initialize your
7104target's package database(s) later once your image is up and running.
7105
7106Whenever you perform any sort of build step that can potentially
7107generate a package or modify existing package, it is always a good idea
7108to re-generate the package index after the build by using the following
Andrew Geisslerc926e172021-05-07 16:11:35 -05007109command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007110
7111 $ bitbake package-index
7112
7113It might be tempting to build the
7114package and the package index at the same time with a command such as
Andrew Geisslerc926e172021-05-07 16:11:35 -05007115the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007116
7117 $ bitbake some-package package-index
7118
7119Do not do this as
7120BitBake does not schedule the package index for after the completion of
7121the package you are building. Consequently, you cannot be sure of the
7122package index including information for the package you just built.
7123Thus, be sure to run the package update step separately after building
7124any packages.
7125
7126You can use the
7127:term:`PACKAGE_FEED_ARCHS`,
7128:term:`PACKAGE_FEED_BASE_PATHS`,
7129and
7130:term:`PACKAGE_FEED_URIS`
7131variables to pre-configure target images to use a package feed. If you
7132do not define these variables, then manual steps as described in the
7133subsequent sections are necessary to configure the target. You should
7134set these variables before building the image in order to produce a
7135correctly configured image.
7136
7137When your build is complete, your packages reside in the
7138``${TMPDIR}/deploy/packageformat`` directory. For example, if
7139``${``\ :term:`TMPDIR`\ ``}`` is
7140``tmp`` and your selected package type is RPM, then your RPM packages
7141are available in ``tmp/deploy/rpm``.
7142
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007143Host or Server Machine Setup
7144~~~~~~~~~~~~~~~~~~~~~~~~~~~~
7145
7146Although other protocols are possible, a server using HTTP typically
7147serves packages. If you want to use HTTP, then set up and configure a
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007148web server such as Apache 2, lighttpd, or Python web server on the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007149machine serving the packages.
7150
7151To keep things simple, this section describes how to set up a
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007152Python web server to share package feeds from the developer's
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007153machine. Although this server might not be the best for a production
7154environment, the setup is simple and straight forward. Should you want
7155to use a different server more suited for production (e.g. Apache 2,
7156Lighttpd, or Nginx), take the appropriate steps to do so.
7157
7158From within the build directory where you have built an image based on
7159your packaging choice (i.e. the
7160:term:`PACKAGE_CLASSES`
7161setting), simply start the server. The following example assumes a build
Andrew Geissler09036742021-06-25 14:25:14 -05007162directory of ``poky/build/tmp/deploy/rpm`` and a :term:`PACKAGE_CLASSES`
Andrew Geisslerc926e172021-05-07 16:11:35 -05007163setting of "package_rpm"::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007164
Andrew Geissler95ac1b82021-03-31 14:34:31 -05007165 $ cd poky/build/tmp/deploy/rpm
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007166 $ python3 -m http.server
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007167
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007168Target Setup
7169~~~~~~~~~~~~
7170
7171Setting up the target differs depending on the package management
7172system. This section provides information for RPM, IPK, and DEB.
7173
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007174Using RPM
7175^^^^^^^^^
7176
7177The `Dandified Packaging
7178Tool <https://en.wikipedia.org/wiki/DNF_(software)>`__ (DNF) performs
7179runtime package management of RPM packages. In order to use DNF for
7180runtime package management, you must perform an initial setup on the
7181target machine for cases where the ``PACKAGE_FEED_*`` variables were not
7182set as part of the image that is running on the target. This means if
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05007183you built your image and did not use these variables as part of the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007184build and your image is now running on the target, you need to perform
7185the steps in this section if you want to use runtime package management.
7186
7187.. note::
7188
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007189 For information on the ``PACKAGE_FEED_*`` variables, see
7190 :term:`PACKAGE_FEED_ARCHS`, :term:`PACKAGE_FEED_BASE_PATHS`, and
7191 :term:`PACKAGE_FEED_URIS` in the Yocto Project Reference Manual variables
7192 glossary.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007193
7194On the target, you must inform DNF that package databases are available.
7195You do this by creating a file named
7196``/etc/yum.repos.d/oe-packages.repo`` and defining the ``oe-packages``.
7197
7198As an example, assume the target is able to use the following package
7199databases: ``all``, ``i586``, and ``qemux86`` from a server named
7200``my.server``. The specifics for setting up the web server are up to
7201you. The critical requirement is that the URIs in the target repository
7202configuration point to the correct remote location for the feeds.
7203
7204.. note::
7205
7206 For development purposes, you can point the web server to the build
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007207 system's ``deploy`` directory. However, for production use, it is better to
7208 copy the package directories to a location outside of the build area and use
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007209 that location. Doing so avoids situations where the build system
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007210 overwrites or changes the ``deploy`` directory.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007211
7212When telling DNF where to look for the package databases, you must
7213declare individual locations per architecture or a single location used
7214for all architectures. You cannot do both:
7215
7216- *Create an Explicit List of Architectures:* Define individual base
7217 URLs to identify where each package database is located:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007218
7219 .. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007220
7221 [oe-packages]
7222 baseurl=http://my.server/rpm/i586 http://my.server/rpm/qemux86 http://my.server/rpm/all
7223
7224 This example
7225 informs DNF about individual package databases for all three
7226 architectures.
7227
7228- *Create a Single (Full) Package Index:* Define a single base URL that
Andrew Geisslerc926e172021-05-07 16:11:35 -05007229 identifies where a full package database is located::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007230
7231 [oe-packages]
7232 baseurl=http://my.server/rpm
7233
7234 This example informs DNF about a single
7235 package database that contains all the package index information for
7236 all supported architectures.
7237
7238Once you have informed DNF where to find the package databases, you need
7239to fetch them:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007240
7241.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007242
7243 # dnf makecache
7244
7245DNF is now able to find, install, and
7246upgrade packages from the specified repository or repositories.
7247
7248.. note::
7249
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007250 See the `DNF documentation <https://dnf.readthedocs.io/en/latest/>`__ for
7251 additional information.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007252
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007253Using IPK
7254^^^^^^^^^
7255
7256The ``opkg`` application performs runtime package management of IPK
7257packages. You must perform an initial setup for ``opkg`` on the target
7258machine if the
7259:term:`PACKAGE_FEED_ARCHS`,
7260:term:`PACKAGE_FEED_BASE_PATHS`,
7261and
7262:term:`PACKAGE_FEED_URIS`
7263variables have not been set or the target image was built before the
7264variables were set.
7265
7266The ``opkg`` application uses configuration files to find available
7267package databases. Thus, you need to create a configuration file inside
Andrew Geissler7e0e3c02022-02-25 20:34:39 +00007268the ``/etc/opkg/`` directory, which informs ``opkg`` of any repository
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007269you want to use.
7270
7271As an example, suppose you are serving packages from a ``ipk/``
7272directory containing the ``i586``, ``all``, and ``qemux86`` databases
7273through an HTTP server named ``my.server``. On the target, create a
7274configuration file (e.g. ``my_repo.conf``) inside the ``/etc/opkg/``
7275directory containing the following:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007276
7277.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007278
7279 src/gz all http://my.server/ipk/all
7280 src/gz i586 http://my.server/ipk/i586
7281 src/gz qemux86 http://my.server/ipk/qemux86
7282
7283Next, instruct ``opkg`` to fetch the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007284repository information:
7285
7286.. code-block:: none
7287
7288 # opkg update
7289
7290The ``opkg`` application is now able to find, install, and upgrade packages
7291from the specified repository.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007292
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007293Using DEB
7294^^^^^^^^^
7295
7296The ``apt`` application performs runtime package management of DEB
7297packages. This application uses a source list file to find available
7298package databases. You must perform an initial setup for ``apt`` on the
7299target machine if the
7300:term:`PACKAGE_FEED_ARCHS`,
7301:term:`PACKAGE_FEED_BASE_PATHS`,
7302and
7303:term:`PACKAGE_FEED_URIS`
7304variables have not been set or the target image was built before the
7305variables were set.
7306
7307To inform ``apt`` of the repository you want to use, you might create a
7308list file (e.g. ``my_repo.list``) inside the
7309``/etc/apt/sources.list.d/`` directory. As an example, suppose you are
7310serving packages from a ``deb/`` directory containing the ``i586``,
7311``all``, and ``qemux86`` databases through an HTTP server named
7312``my.server``. The list file should contain:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007313
7314.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007315
7316 deb http://my.server/deb/all ./
7317 deb http://my.server/deb/i586 ./
7318 deb http://my.server/deb/qemux86 ./
7319
7320Next, instruct the ``apt`` application
7321to fetch the repository information:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007322
7323.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007324
Andrew Geisslereff27472021-10-29 15:35:00 -05007325 $ sudo apt update
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007326
7327After this step,
7328``apt`` is able to find, install, and upgrade packages from the
7329specified repository.
7330
7331Generating and Using Signed Packages
7332------------------------------------
7333
7334In order to add security to RPM packages used during a build, you can
7335take steps to securely sign them. Once a signature is verified, the
7336OpenEmbedded build system can use the package in the build. If security
Andrew Geissler595f6302022-01-24 19:11:47 +00007337fails for a signed package, the build system stops the build.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007338
7339This section describes how to sign RPM packages during a build and how
7340to use signed package feeds (repositories) when doing a build.
7341
7342Signing RPM Packages
7343~~~~~~~~~~~~~~~~~~~~
7344
7345To enable signing RPM packages, you must set up the following
7346configurations in either your ``local.config`` or ``distro.config``
Andrew Geisslerc926e172021-05-07 16:11:35 -05007347file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007348
7349 # Inherit sign_rpm.bbclass to enable signing functionality
7350 INHERIT += " sign_rpm"
7351 # Define the GPG key that will be used for signing.
7352 RPM_GPG_NAME = "key_name"
7353 # Provide passphrase for the key
7354 RPM_GPG_PASSPHRASE = "passphrase"
7355
7356.. note::
7357
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007358 Be sure to supply appropriate values for both `key_name` and
7359 `passphrase`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007360
7361Aside from the ``RPM_GPG_NAME`` and ``RPM_GPG_PASSPHRASE`` variables in
William A. Kennington IIIac69b482021-06-02 12:28:27 -07007362the previous example, two optional variables related to signing are available:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007363
7364- *GPG_BIN:* Specifies a ``gpg`` binary/wrapper that is executed
7365 when the package is signed.
7366
7367- *GPG_PATH:* Specifies the ``gpg`` home directory used when the
7368 package is signed.
7369
7370Processing Package Feeds
7371~~~~~~~~~~~~~~~~~~~~~~~~
7372
7373In addition to being able to sign RPM packages, you can also enable
7374signed package feeds for IPK and RPM packages.
7375
7376The steps you need to take to enable signed package feed use are similar
7377to the steps used to sign RPM packages. You must define the following in
Andrew Geisslerc926e172021-05-07 16:11:35 -05007378your ``local.config`` or ``distro.config`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007379
7380 INHERIT += "sign_package_feed"
7381 PACKAGE_FEED_GPG_NAME = "key_name"
7382 PACKAGE_FEED_GPG_PASSPHRASE_FILE = "path_to_file_containing_passphrase"
7383
William A. Kennington IIIac69b482021-06-02 12:28:27 -07007384For signed package feeds, the passphrase must be specified in a separate file,
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007385which is pointed to by the ``PACKAGE_FEED_GPG_PASSPHRASE_FILE``
7386variable. Regarding security, keeping a plain text passphrase out of the
7387configuration is more secure.
7388
7389Aside from the ``PACKAGE_FEED_GPG_NAME`` and
7390``PACKAGE_FEED_GPG_PASSPHRASE_FILE`` variables, three optional variables
William A. Kennington IIIac69b482021-06-02 12:28:27 -07007391related to signed package feeds are available:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007392
7393- *GPG_BIN* Specifies a ``gpg`` binary/wrapper that is executed
7394 when the package is signed.
7395
7396- *GPG_PATH:* Specifies the ``gpg`` home directory used when the
7397 package is signed.
7398
7399- *PACKAGE_FEED_GPG_SIGNATURE_TYPE:* Specifies the type of ``gpg``
7400 signature. This variable applies only to RPM and IPK package feeds.
7401 Allowable values for the ``PACKAGE_FEED_GPG_SIGNATURE_TYPE`` are
7402 "ASC", which is the default and specifies ascii armored, and "BIN",
7403 which specifies binary.
7404
7405Testing Packages With ptest
7406---------------------------
7407
7408A Package Test (ptest) runs tests against packages built by the
7409OpenEmbedded build system on the target machine. A ptest contains at
7410least two items: the actual test, and a shell script (``run-ptest``)
7411that starts the test. The shell script that starts the test must not
Andrew Geissler615f2f12022-07-15 14:00:58 -05007412contain the actual test --- the script only starts the test. On the other
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007413hand, the test can be anything from a simple shell script that runs a
7414binary and checks the output to an elaborate system of test binaries and
7415data files.
7416
Andrew Geisslerc926e172021-05-07 16:11:35 -05007417The test generates output in the format used by Automake::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007418
7419 result: testname
7420
7421where the result can be ``PASS``, ``FAIL``, or ``SKIP``, and
7422the testname can be any identifying string.
7423
7424For a list of Yocto Project recipes that are already enabled with ptest,
Andrew Geissler09209ee2020-12-13 08:44:15 -06007425see the :yocto_wiki:`Ptest </Ptest>` wiki page.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007426
7427.. note::
7428
7429 A recipe is "ptest-enabled" if it inherits the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007430 :ref:`ptest <ref-classes-ptest>` class.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007431
7432Adding ptest to Your Build
7433~~~~~~~~~~~~~~~~~~~~~~~~~~
7434
7435To add package testing to your build, add the
7436:term:`DISTRO_FEATURES` and
7437:term:`EXTRA_IMAGE_FEATURES`
7438variables to your ``local.conf`` file, which is found in the
Andrew Geisslerc926e172021-05-07 16:11:35 -05007439:term:`Build Directory`::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007440
Patrick Williams0ca19cc2021-08-16 14:03:13 -05007441 DISTRO_FEATURES:append = " ptest"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007442 EXTRA_IMAGE_FEATURES += "ptest-pkgs"
7443
7444Once your build is complete, the ptest files are installed into the
7445``/usr/lib/package/ptest`` directory within the image, where ``package``
7446is the name of the package.
7447
7448Running ptest
7449~~~~~~~~~~~~~
7450
7451The ``ptest-runner`` package installs a shell script that loops through
7452all installed ptest test suites and runs them in sequence. Consequently,
7453you might want to add this package to your image.
7454
7455Getting Your Package Ready
7456~~~~~~~~~~~~~~~~~~~~~~~~~~
7457
7458In order to enable a recipe to run installed ptests on target hardware,
7459you need to prepare the recipes that build the packages you want to
7460test. Here is what you have to do for each recipe:
7461
7462- *Be sure the recipe inherits
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007463 the* :ref:`ptest <ref-classes-ptest>` *class:*
Andrew Geisslerc926e172021-05-07 16:11:35 -05007464 Include the following line in each recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007465
7466 inherit ptest
7467
7468- *Create run-ptest:* This script starts your test. Locate the
7469 script where you will refer to it using
7470 :term:`SRC_URI`. Here is an
Andrew Geisslerc926e172021-05-07 16:11:35 -05007471 example that starts a test for ``dbus``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007472
7473 #!/bin/sh
7474 cd test
7475 make -k runtest-TESTS
7476
7477- *Ensure dependencies are met:* If the test adds build or runtime
7478 dependencies that normally do not exist for the package (such as
7479 requiring "make" to run the test suite), use the
7480 :term:`DEPENDS` and
7481 :term:`RDEPENDS` variables in
7482 your recipe in order for the package to meet the dependencies. Here
Andrew Geisslerc926e172021-05-07 16:11:35 -05007483 is an example where the package has a runtime dependency on "make"::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007484
Patrick Williams0ca19cc2021-08-16 14:03:13 -05007485 RDEPENDS:${PN}-ptest += "make"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007486
7487- *Add a function to build the test suite:* Not many packages support
7488 cross-compilation of their test suites. Consequently, you usually
7489 need to add a cross-compilation function to the package.
7490
7491 Many packages based on Automake compile and run the test suite by
7492 using a single command such as ``make check``. However, the host
7493 ``make check`` builds and runs on the same computer, while
7494 cross-compiling requires that the package is built on the host but
7495 executed for the target architecture (though often, as in the case
7496 for ptest, the execution occurs on the host). The built version of
7497 Automake that ships with the Yocto Project includes a patch that
7498 separates building and execution. Consequently, packages that use the
7499 unaltered, patched version of ``make check`` automatically
7500 cross-compiles.
7501
7502 Regardless, you still must add a ``do_compile_ptest`` function to
7503 build the test suite. Add a function similar to the following to your
Andrew Geisslerc926e172021-05-07 16:11:35 -05007504 recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007505
7506 do_compile_ptest() {
7507 oe_runmake buildtest-TESTS
7508 }
7509
7510- *Ensure special configurations are set:* If the package requires
7511 special configurations prior to compiling the test code, you must
7512 insert a ``do_configure_ptest`` function into the recipe.
7513
7514- *Install the test suite:* The ``ptest`` class automatically copies
7515 the file ``run-ptest`` to the target and then runs make
7516 ``install-ptest`` to run the tests. If this is not enough, you need
7517 to create a ``do_install_ptest`` function and make sure it gets
7518 called after the "make install-ptest" completes.
7519
7520Creating Node Package Manager (NPM) Packages
7521--------------------------------------------
7522
7523`NPM <https://en.wikipedia.org/wiki/Npm_(software)>`__ is a package
7524manager for the JavaScript programming language. The Yocto Project
Andrew Geissler09209ee2020-12-13 08:44:15 -06007525supports the NPM :ref:`fetcher <bitbake:bitbake-user-manual/bitbake-user-manual-fetching:fetchers>`. You can
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007526use this fetcher in combination with
Andrew Geissler09209ee2020-12-13 08:44:15 -06007527:doc:`devtool </ref-manual/devtool-reference>` to create
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007528recipes that produce NPM packages.
7529
William A. Kennington IIIac69b482021-06-02 12:28:27 -07007530There are two workflows that allow you to create NPM packages using
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007531``devtool``: the NPM registry modules method and the NPM project code
7532method.
7533
7534.. note::
7535
7536 While it is possible to create NPM recipes manually, using
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007537 ``devtool`` is far simpler.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007538
7539Additionally, some requirements and caveats exist.
7540
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007541Requirements and Caveats
7542~~~~~~~~~~~~~~~~~~~~~~~~
7543
7544You need to be aware of the following before using ``devtool`` to create
7545NPM packages:
7546
7547- Of the two methods that you can use ``devtool`` to create NPM
7548 packages, the registry approach is slightly simpler. However, you
7549 might consider the project approach because you do not have to
7550 publish your module in the NPM registry
7551 (`npm-registry <https://docs.npmjs.com/misc/registry>`_), which
7552 is NPM's public registry.
7553
7554- Be familiar with
Andrew Geissler09209ee2020-12-13 08:44:15 -06007555 :doc:`devtool </ref-manual/devtool-reference>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007556
7557- The NPM host tools need the native ``nodejs-npm`` package, which is
7558 part of the OpenEmbedded environment. You need to get the package by
7559 cloning the https://github.com/openembedded/meta-openembedded
7560 repository out of GitHub. Be sure to add the path to your local copy
7561 to your ``bblayers.conf`` file.
7562
7563- ``devtool`` cannot detect native libraries in module dependencies.
7564 Consequently, you must manually add packages to your recipe.
7565
7566- While deploying NPM packages, ``devtool`` cannot determine which
7567 dependent packages are missing on the target (e.g. the node runtime
7568 ``nodejs``). Consequently, you need to find out what files are
7569 missing and be sure they are on the target.
7570
7571- Although you might not need NPM to run your node package, it is
7572 useful to have NPM on your target. The NPM package name is
7573 ``nodejs-npm``.
7574
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007575Using the Registry Modules Method
7576~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
7577
7578This section presents an example that uses the ``cute-files`` module,
7579which is a file browser web application.
7580
7581.. note::
7582
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007583 You must know the ``cute-files`` module version.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007584
7585The first thing you need to do is use ``devtool`` and the NPM fetcher to
Andrew Geisslerc926e172021-05-07 16:11:35 -05007586create the recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007587
7588 $ devtool add "npm://registry.npmjs.org;package=cute-files;version=1.0.2"
7589
7590The
7591``devtool add`` command runs ``recipetool create`` and uses the same
7592fetch URI to download each dependency and capture license details where
7593possible. The result is a generated recipe.
7594
Andrew Geissler615f2f12022-07-15 14:00:58 -05007595After running for quite a long time, in particular building the
7596``nodejs-native`` package, the command should end as follows::
7597
7598 INFO: Recipe /home/.../build/workspace/recipes/cute-files/cute-files_1.0.2.bb has been automatically created; further editing may be required to make it fully functional
7599
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007600The recipe file is fairly simple and contains every license that
7601``recipetool`` finds and includes the licenses in the recipe's
7602:term:`LIC_FILES_CHKSUM`
7603variables. You need to examine the variables and look for those with
7604"unknown" in the :term:`LICENSE`
7605field. You need to track down the license information for "unknown"
7606modules and manually add the information to the recipe.
7607
7608``recipetool`` creates a "shrinkwrap" file for your recipe. Shrinkwrap
7609files capture the version of all dependent modules. Many packages do not
Andrew Geissler615f2f12022-07-15 14:00:58 -05007610provide shrinkwrap files but ``recipetool`` will create a shrinkwrap file as it
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007611runs.
7612
7613.. note::
7614
7615 A package is created for each sub-module. This policy is the only
7616 practical way to have the licenses for all of the dependencies
7617 represented in the license manifest of the image.
7618
Andrew Geisslerc926e172021-05-07 16:11:35 -05007619The ``devtool edit-recipe`` command lets you take a look at the recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007620
7621 $ devtool edit-recipe cute-files
Andrew Geissler615f2f12022-07-15 14:00:58 -05007622 # Recipe created by recipetool
7623 # This is the basis of a recipe and may need further editing in order to be fully functional.
7624 # (Feel free to remove these comments when editing.)
7625
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007626 SUMMARY = "Turn any folder on your computer into a cute file browser, available on the local network."
Andrew Geissler615f2f12022-07-15 14:00:58 -05007627 # WARNING: the following LICENSE and LIC_FILES_CHKSUM values are best guesses - it is
7628 # your responsibility to verify that the values are complete and correct.
7629 #
7630 # NOTE: multiple licenses have been detected; they have been separated with &
7631 # in the LICENSE value for now since it is a reasonable assumption that all
7632 # of the licenses apply. If instead there is a choice between the multiple
7633 # licenses then you should change the value to separate the licenses with |
7634 # instead of &. If there is any doubt, check the accompanying documentation
7635 # to determine which situation is applicable.
7636
7637 SUMMARY = "Turn any folder on your computer into a cute file browser, available on the local network."
7638 LICENSE = "BSD-3-Clause & ISC & MIT"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007639 LIC_FILES_CHKSUM = "file://LICENSE;md5=71d98c0a1db42956787b1909c74a86ca \
Andrew Geissler615f2f12022-07-15 14:00:58 -05007640 file://node_modules/accepts/LICENSE;md5=bf1f9ad1e2e1d507aef4883fff7103de \
7641 file://node_modules/array-flatten/LICENSE;md5=44088ba57cb871a58add36ce51b8de08 \
7642 ...
7643 file://node_modules/cookie-signature/Readme.md;md5=57ae8b42de3dd0c1f22d5f4cf191e15a"
7644
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007645 SRC_URI = " \
7646 npm://registry.npmjs.org/;package=cute-files;version=${PV} \
7647 npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \
7648 "
Andrew Geissler615f2f12022-07-15 14:00:58 -05007649
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007650 S = "${WORKDIR}/npm"
Andrew Geissler615f2f12022-07-15 14:00:58 -05007651
Patrick Williams213cb262021-08-07 19:21:33 -05007652 inherit npm
Andrew Geissler615f2f12022-07-15 14:00:58 -05007653
Patrick Williams0ca19cc2021-08-16 14:03:13 -05007654 LICENSE:${PN} = "MIT"
7655 LICENSE:${PN}-accepts = "MIT"
7656 LICENSE:${PN}-array-flatten = "MIT"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007657 ...
Patrick Williams0ca19cc2021-08-16 14:03:13 -05007658 LICENSE:${PN}-vary = "MIT"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007659
William A. Kennington IIIac69b482021-06-02 12:28:27 -07007660Here are three key points in the previous example:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007661
7662- :term:`SRC_URI` uses the NPM
7663 scheme so that the NPM fetcher is used.
7664
7665- ``recipetool`` collects all the license information. If a
7666 sub-module's license is unavailable, the sub-module's name appears in
7667 the comments.
7668
7669- The ``inherit npm`` statement causes the
7670 :ref:`npm <ref-classes-npm>` class to package
7671 up all the modules.
7672
Andrew Geisslerc926e172021-05-07 16:11:35 -05007673You can run the following command to build the ``cute-files`` package::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007674
7675 $ devtool build cute-files
7676
7677Remember that ``nodejs`` must be installed on
7678the target before your package.
7679
7680Assuming 192.168.7.2 for the target's IP address, use the following
Andrew Geisslerc926e172021-05-07 16:11:35 -05007681command to deploy your package::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007682
7683 $ devtool deploy-target -s cute-files root@192.168.7.2
7684
7685Once the package is installed on the target, you can
Andrew Geissler615f2f12022-07-15 14:00:58 -05007686test the application to show the contents of any directory::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007687
7688 $ cd /usr/lib/node_modules/cute-files
Andrew Geissler615f2f12022-07-15 14:00:58 -05007689 $ cute-files
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007690
7691On a browser,
7692go to ``http://192.168.7.2:3000`` and you see the following:
7693
7694.. image:: figures/cute-files-npm-example.png
Andrew Geisslerd5838332022-05-27 11:33:10 -05007695 :width: 100%
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007696
7697You can find the recipe in ``workspace/recipes/cute-files``. You can use
7698the recipe in any layer you choose.
7699
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007700Using the NPM Projects Code Method
7701~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
7702
7703Although it is useful to package modules already in the NPM registry,
7704adding ``node.js`` projects under development is a more common developer
7705use case.
7706
7707This section covers the NPM projects code method, which is very similar
7708to the "registry" approach described in the previous section. In the NPM
7709projects method, you provide ``devtool`` with an URL that points to the
7710source files.
7711
7712Replicating the same example, (i.e. ``cute-files``) use the following
Andrew Geisslerc926e172021-05-07 16:11:35 -05007713command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007714
7715 $ devtool add https://github.com/martinaglv/cute-files.git
7716
Andrew Geissler615f2f12022-07-15 14:00:58 -05007717The recipe this command generates is very similar to the recipe created in
Andrew Geissler09036742021-06-25 14:25:14 -05007718the previous section. However, the :term:`SRC_URI` looks like the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007719
7720 SRC_URI = " \
Andrew Geissler615f2f12022-07-15 14:00:58 -05007721 git://github.com/martinaglv/cute-files.git;protocol=https;branch=master \
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007722 npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \
7723 "
7724
7725In this example,
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007726the main module is taken from the Git repository and dependencies are
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007727taken from the NPM registry. Other than those differences, the recipe is
7728basically the same between the two methods. You can build and deploy the
7729package exactly as described in the previous section that uses the
7730registry modules method.
7731
7732Adding custom metadata to packages
7733----------------------------------
7734
7735The variable
7736:term:`PACKAGE_ADD_METADATA`
7737can be used to add additional metadata to packages. This is reflected in
7738the package control/spec file. To take the ipk format for example, the
7739CONTROL file stored inside would contain the additional metadata as
7740additional lines.
7741
7742The variable can be used in multiple ways, including using suffixes to
7743set it for a specific package type and/or package. Note that the order
7744of precedence is the same as this list:
7745
Patrick Williams0ca19cc2021-08-16 14:03:13 -05007746- ``PACKAGE_ADD_METADATA_<PKGTYPE>:<PN>``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007747
7748- ``PACKAGE_ADD_METADATA_<PKGTYPE>``
7749
Patrick Williams0ca19cc2021-08-16 14:03:13 -05007750- ``PACKAGE_ADD_METADATA:<PN>``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007751
Andrew Geissler09036742021-06-25 14:25:14 -05007752- :term:`PACKAGE_ADD_METADATA`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007753
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007754`<PKGTYPE>` is a parameter and expected to be a distinct name of specific
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007755package type:
7756
7757- IPK for .ipk packages
7758
7759- DEB for .deb packages
7760
7761- RPM for .rpm packages
7762
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007763`<PN>` is a parameter and expected to be a package name.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007764
7765The variable can contain multiple [one-line] metadata fields separated
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007766by the literal sequence '\\n'. The separator can be redefined using the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007767variable flag ``separator``.
7768
William A. Kennington IIIac69b482021-06-02 12:28:27 -07007769Here is an example that adds two custom fields for ipk
Andrew Geisslerc926e172021-05-07 16:11:35 -05007770packages::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007771
7772 PACKAGE_ADD_METADATA_IPK = "Vendor: CustomIpk\nGroup:Applications/Spreadsheets"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007773
7774Efficiently Fetching Source Files During a Build
7775================================================
7776
7777The OpenEmbedded build system works with source files located through
7778the :term:`SRC_URI` variable. When
7779you build something using BitBake, a big part of the operation is
7780locating and downloading all the source tarballs. For images,
7781downloading all the source for various packages can take a significant
7782amount of time.
7783
7784This section shows you how you can use mirrors to speed up fetching
7785source files and how you can pre-fetch files all of which leads to more
7786efficient use of resources and time.
7787
7788Setting up Effective Mirrors
7789----------------------------
7790
7791A good deal that goes into a Yocto Project build is simply downloading
7792all of the source tarballs. Maybe you have been working with another
Andrew Geissler7e0e3c02022-02-25 20:34:39 +00007793build system for which you have built up a
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007794sizable directory of source tarballs. Or, perhaps someone else has such
7795a directory for which you have read access. If so, you can save time by
7796adding statements to your configuration file so that the build process
7797checks local directories first for existing tarballs before checking the
7798Internet.
7799
Andrew Geisslerc926e172021-05-07 16:11:35 -05007800Here is an efficient way to set it up in your ``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007801
7802 SOURCE_MIRROR_URL ?= "file:///home/you/your-download-dir/"
7803 INHERIT += "own-mirrors"
7804 BB_GENERATE_MIRROR_TARBALLS = "1"
7805 # BB_NO_NETWORK = "1"
7806
7807In the previous example, the
7808:term:`BB_GENERATE_MIRROR_TARBALLS`
7809variable causes the OpenEmbedded build system to generate tarballs of
7810the Git repositories and store them in the
7811:term:`DL_DIR` directory. Due to
7812performance reasons, generating and storing these tarballs is not the
7813build system's default behavior.
7814
7815You can also use the
7816:term:`PREMIRRORS` variable. For
7817an example, see the variable's glossary entry in the Yocto Project
7818Reference Manual.
7819
7820Getting Source Files and Suppressing the Build
7821----------------------------------------------
7822
7823Another technique you can use to ready yourself for a successive string
7824of build operations, is to pre-fetch all the source files without
7825actually starting a build. This technique lets you work through any
7826download issues and ultimately gathers all the source files into your
7827download directory :ref:`structure-build-downloads`,
7828which is located with :term:`DL_DIR`.
7829
7830Use the following BitBake command form to fetch all the necessary
Andrew Geisslerc926e172021-05-07 16:11:35 -05007831sources without starting the build::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007832
7833 $ bitbake target --runall=fetch
7834
7835This
7836variation of the BitBake command guarantees that you have all the
7837sources for that BitBake target should you disconnect from the Internet
7838and want to do the build later offline.
7839
7840Selecting an Initialization Manager
7841===================================
7842
7843By default, the Yocto Project uses SysVinit as the initialization
William A. Kennington IIIac69b482021-06-02 12:28:27 -07007844manager. However, there is also support for systemd, which is a full
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007845replacement for init with parallel starting of services, reduced shell
7846overhead and other features that are used by many distributions.
7847
7848Within the system, SysVinit treats system components as services. These
7849services are maintained as shell scripts stored in the ``/etc/init.d/``
7850directory. Services organize into different run levels. This
7851organization is maintained by putting links to the services in the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007852``/etc/rcN.d/`` directories, where `N/` is one of the following options:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007853"S", "0", "1", "2", "3", "4", "5", or "6".
7854
7855.. note::
7856
7857 Each runlevel has a dependency on the previous runlevel. This
7858 dependency allows the services to work properly.
7859
7860In comparison, systemd treats components as units. Using units is a
7861broader concept as compared to using a service. A unit includes several
7862different types of entities. Service is one of the types of entities.
7863The runlevel concept in SysVinit corresponds to the concept of a target
7864in systemd, where target is also a type of supported unit.
7865
7866In a SysVinit-based system, services load sequentially (i.e. one by one)
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007867during init and parallelization is not supported. With systemd, services
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007868start in parallel. Needless to say, the method can have an impact on
7869system startup performance.
7870
7871If you want to use SysVinit, you do not have to do anything. But, if you
7872want to use systemd, you must take some steps as described in the
7873following sections.
7874
7875Using systemd Exclusively
7876-------------------------
7877
Andrew Geisslerc926e172021-05-07 16:11:35 -05007878Set these variables in your distribution configuration file as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007879
Patrick Williams0ca19cc2021-08-16 14:03:13 -05007880 DISTRO_FEATURES:append = " systemd"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007881 VIRTUAL-RUNTIME_init_manager = "systemd"
7882
7883You can also prevent the SysVinit distribution feature from
Andrew Geisslerc926e172021-05-07 16:11:35 -05007884being automatically enabled as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007885
7886 DISTRO_FEATURES_BACKFILL_CONSIDERED = "sysvinit"
7887
7888Doing so removes any
7889redundant SysVinit scripts.
7890
7891To remove initscripts from your image altogether, set this variable
Andrew Geisslerc926e172021-05-07 16:11:35 -05007892also::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007893
7894 VIRTUAL-RUNTIME_initscripts = ""
7895
7896For information on the backfill variable, see
7897:term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`.
7898
7899Using systemd for the Main Image and Using SysVinit for the Rescue Image
7900------------------------------------------------------------------------
7901
Andrew Geisslerc926e172021-05-07 16:11:35 -05007902Set these variables in your distribution configuration file as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007903
Patrick Williams0ca19cc2021-08-16 14:03:13 -05007904 DISTRO_FEATURES:append = " systemd"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007905 VIRTUAL-RUNTIME_init_manager = "systemd"
7906
7907Doing so causes your main image to use the
7908``packagegroup-core-boot.bb`` recipe and systemd. The rescue/minimal
7909image cannot use this package group. However, it can install SysVinit
7910and the appropriate packages will have support for both systemd and
7911SysVinit.
7912
Andrew Geissler9aee5002022-03-30 16:27:02 +00007913Using systemd-journald without a traditional syslog daemon
7914----------------------------------------------------------
7915
7916Counter-intuitively, ``systemd-journald`` is not a syslog runtime or provider,
7917and the proper way to use systemd-journald as your sole logging mechanism is to
7918effectively disable syslog entirely by setting these variables in your distribution
7919configuration file::
7920
7921 VIRTUAL-RUNTIME_syslog = ""
7922 VIRTUAL-RUNTIME_base-utils-syslog = ""
7923
7924Doing so will prevent ``rsyslog`` / ``busybox-syslog`` from being pulled in by
7925default, leaving only ``journald``.
7926
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007927Selecting a Device Manager
7928==========================
7929
7930The Yocto Project provides multiple ways to manage the device manager
7931(``/dev``):
7932
Andrew Geissler5199d832021-09-24 16:47:35 -05007933- Persistent and Pre-Populated ``/dev``: For this case, the ``/dev``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007934 directory is persistent and the required device nodes are created
7935 during the build.
7936
7937- Use ``devtmpfs`` with a Device Manager: For this case, the ``/dev``
7938 directory is provided by the kernel as an in-memory file system and
7939 is automatically populated by the kernel at runtime. Additional
7940 configuration of device nodes is done in user space by a device
7941 manager like ``udev`` or ``busybox-mdev``.
7942
Andrew Geissler5199d832021-09-24 16:47:35 -05007943Using Persistent and Pre-Populated ``/dev``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007944--------------------------------------------
7945
7946To use the static method for device population, you need to set the
7947:term:`USE_DEVFS` variable to "0"
Andrew Geisslerc926e172021-05-07 16:11:35 -05007948as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007949
7950 USE_DEVFS = "0"
7951
7952The content of the resulting ``/dev`` directory is defined in a Device
7953Table file. The
7954:term:`IMAGE_DEVICE_TABLES`
7955variable defines the Device Table to use and should be set in the
7956machine or distro configuration file. Alternatively, you can set this
7957variable in your ``local.conf`` configuration file.
7958
Andrew Geissler09036742021-06-25 14:25:14 -05007959If you do not define the :term:`IMAGE_DEVICE_TABLES` variable, the default
Andrew Geisslerc926e172021-05-07 16:11:35 -05007960``device_table-minimal.txt`` is used::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007961
7962 IMAGE_DEVICE_TABLES = "device_table-mymachine.txt"
7963
7964The population is handled by the ``makedevs`` utility during image
7965creation:
7966
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007967Using ``devtmpfs`` and a Device Manager
7968---------------------------------------
7969
7970To use the dynamic method for device population, you need to use (or be
7971sure to set) the :term:`USE_DEVFS`
Andrew Geisslerc926e172021-05-07 16:11:35 -05007972variable to "1", which is the default::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007973
7974 USE_DEVFS = "1"
7975
7976With this
7977setting, the resulting ``/dev`` directory is populated by the kernel
7978using ``devtmpfs``. Make sure the corresponding kernel configuration
7979variable ``CONFIG_DEVTMPFS`` is set when building you build a Linux
7980kernel.
7981
7982All devices created by ``devtmpfs`` will be owned by ``root`` and have
7983permissions ``0600``.
7984
7985To have more control over the device nodes, you can use a device manager
7986like ``udev`` or ``busybox-mdev``. You choose the device manager by
7987defining the ``VIRTUAL-RUNTIME_dev_manager`` variable in your machine or
7988distro configuration file. Alternatively, you can set this variable in
Andrew Geisslerc926e172021-05-07 16:11:35 -05007989your ``local.conf`` configuration file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007990
7991 VIRTUAL-RUNTIME_dev_manager = "udev"
7992
7993 # Some alternative values
7994 # VIRTUAL-RUNTIME_dev_manager = "busybox-mdev"
7995 # VIRTUAL-RUNTIME_dev_manager = "systemd"
7996
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007997Using an External SCM
7998=====================
7999
8000If you're working on a recipe that pulls from an external Source Code
8001Manager (SCM), it is possible to have the OpenEmbedded build system
8002notice new recipe changes added to the SCM and then build the resulting
8003packages that depend on the new recipes by using the latest versions.
8004This only works for SCMs from which it is possible to get a sensible
8005revision number for changes. Currently, you can do this with Apache
8006Subversion (SVN), Git, and Bazaar (BZR) repositories.
8007
8008To enable this behavior, the :term:`PV` of
8009the recipe needs to reference
Andrew Geisslerc926e172021-05-07 16:11:35 -05008010:term:`SRCPV`. Here is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008011
8012 PV = "1.2.3+git${SRCPV}"
8013
8014Then, you can add the following to your
Andrew Geisslerc926e172021-05-07 16:11:35 -05008015``local.conf``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008016
Patrick Williams0ca19cc2021-08-16 14:03:13 -05008017 SRCREV:pn-PN = "${AUTOREV}"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008018
8019:term:`PN` is the name of the recipe for
8020which you want to enable automatic source revision updating.
8021
8022If you do not want to update your local configuration file, you can add
Andrew Geisslerc926e172021-05-07 16:11:35 -05008023the following directly to the recipe to finish enabling the feature::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008024
8025 SRCREV = "${AUTOREV}"
8026
8027The Yocto Project provides a distribution named ``poky-bleeding``, whose
Andrew Geisslerc926e172021-05-07 16:11:35 -05008028configuration file contains the line::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008029
8030 require conf/distro/include/poky-floating-revisions.inc
8031
8032This line pulls in the
Andrew Geisslerc926e172021-05-07 16:11:35 -05008033listed include file that contains numerous lines of exactly that form::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008034
Patrick Williams0ca19cc2021-08-16 14:03:13 -05008035 #SRCREV:pn-opkg-native ?= "${AUTOREV}"
8036 #SRCREV:pn-opkg-sdk ?= "${AUTOREV}"
8037 #SRCREV:pn-opkg ?= "${AUTOREV}"
8038 #SRCREV:pn-opkg-utils-native ?= "${AUTOREV}"
8039 #SRCREV:pn-opkg-utils ?= "${AUTOREV}"
8040 SRCREV:pn-gconf-dbus ?= "${AUTOREV}"
8041 SRCREV:pn-matchbox-common ?= "${AUTOREV}"
8042 SRCREV:pn-matchbox-config-gtk ?= "${AUTOREV}"
8043 SRCREV:pn-matchbox-desktop ?= "${AUTOREV}"
8044 SRCREV:pn-matchbox-keyboard ?= "${AUTOREV}"
8045 SRCREV:pn-matchbox-panel-2 ?= "${AUTOREV}"
8046 SRCREV:pn-matchbox-themes-extra ?= "${AUTOREV}"
8047 SRCREV:pn-matchbox-terminal ?= "${AUTOREV}"
8048 SRCREV:pn-matchbox-wm ?= "${AUTOREV}"
8049 SRCREV:pn-settings-daemon ?= "${AUTOREV}"
8050 SRCREV:pn-screenshot ?= "${AUTOREV}"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008051 . . .
8052
8053These lines allow you to
8054experiment with building a distribution that tracks the latest
8055development source for numerous packages.
8056
8057.. note::
8058
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008059 The ``poky-bleeding`` distribution is not tested on a regular basis. Keep
8060 this in mind if you use it.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008061
8062Creating a Read-Only Root Filesystem
8063====================================
8064
8065Suppose, for security reasons, you need to disable your target device's
8066root filesystem's write permissions (i.e. you need a read-only root
8067filesystem). Or, perhaps you are running the device's operating system
8068from a read-only storage device. For either case, you can customize your
8069image for that behavior.
8070
8071.. note::
8072
8073 Supporting a read-only root filesystem requires that the system and
8074 applications do not try to write to the root filesystem. You must
8075 configure all parts of the target system to write elsewhere, or to
8076 gracefully fail in the event of attempting to write to the root
8077 filesystem.
8078
8079Creating the Root Filesystem
8080----------------------------
8081
8082To create the read-only root filesystem, simply add the
8083"read-only-rootfs" feature to your image, normally in one of two ways.
8084The first way is to add the "read-only-rootfs" image feature in the
Andrew Geissler09036742021-06-25 14:25:14 -05008085image's recipe file via the :term:`IMAGE_FEATURES` variable::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008086
8087 IMAGE_FEATURES += "read-only-rootfs"
8088
8089As an alternative, you can add the same feature
8090from within your build directory's ``local.conf`` file with the
Andrew Geissler09036742021-06-25 14:25:14 -05008091associated :term:`EXTRA_IMAGE_FEATURES` variable, as in::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008092
8093 EXTRA_IMAGE_FEATURES = "read-only-rootfs"
8094
8095For more information on how to use these variables, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06008096":ref:`dev-manual/common-tasks:Customizing Images Using Custom \`\`IMAGE_FEATURES\`\` and \`\`EXTRA_IMAGE_FEATURES\`\``"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008097section. For information on the variables, see
8098:term:`IMAGE_FEATURES` and
8099:term:`EXTRA_IMAGE_FEATURES`.
8100
8101Post-Installation Scripts and Read-Only Root Filesystem
8102-------------------------------------------------------
8103
8104It is very important that you make sure all post-Installation
8105(``pkg_postinst``) scripts for packages that are installed into the
8106image can be run at the time when the root filesystem is created during
8107the build on the host system. These scripts cannot attempt to run during
Patrick Williams0ca19cc2021-08-16 14:03:13 -05008108the first boot on the target device. With the "read-only-rootfs" feature
8109enabled, the build system makes sure that all post-installation scripts
8110succeed at file system creation time. If any of these scripts
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008111still need to be run after the root filesystem is created, the build
8112immediately fails. These build-time checks ensure that the build fails
8113rather than the target device fails later during its initial boot
8114operation.
8115
8116Most of the common post-installation scripts generated by the build
8117system for the out-of-the-box Yocto Project are engineered so that they
8118can run during root filesystem creation (e.g. post-installation scripts
8119for caching fonts). However, if you create and add custom scripts, you
8120need to be sure they can be run during this file system creation.
8121
8122Here are some common problems that prevent post-installation scripts
8123from running during root filesystem creation:
8124
8125- *Not using $D in front of absolute paths:* The build system defines
8126 ``$``\ :term:`D` when the root
8127 filesystem is created. Furthermore, ``$D`` is blank when the script
8128 is run on the target device. This implies two purposes for ``$D``:
8129 ensuring paths are valid in both the host and target environments,
8130 and checking to determine which environment is being used as a method
8131 for taking appropriate actions.
8132
8133- *Attempting to run processes that are specific to or dependent on the
8134 target architecture:* You can work around these attempts by using
8135 native tools, which run on the host system, to accomplish the same
8136 tasks, or by alternatively running the processes under QEMU, which
8137 has the ``qemu_run_binary`` function. For more information, see the
8138 :ref:`qemu <ref-classes-qemu>` class.
8139
8140Areas With Write Access
8141-----------------------
8142
8143With the "read-only-rootfs" feature enabled, any attempt by the target
8144to write to the root filesystem at runtime fails. Consequently, you must
8145make sure that you configure processes and applications that attempt
8146these types of writes do so to directories with write access (e.g.
8147``/tmp`` or ``/var/run``).
8148
8149Maintaining Build Output Quality
8150================================
8151
8152Many factors can influence the quality of a build. For example, if you
8153upgrade a recipe to use a new version of an upstream software package or
8154you experiment with some new configuration options, subtle changes can
8155occur that you might not detect until later. Consider the case where
8156your recipe is using a newer version of an upstream package. In this
8157case, a new version of a piece of software might introduce an optional
8158dependency on another library, which is auto-detected. If that library
8159has already been built when the software is building, the software will
8160link to the built library and that library will be pulled into your
8161image along with the new software even if you did not want the library.
8162
8163The :ref:`buildhistory <ref-classes-buildhistory>`
William A. Kennington IIIac69b482021-06-02 12:28:27 -07008164class helps you maintain the quality of your build output. You
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008165can use the class to highlight unexpected and possibly unwanted changes
8166in the build output. When you enable build history, it records
8167information about the contents of each package and image and then
8168commits that information to a local Git repository where you can examine
8169the information.
8170
8171The remainder of this section describes the following:
8172
Andrew Geissler09209ee2020-12-13 08:44:15 -06008173- :ref:`How you can enable and disable build history <dev-manual/common-tasks:enabling and disabling build history>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008174
Andrew Geissler09209ee2020-12-13 08:44:15 -06008175- :ref:`How to understand what the build history contains <dev-manual/common-tasks:understanding what the build history contains>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008176
Andrew Geissler09209ee2020-12-13 08:44:15 -06008177- :ref:`How to limit the information used for build history <dev-manual/common-tasks:using build history to gather image information only>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008178
Andrew Geissler09209ee2020-12-13 08:44:15 -06008179- :ref:`How to examine the build history from both a command-line and web interface <dev-manual/common-tasks:examining build history information>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008180
8181Enabling and Disabling Build History
8182------------------------------------
8183
8184Build history is disabled by default. To enable it, add the following
Andrew Geissler09036742021-06-25 14:25:14 -05008185:term:`INHERIT` statement and set the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008186:term:`BUILDHISTORY_COMMIT`
8187variable to "1" at the end of your ``conf/local.conf`` file found in the
Andrew Geisslerc926e172021-05-07 16:11:35 -05008188:term:`Build Directory`::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008189
8190 INHERIT += "buildhistory"
8191 BUILDHISTORY_COMMIT = "1"
8192
8193Enabling build history as
8194previously described causes the OpenEmbedded build system to collect
8195build output information and commit it as a single commit to a local
Andrew Geissler09209ee2020-12-13 08:44:15 -06008196:ref:`overview-manual/development-environment:git` repository.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008197
8198.. note::
8199
8200 Enabling build history increases your build times slightly,
8201 particularly for images, and increases the amount of disk space used
8202 during the build.
8203
8204You can disable build history by removing the previous statements from
8205your ``conf/local.conf`` file.
8206
8207Understanding What the Build History Contains
8208---------------------------------------------
8209
8210Build history information is kept in
8211``${``\ :term:`TOPDIR`\ ``}/buildhistory``
8212in the Build Directory as defined by the
8213:term:`BUILDHISTORY_DIR`
William A. Kennington IIIac69b482021-06-02 12:28:27 -07008214variable. Here is an example abbreviated listing:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008215
8216.. image:: figures/buildhistory.png
8217 :align: center
Andrew Geisslerd5838332022-05-27 11:33:10 -05008218 :width: 50%
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008219
William A. Kennington IIIac69b482021-06-02 12:28:27 -07008220At the top level, there is a ``metadata-revs`` file that lists the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008221revisions of the repositories for the enabled layers when the build was
8222produced. The rest of the data splits into separate ``packages``,
8223``images`` and ``sdk`` directories, the contents of which are described
8224as follows.
8225
8226Build History Package Information
8227~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8228
8229The history for each package contains a text file that has name-value
8230pairs with information about the package. For example,
8231``buildhistory/packages/i586-poky-linux/busybox/busybox/latest``
8232contains the following:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008233
8234.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008235
8236 PV = 1.22.1
8237 PR = r32
8238 RPROVIDES =
8239 RDEPENDS = glibc (>= 2.20) update-alternatives-opkg
8240 RRECOMMENDS = busybox-syslog busybox-udhcpc update-rc.d
8241 PKGSIZE = 540168
8242 FILES = /usr/bin/* /usr/sbin/* /usr/lib/busybox/* /usr/lib/lib*.so.* \
8243 /etc /com /var /bin/* /sbin/* /lib/*.so.* /lib/udev/rules.d \
8244 /usr/lib/udev/rules.d /usr/share/busybox /usr/lib/busybox/* \
8245 /usr/share/pixmaps /usr/share/applications /usr/share/idl \
8246 /usr/share/omf /usr/share/sounds /usr/lib/bonobo/servers
8247 FILELIST = /bin/busybox /bin/busybox.nosuid /bin/busybox.suid /bin/sh \
8248 /etc/busybox.links.nosuid /etc/busybox.links.suid
8249
8250Most of these
8251name-value pairs correspond to variables used to produce the package.
8252The exceptions are ``FILELIST``, which is the actual list of files in
8253the package, and ``PKGSIZE``, which is the total size of files in the
8254package in bytes.
8255
William A. Kennington IIIac69b482021-06-02 12:28:27 -07008256There is also a file that corresponds to the recipe from which the package
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008257came (e.g. ``buildhistory/packages/i586-poky-linux/busybox/latest``):
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008258
8259.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008260
8261 PV = 1.22.1
8262 PR = r32
8263 DEPENDS = initscripts kern-tools-native update-rc.d-native \
8264 virtual/i586-poky-linux-compilerlibs virtual/i586-poky-linux-gcc \
8265 virtual/libc virtual/update-alternatives
8266 PACKAGES = busybox-ptest busybox-httpd busybox-udhcpd busybox-udhcpc \
8267 busybox-syslog busybox-mdev busybox-hwclock busybox-dbg \
8268 busybox-staticdev busybox-dev busybox-doc busybox-locale busybox
8269
8270Finally, for those recipes fetched from a version control system (e.g.,
William A. Kennington IIIac69b482021-06-02 12:28:27 -07008271Git), there is a file that lists source revisions that are specified in
8272the recipe and the actual revisions used during the build. Listed
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008273and actual revisions might differ when
8274:term:`SRCREV` is set to
8275${:term:`AUTOREV`}. Here is an
8276example assuming
Andrew Geisslerc926e172021-05-07 16:11:35 -05008277``buildhistory/packages/qemux86-poky-linux/linux-yocto/latest_srcrev``)::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008278
8279 # SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
8280 SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
8281 # SRCREV_meta = "a227f20eff056e511d504b2e490f3774ab260d6f"
8282 SRCREV_meta ="a227f20eff056e511d504b2e490f3774ab260d6f"
8283
8284You can use the
8285``buildhistory-collect-srcrevs`` command with the ``-a`` option to
Andrew Geissler09036742021-06-25 14:25:14 -05008286collect the stored :term:`SRCREV` values from build history and report them
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008287in a format suitable for use in global configuration (e.g.,
8288``local.conf`` or a distro include file) to override floating
Andrew Geissler09036742021-06-25 14:25:14 -05008289:term:`AUTOREV` values to a fixed set of revisions. Here is some example
Andrew Geisslerc926e172021-05-07 16:11:35 -05008290output from this command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008291
8292 $ buildhistory-collect-srcrevs -a
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008293 # all-poky-linux
Andrew Geissler9aee5002022-03-30 16:27:02 +00008294 SRCREV:pn-ca-certificates = "07de54fdcc5806bde549e1edf60738c6bccf50e8"
8295 SRCREV:pn-update-rc.d = "8636cf478d426b568c1be11dbd9346f67e03adac"
8296 # core2-64-poky-linux
8297 SRCREV:pn-binutils = "87d4632d36323091e731eb07b8aa65f90293da66"
8298 SRCREV:pn-btrfs-tools = "8ad326b2f28c044cb6ed9016d7c3285e23b673c8"
8299 SRCREV_bzip2-tests:pn-bzip2 = "f9061c030a25de5b6829e1abf373057309c734c0"
8300 SRCREV:pn-e2fsprogs = "02540dedd3ddc52c6ae8aaa8a95ce75c3f8be1c0"
8301 SRCREV:pn-file = "504206e53a89fd6eed71aeaf878aa3512418eab1"
8302 SRCREV_glibc:pn-glibc = "24962427071fa532c3c48c918e9d64d719cc8a6c"
8303 SRCREV:pn-gnome-desktop-testing = "e346cd4ed2e2102c9b195b614f3c642d23f5f6e7"
8304 SRCREV:pn-init-system-helpers = "dbd9197569c0935029acd5c9b02b84c68fd937ee"
8305 SRCREV:pn-kmod = "b6ecfc916a17eab8f93be5b09f4e4f845aabd3d1"
8306 SRCREV:pn-libnsl2 = "82245c0c58add79a8e34ab0917358217a70e5100"
8307 SRCREV:pn-libseccomp = "57357d2741a3b3d3e8425889a6b79a130e0fa2f3"
8308 SRCREV:pn-libxcrypt = "50cf2b6dd4fdf04309445f2eec8de7051d953abf"
8309 SRCREV:pn-ncurses = "51d0fd9cc3edb975f04224f29f777f8f448e8ced"
8310 SRCREV:pn-procps = "19a508ea121c0c4ac6d0224575a036de745eaaf8"
8311 SRCREV:pn-psmisc = "5fab6b7ab385080f1db725d6803136ec1841a15f"
8312 SRCREV:pn-ptest-runner = "bcb82804daa8f725b6add259dcef2067e61a75aa"
8313 SRCREV:pn-shared-mime-info = "18e558fa1c8b90b86757ade09a4ba4d6a6cf8f70"
8314 SRCREV:pn-zstd = "e47e674cd09583ff0503f0f6defd6d23d8b718d3"
8315 # qemux86_64-poky-linux
8316 SRCREV_machine:pn-linux-yocto = "20301aeb1a64164b72bc72af58802b315e025c9c"
8317 SRCREV_meta:pn-linux-yocto = "2d38a472b21ae343707c8bd64ac68a9eaca066a0"
8318 # x86_64-linux
8319 SRCREV:pn-binutils-cross-x86_64 = "87d4632d36323091e731eb07b8aa65f90293da66"
8320 SRCREV_glibc:pn-cross-localedef-native = "24962427071fa532c3c48c918e9d64d719cc8a6c"
8321 SRCREV_localedef:pn-cross-localedef-native = "794da69788cbf9bf57b59a852f9f11307663fa87"
8322 SRCREV:pn-debianutils-native = "de14223e5bffe15e374a441302c528ffc1cbed57"
8323 SRCREV:pn-libmodulemd-native = "ee80309bc766d781a144e6879419b29f444d94eb"
8324 SRCREV:pn-virglrenderer-native = "363915595e05fb252e70d6514be2f0c0b5ca312b"
8325 SRCREV:pn-zstd-native = "e47e674cd09583ff0503f0f6defd6d23d8b718d3"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008326
8327.. note::
8328
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008329 Here are some notes on using the ``buildhistory-collect-srcrevs`` command:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008330
Andrew Geissler09036742021-06-25 14:25:14 -05008331 - By default, only values where the :term:`SRCREV` was not hardcoded
Andrew Geissler5f350902021-07-23 13:09:54 -04008332 (usually when :term:`AUTOREV` is used) are reported. Use the ``-a``
8333 option to see all :term:`SRCREV` values.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008334
8335 - The output statements might not have any effect if overrides are
8336 applied elsewhere in the build system configuration. Use the
8337 ``-f`` option to add the ``forcevariable`` override to each output
8338 line if you need to work around this restriction.
8339
8340 - The script does apply special handling when building for multiple
8341 machines. However, the script does place a comment before each set
8342 of values that specifies which triplet to which they belong as
8343 previously shown (e.g., ``i586-poky-linux``).
8344
8345Build History Image Information
8346~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8347
8348The files produced for each image are as follows:
8349
8350- ``image-files:`` A directory containing selected files from the root
8351 filesystem. The files are defined by
8352 :term:`BUILDHISTORY_IMAGE_FILES`.
8353
8354- ``build-id.txt:`` Human-readable information about the build
8355 configuration and metadata source revisions. This file contains the
8356 full build header as printed by BitBake.
8357
8358- ``*.dot:`` Dependency graphs for the image that are compatible with
8359 ``graphviz``.
8360
8361- ``files-in-image.txt:`` A list of files in the image with
8362 permissions, owner, group, size, and symlink information.
8363
8364- ``image-info.txt:`` A text file containing name-value pairs with
8365 information about the image. See the following listing example for
8366 more information.
8367
8368- ``installed-package-names.txt:`` A list of installed packages by name
8369 only.
8370
8371- ``installed-package-sizes.txt:`` A list of installed packages ordered
8372 by size.
8373
8374- ``installed-packages.txt:`` A list of installed packages with full
8375 package filenames.
8376
8377.. note::
8378
8379 Installed package information is able to be gathered and produced
8380 even if package management is disabled for the final image.
8381
8382Here is an example of ``image-info.txt``:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008383
8384.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008385
8386 DISTRO = poky
Andrew Geissler9aee5002022-03-30 16:27:02 +00008387 DISTRO_VERSION = 3.4+snapshot-a0245d7be08f3d24ea1875e9f8872aa6bbff93be
8388 USER_CLASSES = buildstats
8389 IMAGE_CLASSES = qemuboot qemuboot license_image
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008390 IMAGE_FEATURES = debug-tweaks
8391 IMAGE_LINGUAS =
Andrew Geissler9aee5002022-03-30 16:27:02 +00008392 IMAGE_INSTALL = packagegroup-core-boot speex speexdsp
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008393 BAD_RECOMMENDATIONS =
8394 NO_RECOMMENDATIONS =
8395 PACKAGE_EXCLUDE =
Andrew Geissler9aee5002022-03-30 16:27:02 +00008396 ROOTFS_POSTPROCESS_COMMAND = write_package_manifest; license_create_manifest; cve_check_write_rootfs_manifest; ssh_allow_empty_password; ssh_allow_root_login; postinst_enable_logging; rootfs_update_timestamp; write_image_test_data; empty_var_volatile; sort_passwd; rootfs_reproducible;
8397 IMAGE_POSTPROCESS_COMMAND = buildhistory_get_imageinfo ;
8398 IMAGESIZE = 9265
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008399
8400Other than ``IMAGESIZE``,
8401which is the total size of the files in the image in Kbytes, the
8402name-value pairs are variables that may have influenced the content of
8403the image. This information is often useful when you are trying to
8404determine why a change in the package or file listings has occurred.
8405
8406Using Build History to Gather Image Information Only
8407~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8408
8409As you can see, build history produces image information, including
8410dependency graphs, so you can see why something was pulled into the
8411image. If you are just interested in this information and not interested
8412in collecting specific package or SDK information, you can enable
8413writing only image information without any history by adding the
8414following to your ``conf/local.conf`` file found in the
Andrew Geisslerc926e172021-05-07 16:11:35 -05008415:term:`Build Directory`::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008416
8417 INHERIT += "buildhistory"
8418 BUILDHISTORY_COMMIT = "0"
8419 BUILDHISTORY_FEATURES = "image"
8420
8421Here, you set the
8422:term:`BUILDHISTORY_FEATURES`
8423variable to use the image feature only.
8424
8425Build History SDK Information
8426~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8427
8428Build history collects similar information on the contents of SDKs (e.g.
8429``bitbake -c populate_sdk imagename``) as compared to information it
8430collects for images. Furthermore, this information differs depending on
8431whether an extensible or standard SDK is being produced.
8432
8433The following list shows the files produced for SDKs:
8434
8435- ``files-in-sdk.txt:`` A list of files in the SDK with permissions,
8436 owner, group, size, and symlink information. This list includes both
8437 the host and target parts of the SDK.
8438
8439- ``sdk-info.txt:`` A text file containing name-value pairs with
8440 information about the SDK. See the following listing example for more
8441 information.
8442
8443- ``sstate-task-sizes.txt:`` A text file containing name-value pairs
Patrick Williams2194f502022-10-16 14:26:09 -05008444 with information about task group sizes (e.g. :ref:`ref-tasks-populate_sysroot`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008445 tasks have a total size). The ``sstate-task-sizes.txt`` file exists
8446 only when an extensible SDK is created.
8447
8448- ``sstate-package-sizes.txt:`` A text file containing name-value pairs
8449 with information for the shared-state packages and sizes in the SDK.
8450 The ``sstate-package-sizes.txt`` file exists only when an extensible
8451 SDK is created.
8452
8453- ``sdk-files:`` A folder that contains copies of the files mentioned
8454 in ``BUILDHISTORY_SDK_FILES`` if the files are present in the output.
8455 Additionally, the default value of ``BUILDHISTORY_SDK_FILES`` is
8456 specific to the extensible SDK although you can set it differently if
8457 you would like to pull in specific files from the standard SDK.
8458
8459 The default files are ``conf/local.conf``, ``conf/bblayers.conf``,
8460 ``conf/auto.conf``, ``conf/locked-sigs.inc``, and
8461 ``conf/devtool.conf``. Thus, for an extensible SDK, these files get
8462 copied into the ``sdk-files`` directory.
8463
8464- The following information appears under each of the ``host`` and
8465 ``target`` directories for the portions of the SDK that run on the
8466 host and on the target, respectively:
8467
8468 .. note::
8469
8470 The following files for the most part are empty when producing an
8471 extensible SDK because this type of SDK is not constructed from
8472 packages as is the standard SDK.
8473
8474 - ``depends.dot:`` Dependency graph for the SDK that is compatible
8475 with ``graphviz``.
8476
8477 - ``installed-package-names.txt:`` A list of installed packages by
8478 name only.
8479
8480 - ``installed-package-sizes.txt:`` A list of installed packages
8481 ordered by size.
8482
8483 - ``installed-packages.txt:`` A list of installed packages with full
8484 package filenames.
8485
8486Here is an example of ``sdk-info.txt``:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008487
8488.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008489
8490 DISTRO = poky
8491 DISTRO_VERSION = 1.3+snapshot-20130327
8492 SDK_NAME = poky-glibc-i686-arm
8493 SDK_VERSION = 1.3+snapshot
8494 SDKMACHINE =
8495 SDKIMAGE_FEATURES = dev-pkgs dbg-pkgs
8496 BAD_RECOMMENDATIONS =
8497 SDKSIZE = 352712
8498
8499Other than ``SDKSIZE``, which is
8500the total size of the files in the SDK in Kbytes, the name-value pairs
8501are variables that might have influenced the content of the SDK. This
8502information is often useful when you are trying to determine why a
8503change in the package or file listings has occurred.
8504
8505Examining Build History Information
8506~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8507
8508You can examine build history output from the command line or from a web
8509interface.
8510
8511To see any changes that have occurred (assuming you have
8512:term:`BUILDHISTORY_COMMIT` = "1"),
8513you can simply use any Git command that allows you to view the history
Andrew Geisslerc926e172021-05-07 16:11:35 -05008514of a repository. Here is one method::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008515
8516 $ git log -p
8517
8518You need to realize,
8519however, that this method does show changes that are not significant
8520(e.g. a package's size changing by a few bytes).
8521
William A. Kennington IIIac69b482021-06-02 12:28:27 -07008522There is a command-line tool called ``buildhistory-diff``, though,
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008523that queries the Git repository and prints just the differences that
Andrew Geisslerc926e172021-05-07 16:11:35 -05008524might be significant in human-readable form. Here is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008525
Andrew Geissler95ac1b82021-03-31 14:34:31 -05008526 $ poky/poky/scripts/buildhistory-diff . HEAD^
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008527 Changes to images/qemux86_64/glibc/core-image-minimal (files-in-image.txt):
8528 /etc/anotherpkg.conf was added
8529 /sbin/anotherpkg was added
8530 * (installed-package-names.txt):
8531 * anotherpkg was added
8532 Changes to images/qemux86_64/glibc/core-image-minimal (installed-package-names.txt):
8533 anotherpkg was added
8534 packages/qemux86_64-poky-linux/v86d: PACKAGES: added "v86d-extras"
8535 * PR changed from "r0" to "r1"
8536 * PV changed from "0.1.10" to "0.1.12"
8537 packages/qemux86_64-poky-linux/v86d/v86d: PKGSIZE changed from 110579 to 144381 (+30%)
8538 * PR changed from "r0" to "r1"
8539 * PV changed from "0.1.10" to "0.1.12"
8540
8541.. note::
8542
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008543 The ``buildhistory-diff`` tool requires the ``GitPython``
Andrew Geisslerc926e172021-05-07 16:11:35 -05008544 package. Be sure to install it using Pip3 as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008545
8546 $ pip3 install GitPython --user
8547
8548
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008549 Alternatively, you can install ``python3-git`` using the appropriate
Andrew Geisslereff27472021-10-29 15:35:00 -05008550 distribution package manager (e.g. ``apt``, ``dnf``, or ``zipper``).
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008551
8552To see changes to the build history using a web interface, follow the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008553instruction in the ``README`` file
Andrew Geissler09209ee2020-12-13 08:44:15 -06008554:yocto_git:`here </buildhistory-web/>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008555
8556Here is a sample screenshot of the interface:
8557
8558.. image:: figures/buildhistory-web.png
Andrew Geisslerd5838332022-05-27 11:33:10 -05008559 :width: 100%
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008560
8561Performing Automated Runtime Testing
8562====================================
8563
8564The OpenEmbedded build system makes available a series of automated
8565tests for images to verify runtime functionality. You can run these
8566tests on either QEMU or actual target hardware. Tests are written in
8567Python making use of the ``unittest`` module, and the majority of them
8568run commands on the target system over SSH. This section describes how
8569you set up the environment to use these tests, run available tests, and
8570write and add your own tests.
8571
8572For information on the test and QA infrastructure available within the
Andrew Geissler09209ee2020-12-13 08:44:15 -06008573Yocto Project, see the ":ref:`ref-manual/release-process:testing and quality assurance`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008574section in the Yocto Project Reference Manual.
8575
8576Enabling Tests
8577--------------
8578
8579Depending on whether you are planning to run tests using QEMU or on the
8580hardware, you have to take different steps to enable the tests. See the
8581following subsections for information on how to enable both types of
8582tests.
8583
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008584Enabling Runtime Tests on QEMU
8585~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8586
8587In order to run tests, you need to do the following:
8588
8589- *Set up to avoid interaction with sudo for networking:* To
8590 accomplish this, you must do one of the following:
8591
8592 - Add ``NOPASSWD`` for your user in ``/etc/sudoers`` either for all
8593 commands or just for ``runqemu-ifup``. You must provide the full
8594 path as that can change if you are using multiple clones of the
8595 source repository.
8596
8597 .. note::
8598
8599 On some distributions, you also need to comment out "Defaults
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008600 requiretty" in ``/etc/sudoers``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008601
8602 - Manually configure a tap interface for your system.
8603
8604 - Run as root the script in ``scripts/runqemu-gen-tapdevs``, which
8605 should generate a list of tap devices. This is the option
8606 typically chosen for Autobuilder-type environments.
8607
8608 .. note::
8609
8610 - Be sure to use an absolute path when calling this script
8611 with sudo.
8612
8613 - The package recipe ``qemu-helper-native`` is required to run
Andrew Geisslerc926e172021-05-07 16:11:35 -05008614 this script. Build the package using the following command::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008615
8616 $ bitbake qemu-helper-native
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008617
8618- *Set the DISPLAY variable:* You need to set this variable so that
8619 you have an X server available (e.g. start ``vncserver`` for a
8620 headless machine).
8621
8622- *Be sure your host's firewall accepts incoming connections from
8623 192.168.7.0/24:* Some of the tests (in particular DNF tests) start an
8624 HTTP server on a random high number port, which is used to serve
8625 files to the target. The DNF module serves
8626 ``${WORKDIR}/oe-rootfs-repo`` so it can run DNF channel commands.
8627 That means your host's firewall must accept incoming connections from
8628 192.168.7.0/24, which is the default IP range used for tap devices by
8629 ``runqemu``.
8630
8631- *Be sure your host has the correct packages installed:* Depending
8632 your host's distribution, you need to have the following packages
8633 installed:
8634
8635 - Ubuntu and Debian: ``sysstat`` and ``iproute2``
8636
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05008637 - openSUSE: ``sysstat`` and ``iproute2``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008638
8639 - Fedora: ``sysstat`` and ``iproute``
8640
8641 - CentOS: ``sysstat`` and ``iproute``
8642
8643Once you start running the tests, the following happens:
8644
86451. A copy of the root filesystem is written to ``${WORKDIR}/testimage``.
8646
86472. The image is booted under QEMU using the standard ``runqemu`` script.
8648
86493. A default timeout of 500 seconds occurs to allow for the boot process
8650 to reach the login prompt. You can change the timeout period by
8651 setting
8652 :term:`TEST_QEMUBOOT_TIMEOUT`
8653 in the ``local.conf`` file.
8654
86554. Once the boot process is reached and the login prompt appears, the
8656 tests run. The full boot log is written to
8657 ``${WORKDIR}/testimage/qemu_boot_log``.
8658
Andrew Geissler09036742021-06-25 14:25:14 -050086595. Each test module loads in the order found in :term:`TEST_SUITES`. You can
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008660 find the full output of the commands run over SSH in
8661 ``${WORKDIR}/testimgage/ssh_target_log``.
8662
86636. If no failures occur, the task running the tests ends successfully.
8664 You can find the output from the ``unittest`` in the task log at
8665 ``${WORKDIR}/temp/log.do_testimage``.
8666
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008667Enabling Runtime Tests on Hardware
8668~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8669
8670The OpenEmbedded build system can run tests on real hardware, and for
8671certain devices it can also deploy the image to be tested onto the
8672device beforehand.
8673
Andrew Geissler595f6302022-01-24 19:11:47 +00008674For automated deployment, a "controller image" is installed onto the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008675hardware once as part of setup. Then, each time tests are to be run, the
8676following occurs:
8677
Andrew Geissler595f6302022-01-24 19:11:47 +000086781. The controller image is booted into and used to write the image to be
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008679 tested to a second partition.
8680
86812. The device is then rebooted using an external script that you need to
8682 provide.
8683
86843. The device boots into the image to be tested.
8685
8686When running tests (independent of whether the image has been deployed
8687automatically or not), the device is expected to be connected to a
8688network on a pre-determined IP address. You can either use static IP
8689addresses written into the image, or set the image to use DHCP and have
8690your DHCP server on the test network assign a known IP address based on
8691the MAC address of the device.
8692
Andrew Geissler09036742021-06-25 14:25:14 -05008693In order to run tests on hardware, you need to set :term:`TEST_TARGET` to an
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008694appropriate value. For QEMU, you do not have to change anything, the
8695default value is "qemu". For running tests on hardware, the following
William A. Kennington IIIac69b482021-06-02 12:28:27 -07008696options are available:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008697
8698- *"simpleremote":* Choose "simpleremote" if you are going to run tests
8699 on a target system that is already running the image to be tested and
8700 is available on the network. You can use "simpleremote" in
8701 conjunction with either real hardware or an image running within a
8702 separately started QEMU or any other virtual machine manager.
8703
8704- *"SystemdbootTarget":* Choose "SystemdbootTarget" if your hardware is
8705 an EFI-based machine with ``systemd-boot`` as bootloader and
8706 ``core-image-testmaster`` (or something similar) is installed. Also,
8707 your hardware under test must be in a DHCP-enabled network that gives
8708 it the same IP address for each reboot.
8709
8710 If you choose "SystemdbootTarget", there are additional requirements
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05008711 and considerations. See the
8712 ":ref:`dev-manual/common-tasks:selecting systemdboottarget`" section, which
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008713 follows, for more information.
8714
8715- *"BeagleBoneTarget":* Choose "BeagleBoneTarget" if you are deploying
8716 images and running tests on the BeagleBone "Black" or original
8717 "White" hardware. For information on how to use these tests, see the
8718 comments at the top of the BeagleBoneTarget
8719 ``meta-yocto-bsp/lib/oeqa/controllers/beaglebonetarget.py`` file.
8720
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008721- *"EdgeRouterTarget":* Choose "EdgeRouterTarget" if you are deploying
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008722 images and running tests on the Ubiquiti Networks EdgeRouter Lite.
8723 For information on how to use these tests, see the comments at the
8724 top of the EdgeRouterTarget
8725 ``meta-yocto-bsp/lib/oeqa/controllers/edgeroutertarget.py`` file.
8726
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008727- *"GrubTarget":* Choose "GrubTarget" if you are deploying images and running
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008728 tests on any generic PC that boots using GRUB. For information on how
8729 to use these tests, see the comments at the top of the GrubTarget
8730 ``meta-yocto-bsp/lib/oeqa/controllers/grubtarget.py`` file.
8731
8732- *"your-target":* Create your own custom target if you want to run
8733 tests when you are deploying images and running tests on a custom
8734 machine within your BSP layer. To do this, you need to add a Python
8735 unit that defines the target class under ``lib/oeqa/controllers/``
8736 within your layer. You must also provide an empty ``__init__.py``.
8737 For examples, see files in ``meta-yocto-bsp/lib/oeqa/controllers/``.
8738
8739Selecting SystemdbootTarget
8740~~~~~~~~~~~~~~~~~~~~~~~~~~~
8741
Andrew Geissler09036742021-06-25 14:25:14 -05008742If you did not set :term:`TEST_TARGET` to "SystemdbootTarget", then you do
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008743not need any information in this section. You can skip down to the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05008744":ref:`dev-manual/common-tasks:running tests`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008745
Andrew Geissler09036742021-06-25 14:25:14 -05008746If you did set :term:`TEST_TARGET` to "SystemdbootTarget", you also need to
Andrew Geissler595f6302022-01-24 19:11:47 +00008747perform a one-time setup of your controller image by doing the following:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008748
Andrew Geissler09036742021-06-25 14:25:14 -050087491. *Set EFI_PROVIDER:* Be sure that :term:`EFI_PROVIDER` is as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008750
8751 EFI_PROVIDER = "systemd-boot"
8752
Andrew Geissler595f6302022-01-24 19:11:47 +000087532. *Build the controller image:* Build the ``core-image-testmaster`` image.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008754 The ``core-image-testmaster`` recipe is provided as an example for a
Andrew Geissler595f6302022-01-24 19:11:47 +00008755 "controller" image and you can customize the image recipe as you would
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008756 any other recipe.
8757
8758 Here are the image recipe requirements:
8759
8760 - Inherits ``core-image`` so that kernel modules are installed.
8761
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05008762 - Installs normal linux utilities not BusyBox ones (e.g. ``bash``,
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008763 ``coreutils``, ``tar``, ``gzip``, and ``kmod``).
8764
Patrick Williams2194f502022-10-16 14:26:09 -05008765 - Uses a custom :term:`Initramfs` image with a custom
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008766 installer. A normal image that you can install usually creates a
Andrew Geissler595f6302022-01-24 19:11:47 +00008767 single root filesystem partition. This image uses another installer that
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008768 creates a specific partition layout. Not all Board Support
8769 Packages (BSPs) can use an installer. For such cases, you need to
8770 manually create the following partition layout on the target:
8771
8772 - First partition mounted under ``/boot``, labeled "boot".
8773
Andrew Geissler595f6302022-01-24 19:11:47 +00008774 - The main root filesystem partition where this image gets installed,
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008775 which is mounted under ``/``.
8776
8777 - Another partition labeled "testrootfs" where test images get
8778 deployed.
8779
87803. *Install image:* Install the image that you just built on the target
8781 system.
8782
Andrew Geissler09036742021-06-25 14:25:14 -05008783The final thing you need to do when setting :term:`TEST_TARGET` to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008784"SystemdbootTarget" is to set up the test image:
8785
87861. *Set up your local.conf file:* Make sure you have the following
Andrew Geisslerc926e172021-05-07 16:11:35 -05008787 statements in your ``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008788
8789 IMAGE_FSTYPES += "tar.gz"
8790 INHERIT += "testimage"
8791 TEST_TARGET = "SystemdbootTarget"
8792 TEST_TARGET_IP = "192.168.2.3"
8793
Andrew Geisslerc926e172021-05-07 16:11:35 -050087942. *Build your test image:* Use BitBake to build the image::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008795
8796 $ bitbake core-image-sato
8797
8798Power Control
8799~~~~~~~~~~~~~
8800
8801For most hardware targets other than "simpleremote", you can control
8802power:
8803
Andrew Geissler09036742021-06-25 14:25:14 -05008804- You can use :term:`TEST_POWERCONTROL_CMD` together with
8805 :term:`TEST_POWERCONTROL_EXTRA_ARGS` as a command that runs on the host
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008806 and does power cycling. The test code passes one argument to that
8807 command: off, on or cycle (off then on). Here is an example that
Andrew Geisslerc926e172021-05-07 16:11:35 -05008808 could appear in your ``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008809
8810 TEST_POWERCONTROL_CMD = "powercontrol.exp test 10.11.12.1 nuc1"
8811
8812 In this example, the expect
8813 script does the following:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008814
8815 .. code-block:: shell
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008816
8817 ssh test@10.11.12.1 "pyctl nuc1 arg"
8818
8819 It then runs a Python script that controls power for a label called
8820 ``nuc1``.
8821
8822 .. note::
8823
Andrew Geissler09036742021-06-25 14:25:14 -05008824 You need to customize :term:`TEST_POWERCONTROL_CMD` and
8825 :term:`TEST_POWERCONTROL_EXTRA_ARGS` for your own setup. The one requirement
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008826 is that it accepts "on", "off", and "cycle" as the last argument.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008827
8828- When no command is defined, it connects to the device over SSH and
8829 uses the classic reboot command to reboot the device. Classic reboot
8830 is fine as long as the machine actually reboots (i.e. the SSH test
8831 has not failed). It is useful for scenarios where you have a simple
8832 setup, typically with a single board, and where some manual
8833 interaction is okay from time to time.
8834
8835If you have no hardware to automatically perform power control but still
8836wish to experiment with automated hardware testing, you can use the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008837``dialog-power-control`` script that shows a dialog prompting you to perform
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008838the required power action. This script requires either KDialog or Zenity
8839to be installed. To use this script, set the
8840:term:`TEST_POWERCONTROL_CMD`
Andrew Geisslerc926e172021-05-07 16:11:35 -05008841variable as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008842
8843 TEST_POWERCONTROL_CMD = "${COREBASE}/scripts/contrib/dialog-power-control"
8844
8845Serial Console Connection
8846~~~~~~~~~~~~~~~~~~~~~~~~~
8847
8848For test target classes requiring a serial console to interact with the
8849bootloader (e.g. BeagleBoneTarget, EdgeRouterTarget, and GrubTarget),
8850you need to specify a command to use to connect to the serial console of
8851the target machine by using the
8852:term:`TEST_SERIALCONTROL_CMD`
8853variable and optionally the
8854:term:`TEST_SERIALCONTROL_EXTRA_ARGS`
8855variable.
8856
8857These cases could be a serial terminal program if the machine is
8858connected to a local serial port, or a ``telnet`` or ``ssh`` command
8859connecting to a remote console server. Regardless of the case, the
8860command simply needs to connect to the serial console and forward that
8861connection to standard input and output as any normal terminal program
8862does. For example, to use the picocom terminal program on serial device
Andrew Geisslerc926e172021-05-07 16:11:35 -05008863``/dev/ttyUSB0`` at 115200bps, you would set the variable as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008864
8865 TEST_SERIALCONTROL_CMD = "picocom /dev/ttyUSB0 -b 115200"
8866
8867For local
8868devices where the serial port device disappears when the device reboots,
8869an additional "serdevtry" wrapper script is provided. To use this
8870wrapper, simply prefix the terminal command with
Andrew Geisslerc926e172021-05-07 16:11:35 -05008871``${COREBASE}/scripts/contrib/serdevtry``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008872
8873 TEST_SERIALCONTROL_CMD = "${COREBASE}/scripts/contrib/serdevtry picocom -b 115200 /dev/ttyUSB0"
8874
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008875Running Tests
8876-------------
8877
8878You can start the tests automatically or manually:
8879
8880- *Automatically running tests:* To run the tests automatically after
8881 the OpenEmbedded build system successfully creates an image, first
8882 set the
8883 :term:`TESTIMAGE_AUTO`
8884 variable to "1" in your ``local.conf`` file in the
Andrew Geisslerc926e172021-05-07 16:11:35 -05008885 :term:`Build Directory`::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008886
8887 TESTIMAGE_AUTO = "1"
8888
8889 Next, build your image. If the image successfully builds, the
Andrew Geisslerc926e172021-05-07 16:11:35 -05008890 tests run::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008891
8892 bitbake core-image-sato
8893
8894- *Manually running tests:* To manually run the tests, first globally
8895 inherit the
Patrick Williams975a06f2022-10-21 14:42:47 -05008896 :ref:`testimage <ref-classes-testimage>` class
Andrew Geisslerc926e172021-05-07 16:11:35 -05008897 by editing your ``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008898
8899 INHERIT += "testimage"
8900
Andrew Geisslerc926e172021-05-07 16:11:35 -05008901 Next, use BitBake to run the tests::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008902
8903 bitbake -c testimage image
8904
8905All test files reside in ``meta/lib/oeqa/runtime`` in the
8906:term:`Source Directory`. A test name maps
8907directly to a Python module. Each test module may contain a number of
8908individual tests. Tests are usually grouped together by the area tested
8909(e.g tests for systemd reside in ``meta/lib/oeqa/runtime/systemd.py``).
8910
8911You can add tests to any layer provided you place them in the proper
8912area and you extend :term:`BBPATH` in
8913the ``local.conf`` file as normal. Be sure that tests reside in
8914``layer/lib/oeqa/runtime``.
8915
8916.. note::
8917
8918 Be sure that module names do not collide with module names used in
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008919 the default set of test modules in ``meta/lib/oeqa/runtime``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008920
8921You can change the set of tests run by appending or overriding
8922:term:`TEST_SUITES` variable in
Andrew Geissler09036742021-06-25 14:25:14 -05008923``local.conf``. Each name in :term:`TEST_SUITES` represents a required test
8924for the image. Test modules named within :term:`TEST_SUITES` cannot be
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008925skipped even if a test is not suitable for an image (e.g. running the
8926RPM tests on an image without ``rpm``). Appending "auto" to
Andrew Geissler09036742021-06-25 14:25:14 -05008927:term:`TEST_SUITES` causes the build system to try to run all tests that are
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008928suitable for the image (i.e. each test module may elect to skip itself).
8929
Andrew Geissler09036742021-06-25 14:25:14 -05008930The order you list tests in :term:`TEST_SUITES` is important and influences
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008931test dependencies. Consequently, tests that depend on other tests should
8932be added after the test on which they depend. For example, since the
8933``ssh`` test depends on the ``ping`` test, "ssh" needs to come after
8934"ping" in the list. The test class provides no re-ordering or dependency
8935handling.
8936
8937.. note::
8938
8939 Each module can have multiple classes with multiple test methods.
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008940 And, Python ``unittest`` rules apply.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008941
8942Here are some things to keep in mind when running tests:
8943
Andrew Geisslerc926e172021-05-07 16:11:35 -05008944- The default tests for the image are defined as::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008945
Patrick Williams0ca19cc2021-08-16 14:03:13 -05008946 DEFAULT_TEST_SUITES:pn-image = "ping ssh df connman syslog xorg scp vnc date rpm dnf dmesg"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008947
Andrew Geisslerc926e172021-05-07 16:11:35 -05008948- Add your own test to the list of the by using the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008949
Patrick Williams0ca19cc2021-08-16 14:03:13 -05008950 TEST_SUITES:append = " mytest"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008951
Andrew Geisslerc926e172021-05-07 16:11:35 -05008952- Run a specific list of tests as follows::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008953
8954 TEST_SUITES = "test1 test2 test3"
8955
8956 Remember, order is important. Be sure to place a test that is
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008957 dependent on another test later in the order.
8958
8959Exporting Tests
8960---------------
8961
8962You can export tests so that they can run independently of the build
8963system. Exporting tests is required if you want to be able to hand the
8964test execution off to a scheduler. You can only export tests that are
8965defined in :term:`TEST_SUITES`.
8966
8967If your image is already built, make sure the following are set in your
Andrew Geisslerc926e172021-05-07 16:11:35 -05008968``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008969
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008970 INHERIT += "testexport"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008971 TEST_TARGET_IP = "IP-address-for-the-test-target"
8972 TEST_SERVER_IP = "IP-address-for-the-test-server"
8973
8974You can then export the tests with the
Andrew Geisslerc926e172021-05-07 16:11:35 -05008975following BitBake command form::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008976
8977 $ bitbake image -c testexport
8978
8979Exporting the tests places them in the
8980:term:`Build Directory` in
8981``tmp/testexport/``\ image, which is controlled by the
Andrew Geissler09036742021-06-25 14:25:14 -05008982:term:`TEST_EXPORT_DIR` variable.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008983
Andrew Geisslerc926e172021-05-07 16:11:35 -05008984You can now run the tests outside of the build environment::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008985
8986 $ cd tmp/testexport/image
8987 $ ./runexported.py testdata.json
8988
8989Here is a complete example that shows IP addresses and uses the
Andrew Geisslerc926e172021-05-07 16:11:35 -05008990``core-image-sato`` image::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008991
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008992 INHERIT += "testexport"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008993 TEST_TARGET_IP = "192.168.7.2"
8994 TEST_SERVER_IP = "192.168.7.1"
8995
Andrew Geisslerc926e172021-05-07 16:11:35 -05008996Use BitBake to export the tests::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008997
8998 $ bitbake core-image-sato -c testexport
8999
9000Run the tests outside of
Andrew Geisslerc926e172021-05-07 16:11:35 -05009001the build environment using the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009002
9003 $ cd tmp/testexport/core-image-sato
9004 $ ./runexported.py testdata.json
9005
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009006Writing New Tests
9007-----------------
9008
9009As mentioned previously, all new test files need to be in the proper
9010place for the build system to find them. New tests for additional
9011functionality outside of the core should be added to the layer that adds
9012the functionality, in ``layer/lib/oeqa/runtime`` (as long as
9013:term:`BBPATH` is extended in the
9014layer's ``layer.conf`` file as normal). Just remember the following:
9015
9016- Filenames need to map directly to test (module) names.
9017
9018- Do not use module names that collide with existing core tests.
9019
William A. Kennington IIIac69b482021-06-02 12:28:27 -07009020- Minimally, an empty ``__init__.py`` file must be present in the runtime
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009021 directory.
9022
9023To create a new test, start by copying an existing module (e.g.
9024``syslog.py`` or ``gcc.py`` are good ones to use). Test modules can use
9025code from ``meta/lib/oeqa/utils``, which are helper classes.
9026
9027.. note::
9028
9029 Structure shell commands such that you rely on them and they return a
9030 single code for success. Be aware that sometimes you will need to
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009031 parse the output. See the ``df.py`` and ``date.py`` modules for examples.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009032
9033You will notice that all test classes inherit ``oeRuntimeTest``, which
9034is found in ``meta/lib/oetest.py``. This base class offers some helper
9035attributes, which are described in the following sections:
9036
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009037Class Methods
9038~~~~~~~~~~~~~
9039
9040Class methods are as follows:
9041
9042- *hasPackage(pkg):* Returns "True" if ``pkg`` is in the installed
9043 package list of the image, which is based on the manifest file that
Patrick Williams2194f502022-10-16 14:26:09 -05009044 is generated during the :ref:`ref-tasks-rootfs` task.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009045
9046- *hasFeature(feature):* Returns "True" if the feature is in
9047 :term:`IMAGE_FEATURES` or
9048 :term:`DISTRO_FEATURES`.
9049
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009050Class Attributes
9051~~~~~~~~~~~~~~~~
9052
9053Class attributes are as follows:
9054
9055- *pscmd:* Equals "ps -ef" if ``procps`` is installed in the image.
9056 Otherwise, ``pscmd`` equals "ps" (busybox).
9057
9058- *tc:* The called test context, which gives access to the
9059 following attributes:
9060
9061 - *d:* The BitBake datastore, which allows you to use stuff such
9062 as ``oeRuntimeTest.tc.d.getVar("VIRTUAL-RUNTIME_init_manager")``.
9063
9064 - *testslist and testsrequired:* Used internally. The tests
9065 do not need these.
9066
9067 - *filesdir:* The absolute path to
9068 ``meta/lib/oeqa/runtime/files``, which contains helper files for
9069 tests meant for copying on the target such as small files written
9070 in C for compilation.
9071
9072 - *target:* The target controller object used to deploy and
9073 start an image on a particular target (e.g. Qemu, SimpleRemote,
9074 and SystemdbootTarget). Tests usually use the following:
9075
9076 - *ip:* The target's IP address.
9077
9078 - *server_ip:* The host's IP address, which is usually used
9079 by the DNF test suite.
9080
9081 - *run(cmd, timeout=None):* The single, most used method.
9082 This command is a wrapper for: ``ssh root@host "cmd"``. The
9083 command returns a tuple: (status, output), which are what their
9084 names imply - the return code of "cmd" and whatever output it
9085 produces. The optional timeout argument represents the number
9086 of seconds the test should wait for "cmd" to return. If the
9087 argument is "None", the test uses the default instance's
9088 timeout period, which is 300 seconds. If the argument is "0",
9089 the test runs until the command returns.
9090
9091 - *copy_to(localpath, remotepath):*
9092 ``scp localpath root@ip:remotepath``.
9093
9094 - *copy_from(remotepath, localpath):*
9095 ``scp root@host:remotepath localpath``.
9096
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009097Instance Attributes
9098~~~~~~~~~~~~~~~~~~~
9099
William A. Kennington IIIac69b482021-06-02 12:28:27 -07009100There is a single instance attribute, which is ``target``. The ``target``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009101instance attribute is identical to the class attribute of the same name,
9102which is described in the previous section. This attribute exists as
9103both an instance and class attribute so tests can use
9104``self.target.run(cmd)`` in instance methods instead of
9105``oeRuntimeTest.tc.target.run(cmd)``.
9106
9107Installing Packages in the DUT Without the Package Manager
9108----------------------------------------------------------
9109
9110When a test requires a package built by BitBake, it is possible to
9111install that package. Installing the package does not require a package
9112manager be installed in the device under test (DUT). It does, however,
9113require an SSH connection and the target must be using the
9114``sshcontrol`` class.
9115
9116.. note::
9117
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009118 This method uses ``scp`` to copy files from the host to the target, which
9119 causes permissions and special attributes to be lost.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009120
9121A JSON file is used to define the packages needed by a test. This file
9122must be in the same path as the file used to define the tests.
9123Furthermore, the filename must map directly to the test module name with
9124a ``.json`` extension.
9125
9126The JSON file must include an object with the test name as keys of an
9127object or an array. This object (or array of objects) uses the following
9128data:
9129
Andrew Geissler615f2f12022-07-15 14:00:58 -05009130- "pkg" --- a mandatory string that is the name of the package to be
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009131 installed.
9132
Andrew Geissler615f2f12022-07-15 14:00:58 -05009133- "rm" --- an optional boolean, which defaults to "false", that specifies
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009134 to remove the package after the test.
9135
Andrew Geissler615f2f12022-07-15 14:00:58 -05009136- "extract" --- an optional boolean, which defaults to "false", that
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009137 specifies if the package must be extracted from the package format.
9138 When set to "true", the package is not automatically installed into
9139 the DUT.
9140
9141Following is an example JSON file that handles test "foo" installing
9142package "bar" and test "foobar" installing packages "foo" and "bar".
9143Once the test is complete, the packages are removed from the DUT.
9144::
9145
9146 {
9147 "foo": {
9148 "pkg": "bar"
9149 },
9150 "foobar": [
9151 {
9152 "pkg": "foo",
9153 "rm": true
9154 },
9155 {
9156 "pkg": "bar",
9157 "rm": true
9158 }
9159 ]
9160 }
9161
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009162Debugging Tools and Techniques
9163==============================
9164
9165The exact method for debugging build failures depends on the nature of
9166the problem and on the system's area from which the bug originates.
9167Standard debugging practices such as comparison against the last known
9168working version with examination of the changes and the re-application
9169of steps to identify the one causing the problem are valid for the Yocto
9170Project just as they are for any other system. Even though it is
9171impossible to detail every possible potential failure, this section
9172provides some general tips to aid in debugging given a variety of
9173situations.
9174
9175.. note::
9176
9177 A useful feature for debugging is the error reporting tool.
9178 Configuring the Yocto Project to use this tool causes the
9179 OpenEmbedded build system to produce error reporting commands as part
9180 of the console output. You can enter the commands after the build
9181 completes to log error information into a common database, that can
9182 help you figure out what might be going wrong. For information on how
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009183 to enable and use this feature, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06009184 ":ref:`dev-manual/common-tasks:using the error reporting tool`"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009185 section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009186
9187The following list shows the debugging topics in the remainder of this
9188section:
9189
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05009190- ":ref:`dev-manual/common-tasks:viewing logs from failed tasks`" describes
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009191 how to find and view logs from tasks that failed during the build
9192 process.
9193
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05009194- ":ref:`dev-manual/common-tasks:viewing variable values`" describes how to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009195 use the BitBake ``-e`` option to examine variable values after a
9196 recipe has been parsed.
9197
Andrew Geissler09209ee2020-12-13 08:44:15 -06009198- ":ref:`dev-manual/common-tasks:viewing package information with \`\`oe-pkgdata-util\`\``"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009199 describes how to use the ``oe-pkgdata-util`` utility to query
9200 :term:`PKGDATA_DIR` and
9201 display package-related information for built packages.
9202
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05009203- ":ref:`dev-manual/common-tasks:viewing dependencies between recipes and tasks`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009204 describes how to use the BitBake ``-g`` option to display recipe
9205 dependency information used during the build.
9206
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05009207- ":ref:`dev-manual/common-tasks:viewing task variable dependencies`" describes
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009208 how to use the ``bitbake-dumpsig`` command in conjunction with key
9209 subdirectories in the
9210 :term:`Build Directory` to determine
9211 variable dependencies.
9212
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05009213- ":ref:`dev-manual/common-tasks:running specific tasks`" describes
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009214 how to use several BitBake options (e.g. ``-c``, ``-C``, and ``-f``)
9215 to run specific tasks in the build chain. It can be useful to run
9216 tasks "out-of-order" when trying isolate build issues.
9217
Andrew Geisslerd5838332022-05-27 11:33:10 -05009218- ":ref:`dev-manual/common-tasks:general BitBake problems`" describes how
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009219 to use BitBake's ``-D`` debug output option to reveal more about what
9220 BitBake is doing during the build.
9221
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05009222- ":ref:`dev-manual/common-tasks:building with no dependencies`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009223 describes how to use the BitBake ``-b`` option to build a recipe
9224 while ignoring dependencies.
9225
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05009226- ":ref:`dev-manual/common-tasks:recipe logging mechanisms`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009227 describes how to use the many recipe logging functions to produce
9228 debugging output and report errors and warnings.
9229
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05009230- ":ref:`dev-manual/common-tasks:debugging parallel make races`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009231 describes how to debug situations where the build consists of several
9232 parts that are run simultaneously and when the output or result of
9233 one part is not ready for use with a different part of the build that
9234 depends on that output.
9235
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05009236- ":ref:`dev-manual/common-tasks:debugging with the gnu project debugger (gdb) remotely`"
9237 describes how to use GDB to allow you to examine running programs, which can
9238 help you fix problems.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009239
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05009240- ":ref:`dev-manual/common-tasks:debugging with the gnu project debugger (gdb) on the target`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009241 describes how to use GDB directly on target hardware for debugging.
9242
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05009243- ":ref:`dev-manual/common-tasks:other debugging tips`" describes
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009244 miscellaneous debugging tips that can be useful.
9245
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009246Viewing Logs from Failed Tasks
9247------------------------------
9248
9249You can find the log for a task in the file
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009250``${``\ :term:`WORKDIR`\ ``}/temp/log.do_``\ `taskname`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009251For example, the log for the
9252:ref:`ref-tasks-compile` task of the
9253QEMU minimal image for the x86 machine (``qemux86``) might be in
9254``tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/temp/log.do_compile``.
9255To see the commands :term:`BitBake` ran
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009256to generate a log, look at the corresponding ``run.do_``\ `taskname` file
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009257in the same directory.
9258
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009259``log.do_``\ `taskname` and ``run.do_``\ `taskname` are actually symbolic
9260links to ``log.do_``\ `taskname`\ ``.``\ `pid` and
9261``log.run_``\ `taskname`\ ``.``\ `pid`, where `pid` is the PID the task had
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009262when it ran. The symlinks always point to the files corresponding to the
9263most recent run.
9264
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009265Viewing Variable Values
9266-----------------------
9267
9268Sometimes you need to know the value of a variable as a result of
9269BitBake's parsing step. This could be because some unexpected behavior
9270occurred in your project. Perhaps an attempt to :ref:`modify a variable
9271<bitbake:bitbake-user-manual/bitbake-user-manual-metadata:modifying existing
9272variables>` did not work out as expected.
9273
9274BitBake's ``-e`` option is used to display variable values after
9275parsing. The following command displays the variable values after the
9276configuration files (i.e. ``local.conf``, ``bblayers.conf``,
Andrew Geisslerc926e172021-05-07 16:11:35 -05009277``bitbake.conf`` and so forth) have been parsed::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009278
9279 $ bitbake -e
9280
9281The following command displays variable values after a specific recipe has
Andrew Geisslerc926e172021-05-07 16:11:35 -05009282been parsed. The variables include those from the configuration as well::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009283
9284 $ bitbake -e recipename
9285
9286.. note::
9287
9288 Each recipe has its own private set of variables (datastore).
9289 Internally, after parsing the configuration, a copy of the resulting
9290 datastore is made prior to parsing each recipe. This copying implies
9291 that variables set in one recipe will not be visible to other
9292 recipes.
9293
9294 Likewise, each task within a recipe gets a private datastore based on
9295 the recipe datastore, which means that variables set within one task
9296 will not be visible to other tasks.
9297
9298In the output of ``bitbake -e``, each variable is preceded by a
9299description of how the variable got its value, including temporary
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009300values that were later overridden. This description also includes
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009301variable flags (varflags) set on the variable. The output can be very
9302helpful during debugging.
9303
9304Variables that are exported to the environment are preceded by
Andrew Geisslerc926e172021-05-07 16:11:35 -05009305``export`` in the output of ``bitbake -e``. See the following example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009306
9307 export CC="i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/ulf/poky/build/tmp/sysroots/qemux86"
9308
9309In addition to variable values, the output of the ``bitbake -e`` and
9310``bitbake -e`` recipe commands includes the following information:
9311
9312- The output starts with a tree listing all configuration files and
9313 classes included globally, recursively listing the files they include
9314 or inherit in turn. Much of the behavior of the OpenEmbedded build
Andrew Geissler09209ee2020-12-13 08:44:15 -06009315 system (including the behavior of the :ref:`ref-manual/tasks:normal recipe build tasks`) is
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009316 implemented in the
9317 :ref:`base <ref-classes-base>` class and the
9318 classes it inherits, rather than being built into BitBake itself.
9319
9320- After the variable values, all functions appear in the output. For
9321 shell functions, variables referenced within the function body are
9322 expanded. If a function has been modified using overrides or using
Patrick Williams0ca19cc2021-08-16 14:03:13 -05009323 override-style operators like ``:append`` and ``:prepend``, then the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009324 final assembled function body appears in the output.
9325
9326Viewing Package Information with ``oe-pkgdata-util``
9327----------------------------------------------------
9328
9329You can use the ``oe-pkgdata-util`` command-line utility to query
9330:term:`PKGDATA_DIR` and display
9331various package-related information. When you use the utility, you must
9332use it to view information on packages that have already been built.
9333
9334Following are a few of the available ``oe-pkgdata-util`` subcommands.
9335
9336.. note::
9337
9338 You can use the standard \* and ? globbing wildcards as part of
9339 package names and paths.
9340
9341- ``oe-pkgdata-util list-pkgs [pattern]``: Lists all packages
9342 that have been built, optionally limiting the match to packages that
9343 match pattern.
9344
9345- ``oe-pkgdata-util list-pkg-files package ...``: Lists the
9346 files and directories contained in the given packages.
9347
9348 .. note::
9349
9350 A different way to view the contents of a package is to look at
9351 the
9352 ``${``\ :term:`WORKDIR`\ ``}/packages-split``
9353 directory of the recipe that generates the package. This directory
9354 is created by the
9355 :ref:`ref-tasks-package` task
9356 and has one subdirectory for each package the recipe generates,
9357 which contains the files stored in that package.
9358
9359 If you want to inspect the ``${WORKDIR}/packages-split``
9360 directory, make sure that
9361 :ref:`rm_work <ref-classes-rm-work>` is not
9362 enabled when you build the recipe.
9363
9364- ``oe-pkgdata-util find-path path ...``: Lists the names of
9365 the packages that contain the given paths. For example, the following
9366 tells us that ``/usr/share/man/man1/make.1`` is contained in the
Andrew Geisslerc926e172021-05-07 16:11:35 -05009367 ``make-doc`` package::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009368
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009369 $ oe-pkgdata-util find-path /usr/share/man/man1/make.1
9370 make-doc: /usr/share/man/man1/make.1
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009371
9372- ``oe-pkgdata-util lookup-recipe package ...``: Lists the name
9373 of the recipes that produce the given packages.
9374
9375For more information on the ``oe-pkgdata-util`` command, use the help
Andrew Geisslerc926e172021-05-07 16:11:35 -05009376facility::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009377
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009378 $ oe-pkgdata-util --help
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009379 $ oe-pkgdata-util subcommand --help
9380
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009381Viewing Dependencies Between Recipes and Tasks
9382----------------------------------------------
9383
9384Sometimes it can be hard to see why BitBake wants to build other recipes
9385before the one you have specified. Dependency information can help you
9386understand why a recipe is built.
9387
9388To generate dependency information for a recipe, run the following
Andrew Geisslerc926e172021-05-07 16:11:35 -05009389command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009390
9391 $ bitbake -g recipename
9392
9393This command writes the following files in the current directory:
9394
9395- ``pn-buildlist``: A list of recipes/targets involved in building
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009396 `recipename`. "Involved" here means that at least one task from the
9397 recipe needs to run when building `recipename` from scratch. Targets
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009398 that are in
9399 :term:`ASSUME_PROVIDED`
9400 are not listed.
9401
9402- ``task-depends.dot``: A graph showing dependencies between tasks.
9403
9404The graphs are in
9405`DOT <https://en.wikipedia.org/wiki/DOT_%28graph_description_language%29>`__
9406format and can be converted to images (e.g. using the ``dot`` tool from
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009407`Graphviz <https://www.graphviz.org/>`__).
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009408
9409.. note::
9410
9411 - DOT files use a plain text format. The graphs generated using the
9412 ``bitbake -g`` command are often so large as to be difficult to
Andrew Geisslerd5838332022-05-27 11:33:10 -05009413 read without special pruning (e.g. with BitBake's ``-I`` option)
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009414 and processing. Despite the form and size of the graphs, the
9415 corresponding ``.dot`` files can still be possible to read and
9416 provide useful information.
9417
9418 As an example, the ``task-depends.dot`` file contains lines such
Andrew Geisslerc926e172021-05-07 16:11:35 -05009419 as the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009420
9421 "libxslt.do_configure" -> "libxml2.do_populate_sysroot"
9422
9423 The above example line reveals that the
9424 :ref:`ref-tasks-configure`
9425 task in ``libxslt`` depends on the
9426 :ref:`ref-tasks-populate_sysroot`
9427 task in ``libxml2``, which is a normal
9428 :term:`DEPENDS` dependency
9429 between the two recipes.
9430
9431 - For an example of how ``.dot`` files can be processed, see the
9432 ``scripts/contrib/graph-tool`` Python script, which finds and
9433 displays paths between graph nodes.
9434
9435You can use a different method to view dependency information by using
Andrew Geisslerc926e172021-05-07 16:11:35 -05009436the following command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009437
9438 $ bitbake -g -u taskexp recipename
9439
9440This command
9441displays a GUI window from which you can view build-time and runtime
9442dependencies for the recipes involved in building recipename.
9443
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009444Viewing Task Variable Dependencies
9445----------------------------------
9446
9447As mentioned in the
9448":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-execution:checksums (signatures)`" section of the BitBake
9449User Manual, BitBake tries to automatically determine what variables a
9450task depends on so that it can rerun the task if any values of the
9451variables change. This determination is usually reliable. However, if
9452you do things like construct variable names at runtime, then you might
9453have to manually declare dependencies on those variables using
9454``vardeps`` as described in the
9455":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags`" section of the BitBake
9456User Manual.
9457
9458If you are unsure whether a variable dependency is being picked up
9459automatically for a given task, you can list the variable dependencies
9460BitBake has determined by doing the following:
9461
Andrew Geisslerc926e172021-05-07 16:11:35 -050094621. Build the recipe containing the task::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009463
9464 $ bitbake recipename
9465
94662. Inside the :term:`STAMPS_DIR`
9467 directory, find the signature data (``sigdata``) file that
9468 corresponds to the task. The ``sigdata`` files contain a pickled
9469 Python database of all the metadata that went into creating the input
9470 checksum for the task. As an example, for the
9471 :ref:`ref-tasks-fetch` task of the
9472 ``db`` recipe, the ``sigdata`` file might be found in the following
Andrew Geisslerc926e172021-05-07 16:11:35 -05009473 location::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009474
9475 ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1
9476
9477 For tasks that are accelerated through the shared state
Andrew Geissler09209ee2020-12-13 08:44:15 -06009478 (:ref:`sstate <overview-manual/concepts:shared state cache>`) cache, an
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009479 additional ``siginfo`` file is written into
9480 :term:`SSTATE_DIR` along with
9481 the cached task output. The ``siginfo`` files contain exactly the
9482 same information as ``sigdata`` files.
9483
94843. Run ``bitbake-dumpsig`` on the ``sigdata`` or ``siginfo`` file. Here
Andrew Geisslerc926e172021-05-07 16:11:35 -05009485 is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009486
9487 $ bitbake-dumpsig ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1
9488
9489 In the output of the above command, you will find a line like the
9490 following, which lists all the (inferred) variable dependencies for
9491 the task. This list also includes indirect dependencies from
9492 variables depending on other variables, recursively.
9493 ::
9494
9495 Task dependencies: ['PV', 'SRCREV', 'SRC_URI', 'SRC_URI[md5sum]', 'SRC_URI[sha256sum]', 'base_do_fetch']
9496
9497 .. note::
9498
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009499 Functions (e.g. ``base_do_fetch``) also count as variable dependencies.
9500 These functions in turn depend on the variables they reference.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009501
9502 The output of ``bitbake-dumpsig`` also includes the value each
9503 variable had, a list of dependencies for each variable, and
Andrew Geissler7e0e3c02022-02-25 20:34:39 +00009504 :term:`BB_BASEHASH_IGNORE_VARS`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009505 information.
9506
9507There is also a ``bitbake-diffsigs`` command for comparing two
9508``siginfo`` or ``sigdata`` files. This command can be helpful when
9509trying to figure out what changed between two versions of a task. If you
9510call ``bitbake-diffsigs`` with just one file, the command behaves like
9511``bitbake-dumpsig``.
9512
9513You can also use BitBake to dump out the signature construction
9514information without executing tasks by using either of the following
Andrew Geisslerc926e172021-05-07 16:11:35 -05009515BitBake command-line options::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009516
9517 ‐‐dump-signatures=SIGNATURE_HANDLER
9518 -S SIGNATURE_HANDLER
9519
9520
9521.. note::
9522
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009523 Two common values for `SIGNATURE_HANDLER` are "none" and "printdiff", which
9524 dump only the signature or compare the dumped signature with the cached one,
9525 respectively.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009526
9527Using BitBake with either of these options causes BitBake to dump out
9528``sigdata`` files in the ``stamps`` directory for every task it would
9529have executed instead of building the specified target package.
9530
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009531Viewing Metadata Used to Create the Input Signature of a Shared State Task
9532--------------------------------------------------------------------------
9533
9534Seeing what metadata went into creating the input signature of a shared
9535state (sstate) task can be a useful debugging aid. This information is
9536available in signature information (``siginfo``) files in
9537:term:`SSTATE_DIR`. For
9538information on how to view and interpret information in ``siginfo``
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05009539files, see the
9540":ref:`dev-manual/common-tasks:viewing task variable dependencies`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009541
9542For conceptual information on shared state, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06009543":ref:`overview-manual/concepts:shared state`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009544section in the Yocto Project Overview and Concepts Manual.
9545
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009546Invalidating Shared State to Force a Task to Run
9547------------------------------------------------
9548
9549The OpenEmbedded build system uses
Andrew Geissler09209ee2020-12-13 08:44:15 -06009550:ref:`checksums <overview-manual/concepts:checksums (signatures)>` and
9551:ref:`overview-manual/concepts:shared state` cache to avoid unnecessarily
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009552rebuilding tasks. Collectively, this scheme is known as "shared state
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009553code".
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009554
9555As with all schemes, this one has some drawbacks. It is possible that
9556you could make implicit changes to your code that the checksum
9557calculations do not take into account. These implicit changes affect a
9558task's output but do not trigger the shared state code into rebuilding a
9559recipe. Consider an example during which a tool changes its output.
9560Assume that the output of ``rpmdeps`` changes. The result of the change
9561should be that all the ``package`` and ``package_write_rpm`` shared
9562state cache items become invalid. However, because the change to the
9563output is external to the code and therefore implicit, the associated
9564shared state cache items do not become invalidated. In this case, the
9565build process uses the cached items rather than running the task again.
9566Obviously, these types of implicit changes can cause problems.
9567
9568To avoid these problems during the build, you need to understand the
9569effects of any changes you make. Realize that changes you make directly
9570to a function are automatically factored into the checksum calculation.
9571Thus, these explicit changes invalidate the associated area of shared
9572state cache. However, you need to be aware of any implicit changes that
9573are not obvious changes to the code and could affect the output of a
9574given task.
9575
9576When you identify an implicit change, you can easily take steps to
9577invalidate the cache and force the tasks to run. The steps you can take
9578are as simple as changing a function's comments in the source code. For
9579example, to invalidate package shared state files, change the comment
9580statements of
9581:ref:`ref-tasks-package` or the
9582comments of one of the functions it calls. Even though the change is
9583purely cosmetic, it causes the checksum to be recalculated and forces
9584the build system to run the task again.
9585
9586.. note::
9587
9588 For an example of a commit that makes a cosmetic change to invalidate
9589 shared state, see this
Andrew Geissler09209ee2020-12-13 08:44:15 -06009590 :yocto_git:`commit </poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009591
9592Running Specific Tasks
9593----------------------
9594
9595Any given recipe consists of a set of tasks. The standard BitBake
Patrick Williams2194f502022-10-16 14:26:09 -05009596behavior in most cases is: :ref:`ref-tasks-fetch`, :ref:`ref-tasks-unpack`, :ref:`ref-tasks-patch`,
9597:ref:`ref-tasks-configure`, :ref:`ref-tasks-compile`, :ref:`ref-tasks-install`, :ref:`ref-tasks-package`,
9598:ref:`do_package_write_* <ref-tasks-package_write_deb>`, and :ref:`ref-tasks-build`. The default task is
9599:ref:`ref-tasks-build` and any tasks on which it depends build first. Some tasks,
9600such as :ref:`ref-tasks-devshell`, are not part of the default build chain. If you
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009601wish to run a task that is not part of the default build chain, you can
Andrew Geisslerc926e172021-05-07 16:11:35 -05009602use the ``-c`` option in BitBake. Here is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009603
9604 $ bitbake matchbox-desktop -c devshell
9605
9606The ``-c`` option respects task dependencies, which means that all other
9607tasks (including tasks from other recipes) that the specified task
9608depends on will be run before the task. Even when you manually specify a
9609task to run with ``-c``, BitBake will only run the task if it considers
9610it "out of date". See the
Andrew Geissler09209ee2020-12-13 08:44:15 -06009611":ref:`overview-manual/concepts:stamp files and the rerunning of tasks`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009612section in the Yocto Project Overview and Concepts Manual for how
9613BitBake determines whether a task is "out of date".
9614
9615If you want to force an up-to-date task to be rerun (e.g. because you
9616made manual modifications to the recipe's
9617:term:`WORKDIR` that you want to try
9618out), then you can use the ``-f`` option.
9619
9620.. note::
9621
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009622 The reason ``-f`` is never required when running the
9623 :ref:`ref-tasks-devshell` task is because the
9624 [\ :ref:`nostamp <bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ]
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009625 variable flag is already set for the task.
9626
Andrew Geisslerc926e172021-05-07 16:11:35 -05009627The following example shows one way you can use the ``-f`` option::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009628
9629 $ bitbake matchbox-desktop
9630 .
9631 .
9632 make some changes to the source code in the work directory
9633 .
9634 .
9635 $ bitbake matchbox-desktop -c compile -f
9636 $ bitbake matchbox-desktop
9637
9638This sequence first builds and then recompiles ``matchbox-desktop``. The
9639last command reruns all tasks (basically the packaging tasks) after the
Patrick Williams2194f502022-10-16 14:26:09 -05009640compile. BitBake recognizes that the :ref:`ref-tasks-compile` task was rerun and
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009641therefore understands that the other tasks also need to be run again.
9642
9643Another, shorter way to rerun a task and all
Andrew Geissler09209ee2020-12-13 08:44:15 -06009644:ref:`ref-manual/tasks:normal recipe build tasks`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009645that depend on it is to use the ``-C`` option.
9646
9647.. note::
9648
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009649 This option is upper-cased and is separate from the ``-c``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009650 option, which is lower-cased.
9651
9652Using this option invalidates the given task and then runs the
9653:ref:`ref-tasks-build` task, which is
9654the default task if no task is given, and the tasks on which it depends.
9655You could replace the final two commands in the previous example with
Andrew Geisslerc926e172021-05-07 16:11:35 -05009656the following single command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009657
9658 $ bitbake matchbox-desktop -C compile
9659
9660Internally, the ``-f`` and ``-C`` options work by tainting (modifying)
9661the input checksum of the specified task. This tainting indirectly
9662causes the task and its dependent tasks to be rerun through the normal
9663task dependency mechanisms.
9664
9665.. note::
9666
9667 BitBake explicitly keeps track of which tasks have been tainted in
9668 this fashion, and will print warnings such as the following for
9669 builds involving such tasks:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009670
9671 .. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009672
9673 WARNING: /home/ulf/poky/meta/recipes-sato/matchbox-desktop/matchbox-desktop_2.1.bb.do_compile is tainted from a forced run
9674
9675
9676 The purpose of the warning is to let you know that the work directory
9677 and build output might not be in the clean state they would be in for
9678 a "normal" build, depending on what actions you took. To get rid of
9679 such warnings, you can remove the work directory and rebuild the
Andrew Geisslerc926e172021-05-07 16:11:35 -05009680 recipe, as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009681
9682 $ bitbake matchbox-desktop -c clean
9683 $ bitbake matchbox-desktop
9684
9685
9686You can view a list of tasks in a given package by running the
Patrick Williams2194f502022-10-16 14:26:09 -05009687:ref:`ref-tasks-listtasks` task as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009688
9689 $ bitbake matchbox-desktop -c listtasks
9690
9691The results appear as output to the console and are also in
9692the file ``${WORKDIR}/temp/log.do_listtasks``.
9693
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009694General BitBake Problems
9695------------------------
9696
9697You can see debug output from BitBake by using the ``-D`` option. The
9698debug output gives more information about what BitBake is doing and the
9699reason behind it. Each ``-D`` option you use increases the logging
9700level. The most common usage is ``-DDD``.
9701
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009702The output from ``bitbake -DDD -v targetname`` can reveal why BitBake
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009703chose a certain version of a package or why BitBake picked a certain
9704provider. This command could also help you in a situation where you
9705think BitBake did something unexpected.
9706
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009707Building with No Dependencies
9708-----------------------------
9709
9710To build a specific recipe (``.bb`` file), you can use the following
Andrew Geisslerc926e172021-05-07 16:11:35 -05009711command form::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009712
9713 $ bitbake -b somepath/somerecipe.bb
9714
9715This command form does
9716not check for dependencies. Consequently, you should use it only when
9717you know existing dependencies have been met.
9718
9719.. note::
9720
9721 You can also specify fragments of the filename. In this case, BitBake
9722 checks for a unique match.
9723
9724Recipe Logging Mechanisms
9725-------------------------
9726
9727The Yocto Project provides several logging functions for producing
9728debugging output and reporting errors and warnings. For Python
William A. Kennington IIIac69b482021-06-02 12:28:27 -07009729functions, the following logging functions are available. All of these functions
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009730log to ``${T}/log.do_``\ `task`, and can also log to standard output
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009731(stdout) with the right settings:
9732
9733- ``bb.plain(msg)``: Writes msg as is to the log while also
9734 logging to stdout.
9735
9736- ``bb.note(msg)``: Writes "NOTE: msg" to the log. Also logs to
9737 stdout if BitBake is called with "-v".
9738
9739- ``bb.debug(level, msg)``: Writes "DEBUG: msg" to the
9740 log. Also logs to stdout if the log level is greater than or equal to
Patrick Williams213cb262021-08-07 19:21:33 -05009741 level. See the ":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-intro:usage and syntax`" option
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009742 in the BitBake User Manual for more information.
9743
9744- ``bb.warn(msg)``: Writes "WARNING: msg" to the log while also
9745 logging to stdout.
9746
9747- ``bb.error(msg)``: Writes "ERROR: msg" to the log while also
9748 logging to standard out (stdout).
9749
9750 .. note::
9751
9752 Calling this function does not cause the task to fail.
9753
Andrew Geisslereff27472021-10-29 15:35:00 -05009754- ``bb.fatal(msg)``: This logging function is similar to
9755 ``bb.error(msg)`` but also causes the calling task to fail.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009756
9757 .. note::
9758
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009759 ``bb.fatal()`` raises an exception, which means you do not need to put a
9760 "return" statement after the function.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009761
9762The same logging functions are also available in shell functions, under
9763the names ``bbplain``, ``bbnote``, ``bbdebug``, ``bbwarn``, ``bberror``,
9764and ``bbfatal``. The
9765:ref:`logging <ref-classes-logging>` class
9766implements these functions. See that class in the ``meta/classes``
9767folder of the :term:`Source Directory` for information.
9768
9769Logging With Python
9770~~~~~~~~~~~~~~~~~~~
9771
9772When creating recipes using Python and inserting code that handles build
9773logs, keep in mind the goal is to have informative logs while keeping
9774the console as "silent" as possible. Also, if you want status messages
9775in the log, use the "debug" loglevel.
9776
9777Following is an example written in Python. The code handles logging for
9778a function that determines the number of tasks needed to be run. See the
9779":ref:`ref-tasks-listtasks`"
Andrew Geisslerc926e172021-05-07 16:11:35 -05009780section for additional information::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009781
9782 python do_listtasks() {
9783 bb.debug(2, "Starting to figure out the task list")
9784 if noteworthy_condition:
9785 bb.note("There are 47 tasks to run")
9786 bb.debug(2, "Got to point xyz")
9787 if warning_trigger:
9788 bb.warn("Detected warning_trigger, this might be a problem later.")
9789 if recoverable_error:
9790 bb.error("Hit recoverable_error, you really need to fix this!")
9791 if fatal_error:
9792 bb.fatal("fatal_error detected, unable to print the task list")
9793 bb.plain("The tasks present are abc")
9794 bb.debug(2, "Finished figuring out the tasklist")
9795 }
9796
9797Logging With Bash
9798~~~~~~~~~~~~~~~~~
9799
9800When creating recipes using Bash and inserting code that handles build
Andrew Geissler615f2f12022-07-15 14:00:58 -05009801logs, you have the same goals --- informative with minimal console output.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009802The syntax you use for recipes written in Bash is similar to that of
9803recipes written in Python described in the previous section.
9804
9805Following is an example written in Bash. The code logs the progress of
9806the ``do_my_function`` function.
9807::
9808
9809 do_my_function() {
9810 bbdebug 2 "Running do_my_function"
9811 if [ exceptional_condition ]; then
9812 bbnote "Hit exceptional_condition"
9813 fi
9814 bbdebug 2 "Got to point xyz"
9815 if [ warning_trigger ]; then
9816 bbwarn "Detected warning_trigger, this might cause a problem later."
9817 fi
9818 if [ recoverable_error ]; then
9819 bberror "Hit recoverable_error, correcting"
9820 fi
9821 if [ fatal_error ]; then
9822 bbfatal "fatal_error detected"
9823 fi
9824 bbdebug 2 "Completed do_my_function"
9825 }
9826
9827
9828Debugging Parallel Make Races
9829-----------------------------
9830
9831A parallel ``make`` race occurs when the build consists of several parts
9832that are run simultaneously and a situation occurs when the output or
9833result of one part is not ready for use with a different part of the
9834build that depends on that output. Parallel make races are annoying and
William A. Kennington IIIac69b482021-06-02 12:28:27 -07009835can sometimes be difficult to reproduce and fix. However, there are some simple
9836tips and tricks that can help you debug and fix them. This section
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009837presents a real-world example of an error encountered on the Yocto
9838Project autobuilder and the process used to fix it.
9839
9840.. note::
9841
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009842 If you cannot properly fix a ``make`` race condition, you can work around it
9843 by clearing either the :term:`PARALLEL_MAKE` or :term:`PARALLEL_MAKEINST`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009844 variables.
9845
9846The Failure
9847~~~~~~~~~~~
9848
9849For this example, assume that you are building an image that depends on
9850the "neard" package. And, during the build, BitBake runs into problems
9851and creates the following output.
9852
9853.. note::
9854
9855 This example log file has longer lines artificially broken to make
9856 the listing easier to read.
9857
9858If you examine the output or the log file, you see the failure during
9859``make``:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009860
9861.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009862
9863 | DEBUG: SITE files ['endian-little', 'bit-32', 'ix86-common', 'common-linux', 'common-glibc', 'i586-linux', 'common']
9864 | DEBUG: Executing shell function do_compile
9865 | NOTE: make -j 16
9866 | make --no-print-directory all-am
9867 | /bin/mkdir -p include/near
9868 | /bin/mkdir -p include/near
9869 | /bin/mkdir -p include/near
Andrew Geissler595f6302022-01-24 19:11:47 +00009870 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009871 0.14-r0/neard-0.14/include/types.h include/near/types.h
Andrew Geissler595f6302022-01-24 19:11:47 +00009872 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009873 0.14-r0/neard-0.14/include/log.h include/near/log.h
Andrew Geissler595f6302022-01-24 19:11:47 +00009874 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009875 0.14-r0/neard-0.14/include/plugin.h include/near/plugin.h
9876 | /bin/mkdir -p include/near
9877 | /bin/mkdir -p include/near
9878 | /bin/mkdir -p include/near
Andrew Geissler595f6302022-01-24 19:11:47 +00009879 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009880 0.14-r0/neard-0.14/include/tag.h include/near/tag.h
9881 | /bin/mkdir -p include/near
Andrew Geissler595f6302022-01-24 19:11:47 +00009882 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009883 0.14-r0/neard-0.14/include/adapter.h include/near/adapter.h
9884 | /bin/mkdir -p include/near
Andrew Geissler595f6302022-01-24 19:11:47 +00009885 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009886 0.14-r0/neard-0.14/include/ndef.h include/near/ndef.h
Andrew Geissler595f6302022-01-24 19:11:47 +00009887 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009888 0.14-r0/neard-0.14/include/tlv.h include/near/tlv.h
9889 | /bin/mkdir -p include/near
9890 | /bin/mkdir -p include/near
Andrew Geissler595f6302022-01-24 19:11:47 +00009891 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009892 0.14-r0/neard-0.14/include/setting.h include/near/setting.h
9893 | /bin/mkdir -p include/near
9894 | /bin/mkdir -p include/near
9895 | /bin/mkdir -p include/near
Andrew Geissler595f6302022-01-24 19:11:47 +00009896 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009897 0.14-r0/neard-0.14/include/device.h include/near/device.h
Andrew Geissler595f6302022-01-24 19:11:47 +00009898 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009899 0.14-r0/neard-0.14/include/nfc_copy.h include/near/nfc_copy.h
Andrew Geissler595f6302022-01-24 19:11:47 +00009900 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009901 0.14-r0/neard-0.14/include/snep.h include/near/snep.h
Andrew Geissler595f6302022-01-24 19:11:47 +00009902 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009903 0.14-r0/neard-0.14/include/version.h include/near/version.h
Andrew Geissler595f6302022-01-24 19:11:47 +00009904 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009905 0.14-r0/neard-0.14/include/dbus.h include/near/dbus.h
9906 | ./src/genbuiltin nfctype1 nfctype2 nfctype3 nfctype4 p2p > src/builtin.h
Andrew Geissler595f6302022-01-24 19:11:47 +00009907 | i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/pokybuild/yocto-autobuilder/nightly-x86/
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009908 build/build/tmp/sysroots/qemux86 -DHAVE_CONFIG_H -I. -I./include -I./src -I./gdbus -I/home/pokybuild/
Andrew Geissler595f6302022-01-24 19:11:47 +00009909 yocto-autobuilder/nightly-x86/build/build/tmp/sysroots/qemux86/usr/include/glib-2.0
9910 -I/home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/sysroots/qemux86/usr/
9911 lib/glib-2.0/include -I/home/pokybuild/yocto-autobuilder/nightly-x86/build/build/
9912 tmp/sysroots/qemux86/usr/include/dbus-1.0 -I/home/pokybuild/yocto-autobuilder/
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009913 nightly-x86/build/build/tmp/sysroots/qemux86/usr/lib/dbus-1.0/include -I/home/pokybuild/yocto-autobuilder/
Andrew Geissler595f6302022-01-24 19:11:47 +00009914 nightly-x86/build/build/tmp/sysroots/qemux86/usr/include/libnl3
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009915 -DNEAR_PLUGIN_BUILTIN -DPLUGINDIR=\""/usr/lib/near/plugins"\"
9916 -DCONFIGDIR=\""/etc/neard\"" -O2 -pipe -g -feliminate-unused-debug-types -c
9917 -o tools/snep-send.o tools/snep-send.c
9918 | In file included from tools/snep-send.c:16:0:
9919 | tools/../src/near.h:41:23: fatal error: near/dbus.h: No such file or directory
9920 | #include <near/dbus.h>
9921 | ^
9922 | compilation terminated.
9923 | make[1]: *** [tools/snep-send.o] Error 1
9924 | make[1]: *** Waiting for unfinished jobs....
9925 | make: *** [all] Error 2
9926 | ERROR: oe_runmake failed
9927
9928Reproducing the Error
9929~~~~~~~~~~~~~~~~~~~~~
9930
9931Because race conditions are intermittent, they do not manifest
9932themselves every time you do the build. In fact, most times the build
9933will complete without problems even though the potential race condition
9934exists. Thus, once the error surfaces, you need a way to reproduce it.
9935
9936In this example, compiling the "neard" package is causing the problem.
9937So the first thing to do is build "neard" locally. Before you start the
9938build, set the
9939:term:`PARALLEL_MAKE` variable
9940in your ``local.conf`` file to a high number (e.g. "-j 20"). Using a
Andrew Geissler09036742021-06-25 14:25:14 -05009941high value for :term:`PARALLEL_MAKE` increases the chances of the race
Andrew Geisslerc926e172021-05-07 16:11:35 -05009942condition showing up::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009943
9944 $ bitbake neard
9945
Andrew Geisslerc926e172021-05-07 16:11:35 -05009946Once the local build for "neard" completes, start a ``devshell`` build::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009947
9948 $ bitbake neard -c devshell
9949
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05009950For information on how to use a ``devshell``, see the
9951":ref:`dev-manual/common-tasks:using a development shell`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009952
Andrew Geisslerc926e172021-05-07 16:11:35 -05009953In the ``devshell``, do the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009954
9955 $ make clean
9956 $ make tools/snep-send.o
9957
9958The ``devshell`` commands cause the failure to clearly
William A. Kennington IIIac69b482021-06-02 12:28:27 -07009959be visible. In this case, there is a missing dependency for the ``neard``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009960Makefile target. Here is some abbreviated, sample output with the
Andrew Geisslerc926e172021-05-07 16:11:35 -05009961missing dependency clearly visible at the end::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009962
9963 i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/scott-lenovo/......
9964 .
9965 .
9966 .
9967 tools/snep-send.c
9968 In file included from tools/snep-send.c:16:0:
9969 tools/../src/near.h:41:23: fatal error: near/dbus.h: No such file or directory
9970 #include <near/dbus.h>
9971 ^
9972 compilation terminated.
9973 make: *** [tools/snep-send.o] Error 1
9974 $
9975
9976
9977Creating a Patch for the Fix
9978~~~~~~~~~~~~~~~~~~~~~~~~~~~~
9979
9980Because there is a missing dependency for the Makefile target, you need
9981to patch the ``Makefile.am`` file, which is generated from
Andrew Geisslerc926e172021-05-07 16:11:35 -05009982``Makefile.in``. You can use Quilt to create the patch::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009983
9984 $ quilt new parallelmake.patch
9985 Patch patches/parallelmake.patch is now on top
9986 $ quilt add Makefile.am
9987 File Makefile.am added to patch patches/parallelmake.patch
9988
9989For more information on using Quilt, see the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05009990":ref:`dev-manual/common-tasks:using quilt in your workflow`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009991
9992At this point you need to make the edits to ``Makefile.am`` to add the
9993missing dependency. For our example, you have to add the following line
Andrew Geisslerc926e172021-05-07 16:11:35 -05009994to the file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009995
9996 tools/snep-send.$(OBJEXT): include/near/dbus.h
9997
9998Once you have edited the file, use the ``refresh`` command to create the
Andrew Geisslerc926e172021-05-07 16:11:35 -05009999patch::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010000
10001 $ quilt refresh
10002 Refreshed patch patches/parallelmake.patch
10003
William A. Kennington IIIac69b482021-06-02 12:28:27 -070010004Once the patch file is created, you need to add it back to the originating
10005recipe folder. Here is an example assuming a top-level
Andrew Geisslerc926e172021-05-07 16:11:35 -050010006:term:`Source Directory` named ``poky``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010007
10008 $ cp patches/parallelmake.patch poky/meta/recipes-connectivity/neard/neard
10009
10010The final thing you need to do to implement the fix in the build is to
10011update the "neard" recipe (i.e. ``neard-0.14.bb``) so that the
10012:term:`SRC_URI` statement includes
10013the patch file. The recipe file is in the folder above the patch. Here
Andrew Geissler09036742021-06-25 14:25:14 -050010014is what the edited :term:`SRC_URI` statement would look like::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010015
10016 SRC_URI = "${KERNELORG_MIRROR}/linux/network/nfc/${BPN}-${PV}.tar.xz \
10017 file://neard.in \
10018 file://neard.service.in \
10019 file://parallelmake.patch \
10020 "
10021
10022With the patch complete and moved to the correct folder and the
Andrew Geissler09036742021-06-25 14:25:14 -050010023:term:`SRC_URI` statement updated, you can exit the ``devshell``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010024
10025 $ exit
10026
10027Testing the Build
10028~~~~~~~~~~~~~~~~~
10029
10030With everything in place, you can get back to trying the build again
Andrew Geisslerc926e172021-05-07 16:11:35 -050010031locally::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010032
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010033 $ bitbake neard
10034
10035This build should succeed.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010036
10037Now you can open up a ``devshell`` again and repeat the clean and make
Andrew Geisslerc926e172021-05-07 16:11:35 -050010038operations as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010039
10040 $ bitbake neard -c devshell
10041 $ make clean
10042 $ make tools/snep-send.o
10043
10044The build should work without issue.
10045
10046As with all solved problems, if they originated upstream, you need to
10047submit the fix for the recipe in OE-Core and upstream so that the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -050010048problem is taken care of at its source. See the
10049":ref:`dev-manual/common-tasks:submitting a change to the yocto project`"
10050section for more information.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010051
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010052Debugging With the GNU Project Debugger (GDB) Remotely
10053------------------------------------------------------
10054
10055GDB allows you to examine running programs, which in turn helps you to
10056understand and fix problems. It also allows you to perform post-mortem
10057style analysis of program crashes. GDB is available as a package within
10058the Yocto Project and is installed in SDK images by default. See the
Andrew Geissler09209ee2020-12-13 08:44:15 -060010059":ref:`ref-manual/images:Images`" chapter in the Yocto
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010060Project Reference Manual for a description of these images. You can find
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010061information on GDB at https://sourceware.org/gdb/.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010062
10063.. note::
10064
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010065 For best results, install debug (``-dbg``) packages for the applications you
10066 are going to debug. Doing so makes extra debug symbols available that give
10067 you more meaningful output.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010068
10069Sometimes, due to memory or disk space constraints, it is not possible
10070to use GDB directly on the remote target to debug applications. These
10071constraints arise because GDB needs to load the debugging information
10072and the binaries of the process being debugged. Additionally, GDB needs
10073to perform many computations to locate information such as function
Andrew Geissler615f2f12022-07-15 14:00:58 -050010074names, variable names and values, stack traces and so forth --- even
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010075before starting the debugging process. These extra computations place
10076more load on the target system and can alter the characteristics of the
10077program being debugged.
10078
Andrew Geissler95ac1b82021-03-31 14:34:31 -050010079To help get past the previously mentioned constraints, there are two
10080methods you can use: running a debuginfod server and using gdbserver.
10081
10082Using the debuginfod server method
10083~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10084
Andrew Geisslerc926e172021-05-07 16:11:35 -050010085``debuginfod`` from ``elfutils`` is a way to distribute ``debuginfo`` files.
10086Running a ``debuginfod`` server makes debug symbols readily available,
Andrew Geissler95ac1b82021-03-31 14:34:31 -050010087which means you don't need to download debugging information
10088and the binaries of the process being debugged. You can just fetch
10089debug symbols from the server.
10090
Andrew Geisslerc926e172021-05-07 16:11:35 -050010091To run a ``debuginfod`` server, you need to do the following:
Andrew Geissler95ac1b82021-03-31 14:34:31 -050010092
Andrew Geisslerc926e172021-05-07 16:11:35 -050010093- Ensure that ``debuginfod`` is present in :term:`DISTRO_FEATURES`
10094 (it already is in ``OpenEmbedded-core`` defaults and ``poky`` reference distribution).
10095 If not, set in your distro config file or in ``local.conf``::
Andrew Geissler95ac1b82021-03-31 14:34:31 -050010096
Patrick Williams0ca19cc2021-08-16 14:03:13 -050010097 DISTRO_FEATURES:append = " debuginfod"
Andrew Geissler95ac1b82021-03-31 14:34:31 -050010098
Andrew Geisslerc926e172021-05-07 16:11:35 -050010099 This distro feature enables the server and client library in ``elfutils``,
10100 and enables ``debuginfod`` support in clients (at the moment, ``gdb`` and ``binutils``).
Andrew Geissler95ac1b82021-03-31 14:34:31 -050010101
Andrew Geisslerc926e172021-05-07 16:11:35 -050010102- Run the following commands to launch the ``debuginfod`` server on the host::
Andrew Geissler95ac1b82021-03-31 14:34:31 -050010103
10104 $ oe-debuginfod
10105
Andrew Geisslerc926e172021-05-07 16:11:35 -050010106- To use ``debuginfod`` on the target, you need to know the ip:port where
10107 ``debuginfod`` is listening on the host (port defaults to 8002), and export
10108 that into the shell environment, for example in ``qemu``::
Andrew Geissler95ac1b82021-03-31 14:34:31 -050010109
Andrew Geisslerc926e172021-05-07 16:11:35 -050010110 root@qemux86-64:~# export DEBUGINFOD_URLS="http://192.168.7.1:8002/"
Andrew Geissler95ac1b82021-03-31 14:34:31 -050010111
Andrew Geisslerc926e172021-05-07 16:11:35 -050010112- Then debug info fetching should simply work when running the target ``gdb``,
10113 ``readelf`` or ``objdump``, for example::
Andrew Geissler95ac1b82021-03-31 14:34:31 -050010114
Andrew Geisslerc926e172021-05-07 16:11:35 -050010115 root@qemux86-64:~# gdb /bin/cat
10116 ...
10117 Reading symbols from /bin/cat...
10118 Downloading separate debug info for /bin/cat...
10119 Reading symbols from /home/root/.cache/debuginfod_client/923dc4780cfbc545850c616bffa884b6b5eaf322/debuginfo...
Andrew Geissler95ac1b82021-03-31 14:34:31 -050010120
Andrew Geisslerc926e172021-05-07 16:11:35 -050010121- It's also possible to use ``debuginfod-find`` to just query the server::
Andrew Geissler95ac1b82021-03-31 14:34:31 -050010122
Andrew Geisslerc926e172021-05-07 16:11:35 -050010123 root@qemux86-64:~# debuginfod-find debuginfo /bin/ls
10124 /home/root/.cache/debuginfod_client/356edc585f7f82d46f94fcb87a86a3fe2d2e60bd/debuginfo
Andrew Geissler95ac1b82021-03-31 14:34:31 -050010125
Andrew Geissler95ac1b82021-03-31 14:34:31 -050010126
10127Using the gdbserver method
10128~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10129
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010130gdbserver, which runs on the remote target and does not load any
10131debugging information from the debugged process. Instead, a GDB instance
10132processes the debugging information that is run on a remote computer -
10133the host GDB. The host GDB then sends control commands to gdbserver to
10134make it stop or start the debugged program, as well as read or write
10135memory regions of that debugged program. All the debugging information
10136loaded and processed as well as all the heavy debugging is done by the
10137host GDB. Offloading these processes gives the gdbserver running on the
10138target a chance to remain small and fast.
10139
10140Because the host GDB is responsible for loading the debugging
10141information and for doing the necessary processing to make actual
10142debugging happen, you have to make sure the host can access the
10143unstripped binaries complete with their debugging information and also
10144be sure the target is compiled with no optimizations. The host GDB must
10145also have local access to all the libraries used by the debugged
10146program. Because gdbserver does not need any local debugging
10147information, the binaries on the remote target can remain stripped.
10148However, the binaries must also be compiled without optimization so they
10149match the host's binaries.
10150
10151To remain consistent with GDB documentation and terminology, the binary
10152being debugged on the remote target machine is referred to as the
10153"inferior" binary. For documentation on GDB see the `GDB
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010154site <https://sourceware.org/gdb/documentation/>`__.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010155
10156The following steps show you how to debug using the GNU project
10157debugger.
10158
101591. *Configure your build system to construct the companion debug
10160 filesystem:*
10161
Andrew Geisslerc926e172021-05-07 16:11:35 -050010162 In your ``local.conf`` file, set the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010163
10164 IMAGE_GEN_DEBUGFS = "1"
10165 IMAGE_FSTYPES_DEBUGFS = "tar.bz2"
10166
10167 These options cause the
10168 OpenEmbedded build system to generate a special companion filesystem
10169 fragment, which contains the matching source and debug symbols to
10170 your deployable filesystem. The build system does this by looking at
10171 what is in the deployed filesystem, and pulling the corresponding
10172 ``-dbg`` packages.
10173
10174 The companion debug filesystem is not a complete filesystem, but only
10175 contains the debug fragments. This filesystem must be combined with
10176 the full filesystem for debugging. Subsequent steps in this procedure
10177 show how to combine the partial filesystem with the full filesystem.
10178
101792. *Configure the system to include gdbserver in the target filesystem:*
10180
Andrew Geisslerd5838332022-05-27 11:33:10 -050010181 Make the following addition in your ``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010182
Andrew Geisslerd5838332022-05-27 11:33:10 -050010183 EXTRA_IMAGE_FEATURES:append = " tools-debug"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010184
10185 The change makes
10186 sure the ``gdbserver`` package is included.
10187
101883. *Build the environment:*
10189
10190 Use the following command to construct the image and the companion
Andrew Geisslerc926e172021-05-07 16:11:35 -050010191 Debug Filesystem::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010192
10193 $ bitbake image
10194
10195 Build the cross GDB component and
10196 make it available for debugging. Build the SDK that matches the
10197 image. Building the SDK is best for a production build that can be
Andrew Geisslerc926e172021-05-07 16:11:35 -050010198 used later for debugging, especially during long term maintenance::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010199
10200 $ bitbake -c populate_sdk image
10201
10202 Alternatively, you can build the minimal toolchain components that
10203 match the target. Doing so creates a smaller than typical SDK and
10204 only contains a minimal set of components with which to build simple
Andrew Geisslerc926e172021-05-07 16:11:35 -050010205 test applications, as well as run the debugger::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010206
10207 $ bitbake meta-toolchain
10208
Andrew Geisslerc926e172021-05-07 16:11:35 -050010209 A final method is to build Gdb itself within the build system::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010210
10211 $ bitbake gdb-cross-<architecture>
10212
10213 Doing so produces a temporary copy of
10214 ``cross-gdb`` you can use for debugging during development. While
10215 this is the quickest approach, the two previous methods in this step
10216 are better when considering long-term maintenance strategies.
10217
10218 .. note::
10219
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010220 If you run ``bitbake gdb-cross``, the OpenEmbedded build system suggests
10221 the actual image (e.g. ``gdb-cross-i586``). The suggestion is usually the
10222 actual name you want to use.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010223
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500102244. *Set up the* ``debugfs``\ *:*
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010225
Andrew Geisslerc926e172021-05-07 16:11:35 -050010226 Run the following commands to set up the ``debugfs``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010227
10228 $ mkdir debugfs
10229 $ cd debugfs
Andrew Geisslerd5838332022-05-27 11:33:10 -050010230 $ tar xvfj build-dir/tmp/deploy/images/machine/image.rootfs.tar.bz2
10231 $ tar xvfj build-dir/tmp/deploy/images/machine/image-dbg.rootfs.tar.bz2
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010232
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500102335. *Set up GDB:*
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010234
10235 Install the SDK (if you built one) and then source the correct
10236 environment file. Sourcing the environment file puts the SDK in your
Andrew Geisslerd5838332022-05-27 11:33:10 -050010237 ``PATH`` environment variable and sets ``$GDB`` to the SDK's debugger.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010238
10239 If you are using the build system, Gdb is located in
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010240 `build-dir`\ ``/tmp/sysroots/``\ `host`\ ``/usr/bin/``\ `architecture`\ ``/``\ `architecture`\ ``-gdb``
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010241
102426. *Boot the target:*
10243
10244 For information on how to run QEMU, see the `QEMU
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010245 Documentation <https://wiki.qemu.org/Documentation/GettingStartedDevelopers>`__.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010246
10247 .. note::
10248
10249 Be sure to verify that your host can access the target via TCP.
10250
102517. *Debug a program:*
10252
10253 Debugging a program involves running gdbserver on the target and then
10254 running Gdb on the host. The example in this step debugs ``gzip``:
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010255
10256 .. code-block:: shell
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010257
10258 root@qemux86:~# gdbserver localhost:1234 /bin/gzip —help
10259
10260 For
10261 additional gdbserver options, see the `GDB Server
10262 Documentation <https://www.gnu.org/software/gdb/documentation/>`__.
10263
10264 After running gdbserver on the target, you need to run Gdb on the
Andrew Geisslerc926e172021-05-07 16:11:35 -050010265 host and configure it and connect to the target. Use these commands::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010266
10267 $ cd directory-holding-the-debugfs-directory
10268 $ arch-gdb
10269 (gdb) set sysroot debugfs
10270 (gdb) set substitute-path /usr/src/debug debugfs/usr/src/debug
10271 (gdb) target remote IP-of-target:1234
10272
10273 At this
10274 point, everything should automatically load (i.e. matching binaries,
10275 symbols and headers).
10276
10277 .. note::
10278
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010279 The Gdb ``set`` commands in the previous example can be placed into the
10280 users ``~/.gdbinit`` file. Upon starting, Gdb automatically runs whatever
10281 commands are in that file.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010282
102838. *Deploying without a full image rebuild:*
10284
10285 In many cases, during development you want a quick method to deploy a
10286 new binary to the target and debug it, without waiting for a full
10287 image build.
10288
10289 One approach to solving this situation is to just build the component
10290 you want to debug. Once you have built the component, copy the
10291 executable directly to both the target and the host ``debugfs``.
10292
10293 If the binary is processed through the debug splitting in
10294 OpenEmbedded, you should also copy the debug items (i.e. ``.debug``
10295 contents and corresponding ``/usr/src/debug`` files) from the work
Andrew Geisslerc926e172021-05-07 16:11:35 -050010296 directory. Here is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010297
10298 $ bitbake bash
10299 $ bitbake -c devshell bash
10300 $ cd ..
10301 $ scp packages-split/bash/bin/bash target:/bin/bash
10302 $ cp -a packages-split/bash-dbg/\* path/debugfs
10303
10304Debugging with the GNU Project Debugger (GDB) on the Target
10305-----------------------------------------------------------
10306
10307The previous section addressed using GDB remotely for debugging
10308purposes, which is the most usual case due to the inherent hardware
10309limitations on many embedded devices. However, debugging in the target
10310hardware itself is also possible with more powerful devices. This
10311section describes what you need to do in order to support using GDB to
10312debug on the target hardware.
10313
10314To support this kind of debugging, you need do the following:
10315
Andrew Geisslerd5838332022-05-27 11:33:10 -050010316- Ensure that GDB is on the target. You can do this by making
10317 the following addition to your ``local.conf`` file::
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010318
Andrew Geisslerd5838332022-05-27 11:33:10 -050010319 EXTRA_IMAGE_FEATURES:append = " tools-debug"
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010320
Andrew Geisslerd5838332022-05-27 11:33:10 -050010321- Ensure that debug symbols are present. You can do so by adding the
10322 corresponding ``-dbg`` package to :term:`IMAGE_INSTALL`::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010323
Andrew Geisslerd5838332022-05-27 11:33:10 -050010324 IMAGE_INSTALL:append = " packagename-dbg"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010325
Andrew Geisslerd5838332022-05-27 11:33:10 -050010326 Alternatively, you can add the following to ``local.conf`` to include
Andrew Geisslerc926e172021-05-07 16:11:35 -050010327 all the debug symbols::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010328
Andrew Geisslerd5838332022-05-27 11:33:10 -050010329 EXTRA_IMAGE_FEATURES:append = " dbg-pkgs"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010330
10331.. note::
10332
10333 To improve the debug information accuracy, you can reduce the level
10334 of optimization used by the compiler. For example, when adding the
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010335 following line to your ``local.conf`` file, you will reduce optimization
10336 from :term:`FULL_OPTIMIZATION` of "-O2" to :term:`DEBUG_OPTIMIZATION`
Andrew Geisslerc926e172021-05-07 16:11:35 -050010337 of "-O -fno-omit-frame-pointer"::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010338
10339 DEBUG_BUILD = "1"
10340
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010341 Consider that this will reduce the application's performance and is
10342 recommended only for debugging purposes.
10343
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010344Other Debugging Tips
10345--------------------
10346
10347Here are some other tips that you might find useful:
10348
10349- When adding new packages, it is worth watching for undesirable items
10350 making their way into compiler command lines. For example, you do not
10351 want references to local system files like ``/usr/lib/`` or
10352 ``/usr/include/``.
10353
10354- If you want to remove the ``psplash`` boot splashscreen, add
10355 ``psplash=false`` to the kernel command line. Doing so prevents
10356 ``psplash`` from loading and thus allows you to see the console. It
10357 is also possible to switch out of the splashscreen by switching the
10358 virtual console (e.g. Fn+Left or Fn+Right on a Zaurus).
10359
10360- Removing :term:`TMPDIR` (usually
10361 ``tmp/``, within the
10362 :term:`Build Directory`) can often fix
Andrew Geissler09036742021-06-25 14:25:14 -050010363 temporary build issues. Removing :term:`TMPDIR` is usually a relatively
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010364 cheap operation, because task output will be cached in
10365 :term:`SSTATE_DIR` (usually
10366 ``sstate-cache/``, which is also in the Build Directory).
10367
10368 .. note::
10369
Andrew Geissler09036742021-06-25 14:25:14 -050010370 Removing :term:`TMPDIR` might be a workaround rather than a fix.
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010371 Consequently, trying to determine the underlying cause of an issue before
10372 removing the directory is a good idea.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010373
10374- Understanding how a feature is used in practice within existing
10375 recipes can be very helpful. It is recommended that you configure
10376 some method that allows you to quickly search through files.
10377
10378 Using GNU Grep, you can use the following shell function to
10379 recursively search through common recipe-related files, skipping
10380 binary files, ``.git`` directories, and the Build Directory (assuming
Andrew Geisslerc926e172021-05-07 16:11:35 -050010381 its name starts with "build")::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010382
10383 g() {
10384 grep -Ir \
10385 --exclude-dir=.git \
10386 --exclude-dir='build*' \
10387 --include='*.bb*' \
10388 --include='*.inc*' \
10389 --include='*.conf*' \
10390 --include='*.py*' \
10391 "$@"
10392 }
10393
Andrew Geisslerc926e172021-05-07 16:11:35 -050010394 Following are some usage examples::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010395
10396 $ g FOO # Search recursively for "FOO"
10397 $ g -i foo # Search recursively for "foo", ignoring case
10398 $ g -w FOO # Search recursively for "FOO" as a word, ignoring e.g. "FOOBAR"
10399
10400 If figuring
10401 out how some feature works requires a lot of searching, it might
10402 indicate that the documentation should be extended or improved. In
10403 such cases, consider filing a documentation bug using the Yocto
10404 Project implementation of
10405 :yocto_bugs:`Bugzilla <>`. For information on
10406 how to submit a bug against the Yocto Project, see the Yocto Project
Andrew Geissler09209ee2020-12-13 08:44:15 -060010407 Bugzilla :yocto_wiki:`wiki page </Bugzilla_Configuration_and_Bug_Tracking>`
Andrew Geissler3b8a17c2021-04-15 15:55:55 -050010408 and the
10409 ":ref:`dev-manual/common-tasks:submitting a defect against the yocto project`"
10410 section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010411
10412 .. note::
10413
10414 The manuals might not be the right place to document variables
10415 that are purely internal and have a limited scope (e.g. internal
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010416 variables used to implement a single ``.bbclass`` file).
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010417
10418Making Changes to the Yocto Project
10419===================================
10420
10421Because the Yocto Project is an open-source, community-based project,
10422you can effect changes to the project. This section presents procedures
10423that show you how to submit a defect against the project and how to
10424submit a change.
10425
10426Submitting a Defect Against the Yocto Project
10427---------------------------------------------
10428
10429Use the Yocto Project implementation of
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010430`Bugzilla <https://www.bugzilla.org/about/>`__ to submit a defect (bug)
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010431against the Yocto Project. For additional information on this
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010432implementation of Bugzilla see the ":ref:`Yocto Project
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010433Bugzilla <resources-bugtracker>`" section in the
10434Yocto Project Reference Manual. For more detail on any of the following
10435steps, see the Yocto Project
Andrew Geissler09209ee2020-12-13 08:44:15 -060010436:yocto_wiki:`Bugzilla wiki page </Bugzilla_Configuration_and_Bug_Tracking>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010437
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010438Use the following general steps to submit a bug:
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010439
104401. Open the Yocto Project implementation of :yocto_bugs:`Bugzilla <>`.
10441
104422. Click "File a Bug" to enter a new bug.
10443
104443. Choose the appropriate "Classification", "Product", and "Component"
10445 for which the bug was found. Bugs for the Yocto Project fall into
10446 one of several classifications, which in turn break down into
10447 several products and components. For example, for a bug against the
10448 ``meta-intel`` layer, you would choose "Build System, Metadata &
10449 Runtime", "BSPs", and "bsps-meta-intel", respectively.
10450
104514. Choose the "Version" of the Yocto Project for which you found the
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010452 bug (e.g. &DISTRO;).
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010453
104545. Determine and select the "Severity" of the bug. The severity
10455 indicates how the bug impacted your work.
10456
104576. Choose the "Hardware" that the bug impacts.
10458
104597. Choose the "Architecture" that the bug impacts.
10460
104618. Choose a "Documentation change" item for the bug. Fixing a bug might
10462 or might not affect the Yocto Project documentation. If you are
10463 unsure of the impact to the documentation, select "Don't Know".
10464
104659. Provide a brief "Summary" of the bug. Try to limit your summary to
10466 just a line or two and be sure to capture the essence of the bug.
10467
1046810. Provide a detailed "Description" of the bug. You should provide as
10469 much detail as you can about the context, behavior, output, and so
10470 forth that surrounds the bug. You can even attach supporting files
10471 for output from logs by using the "Add an attachment" button.
10472
1047311. Click the "Submit Bug" button submit the bug. A new Bugzilla number
10474 is assigned to the bug and the defect is logged in the bug tracking
10475 system.
10476
10477Once you file a bug, the bug is processed by the Yocto Project Bug
10478Triage Team and further details concerning the bug are assigned (e.g.
10479priority and owner). You are the "Submitter" of the bug and any further
10480categorization, progress, or comments on the bug result in Bugzilla
10481sending you an automated email concerning the particular change or
10482progress to the bug.
10483
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010484Submitting a Change to the Yocto Project
10485----------------------------------------
10486
10487Contributions to the Yocto Project and OpenEmbedded are very welcome.
10488Because the system is extremely configurable and flexible, we recognize
10489that developers will want to extend, configure or optimize it for their
10490specific uses.
10491
10492The Yocto Project uses a mailing list and a patch-based workflow that is
10493similar to the Linux kernel but contains important differences. In
William A. Kennington IIIac69b482021-06-02 12:28:27 -070010494general, there is a mailing list through which you can submit patches. You
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010495should send patches to the appropriate mailing list so that they can be
10496reviewed and merged by the appropriate maintainer. The specific mailing
10497list you need to use depends on the location of the code you are
10498changing. Each component (e.g. layer) should have a ``README`` file that
10499indicates where to send the changes and which process to follow.
10500
10501You can send the patch to the mailing list using whichever approach you
10502feel comfortable with to generate the patch. Once sent, the patch is
10503usually reviewed by the community at large. If somebody has concerns
10504with the patch, they will usually voice their concern over the mailing
10505list. If a patch does not receive any negative reviews, the maintainer
10506of the affected layer typically takes the patch, tests it, and then
10507based on successful testing, merges the patch.
10508
10509The "poky" repository, which is the Yocto Project's reference build
10510environment, is a hybrid repository that contains several individual
10511pieces (e.g. BitBake, Metadata, documentation, and so forth) built using
10512the combo-layer tool. The upstream location used for submitting changes
10513varies by component:
10514
10515- *Core Metadata:* Send your patch to the
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010516 :oe_lists:`openembedded-core </g/openembedded-core>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010517 mailing list. For example, a change to anything under the ``meta`` or
10518 ``scripts`` directories should be sent to this mailing list.
10519
10520- *BitBake:* For changes to BitBake (i.e. anything under the
10521 ``bitbake`` directory), send your patch to the
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010522 :oe_lists:`bitbake-devel </g/bitbake-devel>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010523 mailing list.
10524
Andrew Geisslerc3d88e42020-10-02 09:45:00 -050010525- *"meta-\*" trees:* These trees contain Metadata. Use the
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010526 :yocto_lists:`poky </g/poky>` mailing list.
Andrew Geisslerc3d88e42020-10-02 09:45:00 -050010527
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010528- *Documentation*: For changes to the Yocto Project documentation, use the
10529 :yocto_lists:`docs </g/docs>` mailing list.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010530
10531For changes to other layers hosted in the Yocto Project source
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010532repositories (i.e. ``yoctoproject.org``) and tools use the
10533:yocto_lists:`Yocto Project </g/yocto/>` general mailing list.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010534
10535.. note::
10536
10537 Sometimes a layer's documentation specifies to use a particular
10538 mailing list. If so, use that list.
10539
10540For additional recipes that do not fit into the core Metadata, you
10541should determine which layer the recipe should go into and submit the
10542change in the manner recommended by the documentation (e.g. the
10543``README`` file) supplied with the layer. If in doubt, please ask on the
10544Yocto general mailing list or on the openembedded-devel mailing list.
10545
10546You can also push a change upstream and request a maintainer to pull the
10547change into the component's upstream repository. You do this by pushing
Andrew Geissler09209ee2020-12-13 08:44:15 -060010548to a contribution repository that is upstream. See the
10549":ref:`overview-manual/development-environment:git workflows and the yocto project`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010550section in the Yocto Project Overview and Concepts Manual for additional
10551concepts on working in the Yocto Project development environment.
10552
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010553Maintainers commonly use ``-next`` branches to test submissions prior to
10554merging patches. Thus, you can get an idea of the status of a patch based on
10555whether the patch has been merged into one of these branches. The commonly
10556used testing branches for OpenEmbedded-Core are as follows:
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010557
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010558- *openembedded-core "master-next" branch:* This branch is part of the
10559 :oe_git:`openembedded-core </openembedded-core/>` repository and contains
10560 proposed changes to the core metadata.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010561
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010562- *poky "master-next" branch:* This branch is part of the
Andrew Geissler09209ee2020-12-13 08:44:15 -060010563 :yocto_git:`poky </poky/>` repository and combines proposed
Andrew Geisslerd5838332022-05-27 11:33:10 -050010564 changes to BitBake, the core metadata and the poky distro.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010565
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010566Similarly, stable branches maintained by the project may have corresponding
10567``-next`` branches which collect proposed changes. For example,
10568``&DISTRO_NAME_NO_CAP;-next`` and ``&DISTRO_NAME_NO_CAP_MINUS_ONE;-next``
10569branches in both the "openembdedded-core" and "poky" repositories.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010570
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010571Other layers may have similar testing branches but there is no formal
10572requirement or standard for these so please check the documentation for the
10573layers you are contributing to.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010574
10575The following sections provide procedures for submitting a change.
10576
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010577Preparing Changes for Submission
10578~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010579
105801. *Make Your Changes Locally:* Make your changes in your local Git
10581 repository. You should make small, controlled, isolated changes.
10582 Keeping changes small and isolated aids review, makes
10583 merging/rebasing easier and keeps the change history clean should
10584 anyone need to refer to it in future.
10585
105862. *Stage Your Changes:* Stage your changes by using the ``git add``
10587 command on each file you changed.
10588
105893. *Commit Your Changes:* Commit the change by using the ``git commit``
10590 command. Make sure your commit information follows standards by
10591 following these accepted conventions:
10592
10593 - Be sure to include a "Signed-off-by:" line in the same style as
Patrick Williams03907ee2022-05-01 06:28:52 -050010594 required by the Linux kernel. This can be done by using the
10595 ``git commit -s`` command. Adding this line signifies that you,
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010596 the submitter, have agreed to the Developer's Certificate of
10597 Origin 1.1 as follows:
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010598
10599 .. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010600
10601 Developer's Certificate of Origin 1.1
10602
10603 By making a contribution to this project, I certify that:
10604
10605 (a) The contribution was created in whole or in part by me and I
10606 have the right to submit it under the open source license
10607 indicated in the file; or
10608
10609 (b) The contribution is based upon previous work that, to the best
10610 of my knowledge, is covered under an appropriate open source
10611 license and I have the right under that license to submit that
10612 work with modifications, whether created in whole or in part
10613 by me, under the same open source license (unless I am
10614 permitted to submit under a different license), as indicated
10615 in the file; or
10616
10617 (c) The contribution was provided directly to me by some other
10618 person who certified (a), (b) or (c) and I have not modified
10619 it.
10620
10621 (d) I understand and agree that this project and the contribution
10622 are public and that a record of the contribution (including all
10623 personal information I submit with it, including my sign-off) is
10624 maintained indefinitely and may be redistributed consistent with
10625 this project or the open source license(s) involved.
10626
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010627 - Provide a single-line summary of the change and, if more
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010628 explanation is needed, provide more detail in the body of the
10629 commit. This summary is typically viewable in the "shortlist" of
10630 changes. Thus, providing something short and descriptive that
10631 gives the reader a summary of the change is useful when viewing a
10632 list of many commits. You should prefix this short description
10633 with the recipe name (if changing a recipe), or else with the
10634 short form path to the file being changed.
10635
10636 - For the body of the commit message, provide detailed information
10637 that describes what you changed, why you made the change, and the
10638 approach you used. It might also be helpful if you mention how you
10639 tested the change. Provide as much detail as you can in the body
10640 of the commit message.
10641
10642 .. note::
10643
10644 You do not need to provide a more detailed explanation of a
10645 change if the change is minor to the point of the single line
10646 summary providing all the information.
10647
10648 - If the change addresses a specific bug or issue that is associated
10649 with a bug-tracking ID, include a reference to that ID in your
10650 detailed description. For example, the Yocto Project uses a
Andrew Geissler615f2f12022-07-15 14:00:58 -050010651 specific convention for bug references --- any commit that addresses
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010652 a specific bug should use the following form for the detailed
10653 description. Be sure to use the actual bug-tracking ID from
Andrew Geisslerc926e172021-05-07 16:11:35 -050010654 Bugzilla for bug-id::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010655
10656 Fixes [YOCTO #bug-id]
10657
10658 detailed description of change
10659
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010660Using Email to Submit a Patch
10661~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10662
10663Depending on the components changed, you need to submit the email to a
10664specific mailing list. For some guidance on which mailing list to use,
Andrew Geissler3b8a17c2021-04-15 15:55:55 -050010665see the
10666:ref:`list <dev-manual/common-tasks:submitting a change to the yocto project>`
10667at the beginning of this section. For a description of all the available
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010668mailing lists, see the ":ref:`Mailing Lists <resources-mailinglist>`" section in the
10669Yocto Project Reference Manual.
10670
10671Here is the general procedure on how to submit a patch through email
10672without using the scripts once the steps in
Andrew Geissler09209ee2020-12-13 08:44:15 -060010673:ref:`dev-manual/common-tasks:preparing changes for submission` have been followed:
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010674
106751. *Format the Commit:* Format the commit into an email message. To
10676 format commits, use the ``git format-patch`` command. When you
10677 provide the command, you must include a revision list or a number of
10678 patches as part of the command. For example, either of these two
10679 commands takes your most recent single commit and formats it as an
Andrew Geisslerc926e172021-05-07 16:11:35 -050010680 email message in the current directory::
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010681
10682 $ git format-patch -1
10683
10684 or ::
10685
10686 $ git format-patch HEAD~
10687
10688 After the command is run, the current directory contains a numbered
10689 ``.patch`` file for the commit.
10690
10691 If you provide several commits as part of the command, the
10692 ``git format-patch`` command produces a series of numbered files in
10693 the current directory – one for each commit. If you have more than
10694 one patch, you should also use the ``--cover`` option with the
10695 command, which generates a cover letter as the first "patch" in the
10696 series. You can then edit the cover letter to provide a description
10697 for the series of patches. For information on the
10698 ``git format-patch`` command, see ``GIT_FORMAT_PATCH(1)`` displayed
10699 using the ``man git-format-patch`` command.
10700
10701 .. note::
10702
10703 If you are or will be a frequent contributor to the Yocto Project
10704 or to OpenEmbedded, you might consider requesting a contrib area
10705 and the necessary associated rights.
10706
107072. *Send the patches via email:* Send the patches to the recipients and
10708 relevant mailing lists by using the ``git send-email`` command.
10709
10710 .. note::
10711
10712 In order to use ``git send-email``, you must have the proper Git packages
10713 installed on your host.
10714 For Ubuntu, Debian, and Fedora the package is ``git-email``.
10715
10716 The ``git send-email`` command sends email by using a local or remote
10717 Mail Transport Agent (MTA) such as ``msmtp``, ``sendmail``, or
10718 through a direct ``smtp`` configuration in your Git ``~/.gitconfig``
10719 file. If you are submitting patches through email only, it is very
10720 important that you submit them without any whitespace or HTML
10721 formatting that either you or your mailer introduces. The maintainer
10722 that receives your patches needs to be able to save and apply them
10723 directly from your emails. A good way to verify that what you are
10724 sending will be applicable by the maintainer is to do a dry run and
10725 send them to yourself and then save and apply them as the maintainer
10726 would.
10727
10728 The ``git send-email`` command is the preferred method for sending
10729 your patches using email since there is no risk of compromising
10730 whitespace in the body of the message, which can occur when you use
10731 your own mail client. The command also has several options that let
10732 you specify recipients and perform further editing of the email
10733 message. For information on how to use the ``git send-email``
10734 command, see ``GIT-SEND-EMAIL(1)`` displayed using the
10735 ``man git-send-email`` command.
10736
10737The Yocto Project uses a `Patchwork instance <https://patchwork.openembedded.org/>`__
10738to track the status of patches submitted to the various mailing lists and to
10739support automated patch testing. Each submitted patch is checked for common
10740mistakes and deviations from the expected patch format and submitters are
10741notified by patchtest if such mistakes are found. This process helps to
10742reduce the burden of patch review on maintainers.
10743
10744.. note::
10745
10746 This system is imperfect and changes can sometimes get lost in the flow.
10747 Asking about the status of a patch or change is reasonable if the change
10748 has been idle for a while with no feedback.
10749
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010750Using Scripts to Push a Change Upstream and Request a Pull
10751~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10752
10753For larger patch series it is preferable to send a pull request which not
10754only includes the patch but also a pointer to a branch that can be pulled
10755from. This involves making a local branch for your changes, pushing this
10756branch to an accessible repository and then using the ``create-pull-request``
10757and ``send-pull-request`` scripts from openembedded-core to create and send a
10758patch series with a link to the branch for review.
10759
10760Follow this procedure to push a change to an upstream "contrib" Git
Andrew Geissler09209ee2020-12-13 08:44:15 -060010761repository once the steps in :ref:`dev-manual/common-tasks:preparing changes for submission` have
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010762been followed:
10763
10764.. note::
10765
10766 You can find general Git information on how to push a change upstream
10767 in the
10768 `Git Community Book <https://git-scm.com/book/en/v2/Distributed-Git-Distributed-Workflows>`__.
10769
107701. *Push Your Commits to a "Contrib" Upstream:* If you have arranged for
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010771 permissions to push to an upstream contrib repository, push the
Andrew Geisslerc926e172021-05-07 16:11:35 -050010772 change to that repository::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010773
10774 $ git push upstream_remote_repo local_branch_name
10775
10776 For example, suppose you have permissions to push
10777 into the upstream ``meta-intel-contrib`` repository and you are
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010778 working in a local branch named `your_name`\ ``/README``. The following
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010779 command pushes your local commits to the ``meta-intel-contrib``
10780 upstream repository and puts the commit in a branch named
Andrew Geisslerc926e172021-05-07 16:11:35 -050010781 `your_name`\ ``/README``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010782
10783 $ git push meta-intel-contrib your_name/README
10784
Andrew Geissler6ce62a22020-11-30 19:58:47 -0600107852. *Determine Who to Notify:* Determine the maintainer or the mailing
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010786 list that you need to notify for the change.
10787
10788 Before submitting any change, you need to be sure who the maintainer
10789 is or what mailing list that you need to notify. Use either these
10790 methods to find out:
10791
10792 - *Maintenance File:* Examine the ``maintainers.inc`` file, which is
10793 located in the :term:`Source Directory` at
10794 ``meta/conf/distro/include``, to see who is responsible for code.
10795
Andrew Geissler09209ee2020-12-13 08:44:15 -060010796 - *Search by File:* Using :ref:`overview-manual/development-environment:git`, you can
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010797 enter the following command to bring up a short list of all
Andrew Geisslerc926e172021-05-07 16:11:35 -050010798 commits against a specific file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010799
10800 git shortlog -- filename
10801
10802 Just provide the name of the file for which you are interested. The
10803 information returned is not ordered by history but does include a
10804 list of everyone who has committed grouped by name. From the list,
10805 you can see who is responsible for the bulk of the changes against
10806 the file.
10807
10808 - *Examine the List of Mailing Lists:* For a list of the Yocto
10809 Project and related mailing lists, see the ":ref:`Mailing
10810 lists <resources-mailinglist>`" section in
10811 the Yocto Project Reference Manual.
10812
Andrew Geissler6ce62a22020-11-30 19:58:47 -0600108133. *Make a Pull Request:* Notify the maintainer or the mailing list that
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010814 you have pushed a change by making a pull request.
10815
10816 The Yocto Project provides two scripts that conveniently let you
10817 generate and send pull requests to the Yocto Project. These scripts
10818 are ``create-pull-request`` and ``send-pull-request``. You can find
10819 these scripts in the ``scripts`` directory within the
10820 :term:`Source Directory` (e.g.
Andrew Geissler95ac1b82021-03-31 14:34:31 -050010821 ``poky/scripts``).
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010822
10823 Using these scripts correctly formats the requests without
10824 introducing any whitespace or HTML formatting. The maintainer that
10825 receives your patches either directly or through the mailing list
10826 needs to be able to save and apply them directly from your emails.
10827 Using these scripts is the preferred method for sending patches.
10828
10829 First, create the pull request. For example, the following command
10830 runs the script, specifies the upstream repository in the contrib
10831 directory into which you pushed the change, and provides a subject
Andrew Geisslerc926e172021-05-07 16:11:35 -050010832 line in the created patch files::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010833
Andrew Geissler95ac1b82021-03-31 14:34:31 -050010834 $ poky/scripts/create-pull-request -u meta-intel-contrib -s "Updated Manual Section Reference in README"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010835
10836 Running this script forms ``*.patch`` files in a folder named
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010837 ``pull-``\ `PID` in the current directory. One of the patch files is a
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010838 cover letter.
10839
10840 Before running the ``send-pull-request`` script, you must edit the
10841 cover letter patch to insert information about your change. After
10842 editing the cover letter, send the pull request. For example, the
10843 following command runs the script and specifies the patch directory
10844 and email address. In this example, the email address is a mailing
Andrew Geisslerc926e172021-05-07 16:11:35 -050010845 list::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010846
Andrew Geissler5199d832021-09-24 16:47:35 -050010847 $ poky/scripts/send-pull-request -p ~/meta-intel/pull-10565 -t meta-intel@lists.yoctoproject.org
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010848
10849 You need to follow the prompts as the script is interactive.
10850
10851 .. note::
10852
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010853 For help on using these scripts, simply provide the ``-h``
Andrew Geisslerc926e172021-05-07 16:11:35 -050010854 argument as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010855
10856 $ poky/scripts/create-pull-request -h
10857 $ poky/scripts/send-pull-request -h
10858
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010859Responding to Patch Review
10860~~~~~~~~~~~~~~~~~~~~~~~~~~
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010861
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010862You may get feedback on your submitted patches from other community members
10863or from the automated patchtest service. If issues are identified in your
10864patch then it is usually necessary to address these before the patch will be
10865accepted into the project. In this case you should amend the patch according
10866to the feedback and submit an updated version to the relevant mailing list,
10867copying in the reviewers who provided feedback to the previous version of the
10868patch.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010869
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010870The patch should be amended using ``git commit --amend`` or perhaps ``git
10871rebase`` for more expert git users. You should also modify the ``[PATCH]``
10872tag in the email subject line when sending the revised patch to mark the new
10873iteration as ``[PATCH v2]``, ``[PATCH v3]``, etc as appropriate. This can be
10874done by passing the ``-v`` argument to ``git format-patch`` with a version
10875number.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010876
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010877Lastly please ensure that you also test your revised changes. In particular
10878please don't just edit the patch file written out by ``git format-patch`` and
10879resend it.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010880
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010881Submitting Changes to Stable Release Branches
10882~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010883
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010884The process for proposing changes to a Yocto Project stable branch differs
10885from the steps described above. Changes to a stable branch must address
10886identified bugs or CVEs and should be made carefully in order to avoid the
10887risk of introducing new bugs or breaking backwards compatibility. Typically
10888bug fixes must already be accepted into the master branch before they can be
10889backported to a stable branch unless the bug in question does not affect the
10890master branch or the fix on the master branch is unsuitable for backporting.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010891
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010892The list of stable branches along with the status and maintainer for each
10893branch can be obtained from the
Andrew Geissler09209ee2020-12-13 08:44:15 -060010894:yocto_wiki:`Releases wiki page </Releases>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010895
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010896.. note::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010897
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010898 Changes will not typically be accepted for branches which are marked as
10899 End-Of-Life (EOL).
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010900
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010901With this in mind, the steps to submit a change for a stable branch are as
10902follows:
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010903
Andrew Geissler6ce62a22020-11-30 19:58:47 -0600109041. *Identify the bug or CVE to be fixed:* This information should be
10905 collected so that it can be included in your submission.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010906
Patrick Williams213cb262021-08-07 19:21:33 -050010907 See :ref:`dev-manual/common-tasks:checking for vulnerabilities`
10908 for details about CVE tracking.
10909
Andrew Geissler6ce62a22020-11-30 19:58:47 -0600109102. *Check if the fix is already present in the master branch:* This will
10911 result in the most straightforward path into the stable branch for the
10912 fix.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010913
Andrew Geissler615f2f12022-07-15 14:00:58 -050010914 a. *If the fix is present in the master branch --- submit a backport request
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010915 by email:* You should send an email to the relevant stable branch
10916 maintainer and the mailing list with details of the bug or CVE to be
10917 fixed, the commit hash on the master branch that fixes the issue and
10918 the stable branches which you would like this fix to be backported to.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010919
Andrew Geissler615f2f12022-07-15 14:00:58 -050010920 b. *If the fix is not present in the master branch --- submit the fix to the
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010921 master branch first:* This will ensure that the fix passes through the
10922 project's usual patch review and test processes before being accepted.
10923 It will also ensure that bugs are not left unresolved in the master
10924 branch itself. Once the fix is accepted in the master branch a backport
10925 request can be submitted as above.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010926
Andrew Geissler615f2f12022-07-15 14:00:58 -050010927 c. *If the fix is unsuitable for the master branch --- submit a patch
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010928 directly for the stable branch:* This method should be considered as a
10929 last resort. It is typically necessary when the master branch is using
10930 a newer version of the software which includes an upstream fix for the
10931 issue or when the issue has been fixed on the master branch in a way
10932 that introduces backwards incompatible changes. In this case follow the
Andrew Geissler09209ee2020-12-13 08:44:15 -060010933 steps in :ref:`dev-manual/common-tasks:preparing changes for submission` and
10934 :ref:`dev-manual/common-tasks:using email to submit a patch` but modify the subject header of your patch
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010935 email to include the name of the stable branch which you are
10936 targetting. This can be done using the ``--subject-prefix`` argument to
10937 ``git format-patch``, for example to submit a patch to the dunfell
10938 branch use
10939 ``git format-patch --subject-prefix='&DISTRO_NAME_NO_CAP_MINUS_ONE;][PATCH' ...``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010940
10941Working With Licenses
10942=====================
10943
Andrew Geissler09209ee2020-12-13 08:44:15 -060010944As mentioned in the ":ref:`overview-manual/development-environment:licensing`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010945section in the Yocto Project Overview and Concepts Manual, open source
10946projects are open to the public and they consequently have different
10947licensing structures in place. This section describes the mechanism by
10948which the :term:`OpenEmbedded Build System`
10949tracks changes to
10950licensing text and covers how to maintain open source license compliance
10951during your project's lifecycle. The section also describes how to
10952enable commercially licensed recipes, which by default are disabled.
10953
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010954Tracking License Changes
10955------------------------
10956
10957The license of an upstream project might change in the future. In order
10958to prevent these changes going unnoticed, the
10959:term:`LIC_FILES_CHKSUM`
10960variable tracks changes to the license text. The checksums are validated
10961at the end of the configure step, and if the checksums do not match, the
10962build will fail.
10963
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010964Specifying the ``LIC_FILES_CHKSUM`` Variable
10965~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10966
Andrew Geissler09036742021-06-25 14:25:14 -050010967The :term:`LIC_FILES_CHKSUM` variable contains checksums of the license text
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010968in the source code for the recipe. Following is an example of how to
Andrew Geissler09036742021-06-25 14:25:14 -050010969specify :term:`LIC_FILES_CHKSUM`::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010970
10971 LIC_FILES_CHKSUM = "file://COPYING;md5=xxxx \
10972 file://licfile1.txt;beginline=5;endline=29;md5=yyyy \
10973 file://licfile2.txt;endline=50;md5=zzzz \
10974 ..."
10975
10976.. note::
10977
10978 - When using "beginline" and "endline", realize that line numbering
10979 begins with one and not zero. Also, the included lines are
10980 inclusive (i.e. lines five through and including 29 in the
10981 previous example for ``licfile1.txt``).
10982
10983 - When a license check fails, the selected license text is included
10984 as part of the QA message. Using this output, you can determine
10985 the exact start and finish for the needed license text.
10986
10987The build system uses the :term:`S`
10988variable as the default directory when searching files listed in
Andrew Geissler09036742021-06-25 14:25:14 -050010989:term:`LIC_FILES_CHKSUM`. The previous example employs the default
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010990directory.
10991
Andrew Geisslerc926e172021-05-07 16:11:35 -050010992Consider this next example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010993
10994 LIC_FILES_CHKSUM = "file://src/ls.c;beginline=5;endline=16;\
10995 md5=bb14ed3c4cda583abc85401304b5cd4e"
10996 LIC_FILES_CHKSUM = "file://${WORKDIR}/license.html;md5=5c94767cedb5d6987c902ac850ded2c6"
10997
10998The first line locates a file in ``${S}/src/ls.c`` and isolates lines
10999five through 16 as license text. The second line refers to a file in
11000:term:`WORKDIR`.
11001
Andrew Geissler09036742021-06-25 14:25:14 -050011002Note that :term:`LIC_FILES_CHKSUM` variable is mandatory for all recipes,
11003unless the :term:`LICENSE` variable is set to "CLOSED".
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011004
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011005Explanation of Syntax
11006~~~~~~~~~~~~~~~~~~~~~
11007
Andrew Geissler09036742021-06-25 14:25:14 -050011008As mentioned in the previous section, the :term:`LIC_FILES_CHKSUM` variable
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011009lists all the important files that contain the license text for the
11010source code. It is possible to specify a checksum for an entire file, or
11011a specific section of a file (specified by beginning and ending line
11012numbers with the "beginline" and "endline" parameters, respectively).
11013The latter is useful for source files with a license notice header,
11014README documents, and so forth. If you do not use the "beginline"
11015parameter, then it is assumed that the text begins on the first line of
11016the file. Similarly, if you do not use the "endline" parameter, it is
11017assumed that the license text ends with the last line of the file.
11018
11019The "md5" parameter stores the md5 checksum of the license text. If the
11020license text changes in any way as compared to this parameter then a
11021mismatch occurs. This mismatch triggers a build failure and notifies the
11022developer. Notification allows the developer to review and address the
11023license text changes. Also note that if a mismatch occurs during the
11024build, the correct md5 checksum is placed in the build log and can be
11025easily copied to the recipe.
11026
11027There is no limit to how many files you can specify using the
Andrew Geissler09036742021-06-25 14:25:14 -050011028:term:`LIC_FILES_CHKSUM` variable. Generally, however, every project
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011029requires a few specifications for license tracking. Many projects have a
11030"COPYING" file that stores the license information for all the source
11031code files. This practice allows you to just track the "COPYING" file as
11032long as it is kept up to date.
11033
11034.. note::
11035
11036 - If you specify an empty or invalid "md5" parameter,
11037 :term:`BitBake` returns an md5
11038 mis-match error and displays the correct "md5" parameter value
11039 during the build. The correct parameter is also captured in the
11040 build log.
11041
11042 - If the whole file contains only license text, you do not need to
11043 use the "beginline" and "endline" parameters.
11044
11045Enabling Commercially Licensed Recipes
11046--------------------------------------
11047
11048By default, the OpenEmbedded build system disables components that have
11049commercial or other special licensing requirements. Such requirements
11050are defined on a recipe-by-recipe basis through the
11051:term:`LICENSE_FLAGS` variable
11052definition in the affected recipe. For instance, the
11053``poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly`` recipe
Andrew Geisslerc926e172021-05-07 16:11:35 -050011054contains the following statement::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011055
11056 LICENSE_FLAGS = "commercial"
11057
11058Here is a
11059slightly more complicated example that contains both an explicit recipe
Andrew Geisslerc926e172021-05-07 16:11:35 -050011060name and version (after variable expansion)::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011061
11062 LICENSE_FLAGS = "license_${PN}_${PV}"
11063
11064In order for a component restricted by a
Andrew Geissler09036742021-06-25 14:25:14 -050011065:term:`LICENSE_FLAGS` definition to be enabled and included in an image, it
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011066needs to have a matching entry in the global
Andrew Geissler9aee5002022-03-30 16:27:02 +000011067:term:`LICENSE_FLAGS_ACCEPTED`
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011068variable, which is a variable typically defined in your ``local.conf``
11069file. For example, to enable the
11070``poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly`` package, you
11071could add either the string "commercial_gst-plugins-ugly" or the more
Andrew Geissler9aee5002022-03-30 16:27:02 +000011072general string "commercial" to :term:`LICENSE_FLAGS_ACCEPTED`. See the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -050011073":ref:`dev-manual/common-tasks:license flag matching`" section for a full
Andrew Geissler09036742021-06-25 14:25:14 -050011074explanation of how :term:`LICENSE_FLAGS` matching works. Here is the
Andrew Geisslerc926e172021-05-07 16:11:35 -050011075example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011076
Andrew Geissler9aee5002022-03-30 16:27:02 +000011077 LICENSE_FLAGS_ACCEPTED = "commercial_gst-plugins-ugly"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011078
11079Likewise, to additionally enable the package built from the recipe
11080containing ``LICENSE_FLAGS = "license_${PN}_${PV}"``, and assuming that
11081the actual recipe name was ``emgd_1.10.bb``, the following string would
11082enable that package as well as the original ``gst-plugins-ugly``
Andrew Geisslerc926e172021-05-07 16:11:35 -050011083package::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011084
Andrew Geissler9aee5002022-03-30 16:27:02 +000011085 LICENSE_FLAGS_ACCEPTED = "commercial_gst-plugins-ugly license_emgd_1.10"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011086
11087As a convenience, you do not need to specify the
Andrew Geissler595f6302022-01-24 19:11:47 +000011088complete license string for every package. You can use
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011089an abbreviated form, which consists of just the first portion or
11090portions of the license string before the initial underscore character
11091or characters. A partial string will match any license that contains the
11092given string as the first portion of its license. For example, the
Andrew Geissler595f6302022-01-24 19:11:47 +000011093following value will also match both of the packages
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011094previously mentioned as well as any other packages that have licenses
11095starting with "commercial" or "license".
11096::
11097
Andrew Geissler9aee5002022-03-30 16:27:02 +000011098 LICENSE_FLAGS_ACCEPTED = "commercial license"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011099
11100License Flag Matching
11101~~~~~~~~~~~~~~~~~~~~~
11102
11103License flag matching allows you to control what recipes the
11104OpenEmbedded build system includes in the build. Fundamentally, the
Andrew Geissler09036742021-06-25 14:25:14 -050011105build system attempts to match :term:`LICENSE_FLAGS` strings found in
Andrew Geissler9aee5002022-03-30 16:27:02 +000011106recipes against strings found in :term:`LICENSE_FLAGS_ACCEPTED`.
Andrew Geissler595f6302022-01-24 19:11:47 +000011107A match causes the build system to include a recipe in the
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011108build, while failure to find a match causes the build system to exclude
11109a recipe.
11110
11111In general, license flag matching is simple. However, understanding some
11112concepts will help you correctly and effectively use matching.
11113
11114Before a flag defined by a particular recipe is tested against the
Andrew Geissler9aee5002022-03-30 16:27:02 +000011115entries of :term:`LICENSE_FLAGS_ACCEPTED`, the expanded
Andrew Geissler595f6302022-01-24 19:11:47 +000011116string ``_${PN}`` is appended to the flag. This expansion makes each
11117:term:`LICENSE_FLAGS` value recipe-specific. After expansion, the
11118string is then matched against the entries. Thus, specifying
11119``LICENSE_FLAGS = "commercial"`` in recipe "foo", for example, results
11120in the string ``"commercial_foo"``. And, to create a match, that string
Andrew Geissler9aee5002022-03-30 16:27:02 +000011121must appear among the entries of :term:`LICENSE_FLAGS_ACCEPTED`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011122
Andrew Geissler09036742021-06-25 14:25:14 -050011123Judicious use of the :term:`LICENSE_FLAGS` strings and the contents of the
Andrew Geissler9aee5002022-03-30 16:27:02 +000011124:term:`LICENSE_FLAGS_ACCEPTED` variable allows you a lot of flexibility for
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011125including or excluding recipes based on licensing. For example, you can
11126broaden the matching capabilities by using license flags string subsets
Andrew Geissler9aee5002022-03-30 16:27:02 +000011127in :term:`LICENSE_FLAGS_ACCEPTED`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011128
11129.. note::
11130
11131 When using a string subset, be sure to use the part of the expanded
11132 string that precedes the appended underscore character (e.g.
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011133 ``usethispart_1.3``, ``usethispart_1.4``, and so forth).
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011134
Andrew Geissler595f6302022-01-24 19:11:47 +000011135For example, simply specifying the string "commercial" in the
Andrew Geissler9aee5002022-03-30 16:27:02 +000011136:term:`LICENSE_FLAGS_ACCEPTED` variable matches any expanded
Andrew Geissler595f6302022-01-24 19:11:47 +000011137:term:`LICENSE_FLAGS` definition that starts with the string
11138"commercial" such as "commercial_foo" and "commercial_bar", which
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011139are the strings the build system automatically generates for
11140hypothetical recipes named "foo" and "bar" assuming those recipes simply
Andrew Geisslerc926e172021-05-07 16:11:35 -050011141specify the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011142
11143 LICENSE_FLAGS = "commercial"
11144
Andrew Geissler595f6302022-01-24 19:11:47 +000011145Thus, you can choose to exhaustively enumerate each license flag in the
11146list and allow only specific recipes into the image, or you can use a
11147string subset that causes a broader range of matches to allow a range of
11148recipes into the image.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011149
Andrew Geissler09036742021-06-25 14:25:14 -050011150This scheme works even if the :term:`LICENSE_FLAGS` string already has
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011151``_${PN}`` appended. For example, the build system turns the license
11152flag "commercial_1.2_foo" into "commercial_1.2_foo_foo" and would match
11153both the general "commercial" and the specific "commercial_1.2_foo"
Andrew Geissler9aee5002022-03-30 16:27:02 +000011154strings found in the :term:`LICENSE_FLAGS_ACCEPTED` variable, as expected.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011155
11156Here are some other scenarios:
11157
11158- You can specify a versioned string in the recipe such as
11159 "commercial_foo_1.2" in a "foo" recipe. The build system expands this
11160 string to "commercial_foo_1.2_foo". Combine this license flag with a
Andrew Geissler9aee5002022-03-30 16:27:02 +000011161 :term:`LICENSE_FLAGS_ACCEPTED` variable that has the string
Andrew Geissler595f6302022-01-24 19:11:47 +000011162 "commercial" and you match the flag along with any other flag that
11163 starts with the string "commercial".
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011164
Andrew Geissler595f6302022-01-24 19:11:47 +000011165- Under the same circumstances, you can add "commercial_foo" in the
Andrew Geissler9aee5002022-03-30 16:27:02 +000011166 :term:`LICENSE_FLAGS_ACCEPTED` variable and the build system not only
Andrew Geissler595f6302022-01-24 19:11:47 +000011167 matches "commercial_foo_1.2" but also matches any license flag with
11168 the string "commercial_foo", regardless of the version.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011169
11170- You can be very specific and use both the package and version parts
Andrew Geissler9aee5002022-03-30 16:27:02 +000011171 in the :term:`LICENSE_FLAGS_ACCEPTED` list (e.g.
Andrew Geissler595f6302022-01-24 19:11:47 +000011172 "commercial_foo_1.2") to specifically match a versioned recipe.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011173
11174Other Variables Related to Commercial Licenses
11175~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
11176
William A. Kennington IIIac69b482021-06-02 12:28:27 -070011177There are other helpful variables related to commercial license handling,
11178defined in the
Andrew Geisslerc926e172021-05-07 16:11:35 -050011179``poky/meta/conf/distro/include/default-distrovars.inc`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011180
11181 COMMERCIAL_AUDIO_PLUGINS ?= ""
11182 COMMERCIAL_VIDEO_PLUGINS ?= ""
11183
11184If you
11185want to enable these components, you can do so by making sure you have
11186statements similar to the following in your ``local.conf`` configuration
Andrew Geisslerc926e172021-05-07 16:11:35 -050011187file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011188
11189 COMMERCIAL_AUDIO_PLUGINS = "gst-plugins-ugly-mad \
11190 gst-plugins-ugly-mpegaudioparse"
11191 COMMERCIAL_VIDEO_PLUGINS = "gst-plugins-ugly-mpeg2dec \
11192 gst-plugins-ugly-mpegstream gst-plugins-bad-mpegvideoparse"
Andrew Geissler9aee5002022-03-30 16:27:02 +000011193 LICENSE_FLAGS_ACCEPTED = "commercial_gst-plugins-ugly commercial_gst-plugins-bad commercial_qmmp"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011194
11195
Andrew Geissler595f6302022-01-24 19:11:47 +000011196Of course, you could also create a matching list for those
11197components using the more general "commercial" in the
Andrew Geissler9aee5002022-03-30 16:27:02 +000011198:term:`LICENSE_FLAGS_ACCEPTED` variable, but that would also enable all
Andrew Geissler595f6302022-01-24 19:11:47 +000011199the other packages with :term:`LICENSE_FLAGS`
Andrew Geisslerc926e172021-05-07 16:11:35 -050011200containing "commercial", which you may or may not want::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011201
Andrew Geissler9aee5002022-03-30 16:27:02 +000011202 LICENSE_FLAGS_ACCEPTED = "commercial"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011203
11204Specifying audio and video plugins as part of the
11205``COMMERCIAL_AUDIO_PLUGINS`` and ``COMMERCIAL_VIDEO_PLUGINS`` statements
Andrew Geissler9aee5002022-03-30 16:27:02 +000011206(along with the enabling :term:`LICENSE_FLAGS_ACCEPTED`) includes the
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011207plugins or components into built images, thus adding support for media
11208formats or components.
11209
11210Maintaining Open Source License Compliance During Your Product's Lifecycle
11211--------------------------------------------------------------------------
11212
11213One of the concerns for a development organization using open source
11214software is how to maintain compliance with various open source
11215licensing during the lifecycle of the product. While this section does
11216not provide legal advice or comprehensively cover all scenarios, it does
11217present methods that you can use to assist you in meeting the compliance
11218requirements during a software release.
11219
11220With hundreds of different open source licenses that the Yocto Project
11221tracks, it is difficult to know the requirements of each and every
11222license. However, the requirements of the major FLOSS licenses can begin
William A. Kennington IIIac69b482021-06-02 12:28:27 -070011223to be covered by assuming that there are three main areas of concern:
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011224
11225- Source code must be provided.
11226
11227- License text for the software must be provided.
11228
11229- Compilation scripts and modifications to the source code must be
11230 provided.
11231
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011232- spdx files can be provided.
11233
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011234There are other requirements beyond the scope of these three and the
11235methods described in this section (e.g. the mechanism through which
11236source code is distributed).
11237
11238As different organizations have different methods of complying with open
11239source licensing, this section is not meant to imply that there is only
11240one single way to meet your compliance obligations, but rather to
11241describe one method of achieving compliance. The remainder of this
11242section describes methods supported to meet the previously mentioned
11243three requirements. Once you take steps to meet these requirements, and
11244prior to releasing images, sources, and the build system, you should
11245audit all artifacts to ensure completeness.
11246
11247.. note::
11248
11249 The Yocto Project generates a license manifest during image creation
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011250 that is located in ``${DEPLOY_DIR}/licenses/``\ `image_name`\ ``-``\ `datestamp`
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011251 to assist with any audits.
11252
11253Providing the Source Code
11254~~~~~~~~~~~~~~~~~~~~~~~~~
11255
11256Compliance activities should begin before you generate the final image.
11257The first thing you should look at is the requirement that tops the list
Andrew Geissler615f2f12022-07-15 14:00:58 -050011258for most compliance groups --- providing the source. The Yocto Project has
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011259a few ways of meeting this requirement.
11260
11261One of the easiest ways to meet this requirement is to provide the
11262entire :term:`DL_DIR` used by the
11263build. This method, however, has a few issues. The most obvious is the
11264size of the directory since it includes all sources used in the build
11265and not just the source used in the released image. It will include
11266toolchain source, and other artifacts, which you would not generally
11267release. However, the more serious issue for most companies is
11268accidental release of proprietary software. The Yocto Project provides
11269an :ref:`archiver <ref-classes-archiver>` class to
11270help avoid some of these concerns.
11271
Andrew Geissler595f6302022-01-24 19:11:47 +000011272Before you employ :term:`DL_DIR` or the :ref:`archiver <ref-classes-archiver>` class, you need to
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011273decide how you choose to provide source. The source ``archiver`` class
11274can generate tarballs and SRPMs and can create them with various levels
11275of compliance in mind.
11276
11277One way of doing this (but certainly not the only way) is to release
11278just the source as a tarball. You can do this by adding the following to
11279the ``local.conf`` file found in the
Andrew Geisslerc926e172021-05-07 16:11:35 -050011280:term:`Build Directory`::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011281
11282 INHERIT += "archiver"
11283 ARCHIVER_MODE[src] = "original"
11284
11285During the creation of your
11286image, the source from all recipes that deploy packages to the image is
11287placed within subdirectories of ``DEPLOY_DIR/sources`` based on the
11288:term:`LICENSE` for each recipe.
11289Releasing the entire directory enables you to comply with requirements
11290concerning providing the unmodified source. It is important to note that
11291the size of the directory can get large.
11292
11293A way to help mitigate the size issue is to only release tarballs for
11294licenses that require the release of source. Let us assume you are only
11295concerned with GPL code as identified by running the following script:
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011296
11297.. code-block:: shell
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011298
11299 # Script to archive a subset of packages matching specific license(s)
11300 # Source and license files are copied into sub folders of package folder
11301 # Must be run from build folder
11302 #!/bin/bash
11303 src_release_dir="source-release"
11304 mkdir -p $src_release_dir
11305 for a in tmp/deploy/sources/*; do
11306 for d in $a/*; do
11307 # Get package name from path
11308 p=`basename $d`
11309 p=${p%-*}
11310 p=${p%-*}
11311 # Only archive GPL packages (update *GPL* regex for your license check)
11312 numfiles=`ls tmp/deploy/licenses/$p/*GPL* 2> /dev/null | wc -l`
Patrick Williams213cb262021-08-07 19:21:33 -050011313 if [ $numfiles -ge 1 ]; then
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011314 echo Archiving $p
11315 mkdir -p $src_release_dir/$p/source
11316 cp $d/* $src_release_dir/$p/source 2> /dev/null
11317 mkdir -p $src_release_dir/$p/license
11318 cp tmp/deploy/licenses/$p/* $src_release_dir/$p/license 2> /dev/null
11319 fi
11320 done
11321 done
11322
11323At this point, you
11324could create a tarball from the ``gpl_source_release`` directory and
11325provide that to the end user. This method would be a step toward
11326achieving compliance with section 3a of GPLv2 and with section 6 of
11327GPLv3.
11328
11329Providing License Text
11330~~~~~~~~~~~~~~~~~~~~~~
11331
11332One requirement that is often overlooked is inclusion of license text.
11333This requirement also needs to be dealt with prior to generating the
11334final image. Some licenses require the license text to accompany the
11335binary. You can achieve this by adding the following to your
Andrew Geisslerc926e172021-05-07 16:11:35 -050011336``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011337
11338 COPY_LIC_MANIFEST = "1"
11339 COPY_LIC_DIRS = "1"
11340 LICENSE_CREATE_PACKAGE = "1"
11341
11342Adding these statements to the
11343configuration file ensures that the licenses collected during package
11344generation are included on your image.
11345
11346.. note::
11347
11348 Setting all three variables to "1" results in the image having two
11349 copies of the same license file. One copy resides in
11350 ``/usr/share/common-licenses`` and the other resides in
11351 ``/usr/share/license``.
11352
11353 The reason for this behavior is because
11354 :term:`COPY_LIC_DIRS` and
11355 :term:`COPY_LIC_MANIFEST`
11356 add a copy of the license when the image is built but do not offer a
11357 path for adding licenses for newly installed packages to an image.
11358 :term:`LICENSE_CREATE_PACKAGE`
11359 adds a separate package and an upgrade path for adding licenses to an
11360 image.
11361
11362As the source ``archiver`` class has already archived the original
11363unmodified source that contains the license files, you would have
11364already met the requirements for inclusion of the license information
11365with source as defined by the GPL and other open source licenses.
11366
11367Providing Compilation Scripts and Source Code Modifications
11368~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
11369
11370At this point, we have addressed all we need to prior to generating the
11371image. The next two requirements are addressed during the final
11372packaging of the release.
11373
11374By releasing the version of the OpenEmbedded build system and the layers
11375used during the build, you will be providing both compilation scripts
11376and the source code modifications in one step.
11377
Andrew Geissler09209ee2020-12-13 08:44:15 -060011378If the deployment team has a :ref:`overview-manual/concepts:bsp layer`
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011379and a distro layer, and those
11380those layers are used to patch, compile, package, or modify (in any way)
11381any open source software included in your released images, you might be
11382required to release those layers under section 3 of GPLv2 or section 1
11383of GPLv3. One way of doing that is with a clean checkout of the version
11384of the Yocto Project and layers used during your build. Here is an
11385example:
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011386
11387.. code-block:: shell
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011388
11389 # We built using the dunfell branch of the poky repo
11390 $ git clone -b dunfell git://git.yoctoproject.org/poky
11391 $ cd poky
11392 # We built using the release_branch for our layers
11393 $ git clone -b release_branch git://git.mycompany.com/meta-my-bsp-layer
11394 $ git clone -b release_branch git://git.mycompany.com/meta-my-software-layer
11395 # clean up the .git repos
11396 $ find . -name ".git" -type d -exec rm -rf {} \;
11397
Andrew Geissler87f5cff2022-09-30 13:13:31 -050011398One thing a development organization might want to consider for end-user
11399convenience is to modify
11400``meta-poky/conf/templates/default/bblayers.conf.sample`` to ensure that when
11401the end user utilizes the released build system to build an image, the
11402development organization's layers are included in the ``bblayers.conf`` file
11403automatically::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011404
11405 # POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
11406 # changes incompatibly
11407 POKY_BBLAYERS_CONF_VERSION = "2"
11408
11409 BBPATH = "${TOPDIR}"
11410 BBFILES ?= ""
11411
11412 BBLAYERS ?= " \
11413 ##OEROOT##/meta \
11414 ##OEROOT##/meta-poky \
11415 ##OEROOT##/meta-yocto-bsp \
11416 ##OEROOT##/meta-mylayer \
11417 "
11418
11419Creating and
11420providing an archive of the :term:`Metadata`
11421layers (recipes, configuration files, and so forth) enables you to meet
11422your requirements to include the scripts to control compilation as well
11423as any modifications to the original source.
11424
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011425Providing spdx files
11426~~~~~~~~~~~~~~~~~~~~~~~~~
11427
11428The spdx module has been integrated to a layer named meta-spdxscanner.
11429meta-spdxscanner provides several kinds of scanner. If you want to enable
11430this function, you have to follow the following steps:
11431
Andrew Geissler87f5cff2022-09-30 13:13:31 -0500114321. Add meta-spdxscanner layer into ``bblayers.conf``.
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011433
Andrew Geissler87f5cff2022-09-30 13:13:31 -0500114342. Refer to the README in meta-spdxscanner to setup the environment (e.g,
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011435 setup a fossology server) needed for the scanner.
11436
114373. Meta-spdxscanner provides several methods within the bbclass to create spdx files.
11438 Please choose one that you want to use and enable the spdx task. You have to
11439 add some config options in ``local.conf`` file in your :term:`Build
William A. Kennington IIIac69b482021-06-02 12:28:27 -070011440 Directory`. Here is an example showing how to generate spdx files
Andrew Geisslerd5838332022-05-27 11:33:10 -050011441 during BitBake using the fossology-python.bbclass::
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011442
11443 # Select fossology-python.bbclass.
11444 INHERIT += "fossology-python"
11445 # For fossology-python.bbclass, TOKEN is necessary, so, after setup a
11446 # Fossology server, you have to create a token.
11447 TOKEN = "eyJ0eXAiO..."
11448 # The fossology server is necessary for fossology-python.bbclass.
11449 FOSSOLOGY_SERVER = "http://xx.xx.xx.xx:8081/repo"
11450 # If you want to upload the source code to a special folder:
11451 FOLDER_NAME = "xxxx" //Optional
11452 # If you don't want to put spdx files in tmp/deploy/spdx, you can enable:
11453 SPDX_DEPLOY_DIR = "${DEPLOY_DIR}" //Optional
11454
11455For more usage information refer to :yocto_git:`the meta-spdxscanner repository
Andrew Geissler09209ee2020-12-13 08:44:15 -060011456</meta-spdxscanner/>`.
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011457
Andrew Geisslereff27472021-10-29 15:35:00 -050011458Compliance Limitations with Executables Built from Static Libraries
11459~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011460
Andrew Geisslereff27472021-10-29 15:35:00 -050011461When package A is added to an image via the :term:`RDEPENDS` or :term:`RRECOMMENDS`
11462mechanisms as well as explicitly included in the image recipe with
11463:term:`IMAGE_INSTALL`, and depends on a static linked library recipe B
11464(``DEPENDS += "B"``), package B will neither appear in the generated license
11465manifest nor in the generated source tarballs. This occurs as the
11466:ref:`license <ref-classes-license>` and :ref:`archiver <ref-classes-archiver>`
11467classes assume that only packages included via :term:`RDEPENDS` or :term:`RRECOMMENDS`
11468end up in the image.
11469
11470As a result, potential obligations regarding license compliance for package B
11471may not be met.
11472
11473The Yocto Project doesn't enable static libraries by default, in part because
11474of this issue. Before a solution to this limitation is found, you need to
11475keep in mind that if your root filesystem is built from static libraries,
11476you will need to manually ensure that your deliveries are compliant
11477with the licenses of these libraries.
11478
11479Copying Non Standard Licenses
11480-----------------------------
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011481
11482Some packages, such as the linux-firmware package, have many licenses
11483that are not in any way common. You can avoid adding a lot of these
11484types of common license files, which are only applicable to a specific
11485package, by using the
11486:term:`NO_GENERIC_LICENSE`
11487variable. Using this variable also avoids QA errors when you use a
11488non-common, non-CLOSED license in a recipe.
11489
William A. Kennington IIIac69b482021-06-02 12:28:27 -070011490Here is an example that uses the ``LICENSE.Abilis.txt`` file as
Andrew Geisslerc926e172021-05-07 16:11:35 -050011491the license from the fetched source::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011492
11493 NO_GENERIC_LICENSE[Firmware-Abilis] = "LICENSE.Abilis.txt"
11494
Patrick Williams213cb262021-08-07 19:21:33 -050011495Checking for Vulnerabilities
11496============================
11497
11498Vulnerabilities in images
11499-------------------------
11500
11501The Yocto Project has an infrastructure to track and address unfixed
11502known security vulnerabilities, as tracked by the public
11503`Common Vulnerabilities and Exposures (CVE) <https://en.wikipedia.org/wiki/Common_Vulnerabilities_and_Exposures>`__
11504database.
11505
Andrew Geissler615f2f12022-07-15 14:00:58 -050011506The Yocto Project maintains a `list of known vulnerabilities
11507<https://autobuilder.yocto.io/pub/non-release/patchmetrics/>`__
11508for packages in Poky and OE-Core, tracking the evolution of the number of
11509unpatched CVEs and the status of patches. Such information is available for
11510the current development version and for each supported release.
11511
11512To know which packages are vulnerable to known security vulnerabilities
11513in the specific image you are building, add the following setting to your
11514configuration::
Patrick Williams213cb262021-08-07 19:21:33 -050011515
11516 INHERIT += "cve-check"
11517
11518This way, at build time, BitBake will warn you about known CVEs
11519as in the example below::
11520
11521 WARNING: flex-2.6.4-r0 do_cve_check: Found unpatched CVE (CVE-2019-6293), for more information check /poky/build/tmp/work/core2-64-poky-linux/flex/2.6.4-r0/temp/cve.log
11522 WARNING: libarchive-3.5.1-r0 do_cve_check: Found unpatched CVE (CVE-2021-36976), for more information check /poky/build/tmp/work/core2-64-poky-linux/libarchive/3.5.1-r0/temp/cve.log
11523
11524It is also possible to check the CVE status of individual packages as follows::
11525
11526 bitbake -c cve_check flex libarchive
11527
11528Note that OpenEmbedded-Core keeps a list of known unfixed CVE issues which can
11529be ignored. You can pass this list to the check as follows::
11530
11531 bitbake -c cve_check libarchive -R conf/distro/include/cve-extra-exclusions.inc
11532
11533Enabling vulnerabily tracking in recipes
11534----------------------------------------
11535
11536The :term:`CVE_PRODUCT` variable defines the name used to match the recipe name
11537against the name in the upstream `NIST CVE database <https://nvd.nist.gov/>`__.
11538
Patrick Williams0ca19cc2021-08-16 14:03:13 -050011539Editing recipes to fix vulnerabilities
11540--------------------------------------
11541
11542To fix a given known vulnerability, you need to add a patch file to your recipe. Here's
11543an example from the :oe_layerindex:`ffmpeg recipe</layerindex/recipe/47350>`::
11544
11545 SRC_URI = "https://www.ffmpeg.org/releases/${BP}.tar.xz \
11546 file://0001-libavutil-include-assembly-with-full-path-from-sourc.patch \
11547 file://fix-CVE-2020-20446.patch \
11548 file://fix-CVE-2020-20453.patch \
11549 file://fix-CVE-2020-22015.patch \
11550 file://fix-CVE-2020-22021.patch \
11551 file://fix-CVE-2020-22033-CVE-2020-22019.patch \
11552 file://fix-CVE-2021-33815.patch \
11553
11554The :ref:`cve-check <ref-classes-cve-check>` class defines two ways of
11555supplying a patch for a given CVE. The first
11556way is to use a patch filename that matches the below pattern::
11557
11558 cve_file_name_match = re.compile(".*([Cc][Vv][Ee]\-\d{4}\-\d+)")
11559
11560As shown in the example above, multiple CVE IDs can appear in a patch filename,
11561but the :ref:`cve-check <ref-classes-cve-check>` class will only consider
11562the last CVE ID in the filename as patched.
11563
11564The second way to recognize a patched CVE ID is when a line matching the
11565below pattern is found in any patch file provided by the recipe::
11566
11567 cve_match = re.compile("CVE:( CVE\-\d{4}\-\d+)+")
11568
11569This allows a single patch file to address multiple CVE IDs at the same time.
11570
11571Of course, another way to fix vulnerabilities is to upgrade to a version
11572of the package which is not impacted, typically a more recent one.
11573The NIST database knows which versions are vulnerable and which ones
11574are not.
11575
11576Last but not least, you can choose to ignore vulnerabilities through
Andrew Geissler9aee5002022-03-30 16:27:02 +000011577the :term:`CVE_CHECK_SKIP_RECIPE` and :term:`CVE_CHECK_IGNORE`
Patrick Williams0ca19cc2021-08-16 14:03:13 -050011578variables.
11579
11580Implementation details
11581----------------------
11582
11583Here's what the :ref:`cve-check <ref-classes-cve-check>` class does to
11584find unpatched CVE IDs.
11585
11586First the code goes through each patch file provided by a recipe. If a valid CVE ID
11587is found in the name of the file, the corresponding CVE is considered as patched.
11588Don't forget that if multiple CVE IDs are found in the filename, only the last
11589one is considered. Then, the code looks for ``CVE: CVE-ID`` lines in the patch
11590file. The found CVE IDs are also considered as patched.
11591
11592Then, the code looks up all the CVE IDs in the NIST database for all the
11593products defined in :term:`CVE_PRODUCT`. Then, for each found CVE:
11594
11595 - If the package name (:term:`PN`) is part of
Andrew Geissler9aee5002022-03-30 16:27:02 +000011596 :term:`CVE_CHECK_SKIP_RECIPE`, it is considered as patched.
Patrick Williams0ca19cc2021-08-16 14:03:13 -050011597
Andrew Geissler9aee5002022-03-30 16:27:02 +000011598 - If the CVE ID is part of :term:`CVE_CHECK_IGNORE`, it is
Patrick Williams0ca19cc2021-08-16 14:03:13 -050011599 considered as patched too.
11600
11601 - If the CVE ID is part of the patched CVE for the recipe, it is
11602 already considered as patched.
11603
11604 - Otherwise, the code checks whether the recipe version (:term:`PV`)
11605 is within the range of versions impacted by the CVE. If so, the CVE
11606 is considered as unpatched.
11607
Patrick Williams213cb262021-08-07 19:21:33 -050011608The CVE database is stored in :term:`DL_DIR` and can be inspected using
11609``sqlite3`` command as follows::
11610
11611 sqlite3 downloads/CVE_CHECK/nvdcve_1.1.db .dump | grep CVE-2021-37462
11612
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011613Using the Error Reporting Tool
11614==============================
11615
11616The error reporting tool allows you to submit errors encountered during
11617builds to a central database. Outside of the build environment, you can
11618use a web interface to browse errors, view statistics, and query for
11619errors. The tool works using a client-server system where the client
11620portion is integrated with the installed Yocto Project
11621:term:`Source Directory` (e.g. ``poky``).
11622The server receives the information collected and saves it in a
11623database.
11624
William A. Kennington IIIac69b482021-06-02 12:28:27 -070011625There is a live instance of the error reporting server at
11626https://errors.yoctoproject.org.
11627When you want to get help with build failures, you can submit all of the
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011628information on the failure easily and then point to the URL in your bug
11629report or send an email to the mailing list.
11630
11631.. note::
11632
11633 If you send error reports to this server, the reports become publicly
11634 visible.
11635
11636Enabling and Using the Tool
11637---------------------------
11638
11639By default, the error reporting tool is disabled. You can enable it by
11640inheriting the
11641:ref:`report-error <ref-classes-report-error>`
11642class by adding the following statement to the end of your
11643``local.conf`` file in your
11644:term:`Build Directory`.
11645::
11646
11647 INHERIT += "report-error"
11648
11649By default, the error reporting feature stores information in
11650``${``\ :term:`LOG_DIR`\ ``}/error-report``.
11651However, you can specify a directory to use by adding the following to
Andrew Geisslerc926e172021-05-07 16:11:35 -050011652your ``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011653
11654 ERR_REPORT_DIR = "path"
11655
11656Enabling error
11657reporting causes the build process to collect the errors and store them
11658in a file as previously described. When the build system encounters an
11659error, it includes a command as part of the console output. You can run
11660the command to send the error file to the server. For example, the
Andrew Geisslerc926e172021-05-07 16:11:35 -050011661following command sends the errors to an upstream server::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011662
11663 $ send-error-report /home/brandusa/project/poky/build/tmp/log/error-report/error_report_201403141617.txt
11664
11665In the previous example, the errors are sent to a public database
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011666available at https://errors.yoctoproject.org, which is used by the
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011667entire community. If you specify a particular server, you can send the
11668errors to a different database. Use the following command for more
Andrew Geisslerc926e172021-05-07 16:11:35 -050011669information on available options::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011670
11671 $ send-error-report --help
11672
11673When sending the error file, you are prompted to review the data being
11674sent as well as to provide a name and optional email address. Once you
11675satisfy these prompts, the command returns a link from the server that
11676corresponds to your entry in the database. For example, here is a
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011677typical link: https://errors.yoctoproject.org/Errors/Details/9522/
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011678
11679Following the link takes you to a web interface where you can browse,
11680query the errors, and view statistics.
11681
11682Disabling the Tool
11683------------------
11684
11685To disable the error reporting feature, simply remove or comment out the
11686following statement from the end of your ``local.conf`` file in your
11687:term:`Build Directory`.
11688::
11689
11690 INHERIT += "report-error"
11691
11692Setting Up Your Own Error Reporting Server
11693------------------------------------------
11694
11695If you want to set up your own error reporting server, you can obtain
Andrew Geissler09209ee2020-12-13 08:44:15 -060011696the code from the Git repository at :yocto_git:`/error-report-web/`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011697Instructions on how to set it up are in the README document.
11698
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011699Using Wayland and Weston
11700========================
11701
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011702`Wayland <https://en.wikipedia.org/wiki/Wayland_(display_server_protocol)>`__
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011703is a computer display server protocol that provides a method for
11704compositing window managers to communicate directly with applications
11705and video hardware and expects them to communicate with input hardware
11706using other libraries. Using Wayland with supporting targets can result
11707in better control over graphics frame rendering than an application
11708might otherwise achieve.
11709
11710The Yocto Project provides the Wayland protocol libraries and the
11711reference
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011712`Weston <https://en.wikipedia.org/wiki/Wayland_(display_server_protocol)#Weston>`__
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011713compositor as part of its release. You can find the integrated packages
11714in the ``meta`` layer of the :term:`Source Directory`.
11715Specifically, you
11716can find the recipes that build both Wayland and Weston at
11717``meta/recipes-graphics/wayland``.
11718
11719You can build both the Wayland and Weston packages for use only with
11720targets that accept the `Mesa 3D and Direct Rendering
11721Infrastructure <https://en.wikipedia.org/wiki/Mesa_(computer_graphics)>`__,
11722which is also known as Mesa DRI. This implies that you cannot build and
11723use the packages if your target uses, for example, the Intel Embedded
11724Media and Graphics Driver (Intel EMGD) that overrides Mesa DRI.
11725
11726.. note::
11727
11728 Due to lack of EGL support, Weston 1.0.3 will not run directly on the
11729 emulated QEMU hardware. However, this version of Weston will run
11730 under X emulation without issues.
11731
11732This section describes what you need to do to implement Wayland and use
11733the Weston compositor when building an image for a supporting target.
11734
11735Enabling Wayland in an Image
11736----------------------------
11737
11738To enable Wayland, you need to enable it to be built and enable it to be
11739included (installed) in the image.
11740
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011741Building Wayland
11742~~~~~~~~~~~~~~~~
11743
11744To cause Mesa to build the ``wayland-egl`` platform and Weston to build
11745Wayland with Kernel Mode Setting
11746(`KMS <https://wiki.archlinux.org/index.php/Kernel_Mode_Setting>`__)
11747support, include the "wayland" flag in the
11748:term:`DISTRO_FEATURES`
Andrew Geisslerc926e172021-05-07 16:11:35 -050011749statement in your ``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011750
Patrick Williams0ca19cc2021-08-16 14:03:13 -050011751 DISTRO_FEATURES:append = " wayland"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011752
11753.. note::
11754
11755 If X11 has been enabled elsewhere, Weston will build Wayland with X11
11756 support
11757
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011758Installing Wayland and Weston
11759~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
11760
11761To install the Wayland feature into an image, you must include the
11762following
11763:term:`CORE_IMAGE_EXTRA_INSTALL`
Andrew Geisslerc926e172021-05-07 16:11:35 -050011764statement in your ``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011765
11766 CORE_IMAGE_EXTRA_INSTALL += "wayland weston"
11767
11768Running Weston
11769--------------
11770
11771To run Weston inside X11, enabling it as described earlier and building
11772a Sato image is sufficient. If you are running your image under Sato, a
11773Weston Launcher appears in the "Utility" category.
11774
11775Alternatively, you can run Weston through the command-line interpretor
11776(CLI), which is better suited for development work. To run Weston under
11777the CLI, you need to do the following after your image is built:
11778
Andrew Geisslerc926e172021-05-07 16:11:35 -0500117791. Run these commands to export ``XDG_RUNTIME_DIR``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011780
11781 mkdir -p /tmp/$USER-weston
11782 chmod 0700 /tmp/$USER-weston
11783 export XDG_RUNTIME_DIR=/tmp/$USER-weston
11784
Andrew Geisslerc926e172021-05-07 16:11:35 -0500117852. Launch Weston in the shell::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011786
11787 weston