blob: 3eead147a33d321ac25bbe5e05f113e3f87d1664 [file] [log] [blame]
Andrew Geisslerf0343792020-11-18 10:42:21 -06001.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002
3************
4Common Tasks
5************
6
7This chapter describes fundamental procedures such as creating layers,
8adding new software packages, extending or customizing images, porting
9work to new hardware (adding a new machine), and so forth. You will find
10that the procedures documented here occur often in the development cycle
11using the Yocto Project.
12
13Understanding and Creating Layers
14=================================
15
16The OpenEmbedded build system supports organizing
17:term:`Metadata` into multiple layers.
18Layers allow you to isolate different types of customizations from each
19other. For introductory information on the Yocto Project Layer Model,
20see the
Andrew Geissler09209ee2020-12-13 08:44:15 -060021":ref:`overview-manual/yp-intro:the yocto project layer model`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050022section in the Yocto Project Overview and Concepts Manual.
23
24Creating Your Own Layer
25-----------------------
26
27It is very easy to create your own layers to use with the OpenEmbedded
Andrew Geissler5199d832021-09-24 16:47:35 -050028build system, as the Yocto Project ships with tools that speed up creating
Andrew Geisslerc9f78652020-09-18 14:11:35 -050029layers. This section describes the steps you perform by hand to create
30layers so that you can better understand them. For information about the
31layer-creation tools, see the
32":ref:`bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script`"
33section in the Yocto Project Board Support Package (BSP) Developer's
Andrew Geissler09209ee2020-12-13 08:44:15 -060034Guide and the ":ref:`dev-manual/common-tasks:creating a general layer using the \`\`bitbake-layers\`\` script`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050035section further down in this manual.
36
37Follow these general steps to create your layer without using tools:
38
391. *Check Existing Layers:* Before creating a new layer, you should be
40 sure someone has not already created a layer containing the Metadata
Andrew Geisslerd1e89492021-02-12 15:35:20 -060041 you need. You can see the :oe_layerindex:`OpenEmbedded Metadata Index <>`
42 for a list of layers from the OpenEmbedded community that can be used in
Andrew Geisslerc9f78652020-09-18 14:11:35 -050043 the Yocto Project. You could find a layer that is identical or close
44 to what you need.
45
462. *Create a Directory:* Create the directory for your layer. When you
47 create the layer, be sure to create the directory in an area not
48 associated with the Yocto Project :term:`Source Directory`
49 (e.g. the cloned ``poky`` repository).
50
51 While not strictly required, prepend the name of the directory with
Andrew Geisslerc926e172021-05-07 16:11:35 -050052 the string "meta-". For example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050053
54 meta-mylayer
55 meta-GUI_xyz
56 meta-mymachine
57
Andrew Geisslerc926e172021-05-07 16:11:35 -050058 With rare exceptions, a layer's name follows this form::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050059
60 meta-root_name
61
62 Following this layer naming convention can save
63 you trouble later when tools, components, or variables "assume" your
64 layer name begins with "meta-". A notable example is in configuration
65 files as shown in the following step where layer names without the
66 "meta-" string are appended to several variables used in the
67 configuration.
68
693. *Create a Layer Configuration File:* Inside your new layer folder,
70 you need to create a ``conf/layer.conf`` file. It is easiest to take
71 an existing layer configuration file and copy that to your layer's
72 ``conf`` directory and then modify the file as needed.
73
74 The ``meta-yocto-bsp/conf/layer.conf`` file in the Yocto Project
Andrew Geissler09209ee2020-12-13 08:44:15 -060075 :yocto_git:`Source Repositories </poky/tree/meta-yocto-bsp/conf>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -050076 demonstrates the required syntax. For your layer, you need to replace
77 "yoctobsp" with a unique identifier for your layer (e.g. "machinexyz"
Andrew Geisslerc926e172021-05-07 16:11:35 -050078 for a layer named "meta-machinexyz")::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050079
80 # We have a conf and classes directory, add to BBPATH
81 BBPATH .= ":${LAYERDIR}"
82
Andrew Geissler4c19ea12020-10-27 13:52:24 -050083 # We have recipes-* directories, add to BBFILES
Andrew Geisslerc9f78652020-09-18 14:11:35 -050084 BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
85 ${LAYERDIR}/recipes-*/*/*.bbappend"
86
87 BBFILE_COLLECTIONS += "yoctobsp"
88 BBFILE_PATTERN_yoctobsp = "^${LAYERDIR}/"
89 BBFILE_PRIORITY_yoctobsp = "5"
90 LAYERVERSION_yoctobsp = "4"
91 LAYERSERIES_COMPAT_yoctobsp = "dunfell"
92
93 Following is an explanation of the layer configuration file:
94
95 - :term:`BBPATH`: Adds the layer's
96 root directory to BitBake's search path. Through the use of the
Andrew Geissler09036742021-06-25 14:25:14 -050097 :term:`BBPATH` variable, BitBake locates class files (``.bbclass``),
Andrew Geisslerc9f78652020-09-18 14:11:35 -050098 configuration files, and files that are included with ``include``
99 and ``require`` statements. For these cases, BitBake uses the
Andrew Geissler09036742021-06-25 14:25:14 -0500100 first file that matches the name found in :term:`BBPATH`. This is
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500101 similar to the way the ``PATH`` variable is used for binaries. It
102 is recommended, therefore, that you use unique class and
103 configuration filenames in your custom layer.
104
105 - :term:`BBFILES`: Defines the
106 location for all recipes in the layer.
107
108 - :term:`BBFILE_COLLECTIONS`:
109 Establishes the current layer through a unique identifier that is
110 used throughout the OpenEmbedded build system to refer to the
111 layer. In this example, the identifier "yoctobsp" is the
112 representation for the container layer named "meta-yocto-bsp".
113
114 - :term:`BBFILE_PATTERN`:
115 Expands immediately during parsing to provide the directory of the
116 layer.
117
118 - :term:`BBFILE_PRIORITY`:
119 Establishes a priority to use for recipes in the layer when the
120 OpenEmbedded build finds recipes of the same name in different
121 layers.
122
123 - :term:`LAYERVERSION`:
124 Establishes a version number for the layer. You can use this
125 version number to specify this exact version of the layer as a
126 dependency when using the
127 :term:`LAYERDEPENDS`
128 variable.
129
130 - :term:`LAYERDEPENDS`:
131 Lists all layers on which this layer depends (if any).
132
133 - :term:`LAYERSERIES_COMPAT`:
Andrew Geissler09209ee2020-12-13 08:44:15 -0600134 Lists the :yocto_wiki:`Yocto Project </Releases>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500135 releases for which the current version is compatible. This
136 variable is a good way to indicate if your particular layer is
137 current.
138
1394. *Add Content:* Depending on the type of layer, add the content. If
140 the layer adds support for a machine, add the machine configuration
141 in a ``conf/machine/`` file within the layer. If the layer adds
142 distro policy, add the distro configuration in a ``conf/distro/``
143 file within the layer. If the layer introduces new recipes, put the
144 recipes you need in ``recipes-*`` subdirectories within the layer.
145
146 .. note::
147
148 For an explanation of layer hierarchy that is compliant with the
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500149 Yocto Project, see the ":ref:`bsp-guide/bsp:example filesystem layout`"
150 section in the Yocto Project Board Support Package (BSP) Developer's Guide.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500151
1525. *Optionally Test for Compatibility:* If you want permission to use
153 the Yocto Project Compatibility logo with your layer or application
154 that uses your layer, perform the steps to apply for compatibility.
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500155 See the
156 ":ref:`dev-manual/common-tasks:making sure your layer is compatible with yocto project`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500157 section for more information.
158
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500159Following Best Practices When Creating Layers
160---------------------------------------------
161
162To create layers that are easier to maintain and that will not impact
163builds for other machines, you should consider the information in the
164following list:
165
166- *Avoid "Overlaying" Entire Recipes from Other Layers in Your
167 Configuration:* In other words, do not copy an entire recipe into
168 your layer and then modify it. Rather, use an append file
169 (``.bbappend``) to override only those parts of the original recipe
170 you need to modify.
171
172- *Avoid Duplicating Include Files:* Use append files (``.bbappend``)
173 for each recipe that uses an include file. Or, if you are introducing
174 a new recipe that requires the included file, use the path relative
175 to the original layer directory to refer to the file. For example,
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500176 use ``require recipes-core/``\ `package`\ ``/``\ `file`\ ``.inc`` instead
177 of ``require`` `file`\ ``.inc``. If you're finding you have to overlay
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500178 the include file, it could indicate a deficiency in the include file
179 in the layer to which it originally belongs. If this is the case, you
180 should try to address that deficiency instead of overlaying the
181 include file. For example, you could address this by getting the
182 maintainer of the include file to add a variable or variables to make
183 it easy to override the parts needing to be overridden.
184
185- *Structure Your Layers:* Proper use of overrides within append files
186 and placement of machine-specific files within your layer can ensure
187 that a build is not using the wrong Metadata and negatively impacting
188 a build for a different machine. Following are some examples:
189
190 - *Modify Variables to Support a Different Machine:* Suppose you
191 have a layer named ``meta-one`` that adds support for building
192 machine "one". To do so, you use an append file named
193 ``base-files.bbappend`` and create a dependency on "foo" by
194 altering the :term:`DEPENDS`
Andrew Geisslerc926e172021-05-07 16:11:35 -0500195 variable::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500196
197 DEPENDS = "foo"
198
199 The dependency is created during any
200 build that includes the layer ``meta-one``. However, you might not
201 want this dependency for all machines. For example, suppose you
202 are building for machine "two" but your ``bblayers.conf`` file has
203 the ``meta-one`` layer included. During the build, the
204 ``base-files`` for machine "two" will also have the dependency on
205 ``foo``.
206
207 To make sure your changes apply only when building machine "one",
Andrew Geissler09036742021-06-25 14:25:14 -0500208 use a machine override with the :term:`DEPENDS` statement::
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500209
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500210 DEPENDS:one = "foo"
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500211
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500212 You should follow the same strategy when using ``:append``
213 and ``:prepend`` operations::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500214
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500215 DEPENDS:append:one = " foo"
216 DEPENDS:prepend:one = "foo "
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500217
218 As an actual example, here's a
219 snippet from the generic kernel include file ``linux-yocto.inc``,
220 wherein the kernel compile and link options are adjusted in the
Andrew Geisslerc926e172021-05-07 16:11:35 -0500221 case of a subset of the supported architectures::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500222
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500223 DEPENDS:append:aarch64 = " libgcc"
224 KERNEL_CC:append:aarch64 = " ${TOOLCHAIN_OPTIONS}"
225 KERNEL_LD:append:aarch64 = " ${TOOLCHAIN_OPTIONS}"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500226
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500227 DEPENDS:append:nios2 = " libgcc"
228 KERNEL_CC:append:nios2 = " ${TOOLCHAIN_OPTIONS}"
229 KERNEL_LD:append:nios2 = " ${TOOLCHAIN_OPTIONS}"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500230
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500231 DEPENDS:append:arc = " libgcc"
232 KERNEL_CC:append:arc = " ${TOOLCHAIN_OPTIONS}"
233 KERNEL_LD:append:arc = " ${TOOLCHAIN_OPTIONS}"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500234
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500235 KERNEL_FEATURES:append:qemuall=" features/debug/printk.scc"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500236
237 .. note::
238
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500239 Avoiding "+=" and "=+" and using machine-specific ``:append``
240 and ``:prepend`` operations is recommended as well.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500241
242 - *Place Machine-Specific Files in Machine-Specific Locations:* When
243 you have a base recipe, such as ``base-files.bb``, that contains a
244 :term:`SRC_URI` statement to a
245 file, you can use an append file to cause the build to use your
246 own version of the file. For example, an append file in your layer
247 at ``meta-one/recipes-core/base-files/base-files.bbappend`` could
Andrew Geisslerc926e172021-05-07 16:11:35 -0500248 extend :term:`FILESPATH` using :term:`FILESEXTRAPATHS` as follows::
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500249
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500250 FILESEXTRAPATHS:prepend := "${THISDIR}/${BPN}:"
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500251
252 The build for machine "one" will pick up your machine-specific file as
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500253 long as you have the file in
254 ``meta-one/recipes-core/base-files/base-files/``. However, if you
255 are building for a different machine and the ``bblayers.conf``
256 file includes the ``meta-one`` layer and the location of your
257 machine-specific file is the first location where that file is
Andrew Geissler09036742021-06-25 14:25:14 -0500258 found according to :term:`FILESPATH`, builds for all machines will
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500259 also use that machine-specific file.
260
261 You can make sure that a machine-specific file is used for a
262 particular machine by putting the file in a subdirectory specific
263 to the machine. For example, rather than placing the file in
264 ``meta-one/recipes-core/base-files/base-files/`` as shown above,
265 put it in ``meta-one/recipes-core/base-files/base-files/one/``.
266 Not only does this make sure the file is used only when building
267 for machine "one", but the build process locates the file more
268 quickly.
269
270 In summary, you need to place all files referenced from
Andrew Geissler5f350902021-07-23 13:09:54 -0400271 :term:`SRC_URI` in a machine-specific subdirectory within the layer in
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500272 order to restrict those files to machine-specific builds.
273
274- *Perform Steps to Apply for Yocto Project Compatibility:* If you want
275 permission to use the Yocto Project Compatibility logo with your
276 layer or application that uses your layer, perform the steps to apply
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500277 for compatibility. See the
278 ":ref:`dev-manual/common-tasks:making sure your layer is compatible with yocto project`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500279 section for more information.
280
281- *Follow the Layer Naming Convention:* Store custom layers in a Git
282 repository that use the ``meta-layer_name`` format.
283
284- *Group Your Layers Locally:* Clone your repository alongside other
285 cloned ``meta`` directories from the :term:`Source Directory`.
286
287Making Sure Your Layer is Compatible With Yocto Project
288-------------------------------------------------------
289
290When you create a layer used with the Yocto Project, it is advantageous
291to make sure that the layer interacts well with existing Yocto Project
292layers (i.e. the layer is compatible with the Yocto Project). Ensuring
293compatibility makes the layer easy to be consumed by others in the Yocto
294Project community and could allow you permission to use the Yocto
295Project Compatible Logo.
296
297.. note::
298
299 Only Yocto Project member organizations are permitted to use the
300 Yocto Project Compatible Logo. The logo is not available for general
301 use. For information on how to become a Yocto Project member
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500302 organization, see the :yocto_home:`Yocto Project Website <>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500303
304The Yocto Project Compatibility Program consists of a layer application
305process that requests permission to use the Yocto Project Compatibility
306Logo for your layer and application. The process consists of two parts:
307
3081. Successfully passing a script (``yocto-check-layer``) that when run
309 against your layer, tests it against constraints based on experiences
310 of how layers have worked in the real world and where pitfalls have
311 been found. Getting a "PASS" result from the script is required for
312 successful compatibility registration.
313
3142. Completion of an application acceptance form, which you can find at
Andrew Geisslerd1e89492021-02-12 15:35:20 -0600315 :yocto_home:`/webform/yocto-project-compatible-registration`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500316
317To be granted permission to use the logo, you need to satisfy the
318following:
319
320- Be able to check the box indicating that you got a "PASS" when
321 running the script against your layer.
322
323- Answer "Yes" to the questions on the form or have an acceptable
324 explanation for any questions answered "No".
325
326- Be a Yocto Project Member Organization.
327
328The remainder of this section presents information on the registration
329form and on the ``yocto-check-layer`` script.
330
331Yocto Project Compatible Program Application
332~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
333
334Use the form to apply for your layer's approval. Upon successful
335application, you can use the Yocto Project Compatibility Logo with your
336layer and the application that uses your layer.
337
338To access the form, use this link:
Andrew Geisslerd1e89492021-02-12 15:35:20 -0600339:yocto_home:`/webform/yocto-project-compatible-registration`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500340Follow the instructions on the form to complete your application.
341
342The application consists of the following sections:
343
344- *Contact Information:* Provide your contact information as the fields
345 require. Along with your information, provide the released versions
346 of the Yocto Project for which your layer is compatible.
347
348- *Acceptance Criteria:* Provide "Yes" or "No" answers for each of the
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700349 items in the checklist. There is space at the bottom of the form for
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500350 any explanations for items for which you answered "No".
351
352- *Recommendations:* Provide answers for the questions regarding Linux
353 kernel use and build success.
354
355``yocto-check-layer`` Script
356~~~~~~~~~~~~~~~~~~~~~~~~~~~~
357
358The ``yocto-check-layer`` script provides you a way to assess how
359compatible your layer is with the Yocto Project. You should run this
360script prior to using the form to apply for compatibility as described
361in the previous section. You need to achieve a "PASS" result in order to
362have your application form successfully processed.
363
364The script divides tests into three areas: COMMON, BSP, and DISTRO. For
365example, given a distribution layer (DISTRO), the layer must pass both
366the COMMON and DISTRO related tests. Furthermore, if your layer is a BSP
367layer, the layer must pass the COMMON and BSP set of tests.
368
369To execute the script, enter the following commands from your build
Andrew Geisslerc926e172021-05-07 16:11:35 -0500370directory::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500371
372 $ source oe-init-build-env
373 $ yocto-check-layer your_layer_directory
374
375Be sure to provide the actual directory for your
376layer as part of the command.
377
378Entering the command causes the script to determine the type of layer
379and then to execute a set of specific tests against the layer. The
380following list overviews the test:
381
382- ``common.test_readme``: Tests if a ``README`` file exists in the
383 layer and the file is not empty.
384
385- ``common.test_parse``: Tests to make sure that BitBake can parse the
386 files without error (i.e. ``bitbake -p``).
387
388- ``common.test_show_environment``: Tests that the global or per-recipe
389 environment is in order without errors (i.e. ``bitbake -e``).
390
391- ``common.test_world``: Verifies that ``bitbake world`` works.
392
393- ``common.test_signatures``: Tests to be sure that BSP and DISTRO
394 layers do not come with recipes that change signatures.
395
396- ``common.test_layerseries_compat``: Verifies layer compatibility is
397 set properly.
398
399- ``bsp.test_bsp_defines_machines``: Tests if a BSP layer has machine
400 configurations.
401
402- ``bsp.test_bsp_no_set_machine``: Tests to ensure a BSP layer does not
403 set the machine when the layer is added.
404
405- ``bsp.test_machine_world``: Verifies that ``bitbake world`` works
406 regardless of which machine is selected.
407
408- ``bsp.test_machine_signatures``: Verifies that building for a
409 particular machine affects only the signature of tasks specific to
410 that machine.
411
412- ``distro.test_distro_defines_distros``: Tests if a DISTRO layer has
413 distro configurations.
414
415- ``distro.test_distro_no_set_distros``: Tests to ensure a DISTRO layer
416 does not set the distribution when the layer is added.
417
418Enabling Your Layer
419-------------------
420
421Before the OpenEmbedded build system can use your new layer, you need to
422enable it. To enable your layer, simply add your layer's path to the
Andrew Geissler09036742021-06-25 14:25:14 -0500423:term:`BBLAYERS` variable in your ``conf/bblayers.conf`` file, which is
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500424found in the :term:`Build Directory`.
Andrew Geissler5199d832021-09-24 16:47:35 -0500425The following example shows how to enable your new
426``meta-mylayer`` layer (note how your new layer exists outside of
427the official ``poky`` repository which you would have checked out earlier)::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500428
429 # POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
430 # changes incompatibly
431 POKY_BBLAYERS_CONF_VERSION = "2"
432 BBPATH = "${TOPDIR}"
433 BBFILES ?= ""
434 BBLAYERS ?= " \
435 /home/user/poky/meta \
436 /home/user/poky/meta-poky \
437 /home/user/poky/meta-yocto-bsp \
Andrew Geissler5199d832021-09-24 16:47:35 -0500438 /home/user/mystuff/meta-mylayer \
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500439 "
440
441BitBake parses each ``conf/layer.conf`` file from the top down as
Andrew Geissler09036742021-06-25 14:25:14 -0500442specified in the :term:`BBLAYERS` variable within the ``conf/bblayers.conf``
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500443file. During the processing of each ``conf/layer.conf`` file, BitBake
444adds the recipes, classes and configurations contained within the
445particular layer to the source directory.
446
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500447Appending Other Layers Metadata With Your Layer
448-----------------------------------------------
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500449
450A recipe that appends Metadata to another recipe is called a BitBake
451append file. A BitBake append file uses the ``.bbappend`` file type
452suffix, while the corresponding recipe to which Metadata is being
453appended uses the ``.bb`` file type suffix.
454
455You can use a ``.bbappend`` file in your layer to make additions or
456changes to the content of another layer's recipe without having to copy
457the other layer's recipe into your layer. Your ``.bbappend`` file
458resides in your layer, while the main ``.bb`` recipe file to which you
459are appending Metadata resides in a different layer.
460
461Being able to append information to an existing recipe not only avoids
462duplication, but also automatically applies recipe changes from a
463different layer into your layer. If you were copying recipes, you would
464have to manually merge changes as they occur.
465
466When you create an append file, you must use the same root name as the
467corresponding recipe file. For example, the append file
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500468``someapp_3.1.bbappend`` must apply to ``someapp_3.1.bb``. This
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500469means the original recipe and append filenames are version
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500470number-specific. If the corresponding recipe is renamed to update to a
471newer version, you must also rename and possibly update the
472corresponding ``.bbappend`` as well. During the build process, BitBake
473displays an error on starting if it detects a ``.bbappend`` file that
474does not have a corresponding recipe with a matching name. See the
475:term:`BB_DANGLINGAPPENDS_WARNONLY`
476variable for information on how to handle this error.
477
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500478Overlaying a File Using Your Layer
479~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
480
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500481As an example, consider the main formfactor recipe and a corresponding
482formfactor append file both from the :term:`Source Directory`.
483Here is the main
484formfactor recipe, which is named ``formfactor_0.0.bb`` and located in
Andrew Geisslerc926e172021-05-07 16:11:35 -0500485the "meta" layer at ``meta/recipes-bsp/formfactor``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500486
487 SUMMARY = "Device formfactor information"
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500488 DESCRIPTION = "A formfactor configuration file provides information about the \
489 target hardware for which the image is being built and information that the \
490 build system cannot obtain from other sources such as the kernel."
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500491 SECTION = "base"
492 LICENSE = "MIT"
493 LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
494 PR = "r45"
495
496 SRC_URI = "file://config file://machconfig"
497 S = "${WORKDIR}"
498
499 PACKAGE_ARCH = "${MACHINE_ARCH}"
500 INHIBIT_DEFAULT_DEPS = "1"
501
502 do_install() {
503 # Install file only if it has contents
504 install -d ${D}${sysconfdir}/formfactor/
505 install -m 0644 ${S}/config ${D}${sysconfdir}/formfactor/
506 if [ -s "${S}/machconfig" ]; then
507 install -m 0644 ${S}/machconfig ${D}${sysconfdir}/formfactor/
508 fi
509 }
510
511In the main recipe, note the :term:`SRC_URI`
512variable, which tells the OpenEmbedded build system where to find files
513during the build.
514
515Following is the append file, which is named ``formfactor_0.0.bbappend``
516and is from the Raspberry Pi BSP Layer named ``meta-raspberrypi``. The
Andrew Geisslerc926e172021-05-07 16:11:35 -0500517file is in the layer at ``recipes-bsp/formfactor``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500518
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500519 FILESEXTRAPATHS:prepend := "${THISDIR}/${PN}:"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500520
521By default, the build system uses the
522:term:`FILESPATH` variable to
523locate files. This append file extends the locations by setting the
524:term:`FILESEXTRAPATHS`
525variable. Setting this variable in the ``.bbappend`` file is the most
526reliable and recommended method for adding directories to the search
527path used by the build system to find files.
528
529The statement in this example extends the directories to include
530``${``\ :term:`THISDIR`\ ``}/${``\ :term:`PN`\ ``}``,
531which resolves to a directory named ``formfactor`` in the same directory
532in which the append file resides (i.e.
533``meta-raspberrypi/recipes-bsp/formfactor``. This implies that you must
534have the supporting directory structure set up that will contain any
535files or patches you will be including from the layer.
536
537Using the immediate expansion assignment operator ``:=`` is important
Andrew Geissler09036742021-06-25 14:25:14 -0500538because of the reference to :term:`THISDIR`. The trailing colon character is
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500539important as it ensures that items in the list remain colon-separated.
540
541.. note::
542
Andrew Geissler09036742021-06-25 14:25:14 -0500543 BitBake automatically defines the :term:`THISDIR` variable. You should
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500544 never set this variable yourself. Using ":prepend" as part of the
Andrew Geissler09036742021-06-25 14:25:14 -0500545 :term:`FILESEXTRAPATHS` ensures your path will be searched prior to other
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500546 paths in the final list.
547
548 Also, not all append files add extra files. Many append files simply
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700549 allow to add build options (e.g. ``systemd``). For these cases, your
Andrew Geissler09036742021-06-25 14:25:14 -0500550 append file would not even use the :term:`FILESEXTRAPATHS` statement.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500551
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500552The end result of this ``.bbappend`` file is that on a Raspberry Pi, where
553``rpi`` will exist in the list of :term:`OVERRIDES`, the file
554``meta-raspberrypi/recipes-bsp/formfactor/formfactor/rpi/machconfig`` will be
555used during :ref:`ref-tasks-fetch` and the test for a non-zero file size in
556:ref:`ref-tasks-install` will return true, and the file will be installed.
557
Andrew Geissler5199d832021-09-24 16:47:35 -0500558Installing Additional Files Using Your Layer
559~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
560
561As another example, consider the main ``xserver-xf86-config`` recipe and a
562corresponding ``xserver-xf86-config`` append file both from the :term:`Source
563Directory`. Here is the main ``xserver-xf86-config`` recipe, which is named
564``xserver-xf86-config_0.1.bb`` and located in the "meta" layer at
565``meta/recipes-graphics/xorg-xserver``::
566
567 SUMMARY = "X.Org X server configuration file"
568 HOMEPAGE = "http://www.x.org"
569 SECTION = "x11/base"
570 LICENSE = "MIT-X"
571 LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
572 PR = "r33"
573
574 SRC_URI = "file://xorg.conf"
575
576 S = "${WORKDIR}"
577
578 CONFFILES:${PN} = "${sysconfdir}/X11/xorg.conf"
579
580 PACKAGE_ARCH = "${MACHINE_ARCH}"
581 ALLOW_EMPTY:${PN} = "1"
582
583 do_install () {
584 if test -s ${WORKDIR}/xorg.conf; then
585 install -d ${D}/${sysconfdir}/X11
586 install -m 0644 ${WORKDIR}/xorg.conf ${D}/${sysconfdir}/X11/
587 fi
588 }
589
590Following is the append file, which is named ``xserver-xf86-config_%.bbappend``
591and is from the Raspberry Pi BSP Layer named ``meta-raspberrypi``. The
592file is in the layer at ``recipes-graphics/xorg-xserver``::
593
594 FILESEXTRAPATHS:prepend := "${THISDIR}/${PN}:"
595
596 SRC_URI:append:rpi = " \
597 file://xorg.conf.d/98-pitft.conf \
598 file://xorg.conf.d/99-calibration.conf \
599 "
600 do_install:append:rpi () {
601 PITFT="${@bb.utils.contains("MACHINE_FEATURES", "pitft", "1", "0", d)}"
602 if [ "${PITFT}" = "1" ]; then
603 install -d ${D}/${sysconfdir}/X11/xorg.conf.d/
604 install -m 0644 ${WORKDIR}/xorg.conf.d/98-pitft.conf ${D}/${sysconfdir}/X11/xorg.conf.d/
605 install -m 0644 ${WORKDIR}/xorg.conf.d/99-calibration.conf ${D}/${sysconfdir}/X11/xorg.conf.d/
606 fi
607 }
608
609 FILES:${PN}:append:rpi = " ${sysconfdir}/X11/xorg.conf.d/*"
610
611Building off of the previous example, we once again are setting the
612:term:`FILESEXTRAPATHS` variable. In this case we are also using
613:term:`SRC_URI` to list additional source files to use when ``rpi`` is found in
614the list of :term:`OVERRIDES`. The :ref:`ref-tasks-install` task will then perform a
615check for an additional :term:`MACHINE_FEATURES` that if set will cause these
616additional files to be installed. These additional files are listed in
617:term:`FILES` so that they will be packaged.
618
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500619Prioritizing Your Layer
620-----------------------
621
622Each layer is assigned a priority value. Priority values control which
623layer takes precedence if there are recipe files with the same name in
624multiple layers. For these cases, the recipe file from the layer with a
625higher priority number takes precedence. Priority values also affect the
626order in which multiple ``.bbappend`` files for the same recipe are
627applied. You can either specify the priority manually, or allow the
628build system to calculate it based on the layer's dependencies.
629
630To specify the layer's priority manually, use the
631:term:`BBFILE_PRIORITY`
Andrew Geisslerc926e172021-05-07 16:11:35 -0500632variable and append the layer's root name::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500633
634 BBFILE_PRIORITY_mylayer = "1"
635
636.. note::
637
638 It is possible for a recipe with a lower version number
639 :term:`PV` in a layer that has a higher
640 priority to take precedence.
641
642 Also, the layer priority does not currently affect the precedence
643 order of ``.conf`` or ``.bbclass`` files. Future versions of BitBake
644 might address this.
645
646Managing Layers
647---------------
648
649You can use the BitBake layer management tool ``bitbake-layers`` to
650provide a view into the structure of recipes across a multi-layer
651project. Being able to generate output that reports on configured layers
652with their paths and priorities and on ``.bbappend`` files and their
653applicable recipes can help to reveal potential problems.
654
655For help on the BitBake layer management tool, use the following
Andrew Geisslerc926e172021-05-07 16:11:35 -0500656command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500657
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500658 $ bitbake-layers --help
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500659 NOTE: Starting bitbake server...
660 usage: bitbake-layers [-d] [-q] [-F] [--color COLOR] [-h] <subcommand> ...
661
662 BitBake layers utility
663
664 optional arguments:
665 -d, --debug Enable debug output
666 -q, --quiet Print only errors
667 -F, --force Force add without recipe parse verification
668 --color COLOR Colorize output (where COLOR is auto, always, never)
669 -h, --help show this help message and exit
670
671 subcommands:
672 <subcommand>
673 layerindex-fetch Fetches a layer from a layer index along with its
674 dependent layers, and adds them to conf/bblayers.conf.
675 layerindex-show-depends
676 Find layer dependencies from layer index.
677 add-layer Add one or more layers to bblayers.conf.
678 remove-layer Remove one or more layers from bblayers.conf.
679 flatten flatten layer configuration into a separate output
680 directory.
681 show-layers show current configured layers.
682 show-overlayed list overlayed recipes (where the same recipe exists
683 in another layer)
684 show-recipes list available recipes, showing the layer they are
685 provided by
686 show-appends list bbappend files and recipe files they apply to
687 show-cross-depends Show dependencies between recipes that cross layer
688 boundaries.
689 create-layer Create a basic layer
690
691 Use bitbake-layers <subcommand> --help to get help on a specific command
692
693The following list describes the available commands:
694
695- ``help:`` Displays general help or help on a specified command.
696
697- ``show-layers:`` Shows the current configured layers.
698
699- ``show-overlayed:`` Lists overlayed recipes. A recipe is overlayed
700 when a recipe with the same name exists in another layer that has a
701 higher layer priority.
702
703- ``show-recipes:`` Lists available recipes and the layers that
704 provide them.
705
706- ``show-appends:`` Lists ``.bbappend`` files and the recipe files to
707 which they apply.
708
709- ``show-cross-depends:`` Lists dependency relationships between
710 recipes that cross layer boundaries.
711
712- ``add-layer:`` Adds a layer to ``bblayers.conf``.
713
714- ``remove-layer:`` Removes a layer from ``bblayers.conf``
715
716- ``flatten:`` Flattens the layer configuration into a separate
717 output directory. Flattening your layer configuration builds a
718 "flattened" directory that contains the contents of all layers, with
719 any overlayed recipes removed and any ``.bbappend`` files appended to
720 the corresponding recipes. You might have to perform some manual
721 cleanup of the flattened layer as follows:
722
723 - Non-recipe files (such as patches) are overwritten. The flatten
724 command shows a warning for these files.
725
726 - Anything beyond the normal layer setup has been added to the
727 ``layer.conf`` file. Only the lowest priority layer's
728 ``layer.conf`` is used.
729
730 - Overridden and appended items from ``.bbappend`` files need to be
731 cleaned up. The contents of each ``.bbappend`` end up in the
732 flattened recipe. However, if there are appended or changed
733 variable values, you need to tidy these up yourself. Consider the
734 following example. Here, the ``bitbake-layers`` command adds the
735 line ``#### bbappended ...`` so that you know where the following
Andrew Geisslerc926e172021-05-07 16:11:35 -0500736 lines originate::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500737
738 ...
739 DESCRIPTION = "A useful utility"
740 ...
741 EXTRA_OECONF = "--enable-something"
742 ...
743
744 #### bbappended from meta-anotherlayer ####
745
746 DESCRIPTION = "Customized utility"
747 EXTRA_OECONF += "--enable-somethingelse"
748
749
Andrew Geisslerc926e172021-05-07 16:11:35 -0500750 Ideally, you would tidy up these utilities as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500751
752 ...
753 DESCRIPTION = "Customized utility"
754 ...
755 EXTRA_OECONF = "--enable-something --enable-somethingelse"
756 ...
757
758- ``layerindex-fetch``: Fetches a layer from a layer index, along
759 with its dependent layers, and adds the layers to the
760 ``conf/bblayers.conf`` file.
761
762- ``layerindex-show-depends``: Finds layer dependencies from the
763 layer index.
764
765- ``create-layer``: Creates a basic layer.
766
767Creating a General Layer Using the ``bitbake-layers`` Script
768------------------------------------------------------------
769
770The ``bitbake-layers`` script with the ``create-layer`` subcommand
771simplifies creating a new general layer.
772
773.. note::
774
775 - For information on BSP layers, see the ":ref:`bsp-guide/bsp:bsp layers`"
776 section in the Yocto
777 Project Board Specific (BSP) Developer's Guide.
778
779 - In order to use a layer with the OpenEmbedded build system, you
780 need to add the layer to your ``bblayers.conf`` configuration
Andrew Geissler09209ee2020-12-13 08:44:15 -0600781 file. See the ":ref:`dev-manual/common-tasks:adding a layer using the \`\`bitbake-layers\`\` script`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500782 section for more information.
783
784The default mode of the script's operation with this subcommand is to
785create a layer with the following:
786
787- A layer priority of 6.
788
789- A ``conf`` subdirectory that contains a ``layer.conf`` file.
790
791- A ``recipes-example`` subdirectory that contains a further
792 subdirectory named ``example``, which contains an ``example.bb``
793 recipe file.
794
795- A ``COPYING.MIT``, which is the license statement for the layer. The
796 script assumes you want to use the MIT license, which is typical for
797 most layers, for the contents of the layer itself.
798
799- A ``README`` file, which is a file describing the contents of your
800 new layer.
801
802In its simplest form, you can use the following command form to create a
803layer. The command creates a layer whose name corresponds to
Andrew Geisslerc926e172021-05-07 16:11:35 -0500804"your_layer_name" in the current directory::
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500805
806 $ bitbake-layers create-layer your_layer_name
807
808As an example, the following command creates a layer named ``meta-scottrif``
Andrew Geisslerc926e172021-05-07 16:11:35 -0500809in your home directory::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500810
811 $ cd /usr/home
812 $ bitbake-layers create-layer meta-scottrif
813 NOTE: Starting bitbake server...
814 Add your new layer with 'bitbake-layers add-layer meta-scottrif'
815
816If you want to set the priority of the layer to other than the default
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500817value of "6", you can either use the ``--priority`` option or you
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500818can edit the
819:term:`BBFILE_PRIORITY` value
820in the ``conf/layer.conf`` after the script creates it. Furthermore, if
821you want to give the example recipe file some name other than the
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500822default, you can use the ``--example-recipe-name`` option.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500823
824The easiest way to see how the ``bitbake-layers create-layer`` command
825works is to experiment with the script. You can also read the usage
Andrew Geisslerc926e172021-05-07 16:11:35 -0500826information by entering the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500827
828 $ bitbake-layers create-layer --help
829 NOTE: Starting bitbake server...
830 usage: bitbake-layers create-layer [-h] [--priority PRIORITY]
831 [--example-recipe-name EXAMPLERECIPE]
832 layerdir
833
834 Create a basic layer
835
836 positional arguments:
837 layerdir Layer directory to create
838
839 optional arguments:
840 -h, --help show this help message and exit
841 --priority PRIORITY, -p PRIORITY
842 Layer directory to create
843 --example-recipe-name EXAMPLERECIPE, -e EXAMPLERECIPE
844 Filename of the example recipe
845
846Adding a Layer Using the ``bitbake-layers`` Script
847--------------------------------------------------
848
849Once you create your general layer, you must add it to your
850``bblayers.conf`` file. Adding the layer to this configuration file
851makes the OpenEmbedded build system aware of your layer so that it can
852search it for metadata.
853
Andrew Geisslerc926e172021-05-07 16:11:35 -0500854Add your layer by using the ``bitbake-layers add-layer`` command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500855
856 $ bitbake-layers add-layer your_layer_name
857
858Here is an example that adds a
859layer named ``meta-scottrif`` to the configuration file. Following the
860command that adds the layer is another ``bitbake-layers`` command that
Andrew Geisslerc926e172021-05-07 16:11:35 -0500861shows the layers that are in your ``bblayers.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500862
863 $ bitbake-layers add-layer meta-scottrif
864 NOTE: Starting bitbake server...
865 Parsing recipes: 100% |##########################################################| Time: 0:00:49
866 Parsing of 1441 .bb files complete (0 cached, 1441 parsed). 2055 targets, 56 skipped, 0 masked, 0 errors.
867 $ bitbake-layers show-layers
868 NOTE: Starting bitbake server...
869 layer path priority
870 ==========================================================================
871 meta /home/scottrif/poky/meta 5
872 meta-poky /home/scottrif/poky/meta-poky 5
873 meta-yocto-bsp /home/scottrif/poky/meta-yocto-bsp 5
874 workspace /home/scottrif/poky/build/workspace 99
875 meta-scottrif /home/scottrif/poky/build/meta-scottrif 6
876
877
878Adding the layer to this file
879enables the build system to locate the layer during the build.
880
881.. note::
882
883 During a build, the OpenEmbedded build system looks in the layers
884 from the top of the list down to the bottom in that order.
885
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500886Customizing Images
887==================
888
889You can customize images to satisfy particular requirements. This
890section describes several methods and provides guidelines for each.
891
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500892Customizing Images Using ``local.conf``
893---------------------------------------
894
895Probably the easiest way to customize an image is to add a package by
896way of the ``local.conf`` configuration file. Because it is limited to
897local use, this method generally only allows you to add packages and is
898not as flexible as creating your own customized image. When you add
899packages using local variables this way, you need to realize that these
900variable changes are in effect for every build and consequently affect
901all images, which might not be what you require.
902
903To add a package to your image using the local configuration file, use
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500904the :term:`IMAGE_INSTALL` variable with the ``:append`` operator::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500905
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500906 IMAGE_INSTALL:append = " strace"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500907
Andrew Geissler5199d832021-09-24 16:47:35 -0500908Use of the syntax is important; specifically, the leading space
909after the opening quote and before the package name, which is
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500910``strace`` in this example. This space is required since the ``:append``
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500911operator does not add the space.
912
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500913Furthermore, you must use ``:append`` instead of the ``+=`` operator if
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500914you want to avoid ordering issues. The reason for this is because doing
915so unconditionally appends to the variable and avoids ordering problems
916due to the variable being set in image recipes and ``.bbclass`` files
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500917with operators like ``?=``. Using ``:append`` ensures the operation
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500918takes effect.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500919
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500920As shown in its simplest use, ``IMAGE_INSTALL:append`` affects all
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500921images. It is possible to extend the syntax so that the variable applies
Andrew Geisslerc926e172021-05-07 16:11:35 -0500922to a specific image only. Here is an example::
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500923
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500924 IMAGE_INSTALL:append:pn-core-image-minimal = " strace"
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500925
926This example adds ``strace`` to the ``core-image-minimal`` image only.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500927
928You can add packages using a similar approach through the
Andrew Geissler09036742021-06-25 14:25:14 -0500929:term:`CORE_IMAGE_EXTRA_INSTALL` variable. If you use this variable, only
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500930``core-image-*`` images are affected.
931
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500932Customizing Images Using Custom ``IMAGE_FEATURES`` and ``EXTRA_IMAGE_FEATURES``
933-------------------------------------------------------------------------------
934
935Another method for customizing your image is to enable or disable
936high-level image features by using the
937:term:`IMAGE_FEATURES` and
938:term:`EXTRA_IMAGE_FEATURES`
939variables. Although the functions for both variables are nearly
Andrew Geissler09036742021-06-25 14:25:14 -0500940equivalent, best practices dictate using :term:`IMAGE_FEATURES` from within
941a recipe and using :term:`EXTRA_IMAGE_FEATURES` from within your
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500942``local.conf`` file, which is found in the
943:term:`Build Directory`.
944
945To understand how these features work, the best reference is
Patrick Williams213cb262021-08-07 19:21:33 -0500946``meta/classes/image.bbclass``. This class lists out the available
Andrew Geissler09036742021-06-25 14:25:14 -0500947:term:`IMAGE_FEATURES` of which most map to package groups while some, such
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500948as ``debug-tweaks`` and ``read-only-rootfs``, resolve as general
949configuration settings.
950
Andrew Geissler09036742021-06-25 14:25:14 -0500951In summary, the file looks at the contents of the :term:`IMAGE_FEATURES`
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500952variable and then maps or configures the feature accordingly. Based on
953this information, the build system automatically adds the appropriate
954packages or configurations to the
955:term:`IMAGE_INSTALL` variable.
956Effectively, you are enabling extra features by extending the class or
957creating a custom class for use with specialized image ``.bb`` files.
958
Andrew Geissler09036742021-06-25 14:25:14 -0500959Use the :term:`EXTRA_IMAGE_FEATURES` variable from within your local
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500960configuration file. Using a separate area from which to enable features
961with this variable helps you avoid overwriting the features in the image
Andrew Geissler09036742021-06-25 14:25:14 -0500962recipe that are enabled with :term:`IMAGE_FEATURES`. The value of
963:term:`EXTRA_IMAGE_FEATURES` is added to :term:`IMAGE_FEATURES` within
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500964``meta/conf/bitbake.conf``.
965
966To illustrate how you can use these variables to modify your image,
967consider an example that selects the SSH server. The Yocto Project ships
968with two SSH servers you can use with your images: Dropbear and OpenSSH.
969Dropbear is a minimal SSH server appropriate for resource-constrained
970environments, while OpenSSH is a well-known standard SSH server
971implementation. By default, the ``core-image-sato`` image is configured
972to use Dropbear. The ``core-image-full-cmdline`` and ``core-image-lsb``
973images both include OpenSSH. The ``core-image-minimal`` image does not
974contain an SSH server.
975
976You can customize your image and change these defaults. Edit the
Andrew Geissler09036742021-06-25 14:25:14 -0500977:term:`IMAGE_FEATURES` variable in your recipe or use the
978:term:`EXTRA_IMAGE_FEATURES` in your ``local.conf`` file so that it
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500979configures the image you are working with to include
980``ssh-server-dropbear`` or ``ssh-server-openssh``.
981
982.. note::
983
Andrew Geissler09209ee2020-12-13 08:44:15 -0600984 See the ":ref:`ref-manual/features:image features`" section in the Yocto
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500985 Project Reference Manual for a complete list of image features that ship
986 with the Yocto Project.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500987
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500988Customizing Images Using Custom .bb Files
989-----------------------------------------
990
991You can also customize an image by creating a custom recipe that defines
992additional software as part of the image. The following example shows
Andrew Geisslerc926e172021-05-07 16:11:35 -0500993the form for the two lines you need::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500994
995 IMAGE_INSTALL = "packagegroup-core-x11-base package1 package2"
996 inherit core-image
997
998Defining the software using a custom recipe gives you total control over
999the contents of the image. It is important to use the correct names of
Andrew Geissler09036742021-06-25 14:25:14 -05001000packages in the :term:`IMAGE_INSTALL` variable. You must use the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001001OpenEmbedded notation and not the Debian notation for the names (e.g.
1002``glibc-dev`` instead of ``libc6-dev``).
1003
1004The other method for creating a custom image is to base it on an
1005existing image. For example, if you want to create an image based on
1006``core-image-sato`` but add the additional package ``strace`` to the
1007image, copy the ``meta/recipes-sato/images/core-image-sato.bb`` to a new
Andrew Geisslerc926e172021-05-07 16:11:35 -05001008``.bb`` and add the following line to the end of the copy::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001009
1010 IMAGE_INSTALL += "strace"
1011
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001012Customizing Images Using Custom Package Groups
1013----------------------------------------------
1014
1015For complex custom images, the best approach for customizing an image is
1016to create a custom package group recipe that is used to build the image
1017or images. A good example of a package group recipe is
1018``meta/recipes-core/packagegroups/packagegroup-base.bb``.
1019
Andrew Geissler09036742021-06-25 14:25:14 -05001020If you examine that recipe, you see that the :term:`PACKAGES` variable lists
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001021the package group packages to produce. The ``inherit packagegroup``
1022statement sets appropriate default values and automatically adds
1023``-dev``, ``-dbg``, and ``-ptest`` complementary packages for each
Andrew Geissler09036742021-06-25 14:25:14 -05001024package specified in the :term:`PACKAGES` statement.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001025
1026.. note::
1027
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001028 The ``inherit packagegroup`` line should be located near the top of the
Andrew Geissler09036742021-06-25 14:25:14 -05001029 recipe, certainly before the :term:`PACKAGES` statement.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001030
Andrew Geissler09036742021-06-25 14:25:14 -05001031For each package you specify in :term:`PACKAGES`, you can use :term:`RDEPENDS`
1032and :term:`RRECOMMENDS` entries to provide a list of packages the parent
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001033task package should contain. You can see examples of these further down
1034in the ``packagegroup-base.bb`` recipe.
1035
1036Here is a short, fabricated example showing the same basic pieces for a
1037hypothetical packagegroup defined in ``packagegroup-custom.bb``, where
Andrew Geissler09036742021-06-25 14:25:14 -05001038the variable :term:`PN` is the standard way to abbreviate the reference to
Andrew Geisslerc926e172021-05-07 16:11:35 -05001039the full packagegroup name ``packagegroup-custom``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001040
1041 DESCRIPTION = "My Custom Package Groups"
1042
1043 inherit packagegroup
1044
1045 PACKAGES = "\
1046 ${PN}-apps \
1047 ${PN}-tools \
1048 "
1049
Patrick Williams0ca19cc2021-08-16 14:03:13 -05001050 RDEPENDS:${PN}-apps = "\
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001051 dropbear \
1052 portmap \
1053 psplash"
1054
Patrick Williams0ca19cc2021-08-16 14:03:13 -05001055 RDEPENDS:${PN}-tools = "\
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001056 oprofile \
1057 oprofileui-server \
1058 lttng-tools"
1059
Patrick Williams0ca19cc2021-08-16 14:03:13 -05001060 RRECOMMENDS:${PN}-tools = "\
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001061 kernel-module-oprofile"
1062
1063In the previous example, two package group packages are created with
1064their dependencies and their recommended package dependencies listed:
1065``packagegroup-custom-apps``, and ``packagegroup-custom-tools``. To
1066build an image using these package group packages, you need to add
1067``packagegroup-custom-apps`` and/or ``packagegroup-custom-tools`` to
Andrew Geissler09036742021-06-25 14:25:14 -05001068:term:`IMAGE_INSTALL`. For other forms of image dependencies see the other
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001069areas of this section.
1070
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001071Customizing an Image Hostname
1072-----------------------------
1073
1074By default, the configured hostname (i.e. ``/etc/hostname``) in an image
1075is the same as the machine name. For example, if
1076:term:`MACHINE` equals "qemux86", the
1077configured hostname written to ``/etc/hostname`` is "qemux86".
1078
1079You can customize this name by altering the value of the "hostname"
1080variable in the ``base-files`` recipe using either an append file or a
Andrew Geisslerc926e172021-05-07 16:11:35 -05001081configuration file. Use the following in an append file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001082
1083 hostname = "myhostname"
1084
Andrew Geisslerc926e172021-05-07 16:11:35 -05001085Use the following in a configuration file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001086
Patrick Williams0ca19cc2021-08-16 14:03:13 -05001087 hostname:pn-base-files = "myhostname"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001088
1089Changing the default value of the variable "hostname" can be useful in
1090certain situations. For example, suppose you need to do extensive
1091testing on an image and you would like to easily identify the image
1092under test from existing images with typical default hostnames. In this
1093situation, you could change the default hostname to "testme", which
1094results in all the images using the name "testme". Once testing is
1095complete and you do not need to rebuild the image for test any longer,
1096you can easily reset the default hostname.
1097
1098Another point of interest is that if you unset the variable, the image
1099will have no default hostname in the filesystem. Here is an example that
Andrew Geisslerc926e172021-05-07 16:11:35 -05001100unsets the variable in a configuration file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001101
Patrick Williams0ca19cc2021-08-16 14:03:13 -05001102 hostname:pn-base-files = ""
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001103
1104Having no default hostname in the filesystem is suitable for
1105environments that use dynamic hostnames such as virtual machines.
1106
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001107Writing a New Recipe
1108====================
1109
1110Recipes (``.bb`` files) are fundamental components in the Yocto Project
1111environment. Each software component built by the OpenEmbedded build
1112system requires a recipe to define the component. This section describes
1113how to create, write, and test a new recipe.
1114
1115.. note::
1116
1117 For information on variables that are useful for recipes and for
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001118 information about recipe naming issues, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001119 ":ref:`ref-manual/varlocality:recipes`" section of the Yocto Project
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001120 Reference Manual.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001121
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001122Overview
1123--------
1124
1125The following figure shows the basic process for creating a new recipe.
1126The remainder of the section provides details for the steps.
1127
1128.. image:: figures/recipe-workflow.png
1129 :align: center
1130
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001131Locate or Automatically Create a Base Recipe
1132--------------------------------------------
1133
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001134You can always write a recipe from scratch. However, there are three choices
1135that can help you quickly get started with a new recipe:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001136
1137- ``devtool add``: A command that assists in creating a recipe and an
1138 environment conducive to development.
1139
1140- ``recipetool create``: A command provided by the Yocto Project that
1141 automates creation of a base recipe based on the source files.
1142
1143- *Existing Recipes:* Location and modification of an existing recipe
1144 that is similar in function to the recipe you need.
1145
1146.. note::
1147
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001148 For information on recipe syntax, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001149 ":ref:`dev-manual/common-tasks:recipe syntax`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001150
1151Creating the Base Recipe Using ``devtool add``
1152~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1153
1154The ``devtool add`` command uses the same logic for auto-creating the
1155recipe as ``recipetool create``, which is listed below. Additionally,
1156however, ``devtool add`` sets up an environment that makes it easy for
1157you to patch the source and to make changes to the recipe as is often
1158necessary when adding a recipe to build a new piece of software to be
1159included in a build.
1160
1161You can find a complete description of the ``devtool add`` command in
Andrew Geissler09209ee2020-12-13 08:44:15 -06001162the ":ref:`sdk-manual/extensible:a closer look at \`\`devtool add\`\``" section
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001163in the Yocto Project Application Development and the Extensible Software
1164Development Kit (eSDK) manual.
1165
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001166Creating the Base Recipe Using ``recipetool create``
1167~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1168
1169``recipetool create`` automates creation of a base recipe given a set of
1170source code files. As long as you can extract or point to the source
1171files, the tool will construct a recipe and automatically configure all
1172pre-build information into the recipe. For example, suppose you have an
1173application that builds using Autotools. Creating the base recipe using
1174``recipetool`` results in a recipe that has the pre-build dependencies,
1175license requirements, and checksums configured.
1176
1177To run the tool, you just need to be in your
1178:term:`Build Directory` and have sourced the
1179build environment setup script (i.e.
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001180:ref:`structure-core-script`).
Andrew Geisslerc926e172021-05-07 16:11:35 -05001181To get help on the tool, use the following command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001182
1183 $ recipetool -h
1184 NOTE: Starting bitbake server...
1185 usage: recipetool [-d] [-q] [--color COLOR] [-h] <subcommand> ...
1186
1187 OpenEmbedded recipe tool
1188
1189 options:
1190 -d, --debug Enable debug output
1191 -q, --quiet Print only errors
1192 --color COLOR Colorize output (where COLOR is auto, always, never)
1193 -h, --help show this help message and exit
1194
1195 subcommands:
1196 create Create a new recipe
1197 newappend Create a bbappend for the specified target in the specified
1198 layer
1199 setvar Set a variable within a recipe
1200 appendfile Create/update a bbappend to replace a target file
1201 appendsrcfiles Create/update a bbappend to add or replace source files
1202 appendsrcfile Create/update a bbappend to add or replace a source file
1203 Use recipetool <subcommand> --help to get help on a specific command
1204
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001205Running ``recipetool create -o OUTFILE`` creates the base recipe and
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001206locates it properly in the layer that contains your source files.
1207Following are some syntax examples:
1208
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001209 - Use this syntax to generate a recipe based on source. Once generated,
Andrew Geisslerc926e172021-05-07 16:11:35 -05001210 the recipe resides in the existing source code layer::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001211
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001212 recipetool create -o OUTFILE source
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001213
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001214 - Use this syntax to generate a recipe using code that
1215 you extract from source. The extracted code is placed in its own layer
Andrew Geissler09036742021-06-25 14:25:14 -05001216 defined by :term:`EXTERNALSRC`.
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001217 ::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001218
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001219 recipetool create -o OUTFILE -x EXTERNALSRC source
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001220
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001221 - Use this syntax to generate a recipe based on source. The options
1222 direct ``recipetool`` to generate debugging information. Once generated,
Andrew Geisslerc926e172021-05-07 16:11:35 -05001223 the recipe resides in the existing source code layer::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001224
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001225 recipetool create -d -o OUTFILE source
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001226
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001227Locating and Using a Similar Recipe
1228~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1229
1230Before writing a recipe from scratch, it is often useful to discover
1231whether someone else has already written one that meets (or comes close
1232to meeting) your needs. The Yocto Project and OpenEmbedded communities
1233maintain many recipes that might be candidates for what you are doing.
Andrew Geisslerd1e89492021-02-12 15:35:20 -06001234You can find a good central index of these recipes in the
1235:oe_layerindex:`OpenEmbedded Layer Index <>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001236
1237Working from an existing recipe or a skeleton recipe is the best way to
1238get started. Here are some points on both methods:
1239
1240- *Locate and modify a recipe that is close to what you want to do:*
1241 This method works when you are familiar with the current recipe
1242 space. The method does not work so well for those new to the Yocto
1243 Project or writing recipes.
1244
1245 Some risks associated with this method are using a recipe that has
1246 areas totally unrelated to what you are trying to accomplish with
1247 your recipe, not recognizing areas of the recipe that you might have
1248 to add from scratch, and so forth. All these risks stem from
1249 unfamiliarity with the existing recipe space.
1250
1251- *Use and modify the following skeleton recipe:* If for some reason
1252 you do not want to use ``recipetool`` and you cannot find an existing
1253 recipe that is close to meeting your needs, you can use the following
1254 structure to provide the fundamental areas of a new recipe.
1255 ::
1256
1257 DESCRIPTION = ""
1258 HOMEPAGE = ""
1259 LICENSE = ""
1260 SECTION = ""
1261 DEPENDS = ""
1262 LIC_FILES_CHKSUM = ""
1263
1264 SRC_URI = ""
1265
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001266Storing and Naming the Recipe
1267-----------------------------
1268
1269Once you have your base recipe, you should put it in your own layer and
1270name it appropriately. Locating it correctly ensures that the
1271OpenEmbedded build system can find it when you use BitBake to process
1272the recipe.
1273
1274- *Storing Your Recipe:* The OpenEmbedded build system locates your
1275 recipe through the layer's ``conf/layer.conf`` file and the
1276 :term:`BBFILES` variable. This
1277 variable sets up a path from which the build system can locate
Andrew Geisslerc926e172021-05-07 16:11:35 -05001278 recipes. Here is the typical use::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001279
1280 BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
1281 ${LAYERDIR}/recipes-*/*/*.bbappend"
1282
1283 Consequently, you need to be sure you locate your new recipe inside
1284 your layer such that it can be found.
1285
1286 You can find more information on how layers are structured in the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001287 ":ref:`dev-manual/common-tasks:understanding and creating layers`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001288
1289- *Naming Your Recipe:* When you name your recipe, you need to follow
Andrew Geisslerc926e172021-05-07 16:11:35 -05001290 this naming convention::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001291
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001292 basename_version.bb
1293
1294 Use lower-cased characters and do not include the reserved suffixes
1295 ``-native``, ``-cross``, ``-initial``, or ``-dev`` casually (i.e. do not use
1296 them as part of your recipe name unless the string applies). Here are some
1297 examples:
1298
1299 .. code-block:: none
1300
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001301 cups_1.7.0.bb
1302 gawk_4.0.2.bb
1303 irssi_0.8.16-rc1.bb
1304
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001305Running a Build on the Recipe
1306-----------------------------
1307
1308Creating a new recipe is usually an iterative process that requires
1309using BitBake to process the recipe multiple times in order to
1310progressively discover and add information to the recipe file.
1311
1312Assuming you have sourced the build environment setup script (i.e.
1313:ref:`structure-core-script`) and you are in
1314the :term:`Build Directory`, use
1315BitBake to process your recipe. All you need to provide is the
Andrew Geisslerc926e172021-05-07 16:11:35 -05001316``basename`` of the recipe as described in the previous section::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001317
1318 $ bitbake basename
1319
1320During the build, the OpenEmbedded build system creates a temporary work
1321directory for each recipe
1322(``${``\ :term:`WORKDIR`\ ``}``)
1323where it keeps extracted source files, log files, intermediate
1324compilation and packaging files, and so forth.
1325
1326The path to the per-recipe temporary work directory depends on the
1327context in which it is being built. The quickest way to find this path
Andrew Geisslerc926e172021-05-07 16:11:35 -05001328is to have BitBake return it by running the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001329
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001330 $ bitbake -e basename | grep ^WORKDIR=
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001331
1332As an example, assume a Source Directory
1333top-level folder named ``poky``, a default Build Directory at
1334``poky/build``, and a ``qemux86-poky-linux`` machine target system.
1335Furthermore, suppose your recipe is named ``foo_1.3.0.bb``. In this
1336case, the work directory the build system uses to build the package
Andrew Geisslerc926e172021-05-07 16:11:35 -05001337would be as follows::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001338
1339 poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0
1340
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001341Inside this directory you can find sub-directories such as ``image``,
1342``packages-split``, and ``temp``. After the build, you can examine these
1343to determine how well the build went.
1344
1345.. note::
1346
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001347 You can find log files for each task in the recipe's ``temp``
1348 directory (e.g. ``poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0/temp``).
1349 Log files are named ``log.taskname`` (e.g. ``log.do_configure``,
1350 ``log.do_fetch``, and ``log.do_compile``).
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001351
1352You can find more information about the build process in
Andrew Geissler09209ee2020-12-13 08:44:15 -06001353":doc:`/overview-manual/development-environment`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001354chapter of the Yocto Project Overview and Concepts Manual.
1355
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001356Fetching Code
1357-------------
1358
1359The first thing your recipe must do is specify how to fetch the source
1360files. Fetching is controlled mainly through the
1361:term:`SRC_URI` variable. Your recipe
Andrew Geissler09036742021-06-25 14:25:14 -05001362must have a :term:`SRC_URI` variable that points to where the source is
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001363located. For a graphical representation of source locations, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001364":ref:`overview-manual/concepts:sources`" section in
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001365the Yocto Project Overview and Concepts Manual.
1366
1367The :ref:`ref-tasks-fetch` task uses
Andrew Geissler09036742021-06-25 14:25:14 -05001368the prefix of each entry in the :term:`SRC_URI` variable value to determine
Andrew Geissler09209ee2020-12-13 08:44:15 -06001369which :ref:`fetcher <bitbake:bitbake-user-manual/bitbake-user-manual-fetching:fetchers>` to use to get your
Andrew Geissler09036742021-06-25 14:25:14 -05001370source files. It is the :term:`SRC_URI` variable that triggers the fetcher.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001371The :ref:`ref-tasks-patch` task uses
1372the variable after source is fetched to apply patches. The OpenEmbedded
1373build system uses
1374:term:`FILESOVERRIDES` for
Andrew Geissler09036742021-06-25 14:25:14 -05001375scanning directory locations for local files in :term:`SRC_URI`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001376
Andrew Geissler09036742021-06-25 14:25:14 -05001377The :term:`SRC_URI` variable in your recipe must define each unique location
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001378for your source files. It is good practice to not hard-code version
Andrew Geissler5f350902021-07-23 13:09:54 -04001379numbers in a URL used in :term:`SRC_URI`. Rather than hard-code these
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001380values, use ``${``\ :term:`PV`\ ``}``,
1381which causes the fetch process to use the version specified in the
1382recipe filename. Specifying the version in this manner means that
1383upgrading the recipe to a future version is as simple as renaming the
1384recipe to match the new version.
1385
1386Here is a simple example from the
1387``meta/recipes-devtools/strace/strace_5.5.bb`` recipe where the source
1388comes from a single tarball. Notice the use of the
Andrew Geisslerc926e172021-05-07 16:11:35 -05001389:term:`PV` variable::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001390
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001391 SRC_URI = "https://strace.io/files/${PV}/strace-${PV}.tar.xz \
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001392
Andrew Geissler09036742021-06-25 14:25:14 -05001393Files mentioned in :term:`SRC_URI` whose names end in a typical archive
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001394extension (e.g. ``.tar``, ``.tar.gz``, ``.tar.bz2``, ``.zip``, and so
1395forth), are automatically extracted during the
1396:ref:`ref-tasks-unpack` task. For
1397another example that specifies these types of files, see the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001398":ref:`dev-manual/common-tasks:autotooled package`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001399
1400Another way of specifying source is from an SCM. For Git repositories,
1401you must specify :term:`SRCREV` and
1402you should specify :term:`PV` to include
1403the revision with :term:`SRCPV`. Here
1404is an example from the recipe
Andrew Geisslerc926e172021-05-07 16:11:35 -05001405``meta/recipes-kernel/blktrace/blktrace_git.bb``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001406
1407 SRCREV = "d6918c8832793b4205ed3bfede78c2f915c23385"
1408
1409 PR = "r6"
1410 PV = "1.0.5+git${SRCPV}"
1411
1412 SRC_URI = "git://git.kernel.dk/blktrace.git \
1413 file://ldflags.patch"
1414
Andrew Geissler09036742021-06-25 14:25:14 -05001415If your :term:`SRC_URI` statement includes URLs pointing to individual files
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001416fetched from a remote server other than a version control system,
1417BitBake attempts to verify the files against checksums defined in your
1418recipe to ensure they have not been tampered with or otherwise modified
1419since the recipe was written. Two checksums are used:
1420``SRC_URI[md5sum]`` and ``SRC_URI[sha256sum]``.
1421
Andrew Geissler09036742021-06-25 14:25:14 -05001422If your :term:`SRC_URI` variable points to more than a single URL (excluding
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001423SCM URLs), you need to provide the ``md5`` and ``sha256`` checksums for
1424each URL. For these cases, you provide a name for each URL as part of
Andrew Geissler09036742021-06-25 14:25:14 -05001425the :term:`SRC_URI` and then reference that name in the subsequent checksum
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001426statements. Here is an example combining lines from the files
Andrew Geisslerc926e172021-05-07 16:11:35 -05001427``git.inc`` and ``git_2.24.1.bb``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001428
1429 SRC_URI = "${KERNELORG_MIRROR}/software/scm/git/git-${PV}.tar.gz;name=tarball \
1430 ${KERNELORG_MIRROR}/software/scm/git/git-manpages-${PV}.tar.gz;name=manpages"
1431
1432 SRC_URI[tarball.md5sum] = "166bde96adbbc11c8843d4f8f4f9811b"
1433 SRC_URI[tarball.sha256sum] = "ad5334956301c86841eb1e5b1bb20884a6bad89a10a6762c958220c7cf64da02"
1434 SRC_URI[manpages.md5sum] = "31c2272a8979022497ba3d4202df145d"
1435 SRC_URI[manpages.sha256sum] = "9a7ae3a093bea39770eb96ca3e5b40bff7af0b9f6123f089d7821d0e5b8e1230"
1436
1437Proper values for ``md5`` and ``sha256`` checksums might be available
1438with other signatures on the download page for the upstream source (e.g.
1439``md5``, ``sha1``, ``sha256``, ``GPG``, and so forth). Because the
1440OpenEmbedded build system only deals with ``sha256sum`` and ``md5sum``,
1441you should verify all the signatures you find by hand.
1442
Andrew Geissler09036742021-06-25 14:25:14 -05001443If no :term:`SRC_URI` checksums are specified when you attempt to build the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001444recipe, or you provide an incorrect checksum, the build will produce an
1445error for each missing or incorrect checksum. As part of the error
1446message, the build system provides the checksum string corresponding to
1447the fetched file. Once you have the correct checksums, you can copy and
1448paste them into your recipe and then run the build again to continue.
1449
1450.. note::
1451
1452 As mentioned, if the upstream source provides signatures for
1453 verifying the downloaded source code, you should verify those
1454 manually before setting the checksum values in the recipe and
1455 continuing with the build.
1456
1457This final example is a bit more complicated and is from the
1458``meta/recipes-sato/rxvt-unicode/rxvt-unicode_9.20.bb`` recipe. The
Andrew Geissler09036742021-06-25 14:25:14 -05001459example's :term:`SRC_URI` statement identifies multiple files as the source
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001460files for the recipe: a tarball, a patch file, a desktop file, and an
1461icon.
1462::
1463
1464 SRC_URI = "http://dist.schmorp.de/rxvt-unicode/Attic/rxvt-unicode-${PV}.tar.bz2 \
1465 file://xwc.patch \
1466 file://rxvt.desktop \
1467 file://rxvt.png"
1468
1469When you specify local files using the ``file://`` URI protocol, the
1470build system fetches files from the local machine. The path is relative
1471to the :term:`FILESPATH` variable
1472and searches specific directories in a certain order:
1473``${``\ :term:`BP`\ ``}``,
1474``${``\ :term:`BPN`\ ``}``, and
1475``files``. The directories are assumed to be subdirectories of the
1476directory in which the recipe or append file resides. For another
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001477example that specifies these types of files, see the
1478":ref:`dev-manual/common-tasks:single .c file package (hello world!)`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001479
1480The previous example also specifies a patch file. Patch files are files
1481whose names usually end in ``.patch`` or ``.diff`` but can end with
1482compressed suffixes such as ``diff.gz`` and ``patch.bz2``, for example.
1483The build system automatically applies patches as described in the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001484":ref:`dev-manual/common-tasks:patching code`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001485
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001486Unpacking Code
1487--------------
1488
1489During the build, the
1490:ref:`ref-tasks-unpack` task unpacks
1491the source with ``${``\ :term:`S`\ ``}``
1492pointing to where it is unpacked.
1493
1494If you are fetching your source files from an upstream source archived
1495tarball and the tarball's internal structure matches the common
1496convention of a top-level subdirectory named
1497``${``\ :term:`BPN`\ ``}-${``\ :term:`PV`\ ``}``,
Andrew Geissler09036742021-06-25 14:25:14 -05001498then you do not need to set :term:`S`. However, if :term:`SRC_URI` specifies to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001499fetch source from an archive that does not use this convention, or from
Andrew Geissler09036742021-06-25 14:25:14 -05001500an SCM like Git or Subversion, your recipe needs to define :term:`S`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001501
1502If processing your recipe using BitBake successfully unpacks the source
1503files, you need to be sure that the directory pointed to by ``${S}``
1504matches the structure of the source.
1505
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001506Patching Code
1507-------------
1508
1509Sometimes it is necessary to patch code after it has been fetched. Any
Andrew Geissler09036742021-06-25 14:25:14 -05001510files mentioned in :term:`SRC_URI` whose names end in ``.patch`` or
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001511``.diff`` or compressed versions of these suffixes (e.g. ``diff.gz`` are
1512treated as patches. The
1513:ref:`ref-tasks-patch` task
1514automatically applies these patches.
1515
1516The build system should be able to apply patches with the "-p1" option
1517(i.e. one directory level in the path will be stripped off). If your
1518patch needs to have more directory levels stripped off, specify the
Andrew Geissler09036742021-06-25 14:25:14 -05001519number of levels using the "striplevel" option in the :term:`SRC_URI` entry
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001520for the patch. Alternatively, if your patch needs to be applied in a
1521specific subdirectory that is not specified in the patch file, use the
1522"patchdir" option in the entry.
1523
1524As with all local files referenced in
1525:term:`SRC_URI` using ``file://``,
1526you should place patch files in a directory next to the recipe either
1527named the same as the base name of the recipe
1528(:term:`BP` and
1529:term:`BPN`) or "files".
1530
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001531Licensing
1532---------
1533
1534Your recipe needs to have both the
1535:term:`LICENSE` and
1536:term:`LIC_FILES_CHKSUM`
1537variables:
1538
Andrew Geissler09036742021-06-25 14:25:14 -05001539- :term:`LICENSE`: This variable specifies the license for the software.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001540 If you do not know the license under which the software you are
1541 building is distributed, you should go to the source code and look
1542 for that information. Typical files containing this information
Andrew Geissler09036742021-06-25 14:25:14 -05001543 include ``COPYING``, :term:`LICENSE`, and ``README`` files. You could
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001544 also find the information near the top of a source file. For example,
1545 given a piece of software licensed under the GNU General Public
Andrew Geissler09036742021-06-25 14:25:14 -05001546 License version 2, you would set :term:`LICENSE` as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001547
1548 LICENSE = "GPLv2"
1549
Andrew Geissler09036742021-06-25 14:25:14 -05001550 The licenses you specify within :term:`LICENSE` can have any name as long
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001551 as you do not use spaces, since spaces are used as separators between
1552 license names. For standard licenses, use the names of the files in
Andrew Geissler09036742021-06-25 14:25:14 -05001553 ``meta/files/common-licenses/`` or the :term:`SPDXLICENSEMAP` flag names
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001554 defined in ``meta/conf/licenses.conf``.
1555
Andrew Geissler09036742021-06-25 14:25:14 -05001556- :term:`LIC_FILES_CHKSUM`: The OpenEmbedded build system uses this
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001557 variable to make sure the license text has not changed. If it has,
1558 the build produces an error and it affords you the chance to figure
1559 it out and correct the problem.
1560
1561 You need to specify all applicable licensing files for the software.
1562 At the end of the configuration step, the build process will compare
1563 the checksums of the files to be sure the text has not changed. Any
1564 differences result in an error with the message containing the
1565 current checksum. For more explanation and examples of how to set the
Andrew Geissler09036742021-06-25 14:25:14 -05001566 :term:`LIC_FILES_CHKSUM` variable, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001567 ":ref:`dev-manual/common-tasks:tracking license changes`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001568
1569 To determine the correct checksum string, you can list the
Andrew Geissler09036742021-06-25 14:25:14 -05001570 appropriate files in the :term:`LIC_FILES_CHKSUM` variable with incorrect
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001571 md5 strings, attempt to build the software, and then note the
1572 resulting error messages that will report the correct md5 strings.
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001573 See the ":ref:`dev-manual/common-tasks:fetching code`" section for
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001574 additional information.
1575
Andrew Geisslerc926e172021-05-07 16:11:35 -05001576 Here is an example that assumes the software has a ``COPYING`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001577
1578 LIC_FILES_CHKSUM = "file://COPYING;md5=xxx"
1579
1580 When you try to build the
1581 software, the build system will produce an error and give you the
1582 correct string that you can substitute into the recipe file for a
1583 subsequent build.
1584
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001585Dependencies
1586------------
1587
1588Most software packages have a short list of other packages that they
1589require, which are called dependencies. These dependencies fall into two
1590main categories: build-time dependencies, which are required when the
1591software is built; and runtime dependencies, which are required to be
1592installed on the target in order for the software to run.
1593
1594Within a recipe, you specify build-time dependencies using the
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001595:term:`DEPENDS` variable. Although there are nuances,
Andrew Geissler09036742021-06-25 14:25:14 -05001596items specified in :term:`DEPENDS` should be names of other
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001597recipes. It is important that you specify all build-time dependencies
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001598explicitly.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001599
1600Another consideration is that configure scripts might automatically
1601check for optional dependencies and enable corresponding functionality
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001602if those dependencies are found. If you wish to make a recipe that is
1603more generally useful (e.g. publish the recipe in a layer for others to
1604use), instead of hard-disabling the functionality, you can use the
1605:term:`PACKAGECONFIG` variable to allow functionality and the
1606corresponding dependencies to be enabled and disabled easily by other
1607users of the recipe.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001608
1609Similar to build-time dependencies, you specify runtime dependencies
1610through a variable -
1611:term:`RDEPENDS`, which is
1612package-specific. All variables that are package-specific need to have
1613the name of the package added to the end as an override. Since the main
1614package for a recipe has the same name as the recipe, and the recipe's
1615name can be found through the
1616``${``\ :term:`PN`\ ``}`` variable, then
1617you specify the dependencies for the main package by setting
Patrick Williams0ca19cc2021-08-16 14:03:13 -05001618``RDEPENDS:${PN}``. If the package were named ``${PN}-tools``, then you
1619would set ``RDEPENDS:${PN}-tools``, and so forth.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001620
1621Some runtime dependencies will be set automatically at packaging time.
1622These dependencies include any shared library dependencies (i.e. if a
1623package "example" contains "libexample" and another package "mypackage"
1624contains a binary that links to "libexample" then the OpenEmbedded build
1625system will automatically add a runtime dependency to "mypackage" on
1626"example"). See the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001627":ref:`overview-manual/concepts:automatically added runtime dependencies`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001628section in the Yocto Project Overview and Concepts Manual for further
1629details.
1630
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001631Configuring the Recipe
1632----------------------
1633
1634Most software provides some means of setting build-time configuration
1635options before compilation. Typically, setting these options is
1636accomplished by running a configure script with options, or by modifying
1637a build configuration file.
1638
1639.. note::
1640
1641 As of Yocto Project Release 1.7, some of the core recipes that
1642 package binary configuration scripts now disable the scripts due to
1643 the scripts previously requiring error-prone path substitution. The
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001644 OpenEmbedded build system uses ``pkg-config`` now, which is much more
1645 robust. You can find a list of the ``*-config`` scripts that are disabled
1646 in the ":ref:`migration-1.7-binary-configuration-scripts-disabled`" section
1647 in the Yocto Project Reference Manual.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001648
1649A major part of build-time configuration is about checking for
1650build-time dependencies and possibly enabling optional functionality as
1651a result. You need to specify any build-time dependencies for the
1652software you are building in your recipe's
1653:term:`DEPENDS` value, in terms of
1654other recipes that satisfy those dependencies. You can often find
1655build-time or runtime dependencies described in the software's
1656documentation.
1657
1658The following list provides configuration items of note based on how
1659your software is built:
1660
1661- *Autotools:* If your source files have a ``configure.ac`` file, then
1662 your software is built using Autotools. If this is the case, you just
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001663 need to modify the configuration.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001664
1665 When using Autotools, your recipe needs to inherit the
1666 :ref:`autotools <ref-classes-autotools>` class
1667 and your recipe does not have to contain a
1668 :ref:`ref-tasks-configure` task.
1669 However, you might still want to make some adjustments. For example,
1670 you can set
1671 :term:`EXTRA_OECONF` or
1672 :term:`PACKAGECONFIG_CONFARGS`
1673 to pass any needed configure options that are specific to the recipe.
1674
1675- *CMake:* If your source files have a ``CMakeLists.txt`` file, then
1676 your software is built using CMake. If this is the case, you just
William A. Kennington IIIac69b482021-06-02 12:28:27 -07001677 need to modify the configuration.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001678
1679 When you use CMake, your recipe needs to inherit the
1680 :ref:`cmake <ref-classes-cmake>` class and your
1681 recipe does not have to contain a
1682 :ref:`ref-tasks-configure` task.
1683 You can make some adjustments by setting
1684 :term:`EXTRA_OECMAKE` to
1685 pass any needed configure options that are specific to the recipe.
1686
1687 .. note::
1688
1689 If you need to install one or more custom CMake toolchain files
1690 that are supplied by the application you are building, install the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001691 files to ``${D}${datadir}/cmake/Modules`` during ``do_install``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001692
1693- *Other:* If your source files do not have a ``configure.ac`` or
1694 ``CMakeLists.txt`` file, then your software is built using some
1695 method other than Autotools or CMake. If this is the case, you
1696 normally need to provide a
1697 :ref:`ref-tasks-configure` task
1698 in your recipe unless, of course, there is nothing to configure.
1699
1700 Even if your software is not being built by Autotools or CMake, you
1701 still might not need to deal with any configuration issues. You need
1702 to determine if configuration is even a required step. You might need
1703 to modify a Makefile or some configuration file used for the build to
1704 specify necessary build options. Or, perhaps you might need to run a
1705 provided, custom configure script with the appropriate options.
1706
1707 For the case involving a custom configure script, you would run
1708 ``./configure --help`` and look for the options you need to set.
1709
1710Once configuration succeeds, it is always good practice to look at the
1711``log.do_configure`` file to ensure that the appropriate options have
1712been enabled and no additional build-time dependencies need to be added
Andrew Geissler09036742021-06-25 14:25:14 -05001713to :term:`DEPENDS`. For example, if the configure script reports that it
1714found something not mentioned in :term:`DEPENDS`, or that it did not find
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001715something that it needed for some desired optional functionality, then
Andrew Geissler09036742021-06-25 14:25:14 -05001716you would need to add those to :term:`DEPENDS`. Looking at the log might
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001717also reveal items being checked for, enabled, or both that you do not
Andrew Geissler09036742021-06-25 14:25:14 -05001718want, or items not being found that are in :term:`DEPENDS`, in which case
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001719you would need to look at passing extra options to the configure script
1720as needed. For reference information on configure options specific to
1721the software you are building, you can consult the output of the
1722``./configure --help`` command within ``${S}`` or consult the software's
1723upstream documentation.
1724
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001725Using Headers to Interface with Devices
1726---------------------------------------
1727
1728If your recipe builds an application that needs to communicate with some
1729device or needs an API into a custom kernel, you will need to provide
1730appropriate header files. Under no circumstances should you ever modify
1731the existing
1732``meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc`` file.
1733These headers are used to build ``libc`` and must not be compromised
1734with custom or machine-specific header information. If you customize
1735``libc`` through modified headers all other applications that use
1736``libc`` thus become affected.
1737
1738.. note::
1739
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001740 Never copy and customize the ``libc`` header file (i.e.
1741 ``meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc``).
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001742
1743The correct way to interface to a device or custom kernel is to use a
1744separate package that provides the additional headers for the driver or
1745other unique interfaces. When doing so, your application also becomes
1746responsible for creating a dependency on that specific provider.
1747
1748Consider the following:
1749
1750- Never modify ``linux-libc-headers.inc``. Consider that file to be
1751 part of the ``libc`` system, and not something you use to access the
1752 kernel directly. You should access ``libc`` through specific ``libc``
1753 calls.
1754
1755- Applications that must talk directly to devices should either provide
1756 necessary headers themselves, or establish a dependency on a special
1757 headers package that is specific to that driver.
1758
1759For example, suppose you want to modify an existing header that adds I/O
1760control or network support. If the modifications are used by a small
1761number programs, providing a unique version of a header is easy and has
1762little impact. When doing so, bear in mind the guidelines in the
1763previous list.
1764
1765.. note::
1766
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001767 If for some reason your changes need to modify the behavior of the ``libc``,
1768 and subsequently all other applications on the system, use a ``.bbappend``
1769 to modify the ``linux-kernel-headers.inc`` file. However, take care to not
1770 make the changes machine specific.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001771
1772Consider a case where your kernel is older and you need an older
1773``libc`` ABI. The headers installed by your recipe should still be a
1774standard mainline kernel, not your own custom one.
1775
1776When you use custom kernel headers you need to get them from
1777:term:`STAGING_KERNEL_DIR`,
1778which is the directory with kernel headers that are required to build
Andrew Geisslerc926e172021-05-07 16:11:35 -05001779out-of-tree modules. Your recipe will also need the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001780
1781 do_configure[depends] += "virtual/kernel:do_shared_workdir"
1782
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001783Compilation
1784-----------
1785
1786During a build, the ``do_compile`` task happens after source is fetched,
1787unpacked, and configured. If the recipe passes through ``do_compile``
1788successfully, nothing needs to be done.
1789
1790However, if the compile step fails, you need to diagnose the failure.
1791Here are some common issues that cause failures.
1792
1793.. note::
1794
1795 For cases where improper paths are detected for configuration files
1796 or for when libraries/headers cannot be found, be sure you are using
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001797 the more robust ``pkg-config``. See the note in section
Andrew Geissler09209ee2020-12-13 08:44:15 -06001798 ":ref:`dev-manual/common-tasks:Configuring the Recipe`" for additional information.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001799
1800- *Parallel build failures:* These failures manifest themselves as
1801 intermittent errors, or errors reporting that a file or directory
1802 that should be created by some other part of the build process could
1803 not be found. This type of failure can occur even if, upon
1804 inspection, the file or directory does exist after the build has
1805 failed, because that part of the build process happened in the wrong
1806 order.
1807
1808 To fix the problem, you need to either satisfy the missing dependency
1809 in the Makefile or whatever script produced the Makefile, or (as a
Andrew Geisslerc926e172021-05-07 16:11:35 -05001810 workaround) set :term:`PARALLEL_MAKE` to an empty string::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001811
1812 PARALLEL_MAKE = ""
1813
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001814 For information on parallel Makefile issues, see the
1815 ":ref:`dev-manual/common-tasks:debugging parallel make races`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001816
1817- *Improper host path usage:* This failure applies to recipes building
1818 for the target or ``nativesdk`` only. The failure occurs when the
1819 compilation process uses improper headers, libraries, or other files
1820 from the host system when cross-compiling for the target.
1821
1822 To fix the problem, examine the ``log.do_compile`` file to identify
1823 the host paths being used (e.g. ``/usr/include``, ``/usr/lib``, and
1824 so forth) and then either add configure options, apply a patch, or do
1825 both.
1826
1827- *Failure to find required libraries/headers:* If a build-time
1828 dependency is missing because it has not been declared in
1829 :term:`DEPENDS`, or because the
1830 dependency exists but the path used by the build process to find the
1831 file is incorrect and the configure step did not detect it, the
1832 compilation process could fail. For either of these failures, the
1833 compilation process notes that files could not be found. In these
1834 cases, you need to go back and add additional options to the
1835 configure script as well as possibly add additional build-time
Andrew Geissler09036742021-06-25 14:25:14 -05001836 dependencies to :term:`DEPENDS`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001837
1838 Occasionally, it is necessary to apply a patch to the source to
1839 ensure the correct paths are used. If you need to specify paths to
1840 find files staged into the sysroot from other recipes, use the
1841 variables that the OpenEmbedded build system provides (e.g.
Andrew Geissler09036742021-06-25 14:25:14 -05001842 :term:`STAGING_BINDIR`, :term:`STAGING_INCDIR`, :term:`STAGING_DATADIR`, and so
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001843 forth).
1844
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001845Installing
1846----------
1847
1848During ``do_install``, the task copies the built files along with their
1849hierarchy to locations that would mirror their locations on the target
1850device. The installation process copies files from the
1851``${``\ :term:`S`\ ``}``,
1852``${``\ :term:`B`\ ``}``, and
1853``${``\ :term:`WORKDIR`\ ``}``
1854directories to the ``${``\ :term:`D`\ ``}``
1855directory to create the structure as it should appear on the target
1856system.
1857
1858How your software is built affects what you must do to be sure your
1859software is installed correctly. The following list describes what you
1860must do for installation depending on the type of build system used by
1861the software being built:
1862
1863- *Autotools and CMake:* If the software your recipe is building uses
1864 Autotools or CMake, the OpenEmbedded build system understands how to
1865 install the software. Consequently, you do not have to have a
1866 ``do_install`` task as part of your recipe. You just need to make
1867 sure the install portion of the build completes with no issues.
1868 However, if you wish to install additional files not already being
1869 installed by ``make install``, you should do this using a
Patrick Williams0ca19cc2021-08-16 14:03:13 -05001870 ``do_install:append`` function using the install command as described
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001871 in the "Manual" bulleted item later in this list.
1872
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001873- *Other (using* ``make install``\ *)*: You need to define a ``do_install``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001874 function in your recipe. The function should call
1875 ``oe_runmake install`` and will likely need to pass in the
1876 destination directory as well. How you pass that path is dependent on
1877 how the ``Makefile`` being run is written (e.g. ``DESTDIR=${D}``,
1878 ``PREFIX=${D}``, ``INSTALLROOT=${D}``, and so forth).
1879
1880 For an example recipe using ``make install``, see the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05001881 ":ref:`dev-manual/common-tasks:makefile-based package`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001882
1883- *Manual:* You need to define a ``do_install`` function in your
1884 recipe. The function must first use ``install -d`` to create the
1885 directories under
1886 ``${``\ :term:`D`\ ``}``. Once the
1887 directories exist, your function can use ``install`` to manually
1888 install the built software into the directories.
1889
1890 You can find more information on ``install`` at
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001891 https://www.gnu.org/software/coreutils/manual/html_node/install-invocation.html.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001892
1893For the scenarios that do not use Autotools or CMake, you need to track
1894the installation and diagnose and fix any issues until everything
1895installs correctly. You need to look in the default location of
1896``${D}``, which is ``${WORKDIR}/image``, to be sure your files have been
1897installed correctly.
1898
1899.. note::
1900
1901 - During the installation process, you might need to modify some of
1902 the installed files to suit the target layout. For example, you
1903 might need to replace hard-coded paths in an initscript with
1904 values of variables provided by the build system, such as
1905 replacing ``/usr/bin/`` with ``${bindir}``. If you do perform such
1906 modifications during ``do_install``, be sure to modify the
1907 destination file after copying rather than before copying.
1908 Modifying after copying ensures that the build system can
1909 re-execute ``do_install`` if needed.
1910
1911 - ``oe_runmake install``, which can be run directly or can be run
1912 indirectly by the
1913 :ref:`autotools <ref-classes-autotools>` and
1914 :ref:`cmake <ref-classes-cmake>` classes,
1915 runs ``make install`` in parallel. Sometimes, a Makefile can have
1916 missing dependencies between targets that can result in race
1917 conditions. If you experience intermittent failures during
1918 ``do_install``, you might be able to work around them by disabling
Andrew Geisslerc926e172021-05-07 16:11:35 -05001919 parallel Makefile installs by adding the following to the recipe::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001920
1921 PARALLEL_MAKEINST = ""
1922
1923 See :term:`PARALLEL_MAKEINST` for additional information.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001924
1925 - If you need to install one or more custom CMake toolchain files
1926 that are supplied by the application you are building, install the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001927 files to ``${D}${datadir}/cmake/Modules`` during
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001928 :ref:`ref-tasks-install`.
1929
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001930Enabling System Services
1931------------------------
1932
1933If you want to install a service, which is a process that usually starts
1934on boot and runs in the background, then you must include some
1935additional definitions in your recipe.
1936
1937If you are adding services and the service initialization script or the
1938service file itself is not installed, you must provide for that
Patrick Williams0ca19cc2021-08-16 14:03:13 -05001939installation in your recipe using a ``do_install:append`` function. If
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001940your recipe already has a ``do_install`` function, update the function
Patrick Williams0ca19cc2021-08-16 14:03:13 -05001941near its end rather than adding an additional ``do_install:append``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001942function.
1943
1944When you create the installation for your services, you need to
1945accomplish what is normally done by ``make install``. In other words,
1946make sure your installation arranges the output similar to how it is
1947arranged on the target system.
1948
1949The OpenEmbedded build system provides support for starting services two
1950different ways:
1951
1952- *SysVinit:* SysVinit is a system and service manager that manages the
1953 init system used to control the very basic functions of your system.
1954 The init program is the first program started by the Linux kernel
1955 when the system boots. Init then controls the startup, running and
1956 shutdown of all other programs.
1957
1958 To enable a service using SysVinit, your recipe needs to inherit the
1959 :ref:`update-rc.d <ref-classes-update-rc.d>`
1960 class. The class helps facilitate safely installing the package on
1961 the target.
1962
1963 You will need to set the
1964 :term:`INITSCRIPT_PACKAGES`,
1965 :term:`INITSCRIPT_NAME`,
1966 and
1967 :term:`INITSCRIPT_PARAMS`
1968 variables within your recipe.
1969
1970- *systemd:* System Management Daemon (systemd) was designed to replace
1971 SysVinit and to provide enhanced management of services. For more
1972 information on systemd, see the systemd homepage at
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001973 https://freedesktop.org/wiki/Software/systemd/.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001974
1975 To enable a service using systemd, your recipe needs to inherit the
1976 :ref:`systemd <ref-classes-systemd>` class. See
1977 the ``systemd.bbclass`` file located in your :term:`Source Directory`
1978 section for
1979 more information.
1980
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001981Packaging
1982---------
1983
1984Successful packaging is a combination of automated processes performed
1985by the OpenEmbedded build system and some specific steps you need to
1986take. The following list describes the process:
1987
1988- *Splitting Files*: The ``do_package`` task splits the files produced
1989 by the recipe into logical components. Even software that produces a
1990 single binary might still have debug symbols, documentation, and
1991 other logical components that should be split out. The ``do_package``
1992 task ensures that files are split up and packaged correctly.
1993
1994- *Running QA Checks*: The
1995 :ref:`insane <ref-classes-insane>` class adds a
1996 step to the package generation process so that output quality
1997 assurance checks are generated by the OpenEmbedded build system. This
1998 step performs a range of checks to be sure the build's output is free
1999 of common problems that show up during runtime. For information on
2000 these checks, see the
2001 :ref:`insane <ref-classes-insane>` class and
Andrew Geissler09209ee2020-12-13 08:44:15 -06002002 the ":ref:`ref-manual/qa-checks:qa error and warning messages`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002003 chapter in the Yocto Project Reference Manual.
2004
2005- *Hand-Checking Your Packages*: After you build your software, you
2006 need to be sure your packages are correct. Examine the
2007 ``${``\ :term:`WORKDIR`\ ``}/packages-split``
2008 directory and make sure files are where you expect them to be. If you
2009 discover problems, you can set
2010 :term:`PACKAGES`,
2011 :term:`FILES`,
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002012 ``do_install(:append)``, and so forth as needed.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002013
2014- *Splitting an Application into Multiple Packages*: If you need to
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05002015 split an application into several packages, see the
2016 ":ref:`dev-manual/common-tasks:splitting an application into multiple packages`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002017 section for an example.
2018
2019- *Installing a Post-Installation Script*: For an example showing how
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05002020 to install a post-installation script, see the
2021 ":ref:`dev-manual/common-tasks:post-installation scripts`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002022
2023- *Marking Package Architecture*: Depending on what your recipe is
2024 building and how it is configured, it might be important to mark the
2025 packages produced as being specific to a particular machine, or to
2026 mark them as not being specific to a particular machine or
2027 architecture at all.
2028
2029 By default, packages apply to any machine with the same architecture
2030 as the target machine. When a recipe produces packages that are
2031 machine-specific (e.g. the
2032 :term:`MACHINE` value is passed
2033 into the configure script or a patch is applied only for a particular
2034 machine), you should mark them as such by adding the following to the
Andrew Geisslerc926e172021-05-07 16:11:35 -05002035 recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002036
2037 PACKAGE_ARCH = "${MACHINE_ARCH}"
2038
2039 On the other hand, if the recipe produces packages that do not
2040 contain anything specific to the target machine or architecture at
2041 all (e.g. recipes that simply package script files or configuration
2042 files), you should use the
2043 :ref:`allarch <ref-classes-allarch>` class to
Andrew Geisslerc926e172021-05-07 16:11:35 -05002044 do this for you by adding this to your recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002045
2046 inherit allarch
2047
2048 Ensuring that the package architecture is correct is not critical
2049 while you are doing the first few builds of your recipe. However, it
2050 is important in order to ensure that your recipe rebuilds (or does
2051 not rebuild) appropriately in response to changes in configuration,
2052 and to ensure that you get the appropriate packages installed on the
2053 target machine, particularly if you run separate builds for more than
2054 one target machine.
2055
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002056Sharing Files Between Recipes
2057-----------------------------
2058
2059Recipes often need to use files provided by other recipes on the build
2060host. For example, an application linking to a common library needs
2061access to the library itself and its associated headers. The way this
2062access is accomplished is by populating a sysroot with files. Each
2063recipe has two sysroots in its work directory, one for target files
2064(``recipe-sysroot``) and one for files that are native to the build host
2065(``recipe-sysroot-native``).
2066
2067.. note::
2068
2069 You could find the term "staging" used within the Yocto project
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002070 regarding files populating sysroots (e.g. the :term:`STAGING_DIR`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002071 variable).
2072
2073Recipes should never populate the sysroot directly (i.e. write files
2074into sysroot). Instead, files should be installed into standard
2075locations during the
2076:ref:`ref-tasks-install` task within
2077the ``${``\ :term:`D`\ ``}`` directory. The
2078reason for this limitation is that almost all files that populate the
2079sysroot are cataloged in manifests in order to ensure the files can be
2080removed later when a recipe is either modified or removed. Thus, the
2081sysroot is able to remain free from stale files.
2082
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05002083A subset of the files installed by the :ref:`ref-tasks-install` task are
2084used by the :ref:`ref-tasks-populate_sysroot` task as defined by the the
2085:term:`SYSROOT_DIRS` variable to automatically populate the sysroot. It
2086is possible to modify the list of directories that populate the sysroot.
2087The following example shows how you could add the ``/opt`` directory to
Andrew Geisslerc926e172021-05-07 16:11:35 -05002088the list of directories within a recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002089
2090 SYSROOT_DIRS += "/opt"
2091
Andrew Geisslerd1e89492021-02-12 15:35:20 -06002092.. note::
2093
2094 The `/sysroot-only` is to be used by recipes that generate artifacts
2095 that are not included in the target filesystem, allowing them to share
Andrew Geissler09036742021-06-25 14:25:14 -05002096 these artifacts without needing to use the :term:`DEPLOY_DIR`.
Andrew Geisslerd1e89492021-02-12 15:35:20 -06002097
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05002098For a more complete description of the :ref:`ref-tasks-populate_sysroot`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002099task and its associated functions, see the
2100:ref:`staging <ref-classes-staging>` class.
2101
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002102Using Virtual Providers
2103-----------------------
2104
2105Prior to a build, if you know that several different recipes provide the
2106same functionality, you can use a virtual provider (i.e. ``virtual/*``)
2107as a placeholder for the actual provider. The actual provider is
2108determined at build-time.
2109
2110A common scenario where a virtual provider is used would be for the
2111kernel recipe. Suppose you have three kernel recipes whose
2112:term:`PN` values map to ``kernel-big``,
2113``kernel-mid``, and ``kernel-small``. Furthermore, each of these recipes
2114in some way uses a :term:`PROVIDES`
2115statement that essentially identifies itself as being able to provide
2116``virtual/kernel``. Here is one way through the
Andrew Geisslerc926e172021-05-07 16:11:35 -05002117:ref:`kernel <ref-classes-kernel>` class::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002118
2119 PROVIDES += "${@ "virtual/kernel" if (d.getVar("KERNEL_PACKAGE_NAME") == "kernel") else "" }"
2120
2121Any recipe that inherits the ``kernel`` class is
Andrew Geissler09036742021-06-25 14:25:14 -05002122going to utilize a :term:`PROVIDES` statement that identifies that recipe as
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002123being able to provide the ``virtual/kernel`` item.
2124
2125Now comes the time to actually build an image and you need a kernel
2126recipe, but which one? You can configure your build to call out the
Andrew Geissler09209ee2020-12-13 08:44:15 -06002127kernel recipe you want by using the :term:`PREFERRED_PROVIDER` variable. As
2128an example, consider the :yocto_git:`x86-base.inc
Andrew Geisslerd159c7f2021-09-02 21:05:58 -05002129</poky/tree/meta/conf/machine/include/x86/x86-base.inc>` include file, which is a
Andrew Geissler09209ee2020-12-13 08:44:15 -06002130machine (i.e. :term:`MACHINE`) configuration file. This include file is the
2131reason all x86-based machines use the ``linux-yocto`` kernel. Here are the
Andrew Geisslerc926e172021-05-07 16:11:35 -05002132relevant lines from the include file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002133
2134 PREFERRED_PROVIDER_virtual/kernel ??= "linux-yocto"
2135 PREFERRED_VERSION_linux-yocto ??= "4.15%"
2136
2137When you use a virtual provider, you do not have to "hard code" a recipe
2138name as a build dependency. You can use the
2139:term:`DEPENDS` variable to state the
Andrew Geisslerc926e172021-05-07 16:11:35 -05002140build is dependent on ``virtual/kernel`` for example::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002141
2142 DEPENDS = "virtual/kernel"
2143
2144During the build, the OpenEmbedded build system picks
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002145the correct recipe needed for the ``virtual/kernel`` dependency based on
Andrew Geissler09036742021-06-25 14:25:14 -05002146the :term:`PREFERRED_PROVIDER` variable. If you want to use the small kernel
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002147mentioned at the beginning of this section, configure your build as
Andrew Geisslerc926e172021-05-07 16:11:35 -05002148follows::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002149
2150 PREFERRED_PROVIDER_virtual/kernel ??= "kernel-small"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002151
2152.. note::
2153
Andrew Geissler09036742021-06-25 14:25:14 -05002154 Any recipe that :term:`PROVIDES` a ``virtual/*`` item that is ultimately not
2155 selected through :term:`PREFERRED_PROVIDER` does not get built. Preventing these
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002156 recipes from building is usually the desired behavior since this mechanism's
2157 purpose is to select between mutually exclusive alternative providers.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002158
2159The following lists specific examples of virtual providers:
2160
2161- ``virtual/kernel``: Provides the name of the kernel recipe to use
2162 when building a kernel image.
2163
2164- ``virtual/bootloader``: Provides the name of the bootloader to use
2165 when building an image.
2166
2167- ``virtual/libgbm``: Provides ``gbm.pc``.
2168
2169- ``virtual/egl``: Provides ``egl.pc`` and possibly ``wayland-egl.pc``.
2170
2171- ``virtual/libgl``: Provides ``gl.pc`` (i.e. libGL).
2172
2173- ``virtual/libgles1``: Provides ``glesv1_cm.pc`` (i.e. libGLESv1_CM).
2174
2175- ``virtual/libgles2``: Provides ``glesv2.pc`` (i.e. libGLESv2).
2176
2177.. note::
2178
2179 Virtual providers only apply to build time dependencies specified with
2180 :term:`PROVIDES` and :term:`DEPENDS`. They do not apply to runtime
2181 dependencies specified with :term:`RPROVIDES` and :term:`RDEPENDS`.
2182
2183Properly Versioning Pre-Release Recipes
2184---------------------------------------
2185
2186Sometimes the name of a recipe can lead to versioning problems when the
2187recipe is upgraded to a final release. For example, consider the
2188``irssi_0.8.16-rc1.bb`` recipe file in the list of example recipes in
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05002189the ":ref:`dev-manual/common-tasks:storing and naming the recipe`" section.
2190This recipe is at a release candidate stage (i.e. "rc1"). When the recipe is
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002191released, the recipe filename becomes ``irssi_0.8.16.bb``. The version
2192change from ``0.8.16-rc1`` to ``0.8.16`` is seen as a decrease by the
2193build system and package managers, so the resulting packages will not
2194correctly trigger an upgrade.
2195
2196In order to ensure the versions compare properly, the recommended
2197convention is to set :term:`PV` within the
2198recipe to "previous_version+current_version". You can use an additional
2199variable so that you can use the current version elsewhere. Here is an
Andrew Geisslerc926e172021-05-07 16:11:35 -05002200example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002201
2202 REALPV = "0.8.16-rc1"
2203 PV = "0.8.15+${REALPV}"
2204
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002205Post-Installation Scripts
2206-------------------------
2207
2208Post-installation scripts run immediately after installing a package on
2209the target or during image creation when a package is included in an
2210image. To add a post-installation script to a package, add a
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002211``pkg_postinst:``\ `PACKAGENAME`\ ``()`` function to the recipe file
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002212(``.bb``) and replace `PACKAGENAME` with the name of the package you want
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002213to attach to the ``postinst`` script. To apply the post-installation
2214script to the main package for the recipe, which is usually what is
2215required, specify
2216``${``\ :term:`PN`\ ``}`` in place of
2217PACKAGENAME.
2218
Andrew Geisslerc926e172021-05-07 16:11:35 -05002219A post-installation function has the following structure::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002220
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002221 pkg_postinst:PACKAGENAME() {
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002222 # Commands to carry out
2223 }
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002224
2225The script defined in the post-installation function is called when the
2226root filesystem is created. If the script succeeds, the package is
2227marked as installed.
2228
2229.. note::
2230
2231 Any RPM post-installation script that runs on the target should
2232 return a 0 exit code. RPM does not allow non-zero exit codes for
2233 these scripts, and the RPM package manager will cause the package to
2234 fail installation on the target.
2235
2236Sometimes it is necessary for the execution of a post-installation
2237script to be delayed until the first boot. For example, the script might
2238need to be executed on the device itself. To delay script execution
2239until boot time, you must explicitly mark post installs to defer to the
2240target. You can use ``pkg_postinst_ontarget()`` or call
2241``postinst_intercept delay_to_first_boot`` from ``pkg_postinst()``. Any
2242failure of a ``pkg_postinst()`` script (including exit 1) triggers an
2243error during the
2244:ref:`ref-tasks-rootfs` task.
2245
2246If you have recipes that use ``pkg_postinst`` function and they require
2247the use of non-standard native tools that have dependencies during
2248rootfs construction, you need to use the
2249:term:`PACKAGE_WRITE_DEPS`
2250variable in your recipe to list these tools. If you do not use this
2251variable, the tools might be missing and execution of the
2252post-installation script is deferred until first boot. Deferring the
2253script to first boot is undesirable and for read-only rootfs impossible.
2254
2255.. note::
2256
William A. Kennington IIIac69b482021-06-02 12:28:27 -07002257 There is equivalent support for pre-install, pre-uninstall, and post-uninstall
2258 scripts by way of ``pkg_preinst``, ``pkg_prerm``, and ``pkg_postrm``,
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002259 respectively. These scrips work in exactly the same way as does
2260 ``pkg_postinst`` with the exception that they run at different times. Also,
2261 because of when they run, they are not applicable to being run at image
2262 creation time like ``pkg_postinst``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002263
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002264Testing
2265-------
2266
2267The final step for completing your recipe is to be sure that the
2268software you built runs correctly. To accomplish runtime testing, add
2269the build's output packages to your image and test them on the target.
2270
2271For information on how to customize your image by adding specific
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05002272packages, see ":ref:`dev-manual/common-tasks:customizing images`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002273
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002274Examples
2275--------
2276
2277To help summarize how to write a recipe, this section provides some
2278examples given various scenarios:
2279
2280- Recipes that use local files
2281
2282- Using an Autotooled package
2283
2284- Using a Makefile-based package
2285
2286- Splitting an application into multiple packages
2287
2288- Adding binaries to an image
2289
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002290Single .c File Package (Hello World!)
2291~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2292
2293Building an application from a single file that is stored locally (e.g.
2294under ``files``) requires a recipe that has the file listed in the
Andrew Geissler09036742021-06-25 14:25:14 -05002295:term:`SRC_URI` variable. Additionally, you need to manually write the
2296``do_compile`` and ``do_install`` tasks. The :term:`S` variable defines the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002297directory containing the source code, which is set to
2298:term:`WORKDIR` in this case - the
2299directory BitBake uses for the build.
2300::
2301
2302 SUMMARY = "Simple helloworld application"
2303 SECTION = "examples"
2304 LICENSE = "MIT"
2305 LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
2306
2307 SRC_URI = "file://helloworld.c"
2308
2309 S = "${WORKDIR}"
2310
2311 do_compile() {
Andrew Geisslerd1e89492021-02-12 15:35:20 -06002312 ${CC} ${LDFLAGS} helloworld.c -o helloworld
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002313 }
2314
2315 do_install() {
2316 install -d ${D}${bindir}
2317 install -m 0755 helloworld ${D}${bindir}
2318 }
2319
2320By default, the ``helloworld``, ``helloworld-dbg``, and
2321``helloworld-dev`` packages are built. For information on how to
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05002322customize the packaging process, see the
2323":ref:`dev-manual/common-tasks:splitting an application into multiple packages`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002324section.
2325
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002326Autotooled Package
2327~~~~~~~~~~~~~~~~~~
2328
2329Applications that use Autotools such as ``autoconf`` and ``automake``
Andrew Geissler09036742021-06-25 14:25:14 -05002330require a recipe that has a source archive listed in :term:`SRC_URI` and
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002331also inherit the
2332:ref:`autotools <ref-classes-autotools>` class,
2333which contains the definitions of all the steps needed to build an
2334Autotool-based application. The result of the build is automatically
2335packaged. And, if the application uses NLS for localization, packages
2336with local information are generated (one package per language).
2337Following is one example: (``hello_2.3.bb``)
2338::
2339
2340 SUMMARY = "GNU Helloworld application"
2341 SECTION = "examples"
2342 LICENSE = "GPLv2+"
2343 LIC_FILES_CHKSUM = "file://COPYING;md5=751419260aa954499f7abaabaa882bbe"
2344
2345 SRC_URI = "${GNU_MIRROR}/hello/hello-${PV}.tar.gz"
2346
2347 inherit autotools gettext
2348
Andrew Geissler09036742021-06-25 14:25:14 -05002349The variable :term:`LIC_FILES_CHKSUM` is used to track source license
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002350changes as described in the
Andrew Geissler09209ee2020-12-13 08:44:15 -06002351":ref:`dev-manual/common-tasks:tracking license changes`" section in
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002352the Yocto Project Overview and Concepts Manual. You can quickly create
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002353Autotool-based recipes in a manner similar to the previous example.
2354
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002355Makefile-Based Package
2356~~~~~~~~~~~~~~~~~~~~~~
2357
2358Applications that use GNU ``make`` also require a recipe that has the
Andrew Geissler09036742021-06-25 14:25:14 -05002359source archive listed in :term:`SRC_URI`. You do not need to add a
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002360``do_compile`` step since by default BitBake starts the ``make`` command
2361to compile the application. If you need additional ``make`` options, you
2362should store them in the
2363:term:`EXTRA_OEMAKE` or
2364:term:`PACKAGECONFIG_CONFARGS`
2365variables. BitBake passes these options into the GNU ``make``
2366invocation. Note that a ``do_install`` task is still required.
2367Otherwise, BitBake runs an empty ``do_install`` task by default.
2368
2369Some applications might require extra parameters to be passed to the
2370compiler. For example, the application might need an additional header
Andrew Geissler09036742021-06-25 14:25:14 -05002371path. You can accomplish this by adding to the :term:`CFLAGS` variable. The
Andrew Geisslerc926e172021-05-07 16:11:35 -05002372following example shows this::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002373
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002374 CFLAGS:prepend = "-I ${S}/include "
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002375
Andrew Geisslerc926e172021-05-07 16:11:35 -05002376In the following example, ``mtd-utils`` is a makefile-based package::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002377
2378 SUMMARY = "Tools for managing memory technology devices"
2379 SECTION = "base"
2380 DEPENDS = "zlib lzo e2fsprogs util-linux"
2381 HOMEPAGE = "http://www.linux-mtd.infradead.org/"
2382 LICENSE = "GPLv2+"
2383 LIC_FILES_CHKSUM = "file://COPYING;md5=0636e73ff0215e8d672dc4c32c317bb3 \
2384 file://include/common.h;beginline=1;endline=17;md5=ba05b07912a44ea2bf81ce409380049c"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002385
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002386 # Use the latest version at 26 Oct, 2013
2387 SRCREV = "9f107132a6a073cce37434ca9cda6917dd8d866b"
2388 SRC_URI = "git://git.infradead.org/mtd-utils.git \
2389 file://add-exclusion-to-mkfs-jffs2-git-2.patch \
2390 "
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002391
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002392 PV = "1.5.1+git${SRCPV}"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002393
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002394 S = "${WORKDIR}/git"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002395
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002396 EXTRA_OEMAKE = "'CC=${CC}' 'RANLIB=${RANLIB}' 'AR=${AR}' 'CFLAGS=${CFLAGS} -I${S}/include -DWITHOUT_XATTR' 'BUILDDIR=${S}'"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002397
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002398 do_install () {
2399 oe_runmake install DESTDIR=${D} SBINDIR=${sbindir} MANDIR=${mandir} INCLUDEDIR=${includedir}
2400 }
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002401
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002402 PACKAGES =+ "mtd-utils-jffs2 mtd-utils-ubifs mtd-utils-misc"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002403
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002404 FILES:mtd-utils-jffs2 = "${sbindir}/mkfs.jffs2 ${sbindir}/jffs2dump ${sbindir}/jffs2reader ${sbindir}/sumtool"
2405 FILES:mtd-utils-ubifs = "${sbindir}/mkfs.ubifs ${sbindir}/ubi*"
2406 FILES:mtd-utils-misc = "${sbindir}/nftl* ${sbindir}/ftl* ${sbindir}/rfd* ${sbindir}/doc* ${sbindir}/serve_image ${sbindir}/recv_image"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002407
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002408 PARALLEL_MAKE = ""
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002409
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002410 BBCLASSEXTEND = "native"
2411
2412Splitting an Application into Multiple Packages
2413~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2414
Andrew Geissler09036742021-06-25 14:25:14 -05002415You can use the variables :term:`PACKAGES` and :term:`FILES` to split an
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002416application into multiple packages.
2417
2418Following is an example that uses the ``libxpm`` recipe. By default,
2419this recipe generates a single package that contains the library along
2420with a few binaries. You can modify the recipe to split the binaries
Andrew Geisslerc926e172021-05-07 16:11:35 -05002421into separate packages::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002422
2423 require xorg-lib-common.inc
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002424
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002425 SUMMARY = "Xpm: X Pixmap extension library"
Andrew Geissler5199d832021-09-24 16:47:35 -05002426 LICENSE = "MIT"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002427 LIC_FILES_CHKSUM = "file://COPYING;md5=51f4270b012ecd4ab1a164f5f4ed6cf7"
2428 DEPENDS += "libxext libsm libxt"
2429 PE = "1"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002430
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002431 XORG_PN = "libXpm"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002432
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002433 PACKAGES =+ "sxpm cxpm"
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002434 FILES:cxpm = "${bindir}/cxpm"
2435 FILES:sxpm = "${bindir}/sxpm"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002436
2437In the previous example, we want to ship the ``sxpm`` and ``cxpm``
2438binaries in separate packages. Since ``bindir`` would be packaged into
Andrew Geissler09036742021-06-25 14:25:14 -05002439the main :term:`PN` package by default, we prepend the :term:`PACKAGES` variable
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002440so additional package names are added to the start of list. This results
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002441in the extra ``FILES:*`` variables then containing information that
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002442define which files and directories go into which packages. Files
2443included by earlier packages are skipped by latter packages. Thus, the
Andrew Geissler09036742021-06-25 14:25:14 -05002444main :term:`PN` package does not include the above listed files.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002445
2446Packaging Externally Produced Binaries
2447~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2448
2449Sometimes, you need to add pre-compiled binaries to an image. For
William A. Kennington IIIac69b482021-06-02 12:28:27 -07002450example, suppose that there are binaries for proprietary code,
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002451created by a particular division of a company. Your part of the company
2452needs to use those binaries as part of an image that you are building
2453using the OpenEmbedded build system. Since you only have the binaries
2454and not the source code, you cannot use a typical recipe that expects to
2455fetch the source specified in
2456:term:`SRC_URI` and then compile it.
2457
2458One method is to package the binaries and then install them as part of
2459the image. Generally, it is not a good idea to package binaries since,
2460among other things, it can hinder the ability to reproduce builds and
2461could lead to compatibility problems with ABI in the future. However,
2462sometimes you have no choice.
2463
2464The easiest solution is to create a recipe that uses the
2465:ref:`bin_package <ref-classes-bin-package>` class
2466and to be sure that you are using default locations for build artifacts.
2467In most cases, the ``bin_package`` class handles "skipping" the
2468configure and compile steps as well as sets things up to grab packages
2469from the appropriate area. In particular, this class sets ``noexec`` on
2470both the :ref:`ref-tasks-configure`
2471and :ref:`ref-tasks-compile` tasks,
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002472sets ``FILES:${PN}`` to "/" so that it picks up all files, and sets up a
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002473:ref:`ref-tasks-install` task, which
2474effectively copies all files from ``${S}`` to ``${D}``. The
2475``bin_package`` class works well when the files extracted into ``${S}``
2476are already laid out in the way they should be laid out on the target.
2477For more information on these variables, see the
2478:term:`FILES`,
2479:term:`PN`,
2480:term:`S`, and
2481:term:`D` variables in the Yocto Project
2482Reference Manual's variable glossary.
2483
2484.. note::
2485
2486 - Using :term:`DEPENDS` is a good
2487 idea even for components distributed in binary form, and is often
2488 necessary for shared libraries. For a shared library, listing the
Andrew Geissler09036742021-06-25 14:25:14 -05002489 library dependencies in :term:`DEPENDS` makes sure that the libraries
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002490 are available in the staging sysroot when other recipes link
2491 against the library, which might be necessary for successful
2492 linking.
2493
Andrew Geissler09036742021-06-25 14:25:14 -05002494 - Using :term:`DEPENDS` also allows runtime dependencies between
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002495 packages to be added automatically. See the
Andrew Geissler09209ee2020-12-13 08:44:15 -06002496 ":ref:`overview-manual/concepts:automatically added runtime dependencies`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002497 section in the Yocto Project Overview and Concepts Manual for more
2498 information.
2499
2500If you cannot use the ``bin_package`` class, you need to be sure you are
2501doing the following:
2502
2503- Create a recipe where the
2504 :ref:`ref-tasks-configure` and
2505 :ref:`ref-tasks-compile` tasks do
2506 nothing: It is usually sufficient to just not define these tasks in
2507 the recipe, because the default implementations do nothing unless a
2508 Makefile is found in
2509 ``${``\ :term:`S`\ ``}``.
2510
2511 If ``${S}`` might contain a Makefile, or if you inherit some class
2512 that replaces ``do_configure`` and ``do_compile`` with custom
2513 versions, then you can use the
2514 ``[``\ :ref:`noexec <bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ``]``
Andrew Geisslerc926e172021-05-07 16:11:35 -05002515 flag to turn the tasks into no-ops, as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002516
2517 do_configure[noexec] = "1"
2518 do_compile[noexec] = "1"
2519
2520 Unlike
2521 :ref:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata:deleting a task`,
2522 using the flag preserves the dependency chain from the
2523 :ref:`ref-tasks-fetch`,
2524 :ref:`ref-tasks-unpack`, and
2525 :ref:`ref-tasks-patch` tasks to the
2526 :ref:`ref-tasks-install` task.
2527
2528- Make sure your ``do_install`` task installs the binaries
2529 appropriately.
2530
2531- Ensure that you set up :term:`FILES`
2532 (usually
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002533 ``FILES:${``\ :term:`PN`\ ``}``) to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002534 point to the files you have installed, which of course depends on
2535 where you have installed them and whether those files are in
2536 different locations than the defaults.
2537
Andrew Geissler6ce62a22020-11-30 19:58:47 -06002538.. note::
2539
2540 If image prelinking is enabled (e.g. "image-prelink" is in :term:`USER_CLASSES`
2541 which it is by default), prelink will change the binaries in the generated images
2542 and this often catches people out. Remove that class to ensure binaries are
2543 preserved exactly if that is necessary.
2544
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002545Following Recipe Style Guidelines
2546---------------------------------
2547
2548When writing recipes, it is good to conform to existing style
Andrew Geisslerd1e89492021-02-12 15:35:20 -06002549guidelines. The :oe_wiki:`OpenEmbedded Styleguide </Styleguide>` wiki page
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002550provides rough guidelines for preferred recipe style.
2551
2552It is common for existing recipes to deviate a bit from this style.
2553However, aiming for at least a consistent style is a good idea. Some
2554practices, such as omitting spaces around ``=`` operators in assignments
2555or ordering recipe components in an erratic way, are widely seen as poor
2556style.
2557
2558Recipe Syntax
2559-------------
2560
2561Understanding recipe file syntax is important for writing recipes. The
2562following list overviews the basic items that make up a BitBake recipe
2563file. For more complete BitBake syntax descriptions, see the
2564":doc:`bitbake-user-manual/bitbake-user-manual-metadata`"
2565chapter of the BitBake User Manual.
2566
2567- *Variable Assignments and Manipulations:* Variable assignments allow
2568 a value to be assigned to a variable. The assignment can be static
2569 text or might include the contents of other variables. In addition to
2570 the assignment, appending and prepending operations are also
2571 supported.
2572
2573 The following example shows some of the ways you can use variables in
Andrew Geisslerc926e172021-05-07 16:11:35 -05002574 recipes::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002575
2576 S = "${WORKDIR}/postfix-${PV}"
2577 CFLAGS += "-DNO_ASM"
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002578 SRC_URI:append = " file://fixup.patch"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002579
2580- *Functions:* Functions provide a series of actions to be performed.
2581 You usually use functions to override the default implementation of a
2582 task function or to complement a default function (i.e. append or
2583 prepend to an existing function). Standard functions use ``sh`` shell
2584 syntax, although access to OpenEmbedded variables and internal
2585 methods are also available.
2586
William A. Kennington IIIac69b482021-06-02 12:28:27 -07002587 Here is an example function from the ``sed`` recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002588
2589 do_install () {
2590 autotools_do_install
2591 install -d ${D}${base_bindir}
2592 mv ${D}${bindir}/sed ${D}${base_bindir}/sed
2593 rmdir ${D}${bindir}/
2594 }
2595
2596 It is
2597 also possible to implement new functions that are called between
2598 existing tasks as long as the new functions are not replacing or
2599 complementing the default functions. You can implement functions in
2600 Python instead of shell. Both of these options are not seen in the
2601 majority of recipes.
2602
2603- *Keywords:* BitBake recipes use only a few keywords. You use keywords
2604 to include common functions (``inherit``), load parts of a recipe
2605 from other files (``include`` and ``require``) and export variables
2606 to the environment (``export``).
2607
Andrew Geisslerc926e172021-05-07 16:11:35 -05002608 The following example shows the use of some of these keywords::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002609
2610 export POSTCONF = "${STAGING_BINDIR}/postconf"
2611 inherit autoconf
2612 require otherfile.inc
2613
2614- *Comments (#):* Any lines that begin with the hash character (``#``)
Andrew Geisslerc926e172021-05-07 16:11:35 -05002615 are treated as comment lines and are ignored::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002616
2617 # This is a comment
2618
2619This next list summarizes the most important and most commonly used
2620parts of the recipe syntax. For more information on these parts of the
2621syntax, you can reference the
2622:doc:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata` chapter
2623in the BitBake User Manual.
2624
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002625- *Line Continuation (\\):* Use the backward slash (``\``) character to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002626 split a statement over multiple lines. Place the slash character at
Andrew Geisslerc926e172021-05-07 16:11:35 -05002627 the end of the line that is to be continued on the next line::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002628
2629 VAR = "A really long \
2630 line"
2631
2632 .. note::
2633
2634 You cannot have any characters including spaces or tabs after the
2635 slash character.
2636
2637- *Using Variables (${VARNAME}):* Use the ``${VARNAME}`` syntax to
Andrew Geisslerc926e172021-05-07 16:11:35 -05002638 access the contents of a variable::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002639
2640 SRC_URI = "${SOURCEFORGE_MIRROR}/libpng/zlib-${PV}.tar.gz"
2641
2642 .. note::
2643
2644 It is important to understand that the value of a variable
2645 expressed in this form does not get substituted automatically. The
2646 expansion of these expressions happens on-demand later (e.g.
2647 usually when a function that makes reference to the variable
2648 executes). This behavior ensures that the values are most
2649 appropriate for the context in which they are finally used. On the
2650 rare occasion that you do need the variable expression to be
2651 expanded immediately, you can use the
2652 :=
2653 operator instead of
2654 =
2655 when you make the assignment, but this is not generally needed.
2656
2657- *Quote All Assignments ("value"):* Use double quotes around values in
Andrew Geisslerc926e172021-05-07 16:11:35 -05002658 all variable assignments (e.g. ``"value"``). Following is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002659
2660 VAR1 = "${OTHERVAR}"
2661 VAR2 = "The version is ${PV}"
2662
2663- *Conditional Assignment (?=):* Conditional assignment is used to
2664 assign a value to a variable, but only when the variable is currently
2665 unset. Use the question mark followed by the equal sign (``?=``) to
2666 make a "soft" assignment used for conditional assignment. Typically,
2667 "soft" assignments are used in the ``local.conf`` file for variables
2668 that are allowed to come through from the external environment.
2669
2670 Here is an example where ``VAR1`` is set to "New value" if it is
2671 currently empty. However, if ``VAR1`` has already been set, it
Andrew Geisslerc926e172021-05-07 16:11:35 -05002672 remains unchanged::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002673
2674 VAR1 ?= "New value"
2675
Andrew Geisslerc926e172021-05-07 16:11:35 -05002676 In this next example, ``VAR1`` is left with the value "Original value"::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002677
2678 VAR1 = "Original value"
2679 VAR1 ?= "New value"
2680
2681- *Appending (+=):* Use the plus character followed by the equals sign
2682 (``+=``) to append values to existing variables.
2683
2684 .. note::
2685
2686 This operator adds a space between the existing content of the
2687 variable and the new content.
2688
Andrew Geisslerc926e172021-05-07 16:11:35 -05002689 Here is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002690
2691 SRC_URI += "file://fix-makefile.patch"
2692
2693- *Prepending (=+):* Use the equals sign followed by the plus character
2694 (``=+``) to prepend values to existing variables.
2695
2696 .. note::
2697
2698 This operator adds a space between the new content and the
2699 existing content of the variable.
2700
Andrew Geisslerc926e172021-05-07 16:11:35 -05002701 Here is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002702
2703 VAR =+ "Starts"
2704
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002705- *Appending (:append):* Use the ``:append`` operator to append values
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002706 to existing variables. This operator does not add any additional
2707 space. Also, the operator is applied after all the ``+=``, and ``=+``
2708 operators have been applied and after all ``=`` assignments have
2709 occurred.
2710
2711 The following example shows the space being explicitly added to the
2712 start to ensure the appended value is not merged with the existing
Andrew Geisslerc926e172021-05-07 16:11:35 -05002713 value::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002714
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002715 SRC_URI:append = " file://fix-makefile.patch"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002716
2717 You can also use
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002718 the ``:append`` operator with overrides, which results in the actions
Andrew Geisslerc926e172021-05-07 16:11:35 -05002719 only being performed for the specified target or machine::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002720
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002721 SRC_URI:append:sh4 = " file://fix-makefile.patch"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002722
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002723- *Prepending (:prepend):* Use the ``:prepend`` operator to prepend
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002724 values to existing variables. This operator does not add any
2725 additional space. Also, the operator is applied after all the ``+=``,
2726 and ``=+`` operators have been applied and after all ``=``
2727 assignments have occurred.
2728
2729 The following example shows the space being explicitly added to the
2730 end to ensure the prepended value is not merged with the existing
Andrew Geisslerc926e172021-05-07 16:11:35 -05002731 value::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002732
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002733 CFLAGS:prepend = "-I${S}/myincludes "
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002734
2735 You can also use the
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002736 ``:prepend`` operator with overrides, which results in the actions
Andrew Geisslerc926e172021-05-07 16:11:35 -05002737 only being performed for the specified target or machine::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002738
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002739 CFLAGS:prepend:sh4 = "-I${S}/myincludes "
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002740
2741- *Overrides:* You can use overrides to set a value conditionally,
2742 typically based on how the recipe is being built. For example, to set
2743 the :term:`KBRANCH` variable's
2744 value to "standard/base" for any target
2745 :term:`MACHINE`, except for
2746 qemuarm where it should be set to "standard/arm-versatile-926ejs",
Andrew Geisslerc926e172021-05-07 16:11:35 -05002747 you would do the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002748
2749 KBRANCH = "standard/base"
Patrick Williams0ca19cc2021-08-16 14:03:13 -05002750 KBRANCH:qemuarm = "standard/arm-versatile-926ejs"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002751
2752 Overrides are also used to separate
2753 alternate values of a variable in other situations. For example, when
2754 setting variables such as
2755 :term:`FILES` and
2756 :term:`RDEPENDS` that are
2757 specific to individual packages produced by a recipe, you should
2758 always use an override that specifies the name of the package.
2759
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002760- *Indentation:* Use spaces for indentation rather than tabs. For
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002761 shell functions, both currently work. However, it is a policy
2762 decision of the Yocto Project to use tabs in shell functions. Realize
2763 that some layers have a policy to use spaces for all indentation.
2764
2765- *Using Python for Complex Operations:* For more advanced processing,
2766 it is possible to use Python code during variable assignments (e.g.
2767 search and replacement on a variable).
2768
2769 You indicate Python code using the ``${@python_code}`` syntax for the
Andrew Geisslerc926e172021-05-07 16:11:35 -05002770 variable assignment::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002771
2772 SRC_URI = "ftp://ftp.info-zip.org/pub/infozip/src/zip${@d.getVar('PV',1).replace('.', '')}.tgz
2773
2774- *Shell Function Syntax:* Write shell functions as if you were writing
2775 a shell script when you describe a list of actions to take. You
2776 should ensure that your script works with a generic ``sh`` and that
2777 it does not require any ``bash`` or other shell-specific
2778 functionality. The same considerations apply to various system
2779 utilities (e.g. ``sed``, ``grep``, ``awk``, and so forth) that you
2780 might wish to use. If in doubt, you should check with multiple
2781 implementations - including those from BusyBox.
2782
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002783Adding a New Machine
2784====================
2785
2786Adding a new machine to the Yocto Project is a straightforward process.
2787This section describes how to add machines that are similar to those
2788that the Yocto Project already supports.
2789
2790.. note::
2791
2792 Although well within the capabilities of the Yocto Project, adding a
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002793 totally new architecture might require changes to ``gcc``/``glibc``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002794 and to the site information, which is beyond the scope of this
2795 manual.
2796
2797For a complete example that shows how to add a new machine, see the
2798":ref:`bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script`"
2799section in the Yocto Project Board Support Package (BSP) Developer's
2800Guide.
2801
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002802Adding the Machine Configuration File
2803-------------------------------------
2804
2805To add a new machine, you need to add a new machine configuration file
2806to the layer's ``conf/machine`` directory. This configuration file
2807provides details about the device you are adding.
2808
2809The OpenEmbedded build system uses the root name of the machine
2810configuration file to reference the new machine. For example, given a
2811machine configuration file named ``crownbay.conf``, the build system
2812recognizes the machine as "crownbay".
2813
2814The most important variables you must set in your machine configuration
2815file or include from a lower-level configuration file are as follows:
2816
Andrew Geissler5f350902021-07-23 13:09:54 -04002817- :term:`TARGET_ARCH` (e.g. "arm")
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002818
2819- ``PREFERRED_PROVIDER_virtual/kernel``
2820
Andrew Geissler09036742021-06-25 14:25:14 -05002821- :term:`MACHINE_FEATURES` (e.g. "apm screen wifi")
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002822
2823You might also need these variables:
2824
Andrew Geissler5f350902021-07-23 13:09:54 -04002825- :term:`SERIAL_CONSOLES` (e.g. "115200;ttyS0 115200;ttyS1")
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002826
Andrew Geissler5f350902021-07-23 13:09:54 -04002827- :term:`KERNEL_IMAGETYPE` (e.g. "zImage")
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002828
Andrew Geissler09036742021-06-25 14:25:14 -05002829- :term:`IMAGE_FSTYPES` (e.g. "tar.gz jffs2")
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002830
2831You can find full details on these variables in the reference section.
2832You can leverage existing machine ``.conf`` files from
2833``meta-yocto-bsp/conf/machine/``.
2834
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002835Adding a Kernel for the Machine
2836-------------------------------
2837
2838The OpenEmbedded build system needs to be able to build a kernel for the
2839machine. You need to either create a new kernel recipe for this machine,
2840or extend an existing kernel recipe. You can find several kernel recipe
2841examples in the Source Directory at ``meta/recipes-kernel/linux`` that
2842you can use as references.
2843
2844If you are creating a new kernel recipe, normal recipe-writing rules
Andrew Geissler09036742021-06-25 14:25:14 -05002845apply for setting up a :term:`SRC_URI`. Thus, you need to specify any
2846necessary patches and set :term:`S` to point at the source code. You need to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002847create a ``do_configure`` task that configures the unpacked kernel with
2848a ``defconfig`` file. You can do this by using a ``make defconfig``
2849command or, more commonly, by copying in a suitable ``defconfig`` file
2850and then running ``make oldconfig``. By making use of ``inherit kernel``
2851and potentially some of the ``linux-*.inc`` files, most other
2852functionality is centralized and the defaults of the class normally work
2853well.
2854
2855If you are extending an existing kernel recipe, it is usually a matter
2856of adding a suitable ``defconfig`` file. The file needs to be added into
2857a location similar to ``defconfig`` files used for other machines in a
2858given kernel recipe. A possible way to do this is by listing the file in
Andrew Geissler09036742021-06-25 14:25:14 -05002859the :term:`SRC_URI` and adding the machine to the expression in
2860:term:`COMPATIBLE_MACHINE`::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002861
2862 COMPATIBLE_MACHINE = '(qemux86|qemumips)'
2863
2864For more information on ``defconfig`` files, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06002865":ref:`kernel-dev/common:changing the configuration`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002866section in the Yocto Project Linux Kernel Development Manual.
2867
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002868Adding a Formfactor Configuration File
2869--------------------------------------
2870
2871A formfactor configuration file provides information about the target
2872hardware for which the image is being built and information that the
2873build system cannot obtain from other sources such as the kernel. Some
2874examples of information contained in a formfactor configuration file
2875include framebuffer orientation, whether or not the system has a
2876keyboard, the positioning of the keyboard in relation to the screen, and
2877the screen resolution.
2878
2879The build system uses reasonable defaults in most cases. However, if
2880customization is necessary, you need to create a ``machconfig`` file in
2881the ``meta/recipes-bsp/formfactor/files`` directory. This directory
2882contains directories for specific machines such as ``qemuarm`` and
2883``qemux86``. For information about the settings available and the
2884defaults, see the ``meta/recipes-bsp/formfactor/files/config`` file
2885found in the same area.
2886
Andrew Geisslerc926e172021-05-07 16:11:35 -05002887Following is an example for "qemuarm" machine::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002888
2889 HAVE_TOUCHSCREEN=1
2890 HAVE_KEYBOARD=1
2891 DISPLAY_CAN_ROTATE=0
2892 DISPLAY_ORIENTATION=0
2893 #DISPLAY_WIDTH_PIXELS=640
2894 #DISPLAY_HEIGHT_PIXELS=480
2895 #DISPLAY_BPP=16
2896 DISPLAY_DPI=150
2897 DISPLAY_SUBPIXEL_ORDER=vrgb
2898
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002899Upgrading Recipes
2900=================
2901
2902Over time, upstream developers publish new versions for software built
2903by layer recipes. It is recommended to keep recipes up-to-date with
2904upstream version releases.
2905
William A. Kennington IIIac69b482021-06-02 12:28:27 -07002906While there are several methods to upgrade a recipe, you might
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002907consider checking on the upgrade status of a recipe first. You can do so
2908using the ``devtool check-upgrade-status`` command. See the
2909":ref:`devtool-checking-on-the-upgrade-status-of-a-recipe`"
2910section in the Yocto Project Reference Manual for more information.
2911
2912The remainder of this section describes three ways you can upgrade a
2913recipe. You can use the Automated Upgrade Helper (AUH) to set up
2914automatic version upgrades. Alternatively, you can use
2915``devtool upgrade`` to set up semi-automatic version upgrades. Finally,
2916you can manually upgrade a recipe by editing the recipe itself.
2917
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002918Using the Auto Upgrade Helper (AUH)
2919-----------------------------------
2920
2921The AUH utility works in conjunction with the OpenEmbedded build system
2922in order to automatically generate upgrades for recipes based on new
2923versions being published upstream. Use AUH when you want to create a
2924service that performs the upgrades automatically and optionally sends
2925you an email with the results.
2926
2927AUH allows you to update several recipes with a single use. You can also
2928optionally perform build and integration tests using images with the
2929results saved to your hard drive and emails of results optionally sent
2930to recipe maintainers. Finally, AUH creates Git commits with appropriate
2931commit messages in the layer's tree for the changes made to recipes.
2932
2933.. note::
2934
William A. Kennington IIIac69b482021-06-02 12:28:27 -07002935 In some conditions, you should not use AUH to upgrade recipes
2936 and should instead use either ``devtool upgrade`` or upgrade your
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002937 recipes manually:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002938
2939 - When AUH cannot complete the upgrade sequence. This situation
2940 usually results because custom patches carried by the recipe
2941 cannot be automatically rebased to the new version. In this case,
2942 ``devtool upgrade`` allows you to manually resolve conflicts.
2943
2944 - When for any reason you want fuller control over the upgrade
2945 process. For example, when you want special arrangements for
2946 testing.
2947
2948The following steps describe how to set up the AUH utility:
2949
29501. *Be Sure the Development Host is Set Up:* You need to be sure that
2951 your development host is set up to use the Yocto Project. For
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002952 information on how to set up your host, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06002953 ":ref:`dev-manual/start:Preparing the Build Host`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002954
29552. *Make Sure Git is Configured:* The AUH utility requires Git to be
2956 configured because AUH uses Git to save upgrades. Thus, you must have
2957 Git user and email configured. The following command shows your
Andrew Geisslerc926e172021-05-07 16:11:35 -05002958 configurations::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002959
2960 $ git config --list
2961
2962 If you do not have the user and
Andrew Geisslerc926e172021-05-07 16:11:35 -05002963 email configured, you can use the following commands to do so::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002964
2965 $ git config --global user.name some_name
2966 $ git config --global user.email username@domain.com
2967
29683. *Clone the AUH Repository:* To use AUH, you must clone the repository
2969 onto your development host. The following command uses Git to create
Andrew Geisslerc926e172021-05-07 16:11:35 -05002970 a local copy of the repository on your system::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002971
2972 $ git clone git://git.yoctoproject.org/auto-upgrade-helper
2973 Cloning into 'auto-upgrade-helper'... remote: Counting objects: 768, done.
2974 remote: Compressing objects: 100% (300/300), done.
2975 remote: Total 768 (delta 499), reused 703 (delta 434)
2976 Receiving objects: 100% (768/768), 191.47 KiB | 98.00 KiB/s, done.
2977 Resolving deltas: 100% (499/499), done.
2978 Checking connectivity... done.
2979
2980 AUH is not part of the :term:`OpenEmbedded-Core (OE-Core)` or
2981 :term:`Poky` repositories.
2982
29834. *Create a Dedicated Build Directory:* Run the
2984 :ref:`structure-core-script`
2985 script to create a fresh build directory that you use exclusively for
Andrew Geisslerc926e172021-05-07 16:11:35 -05002986 running the AUH utility::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002987
Andrew Geissler95ac1b82021-03-31 14:34:31 -05002988 $ cd poky
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002989 $ source oe-init-build-env your_AUH_build_directory
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002990
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002991 Re-using an existing build directory and its configurations is not
2992 recommended as existing settings could cause AUH to fail or behave
2993 undesirably.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002994
29955. *Make Configurations in Your Local Configuration File:* Several
William A. Kennington IIIac69b482021-06-02 12:28:27 -07002996 settings are needed in the ``local.conf`` file in the build
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002997 directory you just created for AUH. Make these following
2998 configurations:
2999
3000 - If you want to enable :ref:`Build
Andrew Geissler09209ee2020-12-13 08:44:15 -06003001 History <dev-manual/common-tasks:maintaining build output quality>`,
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003002 which is optional, you need the following lines in the
Andrew Geisslerc926e172021-05-07 16:11:35 -05003003 ``conf/local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003004
3005 INHERIT =+ "buildhistory"
3006 BUILDHISTORY_COMMIT = "1"
3007
3008 With this configuration and a successful
3009 upgrade, a build history "diff" file appears in the
3010 ``upgrade-helper/work/recipe/buildhistory-diff.txt`` file found in
3011 your build directory.
3012
3013 - If you want to enable testing through the
3014 :ref:`testimage <ref-classes-testimage*>`
3015 class, which is optional, you need to have the following set in
Andrew Geisslerc926e172021-05-07 16:11:35 -05003016 your ``conf/local.conf`` file::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003017
3018 INHERIT += "testimage"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003019
3020 .. note::
3021
3022 If your distro does not enable by default ptest, which Poky
Andrew Geisslerc926e172021-05-07 16:11:35 -05003023 does, you need the following in your ``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003024
Patrick Williams0ca19cc2021-08-16 14:03:13 -05003025 DISTRO_FEATURES:append = " ptest"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003026
3027
30286. *Optionally Start a vncserver:* If you are running in a server
Andrew Geisslerc926e172021-05-07 16:11:35 -05003029 without an X11 session, you need to start a vncserver::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003030
3031 $ vncserver :1
3032 $ export DISPLAY=:1
3033
30347. *Create and Edit an AUH Configuration File:* You need to have the
3035 ``upgrade-helper/upgrade-helper.conf`` configuration file in your
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003036 build directory. You can find a sample configuration file in the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003037 :yocto_git:`AUH source repository </auto-upgrade-helper/tree/>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003038
3039 Read through the sample file and make configurations as needed. For
3040 example, if you enabled build history in your ``local.conf`` as
3041 described earlier, you must enable it in ``upgrade-helper.conf``.
3042
3043 Also, if you are using the default ``maintainers.inc`` file supplied
3044 with Poky and located in ``meta-yocto`` and you do not set a
3045 "maintainers_whitelist" or "global_maintainer_override" in the
3046 ``upgrade-helper.conf`` configuration, and you specify "-e all" on
3047 the AUH command-line, the utility automatically sends out emails to
3048 all the default maintainers. Please avoid this.
3049
3050This next set of examples describes how to use the AUH:
3051
3052- *Upgrading a Specific Recipe:* To upgrade a specific recipe, use the
Andrew Geisslerc926e172021-05-07 16:11:35 -05003053 following form::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003054
3055 $ upgrade-helper.py recipe_name
3056
Andrew Geisslerc926e172021-05-07 16:11:35 -05003057 For example, this command upgrades the ``xmodmap`` recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003058
3059 $ upgrade-helper.py xmodmap
3060
3061- *Upgrading a Specific Recipe to a Particular Version:* To upgrade a
Andrew Geisslerc926e172021-05-07 16:11:35 -05003062 specific recipe to a particular version, use the following form::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003063
3064 $ upgrade-helper.py recipe_name -t version
3065
Andrew Geisslerc926e172021-05-07 16:11:35 -05003066 For example, this command upgrades the ``xmodmap`` recipe to version 1.2.3::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003067
3068 $ upgrade-helper.py xmodmap -t 1.2.3
3069
3070- *Upgrading all Recipes to the Latest Versions and Suppressing Email
3071 Notifications:* To upgrade all recipes to their most recent versions
Andrew Geisslerc926e172021-05-07 16:11:35 -05003072 and suppress the email notifications, use the following command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003073
3074 $ upgrade-helper.py all
3075
3076- *Upgrading all Recipes to the Latest Versions and Send Email
3077 Notifications:* To upgrade all recipes to their most recent versions
3078 and send email messages to maintainers for each attempted recipe as
Andrew Geisslerc926e172021-05-07 16:11:35 -05003079 well as a status email, use the following command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003080
3081 $ upgrade-helper.py -e all
3082
3083Once you have run the AUH utility, you can find the results in the AUH
Andrew Geisslerc926e172021-05-07 16:11:35 -05003084build directory::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003085
3086 ${BUILDDIR}/upgrade-helper/timestamp
3087
3088The AUH utility
3089also creates recipe update commits from successful upgrade attempts in
3090the layer tree.
3091
3092You can easily set up to run the AUH utility on a regular basis by using
3093a cron job. See the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003094:yocto_git:`weeklyjob.sh </auto-upgrade-helper/tree/weeklyjob.sh>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003095file distributed with the utility for an example.
3096
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003097Using ``devtool upgrade``
3098-------------------------
3099
3100As mentioned earlier, an alternative method for upgrading recipes to
3101newer versions is to use
Andrew Geissler09209ee2020-12-13 08:44:15 -06003102:doc:`devtool upgrade </ref-manual/devtool-reference>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003103You can read about ``devtool upgrade`` in general in the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003104":ref:`sdk-manual/extensible:use \`\`devtool upgrade\`\` to create a version of the recipe that supports a newer version of the software`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003105section in the Yocto Project Application Development and the Extensible
3106Software Development Kit (eSDK) Manual.
3107
3108To see all the command-line options available with ``devtool upgrade``,
Andrew Geisslerc926e172021-05-07 16:11:35 -05003109use the following help command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003110
3111 $ devtool upgrade -h
3112
3113If you want to find out what version a recipe is currently at upstream
3114without any attempt to upgrade your local version of the recipe, you can
Andrew Geisslerc926e172021-05-07 16:11:35 -05003115use the following command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003116
3117 $ devtool latest-version recipe_name
3118
3119As mentioned in the previous section describing AUH, ``devtool upgrade``
3120works in a less-automated manner than AUH. Specifically,
3121``devtool upgrade`` only works on a single recipe that you name on the
3122command line, cannot perform build and integration testing using images,
3123and does not automatically generate commits for changes in the source
3124tree. Despite all these "limitations", ``devtool upgrade`` updates the
3125recipe file to the new upstream version and attempts to rebase custom
3126patches contained by the recipe as needed.
3127
3128.. note::
3129
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003130 AUH uses much of ``devtool upgrade`` behind the scenes making AUH somewhat
3131 of a "wrapper" application for ``devtool upgrade``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003132
3133A typical scenario involves having used Git to clone an upstream
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003134repository that you use during build operations. Because you have built the
3135recipe in the past, the layer is likely added to your
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003136configuration already. If for some reason, the layer is not added, you
3137could add it easily using the
3138":ref:`bitbake-layers <bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script>`"
3139script. For example, suppose you use the ``nano.bb`` recipe from the
3140``meta-oe`` layer in the ``meta-openembedded`` repository. For this
Andrew Geisslerc926e172021-05-07 16:11:35 -05003141example, assume that the layer has been cloned into following area::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003142
3143 /home/scottrif/meta-openembedded
3144
3145The following command from your
3146:term:`Build Directory` adds the layer to
Andrew Geisslerc926e172021-05-07 16:11:35 -05003147your build configuration (i.e. ``${BUILDDIR}/conf/bblayers.conf``)::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003148
3149 $ bitbake-layers add-layer /home/scottrif/meta-openembedded/meta-oe
3150 NOTE: Starting bitbake server...
3151 Parsing recipes: 100% |##########################################| Time: 0:00:55
3152 Parsing of 1431 .bb files complete (0 cached, 1431 parsed). 2040 targets, 56 skipped, 0 masked, 0 errors.
3153 Removing 12 recipes from the x86_64 sysroot: 100% |##############| Time: 0:00:00
3154 Removing 1 recipes from the x86_64_i586 sysroot: 100% |##########| Time: 0:00:00
3155 Removing 5 recipes from the i586 sysroot: 100% |#################| Time: 0:00:00
3156 Removing 5 recipes from the qemux86 sysroot: 100% |##############| Time: 0:00:00
3157
3158For this example, assume that the ``nano.bb`` recipe that
3159is upstream has a 2.9.3 version number. However, the version in the
3160local repository is 2.7.4. The following command from your build
3161directory automatically upgrades the recipe for you:
3162
3163.. note::
3164
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003165 Using the ``-V`` option is not necessary. Omitting the version number causes
3166 ``devtool upgrade`` to upgrade the recipe to the most recent version.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003167
3168::
3169
3170 $ devtool upgrade nano -V 2.9.3
3171 NOTE: Starting bitbake server...
3172 NOTE: Creating workspace layer in /home/scottrif/poky/build/workspace
3173 Parsing recipes: 100% |##########################################| Time: 0:00:46
3174 Parsing of 1431 .bb files complete (0 cached, 1431 parsed). 2040 targets, 56 skipped, 0 masked, 0 errors.
3175 NOTE: Extracting current version source...
3176 NOTE: Resolving any missing task queue dependencies
3177 .
3178 .
3179 .
3180 NOTE: Executing SetScene Tasks
3181 NOTE: Executing RunQueue Tasks
3182 NOTE: Tasks Summary: Attempted 74 tasks of which 72 didn't need to be rerun and all succeeded.
3183 Adding changed files: 100% |#####################################| Time: 0:00:00
3184 NOTE: Upgraded source extracted to /home/scottrif/poky/build/workspace/sources/nano
3185 NOTE: New recipe is /home/scottrif/poky/build/workspace/recipes/nano/nano_2.9.3.bb
3186
3187Continuing with this example, you can use ``devtool build`` to build the
Andrew Geisslerc926e172021-05-07 16:11:35 -05003188newly upgraded recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003189
3190 $ devtool build nano
3191 NOTE: Starting bitbake server...
3192 Loading cache: 100% |################################################################################################| Time: 0:00:01
3193 Loaded 2040 entries from dependency cache.
3194 Parsing recipes: 100% |##############################################################################################| Time: 0:00:00
3195 Parsing of 1432 .bb files complete (1431 cached, 1 parsed). 2041 targets, 56 skipped, 0 masked, 0 errors.
3196 NOTE: Resolving any missing task queue dependencies
3197 .
3198 .
3199 .
3200 NOTE: Executing SetScene Tasks
3201 NOTE: Executing RunQueue Tasks
3202 NOTE: nano: compiling from external source tree /home/scottrif/poky/build/workspace/sources/nano
3203 NOTE: Tasks Summary: Attempted 520 tasks of which 304 didn't need to be rerun and all succeeded.
3204
William A. Kennington IIIac69b482021-06-02 12:28:27 -07003205Within the ``devtool upgrade`` workflow, you can
3206deploy and test your rebuilt software. For this example,
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003207however, running ``devtool finish`` cleans up the workspace once the
3208source in your workspace is clean. This usually means using Git to stage
3209and submit commits for the changes generated by the upgrade process.
3210
3211Once the tree is clean, you can clean things up in this example with the
3212following command from the ``${BUILDDIR}/workspace/sources/nano``
Andrew Geisslerc926e172021-05-07 16:11:35 -05003213directory::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003214
3215 $ devtool finish nano meta-oe
3216 NOTE: Starting bitbake server...
3217 Loading cache: 100% |################################################################################################| Time: 0:00:00
3218 Loaded 2040 entries from dependency cache.
3219 Parsing recipes: 100% |##############################################################################################| Time: 0:00:01
3220 Parsing of 1432 .bb files complete (1431 cached, 1 parsed). 2041 targets, 56 skipped, 0 masked, 0 errors.
3221 NOTE: Adding new patch 0001-nano.bb-Stuff-I-changed-when-upgrading-nano.bb.patch
3222 NOTE: Updating recipe nano_2.9.3.bb
3223 NOTE: Removing file /home/scottrif/meta-openembedded/meta-oe/recipes-support/nano/nano_2.7.4.bb
3224 NOTE: Moving recipe file to /home/scottrif/meta-openembedded/meta-oe/recipes-support/nano
3225 NOTE: Leaving source tree /home/scottrif/poky/build/workspace/sources/nano as-is; if you no longer need it then please delete it manually
3226
3227
3228Using the ``devtool finish`` command cleans up the workspace and creates a patch
3229file based on your commits. The tool puts all patch files back into the
3230source directory in a sub-directory named ``nano`` in this case.
3231
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003232Manually Upgrading a Recipe
3233---------------------------
3234
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003235If for some reason you choose not to upgrade recipes using
Andrew Geissler09209ee2020-12-13 08:44:15 -06003236:ref:`dev-manual/common-tasks:Using the Auto Upgrade Helper (AUH)` or
3237by :ref:`dev-manual/common-tasks:Using \`\`devtool upgrade\`\``,
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003238you can manually edit the recipe files to upgrade the versions.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003239
3240.. note::
3241
3242 Manually updating multiple recipes scales poorly and involves many
3243 steps. The recommendation to upgrade recipe versions is through AUH
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003244 or ``devtool upgrade``, both of which automate some steps and provide
3245 guidance for others needed for the manual process.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003246
3247To manually upgrade recipe versions, follow these general steps:
3248
32491. *Change the Version:* Rename the recipe such that the version (i.e.
3250 the :term:`PV` part of the recipe name)
3251 changes appropriately. If the version is not part of the recipe name,
Andrew Geissler09036742021-06-25 14:25:14 -05003252 change the value as it is set for :term:`PV` within the recipe itself.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003253
Andrew Geissler09036742021-06-25 14:25:14 -050032542. *Update* :term:`SRCREV` *if Needed*: If the source code your recipe builds
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003255 is fetched from Git or some other version control system, update
3256 :term:`SRCREV` to point to the
3257 commit hash that matches the new version.
3258
32593. *Build the Software:* Try to build the recipe using BitBake. Typical
3260 build failures include the following:
3261
3262 - License statements were updated for the new version. For this
3263 case, you need to review any changes to the license and update the
3264 values of :term:`LICENSE` and
3265 :term:`LIC_FILES_CHKSUM`
3266 as needed.
3267
3268 .. note::
3269
3270 License changes are often inconsequential. For example, the
3271 license text's copyright year might have changed.
3272
3273 - Custom patches carried by the older version of the recipe might
3274 fail to apply to the new version. For these cases, you need to
3275 review the failures. Patches might not be necessary for the new
3276 version of the software if the upgraded version has fixed those
3277 issues. If a patch is necessary and failing, you need to rebase it
3278 into the new version.
3279
32804. *Optionally Attempt to Build for Several Architectures:* Once you
3281 successfully build the new software for a given architecture, you
3282 could test the build for other architectures by changing the
3283 :term:`MACHINE` variable and
3284 rebuilding the software. This optional step is especially important
3285 if the recipe is to be released publicly.
3286
32875. *Check the Upstream Change Log or Release Notes:* Checking both these
William A. Kennington IIIac69b482021-06-02 12:28:27 -07003288 reveals if there are new features that could break
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003289 backwards-compatibility. If so, you need to take steps to mitigate or
3290 eliminate that situation.
3291
32926. *Optionally Create a Bootable Image and Test:* If you want, you can
3293 test the new software by booting it onto actual hardware.
3294
32957. *Create a Commit with the Change in the Layer Repository:* After all
3296 builds work and any testing is successful, you can create commits for
3297 any changes in the layer holding your upgraded recipe.
3298
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003299Finding Temporary Source Code
3300=============================
3301
3302You might find it helpful during development to modify the temporary
3303source code used by recipes to build packages. For example, suppose you
3304are developing a patch and you need to experiment a bit to figure out
3305your solution. After you have initially built the package, you can
3306iteratively tweak the source code, which is located in the
3307:term:`Build Directory`, and then you can
3308force a re-compile and quickly test your altered code. Once you settle
3309on a solution, you can then preserve your changes in the form of
3310patches.
3311
3312During a build, the unpacked temporary source code used by recipes to
3313build packages is available in the Build Directory as defined by the
3314:term:`S` variable. Below is the default
Andrew Geissler09036742021-06-25 14:25:14 -05003315value for the :term:`S` variable as defined in the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003316``meta/conf/bitbake.conf`` configuration file in the
Andrew Geisslerc926e172021-05-07 16:11:35 -05003317:term:`Source Directory`::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003318
3319 S = "${WORKDIR}/${BP}"
3320
3321You should be aware that many recipes override the
Andrew Geissler09036742021-06-25 14:25:14 -05003322:term:`S` variable. For example, recipes that fetch their source from Git
3323usually set :term:`S` to ``${WORKDIR}/git``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003324
3325.. note::
3326
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003327 The :term:`BP` represents the base recipe name, which consists of the name
Andrew Geisslerc926e172021-05-07 16:11:35 -05003328 and version::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003329
3330 BP = "${BPN}-${PV}"
3331
3332
3333The path to the work directory for the recipe
3334(:term:`WORKDIR`) is defined as
Andrew Geisslerc926e172021-05-07 16:11:35 -05003335follows::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003336
3337 ${TMPDIR}/work/${MULTIMACH_TARGET_SYS}/${PN}/${EXTENDPE}${PV}-${PR}
3338
3339The actual directory depends on several things:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003340
3341- :term:`TMPDIR`: The top-level build
3342 output directory.
3343
3344- :term:`MULTIMACH_TARGET_SYS`:
3345 The target system identifier.
3346
3347- :term:`PN`: The recipe name.
3348
3349- :term:`EXTENDPE`: The epoch - (if
3350 :term:`PE` is not specified, which is
Andrew Geissler5f350902021-07-23 13:09:54 -04003351 usually the case for most recipes, then :term:`EXTENDPE` is blank).
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003352
3353- :term:`PV`: The recipe version.
3354
3355- :term:`PR`: The recipe revision.
3356
3357As an example, assume a Source Directory top-level folder named
3358``poky``, a default Build Directory at ``poky/build``, and a
3359``qemux86-poky-linux`` machine target system. Furthermore, suppose your
3360recipe is named ``foo_1.3.0.bb``. In this case, the work directory the
Andrew Geisslerc926e172021-05-07 16:11:35 -05003361build system uses to build the package would be as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003362
3363 poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0
3364
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003365Using Quilt in Your Workflow
3366============================
3367
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003368`Quilt <https://savannah.nongnu.org/projects/quilt>`__ is a powerful tool
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003369that allows you to capture source code changes without having a clean
3370source tree. This section outlines the typical workflow you can use to
3371modify source code, test changes, and then preserve the changes in the
3372form of a patch all using Quilt.
3373
3374.. note::
3375
3376 With regard to preserving changes to source files, if you clean a
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003377 recipe or have ``rm_work`` enabled, the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003378 :ref:`devtool workflow <sdk-manual/extensible:using \`\`devtool\`\` in your sdk workflow>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003379 as described in the Yocto Project Application Development and the
3380 Extensible Software Development Kit (eSDK) manual is a safer
3381 development flow than the flow that uses Quilt.
3382
3383Follow these general steps:
3384
33851. *Find the Source Code:* Temporary source code used by the
3386 OpenEmbedded build system is kept in the
3387 :term:`Build Directory`. See the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05003388 ":ref:`dev-manual/common-tasks:finding temporary source code`" section to
3389 learn how to locate the directory that has the temporary source code for a
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003390 particular package.
3391
33922. *Change Your Working Directory:* You need to be in the directory that
3393 has the temporary source code. That directory is defined by the
3394 :term:`S` variable.
3395
33963. *Create a New Patch:* Before modifying source code, you need to
3397 create a new patch. To create a new patch file, use ``quilt new`` as
Andrew Geisslerc926e172021-05-07 16:11:35 -05003398 below::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003399
3400 $ quilt new my_changes.patch
3401
34024. *Notify Quilt and Add Files:* After creating the patch, you need to
3403 notify Quilt about the files you plan to edit. You notify Quilt by
Andrew Geisslerc926e172021-05-07 16:11:35 -05003404 adding the files to the patch you just created::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003405
3406 $ quilt add file1.c file2.c file3.c
3407
34085. *Edit the Files:* Make your changes in the source code to the files
3409 you added to the patch.
3410
34116. *Test Your Changes:* Once you have modified the source code, the
3412 easiest way to test your changes is by calling the ``do_compile``
Andrew Geisslerc926e172021-05-07 16:11:35 -05003413 task as shown in the following example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003414
3415 $ bitbake -c compile -f package
3416
3417 The ``-f`` or ``--force`` option forces the specified task to
3418 execute. If you find problems with your code, you can just keep
3419 editing and re-testing iteratively until things work as expected.
3420
3421 .. note::
3422
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003423 All the modifications you make to the temporary source code disappear
3424 once you run the ``do_clean`` or ``do_cleanall`` tasks using BitBake
3425 (i.e. ``bitbake -c clean package`` and ``bitbake -c cleanall package``).
3426 Modifications will also disappear if you use the ``rm_work`` feature as
3427 described in the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003428 ":ref:`dev-manual/common-tasks:conserving disk space during builds`"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003429 section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003430
34317. *Generate the Patch:* Once your changes work as expected, you need to
3432 use Quilt to generate the final patch that contains all your
3433 modifications.
3434 ::
3435
3436 $ quilt refresh
3437
3438 At this point, the
3439 ``my_changes.patch`` file has all your edits made to the ``file1.c``,
3440 ``file2.c``, and ``file3.c`` files.
3441
3442 You can find the resulting patch file in the ``patches/``
Andrew Geissler09036742021-06-25 14:25:14 -05003443 subdirectory of the source (:term:`S`) directory.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003444
34458. *Copy the Patch File:* For simplicity, copy the patch file into a
3446 directory named ``files``, which you can create in the same directory
3447 that holds the recipe (``.bb``) file or the append (``.bbappend``)
3448 file. Placing the patch here guarantees that the OpenEmbedded build
Andrew Geissler09036742021-06-25 14:25:14 -05003449 system will find the patch. Next, add the patch into the :term:`SRC_URI`
Andrew Geisslerc926e172021-05-07 16:11:35 -05003450 of the recipe. Here is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003451
3452 SRC_URI += "file://my_changes.patch"
3453
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003454Using a Development Shell
3455=========================
3456
3457When debugging certain commands or even when just editing packages,
3458``devshell`` can be a useful tool. When you invoke ``devshell``, all
3459tasks up to and including
3460:ref:`ref-tasks-patch` are run for the
3461specified target. Then, a new terminal is opened and you are placed in
3462``${``\ :term:`S`\ ``}``, the source
3463directory. In the new terminal, all the OpenEmbedded build-related
3464environment variables are still defined so you can use commands such as
3465``configure`` and ``make``. The commands execute just as if the
3466OpenEmbedded build system were executing them. Consequently, working
3467this way can be helpful when debugging a build or preparing software to
3468be used with the OpenEmbedded build system.
3469
3470Following is an example that uses ``devshell`` on a target named
Andrew Geisslerc926e172021-05-07 16:11:35 -05003471``matchbox-desktop``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003472
3473 $ bitbake matchbox-desktop -c devshell
3474
3475This command spawns a terminal with a shell prompt within the
3476OpenEmbedded build environment. The
3477:term:`OE_TERMINAL` variable
3478controls what type of shell is opened.
3479
3480For spawned terminals, the following occurs:
3481
3482- The ``PATH`` variable includes the cross-toolchain.
3483
3484- The ``pkgconfig`` variables find the correct ``.pc`` files.
3485
3486- The ``configure`` command finds the Yocto Project site files as well
3487 as any other necessary files.
3488
3489Within this environment, you can run configure or compile commands as if
3490they were being run by the OpenEmbedded build system itself. As noted
3491earlier, the working directory also automatically changes to the Source
3492Directory (:term:`S`).
3493
3494To manually run a specific task using ``devshell``, run the
3495corresponding ``run.*`` script in the
3496``${``\ :term:`WORKDIR`\ ``}/temp``
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003497directory (e.g., ``run.do_configure.``\ `pid`). If a task's script does
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003498not exist, which would be the case if the task was skipped by way of the
3499sstate cache, you can create the task by first running it outside of the
Andrew Geisslerc926e172021-05-07 16:11:35 -05003500``devshell``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003501
3502 $ bitbake -c task
3503
3504.. note::
3505
3506 - Execution of a task's ``run.*`` script and BitBake's execution of
3507 a task are identical. In other words, running the script re-runs
3508 the task just as it would be run using the ``bitbake -c`` command.
3509
3510 - Any ``run.*`` file that does not have a ``.pid`` extension is a
3511 symbolic link (symlink) to the most recent version of that file.
3512
3513Remember, that the ``devshell`` is a mechanism that allows you to get
3514into the BitBake task execution environment. And as such, all commands
3515must be called just as BitBake would call them. That means you need to
3516provide the appropriate options for cross-compilation and so forth as
3517applicable.
3518
3519When you are finished using ``devshell``, exit the shell or close the
3520terminal window.
3521
3522.. note::
3523
3524 - It is worth remembering that when using ``devshell`` you need to
3525 use the full compiler name such as ``arm-poky-linux-gnueabi-gcc``
3526 instead of just using ``gcc``. The same applies to other
3527 applications such as ``binutils``, ``libtool`` and so forth.
Andrew Geissler09036742021-06-25 14:25:14 -05003528 BitBake sets up environment variables such as :term:`CC` to assist
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003529 applications, such as ``make`` to find the correct tools.
3530
3531 - It is also worth noting that ``devshell`` still works over X11
3532 forwarding and similar situations.
3533
Andrew Geisslereff27472021-10-29 15:35:00 -05003534Using a Python Development Shell
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003535================================
3536
3537Similar to working within a development shell as described in the
3538previous section, you can also spawn and work within an interactive
3539Python development shell. When debugging certain commands or even when
Andrew Geisslereff27472021-10-29 15:35:00 -05003540just editing packages, ``pydevshell`` can be a useful tool. When you
3541invoke the ``pydevshell`` task, all tasks up to and including
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003542:ref:`ref-tasks-patch` are run for the
3543specified target. Then a new terminal is opened. Additionally, key
3544Python objects and code are available in the same way they are to
3545BitBake tasks, in particular, the data store 'd'. So, commands such as
3546the following are useful when exploring the data store and running
Andrew Geisslerc926e172021-05-07 16:11:35 -05003547functions::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003548
3549 pydevshell> d.getVar("STAGING_DIR")
3550 '/media/build1/poky/build/tmp/sysroots'
Andrew Geissler5199d832021-09-24 16:47:35 -05003551 pydevshell> d.getVar("STAGING_DIR", False)
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003552 '${TMPDIR}/sysroots'
3553 pydevshell> d.setVar("FOO", "bar")
3554 pydevshell> d.getVar("FOO")
3555 'bar'
3556 pydevshell> d.delVar("FOO")
3557 pydevshell> d.getVar("FOO")
3558 pydevshell> bb.build.exec_func("do_unpack", d)
3559 pydevshell>
3560
3561The commands execute just as if the OpenEmbedded build
3562system were executing them. Consequently, working this way can be
3563helpful when debugging a build or preparing software to be used with the
3564OpenEmbedded build system.
3565
Andrew Geisslereff27472021-10-29 15:35:00 -05003566Following is an example that uses ``pydevshell`` on a target named
Andrew Geisslerc926e172021-05-07 16:11:35 -05003567``matchbox-desktop``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003568
Andrew Geisslereff27472021-10-29 15:35:00 -05003569 $ bitbake matchbox-desktop -c pydevshell
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003570
3571This command spawns a terminal and places you in an interactive Python
3572interpreter within the OpenEmbedded build environment. The
3573:term:`OE_TERMINAL` variable
3574controls what type of shell is opened.
3575
Andrew Geisslereff27472021-10-29 15:35:00 -05003576When you are finished using ``pydevshell``, you can exit the shell
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003577either by using Ctrl+d or closing the terminal window.
3578
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003579Building
3580========
3581
Andrew Geissler5199d832021-09-24 16:47:35 -05003582This section describes various build procedures, such as the steps
3583needed for a simple build, building a target for multiple configurations,
3584generating an image for more than one machine, and so forth.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003585
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003586Building a Simple Image
3587-----------------------
3588
3589In the development environment, you need to build an image whenever you
3590change hardware support, add or change system libraries, or add or
William A. Kennington IIIac69b482021-06-02 12:28:27 -07003591change services that have dependencies. There are several methods that allow
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003592you to build an image within the Yocto Project. This section presents
3593the basic steps you need to build a simple image using BitBake from a
3594build host running Linux.
3595
3596.. note::
3597
3598 - For information on how to build an image using
3599 :term:`Toaster`, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003600 :doc:`/toaster-manual/index`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003601
3602 - For information on how to use ``devtool`` to build images, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003603 ":ref:`sdk-manual/extensible:using \`\`devtool\`\` in your sdk workflow`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003604 section in the Yocto Project Application Development and the
3605 Extensible Software Development Kit (eSDK) manual.
3606
3607 - For a quick example on how to build an image using the
3608 OpenEmbedded build system, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003609 :doc:`/brief-yoctoprojectqs/index` document.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003610
3611The build process creates an entire Linux distribution from source and
3612places it in your :term:`Build Directory` under
3613``tmp/deploy/images``. For detailed information on the build process
Andrew Geissler09209ee2020-12-13 08:44:15 -06003614using BitBake, see the ":ref:`overview-manual/concepts:images`" section in the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003615Yocto Project Overview and Concepts Manual.
3616
3617The following figure and list overviews the build process:
3618
3619.. image:: figures/bitbake-build-flow.png
3620 :align: center
3621
36221. *Set up Your Host Development System to Support Development Using the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003623 Yocto Project*: See the ":doc:`start`" section for options on how to get a
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003624 build host ready to use the Yocto Project.
3625
36262. *Initialize the Build Environment:* Initialize the build environment
3627 by sourcing the build environment script (i.e.
Andrew Geisslerc926e172021-05-07 16:11:35 -05003628 :ref:`structure-core-script`)::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003629
3630 $ source oe-init-build-env [build_dir]
3631
3632 When you use the initialization script, the OpenEmbedded build system
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003633 uses ``build`` as the default :term:`Build Directory` in your current work
3634 directory. You can use a `build_dir` argument with the script to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003635 specify a different build directory.
3636
3637 .. note::
3638
3639 A common practice is to use a different Build Directory for
Andrew Geissler5199d832021-09-24 16:47:35 -05003640 different targets; for example, ``~/build/x86`` for a ``qemux86``
3641 target, and ``~/build/arm`` for a ``qemuarm`` target. In any
3642 event, it's typically cleaner to locate the build directory
3643 somewhere outside of your source directory.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003644
Andrew Geissler4c19ea12020-10-27 13:52:24 -050036453. *Make Sure Your* ``local.conf`` *File is Correct*: Ensure the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003646 ``conf/local.conf`` configuration file, which is found in the Build
3647 Directory, is set up how you want it. This file defines many aspects
3648 of the build environment including the target machine architecture
Andrew Geissler09036742021-06-25 14:25:14 -05003649 through the :term:`MACHINE` variable, the packaging format used during
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003650 the build
3651 (:term:`PACKAGE_CLASSES`),
3652 and a centralized tarball download directory through the
3653 :term:`DL_DIR` variable.
3654
Andrew Geisslerc926e172021-05-07 16:11:35 -050036554. *Build the Image:* Build the image using the ``bitbake`` command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003656
3657 $ bitbake target
3658
3659 .. note::
3660
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003661 For information on BitBake, see the :doc:`bitbake:index`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003662
3663 The target is the name of the recipe you want to build. Common
3664 targets are the images in ``meta/recipes-core/images``,
3665 ``meta/recipes-sato/images``, and so forth all found in the
Andrew Geissler5199d832021-09-24 16:47:35 -05003666 :term:`Source Directory`. Alternatively, the target
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003667 can be the name of a recipe for a specific piece of software such as
3668 BusyBox. For more details about the images the OpenEmbedded build
3669 system supports, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003670 ":ref:`ref-manual/images:Images`" chapter in the Yocto
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003671 Project Reference Manual.
3672
3673 As an example, the following command builds the
Andrew Geisslerc926e172021-05-07 16:11:35 -05003674 ``core-image-minimal`` image::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003675
3676 $ bitbake core-image-minimal
3677
3678 Once an
3679 image has been built, it often needs to be installed. The images and
3680 kernels built by the OpenEmbedded build system are placed in the
3681 Build Directory in ``tmp/deploy/images``. For information on how to
3682 run pre-built images such as ``qemux86`` and ``qemuarm``, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003683 :doc:`/sdk-manual/index` manual. For
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003684 information about how to install these images, see the documentation
3685 for your particular board or machine.
3686
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003687Building Images for Multiple Targets Using Multiple Configurations
3688------------------------------------------------------------------
3689
3690You can use a single ``bitbake`` command to build multiple images or
3691packages for different targets where each image or package requires a
3692different configuration (multiple configuration builds). The builds, in
3693this scenario, are sometimes referred to as "multiconfigs", and this
3694section uses that term throughout.
3695
3696This section describes how to set up for multiple configuration builds
3697and how to account for cross-build dependencies between the
3698multiconfigs.
3699
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003700Setting Up and Running a Multiple Configuration Build
3701~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3702
3703To accomplish a multiple configuration build, you must define each
3704target's configuration separately using a parallel configuration file in
3705the :term:`Build Directory`, and you
3706must follow a required file hierarchy. Additionally, you must enable the
3707multiple configuration builds in your ``local.conf`` file.
3708
3709Follow these steps to set up and execute multiple configuration builds:
3710
3711- *Create Separate Configuration Files*: You need to create a single
3712 configuration file for each build target (each multiconfig).
3713 Minimally, each configuration file must define the machine and the
3714 temporary directory BitBake uses for the build. Suggested practice
3715 dictates that you do not overlap the temporary directories used
3716 during the builds. However, it is possible that you can share the
3717 temporary directory
3718 (:term:`TMPDIR`). For example,
3719 consider a scenario with two different multiconfigs for the same
3720 :term:`MACHINE`: "qemux86" built
3721 for two distributions such as "poky" and "poky-lsb". In this case,
Andrew Geissler09036742021-06-25 14:25:14 -05003722 you might want to use the same :term:`TMPDIR`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003723
3724 Here is an example showing the minimal statements needed in a
3725 configuration file for a "qemux86" target whose temporary build
Andrew Geisslerc926e172021-05-07 16:11:35 -05003726 directory is ``tmpmultix86``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003727
3728 MACHINE = "qemux86"
3729 TMPDIR = "${TOPDIR}/tmpmultix86"
3730
3731 The location for these multiconfig configuration files is specific.
3732 They must reside in the current build directory in a sub-directory of
3733 ``conf`` named ``multiconfig``. Following is an example that defines
3734 two configuration files for the "x86" and "arm" multiconfigs:
3735
3736 .. image:: figures/multiconfig_files.png
3737 :align: center
3738
Andrew Geissler09036742021-06-25 14:25:14 -05003739 The reason for this required file hierarchy is because the :term:`BBPATH`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003740 variable is not constructed until the layers are parsed.
3741 Consequently, using the configuration file as a pre-configuration
3742 file is not possible unless it is located in the current working
3743 directory.
3744
3745- *Add the BitBake Multi-configuration Variable to the Local
3746 Configuration File*: Use the
3747 :term:`BBMULTICONFIG`
3748 variable in your ``conf/local.conf`` configuration file to specify
3749 each multiconfig. Continuing with the example from the previous
Andrew Geissler09036742021-06-25 14:25:14 -05003750 figure, the :term:`BBMULTICONFIG` variable needs to enable two
Andrew Geisslerc926e172021-05-07 16:11:35 -05003751 multiconfigs: "x86" and "arm" by specifying each configuration file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003752
3753 BBMULTICONFIG = "x86 arm"
3754
3755 .. note::
3756
3757 A "default" configuration already exists by definition. This
3758 configuration is named: "" (i.e. empty string) and is defined by
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003759 the variables coming from your ``local.conf``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003760 file. Consequently, the previous example actually adds two
3761 additional configurations to your build: "arm" and "x86" along
3762 with "".
3763
3764- *Launch BitBake*: Use the following BitBake command form to launch
Andrew Geisslerc926e172021-05-07 16:11:35 -05003765 the multiple configuration build::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003766
3767 $ bitbake [mc:multiconfigname:]target [[[mc:multiconfigname:]target] ... ]
3768
Andrew Geisslerc926e172021-05-07 16:11:35 -05003769 For the example in this section, the following command applies::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003770
3771 $ bitbake mc:x86:core-image-minimal mc:arm:core-image-sato mc::core-image-base
3772
3773 The previous BitBake command builds a ``core-image-minimal`` image
3774 that is configured through the ``x86.conf`` configuration file, a
3775 ``core-image-sato`` image that is configured through the ``arm.conf``
3776 configuration file and a ``core-image-base`` that is configured
3777 through your ``local.conf`` configuration file.
3778
3779.. note::
3780
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003781 Support for multiple configuration builds in the Yocto Project &DISTRO;
3782 (&DISTRO_NAME;) Release does not include Shared State (sstate)
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003783 optimizations. Consequently, if a build uses the same object twice
Andrew Geissler09036742021-06-25 14:25:14 -05003784 in, for example, two different :term:`TMPDIR`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003785 directories, the build either loads from an existing sstate cache for
3786 that build at the start or builds the object fresh.
3787
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003788Enabling Multiple Configuration Build Dependencies
3789~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3790
3791Sometimes dependencies can exist between targets (multiconfigs) in a
3792multiple configuration build. For example, suppose that in order to
3793build a ``core-image-sato`` image for an "x86" multiconfig, the root
3794filesystem of an "arm" multiconfig must exist. This dependency is
3795essentially that the
3796:ref:`ref-tasks-image` task in the
3797``core-image-sato`` recipe depends on the completion of the
3798:ref:`ref-tasks-rootfs` task of the
3799``core-image-minimal`` recipe.
3800
3801To enable dependencies in a multiple configuration build, you must
3802declare the dependencies in the recipe using the following statement
Andrew Geisslerc926e172021-05-07 16:11:35 -05003803form::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003804
3805 task_or_package[mcdepends] = "mc:from_multiconfig:to_multiconfig:recipe_name:task_on_which_to_depend"
3806
3807To better show how to use this statement, consider the example scenario
3808from the first paragraph of this section. The following statement needs
Andrew Geisslerc926e172021-05-07 16:11:35 -05003809to be added to the recipe that builds the ``core-image-sato`` image::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003810
3811 do_image[mcdepends] = "mc:x86:arm:core-image-minimal:do_rootfs"
3812
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003813In this example, the `from_multiconfig` is "x86". The `to_multiconfig` is "arm". The
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003814task on which the ``do_image`` task in the recipe depends is the
3815``do_rootfs`` task from the ``core-image-minimal`` recipe associated
3816with the "arm" multiconfig.
3817
3818Once you set up this dependency, you can build the "x86" multiconfig
Andrew Geisslerc926e172021-05-07 16:11:35 -05003819using a BitBake command as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003820
3821 $ bitbake mc:x86:core-image-sato
3822
3823This command executes all the tasks needed to create the
3824``core-image-sato`` image for the "x86" multiconfig. Because of the
3825dependency, BitBake also executes through the ``do_rootfs`` task for the
3826"arm" multiconfig build.
3827
3828Having a recipe depend on the root filesystem of another build might not
3829seem that useful. Consider this change to the statement in the
Andrew Geisslerc926e172021-05-07 16:11:35 -05003830``core-image-sato`` recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003831
3832 do_image[mcdepends] = "mc:x86:arm:core-image-minimal:do_image"
3833
3834In this case, BitBake must
3835create the ``core-image-minimal`` image for the "arm" build since the
3836"x86" build depends on it.
3837
3838Because "x86" and "arm" are enabled for multiple configuration builds
3839and have separate configuration files, BitBake places the artifacts for
3840each build in the respective temporary build directories (i.e.
3841:term:`TMPDIR`).
3842
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003843Building an Initial RAM Filesystem (initramfs) Image
3844----------------------------------------------------
3845
3846An initial RAM filesystem (initramfs) image provides a temporary root
3847filesystem used for early system initialization (e.g. loading of modules
3848needed to locate and mount the "real" root filesystem).
3849
3850.. note::
3851
3852 The initramfs image is the successor of initial RAM disk (initrd). It
3853 is a "copy in and out" (cpio) archive of the initial filesystem that
3854 gets loaded into memory during the Linux startup process. Because
3855 Linux uses the contents of the archive during initialization, the
3856 initramfs image needs to contain all of the device drivers and tools
3857 needed to mount the final root filesystem.
3858
3859Follow these steps to create an initramfs image:
3860
38611. *Create the initramfs Image Recipe:* You can reference the
3862 ``core-image-minimal-initramfs.bb`` recipe found in the
3863 ``meta/recipes-core`` directory of the :term:`Source Directory`
3864 as an example
3865 from which to work.
3866
38672. *Decide if You Need to Bundle the initramfs Image Into the Kernel
3868 Image:* If you want the initramfs image that is built to be bundled
3869 in with the kernel image, set the
3870 :term:`INITRAMFS_IMAGE_BUNDLE`
3871 variable to "1" in your ``local.conf`` configuration file and set the
3872 :term:`INITRAMFS_IMAGE`
3873 variable in the recipe that builds the kernel image.
3874
3875 .. note::
3876
Andrew Geissler5199d832021-09-24 16:47:35 -05003877 It is recommended that you bundle the initramfs image with the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003878 kernel image to avoid circular dependencies between the kernel
3879 recipe and the initramfs recipe should the initramfs image include
3880 kernel modules.
3881
Andrew Geissler09036742021-06-25 14:25:14 -05003882 Setting the :term:`INITRAMFS_IMAGE_BUNDLE` flag causes the initramfs
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003883 image to be unpacked into the ``${B}/usr/`` directory. The unpacked
3884 initramfs image is then passed to the kernel's ``Makefile`` using the
3885 :term:`CONFIG_INITRAMFS_SOURCE`
3886 variable, allowing the initramfs image to be built into the kernel
3887 normally.
3888
3889 .. note::
3890
Patrick Williams93c203f2021-10-06 16:15:23 -05003891 Bundling the initramfs with the kernel conflates the code in the initramfs
3892 with the GPLv2 licensed Linux kernel binary. Thus only GPLv2 compatible
3893 software may be part of a bundled initramfs.
3894
3895 .. note::
3896
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003897 If you choose to not bundle the initramfs image with the kernel
3898 image, you are essentially using an
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003899 `Initial RAM Disk (initrd) <https://en.wikipedia.org/wiki/Initrd>`__.
3900 Creating an initrd is handled primarily through the :term:`INITRD_IMAGE`,
3901 ``INITRD_LIVE``, and ``INITRD_IMAGE_LIVE`` variables. For more
3902 information, see the :ref:`ref-classes-image-live` file.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003903
39043. *Optionally Add Items to the initramfs Image Through the initramfs
3905 Image Recipe:* If you add items to the initramfs image by way of its
3906 recipe, you should use
3907 :term:`PACKAGE_INSTALL`
3908 rather than
3909 :term:`IMAGE_INSTALL`.
Andrew Geissler09036742021-06-25 14:25:14 -05003910 :term:`PACKAGE_INSTALL` gives more direct control of what is added to the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003911 image as compared to the defaults you might not necessarily want that
3912 are set by the :ref:`image <ref-classes-image>`
3913 or :ref:`core-image <ref-classes-core-image>`
3914 classes.
3915
39164. *Build the Kernel Image and the initramfs Image:* Build your kernel
3917 image using BitBake. Because the initramfs image recipe is a
3918 dependency of the kernel image, the initramfs image is built as well
3919 and bundled with the kernel image if you used the
3920 :term:`INITRAMFS_IMAGE_BUNDLE`
3921 variable described earlier.
3922
3923Building a Tiny System
3924----------------------
3925
3926Very small distributions have some significant advantages such as
3927requiring less on-die or in-package memory (cheaper), better performance
3928through efficient cache usage, lower power requirements due to less
3929memory, faster boot times, and reduced development overhead. Some
3930real-world examples where a very small distribution gives you distinct
3931advantages are digital cameras, medical devices, and small headless
3932systems.
3933
3934This section presents information that shows you how you can trim your
3935distribution to even smaller sizes than the ``poky-tiny`` distribution,
3936which is around 5 Mbytes, that can be built out-of-the-box using the
3937Yocto Project.
3938
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003939Tiny System Overview
3940~~~~~~~~~~~~~~~~~~~~
3941
3942The following list presents the overall steps you need to consider and
3943perform to create distributions with smaller root filesystems, achieve
3944faster boot times, maintain your critical functionality, and avoid
3945initial RAM disks:
3946
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05003947- :ref:`Determine your goals and guiding principles
3948 <dev-manual/common-tasks:goals and guiding principles>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003949
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05003950- :ref:`dev-manual/common-tasks:understand what contributes to your image size`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003951
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05003952- :ref:`Reduce the size of the root filesystem
3953 <dev-manual/common-tasks:trim the root filesystem>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003954
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05003955- :ref:`Reduce the size of the kernel <dev-manual/common-tasks:trim the kernel>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003956
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05003957- :ref:`dev-manual/common-tasks:remove package management requirements`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003958
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05003959- :ref:`dev-manual/common-tasks:look for other ways to minimize size`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003960
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05003961- :ref:`dev-manual/common-tasks:iterate on the process`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003962
3963Goals and Guiding Principles
3964~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3965
3966Before you can reach your destination, you need to know where you are
3967going. Here is an example list that you can use as a guide when creating
3968very small distributions:
3969
3970- Determine how much space you need (e.g. a kernel that is 1 Mbyte or
3971 less and a root filesystem that is 3 Mbytes or less).
3972
3973- Find the areas that are currently taking 90% of the space and
3974 concentrate on reducing those areas.
3975
3976- Do not create any difficult "hacks" to achieve your goals.
3977
3978- Leverage the device-specific options.
3979
3980- Work in a separate layer so that you keep changes isolated. For
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05003981 information on how to create layers, see the
3982 ":ref:`dev-manual/common-tasks:understanding and creating layers`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003983
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003984Understand What Contributes to Your Image Size
3985~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3986
3987It is easiest to have something to start with when creating your own
3988distribution. You can use the Yocto Project out-of-the-box to create the
3989``poky-tiny`` distribution. Ultimately, you will want to make changes in
3990your own distribution that are likely modeled after ``poky-tiny``.
3991
3992.. note::
3993
Andrew Geissler09036742021-06-25 14:25:14 -05003994 To use ``poky-tiny`` in your build, set the :term:`DISTRO` variable in your
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003995 ``local.conf`` file to "poky-tiny" as described in the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003996 ":ref:`dev-manual/common-tasks:creating your own distribution`"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003997 section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003998
3999Understanding some memory concepts will help you reduce the system size.
4000Memory consists of static, dynamic, and temporary memory. Static memory
4001is the TEXT (code), DATA (initialized data in the code), and BSS
4002(uninitialized data) sections. Dynamic memory represents memory that is
4003allocated at runtime: stacks, hash tables, and so forth. Temporary
4004memory is recovered after the boot process. This memory consists of
4005memory used for decompressing the kernel and for the ``__init__``
4006functions.
4007
4008To help you see where you currently are with kernel and root filesystem
4009sizes, you can use two tools found in the :term:`Source Directory`
4010in the
4011``scripts/tiny/`` directory:
4012
4013- ``ksize.py``: Reports component sizes for the kernel build objects.
4014
4015- ``dirsize.py``: Reports component sizes for the root filesystem.
4016
4017This next tool and command help you organize configuration fragments and
4018view file dependencies in a human-readable form:
4019
4020- ``merge_config.sh``: Helps you manage configuration files and
4021 fragments within the kernel. With this tool, you can merge individual
4022 configuration fragments together. The tool allows you to make
4023 overrides and warns you of any missing configuration options. The
4024 tool is ideal for allowing you to iterate on configurations, create
4025 minimal configurations, and create configuration files for different
4026 machines without having to duplicate your process.
4027
4028 The ``merge_config.sh`` script is part of the Linux Yocto kernel Git
4029 repositories (i.e. ``linux-yocto-3.14``, ``linux-yocto-3.10``,
4030 ``linux-yocto-3.8``, and so forth) in the ``scripts/kconfig``
4031 directory.
4032
4033 For more information on configuration fragments, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06004034 ":ref:`kernel-dev/common:creating configuration fragments`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004035 section in the Yocto Project Linux Kernel Development Manual.
4036
4037- ``bitbake -u taskexp -g bitbake_target``: Using the BitBake command
4038 with these options brings up a Dependency Explorer from which you can
4039 view file dependencies. Understanding these dependencies allows you
4040 to make informed decisions when cutting out various pieces of the
4041 kernel and root filesystem.
4042
4043Trim the Root Filesystem
4044~~~~~~~~~~~~~~~~~~~~~~~~
4045
4046The root filesystem is made up of packages for booting, libraries, and
4047applications. To change things, you can configure how the packaging
4048happens, which changes the way you build them. You can also modify the
4049filesystem itself or select a different filesystem.
4050
4051First, find out what is hogging your root filesystem by running the
Andrew Geisslerc926e172021-05-07 16:11:35 -05004052``dirsize.py`` script from your root directory::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004053
4054 $ cd root-directory-of-image
4055 $ dirsize.py 100000 > dirsize-100k.log
4056 $ cat dirsize-100k.log
4057
4058You can apply a filter to the script to ignore files
4059under a certain size. The previous example filters out any files below
4060100 Kbytes. The sizes reported by the tool are uncompressed, and thus
4061will be smaller by a relatively constant factor in a compressed root
4062filesystem. When you examine your log file, you can focus on areas of
4063the root filesystem that take up large amounts of memory.
4064
4065You need to be sure that what you eliminate does not cripple the
4066functionality you need. One way to see how packages relate to each other
Andrew Geisslerc926e172021-05-07 16:11:35 -05004067is by using the Dependency Explorer UI with the BitBake command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004068
4069 $ cd image-directory
4070 $ bitbake -u taskexp -g image
4071
4072Use the interface to
4073select potential packages you wish to eliminate and see their dependency
4074relationships.
4075
4076When deciding how to reduce the size, get rid of packages that result in
4077minimal impact on the feature set. For example, you might not need a VGA
4078display. Or, you might be able to get by with ``devtmpfs`` and ``mdev``
4079instead of ``udev``.
4080
4081Use your ``local.conf`` file to make changes. For example, to eliminate
4082``udev`` and ``glib``, set the following in the local configuration
Andrew Geisslerc926e172021-05-07 16:11:35 -05004083file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004084
4085 VIRTUAL-RUNTIME_dev_manager = ""
4086
4087Finally, you should consider exactly the type of root filesystem you
4088need to meet your needs while also reducing its size. For example,
4089consider ``cramfs``, ``squashfs``, ``ubifs``, ``ext2``, or an
4090``initramfs`` using ``initramfs``. Be aware that ``ext3`` requires a 1
4091Mbyte journal. If you are okay with running read-only, you do not need
4092this journal.
4093
4094.. note::
4095
4096 After each round of elimination, you need to rebuild your system and
4097 then use the tools to see the effects of your reductions.
4098
4099Trim the Kernel
4100~~~~~~~~~~~~~~~
4101
4102The kernel is built by including policies for hardware-independent
4103aspects. What subsystems do you enable? For what architecture are you
4104building? Which drivers do you build by default?
4105
4106.. note::
4107
4108 You can modify the kernel source if you want to help with boot time.
4109
4110Run the ``ksize.py`` script from the top-level Linux build directory to
Andrew Geisslerc926e172021-05-07 16:11:35 -05004111get an idea of what is making up the kernel::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004112
4113 $ cd top-level-linux-build-directory
4114 $ ksize.py > ksize.log
4115 $ cat ksize.log
4116
4117When you examine the log, you will see how much space is taken up with
4118the built-in ``.o`` files for drivers, networking, core kernel files,
4119filesystem, sound, and so forth. The sizes reported by the tool are
4120uncompressed, and thus will be smaller by a relatively constant factor
4121in a compressed kernel image. Look to reduce the areas that are large
4122and taking up around the "90% rule."
4123
4124To examine, or drill down, into any particular area, use the ``-d``
Andrew Geisslerc926e172021-05-07 16:11:35 -05004125option with the script::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004126
4127 $ ksize.py -d > ksize.log
4128
4129Using this option
4130breaks out the individual file information for each area of the kernel
4131(e.g. drivers, networking, and so forth).
4132
4133Use your log file to see what you can eliminate from the kernel based on
4134features you can let go. For example, if you are not going to need
4135sound, you do not need any drivers that support sound.
4136
4137After figuring out what to eliminate, you need to reconfigure the kernel
4138to reflect those changes during the next build. You could run
4139``menuconfig`` and make all your changes at once. However, that makes it
4140difficult to see the effects of your individual eliminations and also
4141makes it difficult to replicate the changes for perhaps another target
4142device. A better method is to start with no configurations using
4143``allnoconfig``, create configuration fragments for individual changes,
4144and then manage the fragments into a single configuration file using
4145``merge_config.sh``. The tool makes it easy for you to iterate using the
4146configuration change and build cycle.
4147
4148Each time you make configuration changes, you need to rebuild the kernel
4149and check to see what impact your changes had on the overall size.
4150
4151Remove Package Management Requirements
4152~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4153
4154Packaging requirements add size to the image. One way to reduce the size
4155of the image is to remove all the packaging requirements from the image.
4156This reduction includes both removing the package manager and its unique
4157dependencies as well as removing the package management data itself.
4158
4159To eliminate all the packaging requirements for an image, be sure that
4160"package-management" is not part of your
4161:term:`IMAGE_FEATURES`
4162statement for the image. When you remove this feature, you are removing
4163the package manager as well as its dependencies from the root
4164filesystem.
4165
4166Look for Other Ways to Minimize Size
4167~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4168
4169Depending on your particular circumstances, other areas that you can
4170trim likely exist. The key to finding these areas is through tools and
4171methods described here combined with experimentation and iteration. Here
4172are a couple of areas to experiment with:
4173
4174- ``glibc``: In general, follow this process:
4175
4176 1. Remove ``glibc`` features from
4177 :term:`DISTRO_FEATURES`
4178 that you think you do not need.
4179
4180 2. Build your distribution.
4181
4182 3. If the build fails due to missing symbols in a package, determine
4183 if you can reconfigure the package to not need those features. For
4184 example, change the configuration to not support wide character
4185 support as is done for ``ncurses``. Or, if support for those
4186 characters is needed, determine what ``glibc`` features provide
4187 the support and restore the configuration.
4188
4189 4. Rebuild and repeat the process.
4190
4191- ``busybox``: For BusyBox, use a process similar as described for
4192 ``glibc``. A difference is you will need to boot the resulting system
4193 to see if you are able to do everything you expect from the running
4194 system. You need to be sure to integrate configuration fragments into
4195 Busybox because BusyBox handles its own core features and then allows
4196 you to add configuration fragments on top.
4197
4198Iterate on the Process
4199~~~~~~~~~~~~~~~~~~~~~~
4200
4201If you have not reached your goals on system size, you need to iterate
4202on the process. The process is the same. Use the tools and see just what
4203is taking up 90% of the root filesystem and the kernel. Decide what you
4204can eliminate without limiting your device beyond what you need.
4205
4206Depending on your system, a good place to look might be Busybox, which
4207provides a stripped down version of Unix tools in a single, executable
4208file. You might be able to drop virtual terminal services or perhaps
4209ipv6.
4210
4211Building Images for More than One Machine
4212-----------------------------------------
4213
4214A common scenario developers face is creating images for several
4215different machines that use the same software environment. In this
4216situation, it is tempting to set the tunings and optimization flags for
4217each build specifically for the targeted hardware (i.e. "maxing out" the
4218tunings). Doing so can considerably add to build times and package feed
4219maintenance collectively for the machines. For example, selecting tunes
4220that are extremely specific to a CPU core used in a system might enable
4221some micro optimizations in GCC for that particular system but would
4222otherwise not gain you much of a performance difference across the other
4223systems as compared to using a more general tuning across all the builds
4224(e.g. setting :term:`DEFAULTTUNE`
4225specifically for each machine's build). Rather than "max out" each
4226build's tunings, you can take steps that cause the OpenEmbedded build
4227system to reuse software across the various machines where it makes
4228sense.
4229
4230If build speed and package feed maintenance are considerations, you
4231should consider the points in this section that can help you optimize
4232your tunings to best consider build times and package feed maintenance.
4233
4234- *Share the Build Directory:* If at all possible, share the
4235 :term:`TMPDIR` across builds. The
4236 Yocto Project supports switching between different
4237 :term:`MACHINE` values in the same
Andrew Geissler09036742021-06-25 14:25:14 -05004238 :term:`TMPDIR`. This practice is well supported and regularly used by
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004239 developers when building for multiple machines. When you use the same
Andrew Geissler09036742021-06-25 14:25:14 -05004240 :term:`TMPDIR` for multiple machine builds, the OpenEmbedded build system
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004241 can reuse the existing native and often cross-recipes for multiple
4242 machines. Thus, build time decreases.
4243
4244 .. note::
4245
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004246 If :term:`DISTRO` settings change or fundamental configuration settings
Andrew Geissler09036742021-06-25 14:25:14 -05004247 such as the filesystem layout, you need to work with a clean :term:`TMPDIR`.
4248 Sharing :term:`TMPDIR` under these circumstances might work but since it is
Andrew Geissler5f350902021-07-23 13:09:54 -04004249 not guaranteed, you should use a clean :term:`TMPDIR`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004250
4251- *Enable the Appropriate Package Architecture:* By default, the
4252 OpenEmbedded build system enables three levels of package
4253 architectures: "all", "tune" or "package", and "machine". Any given
4254 recipe usually selects one of these package architectures (types) for
4255 its output. Depending for what a given recipe creates packages,
4256 making sure you enable the appropriate package architecture can
4257 directly impact the build time.
4258
4259 A recipe that just generates scripts can enable "all" architecture
4260 because there are no binaries to build. To specifically enable "all"
4261 architecture, be sure your recipe inherits the
4262 :ref:`allarch <ref-classes-allarch>` class.
4263 This class is useful for "all" architectures because it configures
4264 many variables so packages can be used across multiple architectures.
4265
4266 If your recipe needs to generate packages that are machine-specific
4267 or when one of the build or runtime dependencies is already
4268 machine-architecture dependent, which makes your recipe also
4269 machine-architecture dependent, make sure your recipe enables the
4270 "machine" package architecture through the
4271 :term:`MACHINE_ARCH`
Andrew Geisslerc926e172021-05-07 16:11:35 -05004272 variable::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004273
4274 PACKAGE_ARCH = "${MACHINE_ARCH}"
4275
4276 When you do not
4277 specifically enable a package architecture through the
4278 :term:`PACKAGE_ARCH`, The
4279 OpenEmbedded build system defaults to the
Andrew Geisslerc926e172021-05-07 16:11:35 -05004280 :term:`TUNE_PKGARCH` setting::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004281
4282 PACKAGE_ARCH = "${TUNE_PKGARCH}"
4283
4284- *Choose a Generic Tuning File if Possible:* Some tunes are more
4285 generic and can run on multiple targets (e.g. an ``armv5`` set of
4286 packages could run on ``armv6`` and ``armv7`` processors in most
4287 cases). Similarly, ``i486`` binaries could work on ``i586`` and
4288 higher processors. You should realize, however, that advances on
4289 newer processor versions would not be used.
4290
4291 If you select the same tune for several different machines, the
4292 OpenEmbedded build system reuses software previously built, thus
4293 speeding up the overall build time. Realize that even though a new
4294 sysroot for each machine is generated, the software is not recompiled
4295 and only one package feed exists.
4296
William A. Kennington IIIac69b482021-06-02 12:28:27 -07004297- *Manage Granular Level Packaging:* Sometimes there are cases where
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004298 injecting another level of package architecture beyond the three
4299 higher levels noted earlier can be useful. For example, consider how
4300 NXP (formerly Freescale) allows for the easy reuse of binary packages
4301 in their layer
Andrew Geissler09209ee2020-12-13 08:44:15 -06004302 :yocto_git:`meta-freescale </meta-freescale/>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004303 In this example, the
Andrew Geissler09209ee2020-12-13 08:44:15 -06004304 :yocto_git:`fsl-dynamic-packagearch </meta-freescale/tree/classes/fsl-dynamic-packagearch.bbclass>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004305 class shares GPU packages for i.MX53 boards because all boards share
4306 the AMD GPU. The i.MX6-based boards can do the same because all
4307 boards share the Vivante GPU. This class inspects the BitBake
4308 datastore to identify if the package provides or depends on one of
4309 the sub-architecture values. If so, the class sets the
4310 :term:`PACKAGE_ARCH` value
4311 based on the ``MACHINE_SUBARCH`` value. If the package does not
4312 provide or depend on one of the sub-architecture values but it
4313 matches a value in the machine-specific filter, it sets
4314 :term:`MACHINE_ARCH`. This
4315 behavior reduces the number of packages built and saves build time by
4316 reusing binaries.
4317
4318- *Use Tools to Debug Issues:* Sometimes you can run into situations
4319 where software is being rebuilt when you think it should not be. For
4320 example, the OpenEmbedded build system might not be using shared
4321 state between machines when you think it should be. These types of
4322 situations are usually due to references to machine-specific
4323 variables such as :term:`MACHINE`,
4324 :term:`SERIAL_CONSOLES`,
4325 :term:`XSERVER`,
4326 :term:`MACHINE_FEATURES`,
4327 and so forth in code that is supposed to only be tune-specific or
4328 when the recipe depends
4329 (:term:`DEPENDS`,
4330 :term:`RDEPENDS`,
4331 :term:`RRECOMMENDS`,
4332 :term:`RSUGGESTS`, and so forth)
4333 on some other recipe that already has
4334 :term:`PACKAGE_ARCH` defined
4335 as "${MACHINE_ARCH}".
4336
4337 .. note::
4338
4339 Patches to fix any issues identified are most welcome as these
4340 issues occasionally do occur.
4341
4342 For such cases, you can use some tools to help you sort out the
4343 situation:
4344
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004345 - ``state-diff-machines.sh``*:* You can find this tool in the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004346 ``scripts`` directory of the Source Repositories. See the comments
4347 in the script for information on how to use the tool.
4348
4349 - *BitBake's "-S printdiff" Option:* Using this option causes
4350 BitBake to try to establish the closest signature match it can
4351 (e.g. in the shared state cache) and then run ``bitbake-diffsigs``
4352 over the matches to determine the stamps and delta where these two
4353 stamp trees diverge.
4354
4355Building Software from an External Source
4356-----------------------------------------
4357
4358By default, the OpenEmbedded build system uses the
4359:term:`Build Directory` when building source
4360code. The build process involves fetching the source files, unpacking
4361them, and then patching them if necessary before the build takes place.
4362
William A. Kennington IIIac69b482021-06-02 12:28:27 -07004363There are situations where you might want to build software from source
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004364files that are external to and thus outside of the OpenEmbedded build
4365system. For example, suppose you have a project that includes a new BSP
4366with a heavily customized kernel. And, you want to minimize exposing the
4367build system to the development team so that they can focus on their
4368project and maintain everyone's workflow as much as possible. In this
4369case, you want a kernel source directory on the development machine
4370where the development occurs. You want the recipe's
4371:term:`SRC_URI` variable to point to
4372the external directory and use it as is, not copy it.
4373
4374To build from software that comes from an external source, all you need
4375to do is inherit the
4376:ref:`externalsrc <ref-classes-externalsrc>` class
4377and then set the
4378:term:`EXTERNALSRC` variable to
4379point to your external source code. Here are the statements to put in
Andrew Geisslerc926e172021-05-07 16:11:35 -05004380your ``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004381
4382 INHERIT += "externalsrc"
Patrick Williams0ca19cc2021-08-16 14:03:13 -05004383 EXTERNALSRC:pn-myrecipe = "path-to-your-source-tree"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004384
4385This next example shows how to accomplish the same thing by setting
Andrew Geissler09036742021-06-25 14:25:14 -05004386:term:`EXTERNALSRC` in the recipe itself or in the recipe's append file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004387
4388 EXTERNALSRC = "path"
4389 EXTERNALSRC_BUILD = "path"
4390
4391.. note::
4392
4393 In order for these settings to take effect, you must globally or
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004394 locally inherit the :ref:`externalsrc <ref-classes-externalsrc>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004395 class.
4396
4397By default, ``externalsrc.bbclass`` builds the source code in a
4398directory separate from the external source directory as specified by
4399:term:`EXTERNALSRC`. If you need
4400to have the source built in the same directory in which it resides, or
4401some other nominated directory, you can set
4402:term:`EXTERNALSRC_BUILD`
Andrew Geisslerc926e172021-05-07 16:11:35 -05004403to point to that directory::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004404
Patrick Williams0ca19cc2021-08-16 14:03:13 -05004405 EXTERNALSRC_BUILD:pn-myrecipe = "path-to-your-source-tree"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004406
4407Replicating a Build Offline
4408---------------------------
4409
4410It can be useful to take a "snapshot" of upstream sources used in a
4411build and then use that "snapshot" later to replicate the build offline.
4412To do so, you need to first prepare and populate your downloads
4413directory your "snapshot" of files. Once your downloads directory is
4414ready, you can use it at any time and from any machine to replicate your
4415build.
4416
4417Follow these steps to populate your Downloads directory:
4418
44191. *Create a Clean Downloads Directory:* Start with an empty downloads
4420 directory (:term:`DL_DIR`). You
4421 start with an empty downloads directory by either removing the files
Andrew Geissler09036742021-06-25 14:25:14 -05004422 in the existing directory or by setting :term:`DL_DIR` to point to either
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004423 an empty location or one that does not yet exist.
4424
44252. *Generate Tarballs of the Source Git Repositories:* Edit your
Andrew Geisslerc926e172021-05-07 16:11:35 -05004426 ``local.conf`` configuration file as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004427
4428 DL_DIR = "/home/your-download-dir/"
4429 BB_GENERATE_MIRROR_TARBALLS = "1"
4430
4431 During
4432 the fetch process in the next step, BitBake gathers the source files
Andrew Geissler09036742021-06-25 14:25:14 -05004433 and creates tarballs in the directory pointed to by :term:`DL_DIR`. See
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004434 the
4435 :term:`BB_GENERATE_MIRROR_TARBALLS`
4436 variable for more information.
4437
44383. *Populate Your Downloads Directory Without Building:* Use BitBake to
Andrew Geisslerc926e172021-05-07 16:11:35 -05004439 fetch your sources but inhibit the build::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004440
4441 $ bitbake target --runonly=fetch
4442
4443 The downloads directory (i.e. ``${DL_DIR}``) now has
4444 a "snapshot" of the source files in the form of tarballs, which can
4445 be used for the build.
4446
44474. *Optionally Remove Any Git or other SCM Subdirectories From the
4448 Downloads Directory:* If you want, you can clean up your downloads
4449 directory by removing any Git or other Source Control Management
4450 (SCM) subdirectories such as ``${DL_DIR}/git2/*``. The tarballs
4451 already contain these subdirectories.
4452
4453Once your downloads directory has everything it needs regarding source
4454files, you can create your "own-mirror" and build your target.
4455Understand that you can use the files to build the target offline from
4456any machine and at any time.
4457
4458Follow these steps to build your target using the files in the downloads
4459directory:
4460
44611. *Using Local Files Only:* Inside your ``local.conf`` file, add the
4462 :term:`SOURCE_MIRROR_URL`
4463 variable, inherit the
4464 :ref:`own-mirrors <ref-classes-own-mirrors>`
4465 class, and use the
Patrick Williams213cb262021-08-07 19:21:33 -05004466 :term:`BB_NO_NETWORK`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004467 variable to your ``local.conf``.
4468 ::
4469
4470 SOURCE_MIRROR_URL ?= "file:///home/your-download-dir/"
4471 INHERIT += "own-mirrors"
4472 BB_NO_NETWORK = "1"
4473
Andrew Geissler5f350902021-07-23 13:09:54 -04004474 The :term:`SOURCE_MIRROR_URL` and ``own-mirror``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004475 class set up the system to use the downloads directory as your "own
Andrew Geissler09036742021-06-25 14:25:14 -05004476 mirror". Using the :term:`BB_NO_NETWORK` variable makes sure that
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004477 BitBake's fetching process in step 3 stays local, which means files
4478 from your "own-mirror" are used.
4479
44802. *Start With a Clean Build:* You can start with a clean build by
4481 removing the
4482 ``${``\ :term:`TMPDIR`\ ``}``
4483 directory or using a new :term:`Build Directory`.
4484
Andrew Geisslerc926e172021-05-07 16:11:35 -050044853. *Build Your Target:* Use BitBake to build your target::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004486
4487 $ bitbake target
4488
4489 The build completes using the known local "snapshot" of source
4490 files from your mirror. The resulting tarballs for your "snapshot" of
4491 source files are in the downloads directory.
4492
4493 .. note::
4494
4495 The offline build does not work if recipes attempt to find the
4496 latest version of software by setting
4497 :term:`SRCREV` to
Andrew Geisslerc926e172021-05-07 16:11:35 -05004498 ``${``\ :term:`AUTOREV`\ ``}``::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004499
4500 SRCREV = "${AUTOREV}"
4501
Andrew Geissler09036742021-06-25 14:25:14 -05004502 When a recipe sets :term:`SRCREV` to
Andrew Geissler5199d832021-09-24 16:47:35 -05004503 ``${``\ :term:`AUTOREV`\ ``}``, the build system accesses the network in an
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004504 attempt to determine the latest version of software from the SCM.
Andrew Geissler09036742021-06-25 14:25:14 -05004505 Typically, recipes that use :term:`AUTOREV` are custom or modified
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004506 recipes. Recipes that reside in public repositories usually do not
Andrew Geissler09036742021-06-25 14:25:14 -05004507 use :term:`AUTOREV`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004508
Andrew Geissler09036742021-06-25 14:25:14 -05004509 If you do have recipes that use :term:`AUTOREV`, you can take steps to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004510 still use the recipes in an offline build. Do the following:
4511
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05004512 1. Use a configuration generated by enabling :ref:`build
4513 history <dev-manual/common-tasks:maintaining build output quality>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004514
4515 2. Use the ``buildhistory-collect-srcrevs`` command to collect the
Andrew Geissler09036742021-06-25 14:25:14 -05004516 stored :term:`SRCREV` values from the build's history. For more
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05004517 information on collecting these values, see the
4518 ":ref:`dev-manual/common-tasks:build history package information`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004519 section.
4520
4521 3. Once you have the correct source revisions, you can modify
Andrew Geissler09036742021-06-25 14:25:14 -05004522 those recipes to set :term:`SRCREV` to specific versions of the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004523 software.
4524
4525Speeding Up a Build
4526===================
4527
4528Build time can be an issue. By default, the build system uses simple
4529controls to try and maximize build efficiency. In general, the default
4530settings for all the following variables result in the most efficient
4531build times when dealing with single socket systems (i.e. a single CPU).
4532If you have multiple CPUs, you might try increasing the default values
4533to gain more speed. See the descriptions in the glossary for each
4534variable for more information:
4535
4536- :term:`BB_NUMBER_THREADS`:
4537 The maximum number of threads BitBake simultaneously executes.
4538
Patrick Williams213cb262021-08-07 19:21:33 -05004539- :term:`BB_NUMBER_PARSE_THREADS`:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004540 The number of threads BitBake uses during parsing.
4541
4542- :term:`PARALLEL_MAKE`: Extra
4543 options passed to the ``make`` command during the
4544 :ref:`ref-tasks-compile` task in
4545 order to specify parallel compilation on the local build host.
4546
4547- :term:`PARALLEL_MAKEINST`:
4548 Extra options passed to the ``make`` command during the
4549 :ref:`ref-tasks-install` task in
4550 order to specify parallel installation on the local build host.
4551
4552As mentioned, these variables all scale to the number of processor cores
4553available on the build system. For single socket systems, this
4554auto-scaling ensures that the build system fundamentally takes advantage
4555of potential parallel operations during the build based on the build
4556machine's capabilities.
4557
4558Following are additional factors that can affect build speed:
4559
4560- File system type: The file system type that the build is being
4561 performed on can also influence performance. Using ``ext4`` is
4562 recommended as compared to ``ext2`` and ``ext3`` due to ``ext4``
4563 improved features such as extents.
4564
4565- Disabling the updating of access time using ``noatime``: The
4566 ``noatime`` mount option prevents the build system from updating file
4567 and directory access times.
4568
4569- Setting a longer commit: Using the "commit=" mount option increases
4570 the interval in seconds between disk cache writes. Changing this
4571 interval from the five second default to something longer increases
4572 the risk of data loss but decreases the need to write to the disk,
4573 thus increasing the build performance.
4574
4575- Choosing the packaging backend: Of the available packaging backends,
4576 IPK is the fastest. Additionally, selecting a singular packaging
4577 backend also helps.
4578
4579- Using ``tmpfs`` for :term:`TMPDIR`
4580 as a temporary file system: While this can help speed up the build,
4581 the benefits are limited due to the compiler using ``-pipe``. The
4582 build system goes to some lengths to avoid ``sync()`` calls into the
4583 file system on the principle that if there was a significant failure,
4584 the :term:`Build Directory`
4585 contents could easily be rebuilt.
4586
4587- Inheriting the
4588 :ref:`rm_work <ref-classes-rm-work>` class:
4589 Inheriting this class has shown to speed up builds due to
4590 significantly lower amounts of data stored in the data cache as well
4591 as on disk. Inheriting this class also makes cleanup of
4592 :term:`TMPDIR` faster, at the
4593 expense of being easily able to dive into the source code. File
4594 system maintainers have recommended that the fastest way to clean up
4595 large numbers of files is to reformat partitions rather than delete
4596 files due to the linear nature of partitions. This, of course,
4597 assumes you structure the disk partitions and file systems in a way
4598 that this is practical.
4599
4600Aside from the previous list, you should keep some trade offs in mind
4601that can help you speed up the build:
4602
4603- Remove items from
4604 :term:`DISTRO_FEATURES`
4605 that you might not need.
4606
4607- Exclude debug symbols and other debug information: If you do not need
4608 these symbols and other debug information, disabling the ``*-dbg``
4609 package generation can speed up the build. You can disable this
4610 generation by setting the
4611 :term:`INHIBIT_PACKAGE_DEBUG_SPLIT`
4612 variable to "1".
4613
4614- Disable static library generation for recipes derived from
4615 ``autoconf`` or ``libtool``: Following is an example showing how to
4616 disable static libraries and still provide an override to handle
Andrew Geisslerc926e172021-05-07 16:11:35 -05004617 exceptions::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004618
4619 STATICLIBCONF = "--disable-static"
Patrick Williams0ca19cc2021-08-16 14:03:13 -05004620 STATICLIBCONF:sqlite3-native = ""
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004621 EXTRA_OECONF += "${STATICLIBCONF}"
4622
4623 .. note::
4624
4625 - Some recipes need static libraries in order to work correctly
4626 (e.g. ``pseudo-native`` needs ``sqlite3-native``). Overrides,
4627 as in the previous example, account for these kinds of
4628 exceptions.
4629
4630 - Some packages have packaging code that assumes the presence of
4631 the static libraries. If so, you might need to exclude them as
4632 well.
4633
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004634Working With Libraries
4635======================
4636
4637Libraries are an integral part of your system. This section describes
4638some common practices you might find helpful when working with libraries
4639to build your system:
4640
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05004641- :ref:`How to include static library files
4642 <dev-manual/common-tasks:including static library files>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004643
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05004644- :ref:`How to use the Multilib feature to combine multiple versions of
4645 library files into a single image
4646 <dev-manual/common-tasks:combining multiple versions of library files into one image>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004647
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05004648- :ref:`How to install multiple versions of the same library in parallel on
4649 the same system
4650 <dev-manual/common-tasks:installing multiple versions of the same library>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004651
4652Including Static Library Files
4653------------------------------
4654
4655If you are building a library and the library offers static linking, you
4656can control which static library files (``*.a`` files) get included in
4657the built library.
4658
4659The :term:`PACKAGES` and
Patrick Williams0ca19cc2021-08-16 14:03:13 -05004660:term:`FILES:* <FILES>` variables in the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004661``meta/conf/bitbake.conf`` configuration file define how files installed
Andrew Geissler09036742021-06-25 14:25:14 -05004662by the ``do_install`` task are packaged. By default, the :term:`PACKAGES`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004663variable includes ``${PN}-staticdev``, which represents all static
4664library files.
4665
4666.. note::
4667
4668 Some previously released versions of the Yocto Project defined the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004669 static library files through ``${PN}-dev``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004670
4671Following is part of the BitBake configuration file, where you can see
Andrew Geisslerc926e172021-05-07 16:11:35 -05004672how the static library files are defined::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004673
4674 PACKAGE_BEFORE_PN ?= ""
4675 PACKAGES = "${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc ${PN}-locale ${PACKAGE_BEFORE_PN} ${PN}"
4676 PACKAGES_DYNAMIC = "^${PN}-locale-.*"
4677 FILES = ""
4678
Patrick Williams0ca19cc2021-08-16 14:03:13 -05004679 FILES:${PN} = "${bindir}/* ${sbindir}/* ${libexecdir}/* ${libdir}/lib*${SOLIBS} \
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004680 ${sysconfdir} ${sharedstatedir} ${localstatedir} \
4681 ${base_bindir}/* ${base_sbindir}/* \
4682 ${base_libdir}/*${SOLIBS} \
4683 ${base_prefix}/lib/udev/rules.d ${prefix}/lib/udev/rules.d \
4684 ${datadir}/${BPN} ${libdir}/${BPN}/* \
4685 ${datadir}/pixmaps ${datadir}/applications \
4686 ${datadir}/idl ${datadir}/omf ${datadir}/sounds \
4687 ${libdir}/bonobo/servers"
4688
Patrick Williams0ca19cc2021-08-16 14:03:13 -05004689 FILES:${PN}-bin = "${bindir}/* ${sbindir}/*"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004690
Patrick Williams0ca19cc2021-08-16 14:03:13 -05004691 FILES:${PN}-doc = "${docdir} ${mandir} ${infodir} ${datadir}/gtk-doc \
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004692 ${datadir}/gnome/help"
Patrick Williams0ca19cc2021-08-16 14:03:13 -05004693 SECTION:${PN}-doc = "doc"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004694
4695 FILES_SOLIBSDEV ?= "${base_libdir}/lib*${SOLIBSDEV} ${libdir}/lib*${SOLIBSDEV}"
Patrick Williams0ca19cc2021-08-16 14:03:13 -05004696 FILES:${PN}-dev = "${includedir} ${FILES_SOLIBSDEV} ${libdir}/*.la \
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004697 ${libdir}/*.o ${libdir}/pkgconfig ${datadir}/pkgconfig \
4698 ${datadir}/aclocal ${base_libdir}/*.o \
4699 ${libdir}/${BPN}/*.la ${base_libdir}/*.la"
Patrick Williams0ca19cc2021-08-16 14:03:13 -05004700 SECTION:${PN}-dev = "devel"
4701 ALLOW_EMPTY:${PN}-dev = "1"
4702 RDEPENDS:${PN}-dev = "${PN} (= ${EXTENDPKGV})"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004703
Patrick Williams0ca19cc2021-08-16 14:03:13 -05004704 FILES:${PN}-staticdev = "${libdir}/*.a ${base_libdir}/*.a ${libdir}/${BPN}/*.a"
4705 SECTION:${PN}-staticdev = "devel"
4706 RDEPENDS:${PN}-staticdev = "${PN}-dev (= ${EXTENDPKGV})"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004707
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004708Combining Multiple Versions of Library Files into One Image
4709-----------------------------------------------------------
4710
4711The build system offers the ability to build libraries with different
4712target optimizations or architecture formats and combine these together
4713into one system image. You can link different binaries in the image
4714against the different libraries as needed for specific use cases. This
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004715feature is called "Multilib".
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004716
4717An example would be where you have most of a system compiled in 32-bit
4718mode using 32-bit libraries, but you have something large, like a
4719database engine, that needs to be a 64-bit application and uses 64-bit
4720libraries. Multilib allows you to get the best of both 32-bit and 64-bit
4721libraries.
4722
4723While the Multilib feature is most commonly used for 32 and 64-bit
4724differences, the approach the build system uses facilitates different
4725target optimizations. You could compile some binaries to use one set of
4726libraries and other binaries to use a different set of libraries. The
4727libraries could differ in architecture, compiler options, or other
4728optimizations.
4729
William A. Kennington IIIac69b482021-06-02 12:28:27 -07004730There are several examples in the ``meta-skeleton`` layer found in the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004731:term:`Source Directory`:
4732
4733- ``conf/multilib-example.conf`` configuration file
4734
4735- ``conf/multilib-example2.conf`` configuration file
4736
4737- ``recipes-multilib/images/core-image-multilib-example.bb`` recipe
4738
4739Preparing to Use Multilib
4740~~~~~~~~~~~~~~~~~~~~~~~~~
4741
4742User-specific requirements drive the Multilib feature. Consequently,
William A. Kennington IIIac69b482021-06-02 12:28:27 -07004743there is no one "out-of-the-box" configuration that would
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004744meet your needs.
4745
4746In order to enable Multilib, you first need to ensure your recipe is
4747extended to support multiple libraries. Many standard recipes are
4748already extended and support multiple libraries. You can check in the
4749``meta/conf/multilib.conf`` configuration file in the
4750:term:`Source Directory` to see how this is
4751done using the
4752:term:`BBCLASSEXTEND` variable.
4753Eventually, all recipes will be covered and this list will not be
4754needed.
4755
4756For the most part, the Multilib class extension works automatically to
4757extend the package name from ``${PN}`` to ``${MLPREFIX}${PN}``, where
Andrew Geissler5f350902021-07-23 13:09:54 -04004758:term:`MLPREFIX` is the particular multilib (e.g. "lib32-" or "lib64-").
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004759Standard variables such as
4760:term:`DEPENDS`,
4761:term:`RDEPENDS`,
4762:term:`RPROVIDES`,
4763:term:`RRECOMMENDS`,
4764:term:`PACKAGES`, and
4765:term:`PACKAGES_DYNAMIC` are
4766automatically extended by the system. If you are extending any manual
4767code in the recipe, you can use the ``${MLPREFIX}`` variable to ensure
4768those names are extended correctly. This automatic extension code
4769resides in ``multilib.bbclass``.
4770
4771Using Multilib
4772~~~~~~~~~~~~~~
4773
4774After you have set up the recipes, you need to define the actual
4775combination of multiple libraries you want to build. You accomplish this
4776through your ``local.conf`` configuration file in the
4777:term:`Build Directory`. An example
Andrew Geisslerc926e172021-05-07 16:11:35 -05004778configuration would be as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004779
4780 MACHINE = "qemux86-64"
4781 require conf/multilib.conf
4782 MULTILIBS = "multilib:lib32"
Patrick Williams0ca19cc2021-08-16 14:03:13 -05004783 DEFAULTTUNE:virtclass-multilib-lib32 = "x86"
4784 IMAGE_INSTALL:append = "lib32-glib-2.0"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004785
4786This example enables an additional library named
4787``lib32`` alongside the normal target packages. When combining these
4788"lib32" alternatives, the example uses "x86" for tuning. For information
4789on this particular tuning, see
4790``meta/conf/machine/include/ia32/arch-ia32.inc``.
4791
4792The example then includes ``lib32-glib-2.0`` in all the images, which
4793illustrates one method of including a multiple library dependency. You
Andrew Geisslerc926e172021-05-07 16:11:35 -05004794can use a normal image build to include this dependency, for example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004795
4796 $ bitbake core-image-sato
4797
4798You can also build Multilib packages
Andrew Geisslerc926e172021-05-07 16:11:35 -05004799specifically with a command like this::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004800
4801 $ bitbake lib32-glib-2.0
4802
4803Additional Implementation Details
4804~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4805
William A. Kennington IIIac69b482021-06-02 12:28:27 -07004806There are generic implementation details as well as details that are specific to
4807package management systems. Following are implementation details
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004808that exist regardless of the package management system:
4809
4810- The typical convention used for the class extension code as used by
4811 Multilib assumes that all package names specified in
4812 :term:`PACKAGES` that contain
4813 ``${PN}`` have ``${PN}`` at the start of the name. When that
4814 convention is not followed and ``${PN}`` appears at the middle or the
4815 end of a name, problems occur.
4816
4817- The :term:`TARGET_VENDOR`
4818 value under Multilib will be extended to "-vendormlmultilib" (e.g.
4819 "-pokymllib32" for a "lib32" Multilib with Poky). The reason for this
4820 slightly unwieldy contraction is that any "-" characters in the
4821 vendor string presently break Autoconf's ``config.sub``, and other
4822 separators are problematic for different reasons.
4823
William A. Kennington IIIac69b482021-06-02 12:28:27 -07004824Here are the implementation details for the RPM Package Management System:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004825
4826- A unique architecture is defined for the Multilib packages, along
4827 with creating a unique deploy folder under ``tmp/deploy/rpm`` in the
4828 :term:`Build Directory`. For
4829 example, consider ``lib32`` in a ``qemux86-64`` image. The possible
4830 architectures in the system are "all", "qemux86_64",
Patrick Williams0ca19cc2021-08-16 14:03:13 -05004831 "lib32:qemux86_64", and "lib32:x86".
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004832
4833- The ``${MLPREFIX}`` variable is stripped from ``${PN}`` during RPM
4834 packaging. The naming for a normal RPM package and a Multilib RPM
4835 package in a ``qemux86-64`` system resolves to something similar to
4836 ``bash-4.1-r2.x86_64.rpm`` and ``bash-4.1.r2.lib32_x86.rpm``,
4837 respectively.
4838
4839- When installing a Multilib image, the RPM backend first installs the
4840 base image and then installs the Multilib libraries.
4841
4842- The build system relies on RPM to resolve the identical files in the
4843 two (or more) Multilib packages.
4844
William A. Kennington IIIac69b482021-06-02 12:28:27 -07004845Here are the implementation details for the IPK Package Management System:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004846
4847- The ``${MLPREFIX}`` is not stripped from ``${PN}`` during IPK
4848 packaging. The naming for a normal RPM package and a Multilib IPK
4849 package in a ``qemux86-64`` system resolves to something like
Patrick Williams0ca19cc2021-08-16 14:03:13 -05004850 ``bash_4.1-r2.x86_64.ipk`` and ``lib32-bash_4.1-rw:x86.ipk``,
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004851 respectively.
4852
4853- The IPK deploy folder is not modified with ``${MLPREFIX}`` because
4854 packages with and without the Multilib feature can exist in the same
4855 folder due to the ``${PN}`` differences.
4856
4857- IPK defines a sanity check for Multilib installation using certain
4858 rules for file comparison, overridden, etc.
4859
4860Installing Multiple Versions of the Same Library
4861------------------------------------------------
4862
William A. Kennington IIIac69b482021-06-02 12:28:27 -07004863There are be situations where you need to install and use multiple versions
4864of the same library on the same system at the same time. This
4865almost always happens when a library API changes and you have
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004866multiple pieces of software that depend on the separate versions of the
4867library. To accommodate these situations, you can install multiple
4868versions of the same library in parallel on the same system.
4869
4870The process is straightforward as long as the libraries use proper
4871versioning. With properly versioned libraries, all you need to do to
4872individually specify the libraries is create separate, appropriately
4873named recipes where the :term:`PN` part of
4874the name includes a portion that differentiates each library version
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004875(e.g. the major part of the version number). Thus, instead of having a
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004876single recipe that loads one version of a library (e.g. ``clutter``),
4877you provide multiple recipes that result in different versions of the
4878libraries you want. As an example, the following two recipes would allow
4879the two separate versions of the ``clutter`` library to co-exist on the
4880same system:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004881
4882.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004883
4884 clutter-1.6_1.6.20.bb
4885 clutter-1.8_1.8.4.bb
4886
4887Additionally, if
4888you have other recipes that depend on a given library, you need to use
4889the :term:`DEPENDS` variable to
4890create the dependency. Continuing with the same example, if you want to
4891have a recipe depend on the 1.8 version of the ``clutter`` library, use
Andrew Geisslerc926e172021-05-07 16:11:35 -05004892the following in your recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004893
4894 DEPENDS = "clutter-1.8"
4895
4896Using x32 psABI
4897===============
4898
4899x32 processor-specific Application Binary Interface (`x32
4900psABI <https://software.intel.com/en-us/node/628948>`__) is a native
490132-bit processor-specific ABI for Intel 64 (x86-64) architectures. An
4902ABI defines the calling conventions between functions in a processing
4903environment. The interface determines what registers are used and what
4904the sizes are for various C data types.
4905
4906Some processing environments prefer using 32-bit applications even when
4907running on Intel 64-bit platforms. Consider the i386 psABI, which is a
4908very old 32-bit ABI for Intel 64-bit platforms. The i386 psABI does not
4909provide efficient use and access of the Intel 64-bit processor
4910resources, leaving the system underutilized. Now consider the x86_64
4911psABI. This ABI is newer and uses 64-bits for data sizes and program
4912pointers. The extra bits increase the footprint size of the programs,
4913libraries, and also increases the memory and file system size
4914requirements. Executing under the x32 psABI enables user programs to
4915utilize CPU and system resources more efficiently while keeping the
4916memory footprint of the applications low. Extra bits are used for
4917registers but not for addressing mechanisms.
4918
4919The Yocto Project supports the final specifications of x32 psABI as
4920follows:
4921
4922- You can create packages and images in x32 psABI format on x86_64
4923 architecture targets.
4924
4925- You can successfully build recipes with the x32 toolchain.
4926
4927- You can create and boot ``core-image-minimal`` and
4928 ``core-image-sato`` images.
4929
William A. Kennington IIIac69b482021-06-02 12:28:27 -07004930- There is RPM Package Manager (RPM) support for x32 binaries.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004931
William A. Kennington IIIac69b482021-06-02 12:28:27 -07004932- There is support for large images.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004933
4934To use the x32 psABI, you need to edit your ``conf/local.conf``
Andrew Geisslerc926e172021-05-07 16:11:35 -05004935configuration file as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004936
4937 MACHINE = "qemux86-64"
4938 DEFAULTTUNE = "x86-64-x32"
Patrick Williams0ca19cc2021-08-16 14:03:13 -05004939 baselib = "${@d.getVar('BASE_LIB:tune-' + (d.getVar('DEFAULTTUNE') \
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004940 or 'INVALID')) or 'lib'}"
4941
4942Once you have set
4943up your configuration file, use BitBake to build an image that supports
Andrew Geisslerc926e172021-05-07 16:11:35 -05004944the x32 psABI. Here is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004945
4946 $ bitbake core-image-sato
4947
4948Enabling GObject Introspection Support
4949======================================
4950
4951`GObject
4952introspection <https://wiki.gnome.org/Projects/GObjectIntrospection>`__
4953is the standard mechanism for accessing GObject-based software from
4954runtime environments. GObject is a feature of the GLib library that
4955provides an object framework for the GNOME desktop and related software.
4956GObject Introspection adds information to GObject that allows objects
4957created within it to be represented across different programming
4958languages. If you want to construct GStreamer pipelines using Python, or
4959control UPnP infrastructure using Javascript and GUPnP, GObject
4960introspection is the only way to do it.
4961
4962This section describes the Yocto Project support for generating and
4963packaging GObject introspection data. GObject introspection data is a
4964description of the API provided by libraries built on top of GLib
4965framework, and, in particular, that framework's GObject mechanism.
4966GObject Introspection Repository (GIR) files go to ``-dev`` packages,
4967``typelib`` files go to main packages as they are packaged together with
4968libraries that are introspected.
4969
4970The data is generated when building such a library, by linking the
4971library with a small executable binary that asks the library to describe
4972itself, and then executing the binary and processing its output.
4973
4974Generating this data in a cross-compilation environment is difficult
4975because the library is produced for the target architecture, but its
4976code needs to be executed on the build host. This problem is solved with
4977the OpenEmbedded build system by running the code through QEMU, which
4978allows precisely that. Unfortunately, QEMU does not always work
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05004979perfectly as mentioned in the ":ref:`dev-manual/common-tasks:known issues`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004980section.
4981
4982Enabling the Generation of Introspection Data
4983---------------------------------------------
4984
4985Enabling the generation of introspection data (GIR files) in your
4986library package involves the following:
4987
49881. Inherit the
4989 :ref:`gobject-introspection <ref-classes-gobject-introspection>`
4990 class.
4991
49922. Make sure introspection is not disabled anywhere in the recipe or
4993 from anything the recipe includes. Also, make sure that
4994 "gobject-introspection-data" is not in
4995 :term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`
4996 and that "qemu-usermode" is not in
4997 :term:`MACHINE_FEATURES_BACKFILL_CONSIDERED`.
William A. Kennington IIIac69b482021-06-02 12:28:27 -07004998 In either of these conditions, nothing will happen.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004999
50003. Try to build the recipe. If you encounter build errors that look like
5001 something is unable to find ``.so`` libraries, check where these
5002 libraries are located in the source tree and add the following to the
Andrew Geisslerc926e172021-05-07 16:11:35 -05005003 recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005004
5005 GIR_EXTRA_LIBS_PATH = "${B}/something/.libs"
5006
5007 .. note::
5008
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005009 See recipes in the ``oe-core`` repository that use that
5010 ``GIR_EXTRA_LIBS_PATH`` variable as an example.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005011
50124. Look for any other errors, which probably mean that introspection
5013 support in a package is not entirely standard, and thus breaks down
5014 in a cross-compilation environment. For such cases, custom-made fixes
5015 are needed. A good place to ask and receive help in these cases is
5016 the :ref:`Yocto Project mailing
5017 lists <resources-mailinglist>`.
5018
5019.. note::
5020
5021 Using a library that no longer builds against the latest Yocto
5022 Project release and prints introspection related errors is a good
5023 candidate for the previous procedure.
5024
5025Disabling the Generation of Introspection Data
5026----------------------------------------------
5027
5028You might find that you do not want to generate introspection data. Or,
5029perhaps QEMU does not work on your build host and target architecture
5030combination. If so, you can use either of the following methods to
5031disable GIR file generations:
5032
Andrew Geisslerc926e172021-05-07 16:11:35 -05005033- Add the following to your distro configuration::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005034
5035 DISTRO_FEATURES_BACKFILL_CONSIDERED = "gobject-introspection-data"
5036
5037 Adding this statement disables generating introspection data using
5038 QEMU but will still enable building introspection tools and libraries
5039 (i.e. building them does not require the use of QEMU).
5040
Andrew Geisslerc926e172021-05-07 16:11:35 -05005041- Add the following to your machine configuration::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005042
5043 MACHINE_FEATURES_BACKFILL_CONSIDERED = "qemu-usermode"
5044
5045 Adding this statement disables the use of QEMU when building packages for your
5046 machine. Currently, this feature is used only by introspection
5047 recipes and has the same effect as the previously described option.
5048
5049 .. note::
5050
5051 Future releases of the Yocto Project might have other features
5052 affected by this option.
5053
5054If you disable introspection data, you can still obtain it through other
5055means such as copying the data from a suitable sysroot, or by generating
5056it on the target hardware. The OpenEmbedded build system does not
5057currently provide specific support for these techniques.
5058
5059Testing that Introspection Works in an Image
5060--------------------------------------------
5061
5062Use the following procedure to test if generating introspection data is
5063working in an image:
5064
50651. Make sure that "gobject-introspection-data" is not in
5066 :term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`
5067 and that "qemu-usermode" is not in
5068 :term:`MACHINE_FEATURES_BACKFILL_CONSIDERED`.
5069
50702. Build ``core-image-sato``.
5071
50723. Launch a Terminal and then start Python in the terminal.
5073
Andrew Geisslerc926e172021-05-07 16:11:35 -050050744. Enter the following in the terminal::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005075
5076 >>> from gi.repository import GLib
5077 >>> GLib.get_host_name()
5078
50795. For something a little more advanced, enter the following see:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005080 https://python-gtk-3-tutorial.readthedocs.io/en/latest/introduction.html
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005081
5082Known Issues
5083------------
5084
William A. Kennington IIIac69b482021-06-02 12:28:27 -07005085Here are know issues in GObject Introspection Support:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005086
5087- ``qemu-ppc64`` immediately crashes. Consequently, you cannot build
5088 introspection data on that architecture.
5089
5090- x32 is not supported by QEMU. Consequently, introspection data is
5091 disabled.
5092
5093- musl causes transient GLib binaries to crash on assertion failures.
5094 Consequently, generating introspection data is disabled.
5095
5096- Because QEMU is not able to run the binaries correctly, introspection
5097 is disabled for some specific packages under specific architectures
5098 (e.g. ``gcr``, ``libsecret``, and ``webkit``).
5099
5100- QEMU usermode might not work properly when running 64-bit binaries
5101 under 32-bit host machines. In particular, "qemumips64" is known to
5102 not work under i686.
5103
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005104Optionally Using an External Toolchain
5105======================================
5106
5107You might want to use an external toolchain as part of your development.
5108If this is the case, the fundamental steps you need to accomplish are as
5109follows:
5110
5111- Understand where the installed toolchain resides. For cases where you
5112 need to build the external toolchain, you would need to take separate
5113 steps to build and install the toolchain.
5114
5115- Make sure you add the layer that contains the toolchain to your
5116 ``bblayers.conf`` file through the
5117 :term:`BBLAYERS` variable.
5118
5119- Set the ``EXTERNAL_TOOLCHAIN`` variable in your ``local.conf`` file
5120 to the location in which you installed the toolchain.
5121
5122A good example of an external toolchain used with the Yocto Project is
5123Mentor Graphics Sourcery G++ Toolchain. You can see information on how
5124to use that particular layer in the ``README`` file at
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005125https://github.com/MentorEmbedded/meta-sourcery/. You can find
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005126further information by reading about the
5127:term:`TCMODE` variable in the Yocto
5128Project Reference Manual's variable glossary.
5129
5130Creating Partitioned Images Using Wic
5131=====================================
5132
5133Creating an image for a particular hardware target using the
5134OpenEmbedded build system does not necessarily mean you can boot that
5135image as is on your device. Physical devices accept and boot images in
5136various ways depending on the specifics of the device. Usually,
5137information about the hardware can tell you what image format the device
5138requires. Should your device require multiple partitions on an SD card,
5139flash, or an HDD, you can use the OpenEmbedded Image Creator, Wic, to
5140create the properly partitioned image.
5141
5142The ``wic`` command generates partitioned images from existing
5143OpenEmbedded build artifacts. Image generation is driven by partitioning
5144commands contained in an Openembedded kickstart file (``.wks``)
5145specified either directly on the command line or as one of a selection
5146of canned kickstart files as shown with the ``wic list images`` command
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05005147in the
5148":ref:`dev-manual/common-tasks:generate an image using an existing kickstart file`"
5149section. When you apply the command to a given set of build artifacts, the
5150result is an image or set of images that can be directly written onto media and
5151used on a particular system.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005152
5153.. note::
5154
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005155 For a kickstart file reference, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06005156 ":ref:`ref-manual/kickstart:openembedded kickstart (\`\`.wks\`\`) reference`"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005157 Chapter in the Yocto Project Reference Manual.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005158
5159The ``wic`` command and the infrastructure it is based on is by
5160definition incomplete. The purpose of the command is to allow the
5161generation of customized images, and as such, was designed to be
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05005162completely extensible through a plugin interface. See the
5163":ref:`dev-manual/common-tasks:using the wic plugin interface`" section
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005164for information on these plugins.
5165
5166This section provides some background information on Wic, describes what
5167you need to have in place to run the tool, provides instruction on how
5168to use the Wic utility, provides information on using the Wic plugins
5169interface, and provides several examples that show how to use Wic.
5170
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005171Background
5172----------
5173
5174This section provides some background on the Wic utility. While none of
5175this information is required to use Wic, you might find it interesting.
5176
5177- The name "Wic" is derived from OpenEmbedded Image Creator (oeic). The
5178 "oe" diphthong in "oeic" was promoted to the letter "w", because
5179 "oeic" is both difficult to remember and to pronounce.
5180
5181- Wic is loosely based on the Meego Image Creator (``mic``) framework.
5182 The Wic implementation has been heavily modified to make direct use
5183 of OpenEmbedded build artifacts instead of package installation and
5184 configuration, which are already incorporated within the OpenEmbedded
5185 artifacts.
5186
5187- Wic is a completely independent standalone utility that initially
5188 provides easier-to-use and more flexible replacements for an existing
5189 functionality in OE-Core's
5190 :ref:`image-live <ref-classes-image-live>`
5191 class. The difference between Wic and those examples is that with Wic
5192 the functionality of those scripts is implemented by a
5193 general-purpose partitioning language, which is based on Redhat
5194 kickstart syntax.
5195
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005196Requirements
5197------------
5198
5199In order to use the Wic utility with the OpenEmbedded Build system, your
5200system needs to meet the following requirements:
5201
5202- The Linux distribution on your development host must support the
5203 Yocto Project. See the ":ref:`detailed-supported-distros`"
5204 section in the Yocto Project Reference Manual for the list of
5205 distributions that support the Yocto Project.
5206
5207- The standard system utilities, such as ``cp``, must be installed on
5208 your development host system.
5209
5210- You must have sourced the build environment setup script (i.e.
5211 :ref:`structure-core-script`) found in the
5212 :term:`Build Directory`.
5213
5214- You need to have the build artifacts already available, which
5215 typically means that you must have already created an image using the
5216 Openembedded build system (e.g. ``core-image-minimal``). While it
5217 might seem redundant to generate an image in order to create an image
5218 using Wic, the current version of Wic requires the artifacts in the
5219 form generated by the OpenEmbedded build system.
5220
5221- You must build several native tools, which are built to run on the
Andrew Geisslerc926e172021-05-07 16:11:35 -05005222 build system::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005223
5224 $ bitbake parted-native dosfstools-native mtools-native
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005225
5226- Include "wic" as part of the
5227 :term:`IMAGE_FSTYPES`
5228 variable.
5229
5230- Include the name of the :ref:`wic kickstart file <openembedded-kickstart-wks-reference>`
5231 as part of the :term:`WKS_FILE` variable
5232
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005233Getting Help
5234------------
5235
5236You can get general help for the ``wic`` command by entering the ``wic``
5237command by itself or by entering the command with a help argument as
Andrew Geisslerc926e172021-05-07 16:11:35 -05005238follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005239
5240 $ wic -h
5241 $ wic --help
5242 $ wic help
5243
5244Currently, Wic supports seven commands: ``cp``, ``create``, ``help``,
5245``list``, ``ls``, ``rm``, and ``write``. You can get help for all these
Andrew Geisslerc926e172021-05-07 16:11:35 -05005246commands except "help" by using the following form::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005247
5248 $ wic help command
5249
5250For example, the following command returns help for the ``write``
Andrew Geisslerc926e172021-05-07 16:11:35 -05005251command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005252
5253 $ wic help write
5254
5255Wic supports help for three topics: ``overview``, ``plugins``, and
Andrew Geisslerc926e172021-05-07 16:11:35 -05005256``kickstart``. You can get help for any topic using the following form::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005257
5258 $ wic help topic
5259
Andrew Geisslerc926e172021-05-07 16:11:35 -05005260For example, the following returns overview help for Wic::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005261
5262 $ wic help overview
5263
William A. Kennington IIIac69b482021-06-02 12:28:27 -07005264There is one additional level of help for Wic. You can get help on
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005265individual images through the ``list`` command. You can use the ``list``
Andrew Geisslerc926e172021-05-07 16:11:35 -05005266command to return the available Wic images as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005267
5268 $ wic list images
5269 genericx86 Create an EFI disk image for genericx86*
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005270 edgerouter Create SD card image for Edgerouter
Andrew Geissler5199d832021-09-24 16:47:35 -05005271 beaglebone-yocto Create SD card image for Beaglebone
5272 qemux86-directdisk Create a qemu machine 'pcbios' direct disk image
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005273 systemd-bootdisk Create an EFI disk image with systemd-boot
5274 mkhybridiso Create a hybrid ISO image
Andrew Geissler5199d832021-09-24 16:47:35 -05005275 mkefidisk Create an EFI disk image
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005276 sdimage-bootpart Create SD card image with a boot partition
5277 directdisk-multi-rootfs Create multi rootfs image using rootfs plugin
Andrew Geissler5199d832021-09-24 16:47:35 -05005278 directdisk Create a 'pcbios' direct disk image
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005279 directdisk-bootloader-config Create a 'pcbios' direct disk image with custom bootloader config
Andrew Geissler5199d832021-09-24 16:47:35 -05005280 qemuriscv Create qcow2 image for RISC-V QEMU machines
5281 directdisk-gpt Create a 'pcbios' direct disk image
5282 efi-bootdisk
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005283
5284Once you know the list of available
5285Wic images, you can use ``help`` with the command to get help on a
5286particular image. For example, the following command returns help on the
Andrew Geisslerc926e172021-05-07 16:11:35 -05005287"beaglebone-yocto" image::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005288
5289 $ wic list beaglebone-yocto help
5290
5291 Creates a partitioned SD card image for Beaglebone.
5292 Boot files are located in the first vfat partition.
5293
5294Operational Modes
5295-----------------
5296
5297You can use Wic in two different modes, depending on how much control
5298you need for specifying the Openembedded build artifacts that are used
5299for creating the image: Raw and Cooked:
5300
5301- *Raw Mode:* You explicitly specify build artifacts through Wic
5302 command-line arguments.
5303
5304- *Cooked Mode:* The current
5305 :term:`MACHINE` setting and image
5306 name are used to automatically locate and provide the build
5307 artifacts. You just supply a kickstart file and the name of the image
5308 from which to use artifacts.
5309
5310Regardless of the mode you use, you need to have the build artifacts
5311ready and available.
5312
5313Raw Mode
5314~~~~~~~~
5315
5316Running Wic in raw mode allows you to specify all the partitions through
5317the ``wic`` command line. The primary use for raw mode is if you have
5318built your kernel outside of the Yocto Project
5319:term:`Build Directory`. In other words, you
5320can point to arbitrary kernel, root filesystem locations, and so forth.
5321Contrast this behavior with cooked mode where Wic looks in the Build
5322Directory (e.g. ``tmp/deploy/images/``\ machine).
5323
Andrew Geisslerc926e172021-05-07 16:11:35 -05005324The general form of the ``wic`` command in raw mode is::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005325
5326 $ wic create wks_file options ...
5327
5328 Where:
5329
5330 wks_file:
5331 An OpenEmbedded kickstart file. You can provide
5332 your own custom file or use a file from a set of
5333 existing files as described by further options.
5334
5335 optional arguments:
5336 -h, --help show this help message and exit
5337 -o OUTDIR, --outdir OUTDIR
5338 name of directory to create image in
5339 -e IMAGE_NAME, --image-name IMAGE_NAME
5340 name of the image to use the artifacts from e.g. core-
5341 image-sato
5342 -r ROOTFS_DIR, --rootfs-dir ROOTFS_DIR
5343 path to the /rootfs dir to use as the .wks rootfs
5344 source
5345 -b BOOTIMG_DIR, --bootimg-dir BOOTIMG_DIR
5346 path to the dir containing the boot artifacts (e.g.
5347 /EFI or /syslinux dirs) to use as the .wks bootimg
5348 source
5349 -k KERNEL_DIR, --kernel-dir KERNEL_DIR
5350 path to the dir containing the kernel to use in the
5351 .wks bootimg
5352 -n NATIVE_SYSROOT, --native-sysroot NATIVE_SYSROOT
5353 path to the native sysroot containing the tools to use
5354 to build the image
5355 -s, --skip-build-check
5356 skip the build check
5357 -f, --build-rootfs build rootfs
5358 -c {gzip,bzip2,xz}, --compress-with {gzip,bzip2,xz}
5359 compress image with specified compressor
5360 -m, --bmap generate .bmap
5361 --no-fstab-update Do not change fstab file.
5362 -v VARS_DIR, --vars VARS_DIR
5363 directory with <image>.env files that store bitbake
5364 variables
5365 -D, --debug output debug information
5366
5367.. note::
5368
5369 You do not need root privileges to run Wic. In fact, you should not
5370 run as root when using the utility.
5371
5372Cooked Mode
5373~~~~~~~~~~~
5374
5375Running Wic in cooked mode leverages off artifacts in the Build
5376Directory. In other words, you do not have to specify kernel or root
5377filesystem locations as part of the command. All you need to provide is
5378a kickstart file and the name of the image from which to use artifacts
5379by using the "-e" option. Wic looks in the Build Directory (e.g.
5380``tmp/deploy/images/``\ machine) for artifacts.
5381
Andrew Geisslerc926e172021-05-07 16:11:35 -05005382The general form of the ``wic`` command using Cooked Mode is as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005383
5384 $ wic create wks_file -e IMAGE_NAME
5385
5386 Where:
5387
5388 wks_file:
5389 An OpenEmbedded kickstart file. You can provide
5390 your own custom file or use a file from a set of
5391 existing files provided with the Yocto Project
5392 release.
5393
5394 required argument:
5395 -e IMAGE_NAME, --image-name IMAGE_NAME
5396 name of the image to use the artifacts from e.g. core-
5397 image-sato
5398
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005399Using an Existing Kickstart File
5400--------------------------------
5401
5402If you do not want to create your own kickstart file, you can use an
5403existing file provided by the Wic installation. As shipped, kickstart
Andrew Geissler09209ee2020-12-13 08:44:15 -06005404files can be found in the :ref:`overview-manual/development-environment:yocto project source repositories` in the
Andrew Geisslerc926e172021-05-07 16:11:35 -05005405following two locations::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005406
5407 poky/meta-yocto-bsp/wic
5408 poky/scripts/lib/wic/canned-wks
5409
Andrew Geisslerc926e172021-05-07 16:11:35 -05005410Use the following command to list the available kickstart files::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005411
5412 $ wic list images
5413 genericx86 Create an EFI disk image for genericx86*
5414 beaglebone-yocto Create SD card image for Beaglebone
5415 edgerouter Create SD card image for Edgerouter
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05005416 qemux86-directdisk Create a QEMU machine 'pcbios' direct disk image
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005417 directdisk-gpt Create a 'pcbios' direct disk image
5418 mkefidisk Create an EFI disk image
5419 directdisk Create a 'pcbios' direct disk image
5420 systemd-bootdisk Create an EFI disk image with systemd-boot
5421 mkhybridiso Create a hybrid ISO image
5422 sdimage-bootpart Create SD card image with a boot partition
5423 directdisk-multi-rootfs Create multi rootfs image using rootfs plugin
5424 directdisk-bootloader-config Create a 'pcbios' direct disk image with custom bootloader config
5425
5426When you use an existing file, you
5427do not have to use the ``.wks`` extension. Here is an example in Raw
Andrew Geisslerc926e172021-05-07 16:11:35 -05005428Mode that uses the ``directdisk`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005429
5430 $ wic create directdisk -r rootfs_dir -b bootimg_dir \
5431 -k kernel_dir -n native_sysroot
5432
5433Here are the actual partition language commands used in the
Andrew Geisslerc926e172021-05-07 16:11:35 -05005434``genericx86.wks`` file to generate an image::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005435
5436 # short-description: Create an EFI disk image for genericx86*
5437 # long-description: Creates a partitioned EFI disk image for genericx86* machines
5438 part /boot --source bootimg-efi --sourceparams="loader=grub-efi" --ondisk sda --label msdos --active --align 1024
5439 part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 1024 --use-uuid
5440 part swap --ondisk sda --size 44 --label swap1 --fstype=swap
5441
5442 bootloader --ptable gpt --timeout=5 --append="rootfstype=ext4 console=ttyS0,115200 console=tty0"
5443
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005444Using the Wic Plugin Interface
5445------------------------------
5446
5447You can extend and specialize Wic functionality by using Wic plugins.
5448This section explains the Wic plugin interface.
5449
5450.. note::
5451
5452 Wic plugins consist of "source" and "imager" plugins. Imager plugins
5453 are beyond the scope of this section.
5454
5455Source plugins provide a mechanism to customize partition content during
5456the Wic image generation process. You can use source plugins to map
5457values that you specify using ``--source`` commands in kickstart files
5458(i.e. ``*.wks``) to a plugin implementation used to populate a given
5459partition.
5460
5461.. note::
5462
5463 If you use plugins that have build-time dependencies (e.g. native
5464 tools, bootloaders, and so forth) when building a Wic image, you need
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005465 to specify those dependencies using the :term:`WKS_FILE_DEPENDS`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005466 variable.
5467
5468Source plugins are subclasses defined in plugin files. As shipped, the
5469Yocto Project provides several plugin files. You can see the source
5470plugin files that ship with the Yocto Project
Andrew Geissler09209ee2020-12-13 08:44:15 -06005471:yocto_git:`here </poky/tree/scripts/lib/wic/plugins/source>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005472Each of these plugin files contains source plugins that are designed to
5473populate a specific Wic image partition.
5474
5475Source plugins are subclasses of the ``SourcePlugin`` class, which is
5476defined in the ``poky/scripts/lib/wic/pluginbase.py`` file. For example,
5477the ``BootimgEFIPlugin`` source plugin found in the ``bootimg-efi.py``
5478file is a subclass of the ``SourcePlugin`` class, which is found in the
5479``pluginbase.py`` file.
5480
5481You can also implement source plugins in a layer outside of the Source
5482Repositories (external layer). To do so, be sure that your plugin files
5483are located in a directory whose path is
5484``scripts/lib/wic/plugins/source/`` within your external layer. When the
5485plugin files are located there, the source plugins they contain are made
5486available to Wic.
5487
5488When the Wic implementation needs to invoke a partition-specific
5489implementation, it looks for the plugin with the same name as the
5490``--source`` parameter used in the kickstart file given to that
5491partition. For example, if the partition is set up using the following
Andrew Geisslerc926e172021-05-07 16:11:35 -05005492command in a kickstart file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005493
5494 part /boot --source bootimg-pcbios --ondisk sda --label boot --active --align 1024
5495
5496The methods defined as class
5497members of the matching source plugin (i.e. ``bootimg-pcbios``) in the
5498``bootimg-pcbios.py`` plugin file are used.
5499
5500To be more concrete, here is the corresponding plugin definition from
5501the ``bootimg-pcbios.py`` file for the previous command along with an
5502example method called by the Wic implementation when it needs to prepare
Andrew Geisslerc926e172021-05-07 16:11:35 -05005503a partition using an implementation-specific function::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005504
5505 .
5506 .
5507 .
5508 class BootimgPcbiosPlugin(SourcePlugin):
5509 """
5510 Create MBR boot partition and install syslinux on it.
5511 """
5512
5513 name = 'bootimg-pcbios'
5514 .
5515 .
5516 .
5517 @classmethod
5518 def do_prepare_partition(cls, part, source_params, creator, cr_workdir,
5519 oe_builddir, bootimg_dir, kernel_dir,
5520 rootfs_dir, native_sysroot):
5521 """
5522 Called to do the actual content population for a partition i.e. it
5523 'prepares' the partition to be incorporated into the image.
5524 In this case, prepare content for legacy bios boot partition.
5525 """
5526 .
5527 .
5528 .
5529
5530If a
5531subclass (plugin) itself does not implement a particular function, Wic
5532locates and uses the default version in the superclass. It is for this
5533reason that all source plugins are derived from the ``SourcePlugin``
5534class.
5535
5536The ``SourcePlugin`` class defined in the ``pluginbase.py`` file defines
5537a set of methods that source plugins can implement or override. Any
5538plugins (subclass of ``SourcePlugin``) that do not implement a
5539particular method inherit the implementation of the method from the
5540``SourcePlugin`` class. For more information, see the ``SourcePlugin``
5541class in the ``pluginbase.py`` file for details:
5542
5543The following list describes the methods implemented in the
5544``SourcePlugin`` class:
5545
5546- ``do_prepare_partition()``: Called to populate a partition with
5547 actual content. In other words, the method prepares the final
5548 partition image that is incorporated into the disk image.
5549
5550- ``do_configure_partition()``: Called before
5551 ``do_prepare_partition()`` to create custom configuration files for a
5552 partition (e.g. syslinux or grub configuration files).
5553
5554- ``do_install_disk()``: Called after all partitions have been
5555 prepared and assembled into a disk image. This method provides a hook
5556 to allow finalization of a disk image (e.g. writing an MBR).
5557
5558- ``do_stage_partition()``: Special content-staging hook called
5559 before ``do_prepare_partition()``. This method is normally empty.
5560
5561 Typically, a partition just uses the passed-in parameters (e.g. the
5562 unmodified value of ``bootimg_dir``). However, in some cases, things
5563 might need to be more tailored. As an example, certain files might
5564 additionally need to be taken from ``bootimg_dir + /boot``. This hook
5565 allows those files to be staged in a customized fashion.
5566
5567 .. note::
5568
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005569 ``get_bitbake_var()`` allows you to access non-standard variables that
5570 you might want to use for this behavior.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005571
5572You can extend the source plugin mechanism. To add more hooks, create
5573more source plugin methods within ``SourcePlugin`` and the corresponding
5574derived subclasses. The code that calls the plugin methods uses the
5575``plugin.get_source_plugin_methods()`` function to find the method or
5576methods needed by the call. Retrieval of those methods is accomplished
5577by filling up a dict with keys that contain the method names of
5578interest. On success, these will be filled in with the actual methods.
5579See the Wic implementation for examples and details.
5580
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005581Wic Examples
5582------------
5583
5584This section provides several examples that show how to use the Wic
5585utility. All the examples assume the list of requirements in the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05005586":ref:`dev-manual/common-tasks:requirements`" section have been met. The
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005587examples assume the previously generated image is
5588``core-image-minimal``.
5589
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005590Generate an Image using an Existing Kickstart File
5591~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5592
5593This example runs in Cooked Mode and uses the ``mkefidisk`` kickstart
Andrew Geisslerc926e172021-05-07 16:11:35 -05005594file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005595
5596 $ wic create mkefidisk -e core-image-minimal
5597 INFO: Building wic-tools...
5598 .
5599 .
5600 .
5601 INFO: The new image(s) can be found here:
5602 ./mkefidisk-201804191017-sda.direct
5603
5604 The following build artifacts were used to create the image(s):
5605 ROOTFS_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
5606 BOOTIMG_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
5607 KERNEL_DIR: /home/stephano/build/master/build/tmp-glibc/deploy/images/qemux86
5608 NATIVE_SYSROOT: /home/stephano/build/master/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
5609
5610 INFO: The image(s) were created using OE kickstart file:
5611 /home/stephano/build/master/openembedded-core/scripts/lib/wic/canned-wks/mkefidisk.wks
5612
5613The previous example shows the easiest way to create an image by running
5614in cooked mode and supplying a kickstart file and the "-e" option to
5615point to the existing build artifacts. Your ``local.conf`` file needs to
5616have the :term:`MACHINE` variable set
5617to the machine you are using, which is "qemux86" in this example.
5618
5619Once the image builds, the output provides image location, artifact use,
5620and kickstart file information.
5621
5622.. note::
5623
5624 You should always verify the details provided in the output to make
5625 sure that the image was indeed created exactly as expected.
5626
5627Continuing with the example, you can now write the image from the Build
5628Directory onto a USB stick, or whatever media for which you built your
5629image, and boot from the media. You can write the image by using
Andrew Geisslerc926e172021-05-07 16:11:35 -05005630``bmaptool`` or ``dd``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005631
5632 $ oe-run-native bmaptool copy mkefidisk-201804191017-sda.direct /dev/sdX
5633
5634or ::
5635
5636 $ sudo dd if=mkefidisk-201804191017-sda.direct of=/dev/sdX
5637
5638.. note::
5639
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005640 For more information on how to use the ``bmaptool``
5641 to flash a device with an image, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06005642 ":ref:`dev-manual/common-tasks:flashing images using \`\`bmaptool\`\``"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005643 section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005644
5645Using a Modified Kickstart File
5646~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5647
5648Because partitioned image creation is driven by the kickstart file, it
5649is easy to affect image creation by changing the parameters in the file.
5650This next example demonstrates that through modification of the
5651``directdisk-gpt`` kickstart file.
5652
5653As mentioned earlier, you can use the command ``wic list images`` to
5654show the list of existing kickstart files. The directory in which the
5655``directdisk-gpt.wks`` file resides is
5656``scripts/lib/image/canned-wks/``, which is located in the
5657:term:`Source Directory` (e.g. ``poky``).
5658Because available files reside in this directory, you can create and add
5659your own custom files to the directory. Subsequent use of the
5660``wic list images`` command would then include your kickstart files.
5661
5662In this example, the existing ``directdisk-gpt`` file already does most
5663of what is needed. However, for the hardware in this example, the image
5664will need to boot from ``sdb`` instead of ``sda``, which is what the
5665``directdisk-gpt`` kickstart file uses.
5666
5667The example begins by making a copy of the ``directdisk-gpt.wks`` file
5668in the ``scripts/lib/image/canned-wks`` directory and then by changing
5669the lines that specify the target disk from which to boot.
5670::
5671
5672 $ cp /home/stephano/poky/scripts/lib/wic/canned-wks/directdisk-gpt.wks \
5673 /home/stephano/poky/scripts/lib/wic/canned-wks/directdisksdb-gpt.wks
5674
5675Next, the example modifies the ``directdisksdb-gpt.wks`` file and
5676changes all instances of "``--ondisk sda``" to "``--ondisk sdb``". The
5677example changes the following two lines and leaves the remaining lines
Andrew Geisslerc926e172021-05-07 16:11:35 -05005678untouched::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005679
5680 part /boot --source bootimg-pcbios --ondisk sdb --label boot --active --align 1024
5681 part / --source rootfs --ondisk sdb --fstype=ext4 --label platform --align 1024 --use-uuid
5682
5683Once the lines are changed, the
5684example generates the ``directdisksdb-gpt`` image. The command points
5685the process at the ``core-image-minimal`` artifacts for the Next Unit of
5686Computing (nuc) :term:`MACHINE` the
5687``local.conf``.
5688::
5689
5690 $ wic create directdisksdb-gpt -e core-image-minimal
5691 INFO: Building wic-tools...
5692 .
5693 .
5694 .
5695 Initialising tasks: 100% |#######################################| Time: 0:00:01
5696 NOTE: Executing SetScene Tasks
5697 NOTE: Executing RunQueue Tasks
5698 NOTE: Tasks Summary: Attempted 1161 tasks of which 1157 didn't need to be rerun and all succeeded.
5699 INFO: Creating image(s)...
5700
5701 INFO: The new image(s) can be found here:
5702 ./directdisksdb-gpt-201710090938-sdb.direct
5703
5704 The following build artifacts were used to create the image(s):
5705 ROOTFS_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
5706 BOOTIMG_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
5707 KERNEL_DIR: /home/stephano/build/master/build/tmp-glibc/deploy/images/qemux86
5708 NATIVE_SYSROOT: /home/stephano/build/master/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
5709
5710 INFO: The image(s) were created using OE kickstart file:
5711 /home/stephano/poky/scripts/lib/wic/canned-wks/directdisksdb-gpt.wks
5712
5713Continuing with the example, you can now directly ``dd`` the image to a
5714USB stick, or whatever media for which you built your image, and boot
Andrew Geisslerc926e172021-05-07 16:11:35 -05005715the resulting media::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005716
5717 $ sudo dd if=directdisksdb-gpt-201710090938-sdb.direct of=/dev/sdb
5718 140966+0 records in
5719 140966+0 records out
5720 72174592 bytes (72 MB, 69 MiB) copied, 78.0282 s, 925 kB/s
5721 $ sudo eject /dev/sdb
5722
5723Using a Modified Kickstart File and Running in Raw Mode
5724~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5725
5726This next example manually specifies each build artifact (runs in Raw
5727Mode) and uses a modified kickstart file. The example also uses the
5728``-o`` option to cause Wic to create the output somewhere other than the
Andrew Geisslerc926e172021-05-07 16:11:35 -05005729default output directory, which is the current directory::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005730
5731 $ wic create /home/stephano/my_yocto/test.wks -o /home/stephano/testwic \
5732 --rootfs-dir /home/stephano/build/master/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/rootfs \
5733 --bootimg-dir /home/stephano/build/master/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share \
5734 --kernel-dir /home/stephano/build/master/build/tmp/deploy/images/qemux86 \
5735 --native-sysroot /home/stephano/build/master/build/tmp/work/i586-poky-linux/wic-tools/1.0-r0/recipe-sysroot-native
5736
5737 INFO: Creating image(s)...
5738
5739 INFO: The new image(s) can be found here:
5740 /home/stephano/testwic/test-201710091445-sdb.direct
5741
5742 The following build artifacts were used to create the image(s):
5743 ROOTFS_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
5744 BOOTIMG_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
5745 KERNEL_DIR: /home/stephano/build/master/build/tmp-glibc/deploy/images/qemux86
5746 NATIVE_SYSROOT: /home/stephano/build/master/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
5747
5748 INFO: The image(s) were created using OE kickstart file:
5749 /home/stephano/my_yocto/test.wks
5750
5751For this example,
5752:term:`MACHINE` did not have to be
5753specified in the ``local.conf`` file since the artifact is manually
5754specified.
5755
5756Using Wic to Manipulate an Image
5757~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5758
5759Wic image manipulation allows you to shorten turnaround time during
5760image development. For example, you can use Wic to delete the kernel
5761partition of a Wic image and then insert a newly built kernel. This
5762saves you time from having to rebuild the entire image each time you
5763modify the kernel.
5764
5765.. note::
5766
5767 In order to use Wic to manipulate a Wic image as in this example,
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005768 your development machine must have the ``mtools`` package installed.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005769
5770The following example examines the contents of the Wic image, deletes
5771the existing kernel, and then inserts a new kernel:
5772
57731. *List the Partitions:* Use the ``wic ls`` command to list all the
Andrew Geisslerc926e172021-05-07 16:11:35 -05005774 partitions in the Wic image::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005775
5776 $ wic ls tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic
5777 Num Start End Size Fstype
5778 1 1048576 25041919 23993344 fat16
5779 2 25165824 72157183 46991360 ext4
5780
5781 The previous output shows two partitions in the
5782 ``core-image-minimal-qemux86.wic`` image.
5783
57842. *Examine a Particular Partition:* Use the ``wic ls`` command again
5785 but in a different form to examine a particular partition.
5786
5787 .. note::
5788
5789 You can get command usage on any Wic command using the following
Andrew Geisslerc926e172021-05-07 16:11:35 -05005790 form::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005791
5792 $ wic help command
5793
5794
5795 For example, the following command shows you the various ways to
5796 use the
5797 wic ls
Andrew Geisslerc926e172021-05-07 16:11:35 -05005798 command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005799
5800 $ wic help ls
5801
5802
Andrew Geisslerc926e172021-05-07 16:11:35 -05005803 The following command shows what is in partition one::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005804
5805 $ wic ls tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1
5806 Volume in drive : is boot
5807 Volume Serial Number is E894-1809
5808 Directory for ::/
5809
5810 libcom32 c32 186500 2017-10-09 16:06
5811 libutil c32 24148 2017-10-09 16:06
5812 syslinux cfg 220 2017-10-09 16:06
5813 vesamenu c32 27104 2017-10-09 16:06
5814 vmlinuz 6904608 2017-10-09 16:06
5815 5 files 7 142 580 bytes
5816 16 582 656 bytes free
5817
5818 The previous output shows five files, with the
5819 ``vmlinuz`` being the kernel.
5820
5821 .. note::
5822
5823 If you see the following error, you need to update or create a
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005824 ``~/.mtoolsrc`` file and be sure to have the line "mtools_skip_check=1"
Andrew Geisslerc926e172021-05-07 16:11:35 -05005825 in the file. Then, run the Wic command again::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005826
5827 ERROR: _exec_cmd: /usr/bin/mdir -i /tmp/wic-parttfokuwra ::/ returned '1' instead of 0
5828 output: Total number of sectors (47824) not a multiple of sectors per track (32)!
5829 Add mtools_skip_check=1 to your .mtoolsrc file to skip this test
5830
5831
58323. *Remove the Old Kernel:* Use the ``wic rm`` command to remove the
Andrew Geisslerc926e172021-05-07 16:11:35 -05005833 ``vmlinuz`` file (kernel)::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005834
5835 $ wic rm tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1/vmlinuz
5836
58374. *Add In the New Kernel:* Use the ``wic cp`` command to add the
5838 updated kernel to the Wic image. Depending on how you built your
5839 kernel, it could be in different places. If you used ``devtool`` and
5840 an SDK to build your kernel, it resides in the ``tmp/work`` directory
5841 of the extensible SDK. If you used ``make`` to build the kernel, the
5842 kernel will be in the ``workspace/sources`` area.
5843
5844 The following example assumes ``devtool`` was used to build the
Andrew Geisslerc926e172021-05-07 16:11:35 -05005845 kernel::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005846
Andrew Geissler95ac1b82021-03-31 14:34:31 -05005847 $ wic cp poky_sdk/tmp/work/qemux86-poky-linux/linux-yocto/4.12.12+git999-r0/linux-yocto-4.12.12+git999/arch/x86/boot/bzImage \
5848 poky/build/tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1/vmlinuz
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005849
5850 Once the new kernel is added back into the image, you can use the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005851 ``dd`` command or :ref:`bmaptool
Andrew Geissler09209ee2020-12-13 08:44:15 -06005852 <dev-manual/common-tasks:flashing images using \`\`bmaptool\`\`>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005853 to flash your wic image onto an SD card or USB stick and test your
5854 target.
5855
5856 .. note::
5857
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005858 Using ``bmaptool`` is generally 10 to 20 times faster than using ``dd``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005859
5860Flashing Images Using ``bmaptool``
5861==================================
5862
5863A fast and easy way to flash an image to a bootable device is to use
5864Bmaptool, which is integrated into the OpenEmbedded build system.
5865Bmaptool is a generic tool that creates a file's block map (bmap) and
5866then uses that map to copy the file. As compared to traditional tools
5867such as dd or cp, Bmaptool can copy (or flash) large files like raw
5868system image files much faster.
5869
5870.. note::
5871
5872 - If you are using Ubuntu or Debian distributions, you can install
5873 the ``bmap-tools`` package using the following command and then
5874 use the tool without specifying ``PATH`` even from the root
Andrew Geisslerc926e172021-05-07 16:11:35 -05005875 account::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005876
Andrew Geisslereff27472021-10-29 15:35:00 -05005877 $ sudo apt install bmap-tools
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005878
5879 - If you are unable to install the ``bmap-tools`` package, you will
Andrew Geisslerc926e172021-05-07 16:11:35 -05005880 need to build Bmaptool before using it. Use the following command::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005881
5882 $ bitbake bmap-tools-native
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005883
5884Following, is an example that shows how to flash a Wic image. Realize
5885that while this example uses a Wic image, you can use Bmaptool to flash
5886any type of image. Use these steps to flash an image using Bmaptool:
5887
58881. *Update your local.conf File:* You need to have the following set
Andrew Geisslerc926e172021-05-07 16:11:35 -05005889 in your ``local.conf`` file before building your image::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005890
5891 IMAGE_FSTYPES += "wic wic.bmap"
5892
58932. *Get Your Image:* Either have your image ready (pre-built with the
5894 :term:`IMAGE_FSTYPES`
Andrew Geisslerc926e172021-05-07 16:11:35 -05005895 setting previously mentioned) or take the step to build the image::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005896
5897 $ bitbake image
5898
58993. *Flash the Device:* Flash the device with the image by using Bmaptool
5900 depending on your particular setup. The following commands assume the
5901 image resides in the Build Directory's ``deploy/images/`` area:
5902
Andrew Geisslerc926e172021-05-07 16:11:35 -05005903 - If you have write access to the media, use this command form::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005904
5905 $ oe-run-native bmap-tools-native bmaptool copy build-directory/tmp/deploy/images/machine/image.wic /dev/sdX
5906
5907 - If you do not have write access to the media, set your permissions
Andrew Geisslerc926e172021-05-07 16:11:35 -05005908 first and then use the same command form::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005909
5910 $ sudo chmod 666 /dev/sdX
5911 $ oe-run-native bmap-tools-native bmaptool copy build-directory/tmp/deploy/images/machine/image.wic /dev/sdX
5912
Andrew Geisslerc926e172021-05-07 16:11:35 -05005913For help on the ``bmaptool`` command, use the following command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005914
5915 $ bmaptool --help
5916
5917Making Images More Secure
5918=========================
5919
5920Security is of increasing concern for embedded devices. Consider the
5921issues and problems discussed in just this sampling of work found across
5922the Internet:
5923
5924- *"*\ `Security Risks of Embedded
5925 Systems <https://www.schneier.com/blog/archives/2014/01/security_risks_9.html>`__\ *"*
5926 by Bruce Schneier
5927
5928- *"*\ `Internet Census
5929 2012 <http://census2012.sourceforge.net/paper.html>`__\ *"* by Carna
5930 Botnet
5931
5932- *"*\ `Security Issues for Embedded
Andrew Geisslerd1e89492021-02-12 15:35:20 -06005933 Devices <https://elinux.org/images/6/6f/Security-issues.pdf>`__\ *"*
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005934 by Jake Edge
5935
5936When securing your image is of concern, there are steps, tools, and
5937variables that you can consider to help you reach the security goals you
5938need for your particular device. Not all situations are identical when
5939it comes to making an image secure. Consequently, this section provides
5940some guidance and suggestions for consideration when you want to make
5941your image more secure.
5942
5943.. note::
5944
5945 Because the security requirements and risks are different for every
5946 type of device, this section cannot provide a complete reference on
5947 securing your custom OS. It is strongly recommended that you also
5948 consult other sources of information on embedded Linux system
5949 hardening and on security.
5950
5951General Considerations
5952----------------------
5953
William A. Kennington IIIac69b482021-06-02 12:28:27 -07005954There are general considerations that help you create more secure images.
5955You should consider the following suggestions to make your device
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005956more secure:
5957
5958- Scan additional code you are adding to the system (e.g. application
5959 code) by using static analysis tools. Look for buffer overflows and
5960 other potential security problems.
5961
5962- Pay particular attention to the security for any web-based
5963 administration interface.
5964
5965 Web interfaces typically need to perform administrative functions and
5966 tend to need to run with elevated privileges. Thus, the consequences
5967 resulting from the interface's security becoming compromised can be
5968 serious. Look for common web vulnerabilities such as
5969 cross-site-scripting (XSS), unvalidated inputs, and so forth.
5970
5971 As with system passwords, the default credentials for accessing a
5972 web-based interface should not be the same across all devices. This
5973 is particularly true if the interface is enabled by default as it can
5974 be assumed that many end-users will not change the credentials.
5975
5976- Ensure you can update the software on the device to mitigate
5977 vulnerabilities discovered in the future. This consideration
5978 especially applies when your device is network-enabled.
5979
5980- Ensure you remove or disable debugging functionality before producing
5981 the final image. For information on how to do this, see the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05005982 ":ref:`dev-manual/common-tasks:considerations specific to the openembedded build system`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005983 section.
5984
5985- Ensure you have no network services listening that are not needed.
5986
5987- Remove any software from the image that is not needed.
5988
5989- Enable hardware support for secure boot functionality when your
5990 device supports this functionality.
5991
5992Security Flags
5993--------------
5994
5995The Yocto Project has security flags that you can enable that help make
5996your build output more secure. The security flags are in the
5997``meta/conf/distro/include/security_flags.inc`` file in your
5998:term:`Source Directory` (e.g. ``poky``).
5999
6000.. note::
6001
6002 Depending on the recipe, certain security flags are enabled and
6003 disabled by default.
6004
6005Use the following line in your ``local.conf`` file or in your custom
6006distribution configuration file to enable the security compiler and
Andrew Geisslerc926e172021-05-07 16:11:35 -05006007linker flags for your build::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006008
6009 require conf/distro/include/security_flags.inc
6010
6011Considerations Specific to the OpenEmbedded Build System
6012--------------------------------------------------------
6013
6014You can take some steps that are specific to the OpenEmbedded build
6015system to make your images more secure:
6016
6017- Ensure "debug-tweaks" is not one of your selected
6018 :term:`IMAGE_FEATURES`.
6019 When creating a new project, the default is to provide you with an
6020 initial ``local.conf`` file that enables this feature using the
6021 :term:`EXTRA_IMAGE_FEATURES`
Andrew Geisslerc926e172021-05-07 16:11:35 -05006022 variable with the line::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006023
6024 EXTRA_IMAGE_FEATURES = "debug-tweaks"
6025
6026 To disable that feature, simply comment out that line in your
Andrew Geissler09036742021-06-25 14:25:14 -05006027 ``local.conf`` file, or make sure :term:`IMAGE_FEATURES` does not contain
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006028 "debug-tweaks" before producing your final image. Among other things,
6029 leaving this in place sets the root password as blank, which makes
6030 logging in for debugging or inspection easy during development but
6031 also means anyone can easily log in during production.
6032
6033- It is possible to set a root password for the image and also to set
6034 passwords for any extra users you might add (e.g. administrative or
6035 service type users). When you set up passwords for multiple images or
6036 users, you should not duplicate passwords.
6037
6038 To set up passwords, use the
6039 :ref:`extrausers <ref-classes-extrausers>`
6040 class, which is the preferred method. For an example on how to set up
6041 both root and user passwords, see the
6042 ":ref:`extrausers.bbclass <ref-classes-extrausers>`"
6043 section.
6044
6045 .. note::
6046
6047 When adding extra user accounts or setting a root password, be
6048 cautious about setting the same password on every device. If you
6049 do this, and the password you have set is exposed, then every
6050 device is now potentially compromised. If you need this access but
6051 want to ensure security, consider setting a different, random
6052 password for each device. Typically, you do this as a separate
6053 step after you deploy the image onto the device.
6054
6055- Consider enabling a Mandatory Access Control (MAC) framework such as
6056 SMACK or SELinux and tuning it appropriately for your device's usage.
6057 You can find more information in the
Andrew Geissler09209ee2020-12-13 08:44:15 -06006058 :yocto_git:`meta-selinux </meta-selinux/>` layer.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006059
6060Tools for Hardening Your Image
6061------------------------------
6062
6063The Yocto Project provides tools for making your image more secure. You
6064can find these tools in the ``meta-security`` layer of the
6065:yocto_git:`Yocto Project Source Repositories <>`.
6066
6067Creating Your Own Distribution
6068==============================
6069
6070When you build an image using the Yocto Project and do not alter any
6071distribution :term:`Metadata`, you are
6072creating a Poky distribution. If you wish to gain more control over
6073package alternative selections, compile-time options, and other
6074low-level configurations, you can create your own distribution.
6075
6076To create your own distribution, the basic steps consist of creating
6077your own distribution layer, creating your own distribution
6078configuration file, and then adding any needed code and Metadata to the
6079layer. The following steps provide some more detail:
6080
6081- *Create a layer for your new distro:* Create your distribution layer
6082 so that you can keep your Metadata and code for the distribution
6083 separate. It is strongly recommended that you create and use your own
6084 layer for configuration and code. Using your own layer as compared to
6085 just placing configurations in a ``local.conf`` configuration file
6086 makes it easier to reproduce the same build configuration when using
6087 multiple build machines. See the
Andrew Geissler09209ee2020-12-13 08:44:15 -06006088 ":ref:`dev-manual/common-tasks:creating a general layer using the \`\`bitbake-layers\`\` script`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006089 section for information on how to quickly set up a layer.
6090
6091- *Create the distribution configuration file:* The distribution
6092 configuration file needs to be created in the ``conf/distro``
6093 directory of your layer. You need to name it using your distribution
6094 name (e.g. ``mydistro.conf``).
6095
6096 .. note::
6097
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006098 The :term:`DISTRO` variable in your ``local.conf`` file determines the
6099 name of your distribution.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006100
6101 You can split out parts of your configuration file into include files
6102 and then "require" them from within your distribution configuration
6103 file. Be sure to place the include files in the
6104 ``conf/distro/include`` directory of your layer. A common example
6105 usage of include files would be to separate out the selection of
6106 desired version and revisions for individual recipes.
6107
6108 Your configuration file needs to set the following required
6109 variables:
6110
6111 - :term:`DISTRO_NAME`
6112
6113 - :term:`DISTRO_VERSION`
6114
6115 These following variables are optional and you typically set them
6116 from the distribution configuration file:
6117
6118 - :term:`DISTRO_FEATURES`
6119
6120 - :term:`DISTRO_EXTRA_RDEPENDS`
6121
6122 - :term:`DISTRO_EXTRA_RRECOMMENDS`
6123
6124 - :term:`TCLIBC`
6125
6126 .. tip::
6127
6128 If you want to base your distribution configuration file on the
6129 very basic configuration from OE-Core, you can use
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006130 ``conf/distro/defaultsetup.conf`` as a reference and just include
6131 variables that differ as compared to ``defaultsetup.conf``.
6132 Alternatively, you can create a distribution configuration file
6133 from scratch using the ``defaultsetup.conf`` file or configuration files
6134 from other distributions such as Poky or Angstrom as references.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006135
6136- *Provide miscellaneous variables:* Be sure to define any other
6137 variables for which you want to create a default or enforce as part
6138 of the distribution configuration. You can include nearly any
6139 variable from the ``local.conf`` file. The variables you use are not
6140 limited to the list in the previous bulleted item.
6141
6142- *Point to Your distribution configuration file:* In your
6143 ``local.conf`` file in the :term:`Build Directory`,
6144 set your
6145 :term:`DISTRO` variable to point to
6146 your distribution's configuration file. For example, if your
6147 distribution's configuration file is named ``mydistro.conf``, then
Andrew Geisslerc926e172021-05-07 16:11:35 -05006148 you point to it as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006149
6150 DISTRO = "mydistro"
6151
6152- *Add more to the layer if necessary:* Use your layer to hold other
6153 information needed for the distribution:
6154
6155 - Add recipes for installing distro-specific configuration files
6156 that are not already installed by another recipe. If you have
6157 distro-specific configuration files that are included by an
6158 existing recipe, you should add an append file (``.bbappend``) for
6159 those. For general information and recommendations on how to add
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006160 recipes to your layer, see the
6161 ":ref:`dev-manual/common-tasks:creating your own layer`" and
6162 ":ref:`dev-manual/common-tasks:following best practices when creating layers`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006163 sections.
6164
6165 - Add any image recipes that are specific to your distribution.
6166
6167 - Add a ``psplash`` append file for a branded splash screen. For
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006168 information on append files, see the
Patrick Williams0ca19cc2021-08-16 14:03:13 -05006169 ":ref:`dev-manual/common-tasks:appending other layers metadata with your layer`"
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006170 section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006171
6172 - Add any other append files to make custom changes that are
6173 specific to individual recipes.
6174
6175Creating a Custom Template Configuration Directory
6176==================================================
6177
6178If you are producing your own customized version of the build system for
6179use by other users, you might want to customize the message shown by the
6180setup script or you might want to change the template configuration
6181files (i.e. ``local.conf`` and ``bblayers.conf``) that are created in a
6182new build directory.
6183
6184The OpenEmbedded build system uses the environment variable
6185``TEMPLATECONF`` to locate the directory from which it gathers
6186configuration information that ultimately ends up in the
6187:term:`Build Directory` ``conf`` directory.
6188By default, ``TEMPLATECONF`` is set as follows in the ``poky``
Andrew Geisslerc926e172021-05-07 16:11:35 -05006189repository::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006190
6191 TEMPLATECONF=${TEMPLATECONF:-meta-poky/conf}
6192
6193This is the
6194directory used by the build system to find templates from which to build
6195some key configuration files. If you look at this directory, you will
6196see the ``bblayers.conf.sample``, ``local.conf.sample``, and
6197``conf-notes.txt`` files. The build system uses these files to form the
6198respective ``bblayers.conf`` file, ``local.conf`` file, and display the
6199list of BitBake targets when running the setup script.
6200
6201To override these default configuration files with configurations you
6202want used within every new Build Directory, simply set the
6203``TEMPLATECONF`` variable to your directory. The ``TEMPLATECONF``
6204variable is set in the ``.templateconf`` file, which is in the top-level
6205:term:`Source Directory` folder
6206(e.g. ``poky``). Edit the ``.templateconf`` so that it can locate your
6207directory.
6208
6209Best practices dictate that you should keep your template configuration
6210directory in your custom distribution layer. For example, suppose you
6211have a layer named ``meta-mylayer`` located in your home directory and
6212you want your template configuration directory named ``myconf``.
6213Changing the ``.templateconf`` as follows causes the OpenEmbedded build
6214system to look in your directory and base its configuration files on the
6215``*.sample`` configuration files it finds. The final configuration files
6216(i.e. ``local.conf`` and ``bblayers.conf`` ultimately still end up in
6217your Build Directory, but they are based on your ``*.sample`` files.
6218::
6219
6220 TEMPLATECONF=${TEMPLATECONF:-meta-mylayer/myconf}
6221
6222Aside from the ``*.sample`` configuration files, the ``conf-notes.txt``
6223also resides in the default ``meta-poky/conf`` directory. The script
6224that sets up the build environment (i.e.
6225:ref:`structure-core-script`) uses this file to
6226display BitBake targets as part of the script output. Customizing this
6227``conf-notes.txt`` file is a good way to make sure your list of custom
6228targets appears as part of the script's output.
6229
6230Here is the default list of targets displayed as a result of running
Andrew Geisslerc926e172021-05-07 16:11:35 -05006231either of the setup scripts::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006232
6233 You can now run 'bitbake <target>'
6234
6235 Common targets are:
6236 core-image-minimal
6237 core-image-sato
6238 meta-toolchain
6239 meta-ide-support
6240
6241Changing the listed common targets is as easy as editing your version of
6242``conf-notes.txt`` in your custom template configuration directory and
6243making sure you have ``TEMPLATECONF`` set to your directory.
6244
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006245Conserving Disk Space During Builds
6246===================================
6247
6248To help conserve disk space during builds, you can add the following
6249statement to your project's ``local.conf`` configuration file found in
Andrew Geisslerc926e172021-05-07 16:11:35 -05006250the :term:`Build Directory`::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006251
6252 INHERIT += "rm_work"
6253
6254Adding this statement deletes the work directory used for
6255building a recipe once the recipe is built. For more information on
6256"rm_work", see the
6257:ref:`rm_work <ref-classes-rm-work>` class in the
6258Yocto Project Reference Manual.
6259
6260Working with Packages
6261=====================
6262
6263This section describes a few tasks that involve packages:
6264
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006265- :ref:`dev-manual/common-tasks:excluding packages from an image`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006266
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006267- :ref:`dev-manual/common-tasks:incrementing a package version`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006268
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006269- :ref:`dev-manual/common-tasks:handling optional module packaging`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006270
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006271- :ref:`dev-manual/common-tasks:using runtime package management`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006272
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006273- :ref:`dev-manual/common-tasks:generating and using signed packages`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006274
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006275- :ref:`Setting up and running package test
6276 (ptest) <dev-manual/common-tasks:testing packages with ptest>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006277
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006278- :ref:`dev-manual/common-tasks:creating node package manager (npm) packages`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006279
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006280- :ref:`dev-manual/common-tasks:adding custom metadata to packages`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006281
6282Excluding Packages from an Image
6283--------------------------------
6284
6285You might find it necessary to prevent specific packages from being
6286installed into an image. If so, you can use several variables to direct
6287the build system to essentially ignore installing recommended packages
6288or to not install a package at all.
6289
6290The following list introduces variables you can use to prevent packages
6291from being installed into your image. Each of these variables only works
William A. Kennington IIIac69b482021-06-02 12:28:27 -07006292with IPK and RPM package types, not for Debian packages.
6293Also, you can use these variables from your ``local.conf`` file
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006294or attach them to a specific image recipe by using a recipe name
6295override. For more detail on the variables, see the descriptions in the
6296Yocto Project Reference Manual's glossary chapter.
6297
6298- :term:`BAD_RECOMMENDATIONS`:
6299 Use this variable to specify "recommended-only" packages that you do
6300 not want installed.
6301
6302- :term:`NO_RECOMMENDATIONS`:
6303 Use this variable to prevent all "recommended-only" packages from
6304 being installed.
6305
6306- :term:`PACKAGE_EXCLUDE`:
6307 Use this variable to prevent specific packages from being installed
6308 regardless of whether they are "recommended-only" or not. You need to
6309 realize that the build process could fail with an error when you
6310 prevent the installation of a package whose presence is required by
6311 an installed package.
6312
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006313Incrementing a Package Version
6314------------------------------
6315
6316This section provides some background on how binary package versioning
6317is accomplished and presents some of the services, variables, and
6318terminology involved.
6319
6320In order to understand binary package versioning, you need to consider
6321the following:
6322
6323- Binary Package: The binary package that is eventually built and
6324 installed into an image.
6325
6326- Binary Package Version: The binary package version is composed of two
6327 components - a version and a revision.
6328
6329 .. note::
6330
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006331 Technically, a third component, the "epoch" (i.e. :term:`PE`) is involved
Andrew Geissler09036742021-06-25 14:25:14 -05006332 but this discussion for the most part ignores :term:`PE`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006333
6334 The version and revision are taken from the
6335 :term:`PV` and
6336 :term:`PR` variables, respectively.
6337
Andrew Geissler09036742021-06-25 14:25:14 -05006338- :term:`PV`: The recipe version. :term:`PV` represents the version of the
6339 software being packaged. Do not confuse :term:`PV` with the binary
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006340 package version.
6341
Andrew Geissler5f350902021-07-23 13:09:54 -04006342- :term:`PR`: The recipe revision.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006343
6344- :term:`SRCPV`: The OpenEmbedded
Andrew Geissler09036742021-06-25 14:25:14 -05006345 build system uses this string to help define the value of :term:`PV` when
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006346 the source code revision needs to be included in it.
6347
Andrew Geissler09209ee2020-12-13 08:44:15 -06006348- :yocto_wiki:`PR Service </PR_Service>`: A
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006349 network-based service that helps automate keeping package feeds
6350 compatible with existing package manager applications such as RPM,
6351 APT, and OPKG.
6352
6353Whenever the binary package content changes, the binary package version
6354must change. Changing the binary package version is accomplished by
Andrew Geissler09036742021-06-25 14:25:14 -05006355changing or "bumping" the :term:`PR` and/or :term:`PV` values. Increasing these
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006356values occurs one of two ways:
6357
6358- Automatically using a Package Revision Service (PR Service).
6359
Andrew Geissler09036742021-06-25 14:25:14 -05006360- Manually incrementing the :term:`PR` and/or :term:`PV` variables.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006361
6362Given a primary challenge of any build system and its users is how to
6363maintain a package feed that is compatible with existing package manager
6364applications such as RPM, APT, and OPKG, using an automated system is
6365much preferred over a manual system. In either system, the main
6366requirement is that binary package version numbering increases in a
William A. Kennington IIIac69b482021-06-02 12:28:27 -07006367linear fashion and that there is a number of version components that
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006368support that linear progression. For information on how to ensure
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006369package revisioning remains linear, see the
6370":ref:`dev-manual/common-tasks:automatically incrementing a package version number`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006371section.
6372
6373The following three sections provide related information on the PR
Andrew Geissler09036742021-06-25 14:25:14 -05006374Service, the manual method for "bumping" :term:`PR` and/or :term:`PV`, and on
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006375how to ensure binary package revisioning remains linear.
6376
6377Working With a PR Service
6378~~~~~~~~~~~~~~~~~~~~~~~~~
6379
6380As mentioned, attempting to maintain revision numbers in the
6381:term:`Metadata` is error prone, inaccurate,
6382and causes problems for people submitting recipes. Conversely, the PR
6383Service automatically generates increasing numbers, particularly the
6384revision field, which removes the human element.
6385
6386.. note::
6387
6388 For additional information on using a PR Service, you can see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06006389 :yocto_wiki:`PR Service </PR_Service>` wiki page.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006390
6391The Yocto Project uses variables in order of decreasing priority to
6392facilitate revision numbering (i.e.
6393:term:`PE`,
6394:term:`PV`, and
6395:term:`PR` for epoch, version, and
6396revision, respectively). The values are highly dependent on the policies
6397and procedures of a given distribution and package feed.
6398
6399Because the OpenEmbedded build system uses
Andrew Geissler09209ee2020-12-13 08:44:15 -06006400":ref:`signatures <overview-manual/concepts:checksums (signatures)>`", which are
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006401unique to a given build, the build system knows when to rebuild
6402packages. All the inputs into a given task are represented by a
6403signature, which can trigger a rebuild when different. Thus, the build
Andrew Geissler09036742021-06-25 14:25:14 -05006404system itself does not rely on the :term:`PR`, :term:`PV`, and :term:`PE` numbers to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006405trigger a rebuild. The signatures, however, can be used to generate
6406these values.
6407
6408The PR Service works with both ``OEBasic`` and ``OEBasicHash``
Andrew Geissler09036742021-06-25 14:25:14 -05006409generators. The value of :term:`PR` bumps when the checksum changes and the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006410different generator mechanisms change signatures under different
6411circumstances.
6412
6413As implemented, the build system includes values from the PR Service
Andrew Geissler09036742021-06-25 14:25:14 -05006414into the :term:`PR` field as an addition using the form "``.x``" so ``r0``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006415becomes ``r0.1``, ``r0.2`` and so forth. This scheme allows existing
Andrew Geissler09036742021-06-25 14:25:14 -05006416:term:`PR` values to be used for whatever reasons, which include manual
6417:term:`PR` bumps, should it be necessary.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006418
6419By default, the PR Service is not enabled or running. Thus, the packages
6420generated are just "self consistent". The build system adds and removes
6421packages and there are no guarantees about upgrade paths but images will
6422be consistent and correct with the latest changes.
6423
William A. Kennington IIIac69b482021-06-02 12:28:27 -07006424The simplest form for a PR Service is for a single host
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006425development system that builds the package feed (building system). For
6426this scenario, you can enable a local PR Service by setting
6427:term:`PRSERV_HOST` in your
Andrew Geisslerc926e172021-05-07 16:11:35 -05006428``local.conf`` file in the :term:`Build Directory`::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006429
6430 PRSERV_HOST = "localhost:0"
6431
6432Once the service is started, packages will automatically
Andrew Geissler09036742021-06-25 14:25:14 -05006433get increasing :term:`PR` values and BitBake takes care of starting and
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006434stopping the server.
6435
6436If you have a more complex setup where multiple host development systems
6437work against a common, shared package feed, you have a single PR Service
6438running and it is connected to each building system. For this scenario,
Andrew Geisslerc926e172021-05-07 16:11:35 -05006439you need to start the PR Service using the ``bitbake-prserv`` command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006440
6441 bitbake-prserv --host ip --port port --start
6442
6443In addition to
6444hand-starting the service, you need to update the ``local.conf`` file of
6445each building system as described earlier so each system points to the
6446server and port.
6447
6448It is also recommended you use build history, which adds some sanity
6449checks to binary package versions, in conjunction with the server that
6450is running the PR Service. To enable build history, add the following to
Andrew Geisslerc926e172021-05-07 16:11:35 -05006451each building system's ``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006452
6453 # It is recommended to activate "buildhistory" for testing the PR service
6454 INHERIT += "buildhistory"
6455 BUILDHISTORY_COMMIT = "1"
6456
6457For information on build
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006458history, see the
6459":ref:`dev-manual/common-tasks:maintaining build output quality`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006460
6461.. note::
6462
Andrew Geissler09036742021-06-25 14:25:14 -05006463 The OpenEmbedded build system does not maintain :term:`PR` information as
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006464 part of the shared state (sstate) packages. If you maintain an sstate
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006465 feed, it's expected that either all your building systems that
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006466 contribute to the sstate feed use a shared PR Service, or you do not
6467 run a PR Service on any of your building systems. Having some systems
6468 use a PR Service while others do not leads to obvious problems.
6469
6470 For more information on shared state, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06006471 ":ref:`overview-manual/concepts:shared state cache`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006472 section in the Yocto Project Overview and Concepts Manual.
6473
6474Manually Bumping PR
6475~~~~~~~~~~~~~~~~~~~
6476
6477The alternative to setting up a PR Service is to manually "bump" the
6478:term:`PR` variable.
6479
6480If a committed change results in changing the package output, then the
6481value of the PR variable needs to be increased (or "bumped") as part of
Andrew Geissler09036742021-06-25 14:25:14 -05006482that commit. For new recipes you should add the :term:`PR` variable and set
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006483its initial value equal to "r0", which is the default. Even though the
6484default value is "r0", the practice of adding it to a new recipe makes
6485it harder to forget to bump the variable when you make changes to the
6486recipe in future.
6487
6488If you are sharing a common ``.inc`` file with multiple recipes, you can
Andrew Geissler09036742021-06-25 14:25:14 -05006489also use the :term:`INC_PR` variable to ensure that the recipes sharing the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006490``.inc`` file are rebuilt when the ``.inc`` file itself is changed. The
Andrew Geissler09036742021-06-25 14:25:14 -05006491``.inc`` file must set :term:`INC_PR` (initially to "r0"), and all recipes
6492referring to it should set :term:`PR` to "${INC_PR}.0" initially,
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006493incrementing the last number when the recipe is changed. If the ``.inc``
Andrew Geissler09036742021-06-25 14:25:14 -05006494file is changed then its :term:`INC_PR` should be incremented.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006495
Andrew Geissler09036742021-06-25 14:25:14 -05006496When upgrading the version of a binary package, assuming the :term:`PV`
6497changes, the :term:`PR` variable should be reset to "r0" (or "${INC_PR}.0"
6498if you are using :term:`INC_PR`).
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006499
6500Usually, version increases occur only to binary packages. However, if
Andrew Geissler09036742021-06-25 14:25:14 -05006501for some reason :term:`PV` changes but does not increase, you can increase
6502the :term:`PE` variable (Package Epoch). The :term:`PE` variable defaults to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006503"0".
6504
6505Binary package version numbering strives to follow the `Debian Version
6506Field Policy
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006507Guidelines <https://www.debian.org/doc/debian-policy/ch-controlfields.html>`__.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006508These guidelines define how versions are compared and what "increasing"
6509a version means.
6510
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006511Automatically Incrementing a Package Version Number
6512~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
6513
6514When fetching a repository, BitBake uses the
6515:term:`SRCREV` variable to determine
6516the specific source code revision from which to build. You set the
Andrew Geissler09036742021-06-25 14:25:14 -05006517:term:`SRCREV` variable to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006518:term:`AUTOREV` to cause the
6519OpenEmbedded build system to automatically use the latest revision of
Andrew Geisslerc926e172021-05-07 16:11:35 -05006520the software::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006521
6522 SRCREV = "${AUTOREV}"
6523
Andrew Geissler09036742021-06-25 14:25:14 -05006524Furthermore, you need to reference :term:`SRCPV` in :term:`PV` in order to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006525automatically update the version whenever the revision of the source
Andrew Geisslerc926e172021-05-07 16:11:35 -05006526code changes. Here is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006527
6528 PV = "1.0+git${SRCPV}"
6529
Andrew Geissler09036742021-06-25 14:25:14 -05006530The OpenEmbedded build system substitutes :term:`SRCPV` with the following:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006531
6532.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006533
6534 AUTOINC+source_code_revision
6535
6536The build system replaces the ``AUTOINC``
6537with a number. The number used depends on the state of the PR Service:
6538
6539- If PR Service is enabled, the build system increments the number,
6540 which is similar to the behavior of
6541 :term:`PR`. This behavior results in
6542 linearly increasing package versions, which is desirable. Here is an
6543 example:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006544
6545 .. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006546
6547 hello-world-git_0.0+git0+b6558dd387-r0.0_armv7a-neon.ipk
6548 hello-world-git_0.0+git1+dd2f5c3565-r0.0_armv7a-neon.ipk
6549
6550- If PR Service is not enabled, the build system replaces the
6551 ``AUTOINC`` placeholder with zero (i.e. "0"). This results in
6552 changing the package version since the source revision is included.
6553 However, package versions are not increased linearly. Here is an
6554 example:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006555
6556 .. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006557
6558 hello-world-git_0.0+git0+b6558dd387-r0.0_armv7a-neon.ipk
6559 hello-world-git_0.0+git0+dd2f5c3565-r0.0_armv7a-neon.ipk
6560
6561In summary, the OpenEmbedded build system does not track the history of
6562binary package versions for this purpose. ``AUTOINC``, in this case, is
Andrew Geissler09036742021-06-25 14:25:14 -05006563comparable to :term:`PR`. If PR server is not enabled, ``AUTOINC`` in the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006564package version is simply replaced by "0". If PR server is enabled, the
6565build system keeps track of the package versions and bumps the number
6566when the package revision changes.
6567
6568Handling Optional Module Packaging
6569----------------------------------
6570
6571Many pieces of software split functionality into optional modules (or
6572plugins) and the plugins that are built might depend on configuration
6573options. To avoid having to duplicate the logic that determines what
6574modules are available in your recipe or to avoid having to package each
6575module by hand, the OpenEmbedded build system provides functionality to
6576handle module packaging dynamically.
6577
6578To handle optional module packaging, you need to do two things:
6579
6580- Ensure the module packaging is actually done.
6581
6582- Ensure that any dependencies on optional modules from other recipes
6583 are satisfied by your recipe.
6584
6585Making Sure the Packaging is Done
6586~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
6587
6588To ensure the module packaging actually gets done, you use the
6589``do_split_packages`` function within the ``populate_packages`` Python
6590function in your recipe. The ``do_split_packages`` function searches for
6591a pattern of files or directories under a specified path and creates a
6592package for each one it finds by appending to the
6593:term:`PACKAGES` variable and
Patrick Williams0ca19cc2021-08-16 14:03:13 -05006594setting the appropriate values for ``FILES:packagename``,
6595``RDEPENDS:packagename``, ``DESCRIPTION:packagename``, and so forth.
Andrew Geisslerc926e172021-05-07 16:11:35 -05006596Here is an example from the ``lighttpd`` recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006597
Patrick Williams0ca19cc2021-08-16 14:03:13 -05006598 python populate_packages:prepend () {
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006599 lighttpd_libdir = d.expand('${libdir}')
6600 do_split_packages(d, lighttpd_libdir, '^mod_(.*).so$',
6601 'lighttpd-module-%s', 'Lighttpd module for %s',
6602 extra_depends='')
6603 }
6604
6605The previous example specifies a number of things in the call to
6606``do_split_packages``.
6607
6608- A directory within the files installed by your recipe through
6609 ``do_install`` in which to search.
6610
6611- A regular expression used to match module files in that directory. In
6612 the example, note the parentheses () that mark the part of the
6613 expression from which the module name should be derived.
6614
6615- A pattern to use for the package names.
6616
6617- A description for each package.
6618
6619- An empty string for ``extra_depends``, which disables the default
6620 dependency on the main ``lighttpd`` package. Thus, if a file in
6621 ``${libdir}`` called ``mod_alias.so`` is found, a package called
6622 ``lighttpd-module-alias`` is created for it and the
6623 :term:`DESCRIPTION` is set to
6624 "Lighttpd module for alias".
6625
6626Often, packaging modules is as simple as the previous example. However,
William A. Kennington IIIac69b482021-06-02 12:28:27 -07006627there are more advanced options that you can use within
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006628``do_split_packages`` to modify its behavior. And, if you need to, you
6629can add more logic by specifying a hook function that is called for each
6630package. It is also perfectly acceptable to call ``do_split_packages``
6631multiple times if you have more than one set of modules to package.
6632
6633For more examples that show how to use ``do_split_packages``, see the
6634``connman.inc`` file in the ``meta/recipes-connectivity/connman/``
Andrew Geissler09209ee2020-12-13 08:44:15 -06006635directory of the ``poky`` :ref:`source repository <overview-manual/development-environment:yocto project source repositories>`. You can
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006636also find examples in ``meta/classes/kernel.bbclass``.
6637
6638Following is a reference that shows ``do_split_packages`` mandatory and
Andrew Geisslerc926e172021-05-07 16:11:35 -05006639optional arguments::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006640
6641 Mandatory arguments
6642
6643 root
6644 The path in which to search
6645 file_regex
6646 Regular expression to match searched files.
6647 Use parentheses () to mark the part of this
6648 expression that should be used to derive the
6649 module name (to be substituted where %s is
6650 used in other function arguments as noted below)
6651 output_pattern
6652 Pattern to use for the package names. Must
6653 include %s.
6654 description
6655 Description to set for each package. Must
6656 include %s.
6657
6658 Optional arguments
6659
6660 postinst
6661 Postinstall script to use for all packages
6662 (as a string)
6663 recursive
6664 True to perform a recursive search - default
6665 False
6666 hook
6667 A hook function to be called for every match.
6668 The function will be called with the following
6669 arguments (in the order listed):
6670
6671 f
6672 Full path to the file/directory match
6673 pkg
6674 The package name
6675 file_regex
6676 As above
6677 output_pattern
6678 As above
6679 modulename
6680 The module name derived using file_regex
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006681 extra_depends
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006682 Extra runtime dependencies (RDEPENDS) to be
6683 set for all packages. The default value of None
6684 causes a dependency on the main package
6685 (${PN}) - if you do not want this, pass empty
6686 string '' for this parameter.
6687 aux_files_pattern
6688 Extra item(s) to be added to FILES for each
6689 package. Can be a single string item or a list
6690 of strings for multiple items. Must include %s.
6691 postrm
6692 postrm script to use for all packages (as a
6693 string)
6694 allow_dirs
6695 True to allow directories to be matched -
6696 default False
6697 prepend
6698 If True, prepend created packages to PACKAGES
6699 instead of the default False which appends them
6700 match_path
6701 match file_regex on the whole relative path to
Patrick Williams0ca19cc2021-08-16 14:03:13 -05006702 the root rather than just the filename
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006703 aux_files_pattern_verbatim
6704 Extra item(s) to be added to FILES for each
6705 package, using the actual derived module name
6706 rather than converting it to something legal
6707 for a package name. Can be a single string item
6708 or a list of strings for multiple items. Must
6709 include %s.
6710 allow_links
6711 True to allow symlinks to be matched - default
6712 False
6713 summary
6714 Summary to set for each package. Must include %s;
6715 defaults to description if not set.
6716
6717
6718
6719Satisfying Dependencies
6720~~~~~~~~~~~~~~~~~~~~~~~
6721
6722The second part for handling optional module packaging is to ensure that
6723any dependencies on optional modules from other recipes are satisfied by
6724your recipe. You can be sure these dependencies are satisfied by using
6725the :term:`PACKAGES_DYNAMIC`
6726variable. Here is an example that continues with the ``lighttpd`` recipe
Andrew Geisslerc926e172021-05-07 16:11:35 -05006727shown earlier::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006728
6729 PACKAGES_DYNAMIC = "lighttpd-module-.*"
6730
6731The name
6732specified in the regular expression can of course be anything. In this
6733example, it is ``lighttpd-module-`` and is specified as the prefix to
6734ensure that any :term:`RDEPENDS` and
6735:term:`RRECOMMENDS` on a package
6736name starting with the prefix are satisfied during build time. If you
6737are using ``do_split_packages`` as described in the previous section,
Andrew Geissler09036742021-06-25 14:25:14 -05006738the value you put in :term:`PACKAGES_DYNAMIC` should correspond to the name
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006739pattern specified in the call to ``do_split_packages``.
6740
6741Using Runtime Package Management
6742--------------------------------
6743
6744During a build, BitBake always transforms a recipe into one or more
6745packages. For example, BitBake takes the ``bash`` recipe and produces a
6746number of packages (e.g. ``bash``, ``bash-bashbug``,
6747``bash-completion``, ``bash-completion-dbg``, ``bash-completion-dev``,
6748``bash-completion-extra``, ``bash-dbg``, and so forth). Not all
6749generated packages are included in an image.
6750
6751In several situations, you might need to update, add, remove, or query
6752the packages on a target device at runtime (i.e. without having to
6753generate a new image). Examples of such situations include:
6754
6755- You want to provide in-the-field updates to deployed devices (e.g.
6756 security updates).
6757
6758- You want to have a fast turn-around development cycle for one or more
6759 applications that run on your device.
6760
6761- You want to temporarily install the "debug" packages of various
6762 applications on your device so that debugging can be greatly improved
6763 by allowing access to symbols and source debugging.
6764
6765- You want to deploy a more minimal package selection of your device
6766 but allow in-the-field updates to add a larger selection for
6767 customization.
6768
6769In all these situations, you have something similar to a more
6770traditional Linux distribution in that in-field devices are able to
6771receive pre-compiled packages from a server for installation or update.
6772Being able to install these packages on a running, in-field device is
6773what is termed "runtime package management".
6774
6775In order to use runtime package management, you need a host or server
6776machine that serves up the pre-compiled packages plus the required
6777metadata. You also need package manipulation tools on the target. The
6778build machine is a likely candidate to act as the server. However, that
6779machine does not necessarily have to be the package server. The build
6780machine could push its artifacts to another machine that acts as the
6781server (e.g. Internet-facing). In fact, doing so is advantageous for a
6782production environment as getting the packages away from the development
6783system's build directory prevents accidental overwrites.
6784
6785A simple build that targets just one device produces more than one
6786package database. In other words, the packages produced by a build are
6787separated out into a couple of different package groupings based on
6788criteria such as the target's CPU architecture, the target board, or the
6789C library used on the target. For example, a build targeting the
6790``qemux86`` device produces the following three package databases:
6791``noarch``, ``i586``, and ``qemux86``. If you wanted your ``qemux86``
6792device to be aware of all the packages that were available to it, you
6793would need to point it to each of these databases individually. In a
6794similar way, a traditional Linux distribution usually is configured to
6795be aware of a number of software repositories from which it retrieves
6796packages.
6797
6798Using runtime package management is completely optional and not required
6799for a successful build or deployment in any way. But if you want to make
6800use of runtime package management, you need to do a couple things above
6801and beyond the basics. The remainder of this section describes what you
6802need to do.
6803
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006804Build Considerations
6805~~~~~~~~~~~~~~~~~~~~
6806
6807This section describes build considerations of which you need to be
6808aware in order to provide support for runtime package management.
6809
6810When BitBake generates packages, it needs to know what format or formats
6811to use. In your configuration, you use the
6812:term:`PACKAGE_CLASSES`
6813variable to specify the format:
6814
68151. Open the ``local.conf`` file inside your
6816 :term:`Build Directory` (e.g.
Andrew Geissler95ac1b82021-03-31 14:34:31 -05006817 ``poky/build/conf/local.conf``).
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006818
Andrew Geisslerc926e172021-05-07 16:11:35 -050068192. Select the desired package format as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006820
6821 PACKAGE_CLASSES ?= "package_packageformat"
6822
6823 where packageformat can be "ipk", "rpm",
6824 "deb", or "tar" which are the supported package formats.
6825
6826 .. note::
6827
6828 Because the Yocto Project supports four different package formats,
6829 you can set the variable with more than one argument. However, the
6830 OpenEmbedded build system only uses the first argument when
6831 creating an image or Software Development Kit (SDK).
6832
6833If you would like your image to start off with a basic package database
6834containing the packages in your current build as well as to have the
6835relevant tools available on the target for runtime package management,
6836you can include "package-management" in the
6837:term:`IMAGE_FEATURES`
6838variable. Including "package-management" in this configuration variable
6839ensures that when the image is assembled for your target, the image
6840includes the currently-known package databases as well as the
6841target-specific tools required for runtime package management to be
6842performed on the target. However, this is not strictly necessary. You
6843could start your image off without any databases but only include the
6844required on-target package tool(s). As an example, you could include
6845"opkg" in your
6846:term:`IMAGE_INSTALL` variable
6847if you are using the IPK package format. You can then initialize your
6848target's package database(s) later once your image is up and running.
6849
6850Whenever you perform any sort of build step that can potentially
6851generate a package or modify existing package, it is always a good idea
6852to re-generate the package index after the build by using the following
Andrew Geisslerc926e172021-05-07 16:11:35 -05006853command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006854
6855 $ bitbake package-index
6856
6857It might be tempting to build the
6858package and the package index at the same time with a command such as
Andrew Geisslerc926e172021-05-07 16:11:35 -05006859the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006860
6861 $ bitbake some-package package-index
6862
6863Do not do this as
6864BitBake does not schedule the package index for after the completion of
6865the package you are building. Consequently, you cannot be sure of the
6866package index including information for the package you just built.
6867Thus, be sure to run the package update step separately after building
6868any packages.
6869
6870You can use the
6871:term:`PACKAGE_FEED_ARCHS`,
6872:term:`PACKAGE_FEED_BASE_PATHS`,
6873and
6874:term:`PACKAGE_FEED_URIS`
6875variables to pre-configure target images to use a package feed. If you
6876do not define these variables, then manual steps as described in the
6877subsequent sections are necessary to configure the target. You should
6878set these variables before building the image in order to produce a
6879correctly configured image.
6880
6881When your build is complete, your packages reside in the
6882``${TMPDIR}/deploy/packageformat`` directory. For example, if
6883``${``\ :term:`TMPDIR`\ ``}`` is
6884``tmp`` and your selected package type is RPM, then your RPM packages
6885are available in ``tmp/deploy/rpm``.
6886
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006887Host or Server Machine Setup
6888~~~~~~~~~~~~~~~~~~~~~~~~~~~~
6889
6890Although other protocols are possible, a server using HTTP typically
6891serves packages. If you want to use HTTP, then set up and configure a
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006892web server such as Apache 2, lighttpd, or Python web server on the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006893machine serving the packages.
6894
6895To keep things simple, this section describes how to set up a
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006896Python web server to share package feeds from the developer's
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006897machine. Although this server might not be the best for a production
6898environment, the setup is simple and straight forward. Should you want
6899to use a different server more suited for production (e.g. Apache 2,
6900Lighttpd, or Nginx), take the appropriate steps to do so.
6901
6902From within the build directory where you have built an image based on
6903your packaging choice (i.e. the
6904:term:`PACKAGE_CLASSES`
6905setting), simply start the server. The following example assumes a build
Andrew Geissler09036742021-06-25 14:25:14 -05006906directory of ``poky/build/tmp/deploy/rpm`` and a :term:`PACKAGE_CLASSES`
Andrew Geisslerc926e172021-05-07 16:11:35 -05006907setting of "package_rpm"::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006908
Andrew Geissler95ac1b82021-03-31 14:34:31 -05006909 $ cd poky/build/tmp/deploy/rpm
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006910 $ python3 -m http.server
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006911
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006912Target Setup
6913~~~~~~~~~~~~
6914
6915Setting up the target differs depending on the package management
6916system. This section provides information for RPM, IPK, and DEB.
6917
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006918Using RPM
6919^^^^^^^^^
6920
6921The `Dandified Packaging
6922Tool <https://en.wikipedia.org/wiki/DNF_(software)>`__ (DNF) performs
6923runtime package management of RPM packages. In order to use DNF for
6924runtime package management, you must perform an initial setup on the
6925target machine for cases where the ``PACKAGE_FEED_*`` variables were not
6926set as part of the image that is running on the target. This means if
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05006927you built your image and did not use these variables as part of the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006928build and your image is now running on the target, you need to perform
6929the steps in this section if you want to use runtime package management.
6930
6931.. note::
6932
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006933 For information on the ``PACKAGE_FEED_*`` variables, see
6934 :term:`PACKAGE_FEED_ARCHS`, :term:`PACKAGE_FEED_BASE_PATHS`, and
6935 :term:`PACKAGE_FEED_URIS` in the Yocto Project Reference Manual variables
6936 glossary.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006937
6938On the target, you must inform DNF that package databases are available.
6939You do this by creating a file named
6940``/etc/yum.repos.d/oe-packages.repo`` and defining the ``oe-packages``.
6941
6942As an example, assume the target is able to use the following package
6943databases: ``all``, ``i586``, and ``qemux86`` from a server named
6944``my.server``. The specifics for setting up the web server are up to
6945you. The critical requirement is that the URIs in the target repository
6946configuration point to the correct remote location for the feeds.
6947
6948.. note::
6949
6950 For development purposes, you can point the web server to the build
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006951 system's ``deploy`` directory. However, for production use, it is better to
6952 copy the package directories to a location outside of the build area and use
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006953 that location. Doing so avoids situations where the build system
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006954 overwrites or changes the ``deploy`` directory.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006955
6956When telling DNF where to look for the package databases, you must
6957declare individual locations per architecture or a single location used
6958for all architectures. You cannot do both:
6959
6960- *Create an Explicit List of Architectures:* Define individual base
6961 URLs to identify where each package database is located:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006962
6963 .. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006964
6965 [oe-packages]
6966 baseurl=http://my.server/rpm/i586 http://my.server/rpm/qemux86 http://my.server/rpm/all
6967
6968 This example
6969 informs DNF about individual package databases for all three
6970 architectures.
6971
6972- *Create a Single (Full) Package Index:* Define a single base URL that
Andrew Geisslerc926e172021-05-07 16:11:35 -05006973 identifies where a full package database is located::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006974
6975 [oe-packages]
6976 baseurl=http://my.server/rpm
6977
6978 This example informs DNF about a single
6979 package database that contains all the package index information for
6980 all supported architectures.
6981
6982Once you have informed DNF where to find the package databases, you need
6983to fetch them:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006984
6985.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006986
6987 # dnf makecache
6988
6989DNF is now able to find, install, and
6990upgrade packages from the specified repository or repositories.
6991
6992.. note::
6993
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006994 See the `DNF documentation <https://dnf.readthedocs.io/en/latest/>`__ for
6995 additional information.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006996
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006997Using IPK
6998^^^^^^^^^
6999
7000The ``opkg`` application performs runtime package management of IPK
7001packages. You must perform an initial setup for ``opkg`` on the target
7002machine if the
7003:term:`PACKAGE_FEED_ARCHS`,
7004:term:`PACKAGE_FEED_BASE_PATHS`,
7005and
7006:term:`PACKAGE_FEED_URIS`
7007variables have not been set or the target image was built before the
7008variables were set.
7009
7010The ``opkg`` application uses configuration files to find available
7011package databases. Thus, you need to create a configuration file inside
7012the ``/etc/opkg/`` direction, which informs ``opkg`` of any repository
7013you want to use.
7014
7015As an example, suppose you are serving packages from a ``ipk/``
7016directory containing the ``i586``, ``all``, and ``qemux86`` databases
7017through an HTTP server named ``my.server``. On the target, create a
7018configuration file (e.g. ``my_repo.conf``) inside the ``/etc/opkg/``
7019directory containing the following:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007020
7021.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007022
7023 src/gz all http://my.server/ipk/all
7024 src/gz i586 http://my.server/ipk/i586
7025 src/gz qemux86 http://my.server/ipk/qemux86
7026
7027Next, instruct ``opkg`` to fetch the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007028repository information:
7029
7030.. code-block:: none
7031
7032 # opkg update
7033
7034The ``opkg`` application is now able to find, install, and upgrade packages
7035from the specified repository.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007036
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007037Using DEB
7038^^^^^^^^^
7039
7040The ``apt`` application performs runtime package management of DEB
7041packages. This application uses a source list file to find available
7042package databases. You must perform an initial setup for ``apt`` on the
7043target machine if the
7044:term:`PACKAGE_FEED_ARCHS`,
7045:term:`PACKAGE_FEED_BASE_PATHS`,
7046and
7047:term:`PACKAGE_FEED_URIS`
7048variables have not been set or the target image was built before the
7049variables were set.
7050
7051To inform ``apt`` of the repository you want to use, you might create a
7052list file (e.g. ``my_repo.list``) inside the
7053``/etc/apt/sources.list.d/`` directory. As an example, suppose you are
7054serving packages from a ``deb/`` directory containing the ``i586``,
7055``all``, and ``qemux86`` databases through an HTTP server named
7056``my.server``. The list file should contain:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007057
7058.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007059
7060 deb http://my.server/deb/all ./
7061 deb http://my.server/deb/i586 ./
7062 deb http://my.server/deb/qemux86 ./
7063
7064Next, instruct the ``apt`` application
7065to fetch the repository information:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007066
7067.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007068
Andrew Geisslereff27472021-10-29 15:35:00 -05007069 $ sudo apt update
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007070
7071After this step,
7072``apt`` is able to find, install, and upgrade packages from the
7073specified repository.
7074
7075Generating and Using Signed Packages
7076------------------------------------
7077
7078In order to add security to RPM packages used during a build, you can
7079take steps to securely sign them. Once a signature is verified, the
7080OpenEmbedded build system can use the package in the build. If security
7081fails for a signed package, the build system aborts the build.
7082
7083This section describes how to sign RPM packages during a build and how
7084to use signed package feeds (repositories) when doing a build.
7085
7086Signing RPM Packages
7087~~~~~~~~~~~~~~~~~~~~
7088
7089To enable signing RPM packages, you must set up the following
7090configurations in either your ``local.config`` or ``distro.config``
Andrew Geisslerc926e172021-05-07 16:11:35 -05007091file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007092
7093 # Inherit sign_rpm.bbclass to enable signing functionality
7094 INHERIT += " sign_rpm"
7095 # Define the GPG key that will be used for signing.
7096 RPM_GPG_NAME = "key_name"
7097 # Provide passphrase for the key
7098 RPM_GPG_PASSPHRASE = "passphrase"
7099
7100.. note::
7101
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007102 Be sure to supply appropriate values for both `key_name` and
7103 `passphrase`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007104
7105Aside from the ``RPM_GPG_NAME`` and ``RPM_GPG_PASSPHRASE`` variables in
William A. Kennington IIIac69b482021-06-02 12:28:27 -07007106the previous example, two optional variables related to signing are available:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007107
7108- *GPG_BIN:* Specifies a ``gpg`` binary/wrapper that is executed
7109 when the package is signed.
7110
7111- *GPG_PATH:* Specifies the ``gpg`` home directory used when the
7112 package is signed.
7113
7114Processing Package Feeds
7115~~~~~~~~~~~~~~~~~~~~~~~~
7116
7117In addition to being able to sign RPM packages, you can also enable
7118signed package feeds for IPK and RPM packages.
7119
7120The steps you need to take to enable signed package feed use are similar
7121to the steps used to sign RPM packages. You must define the following in
Andrew Geisslerc926e172021-05-07 16:11:35 -05007122your ``local.config`` or ``distro.config`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007123
7124 INHERIT += "sign_package_feed"
7125 PACKAGE_FEED_GPG_NAME = "key_name"
7126 PACKAGE_FEED_GPG_PASSPHRASE_FILE = "path_to_file_containing_passphrase"
7127
William A. Kennington IIIac69b482021-06-02 12:28:27 -07007128For signed package feeds, the passphrase must be specified in a separate file,
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007129which is pointed to by the ``PACKAGE_FEED_GPG_PASSPHRASE_FILE``
7130variable. Regarding security, keeping a plain text passphrase out of the
7131configuration is more secure.
7132
7133Aside from the ``PACKAGE_FEED_GPG_NAME`` and
7134``PACKAGE_FEED_GPG_PASSPHRASE_FILE`` variables, three optional variables
William A. Kennington IIIac69b482021-06-02 12:28:27 -07007135related to signed package feeds are available:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007136
7137- *GPG_BIN* Specifies a ``gpg`` binary/wrapper that is executed
7138 when the package is signed.
7139
7140- *GPG_PATH:* Specifies the ``gpg`` home directory used when the
7141 package is signed.
7142
7143- *PACKAGE_FEED_GPG_SIGNATURE_TYPE:* Specifies the type of ``gpg``
7144 signature. This variable applies only to RPM and IPK package feeds.
7145 Allowable values for the ``PACKAGE_FEED_GPG_SIGNATURE_TYPE`` are
7146 "ASC", which is the default and specifies ascii armored, and "BIN",
7147 which specifies binary.
7148
7149Testing Packages With ptest
7150---------------------------
7151
7152A Package Test (ptest) runs tests against packages built by the
7153OpenEmbedded build system on the target machine. A ptest contains at
7154least two items: the actual test, and a shell script (``run-ptest``)
7155that starts the test. The shell script that starts the test must not
7156contain the actual test - the script only starts the test. On the other
7157hand, the test can be anything from a simple shell script that runs a
7158binary and checks the output to an elaborate system of test binaries and
7159data files.
7160
Andrew Geisslerc926e172021-05-07 16:11:35 -05007161The test generates output in the format used by Automake::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007162
7163 result: testname
7164
7165where the result can be ``PASS``, ``FAIL``, or ``SKIP``, and
7166the testname can be any identifying string.
7167
7168For a list of Yocto Project recipes that are already enabled with ptest,
Andrew Geissler09209ee2020-12-13 08:44:15 -06007169see the :yocto_wiki:`Ptest </Ptest>` wiki page.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007170
7171.. note::
7172
7173 A recipe is "ptest-enabled" if it inherits the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007174 :ref:`ptest <ref-classes-ptest>` class.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007175
7176Adding ptest to Your Build
7177~~~~~~~~~~~~~~~~~~~~~~~~~~
7178
7179To add package testing to your build, add the
7180:term:`DISTRO_FEATURES` and
7181:term:`EXTRA_IMAGE_FEATURES`
7182variables to your ``local.conf`` file, which is found in the
Andrew Geisslerc926e172021-05-07 16:11:35 -05007183:term:`Build Directory`::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007184
Patrick Williams0ca19cc2021-08-16 14:03:13 -05007185 DISTRO_FEATURES:append = " ptest"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007186 EXTRA_IMAGE_FEATURES += "ptest-pkgs"
7187
7188Once your build is complete, the ptest files are installed into the
7189``/usr/lib/package/ptest`` directory within the image, where ``package``
7190is the name of the package.
7191
7192Running ptest
7193~~~~~~~~~~~~~
7194
7195The ``ptest-runner`` package installs a shell script that loops through
7196all installed ptest test suites and runs them in sequence. Consequently,
7197you might want to add this package to your image.
7198
7199Getting Your Package Ready
7200~~~~~~~~~~~~~~~~~~~~~~~~~~
7201
7202In order to enable a recipe to run installed ptests on target hardware,
7203you need to prepare the recipes that build the packages you want to
7204test. Here is what you have to do for each recipe:
7205
7206- *Be sure the recipe inherits
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007207 the* :ref:`ptest <ref-classes-ptest>` *class:*
Andrew Geisslerc926e172021-05-07 16:11:35 -05007208 Include the following line in each recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007209
7210 inherit ptest
7211
7212- *Create run-ptest:* This script starts your test. Locate the
7213 script where you will refer to it using
7214 :term:`SRC_URI`. Here is an
Andrew Geisslerc926e172021-05-07 16:11:35 -05007215 example that starts a test for ``dbus``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007216
7217 #!/bin/sh
7218 cd test
7219 make -k runtest-TESTS
7220
7221- *Ensure dependencies are met:* If the test adds build or runtime
7222 dependencies that normally do not exist for the package (such as
7223 requiring "make" to run the test suite), use the
7224 :term:`DEPENDS` and
7225 :term:`RDEPENDS` variables in
7226 your recipe in order for the package to meet the dependencies. Here
Andrew Geisslerc926e172021-05-07 16:11:35 -05007227 is an example where the package has a runtime dependency on "make"::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007228
Patrick Williams0ca19cc2021-08-16 14:03:13 -05007229 RDEPENDS:${PN}-ptest += "make"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007230
7231- *Add a function to build the test suite:* Not many packages support
7232 cross-compilation of their test suites. Consequently, you usually
7233 need to add a cross-compilation function to the package.
7234
7235 Many packages based on Automake compile and run the test suite by
7236 using a single command such as ``make check``. However, the host
7237 ``make check`` builds and runs on the same computer, while
7238 cross-compiling requires that the package is built on the host but
7239 executed for the target architecture (though often, as in the case
7240 for ptest, the execution occurs on the host). The built version of
7241 Automake that ships with the Yocto Project includes a patch that
7242 separates building and execution. Consequently, packages that use the
7243 unaltered, patched version of ``make check`` automatically
7244 cross-compiles.
7245
7246 Regardless, you still must add a ``do_compile_ptest`` function to
7247 build the test suite. Add a function similar to the following to your
Andrew Geisslerc926e172021-05-07 16:11:35 -05007248 recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007249
7250 do_compile_ptest() {
7251 oe_runmake buildtest-TESTS
7252 }
7253
7254- *Ensure special configurations are set:* If the package requires
7255 special configurations prior to compiling the test code, you must
7256 insert a ``do_configure_ptest`` function into the recipe.
7257
7258- *Install the test suite:* The ``ptest`` class automatically copies
7259 the file ``run-ptest`` to the target and then runs make
7260 ``install-ptest`` to run the tests. If this is not enough, you need
7261 to create a ``do_install_ptest`` function and make sure it gets
7262 called after the "make install-ptest" completes.
7263
7264Creating Node Package Manager (NPM) Packages
7265--------------------------------------------
7266
7267`NPM <https://en.wikipedia.org/wiki/Npm_(software)>`__ is a package
7268manager for the JavaScript programming language. The Yocto Project
Andrew Geissler09209ee2020-12-13 08:44:15 -06007269supports the NPM :ref:`fetcher <bitbake:bitbake-user-manual/bitbake-user-manual-fetching:fetchers>`. You can
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007270use this fetcher in combination with
Andrew Geissler09209ee2020-12-13 08:44:15 -06007271:doc:`devtool </ref-manual/devtool-reference>` to create
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007272recipes that produce NPM packages.
7273
William A. Kennington IIIac69b482021-06-02 12:28:27 -07007274There are two workflows that allow you to create NPM packages using
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007275``devtool``: the NPM registry modules method and the NPM project code
7276method.
7277
7278.. note::
7279
7280 While it is possible to create NPM recipes manually, using
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007281 ``devtool`` is far simpler.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007282
7283Additionally, some requirements and caveats exist.
7284
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007285Requirements and Caveats
7286~~~~~~~~~~~~~~~~~~~~~~~~
7287
7288You need to be aware of the following before using ``devtool`` to create
7289NPM packages:
7290
7291- Of the two methods that you can use ``devtool`` to create NPM
7292 packages, the registry approach is slightly simpler. However, you
7293 might consider the project approach because you do not have to
7294 publish your module in the NPM registry
7295 (`npm-registry <https://docs.npmjs.com/misc/registry>`_), which
7296 is NPM's public registry.
7297
7298- Be familiar with
Andrew Geissler09209ee2020-12-13 08:44:15 -06007299 :doc:`devtool </ref-manual/devtool-reference>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007300
7301- The NPM host tools need the native ``nodejs-npm`` package, which is
7302 part of the OpenEmbedded environment. You need to get the package by
7303 cloning the https://github.com/openembedded/meta-openembedded
7304 repository out of GitHub. Be sure to add the path to your local copy
7305 to your ``bblayers.conf`` file.
7306
7307- ``devtool`` cannot detect native libraries in module dependencies.
7308 Consequently, you must manually add packages to your recipe.
7309
7310- While deploying NPM packages, ``devtool`` cannot determine which
7311 dependent packages are missing on the target (e.g. the node runtime
7312 ``nodejs``). Consequently, you need to find out what files are
7313 missing and be sure they are on the target.
7314
7315- Although you might not need NPM to run your node package, it is
7316 useful to have NPM on your target. The NPM package name is
7317 ``nodejs-npm``.
7318
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007319Using the Registry Modules Method
7320~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
7321
7322This section presents an example that uses the ``cute-files`` module,
7323which is a file browser web application.
7324
7325.. note::
7326
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007327 You must know the ``cute-files`` module version.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007328
7329The first thing you need to do is use ``devtool`` and the NPM fetcher to
Andrew Geisslerc926e172021-05-07 16:11:35 -05007330create the recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007331
7332 $ devtool add "npm://registry.npmjs.org;package=cute-files;version=1.0.2"
7333
7334The
7335``devtool add`` command runs ``recipetool create`` and uses the same
7336fetch URI to download each dependency and capture license details where
7337possible. The result is a generated recipe.
7338
7339The recipe file is fairly simple and contains every license that
7340``recipetool`` finds and includes the licenses in the recipe's
7341:term:`LIC_FILES_CHKSUM`
7342variables. You need to examine the variables and look for those with
7343"unknown" in the :term:`LICENSE`
7344field. You need to track down the license information for "unknown"
7345modules and manually add the information to the recipe.
7346
7347``recipetool`` creates a "shrinkwrap" file for your recipe. Shrinkwrap
7348files capture the version of all dependent modules. Many packages do not
7349provide shrinkwrap files. ``recipetool`` create a shrinkwrap file as it
7350runs.
7351
7352.. note::
7353
7354 A package is created for each sub-module. This policy is the only
7355 practical way to have the licenses for all of the dependencies
7356 represented in the license manifest of the image.
7357
Andrew Geisslerc926e172021-05-07 16:11:35 -05007358The ``devtool edit-recipe`` command lets you take a look at the recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007359
7360 $ devtool edit-recipe cute-files
7361 SUMMARY = "Turn any folder on your computer into a cute file browser, available on the local network."
7362 LICENSE = "MIT & ISC & Unknown"
7363 LIC_FILES_CHKSUM = "file://LICENSE;md5=71d98c0a1db42956787b1909c74a86ca \
7364 file://node_modules/toidentifier/LICENSE;md5=1a261071a044d02eb6f2bb47f51a3502 \
7365 file://node_modules/debug/LICENSE;md5=ddd815a475e7338b0be7a14d8ee35a99 \
7366 ...
7367 SRC_URI = " \
7368 npm://registry.npmjs.org/;package=cute-files;version=${PV} \
7369 npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \
7370 "
7371 S = "${WORKDIR}/npm"
Patrick Williams213cb262021-08-07 19:21:33 -05007372 inherit npm
Patrick Williams0ca19cc2021-08-16 14:03:13 -05007373 LICENSE:${PN} = "MIT"
7374 LICENSE:${PN}-accepts = "MIT"
7375 LICENSE:${PN}-array-flatten = "MIT"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007376 ...
Patrick Williams0ca19cc2021-08-16 14:03:13 -05007377 LICENSE:${PN}-vary = "MIT"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007378
William A. Kennington IIIac69b482021-06-02 12:28:27 -07007379Here are three key points in the previous example:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007380
7381- :term:`SRC_URI` uses the NPM
7382 scheme so that the NPM fetcher is used.
7383
7384- ``recipetool`` collects all the license information. If a
7385 sub-module's license is unavailable, the sub-module's name appears in
7386 the comments.
7387
7388- The ``inherit npm`` statement causes the
7389 :ref:`npm <ref-classes-npm>` class to package
7390 up all the modules.
7391
Andrew Geisslerc926e172021-05-07 16:11:35 -05007392You can run the following command to build the ``cute-files`` package::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007393
7394 $ devtool build cute-files
7395
7396Remember that ``nodejs`` must be installed on
7397the target before your package.
7398
7399Assuming 192.168.7.2 for the target's IP address, use the following
Andrew Geisslerc926e172021-05-07 16:11:35 -05007400command to deploy your package::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007401
7402 $ devtool deploy-target -s cute-files root@192.168.7.2
7403
7404Once the package is installed on the target, you can
7405test the application:
7406
7407.. note::
7408
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007409 Because of a known issue, you cannot simply run ``cute-files`` as you would
7410 if you had run ``npm install``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007411
7412::
7413
7414 $ cd /usr/lib/node_modules/cute-files
7415 $ node cute-files.js
7416
7417On a browser,
7418go to ``http://192.168.7.2:3000`` and you see the following:
7419
7420.. image:: figures/cute-files-npm-example.png
7421 :align: center
7422
7423You can find the recipe in ``workspace/recipes/cute-files``. You can use
7424the recipe in any layer you choose.
7425
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007426Using the NPM Projects Code Method
7427~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
7428
7429Although it is useful to package modules already in the NPM registry,
7430adding ``node.js`` projects under development is a more common developer
7431use case.
7432
7433This section covers the NPM projects code method, which is very similar
7434to the "registry" approach described in the previous section. In the NPM
7435projects method, you provide ``devtool`` with an URL that points to the
7436source files.
7437
7438Replicating the same example, (i.e. ``cute-files``) use the following
Andrew Geisslerc926e172021-05-07 16:11:35 -05007439command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007440
7441 $ devtool add https://github.com/martinaglv/cute-files.git
7442
7443The
7444recipe this command generates is very similar to the recipe created in
Andrew Geissler09036742021-06-25 14:25:14 -05007445the previous section. However, the :term:`SRC_URI` looks like the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007446
7447 SRC_URI = " \
7448 git://github.com/martinaglv/cute-files.git;protocol=https \
7449 npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \
7450 "
7451
7452In this example,
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007453the main module is taken from the Git repository and dependencies are
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007454taken from the NPM registry. Other than those differences, the recipe is
7455basically the same between the two methods. You can build and deploy the
7456package exactly as described in the previous section that uses the
7457registry modules method.
7458
7459Adding custom metadata to packages
7460----------------------------------
7461
7462The variable
7463:term:`PACKAGE_ADD_METADATA`
7464can be used to add additional metadata to packages. This is reflected in
7465the package control/spec file. To take the ipk format for example, the
7466CONTROL file stored inside would contain the additional metadata as
7467additional lines.
7468
7469The variable can be used in multiple ways, including using suffixes to
7470set it for a specific package type and/or package. Note that the order
7471of precedence is the same as this list:
7472
Patrick Williams0ca19cc2021-08-16 14:03:13 -05007473- ``PACKAGE_ADD_METADATA_<PKGTYPE>:<PN>``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007474
7475- ``PACKAGE_ADD_METADATA_<PKGTYPE>``
7476
Patrick Williams0ca19cc2021-08-16 14:03:13 -05007477- ``PACKAGE_ADD_METADATA:<PN>``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007478
Andrew Geissler09036742021-06-25 14:25:14 -05007479- :term:`PACKAGE_ADD_METADATA`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007480
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007481`<PKGTYPE>` is a parameter and expected to be a distinct name of specific
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007482package type:
7483
7484- IPK for .ipk packages
7485
7486- DEB for .deb packages
7487
7488- RPM for .rpm packages
7489
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007490`<PN>` is a parameter and expected to be a package name.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007491
7492The variable can contain multiple [one-line] metadata fields separated
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007493by the literal sequence '\\n'. The separator can be redefined using the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007494variable flag ``separator``.
7495
William A. Kennington IIIac69b482021-06-02 12:28:27 -07007496Here is an example that adds two custom fields for ipk
Andrew Geisslerc926e172021-05-07 16:11:35 -05007497packages::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007498
7499 PACKAGE_ADD_METADATA_IPK = "Vendor: CustomIpk\nGroup:Applications/Spreadsheets"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007500
7501Efficiently Fetching Source Files During a Build
7502================================================
7503
7504The OpenEmbedded build system works with source files located through
7505the :term:`SRC_URI` variable. When
7506you build something using BitBake, a big part of the operation is
7507locating and downloading all the source tarballs. For images,
7508downloading all the source for various packages can take a significant
7509amount of time.
7510
7511This section shows you how you can use mirrors to speed up fetching
7512source files and how you can pre-fetch files all of which leads to more
7513efficient use of resources and time.
7514
7515Setting up Effective Mirrors
7516----------------------------
7517
7518A good deal that goes into a Yocto Project build is simply downloading
7519all of the source tarballs. Maybe you have been working with another
7520build system (OpenEmbedded or Angstrom) for which you have built up a
7521sizable directory of source tarballs. Or, perhaps someone else has such
7522a directory for which you have read access. If so, you can save time by
7523adding statements to your configuration file so that the build process
7524checks local directories first for existing tarballs before checking the
7525Internet.
7526
Andrew Geisslerc926e172021-05-07 16:11:35 -05007527Here is an efficient way to set it up in your ``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007528
7529 SOURCE_MIRROR_URL ?= "file:///home/you/your-download-dir/"
7530 INHERIT += "own-mirrors"
7531 BB_GENERATE_MIRROR_TARBALLS = "1"
7532 # BB_NO_NETWORK = "1"
7533
7534In the previous example, the
7535:term:`BB_GENERATE_MIRROR_TARBALLS`
7536variable causes the OpenEmbedded build system to generate tarballs of
7537the Git repositories and store them in the
7538:term:`DL_DIR` directory. Due to
7539performance reasons, generating and storing these tarballs is not the
7540build system's default behavior.
7541
7542You can also use the
7543:term:`PREMIRRORS` variable. For
7544an example, see the variable's glossary entry in the Yocto Project
7545Reference Manual.
7546
7547Getting Source Files and Suppressing the Build
7548----------------------------------------------
7549
7550Another technique you can use to ready yourself for a successive string
7551of build operations, is to pre-fetch all the source files without
7552actually starting a build. This technique lets you work through any
7553download issues and ultimately gathers all the source files into your
7554download directory :ref:`structure-build-downloads`,
7555which is located with :term:`DL_DIR`.
7556
7557Use the following BitBake command form to fetch all the necessary
Andrew Geisslerc926e172021-05-07 16:11:35 -05007558sources without starting the build::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007559
7560 $ bitbake target --runall=fetch
7561
7562This
7563variation of the BitBake command guarantees that you have all the
7564sources for that BitBake target should you disconnect from the Internet
7565and want to do the build later offline.
7566
7567Selecting an Initialization Manager
7568===================================
7569
7570By default, the Yocto Project uses SysVinit as the initialization
William A. Kennington IIIac69b482021-06-02 12:28:27 -07007571manager. However, there is also support for systemd, which is a full
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007572replacement for init with parallel starting of services, reduced shell
7573overhead and other features that are used by many distributions.
7574
7575Within the system, SysVinit treats system components as services. These
7576services are maintained as shell scripts stored in the ``/etc/init.d/``
7577directory. Services organize into different run levels. This
7578organization is maintained by putting links to the services in the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007579``/etc/rcN.d/`` directories, where `N/` is one of the following options:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007580"S", "0", "1", "2", "3", "4", "5", or "6".
7581
7582.. note::
7583
7584 Each runlevel has a dependency on the previous runlevel. This
7585 dependency allows the services to work properly.
7586
7587In comparison, systemd treats components as units. Using units is a
7588broader concept as compared to using a service. A unit includes several
7589different types of entities. Service is one of the types of entities.
7590The runlevel concept in SysVinit corresponds to the concept of a target
7591in systemd, where target is also a type of supported unit.
7592
7593In a SysVinit-based system, services load sequentially (i.e. one by one)
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007594during init and parallelization is not supported. With systemd, services
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007595start in parallel. Needless to say, the method can have an impact on
7596system startup performance.
7597
7598If you want to use SysVinit, you do not have to do anything. But, if you
7599want to use systemd, you must take some steps as described in the
7600following sections.
7601
7602Using systemd Exclusively
7603-------------------------
7604
Andrew Geisslerc926e172021-05-07 16:11:35 -05007605Set these variables in your distribution configuration file as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007606
Patrick Williams0ca19cc2021-08-16 14:03:13 -05007607 DISTRO_FEATURES:append = " systemd"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007608 VIRTUAL-RUNTIME_init_manager = "systemd"
7609
7610You can also prevent the SysVinit distribution feature from
Andrew Geisslerc926e172021-05-07 16:11:35 -05007611being automatically enabled as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007612
7613 DISTRO_FEATURES_BACKFILL_CONSIDERED = "sysvinit"
7614
7615Doing so removes any
7616redundant SysVinit scripts.
7617
7618To remove initscripts from your image altogether, set this variable
Andrew Geisslerc926e172021-05-07 16:11:35 -05007619also::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007620
7621 VIRTUAL-RUNTIME_initscripts = ""
7622
7623For information on the backfill variable, see
7624:term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`.
7625
7626Using systemd for the Main Image and Using SysVinit for the Rescue Image
7627------------------------------------------------------------------------
7628
Andrew Geisslerc926e172021-05-07 16:11:35 -05007629Set these variables in your distribution configuration file as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007630
Patrick Williams0ca19cc2021-08-16 14:03:13 -05007631 DISTRO_FEATURES:append = " systemd"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007632 VIRTUAL-RUNTIME_init_manager = "systemd"
7633
7634Doing so causes your main image to use the
7635``packagegroup-core-boot.bb`` recipe and systemd. The rescue/minimal
7636image cannot use this package group. However, it can install SysVinit
7637and the appropriate packages will have support for both systemd and
7638SysVinit.
7639
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007640Selecting a Device Manager
7641==========================
7642
7643The Yocto Project provides multiple ways to manage the device manager
7644(``/dev``):
7645
Andrew Geissler5199d832021-09-24 16:47:35 -05007646- Persistent and Pre-Populated ``/dev``: For this case, the ``/dev``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007647 directory is persistent and the required device nodes are created
7648 during the build.
7649
7650- Use ``devtmpfs`` with a Device Manager: For this case, the ``/dev``
7651 directory is provided by the kernel as an in-memory file system and
7652 is automatically populated by the kernel at runtime. Additional
7653 configuration of device nodes is done in user space by a device
7654 manager like ``udev`` or ``busybox-mdev``.
7655
Andrew Geissler5199d832021-09-24 16:47:35 -05007656Using Persistent and Pre-Populated ``/dev``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007657--------------------------------------------
7658
7659To use the static method for device population, you need to set the
7660:term:`USE_DEVFS` variable to "0"
Andrew Geisslerc926e172021-05-07 16:11:35 -05007661as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007662
7663 USE_DEVFS = "0"
7664
7665The content of the resulting ``/dev`` directory is defined in a Device
7666Table file. The
7667:term:`IMAGE_DEVICE_TABLES`
7668variable defines the Device Table to use and should be set in the
7669machine or distro configuration file. Alternatively, you can set this
7670variable in your ``local.conf`` configuration file.
7671
Andrew Geissler09036742021-06-25 14:25:14 -05007672If you do not define the :term:`IMAGE_DEVICE_TABLES` variable, the default
Andrew Geisslerc926e172021-05-07 16:11:35 -05007673``device_table-minimal.txt`` is used::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007674
7675 IMAGE_DEVICE_TABLES = "device_table-mymachine.txt"
7676
7677The population is handled by the ``makedevs`` utility during image
7678creation:
7679
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007680Using ``devtmpfs`` and a Device Manager
7681---------------------------------------
7682
7683To use the dynamic method for device population, you need to use (or be
7684sure to set) the :term:`USE_DEVFS`
Andrew Geisslerc926e172021-05-07 16:11:35 -05007685variable to "1", which is the default::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007686
7687 USE_DEVFS = "1"
7688
7689With this
7690setting, the resulting ``/dev`` directory is populated by the kernel
7691using ``devtmpfs``. Make sure the corresponding kernel configuration
7692variable ``CONFIG_DEVTMPFS`` is set when building you build a Linux
7693kernel.
7694
7695All devices created by ``devtmpfs`` will be owned by ``root`` and have
7696permissions ``0600``.
7697
7698To have more control over the device nodes, you can use a device manager
7699like ``udev`` or ``busybox-mdev``. You choose the device manager by
7700defining the ``VIRTUAL-RUNTIME_dev_manager`` variable in your machine or
7701distro configuration file. Alternatively, you can set this variable in
Andrew Geisslerc926e172021-05-07 16:11:35 -05007702your ``local.conf`` configuration file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007703
7704 VIRTUAL-RUNTIME_dev_manager = "udev"
7705
7706 # Some alternative values
7707 # VIRTUAL-RUNTIME_dev_manager = "busybox-mdev"
7708 # VIRTUAL-RUNTIME_dev_manager = "systemd"
7709
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007710Using an External SCM
7711=====================
7712
7713If you're working on a recipe that pulls from an external Source Code
7714Manager (SCM), it is possible to have the OpenEmbedded build system
7715notice new recipe changes added to the SCM and then build the resulting
7716packages that depend on the new recipes by using the latest versions.
7717This only works for SCMs from which it is possible to get a sensible
7718revision number for changes. Currently, you can do this with Apache
7719Subversion (SVN), Git, and Bazaar (BZR) repositories.
7720
7721To enable this behavior, the :term:`PV` of
7722the recipe needs to reference
Andrew Geisslerc926e172021-05-07 16:11:35 -05007723:term:`SRCPV`. Here is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007724
7725 PV = "1.2.3+git${SRCPV}"
7726
7727Then, you can add the following to your
Andrew Geisslerc926e172021-05-07 16:11:35 -05007728``local.conf``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007729
Patrick Williams0ca19cc2021-08-16 14:03:13 -05007730 SRCREV:pn-PN = "${AUTOREV}"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007731
7732:term:`PN` is the name of the recipe for
7733which you want to enable automatic source revision updating.
7734
7735If you do not want to update your local configuration file, you can add
Andrew Geisslerc926e172021-05-07 16:11:35 -05007736the following directly to the recipe to finish enabling the feature::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007737
7738 SRCREV = "${AUTOREV}"
7739
7740The Yocto Project provides a distribution named ``poky-bleeding``, whose
Andrew Geisslerc926e172021-05-07 16:11:35 -05007741configuration file contains the line::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007742
7743 require conf/distro/include/poky-floating-revisions.inc
7744
7745This line pulls in the
Andrew Geisslerc926e172021-05-07 16:11:35 -05007746listed include file that contains numerous lines of exactly that form::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007747
Patrick Williams0ca19cc2021-08-16 14:03:13 -05007748 #SRCREV:pn-opkg-native ?= "${AUTOREV}"
7749 #SRCREV:pn-opkg-sdk ?= "${AUTOREV}"
7750 #SRCREV:pn-opkg ?= "${AUTOREV}"
7751 #SRCREV:pn-opkg-utils-native ?= "${AUTOREV}"
7752 #SRCREV:pn-opkg-utils ?= "${AUTOREV}"
7753 SRCREV:pn-gconf-dbus ?= "${AUTOREV}"
7754 SRCREV:pn-matchbox-common ?= "${AUTOREV}"
7755 SRCREV:pn-matchbox-config-gtk ?= "${AUTOREV}"
7756 SRCREV:pn-matchbox-desktop ?= "${AUTOREV}"
7757 SRCREV:pn-matchbox-keyboard ?= "${AUTOREV}"
7758 SRCREV:pn-matchbox-panel-2 ?= "${AUTOREV}"
7759 SRCREV:pn-matchbox-themes-extra ?= "${AUTOREV}"
7760 SRCREV:pn-matchbox-terminal ?= "${AUTOREV}"
7761 SRCREV:pn-matchbox-wm ?= "${AUTOREV}"
7762 SRCREV:pn-settings-daemon ?= "${AUTOREV}"
7763 SRCREV:pn-screenshot ?= "${AUTOREV}"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007764 . . .
7765
7766These lines allow you to
7767experiment with building a distribution that tracks the latest
7768development source for numerous packages.
7769
7770.. note::
7771
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007772 The ``poky-bleeding`` distribution is not tested on a regular basis. Keep
7773 this in mind if you use it.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007774
7775Creating a Read-Only Root Filesystem
7776====================================
7777
7778Suppose, for security reasons, you need to disable your target device's
7779root filesystem's write permissions (i.e. you need a read-only root
7780filesystem). Or, perhaps you are running the device's operating system
7781from a read-only storage device. For either case, you can customize your
7782image for that behavior.
7783
7784.. note::
7785
7786 Supporting a read-only root filesystem requires that the system and
7787 applications do not try to write to the root filesystem. You must
7788 configure all parts of the target system to write elsewhere, or to
7789 gracefully fail in the event of attempting to write to the root
7790 filesystem.
7791
7792Creating the Root Filesystem
7793----------------------------
7794
7795To create the read-only root filesystem, simply add the
7796"read-only-rootfs" feature to your image, normally in one of two ways.
7797The first way is to add the "read-only-rootfs" image feature in the
Andrew Geissler09036742021-06-25 14:25:14 -05007798image's recipe file via the :term:`IMAGE_FEATURES` variable::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007799
7800 IMAGE_FEATURES += "read-only-rootfs"
7801
7802As an alternative, you can add the same feature
7803from within your build directory's ``local.conf`` file with the
Andrew Geissler09036742021-06-25 14:25:14 -05007804associated :term:`EXTRA_IMAGE_FEATURES` variable, as in::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007805
7806 EXTRA_IMAGE_FEATURES = "read-only-rootfs"
7807
7808For more information on how to use these variables, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06007809":ref:`dev-manual/common-tasks:Customizing Images Using Custom \`\`IMAGE_FEATURES\`\` and \`\`EXTRA_IMAGE_FEATURES\`\``"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007810section. For information on the variables, see
7811:term:`IMAGE_FEATURES` and
7812:term:`EXTRA_IMAGE_FEATURES`.
7813
7814Post-Installation Scripts and Read-Only Root Filesystem
7815-------------------------------------------------------
7816
7817It is very important that you make sure all post-Installation
7818(``pkg_postinst``) scripts for packages that are installed into the
7819image can be run at the time when the root filesystem is created during
7820the build on the host system. These scripts cannot attempt to run during
Patrick Williams0ca19cc2021-08-16 14:03:13 -05007821the first boot on the target device. With the "read-only-rootfs" feature
7822enabled, the build system makes sure that all post-installation scripts
7823succeed at file system creation time. If any of these scripts
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007824still need to be run after the root filesystem is created, the build
7825immediately fails. These build-time checks ensure that the build fails
7826rather than the target device fails later during its initial boot
7827operation.
7828
7829Most of the common post-installation scripts generated by the build
7830system for the out-of-the-box Yocto Project are engineered so that they
7831can run during root filesystem creation (e.g. post-installation scripts
7832for caching fonts). However, if you create and add custom scripts, you
7833need to be sure they can be run during this file system creation.
7834
7835Here are some common problems that prevent post-installation scripts
7836from running during root filesystem creation:
7837
7838- *Not using $D in front of absolute paths:* The build system defines
7839 ``$``\ :term:`D` when the root
7840 filesystem is created. Furthermore, ``$D`` is blank when the script
7841 is run on the target device. This implies two purposes for ``$D``:
7842 ensuring paths are valid in both the host and target environments,
7843 and checking to determine which environment is being used as a method
7844 for taking appropriate actions.
7845
7846- *Attempting to run processes that are specific to or dependent on the
7847 target architecture:* You can work around these attempts by using
7848 native tools, which run on the host system, to accomplish the same
7849 tasks, or by alternatively running the processes under QEMU, which
7850 has the ``qemu_run_binary`` function. For more information, see the
7851 :ref:`qemu <ref-classes-qemu>` class.
7852
7853Areas With Write Access
7854-----------------------
7855
7856With the "read-only-rootfs" feature enabled, any attempt by the target
7857to write to the root filesystem at runtime fails. Consequently, you must
7858make sure that you configure processes and applications that attempt
7859these types of writes do so to directories with write access (e.g.
7860``/tmp`` or ``/var/run``).
7861
7862Maintaining Build Output Quality
7863================================
7864
7865Many factors can influence the quality of a build. For example, if you
7866upgrade a recipe to use a new version of an upstream software package or
7867you experiment with some new configuration options, subtle changes can
7868occur that you might not detect until later. Consider the case where
7869your recipe is using a newer version of an upstream package. In this
7870case, a new version of a piece of software might introduce an optional
7871dependency on another library, which is auto-detected. If that library
7872has already been built when the software is building, the software will
7873link to the built library and that library will be pulled into your
7874image along with the new software even if you did not want the library.
7875
7876The :ref:`buildhistory <ref-classes-buildhistory>`
William A. Kennington IIIac69b482021-06-02 12:28:27 -07007877class helps you maintain the quality of your build output. You
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007878can use the class to highlight unexpected and possibly unwanted changes
7879in the build output. When you enable build history, it records
7880information about the contents of each package and image and then
7881commits that information to a local Git repository where you can examine
7882the information.
7883
7884The remainder of this section describes the following:
7885
Andrew Geissler09209ee2020-12-13 08:44:15 -06007886- :ref:`How you can enable and disable build history <dev-manual/common-tasks:enabling and disabling build history>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007887
Andrew Geissler09209ee2020-12-13 08:44:15 -06007888- :ref:`How to understand what the build history contains <dev-manual/common-tasks:understanding what the build history contains>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007889
Andrew Geissler09209ee2020-12-13 08:44:15 -06007890- :ref:`How to limit the information used for build history <dev-manual/common-tasks:using build history to gather image information only>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007891
Andrew Geissler09209ee2020-12-13 08:44:15 -06007892- :ref:`How to examine the build history from both a command-line and web interface <dev-manual/common-tasks:examining build history information>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007893
7894Enabling and Disabling Build History
7895------------------------------------
7896
7897Build history is disabled by default. To enable it, add the following
Andrew Geissler09036742021-06-25 14:25:14 -05007898:term:`INHERIT` statement and set the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007899:term:`BUILDHISTORY_COMMIT`
7900variable to "1" at the end of your ``conf/local.conf`` file found in the
Andrew Geisslerc926e172021-05-07 16:11:35 -05007901:term:`Build Directory`::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007902
7903 INHERIT += "buildhistory"
7904 BUILDHISTORY_COMMIT = "1"
7905
7906Enabling build history as
7907previously described causes the OpenEmbedded build system to collect
7908build output information and commit it as a single commit to a local
Andrew Geissler09209ee2020-12-13 08:44:15 -06007909:ref:`overview-manual/development-environment:git` repository.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007910
7911.. note::
7912
7913 Enabling build history increases your build times slightly,
7914 particularly for images, and increases the amount of disk space used
7915 during the build.
7916
7917You can disable build history by removing the previous statements from
7918your ``conf/local.conf`` file.
7919
7920Understanding What the Build History Contains
7921---------------------------------------------
7922
7923Build history information is kept in
7924``${``\ :term:`TOPDIR`\ ``}/buildhistory``
7925in the Build Directory as defined by the
7926:term:`BUILDHISTORY_DIR`
William A. Kennington IIIac69b482021-06-02 12:28:27 -07007927variable. Here is an example abbreviated listing:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007928
7929.. image:: figures/buildhistory.png
7930 :align: center
7931
William A. Kennington IIIac69b482021-06-02 12:28:27 -07007932At the top level, there is a ``metadata-revs`` file that lists the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007933revisions of the repositories for the enabled layers when the build was
7934produced. The rest of the data splits into separate ``packages``,
7935``images`` and ``sdk`` directories, the contents of which are described
7936as follows.
7937
7938Build History Package Information
7939~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
7940
7941The history for each package contains a text file that has name-value
7942pairs with information about the package. For example,
7943``buildhistory/packages/i586-poky-linux/busybox/busybox/latest``
7944contains the following:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007945
7946.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007947
7948 PV = 1.22.1
7949 PR = r32
7950 RPROVIDES =
7951 RDEPENDS = glibc (>= 2.20) update-alternatives-opkg
7952 RRECOMMENDS = busybox-syslog busybox-udhcpc update-rc.d
7953 PKGSIZE = 540168
7954 FILES = /usr/bin/* /usr/sbin/* /usr/lib/busybox/* /usr/lib/lib*.so.* \
7955 /etc /com /var /bin/* /sbin/* /lib/*.so.* /lib/udev/rules.d \
7956 /usr/lib/udev/rules.d /usr/share/busybox /usr/lib/busybox/* \
7957 /usr/share/pixmaps /usr/share/applications /usr/share/idl \
7958 /usr/share/omf /usr/share/sounds /usr/lib/bonobo/servers
7959 FILELIST = /bin/busybox /bin/busybox.nosuid /bin/busybox.suid /bin/sh \
7960 /etc/busybox.links.nosuid /etc/busybox.links.suid
7961
7962Most of these
7963name-value pairs correspond to variables used to produce the package.
7964The exceptions are ``FILELIST``, which is the actual list of files in
7965the package, and ``PKGSIZE``, which is the total size of files in the
7966package in bytes.
7967
William A. Kennington IIIac69b482021-06-02 12:28:27 -07007968There is also a file that corresponds to the recipe from which the package
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007969came (e.g. ``buildhistory/packages/i586-poky-linux/busybox/latest``):
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007970
7971.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007972
7973 PV = 1.22.1
7974 PR = r32
7975 DEPENDS = initscripts kern-tools-native update-rc.d-native \
7976 virtual/i586-poky-linux-compilerlibs virtual/i586-poky-linux-gcc \
7977 virtual/libc virtual/update-alternatives
7978 PACKAGES = busybox-ptest busybox-httpd busybox-udhcpd busybox-udhcpc \
7979 busybox-syslog busybox-mdev busybox-hwclock busybox-dbg \
7980 busybox-staticdev busybox-dev busybox-doc busybox-locale busybox
7981
7982Finally, for those recipes fetched from a version control system (e.g.,
William A. Kennington IIIac69b482021-06-02 12:28:27 -07007983Git), there is a file that lists source revisions that are specified in
7984the recipe and the actual revisions used during the build. Listed
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007985and actual revisions might differ when
7986:term:`SRCREV` is set to
7987${:term:`AUTOREV`}. Here is an
7988example assuming
Andrew Geisslerc926e172021-05-07 16:11:35 -05007989``buildhistory/packages/qemux86-poky-linux/linux-yocto/latest_srcrev``)::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007990
7991 # SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
7992 SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
7993 # SRCREV_meta = "a227f20eff056e511d504b2e490f3774ab260d6f"
7994 SRCREV_meta ="a227f20eff056e511d504b2e490f3774ab260d6f"
7995
7996You can use the
7997``buildhistory-collect-srcrevs`` command with the ``-a`` option to
Andrew Geissler09036742021-06-25 14:25:14 -05007998collect the stored :term:`SRCREV` values from build history and report them
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007999in a format suitable for use in global configuration (e.g.,
8000``local.conf`` or a distro include file) to override floating
Andrew Geissler09036742021-06-25 14:25:14 -05008001:term:`AUTOREV` values to a fixed set of revisions. Here is some example
Andrew Geisslerc926e172021-05-07 16:11:35 -05008002output from this command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008003
8004 $ buildhistory-collect-srcrevs -a
8005 # i586-poky-linux
Patrick Williams0ca19cc2021-08-16 14:03:13 -05008006 SRCREV:pn-glibc = "b8079dd0d360648e4e8de48656c5c38972621072"
8007 SRCREV:pn-glibc-initial = "b8079dd0d360648e4e8de48656c5c38972621072"
8008 SRCREV:pn-opkg-utils = "53274f087565fd45d8452c5367997ba6a682a37a"
8009 SRCREV:pn-kmod = "fd56638aed3fe147015bfa10ed4a5f7491303cb4"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008010 # x86_64-linux
Patrick Williams0ca19cc2021-08-16 14:03:13 -05008011 SRCREV:pn-gtk-doc-stub-native = "1dea266593edb766d6d898c79451ef193eb17cfa"
8012 SRCREV:pn-dtc-native = "65cc4d2748a2c2e6f27f1cf39e07a5dbabd80ebf"
8013 SRCREV:pn-update-rc.d-native = "eca680ddf28d024954895f59a241a622dd575c11"
8014 SRCREV_glibc:pn-cross-localedef-native = "b8079dd0d360648e4e8de48656c5c38972621072"
8015 SRCREV_localedef:pn-cross-localedef-native = "c833367348d39dad7ba018990bfdaffaec8e9ed3"
8016 SRCREV:pn-prelink-native = "faa069deec99bf61418d0bab831c83d7c1b797ca"
8017 SRCREV:pn-opkg-utils-native = "53274f087565fd45d8452c5367997ba6a682a37a"
8018 SRCREV:pn-kern-tools-native = "23345b8846fe4bd167efdf1bd8a1224b2ba9a5ff"
8019 SRCREV:pn-kmod-native = "fd56638aed3fe147015bfa10ed4a5f7491303cb4"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008020 # qemux86-poky-linux
Patrick Williams0ca19cc2021-08-16 14:03:13 -05008021 SRCREV_machine:pn-linux-yocto = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
8022 SRCREV_meta:pn-linux-yocto = "a227f20eff056e511d504b2e490f3774ab260d6f"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008023 # all-poky-linux
Patrick Williams0ca19cc2021-08-16 14:03:13 -05008024 SRCREV:pn-update-rc.d = "eca680ddf28d024954895f59a241a622dd575c11"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008025
8026.. note::
8027
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008028 Here are some notes on using the ``buildhistory-collect-srcrevs`` command:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008029
Andrew Geissler09036742021-06-25 14:25:14 -05008030 - By default, only values where the :term:`SRCREV` was not hardcoded
Andrew Geissler5f350902021-07-23 13:09:54 -04008031 (usually when :term:`AUTOREV` is used) are reported. Use the ``-a``
8032 option to see all :term:`SRCREV` values.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008033
8034 - The output statements might not have any effect if overrides are
8035 applied elsewhere in the build system configuration. Use the
8036 ``-f`` option to add the ``forcevariable`` override to each output
8037 line if you need to work around this restriction.
8038
8039 - The script does apply special handling when building for multiple
8040 machines. However, the script does place a comment before each set
8041 of values that specifies which triplet to which they belong as
8042 previously shown (e.g., ``i586-poky-linux``).
8043
8044Build History Image Information
8045~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8046
8047The files produced for each image are as follows:
8048
8049- ``image-files:`` A directory containing selected files from the root
8050 filesystem. The files are defined by
8051 :term:`BUILDHISTORY_IMAGE_FILES`.
8052
8053- ``build-id.txt:`` Human-readable information about the build
8054 configuration and metadata source revisions. This file contains the
8055 full build header as printed by BitBake.
8056
8057- ``*.dot:`` Dependency graphs for the image that are compatible with
8058 ``graphviz``.
8059
8060- ``files-in-image.txt:`` A list of files in the image with
8061 permissions, owner, group, size, and symlink information.
8062
8063- ``image-info.txt:`` A text file containing name-value pairs with
8064 information about the image. See the following listing example for
8065 more information.
8066
8067- ``installed-package-names.txt:`` A list of installed packages by name
8068 only.
8069
8070- ``installed-package-sizes.txt:`` A list of installed packages ordered
8071 by size.
8072
8073- ``installed-packages.txt:`` A list of installed packages with full
8074 package filenames.
8075
8076.. note::
8077
8078 Installed package information is able to be gathered and produced
8079 even if package management is disabled for the final image.
8080
8081Here is an example of ``image-info.txt``:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008082
8083.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008084
8085 DISTRO = poky
8086 DISTRO_VERSION = 1.7
Andrew Geissler5f350902021-07-23 13:09:54 -04008087 USER_CLASSES = buildstats image-prelink
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008088 IMAGE_CLASSES = image_types
8089 IMAGE_FEATURES = debug-tweaks
8090 IMAGE_LINGUAS =
8091 IMAGE_INSTALL = packagegroup-core-boot run-postinsts
8092 BAD_RECOMMENDATIONS =
8093 NO_RECOMMENDATIONS =
8094 PACKAGE_EXCLUDE =
8095 ROOTFS_POSTPROCESS_COMMAND = write_package_manifest; license_create_manifest; \
8096 write_image_manifest ; buildhistory_list_installed_image ; \
8097 buildhistory_get_image_installed ; ssh_allow_empty_password; \
8098 postinst_enable_logging; rootfs_update_timestamp ; ssh_disable_dns_lookup ;
8099 IMAGE_POSTPROCESS_COMMAND = buildhistory_get_imageinfo ;
8100 IMAGESIZE = 6900
8101
8102Other than ``IMAGESIZE``,
8103which is the total size of the files in the image in Kbytes, the
8104name-value pairs are variables that may have influenced the content of
8105the image. This information is often useful when you are trying to
8106determine why a change in the package or file listings has occurred.
8107
8108Using Build History to Gather Image Information Only
8109~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8110
8111As you can see, build history produces image information, including
8112dependency graphs, so you can see why something was pulled into the
8113image. If you are just interested in this information and not interested
8114in collecting specific package or SDK information, you can enable
8115writing only image information without any history by adding the
8116following to your ``conf/local.conf`` file found in the
Andrew Geisslerc926e172021-05-07 16:11:35 -05008117:term:`Build Directory`::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008118
8119 INHERIT += "buildhistory"
8120 BUILDHISTORY_COMMIT = "0"
8121 BUILDHISTORY_FEATURES = "image"
8122
8123Here, you set the
8124:term:`BUILDHISTORY_FEATURES`
8125variable to use the image feature only.
8126
8127Build History SDK Information
8128~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8129
8130Build history collects similar information on the contents of SDKs (e.g.
8131``bitbake -c populate_sdk imagename``) as compared to information it
8132collects for images. Furthermore, this information differs depending on
8133whether an extensible or standard SDK is being produced.
8134
8135The following list shows the files produced for SDKs:
8136
8137- ``files-in-sdk.txt:`` A list of files in the SDK with permissions,
8138 owner, group, size, and symlink information. This list includes both
8139 the host and target parts of the SDK.
8140
8141- ``sdk-info.txt:`` A text file containing name-value pairs with
8142 information about the SDK. See the following listing example for more
8143 information.
8144
8145- ``sstate-task-sizes.txt:`` A text file containing name-value pairs
8146 with information about task group sizes (e.g. ``do_populate_sysroot``
8147 tasks have a total size). The ``sstate-task-sizes.txt`` file exists
8148 only when an extensible SDK is created.
8149
8150- ``sstate-package-sizes.txt:`` A text file containing name-value pairs
8151 with information for the shared-state packages and sizes in the SDK.
8152 The ``sstate-package-sizes.txt`` file exists only when an extensible
8153 SDK is created.
8154
8155- ``sdk-files:`` A folder that contains copies of the files mentioned
8156 in ``BUILDHISTORY_SDK_FILES`` if the files are present in the output.
8157 Additionally, the default value of ``BUILDHISTORY_SDK_FILES`` is
8158 specific to the extensible SDK although you can set it differently if
8159 you would like to pull in specific files from the standard SDK.
8160
8161 The default files are ``conf/local.conf``, ``conf/bblayers.conf``,
8162 ``conf/auto.conf``, ``conf/locked-sigs.inc``, and
8163 ``conf/devtool.conf``. Thus, for an extensible SDK, these files get
8164 copied into the ``sdk-files`` directory.
8165
8166- The following information appears under each of the ``host`` and
8167 ``target`` directories for the portions of the SDK that run on the
8168 host and on the target, respectively:
8169
8170 .. note::
8171
8172 The following files for the most part are empty when producing an
8173 extensible SDK because this type of SDK is not constructed from
8174 packages as is the standard SDK.
8175
8176 - ``depends.dot:`` Dependency graph for the SDK that is compatible
8177 with ``graphviz``.
8178
8179 - ``installed-package-names.txt:`` A list of installed packages by
8180 name only.
8181
8182 - ``installed-package-sizes.txt:`` A list of installed packages
8183 ordered by size.
8184
8185 - ``installed-packages.txt:`` A list of installed packages with full
8186 package filenames.
8187
8188Here is an example of ``sdk-info.txt``:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008189
8190.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008191
8192 DISTRO = poky
8193 DISTRO_VERSION = 1.3+snapshot-20130327
8194 SDK_NAME = poky-glibc-i686-arm
8195 SDK_VERSION = 1.3+snapshot
8196 SDKMACHINE =
8197 SDKIMAGE_FEATURES = dev-pkgs dbg-pkgs
8198 BAD_RECOMMENDATIONS =
8199 SDKSIZE = 352712
8200
8201Other than ``SDKSIZE``, which is
8202the total size of the files in the SDK in Kbytes, the name-value pairs
8203are variables that might have influenced the content of the SDK. This
8204information is often useful when you are trying to determine why a
8205change in the package or file listings has occurred.
8206
8207Examining Build History Information
8208~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8209
8210You can examine build history output from the command line or from a web
8211interface.
8212
8213To see any changes that have occurred (assuming you have
8214:term:`BUILDHISTORY_COMMIT` = "1"),
8215you can simply use any Git command that allows you to view the history
Andrew Geisslerc926e172021-05-07 16:11:35 -05008216of a repository. Here is one method::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008217
8218 $ git log -p
8219
8220You need to realize,
8221however, that this method does show changes that are not significant
8222(e.g. a package's size changing by a few bytes).
8223
William A. Kennington IIIac69b482021-06-02 12:28:27 -07008224There is a command-line tool called ``buildhistory-diff``, though,
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008225that queries the Git repository and prints just the differences that
Andrew Geisslerc926e172021-05-07 16:11:35 -05008226might be significant in human-readable form. Here is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008227
Andrew Geissler95ac1b82021-03-31 14:34:31 -05008228 $ poky/poky/scripts/buildhistory-diff . HEAD^
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008229 Changes to images/qemux86_64/glibc/core-image-minimal (files-in-image.txt):
8230 /etc/anotherpkg.conf was added
8231 /sbin/anotherpkg was added
8232 * (installed-package-names.txt):
8233 * anotherpkg was added
8234 Changes to images/qemux86_64/glibc/core-image-minimal (installed-package-names.txt):
8235 anotherpkg was added
8236 packages/qemux86_64-poky-linux/v86d: PACKAGES: added "v86d-extras"
8237 * PR changed from "r0" to "r1"
8238 * PV changed from "0.1.10" to "0.1.12"
8239 packages/qemux86_64-poky-linux/v86d/v86d: PKGSIZE changed from 110579 to 144381 (+30%)
8240 * PR changed from "r0" to "r1"
8241 * PV changed from "0.1.10" to "0.1.12"
8242
8243.. note::
8244
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008245 The ``buildhistory-diff`` tool requires the ``GitPython``
Andrew Geisslerc926e172021-05-07 16:11:35 -05008246 package. Be sure to install it using Pip3 as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008247
8248 $ pip3 install GitPython --user
8249
8250
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008251 Alternatively, you can install ``python3-git`` using the appropriate
Andrew Geisslereff27472021-10-29 15:35:00 -05008252 distribution package manager (e.g. ``apt``, ``dnf``, or ``zipper``).
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008253
8254To see changes to the build history using a web interface, follow the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008255instruction in the ``README`` file
Andrew Geissler09209ee2020-12-13 08:44:15 -06008256:yocto_git:`here </buildhistory-web/>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008257
8258Here is a sample screenshot of the interface:
8259
8260.. image:: figures/buildhistory-web.png
8261 :align: center
8262
8263Performing Automated Runtime Testing
8264====================================
8265
8266The OpenEmbedded build system makes available a series of automated
8267tests for images to verify runtime functionality. You can run these
8268tests on either QEMU or actual target hardware. Tests are written in
8269Python making use of the ``unittest`` module, and the majority of them
8270run commands on the target system over SSH. This section describes how
8271you set up the environment to use these tests, run available tests, and
8272write and add your own tests.
8273
8274For information on the test and QA infrastructure available within the
Andrew Geissler09209ee2020-12-13 08:44:15 -06008275Yocto Project, see the ":ref:`ref-manual/release-process:testing and quality assurance`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008276section in the Yocto Project Reference Manual.
8277
8278Enabling Tests
8279--------------
8280
8281Depending on whether you are planning to run tests using QEMU or on the
8282hardware, you have to take different steps to enable the tests. See the
8283following subsections for information on how to enable both types of
8284tests.
8285
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008286Enabling Runtime Tests on QEMU
8287~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8288
8289In order to run tests, you need to do the following:
8290
8291- *Set up to avoid interaction with sudo for networking:* To
8292 accomplish this, you must do one of the following:
8293
8294 - Add ``NOPASSWD`` for your user in ``/etc/sudoers`` either for all
8295 commands or just for ``runqemu-ifup``. You must provide the full
8296 path as that can change if you are using multiple clones of the
8297 source repository.
8298
8299 .. note::
8300
8301 On some distributions, you also need to comment out "Defaults
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008302 requiretty" in ``/etc/sudoers``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008303
8304 - Manually configure a tap interface for your system.
8305
8306 - Run as root the script in ``scripts/runqemu-gen-tapdevs``, which
8307 should generate a list of tap devices. This is the option
8308 typically chosen for Autobuilder-type environments.
8309
8310 .. note::
8311
8312 - Be sure to use an absolute path when calling this script
8313 with sudo.
8314
8315 - The package recipe ``qemu-helper-native`` is required to run
Andrew Geisslerc926e172021-05-07 16:11:35 -05008316 this script. Build the package using the following command::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008317
8318 $ bitbake qemu-helper-native
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008319
8320- *Set the DISPLAY variable:* You need to set this variable so that
8321 you have an X server available (e.g. start ``vncserver`` for a
8322 headless machine).
8323
8324- *Be sure your host's firewall accepts incoming connections from
8325 192.168.7.0/24:* Some of the tests (in particular DNF tests) start an
8326 HTTP server on a random high number port, which is used to serve
8327 files to the target. The DNF module serves
8328 ``${WORKDIR}/oe-rootfs-repo`` so it can run DNF channel commands.
8329 That means your host's firewall must accept incoming connections from
8330 192.168.7.0/24, which is the default IP range used for tap devices by
8331 ``runqemu``.
8332
8333- *Be sure your host has the correct packages installed:* Depending
8334 your host's distribution, you need to have the following packages
8335 installed:
8336
8337 - Ubuntu and Debian: ``sysstat`` and ``iproute2``
8338
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05008339 - openSUSE: ``sysstat`` and ``iproute2``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008340
8341 - Fedora: ``sysstat`` and ``iproute``
8342
8343 - CentOS: ``sysstat`` and ``iproute``
8344
8345Once you start running the tests, the following happens:
8346
83471. A copy of the root filesystem is written to ``${WORKDIR}/testimage``.
8348
83492. The image is booted under QEMU using the standard ``runqemu`` script.
8350
83513. A default timeout of 500 seconds occurs to allow for the boot process
8352 to reach the login prompt. You can change the timeout period by
8353 setting
8354 :term:`TEST_QEMUBOOT_TIMEOUT`
8355 in the ``local.conf`` file.
8356
83574. Once the boot process is reached and the login prompt appears, the
8358 tests run. The full boot log is written to
8359 ``${WORKDIR}/testimage/qemu_boot_log``.
8360
Andrew Geissler09036742021-06-25 14:25:14 -050083615. Each test module loads in the order found in :term:`TEST_SUITES`. You can
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008362 find the full output of the commands run over SSH in
8363 ``${WORKDIR}/testimgage/ssh_target_log``.
8364
83656. If no failures occur, the task running the tests ends successfully.
8366 You can find the output from the ``unittest`` in the task log at
8367 ``${WORKDIR}/temp/log.do_testimage``.
8368
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008369Enabling Runtime Tests on Hardware
8370~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8371
8372The OpenEmbedded build system can run tests on real hardware, and for
8373certain devices it can also deploy the image to be tested onto the
8374device beforehand.
8375
8376For automated deployment, a "master image" is installed onto the
8377hardware once as part of setup. Then, each time tests are to be run, the
8378following occurs:
8379
83801. The master image is booted into and used to write the image to be
8381 tested to a second partition.
8382
83832. The device is then rebooted using an external script that you need to
8384 provide.
8385
83863. The device boots into the image to be tested.
8387
8388When running tests (independent of whether the image has been deployed
8389automatically or not), the device is expected to be connected to a
8390network on a pre-determined IP address. You can either use static IP
8391addresses written into the image, or set the image to use DHCP and have
8392your DHCP server on the test network assign a known IP address based on
8393the MAC address of the device.
8394
Andrew Geissler09036742021-06-25 14:25:14 -05008395In order to run tests on hardware, you need to set :term:`TEST_TARGET` to an
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008396appropriate value. For QEMU, you do not have to change anything, the
8397default value is "qemu". For running tests on hardware, the following
William A. Kennington IIIac69b482021-06-02 12:28:27 -07008398options are available:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008399
8400- *"simpleremote":* Choose "simpleremote" if you are going to run tests
8401 on a target system that is already running the image to be tested and
8402 is available on the network. You can use "simpleremote" in
8403 conjunction with either real hardware or an image running within a
8404 separately started QEMU or any other virtual machine manager.
8405
8406- *"SystemdbootTarget":* Choose "SystemdbootTarget" if your hardware is
8407 an EFI-based machine with ``systemd-boot`` as bootloader and
8408 ``core-image-testmaster`` (or something similar) is installed. Also,
8409 your hardware under test must be in a DHCP-enabled network that gives
8410 it the same IP address for each reboot.
8411
8412 If you choose "SystemdbootTarget", there are additional requirements
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05008413 and considerations. See the
8414 ":ref:`dev-manual/common-tasks:selecting systemdboottarget`" section, which
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008415 follows, for more information.
8416
8417- *"BeagleBoneTarget":* Choose "BeagleBoneTarget" if you are deploying
8418 images and running tests on the BeagleBone "Black" or original
8419 "White" hardware. For information on how to use these tests, see the
8420 comments at the top of the BeagleBoneTarget
8421 ``meta-yocto-bsp/lib/oeqa/controllers/beaglebonetarget.py`` file.
8422
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008423- *"EdgeRouterTarget":* Choose "EdgeRouterTarget" if you are deploying
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008424 images and running tests on the Ubiquiti Networks EdgeRouter Lite.
8425 For information on how to use these tests, see the comments at the
8426 top of the EdgeRouterTarget
8427 ``meta-yocto-bsp/lib/oeqa/controllers/edgeroutertarget.py`` file.
8428
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008429- *"GrubTarget":* Choose "GrubTarget" if you are deploying images and running
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008430 tests on any generic PC that boots using GRUB. For information on how
8431 to use these tests, see the comments at the top of the GrubTarget
8432 ``meta-yocto-bsp/lib/oeqa/controllers/grubtarget.py`` file.
8433
8434- *"your-target":* Create your own custom target if you want to run
8435 tests when you are deploying images and running tests on a custom
8436 machine within your BSP layer. To do this, you need to add a Python
8437 unit that defines the target class under ``lib/oeqa/controllers/``
8438 within your layer. You must also provide an empty ``__init__.py``.
8439 For examples, see files in ``meta-yocto-bsp/lib/oeqa/controllers/``.
8440
8441Selecting SystemdbootTarget
8442~~~~~~~~~~~~~~~~~~~~~~~~~~~
8443
Andrew Geissler09036742021-06-25 14:25:14 -05008444If you did not set :term:`TEST_TARGET` to "SystemdbootTarget", then you do
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008445not need any information in this section. You can skip down to the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05008446":ref:`dev-manual/common-tasks:running tests`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008447
Andrew Geissler09036742021-06-25 14:25:14 -05008448If you did set :term:`TEST_TARGET` to "SystemdbootTarget", you also need to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008449perform a one-time setup of your master image by doing the following:
8450
Andrew Geissler09036742021-06-25 14:25:14 -050084511. *Set EFI_PROVIDER:* Be sure that :term:`EFI_PROVIDER` is as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008452
8453 EFI_PROVIDER = "systemd-boot"
8454
84552. *Build the master image:* Build the ``core-image-testmaster`` image.
8456 The ``core-image-testmaster`` recipe is provided as an example for a
8457 "master" image and you can customize the image recipe as you would
8458 any other recipe.
8459
8460 Here are the image recipe requirements:
8461
8462 - Inherits ``core-image`` so that kernel modules are installed.
8463
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05008464 - Installs normal linux utilities not BusyBox ones (e.g. ``bash``,
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008465 ``coreutils``, ``tar``, ``gzip``, and ``kmod``).
8466
8467 - Uses a custom Initial RAM Disk (initramfs) image with a custom
8468 installer. A normal image that you can install usually creates a
8469 single rootfs partition. This image uses another installer that
8470 creates a specific partition layout. Not all Board Support
8471 Packages (BSPs) can use an installer. For such cases, you need to
8472 manually create the following partition layout on the target:
8473
8474 - First partition mounted under ``/boot``, labeled "boot".
8475
8476 - The main rootfs partition where this image gets installed,
8477 which is mounted under ``/``.
8478
8479 - Another partition labeled "testrootfs" where test images get
8480 deployed.
8481
84823. *Install image:* Install the image that you just built on the target
8483 system.
8484
Andrew Geissler09036742021-06-25 14:25:14 -05008485The final thing you need to do when setting :term:`TEST_TARGET` to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008486"SystemdbootTarget" is to set up the test image:
8487
84881. *Set up your local.conf file:* Make sure you have the following
Andrew Geisslerc926e172021-05-07 16:11:35 -05008489 statements in your ``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008490
8491 IMAGE_FSTYPES += "tar.gz"
8492 INHERIT += "testimage"
8493 TEST_TARGET = "SystemdbootTarget"
8494 TEST_TARGET_IP = "192.168.2.3"
8495
Andrew Geisslerc926e172021-05-07 16:11:35 -050084962. *Build your test image:* Use BitBake to build the image::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008497
8498 $ bitbake core-image-sato
8499
8500Power Control
8501~~~~~~~~~~~~~
8502
8503For most hardware targets other than "simpleremote", you can control
8504power:
8505
Andrew Geissler09036742021-06-25 14:25:14 -05008506- You can use :term:`TEST_POWERCONTROL_CMD` together with
8507 :term:`TEST_POWERCONTROL_EXTRA_ARGS` as a command that runs on the host
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008508 and does power cycling. The test code passes one argument to that
8509 command: off, on or cycle (off then on). Here is an example that
Andrew Geisslerc926e172021-05-07 16:11:35 -05008510 could appear in your ``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008511
8512 TEST_POWERCONTROL_CMD = "powercontrol.exp test 10.11.12.1 nuc1"
8513
8514 In this example, the expect
8515 script does the following:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008516
8517 .. code-block:: shell
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008518
8519 ssh test@10.11.12.1 "pyctl nuc1 arg"
8520
8521 It then runs a Python script that controls power for a label called
8522 ``nuc1``.
8523
8524 .. note::
8525
Andrew Geissler09036742021-06-25 14:25:14 -05008526 You need to customize :term:`TEST_POWERCONTROL_CMD` and
8527 :term:`TEST_POWERCONTROL_EXTRA_ARGS` for your own setup. The one requirement
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008528 is that it accepts "on", "off", and "cycle" as the last argument.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008529
8530- When no command is defined, it connects to the device over SSH and
8531 uses the classic reboot command to reboot the device. Classic reboot
8532 is fine as long as the machine actually reboots (i.e. the SSH test
8533 has not failed). It is useful for scenarios where you have a simple
8534 setup, typically with a single board, and where some manual
8535 interaction is okay from time to time.
8536
8537If you have no hardware to automatically perform power control but still
8538wish to experiment with automated hardware testing, you can use the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008539``dialog-power-control`` script that shows a dialog prompting you to perform
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008540the required power action. This script requires either KDialog or Zenity
8541to be installed. To use this script, set the
8542:term:`TEST_POWERCONTROL_CMD`
Andrew Geisslerc926e172021-05-07 16:11:35 -05008543variable as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008544
8545 TEST_POWERCONTROL_CMD = "${COREBASE}/scripts/contrib/dialog-power-control"
8546
8547Serial Console Connection
8548~~~~~~~~~~~~~~~~~~~~~~~~~
8549
8550For test target classes requiring a serial console to interact with the
8551bootloader (e.g. BeagleBoneTarget, EdgeRouterTarget, and GrubTarget),
8552you need to specify a command to use to connect to the serial console of
8553the target machine by using the
8554:term:`TEST_SERIALCONTROL_CMD`
8555variable and optionally the
8556:term:`TEST_SERIALCONTROL_EXTRA_ARGS`
8557variable.
8558
8559These cases could be a serial terminal program if the machine is
8560connected to a local serial port, or a ``telnet`` or ``ssh`` command
8561connecting to a remote console server. Regardless of the case, the
8562command simply needs to connect to the serial console and forward that
8563connection to standard input and output as any normal terminal program
8564does. For example, to use the picocom terminal program on serial device
Andrew Geisslerc926e172021-05-07 16:11:35 -05008565``/dev/ttyUSB0`` at 115200bps, you would set the variable as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008566
8567 TEST_SERIALCONTROL_CMD = "picocom /dev/ttyUSB0 -b 115200"
8568
8569For local
8570devices where the serial port device disappears when the device reboots,
8571an additional "serdevtry" wrapper script is provided. To use this
8572wrapper, simply prefix the terminal command with
Andrew Geisslerc926e172021-05-07 16:11:35 -05008573``${COREBASE}/scripts/contrib/serdevtry``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008574
8575 TEST_SERIALCONTROL_CMD = "${COREBASE}/scripts/contrib/serdevtry picocom -b 115200 /dev/ttyUSB0"
8576
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008577Running Tests
8578-------------
8579
8580You can start the tests automatically or manually:
8581
8582- *Automatically running tests:* To run the tests automatically after
8583 the OpenEmbedded build system successfully creates an image, first
8584 set the
8585 :term:`TESTIMAGE_AUTO`
8586 variable to "1" in your ``local.conf`` file in the
Andrew Geisslerc926e172021-05-07 16:11:35 -05008587 :term:`Build Directory`::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008588
8589 TESTIMAGE_AUTO = "1"
8590
8591 Next, build your image. If the image successfully builds, the
Andrew Geisslerc926e172021-05-07 16:11:35 -05008592 tests run::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008593
8594 bitbake core-image-sato
8595
8596- *Manually running tests:* To manually run the tests, first globally
8597 inherit the
8598 :ref:`testimage <ref-classes-testimage*>` class
Andrew Geisslerc926e172021-05-07 16:11:35 -05008599 by editing your ``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008600
8601 INHERIT += "testimage"
8602
Andrew Geisslerc926e172021-05-07 16:11:35 -05008603 Next, use BitBake to run the tests::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008604
8605 bitbake -c testimage image
8606
8607All test files reside in ``meta/lib/oeqa/runtime`` in the
8608:term:`Source Directory`. A test name maps
8609directly to a Python module. Each test module may contain a number of
8610individual tests. Tests are usually grouped together by the area tested
8611(e.g tests for systemd reside in ``meta/lib/oeqa/runtime/systemd.py``).
8612
8613You can add tests to any layer provided you place them in the proper
8614area and you extend :term:`BBPATH` in
8615the ``local.conf`` file as normal. Be sure that tests reside in
8616``layer/lib/oeqa/runtime``.
8617
8618.. note::
8619
8620 Be sure that module names do not collide with module names used in
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008621 the default set of test modules in ``meta/lib/oeqa/runtime``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008622
8623You can change the set of tests run by appending or overriding
8624:term:`TEST_SUITES` variable in
Andrew Geissler09036742021-06-25 14:25:14 -05008625``local.conf``. Each name in :term:`TEST_SUITES` represents a required test
8626for the image. Test modules named within :term:`TEST_SUITES` cannot be
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008627skipped even if a test is not suitable for an image (e.g. running the
8628RPM tests on an image without ``rpm``). Appending "auto" to
Andrew Geissler09036742021-06-25 14:25:14 -05008629:term:`TEST_SUITES` causes the build system to try to run all tests that are
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008630suitable for the image (i.e. each test module may elect to skip itself).
8631
Andrew Geissler09036742021-06-25 14:25:14 -05008632The order you list tests in :term:`TEST_SUITES` is important and influences
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008633test dependencies. Consequently, tests that depend on other tests should
8634be added after the test on which they depend. For example, since the
8635``ssh`` test depends on the ``ping`` test, "ssh" needs to come after
8636"ping" in the list. The test class provides no re-ordering or dependency
8637handling.
8638
8639.. note::
8640
8641 Each module can have multiple classes with multiple test methods.
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008642 And, Python ``unittest`` rules apply.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008643
8644Here are some things to keep in mind when running tests:
8645
Andrew Geisslerc926e172021-05-07 16:11:35 -05008646- The default tests for the image are defined as::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008647
Patrick Williams0ca19cc2021-08-16 14:03:13 -05008648 DEFAULT_TEST_SUITES:pn-image = "ping ssh df connman syslog xorg scp vnc date rpm dnf dmesg"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008649
Andrew Geisslerc926e172021-05-07 16:11:35 -05008650- Add your own test to the list of the by using the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008651
Patrick Williams0ca19cc2021-08-16 14:03:13 -05008652 TEST_SUITES:append = " mytest"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008653
Andrew Geisslerc926e172021-05-07 16:11:35 -05008654- Run a specific list of tests as follows::
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008655
8656 TEST_SUITES = "test1 test2 test3"
8657
8658 Remember, order is important. Be sure to place a test that is
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008659 dependent on another test later in the order.
8660
8661Exporting Tests
8662---------------
8663
8664You can export tests so that they can run independently of the build
8665system. Exporting tests is required if you want to be able to hand the
8666test execution off to a scheduler. You can only export tests that are
8667defined in :term:`TEST_SUITES`.
8668
8669If your image is already built, make sure the following are set in your
Andrew Geisslerc926e172021-05-07 16:11:35 -05008670``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008671
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008672 INHERIT += "testexport"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008673 TEST_TARGET_IP = "IP-address-for-the-test-target"
8674 TEST_SERVER_IP = "IP-address-for-the-test-server"
8675
8676You can then export the tests with the
Andrew Geisslerc926e172021-05-07 16:11:35 -05008677following BitBake command form::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008678
8679 $ bitbake image -c testexport
8680
8681Exporting the tests places them in the
8682:term:`Build Directory` in
8683``tmp/testexport/``\ image, which is controlled by the
Andrew Geissler09036742021-06-25 14:25:14 -05008684:term:`TEST_EXPORT_DIR` variable.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008685
Andrew Geisslerc926e172021-05-07 16:11:35 -05008686You can now run the tests outside of the build environment::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008687
8688 $ cd tmp/testexport/image
8689 $ ./runexported.py testdata.json
8690
8691Here is a complete example that shows IP addresses and uses the
Andrew Geisslerc926e172021-05-07 16:11:35 -05008692``core-image-sato`` image::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008693
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008694 INHERIT += "testexport"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008695 TEST_TARGET_IP = "192.168.7.2"
8696 TEST_SERVER_IP = "192.168.7.1"
8697
Andrew Geisslerc926e172021-05-07 16:11:35 -05008698Use BitBake to export the tests::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008699
8700 $ bitbake core-image-sato -c testexport
8701
8702Run the tests outside of
Andrew Geisslerc926e172021-05-07 16:11:35 -05008703the build environment using the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008704
8705 $ cd tmp/testexport/core-image-sato
8706 $ ./runexported.py testdata.json
8707
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008708Writing New Tests
8709-----------------
8710
8711As mentioned previously, all new test files need to be in the proper
8712place for the build system to find them. New tests for additional
8713functionality outside of the core should be added to the layer that adds
8714the functionality, in ``layer/lib/oeqa/runtime`` (as long as
8715:term:`BBPATH` is extended in the
8716layer's ``layer.conf`` file as normal). Just remember the following:
8717
8718- Filenames need to map directly to test (module) names.
8719
8720- Do not use module names that collide with existing core tests.
8721
William A. Kennington IIIac69b482021-06-02 12:28:27 -07008722- Minimally, an empty ``__init__.py`` file must be present in the runtime
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008723 directory.
8724
8725To create a new test, start by copying an existing module (e.g.
8726``syslog.py`` or ``gcc.py`` are good ones to use). Test modules can use
8727code from ``meta/lib/oeqa/utils``, which are helper classes.
8728
8729.. note::
8730
8731 Structure shell commands such that you rely on them and they return a
8732 single code for success. Be aware that sometimes you will need to
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008733 parse the output. See the ``df.py`` and ``date.py`` modules for examples.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008734
8735You will notice that all test classes inherit ``oeRuntimeTest``, which
8736is found in ``meta/lib/oetest.py``. This base class offers some helper
8737attributes, which are described in the following sections:
8738
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008739Class Methods
8740~~~~~~~~~~~~~
8741
8742Class methods are as follows:
8743
8744- *hasPackage(pkg):* Returns "True" if ``pkg`` is in the installed
8745 package list of the image, which is based on the manifest file that
8746 is generated during the ``do_rootfs`` task.
8747
8748- *hasFeature(feature):* Returns "True" if the feature is in
8749 :term:`IMAGE_FEATURES` or
8750 :term:`DISTRO_FEATURES`.
8751
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008752Class Attributes
8753~~~~~~~~~~~~~~~~
8754
8755Class attributes are as follows:
8756
8757- *pscmd:* Equals "ps -ef" if ``procps`` is installed in the image.
8758 Otherwise, ``pscmd`` equals "ps" (busybox).
8759
8760- *tc:* The called test context, which gives access to the
8761 following attributes:
8762
8763 - *d:* The BitBake datastore, which allows you to use stuff such
8764 as ``oeRuntimeTest.tc.d.getVar("VIRTUAL-RUNTIME_init_manager")``.
8765
8766 - *testslist and testsrequired:* Used internally. The tests
8767 do not need these.
8768
8769 - *filesdir:* The absolute path to
8770 ``meta/lib/oeqa/runtime/files``, which contains helper files for
8771 tests meant for copying on the target such as small files written
8772 in C for compilation.
8773
8774 - *target:* The target controller object used to deploy and
8775 start an image on a particular target (e.g. Qemu, SimpleRemote,
8776 and SystemdbootTarget). Tests usually use the following:
8777
8778 - *ip:* The target's IP address.
8779
8780 - *server_ip:* The host's IP address, which is usually used
8781 by the DNF test suite.
8782
8783 - *run(cmd, timeout=None):* The single, most used method.
8784 This command is a wrapper for: ``ssh root@host "cmd"``. The
8785 command returns a tuple: (status, output), which are what their
8786 names imply - the return code of "cmd" and whatever output it
8787 produces. The optional timeout argument represents the number
8788 of seconds the test should wait for "cmd" to return. If the
8789 argument is "None", the test uses the default instance's
8790 timeout period, which is 300 seconds. If the argument is "0",
8791 the test runs until the command returns.
8792
8793 - *copy_to(localpath, remotepath):*
8794 ``scp localpath root@ip:remotepath``.
8795
8796 - *copy_from(remotepath, localpath):*
8797 ``scp root@host:remotepath localpath``.
8798
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008799Instance Attributes
8800~~~~~~~~~~~~~~~~~~~
8801
William A. Kennington IIIac69b482021-06-02 12:28:27 -07008802There is a single instance attribute, which is ``target``. The ``target``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008803instance attribute is identical to the class attribute of the same name,
8804which is described in the previous section. This attribute exists as
8805both an instance and class attribute so tests can use
8806``self.target.run(cmd)`` in instance methods instead of
8807``oeRuntimeTest.tc.target.run(cmd)``.
8808
8809Installing Packages in the DUT Without the Package Manager
8810----------------------------------------------------------
8811
8812When a test requires a package built by BitBake, it is possible to
8813install that package. Installing the package does not require a package
8814manager be installed in the device under test (DUT). It does, however,
8815require an SSH connection and the target must be using the
8816``sshcontrol`` class.
8817
8818.. note::
8819
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008820 This method uses ``scp`` to copy files from the host to the target, which
8821 causes permissions and special attributes to be lost.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008822
8823A JSON file is used to define the packages needed by a test. This file
8824must be in the same path as the file used to define the tests.
8825Furthermore, the filename must map directly to the test module name with
8826a ``.json`` extension.
8827
8828The JSON file must include an object with the test name as keys of an
8829object or an array. This object (or array of objects) uses the following
8830data:
8831
8832- "pkg" - A mandatory string that is the name of the package to be
8833 installed.
8834
8835- "rm" - An optional boolean, which defaults to "false", that specifies
8836 to remove the package after the test.
8837
8838- "extract" - An optional boolean, which defaults to "false", that
8839 specifies if the package must be extracted from the package format.
8840 When set to "true", the package is not automatically installed into
8841 the DUT.
8842
8843Following is an example JSON file that handles test "foo" installing
8844package "bar" and test "foobar" installing packages "foo" and "bar".
8845Once the test is complete, the packages are removed from the DUT.
8846::
8847
8848 {
8849 "foo": {
8850 "pkg": "bar"
8851 },
8852 "foobar": [
8853 {
8854 "pkg": "foo",
8855 "rm": true
8856 },
8857 {
8858 "pkg": "bar",
8859 "rm": true
8860 }
8861 ]
8862 }
8863
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008864Debugging Tools and Techniques
8865==============================
8866
8867The exact method for debugging build failures depends on the nature of
8868the problem and on the system's area from which the bug originates.
8869Standard debugging practices such as comparison against the last known
8870working version with examination of the changes and the re-application
8871of steps to identify the one causing the problem are valid for the Yocto
8872Project just as they are for any other system. Even though it is
8873impossible to detail every possible potential failure, this section
8874provides some general tips to aid in debugging given a variety of
8875situations.
8876
8877.. note::
8878
8879 A useful feature for debugging is the error reporting tool.
8880 Configuring the Yocto Project to use this tool causes the
8881 OpenEmbedded build system to produce error reporting commands as part
8882 of the console output. You can enter the commands after the build
8883 completes to log error information into a common database, that can
8884 help you figure out what might be going wrong. For information on how
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008885 to enable and use this feature, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06008886 ":ref:`dev-manual/common-tasks:using the error reporting tool`"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008887 section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008888
8889The following list shows the debugging topics in the remainder of this
8890section:
8891
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05008892- ":ref:`dev-manual/common-tasks:viewing logs from failed tasks`" describes
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008893 how to find and view logs from tasks that failed during the build
8894 process.
8895
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05008896- ":ref:`dev-manual/common-tasks:viewing variable values`" describes how to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008897 use the BitBake ``-e`` option to examine variable values after a
8898 recipe has been parsed.
8899
Andrew Geissler09209ee2020-12-13 08:44:15 -06008900- ":ref:`dev-manual/common-tasks:viewing package information with \`\`oe-pkgdata-util\`\``"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008901 describes how to use the ``oe-pkgdata-util`` utility to query
8902 :term:`PKGDATA_DIR` and
8903 display package-related information for built packages.
8904
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05008905- ":ref:`dev-manual/common-tasks:viewing dependencies between recipes and tasks`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008906 describes how to use the BitBake ``-g`` option to display recipe
8907 dependency information used during the build.
8908
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05008909- ":ref:`dev-manual/common-tasks:viewing task variable dependencies`" describes
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008910 how to use the ``bitbake-dumpsig`` command in conjunction with key
8911 subdirectories in the
8912 :term:`Build Directory` to determine
8913 variable dependencies.
8914
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05008915- ":ref:`dev-manual/common-tasks:running specific tasks`" describes
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008916 how to use several BitBake options (e.g. ``-c``, ``-C``, and ``-f``)
8917 to run specific tasks in the build chain. It can be useful to run
8918 tasks "out-of-order" when trying isolate build issues.
8919
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05008920- ":ref:`dev-manual/common-tasks:general bitbake problems`" describes how
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008921 to use BitBake's ``-D`` debug output option to reveal more about what
8922 BitBake is doing during the build.
8923
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05008924- ":ref:`dev-manual/common-tasks:building with no dependencies`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008925 describes how to use the BitBake ``-b`` option to build a recipe
8926 while ignoring dependencies.
8927
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05008928- ":ref:`dev-manual/common-tasks:recipe logging mechanisms`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008929 describes how to use the many recipe logging functions to produce
8930 debugging output and report errors and warnings.
8931
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05008932- ":ref:`dev-manual/common-tasks:debugging parallel make races`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008933 describes how to debug situations where the build consists of several
8934 parts that are run simultaneously and when the output or result of
8935 one part is not ready for use with a different part of the build that
8936 depends on that output.
8937
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05008938- ":ref:`dev-manual/common-tasks:debugging with the gnu project debugger (gdb) remotely`"
8939 describes how to use GDB to allow you to examine running programs, which can
8940 help you fix problems.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008941
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05008942- ":ref:`dev-manual/common-tasks:debugging with the gnu project debugger (gdb) on the target`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008943 describes how to use GDB directly on target hardware for debugging.
8944
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05008945- ":ref:`dev-manual/common-tasks:other debugging tips`" describes
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008946 miscellaneous debugging tips that can be useful.
8947
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008948Viewing Logs from Failed Tasks
8949------------------------------
8950
8951You can find the log for a task in the file
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008952``${``\ :term:`WORKDIR`\ ``}/temp/log.do_``\ `taskname`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008953For example, the log for the
8954:ref:`ref-tasks-compile` task of the
8955QEMU minimal image for the x86 machine (``qemux86``) might be in
8956``tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/temp/log.do_compile``.
8957To see the commands :term:`BitBake` ran
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008958to generate a log, look at the corresponding ``run.do_``\ `taskname` file
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008959in the same directory.
8960
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008961``log.do_``\ `taskname` and ``run.do_``\ `taskname` are actually symbolic
8962links to ``log.do_``\ `taskname`\ ``.``\ `pid` and
8963``log.run_``\ `taskname`\ ``.``\ `pid`, where `pid` is the PID the task had
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008964when it ran. The symlinks always point to the files corresponding to the
8965most recent run.
8966
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008967Viewing Variable Values
8968-----------------------
8969
8970Sometimes you need to know the value of a variable as a result of
8971BitBake's parsing step. This could be because some unexpected behavior
8972occurred in your project. Perhaps an attempt to :ref:`modify a variable
8973<bitbake:bitbake-user-manual/bitbake-user-manual-metadata:modifying existing
8974variables>` did not work out as expected.
8975
8976BitBake's ``-e`` option is used to display variable values after
8977parsing. The following command displays the variable values after the
8978configuration files (i.e. ``local.conf``, ``bblayers.conf``,
Andrew Geisslerc926e172021-05-07 16:11:35 -05008979``bitbake.conf`` and so forth) have been parsed::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008980
8981 $ bitbake -e
8982
8983The following command displays variable values after a specific recipe has
Andrew Geisslerc926e172021-05-07 16:11:35 -05008984been parsed. The variables include those from the configuration as well::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008985
8986 $ bitbake -e recipename
8987
8988.. note::
8989
8990 Each recipe has its own private set of variables (datastore).
8991 Internally, after parsing the configuration, a copy of the resulting
8992 datastore is made prior to parsing each recipe. This copying implies
8993 that variables set in one recipe will not be visible to other
8994 recipes.
8995
8996 Likewise, each task within a recipe gets a private datastore based on
8997 the recipe datastore, which means that variables set within one task
8998 will not be visible to other tasks.
8999
9000In the output of ``bitbake -e``, each variable is preceded by a
9001description of how the variable got its value, including temporary
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009002values that were later overridden. This description also includes
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009003variable flags (varflags) set on the variable. The output can be very
9004helpful during debugging.
9005
9006Variables that are exported to the environment are preceded by
Andrew Geisslerc926e172021-05-07 16:11:35 -05009007``export`` in the output of ``bitbake -e``. See the following example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009008
9009 export CC="i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/ulf/poky/build/tmp/sysroots/qemux86"
9010
9011In addition to variable values, the output of the ``bitbake -e`` and
9012``bitbake -e`` recipe commands includes the following information:
9013
9014- The output starts with a tree listing all configuration files and
9015 classes included globally, recursively listing the files they include
9016 or inherit in turn. Much of the behavior of the OpenEmbedded build
Andrew Geissler09209ee2020-12-13 08:44:15 -06009017 system (including the behavior of the :ref:`ref-manual/tasks:normal recipe build tasks`) is
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009018 implemented in the
9019 :ref:`base <ref-classes-base>` class and the
9020 classes it inherits, rather than being built into BitBake itself.
9021
9022- After the variable values, all functions appear in the output. For
9023 shell functions, variables referenced within the function body are
9024 expanded. If a function has been modified using overrides or using
Patrick Williams0ca19cc2021-08-16 14:03:13 -05009025 override-style operators like ``:append`` and ``:prepend``, then the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009026 final assembled function body appears in the output.
9027
9028Viewing Package Information with ``oe-pkgdata-util``
9029----------------------------------------------------
9030
9031You can use the ``oe-pkgdata-util`` command-line utility to query
9032:term:`PKGDATA_DIR` and display
9033various package-related information. When you use the utility, you must
9034use it to view information on packages that have already been built.
9035
9036Following are a few of the available ``oe-pkgdata-util`` subcommands.
9037
9038.. note::
9039
9040 You can use the standard \* and ? globbing wildcards as part of
9041 package names and paths.
9042
9043- ``oe-pkgdata-util list-pkgs [pattern]``: Lists all packages
9044 that have been built, optionally limiting the match to packages that
9045 match pattern.
9046
9047- ``oe-pkgdata-util list-pkg-files package ...``: Lists the
9048 files and directories contained in the given packages.
9049
9050 .. note::
9051
9052 A different way to view the contents of a package is to look at
9053 the
9054 ``${``\ :term:`WORKDIR`\ ``}/packages-split``
9055 directory of the recipe that generates the package. This directory
9056 is created by the
9057 :ref:`ref-tasks-package` task
9058 and has one subdirectory for each package the recipe generates,
9059 which contains the files stored in that package.
9060
9061 If you want to inspect the ``${WORKDIR}/packages-split``
9062 directory, make sure that
9063 :ref:`rm_work <ref-classes-rm-work>` is not
9064 enabled when you build the recipe.
9065
9066- ``oe-pkgdata-util find-path path ...``: Lists the names of
9067 the packages that contain the given paths. For example, the following
9068 tells us that ``/usr/share/man/man1/make.1`` is contained in the
Andrew Geisslerc926e172021-05-07 16:11:35 -05009069 ``make-doc`` package::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009070
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009071 $ oe-pkgdata-util find-path /usr/share/man/man1/make.1
9072 make-doc: /usr/share/man/man1/make.1
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009073
9074- ``oe-pkgdata-util lookup-recipe package ...``: Lists the name
9075 of the recipes that produce the given packages.
9076
9077For more information on the ``oe-pkgdata-util`` command, use the help
Andrew Geisslerc926e172021-05-07 16:11:35 -05009078facility::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009079
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009080 $ oe-pkgdata-util --help
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009081 $ oe-pkgdata-util subcommand --help
9082
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009083Viewing Dependencies Between Recipes and Tasks
9084----------------------------------------------
9085
9086Sometimes it can be hard to see why BitBake wants to build other recipes
9087before the one you have specified. Dependency information can help you
9088understand why a recipe is built.
9089
9090To generate dependency information for a recipe, run the following
Andrew Geisslerc926e172021-05-07 16:11:35 -05009091command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009092
9093 $ bitbake -g recipename
9094
9095This command writes the following files in the current directory:
9096
9097- ``pn-buildlist``: A list of recipes/targets involved in building
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009098 `recipename`. "Involved" here means that at least one task from the
9099 recipe needs to run when building `recipename` from scratch. Targets
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009100 that are in
9101 :term:`ASSUME_PROVIDED`
9102 are not listed.
9103
9104- ``task-depends.dot``: A graph showing dependencies between tasks.
9105
9106The graphs are in
9107`DOT <https://en.wikipedia.org/wiki/DOT_%28graph_description_language%29>`__
9108format and can be converted to images (e.g. using the ``dot`` tool from
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009109`Graphviz <https://www.graphviz.org/>`__).
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009110
9111.. note::
9112
9113 - DOT files use a plain text format. The graphs generated using the
9114 ``bitbake -g`` command are often so large as to be difficult to
9115 read without special pruning (e.g. with Bitbake's ``-I`` option)
9116 and processing. Despite the form and size of the graphs, the
9117 corresponding ``.dot`` files can still be possible to read and
9118 provide useful information.
9119
9120 As an example, the ``task-depends.dot`` file contains lines such
Andrew Geisslerc926e172021-05-07 16:11:35 -05009121 as the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009122
9123 "libxslt.do_configure" -> "libxml2.do_populate_sysroot"
9124
9125 The above example line reveals that the
9126 :ref:`ref-tasks-configure`
9127 task in ``libxslt`` depends on the
9128 :ref:`ref-tasks-populate_sysroot`
9129 task in ``libxml2``, which is a normal
9130 :term:`DEPENDS` dependency
9131 between the two recipes.
9132
9133 - For an example of how ``.dot`` files can be processed, see the
9134 ``scripts/contrib/graph-tool`` Python script, which finds and
9135 displays paths between graph nodes.
9136
9137You can use a different method to view dependency information by using
Andrew Geisslerc926e172021-05-07 16:11:35 -05009138the following command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009139
9140 $ bitbake -g -u taskexp recipename
9141
9142This command
9143displays a GUI window from which you can view build-time and runtime
9144dependencies for the recipes involved in building recipename.
9145
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009146Viewing Task Variable Dependencies
9147----------------------------------
9148
9149As mentioned in the
9150":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-execution:checksums (signatures)`" section of the BitBake
9151User Manual, BitBake tries to automatically determine what variables a
9152task depends on so that it can rerun the task if any values of the
9153variables change. This determination is usually reliable. However, if
9154you do things like construct variable names at runtime, then you might
9155have to manually declare dependencies on those variables using
9156``vardeps`` as described in the
9157":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags`" section of the BitBake
9158User Manual.
9159
9160If you are unsure whether a variable dependency is being picked up
9161automatically for a given task, you can list the variable dependencies
9162BitBake has determined by doing the following:
9163
Andrew Geisslerc926e172021-05-07 16:11:35 -050091641. Build the recipe containing the task::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009165
9166 $ bitbake recipename
9167
91682. Inside the :term:`STAMPS_DIR`
9169 directory, find the signature data (``sigdata``) file that
9170 corresponds to the task. The ``sigdata`` files contain a pickled
9171 Python database of all the metadata that went into creating the input
9172 checksum for the task. As an example, for the
9173 :ref:`ref-tasks-fetch` task of the
9174 ``db`` recipe, the ``sigdata`` file might be found in the following
Andrew Geisslerc926e172021-05-07 16:11:35 -05009175 location::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009176
9177 ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1
9178
9179 For tasks that are accelerated through the shared state
Andrew Geissler09209ee2020-12-13 08:44:15 -06009180 (:ref:`sstate <overview-manual/concepts:shared state cache>`) cache, an
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009181 additional ``siginfo`` file is written into
9182 :term:`SSTATE_DIR` along with
9183 the cached task output. The ``siginfo`` files contain exactly the
9184 same information as ``sigdata`` files.
9185
91863. Run ``bitbake-dumpsig`` on the ``sigdata`` or ``siginfo`` file. Here
Andrew Geisslerc926e172021-05-07 16:11:35 -05009187 is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009188
9189 $ bitbake-dumpsig ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1
9190
9191 In the output of the above command, you will find a line like the
9192 following, which lists all the (inferred) variable dependencies for
9193 the task. This list also includes indirect dependencies from
9194 variables depending on other variables, recursively.
9195 ::
9196
9197 Task dependencies: ['PV', 'SRCREV', 'SRC_URI', 'SRC_URI[md5sum]', 'SRC_URI[sha256sum]', 'base_do_fetch']
9198
9199 .. note::
9200
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009201 Functions (e.g. ``base_do_fetch``) also count as variable dependencies.
9202 These functions in turn depend on the variables they reference.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009203
9204 The output of ``bitbake-dumpsig`` also includes the value each
9205 variable had, a list of dependencies for each variable, and
Patrick Williams213cb262021-08-07 19:21:33 -05009206 :term:`BB_HASHBASE_WHITELIST`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009207 information.
9208
9209There is also a ``bitbake-diffsigs`` command for comparing two
9210``siginfo`` or ``sigdata`` files. This command can be helpful when
9211trying to figure out what changed between two versions of a task. If you
9212call ``bitbake-diffsigs`` with just one file, the command behaves like
9213``bitbake-dumpsig``.
9214
9215You can also use BitBake to dump out the signature construction
9216information without executing tasks by using either of the following
Andrew Geisslerc926e172021-05-07 16:11:35 -05009217BitBake command-line options::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009218
9219 ‐‐dump-signatures=SIGNATURE_HANDLER
9220 -S SIGNATURE_HANDLER
9221
9222
9223.. note::
9224
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009225 Two common values for `SIGNATURE_HANDLER` are "none" and "printdiff", which
9226 dump only the signature or compare the dumped signature with the cached one,
9227 respectively.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009228
9229Using BitBake with either of these options causes BitBake to dump out
9230``sigdata`` files in the ``stamps`` directory for every task it would
9231have executed instead of building the specified target package.
9232
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009233Viewing Metadata Used to Create the Input Signature of a Shared State Task
9234--------------------------------------------------------------------------
9235
9236Seeing what metadata went into creating the input signature of a shared
9237state (sstate) task can be a useful debugging aid. This information is
9238available in signature information (``siginfo``) files in
9239:term:`SSTATE_DIR`. For
9240information on how to view and interpret information in ``siginfo``
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05009241files, see the
9242":ref:`dev-manual/common-tasks:viewing task variable dependencies`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009243
9244For conceptual information on shared state, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06009245":ref:`overview-manual/concepts:shared state`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009246section in the Yocto Project Overview and Concepts Manual.
9247
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009248Invalidating Shared State to Force a Task to Run
9249------------------------------------------------
9250
9251The OpenEmbedded build system uses
Andrew Geissler09209ee2020-12-13 08:44:15 -06009252:ref:`checksums <overview-manual/concepts:checksums (signatures)>` and
9253:ref:`overview-manual/concepts:shared state` cache to avoid unnecessarily
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009254rebuilding tasks. Collectively, this scheme is known as "shared state
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009255code".
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009256
9257As with all schemes, this one has some drawbacks. It is possible that
9258you could make implicit changes to your code that the checksum
9259calculations do not take into account. These implicit changes affect a
9260task's output but do not trigger the shared state code into rebuilding a
9261recipe. Consider an example during which a tool changes its output.
9262Assume that the output of ``rpmdeps`` changes. The result of the change
9263should be that all the ``package`` and ``package_write_rpm`` shared
9264state cache items become invalid. However, because the change to the
9265output is external to the code and therefore implicit, the associated
9266shared state cache items do not become invalidated. In this case, the
9267build process uses the cached items rather than running the task again.
9268Obviously, these types of implicit changes can cause problems.
9269
9270To avoid these problems during the build, you need to understand the
9271effects of any changes you make. Realize that changes you make directly
9272to a function are automatically factored into the checksum calculation.
9273Thus, these explicit changes invalidate the associated area of shared
9274state cache. However, you need to be aware of any implicit changes that
9275are not obvious changes to the code and could affect the output of a
9276given task.
9277
9278When you identify an implicit change, you can easily take steps to
9279invalidate the cache and force the tasks to run. The steps you can take
9280are as simple as changing a function's comments in the source code. For
9281example, to invalidate package shared state files, change the comment
9282statements of
9283:ref:`ref-tasks-package` or the
9284comments of one of the functions it calls. Even though the change is
9285purely cosmetic, it causes the checksum to be recalculated and forces
9286the build system to run the task again.
9287
9288.. note::
9289
9290 For an example of a commit that makes a cosmetic change to invalidate
9291 shared state, see this
Andrew Geissler09209ee2020-12-13 08:44:15 -06009292 :yocto_git:`commit </poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009293
9294Running Specific Tasks
9295----------------------
9296
9297Any given recipe consists of a set of tasks. The standard BitBake
9298behavior in most cases is: ``do_fetch``, ``do_unpack``, ``do_patch``,
9299``do_configure``, ``do_compile``, ``do_install``, ``do_package``,
9300``do_package_write_*``, and ``do_build``. The default task is
9301``do_build`` and any tasks on which it depends build first. Some tasks,
9302such as ``do_devshell``, are not part of the default build chain. If you
9303wish to run a task that is not part of the default build chain, you can
Andrew Geisslerc926e172021-05-07 16:11:35 -05009304use the ``-c`` option in BitBake. Here is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009305
9306 $ bitbake matchbox-desktop -c devshell
9307
9308The ``-c`` option respects task dependencies, which means that all other
9309tasks (including tasks from other recipes) that the specified task
9310depends on will be run before the task. Even when you manually specify a
9311task to run with ``-c``, BitBake will only run the task if it considers
9312it "out of date". See the
Andrew Geissler09209ee2020-12-13 08:44:15 -06009313":ref:`overview-manual/concepts:stamp files and the rerunning of tasks`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009314section in the Yocto Project Overview and Concepts Manual for how
9315BitBake determines whether a task is "out of date".
9316
9317If you want to force an up-to-date task to be rerun (e.g. because you
9318made manual modifications to the recipe's
9319:term:`WORKDIR` that you want to try
9320out), then you can use the ``-f`` option.
9321
9322.. note::
9323
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009324 The reason ``-f`` is never required when running the
9325 :ref:`ref-tasks-devshell` task is because the
9326 [\ :ref:`nostamp <bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ]
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009327 variable flag is already set for the task.
9328
Andrew Geisslerc926e172021-05-07 16:11:35 -05009329The following example shows one way you can use the ``-f`` option::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009330
9331 $ bitbake matchbox-desktop
9332 .
9333 .
9334 make some changes to the source code in the work directory
9335 .
9336 .
9337 $ bitbake matchbox-desktop -c compile -f
9338 $ bitbake matchbox-desktop
9339
9340This sequence first builds and then recompiles ``matchbox-desktop``. The
9341last command reruns all tasks (basically the packaging tasks) after the
9342compile. BitBake recognizes that the ``do_compile`` task was rerun and
9343therefore understands that the other tasks also need to be run again.
9344
9345Another, shorter way to rerun a task and all
Andrew Geissler09209ee2020-12-13 08:44:15 -06009346:ref:`ref-manual/tasks:normal recipe build tasks`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009347that depend on it is to use the ``-C`` option.
9348
9349.. note::
9350
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009351 This option is upper-cased and is separate from the ``-c``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009352 option, which is lower-cased.
9353
9354Using this option invalidates the given task and then runs the
9355:ref:`ref-tasks-build` task, which is
9356the default task if no task is given, and the tasks on which it depends.
9357You could replace the final two commands in the previous example with
Andrew Geisslerc926e172021-05-07 16:11:35 -05009358the following single command::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009359
9360 $ bitbake matchbox-desktop -C compile
9361
9362Internally, the ``-f`` and ``-C`` options work by tainting (modifying)
9363the input checksum of the specified task. This tainting indirectly
9364causes the task and its dependent tasks to be rerun through the normal
9365task dependency mechanisms.
9366
9367.. note::
9368
9369 BitBake explicitly keeps track of which tasks have been tainted in
9370 this fashion, and will print warnings such as the following for
9371 builds involving such tasks:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009372
9373 .. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009374
9375 WARNING: /home/ulf/poky/meta/recipes-sato/matchbox-desktop/matchbox-desktop_2.1.bb.do_compile is tainted from a forced run
9376
9377
9378 The purpose of the warning is to let you know that the work directory
9379 and build output might not be in the clean state they would be in for
9380 a "normal" build, depending on what actions you took. To get rid of
9381 such warnings, you can remove the work directory and rebuild the
Andrew Geisslerc926e172021-05-07 16:11:35 -05009382 recipe, as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009383
9384 $ bitbake matchbox-desktop -c clean
9385 $ bitbake matchbox-desktop
9386
9387
9388You can view a list of tasks in a given package by running the
Andrew Geisslerc926e172021-05-07 16:11:35 -05009389``do_listtasks`` task as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009390
9391 $ bitbake matchbox-desktop -c listtasks
9392
9393The results appear as output to the console and are also in
9394the file ``${WORKDIR}/temp/log.do_listtasks``.
9395
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009396General BitBake Problems
9397------------------------
9398
9399You can see debug output from BitBake by using the ``-D`` option. The
9400debug output gives more information about what BitBake is doing and the
9401reason behind it. Each ``-D`` option you use increases the logging
9402level. The most common usage is ``-DDD``.
9403
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009404The output from ``bitbake -DDD -v targetname`` can reveal why BitBake
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009405chose a certain version of a package or why BitBake picked a certain
9406provider. This command could also help you in a situation where you
9407think BitBake did something unexpected.
9408
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009409Building with No Dependencies
9410-----------------------------
9411
9412To build a specific recipe (``.bb`` file), you can use the following
Andrew Geisslerc926e172021-05-07 16:11:35 -05009413command form::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009414
9415 $ bitbake -b somepath/somerecipe.bb
9416
9417This command form does
9418not check for dependencies. Consequently, you should use it only when
9419you know existing dependencies have been met.
9420
9421.. note::
9422
9423 You can also specify fragments of the filename. In this case, BitBake
9424 checks for a unique match.
9425
9426Recipe Logging Mechanisms
9427-------------------------
9428
9429The Yocto Project provides several logging functions for producing
9430debugging output and reporting errors and warnings. For Python
William A. Kennington IIIac69b482021-06-02 12:28:27 -07009431functions, the following logging functions are available. All of these functions
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009432log to ``${T}/log.do_``\ `task`, and can also log to standard output
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009433(stdout) with the right settings:
9434
9435- ``bb.plain(msg)``: Writes msg as is to the log while also
9436 logging to stdout.
9437
9438- ``bb.note(msg)``: Writes "NOTE: msg" to the log. Also logs to
9439 stdout if BitBake is called with "-v".
9440
9441- ``bb.debug(level, msg)``: Writes "DEBUG: msg" to the
9442 log. Also logs to stdout if the log level is greater than or equal to
Patrick Williams213cb262021-08-07 19:21:33 -05009443 level. See the ":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-intro:usage and syntax`" option
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009444 in the BitBake User Manual for more information.
9445
9446- ``bb.warn(msg)``: Writes "WARNING: msg" to the log while also
9447 logging to stdout.
9448
9449- ``bb.error(msg)``: Writes "ERROR: msg" to the log while also
9450 logging to standard out (stdout).
9451
9452 .. note::
9453
9454 Calling this function does not cause the task to fail.
9455
Andrew Geisslereff27472021-10-29 15:35:00 -05009456- ``bb.fatal(msg)``: This logging function is similar to
9457 ``bb.error(msg)`` but also causes the calling task to fail.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009458
9459 .. note::
9460
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009461 ``bb.fatal()`` raises an exception, which means you do not need to put a
9462 "return" statement after the function.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009463
9464The same logging functions are also available in shell functions, under
9465the names ``bbplain``, ``bbnote``, ``bbdebug``, ``bbwarn``, ``bberror``,
9466and ``bbfatal``. The
9467:ref:`logging <ref-classes-logging>` class
9468implements these functions. See that class in the ``meta/classes``
9469folder of the :term:`Source Directory` for information.
9470
9471Logging With Python
9472~~~~~~~~~~~~~~~~~~~
9473
9474When creating recipes using Python and inserting code that handles build
9475logs, keep in mind the goal is to have informative logs while keeping
9476the console as "silent" as possible. Also, if you want status messages
9477in the log, use the "debug" loglevel.
9478
9479Following is an example written in Python. The code handles logging for
9480a function that determines the number of tasks needed to be run. See the
9481":ref:`ref-tasks-listtasks`"
Andrew Geisslerc926e172021-05-07 16:11:35 -05009482section for additional information::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009483
9484 python do_listtasks() {
9485 bb.debug(2, "Starting to figure out the task list")
9486 if noteworthy_condition:
9487 bb.note("There are 47 tasks to run")
9488 bb.debug(2, "Got to point xyz")
9489 if warning_trigger:
9490 bb.warn("Detected warning_trigger, this might be a problem later.")
9491 if recoverable_error:
9492 bb.error("Hit recoverable_error, you really need to fix this!")
9493 if fatal_error:
9494 bb.fatal("fatal_error detected, unable to print the task list")
9495 bb.plain("The tasks present are abc")
9496 bb.debug(2, "Finished figuring out the tasklist")
9497 }
9498
9499Logging With Bash
9500~~~~~~~~~~~~~~~~~
9501
9502When creating recipes using Bash and inserting code that handles build
9503logs, you have the same goals - informative with minimal console output.
9504The syntax you use for recipes written in Bash is similar to that of
9505recipes written in Python described in the previous section.
9506
9507Following is an example written in Bash. The code logs the progress of
9508the ``do_my_function`` function.
9509::
9510
9511 do_my_function() {
9512 bbdebug 2 "Running do_my_function"
9513 if [ exceptional_condition ]; then
9514 bbnote "Hit exceptional_condition"
9515 fi
9516 bbdebug 2 "Got to point xyz"
9517 if [ warning_trigger ]; then
9518 bbwarn "Detected warning_trigger, this might cause a problem later."
9519 fi
9520 if [ recoverable_error ]; then
9521 bberror "Hit recoverable_error, correcting"
9522 fi
9523 if [ fatal_error ]; then
9524 bbfatal "fatal_error detected"
9525 fi
9526 bbdebug 2 "Completed do_my_function"
9527 }
9528
9529
9530Debugging Parallel Make Races
9531-----------------------------
9532
9533A parallel ``make`` race occurs when the build consists of several parts
9534that are run simultaneously and a situation occurs when the output or
9535result of one part is not ready for use with a different part of the
9536build that depends on that output. Parallel make races are annoying and
William A. Kennington IIIac69b482021-06-02 12:28:27 -07009537can sometimes be difficult to reproduce and fix. However, there are some simple
9538tips and tricks that can help you debug and fix them. This section
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009539presents a real-world example of an error encountered on the Yocto
9540Project autobuilder and the process used to fix it.
9541
9542.. note::
9543
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009544 If you cannot properly fix a ``make`` race condition, you can work around it
9545 by clearing either the :term:`PARALLEL_MAKE` or :term:`PARALLEL_MAKEINST`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009546 variables.
9547
9548The Failure
9549~~~~~~~~~~~
9550
9551For this example, assume that you are building an image that depends on
9552the "neard" package. And, during the build, BitBake runs into problems
9553and creates the following output.
9554
9555.. note::
9556
9557 This example log file has longer lines artificially broken to make
9558 the listing easier to read.
9559
9560If you examine the output or the log file, you see the failure during
9561``make``:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009562
9563.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009564
9565 | DEBUG: SITE files ['endian-little', 'bit-32', 'ix86-common', 'common-linux', 'common-glibc', 'i586-linux', 'common']
9566 | DEBUG: Executing shell function do_compile
9567 | NOTE: make -j 16
9568 | make --no-print-directory all-am
9569 | /bin/mkdir -p include/near
9570 | /bin/mkdir -p include/near
9571 | /bin/mkdir -p include/near
9572 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9573 0.14-r0/neard-0.14/include/types.h include/near/types.h
9574 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9575 0.14-r0/neard-0.14/include/log.h include/near/log.h
9576 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9577 0.14-r0/neard-0.14/include/plugin.h include/near/plugin.h
9578 | /bin/mkdir -p include/near
9579 | /bin/mkdir -p include/near
9580 | /bin/mkdir -p include/near
9581 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9582 0.14-r0/neard-0.14/include/tag.h include/near/tag.h
9583 | /bin/mkdir -p include/near
9584 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9585 0.14-r0/neard-0.14/include/adapter.h include/near/adapter.h
9586 | /bin/mkdir -p include/near
9587 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9588 0.14-r0/neard-0.14/include/ndef.h include/near/ndef.h
9589 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9590 0.14-r0/neard-0.14/include/tlv.h include/near/tlv.h
9591 | /bin/mkdir -p include/near
9592 | /bin/mkdir -p include/near
9593 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9594 0.14-r0/neard-0.14/include/setting.h include/near/setting.h
9595 | /bin/mkdir -p include/near
9596 | /bin/mkdir -p include/near
9597 | /bin/mkdir -p include/near
9598 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9599 0.14-r0/neard-0.14/include/device.h include/near/device.h
9600 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9601 0.14-r0/neard-0.14/include/nfc_copy.h include/near/nfc_copy.h
9602 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9603 0.14-r0/neard-0.14/include/snep.h include/near/snep.h
9604 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9605 0.14-r0/neard-0.14/include/version.h include/near/version.h
9606 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9607 0.14-r0/neard-0.14/include/dbus.h include/near/dbus.h
9608 | ./src/genbuiltin nfctype1 nfctype2 nfctype3 nfctype4 p2p > src/builtin.h
9609 | i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/
9610 build/build/tmp/sysroots/qemux86 -DHAVE_CONFIG_H -I. -I./include -I./src -I./gdbus -I/home/pokybuild/
9611 yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/sysroots/qemux86/usr/include/glib-2.0
9612 -I/home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/sysroots/qemux86/usr/
9613 lib/glib-2.0/include -I/home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/
9614 tmp/sysroots/qemux86/usr/include/dbus-1.0 -I/home/pokybuild/yocto-autobuilder/yocto-slave/
9615 nightly-x86/build/build/tmp/sysroots/qemux86/usr/lib/dbus-1.0/include -I/home/pokybuild/yocto-autobuilder/
9616 yocto-slave/nightly-x86/build/build/tmp/sysroots/qemux86/usr/include/libnl3
9617 -DNEAR_PLUGIN_BUILTIN -DPLUGINDIR=\""/usr/lib/near/plugins"\"
9618 -DCONFIGDIR=\""/etc/neard\"" -O2 -pipe -g -feliminate-unused-debug-types -c
9619 -o tools/snep-send.o tools/snep-send.c
9620 | In file included from tools/snep-send.c:16:0:
9621 | tools/../src/near.h:41:23: fatal error: near/dbus.h: No such file or directory
9622 | #include <near/dbus.h>
9623 | ^
9624 | compilation terminated.
9625 | make[1]: *** [tools/snep-send.o] Error 1
9626 | make[1]: *** Waiting for unfinished jobs....
9627 | make: *** [all] Error 2
9628 | ERROR: oe_runmake failed
9629
9630Reproducing the Error
9631~~~~~~~~~~~~~~~~~~~~~
9632
9633Because race conditions are intermittent, they do not manifest
9634themselves every time you do the build. In fact, most times the build
9635will complete without problems even though the potential race condition
9636exists. Thus, once the error surfaces, you need a way to reproduce it.
9637
9638In this example, compiling the "neard" package is causing the problem.
9639So the first thing to do is build "neard" locally. Before you start the
9640build, set the
9641:term:`PARALLEL_MAKE` variable
9642in your ``local.conf`` file to a high number (e.g. "-j 20"). Using a
Andrew Geissler09036742021-06-25 14:25:14 -05009643high value for :term:`PARALLEL_MAKE` increases the chances of the race
Andrew Geisslerc926e172021-05-07 16:11:35 -05009644condition showing up::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009645
9646 $ bitbake neard
9647
Andrew Geisslerc926e172021-05-07 16:11:35 -05009648Once the local build for "neard" completes, start a ``devshell`` build::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009649
9650 $ bitbake neard -c devshell
9651
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05009652For information on how to use a ``devshell``, see the
9653":ref:`dev-manual/common-tasks:using a development shell`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009654
Andrew Geisslerc926e172021-05-07 16:11:35 -05009655In the ``devshell``, do the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009656
9657 $ make clean
9658 $ make tools/snep-send.o
9659
9660The ``devshell`` commands cause the failure to clearly
William A. Kennington IIIac69b482021-06-02 12:28:27 -07009661be visible. In this case, there is a missing dependency for the ``neard``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009662Makefile target. Here is some abbreviated, sample output with the
Andrew Geisslerc926e172021-05-07 16:11:35 -05009663missing dependency clearly visible at the end::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009664
9665 i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/scott-lenovo/......
9666 .
9667 .
9668 .
9669 tools/snep-send.c
9670 In file included from tools/snep-send.c:16:0:
9671 tools/../src/near.h:41:23: fatal error: near/dbus.h: No such file or directory
9672 #include <near/dbus.h>
9673 ^
9674 compilation terminated.
9675 make: *** [tools/snep-send.o] Error 1
9676 $
9677
9678
9679Creating a Patch for the Fix
9680~~~~~~~~~~~~~~~~~~~~~~~~~~~~
9681
9682Because there is a missing dependency for the Makefile target, you need
9683to patch the ``Makefile.am`` file, which is generated from
Andrew Geisslerc926e172021-05-07 16:11:35 -05009684``Makefile.in``. You can use Quilt to create the patch::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009685
9686 $ quilt new parallelmake.patch
9687 Patch patches/parallelmake.patch is now on top
9688 $ quilt add Makefile.am
9689 File Makefile.am added to patch patches/parallelmake.patch
9690
9691For more information on using Quilt, see the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05009692":ref:`dev-manual/common-tasks:using quilt in your workflow`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009693
9694At this point you need to make the edits to ``Makefile.am`` to add the
9695missing dependency. For our example, you have to add the following line
Andrew Geisslerc926e172021-05-07 16:11:35 -05009696to the file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009697
9698 tools/snep-send.$(OBJEXT): include/near/dbus.h
9699
9700Once you have edited the file, use the ``refresh`` command to create the
Andrew Geisslerc926e172021-05-07 16:11:35 -05009701patch::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009702
9703 $ quilt refresh
9704 Refreshed patch patches/parallelmake.patch
9705
William A. Kennington IIIac69b482021-06-02 12:28:27 -07009706Once the patch file is created, you need to add it back to the originating
9707recipe folder. Here is an example assuming a top-level
Andrew Geisslerc926e172021-05-07 16:11:35 -05009708:term:`Source Directory` named ``poky``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009709
9710 $ cp patches/parallelmake.patch poky/meta/recipes-connectivity/neard/neard
9711
9712The final thing you need to do to implement the fix in the build is to
9713update the "neard" recipe (i.e. ``neard-0.14.bb``) so that the
9714:term:`SRC_URI` statement includes
9715the patch file. The recipe file is in the folder above the patch. Here
Andrew Geissler09036742021-06-25 14:25:14 -05009716is what the edited :term:`SRC_URI` statement would look like::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009717
9718 SRC_URI = "${KERNELORG_MIRROR}/linux/network/nfc/${BPN}-${PV}.tar.xz \
9719 file://neard.in \
9720 file://neard.service.in \
9721 file://parallelmake.patch \
9722 "
9723
9724With the patch complete and moved to the correct folder and the
Andrew Geissler09036742021-06-25 14:25:14 -05009725:term:`SRC_URI` statement updated, you can exit the ``devshell``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009726
9727 $ exit
9728
9729Testing the Build
9730~~~~~~~~~~~~~~~~~
9731
9732With everything in place, you can get back to trying the build again
Andrew Geisslerc926e172021-05-07 16:11:35 -05009733locally::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009734
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009735 $ bitbake neard
9736
9737This build should succeed.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009738
9739Now you can open up a ``devshell`` again and repeat the clean and make
Andrew Geisslerc926e172021-05-07 16:11:35 -05009740operations as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009741
9742 $ bitbake neard -c devshell
9743 $ make clean
9744 $ make tools/snep-send.o
9745
9746The build should work without issue.
9747
9748As with all solved problems, if they originated upstream, you need to
9749submit the fix for the recipe in OE-Core and upstream so that the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -05009750problem is taken care of at its source. See the
9751":ref:`dev-manual/common-tasks:submitting a change to the yocto project`"
9752section for more information.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009753
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009754Debugging With the GNU Project Debugger (GDB) Remotely
9755------------------------------------------------------
9756
9757GDB allows you to examine running programs, which in turn helps you to
9758understand and fix problems. It also allows you to perform post-mortem
9759style analysis of program crashes. GDB is available as a package within
9760the Yocto Project and is installed in SDK images by default. See the
Andrew Geissler09209ee2020-12-13 08:44:15 -06009761":ref:`ref-manual/images:Images`" chapter in the Yocto
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009762Project Reference Manual for a description of these images. You can find
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009763information on GDB at https://sourceware.org/gdb/.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009764
9765.. note::
9766
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009767 For best results, install debug (``-dbg``) packages for the applications you
9768 are going to debug. Doing so makes extra debug symbols available that give
9769 you more meaningful output.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009770
9771Sometimes, due to memory or disk space constraints, it is not possible
9772to use GDB directly on the remote target to debug applications. These
9773constraints arise because GDB needs to load the debugging information
9774and the binaries of the process being debugged. Additionally, GDB needs
9775to perform many computations to locate information such as function
9776names, variable names and values, stack traces and so forth - even
9777before starting the debugging process. These extra computations place
9778more load on the target system and can alter the characteristics of the
9779program being debugged.
9780
Andrew Geissler95ac1b82021-03-31 14:34:31 -05009781To help get past the previously mentioned constraints, there are two
9782methods you can use: running a debuginfod server and using gdbserver.
9783
9784Using the debuginfod server method
9785~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
9786
Andrew Geisslerc926e172021-05-07 16:11:35 -05009787``debuginfod`` from ``elfutils`` is a way to distribute ``debuginfo`` files.
9788Running a ``debuginfod`` server makes debug symbols readily available,
Andrew Geissler95ac1b82021-03-31 14:34:31 -05009789which means you don't need to download debugging information
9790and the binaries of the process being debugged. You can just fetch
9791debug symbols from the server.
9792
Andrew Geisslerc926e172021-05-07 16:11:35 -05009793To run a ``debuginfod`` server, you need to do the following:
Andrew Geissler95ac1b82021-03-31 14:34:31 -05009794
Andrew Geisslerc926e172021-05-07 16:11:35 -05009795- Ensure that ``debuginfod`` is present in :term:`DISTRO_FEATURES`
9796 (it already is in ``OpenEmbedded-core`` defaults and ``poky`` reference distribution).
9797 If not, set in your distro config file or in ``local.conf``::
Andrew Geissler95ac1b82021-03-31 14:34:31 -05009798
Patrick Williams0ca19cc2021-08-16 14:03:13 -05009799 DISTRO_FEATURES:append = " debuginfod"
Andrew Geissler95ac1b82021-03-31 14:34:31 -05009800
Andrew Geisslerc926e172021-05-07 16:11:35 -05009801 This distro feature enables the server and client library in ``elfutils``,
9802 and enables ``debuginfod`` support in clients (at the moment, ``gdb`` and ``binutils``).
Andrew Geissler95ac1b82021-03-31 14:34:31 -05009803
Andrew Geisslerc926e172021-05-07 16:11:35 -05009804- Run the following commands to launch the ``debuginfod`` server on the host::
Andrew Geissler95ac1b82021-03-31 14:34:31 -05009805
9806 $ oe-debuginfod
9807
Andrew Geisslerc926e172021-05-07 16:11:35 -05009808- To use ``debuginfod`` on the target, you need to know the ip:port where
9809 ``debuginfod`` is listening on the host (port defaults to 8002), and export
9810 that into the shell environment, for example in ``qemu``::
Andrew Geissler95ac1b82021-03-31 14:34:31 -05009811
Andrew Geisslerc926e172021-05-07 16:11:35 -05009812 root@qemux86-64:~# export DEBUGINFOD_URLS="http://192.168.7.1:8002/"
Andrew Geissler95ac1b82021-03-31 14:34:31 -05009813
Andrew Geisslerc926e172021-05-07 16:11:35 -05009814- Then debug info fetching should simply work when running the target ``gdb``,
9815 ``readelf`` or ``objdump``, for example::
Andrew Geissler95ac1b82021-03-31 14:34:31 -05009816
Andrew Geisslerc926e172021-05-07 16:11:35 -05009817 root@qemux86-64:~# gdb /bin/cat
9818 ...
9819 Reading symbols from /bin/cat...
9820 Downloading separate debug info for /bin/cat...
9821 Reading symbols from /home/root/.cache/debuginfod_client/923dc4780cfbc545850c616bffa884b6b5eaf322/debuginfo...
Andrew Geissler95ac1b82021-03-31 14:34:31 -05009822
Andrew Geisslerc926e172021-05-07 16:11:35 -05009823- It's also possible to use ``debuginfod-find`` to just query the server::
Andrew Geissler95ac1b82021-03-31 14:34:31 -05009824
Andrew Geisslerc926e172021-05-07 16:11:35 -05009825 root@qemux86-64:~# debuginfod-find debuginfo /bin/ls
9826 /home/root/.cache/debuginfod_client/356edc585f7f82d46f94fcb87a86a3fe2d2e60bd/debuginfo
Andrew Geissler95ac1b82021-03-31 14:34:31 -05009827
Andrew Geissler95ac1b82021-03-31 14:34:31 -05009828
9829Using the gdbserver method
9830~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
9831
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009832gdbserver, which runs on the remote target and does not load any
9833debugging information from the debugged process. Instead, a GDB instance
9834processes the debugging information that is run on a remote computer -
9835the host GDB. The host GDB then sends control commands to gdbserver to
9836make it stop or start the debugged program, as well as read or write
9837memory regions of that debugged program. All the debugging information
9838loaded and processed as well as all the heavy debugging is done by the
9839host GDB. Offloading these processes gives the gdbserver running on the
9840target a chance to remain small and fast.
9841
9842Because the host GDB is responsible for loading the debugging
9843information and for doing the necessary processing to make actual
9844debugging happen, you have to make sure the host can access the
9845unstripped binaries complete with their debugging information and also
9846be sure the target is compiled with no optimizations. The host GDB must
9847also have local access to all the libraries used by the debugged
9848program. Because gdbserver does not need any local debugging
9849information, the binaries on the remote target can remain stripped.
9850However, the binaries must also be compiled without optimization so they
9851match the host's binaries.
9852
9853To remain consistent with GDB documentation and terminology, the binary
9854being debugged on the remote target machine is referred to as the
9855"inferior" binary. For documentation on GDB see the `GDB
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009856site <https://sourceware.org/gdb/documentation/>`__.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009857
9858The following steps show you how to debug using the GNU project
9859debugger.
9860
98611. *Configure your build system to construct the companion debug
9862 filesystem:*
9863
Andrew Geisslerc926e172021-05-07 16:11:35 -05009864 In your ``local.conf`` file, set the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009865
9866 IMAGE_GEN_DEBUGFS = "1"
9867 IMAGE_FSTYPES_DEBUGFS = "tar.bz2"
9868
9869 These options cause the
9870 OpenEmbedded build system to generate a special companion filesystem
9871 fragment, which contains the matching source and debug symbols to
9872 your deployable filesystem. The build system does this by looking at
9873 what is in the deployed filesystem, and pulling the corresponding
9874 ``-dbg`` packages.
9875
9876 The companion debug filesystem is not a complete filesystem, but only
9877 contains the debug fragments. This filesystem must be combined with
9878 the full filesystem for debugging. Subsequent steps in this procedure
9879 show how to combine the partial filesystem with the full filesystem.
9880
98812. *Configure the system to include gdbserver in the target filesystem:*
9882
9883 Make the following addition in either your ``local.conf`` file or in
Andrew Geisslerc926e172021-05-07 16:11:35 -05009884 an image recipe::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009885
Patrick Williams0ca19cc2021-08-16 14:03:13 -05009886 IMAGE_INSTALL:append = " gdbserver"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009887
9888 The change makes
9889 sure the ``gdbserver`` package is included.
9890
98913. *Build the environment:*
9892
9893 Use the following command to construct the image and the companion
Andrew Geisslerc926e172021-05-07 16:11:35 -05009894 Debug Filesystem::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009895
9896 $ bitbake image
9897
9898 Build the cross GDB component and
9899 make it available for debugging. Build the SDK that matches the
9900 image. Building the SDK is best for a production build that can be
Andrew Geisslerc926e172021-05-07 16:11:35 -05009901 used later for debugging, especially during long term maintenance::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009902
9903 $ bitbake -c populate_sdk image
9904
9905 Alternatively, you can build the minimal toolchain components that
9906 match the target. Doing so creates a smaller than typical SDK and
9907 only contains a minimal set of components with which to build simple
Andrew Geisslerc926e172021-05-07 16:11:35 -05009908 test applications, as well as run the debugger::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009909
9910 $ bitbake meta-toolchain
9911
Andrew Geisslerc926e172021-05-07 16:11:35 -05009912 A final method is to build Gdb itself within the build system::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009913
9914 $ bitbake gdb-cross-<architecture>
9915
9916 Doing so produces a temporary copy of
9917 ``cross-gdb`` you can use for debugging during development. While
9918 this is the quickest approach, the two previous methods in this step
9919 are better when considering long-term maintenance strategies.
9920
9921 .. note::
9922
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009923 If you run ``bitbake gdb-cross``, the OpenEmbedded build system suggests
9924 the actual image (e.g. ``gdb-cross-i586``). The suggestion is usually the
9925 actual name you want to use.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009926
Andrew Geissler4c19ea12020-10-27 13:52:24 -050099274. *Set up the* ``debugfs``\ *:*
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009928
Andrew Geisslerc926e172021-05-07 16:11:35 -05009929 Run the following commands to set up the ``debugfs``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009930
9931 $ mkdir debugfs
9932 $ cd debugfs
9933 $ tar xvfj build-dir/tmp-glibc/deploy/images/machine/image.rootfs.tar.bz2
9934 $ tar xvfj build-dir/tmp-glibc/deploy/images/machine/image-dbg.rootfs.tar.bz2
9935
Andrew Geissler4c19ea12020-10-27 13:52:24 -050099365. *Set up GDB:*
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009937
9938 Install the SDK (if you built one) and then source the correct
9939 environment file. Sourcing the environment file puts the SDK in your
9940 ``PATH`` environment variable.
9941
9942 If you are using the build system, Gdb is located in
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009943 `build-dir`\ ``/tmp/sysroots/``\ `host`\ ``/usr/bin/``\ `architecture`\ ``/``\ `architecture`\ ``-gdb``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009944
99456. *Boot the target:*
9946
9947 For information on how to run QEMU, see the `QEMU
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009948 Documentation <https://wiki.qemu.org/Documentation/GettingStartedDevelopers>`__.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009949
9950 .. note::
9951
9952 Be sure to verify that your host can access the target via TCP.
9953
99547. *Debug a program:*
9955
9956 Debugging a program involves running gdbserver on the target and then
9957 running Gdb on the host. The example in this step debugs ``gzip``:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009958
9959 .. code-block:: shell
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009960
9961 root@qemux86:~# gdbserver localhost:1234 /bin/gzip —help
9962
9963 For
9964 additional gdbserver options, see the `GDB Server
9965 Documentation <https://www.gnu.org/software/gdb/documentation/>`__.
9966
9967 After running gdbserver on the target, you need to run Gdb on the
Andrew Geisslerc926e172021-05-07 16:11:35 -05009968 host and configure it and connect to the target. Use these commands::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009969
9970 $ cd directory-holding-the-debugfs-directory
9971 $ arch-gdb
9972 (gdb) set sysroot debugfs
9973 (gdb) set substitute-path /usr/src/debug debugfs/usr/src/debug
9974 (gdb) target remote IP-of-target:1234
9975
9976 At this
9977 point, everything should automatically load (i.e. matching binaries,
9978 symbols and headers).
9979
9980 .. note::
9981
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009982 The Gdb ``set`` commands in the previous example can be placed into the
9983 users ``~/.gdbinit`` file. Upon starting, Gdb automatically runs whatever
9984 commands are in that file.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009985
99868. *Deploying without a full image rebuild:*
9987
9988 In many cases, during development you want a quick method to deploy a
9989 new binary to the target and debug it, without waiting for a full
9990 image build.
9991
9992 One approach to solving this situation is to just build the component
9993 you want to debug. Once you have built the component, copy the
9994 executable directly to both the target and the host ``debugfs``.
9995
9996 If the binary is processed through the debug splitting in
9997 OpenEmbedded, you should also copy the debug items (i.e. ``.debug``
9998 contents and corresponding ``/usr/src/debug`` files) from the work
Andrew Geisslerc926e172021-05-07 16:11:35 -05009999 directory. Here is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010000
10001 $ bitbake bash
10002 $ bitbake -c devshell bash
10003 $ cd ..
10004 $ scp packages-split/bash/bin/bash target:/bin/bash
10005 $ cp -a packages-split/bash-dbg/\* path/debugfs
10006
10007Debugging with the GNU Project Debugger (GDB) on the Target
10008-----------------------------------------------------------
10009
10010The previous section addressed using GDB remotely for debugging
10011purposes, which is the most usual case due to the inherent hardware
10012limitations on many embedded devices. However, debugging in the target
10013hardware itself is also possible with more powerful devices. This
10014section describes what you need to do in order to support using GDB to
10015debug on the target hardware.
10016
10017To support this kind of debugging, you need do the following:
10018
10019- Ensure that GDB is on the target. You can do this by adding "gdb" to
Andrew Geisslerc926e172021-05-07 16:11:35 -050010020 :term:`IMAGE_INSTALL`::
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010021
Patrick Williams0ca19cc2021-08-16 14:03:13 -050010022 IMAGE_INSTALL:append = " gdb"
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010023
Andrew Geisslerc926e172021-05-07 16:11:35 -050010024 Alternatively, you can add "tools-debug" to :term:`IMAGE_FEATURES`::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010025
Patrick Williams0ca19cc2021-08-16 14:03:13 -050010026 IMAGE_FEATURES:append = " tools-debug"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010027
10028- Ensure that debug symbols are present. You can make sure these
Andrew Geisslerc926e172021-05-07 16:11:35 -050010029 symbols are present by installing ``-dbg``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010030
Patrick Williams0ca19cc2021-08-16 14:03:13 -050010031 IMAGE_INSTALL:append = "packagename-dbg"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010032
10033 Alternatively, you can do the following to include
Andrew Geisslerc926e172021-05-07 16:11:35 -050010034 all the debug symbols::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010035
Patrick Williams0ca19cc2021-08-16 14:03:13 -050010036 IMAGE_FEATURES:append = " dbg-pkgs"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010037
10038.. note::
10039
10040 To improve the debug information accuracy, you can reduce the level
10041 of optimization used by the compiler. For example, when adding the
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010042 following line to your ``local.conf`` file, you will reduce optimization
10043 from :term:`FULL_OPTIMIZATION` of "-O2" to :term:`DEBUG_OPTIMIZATION`
Andrew Geisslerc926e172021-05-07 16:11:35 -050010044 of "-O -fno-omit-frame-pointer"::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010045
10046 DEBUG_BUILD = "1"
10047
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010048 Consider that this will reduce the application's performance and is
10049 recommended only for debugging purposes.
10050
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010051Other Debugging Tips
10052--------------------
10053
10054Here are some other tips that you might find useful:
10055
10056- When adding new packages, it is worth watching for undesirable items
10057 making their way into compiler command lines. For example, you do not
10058 want references to local system files like ``/usr/lib/`` or
10059 ``/usr/include/``.
10060
10061- If you want to remove the ``psplash`` boot splashscreen, add
10062 ``psplash=false`` to the kernel command line. Doing so prevents
10063 ``psplash`` from loading and thus allows you to see the console. It
10064 is also possible to switch out of the splashscreen by switching the
10065 virtual console (e.g. Fn+Left or Fn+Right on a Zaurus).
10066
10067- Removing :term:`TMPDIR` (usually
10068 ``tmp/``, within the
10069 :term:`Build Directory`) can often fix
Andrew Geissler09036742021-06-25 14:25:14 -050010070 temporary build issues. Removing :term:`TMPDIR` is usually a relatively
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010071 cheap operation, because task output will be cached in
10072 :term:`SSTATE_DIR` (usually
10073 ``sstate-cache/``, which is also in the Build Directory).
10074
10075 .. note::
10076
Andrew Geissler09036742021-06-25 14:25:14 -050010077 Removing :term:`TMPDIR` might be a workaround rather than a fix.
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010078 Consequently, trying to determine the underlying cause of an issue before
10079 removing the directory is a good idea.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010080
10081- Understanding how a feature is used in practice within existing
10082 recipes can be very helpful. It is recommended that you configure
10083 some method that allows you to quickly search through files.
10084
10085 Using GNU Grep, you can use the following shell function to
10086 recursively search through common recipe-related files, skipping
10087 binary files, ``.git`` directories, and the Build Directory (assuming
Andrew Geisslerc926e172021-05-07 16:11:35 -050010088 its name starts with "build")::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010089
10090 g() {
10091 grep -Ir \
10092 --exclude-dir=.git \
10093 --exclude-dir='build*' \
10094 --include='*.bb*' \
10095 --include='*.inc*' \
10096 --include='*.conf*' \
10097 --include='*.py*' \
10098 "$@"
10099 }
10100
Andrew Geisslerc926e172021-05-07 16:11:35 -050010101 Following are some usage examples::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010102
10103 $ g FOO # Search recursively for "FOO"
10104 $ g -i foo # Search recursively for "foo", ignoring case
10105 $ g -w FOO # Search recursively for "FOO" as a word, ignoring e.g. "FOOBAR"
10106
10107 If figuring
10108 out how some feature works requires a lot of searching, it might
10109 indicate that the documentation should be extended or improved. In
10110 such cases, consider filing a documentation bug using the Yocto
10111 Project implementation of
10112 :yocto_bugs:`Bugzilla <>`. For information on
10113 how to submit a bug against the Yocto Project, see the Yocto Project
Andrew Geissler09209ee2020-12-13 08:44:15 -060010114 Bugzilla :yocto_wiki:`wiki page </Bugzilla_Configuration_and_Bug_Tracking>`
Andrew Geissler3b8a17c2021-04-15 15:55:55 -050010115 and the
10116 ":ref:`dev-manual/common-tasks:submitting a defect against the yocto project`"
10117 section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010118
10119 .. note::
10120
10121 The manuals might not be the right place to document variables
10122 that are purely internal and have a limited scope (e.g. internal
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010123 variables used to implement a single ``.bbclass`` file).
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010124
10125Making Changes to the Yocto Project
10126===================================
10127
10128Because the Yocto Project is an open-source, community-based project,
10129you can effect changes to the project. This section presents procedures
10130that show you how to submit a defect against the project and how to
10131submit a change.
10132
10133Submitting a Defect Against the Yocto Project
10134---------------------------------------------
10135
10136Use the Yocto Project implementation of
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010137`Bugzilla <https://www.bugzilla.org/about/>`__ to submit a defect (bug)
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010138against the Yocto Project. For additional information on this
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010139implementation of Bugzilla see the ":ref:`Yocto Project
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010140Bugzilla <resources-bugtracker>`" section in the
10141Yocto Project Reference Manual. For more detail on any of the following
10142steps, see the Yocto Project
Andrew Geissler09209ee2020-12-13 08:44:15 -060010143:yocto_wiki:`Bugzilla wiki page </Bugzilla_Configuration_and_Bug_Tracking>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010144
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010145Use the following general steps to submit a bug:
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010146
101471. Open the Yocto Project implementation of :yocto_bugs:`Bugzilla <>`.
10148
101492. Click "File a Bug" to enter a new bug.
10150
101513. Choose the appropriate "Classification", "Product", and "Component"
10152 for which the bug was found. Bugs for the Yocto Project fall into
10153 one of several classifications, which in turn break down into
10154 several products and components. For example, for a bug against the
10155 ``meta-intel`` layer, you would choose "Build System, Metadata &
10156 Runtime", "BSPs", and "bsps-meta-intel", respectively.
10157
101584. Choose the "Version" of the Yocto Project for which you found the
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010159 bug (e.g. &DISTRO;).
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010160
101615. Determine and select the "Severity" of the bug. The severity
10162 indicates how the bug impacted your work.
10163
101646. Choose the "Hardware" that the bug impacts.
10165
101667. Choose the "Architecture" that the bug impacts.
10167
101688. Choose a "Documentation change" item for the bug. Fixing a bug might
10169 or might not affect the Yocto Project documentation. If you are
10170 unsure of the impact to the documentation, select "Don't Know".
10171
101729. Provide a brief "Summary" of the bug. Try to limit your summary to
10173 just a line or two and be sure to capture the essence of the bug.
10174
1017510. Provide a detailed "Description" of the bug. You should provide as
10176 much detail as you can about the context, behavior, output, and so
10177 forth that surrounds the bug. You can even attach supporting files
10178 for output from logs by using the "Add an attachment" button.
10179
1018011. Click the "Submit Bug" button submit the bug. A new Bugzilla number
10181 is assigned to the bug and the defect is logged in the bug tracking
10182 system.
10183
10184Once you file a bug, the bug is processed by the Yocto Project Bug
10185Triage Team and further details concerning the bug are assigned (e.g.
10186priority and owner). You are the "Submitter" of the bug and any further
10187categorization, progress, or comments on the bug result in Bugzilla
10188sending you an automated email concerning the particular change or
10189progress to the bug.
10190
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010191Submitting a Change to the Yocto Project
10192----------------------------------------
10193
10194Contributions to the Yocto Project and OpenEmbedded are very welcome.
10195Because the system is extremely configurable and flexible, we recognize
10196that developers will want to extend, configure or optimize it for their
10197specific uses.
10198
10199The Yocto Project uses a mailing list and a patch-based workflow that is
10200similar to the Linux kernel but contains important differences. In
William A. Kennington IIIac69b482021-06-02 12:28:27 -070010201general, there is a mailing list through which you can submit patches. You
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010202should send patches to the appropriate mailing list so that they can be
10203reviewed and merged by the appropriate maintainer. The specific mailing
10204list you need to use depends on the location of the code you are
10205changing. Each component (e.g. layer) should have a ``README`` file that
10206indicates where to send the changes and which process to follow.
10207
10208You can send the patch to the mailing list using whichever approach you
10209feel comfortable with to generate the patch. Once sent, the patch is
10210usually reviewed by the community at large. If somebody has concerns
10211with the patch, they will usually voice their concern over the mailing
10212list. If a patch does not receive any negative reviews, the maintainer
10213of the affected layer typically takes the patch, tests it, and then
10214based on successful testing, merges the patch.
10215
10216The "poky" repository, which is the Yocto Project's reference build
10217environment, is a hybrid repository that contains several individual
10218pieces (e.g. BitBake, Metadata, documentation, and so forth) built using
10219the combo-layer tool. The upstream location used for submitting changes
10220varies by component:
10221
10222- *Core Metadata:* Send your patch to the
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010223 :oe_lists:`openembedded-core </g/openembedded-core>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010224 mailing list. For example, a change to anything under the ``meta`` or
10225 ``scripts`` directories should be sent to this mailing list.
10226
10227- *BitBake:* For changes to BitBake (i.e. anything under the
10228 ``bitbake`` directory), send your patch to the
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010229 :oe_lists:`bitbake-devel </g/bitbake-devel>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010230 mailing list.
10231
Andrew Geisslerc3d88e42020-10-02 09:45:00 -050010232- *"meta-\*" trees:* These trees contain Metadata. Use the
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010233 :yocto_lists:`poky </g/poky>` mailing list.
Andrew Geisslerc3d88e42020-10-02 09:45:00 -050010234
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010235- *Documentation*: For changes to the Yocto Project documentation, use the
10236 :yocto_lists:`docs </g/docs>` mailing list.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010237
10238For changes to other layers hosted in the Yocto Project source
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010239repositories (i.e. ``yoctoproject.org``) and tools use the
10240:yocto_lists:`Yocto Project </g/yocto/>` general mailing list.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010241
10242.. note::
10243
10244 Sometimes a layer's documentation specifies to use a particular
10245 mailing list. If so, use that list.
10246
10247For additional recipes that do not fit into the core Metadata, you
10248should determine which layer the recipe should go into and submit the
10249change in the manner recommended by the documentation (e.g. the
10250``README`` file) supplied with the layer. If in doubt, please ask on the
10251Yocto general mailing list or on the openembedded-devel mailing list.
10252
10253You can also push a change upstream and request a maintainer to pull the
10254change into the component's upstream repository. You do this by pushing
Andrew Geissler09209ee2020-12-13 08:44:15 -060010255to a contribution repository that is upstream. See the
10256":ref:`overview-manual/development-environment:git workflows and the yocto project`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010257section in the Yocto Project Overview and Concepts Manual for additional
10258concepts on working in the Yocto Project development environment.
10259
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010260Maintainers commonly use ``-next`` branches to test submissions prior to
10261merging patches. Thus, you can get an idea of the status of a patch based on
10262whether the patch has been merged into one of these branches. The commonly
10263used testing branches for OpenEmbedded-Core are as follows:
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010264
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010265- *openembedded-core "master-next" branch:* This branch is part of the
10266 :oe_git:`openembedded-core </openembedded-core/>` repository and contains
10267 proposed changes to the core metadata.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010268
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010269- *poky "master-next" branch:* This branch is part of the
Andrew Geissler09209ee2020-12-13 08:44:15 -060010270 :yocto_git:`poky </poky/>` repository and combines proposed
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010271 changes to bitbake, the core metadata and the poky distro.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010272
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010273Similarly, stable branches maintained by the project may have corresponding
10274``-next`` branches which collect proposed changes. For example,
10275``&DISTRO_NAME_NO_CAP;-next`` and ``&DISTRO_NAME_NO_CAP_MINUS_ONE;-next``
10276branches in both the "openembdedded-core" and "poky" repositories.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010277
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010278Other layers may have similar testing branches but there is no formal
10279requirement or standard for these so please check the documentation for the
10280layers you are contributing to.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010281
10282The following sections provide procedures for submitting a change.
10283
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010284Preparing Changes for Submission
10285~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010286
102871. *Make Your Changes Locally:* Make your changes in your local Git
10288 repository. You should make small, controlled, isolated changes.
10289 Keeping changes small and isolated aids review, makes
10290 merging/rebasing easier and keeps the change history clean should
10291 anyone need to refer to it in future.
10292
102932. *Stage Your Changes:* Stage your changes by using the ``git add``
10294 command on each file you changed.
10295
102963. *Commit Your Changes:* Commit the change by using the ``git commit``
10297 command. Make sure your commit information follows standards by
10298 following these accepted conventions:
10299
10300 - Be sure to include a "Signed-off-by:" line in the same style as
10301 required by the Linux kernel. Adding this line signifies that you,
10302 the submitter, have agreed to the Developer's Certificate of
10303 Origin 1.1 as follows:
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010304
10305 .. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010306
10307 Developer's Certificate of Origin 1.1
10308
10309 By making a contribution to this project, I certify that:
10310
10311 (a) The contribution was created in whole or in part by me and I
10312 have the right to submit it under the open source license
10313 indicated in the file; or
10314
10315 (b) The contribution is based upon previous work that, to the best
10316 of my knowledge, is covered under an appropriate open source
10317 license and I have the right under that license to submit that
10318 work with modifications, whether created in whole or in part
10319 by me, under the same open source license (unless I am
10320 permitted to submit under a different license), as indicated
10321 in the file; or
10322
10323 (c) The contribution was provided directly to me by some other
10324 person who certified (a), (b) or (c) and I have not modified
10325 it.
10326
10327 (d) I understand and agree that this project and the contribution
10328 are public and that a record of the contribution (including all
10329 personal information I submit with it, including my sign-off) is
10330 maintained indefinitely and may be redistributed consistent with
10331 this project or the open source license(s) involved.
10332
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010333 - Provide a single-line summary of the change and, if more
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010334 explanation is needed, provide more detail in the body of the
10335 commit. This summary is typically viewable in the "shortlist" of
10336 changes. Thus, providing something short and descriptive that
10337 gives the reader a summary of the change is useful when viewing a
10338 list of many commits. You should prefix this short description
10339 with the recipe name (if changing a recipe), or else with the
10340 short form path to the file being changed.
10341
10342 - For the body of the commit message, provide detailed information
10343 that describes what you changed, why you made the change, and the
10344 approach you used. It might also be helpful if you mention how you
10345 tested the change. Provide as much detail as you can in the body
10346 of the commit message.
10347
10348 .. note::
10349
10350 You do not need to provide a more detailed explanation of a
10351 change if the change is minor to the point of the single line
10352 summary providing all the information.
10353
10354 - If the change addresses a specific bug or issue that is associated
10355 with a bug-tracking ID, include a reference to that ID in your
10356 detailed description. For example, the Yocto Project uses a
10357 specific convention for bug references - any commit that addresses
10358 a specific bug should use the following form for the detailed
10359 description. Be sure to use the actual bug-tracking ID from
Andrew Geisslerc926e172021-05-07 16:11:35 -050010360 Bugzilla for bug-id::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010361
10362 Fixes [YOCTO #bug-id]
10363
10364 detailed description of change
10365
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010366Using Email to Submit a Patch
10367~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10368
10369Depending on the components changed, you need to submit the email to a
10370specific mailing list. For some guidance on which mailing list to use,
Andrew Geissler3b8a17c2021-04-15 15:55:55 -050010371see the
10372:ref:`list <dev-manual/common-tasks:submitting a change to the yocto project>`
10373at the beginning of this section. For a description of all the available
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010374mailing lists, see the ":ref:`Mailing Lists <resources-mailinglist>`" section in the
10375Yocto Project Reference Manual.
10376
10377Here is the general procedure on how to submit a patch through email
10378without using the scripts once the steps in
Andrew Geissler09209ee2020-12-13 08:44:15 -060010379:ref:`dev-manual/common-tasks:preparing changes for submission` have been followed:
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010380
103811. *Format the Commit:* Format the commit into an email message. To
10382 format commits, use the ``git format-patch`` command. When you
10383 provide the command, you must include a revision list or a number of
10384 patches as part of the command. For example, either of these two
10385 commands takes your most recent single commit and formats it as an
Andrew Geisslerc926e172021-05-07 16:11:35 -050010386 email message in the current directory::
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010387
10388 $ git format-patch -1
10389
10390 or ::
10391
10392 $ git format-patch HEAD~
10393
10394 After the command is run, the current directory contains a numbered
10395 ``.patch`` file for the commit.
10396
10397 If you provide several commits as part of the command, the
10398 ``git format-patch`` command produces a series of numbered files in
10399 the current directory – one for each commit. If you have more than
10400 one patch, you should also use the ``--cover`` option with the
10401 command, which generates a cover letter as the first "patch" in the
10402 series. You can then edit the cover letter to provide a description
10403 for the series of patches. For information on the
10404 ``git format-patch`` command, see ``GIT_FORMAT_PATCH(1)`` displayed
10405 using the ``man git-format-patch`` command.
10406
10407 .. note::
10408
10409 If you are or will be a frequent contributor to the Yocto Project
10410 or to OpenEmbedded, you might consider requesting a contrib area
10411 and the necessary associated rights.
10412
104132. *Send the patches via email:* Send the patches to the recipients and
10414 relevant mailing lists by using the ``git send-email`` command.
10415
10416 .. note::
10417
10418 In order to use ``git send-email``, you must have the proper Git packages
10419 installed on your host.
10420 For Ubuntu, Debian, and Fedora the package is ``git-email``.
10421
10422 The ``git send-email`` command sends email by using a local or remote
10423 Mail Transport Agent (MTA) such as ``msmtp``, ``sendmail``, or
10424 through a direct ``smtp`` configuration in your Git ``~/.gitconfig``
10425 file. If you are submitting patches through email only, it is very
10426 important that you submit them without any whitespace or HTML
10427 formatting that either you or your mailer introduces. The maintainer
10428 that receives your patches needs to be able to save and apply them
10429 directly from your emails. A good way to verify that what you are
10430 sending will be applicable by the maintainer is to do a dry run and
10431 send them to yourself and then save and apply them as the maintainer
10432 would.
10433
10434 The ``git send-email`` command is the preferred method for sending
10435 your patches using email since there is no risk of compromising
10436 whitespace in the body of the message, which can occur when you use
10437 your own mail client. The command also has several options that let
10438 you specify recipients and perform further editing of the email
10439 message. For information on how to use the ``git send-email``
10440 command, see ``GIT-SEND-EMAIL(1)`` displayed using the
10441 ``man git-send-email`` command.
10442
10443The Yocto Project uses a `Patchwork instance <https://patchwork.openembedded.org/>`__
10444to track the status of patches submitted to the various mailing lists and to
10445support automated patch testing. Each submitted patch is checked for common
10446mistakes and deviations from the expected patch format and submitters are
10447notified by patchtest if such mistakes are found. This process helps to
10448reduce the burden of patch review on maintainers.
10449
10450.. note::
10451
10452 This system is imperfect and changes can sometimes get lost in the flow.
10453 Asking about the status of a patch or change is reasonable if the change
10454 has been idle for a while with no feedback.
10455
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010456Using Scripts to Push a Change Upstream and Request a Pull
10457~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10458
10459For larger patch series it is preferable to send a pull request which not
10460only includes the patch but also a pointer to a branch that can be pulled
10461from. This involves making a local branch for your changes, pushing this
10462branch to an accessible repository and then using the ``create-pull-request``
10463and ``send-pull-request`` scripts from openembedded-core to create and send a
10464patch series with a link to the branch for review.
10465
10466Follow this procedure to push a change to an upstream "contrib" Git
Andrew Geissler09209ee2020-12-13 08:44:15 -060010467repository once the steps in :ref:`dev-manual/common-tasks:preparing changes for submission` have
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010468been followed:
10469
10470.. note::
10471
10472 You can find general Git information on how to push a change upstream
10473 in the
10474 `Git Community Book <https://git-scm.com/book/en/v2/Distributed-Git-Distributed-Workflows>`__.
10475
104761. *Push Your Commits to a "Contrib" Upstream:* If you have arranged for
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010477 permissions to push to an upstream contrib repository, push the
Andrew Geisslerc926e172021-05-07 16:11:35 -050010478 change to that repository::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010479
10480 $ git push upstream_remote_repo local_branch_name
10481
10482 For example, suppose you have permissions to push
10483 into the upstream ``meta-intel-contrib`` repository and you are
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010484 working in a local branch named `your_name`\ ``/README``. The following
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010485 command pushes your local commits to the ``meta-intel-contrib``
10486 upstream repository and puts the commit in a branch named
Andrew Geisslerc926e172021-05-07 16:11:35 -050010487 `your_name`\ ``/README``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010488
10489 $ git push meta-intel-contrib your_name/README
10490
Andrew Geissler6ce62a22020-11-30 19:58:47 -0600104912. *Determine Who to Notify:* Determine the maintainer or the mailing
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010492 list that you need to notify for the change.
10493
10494 Before submitting any change, you need to be sure who the maintainer
10495 is or what mailing list that you need to notify. Use either these
10496 methods to find out:
10497
10498 - *Maintenance File:* Examine the ``maintainers.inc`` file, which is
10499 located in the :term:`Source Directory` at
10500 ``meta/conf/distro/include``, to see who is responsible for code.
10501
Andrew Geissler09209ee2020-12-13 08:44:15 -060010502 - *Search by File:* Using :ref:`overview-manual/development-environment:git`, you can
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010503 enter the following command to bring up a short list of all
Andrew Geisslerc926e172021-05-07 16:11:35 -050010504 commits against a specific file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010505
10506 git shortlog -- filename
10507
10508 Just provide the name of the file for which you are interested. The
10509 information returned is not ordered by history but does include a
10510 list of everyone who has committed grouped by name. From the list,
10511 you can see who is responsible for the bulk of the changes against
10512 the file.
10513
10514 - *Examine the List of Mailing Lists:* For a list of the Yocto
10515 Project and related mailing lists, see the ":ref:`Mailing
10516 lists <resources-mailinglist>`" section in
10517 the Yocto Project Reference Manual.
10518
Andrew Geissler6ce62a22020-11-30 19:58:47 -0600105193. *Make a Pull Request:* Notify the maintainer or the mailing list that
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010520 you have pushed a change by making a pull request.
10521
10522 The Yocto Project provides two scripts that conveniently let you
10523 generate and send pull requests to the Yocto Project. These scripts
10524 are ``create-pull-request`` and ``send-pull-request``. You can find
10525 these scripts in the ``scripts`` directory within the
10526 :term:`Source Directory` (e.g.
Andrew Geissler95ac1b82021-03-31 14:34:31 -050010527 ``poky/scripts``).
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010528
10529 Using these scripts correctly formats the requests without
10530 introducing any whitespace or HTML formatting. The maintainer that
10531 receives your patches either directly or through the mailing list
10532 needs to be able to save and apply them directly from your emails.
10533 Using these scripts is the preferred method for sending patches.
10534
10535 First, create the pull request. For example, the following command
10536 runs the script, specifies the upstream repository in the contrib
10537 directory into which you pushed the change, and provides a subject
Andrew Geisslerc926e172021-05-07 16:11:35 -050010538 line in the created patch files::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010539
Andrew Geissler95ac1b82021-03-31 14:34:31 -050010540 $ poky/scripts/create-pull-request -u meta-intel-contrib -s "Updated Manual Section Reference in README"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010541
10542 Running this script forms ``*.patch`` files in a folder named
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010543 ``pull-``\ `PID` in the current directory. One of the patch files is a
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010544 cover letter.
10545
10546 Before running the ``send-pull-request`` script, you must edit the
10547 cover letter patch to insert information about your change. After
10548 editing the cover letter, send the pull request. For example, the
10549 following command runs the script and specifies the patch directory
10550 and email address. In this example, the email address is a mailing
Andrew Geisslerc926e172021-05-07 16:11:35 -050010551 list::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010552
Andrew Geissler5199d832021-09-24 16:47:35 -050010553 $ poky/scripts/send-pull-request -p ~/meta-intel/pull-10565 -t meta-intel@lists.yoctoproject.org
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010554
10555 You need to follow the prompts as the script is interactive.
10556
10557 .. note::
10558
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010559 For help on using these scripts, simply provide the ``-h``
Andrew Geisslerc926e172021-05-07 16:11:35 -050010560 argument as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010561
10562 $ poky/scripts/create-pull-request -h
10563 $ poky/scripts/send-pull-request -h
10564
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010565Responding to Patch Review
10566~~~~~~~~~~~~~~~~~~~~~~~~~~
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010567
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010568You may get feedback on your submitted patches from other community members
10569or from the automated patchtest service. If issues are identified in your
10570patch then it is usually necessary to address these before the patch will be
10571accepted into the project. In this case you should amend the patch according
10572to the feedback and submit an updated version to the relevant mailing list,
10573copying in the reviewers who provided feedback to the previous version of the
10574patch.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010575
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010576The patch should be amended using ``git commit --amend`` or perhaps ``git
10577rebase`` for more expert git users. You should also modify the ``[PATCH]``
10578tag in the email subject line when sending the revised patch to mark the new
10579iteration as ``[PATCH v2]``, ``[PATCH v3]``, etc as appropriate. This can be
10580done by passing the ``-v`` argument to ``git format-patch`` with a version
10581number.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010582
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010583Lastly please ensure that you also test your revised changes. In particular
10584please don't just edit the patch file written out by ``git format-patch`` and
10585resend it.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010586
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010587Submitting Changes to Stable Release Branches
10588~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010589
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010590The process for proposing changes to a Yocto Project stable branch differs
10591from the steps described above. Changes to a stable branch must address
10592identified bugs or CVEs and should be made carefully in order to avoid the
10593risk of introducing new bugs or breaking backwards compatibility. Typically
10594bug fixes must already be accepted into the master branch before they can be
10595backported to a stable branch unless the bug in question does not affect the
10596master branch or the fix on the master branch is unsuitable for backporting.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010597
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010598The list of stable branches along with the status and maintainer for each
10599branch can be obtained from the
Andrew Geissler09209ee2020-12-13 08:44:15 -060010600:yocto_wiki:`Releases wiki page </Releases>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010601
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010602.. note::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010603
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010604 Changes will not typically be accepted for branches which are marked as
10605 End-Of-Life (EOL).
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010606
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010607With this in mind, the steps to submit a change for a stable branch are as
10608follows:
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010609
Andrew Geissler6ce62a22020-11-30 19:58:47 -0600106101. *Identify the bug or CVE to be fixed:* This information should be
10611 collected so that it can be included in your submission.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010612
Patrick Williams213cb262021-08-07 19:21:33 -050010613 See :ref:`dev-manual/common-tasks:checking for vulnerabilities`
10614 for details about CVE tracking.
10615
Andrew Geissler6ce62a22020-11-30 19:58:47 -0600106162. *Check if the fix is already present in the master branch:* This will
10617 result in the most straightforward path into the stable branch for the
10618 fix.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010619
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010620 a. *If the fix is present in the master branch - Submit a backport request
10621 by email:* You should send an email to the relevant stable branch
10622 maintainer and the mailing list with details of the bug or CVE to be
10623 fixed, the commit hash on the master branch that fixes the issue and
10624 the stable branches which you would like this fix to be backported to.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010625
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010626 b. *If the fix is not present in the master branch - Submit the fix to the
10627 master branch first:* This will ensure that the fix passes through the
10628 project's usual patch review and test processes before being accepted.
10629 It will also ensure that bugs are not left unresolved in the master
10630 branch itself. Once the fix is accepted in the master branch a backport
10631 request can be submitted as above.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010632
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010633 c. *If the fix is unsuitable for the master branch - Submit a patch
10634 directly for the stable branch:* This method should be considered as a
10635 last resort. It is typically necessary when the master branch is using
10636 a newer version of the software which includes an upstream fix for the
10637 issue or when the issue has been fixed on the master branch in a way
10638 that introduces backwards incompatible changes. In this case follow the
Andrew Geissler09209ee2020-12-13 08:44:15 -060010639 steps in :ref:`dev-manual/common-tasks:preparing changes for submission` and
10640 :ref:`dev-manual/common-tasks:using email to submit a patch` but modify the subject header of your patch
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010641 email to include the name of the stable branch which you are
10642 targetting. This can be done using the ``--subject-prefix`` argument to
10643 ``git format-patch``, for example to submit a patch to the dunfell
10644 branch use
10645 ``git format-patch --subject-prefix='&DISTRO_NAME_NO_CAP_MINUS_ONE;][PATCH' ...``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010646
10647Working With Licenses
10648=====================
10649
Andrew Geissler09209ee2020-12-13 08:44:15 -060010650As mentioned in the ":ref:`overview-manual/development-environment:licensing`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010651section in the Yocto Project Overview and Concepts Manual, open source
10652projects are open to the public and they consequently have different
10653licensing structures in place. This section describes the mechanism by
10654which the :term:`OpenEmbedded Build System`
10655tracks changes to
10656licensing text and covers how to maintain open source license compliance
10657during your project's lifecycle. The section also describes how to
10658enable commercially licensed recipes, which by default are disabled.
10659
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010660Tracking License Changes
10661------------------------
10662
10663The license of an upstream project might change in the future. In order
10664to prevent these changes going unnoticed, the
10665:term:`LIC_FILES_CHKSUM`
10666variable tracks changes to the license text. The checksums are validated
10667at the end of the configure step, and if the checksums do not match, the
10668build will fail.
10669
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010670Specifying the ``LIC_FILES_CHKSUM`` Variable
10671~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10672
Andrew Geissler09036742021-06-25 14:25:14 -050010673The :term:`LIC_FILES_CHKSUM` variable contains checksums of the license text
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010674in the source code for the recipe. Following is an example of how to
Andrew Geissler09036742021-06-25 14:25:14 -050010675specify :term:`LIC_FILES_CHKSUM`::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010676
10677 LIC_FILES_CHKSUM = "file://COPYING;md5=xxxx \
10678 file://licfile1.txt;beginline=5;endline=29;md5=yyyy \
10679 file://licfile2.txt;endline=50;md5=zzzz \
10680 ..."
10681
10682.. note::
10683
10684 - When using "beginline" and "endline", realize that line numbering
10685 begins with one and not zero. Also, the included lines are
10686 inclusive (i.e. lines five through and including 29 in the
10687 previous example for ``licfile1.txt``).
10688
10689 - When a license check fails, the selected license text is included
10690 as part of the QA message. Using this output, you can determine
10691 the exact start and finish for the needed license text.
10692
10693The build system uses the :term:`S`
10694variable as the default directory when searching files listed in
Andrew Geissler09036742021-06-25 14:25:14 -050010695:term:`LIC_FILES_CHKSUM`. The previous example employs the default
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010696directory.
10697
Andrew Geisslerc926e172021-05-07 16:11:35 -050010698Consider this next example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010699
10700 LIC_FILES_CHKSUM = "file://src/ls.c;beginline=5;endline=16;\
10701 md5=bb14ed3c4cda583abc85401304b5cd4e"
10702 LIC_FILES_CHKSUM = "file://${WORKDIR}/license.html;md5=5c94767cedb5d6987c902ac850ded2c6"
10703
10704The first line locates a file in ``${S}/src/ls.c`` and isolates lines
10705five through 16 as license text. The second line refers to a file in
10706:term:`WORKDIR`.
10707
Andrew Geissler09036742021-06-25 14:25:14 -050010708Note that :term:`LIC_FILES_CHKSUM` variable is mandatory for all recipes,
10709unless the :term:`LICENSE` variable is set to "CLOSED".
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010710
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010711Explanation of Syntax
10712~~~~~~~~~~~~~~~~~~~~~
10713
Andrew Geissler09036742021-06-25 14:25:14 -050010714As mentioned in the previous section, the :term:`LIC_FILES_CHKSUM` variable
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010715lists all the important files that contain the license text for the
10716source code. It is possible to specify a checksum for an entire file, or
10717a specific section of a file (specified by beginning and ending line
10718numbers with the "beginline" and "endline" parameters, respectively).
10719The latter is useful for source files with a license notice header,
10720README documents, and so forth. If you do not use the "beginline"
10721parameter, then it is assumed that the text begins on the first line of
10722the file. Similarly, if you do not use the "endline" parameter, it is
10723assumed that the license text ends with the last line of the file.
10724
10725The "md5" parameter stores the md5 checksum of the license text. If the
10726license text changes in any way as compared to this parameter then a
10727mismatch occurs. This mismatch triggers a build failure and notifies the
10728developer. Notification allows the developer to review and address the
10729license text changes. Also note that if a mismatch occurs during the
10730build, the correct md5 checksum is placed in the build log and can be
10731easily copied to the recipe.
10732
10733There is no limit to how many files you can specify using the
Andrew Geissler09036742021-06-25 14:25:14 -050010734:term:`LIC_FILES_CHKSUM` variable. Generally, however, every project
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010735requires a few specifications for license tracking. Many projects have a
10736"COPYING" file that stores the license information for all the source
10737code files. This practice allows you to just track the "COPYING" file as
10738long as it is kept up to date.
10739
10740.. note::
10741
10742 - If you specify an empty or invalid "md5" parameter,
10743 :term:`BitBake` returns an md5
10744 mis-match error and displays the correct "md5" parameter value
10745 during the build. The correct parameter is also captured in the
10746 build log.
10747
10748 - If the whole file contains only license text, you do not need to
10749 use the "beginline" and "endline" parameters.
10750
10751Enabling Commercially Licensed Recipes
10752--------------------------------------
10753
10754By default, the OpenEmbedded build system disables components that have
10755commercial or other special licensing requirements. Such requirements
10756are defined on a recipe-by-recipe basis through the
10757:term:`LICENSE_FLAGS` variable
10758definition in the affected recipe. For instance, the
10759``poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly`` recipe
Andrew Geisslerc926e172021-05-07 16:11:35 -050010760contains the following statement::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010761
10762 LICENSE_FLAGS = "commercial"
10763
10764Here is a
10765slightly more complicated example that contains both an explicit recipe
Andrew Geisslerc926e172021-05-07 16:11:35 -050010766name and version (after variable expansion)::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010767
10768 LICENSE_FLAGS = "license_${PN}_${PV}"
10769
10770In order for a component restricted by a
Andrew Geissler09036742021-06-25 14:25:14 -050010771:term:`LICENSE_FLAGS` definition to be enabled and included in an image, it
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010772needs to have a matching entry in the global
10773:term:`LICENSE_FLAGS_WHITELIST`
10774variable, which is a variable typically defined in your ``local.conf``
10775file. For example, to enable the
10776``poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly`` package, you
10777could add either the string "commercial_gst-plugins-ugly" or the more
Andrew Geissler09036742021-06-25 14:25:14 -050010778general string "commercial" to :term:`LICENSE_FLAGS_WHITELIST`. See the
Andrew Geissler3b8a17c2021-04-15 15:55:55 -050010779":ref:`dev-manual/common-tasks:license flag matching`" section for a full
Andrew Geissler09036742021-06-25 14:25:14 -050010780explanation of how :term:`LICENSE_FLAGS` matching works. Here is the
Andrew Geisslerc926e172021-05-07 16:11:35 -050010781example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010782
10783 LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly"
10784
10785Likewise, to additionally enable the package built from the recipe
10786containing ``LICENSE_FLAGS = "license_${PN}_${PV}"``, and assuming that
10787the actual recipe name was ``emgd_1.10.bb``, the following string would
10788enable that package as well as the original ``gst-plugins-ugly``
Andrew Geisslerc926e172021-05-07 16:11:35 -050010789package::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010790
10791 LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly license_emgd_1.10"
10792
10793As a convenience, you do not need to specify the
10794complete license string in the whitelist for every package. You can use
10795an abbreviated form, which consists of just the first portion or
10796portions of the license string before the initial underscore character
10797or characters. A partial string will match any license that contains the
10798given string as the first portion of its license. For example, the
10799following whitelist string will also match both of the packages
10800previously mentioned as well as any other packages that have licenses
10801starting with "commercial" or "license".
10802::
10803
10804 LICENSE_FLAGS_WHITELIST = "commercial license"
10805
10806License Flag Matching
10807~~~~~~~~~~~~~~~~~~~~~
10808
10809License flag matching allows you to control what recipes the
10810OpenEmbedded build system includes in the build. Fundamentally, the
Andrew Geissler09036742021-06-25 14:25:14 -050010811build system attempts to match :term:`LICENSE_FLAGS` strings found in
10812recipes against :term:`LICENSE_FLAGS_WHITELIST` strings found in the
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010813whitelist. A match causes the build system to include a recipe in the
10814build, while failure to find a match causes the build system to exclude
10815a recipe.
10816
10817In general, license flag matching is simple. However, understanding some
10818concepts will help you correctly and effectively use matching.
10819
10820Before a flag defined by a particular recipe is tested against the
10821contents of the whitelist, the expanded string ``_${PN}`` is appended to
Andrew Geissler09036742021-06-25 14:25:14 -050010822the flag. This expansion makes each :term:`LICENSE_FLAGS` value
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010823recipe-specific. After expansion, the string is then matched against the
10824whitelist. Thus, specifying ``LICENSE_FLAGS = "commercial"`` in recipe
10825"foo", for example, results in the string ``"commercial_foo"``. And, to
10826create a match, that string must appear in the whitelist.
10827
Andrew Geissler09036742021-06-25 14:25:14 -050010828Judicious use of the :term:`LICENSE_FLAGS` strings and the contents of the
10829:term:`LICENSE_FLAGS_WHITELIST` variable allows you a lot of flexibility for
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010830including or excluding recipes based on licensing. For example, you can
10831broaden the matching capabilities by using license flags string subsets
10832in the whitelist.
10833
10834.. note::
10835
10836 When using a string subset, be sure to use the part of the expanded
10837 string that precedes the appended underscore character (e.g.
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010838 ``usethispart_1.3``, ``usethispart_1.4``, and so forth).
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010839
10840For example, simply specifying the string "commercial" in the whitelist
Andrew Geissler09036742021-06-25 14:25:14 -050010841matches any expanded :term:`LICENSE_FLAGS` definition that starts with the
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010842string "commercial" such as "commercial_foo" and "commercial_bar", which
10843are the strings the build system automatically generates for
10844hypothetical recipes named "foo" and "bar" assuming those recipes simply
Andrew Geisslerc926e172021-05-07 16:11:35 -050010845specify the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010846
10847 LICENSE_FLAGS = "commercial"
10848
10849Thus, you can choose
10850to exhaustively enumerate each license flag in the whitelist and allow
10851only specific recipes into the image, or you can use a string subset
10852that causes a broader range of matches to allow a range of recipes into
10853the image.
10854
Andrew Geissler09036742021-06-25 14:25:14 -050010855This scheme works even if the :term:`LICENSE_FLAGS` string already has
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010856``_${PN}`` appended. For example, the build system turns the license
10857flag "commercial_1.2_foo" into "commercial_1.2_foo_foo" and would match
10858both the general "commercial" and the specific "commercial_1.2_foo"
10859strings found in the whitelist, as expected.
10860
10861Here are some other scenarios:
10862
10863- You can specify a versioned string in the recipe such as
10864 "commercial_foo_1.2" in a "foo" recipe. The build system expands this
10865 string to "commercial_foo_1.2_foo". Combine this license flag with a
10866 whitelist that has the string "commercial" and you match the flag
10867 along with any other flag that starts with the string "commercial".
10868
10869- Under the same circumstances, you can use "commercial_foo" in the
10870 whitelist and the build system not only matches "commercial_foo_1.2"
10871 but also matches any license flag with the string "commercial_foo",
10872 regardless of the version.
10873
10874- You can be very specific and use both the package and version parts
10875 in the whitelist (e.g. "commercial_foo_1.2") to specifically match a
10876 versioned recipe.
10877
10878Other Variables Related to Commercial Licenses
10879~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10880
William A. Kennington IIIac69b482021-06-02 12:28:27 -070010881There are other helpful variables related to commercial license handling,
10882defined in the
Andrew Geisslerc926e172021-05-07 16:11:35 -050010883``poky/meta/conf/distro/include/default-distrovars.inc`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010884
10885 COMMERCIAL_AUDIO_PLUGINS ?= ""
10886 COMMERCIAL_VIDEO_PLUGINS ?= ""
10887
10888If you
10889want to enable these components, you can do so by making sure you have
10890statements similar to the following in your ``local.conf`` configuration
Andrew Geisslerc926e172021-05-07 16:11:35 -050010891file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010892
10893 COMMERCIAL_AUDIO_PLUGINS = "gst-plugins-ugly-mad \
10894 gst-plugins-ugly-mpegaudioparse"
10895 COMMERCIAL_VIDEO_PLUGINS = "gst-plugins-ugly-mpeg2dec \
10896 gst-plugins-ugly-mpegstream gst-plugins-bad-mpegvideoparse"
10897 LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly commercial_gst-plugins-bad commercial_qmmp"
10898
10899
10900Of course, you could also create a matching whitelist for those
10901components using the more general "commercial" in the whitelist, but
Andrew Geissler09036742021-06-25 14:25:14 -050010902that would also enable all the other packages with :term:`LICENSE_FLAGS`
Andrew Geisslerc926e172021-05-07 16:11:35 -050010903containing "commercial", which you may or may not want::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010904
10905 LICENSE_FLAGS_WHITELIST = "commercial"
10906
10907Specifying audio and video plugins as part of the
10908``COMMERCIAL_AUDIO_PLUGINS`` and ``COMMERCIAL_VIDEO_PLUGINS`` statements
Andrew Geissler09036742021-06-25 14:25:14 -050010909(along with the enabling :term:`LICENSE_FLAGS_WHITELIST`) includes the
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010910plugins or components into built images, thus adding support for media
10911formats or components.
10912
10913Maintaining Open Source License Compliance During Your Product's Lifecycle
10914--------------------------------------------------------------------------
10915
10916One of the concerns for a development organization using open source
10917software is how to maintain compliance with various open source
10918licensing during the lifecycle of the product. While this section does
10919not provide legal advice or comprehensively cover all scenarios, it does
10920present methods that you can use to assist you in meeting the compliance
10921requirements during a software release.
10922
10923With hundreds of different open source licenses that the Yocto Project
10924tracks, it is difficult to know the requirements of each and every
10925license. However, the requirements of the major FLOSS licenses can begin
William A. Kennington IIIac69b482021-06-02 12:28:27 -070010926to be covered by assuming that there are three main areas of concern:
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010927
10928- Source code must be provided.
10929
10930- License text for the software must be provided.
10931
10932- Compilation scripts and modifications to the source code must be
10933 provided.
10934
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010935- spdx files can be provided.
10936
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010937There are other requirements beyond the scope of these three and the
10938methods described in this section (e.g. the mechanism through which
10939source code is distributed).
10940
10941As different organizations have different methods of complying with open
10942source licensing, this section is not meant to imply that there is only
10943one single way to meet your compliance obligations, but rather to
10944describe one method of achieving compliance. The remainder of this
10945section describes methods supported to meet the previously mentioned
10946three requirements. Once you take steps to meet these requirements, and
10947prior to releasing images, sources, and the build system, you should
10948audit all artifacts to ensure completeness.
10949
10950.. note::
10951
10952 The Yocto Project generates a license manifest during image creation
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010953 that is located in ``${DEPLOY_DIR}/licenses/``\ `image_name`\ ``-``\ `datestamp`
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010954 to assist with any audits.
10955
10956Providing the Source Code
10957~~~~~~~~~~~~~~~~~~~~~~~~~
10958
10959Compliance activities should begin before you generate the final image.
10960The first thing you should look at is the requirement that tops the list
10961for most compliance groups - providing the source. The Yocto Project has
10962a few ways of meeting this requirement.
10963
10964One of the easiest ways to meet this requirement is to provide the
10965entire :term:`DL_DIR` used by the
10966build. This method, however, has a few issues. The most obvious is the
10967size of the directory since it includes all sources used in the build
10968and not just the source used in the released image. It will include
10969toolchain source, and other artifacts, which you would not generally
10970release. However, the more serious issue for most companies is
10971accidental release of proprietary software. The Yocto Project provides
10972an :ref:`archiver <ref-classes-archiver>` class to
10973help avoid some of these concerns.
10974
Andrew Geissler09036742021-06-25 14:25:14 -050010975Before you employ :term:`DL_DIR` or the ``archiver`` class, you need to
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010976decide how you choose to provide source. The source ``archiver`` class
10977can generate tarballs and SRPMs and can create them with various levels
10978of compliance in mind.
10979
10980One way of doing this (but certainly not the only way) is to release
10981just the source as a tarball. You can do this by adding the following to
10982the ``local.conf`` file found in the
Andrew Geisslerc926e172021-05-07 16:11:35 -050010983:term:`Build Directory`::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010984
10985 INHERIT += "archiver"
10986 ARCHIVER_MODE[src] = "original"
10987
10988During the creation of your
10989image, the source from all recipes that deploy packages to the image is
10990placed within subdirectories of ``DEPLOY_DIR/sources`` based on the
10991:term:`LICENSE` for each recipe.
10992Releasing the entire directory enables you to comply with requirements
10993concerning providing the unmodified source. It is important to note that
10994the size of the directory can get large.
10995
10996A way to help mitigate the size issue is to only release tarballs for
10997licenses that require the release of source. Let us assume you are only
10998concerned with GPL code as identified by running the following script:
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010999
11000.. code-block:: shell
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011001
11002 # Script to archive a subset of packages matching specific license(s)
11003 # Source and license files are copied into sub folders of package folder
11004 # Must be run from build folder
11005 #!/bin/bash
11006 src_release_dir="source-release"
11007 mkdir -p $src_release_dir
11008 for a in tmp/deploy/sources/*; do
11009 for d in $a/*; do
11010 # Get package name from path
11011 p=`basename $d`
11012 p=${p%-*}
11013 p=${p%-*}
11014 # Only archive GPL packages (update *GPL* regex for your license check)
11015 numfiles=`ls tmp/deploy/licenses/$p/*GPL* 2> /dev/null | wc -l`
Patrick Williams213cb262021-08-07 19:21:33 -050011016 if [ $numfiles -ge 1 ]; then
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011017 echo Archiving $p
11018 mkdir -p $src_release_dir/$p/source
11019 cp $d/* $src_release_dir/$p/source 2> /dev/null
11020 mkdir -p $src_release_dir/$p/license
11021 cp tmp/deploy/licenses/$p/* $src_release_dir/$p/license 2> /dev/null
11022 fi
11023 done
11024 done
11025
11026At this point, you
11027could create a tarball from the ``gpl_source_release`` directory and
11028provide that to the end user. This method would be a step toward
11029achieving compliance with section 3a of GPLv2 and with section 6 of
11030GPLv3.
11031
11032Providing License Text
11033~~~~~~~~~~~~~~~~~~~~~~
11034
11035One requirement that is often overlooked is inclusion of license text.
11036This requirement also needs to be dealt with prior to generating the
11037final image. Some licenses require the license text to accompany the
11038binary. You can achieve this by adding the following to your
Andrew Geisslerc926e172021-05-07 16:11:35 -050011039``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011040
11041 COPY_LIC_MANIFEST = "1"
11042 COPY_LIC_DIRS = "1"
11043 LICENSE_CREATE_PACKAGE = "1"
11044
11045Adding these statements to the
11046configuration file ensures that the licenses collected during package
11047generation are included on your image.
11048
11049.. note::
11050
11051 Setting all three variables to "1" results in the image having two
11052 copies of the same license file. One copy resides in
11053 ``/usr/share/common-licenses`` and the other resides in
11054 ``/usr/share/license``.
11055
11056 The reason for this behavior is because
11057 :term:`COPY_LIC_DIRS` and
11058 :term:`COPY_LIC_MANIFEST`
11059 add a copy of the license when the image is built but do not offer a
11060 path for adding licenses for newly installed packages to an image.
11061 :term:`LICENSE_CREATE_PACKAGE`
11062 adds a separate package and an upgrade path for adding licenses to an
11063 image.
11064
11065As the source ``archiver`` class has already archived the original
11066unmodified source that contains the license files, you would have
11067already met the requirements for inclusion of the license information
11068with source as defined by the GPL and other open source licenses.
11069
11070Providing Compilation Scripts and Source Code Modifications
11071~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
11072
11073At this point, we have addressed all we need to prior to generating the
11074image. The next two requirements are addressed during the final
11075packaging of the release.
11076
11077By releasing the version of the OpenEmbedded build system and the layers
11078used during the build, you will be providing both compilation scripts
11079and the source code modifications in one step.
11080
Andrew Geissler09209ee2020-12-13 08:44:15 -060011081If the deployment team has a :ref:`overview-manual/concepts:bsp layer`
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011082and a distro layer, and those
11083those layers are used to patch, compile, package, or modify (in any way)
11084any open source software included in your released images, you might be
11085required to release those layers under section 3 of GPLv2 or section 1
11086of GPLv3. One way of doing that is with a clean checkout of the version
11087of the Yocto Project and layers used during your build. Here is an
11088example:
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011089
11090.. code-block:: shell
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011091
11092 # We built using the dunfell branch of the poky repo
11093 $ git clone -b dunfell git://git.yoctoproject.org/poky
11094 $ cd poky
11095 # We built using the release_branch for our layers
11096 $ git clone -b release_branch git://git.mycompany.com/meta-my-bsp-layer
11097 $ git clone -b release_branch git://git.mycompany.com/meta-my-software-layer
11098 # clean up the .git repos
11099 $ find . -name ".git" -type d -exec rm -rf {} \;
11100
11101One
11102thing a development organization might want to consider for end-user
11103convenience is to modify ``meta-poky/conf/bblayers.conf.sample`` to
11104ensure that when the end user utilizes the released build system to
11105build an image, the development organization's layers are included in
Andrew Geisslerc926e172021-05-07 16:11:35 -050011106the ``bblayers.conf`` file automatically::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011107
11108 # POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
11109 # changes incompatibly
11110 POKY_BBLAYERS_CONF_VERSION = "2"
11111
11112 BBPATH = "${TOPDIR}"
11113 BBFILES ?= ""
11114
11115 BBLAYERS ?= " \
11116 ##OEROOT##/meta \
11117 ##OEROOT##/meta-poky \
11118 ##OEROOT##/meta-yocto-bsp \
11119 ##OEROOT##/meta-mylayer \
11120 "
11121
11122Creating and
11123providing an archive of the :term:`Metadata`
11124layers (recipes, configuration files, and so forth) enables you to meet
11125your requirements to include the scripts to control compilation as well
11126as any modifications to the original source.
11127
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011128Providing spdx files
11129~~~~~~~~~~~~~~~~~~~~~~~~~
11130
11131The spdx module has been integrated to a layer named meta-spdxscanner.
11132meta-spdxscanner provides several kinds of scanner. If you want to enable
11133this function, you have to follow the following steps:
11134
111351. Add meta-spdxscanner layer into ``bblayers.conf``.
11136
111372. Refer to the README in meta-spdxscanner to setup the environment (e.g,
11138 setup a fossology server) needed for the scanner.
11139
111403. Meta-spdxscanner provides several methods within the bbclass to create spdx files.
11141 Please choose one that you want to use and enable the spdx task. You have to
11142 add some config options in ``local.conf`` file in your :term:`Build
William A. Kennington IIIac69b482021-06-02 12:28:27 -070011143 Directory`. Here is an example showing how to generate spdx files
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011144 during bitbake using the fossology-python.bbclass::
11145
11146 # Select fossology-python.bbclass.
11147 INHERIT += "fossology-python"
11148 # For fossology-python.bbclass, TOKEN is necessary, so, after setup a
11149 # Fossology server, you have to create a token.
11150 TOKEN = "eyJ0eXAiO..."
11151 # The fossology server is necessary for fossology-python.bbclass.
11152 FOSSOLOGY_SERVER = "http://xx.xx.xx.xx:8081/repo"
11153 # If you want to upload the source code to a special folder:
11154 FOLDER_NAME = "xxxx" //Optional
11155 # If you don't want to put spdx files in tmp/deploy/spdx, you can enable:
11156 SPDX_DEPLOY_DIR = "${DEPLOY_DIR}" //Optional
11157
11158For more usage information refer to :yocto_git:`the meta-spdxscanner repository
Andrew Geissler09209ee2020-12-13 08:44:15 -060011159</meta-spdxscanner/>`.
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011160
Andrew Geisslereff27472021-10-29 15:35:00 -050011161Compliance Limitations with Executables Built from Static Libraries
11162~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011163
Andrew Geisslereff27472021-10-29 15:35:00 -050011164When package A is added to an image via the :term:`RDEPENDS` or :term:`RRECOMMENDS`
11165mechanisms as well as explicitly included in the image recipe with
11166:term:`IMAGE_INSTALL`, and depends on a static linked library recipe B
11167(``DEPENDS += "B"``), package B will neither appear in the generated license
11168manifest nor in the generated source tarballs. This occurs as the
11169:ref:`license <ref-classes-license>` and :ref:`archiver <ref-classes-archiver>`
11170classes assume that only packages included via :term:`RDEPENDS` or :term:`RRECOMMENDS`
11171end up in the image.
11172
11173As a result, potential obligations regarding license compliance for package B
11174may not be met.
11175
11176The Yocto Project doesn't enable static libraries by default, in part because
11177of this issue. Before a solution to this limitation is found, you need to
11178keep in mind that if your root filesystem is built from static libraries,
11179you will need to manually ensure that your deliveries are compliant
11180with the licenses of these libraries.
11181
11182Copying Non Standard Licenses
11183-----------------------------
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011184
11185Some packages, such as the linux-firmware package, have many licenses
11186that are not in any way common. You can avoid adding a lot of these
11187types of common license files, which are only applicable to a specific
11188package, by using the
11189:term:`NO_GENERIC_LICENSE`
11190variable. Using this variable also avoids QA errors when you use a
11191non-common, non-CLOSED license in a recipe.
11192
William A. Kennington IIIac69b482021-06-02 12:28:27 -070011193Here is an example that uses the ``LICENSE.Abilis.txt`` file as
Andrew Geisslerc926e172021-05-07 16:11:35 -050011194the license from the fetched source::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011195
11196 NO_GENERIC_LICENSE[Firmware-Abilis] = "LICENSE.Abilis.txt"
11197
Patrick Williams213cb262021-08-07 19:21:33 -050011198Checking for Vulnerabilities
11199============================
11200
11201Vulnerabilities in images
11202-------------------------
11203
11204The Yocto Project has an infrastructure to track and address unfixed
11205known security vulnerabilities, as tracked by the public
11206`Common Vulnerabilities and Exposures (CVE) <https://en.wikipedia.org/wiki/Common_Vulnerabilities_and_Exposures>`__
11207database.
11208
11209To know which packages are vulnerable to known security vulnerabilities,
11210add the following setting to your configuration::
11211
11212 INHERIT += "cve-check"
11213
11214This way, at build time, BitBake will warn you about known CVEs
11215as in the example below::
11216
11217 WARNING: flex-2.6.4-r0 do_cve_check: Found unpatched CVE (CVE-2019-6293), for more information check /poky/build/tmp/work/core2-64-poky-linux/flex/2.6.4-r0/temp/cve.log
11218 WARNING: libarchive-3.5.1-r0 do_cve_check: Found unpatched CVE (CVE-2021-36976), for more information check /poky/build/tmp/work/core2-64-poky-linux/libarchive/3.5.1-r0/temp/cve.log
11219
11220It is also possible to check the CVE status of individual packages as follows::
11221
11222 bitbake -c cve_check flex libarchive
11223
11224Note that OpenEmbedded-Core keeps a list of known unfixed CVE issues which can
11225be ignored. You can pass this list to the check as follows::
11226
11227 bitbake -c cve_check libarchive -R conf/distro/include/cve-extra-exclusions.inc
11228
11229Enabling vulnerabily tracking in recipes
11230----------------------------------------
11231
11232The :term:`CVE_PRODUCT` variable defines the name used to match the recipe name
11233against the name in the upstream `NIST CVE database <https://nvd.nist.gov/>`__.
11234
Patrick Williams0ca19cc2021-08-16 14:03:13 -050011235Editing recipes to fix vulnerabilities
11236--------------------------------------
11237
11238To fix a given known vulnerability, you need to add a patch file to your recipe. Here's
11239an example from the :oe_layerindex:`ffmpeg recipe</layerindex/recipe/47350>`::
11240
11241 SRC_URI = "https://www.ffmpeg.org/releases/${BP}.tar.xz \
11242 file://0001-libavutil-include-assembly-with-full-path-from-sourc.patch \
11243 file://fix-CVE-2020-20446.patch \
11244 file://fix-CVE-2020-20453.patch \
11245 file://fix-CVE-2020-22015.patch \
11246 file://fix-CVE-2020-22021.patch \
11247 file://fix-CVE-2020-22033-CVE-2020-22019.patch \
11248 file://fix-CVE-2021-33815.patch \
11249
11250The :ref:`cve-check <ref-classes-cve-check>` class defines two ways of
11251supplying a patch for a given CVE. The first
11252way is to use a patch filename that matches the below pattern::
11253
11254 cve_file_name_match = re.compile(".*([Cc][Vv][Ee]\-\d{4}\-\d+)")
11255
11256As shown in the example above, multiple CVE IDs can appear in a patch filename,
11257but the :ref:`cve-check <ref-classes-cve-check>` class will only consider
11258the last CVE ID in the filename as patched.
11259
11260The second way to recognize a patched CVE ID is when a line matching the
11261below pattern is found in any patch file provided by the recipe::
11262
11263 cve_match = re.compile("CVE:( CVE\-\d{4}\-\d+)+")
11264
11265This allows a single patch file to address multiple CVE IDs at the same time.
11266
11267Of course, another way to fix vulnerabilities is to upgrade to a version
11268of the package which is not impacted, typically a more recent one.
11269The NIST database knows which versions are vulnerable and which ones
11270are not.
11271
11272Last but not least, you can choose to ignore vulnerabilities through
11273the :term:`CVE_CHECK_PN_WHITELIST` and :term:`CVE_CHECK_WHITELIST`
11274variables.
11275
11276Implementation details
11277----------------------
11278
11279Here's what the :ref:`cve-check <ref-classes-cve-check>` class does to
11280find unpatched CVE IDs.
11281
11282First the code goes through each patch file provided by a recipe. If a valid CVE ID
11283is found in the name of the file, the corresponding CVE is considered as patched.
11284Don't forget that if multiple CVE IDs are found in the filename, only the last
11285one is considered. Then, the code looks for ``CVE: CVE-ID`` lines in the patch
11286file. The found CVE IDs are also considered as patched.
11287
11288Then, the code looks up all the CVE IDs in the NIST database for all the
11289products defined in :term:`CVE_PRODUCT`. Then, for each found CVE:
11290
11291 - If the package name (:term:`PN`) is part of
11292 :term:`CVE_CHECK_PN_WHITELIST`, it is considered as patched.
11293
11294 - If the CVE ID is part of :term:`CVE_CHECK_WHITELIST`, it is
11295 considered as patched too.
11296
11297 - If the CVE ID is part of the patched CVE for the recipe, it is
11298 already considered as patched.
11299
11300 - Otherwise, the code checks whether the recipe version (:term:`PV`)
11301 is within the range of versions impacted by the CVE. If so, the CVE
11302 is considered as unpatched.
11303
Patrick Williams213cb262021-08-07 19:21:33 -050011304The CVE database is stored in :term:`DL_DIR` and can be inspected using
11305``sqlite3`` command as follows::
11306
11307 sqlite3 downloads/CVE_CHECK/nvdcve_1.1.db .dump | grep CVE-2021-37462
11308
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011309Using the Error Reporting Tool
11310==============================
11311
11312The error reporting tool allows you to submit errors encountered during
11313builds to a central database. Outside of the build environment, you can
11314use a web interface to browse errors, view statistics, and query for
11315errors. The tool works using a client-server system where the client
11316portion is integrated with the installed Yocto Project
11317:term:`Source Directory` (e.g. ``poky``).
11318The server receives the information collected and saves it in a
11319database.
11320
William A. Kennington IIIac69b482021-06-02 12:28:27 -070011321There is a live instance of the error reporting server at
11322https://errors.yoctoproject.org.
11323When you want to get help with build failures, you can submit all of the
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011324information on the failure easily and then point to the URL in your bug
11325report or send an email to the mailing list.
11326
11327.. note::
11328
11329 If you send error reports to this server, the reports become publicly
11330 visible.
11331
11332Enabling and Using the Tool
11333---------------------------
11334
11335By default, the error reporting tool is disabled. You can enable it by
11336inheriting the
11337:ref:`report-error <ref-classes-report-error>`
11338class by adding the following statement to the end of your
11339``local.conf`` file in your
11340:term:`Build Directory`.
11341::
11342
11343 INHERIT += "report-error"
11344
11345By default, the error reporting feature stores information in
11346``${``\ :term:`LOG_DIR`\ ``}/error-report``.
11347However, you can specify a directory to use by adding the following to
Andrew Geisslerc926e172021-05-07 16:11:35 -050011348your ``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011349
11350 ERR_REPORT_DIR = "path"
11351
11352Enabling error
11353reporting causes the build process to collect the errors and store them
11354in a file as previously described. When the build system encounters an
11355error, it includes a command as part of the console output. You can run
11356the command to send the error file to the server. For example, the
Andrew Geisslerc926e172021-05-07 16:11:35 -050011357following command sends the errors to an upstream server::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011358
11359 $ send-error-report /home/brandusa/project/poky/build/tmp/log/error-report/error_report_201403141617.txt
11360
11361In the previous example, the errors are sent to a public database
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011362available at https://errors.yoctoproject.org, which is used by the
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011363entire community. If you specify a particular server, you can send the
11364errors to a different database. Use the following command for more
Andrew Geisslerc926e172021-05-07 16:11:35 -050011365information on available options::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011366
11367 $ send-error-report --help
11368
11369When sending the error file, you are prompted to review the data being
11370sent as well as to provide a name and optional email address. Once you
11371satisfy these prompts, the command returns a link from the server that
11372corresponds to your entry in the database. For example, here is a
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011373typical link: https://errors.yoctoproject.org/Errors/Details/9522/
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011374
11375Following the link takes you to a web interface where you can browse,
11376query the errors, and view statistics.
11377
11378Disabling the Tool
11379------------------
11380
11381To disable the error reporting feature, simply remove or comment out the
11382following statement from the end of your ``local.conf`` file in your
11383:term:`Build Directory`.
11384::
11385
11386 INHERIT += "report-error"
11387
11388Setting Up Your Own Error Reporting Server
11389------------------------------------------
11390
11391If you want to set up your own error reporting server, you can obtain
Andrew Geissler09209ee2020-12-13 08:44:15 -060011392the code from the Git repository at :yocto_git:`/error-report-web/`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011393Instructions on how to set it up are in the README document.
11394
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011395Using Wayland and Weston
11396========================
11397
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011398`Wayland <https://en.wikipedia.org/wiki/Wayland_(display_server_protocol)>`__
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011399is a computer display server protocol that provides a method for
11400compositing window managers to communicate directly with applications
11401and video hardware and expects them to communicate with input hardware
11402using other libraries. Using Wayland with supporting targets can result
11403in better control over graphics frame rendering than an application
11404might otherwise achieve.
11405
11406The Yocto Project provides the Wayland protocol libraries and the
11407reference
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011408`Weston <https://en.wikipedia.org/wiki/Wayland_(display_server_protocol)#Weston>`__
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011409compositor as part of its release. You can find the integrated packages
11410in the ``meta`` layer of the :term:`Source Directory`.
11411Specifically, you
11412can find the recipes that build both Wayland and Weston at
11413``meta/recipes-graphics/wayland``.
11414
11415You can build both the Wayland and Weston packages for use only with
11416targets that accept the `Mesa 3D and Direct Rendering
11417Infrastructure <https://en.wikipedia.org/wiki/Mesa_(computer_graphics)>`__,
11418which is also known as Mesa DRI. This implies that you cannot build and
11419use the packages if your target uses, for example, the Intel Embedded
11420Media and Graphics Driver (Intel EMGD) that overrides Mesa DRI.
11421
11422.. note::
11423
11424 Due to lack of EGL support, Weston 1.0.3 will not run directly on the
11425 emulated QEMU hardware. However, this version of Weston will run
11426 under X emulation without issues.
11427
11428This section describes what you need to do to implement Wayland and use
11429the Weston compositor when building an image for a supporting target.
11430
11431Enabling Wayland in an Image
11432----------------------------
11433
11434To enable Wayland, you need to enable it to be built and enable it to be
11435included (installed) in the image.
11436
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011437Building Wayland
11438~~~~~~~~~~~~~~~~
11439
11440To cause Mesa to build the ``wayland-egl`` platform and Weston to build
11441Wayland with Kernel Mode Setting
11442(`KMS <https://wiki.archlinux.org/index.php/Kernel_Mode_Setting>`__)
11443support, include the "wayland" flag in the
11444:term:`DISTRO_FEATURES`
Andrew Geisslerc926e172021-05-07 16:11:35 -050011445statement in your ``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011446
Patrick Williams0ca19cc2021-08-16 14:03:13 -050011447 DISTRO_FEATURES:append = " wayland"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011448
11449.. note::
11450
11451 If X11 has been enabled elsewhere, Weston will build Wayland with X11
11452 support
11453
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011454Installing Wayland and Weston
11455~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
11456
11457To install the Wayland feature into an image, you must include the
11458following
11459:term:`CORE_IMAGE_EXTRA_INSTALL`
Andrew Geisslerc926e172021-05-07 16:11:35 -050011460statement in your ``local.conf`` file::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011461
11462 CORE_IMAGE_EXTRA_INSTALL += "wayland weston"
11463
11464Running Weston
11465--------------
11466
11467To run Weston inside X11, enabling it as described earlier and building
11468a Sato image is sufficient. If you are running your image under Sato, a
11469Weston Launcher appears in the "Utility" category.
11470
11471Alternatively, you can run Weston through the command-line interpretor
11472(CLI), which is better suited for development work. To run Weston under
11473the CLI, you need to do the following after your image is built:
11474
Andrew Geisslerc926e172021-05-07 16:11:35 -0500114751. Run these commands to export ``XDG_RUNTIME_DIR``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011476
11477 mkdir -p /tmp/$USER-weston
11478 chmod 0700 /tmp/$USER-weston
11479 export XDG_RUNTIME_DIR=/tmp/$USER-weston
11480
Andrew Geisslerc926e172021-05-07 16:11:35 -0500114812. Launch Weston in the shell::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011482
11483 weston