blob: ada3bac7e1fb19fa0be00fbfb9ccccc441cc1b43 [file] [log] [blame]
Andrew Geisslerf0343792020-11-18 10:42:21 -06001.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002
3************
4Common Tasks
5************
6
7This chapter describes fundamental procedures such as creating layers,
8adding new software packages, extending or customizing images, porting
9work to new hardware (adding a new machine), and so forth. You will find
10that the procedures documented here occur often in the development cycle
11using the Yocto Project.
12
13Understanding and Creating Layers
14=================================
15
16The OpenEmbedded build system supports organizing
17:term:`Metadata` into multiple layers.
18Layers allow you to isolate different types of customizations from each
19other. For introductory information on the Yocto Project Layer Model,
20see the
Andrew Geissler09209ee2020-12-13 08:44:15 -060021":ref:`overview-manual/yp-intro:the yocto project layer model`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050022section in the Yocto Project Overview and Concepts Manual.
23
24Creating Your Own Layer
25-----------------------
26
27It is very easy to create your own layers to use with the OpenEmbedded
28build system. The Yocto Project ships with tools that speed up creating
29layers. This section describes the steps you perform by hand to create
30layers so that you can better understand them. For information about the
31layer-creation tools, see the
32":ref:`bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script`"
33section in the Yocto Project Board Support Package (BSP) Developer's
Andrew Geissler09209ee2020-12-13 08:44:15 -060034Guide and the ":ref:`dev-manual/common-tasks:creating a general layer using the \`\`bitbake-layers\`\` script`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050035section further down in this manual.
36
37Follow these general steps to create your layer without using tools:
38
391. *Check Existing Layers:* Before creating a new layer, you should be
40 sure someone has not already created a layer containing the Metadata
41 you need. You can see the `OpenEmbedded Metadata
Andrew Geissler4c19ea12020-10-27 13:52:24 -050042 Index <https://layers.openembedded.org/layerindex/layers/>`__ for a
Andrew Geisslerc9f78652020-09-18 14:11:35 -050043 list of layers from the OpenEmbedded community that can be used in
44 the Yocto Project. You could find a layer that is identical or close
45 to what you need.
46
472. *Create a Directory:* Create the directory for your layer. When you
48 create the layer, be sure to create the directory in an area not
49 associated with the Yocto Project :term:`Source Directory`
50 (e.g. the cloned ``poky`` repository).
51
52 While not strictly required, prepend the name of the directory with
53 the string "meta-". For example:
54 ::
55
56 meta-mylayer
57 meta-GUI_xyz
58 meta-mymachine
59
60 With rare exceptions, a layer's name follows this form:
61 ::
62
63 meta-root_name
64
65 Following this layer naming convention can save
66 you trouble later when tools, components, or variables "assume" your
67 layer name begins with "meta-". A notable example is in configuration
68 files as shown in the following step where layer names without the
69 "meta-" string are appended to several variables used in the
70 configuration.
71
723. *Create a Layer Configuration File:* Inside your new layer folder,
73 you need to create a ``conf/layer.conf`` file. It is easiest to take
74 an existing layer configuration file and copy that to your layer's
75 ``conf`` directory and then modify the file as needed.
76
77 The ``meta-yocto-bsp/conf/layer.conf`` file in the Yocto Project
Andrew Geissler09209ee2020-12-13 08:44:15 -060078 :yocto_git:`Source Repositories </poky/tree/meta-yocto-bsp/conf>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -050079 demonstrates the required syntax. For your layer, you need to replace
80 "yoctobsp" with a unique identifier for your layer (e.g. "machinexyz"
81 for a layer named "meta-machinexyz"):
82 ::
83
84 # We have a conf and classes directory, add to BBPATH
85 BBPATH .= ":${LAYERDIR}"
86
Andrew Geissler4c19ea12020-10-27 13:52:24 -050087 # We have recipes-* directories, add to BBFILES
Andrew Geisslerc9f78652020-09-18 14:11:35 -050088 BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
89 ${LAYERDIR}/recipes-*/*/*.bbappend"
90
91 BBFILE_COLLECTIONS += "yoctobsp"
92 BBFILE_PATTERN_yoctobsp = "^${LAYERDIR}/"
93 BBFILE_PRIORITY_yoctobsp = "5"
94 LAYERVERSION_yoctobsp = "4"
95 LAYERSERIES_COMPAT_yoctobsp = "dunfell"
96
97 Following is an explanation of the layer configuration file:
98
99 - :term:`BBPATH`: Adds the layer's
100 root directory to BitBake's search path. Through the use of the
101 ``BBPATH`` variable, BitBake locates class files (``.bbclass``),
102 configuration files, and files that are included with ``include``
103 and ``require`` statements. For these cases, BitBake uses the
104 first file that matches the name found in ``BBPATH``. This is
105 similar to the way the ``PATH`` variable is used for binaries. It
106 is recommended, therefore, that you use unique class and
107 configuration filenames in your custom layer.
108
109 - :term:`BBFILES`: Defines the
110 location for all recipes in the layer.
111
112 - :term:`BBFILE_COLLECTIONS`:
113 Establishes the current layer through a unique identifier that is
114 used throughout the OpenEmbedded build system to refer to the
115 layer. In this example, the identifier "yoctobsp" is the
116 representation for the container layer named "meta-yocto-bsp".
117
118 - :term:`BBFILE_PATTERN`:
119 Expands immediately during parsing to provide the directory of the
120 layer.
121
122 - :term:`BBFILE_PRIORITY`:
123 Establishes a priority to use for recipes in the layer when the
124 OpenEmbedded build finds recipes of the same name in different
125 layers.
126
127 - :term:`LAYERVERSION`:
128 Establishes a version number for the layer. You can use this
129 version number to specify this exact version of the layer as a
130 dependency when using the
131 :term:`LAYERDEPENDS`
132 variable.
133
134 - :term:`LAYERDEPENDS`:
135 Lists all layers on which this layer depends (if any).
136
137 - :term:`LAYERSERIES_COMPAT`:
Andrew Geissler09209ee2020-12-13 08:44:15 -0600138 Lists the :yocto_wiki:`Yocto Project </Releases>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500139 releases for which the current version is compatible. This
140 variable is a good way to indicate if your particular layer is
141 current.
142
1434. *Add Content:* Depending on the type of layer, add the content. If
144 the layer adds support for a machine, add the machine configuration
145 in a ``conf/machine/`` file within the layer. If the layer adds
146 distro policy, add the distro configuration in a ``conf/distro/``
147 file within the layer. If the layer introduces new recipes, put the
148 recipes you need in ``recipes-*`` subdirectories within the layer.
149
150 .. note::
151
152 For an explanation of layer hierarchy that is compliant with the
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500153 Yocto Project, see the ":ref:`bsp-guide/bsp:example filesystem layout`"
154 section in the Yocto Project Board Support Package (BSP) Developer's Guide.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500155
1565. *Optionally Test for Compatibility:* If you want permission to use
157 the Yocto Project Compatibility logo with your layer or application
158 that uses your layer, perform the steps to apply for compatibility.
159 See the "`Making Sure Your Layer is Compatible With Yocto
160 Project <#making-sure-your-layer-is-compatible-with-yocto-project>`__"
161 section for more information.
162
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500163Following Best Practices When Creating Layers
164---------------------------------------------
165
166To create layers that are easier to maintain and that will not impact
167builds for other machines, you should consider the information in the
168following list:
169
170- *Avoid "Overlaying" Entire Recipes from Other Layers in Your
171 Configuration:* In other words, do not copy an entire recipe into
172 your layer and then modify it. Rather, use an append file
173 (``.bbappend``) to override only those parts of the original recipe
174 you need to modify.
175
176- *Avoid Duplicating Include Files:* Use append files (``.bbappend``)
177 for each recipe that uses an include file. Or, if you are introducing
178 a new recipe that requires the included file, use the path relative
179 to the original layer directory to refer to the file. For example,
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500180 use ``require recipes-core/``\ `package`\ ``/``\ `file`\ ``.inc`` instead
181 of ``require`` `file`\ ``.inc``. If you're finding you have to overlay
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500182 the include file, it could indicate a deficiency in the include file
183 in the layer to which it originally belongs. If this is the case, you
184 should try to address that deficiency instead of overlaying the
185 include file. For example, you could address this by getting the
186 maintainer of the include file to add a variable or variables to make
187 it easy to override the parts needing to be overridden.
188
189- *Structure Your Layers:* Proper use of overrides within append files
190 and placement of machine-specific files within your layer can ensure
191 that a build is not using the wrong Metadata and negatively impacting
192 a build for a different machine. Following are some examples:
193
194 - *Modify Variables to Support a Different Machine:* Suppose you
195 have a layer named ``meta-one`` that adds support for building
196 machine "one". To do so, you use an append file named
197 ``base-files.bbappend`` and create a dependency on "foo" by
198 altering the :term:`DEPENDS`
199 variable:
200 ::
201
202 DEPENDS = "foo"
203
204 The dependency is created during any
205 build that includes the layer ``meta-one``. However, you might not
206 want this dependency for all machines. For example, suppose you
207 are building for machine "two" but your ``bblayers.conf`` file has
208 the ``meta-one`` layer included. During the build, the
209 ``base-files`` for machine "two" will also have the dependency on
210 ``foo``.
211
212 To make sure your changes apply only when building machine "one",
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500213 use a machine override with the ``DEPENDS`` statement:
214 ::
215
216 DEPENDS_one = "foo"
217
218 You should follow the same strategy when using ``_append``
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500219 and ``_prepend`` operations:
220 ::
221
222 DEPENDS_append_one = " foo"
223 DEPENDS_prepend_one = "foo "
224
225 As an actual example, here's a
226 snippet from the generic kernel include file ``linux-yocto.inc``,
227 wherein the kernel compile and link options are adjusted in the
228 case of a subset of the supported architectures:
229 ::
230
231 DEPENDS_append_aarch64 = " libgcc"
232 KERNEL_CC_append_aarch64 = " ${TOOLCHAIN_OPTIONS}"
233 KERNEL_LD_append_aarch64 = " ${TOOLCHAIN_OPTIONS}"
234
235 DEPENDS_append_nios2 = " libgcc"
236 KERNEL_CC_append_nios2 = " ${TOOLCHAIN_OPTIONS}"
237 KERNEL_LD_append_nios2 = " ${TOOLCHAIN_OPTIONS}"
238
239 DEPENDS_append_arc = " libgcc"
240 KERNEL_CC_append_arc = " ${TOOLCHAIN_OPTIONS}"
241 KERNEL_LD_append_arc = " ${TOOLCHAIN_OPTIONS}"
242
243 KERNEL_FEATURES_append_qemuall=" features/debug/printk.scc"
244
245 .. note::
246
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500247 Avoiding "+=" and "=+" and using machine-specific ``_append``
248 and ``_prepend`` operations is recommended as well.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500249
250 - *Place Machine-Specific Files in Machine-Specific Locations:* When
251 you have a base recipe, such as ``base-files.bb``, that contains a
252 :term:`SRC_URI` statement to a
253 file, you can use an append file to cause the build to use your
254 own version of the file. For example, an append file in your layer
255 at ``meta-one/recipes-core/base-files/base-files.bbappend`` could
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500256 extend :term:`FILESPATH` using :term:`FILESEXTRAPATHS` as follows:
257 ::
258
259 FILESEXTRAPATHS_prepend := "${THISDIR}/${BPN}:"
260
261 The build for machine "one" will pick up your machine-specific file as
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500262 long as you have the file in
263 ``meta-one/recipes-core/base-files/base-files/``. However, if you
264 are building for a different machine and the ``bblayers.conf``
265 file includes the ``meta-one`` layer and the location of your
266 machine-specific file is the first location where that file is
267 found according to ``FILESPATH``, builds for all machines will
268 also use that machine-specific file.
269
270 You can make sure that a machine-specific file is used for a
271 particular machine by putting the file in a subdirectory specific
272 to the machine. For example, rather than placing the file in
273 ``meta-one/recipes-core/base-files/base-files/`` as shown above,
274 put it in ``meta-one/recipes-core/base-files/base-files/one/``.
275 Not only does this make sure the file is used only when building
276 for machine "one", but the build process locates the file more
277 quickly.
278
279 In summary, you need to place all files referenced from
280 ``SRC_URI`` in a machine-specific subdirectory within the layer in
281 order to restrict those files to machine-specific builds.
282
283- *Perform Steps to Apply for Yocto Project Compatibility:* If you want
284 permission to use the Yocto Project Compatibility logo with your
285 layer or application that uses your layer, perform the steps to apply
286 for compatibility. See the "`Making Sure Your Layer is Compatible
287 With Yocto
288 Project <#making-sure-your-layer-is-compatible-with-yocto-project>`__"
289 section for more information.
290
291- *Follow the Layer Naming Convention:* Store custom layers in a Git
292 repository that use the ``meta-layer_name`` format.
293
294- *Group Your Layers Locally:* Clone your repository alongside other
295 cloned ``meta`` directories from the :term:`Source Directory`.
296
297Making Sure Your Layer is Compatible With Yocto Project
298-------------------------------------------------------
299
300When you create a layer used with the Yocto Project, it is advantageous
301to make sure that the layer interacts well with existing Yocto Project
302layers (i.e. the layer is compatible with the Yocto Project). Ensuring
303compatibility makes the layer easy to be consumed by others in the Yocto
304Project community and could allow you permission to use the Yocto
305Project Compatible Logo.
306
307.. note::
308
309 Only Yocto Project member organizations are permitted to use the
310 Yocto Project Compatible Logo. The logo is not available for general
311 use. For information on how to become a Yocto Project member
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500312 organization, see the :yocto_home:`Yocto Project Website <>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500313
314The Yocto Project Compatibility Program consists of a layer application
315process that requests permission to use the Yocto Project Compatibility
316Logo for your layer and application. The process consists of two parts:
317
3181. Successfully passing a script (``yocto-check-layer``) that when run
319 against your layer, tests it against constraints based on experiences
320 of how layers have worked in the real world and where pitfalls have
321 been found. Getting a "PASS" result from the script is required for
322 successful compatibility registration.
323
3242. Completion of an application acceptance form, which you can find at
325 https://www.yoctoproject.org/webform/yocto-project-compatible-registration.
326
327To be granted permission to use the logo, you need to satisfy the
328following:
329
330- Be able to check the box indicating that you got a "PASS" when
331 running the script against your layer.
332
333- Answer "Yes" to the questions on the form or have an acceptable
334 explanation for any questions answered "No".
335
336- Be a Yocto Project Member Organization.
337
338The remainder of this section presents information on the registration
339form and on the ``yocto-check-layer`` script.
340
341Yocto Project Compatible Program Application
342~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
343
344Use the form to apply for your layer's approval. Upon successful
345application, you can use the Yocto Project Compatibility Logo with your
346layer and the application that uses your layer.
347
348To access the form, use this link:
349https://www.yoctoproject.org/webform/yocto-project-compatible-registration.
350Follow the instructions on the form to complete your application.
351
352The application consists of the following sections:
353
354- *Contact Information:* Provide your contact information as the fields
355 require. Along with your information, provide the released versions
356 of the Yocto Project for which your layer is compatible.
357
358- *Acceptance Criteria:* Provide "Yes" or "No" answers for each of the
359 items in the checklist. Space exists at the bottom of the form for
360 any explanations for items for which you answered "No".
361
362- *Recommendations:* Provide answers for the questions regarding Linux
363 kernel use and build success.
364
365``yocto-check-layer`` Script
366~~~~~~~~~~~~~~~~~~~~~~~~~~~~
367
368The ``yocto-check-layer`` script provides you a way to assess how
369compatible your layer is with the Yocto Project. You should run this
370script prior to using the form to apply for compatibility as described
371in the previous section. You need to achieve a "PASS" result in order to
372have your application form successfully processed.
373
374The script divides tests into three areas: COMMON, BSP, and DISTRO. For
375example, given a distribution layer (DISTRO), the layer must pass both
376the COMMON and DISTRO related tests. Furthermore, if your layer is a BSP
377layer, the layer must pass the COMMON and BSP set of tests.
378
379To execute the script, enter the following commands from your build
380directory:
381::
382
383 $ source oe-init-build-env
384 $ yocto-check-layer your_layer_directory
385
386Be sure to provide the actual directory for your
387layer as part of the command.
388
389Entering the command causes the script to determine the type of layer
390and then to execute a set of specific tests against the layer. The
391following list overviews the test:
392
393- ``common.test_readme``: Tests if a ``README`` file exists in the
394 layer and the file is not empty.
395
396- ``common.test_parse``: Tests to make sure that BitBake can parse the
397 files without error (i.e. ``bitbake -p``).
398
399- ``common.test_show_environment``: Tests that the global or per-recipe
400 environment is in order without errors (i.e. ``bitbake -e``).
401
402- ``common.test_world``: Verifies that ``bitbake world`` works.
403
404- ``common.test_signatures``: Tests to be sure that BSP and DISTRO
405 layers do not come with recipes that change signatures.
406
407- ``common.test_layerseries_compat``: Verifies layer compatibility is
408 set properly.
409
410- ``bsp.test_bsp_defines_machines``: Tests if a BSP layer has machine
411 configurations.
412
413- ``bsp.test_bsp_no_set_machine``: Tests to ensure a BSP layer does not
414 set the machine when the layer is added.
415
416- ``bsp.test_machine_world``: Verifies that ``bitbake world`` works
417 regardless of which machine is selected.
418
419- ``bsp.test_machine_signatures``: Verifies that building for a
420 particular machine affects only the signature of tasks specific to
421 that machine.
422
423- ``distro.test_distro_defines_distros``: Tests if a DISTRO layer has
424 distro configurations.
425
426- ``distro.test_distro_no_set_distros``: Tests to ensure a DISTRO layer
427 does not set the distribution when the layer is added.
428
429Enabling Your Layer
430-------------------
431
432Before the OpenEmbedded build system can use your new layer, you need to
433enable it. To enable your layer, simply add your layer's path to the
434``BBLAYERS`` variable in your ``conf/bblayers.conf`` file, which is
435found in the :term:`Build Directory`.
436The following example shows how to enable a layer named
437``meta-mylayer``:
438::
439
440 # POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
441 # changes incompatibly
442 POKY_BBLAYERS_CONF_VERSION = "2"
443 BBPATH = "${TOPDIR}"
444 BBFILES ?= ""
445 BBLAYERS ?= " \
446 /home/user/poky/meta \
447 /home/user/poky/meta-poky \
448 /home/user/poky/meta-yocto-bsp \
449 /home/user/poky/meta-mylayer \
450 "
451
452BitBake parses each ``conf/layer.conf`` file from the top down as
453specified in the ``BBLAYERS`` variable within the ``conf/bblayers.conf``
454file. During the processing of each ``conf/layer.conf`` file, BitBake
455adds the recipes, classes and configurations contained within the
456particular layer to the source directory.
457
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500458Using .bbappend Files in Your Layer
459-----------------------------------
460
461A recipe that appends Metadata to another recipe is called a BitBake
462append file. A BitBake append file uses the ``.bbappend`` file type
463suffix, while the corresponding recipe to which Metadata is being
464appended uses the ``.bb`` file type suffix.
465
466You can use a ``.bbappend`` file in your layer to make additions or
467changes to the content of another layer's recipe without having to copy
468the other layer's recipe into your layer. Your ``.bbappend`` file
469resides in your layer, while the main ``.bb`` recipe file to which you
470are appending Metadata resides in a different layer.
471
472Being able to append information to an existing recipe not only avoids
473duplication, but also automatically applies recipe changes from a
474different layer into your layer. If you were copying recipes, you would
475have to manually merge changes as they occur.
476
477When you create an append file, you must use the same root name as the
478corresponding recipe file. For example, the append file
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500479``someapp_3.1.bbappend`` must apply to ``someapp_3.1.bb``. This
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500480means the original recipe and append file names are version
481number-specific. If the corresponding recipe is renamed to update to a
482newer version, you must also rename and possibly update the
483corresponding ``.bbappend`` as well. During the build process, BitBake
484displays an error on starting if it detects a ``.bbappend`` file that
485does not have a corresponding recipe with a matching name. See the
486:term:`BB_DANGLINGAPPENDS_WARNONLY`
487variable for information on how to handle this error.
488
489As an example, consider the main formfactor recipe and a corresponding
490formfactor append file both from the :term:`Source Directory`.
491Here is the main
492formfactor recipe, which is named ``formfactor_0.0.bb`` and located in
493the "meta" layer at ``meta/recipes-bsp/formfactor``:
494::
495
496 SUMMARY = "Device formfactor information"
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500497 DESCRIPTION = "A formfactor configuration file provides information about the \
498 target hardware for which the image is being built and information that the \
499 build system cannot obtain from other sources such as the kernel."
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500500 SECTION = "base"
501 LICENSE = "MIT"
502 LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
503 PR = "r45"
504
505 SRC_URI = "file://config file://machconfig"
506 S = "${WORKDIR}"
507
508 PACKAGE_ARCH = "${MACHINE_ARCH}"
509 INHIBIT_DEFAULT_DEPS = "1"
510
511 do_install() {
512 # Install file only if it has contents
513 install -d ${D}${sysconfdir}/formfactor/
514 install -m 0644 ${S}/config ${D}${sysconfdir}/formfactor/
515 if [ -s "${S}/machconfig" ]; then
516 install -m 0644 ${S}/machconfig ${D}${sysconfdir}/formfactor/
517 fi
518 }
519
520In the main recipe, note the :term:`SRC_URI`
521variable, which tells the OpenEmbedded build system where to find files
522during the build.
523
524Following is the append file, which is named ``formfactor_0.0.bbappend``
525and is from the Raspberry Pi BSP Layer named ``meta-raspberrypi``. The
526file is in the layer at ``recipes-bsp/formfactor``:
527::
528
529 FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"
530
531By default, the build system uses the
532:term:`FILESPATH` variable to
533locate files. This append file extends the locations by setting the
534:term:`FILESEXTRAPATHS`
535variable. Setting this variable in the ``.bbappend`` file is the most
536reliable and recommended method for adding directories to the search
537path used by the build system to find files.
538
539The statement in this example extends the directories to include
540``${``\ :term:`THISDIR`\ ``}/${``\ :term:`PN`\ ``}``,
541which resolves to a directory named ``formfactor`` in the same directory
542in which the append file resides (i.e.
543``meta-raspberrypi/recipes-bsp/formfactor``. This implies that you must
544have the supporting directory structure set up that will contain any
545files or patches you will be including from the layer.
546
547Using the immediate expansion assignment operator ``:=`` is important
548because of the reference to ``THISDIR``. The trailing colon character is
549important as it ensures that items in the list remain colon-separated.
550
551.. note::
552
553 BitBake automatically defines the ``THISDIR`` variable. You should
554 never set this variable yourself. Using "_prepend" as part of the
555 ``FILESEXTRAPATHS`` ensures your path will be searched prior to other
556 paths in the final list.
557
558 Also, not all append files add extra files. Many append files simply
559 exist to add build options (e.g. ``systemd``). For these cases, your
560 append file would not even use the ``FILESEXTRAPATHS`` statement.
561
562Prioritizing Your Layer
563-----------------------
564
565Each layer is assigned a priority value. Priority values control which
566layer takes precedence if there are recipe files with the same name in
567multiple layers. For these cases, the recipe file from the layer with a
568higher priority number takes precedence. Priority values also affect the
569order in which multiple ``.bbappend`` files for the same recipe are
570applied. You can either specify the priority manually, or allow the
571build system to calculate it based on the layer's dependencies.
572
573To specify the layer's priority manually, use the
574:term:`BBFILE_PRIORITY`
575variable and append the layer's root name:
576::
577
578 BBFILE_PRIORITY_mylayer = "1"
579
580.. note::
581
582 It is possible for a recipe with a lower version number
583 :term:`PV` in a layer that has a higher
584 priority to take precedence.
585
586 Also, the layer priority does not currently affect the precedence
587 order of ``.conf`` or ``.bbclass`` files. Future versions of BitBake
588 might address this.
589
590Managing Layers
591---------------
592
593You can use the BitBake layer management tool ``bitbake-layers`` to
594provide a view into the structure of recipes across a multi-layer
595project. Being able to generate output that reports on configured layers
596with their paths and priorities and on ``.bbappend`` files and their
597applicable recipes can help to reveal potential problems.
598
599For help on the BitBake layer management tool, use the following
600command:
601::
602
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500603 $ bitbake-layers --help
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500604 NOTE: Starting bitbake server...
605 usage: bitbake-layers [-d] [-q] [-F] [--color COLOR] [-h] <subcommand> ...
606
607 BitBake layers utility
608
609 optional arguments:
610 -d, --debug Enable debug output
611 -q, --quiet Print only errors
612 -F, --force Force add without recipe parse verification
613 --color COLOR Colorize output (where COLOR is auto, always, never)
614 -h, --help show this help message and exit
615
616 subcommands:
617 <subcommand>
618 layerindex-fetch Fetches a layer from a layer index along with its
619 dependent layers, and adds them to conf/bblayers.conf.
620 layerindex-show-depends
621 Find layer dependencies from layer index.
622 add-layer Add one or more layers to bblayers.conf.
623 remove-layer Remove one or more layers from bblayers.conf.
624 flatten flatten layer configuration into a separate output
625 directory.
626 show-layers show current configured layers.
627 show-overlayed list overlayed recipes (where the same recipe exists
628 in another layer)
629 show-recipes list available recipes, showing the layer they are
630 provided by
631 show-appends list bbappend files and recipe files they apply to
632 show-cross-depends Show dependencies between recipes that cross layer
633 boundaries.
634 create-layer Create a basic layer
635
636 Use bitbake-layers <subcommand> --help to get help on a specific command
637
638The following list describes the available commands:
639
640- ``help:`` Displays general help or help on a specified command.
641
642- ``show-layers:`` Shows the current configured layers.
643
644- ``show-overlayed:`` Lists overlayed recipes. A recipe is overlayed
645 when a recipe with the same name exists in another layer that has a
646 higher layer priority.
647
648- ``show-recipes:`` Lists available recipes and the layers that
649 provide them.
650
651- ``show-appends:`` Lists ``.bbappend`` files and the recipe files to
652 which they apply.
653
654- ``show-cross-depends:`` Lists dependency relationships between
655 recipes that cross layer boundaries.
656
657- ``add-layer:`` Adds a layer to ``bblayers.conf``.
658
659- ``remove-layer:`` Removes a layer from ``bblayers.conf``
660
661- ``flatten:`` Flattens the layer configuration into a separate
662 output directory. Flattening your layer configuration builds a
663 "flattened" directory that contains the contents of all layers, with
664 any overlayed recipes removed and any ``.bbappend`` files appended to
665 the corresponding recipes. You might have to perform some manual
666 cleanup of the flattened layer as follows:
667
668 - Non-recipe files (such as patches) are overwritten. The flatten
669 command shows a warning for these files.
670
671 - Anything beyond the normal layer setup has been added to the
672 ``layer.conf`` file. Only the lowest priority layer's
673 ``layer.conf`` is used.
674
675 - Overridden and appended items from ``.bbappend`` files need to be
676 cleaned up. The contents of each ``.bbappend`` end up in the
677 flattened recipe. However, if there are appended or changed
678 variable values, you need to tidy these up yourself. Consider the
679 following example. Here, the ``bitbake-layers`` command adds the
680 line ``#### bbappended ...`` so that you know where the following
681 lines originate:
682 ::
683
684 ...
685 DESCRIPTION = "A useful utility"
686 ...
687 EXTRA_OECONF = "--enable-something"
688 ...
689
690 #### bbappended from meta-anotherlayer ####
691
692 DESCRIPTION = "Customized utility"
693 EXTRA_OECONF += "--enable-somethingelse"
694
695
696 Ideally, you would tidy up these utilities as follows:
697 ::
698
699 ...
700 DESCRIPTION = "Customized utility"
701 ...
702 EXTRA_OECONF = "--enable-something --enable-somethingelse"
703 ...
704
705- ``layerindex-fetch``: Fetches a layer from a layer index, along
706 with its dependent layers, and adds the layers to the
707 ``conf/bblayers.conf`` file.
708
709- ``layerindex-show-depends``: Finds layer dependencies from the
710 layer index.
711
712- ``create-layer``: Creates a basic layer.
713
714Creating a General Layer Using the ``bitbake-layers`` Script
715------------------------------------------------------------
716
717The ``bitbake-layers`` script with the ``create-layer`` subcommand
718simplifies creating a new general layer.
719
720.. note::
721
722 - For information on BSP layers, see the ":ref:`bsp-guide/bsp:bsp layers`"
723 section in the Yocto
724 Project Board Specific (BSP) Developer's Guide.
725
726 - In order to use a layer with the OpenEmbedded build system, you
727 need to add the layer to your ``bblayers.conf`` configuration
Andrew Geissler09209ee2020-12-13 08:44:15 -0600728 file. See the ":ref:`dev-manual/common-tasks:adding a layer using the \`\`bitbake-layers\`\` script`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500729 section for more information.
730
731The default mode of the script's operation with this subcommand is to
732create a layer with the following:
733
734- A layer priority of 6.
735
736- A ``conf`` subdirectory that contains a ``layer.conf`` file.
737
738- A ``recipes-example`` subdirectory that contains a further
739 subdirectory named ``example``, which contains an ``example.bb``
740 recipe file.
741
742- A ``COPYING.MIT``, which is the license statement for the layer. The
743 script assumes you want to use the MIT license, which is typical for
744 most layers, for the contents of the layer itself.
745
746- A ``README`` file, which is a file describing the contents of your
747 new layer.
748
749In its simplest form, you can use the following command form to create a
750layer. The command creates a layer whose name corresponds to
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500751"your_layer_name" in the current directory:
752::
753
754 $ bitbake-layers create-layer your_layer_name
755
756As an example, the following command creates a layer named ``meta-scottrif``
757in your home directory:
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500758::
759
760 $ cd /usr/home
761 $ bitbake-layers create-layer meta-scottrif
762 NOTE: Starting bitbake server...
763 Add your new layer with 'bitbake-layers add-layer meta-scottrif'
764
765If you want to set the priority of the layer to other than the default
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500766value of "6", you can either use the ``--priority`` option or you
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500767can edit the
768:term:`BBFILE_PRIORITY` value
769in the ``conf/layer.conf`` after the script creates it. Furthermore, if
770you want to give the example recipe file some name other than the
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500771default, you can use the ``--example-recipe-name`` option.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500772
773The easiest way to see how the ``bitbake-layers create-layer`` command
774works is to experiment with the script. You can also read the usage
775information by entering the following:
776::
777
778 $ bitbake-layers create-layer --help
779 NOTE: Starting bitbake server...
780 usage: bitbake-layers create-layer [-h] [--priority PRIORITY]
781 [--example-recipe-name EXAMPLERECIPE]
782 layerdir
783
784 Create a basic layer
785
786 positional arguments:
787 layerdir Layer directory to create
788
789 optional arguments:
790 -h, --help show this help message and exit
791 --priority PRIORITY, -p PRIORITY
792 Layer directory to create
793 --example-recipe-name EXAMPLERECIPE, -e EXAMPLERECIPE
794 Filename of the example recipe
795
796Adding a Layer Using the ``bitbake-layers`` Script
797--------------------------------------------------
798
799Once you create your general layer, you must add it to your
800``bblayers.conf`` file. Adding the layer to this configuration file
801makes the OpenEmbedded build system aware of your layer so that it can
802search it for metadata.
803
804Add your layer by using the ``bitbake-layers add-layer`` command:
805::
806
807 $ bitbake-layers add-layer your_layer_name
808
809Here is an example that adds a
810layer named ``meta-scottrif`` to the configuration file. Following the
811command that adds the layer is another ``bitbake-layers`` command that
812shows the layers that are in your ``bblayers.conf`` file:
813::
814
815 $ bitbake-layers add-layer meta-scottrif
816 NOTE: Starting bitbake server...
817 Parsing recipes: 100% |##########################################################| Time: 0:00:49
818 Parsing of 1441 .bb files complete (0 cached, 1441 parsed). 2055 targets, 56 skipped, 0 masked, 0 errors.
819 $ bitbake-layers show-layers
820 NOTE: Starting bitbake server...
821 layer path priority
822 ==========================================================================
823 meta /home/scottrif/poky/meta 5
824 meta-poky /home/scottrif/poky/meta-poky 5
825 meta-yocto-bsp /home/scottrif/poky/meta-yocto-bsp 5
826 workspace /home/scottrif/poky/build/workspace 99
827 meta-scottrif /home/scottrif/poky/build/meta-scottrif 6
828
829
830Adding the layer to this file
831enables the build system to locate the layer during the build.
832
833.. note::
834
835 During a build, the OpenEmbedded build system looks in the layers
836 from the top of the list down to the bottom in that order.
837
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500838Customizing Images
839==================
840
841You can customize images to satisfy particular requirements. This
842section describes several methods and provides guidelines for each.
843
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500844Customizing Images Using ``local.conf``
845---------------------------------------
846
847Probably the easiest way to customize an image is to add a package by
848way of the ``local.conf`` configuration file. Because it is limited to
849local use, this method generally only allows you to add packages and is
850not as flexible as creating your own customized image. When you add
851packages using local variables this way, you need to realize that these
852variable changes are in effect for every build and consequently affect
853all images, which might not be what you require.
854
855To add a package to your image using the local configuration file, use
856the ``IMAGE_INSTALL`` variable with the ``_append`` operator:
857::
858
859 IMAGE_INSTALL_append = " strace"
860
861Use of the syntax is important -
862specifically, the space between the quote and the package name, which is
863``strace`` in this example. This space is required since the ``_append``
864operator does not add the space.
865
866Furthermore, you must use ``_append`` instead of the ``+=`` operator if
867you want to avoid ordering issues. The reason for this is because doing
868so unconditionally appends to the variable and avoids ordering problems
869due to the variable being set in image recipes and ``.bbclass`` files
870with operators like ``?=``. Using ``_append`` ensures the operation
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500871takes effect.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500872
873As shown in its simplest use, ``IMAGE_INSTALL_append`` affects all
874images. It is possible to extend the syntax so that the variable applies
875to a specific image only. Here is an example:
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500876::
877
878 IMAGE_INSTALL_append_pn-core-image-minimal = " strace"
879
880This example adds ``strace`` to the ``core-image-minimal`` image only.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500881
882You can add packages using a similar approach through the
883``CORE_IMAGE_EXTRA_INSTALL`` variable. If you use this variable, only
884``core-image-*`` images are affected.
885
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500886Customizing Images Using Custom ``IMAGE_FEATURES`` and ``EXTRA_IMAGE_FEATURES``
887-------------------------------------------------------------------------------
888
889Another method for customizing your image is to enable or disable
890high-level image features by using the
891:term:`IMAGE_FEATURES` and
892:term:`EXTRA_IMAGE_FEATURES`
893variables. Although the functions for both variables are nearly
894equivalent, best practices dictate using ``IMAGE_FEATURES`` from within
895a recipe and using ``EXTRA_IMAGE_FEATURES`` from within your
896``local.conf`` file, which is found in the
897:term:`Build Directory`.
898
899To understand how these features work, the best reference is
900``meta/classes/core-image.bbclass``. This class lists out the available
901``IMAGE_FEATURES`` of which most map to package groups while some, such
902as ``debug-tweaks`` and ``read-only-rootfs``, resolve as general
903configuration settings.
904
905In summary, the file looks at the contents of the ``IMAGE_FEATURES``
906variable and then maps or configures the feature accordingly. Based on
907this information, the build system automatically adds the appropriate
908packages or configurations to the
909:term:`IMAGE_INSTALL` variable.
910Effectively, you are enabling extra features by extending the class or
911creating a custom class for use with specialized image ``.bb`` files.
912
913Use the ``EXTRA_IMAGE_FEATURES`` variable from within your local
914configuration file. Using a separate area from which to enable features
915with this variable helps you avoid overwriting the features in the image
916recipe that are enabled with ``IMAGE_FEATURES``. The value of
917``EXTRA_IMAGE_FEATURES`` is added to ``IMAGE_FEATURES`` within
918``meta/conf/bitbake.conf``.
919
920To illustrate how you can use these variables to modify your image,
921consider an example that selects the SSH server. The Yocto Project ships
922with two SSH servers you can use with your images: Dropbear and OpenSSH.
923Dropbear is a minimal SSH server appropriate for resource-constrained
924environments, while OpenSSH is a well-known standard SSH server
925implementation. By default, the ``core-image-sato`` image is configured
926to use Dropbear. The ``core-image-full-cmdline`` and ``core-image-lsb``
927images both include OpenSSH. The ``core-image-minimal`` image does not
928contain an SSH server.
929
930You can customize your image and change these defaults. Edit the
931``IMAGE_FEATURES`` variable in your recipe or use the
932``EXTRA_IMAGE_FEATURES`` in your ``local.conf`` file so that it
933configures the image you are working with to include
934``ssh-server-dropbear`` or ``ssh-server-openssh``.
935
936.. note::
937
Andrew Geissler09209ee2020-12-13 08:44:15 -0600938 See the ":ref:`ref-manual/features:image features`" section in the Yocto
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500939 Project Reference Manual for a complete list of image features that ship
940 with the Yocto Project.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500941
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500942Customizing Images Using Custom .bb Files
943-----------------------------------------
944
945You can also customize an image by creating a custom recipe that defines
946additional software as part of the image. The following example shows
947the form for the two lines you need:
948::
949
950 IMAGE_INSTALL = "packagegroup-core-x11-base package1 package2"
951 inherit core-image
952
953Defining the software using a custom recipe gives you total control over
954the contents of the image. It is important to use the correct names of
955packages in the ``IMAGE_INSTALL`` variable. You must use the
956OpenEmbedded notation and not the Debian notation for the names (e.g.
957``glibc-dev`` instead of ``libc6-dev``).
958
959The other method for creating a custom image is to base it on an
960existing image. For example, if you want to create an image based on
961``core-image-sato`` but add the additional package ``strace`` to the
962image, copy the ``meta/recipes-sato/images/core-image-sato.bb`` to a new
963``.bb`` and add the following line to the end of the copy:
964::
965
966 IMAGE_INSTALL += "strace"
967
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500968Customizing Images Using Custom Package Groups
969----------------------------------------------
970
971For complex custom images, the best approach for customizing an image is
972to create a custom package group recipe that is used to build the image
973or images. A good example of a package group recipe is
974``meta/recipes-core/packagegroups/packagegroup-base.bb``.
975
976If you examine that recipe, you see that the ``PACKAGES`` variable lists
977the package group packages to produce. The ``inherit packagegroup``
978statement sets appropriate default values and automatically adds
979``-dev``, ``-dbg``, and ``-ptest`` complementary packages for each
980package specified in the ``PACKAGES`` statement.
981
982.. note::
983
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500984 The ``inherit packagegroup`` line should be located near the top of the
985 recipe, certainly before the ``PACKAGES`` statement.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500986
987For each package you specify in ``PACKAGES``, you can use ``RDEPENDS``
988and ``RRECOMMENDS`` entries to provide a list of packages the parent
989task package should contain. You can see examples of these further down
990in the ``packagegroup-base.bb`` recipe.
991
992Here is a short, fabricated example showing the same basic pieces for a
993hypothetical packagegroup defined in ``packagegroup-custom.bb``, where
994the variable ``PN`` is the standard way to abbreviate the reference to
995the full packagegroup name ``packagegroup-custom``:
996::
997
998 DESCRIPTION = "My Custom Package Groups"
999
1000 inherit packagegroup
1001
1002 PACKAGES = "\
1003 ${PN}-apps \
1004 ${PN}-tools \
1005 "
1006
1007 RDEPENDS_${PN}-apps = "\
1008 dropbear \
1009 portmap \
1010 psplash"
1011
1012 RDEPENDS_${PN}-tools = "\
1013 oprofile \
1014 oprofileui-server \
1015 lttng-tools"
1016
1017 RRECOMMENDS_${PN}-tools = "\
1018 kernel-module-oprofile"
1019
1020In the previous example, two package group packages are created with
1021their dependencies and their recommended package dependencies listed:
1022``packagegroup-custom-apps``, and ``packagegroup-custom-tools``. To
1023build an image using these package group packages, you need to add
1024``packagegroup-custom-apps`` and/or ``packagegroup-custom-tools`` to
1025``IMAGE_INSTALL``. For other forms of image dependencies see the other
1026areas of this section.
1027
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001028Customizing an Image Hostname
1029-----------------------------
1030
1031By default, the configured hostname (i.e. ``/etc/hostname``) in an image
1032is the same as the machine name. For example, if
1033:term:`MACHINE` equals "qemux86", the
1034configured hostname written to ``/etc/hostname`` is "qemux86".
1035
1036You can customize this name by altering the value of the "hostname"
1037variable in the ``base-files`` recipe using either an append file or a
1038configuration file. Use the following in an append file:
1039::
1040
1041 hostname = "myhostname"
1042
1043Use the following in a configuration file:
1044::
1045
1046 hostname_pn-base-files = "myhostname"
1047
1048Changing the default value of the variable "hostname" can be useful in
1049certain situations. For example, suppose you need to do extensive
1050testing on an image and you would like to easily identify the image
1051under test from existing images with typical default hostnames. In this
1052situation, you could change the default hostname to "testme", which
1053results in all the images using the name "testme". Once testing is
1054complete and you do not need to rebuild the image for test any longer,
1055you can easily reset the default hostname.
1056
1057Another point of interest is that if you unset the variable, the image
1058will have no default hostname in the filesystem. Here is an example that
1059unsets the variable in a configuration file:
1060::
1061
1062 hostname_pn-base-files = ""
1063
1064Having no default hostname in the filesystem is suitable for
1065environments that use dynamic hostnames such as virtual machines.
1066
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001067Writing a New Recipe
1068====================
1069
1070Recipes (``.bb`` files) are fundamental components in the Yocto Project
1071environment. Each software component built by the OpenEmbedded build
1072system requires a recipe to define the component. This section describes
1073how to create, write, and test a new recipe.
1074
1075.. note::
1076
1077 For information on variables that are useful for recipes and for
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001078 information about recipe naming issues, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001079 ":ref:`ref-manual/varlocality:recipes`" section of the Yocto Project
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001080 Reference Manual.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001081
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001082Overview
1083--------
1084
1085The following figure shows the basic process for creating a new recipe.
1086The remainder of the section provides details for the steps.
1087
1088.. image:: figures/recipe-workflow.png
1089 :align: center
1090
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001091Locate or Automatically Create a Base Recipe
1092--------------------------------------------
1093
1094You can always write a recipe from scratch. However, three choices exist
1095that can help you quickly get a start on a new recipe:
1096
1097- ``devtool add``: A command that assists in creating a recipe and an
1098 environment conducive to development.
1099
1100- ``recipetool create``: A command provided by the Yocto Project that
1101 automates creation of a base recipe based on the source files.
1102
1103- *Existing Recipes:* Location and modification of an existing recipe
1104 that is similar in function to the recipe you need.
1105
1106.. note::
1107
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001108 For information on recipe syntax, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001109 ":ref:`dev-manual/common-tasks:recipe syntax`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001110
1111Creating the Base Recipe Using ``devtool add``
1112~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1113
1114The ``devtool add`` command uses the same logic for auto-creating the
1115recipe as ``recipetool create``, which is listed below. Additionally,
1116however, ``devtool add`` sets up an environment that makes it easy for
1117you to patch the source and to make changes to the recipe as is often
1118necessary when adding a recipe to build a new piece of software to be
1119included in a build.
1120
1121You can find a complete description of the ``devtool add`` command in
Andrew Geissler09209ee2020-12-13 08:44:15 -06001122the ":ref:`sdk-manual/extensible:a closer look at \`\`devtool add\`\``" section
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001123in the Yocto Project Application Development and the Extensible Software
1124Development Kit (eSDK) manual.
1125
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001126Creating the Base Recipe Using ``recipetool create``
1127~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1128
1129``recipetool create`` automates creation of a base recipe given a set of
1130source code files. As long as you can extract or point to the source
1131files, the tool will construct a recipe and automatically configure all
1132pre-build information into the recipe. For example, suppose you have an
1133application that builds using Autotools. Creating the base recipe using
1134``recipetool`` results in a recipe that has the pre-build dependencies,
1135license requirements, and checksums configured.
1136
1137To run the tool, you just need to be in your
1138:term:`Build Directory` and have sourced the
1139build environment setup script (i.e.
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001140:ref:`structure-core-script`).
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001141To get help on the tool, use the following command:
1142::
1143
1144 $ recipetool -h
1145 NOTE: Starting bitbake server...
1146 usage: recipetool [-d] [-q] [--color COLOR] [-h] <subcommand> ...
1147
1148 OpenEmbedded recipe tool
1149
1150 options:
1151 -d, --debug Enable debug output
1152 -q, --quiet Print only errors
1153 --color COLOR Colorize output (where COLOR is auto, always, never)
1154 -h, --help show this help message and exit
1155
1156 subcommands:
1157 create Create a new recipe
1158 newappend Create a bbappend for the specified target in the specified
1159 layer
1160 setvar Set a variable within a recipe
1161 appendfile Create/update a bbappend to replace a target file
1162 appendsrcfiles Create/update a bbappend to add or replace source files
1163 appendsrcfile Create/update a bbappend to add or replace a source file
1164 Use recipetool <subcommand> --help to get help on a specific command
1165
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001166Running ``recipetool create -o OUTFILE`` creates the base recipe and
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001167locates it properly in the layer that contains your source files.
1168Following are some syntax examples:
1169
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001170 - Use this syntax to generate a recipe based on source. Once generated,
1171 the recipe resides in the existing source code layer:
1172 ::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001173
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001174 recipetool create -o OUTFILE source
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001175
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001176 - Use this syntax to generate a recipe using code that
1177 you extract from source. The extracted code is placed in its own layer
1178 defined by ``EXTERNALSRC``.
1179 ::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001180
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001181 recipetool create -o OUTFILE -x EXTERNALSRC source
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001182
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001183 - Use this syntax to generate a recipe based on source. The options
1184 direct ``recipetool`` to generate debugging information. Once generated,
1185 the recipe resides in the existing source code layer:
1186 ::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001187
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001188 recipetool create -d -o OUTFILE source
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001189
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001190Locating and Using a Similar Recipe
1191~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1192
1193Before writing a recipe from scratch, it is often useful to discover
1194whether someone else has already written one that meets (or comes close
1195to meeting) your needs. The Yocto Project and OpenEmbedded communities
1196maintain many recipes that might be candidates for what you are doing.
1197You can find a good central index of these recipes in the `OpenEmbedded
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001198Layer Index <https://layers.openembedded.org>`__.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001199
1200Working from an existing recipe or a skeleton recipe is the best way to
1201get started. Here are some points on both methods:
1202
1203- *Locate and modify a recipe that is close to what you want to do:*
1204 This method works when you are familiar with the current recipe
1205 space. The method does not work so well for those new to the Yocto
1206 Project or writing recipes.
1207
1208 Some risks associated with this method are using a recipe that has
1209 areas totally unrelated to what you are trying to accomplish with
1210 your recipe, not recognizing areas of the recipe that you might have
1211 to add from scratch, and so forth. All these risks stem from
1212 unfamiliarity with the existing recipe space.
1213
1214- *Use and modify the following skeleton recipe:* If for some reason
1215 you do not want to use ``recipetool`` and you cannot find an existing
1216 recipe that is close to meeting your needs, you can use the following
1217 structure to provide the fundamental areas of a new recipe.
1218 ::
1219
1220 DESCRIPTION = ""
1221 HOMEPAGE = ""
1222 LICENSE = ""
1223 SECTION = ""
1224 DEPENDS = ""
1225 LIC_FILES_CHKSUM = ""
1226
1227 SRC_URI = ""
1228
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001229Storing and Naming the Recipe
1230-----------------------------
1231
1232Once you have your base recipe, you should put it in your own layer and
1233name it appropriately. Locating it correctly ensures that the
1234OpenEmbedded build system can find it when you use BitBake to process
1235the recipe.
1236
1237- *Storing Your Recipe:* The OpenEmbedded build system locates your
1238 recipe through the layer's ``conf/layer.conf`` file and the
1239 :term:`BBFILES` variable. This
1240 variable sets up a path from which the build system can locate
1241 recipes. Here is the typical use:
1242 ::
1243
1244 BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
1245 ${LAYERDIR}/recipes-*/*/*.bbappend"
1246
1247 Consequently, you need to be sure you locate your new recipe inside
1248 your layer such that it can be found.
1249
1250 You can find more information on how layers are structured in the
1251 "`Understanding and Creating
1252 Layers <#understanding-and-creating-layers>`__" section.
1253
1254- *Naming Your Recipe:* When you name your recipe, you need to follow
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001255 this naming convention:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001256 ::
1257
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001258 basename_version.bb
1259
1260 Use lower-cased characters and do not include the reserved suffixes
1261 ``-native``, ``-cross``, ``-initial``, or ``-dev`` casually (i.e. do not use
1262 them as part of your recipe name unless the string applies). Here are some
1263 examples:
1264
1265 .. code-block:: none
1266
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001267 cups_1.7.0.bb
1268 gawk_4.0.2.bb
1269 irssi_0.8.16-rc1.bb
1270
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001271Running a Build on the Recipe
1272-----------------------------
1273
1274Creating a new recipe is usually an iterative process that requires
1275using BitBake to process the recipe multiple times in order to
1276progressively discover and add information to the recipe file.
1277
1278Assuming you have sourced the build environment setup script (i.e.
1279:ref:`structure-core-script`) and you are in
1280the :term:`Build Directory`, use
1281BitBake to process your recipe. All you need to provide is the
1282``basename`` of the recipe as described in the previous section:
1283::
1284
1285 $ bitbake basename
1286
1287During the build, the OpenEmbedded build system creates a temporary work
1288directory for each recipe
1289(``${``\ :term:`WORKDIR`\ ``}``)
1290where it keeps extracted source files, log files, intermediate
1291compilation and packaging files, and so forth.
1292
1293The path to the per-recipe temporary work directory depends on the
1294context in which it is being built. The quickest way to find this path
1295is to have BitBake return it by running the following:
1296::
1297
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001298 $ bitbake -e basename | grep ^WORKDIR=
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001299
1300As an example, assume a Source Directory
1301top-level folder named ``poky``, a default Build Directory at
1302``poky/build``, and a ``qemux86-poky-linux`` machine target system.
1303Furthermore, suppose your recipe is named ``foo_1.3.0.bb``. In this
1304case, the work directory the build system uses to build the package
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001305would be as follows:
1306::
1307
1308 poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0
1309
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001310Inside this directory you can find sub-directories such as ``image``,
1311``packages-split``, and ``temp``. After the build, you can examine these
1312to determine how well the build went.
1313
1314.. note::
1315
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001316 You can find log files for each task in the recipe's ``temp``
1317 directory (e.g. ``poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0/temp``).
1318 Log files are named ``log.taskname`` (e.g. ``log.do_configure``,
1319 ``log.do_fetch``, and ``log.do_compile``).
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001320
1321You can find more information about the build process in
Andrew Geissler09209ee2020-12-13 08:44:15 -06001322":doc:`/overview-manual/development-environment`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001323chapter of the Yocto Project Overview and Concepts Manual.
1324
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001325Fetching Code
1326-------------
1327
1328The first thing your recipe must do is specify how to fetch the source
1329files. Fetching is controlled mainly through the
1330:term:`SRC_URI` variable. Your recipe
1331must have a ``SRC_URI`` variable that points to where the source is
1332located. For a graphical representation of source locations, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001333":ref:`overview-manual/concepts:sources`" section in
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001334the Yocto Project Overview and Concepts Manual.
1335
1336The :ref:`ref-tasks-fetch` task uses
1337the prefix of each entry in the ``SRC_URI`` variable value to determine
Andrew Geissler09209ee2020-12-13 08:44:15 -06001338which :ref:`fetcher <bitbake:bitbake-user-manual/bitbake-user-manual-fetching:fetchers>` to use to get your
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001339source files. It is the ``SRC_URI`` variable that triggers the fetcher.
1340The :ref:`ref-tasks-patch` task uses
1341the variable after source is fetched to apply patches. The OpenEmbedded
1342build system uses
1343:term:`FILESOVERRIDES` for
1344scanning directory locations for local files in ``SRC_URI``.
1345
1346The ``SRC_URI`` variable in your recipe must define each unique location
1347for your source files. It is good practice to not hard-code version
1348numbers in a URL used in ``SRC_URI``. Rather than hard-code these
1349values, use ``${``\ :term:`PV`\ ``}``,
1350which causes the fetch process to use the version specified in the
1351recipe filename. Specifying the version in this manner means that
1352upgrading the recipe to a future version is as simple as renaming the
1353recipe to match the new version.
1354
1355Here is a simple example from the
1356``meta/recipes-devtools/strace/strace_5.5.bb`` recipe where the source
1357comes from a single tarball. Notice the use of the
1358:term:`PV` variable:
1359::
1360
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001361 SRC_URI = "https://strace.io/files/${PV}/strace-${PV}.tar.xz \
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001362
1363Files mentioned in ``SRC_URI`` whose names end in a typical archive
1364extension (e.g. ``.tar``, ``.tar.gz``, ``.tar.bz2``, ``.zip``, and so
1365forth), are automatically extracted during the
1366:ref:`ref-tasks-unpack` task. For
1367another example that specifies these types of files, see the
1368"`Autotooled Package <#new-recipe-autotooled-package>`__" section.
1369
1370Another way of specifying source is from an SCM. For Git repositories,
1371you must specify :term:`SRCREV` and
1372you should specify :term:`PV` to include
1373the revision with :term:`SRCPV`. Here
1374is an example from the recipe
1375``meta/recipes-kernel/blktrace/blktrace_git.bb``:
1376::
1377
1378 SRCREV = "d6918c8832793b4205ed3bfede78c2f915c23385"
1379
1380 PR = "r6"
1381 PV = "1.0.5+git${SRCPV}"
1382
1383 SRC_URI = "git://git.kernel.dk/blktrace.git \
1384 file://ldflags.patch"
1385
1386If your ``SRC_URI`` statement includes URLs pointing to individual files
1387fetched from a remote server other than a version control system,
1388BitBake attempts to verify the files against checksums defined in your
1389recipe to ensure they have not been tampered with or otherwise modified
1390since the recipe was written. Two checksums are used:
1391``SRC_URI[md5sum]`` and ``SRC_URI[sha256sum]``.
1392
1393If your ``SRC_URI`` variable points to more than a single URL (excluding
1394SCM URLs), you need to provide the ``md5`` and ``sha256`` checksums for
1395each URL. For these cases, you provide a name for each URL as part of
1396the ``SRC_URI`` and then reference that name in the subsequent checksum
1397statements. Here is an example combining lines from the files
1398``git.inc`` and ``git_2.24.1.bb``:
1399::
1400
1401 SRC_URI = "${KERNELORG_MIRROR}/software/scm/git/git-${PV}.tar.gz;name=tarball \
1402 ${KERNELORG_MIRROR}/software/scm/git/git-manpages-${PV}.tar.gz;name=manpages"
1403
1404 SRC_URI[tarball.md5sum] = "166bde96adbbc11c8843d4f8f4f9811b"
1405 SRC_URI[tarball.sha256sum] = "ad5334956301c86841eb1e5b1bb20884a6bad89a10a6762c958220c7cf64da02"
1406 SRC_URI[manpages.md5sum] = "31c2272a8979022497ba3d4202df145d"
1407 SRC_URI[manpages.sha256sum] = "9a7ae3a093bea39770eb96ca3e5b40bff7af0b9f6123f089d7821d0e5b8e1230"
1408
1409Proper values for ``md5`` and ``sha256`` checksums might be available
1410with other signatures on the download page for the upstream source (e.g.
1411``md5``, ``sha1``, ``sha256``, ``GPG``, and so forth). Because the
1412OpenEmbedded build system only deals with ``sha256sum`` and ``md5sum``,
1413you should verify all the signatures you find by hand.
1414
1415If no ``SRC_URI`` checksums are specified when you attempt to build the
1416recipe, or you provide an incorrect checksum, the build will produce an
1417error for each missing or incorrect checksum. As part of the error
1418message, the build system provides the checksum string corresponding to
1419the fetched file. Once you have the correct checksums, you can copy and
1420paste them into your recipe and then run the build again to continue.
1421
1422.. note::
1423
1424 As mentioned, if the upstream source provides signatures for
1425 verifying the downloaded source code, you should verify those
1426 manually before setting the checksum values in the recipe and
1427 continuing with the build.
1428
1429This final example is a bit more complicated and is from the
1430``meta/recipes-sato/rxvt-unicode/rxvt-unicode_9.20.bb`` recipe. The
1431example's ``SRC_URI`` statement identifies multiple files as the source
1432files for the recipe: a tarball, a patch file, a desktop file, and an
1433icon.
1434::
1435
1436 SRC_URI = "http://dist.schmorp.de/rxvt-unicode/Attic/rxvt-unicode-${PV}.tar.bz2 \
1437 file://xwc.patch \
1438 file://rxvt.desktop \
1439 file://rxvt.png"
1440
1441When you specify local files using the ``file://`` URI protocol, the
1442build system fetches files from the local machine. The path is relative
1443to the :term:`FILESPATH` variable
1444and searches specific directories in a certain order:
1445``${``\ :term:`BP`\ ``}``,
1446``${``\ :term:`BPN`\ ``}``, and
1447``files``. The directories are assumed to be subdirectories of the
1448directory in which the recipe or append file resides. For another
1449example that specifies these types of files, see the "`Single .c File
1450Package (Hello
1451World!) <#new-recipe-single-c-file-package-hello-world>`__" section.
1452
1453The previous example also specifies a patch file. Patch files are files
1454whose names usually end in ``.patch`` or ``.diff`` but can end with
1455compressed suffixes such as ``diff.gz`` and ``patch.bz2``, for example.
1456The build system automatically applies patches as described in the
1457"`Patching Code <#new-recipe-patching-code>`__" section.
1458
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001459Unpacking Code
1460--------------
1461
1462During the build, the
1463:ref:`ref-tasks-unpack` task unpacks
1464the source with ``${``\ :term:`S`\ ``}``
1465pointing to where it is unpacked.
1466
1467If you are fetching your source files from an upstream source archived
1468tarball and the tarball's internal structure matches the common
1469convention of a top-level subdirectory named
1470``${``\ :term:`BPN`\ ``}-${``\ :term:`PV`\ ``}``,
1471then you do not need to set ``S``. However, if ``SRC_URI`` specifies to
1472fetch source from an archive that does not use this convention, or from
1473an SCM like Git or Subversion, your recipe needs to define ``S``.
1474
1475If processing your recipe using BitBake successfully unpacks the source
1476files, you need to be sure that the directory pointed to by ``${S}``
1477matches the structure of the source.
1478
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001479Patching Code
1480-------------
1481
1482Sometimes it is necessary to patch code after it has been fetched. Any
1483files mentioned in ``SRC_URI`` whose names end in ``.patch`` or
1484``.diff`` or compressed versions of these suffixes (e.g. ``diff.gz`` are
1485treated as patches. The
1486:ref:`ref-tasks-patch` task
1487automatically applies these patches.
1488
1489The build system should be able to apply patches with the "-p1" option
1490(i.e. one directory level in the path will be stripped off). If your
1491patch needs to have more directory levels stripped off, specify the
1492number of levels using the "striplevel" option in the ``SRC_URI`` entry
1493for the patch. Alternatively, if your patch needs to be applied in a
1494specific subdirectory that is not specified in the patch file, use the
1495"patchdir" option in the entry.
1496
1497As with all local files referenced in
1498:term:`SRC_URI` using ``file://``,
1499you should place patch files in a directory next to the recipe either
1500named the same as the base name of the recipe
1501(:term:`BP` and
1502:term:`BPN`) or "files".
1503
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001504Licensing
1505---------
1506
1507Your recipe needs to have both the
1508:term:`LICENSE` and
1509:term:`LIC_FILES_CHKSUM`
1510variables:
1511
1512- ``LICENSE``: This variable specifies the license for the software.
1513 If you do not know the license under which the software you are
1514 building is distributed, you should go to the source code and look
1515 for that information. Typical files containing this information
1516 include ``COPYING``, ``LICENSE``, and ``README`` files. You could
1517 also find the information near the top of a source file. For example,
1518 given a piece of software licensed under the GNU General Public
1519 License version 2, you would set ``LICENSE`` as follows:
1520 ::
1521
1522 LICENSE = "GPLv2"
1523
1524 The licenses you specify within ``LICENSE`` can have any name as long
1525 as you do not use spaces, since spaces are used as separators between
1526 license names. For standard licenses, use the names of the files in
1527 ``meta/files/common-licenses/`` or the ``SPDXLICENSEMAP`` flag names
1528 defined in ``meta/conf/licenses.conf``.
1529
1530- ``LIC_FILES_CHKSUM``: The OpenEmbedded build system uses this
1531 variable to make sure the license text has not changed. If it has,
1532 the build produces an error and it affords you the chance to figure
1533 it out and correct the problem.
1534
1535 You need to specify all applicable licensing files for the software.
1536 At the end of the configuration step, the build process will compare
1537 the checksums of the files to be sure the text has not changed. Any
1538 differences result in an error with the message containing the
1539 current checksum. For more explanation and examples of how to set the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001540 ``LIC_FILES_CHKSUM`` variable, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001541 ":ref:`dev-manual/common-tasks:tracking license changes`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001542
1543 To determine the correct checksum string, you can list the
1544 appropriate files in the ``LIC_FILES_CHKSUM`` variable with incorrect
1545 md5 strings, attempt to build the software, and then note the
1546 resulting error messages that will report the correct md5 strings.
1547 See the "`Fetching Code <#new-recipe-fetching-code>`__" section for
1548 additional information.
1549
1550 Here is an example that assumes the software has a ``COPYING`` file:
1551 ::
1552
1553 LIC_FILES_CHKSUM = "file://COPYING;md5=xxx"
1554
1555 When you try to build the
1556 software, the build system will produce an error and give you the
1557 correct string that you can substitute into the recipe file for a
1558 subsequent build.
1559
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001560Dependencies
1561------------
1562
1563Most software packages have a short list of other packages that they
1564require, which are called dependencies. These dependencies fall into two
1565main categories: build-time dependencies, which are required when the
1566software is built; and runtime dependencies, which are required to be
1567installed on the target in order for the software to run.
1568
1569Within a recipe, you specify build-time dependencies using the
1570:term:`DEPENDS` variable. Although
1571nuances exist, items specified in ``DEPENDS`` should be names of other
1572recipes. It is important that you specify all build-time dependencies
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001573explicitly.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001574
1575Another consideration is that configure scripts might automatically
1576check for optional dependencies and enable corresponding functionality
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001577if those dependencies are found. If you wish to make a recipe that is
1578more generally useful (e.g. publish the recipe in a layer for others to
1579use), instead of hard-disabling the functionality, you can use the
1580:term:`PACKAGECONFIG` variable to allow functionality and the
1581corresponding dependencies to be enabled and disabled easily by other
1582users of the recipe.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001583
1584Similar to build-time dependencies, you specify runtime dependencies
1585through a variable -
1586:term:`RDEPENDS`, which is
1587package-specific. All variables that are package-specific need to have
1588the name of the package added to the end as an override. Since the main
1589package for a recipe has the same name as the recipe, and the recipe's
1590name can be found through the
1591``${``\ :term:`PN`\ ``}`` variable, then
1592you specify the dependencies for the main package by setting
1593``RDEPENDS_${PN}``. If the package were named ``${PN}-tools``, then you
1594would set ``RDEPENDS_${PN}-tools``, and so forth.
1595
1596Some runtime dependencies will be set automatically at packaging time.
1597These dependencies include any shared library dependencies (i.e. if a
1598package "example" contains "libexample" and another package "mypackage"
1599contains a binary that links to "libexample" then the OpenEmbedded build
1600system will automatically add a runtime dependency to "mypackage" on
1601"example"). See the
Andrew Geissler09209ee2020-12-13 08:44:15 -06001602":ref:`overview-manual/concepts:automatically added runtime dependencies`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001603section in the Yocto Project Overview and Concepts Manual for further
1604details.
1605
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001606Configuring the Recipe
1607----------------------
1608
1609Most software provides some means of setting build-time configuration
1610options before compilation. Typically, setting these options is
1611accomplished by running a configure script with options, or by modifying
1612a build configuration file.
1613
1614.. note::
1615
1616 As of Yocto Project Release 1.7, some of the core recipes that
1617 package binary configuration scripts now disable the scripts due to
1618 the scripts previously requiring error-prone path substitution. The
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001619 OpenEmbedded build system uses ``pkg-config`` now, which is much more
1620 robust. You can find a list of the ``*-config`` scripts that are disabled
1621 in the ":ref:`migration-1.7-binary-configuration-scripts-disabled`" section
1622 in the Yocto Project Reference Manual.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001623
1624A major part of build-time configuration is about checking for
1625build-time dependencies and possibly enabling optional functionality as
1626a result. You need to specify any build-time dependencies for the
1627software you are building in your recipe's
1628:term:`DEPENDS` value, in terms of
1629other recipes that satisfy those dependencies. You can often find
1630build-time or runtime dependencies described in the software's
1631documentation.
1632
1633The following list provides configuration items of note based on how
1634your software is built:
1635
1636- *Autotools:* If your source files have a ``configure.ac`` file, then
1637 your software is built using Autotools. If this is the case, you just
1638 need to worry about modifying the configuration.
1639
1640 When using Autotools, your recipe needs to inherit the
1641 :ref:`autotools <ref-classes-autotools>` class
1642 and your recipe does not have to contain a
1643 :ref:`ref-tasks-configure` task.
1644 However, you might still want to make some adjustments. For example,
1645 you can set
1646 :term:`EXTRA_OECONF` or
1647 :term:`PACKAGECONFIG_CONFARGS`
1648 to pass any needed configure options that are specific to the recipe.
1649
1650- *CMake:* If your source files have a ``CMakeLists.txt`` file, then
1651 your software is built using CMake. If this is the case, you just
1652 need to worry about modifying the configuration.
1653
1654 When you use CMake, your recipe needs to inherit the
1655 :ref:`cmake <ref-classes-cmake>` class and your
1656 recipe does not have to contain a
1657 :ref:`ref-tasks-configure` task.
1658 You can make some adjustments by setting
1659 :term:`EXTRA_OECMAKE` to
1660 pass any needed configure options that are specific to the recipe.
1661
1662 .. note::
1663
1664 If you need to install one or more custom CMake toolchain files
1665 that are supplied by the application you are building, install the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001666 files to ``${D}${datadir}/cmake/Modules`` during ``do_install``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001667
1668- *Other:* If your source files do not have a ``configure.ac`` or
1669 ``CMakeLists.txt`` file, then your software is built using some
1670 method other than Autotools or CMake. If this is the case, you
1671 normally need to provide a
1672 :ref:`ref-tasks-configure` task
1673 in your recipe unless, of course, there is nothing to configure.
1674
1675 Even if your software is not being built by Autotools or CMake, you
1676 still might not need to deal with any configuration issues. You need
1677 to determine if configuration is even a required step. You might need
1678 to modify a Makefile or some configuration file used for the build to
1679 specify necessary build options. Or, perhaps you might need to run a
1680 provided, custom configure script with the appropriate options.
1681
1682 For the case involving a custom configure script, you would run
1683 ``./configure --help`` and look for the options you need to set.
1684
1685Once configuration succeeds, it is always good practice to look at the
1686``log.do_configure`` file to ensure that the appropriate options have
1687been enabled and no additional build-time dependencies need to be added
1688to ``DEPENDS``. For example, if the configure script reports that it
1689found something not mentioned in ``DEPENDS``, or that it did not find
1690something that it needed for some desired optional functionality, then
1691you would need to add those to ``DEPENDS``. Looking at the log might
1692also reveal items being checked for, enabled, or both that you do not
1693want, or items not being found that are in ``DEPENDS``, in which case
1694you would need to look at passing extra options to the configure script
1695as needed. For reference information on configure options specific to
1696the software you are building, you can consult the output of the
1697``./configure --help`` command within ``${S}`` or consult the software's
1698upstream documentation.
1699
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001700Using Headers to Interface with Devices
1701---------------------------------------
1702
1703If your recipe builds an application that needs to communicate with some
1704device or needs an API into a custom kernel, you will need to provide
1705appropriate header files. Under no circumstances should you ever modify
1706the existing
1707``meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc`` file.
1708These headers are used to build ``libc`` and must not be compromised
1709with custom or machine-specific header information. If you customize
1710``libc`` through modified headers all other applications that use
1711``libc`` thus become affected.
1712
1713.. note::
1714
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001715 Never copy and customize the ``libc`` header file (i.e.
1716 ``meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc``).
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001717
1718The correct way to interface to a device or custom kernel is to use a
1719separate package that provides the additional headers for the driver or
1720other unique interfaces. When doing so, your application also becomes
1721responsible for creating a dependency on that specific provider.
1722
1723Consider the following:
1724
1725- Never modify ``linux-libc-headers.inc``. Consider that file to be
1726 part of the ``libc`` system, and not something you use to access the
1727 kernel directly. You should access ``libc`` through specific ``libc``
1728 calls.
1729
1730- Applications that must talk directly to devices should either provide
1731 necessary headers themselves, or establish a dependency on a special
1732 headers package that is specific to that driver.
1733
1734For example, suppose you want to modify an existing header that adds I/O
1735control or network support. If the modifications are used by a small
1736number programs, providing a unique version of a header is easy and has
1737little impact. When doing so, bear in mind the guidelines in the
1738previous list.
1739
1740.. note::
1741
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001742 If for some reason your changes need to modify the behavior of the ``libc``,
1743 and subsequently all other applications on the system, use a ``.bbappend``
1744 to modify the ``linux-kernel-headers.inc`` file. However, take care to not
1745 make the changes machine specific.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001746
1747Consider a case where your kernel is older and you need an older
1748``libc`` ABI. The headers installed by your recipe should still be a
1749standard mainline kernel, not your own custom one.
1750
1751When you use custom kernel headers you need to get them from
1752:term:`STAGING_KERNEL_DIR`,
1753which is the directory with kernel headers that are required to build
1754out-of-tree modules. Your recipe will also need the following:
1755::
1756
1757 do_configure[depends] += "virtual/kernel:do_shared_workdir"
1758
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001759Compilation
1760-----------
1761
1762During a build, the ``do_compile`` task happens after source is fetched,
1763unpacked, and configured. If the recipe passes through ``do_compile``
1764successfully, nothing needs to be done.
1765
1766However, if the compile step fails, you need to diagnose the failure.
1767Here are some common issues that cause failures.
1768
1769.. note::
1770
1771 For cases where improper paths are detected for configuration files
1772 or for when libraries/headers cannot be found, be sure you are using
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001773 the more robust ``pkg-config``. See the note in section
Andrew Geissler09209ee2020-12-13 08:44:15 -06001774 ":ref:`dev-manual/common-tasks:Configuring the Recipe`" for additional information.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001775
1776- *Parallel build failures:* These failures manifest themselves as
1777 intermittent errors, or errors reporting that a file or directory
1778 that should be created by some other part of the build process could
1779 not be found. This type of failure can occur even if, upon
1780 inspection, the file or directory does exist after the build has
1781 failed, because that part of the build process happened in the wrong
1782 order.
1783
1784 To fix the problem, you need to either satisfy the missing dependency
1785 in the Makefile or whatever script produced the Makefile, or (as a
1786 workaround) set :term:`PARALLEL_MAKE` to an empty string:
1787 ::
1788
1789 PARALLEL_MAKE = ""
1790
1791 For information on parallel Makefile issues, see the "`Debugging
1792 Parallel Make Races <#debugging-parallel-make-races>`__" section.
1793
1794- *Improper host path usage:* This failure applies to recipes building
1795 for the target or ``nativesdk`` only. The failure occurs when the
1796 compilation process uses improper headers, libraries, or other files
1797 from the host system when cross-compiling for the target.
1798
1799 To fix the problem, examine the ``log.do_compile`` file to identify
1800 the host paths being used (e.g. ``/usr/include``, ``/usr/lib``, and
1801 so forth) and then either add configure options, apply a patch, or do
1802 both.
1803
1804- *Failure to find required libraries/headers:* If a build-time
1805 dependency is missing because it has not been declared in
1806 :term:`DEPENDS`, or because the
1807 dependency exists but the path used by the build process to find the
1808 file is incorrect and the configure step did not detect it, the
1809 compilation process could fail. For either of these failures, the
1810 compilation process notes that files could not be found. In these
1811 cases, you need to go back and add additional options to the
1812 configure script as well as possibly add additional build-time
1813 dependencies to ``DEPENDS``.
1814
1815 Occasionally, it is necessary to apply a patch to the source to
1816 ensure the correct paths are used. If you need to specify paths to
1817 find files staged into the sysroot from other recipes, use the
1818 variables that the OpenEmbedded build system provides (e.g.
1819 ``STAGING_BINDIR``, ``STAGING_INCDIR``, ``STAGING_DATADIR``, and so
1820 forth).
1821
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001822Installing
1823----------
1824
1825During ``do_install``, the task copies the built files along with their
1826hierarchy to locations that would mirror their locations on the target
1827device. The installation process copies files from the
1828``${``\ :term:`S`\ ``}``,
1829``${``\ :term:`B`\ ``}``, and
1830``${``\ :term:`WORKDIR`\ ``}``
1831directories to the ``${``\ :term:`D`\ ``}``
1832directory to create the structure as it should appear on the target
1833system.
1834
1835How your software is built affects what you must do to be sure your
1836software is installed correctly. The following list describes what you
1837must do for installation depending on the type of build system used by
1838the software being built:
1839
1840- *Autotools and CMake:* If the software your recipe is building uses
1841 Autotools or CMake, the OpenEmbedded build system understands how to
1842 install the software. Consequently, you do not have to have a
1843 ``do_install`` task as part of your recipe. You just need to make
1844 sure the install portion of the build completes with no issues.
1845 However, if you wish to install additional files not already being
1846 installed by ``make install``, you should do this using a
1847 ``do_install_append`` function using the install command as described
1848 in the "Manual" bulleted item later in this list.
1849
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001850- *Other (using* ``make install``\ *)*: You need to define a ``do_install``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001851 function in your recipe. The function should call
1852 ``oe_runmake install`` and will likely need to pass in the
1853 destination directory as well. How you pass that path is dependent on
1854 how the ``Makefile`` being run is written (e.g. ``DESTDIR=${D}``,
1855 ``PREFIX=${D}``, ``INSTALLROOT=${D}``, and so forth).
1856
1857 For an example recipe using ``make install``, see the
1858 "`Makefile-Based Package <#new-recipe-makefile-based-package>`__"
1859 section.
1860
1861- *Manual:* You need to define a ``do_install`` function in your
1862 recipe. The function must first use ``install -d`` to create the
1863 directories under
1864 ``${``\ :term:`D`\ ``}``. Once the
1865 directories exist, your function can use ``install`` to manually
1866 install the built software into the directories.
1867
1868 You can find more information on ``install`` at
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001869 https://www.gnu.org/software/coreutils/manual/html_node/install-invocation.html.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001870
1871For the scenarios that do not use Autotools or CMake, you need to track
1872the installation and diagnose and fix any issues until everything
1873installs correctly. You need to look in the default location of
1874``${D}``, which is ``${WORKDIR}/image``, to be sure your files have been
1875installed correctly.
1876
1877.. note::
1878
1879 - During the installation process, you might need to modify some of
1880 the installed files to suit the target layout. For example, you
1881 might need to replace hard-coded paths in an initscript with
1882 values of variables provided by the build system, such as
1883 replacing ``/usr/bin/`` with ``${bindir}``. If you do perform such
1884 modifications during ``do_install``, be sure to modify the
1885 destination file after copying rather than before copying.
1886 Modifying after copying ensures that the build system can
1887 re-execute ``do_install`` if needed.
1888
1889 - ``oe_runmake install``, which can be run directly or can be run
1890 indirectly by the
1891 :ref:`autotools <ref-classes-autotools>` and
1892 :ref:`cmake <ref-classes-cmake>` classes,
1893 runs ``make install`` in parallel. Sometimes, a Makefile can have
1894 missing dependencies between targets that can result in race
1895 conditions. If you experience intermittent failures during
1896 ``do_install``, you might be able to work around them by disabling
1897 parallel Makefile installs by adding the following to the recipe:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001898 ::
1899
1900 PARALLEL_MAKEINST = ""
1901
1902 See :term:`PARALLEL_MAKEINST` for additional information.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001903
1904 - If you need to install one or more custom CMake toolchain files
1905 that are supplied by the application you are building, install the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001906 files to ``${D}${datadir}/cmake/Modules`` during
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001907 :ref:`ref-tasks-install`.
1908
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001909Enabling System Services
1910------------------------
1911
1912If you want to install a service, which is a process that usually starts
1913on boot and runs in the background, then you must include some
1914additional definitions in your recipe.
1915
1916If you are adding services and the service initialization script or the
1917service file itself is not installed, you must provide for that
1918installation in your recipe using a ``do_install_append`` function. If
1919your recipe already has a ``do_install`` function, update the function
1920near its end rather than adding an additional ``do_install_append``
1921function.
1922
1923When you create the installation for your services, you need to
1924accomplish what is normally done by ``make install``. In other words,
1925make sure your installation arranges the output similar to how it is
1926arranged on the target system.
1927
1928The OpenEmbedded build system provides support for starting services two
1929different ways:
1930
1931- *SysVinit:* SysVinit is a system and service manager that manages the
1932 init system used to control the very basic functions of your system.
1933 The init program is the first program started by the Linux kernel
1934 when the system boots. Init then controls the startup, running and
1935 shutdown of all other programs.
1936
1937 To enable a service using SysVinit, your recipe needs to inherit the
1938 :ref:`update-rc.d <ref-classes-update-rc.d>`
1939 class. The class helps facilitate safely installing the package on
1940 the target.
1941
1942 You will need to set the
1943 :term:`INITSCRIPT_PACKAGES`,
1944 :term:`INITSCRIPT_NAME`,
1945 and
1946 :term:`INITSCRIPT_PARAMS`
1947 variables within your recipe.
1948
1949- *systemd:* System Management Daemon (systemd) was designed to replace
1950 SysVinit and to provide enhanced management of services. For more
1951 information on systemd, see the systemd homepage at
Andrew Geissler4c19ea12020-10-27 13:52:24 -05001952 https://freedesktop.org/wiki/Software/systemd/.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001953
1954 To enable a service using systemd, your recipe needs to inherit the
1955 :ref:`systemd <ref-classes-systemd>` class. See
1956 the ``systemd.bbclass`` file located in your :term:`Source Directory`
1957 section for
1958 more information.
1959
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001960Packaging
1961---------
1962
1963Successful packaging is a combination of automated processes performed
1964by the OpenEmbedded build system and some specific steps you need to
1965take. The following list describes the process:
1966
1967- *Splitting Files*: The ``do_package`` task splits the files produced
1968 by the recipe into logical components. Even software that produces a
1969 single binary might still have debug symbols, documentation, and
1970 other logical components that should be split out. The ``do_package``
1971 task ensures that files are split up and packaged correctly.
1972
1973- *Running QA Checks*: The
1974 :ref:`insane <ref-classes-insane>` class adds a
1975 step to the package generation process so that output quality
1976 assurance checks are generated by the OpenEmbedded build system. This
1977 step performs a range of checks to be sure the build's output is free
1978 of common problems that show up during runtime. For information on
1979 these checks, see the
1980 :ref:`insane <ref-classes-insane>` class and
Andrew Geissler09209ee2020-12-13 08:44:15 -06001981 the ":ref:`ref-manual/qa-checks:qa error and warning messages`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001982 chapter in the Yocto Project Reference Manual.
1983
1984- *Hand-Checking Your Packages*: After you build your software, you
1985 need to be sure your packages are correct. Examine the
1986 ``${``\ :term:`WORKDIR`\ ``}/packages-split``
1987 directory and make sure files are where you expect them to be. If you
1988 discover problems, you can set
1989 :term:`PACKAGES`,
1990 :term:`FILES`,
1991 ``do_install(_append)``, and so forth as needed.
1992
1993- *Splitting an Application into Multiple Packages*: If you need to
1994 split an application into several packages, see the "`Splitting an
1995 Application into Multiple
1996 Packages <#splitting-an-application-into-multiple-packages>`__"
1997 section for an example.
1998
1999- *Installing a Post-Installation Script*: For an example showing how
2000 to install a post-installation script, see the "`Post-Installation
2001 Scripts <#new-recipe-post-installation-scripts>`__" section.
2002
2003- *Marking Package Architecture*: Depending on what your recipe is
2004 building and how it is configured, it might be important to mark the
2005 packages produced as being specific to a particular machine, or to
2006 mark them as not being specific to a particular machine or
2007 architecture at all.
2008
2009 By default, packages apply to any machine with the same architecture
2010 as the target machine. When a recipe produces packages that are
2011 machine-specific (e.g. the
2012 :term:`MACHINE` value is passed
2013 into the configure script or a patch is applied only for a particular
2014 machine), you should mark them as such by adding the following to the
2015 recipe:
2016 ::
2017
2018 PACKAGE_ARCH = "${MACHINE_ARCH}"
2019
2020 On the other hand, if the recipe produces packages that do not
2021 contain anything specific to the target machine or architecture at
2022 all (e.g. recipes that simply package script files or configuration
2023 files), you should use the
2024 :ref:`allarch <ref-classes-allarch>` class to
2025 do this for you by adding this to your recipe:
2026 ::
2027
2028 inherit allarch
2029
2030 Ensuring that the package architecture is correct is not critical
2031 while you are doing the first few builds of your recipe. However, it
2032 is important in order to ensure that your recipe rebuilds (or does
2033 not rebuild) appropriately in response to changes in configuration,
2034 and to ensure that you get the appropriate packages installed on the
2035 target machine, particularly if you run separate builds for more than
2036 one target machine.
2037
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002038Sharing Files Between Recipes
2039-----------------------------
2040
2041Recipes often need to use files provided by other recipes on the build
2042host. For example, an application linking to a common library needs
2043access to the library itself and its associated headers. The way this
2044access is accomplished is by populating a sysroot with files. Each
2045recipe has two sysroots in its work directory, one for target files
2046(``recipe-sysroot``) and one for files that are native to the build host
2047(``recipe-sysroot-native``).
2048
2049.. note::
2050
2051 You could find the term "staging" used within the Yocto project
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002052 regarding files populating sysroots (e.g. the :term:`STAGING_DIR`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002053 variable).
2054
2055Recipes should never populate the sysroot directly (i.e. write files
2056into sysroot). Instead, files should be installed into standard
2057locations during the
2058:ref:`ref-tasks-install` task within
2059the ``${``\ :term:`D`\ ``}`` directory. The
2060reason for this limitation is that almost all files that populate the
2061sysroot are cataloged in manifests in order to ensure the files can be
2062removed later when a recipe is either modified or removed. Thus, the
2063sysroot is able to remain free from stale files.
2064
2065A subset of the files installed by the
2066:ref:`ref-tasks-install` task are
2067used by the
2068:ref:`ref-tasks-populate_sysroot`
2069task as defined by the the
2070:term:`SYSROOT_DIRS` variable to
2071automatically populate the sysroot. It is possible to modify the list of
2072directories that populate the sysroot. The following example shows how
2073you could add the ``/opt`` directory to the list of directories within a
2074recipe:
2075::
2076
2077 SYSROOT_DIRS += "/opt"
2078
2079For a more complete description of the
2080:ref:`ref-tasks-populate_sysroot`
2081task and its associated functions, see the
2082:ref:`staging <ref-classes-staging>` class.
2083
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002084Using Virtual Providers
2085-----------------------
2086
2087Prior to a build, if you know that several different recipes provide the
2088same functionality, you can use a virtual provider (i.e. ``virtual/*``)
2089as a placeholder for the actual provider. The actual provider is
2090determined at build-time.
2091
2092A common scenario where a virtual provider is used would be for the
2093kernel recipe. Suppose you have three kernel recipes whose
2094:term:`PN` values map to ``kernel-big``,
2095``kernel-mid``, and ``kernel-small``. Furthermore, each of these recipes
2096in some way uses a :term:`PROVIDES`
2097statement that essentially identifies itself as being able to provide
2098``virtual/kernel``. Here is one way through the
2099:ref:`kernel <ref-classes-kernel>` class:
2100::
2101
2102 PROVIDES += "${@ "virtual/kernel" if (d.getVar("KERNEL_PACKAGE_NAME") == "kernel") else "" }"
2103
2104Any recipe that inherits the ``kernel`` class is
2105going to utilize a ``PROVIDES`` statement that identifies that recipe as
2106being able to provide the ``virtual/kernel`` item.
2107
2108Now comes the time to actually build an image and you need a kernel
2109recipe, but which one? You can configure your build to call out the
Andrew Geissler09209ee2020-12-13 08:44:15 -06002110kernel recipe you want by using the :term:`PREFERRED_PROVIDER` variable. As
2111an example, consider the :yocto_git:`x86-base.inc
2112</poky/tree/meta/conf/machine/include/x86-base.inc>` include file, which is a
2113machine (i.e. :term:`MACHINE`) configuration file. This include file is the
2114reason all x86-based machines use the ``linux-yocto`` kernel. Here are the
2115relevant lines from the include file:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002116::
2117
2118 PREFERRED_PROVIDER_virtual/kernel ??= "linux-yocto"
2119 PREFERRED_VERSION_linux-yocto ??= "4.15%"
2120
2121When you use a virtual provider, you do not have to "hard code" a recipe
2122name as a build dependency. You can use the
2123:term:`DEPENDS` variable to state the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002124build is dependent on ``virtual/kernel`` for example:
2125::
2126
2127 DEPENDS = "virtual/kernel"
2128
2129During the build, the OpenEmbedded build system picks
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002130the correct recipe needed for the ``virtual/kernel`` dependency based on
2131the ``PREFERRED_PROVIDER`` variable. If you want to use the small kernel
2132mentioned at the beginning of this section, configure your build as
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002133follows:
2134::
2135
2136 PREFERRED_PROVIDER_virtual/kernel ??= "kernel-small"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002137
2138.. note::
2139
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002140 Any recipe that ``PROVIDES`` a ``virtual/*`` item that is ultimately not
2141 selected through ``PREFERRED_PROVIDER`` does not get built. Preventing these
2142 recipes from building is usually the desired behavior since this mechanism's
2143 purpose is to select between mutually exclusive alternative providers.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002144
2145The following lists specific examples of virtual providers:
2146
2147- ``virtual/kernel``: Provides the name of the kernel recipe to use
2148 when building a kernel image.
2149
2150- ``virtual/bootloader``: Provides the name of the bootloader to use
2151 when building an image.
2152
2153- ``virtual/libgbm``: Provides ``gbm.pc``.
2154
2155- ``virtual/egl``: Provides ``egl.pc`` and possibly ``wayland-egl.pc``.
2156
2157- ``virtual/libgl``: Provides ``gl.pc`` (i.e. libGL).
2158
2159- ``virtual/libgles1``: Provides ``glesv1_cm.pc`` (i.e. libGLESv1_CM).
2160
2161- ``virtual/libgles2``: Provides ``glesv2.pc`` (i.e. libGLESv2).
2162
2163.. note::
2164
2165 Virtual providers only apply to build time dependencies specified with
2166 :term:`PROVIDES` and :term:`DEPENDS`. They do not apply to runtime
2167 dependencies specified with :term:`RPROVIDES` and :term:`RDEPENDS`.
2168
2169Properly Versioning Pre-Release Recipes
2170---------------------------------------
2171
2172Sometimes the name of a recipe can lead to versioning problems when the
2173recipe is upgraded to a final release. For example, consider the
2174``irssi_0.8.16-rc1.bb`` recipe file in the list of example recipes in
2175the "`Storing and Naming the
2176Recipe <#new-recipe-storing-and-naming-the-recipe>`__" section. This
2177recipe is at a release candidate stage (i.e. "rc1"). When the recipe is
2178released, the recipe filename becomes ``irssi_0.8.16.bb``. The version
2179change from ``0.8.16-rc1`` to ``0.8.16`` is seen as a decrease by the
2180build system and package managers, so the resulting packages will not
2181correctly trigger an upgrade.
2182
2183In order to ensure the versions compare properly, the recommended
2184convention is to set :term:`PV` within the
2185recipe to "previous_version+current_version". You can use an additional
2186variable so that you can use the current version elsewhere. Here is an
2187example:
2188::
2189
2190 REALPV = "0.8.16-rc1"
2191 PV = "0.8.15+${REALPV}"
2192
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002193Post-Installation Scripts
2194-------------------------
2195
2196Post-installation scripts run immediately after installing a package on
2197the target or during image creation when a package is included in an
2198image. To add a post-installation script to a package, add a
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002199``pkg_postinst_``\ `PACKAGENAME`\ ``()`` function to the recipe file
2200(``.bb``) and replace `PACKAGENAME` with the name of the package you want
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002201to attach to the ``postinst`` script. To apply the post-installation
2202script to the main package for the recipe, which is usually what is
2203required, specify
2204``${``\ :term:`PN`\ ``}`` in place of
2205PACKAGENAME.
2206
2207A post-installation function has the following structure:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002208::
2209
2210 pkg_postinst_PACKAGENAME() {
2211 # Commands to carry out
2212 }
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002213
2214The script defined in the post-installation function is called when the
2215root filesystem is created. If the script succeeds, the package is
2216marked as installed.
2217
2218.. note::
2219
2220 Any RPM post-installation script that runs on the target should
2221 return a 0 exit code. RPM does not allow non-zero exit codes for
2222 these scripts, and the RPM package manager will cause the package to
2223 fail installation on the target.
2224
2225Sometimes it is necessary for the execution of a post-installation
2226script to be delayed until the first boot. For example, the script might
2227need to be executed on the device itself. To delay script execution
2228until boot time, you must explicitly mark post installs to defer to the
2229target. You can use ``pkg_postinst_ontarget()`` or call
2230``postinst_intercept delay_to_first_boot`` from ``pkg_postinst()``. Any
2231failure of a ``pkg_postinst()`` script (including exit 1) triggers an
2232error during the
2233:ref:`ref-tasks-rootfs` task.
2234
2235If you have recipes that use ``pkg_postinst`` function and they require
2236the use of non-standard native tools that have dependencies during
2237rootfs construction, you need to use the
2238:term:`PACKAGE_WRITE_DEPS`
2239variable in your recipe to list these tools. If you do not use this
2240variable, the tools might be missing and execution of the
2241post-installation script is deferred until first boot. Deferring the
2242script to first boot is undesirable and for read-only rootfs impossible.
2243
2244.. note::
2245
2246 Equivalent support for pre-install, pre-uninstall, and post-uninstall
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002247 scripts exist by way of ``pkg_preinst``, ``pkg_prerm``, and ``pkg_postrm``,
2248 respectively. These scrips work in exactly the same way as does
2249 ``pkg_postinst`` with the exception that they run at different times. Also,
2250 because of when they run, they are not applicable to being run at image
2251 creation time like ``pkg_postinst``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002252
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002253Testing
2254-------
2255
2256The final step for completing your recipe is to be sure that the
2257software you built runs correctly. To accomplish runtime testing, add
2258the build's output packages to your image and test them on the target.
2259
2260For information on how to customize your image by adding specific
2261packages, see the "`Customizing
2262Images <#usingpoky-extend-customimage>`__" section.
2263
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002264Examples
2265--------
2266
2267To help summarize how to write a recipe, this section provides some
2268examples given various scenarios:
2269
2270- Recipes that use local files
2271
2272- Using an Autotooled package
2273
2274- Using a Makefile-based package
2275
2276- Splitting an application into multiple packages
2277
2278- Adding binaries to an image
2279
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002280Single .c File Package (Hello World!)
2281~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2282
2283Building an application from a single file that is stored locally (e.g.
2284under ``files``) requires a recipe that has the file listed in the
2285``SRC_URI`` variable. Additionally, you need to manually write the
2286``do_compile`` and ``do_install`` tasks. The ``S`` variable defines the
2287directory containing the source code, which is set to
2288:term:`WORKDIR` in this case - the
2289directory BitBake uses for the build.
2290::
2291
2292 SUMMARY = "Simple helloworld application"
2293 SECTION = "examples"
2294 LICENSE = "MIT"
2295 LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
2296
2297 SRC_URI = "file://helloworld.c"
2298
2299 S = "${WORKDIR}"
2300
2301 do_compile() {
2302 ${CC} helloworld.c -o helloworld
2303 }
2304
2305 do_install() {
2306 install -d ${D}${bindir}
2307 install -m 0755 helloworld ${D}${bindir}
2308 }
2309
2310By default, the ``helloworld``, ``helloworld-dbg``, and
2311``helloworld-dev`` packages are built. For information on how to
2312customize the packaging process, see the "`Splitting an Application into
2313Multiple Packages <#splitting-an-application-into-multiple-packages>`__"
2314section.
2315
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002316Autotooled Package
2317~~~~~~~~~~~~~~~~~~
2318
2319Applications that use Autotools such as ``autoconf`` and ``automake``
2320require a recipe that has a source archive listed in ``SRC_URI`` and
2321also inherit the
2322:ref:`autotools <ref-classes-autotools>` class,
2323which contains the definitions of all the steps needed to build an
2324Autotool-based application. The result of the build is automatically
2325packaged. And, if the application uses NLS for localization, packages
2326with local information are generated (one package per language).
2327Following is one example: (``hello_2.3.bb``)
2328::
2329
2330 SUMMARY = "GNU Helloworld application"
2331 SECTION = "examples"
2332 LICENSE = "GPLv2+"
2333 LIC_FILES_CHKSUM = "file://COPYING;md5=751419260aa954499f7abaabaa882bbe"
2334
2335 SRC_URI = "${GNU_MIRROR}/hello/hello-${PV}.tar.gz"
2336
2337 inherit autotools gettext
2338
2339The variable ``LIC_FILES_CHKSUM`` is used to track source license
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002340changes as described in the
Andrew Geissler09209ee2020-12-13 08:44:15 -06002341":ref:`dev-manual/common-tasks:tracking license changes`" section in
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002342the Yocto Project Overview and Concepts Manual. You can quickly create
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002343Autotool-based recipes in a manner similar to the previous example.
2344
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002345Makefile-Based Package
2346~~~~~~~~~~~~~~~~~~~~~~
2347
2348Applications that use GNU ``make`` also require a recipe that has the
2349source archive listed in ``SRC_URI``. You do not need to add a
2350``do_compile`` step since by default BitBake starts the ``make`` command
2351to compile the application. If you need additional ``make`` options, you
2352should store them in the
2353:term:`EXTRA_OEMAKE` or
2354:term:`PACKAGECONFIG_CONFARGS`
2355variables. BitBake passes these options into the GNU ``make``
2356invocation. Note that a ``do_install`` task is still required.
2357Otherwise, BitBake runs an empty ``do_install`` task by default.
2358
2359Some applications might require extra parameters to be passed to the
2360compiler. For example, the application might need an additional header
2361path. You can accomplish this by adding to the ``CFLAGS`` variable. The
2362following example shows this:
2363::
2364
2365 CFLAGS_prepend = "-I ${S}/include "
2366
2367In the following example, ``mtd-utils`` is a makefile-based package:
2368::
2369
2370 SUMMARY = "Tools for managing memory technology devices"
2371 SECTION = "base"
2372 DEPENDS = "zlib lzo e2fsprogs util-linux"
2373 HOMEPAGE = "http://www.linux-mtd.infradead.org/"
2374 LICENSE = "GPLv2+"
2375 LIC_FILES_CHKSUM = "file://COPYING;md5=0636e73ff0215e8d672dc4c32c317bb3 \
2376 file://include/common.h;beginline=1;endline=17;md5=ba05b07912a44ea2bf81ce409380049c"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002377
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002378 # Use the latest version at 26 Oct, 2013
2379 SRCREV = "9f107132a6a073cce37434ca9cda6917dd8d866b"
2380 SRC_URI = "git://git.infradead.org/mtd-utils.git \
2381 file://add-exclusion-to-mkfs-jffs2-git-2.patch \
2382 "
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002383
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002384 PV = "1.5.1+git${SRCPV}"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002385
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002386 S = "${WORKDIR}/git"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002387
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002388 EXTRA_OEMAKE = "'CC=${CC}' 'RANLIB=${RANLIB}' 'AR=${AR}' 'CFLAGS=${CFLAGS} -I${S}/include -DWITHOUT_XATTR' 'BUILDDIR=${S}'"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002389
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002390 do_install () {
2391 oe_runmake install DESTDIR=${D} SBINDIR=${sbindir} MANDIR=${mandir} INCLUDEDIR=${includedir}
2392 }
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002393
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002394 PACKAGES =+ "mtd-utils-jffs2 mtd-utils-ubifs mtd-utils-misc"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002395
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002396 FILES_mtd-utils-jffs2 = "${sbindir}/mkfs.jffs2 ${sbindir}/jffs2dump ${sbindir}/jffs2reader ${sbindir}/sumtool"
2397 FILES_mtd-utils-ubifs = "${sbindir}/mkfs.ubifs ${sbindir}/ubi*"
2398 FILES_mtd-utils-misc = "${sbindir}/nftl* ${sbindir}/ftl* ${sbindir}/rfd* ${sbindir}/doc* ${sbindir}/serve_image ${sbindir}/recv_image"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002399
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002400 PARALLEL_MAKE = ""
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002401
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002402 BBCLASSEXTEND = "native"
2403
2404Splitting an Application into Multiple Packages
2405~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2406
2407You can use the variables ``PACKAGES`` and ``FILES`` to split an
2408application into multiple packages.
2409
2410Following is an example that uses the ``libxpm`` recipe. By default,
2411this recipe generates a single package that contains the library along
2412with a few binaries. You can modify the recipe to split the binaries
2413into separate packages:
2414::
2415
2416 require xorg-lib-common.inc
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002417
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002418 SUMMARY = "Xpm: X Pixmap extension library"
2419 LICENSE = "BSD"
2420 LIC_FILES_CHKSUM = "file://COPYING;md5=51f4270b012ecd4ab1a164f5f4ed6cf7"
2421 DEPENDS += "libxext libsm libxt"
2422 PE = "1"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002423
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002424 XORG_PN = "libXpm"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002425
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002426 PACKAGES =+ "sxpm cxpm"
2427 FILES_cxpm = "${bindir}/cxpm"
2428 FILES_sxpm = "${bindir}/sxpm"
2429
2430In the previous example, we want to ship the ``sxpm`` and ``cxpm``
2431binaries in separate packages. Since ``bindir`` would be packaged into
2432the main ``PN`` package by default, we prepend the ``PACKAGES`` variable
2433so additional package names are added to the start of list. This results
2434in the extra ``FILES_*`` variables then containing information that
2435define which files and directories go into which packages. Files
2436included by earlier packages are skipped by latter packages. Thus, the
2437main ``PN`` package does not include the above listed files.
2438
2439Packaging Externally Produced Binaries
2440~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2441
2442Sometimes, you need to add pre-compiled binaries to an image. For
2443example, suppose that binaries for proprietary code exist, which are
2444created by a particular division of a company. Your part of the company
2445needs to use those binaries as part of an image that you are building
2446using the OpenEmbedded build system. Since you only have the binaries
2447and not the source code, you cannot use a typical recipe that expects to
2448fetch the source specified in
2449:term:`SRC_URI` and then compile it.
2450
2451One method is to package the binaries and then install them as part of
2452the image. Generally, it is not a good idea to package binaries since,
2453among other things, it can hinder the ability to reproduce builds and
2454could lead to compatibility problems with ABI in the future. However,
2455sometimes you have no choice.
2456
2457The easiest solution is to create a recipe that uses the
2458:ref:`bin_package <ref-classes-bin-package>` class
2459and to be sure that you are using default locations for build artifacts.
2460In most cases, the ``bin_package`` class handles "skipping" the
2461configure and compile steps as well as sets things up to grab packages
2462from the appropriate area. In particular, this class sets ``noexec`` on
2463both the :ref:`ref-tasks-configure`
2464and :ref:`ref-tasks-compile` tasks,
2465sets ``FILES_${PN}`` to "/" so that it picks up all files, and sets up a
2466:ref:`ref-tasks-install` task, which
2467effectively copies all files from ``${S}`` to ``${D}``. The
2468``bin_package`` class works well when the files extracted into ``${S}``
2469are already laid out in the way they should be laid out on the target.
2470For more information on these variables, see the
2471:term:`FILES`,
2472:term:`PN`,
2473:term:`S`, and
2474:term:`D` variables in the Yocto Project
2475Reference Manual's variable glossary.
2476
2477.. note::
2478
2479 - Using :term:`DEPENDS` is a good
2480 idea even for components distributed in binary form, and is often
2481 necessary for shared libraries. For a shared library, listing the
2482 library dependencies in ``DEPENDS`` makes sure that the libraries
2483 are available in the staging sysroot when other recipes link
2484 against the library, which might be necessary for successful
2485 linking.
2486
2487 - Using ``DEPENDS`` also allows runtime dependencies between
2488 packages to be added automatically. See the
Andrew Geissler09209ee2020-12-13 08:44:15 -06002489 ":ref:`overview-manual/concepts:automatically added runtime dependencies`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002490 section in the Yocto Project Overview and Concepts Manual for more
2491 information.
2492
2493If you cannot use the ``bin_package`` class, you need to be sure you are
2494doing the following:
2495
2496- Create a recipe where the
2497 :ref:`ref-tasks-configure` and
2498 :ref:`ref-tasks-compile` tasks do
2499 nothing: It is usually sufficient to just not define these tasks in
2500 the recipe, because the default implementations do nothing unless a
2501 Makefile is found in
2502 ``${``\ :term:`S`\ ``}``.
2503
2504 If ``${S}`` might contain a Makefile, or if you inherit some class
2505 that replaces ``do_configure`` and ``do_compile`` with custom
2506 versions, then you can use the
2507 ``[``\ :ref:`noexec <bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ``]``
2508 flag to turn the tasks into no-ops, as follows:
2509 ::
2510
2511 do_configure[noexec] = "1"
2512 do_compile[noexec] = "1"
2513
2514 Unlike
2515 :ref:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata:deleting a task`,
2516 using the flag preserves the dependency chain from the
2517 :ref:`ref-tasks-fetch`,
2518 :ref:`ref-tasks-unpack`, and
2519 :ref:`ref-tasks-patch` tasks to the
2520 :ref:`ref-tasks-install` task.
2521
2522- Make sure your ``do_install`` task installs the binaries
2523 appropriately.
2524
2525- Ensure that you set up :term:`FILES`
2526 (usually
2527 ``FILES_${``\ :term:`PN`\ ``}``) to
2528 point to the files you have installed, which of course depends on
2529 where you have installed them and whether those files are in
2530 different locations than the defaults.
2531
Andrew Geissler6ce62a22020-11-30 19:58:47 -06002532.. note::
2533
2534 If image prelinking is enabled (e.g. "image-prelink" is in :term:`USER_CLASSES`
2535 which it is by default), prelink will change the binaries in the generated images
2536 and this often catches people out. Remove that class to ensure binaries are
2537 preserved exactly if that is necessary.
2538
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002539Following Recipe Style Guidelines
2540---------------------------------
2541
2542When writing recipes, it is good to conform to existing style
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002543guidelines. The :oe_home:`OpenEmbedded Styleguide </wiki/Styleguide>` wiki page
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002544provides rough guidelines for preferred recipe style.
2545
2546It is common for existing recipes to deviate a bit from this style.
2547However, aiming for at least a consistent style is a good idea. Some
2548practices, such as omitting spaces around ``=`` operators in assignments
2549or ordering recipe components in an erratic way, are widely seen as poor
2550style.
2551
2552Recipe Syntax
2553-------------
2554
2555Understanding recipe file syntax is important for writing recipes. The
2556following list overviews the basic items that make up a BitBake recipe
2557file. For more complete BitBake syntax descriptions, see the
2558":doc:`bitbake-user-manual/bitbake-user-manual-metadata`"
2559chapter of the BitBake User Manual.
2560
2561- *Variable Assignments and Manipulations:* Variable assignments allow
2562 a value to be assigned to a variable. The assignment can be static
2563 text or might include the contents of other variables. In addition to
2564 the assignment, appending and prepending operations are also
2565 supported.
2566
2567 The following example shows some of the ways you can use variables in
2568 recipes:
2569 ::
2570
2571 S = "${WORKDIR}/postfix-${PV}"
2572 CFLAGS += "-DNO_ASM"
2573 SRC_URI_append = " file://fixup.patch"
2574
2575- *Functions:* Functions provide a series of actions to be performed.
2576 You usually use functions to override the default implementation of a
2577 task function or to complement a default function (i.e. append or
2578 prepend to an existing function). Standard functions use ``sh`` shell
2579 syntax, although access to OpenEmbedded variables and internal
2580 methods are also available.
2581
2582 The following is an example function from the ``sed`` recipe:
2583 ::
2584
2585 do_install () {
2586 autotools_do_install
2587 install -d ${D}${base_bindir}
2588 mv ${D}${bindir}/sed ${D}${base_bindir}/sed
2589 rmdir ${D}${bindir}/
2590 }
2591
2592 It is
2593 also possible to implement new functions that are called between
2594 existing tasks as long as the new functions are not replacing or
2595 complementing the default functions. You can implement functions in
2596 Python instead of shell. Both of these options are not seen in the
2597 majority of recipes.
2598
2599- *Keywords:* BitBake recipes use only a few keywords. You use keywords
2600 to include common functions (``inherit``), load parts of a recipe
2601 from other files (``include`` and ``require``) and export variables
2602 to the environment (``export``).
2603
2604 The following example shows the use of some of these keywords:
2605 ::
2606
2607 export POSTCONF = "${STAGING_BINDIR}/postconf"
2608 inherit autoconf
2609 require otherfile.inc
2610
2611- *Comments (#):* Any lines that begin with the hash character (``#``)
2612 are treated as comment lines and are ignored:
2613 ::
2614
2615 # This is a comment
2616
2617This next list summarizes the most important and most commonly used
2618parts of the recipe syntax. For more information on these parts of the
2619syntax, you can reference the
2620:doc:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata` chapter
2621in the BitBake User Manual.
2622
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002623- *Line Continuation (\\):* Use the backward slash (``\``) character to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002624 split a statement over multiple lines. Place the slash character at
2625 the end of the line that is to be continued on the next line:
2626 ::
2627
2628 VAR = "A really long \
2629 line"
2630
2631 .. note::
2632
2633 You cannot have any characters including spaces or tabs after the
2634 slash character.
2635
2636- *Using Variables (${VARNAME}):* Use the ``${VARNAME}`` syntax to
2637 access the contents of a variable:
2638 ::
2639
2640 SRC_URI = "${SOURCEFORGE_MIRROR}/libpng/zlib-${PV}.tar.gz"
2641
2642 .. note::
2643
2644 It is important to understand that the value of a variable
2645 expressed in this form does not get substituted automatically. The
2646 expansion of these expressions happens on-demand later (e.g.
2647 usually when a function that makes reference to the variable
2648 executes). This behavior ensures that the values are most
2649 appropriate for the context in which they are finally used. On the
2650 rare occasion that you do need the variable expression to be
2651 expanded immediately, you can use the
2652 :=
2653 operator instead of
2654 =
2655 when you make the assignment, but this is not generally needed.
2656
2657- *Quote All Assignments ("value"):* Use double quotes around values in
2658 all variable assignments (e.g. ``"value"``). Following is an example:
2659 ::
2660
2661 VAR1 = "${OTHERVAR}"
2662 VAR2 = "The version is ${PV}"
2663
2664- *Conditional Assignment (?=):* Conditional assignment is used to
2665 assign a value to a variable, but only when the variable is currently
2666 unset. Use the question mark followed by the equal sign (``?=``) to
2667 make a "soft" assignment used for conditional assignment. Typically,
2668 "soft" assignments are used in the ``local.conf`` file for variables
2669 that are allowed to come through from the external environment.
2670
2671 Here is an example where ``VAR1`` is set to "New value" if it is
2672 currently empty. However, if ``VAR1`` has already been set, it
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002673 remains unchanged:
2674 ::
2675
2676 VAR1 ?= "New value"
2677
2678 In this next example, ``VAR1`` is left with the value "Original value":
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002679 ::
2680
2681 VAR1 = "Original value"
2682 VAR1 ?= "New value"
2683
2684- *Appending (+=):* Use the plus character followed by the equals sign
2685 (``+=``) to append values to existing variables.
2686
2687 .. note::
2688
2689 This operator adds a space between the existing content of the
2690 variable and the new content.
2691
2692 Here is an example:
2693 ::
2694
2695 SRC_URI += "file://fix-makefile.patch"
2696
2697- *Prepending (=+):* Use the equals sign followed by the plus character
2698 (``=+``) to prepend values to existing variables.
2699
2700 .. note::
2701
2702 This operator adds a space between the new content and the
2703 existing content of the variable.
2704
2705 Here is an example:
2706 ::
2707
2708 VAR =+ "Starts"
2709
2710- *Appending (_append):* Use the ``_append`` operator to append values
2711 to existing variables. This operator does not add any additional
2712 space. Also, the operator is applied after all the ``+=``, and ``=+``
2713 operators have been applied and after all ``=`` assignments have
2714 occurred.
2715
2716 The following example shows the space being explicitly added to the
2717 start to ensure the appended value is not merged with the existing
2718 value:
2719 ::
2720
2721 SRC_URI_append = " file://fix-makefile.patch"
2722
2723 You can also use
2724 the ``_append`` operator with overrides, which results in the actions
2725 only being performed for the specified target or machine:
2726 ::
2727
2728 SRC_URI_append_sh4 = " file://fix-makefile.patch"
2729
2730- *Prepending (_prepend):* Use the ``_prepend`` operator to prepend
2731 values to existing variables. This operator does not add any
2732 additional space. Also, the operator is applied after all the ``+=``,
2733 and ``=+`` operators have been applied and after all ``=``
2734 assignments have occurred.
2735
2736 The following example shows the space being explicitly added to the
2737 end to ensure the prepended value is not merged with the existing
2738 value:
2739 ::
2740
2741 CFLAGS_prepend = "-I${S}/myincludes "
2742
2743 You can also use the
2744 ``_prepend`` operator with overrides, which results in the actions
2745 only being performed for the specified target or machine:
2746 ::
2747
2748 CFLAGS_prepend_sh4 = "-I${S}/myincludes "
2749
2750- *Overrides:* You can use overrides to set a value conditionally,
2751 typically based on how the recipe is being built. For example, to set
2752 the :term:`KBRANCH` variable's
2753 value to "standard/base" for any target
2754 :term:`MACHINE`, except for
2755 qemuarm where it should be set to "standard/arm-versatile-926ejs",
2756 you would do the following:
2757 ::
2758
2759 KBRANCH = "standard/base"
2760 KBRANCH_qemuarm = "standard/arm-versatile-926ejs"
2761
2762 Overrides are also used to separate
2763 alternate values of a variable in other situations. For example, when
2764 setting variables such as
2765 :term:`FILES` and
2766 :term:`RDEPENDS` that are
2767 specific to individual packages produced by a recipe, you should
2768 always use an override that specifies the name of the package.
2769
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002770- *Indentation:* Use spaces for indentation rather than tabs. For
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002771 shell functions, both currently work. However, it is a policy
2772 decision of the Yocto Project to use tabs in shell functions. Realize
2773 that some layers have a policy to use spaces for all indentation.
2774
2775- *Using Python for Complex Operations:* For more advanced processing,
2776 it is possible to use Python code during variable assignments (e.g.
2777 search and replacement on a variable).
2778
2779 You indicate Python code using the ``${@python_code}`` syntax for the
2780 variable assignment:
2781 ::
2782
2783 SRC_URI = "ftp://ftp.info-zip.org/pub/infozip/src/zip${@d.getVar('PV',1).replace('.', '')}.tgz
2784
2785- *Shell Function Syntax:* Write shell functions as if you were writing
2786 a shell script when you describe a list of actions to take. You
2787 should ensure that your script works with a generic ``sh`` and that
2788 it does not require any ``bash`` or other shell-specific
2789 functionality. The same considerations apply to various system
2790 utilities (e.g. ``sed``, ``grep``, ``awk``, and so forth) that you
2791 might wish to use. If in doubt, you should check with multiple
2792 implementations - including those from BusyBox.
2793
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002794Adding a New Machine
2795====================
2796
2797Adding a new machine to the Yocto Project is a straightforward process.
2798This section describes how to add machines that are similar to those
2799that the Yocto Project already supports.
2800
2801.. note::
2802
2803 Although well within the capabilities of the Yocto Project, adding a
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002804 totally new architecture might require changes to ``gcc``/``glibc``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002805 and to the site information, which is beyond the scope of this
2806 manual.
2807
2808For a complete example that shows how to add a new machine, see the
2809":ref:`bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script`"
2810section in the Yocto Project Board Support Package (BSP) Developer's
2811Guide.
2812
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002813Adding the Machine Configuration File
2814-------------------------------------
2815
2816To add a new machine, you need to add a new machine configuration file
2817to the layer's ``conf/machine`` directory. This configuration file
2818provides details about the device you are adding.
2819
2820The OpenEmbedded build system uses the root name of the machine
2821configuration file to reference the new machine. For example, given a
2822machine configuration file named ``crownbay.conf``, the build system
2823recognizes the machine as "crownbay".
2824
2825The most important variables you must set in your machine configuration
2826file or include from a lower-level configuration file are as follows:
2827
2828- ``TARGET_ARCH`` (e.g. "arm")
2829
2830- ``PREFERRED_PROVIDER_virtual/kernel``
2831
2832- ``MACHINE_FEATURES`` (e.g. "apm screen wifi")
2833
2834You might also need these variables:
2835
2836- ``SERIAL_CONSOLES`` (e.g. "115200;ttyS0 115200;ttyS1")
2837
2838- ``KERNEL_IMAGETYPE`` (e.g. "zImage")
2839
2840- ``IMAGE_FSTYPES`` (e.g. "tar.gz jffs2")
2841
2842You can find full details on these variables in the reference section.
2843You can leverage existing machine ``.conf`` files from
2844``meta-yocto-bsp/conf/machine/``.
2845
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002846Adding a Kernel for the Machine
2847-------------------------------
2848
2849The OpenEmbedded build system needs to be able to build a kernel for the
2850machine. You need to either create a new kernel recipe for this machine,
2851or extend an existing kernel recipe. You can find several kernel recipe
2852examples in the Source Directory at ``meta/recipes-kernel/linux`` that
2853you can use as references.
2854
2855If you are creating a new kernel recipe, normal recipe-writing rules
2856apply for setting up a ``SRC_URI``. Thus, you need to specify any
2857necessary patches and set ``S`` to point at the source code. You need to
2858create a ``do_configure`` task that configures the unpacked kernel with
2859a ``defconfig`` file. You can do this by using a ``make defconfig``
2860command or, more commonly, by copying in a suitable ``defconfig`` file
2861and then running ``make oldconfig``. By making use of ``inherit kernel``
2862and potentially some of the ``linux-*.inc`` files, most other
2863functionality is centralized and the defaults of the class normally work
2864well.
2865
2866If you are extending an existing kernel recipe, it is usually a matter
2867of adding a suitable ``defconfig`` file. The file needs to be added into
2868a location similar to ``defconfig`` files used for other machines in a
2869given kernel recipe. A possible way to do this is by listing the file in
2870the ``SRC_URI`` and adding the machine to the expression in
2871``COMPATIBLE_MACHINE``:
2872::
2873
2874 COMPATIBLE_MACHINE = '(qemux86|qemumips)'
2875
2876For more information on ``defconfig`` files, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06002877":ref:`kernel-dev/common:changing the configuration`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002878section in the Yocto Project Linux Kernel Development Manual.
2879
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002880Adding a Formfactor Configuration File
2881--------------------------------------
2882
2883A formfactor configuration file provides information about the target
2884hardware for which the image is being built and information that the
2885build system cannot obtain from other sources such as the kernel. Some
2886examples of information contained in a formfactor configuration file
2887include framebuffer orientation, whether or not the system has a
2888keyboard, the positioning of the keyboard in relation to the screen, and
2889the screen resolution.
2890
2891The build system uses reasonable defaults in most cases. However, if
2892customization is necessary, you need to create a ``machconfig`` file in
2893the ``meta/recipes-bsp/formfactor/files`` directory. This directory
2894contains directories for specific machines such as ``qemuarm`` and
2895``qemux86``. For information about the settings available and the
2896defaults, see the ``meta/recipes-bsp/formfactor/files/config`` file
2897found in the same area.
2898
2899Following is an example for "qemuarm" machine:
2900::
2901
2902 HAVE_TOUCHSCREEN=1
2903 HAVE_KEYBOARD=1
2904 DISPLAY_CAN_ROTATE=0
2905 DISPLAY_ORIENTATION=0
2906 #DISPLAY_WIDTH_PIXELS=640
2907 #DISPLAY_HEIGHT_PIXELS=480
2908 #DISPLAY_BPP=16
2909 DISPLAY_DPI=150
2910 DISPLAY_SUBPIXEL_ORDER=vrgb
2911
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002912Upgrading Recipes
2913=================
2914
2915Over time, upstream developers publish new versions for software built
2916by layer recipes. It is recommended to keep recipes up-to-date with
2917upstream version releases.
2918
2919While several methods exist that allow you upgrade a recipe, you might
2920consider checking on the upgrade status of a recipe first. You can do so
2921using the ``devtool check-upgrade-status`` command. See the
2922":ref:`devtool-checking-on-the-upgrade-status-of-a-recipe`"
2923section in the Yocto Project Reference Manual for more information.
2924
2925The remainder of this section describes three ways you can upgrade a
2926recipe. You can use the Automated Upgrade Helper (AUH) to set up
2927automatic version upgrades. Alternatively, you can use
2928``devtool upgrade`` to set up semi-automatic version upgrades. Finally,
2929you can manually upgrade a recipe by editing the recipe itself.
2930
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002931Using the Auto Upgrade Helper (AUH)
2932-----------------------------------
2933
2934The AUH utility works in conjunction with the OpenEmbedded build system
2935in order to automatically generate upgrades for recipes based on new
2936versions being published upstream. Use AUH when you want to create a
2937service that performs the upgrades automatically and optionally sends
2938you an email with the results.
2939
2940AUH allows you to update several recipes with a single use. You can also
2941optionally perform build and integration tests using images with the
2942results saved to your hard drive and emails of results optionally sent
2943to recipe maintainers. Finally, AUH creates Git commits with appropriate
2944commit messages in the layer's tree for the changes made to recipes.
2945
2946.. note::
2947
2948 Conditions do exist when you should not use AUH to upgrade recipes
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002949 and you should instead use either ``devtool upgrade`` or upgrade your
2950 recipes manually:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002951
2952 - When AUH cannot complete the upgrade sequence. This situation
2953 usually results because custom patches carried by the recipe
2954 cannot be automatically rebased to the new version. In this case,
2955 ``devtool upgrade`` allows you to manually resolve conflicts.
2956
2957 - When for any reason you want fuller control over the upgrade
2958 process. For example, when you want special arrangements for
2959 testing.
2960
2961The following steps describe how to set up the AUH utility:
2962
29631. *Be Sure the Development Host is Set Up:* You need to be sure that
2964 your development host is set up to use the Yocto Project. For
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002965 information on how to set up your host, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06002966 ":ref:`dev-manual/start:Preparing the Build Host`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002967
29682. *Make Sure Git is Configured:* The AUH utility requires Git to be
2969 configured because AUH uses Git to save upgrades. Thus, you must have
2970 Git user and email configured. The following command shows your
2971 configurations:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05002972 ::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002973
2974 $ git config --list
2975
2976 If you do not have the user and
2977 email configured, you can use the following commands to do so:
2978 ::
2979
2980 $ git config --global user.name some_name
2981 $ git config --global user.email username@domain.com
2982
29833. *Clone the AUH Repository:* To use AUH, you must clone the repository
2984 onto your development host. The following command uses Git to create
2985 a local copy of the repository on your system:
2986 ::
2987
2988 $ git clone git://git.yoctoproject.org/auto-upgrade-helper
2989 Cloning into 'auto-upgrade-helper'... remote: Counting objects: 768, done.
2990 remote: Compressing objects: 100% (300/300), done.
2991 remote: Total 768 (delta 499), reused 703 (delta 434)
2992 Receiving objects: 100% (768/768), 191.47 KiB | 98.00 KiB/s, done.
2993 Resolving deltas: 100% (499/499), done.
2994 Checking connectivity... done.
2995
2996 AUH is not part of the :term:`OpenEmbedded-Core (OE-Core)` or
2997 :term:`Poky` repositories.
2998
29994. *Create a Dedicated Build Directory:* Run the
3000 :ref:`structure-core-script`
3001 script to create a fresh build directory that you use exclusively for
3002 running the AUH utility:
3003 ::
3004
3005 $ cd ~/poky
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003006 $ source oe-init-build-env your_AUH_build_directory
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003007
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003008 Re-using an existing build directory and its configurations is not
3009 recommended as existing settings could cause AUH to fail or behave
3010 undesirably.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003011
30125. *Make Configurations in Your Local Configuration File:* Several
3013 settings need to exist in the ``local.conf`` file in the build
3014 directory you just created for AUH. Make these following
3015 configurations:
3016
3017 - If you want to enable :ref:`Build
Andrew Geissler09209ee2020-12-13 08:44:15 -06003018 History <dev-manual/common-tasks:maintaining build output quality>`,
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003019 which is optional, you need the following lines in the
3020 ``conf/local.conf`` file:
3021 ::
3022
3023 INHERIT =+ "buildhistory"
3024 BUILDHISTORY_COMMIT = "1"
3025
3026 With this configuration and a successful
3027 upgrade, a build history "diff" file appears in the
3028 ``upgrade-helper/work/recipe/buildhistory-diff.txt`` file found in
3029 your build directory.
3030
3031 - If you want to enable testing through the
3032 :ref:`testimage <ref-classes-testimage*>`
3033 class, which is optional, you need to have the following set in
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003034 your ``conf/local.conf`` file:
3035 ::
3036
3037 INHERIT += "testimage"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003038
3039 .. note::
3040
3041 If your distro does not enable by default ptest, which Poky
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003042 does, you need the following in your ``local.conf`` file:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003043 ::
3044
3045 DISTRO_FEATURES_append = " ptest"
3046
3047
30486. *Optionally Start a vncserver:* If you are running in a server
3049 without an X11 session, you need to start a vncserver:
3050 ::
3051
3052 $ vncserver :1
3053 $ export DISPLAY=:1
3054
30557. *Create and Edit an AUH Configuration File:* You need to have the
3056 ``upgrade-helper/upgrade-helper.conf`` configuration file in your
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003057 build directory. You can find a sample configuration file in the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003058 :yocto_git:`AUH source repository </auto-upgrade-helper/tree/>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003059
3060 Read through the sample file and make configurations as needed. For
3061 example, if you enabled build history in your ``local.conf`` as
3062 described earlier, you must enable it in ``upgrade-helper.conf``.
3063
3064 Also, if you are using the default ``maintainers.inc`` file supplied
3065 with Poky and located in ``meta-yocto`` and you do not set a
3066 "maintainers_whitelist" or "global_maintainer_override" in the
3067 ``upgrade-helper.conf`` configuration, and you specify "-e all" on
3068 the AUH command-line, the utility automatically sends out emails to
3069 all the default maintainers. Please avoid this.
3070
3071This next set of examples describes how to use the AUH:
3072
3073- *Upgrading a Specific Recipe:* To upgrade a specific recipe, use the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003074 following form:
3075 ::
3076
3077 $ upgrade-helper.py recipe_name
3078
3079 For example, this command upgrades the ``xmodmap`` recipe:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003080 ::
3081
3082 $ upgrade-helper.py xmodmap
3083
3084- *Upgrading a Specific Recipe to a Particular Version:* To upgrade a
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003085 specific recipe to a particular version, use the following form:
3086 ::
3087
3088 $ upgrade-helper.py recipe_name -t version
3089
3090 For example, this command upgrades the ``xmodmap`` recipe to version 1.2.3:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003091 ::
3092
3093 $ upgrade-helper.py xmodmap -t 1.2.3
3094
3095- *Upgrading all Recipes to the Latest Versions and Suppressing Email
3096 Notifications:* To upgrade all recipes to their most recent versions
3097 and suppress the email notifications, use the following command:
3098 ::
3099
3100 $ upgrade-helper.py all
3101
3102- *Upgrading all Recipes to the Latest Versions and Send Email
3103 Notifications:* To upgrade all recipes to their most recent versions
3104 and send email messages to maintainers for each attempted recipe as
3105 well as a status email, use the following command:
3106 ::
3107
3108 $ upgrade-helper.py -e all
3109
3110Once you have run the AUH utility, you can find the results in the AUH
3111build directory:
3112::
3113
3114 ${BUILDDIR}/upgrade-helper/timestamp
3115
3116The AUH utility
3117also creates recipe update commits from successful upgrade attempts in
3118the layer tree.
3119
3120You can easily set up to run the AUH utility on a regular basis by using
3121a cron job. See the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003122:yocto_git:`weeklyjob.sh </auto-upgrade-helper/tree/weeklyjob.sh>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003123file distributed with the utility for an example.
3124
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003125Using ``devtool upgrade``
3126-------------------------
3127
3128As mentioned earlier, an alternative method for upgrading recipes to
3129newer versions is to use
Andrew Geissler09209ee2020-12-13 08:44:15 -06003130:doc:`devtool upgrade </ref-manual/devtool-reference>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003131You can read about ``devtool upgrade`` in general in the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003132":ref:`sdk-manual/extensible:use \`\`devtool upgrade\`\` to create a version of the recipe that supports a newer version of the software`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003133section in the Yocto Project Application Development and the Extensible
3134Software Development Kit (eSDK) Manual.
3135
3136To see all the command-line options available with ``devtool upgrade``,
3137use the following help command:
3138::
3139
3140 $ devtool upgrade -h
3141
3142If you want to find out what version a recipe is currently at upstream
3143without any attempt to upgrade your local version of the recipe, you can
3144use the following command:
3145::
3146
3147 $ devtool latest-version recipe_name
3148
3149As mentioned in the previous section describing AUH, ``devtool upgrade``
3150works in a less-automated manner than AUH. Specifically,
3151``devtool upgrade`` only works on a single recipe that you name on the
3152command line, cannot perform build and integration testing using images,
3153and does not automatically generate commits for changes in the source
3154tree. Despite all these "limitations", ``devtool upgrade`` updates the
3155recipe file to the new upstream version and attempts to rebase custom
3156patches contained by the recipe as needed.
3157
3158.. note::
3159
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003160 AUH uses much of ``devtool upgrade`` behind the scenes making AUH somewhat
3161 of a "wrapper" application for ``devtool upgrade``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003162
3163A typical scenario involves having used Git to clone an upstream
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003164repository that you use during build operations. Because you have built the
3165recipe in the past, the layer is likely added to your
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003166configuration already. If for some reason, the layer is not added, you
3167could add it easily using the
3168":ref:`bitbake-layers <bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script>`"
3169script. For example, suppose you use the ``nano.bb`` recipe from the
3170``meta-oe`` layer in the ``meta-openembedded`` repository. For this
3171example, assume that the layer has been cloned into following area:
3172::
3173
3174 /home/scottrif/meta-openembedded
3175
3176The following command from your
3177:term:`Build Directory` adds the layer to
3178your build configuration (i.e. ``${BUILDDIR}/conf/bblayers.conf``):
3179::
3180
3181 $ bitbake-layers add-layer /home/scottrif/meta-openembedded/meta-oe
3182 NOTE: Starting bitbake server...
3183 Parsing recipes: 100% |##########################################| Time: 0:00:55
3184 Parsing of 1431 .bb files complete (0 cached, 1431 parsed). 2040 targets, 56 skipped, 0 masked, 0 errors.
3185 Removing 12 recipes from the x86_64 sysroot: 100% |##############| Time: 0:00:00
3186 Removing 1 recipes from the x86_64_i586 sysroot: 100% |##########| Time: 0:00:00
3187 Removing 5 recipes from the i586 sysroot: 100% |#################| Time: 0:00:00
3188 Removing 5 recipes from the qemux86 sysroot: 100% |##############| Time: 0:00:00
3189
3190For this example, assume that the ``nano.bb`` recipe that
3191is upstream has a 2.9.3 version number. However, the version in the
3192local repository is 2.7.4. The following command from your build
3193directory automatically upgrades the recipe for you:
3194
3195.. note::
3196
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003197 Using the ``-V`` option is not necessary. Omitting the version number causes
3198 ``devtool upgrade`` to upgrade the recipe to the most recent version.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003199
3200::
3201
3202 $ devtool upgrade nano -V 2.9.3
3203 NOTE: Starting bitbake server...
3204 NOTE: Creating workspace layer in /home/scottrif/poky/build/workspace
3205 Parsing recipes: 100% |##########################################| Time: 0:00:46
3206 Parsing of 1431 .bb files complete (0 cached, 1431 parsed). 2040 targets, 56 skipped, 0 masked, 0 errors.
3207 NOTE: Extracting current version source...
3208 NOTE: Resolving any missing task queue dependencies
3209 .
3210 .
3211 .
3212 NOTE: Executing SetScene Tasks
3213 NOTE: Executing RunQueue Tasks
3214 NOTE: Tasks Summary: Attempted 74 tasks of which 72 didn't need to be rerun and all succeeded.
3215 Adding changed files: 100% |#####################################| Time: 0:00:00
3216 NOTE: Upgraded source extracted to /home/scottrif/poky/build/workspace/sources/nano
3217 NOTE: New recipe is /home/scottrif/poky/build/workspace/recipes/nano/nano_2.9.3.bb
3218
3219Continuing with this example, you can use ``devtool build`` to build the
3220newly upgraded recipe:
3221::
3222
3223 $ devtool build nano
3224 NOTE: Starting bitbake server...
3225 Loading cache: 100% |################################################################################################| Time: 0:00:01
3226 Loaded 2040 entries from dependency cache.
3227 Parsing recipes: 100% |##############################################################################################| Time: 0:00:00
3228 Parsing of 1432 .bb files complete (1431 cached, 1 parsed). 2041 targets, 56 skipped, 0 masked, 0 errors.
3229 NOTE: Resolving any missing task queue dependencies
3230 .
3231 .
3232 .
3233 NOTE: Executing SetScene Tasks
3234 NOTE: Executing RunQueue Tasks
3235 NOTE: nano: compiling from external source tree /home/scottrif/poky/build/workspace/sources/nano
3236 NOTE: Tasks Summary: Attempted 520 tasks of which 304 didn't need to be rerun and all succeeded.
3237
3238Within the ``devtool upgrade`` workflow, opportunity
3239exists to deploy and test your rebuilt software. For this example,
3240however, running ``devtool finish`` cleans up the workspace once the
3241source in your workspace is clean. This usually means using Git to stage
3242and submit commits for the changes generated by the upgrade process.
3243
3244Once the tree is clean, you can clean things up in this example with the
3245following command from the ``${BUILDDIR}/workspace/sources/nano``
3246directory:
3247::
3248
3249 $ devtool finish nano meta-oe
3250 NOTE: Starting bitbake server...
3251 Loading cache: 100% |################################################################################################| Time: 0:00:00
3252 Loaded 2040 entries from dependency cache.
3253 Parsing recipes: 100% |##############################################################################################| Time: 0:00:01
3254 Parsing of 1432 .bb files complete (1431 cached, 1 parsed). 2041 targets, 56 skipped, 0 masked, 0 errors.
3255 NOTE: Adding new patch 0001-nano.bb-Stuff-I-changed-when-upgrading-nano.bb.patch
3256 NOTE: Updating recipe nano_2.9.3.bb
3257 NOTE: Removing file /home/scottrif/meta-openembedded/meta-oe/recipes-support/nano/nano_2.7.4.bb
3258 NOTE: Moving recipe file to /home/scottrif/meta-openembedded/meta-oe/recipes-support/nano
3259 NOTE: Leaving source tree /home/scottrif/poky/build/workspace/sources/nano as-is; if you no longer need it then please delete it manually
3260
3261
3262Using the ``devtool finish`` command cleans up the workspace and creates a patch
3263file based on your commits. The tool puts all patch files back into the
3264source directory in a sub-directory named ``nano`` in this case.
3265
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003266Manually Upgrading a Recipe
3267---------------------------
3268
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003269If for some reason you choose not to upgrade recipes using
Andrew Geissler09209ee2020-12-13 08:44:15 -06003270:ref:`dev-manual/common-tasks:Using the Auto Upgrade Helper (AUH)` or
3271by :ref:`dev-manual/common-tasks:Using \`\`devtool upgrade\`\``,
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003272you can manually edit the recipe files to upgrade the versions.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003273
3274.. note::
3275
3276 Manually updating multiple recipes scales poorly and involves many
3277 steps. The recommendation to upgrade recipe versions is through AUH
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003278 or ``devtool upgrade``, both of which automate some steps and provide
3279 guidance for others needed for the manual process.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003280
3281To manually upgrade recipe versions, follow these general steps:
3282
32831. *Change the Version:* Rename the recipe such that the version (i.e.
3284 the :term:`PV` part of the recipe name)
3285 changes appropriately. If the version is not part of the recipe name,
3286 change the value as it is set for ``PV`` within the recipe itself.
3287
Andrew Geissler4c19ea12020-10-27 13:52:24 -050032882. *Update* ``SRCREV`` *if Needed*: If the source code your recipe builds
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003289 is fetched from Git or some other version control system, update
3290 :term:`SRCREV` to point to the
3291 commit hash that matches the new version.
3292
32933. *Build the Software:* Try to build the recipe using BitBake. Typical
3294 build failures include the following:
3295
3296 - License statements were updated for the new version. For this
3297 case, you need to review any changes to the license and update the
3298 values of :term:`LICENSE` and
3299 :term:`LIC_FILES_CHKSUM`
3300 as needed.
3301
3302 .. note::
3303
3304 License changes are often inconsequential. For example, the
3305 license text's copyright year might have changed.
3306
3307 - Custom patches carried by the older version of the recipe might
3308 fail to apply to the new version. For these cases, you need to
3309 review the failures. Patches might not be necessary for the new
3310 version of the software if the upgraded version has fixed those
3311 issues. If a patch is necessary and failing, you need to rebase it
3312 into the new version.
3313
33144. *Optionally Attempt to Build for Several Architectures:* Once you
3315 successfully build the new software for a given architecture, you
3316 could test the build for other architectures by changing the
3317 :term:`MACHINE` variable and
3318 rebuilding the software. This optional step is especially important
3319 if the recipe is to be released publicly.
3320
33215. *Check the Upstream Change Log or Release Notes:* Checking both these
3322 reveals if new features exist that could break
3323 backwards-compatibility. If so, you need to take steps to mitigate or
3324 eliminate that situation.
3325
33266. *Optionally Create a Bootable Image and Test:* If you want, you can
3327 test the new software by booting it onto actual hardware.
3328
33297. *Create a Commit with the Change in the Layer Repository:* After all
3330 builds work and any testing is successful, you can create commits for
3331 any changes in the layer holding your upgraded recipe.
3332
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003333Finding Temporary Source Code
3334=============================
3335
3336You might find it helpful during development to modify the temporary
3337source code used by recipes to build packages. For example, suppose you
3338are developing a patch and you need to experiment a bit to figure out
3339your solution. After you have initially built the package, you can
3340iteratively tweak the source code, which is located in the
3341:term:`Build Directory`, and then you can
3342force a re-compile and quickly test your altered code. Once you settle
3343on a solution, you can then preserve your changes in the form of
3344patches.
3345
3346During a build, the unpacked temporary source code used by recipes to
3347build packages is available in the Build Directory as defined by the
3348:term:`S` variable. Below is the default
3349value for the ``S`` variable as defined in the
3350``meta/conf/bitbake.conf`` configuration file in the
3351:term:`Source Directory`:
3352::
3353
3354 S = "${WORKDIR}/${BP}"
3355
3356You should be aware that many recipes override the
3357``S`` variable. For example, recipes that fetch their source from Git
3358usually set ``S`` to ``${WORKDIR}/git``.
3359
3360.. note::
3361
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003362 The :term:`BP` represents the base recipe name, which consists of the name
3363 and version:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003364 ::
3365
3366 BP = "${BPN}-${PV}"
3367
3368
3369The path to the work directory for the recipe
3370(:term:`WORKDIR`) is defined as
3371follows:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003372::
3373
3374 ${TMPDIR}/work/${MULTIMACH_TARGET_SYS}/${PN}/${EXTENDPE}${PV}-${PR}
3375
3376The actual directory depends on several things:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003377
3378- :term:`TMPDIR`: The top-level build
3379 output directory.
3380
3381- :term:`MULTIMACH_TARGET_SYS`:
3382 The target system identifier.
3383
3384- :term:`PN`: The recipe name.
3385
3386- :term:`EXTENDPE`: The epoch - (if
3387 :term:`PE` is not specified, which is
3388 usually the case for most recipes, then ``EXTENDPE`` is blank).
3389
3390- :term:`PV`: The recipe version.
3391
3392- :term:`PR`: The recipe revision.
3393
3394As an example, assume a Source Directory top-level folder named
3395``poky``, a default Build Directory at ``poky/build``, and a
3396``qemux86-poky-linux`` machine target system. Furthermore, suppose your
3397recipe is named ``foo_1.3.0.bb``. In this case, the work directory the
3398build system uses to build the package would be as follows:
3399::
3400
3401 poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0
3402
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003403Using Quilt in Your Workflow
3404============================
3405
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003406`Quilt <https://savannah.nongnu.org/projects/quilt>`__ is a powerful tool
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003407that allows you to capture source code changes without having a clean
3408source tree. This section outlines the typical workflow you can use to
3409modify source code, test changes, and then preserve the changes in the
3410form of a patch all using Quilt.
3411
3412.. note::
3413
3414 With regard to preserving changes to source files, if you clean a
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003415 recipe or have ``rm_work`` enabled, the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003416 :ref:`devtool workflow <sdk-manual/extensible:using \`\`devtool\`\` in your sdk workflow>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003417 as described in the Yocto Project Application Development and the
3418 Extensible Software Development Kit (eSDK) manual is a safer
3419 development flow than the flow that uses Quilt.
3420
3421Follow these general steps:
3422
34231. *Find the Source Code:* Temporary source code used by the
3424 OpenEmbedded build system is kept in the
3425 :term:`Build Directory`. See the
3426 "`Finding Temporary Source
3427 Code <#finding-the-temporary-source-code>`__" section to learn how to
3428 locate the directory that has the temporary source code for a
3429 particular package.
3430
34312. *Change Your Working Directory:* You need to be in the directory that
3432 has the temporary source code. That directory is defined by the
3433 :term:`S` variable.
3434
34353. *Create a New Patch:* Before modifying source code, you need to
3436 create a new patch. To create a new patch file, use ``quilt new`` as
3437 below:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003438 ::
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003439
3440 $ quilt new my_changes.patch
3441
34424. *Notify Quilt and Add Files:* After creating the patch, you need to
3443 notify Quilt about the files you plan to edit. You notify Quilt by
3444 adding the files to the patch you just created:
3445 ::
3446
3447 $ quilt add file1.c file2.c file3.c
3448
34495. *Edit the Files:* Make your changes in the source code to the files
3450 you added to the patch.
3451
34526. *Test Your Changes:* Once you have modified the source code, the
3453 easiest way to test your changes is by calling the ``do_compile``
3454 task as shown in the following example:
3455 ::
3456
3457 $ bitbake -c compile -f package
3458
3459 The ``-f`` or ``--force`` option forces the specified task to
3460 execute. If you find problems with your code, you can just keep
3461 editing and re-testing iteratively until things work as expected.
3462
3463 .. note::
3464
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003465 All the modifications you make to the temporary source code disappear
3466 once you run the ``do_clean`` or ``do_cleanall`` tasks using BitBake
3467 (i.e. ``bitbake -c clean package`` and ``bitbake -c cleanall package``).
3468 Modifications will also disappear if you use the ``rm_work`` feature as
3469 described in the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003470 ":ref:`dev-manual/common-tasks:conserving disk space during builds`"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003471 section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003472
34737. *Generate the Patch:* Once your changes work as expected, you need to
3474 use Quilt to generate the final patch that contains all your
3475 modifications.
3476 ::
3477
3478 $ quilt refresh
3479
3480 At this point, the
3481 ``my_changes.patch`` file has all your edits made to the ``file1.c``,
3482 ``file2.c``, and ``file3.c`` files.
3483
3484 You can find the resulting patch file in the ``patches/``
3485 subdirectory of the source (``S``) directory.
3486
34878. *Copy the Patch File:* For simplicity, copy the patch file into a
3488 directory named ``files``, which you can create in the same directory
3489 that holds the recipe (``.bb``) file or the append (``.bbappend``)
3490 file. Placing the patch here guarantees that the OpenEmbedded build
3491 system will find the patch. Next, add the patch into the ``SRC_URI``
3492 of the recipe. Here is an example:
3493 ::
3494
3495 SRC_URI += "file://my_changes.patch"
3496
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003497Using a Development Shell
3498=========================
3499
3500When debugging certain commands or even when just editing packages,
3501``devshell`` can be a useful tool. When you invoke ``devshell``, all
3502tasks up to and including
3503:ref:`ref-tasks-patch` are run for the
3504specified target. Then, a new terminal is opened and you are placed in
3505``${``\ :term:`S`\ ``}``, the source
3506directory. In the new terminal, all the OpenEmbedded build-related
3507environment variables are still defined so you can use commands such as
3508``configure`` and ``make``. The commands execute just as if the
3509OpenEmbedded build system were executing them. Consequently, working
3510this way can be helpful when debugging a build or preparing software to
3511be used with the OpenEmbedded build system.
3512
3513Following is an example that uses ``devshell`` on a target named
3514``matchbox-desktop``:
3515::
3516
3517 $ bitbake matchbox-desktop -c devshell
3518
3519This command spawns a terminal with a shell prompt within the
3520OpenEmbedded build environment. The
3521:term:`OE_TERMINAL` variable
3522controls what type of shell is opened.
3523
3524For spawned terminals, the following occurs:
3525
3526- The ``PATH`` variable includes the cross-toolchain.
3527
3528- The ``pkgconfig`` variables find the correct ``.pc`` files.
3529
3530- The ``configure`` command finds the Yocto Project site files as well
3531 as any other necessary files.
3532
3533Within this environment, you can run configure or compile commands as if
3534they were being run by the OpenEmbedded build system itself. As noted
3535earlier, the working directory also automatically changes to the Source
3536Directory (:term:`S`).
3537
3538To manually run a specific task using ``devshell``, run the
3539corresponding ``run.*`` script in the
3540``${``\ :term:`WORKDIR`\ ``}/temp``
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003541directory (e.g., ``run.do_configure.``\ `pid`). If a task's script does
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003542not exist, which would be the case if the task was skipped by way of the
3543sstate cache, you can create the task by first running it outside of the
3544``devshell``:
3545::
3546
3547 $ bitbake -c task
3548
3549.. note::
3550
3551 - Execution of a task's ``run.*`` script and BitBake's execution of
3552 a task are identical. In other words, running the script re-runs
3553 the task just as it would be run using the ``bitbake -c`` command.
3554
3555 - Any ``run.*`` file that does not have a ``.pid`` extension is a
3556 symbolic link (symlink) to the most recent version of that file.
3557
3558Remember, that the ``devshell`` is a mechanism that allows you to get
3559into the BitBake task execution environment. And as such, all commands
3560must be called just as BitBake would call them. That means you need to
3561provide the appropriate options for cross-compilation and so forth as
3562applicable.
3563
3564When you are finished using ``devshell``, exit the shell or close the
3565terminal window.
3566
3567.. note::
3568
3569 - It is worth remembering that when using ``devshell`` you need to
3570 use the full compiler name such as ``arm-poky-linux-gnueabi-gcc``
3571 instead of just using ``gcc``. The same applies to other
3572 applications such as ``binutils``, ``libtool`` and so forth.
3573 BitBake sets up environment variables such as ``CC`` to assist
3574 applications, such as ``make`` to find the correct tools.
3575
3576 - It is also worth noting that ``devshell`` still works over X11
3577 forwarding and similar situations.
3578
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003579Using a Development Python Shell
3580================================
3581
3582Similar to working within a development shell as described in the
3583previous section, you can also spawn and work within an interactive
3584Python development shell. When debugging certain commands or even when
3585just editing packages, ``devpyshell`` can be a useful tool. When you
3586invoke ``devpyshell``, all tasks up to and including
3587:ref:`ref-tasks-patch` are run for the
3588specified target. Then a new terminal is opened. Additionally, key
3589Python objects and code are available in the same way they are to
3590BitBake tasks, in particular, the data store 'd'. So, commands such as
3591the following are useful when exploring the data store and running
3592functions:
3593::
3594
3595 pydevshell> d.getVar("STAGING_DIR")
3596 '/media/build1/poky/build/tmp/sysroots'
3597 pydevshell> d.getVar("STAGING_DIR")
3598 '${TMPDIR}/sysroots'
3599 pydevshell> d.setVar("FOO", "bar")
3600 pydevshell> d.getVar("FOO")
3601 'bar'
3602 pydevshell> d.delVar("FOO")
3603 pydevshell> d.getVar("FOO")
3604 pydevshell> bb.build.exec_func("do_unpack", d)
3605 pydevshell>
3606
3607The commands execute just as if the OpenEmbedded build
3608system were executing them. Consequently, working this way can be
3609helpful when debugging a build or preparing software to be used with the
3610OpenEmbedded build system.
3611
3612Following is an example that uses ``devpyshell`` on a target named
3613``matchbox-desktop``:
3614::
3615
3616 $ bitbake matchbox-desktop -c devpyshell
3617
3618This command spawns a terminal and places you in an interactive Python
3619interpreter within the OpenEmbedded build environment. The
3620:term:`OE_TERMINAL` variable
3621controls what type of shell is opened.
3622
3623When you are finished using ``devpyshell``, you can exit the shell
3624either by using Ctrl+d or closing the terminal window.
3625
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003626Building
3627========
3628
3629This section describes various build procedures. For example, the steps
3630needed for a simple build, a target that uses multiple configurations,
3631building an image for more than one machine, and so forth.
3632
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003633Building a Simple Image
3634-----------------------
3635
3636In the development environment, you need to build an image whenever you
3637change hardware support, add or change system libraries, or add or
3638change services that have dependencies. Several methods exist that allow
3639you to build an image within the Yocto Project. This section presents
3640the basic steps you need to build a simple image using BitBake from a
3641build host running Linux.
3642
3643.. note::
3644
3645 - For information on how to build an image using
3646 :term:`Toaster`, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003647 :doc:`/toaster-manual/index`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003648
3649 - For information on how to use ``devtool`` to build images, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003650 ":ref:`sdk-manual/extensible:using \`\`devtool\`\` in your sdk workflow`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003651 section in the Yocto Project Application Development and the
3652 Extensible Software Development Kit (eSDK) manual.
3653
3654 - For a quick example on how to build an image using the
3655 OpenEmbedded build system, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003656 :doc:`/brief-yoctoprojectqs/index` document.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003657
3658The build process creates an entire Linux distribution from source and
3659places it in your :term:`Build Directory` under
3660``tmp/deploy/images``. For detailed information on the build process
Andrew Geissler09209ee2020-12-13 08:44:15 -06003661using BitBake, see the ":ref:`overview-manual/concepts:images`" section in the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003662Yocto Project Overview and Concepts Manual.
3663
3664The following figure and list overviews the build process:
3665
3666.. image:: figures/bitbake-build-flow.png
3667 :align: center
3668
36691. *Set up Your Host Development System to Support Development Using the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003670 Yocto Project*: See the ":doc:`start`" section for options on how to get a
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003671 build host ready to use the Yocto Project.
3672
36732. *Initialize the Build Environment:* Initialize the build environment
3674 by sourcing the build environment script (i.e.
3675 :ref:`structure-core-script`):
3676 ::
3677
3678 $ source oe-init-build-env [build_dir]
3679
3680 When you use the initialization script, the OpenEmbedded build system
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003681 uses ``build`` as the default :term:`Build Directory` in your current work
3682 directory. You can use a `build_dir` argument with the script to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003683 specify a different build directory.
3684
3685 .. note::
3686
3687 A common practice is to use a different Build Directory for
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003688 different targets. For example, ``~/build/x86`` for a ``qemux86``
3689 target, and ``~/build/arm`` for a ``qemuarm`` target.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003690
Andrew Geissler4c19ea12020-10-27 13:52:24 -050036913. *Make Sure Your* ``local.conf`` *File is Correct*: Ensure the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003692 ``conf/local.conf`` configuration file, which is found in the Build
3693 Directory, is set up how you want it. This file defines many aspects
3694 of the build environment including the target machine architecture
3695 through the ``MACHINE`` variable, the packaging format used during
3696 the build
3697 (:term:`PACKAGE_CLASSES`),
3698 and a centralized tarball download directory through the
3699 :term:`DL_DIR` variable.
3700
37014. *Build the Image:* Build the image using the ``bitbake`` command:
3702 ::
3703
3704 $ bitbake target
3705
3706 .. note::
3707
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003708 For information on BitBake, see the :doc:`bitbake:index`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003709
3710 The target is the name of the recipe you want to build. Common
3711 targets are the images in ``meta/recipes-core/images``,
3712 ``meta/recipes-sato/images``, and so forth all found in the
3713 :term:`Source Directory`. Or, the target
3714 can be the name of a recipe for a specific piece of software such as
3715 BusyBox. For more details about the images the OpenEmbedded build
3716 system supports, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003717 ":ref:`ref-manual/images:Images`" chapter in the Yocto
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003718 Project Reference Manual.
3719
3720 As an example, the following command builds the
3721 ``core-image-minimal`` image:
3722 ::
3723
3724 $ bitbake core-image-minimal
3725
3726 Once an
3727 image has been built, it often needs to be installed. The images and
3728 kernels built by the OpenEmbedded build system are placed in the
3729 Build Directory in ``tmp/deploy/images``. For information on how to
3730 run pre-built images such as ``qemux86`` and ``qemuarm``, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06003731 :doc:`/sdk-manual/index` manual. For
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003732 information about how to install these images, see the documentation
3733 for your particular board or machine.
3734
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003735Building Images for Multiple Targets Using Multiple Configurations
3736------------------------------------------------------------------
3737
3738You can use a single ``bitbake`` command to build multiple images or
3739packages for different targets where each image or package requires a
3740different configuration (multiple configuration builds). The builds, in
3741this scenario, are sometimes referred to as "multiconfigs", and this
3742section uses that term throughout.
3743
3744This section describes how to set up for multiple configuration builds
3745and how to account for cross-build dependencies between the
3746multiconfigs.
3747
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003748Setting Up and Running a Multiple Configuration Build
3749~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3750
3751To accomplish a multiple configuration build, you must define each
3752target's configuration separately using a parallel configuration file in
3753the :term:`Build Directory`, and you
3754must follow a required file hierarchy. Additionally, you must enable the
3755multiple configuration builds in your ``local.conf`` file.
3756
3757Follow these steps to set up and execute multiple configuration builds:
3758
3759- *Create Separate Configuration Files*: You need to create a single
3760 configuration file for each build target (each multiconfig).
3761 Minimally, each configuration file must define the machine and the
3762 temporary directory BitBake uses for the build. Suggested practice
3763 dictates that you do not overlap the temporary directories used
3764 during the builds. However, it is possible that you can share the
3765 temporary directory
3766 (:term:`TMPDIR`). For example,
3767 consider a scenario with two different multiconfigs for the same
3768 :term:`MACHINE`: "qemux86" built
3769 for two distributions such as "poky" and "poky-lsb". In this case,
3770 you might want to use the same ``TMPDIR``.
3771
3772 Here is an example showing the minimal statements needed in a
3773 configuration file for a "qemux86" target whose temporary build
3774 directory is ``tmpmultix86``:
3775 ::
3776
3777 MACHINE = "qemux86"
3778 TMPDIR = "${TOPDIR}/tmpmultix86"
3779
3780 The location for these multiconfig configuration files is specific.
3781 They must reside in the current build directory in a sub-directory of
3782 ``conf`` named ``multiconfig``. Following is an example that defines
3783 two configuration files for the "x86" and "arm" multiconfigs:
3784
3785 .. image:: figures/multiconfig_files.png
3786 :align: center
3787
3788 The reason for this required file hierarchy is because the ``BBPATH``
3789 variable is not constructed until the layers are parsed.
3790 Consequently, using the configuration file as a pre-configuration
3791 file is not possible unless it is located in the current working
3792 directory.
3793
3794- *Add the BitBake Multi-configuration Variable to the Local
3795 Configuration File*: Use the
3796 :term:`BBMULTICONFIG`
3797 variable in your ``conf/local.conf`` configuration file to specify
3798 each multiconfig. Continuing with the example from the previous
3799 figure, the ``BBMULTICONFIG`` variable needs to enable two
3800 multiconfigs: "x86" and "arm" by specifying each configuration file:
3801 ::
3802
3803 BBMULTICONFIG = "x86 arm"
3804
3805 .. note::
3806
3807 A "default" configuration already exists by definition. This
3808 configuration is named: "" (i.e. empty string) and is defined by
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003809 the variables coming from your ``local.conf``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003810 file. Consequently, the previous example actually adds two
3811 additional configurations to your build: "arm" and "x86" along
3812 with "".
3813
3814- *Launch BitBake*: Use the following BitBake command form to launch
3815 the multiple configuration build:
3816 ::
3817
3818 $ bitbake [mc:multiconfigname:]target [[[mc:multiconfigname:]target] ... ]
3819
3820 For the example in this section, the following command applies:
3821 ::
3822
3823 $ bitbake mc:x86:core-image-minimal mc:arm:core-image-sato mc::core-image-base
3824
3825 The previous BitBake command builds a ``core-image-minimal`` image
3826 that is configured through the ``x86.conf`` configuration file, a
3827 ``core-image-sato`` image that is configured through the ``arm.conf``
3828 configuration file and a ``core-image-base`` that is configured
3829 through your ``local.conf`` configuration file.
3830
3831.. note::
3832
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003833 Support for multiple configuration builds in the Yocto Project &DISTRO;
3834 (&DISTRO_NAME;) Release does not include Shared State (sstate)
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003835 optimizations. Consequently, if a build uses the same object twice
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003836 in, for example, two different ``TMPDIR``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003837 directories, the build either loads from an existing sstate cache for
3838 that build at the start or builds the object fresh.
3839
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003840Enabling Multiple Configuration Build Dependencies
3841~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3842
3843Sometimes dependencies can exist between targets (multiconfigs) in a
3844multiple configuration build. For example, suppose that in order to
3845build a ``core-image-sato`` image for an "x86" multiconfig, the root
3846filesystem of an "arm" multiconfig must exist. This dependency is
3847essentially that the
3848:ref:`ref-tasks-image` task in the
3849``core-image-sato`` recipe depends on the completion of the
3850:ref:`ref-tasks-rootfs` task of the
3851``core-image-minimal`` recipe.
3852
3853To enable dependencies in a multiple configuration build, you must
3854declare the dependencies in the recipe using the following statement
3855form:
3856::
3857
3858 task_or_package[mcdepends] = "mc:from_multiconfig:to_multiconfig:recipe_name:task_on_which_to_depend"
3859
3860To better show how to use this statement, consider the example scenario
3861from the first paragraph of this section. The following statement needs
3862to be added to the recipe that builds the ``core-image-sato`` image:
3863::
3864
3865 do_image[mcdepends] = "mc:x86:arm:core-image-minimal:do_rootfs"
3866
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003867In this example, the `from_multiconfig` is "x86". The `to_multiconfig` is "arm". The
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003868task on which the ``do_image`` task in the recipe depends is the
3869``do_rootfs`` task from the ``core-image-minimal`` recipe associated
3870with the "arm" multiconfig.
3871
3872Once you set up this dependency, you can build the "x86" multiconfig
3873using a BitBake command as follows:
3874::
3875
3876 $ bitbake mc:x86:core-image-sato
3877
3878This command executes all the tasks needed to create the
3879``core-image-sato`` image for the "x86" multiconfig. Because of the
3880dependency, BitBake also executes through the ``do_rootfs`` task for the
3881"arm" multiconfig build.
3882
3883Having a recipe depend on the root filesystem of another build might not
3884seem that useful. Consider this change to the statement in the
3885``core-image-sato`` recipe:
3886::
3887
3888 do_image[mcdepends] = "mc:x86:arm:core-image-minimal:do_image"
3889
3890In this case, BitBake must
3891create the ``core-image-minimal`` image for the "arm" build since the
3892"x86" build depends on it.
3893
3894Because "x86" and "arm" are enabled for multiple configuration builds
3895and have separate configuration files, BitBake places the artifacts for
3896each build in the respective temporary build directories (i.e.
3897:term:`TMPDIR`).
3898
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003899Building an Initial RAM Filesystem (initramfs) Image
3900----------------------------------------------------
3901
3902An initial RAM filesystem (initramfs) image provides a temporary root
3903filesystem used for early system initialization (e.g. loading of modules
3904needed to locate and mount the "real" root filesystem).
3905
3906.. note::
3907
3908 The initramfs image is the successor of initial RAM disk (initrd). It
3909 is a "copy in and out" (cpio) archive of the initial filesystem that
3910 gets loaded into memory during the Linux startup process. Because
3911 Linux uses the contents of the archive during initialization, the
3912 initramfs image needs to contain all of the device drivers and tools
3913 needed to mount the final root filesystem.
3914
3915Follow these steps to create an initramfs image:
3916
39171. *Create the initramfs Image Recipe:* You can reference the
3918 ``core-image-minimal-initramfs.bb`` recipe found in the
3919 ``meta/recipes-core`` directory of the :term:`Source Directory`
3920 as an example
3921 from which to work.
3922
39232. *Decide if You Need to Bundle the initramfs Image Into the Kernel
3924 Image:* If you want the initramfs image that is built to be bundled
3925 in with the kernel image, set the
3926 :term:`INITRAMFS_IMAGE_BUNDLE`
3927 variable to "1" in your ``local.conf`` configuration file and set the
3928 :term:`INITRAMFS_IMAGE`
3929 variable in the recipe that builds the kernel image.
3930
3931 .. note::
3932
3933 It is recommended that you do bundle the initramfs image with the
3934 kernel image to avoid circular dependencies between the kernel
3935 recipe and the initramfs recipe should the initramfs image include
3936 kernel modules.
3937
3938 Setting the ``INITRAMFS_IMAGE_BUNDLE`` flag causes the initramfs
3939 image to be unpacked into the ``${B}/usr/`` directory. The unpacked
3940 initramfs image is then passed to the kernel's ``Makefile`` using the
3941 :term:`CONFIG_INITRAMFS_SOURCE`
3942 variable, allowing the initramfs image to be built into the kernel
3943 normally.
3944
3945 .. note::
3946
3947 If you choose to not bundle the initramfs image with the kernel
3948 image, you are essentially using an
Andrew Geissler4c19ea12020-10-27 13:52:24 -05003949 `Initial RAM Disk (initrd) <https://en.wikipedia.org/wiki/Initrd>`__.
3950 Creating an initrd is handled primarily through the :term:`INITRD_IMAGE`,
3951 ``INITRD_LIVE``, and ``INITRD_IMAGE_LIVE`` variables. For more
3952 information, see the :ref:`ref-classes-image-live` file.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003953
39543. *Optionally Add Items to the initramfs Image Through the initramfs
3955 Image Recipe:* If you add items to the initramfs image by way of its
3956 recipe, you should use
3957 :term:`PACKAGE_INSTALL`
3958 rather than
3959 :term:`IMAGE_INSTALL`.
3960 ``PACKAGE_INSTALL`` gives more direct control of what is added to the
3961 image as compared to the defaults you might not necessarily want that
3962 are set by the :ref:`image <ref-classes-image>`
3963 or :ref:`core-image <ref-classes-core-image>`
3964 classes.
3965
39664. *Build the Kernel Image and the initramfs Image:* Build your kernel
3967 image using BitBake. Because the initramfs image recipe is a
3968 dependency of the kernel image, the initramfs image is built as well
3969 and bundled with the kernel image if you used the
3970 :term:`INITRAMFS_IMAGE_BUNDLE`
3971 variable described earlier.
3972
3973Building a Tiny System
3974----------------------
3975
3976Very small distributions have some significant advantages such as
3977requiring less on-die or in-package memory (cheaper), better performance
3978through efficient cache usage, lower power requirements due to less
3979memory, faster boot times, and reduced development overhead. Some
3980real-world examples where a very small distribution gives you distinct
3981advantages are digital cameras, medical devices, and small headless
3982systems.
3983
3984This section presents information that shows you how you can trim your
3985distribution to even smaller sizes than the ``poky-tiny`` distribution,
3986which is around 5 Mbytes, that can be built out-of-the-box using the
3987Yocto Project.
3988
Andrew Geisslerc9f78652020-09-18 14:11:35 -05003989Tiny System Overview
3990~~~~~~~~~~~~~~~~~~~~
3991
3992The following list presents the overall steps you need to consider and
3993perform to create distributions with smaller root filesystems, achieve
3994faster boot times, maintain your critical functionality, and avoid
3995initial RAM disks:
3996
3997- `Determine your goals and guiding
3998 principles. <#goals-and-guiding-principles>`__
3999
4000- `Understand what contributes to your image
4001 size. <#understand-what-gives-your-image-size>`__
4002
4003- `Reduce the size of the root
4004 filesystem. <#trim-the-root-filesystem>`__
4005
4006- `Reduce the size of the kernel. <#trim-the-kernel>`__
4007
4008- `Eliminate packaging
4009 requirements. <#remove-package-management-requirements>`__
4010
4011- `Look for other ways to minimize
4012 size. <#look-for-other-ways-to-minimize-size>`__
4013
4014- `Iterate on the process. <#iterate-on-the-process>`__
4015
4016Goals and Guiding Principles
4017~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4018
4019Before you can reach your destination, you need to know where you are
4020going. Here is an example list that you can use as a guide when creating
4021very small distributions:
4022
4023- Determine how much space you need (e.g. a kernel that is 1 Mbyte or
4024 less and a root filesystem that is 3 Mbytes or less).
4025
4026- Find the areas that are currently taking 90% of the space and
4027 concentrate on reducing those areas.
4028
4029- Do not create any difficult "hacks" to achieve your goals.
4030
4031- Leverage the device-specific options.
4032
4033- Work in a separate layer so that you keep changes isolated. For
4034 information on how to create layers, see the "`Understanding and
4035 Creating Layers <#understanding-and-creating-layers>`__" section.
4036
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004037Understand What Contributes to Your Image Size
4038~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4039
4040It is easiest to have something to start with when creating your own
4041distribution. You can use the Yocto Project out-of-the-box to create the
4042``poky-tiny`` distribution. Ultimately, you will want to make changes in
4043your own distribution that are likely modeled after ``poky-tiny``.
4044
4045.. note::
4046
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004047 To use ``poky-tiny`` in your build, set the ``DISTRO`` variable in your
4048 ``local.conf`` file to "poky-tiny" as described in the
Andrew Geissler09209ee2020-12-13 08:44:15 -06004049 ":ref:`dev-manual/common-tasks:creating your own distribution`"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004050 section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004051
4052Understanding some memory concepts will help you reduce the system size.
4053Memory consists of static, dynamic, and temporary memory. Static memory
4054is the TEXT (code), DATA (initialized data in the code), and BSS
4055(uninitialized data) sections. Dynamic memory represents memory that is
4056allocated at runtime: stacks, hash tables, and so forth. Temporary
4057memory is recovered after the boot process. This memory consists of
4058memory used for decompressing the kernel and for the ``__init__``
4059functions.
4060
4061To help you see where you currently are with kernel and root filesystem
4062sizes, you can use two tools found in the :term:`Source Directory`
4063in the
4064``scripts/tiny/`` directory:
4065
4066- ``ksize.py``: Reports component sizes for the kernel build objects.
4067
4068- ``dirsize.py``: Reports component sizes for the root filesystem.
4069
4070This next tool and command help you organize configuration fragments and
4071view file dependencies in a human-readable form:
4072
4073- ``merge_config.sh``: Helps you manage configuration files and
4074 fragments within the kernel. With this tool, you can merge individual
4075 configuration fragments together. The tool allows you to make
4076 overrides and warns you of any missing configuration options. The
4077 tool is ideal for allowing you to iterate on configurations, create
4078 minimal configurations, and create configuration files for different
4079 machines without having to duplicate your process.
4080
4081 The ``merge_config.sh`` script is part of the Linux Yocto kernel Git
4082 repositories (i.e. ``linux-yocto-3.14``, ``linux-yocto-3.10``,
4083 ``linux-yocto-3.8``, and so forth) in the ``scripts/kconfig``
4084 directory.
4085
4086 For more information on configuration fragments, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06004087 ":ref:`kernel-dev/common:creating configuration fragments`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004088 section in the Yocto Project Linux Kernel Development Manual.
4089
4090- ``bitbake -u taskexp -g bitbake_target``: Using the BitBake command
4091 with these options brings up a Dependency Explorer from which you can
4092 view file dependencies. Understanding these dependencies allows you
4093 to make informed decisions when cutting out various pieces of the
4094 kernel and root filesystem.
4095
4096Trim the Root Filesystem
4097~~~~~~~~~~~~~~~~~~~~~~~~
4098
4099The root filesystem is made up of packages for booting, libraries, and
4100applications. To change things, you can configure how the packaging
4101happens, which changes the way you build them. You can also modify the
4102filesystem itself or select a different filesystem.
4103
4104First, find out what is hogging your root filesystem by running the
4105``dirsize.py`` script from your root directory:
4106::
4107
4108 $ cd root-directory-of-image
4109 $ dirsize.py 100000 > dirsize-100k.log
4110 $ cat dirsize-100k.log
4111
4112You can apply a filter to the script to ignore files
4113under a certain size. The previous example filters out any files below
4114100 Kbytes. The sizes reported by the tool are uncompressed, and thus
4115will be smaller by a relatively constant factor in a compressed root
4116filesystem. When you examine your log file, you can focus on areas of
4117the root filesystem that take up large amounts of memory.
4118
4119You need to be sure that what you eliminate does not cripple the
4120functionality you need. One way to see how packages relate to each other
4121is by using the Dependency Explorer UI with the BitBake command:
4122::
4123
4124 $ cd image-directory
4125 $ bitbake -u taskexp -g image
4126
4127Use the interface to
4128select potential packages you wish to eliminate and see their dependency
4129relationships.
4130
4131When deciding how to reduce the size, get rid of packages that result in
4132minimal impact on the feature set. For example, you might not need a VGA
4133display. Or, you might be able to get by with ``devtmpfs`` and ``mdev``
4134instead of ``udev``.
4135
4136Use your ``local.conf`` file to make changes. For example, to eliminate
4137``udev`` and ``glib``, set the following in the local configuration
4138file:
4139::
4140
4141 VIRTUAL-RUNTIME_dev_manager = ""
4142
4143Finally, you should consider exactly the type of root filesystem you
4144need to meet your needs while also reducing its size. For example,
4145consider ``cramfs``, ``squashfs``, ``ubifs``, ``ext2``, or an
4146``initramfs`` using ``initramfs``. Be aware that ``ext3`` requires a 1
4147Mbyte journal. If you are okay with running read-only, you do not need
4148this journal.
4149
4150.. note::
4151
4152 After each round of elimination, you need to rebuild your system and
4153 then use the tools to see the effects of your reductions.
4154
4155Trim the Kernel
4156~~~~~~~~~~~~~~~
4157
4158The kernel is built by including policies for hardware-independent
4159aspects. What subsystems do you enable? For what architecture are you
4160building? Which drivers do you build by default?
4161
4162.. note::
4163
4164 You can modify the kernel source if you want to help with boot time.
4165
4166Run the ``ksize.py`` script from the top-level Linux build directory to
4167get an idea of what is making up the kernel:
4168::
4169
4170 $ cd top-level-linux-build-directory
4171 $ ksize.py > ksize.log
4172 $ cat ksize.log
4173
4174When you examine the log, you will see how much space is taken up with
4175the built-in ``.o`` files for drivers, networking, core kernel files,
4176filesystem, sound, and so forth. The sizes reported by the tool are
4177uncompressed, and thus will be smaller by a relatively constant factor
4178in a compressed kernel image. Look to reduce the areas that are large
4179and taking up around the "90% rule."
4180
4181To examine, or drill down, into any particular area, use the ``-d``
4182option with the script:
4183::
4184
4185 $ ksize.py -d > ksize.log
4186
4187Using this option
4188breaks out the individual file information for each area of the kernel
4189(e.g. drivers, networking, and so forth).
4190
4191Use your log file to see what you can eliminate from the kernel based on
4192features you can let go. For example, if you are not going to need
4193sound, you do not need any drivers that support sound.
4194
4195After figuring out what to eliminate, you need to reconfigure the kernel
4196to reflect those changes during the next build. You could run
4197``menuconfig`` and make all your changes at once. However, that makes it
4198difficult to see the effects of your individual eliminations and also
4199makes it difficult to replicate the changes for perhaps another target
4200device. A better method is to start with no configurations using
4201``allnoconfig``, create configuration fragments for individual changes,
4202and then manage the fragments into a single configuration file using
4203``merge_config.sh``. The tool makes it easy for you to iterate using the
4204configuration change and build cycle.
4205
4206Each time you make configuration changes, you need to rebuild the kernel
4207and check to see what impact your changes had on the overall size.
4208
4209Remove Package Management Requirements
4210~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4211
4212Packaging requirements add size to the image. One way to reduce the size
4213of the image is to remove all the packaging requirements from the image.
4214This reduction includes both removing the package manager and its unique
4215dependencies as well as removing the package management data itself.
4216
4217To eliminate all the packaging requirements for an image, be sure that
4218"package-management" is not part of your
4219:term:`IMAGE_FEATURES`
4220statement for the image. When you remove this feature, you are removing
4221the package manager as well as its dependencies from the root
4222filesystem.
4223
4224Look for Other Ways to Minimize Size
4225~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4226
4227Depending on your particular circumstances, other areas that you can
4228trim likely exist. The key to finding these areas is through tools and
4229methods described here combined with experimentation and iteration. Here
4230are a couple of areas to experiment with:
4231
4232- ``glibc``: In general, follow this process:
4233
4234 1. Remove ``glibc`` features from
4235 :term:`DISTRO_FEATURES`
4236 that you think you do not need.
4237
4238 2. Build your distribution.
4239
4240 3. If the build fails due to missing symbols in a package, determine
4241 if you can reconfigure the package to not need those features. For
4242 example, change the configuration to not support wide character
4243 support as is done for ``ncurses``. Or, if support for those
4244 characters is needed, determine what ``glibc`` features provide
4245 the support and restore the configuration.
4246
4247 4. Rebuild and repeat the process.
4248
4249- ``busybox``: For BusyBox, use a process similar as described for
4250 ``glibc``. A difference is you will need to boot the resulting system
4251 to see if you are able to do everything you expect from the running
4252 system. You need to be sure to integrate configuration fragments into
4253 Busybox because BusyBox handles its own core features and then allows
4254 you to add configuration fragments on top.
4255
4256Iterate on the Process
4257~~~~~~~~~~~~~~~~~~~~~~
4258
4259If you have not reached your goals on system size, you need to iterate
4260on the process. The process is the same. Use the tools and see just what
4261is taking up 90% of the root filesystem and the kernel. Decide what you
4262can eliminate without limiting your device beyond what you need.
4263
4264Depending on your system, a good place to look might be Busybox, which
4265provides a stripped down version of Unix tools in a single, executable
4266file. You might be able to drop virtual terminal services or perhaps
4267ipv6.
4268
4269Building Images for More than One Machine
4270-----------------------------------------
4271
4272A common scenario developers face is creating images for several
4273different machines that use the same software environment. In this
4274situation, it is tempting to set the tunings and optimization flags for
4275each build specifically for the targeted hardware (i.e. "maxing out" the
4276tunings). Doing so can considerably add to build times and package feed
4277maintenance collectively for the machines. For example, selecting tunes
4278that are extremely specific to a CPU core used in a system might enable
4279some micro optimizations in GCC for that particular system but would
4280otherwise not gain you much of a performance difference across the other
4281systems as compared to using a more general tuning across all the builds
4282(e.g. setting :term:`DEFAULTTUNE`
4283specifically for each machine's build). Rather than "max out" each
4284build's tunings, you can take steps that cause the OpenEmbedded build
4285system to reuse software across the various machines where it makes
4286sense.
4287
4288If build speed and package feed maintenance are considerations, you
4289should consider the points in this section that can help you optimize
4290your tunings to best consider build times and package feed maintenance.
4291
4292- *Share the Build Directory:* If at all possible, share the
4293 :term:`TMPDIR` across builds. The
4294 Yocto Project supports switching between different
4295 :term:`MACHINE` values in the same
4296 ``TMPDIR``. This practice is well supported and regularly used by
4297 developers when building for multiple machines. When you use the same
4298 ``TMPDIR`` for multiple machine builds, the OpenEmbedded build system
4299 can reuse the existing native and often cross-recipes for multiple
4300 machines. Thus, build time decreases.
4301
4302 .. note::
4303
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004304 If :term:`DISTRO` settings change or fundamental configuration settings
4305 such as the filesystem layout, you need to work with a clean ``TMPDIR``.
4306 Sharing ``TMPDIR`` under these circumstances might work but since it is
4307 not guaranteed, you should use a clean ``TMPDIR``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004308
4309- *Enable the Appropriate Package Architecture:* By default, the
4310 OpenEmbedded build system enables three levels of package
4311 architectures: "all", "tune" or "package", and "machine". Any given
4312 recipe usually selects one of these package architectures (types) for
4313 its output. Depending for what a given recipe creates packages,
4314 making sure you enable the appropriate package architecture can
4315 directly impact the build time.
4316
4317 A recipe that just generates scripts can enable "all" architecture
4318 because there are no binaries to build. To specifically enable "all"
4319 architecture, be sure your recipe inherits the
4320 :ref:`allarch <ref-classes-allarch>` class.
4321 This class is useful for "all" architectures because it configures
4322 many variables so packages can be used across multiple architectures.
4323
4324 If your recipe needs to generate packages that are machine-specific
4325 or when one of the build or runtime dependencies is already
4326 machine-architecture dependent, which makes your recipe also
4327 machine-architecture dependent, make sure your recipe enables the
4328 "machine" package architecture through the
4329 :term:`MACHINE_ARCH`
4330 variable:
4331 ::
4332
4333 PACKAGE_ARCH = "${MACHINE_ARCH}"
4334
4335 When you do not
4336 specifically enable a package architecture through the
4337 :term:`PACKAGE_ARCH`, The
4338 OpenEmbedded build system defaults to the
4339 :term:`TUNE_PKGARCH` setting:
4340 ::
4341
4342 PACKAGE_ARCH = "${TUNE_PKGARCH}"
4343
4344- *Choose a Generic Tuning File if Possible:* Some tunes are more
4345 generic and can run on multiple targets (e.g. an ``armv5`` set of
4346 packages could run on ``armv6`` and ``armv7`` processors in most
4347 cases). Similarly, ``i486`` binaries could work on ``i586`` and
4348 higher processors. You should realize, however, that advances on
4349 newer processor versions would not be used.
4350
4351 If you select the same tune for several different machines, the
4352 OpenEmbedded build system reuses software previously built, thus
4353 speeding up the overall build time. Realize that even though a new
4354 sysroot for each machine is generated, the software is not recompiled
4355 and only one package feed exists.
4356
4357- *Manage Granular Level Packaging:* Sometimes cases exist where
4358 injecting another level of package architecture beyond the three
4359 higher levels noted earlier can be useful. For example, consider how
4360 NXP (formerly Freescale) allows for the easy reuse of binary packages
4361 in their layer
Andrew Geissler09209ee2020-12-13 08:44:15 -06004362 :yocto_git:`meta-freescale </meta-freescale/>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004363 In this example, the
Andrew Geissler09209ee2020-12-13 08:44:15 -06004364 :yocto_git:`fsl-dynamic-packagearch </meta-freescale/tree/classes/fsl-dynamic-packagearch.bbclass>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004365 class shares GPU packages for i.MX53 boards because all boards share
4366 the AMD GPU. The i.MX6-based boards can do the same because all
4367 boards share the Vivante GPU. This class inspects the BitBake
4368 datastore to identify if the package provides or depends on one of
4369 the sub-architecture values. If so, the class sets the
4370 :term:`PACKAGE_ARCH` value
4371 based on the ``MACHINE_SUBARCH`` value. If the package does not
4372 provide or depend on one of the sub-architecture values but it
4373 matches a value in the machine-specific filter, it sets
4374 :term:`MACHINE_ARCH`. This
4375 behavior reduces the number of packages built and saves build time by
4376 reusing binaries.
4377
4378- *Use Tools to Debug Issues:* Sometimes you can run into situations
4379 where software is being rebuilt when you think it should not be. For
4380 example, the OpenEmbedded build system might not be using shared
4381 state between machines when you think it should be. These types of
4382 situations are usually due to references to machine-specific
4383 variables such as :term:`MACHINE`,
4384 :term:`SERIAL_CONSOLES`,
4385 :term:`XSERVER`,
4386 :term:`MACHINE_FEATURES`,
4387 and so forth in code that is supposed to only be tune-specific or
4388 when the recipe depends
4389 (:term:`DEPENDS`,
4390 :term:`RDEPENDS`,
4391 :term:`RRECOMMENDS`,
4392 :term:`RSUGGESTS`, and so forth)
4393 on some other recipe that already has
4394 :term:`PACKAGE_ARCH` defined
4395 as "${MACHINE_ARCH}".
4396
4397 .. note::
4398
4399 Patches to fix any issues identified are most welcome as these
4400 issues occasionally do occur.
4401
4402 For such cases, you can use some tools to help you sort out the
4403 situation:
4404
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004405 - ``state-diff-machines.sh``*:* You can find this tool in the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004406 ``scripts`` directory of the Source Repositories. See the comments
4407 in the script for information on how to use the tool.
4408
4409 - *BitBake's "-S printdiff" Option:* Using this option causes
4410 BitBake to try to establish the closest signature match it can
4411 (e.g. in the shared state cache) and then run ``bitbake-diffsigs``
4412 over the matches to determine the stamps and delta where these two
4413 stamp trees diverge.
4414
4415Building Software from an External Source
4416-----------------------------------------
4417
4418By default, the OpenEmbedded build system uses the
4419:term:`Build Directory` when building source
4420code. The build process involves fetching the source files, unpacking
4421them, and then patching them if necessary before the build takes place.
4422
4423Situations exist where you might want to build software from source
4424files that are external to and thus outside of the OpenEmbedded build
4425system. For example, suppose you have a project that includes a new BSP
4426with a heavily customized kernel. And, you want to minimize exposing the
4427build system to the development team so that they can focus on their
4428project and maintain everyone's workflow as much as possible. In this
4429case, you want a kernel source directory on the development machine
4430where the development occurs. You want the recipe's
4431:term:`SRC_URI` variable to point to
4432the external directory and use it as is, not copy it.
4433
4434To build from software that comes from an external source, all you need
4435to do is inherit the
4436:ref:`externalsrc <ref-classes-externalsrc>` class
4437and then set the
4438:term:`EXTERNALSRC` variable to
4439point to your external source code. Here are the statements to put in
4440your ``local.conf`` file:
4441::
4442
4443 INHERIT += "externalsrc"
4444 EXTERNALSRC_pn-myrecipe = "path-to-your-source-tree"
4445
4446This next example shows how to accomplish the same thing by setting
4447``EXTERNALSRC`` in the recipe itself or in the recipe's append file:
4448::
4449
4450 EXTERNALSRC = "path"
4451 EXTERNALSRC_BUILD = "path"
4452
4453.. note::
4454
4455 In order for these settings to take effect, you must globally or
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004456 locally inherit the :ref:`externalsrc <ref-classes-externalsrc>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004457 class.
4458
4459By default, ``externalsrc.bbclass`` builds the source code in a
4460directory separate from the external source directory as specified by
4461:term:`EXTERNALSRC`. If you need
4462to have the source built in the same directory in which it resides, or
4463some other nominated directory, you can set
4464:term:`EXTERNALSRC_BUILD`
4465to point to that directory:
4466::
4467
4468 EXTERNALSRC_BUILD_pn-myrecipe = "path-to-your-source-tree"
4469
4470Replicating a Build Offline
4471---------------------------
4472
4473It can be useful to take a "snapshot" of upstream sources used in a
4474build and then use that "snapshot" later to replicate the build offline.
4475To do so, you need to first prepare and populate your downloads
4476directory your "snapshot" of files. Once your downloads directory is
4477ready, you can use it at any time and from any machine to replicate your
4478build.
4479
4480Follow these steps to populate your Downloads directory:
4481
44821. *Create a Clean Downloads Directory:* Start with an empty downloads
4483 directory (:term:`DL_DIR`). You
4484 start with an empty downloads directory by either removing the files
4485 in the existing directory or by setting ``DL_DIR`` to point to either
4486 an empty location or one that does not yet exist.
4487
44882. *Generate Tarballs of the Source Git Repositories:* Edit your
4489 ``local.conf`` configuration file as follows:
4490 ::
4491
4492 DL_DIR = "/home/your-download-dir/"
4493 BB_GENERATE_MIRROR_TARBALLS = "1"
4494
4495 During
4496 the fetch process in the next step, BitBake gathers the source files
4497 and creates tarballs in the directory pointed to by ``DL_DIR``. See
4498 the
4499 :term:`BB_GENERATE_MIRROR_TARBALLS`
4500 variable for more information.
4501
45023. *Populate Your Downloads Directory Without Building:* Use BitBake to
4503 fetch your sources but inhibit the build:
4504 ::
4505
4506 $ bitbake target --runonly=fetch
4507
4508 The downloads directory (i.e. ``${DL_DIR}``) now has
4509 a "snapshot" of the source files in the form of tarballs, which can
4510 be used for the build.
4511
45124. *Optionally Remove Any Git or other SCM Subdirectories From the
4513 Downloads Directory:* If you want, you can clean up your downloads
4514 directory by removing any Git or other Source Control Management
4515 (SCM) subdirectories such as ``${DL_DIR}/git2/*``. The tarballs
4516 already contain these subdirectories.
4517
4518Once your downloads directory has everything it needs regarding source
4519files, you can create your "own-mirror" and build your target.
4520Understand that you can use the files to build the target offline from
4521any machine and at any time.
4522
4523Follow these steps to build your target using the files in the downloads
4524directory:
4525
45261. *Using Local Files Only:* Inside your ``local.conf`` file, add the
4527 :term:`SOURCE_MIRROR_URL`
4528 variable, inherit the
4529 :ref:`own-mirrors <ref-classes-own-mirrors>`
4530 class, and use the
4531 :term:`bitbake:BB_NO_NETWORK`
4532 variable to your ``local.conf``.
4533 ::
4534
4535 SOURCE_MIRROR_URL ?= "file:///home/your-download-dir/"
4536 INHERIT += "own-mirrors"
4537 BB_NO_NETWORK = "1"
4538
4539 The ``SOURCE_MIRROR_URL`` and ``own-mirror``
4540 class set up the system to use the downloads directory as your "own
4541 mirror". Using the ``BB_NO_NETWORK`` variable makes sure that
4542 BitBake's fetching process in step 3 stays local, which means files
4543 from your "own-mirror" are used.
4544
45452. *Start With a Clean Build:* You can start with a clean build by
4546 removing the
4547 ``${``\ :term:`TMPDIR`\ ``}``
4548 directory or using a new :term:`Build Directory`.
4549
45503. *Build Your Target:* Use BitBake to build your target:
4551 ::
4552
4553 $ bitbake target
4554
4555 The build completes using the known local "snapshot" of source
4556 files from your mirror. The resulting tarballs for your "snapshot" of
4557 source files are in the downloads directory.
4558
4559 .. note::
4560
4561 The offline build does not work if recipes attempt to find the
4562 latest version of software by setting
4563 :term:`SRCREV` to
4564 ``${``\ :term:`AUTOREV`\ ``}``:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004565 ::
4566
4567 SRCREV = "${AUTOREV}"
4568
4569 When a recipe sets ``SRCREV`` to
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004570 ``${AUTOREV}``, the build system accesses the network in an
4571 attempt to determine the latest version of software from the SCM.
4572 Typically, recipes that use ``AUTOREV`` are custom or modified
4573 recipes. Recipes that reside in public repositories usually do not
4574 use ``AUTOREV``.
4575
4576 If you do have recipes that use ``AUTOREV``, you can take steps to
4577 still use the recipes in an offline build. Do the following:
4578
4579 1. Use a configuration generated by enabling `build
4580 history <#maintaining-build-output-quality>`__.
4581
4582 2. Use the ``buildhistory-collect-srcrevs`` command to collect the
4583 stored ``SRCREV`` values from the build's history. For more
4584 information on collecting these values, see the "`Build History
4585 Package Information <#build-history-package-information>`__"
4586 section.
4587
4588 3. Once you have the correct source revisions, you can modify
4589 those recipes to to set ``SRCREV`` to specific versions of the
4590 software.
4591
4592Speeding Up a Build
4593===================
4594
4595Build time can be an issue. By default, the build system uses simple
4596controls to try and maximize build efficiency. In general, the default
4597settings for all the following variables result in the most efficient
4598build times when dealing with single socket systems (i.e. a single CPU).
4599If you have multiple CPUs, you might try increasing the default values
4600to gain more speed. See the descriptions in the glossary for each
4601variable for more information:
4602
4603- :term:`BB_NUMBER_THREADS`:
4604 The maximum number of threads BitBake simultaneously executes.
4605
4606- :term:`bitbake:BB_NUMBER_PARSE_THREADS`:
4607 The number of threads BitBake uses during parsing.
4608
4609- :term:`PARALLEL_MAKE`: Extra
4610 options passed to the ``make`` command during the
4611 :ref:`ref-tasks-compile` task in
4612 order to specify parallel compilation on the local build host.
4613
4614- :term:`PARALLEL_MAKEINST`:
4615 Extra options passed to the ``make`` command during the
4616 :ref:`ref-tasks-install` task in
4617 order to specify parallel installation on the local build host.
4618
4619As mentioned, these variables all scale to the number of processor cores
4620available on the build system. For single socket systems, this
4621auto-scaling ensures that the build system fundamentally takes advantage
4622of potential parallel operations during the build based on the build
4623machine's capabilities.
4624
4625Following are additional factors that can affect build speed:
4626
4627- File system type: The file system type that the build is being
4628 performed on can also influence performance. Using ``ext4`` is
4629 recommended as compared to ``ext2`` and ``ext3`` due to ``ext4``
4630 improved features such as extents.
4631
4632- Disabling the updating of access time using ``noatime``: The
4633 ``noatime`` mount option prevents the build system from updating file
4634 and directory access times.
4635
4636- Setting a longer commit: Using the "commit=" mount option increases
4637 the interval in seconds between disk cache writes. Changing this
4638 interval from the five second default to something longer increases
4639 the risk of data loss but decreases the need to write to the disk,
4640 thus increasing the build performance.
4641
4642- Choosing the packaging backend: Of the available packaging backends,
4643 IPK is the fastest. Additionally, selecting a singular packaging
4644 backend also helps.
4645
4646- Using ``tmpfs`` for :term:`TMPDIR`
4647 as a temporary file system: While this can help speed up the build,
4648 the benefits are limited due to the compiler using ``-pipe``. The
4649 build system goes to some lengths to avoid ``sync()`` calls into the
4650 file system on the principle that if there was a significant failure,
4651 the :term:`Build Directory`
4652 contents could easily be rebuilt.
4653
4654- Inheriting the
4655 :ref:`rm_work <ref-classes-rm-work>` class:
4656 Inheriting this class has shown to speed up builds due to
4657 significantly lower amounts of data stored in the data cache as well
4658 as on disk. Inheriting this class also makes cleanup of
4659 :term:`TMPDIR` faster, at the
4660 expense of being easily able to dive into the source code. File
4661 system maintainers have recommended that the fastest way to clean up
4662 large numbers of files is to reformat partitions rather than delete
4663 files due to the linear nature of partitions. This, of course,
4664 assumes you structure the disk partitions and file systems in a way
4665 that this is practical.
4666
4667Aside from the previous list, you should keep some trade offs in mind
4668that can help you speed up the build:
4669
4670- Remove items from
4671 :term:`DISTRO_FEATURES`
4672 that you might not need.
4673
4674- Exclude debug symbols and other debug information: If you do not need
4675 these symbols and other debug information, disabling the ``*-dbg``
4676 package generation can speed up the build. You can disable this
4677 generation by setting the
4678 :term:`INHIBIT_PACKAGE_DEBUG_SPLIT`
4679 variable to "1".
4680
4681- Disable static library generation for recipes derived from
4682 ``autoconf`` or ``libtool``: Following is an example showing how to
4683 disable static libraries and still provide an override to handle
4684 exceptions:
4685 ::
4686
4687 STATICLIBCONF = "--disable-static"
4688 STATICLIBCONF_sqlite3-native = ""
4689 EXTRA_OECONF += "${STATICLIBCONF}"
4690
4691 .. note::
4692
4693 - Some recipes need static libraries in order to work correctly
4694 (e.g. ``pseudo-native`` needs ``sqlite3-native``). Overrides,
4695 as in the previous example, account for these kinds of
4696 exceptions.
4697
4698 - Some packages have packaging code that assumes the presence of
4699 the static libraries. If so, you might need to exclude them as
4700 well.
4701
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004702Working With Libraries
4703======================
4704
4705Libraries are an integral part of your system. This section describes
4706some common practices you might find helpful when working with libraries
4707to build your system:
4708
4709- `How to include static library
4710 files <#including-static-library-files>`__
4711
4712- `How to use the Multilib feature to combine multiple versions of
4713 library files into a single
4714 image <#combining-multiple-versions-library-files-into-one-image>`__
4715
4716- `How to install multiple versions of the same library in parallel on
4717 the same
4718 system <#installing-multiple-versions-of-the-same-library>`__
4719
4720Including Static Library Files
4721------------------------------
4722
4723If you are building a library and the library offers static linking, you
4724can control which static library files (``*.a`` files) get included in
4725the built library.
4726
4727The :term:`PACKAGES` and
4728:term:`FILES_* <FILES>` variables in the
4729``meta/conf/bitbake.conf`` configuration file define how files installed
4730by the ``do_install`` task are packaged. By default, the ``PACKAGES``
4731variable includes ``${PN}-staticdev``, which represents all static
4732library files.
4733
4734.. note::
4735
4736 Some previously released versions of the Yocto Project defined the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004737 static library files through ``${PN}-dev``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004738
4739Following is part of the BitBake configuration file, where you can see
4740how the static library files are defined:
4741::
4742
4743 PACKAGE_BEFORE_PN ?= ""
4744 PACKAGES = "${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc ${PN}-locale ${PACKAGE_BEFORE_PN} ${PN}"
4745 PACKAGES_DYNAMIC = "^${PN}-locale-.*"
4746 FILES = ""
4747
4748 FILES_${PN} = "${bindir}/* ${sbindir}/* ${libexecdir}/* ${libdir}/lib*${SOLIBS} \
4749 ${sysconfdir} ${sharedstatedir} ${localstatedir} \
4750 ${base_bindir}/* ${base_sbindir}/* \
4751 ${base_libdir}/*${SOLIBS} \
4752 ${base_prefix}/lib/udev/rules.d ${prefix}/lib/udev/rules.d \
4753 ${datadir}/${BPN} ${libdir}/${BPN}/* \
4754 ${datadir}/pixmaps ${datadir}/applications \
4755 ${datadir}/idl ${datadir}/omf ${datadir}/sounds \
4756 ${libdir}/bonobo/servers"
4757
4758 FILES_${PN}-bin = "${bindir}/* ${sbindir}/*"
4759
4760 FILES_${PN}-doc = "${docdir} ${mandir} ${infodir} ${datadir}/gtk-doc \
4761 ${datadir}/gnome/help"
4762 SECTION_${PN}-doc = "doc"
4763
4764 FILES_SOLIBSDEV ?= "${base_libdir}/lib*${SOLIBSDEV} ${libdir}/lib*${SOLIBSDEV}"
4765 FILES_${PN}-dev = "${includedir} ${FILES_SOLIBSDEV} ${libdir}/*.la \
4766 ${libdir}/*.o ${libdir}/pkgconfig ${datadir}/pkgconfig \
4767 ${datadir}/aclocal ${base_libdir}/*.o \
4768 ${libdir}/${BPN}/*.la ${base_libdir}/*.la"
4769 SECTION_${PN}-dev = "devel"
4770 ALLOW_EMPTY_${PN}-dev = "1"
4771 RDEPENDS_${PN}-dev = "${PN} (= ${EXTENDPKGV})"
4772
4773 FILES_${PN}-staticdev = "${libdir}/*.a ${base_libdir}/*.a ${libdir}/${BPN}/*.a"
4774 SECTION_${PN}-staticdev = "devel"
4775 RDEPENDS_${PN}-staticdev = "${PN}-dev (= ${EXTENDPKGV})"
4776
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004777Combining Multiple Versions of Library Files into One Image
4778-----------------------------------------------------------
4779
4780The build system offers the ability to build libraries with different
4781target optimizations or architecture formats and combine these together
4782into one system image. You can link different binaries in the image
4783against the different libraries as needed for specific use cases. This
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004784feature is called "Multilib".
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004785
4786An example would be where you have most of a system compiled in 32-bit
4787mode using 32-bit libraries, but you have something large, like a
4788database engine, that needs to be a 64-bit application and uses 64-bit
4789libraries. Multilib allows you to get the best of both 32-bit and 64-bit
4790libraries.
4791
4792While the Multilib feature is most commonly used for 32 and 64-bit
4793differences, the approach the build system uses facilitates different
4794target optimizations. You could compile some binaries to use one set of
4795libraries and other binaries to use a different set of libraries. The
4796libraries could differ in architecture, compiler options, or other
4797optimizations.
4798
4799Several examples exist in the ``meta-skeleton`` layer found in the
4800:term:`Source Directory`:
4801
4802- ``conf/multilib-example.conf`` configuration file
4803
4804- ``conf/multilib-example2.conf`` configuration file
4805
4806- ``recipes-multilib/images/core-image-multilib-example.bb`` recipe
4807
4808Preparing to Use Multilib
4809~~~~~~~~~~~~~~~~~~~~~~~~~
4810
4811User-specific requirements drive the Multilib feature. Consequently,
4812there is no one "out-of-the-box" configuration that likely exists to
4813meet your needs.
4814
4815In order to enable Multilib, you first need to ensure your recipe is
4816extended to support multiple libraries. Many standard recipes are
4817already extended and support multiple libraries. You can check in the
4818``meta/conf/multilib.conf`` configuration file in the
4819:term:`Source Directory` to see how this is
4820done using the
4821:term:`BBCLASSEXTEND` variable.
4822Eventually, all recipes will be covered and this list will not be
4823needed.
4824
4825For the most part, the Multilib class extension works automatically to
4826extend the package name from ``${PN}`` to ``${MLPREFIX}${PN}``, where
4827``MLPREFIX`` is the particular multilib (e.g. "lib32-" or "lib64-").
4828Standard variables such as
4829:term:`DEPENDS`,
4830:term:`RDEPENDS`,
4831:term:`RPROVIDES`,
4832:term:`RRECOMMENDS`,
4833:term:`PACKAGES`, and
4834:term:`PACKAGES_DYNAMIC` are
4835automatically extended by the system. If you are extending any manual
4836code in the recipe, you can use the ``${MLPREFIX}`` variable to ensure
4837those names are extended correctly. This automatic extension code
4838resides in ``multilib.bbclass``.
4839
4840Using Multilib
4841~~~~~~~~~~~~~~
4842
4843After you have set up the recipes, you need to define the actual
4844combination of multiple libraries you want to build. You accomplish this
4845through your ``local.conf`` configuration file in the
4846:term:`Build Directory`. An example
4847configuration would be as follows:
4848::
4849
4850 MACHINE = "qemux86-64"
4851 require conf/multilib.conf
4852 MULTILIBS = "multilib:lib32"
4853 DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
4854 IMAGE_INSTALL_append = "lib32-glib-2.0"
4855
4856This example enables an additional library named
4857``lib32`` alongside the normal target packages. When combining these
4858"lib32" alternatives, the example uses "x86" for tuning. For information
4859on this particular tuning, see
4860``meta/conf/machine/include/ia32/arch-ia32.inc``.
4861
4862The example then includes ``lib32-glib-2.0`` in all the images, which
4863illustrates one method of including a multiple library dependency. You
4864can use a normal image build to include this dependency, for example:
4865::
4866
4867 $ bitbake core-image-sato
4868
4869You can also build Multilib packages
4870specifically with a command like this:
4871::
4872
4873 $ bitbake lib32-glib-2.0
4874
4875Additional Implementation Details
4876~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4877
4878Generic implementation details as well as details that are specific to
4879package management systems exist. Following are implementation details
4880that exist regardless of the package management system:
4881
4882- The typical convention used for the class extension code as used by
4883 Multilib assumes that all package names specified in
4884 :term:`PACKAGES` that contain
4885 ``${PN}`` have ``${PN}`` at the start of the name. When that
4886 convention is not followed and ``${PN}`` appears at the middle or the
4887 end of a name, problems occur.
4888
4889- The :term:`TARGET_VENDOR`
4890 value under Multilib will be extended to "-vendormlmultilib" (e.g.
4891 "-pokymllib32" for a "lib32" Multilib with Poky). The reason for this
4892 slightly unwieldy contraction is that any "-" characters in the
4893 vendor string presently break Autoconf's ``config.sub``, and other
4894 separators are problematic for different reasons.
4895
4896For the RPM Package Management System, the following implementation
4897details exist:
4898
4899- A unique architecture is defined for the Multilib packages, along
4900 with creating a unique deploy folder under ``tmp/deploy/rpm`` in the
4901 :term:`Build Directory`. For
4902 example, consider ``lib32`` in a ``qemux86-64`` image. The possible
4903 architectures in the system are "all", "qemux86_64",
4904 "lib32_qemux86_64", and "lib32_x86".
4905
4906- The ``${MLPREFIX}`` variable is stripped from ``${PN}`` during RPM
4907 packaging. The naming for a normal RPM package and a Multilib RPM
4908 package in a ``qemux86-64`` system resolves to something similar to
4909 ``bash-4.1-r2.x86_64.rpm`` and ``bash-4.1.r2.lib32_x86.rpm``,
4910 respectively.
4911
4912- When installing a Multilib image, the RPM backend first installs the
4913 base image and then installs the Multilib libraries.
4914
4915- The build system relies on RPM to resolve the identical files in the
4916 two (or more) Multilib packages.
4917
4918For the IPK Package Management System, the following implementation
4919details exist:
4920
4921- The ``${MLPREFIX}`` is not stripped from ``${PN}`` during IPK
4922 packaging. The naming for a normal RPM package and a Multilib IPK
4923 package in a ``qemux86-64`` system resolves to something like
4924 ``bash_4.1-r2.x86_64.ipk`` and ``lib32-bash_4.1-rw_x86.ipk``,
4925 respectively.
4926
4927- The IPK deploy folder is not modified with ``${MLPREFIX}`` because
4928 packages with and without the Multilib feature can exist in the same
4929 folder due to the ``${PN}`` differences.
4930
4931- IPK defines a sanity check for Multilib installation using certain
4932 rules for file comparison, overridden, etc.
4933
4934Installing Multiple Versions of the Same Library
4935------------------------------------------------
4936
4937Situations can exist where you need to install and use multiple versions
4938of the same library on the same system at the same time. These
4939situations almost always exist when a library API changes and you have
4940multiple pieces of software that depend on the separate versions of the
4941library. To accommodate these situations, you can install multiple
4942versions of the same library in parallel on the same system.
4943
4944The process is straightforward as long as the libraries use proper
4945versioning. With properly versioned libraries, all you need to do to
4946individually specify the libraries is create separate, appropriately
4947named recipes where the :term:`PN` part of
4948the name includes a portion that differentiates each library version
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004949(e.g. the major part of the version number). Thus, instead of having a
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004950single recipe that loads one version of a library (e.g. ``clutter``),
4951you provide multiple recipes that result in different versions of the
4952libraries you want. As an example, the following two recipes would allow
4953the two separate versions of the ``clutter`` library to co-exist on the
4954same system:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05004955
4956.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05004957
4958 clutter-1.6_1.6.20.bb
4959 clutter-1.8_1.8.4.bb
4960
4961Additionally, if
4962you have other recipes that depend on a given library, you need to use
4963the :term:`DEPENDS` variable to
4964create the dependency. Continuing with the same example, if you want to
4965have a recipe depend on the 1.8 version of the ``clutter`` library, use
4966the following in your recipe:
4967::
4968
4969 DEPENDS = "clutter-1.8"
4970
4971Using x32 psABI
4972===============
4973
4974x32 processor-specific Application Binary Interface (`x32
4975psABI <https://software.intel.com/en-us/node/628948>`__) is a native
497632-bit processor-specific ABI for Intel 64 (x86-64) architectures. An
4977ABI defines the calling conventions between functions in a processing
4978environment. The interface determines what registers are used and what
4979the sizes are for various C data types.
4980
4981Some processing environments prefer using 32-bit applications even when
4982running on Intel 64-bit platforms. Consider the i386 psABI, which is a
4983very old 32-bit ABI for Intel 64-bit platforms. The i386 psABI does not
4984provide efficient use and access of the Intel 64-bit processor
4985resources, leaving the system underutilized. Now consider the x86_64
4986psABI. This ABI is newer and uses 64-bits for data sizes and program
4987pointers. The extra bits increase the footprint size of the programs,
4988libraries, and also increases the memory and file system size
4989requirements. Executing under the x32 psABI enables user programs to
4990utilize CPU and system resources more efficiently while keeping the
4991memory footprint of the applications low. Extra bits are used for
4992registers but not for addressing mechanisms.
4993
4994The Yocto Project supports the final specifications of x32 psABI as
4995follows:
4996
4997- You can create packages and images in x32 psABI format on x86_64
4998 architecture targets.
4999
5000- You can successfully build recipes with the x32 toolchain.
5001
5002- You can create and boot ``core-image-minimal`` and
5003 ``core-image-sato`` images.
5004
5005- RPM Package Manager (RPM) support exists for x32 binaries.
5006
5007- Support for large images exists.
5008
5009To use the x32 psABI, you need to edit your ``conf/local.conf``
5010configuration file as follows:
5011::
5012
5013 MACHINE = "qemux86-64"
5014 DEFAULTTUNE = "x86-64-x32"
5015 baselib = "${@d.getVar('BASE_LIB_tune-' + (d.getVar('DEFAULTTUNE') \
5016 or 'INVALID')) or 'lib'}"
5017
5018Once you have set
5019up your configuration file, use BitBake to build an image that supports
5020the x32 psABI. Here is an example:
5021::
5022
5023 $ bitbake core-image-sato
5024
5025Enabling GObject Introspection Support
5026======================================
5027
5028`GObject
5029introspection <https://wiki.gnome.org/Projects/GObjectIntrospection>`__
5030is the standard mechanism for accessing GObject-based software from
5031runtime environments. GObject is a feature of the GLib library that
5032provides an object framework for the GNOME desktop and related software.
5033GObject Introspection adds information to GObject that allows objects
5034created within it to be represented across different programming
5035languages. If you want to construct GStreamer pipelines using Python, or
5036control UPnP infrastructure using Javascript and GUPnP, GObject
5037introspection is the only way to do it.
5038
5039This section describes the Yocto Project support for generating and
5040packaging GObject introspection data. GObject introspection data is a
5041description of the API provided by libraries built on top of GLib
5042framework, and, in particular, that framework's GObject mechanism.
5043GObject Introspection Repository (GIR) files go to ``-dev`` packages,
5044``typelib`` files go to main packages as they are packaged together with
5045libraries that are introspected.
5046
5047The data is generated when building such a library, by linking the
5048library with a small executable binary that asks the library to describe
5049itself, and then executing the binary and processing its output.
5050
5051Generating this data in a cross-compilation environment is difficult
5052because the library is produced for the target architecture, but its
5053code needs to be executed on the build host. This problem is solved with
5054the OpenEmbedded build system by running the code through QEMU, which
5055allows precisely that. Unfortunately, QEMU does not always work
5056perfectly as mentioned in the "`Known Issues <#known-issues>`__"
5057section.
5058
5059Enabling the Generation of Introspection Data
5060---------------------------------------------
5061
5062Enabling the generation of introspection data (GIR files) in your
5063library package involves the following:
5064
50651. Inherit the
5066 :ref:`gobject-introspection <ref-classes-gobject-introspection>`
5067 class.
5068
50692. Make sure introspection is not disabled anywhere in the recipe or
5070 from anything the recipe includes. Also, make sure that
5071 "gobject-introspection-data" is not in
5072 :term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`
5073 and that "qemu-usermode" is not in
5074 :term:`MACHINE_FEATURES_BACKFILL_CONSIDERED`.
5075 If either of these conditions exist, nothing will happen.
5076
50773. Try to build the recipe. If you encounter build errors that look like
5078 something is unable to find ``.so`` libraries, check where these
5079 libraries are located in the source tree and add the following to the
5080 recipe:
5081 ::
5082
5083 GIR_EXTRA_LIBS_PATH = "${B}/something/.libs"
5084
5085 .. note::
5086
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005087 See recipes in the ``oe-core`` repository that use that
5088 ``GIR_EXTRA_LIBS_PATH`` variable as an example.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005089
50904. Look for any other errors, which probably mean that introspection
5091 support in a package is not entirely standard, and thus breaks down
5092 in a cross-compilation environment. For such cases, custom-made fixes
5093 are needed. A good place to ask and receive help in these cases is
5094 the :ref:`Yocto Project mailing
5095 lists <resources-mailinglist>`.
5096
5097.. note::
5098
5099 Using a library that no longer builds against the latest Yocto
5100 Project release and prints introspection related errors is a good
5101 candidate for the previous procedure.
5102
5103Disabling the Generation of Introspection Data
5104----------------------------------------------
5105
5106You might find that you do not want to generate introspection data. Or,
5107perhaps QEMU does not work on your build host and target architecture
5108combination. If so, you can use either of the following methods to
5109disable GIR file generations:
5110
5111- Add the following to your distro configuration:
5112 ::
5113
5114 DISTRO_FEATURES_BACKFILL_CONSIDERED = "gobject-introspection-data"
5115
5116 Adding this statement disables generating introspection data using
5117 QEMU but will still enable building introspection tools and libraries
5118 (i.e. building them does not require the use of QEMU).
5119
5120- Add the following to your machine configuration:
5121 ::
5122
5123 MACHINE_FEATURES_BACKFILL_CONSIDERED = "qemu-usermode"
5124
5125 Adding this statement disables the use of QEMU when building packages for your
5126 machine. Currently, this feature is used only by introspection
5127 recipes and has the same effect as the previously described option.
5128
5129 .. note::
5130
5131 Future releases of the Yocto Project might have other features
5132 affected by this option.
5133
5134If you disable introspection data, you can still obtain it through other
5135means such as copying the data from a suitable sysroot, or by generating
5136it on the target hardware. The OpenEmbedded build system does not
5137currently provide specific support for these techniques.
5138
5139Testing that Introspection Works in an Image
5140--------------------------------------------
5141
5142Use the following procedure to test if generating introspection data is
5143working in an image:
5144
51451. Make sure that "gobject-introspection-data" is not in
5146 :term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`
5147 and that "qemu-usermode" is not in
5148 :term:`MACHINE_FEATURES_BACKFILL_CONSIDERED`.
5149
51502. Build ``core-image-sato``.
5151
51523. Launch a Terminal and then start Python in the terminal.
5153
51544. Enter the following in the terminal:
5155 ::
5156
5157 >>> from gi.repository import GLib
5158 >>> GLib.get_host_name()
5159
51605. For something a little more advanced, enter the following see:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005161 https://python-gtk-3-tutorial.readthedocs.io/en/latest/introduction.html
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005162
5163Known Issues
5164------------
5165
5166The following know issues exist for GObject Introspection Support:
5167
5168- ``qemu-ppc64`` immediately crashes. Consequently, you cannot build
5169 introspection data on that architecture.
5170
5171- x32 is not supported by QEMU. Consequently, introspection data is
5172 disabled.
5173
5174- musl causes transient GLib binaries to crash on assertion failures.
5175 Consequently, generating introspection data is disabled.
5176
5177- Because QEMU is not able to run the binaries correctly, introspection
5178 is disabled for some specific packages under specific architectures
5179 (e.g. ``gcr``, ``libsecret``, and ``webkit``).
5180
5181- QEMU usermode might not work properly when running 64-bit binaries
5182 under 32-bit host machines. In particular, "qemumips64" is known to
5183 not work under i686.
5184
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005185Optionally Using an External Toolchain
5186======================================
5187
5188You might want to use an external toolchain as part of your development.
5189If this is the case, the fundamental steps you need to accomplish are as
5190follows:
5191
5192- Understand where the installed toolchain resides. For cases where you
5193 need to build the external toolchain, you would need to take separate
5194 steps to build and install the toolchain.
5195
5196- Make sure you add the layer that contains the toolchain to your
5197 ``bblayers.conf`` file through the
5198 :term:`BBLAYERS` variable.
5199
5200- Set the ``EXTERNAL_TOOLCHAIN`` variable in your ``local.conf`` file
5201 to the location in which you installed the toolchain.
5202
5203A good example of an external toolchain used with the Yocto Project is
5204Mentor Graphics Sourcery G++ Toolchain. You can see information on how
5205to use that particular layer in the ``README`` file at
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005206https://github.com/MentorEmbedded/meta-sourcery/. You can find
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005207further information by reading about the
5208:term:`TCMODE` variable in the Yocto
5209Project Reference Manual's variable glossary.
5210
5211Creating Partitioned Images Using Wic
5212=====================================
5213
5214Creating an image for a particular hardware target using the
5215OpenEmbedded build system does not necessarily mean you can boot that
5216image as is on your device. Physical devices accept and boot images in
5217various ways depending on the specifics of the device. Usually,
5218information about the hardware can tell you what image format the device
5219requires. Should your device require multiple partitions on an SD card,
5220flash, or an HDD, you can use the OpenEmbedded Image Creator, Wic, to
5221create the properly partitioned image.
5222
5223The ``wic`` command generates partitioned images from existing
5224OpenEmbedded build artifacts. Image generation is driven by partitioning
5225commands contained in an Openembedded kickstart file (``.wks``)
5226specified either directly on the command line or as one of a selection
5227of canned kickstart files as shown with the ``wic list images`` command
5228in the "`Using an Existing Kickstart
5229File <#using-a-provided-kickstart-file>`__" section. When you apply the
5230command to a given set of build artifacts, the result is an image or set
5231of images that can be directly written onto media and used on a
5232particular system.
5233
5234.. note::
5235
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005236 For a kickstart file reference, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06005237 ":ref:`ref-manual/kickstart:openembedded kickstart (\`\`.wks\`\`) reference`"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005238 Chapter in the Yocto Project Reference Manual.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005239
5240The ``wic`` command and the infrastructure it is based on is by
5241definition incomplete. The purpose of the command is to allow the
5242generation of customized images, and as such, was designed to be
5243completely extensible through a plugin interface. See the "`Using the
5244Wic PlugIn Interface <#wic-using-the-wic-plugin-interface>`__" section
5245for information on these plugins.
5246
5247This section provides some background information on Wic, describes what
5248you need to have in place to run the tool, provides instruction on how
5249to use the Wic utility, provides information on using the Wic plugins
5250interface, and provides several examples that show how to use Wic.
5251
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005252Background
5253----------
5254
5255This section provides some background on the Wic utility. While none of
5256this information is required to use Wic, you might find it interesting.
5257
5258- The name "Wic" is derived from OpenEmbedded Image Creator (oeic). The
5259 "oe" diphthong in "oeic" was promoted to the letter "w", because
5260 "oeic" is both difficult to remember and to pronounce.
5261
5262- Wic is loosely based on the Meego Image Creator (``mic``) framework.
5263 The Wic implementation has been heavily modified to make direct use
5264 of OpenEmbedded build artifacts instead of package installation and
5265 configuration, which are already incorporated within the OpenEmbedded
5266 artifacts.
5267
5268- Wic is a completely independent standalone utility that initially
5269 provides easier-to-use and more flexible replacements for an existing
5270 functionality in OE-Core's
5271 :ref:`image-live <ref-classes-image-live>`
5272 class. The difference between Wic and those examples is that with Wic
5273 the functionality of those scripts is implemented by a
5274 general-purpose partitioning language, which is based on Redhat
5275 kickstart syntax.
5276
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005277Requirements
5278------------
5279
5280In order to use the Wic utility with the OpenEmbedded Build system, your
5281system needs to meet the following requirements:
5282
5283- The Linux distribution on your development host must support the
5284 Yocto Project. See the ":ref:`detailed-supported-distros`"
5285 section in the Yocto Project Reference Manual for the list of
5286 distributions that support the Yocto Project.
5287
5288- The standard system utilities, such as ``cp``, must be installed on
5289 your development host system.
5290
5291- You must have sourced the build environment setup script (i.e.
5292 :ref:`structure-core-script`) found in the
5293 :term:`Build Directory`.
5294
5295- You need to have the build artifacts already available, which
5296 typically means that you must have already created an image using the
5297 Openembedded build system (e.g. ``core-image-minimal``). While it
5298 might seem redundant to generate an image in order to create an image
5299 using Wic, the current version of Wic requires the artifacts in the
5300 form generated by the OpenEmbedded build system.
5301
5302- You must build several native tools, which are built to run on the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005303 build system:
5304 ::
5305
5306 $ bitbake parted-native dosfstools-native mtools-native
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005307
5308- Include "wic" as part of the
5309 :term:`IMAGE_FSTYPES`
5310 variable.
5311
5312- Include the name of the :ref:`wic kickstart file <openembedded-kickstart-wks-reference>`
5313 as part of the :term:`WKS_FILE` variable
5314
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005315Getting Help
5316------------
5317
5318You can get general help for the ``wic`` command by entering the ``wic``
5319command by itself or by entering the command with a help argument as
5320follows:
5321::
5322
5323 $ wic -h
5324 $ wic --help
5325 $ wic help
5326
5327Currently, Wic supports seven commands: ``cp``, ``create``, ``help``,
5328``list``, ``ls``, ``rm``, and ``write``. You can get help for all these
5329commands except "help" by using the following form:
5330::
5331
5332 $ wic help command
5333
5334For example, the following command returns help for the ``write``
5335command:
5336::
5337
5338 $ wic help write
5339
5340Wic supports help for three topics: ``overview``, ``plugins``, and
5341``kickstart``. You can get help for any topic using the following form:
5342::
5343
5344 $ wic help topic
5345
5346For example, the following returns overview help for Wic:
5347::
5348
5349 $ wic help overview
5350
5351One additional level of help exists for Wic. You can get help on
5352individual images through the ``list`` command. You can use the ``list``
5353command to return the available Wic images as follows:
5354::
5355
5356 $ wic list images
5357 genericx86 Create an EFI disk image for genericx86*
5358 beaglebone-yocto Create SD card image for Beaglebone
5359 edgerouter Create SD card image for Edgerouter
5360 qemux86-directdisk Create a qemu machine 'pcbios' direct disk image
5361 directdisk-gpt Create a 'pcbios' direct disk image
5362 mkefidisk Create an EFI disk image
5363 directdisk Create a 'pcbios' direct disk image
5364 systemd-bootdisk Create an EFI disk image with systemd-boot
5365 mkhybridiso Create a hybrid ISO image
5366 sdimage-bootpart Create SD card image with a boot partition
5367 directdisk-multi-rootfs Create multi rootfs image using rootfs plugin
5368 directdisk-bootloader-config Create a 'pcbios' direct disk image with custom bootloader config
5369
5370Once you know the list of available
5371Wic images, you can use ``help`` with the command to get help on a
5372particular image. For example, the following command returns help on the
5373"beaglebone-yocto" image:
5374::
5375
5376 $ wic list beaglebone-yocto help
5377
5378 Creates a partitioned SD card image for Beaglebone.
5379 Boot files are located in the first vfat partition.
5380
5381Operational Modes
5382-----------------
5383
5384You can use Wic in two different modes, depending on how much control
5385you need for specifying the Openembedded build artifacts that are used
5386for creating the image: Raw and Cooked:
5387
5388- *Raw Mode:* You explicitly specify build artifacts through Wic
5389 command-line arguments.
5390
5391- *Cooked Mode:* The current
5392 :term:`MACHINE` setting and image
5393 name are used to automatically locate and provide the build
5394 artifacts. You just supply a kickstart file and the name of the image
5395 from which to use artifacts.
5396
5397Regardless of the mode you use, you need to have the build artifacts
5398ready and available.
5399
5400Raw Mode
5401~~~~~~~~
5402
5403Running Wic in raw mode allows you to specify all the partitions through
5404the ``wic`` command line. The primary use for raw mode is if you have
5405built your kernel outside of the Yocto Project
5406:term:`Build Directory`. In other words, you
5407can point to arbitrary kernel, root filesystem locations, and so forth.
5408Contrast this behavior with cooked mode where Wic looks in the Build
5409Directory (e.g. ``tmp/deploy/images/``\ machine).
5410
5411The general form of the ``wic`` command in raw mode is:
5412::
5413
5414 $ wic create wks_file options ...
5415
5416 Where:
5417
5418 wks_file:
5419 An OpenEmbedded kickstart file. You can provide
5420 your own custom file or use a file from a set of
5421 existing files as described by further options.
5422
5423 optional arguments:
5424 -h, --help show this help message and exit
5425 -o OUTDIR, --outdir OUTDIR
5426 name of directory to create image in
5427 -e IMAGE_NAME, --image-name IMAGE_NAME
5428 name of the image to use the artifacts from e.g. core-
5429 image-sato
5430 -r ROOTFS_DIR, --rootfs-dir ROOTFS_DIR
5431 path to the /rootfs dir to use as the .wks rootfs
5432 source
5433 -b BOOTIMG_DIR, --bootimg-dir BOOTIMG_DIR
5434 path to the dir containing the boot artifacts (e.g.
5435 /EFI or /syslinux dirs) to use as the .wks bootimg
5436 source
5437 -k KERNEL_DIR, --kernel-dir KERNEL_DIR
5438 path to the dir containing the kernel to use in the
5439 .wks bootimg
5440 -n NATIVE_SYSROOT, --native-sysroot NATIVE_SYSROOT
5441 path to the native sysroot containing the tools to use
5442 to build the image
5443 -s, --skip-build-check
5444 skip the build check
5445 -f, --build-rootfs build rootfs
5446 -c {gzip,bzip2,xz}, --compress-with {gzip,bzip2,xz}
5447 compress image with specified compressor
5448 -m, --bmap generate .bmap
5449 --no-fstab-update Do not change fstab file.
5450 -v VARS_DIR, --vars VARS_DIR
5451 directory with <image>.env files that store bitbake
5452 variables
5453 -D, --debug output debug information
5454
5455.. note::
5456
5457 You do not need root privileges to run Wic. In fact, you should not
5458 run as root when using the utility.
5459
5460Cooked Mode
5461~~~~~~~~~~~
5462
5463Running Wic in cooked mode leverages off artifacts in the Build
5464Directory. In other words, you do not have to specify kernel or root
5465filesystem locations as part of the command. All you need to provide is
5466a kickstart file and the name of the image from which to use artifacts
5467by using the "-e" option. Wic looks in the Build Directory (e.g.
5468``tmp/deploy/images/``\ machine) for artifacts.
5469
5470The general form of the ``wic`` command using Cooked Mode is as follows:
5471::
5472
5473 $ wic create wks_file -e IMAGE_NAME
5474
5475 Where:
5476
5477 wks_file:
5478 An OpenEmbedded kickstart file. You can provide
5479 your own custom file or use a file from a set of
5480 existing files provided with the Yocto Project
5481 release.
5482
5483 required argument:
5484 -e IMAGE_NAME, --image-name IMAGE_NAME
5485 name of the image to use the artifacts from e.g. core-
5486 image-sato
5487
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005488Using an Existing Kickstart File
5489--------------------------------
5490
5491If you do not want to create your own kickstart file, you can use an
5492existing file provided by the Wic installation. As shipped, kickstart
Andrew Geissler09209ee2020-12-13 08:44:15 -06005493files can be found in the :ref:`overview-manual/development-environment:yocto project source repositories` in the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005494following two locations:
5495::
5496
5497 poky/meta-yocto-bsp/wic
5498 poky/scripts/lib/wic/canned-wks
5499
5500Use the following command to list the available kickstart files:
5501::
5502
5503 $ wic list images
5504 genericx86 Create an EFI disk image for genericx86*
5505 beaglebone-yocto Create SD card image for Beaglebone
5506 edgerouter Create SD card image for Edgerouter
5507 qemux86-directdisk Create a qemu machine 'pcbios' direct disk image
5508 directdisk-gpt Create a 'pcbios' direct disk image
5509 mkefidisk Create an EFI disk image
5510 directdisk Create a 'pcbios' direct disk image
5511 systemd-bootdisk Create an EFI disk image with systemd-boot
5512 mkhybridiso Create a hybrid ISO image
5513 sdimage-bootpart Create SD card image with a boot partition
5514 directdisk-multi-rootfs Create multi rootfs image using rootfs plugin
5515 directdisk-bootloader-config Create a 'pcbios' direct disk image with custom bootloader config
5516
5517When you use an existing file, you
5518do not have to use the ``.wks`` extension. Here is an example in Raw
5519Mode that uses the ``directdisk`` file:
5520::
5521
5522 $ wic create directdisk -r rootfs_dir -b bootimg_dir \
5523 -k kernel_dir -n native_sysroot
5524
5525Here are the actual partition language commands used in the
5526``genericx86.wks`` file to generate an image:
5527::
5528
5529 # short-description: Create an EFI disk image for genericx86*
5530 # long-description: Creates a partitioned EFI disk image for genericx86* machines
5531 part /boot --source bootimg-efi --sourceparams="loader=grub-efi" --ondisk sda --label msdos --active --align 1024
5532 part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 1024 --use-uuid
5533 part swap --ondisk sda --size 44 --label swap1 --fstype=swap
5534
5535 bootloader --ptable gpt --timeout=5 --append="rootfstype=ext4 console=ttyS0,115200 console=tty0"
5536
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005537Using the Wic Plugin Interface
5538------------------------------
5539
5540You can extend and specialize Wic functionality by using Wic plugins.
5541This section explains the Wic plugin interface.
5542
5543.. note::
5544
5545 Wic plugins consist of "source" and "imager" plugins. Imager plugins
5546 are beyond the scope of this section.
5547
5548Source plugins provide a mechanism to customize partition content during
5549the Wic image generation process. You can use source plugins to map
5550values that you specify using ``--source`` commands in kickstart files
5551(i.e. ``*.wks``) to a plugin implementation used to populate a given
5552partition.
5553
5554.. note::
5555
5556 If you use plugins that have build-time dependencies (e.g. native
5557 tools, bootloaders, and so forth) when building a Wic image, you need
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005558 to specify those dependencies using the :term:`WKS_FILE_DEPENDS`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005559 variable.
5560
5561Source plugins are subclasses defined in plugin files. As shipped, the
5562Yocto Project provides several plugin files. You can see the source
5563plugin files that ship with the Yocto Project
Andrew Geissler09209ee2020-12-13 08:44:15 -06005564:yocto_git:`here </poky/tree/scripts/lib/wic/plugins/source>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005565Each of these plugin files contains source plugins that are designed to
5566populate a specific Wic image partition.
5567
5568Source plugins are subclasses of the ``SourcePlugin`` class, which is
5569defined in the ``poky/scripts/lib/wic/pluginbase.py`` file. For example,
5570the ``BootimgEFIPlugin`` source plugin found in the ``bootimg-efi.py``
5571file is a subclass of the ``SourcePlugin`` class, which is found in the
5572``pluginbase.py`` file.
5573
5574You can also implement source plugins in a layer outside of the Source
5575Repositories (external layer). To do so, be sure that your plugin files
5576are located in a directory whose path is
5577``scripts/lib/wic/plugins/source/`` within your external layer. When the
5578plugin files are located there, the source plugins they contain are made
5579available to Wic.
5580
5581When the Wic implementation needs to invoke a partition-specific
5582implementation, it looks for the plugin with the same name as the
5583``--source`` parameter used in the kickstart file given to that
5584partition. For example, if the partition is set up using the following
5585command in a kickstart file:
5586::
5587
5588 part /boot --source bootimg-pcbios --ondisk sda --label boot --active --align 1024
5589
5590The methods defined as class
5591members of the matching source plugin (i.e. ``bootimg-pcbios``) in the
5592``bootimg-pcbios.py`` plugin file are used.
5593
5594To be more concrete, here is the corresponding plugin definition from
5595the ``bootimg-pcbios.py`` file for the previous command along with an
5596example method called by the Wic implementation when it needs to prepare
5597a partition using an implementation-specific function:
5598::
5599
5600 .
5601 .
5602 .
5603 class BootimgPcbiosPlugin(SourcePlugin):
5604 """
5605 Create MBR boot partition and install syslinux on it.
5606 """
5607
5608 name = 'bootimg-pcbios'
5609 .
5610 .
5611 .
5612 @classmethod
5613 def do_prepare_partition(cls, part, source_params, creator, cr_workdir,
5614 oe_builddir, bootimg_dir, kernel_dir,
5615 rootfs_dir, native_sysroot):
5616 """
5617 Called to do the actual content population for a partition i.e. it
5618 'prepares' the partition to be incorporated into the image.
5619 In this case, prepare content for legacy bios boot partition.
5620 """
5621 .
5622 .
5623 .
5624
5625If a
5626subclass (plugin) itself does not implement a particular function, Wic
5627locates and uses the default version in the superclass. It is for this
5628reason that all source plugins are derived from the ``SourcePlugin``
5629class.
5630
5631The ``SourcePlugin`` class defined in the ``pluginbase.py`` file defines
5632a set of methods that source plugins can implement or override. Any
5633plugins (subclass of ``SourcePlugin``) that do not implement a
5634particular method inherit the implementation of the method from the
5635``SourcePlugin`` class. For more information, see the ``SourcePlugin``
5636class in the ``pluginbase.py`` file for details:
5637
5638The following list describes the methods implemented in the
5639``SourcePlugin`` class:
5640
5641- ``do_prepare_partition()``: Called to populate a partition with
5642 actual content. In other words, the method prepares the final
5643 partition image that is incorporated into the disk image.
5644
5645- ``do_configure_partition()``: Called before
5646 ``do_prepare_partition()`` to create custom configuration files for a
5647 partition (e.g. syslinux or grub configuration files).
5648
5649- ``do_install_disk()``: Called after all partitions have been
5650 prepared and assembled into a disk image. This method provides a hook
5651 to allow finalization of a disk image (e.g. writing an MBR).
5652
5653- ``do_stage_partition()``: Special content-staging hook called
5654 before ``do_prepare_partition()``. This method is normally empty.
5655
5656 Typically, a partition just uses the passed-in parameters (e.g. the
5657 unmodified value of ``bootimg_dir``). However, in some cases, things
5658 might need to be more tailored. As an example, certain files might
5659 additionally need to be taken from ``bootimg_dir + /boot``. This hook
5660 allows those files to be staged in a customized fashion.
5661
5662 .. note::
5663
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005664 ``get_bitbake_var()`` allows you to access non-standard variables that
5665 you might want to use for this behavior.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005666
5667You can extend the source plugin mechanism. To add more hooks, create
5668more source plugin methods within ``SourcePlugin`` and the corresponding
5669derived subclasses. The code that calls the plugin methods uses the
5670``plugin.get_source_plugin_methods()`` function to find the method or
5671methods needed by the call. Retrieval of those methods is accomplished
5672by filling up a dict with keys that contain the method names of
5673interest. On success, these will be filled in with the actual methods.
5674See the Wic implementation for examples and details.
5675
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005676Wic Examples
5677------------
5678
5679This section provides several examples that show how to use the Wic
5680utility. All the examples assume the list of requirements in the
5681"`Requirements <#wic-requirements>`__" section have been met. The
5682examples assume the previously generated image is
5683``core-image-minimal``.
5684
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005685Generate an Image using an Existing Kickstart File
5686~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5687
5688This example runs in Cooked Mode and uses the ``mkefidisk`` kickstart
5689file:
5690::
5691
5692 $ wic create mkefidisk -e core-image-minimal
5693 INFO: Building wic-tools...
5694 .
5695 .
5696 .
5697 INFO: The new image(s) can be found here:
5698 ./mkefidisk-201804191017-sda.direct
5699
5700 The following build artifacts were used to create the image(s):
5701 ROOTFS_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
5702 BOOTIMG_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
5703 KERNEL_DIR: /home/stephano/build/master/build/tmp-glibc/deploy/images/qemux86
5704 NATIVE_SYSROOT: /home/stephano/build/master/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
5705
5706 INFO: The image(s) were created using OE kickstart file:
5707 /home/stephano/build/master/openembedded-core/scripts/lib/wic/canned-wks/mkefidisk.wks
5708
5709The previous example shows the easiest way to create an image by running
5710in cooked mode and supplying a kickstart file and the "-e" option to
5711point to the existing build artifacts. Your ``local.conf`` file needs to
5712have the :term:`MACHINE` variable set
5713to the machine you are using, which is "qemux86" in this example.
5714
5715Once the image builds, the output provides image location, artifact use,
5716and kickstart file information.
5717
5718.. note::
5719
5720 You should always verify the details provided in the output to make
5721 sure that the image was indeed created exactly as expected.
5722
5723Continuing with the example, you can now write the image from the Build
5724Directory onto a USB stick, or whatever media for which you built your
5725image, and boot from the media. You can write the image by using
5726``bmaptool`` or ``dd``:
5727::
5728
5729 $ oe-run-native bmaptool copy mkefidisk-201804191017-sda.direct /dev/sdX
5730
5731or ::
5732
5733 $ sudo dd if=mkefidisk-201804191017-sda.direct of=/dev/sdX
5734
5735.. note::
5736
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005737 For more information on how to use the ``bmaptool``
5738 to flash a device with an image, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06005739 ":ref:`dev-manual/common-tasks:flashing images using \`\`bmaptool\`\``"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005740 section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005741
5742Using a Modified Kickstart File
5743~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5744
5745Because partitioned image creation is driven by the kickstart file, it
5746is easy to affect image creation by changing the parameters in the file.
5747This next example demonstrates that through modification of the
5748``directdisk-gpt`` kickstart file.
5749
5750As mentioned earlier, you can use the command ``wic list images`` to
5751show the list of existing kickstart files. The directory in which the
5752``directdisk-gpt.wks`` file resides is
5753``scripts/lib/image/canned-wks/``, which is located in the
5754:term:`Source Directory` (e.g. ``poky``).
5755Because available files reside in this directory, you can create and add
5756your own custom files to the directory. Subsequent use of the
5757``wic list images`` command would then include your kickstart files.
5758
5759In this example, the existing ``directdisk-gpt`` file already does most
5760of what is needed. However, for the hardware in this example, the image
5761will need to boot from ``sdb`` instead of ``sda``, which is what the
5762``directdisk-gpt`` kickstart file uses.
5763
5764The example begins by making a copy of the ``directdisk-gpt.wks`` file
5765in the ``scripts/lib/image/canned-wks`` directory and then by changing
5766the lines that specify the target disk from which to boot.
5767::
5768
5769 $ cp /home/stephano/poky/scripts/lib/wic/canned-wks/directdisk-gpt.wks \
5770 /home/stephano/poky/scripts/lib/wic/canned-wks/directdisksdb-gpt.wks
5771
5772Next, the example modifies the ``directdisksdb-gpt.wks`` file and
5773changes all instances of "``--ondisk sda``" to "``--ondisk sdb``". The
5774example changes the following two lines and leaves the remaining lines
5775untouched:
5776::
5777
5778 part /boot --source bootimg-pcbios --ondisk sdb --label boot --active --align 1024
5779 part / --source rootfs --ondisk sdb --fstype=ext4 --label platform --align 1024 --use-uuid
5780
5781Once the lines are changed, the
5782example generates the ``directdisksdb-gpt`` image. The command points
5783the process at the ``core-image-minimal`` artifacts for the Next Unit of
5784Computing (nuc) :term:`MACHINE` the
5785``local.conf``.
5786::
5787
5788 $ wic create directdisksdb-gpt -e core-image-minimal
5789 INFO: Building wic-tools...
5790 .
5791 .
5792 .
5793 Initialising tasks: 100% |#######################################| Time: 0:00:01
5794 NOTE: Executing SetScene Tasks
5795 NOTE: Executing RunQueue Tasks
5796 NOTE: Tasks Summary: Attempted 1161 tasks of which 1157 didn't need to be rerun and all succeeded.
5797 INFO: Creating image(s)...
5798
5799 INFO: The new image(s) can be found here:
5800 ./directdisksdb-gpt-201710090938-sdb.direct
5801
5802 The following build artifacts were used to create the image(s):
5803 ROOTFS_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
5804 BOOTIMG_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
5805 KERNEL_DIR: /home/stephano/build/master/build/tmp-glibc/deploy/images/qemux86
5806 NATIVE_SYSROOT: /home/stephano/build/master/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
5807
5808 INFO: The image(s) were created using OE kickstart file:
5809 /home/stephano/poky/scripts/lib/wic/canned-wks/directdisksdb-gpt.wks
5810
5811Continuing with the example, you can now directly ``dd`` the image to a
5812USB stick, or whatever media for which you built your image, and boot
5813the resulting media:
5814::
5815
5816 $ sudo dd if=directdisksdb-gpt-201710090938-sdb.direct of=/dev/sdb
5817 140966+0 records in
5818 140966+0 records out
5819 72174592 bytes (72 MB, 69 MiB) copied, 78.0282 s, 925 kB/s
5820 $ sudo eject /dev/sdb
5821
5822Using a Modified Kickstart File and Running in Raw Mode
5823~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5824
5825This next example manually specifies each build artifact (runs in Raw
5826Mode) and uses a modified kickstart file. The example also uses the
5827``-o`` option to cause Wic to create the output somewhere other than the
5828default output directory, which is the current directory:
5829::
5830
5831 $ wic create /home/stephano/my_yocto/test.wks -o /home/stephano/testwic \
5832 --rootfs-dir /home/stephano/build/master/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/rootfs \
5833 --bootimg-dir /home/stephano/build/master/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share \
5834 --kernel-dir /home/stephano/build/master/build/tmp/deploy/images/qemux86 \
5835 --native-sysroot /home/stephano/build/master/build/tmp/work/i586-poky-linux/wic-tools/1.0-r0/recipe-sysroot-native
5836
5837 INFO: Creating image(s)...
5838
5839 INFO: The new image(s) can be found here:
5840 /home/stephano/testwic/test-201710091445-sdb.direct
5841
5842 The following build artifacts were used to create the image(s):
5843 ROOTFS_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
5844 BOOTIMG_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
5845 KERNEL_DIR: /home/stephano/build/master/build/tmp-glibc/deploy/images/qemux86
5846 NATIVE_SYSROOT: /home/stephano/build/master/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
5847
5848 INFO: The image(s) were created using OE kickstart file:
5849 /home/stephano/my_yocto/test.wks
5850
5851For this example,
5852:term:`MACHINE` did not have to be
5853specified in the ``local.conf`` file since the artifact is manually
5854specified.
5855
5856Using Wic to Manipulate an Image
5857~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5858
5859Wic image manipulation allows you to shorten turnaround time during
5860image development. For example, you can use Wic to delete the kernel
5861partition of a Wic image and then insert a newly built kernel. This
5862saves you time from having to rebuild the entire image each time you
5863modify the kernel.
5864
5865.. note::
5866
5867 In order to use Wic to manipulate a Wic image as in this example,
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005868 your development machine must have the ``mtools`` package installed.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005869
5870The following example examines the contents of the Wic image, deletes
5871the existing kernel, and then inserts a new kernel:
5872
58731. *List the Partitions:* Use the ``wic ls`` command to list all the
5874 partitions in the Wic image:
5875 ::
5876
5877 $ wic ls tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic
5878 Num Start End Size Fstype
5879 1 1048576 25041919 23993344 fat16
5880 2 25165824 72157183 46991360 ext4
5881
5882 The previous output shows two partitions in the
5883 ``core-image-minimal-qemux86.wic`` image.
5884
58852. *Examine a Particular Partition:* Use the ``wic ls`` command again
5886 but in a different form to examine a particular partition.
5887
5888 .. note::
5889
5890 You can get command usage on any Wic command using the following
5891 form:
5892 ::
5893
5894 $ wic help command
5895
5896
5897 For example, the following command shows you the various ways to
5898 use the
5899 wic ls
5900 command:
5901 ::
5902
5903 $ wic help ls
5904
5905
5906 The following command shows what is in Partition one:
5907 ::
5908
5909 $ wic ls tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1
5910 Volume in drive : is boot
5911 Volume Serial Number is E894-1809
5912 Directory for ::/
5913
5914 libcom32 c32 186500 2017-10-09 16:06
5915 libutil c32 24148 2017-10-09 16:06
5916 syslinux cfg 220 2017-10-09 16:06
5917 vesamenu c32 27104 2017-10-09 16:06
5918 vmlinuz 6904608 2017-10-09 16:06
5919 5 files 7 142 580 bytes
5920 16 582 656 bytes free
5921
5922 The previous output shows five files, with the
5923 ``vmlinuz`` being the kernel.
5924
5925 .. note::
5926
5927 If you see the following error, you need to update or create a
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005928 ``~/.mtoolsrc`` file and be sure to have the line "mtools_skip_check=1"
5929 in the file. Then, run the Wic command again:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005930 ::
5931
5932 ERROR: _exec_cmd: /usr/bin/mdir -i /tmp/wic-parttfokuwra ::/ returned '1' instead of 0
5933 output: Total number of sectors (47824) not a multiple of sectors per track (32)!
5934 Add mtools_skip_check=1 to your .mtoolsrc file to skip this test
5935
5936
59373. *Remove the Old Kernel:* Use the ``wic rm`` command to remove the
5938 ``vmlinuz`` file (kernel):
5939 ::
5940
5941 $ wic rm tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1/vmlinuz
5942
59434. *Add In the New Kernel:* Use the ``wic cp`` command to add the
5944 updated kernel to the Wic image. Depending on how you built your
5945 kernel, it could be in different places. If you used ``devtool`` and
5946 an SDK to build your kernel, it resides in the ``tmp/work`` directory
5947 of the extensible SDK. If you used ``make`` to build the kernel, the
5948 kernel will be in the ``workspace/sources`` area.
5949
5950 The following example assumes ``devtool`` was used to build the
5951 kernel:
5952 ::
5953
5954 cp ~/poky_sdk/tmp/work/qemux86-poky-linux/linux-yocto/4.12.12+git999-r0/linux-yocto-4.12.12+git999/arch/x86/boot/bzImage \
5955 ~/poky/build/tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1/vmlinuz
5956
5957 Once the new kernel is added back into the image, you can use the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005958 ``dd`` command or :ref:`bmaptool
Andrew Geissler09209ee2020-12-13 08:44:15 -06005959 <dev-manual/common-tasks:flashing images using \`\`bmaptool\`\`>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005960 to flash your wic image onto an SD card or USB stick and test your
5961 target.
5962
5963 .. note::
5964
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005965 Using ``bmaptool`` is generally 10 to 20 times faster than using ``dd``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005966
5967Flashing Images Using ``bmaptool``
5968==================================
5969
5970A fast and easy way to flash an image to a bootable device is to use
5971Bmaptool, which is integrated into the OpenEmbedded build system.
5972Bmaptool is a generic tool that creates a file's block map (bmap) and
5973then uses that map to copy the file. As compared to traditional tools
5974such as dd or cp, Bmaptool can copy (or flash) large files like raw
5975system image files much faster.
5976
5977.. note::
5978
5979 - If you are using Ubuntu or Debian distributions, you can install
5980 the ``bmap-tools`` package using the following command and then
5981 use the tool without specifying ``PATH`` even from the root
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005982 account:
5983 ::
5984
5985 $ sudo apt-get install bmap-tools
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005986
5987 - If you are unable to install the ``bmap-tools`` package, you will
5988 need to build Bmaptool before using it. Use the following command:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05005989 ::
5990
5991 $ bitbake bmap-tools-native
Andrew Geisslerc9f78652020-09-18 14:11:35 -05005992
5993Following, is an example that shows how to flash a Wic image. Realize
5994that while this example uses a Wic image, you can use Bmaptool to flash
5995any type of image. Use these steps to flash an image using Bmaptool:
5996
59971. *Update your local.conf File:* You need to have the following set
5998 in your ``local.conf`` file before building your image:
5999 ::
6000
6001 IMAGE_FSTYPES += "wic wic.bmap"
6002
60032. *Get Your Image:* Either have your image ready (pre-built with the
6004 :term:`IMAGE_FSTYPES`
6005 setting previously mentioned) or take the step to build the image:
6006 ::
6007
6008 $ bitbake image
6009
60103. *Flash the Device:* Flash the device with the image by using Bmaptool
6011 depending on your particular setup. The following commands assume the
6012 image resides in the Build Directory's ``deploy/images/`` area:
6013
6014 - If you have write access to the media, use this command form:
6015 ::
6016
6017 $ oe-run-native bmap-tools-native bmaptool copy build-directory/tmp/deploy/images/machine/image.wic /dev/sdX
6018
6019 - If you do not have write access to the media, set your permissions
6020 first and then use the same command form:
6021 ::
6022
6023 $ sudo chmod 666 /dev/sdX
6024 $ oe-run-native bmap-tools-native bmaptool copy build-directory/tmp/deploy/images/machine/image.wic /dev/sdX
6025
6026For help on the ``bmaptool`` command, use the following command:
6027::
6028
6029 $ bmaptool --help
6030
6031Making Images More Secure
6032=========================
6033
6034Security is of increasing concern for embedded devices. Consider the
6035issues and problems discussed in just this sampling of work found across
6036the Internet:
6037
6038- *"*\ `Security Risks of Embedded
6039 Systems <https://www.schneier.com/blog/archives/2014/01/security_risks_9.html>`__\ *"*
6040 by Bruce Schneier
6041
6042- *"*\ `Internet Census
6043 2012 <http://census2012.sourceforge.net/paper.html>`__\ *"* by Carna
6044 Botnet
6045
6046- *"*\ `Security Issues for Embedded
6047 Devices <http://elinux.org/images/6/6f/Security-issues.pdf>`__\ *"*
6048 by Jake Edge
6049
6050When securing your image is of concern, there are steps, tools, and
6051variables that you can consider to help you reach the security goals you
6052need for your particular device. Not all situations are identical when
6053it comes to making an image secure. Consequently, this section provides
6054some guidance and suggestions for consideration when you want to make
6055your image more secure.
6056
6057.. note::
6058
6059 Because the security requirements and risks are different for every
6060 type of device, this section cannot provide a complete reference on
6061 securing your custom OS. It is strongly recommended that you also
6062 consult other sources of information on embedded Linux system
6063 hardening and on security.
6064
6065General Considerations
6066----------------------
6067
6068General considerations exist that help you create more secure images.
6069You should consider the following suggestions to help make your device
6070more secure:
6071
6072- Scan additional code you are adding to the system (e.g. application
6073 code) by using static analysis tools. Look for buffer overflows and
6074 other potential security problems.
6075
6076- Pay particular attention to the security for any web-based
6077 administration interface.
6078
6079 Web interfaces typically need to perform administrative functions and
6080 tend to need to run with elevated privileges. Thus, the consequences
6081 resulting from the interface's security becoming compromised can be
6082 serious. Look for common web vulnerabilities such as
6083 cross-site-scripting (XSS), unvalidated inputs, and so forth.
6084
6085 As with system passwords, the default credentials for accessing a
6086 web-based interface should not be the same across all devices. This
6087 is particularly true if the interface is enabled by default as it can
6088 be assumed that many end-users will not change the credentials.
6089
6090- Ensure you can update the software on the device to mitigate
6091 vulnerabilities discovered in the future. This consideration
6092 especially applies when your device is network-enabled.
6093
6094- Ensure you remove or disable debugging functionality before producing
6095 the final image. For information on how to do this, see the
6096 "`Considerations Specific to the OpenEmbedded Build
6097 System <#considerations-specific-to-the-openembedded-build-system>`__"
6098 section.
6099
6100- Ensure you have no network services listening that are not needed.
6101
6102- Remove any software from the image that is not needed.
6103
6104- Enable hardware support for secure boot functionality when your
6105 device supports this functionality.
6106
6107Security Flags
6108--------------
6109
6110The Yocto Project has security flags that you can enable that help make
6111your build output more secure. The security flags are in the
6112``meta/conf/distro/include/security_flags.inc`` file in your
6113:term:`Source Directory` (e.g. ``poky``).
6114
6115.. note::
6116
6117 Depending on the recipe, certain security flags are enabled and
6118 disabled by default.
6119
6120Use the following line in your ``local.conf`` file or in your custom
6121distribution configuration file to enable the security compiler and
6122linker flags for your build:
6123::
6124
6125 require conf/distro/include/security_flags.inc
6126
6127Considerations Specific to the OpenEmbedded Build System
6128--------------------------------------------------------
6129
6130You can take some steps that are specific to the OpenEmbedded build
6131system to make your images more secure:
6132
6133- Ensure "debug-tweaks" is not one of your selected
6134 :term:`IMAGE_FEATURES`.
6135 When creating a new project, the default is to provide you with an
6136 initial ``local.conf`` file that enables this feature using the
6137 :term:`EXTRA_IMAGE_FEATURES`
6138 variable with the line:
6139 ::
6140
6141 EXTRA_IMAGE_FEATURES = "debug-tweaks"
6142
6143 To disable that feature, simply comment out that line in your
6144 ``local.conf`` file, or make sure ``IMAGE_FEATURES`` does not contain
6145 "debug-tweaks" before producing your final image. Among other things,
6146 leaving this in place sets the root password as blank, which makes
6147 logging in for debugging or inspection easy during development but
6148 also means anyone can easily log in during production.
6149
6150- It is possible to set a root password for the image and also to set
6151 passwords for any extra users you might add (e.g. administrative or
6152 service type users). When you set up passwords for multiple images or
6153 users, you should not duplicate passwords.
6154
6155 To set up passwords, use the
6156 :ref:`extrausers <ref-classes-extrausers>`
6157 class, which is the preferred method. For an example on how to set up
6158 both root and user passwords, see the
6159 ":ref:`extrausers.bbclass <ref-classes-extrausers>`"
6160 section.
6161
6162 .. note::
6163
6164 When adding extra user accounts or setting a root password, be
6165 cautious about setting the same password on every device. If you
6166 do this, and the password you have set is exposed, then every
6167 device is now potentially compromised. If you need this access but
6168 want to ensure security, consider setting a different, random
6169 password for each device. Typically, you do this as a separate
6170 step after you deploy the image onto the device.
6171
6172- Consider enabling a Mandatory Access Control (MAC) framework such as
6173 SMACK or SELinux and tuning it appropriately for your device's usage.
6174 You can find more information in the
Andrew Geissler09209ee2020-12-13 08:44:15 -06006175 :yocto_git:`meta-selinux </meta-selinux/>` layer.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006176
6177Tools for Hardening Your Image
6178------------------------------
6179
6180The Yocto Project provides tools for making your image more secure. You
6181can find these tools in the ``meta-security`` layer of the
6182:yocto_git:`Yocto Project Source Repositories <>`.
6183
6184Creating Your Own Distribution
6185==============================
6186
6187When you build an image using the Yocto Project and do not alter any
6188distribution :term:`Metadata`, you are
6189creating a Poky distribution. If you wish to gain more control over
6190package alternative selections, compile-time options, and other
6191low-level configurations, you can create your own distribution.
6192
6193To create your own distribution, the basic steps consist of creating
6194your own distribution layer, creating your own distribution
6195configuration file, and then adding any needed code and Metadata to the
6196layer. The following steps provide some more detail:
6197
6198- *Create a layer for your new distro:* Create your distribution layer
6199 so that you can keep your Metadata and code for the distribution
6200 separate. It is strongly recommended that you create and use your own
6201 layer for configuration and code. Using your own layer as compared to
6202 just placing configurations in a ``local.conf`` configuration file
6203 makes it easier to reproduce the same build configuration when using
6204 multiple build machines. See the
Andrew Geissler09209ee2020-12-13 08:44:15 -06006205 ":ref:`dev-manual/common-tasks:creating a general layer using the \`\`bitbake-layers\`\` script`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006206 section for information on how to quickly set up a layer.
6207
6208- *Create the distribution configuration file:* The distribution
6209 configuration file needs to be created in the ``conf/distro``
6210 directory of your layer. You need to name it using your distribution
6211 name (e.g. ``mydistro.conf``).
6212
6213 .. note::
6214
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006215 The :term:`DISTRO` variable in your ``local.conf`` file determines the
6216 name of your distribution.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006217
6218 You can split out parts of your configuration file into include files
6219 and then "require" them from within your distribution configuration
6220 file. Be sure to place the include files in the
6221 ``conf/distro/include`` directory of your layer. A common example
6222 usage of include files would be to separate out the selection of
6223 desired version and revisions for individual recipes.
6224
6225 Your configuration file needs to set the following required
6226 variables:
6227
6228 - :term:`DISTRO_NAME`
6229
6230 - :term:`DISTRO_VERSION`
6231
6232 These following variables are optional and you typically set them
6233 from the distribution configuration file:
6234
6235 - :term:`DISTRO_FEATURES`
6236
6237 - :term:`DISTRO_EXTRA_RDEPENDS`
6238
6239 - :term:`DISTRO_EXTRA_RRECOMMENDS`
6240
6241 - :term:`TCLIBC`
6242
6243 .. tip::
6244
6245 If you want to base your distribution configuration file on the
6246 very basic configuration from OE-Core, you can use
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006247 ``conf/distro/defaultsetup.conf`` as a reference and just include
6248 variables that differ as compared to ``defaultsetup.conf``.
6249 Alternatively, you can create a distribution configuration file
6250 from scratch using the ``defaultsetup.conf`` file or configuration files
6251 from other distributions such as Poky or Angstrom as references.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006252
6253- *Provide miscellaneous variables:* Be sure to define any other
6254 variables for which you want to create a default or enforce as part
6255 of the distribution configuration. You can include nearly any
6256 variable from the ``local.conf`` file. The variables you use are not
6257 limited to the list in the previous bulleted item.
6258
6259- *Point to Your distribution configuration file:* In your
6260 ``local.conf`` file in the :term:`Build Directory`,
6261 set your
6262 :term:`DISTRO` variable to point to
6263 your distribution's configuration file. For example, if your
6264 distribution's configuration file is named ``mydistro.conf``, then
6265 you point to it as follows:
6266 ::
6267
6268 DISTRO = "mydistro"
6269
6270- *Add more to the layer if necessary:* Use your layer to hold other
6271 information needed for the distribution:
6272
6273 - Add recipes for installing distro-specific configuration files
6274 that are not already installed by another recipe. If you have
6275 distro-specific configuration files that are included by an
6276 existing recipe, you should add an append file (``.bbappend``) for
6277 those. For general information and recommendations on how to add
6278 recipes to your layer, see the "`Creating Your Own
6279 Layer <#creating-your-own-layer>`__" and "`Following Best
6280 Practices When Creating
6281 Layers <#best-practices-to-follow-when-creating-layers>`__"
6282 sections.
6283
6284 - Add any image recipes that are specific to your distribution.
6285
6286 - Add a ``psplash`` append file for a branded splash screen. For
6287 information on append files, see the "`Using .bbappend Files in
6288 Your Layer <#using-bbappend-files>`__" section.
6289
6290 - Add any other append files to make custom changes that are
6291 specific to individual recipes.
6292
6293Creating a Custom Template Configuration Directory
6294==================================================
6295
6296If you are producing your own customized version of the build system for
6297use by other users, you might want to customize the message shown by the
6298setup script or you might want to change the template configuration
6299files (i.e. ``local.conf`` and ``bblayers.conf``) that are created in a
6300new build directory.
6301
6302The OpenEmbedded build system uses the environment variable
6303``TEMPLATECONF`` to locate the directory from which it gathers
6304configuration information that ultimately ends up in the
6305:term:`Build Directory` ``conf`` directory.
6306By default, ``TEMPLATECONF`` is set as follows in the ``poky``
6307repository:
6308::
6309
6310 TEMPLATECONF=${TEMPLATECONF:-meta-poky/conf}
6311
6312This is the
6313directory used by the build system to find templates from which to build
6314some key configuration files. If you look at this directory, you will
6315see the ``bblayers.conf.sample``, ``local.conf.sample``, and
6316``conf-notes.txt`` files. The build system uses these files to form the
6317respective ``bblayers.conf`` file, ``local.conf`` file, and display the
6318list of BitBake targets when running the setup script.
6319
6320To override these default configuration files with configurations you
6321want used within every new Build Directory, simply set the
6322``TEMPLATECONF`` variable to your directory. The ``TEMPLATECONF``
6323variable is set in the ``.templateconf`` file, which is in the top-level
6324:term:`Source Directory` folder
6325(e.g. ``poky``). Edit the ``.templateconf`` so that it can locate your
6326directory.
6327
6328Best practices dictate that you should keep your template configuration
6329directory in your custom distribution layer. For example, suppose you
6330have a layer named ``meta-mylayer`` located in your home directory and
6331you want your template configuration directory named ``myconf``.
6332Changing the ``.templateconf`` as follows causes the OpenEmbedded build
6333system to look in your directory and base its configuration files on the
6334``*.sample`` configuration files it finds. The final configuration files
6335(i.e. ``local.conf`` and ``bblayers.conf`` ultimately still end up in
6336your Build Directory, but they are based on your ``*.sample`` files.
6337::
6338
6339 TEMPLATECONF=${TEMPLATECONF:-meta-mylayer/myconf}
6340
6341Aside from the ``*.sample`` configuration files, the ``conf-notes.txt``
6342also resides in the default ``meta-poky/conf`` directory. The script
6343that sets up the build environment (i.e.
6344:ref:`structure-core-script`) uses this file to
6345display BitBake targets as part of the script output. Customizing this
6346``conf-notes.txt`` file is a good way to make sure your list of custom
6347targets appears as part of the script's output.
6348
6349Here is the default list of targets displayed as a result of running
6350either of the setup scripts:
6351::
6352
6353 You can now run 'bitbake <target>'
6354
6355 Common targets are:
6356 core-image-minimal
6357 core-image-sato
6358 meta-toolchain
6359 meta-ide-support
6360
6361Changing the listed common targets is as easy as editing your version of
6362``conf-notes.txt`` in your custom template configuration directory and
6363making sure you have ``TEMPLATECONF`` set to your directory.
6364
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006365Conserving Disk Space During Builds
6366===================================
6367
6368To help conserve disk space during builds, you can add the following
6369statement to your project's ``local.conf`` configuration file found in
6370the :term:`Build Directory`:
6371::
6372
6373 INHERIT += "rm_work"
6374
6375Adding this statement deletes the work directory used for
6376building a recipe once the recipe is built. For more information on
6377"rm_work", see the
6378:ref:`rm_work <ref-classes-rm-work>` class in the
6379Yocto Project Reference Manual.
6380
6381Working with Packages
6382=====================
6383
6384This section describes a few tasks that involve packages:
6385
6386- `Excluding packages from an
6387 image <#excluding-packages-from-an-image>`__
6388
6389- `Incrementing a binary package
6390 version <#incrementing-a-binary-package-version>`__
6391
6392- `Handling optional module
6393 packaging <#handling-optional-module-packaging>`__
6394
6395- `Using runtime package
6396 management <#using-runtime-package-management>`__
6397
6398- `Generating and using signed
6399 packages <#generating-and-using-signed-packages>`__
6400
6401- `Setting up and running package test
6402 (ptest) <#testing-packages-with-ptest>`__
6403
6404- `Creating node package manager (NPM)
6405 packages <#creating-node-package-manager-npm-packages>`__
6406
6407- `Adding custom metadata to
6408 packages <#adding-custom-metadata-to-packages>`__
6409
6410Excluding Packages from an Image
6411--------------------------------
6412
6413You might find it necessary to prevent specific packages from being
6414installed into an image. If so, you can use several variables to direct
6415the build system to essentially ignore installing recommended packages
6416or to not install a package at all.
6417
6418The following list introduces variables you can use to prevent packages
6419from being installed into your image. Each of these variables only works
6420with IPK and RPM package types. Support for Debian packages does not
6421exist. Also, you can use these variables from your ``local.conf`` file
6422or attach them to a specific image recipe by using a recipe name
6423override. For more detail on the variables, see the descriptions in the
6424Yocto Project Reference Manual's glossary chapter.
6425
6426- :term:`BAD_RECOMMENDATIONS`:
6427 Use this variable to specify "recommended-only" packages that you do
6428 not want installed.
6429
6430- :term:`NO_RECOMMENDATIONS`:
6431 Use this variable to prevent all "recommended-only" packages from
6432 being installed.
6433
6434- :term:`PACKAGE_EXCLUDE`:
6435 Use this variable to prevent specific packages from being installed
6436 regardless of whether they are "recommended-only" or not. You need to
6437 realize that the build process could fail with an error when you
6438 prevent the installation of a package whose presence is required by
6439 an installed package.
6440
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006441Incrementing a Package Version
6442------------------------------
6443
6444This section provides some background on how binary package versioning
6445is accomplished and presents some of the services, variables, and
6446terminology involved.
6447
6448In order to understand binary package versioning, you need to consider
6449the following:
6450
6451- Binary Package: The binary package that is eventually built and
6452 installed into an image.
6453
6454- Binary Package Version: The binary package version is composed of two
6455 components - a version and a revision.
6456
6457 .. note::
6458
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006459 Technically, a third component, the "epoch" (i.e. :term:`PE`) is involved
6460 but this discussion for the most part ignores ``PE``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006461
6462 The version and revision are taken from the
6463 :term:`PV` and
6464 :term:`PR` variables, respectively.
6465
6466- ``PV``: The recipe version. ``PV`` represents the version of the
6467 software being packaged. Do not confuse ``PV`` with the binary
6468 package version.
6469
6470- ``PR``: The recipe revision.
6471
6472- :term:`SRCPV`: The OpenEmbedded
6473 build system uses this string to help define the value of ``PV`` when
6474 the source code revision needs to be included in it.
6475
Andrew Geissler09209ee2020-12-13 08:44:15 -06006476- :yocto_wiki:`PR Service </PR_Service>`: A
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006477 network-based service that helps automate keeping package feeds
6478 compatible with existing package manager applications such as RPM,
6479 APT, and OPKG.
6480
6481Whenever the binary package content changes, the binary package version
6482must change. Changing the binary package version is accomplished by
6483changing or "bumping" the ``PR`` and/or ``PV`` values. Increasing these
6484values occurs one of two ways:
6485
6486- Automatically using a Package Revision Service (PR Service).
6487
6488- Manually incrementing the ``PR`` and/or ``PV`` variables.
6489
6490Given a primary challenge of any build system and its users is how to
6491maintain a package feed that is compatible with existing package manager
6492applications such as RPM, APT, and OPKG, using an automated system is
6493much preferred over a manual system. In either system, the main
6494requirement is that binary package version numbering increases in a
6495linear fashion and that a number of version components exist that
6496support that linear progression. For information on how to ensure
6497package revisioning remains linear, see the "`Automatically Incrementing
6498a Binary Package Revision
6499Number <#automatically-incrementing-a-binary-package-revision-number>`__"
6500section.
6501
6502The following three sections provide related information on the PR
6503Service, the manual method for "bumping" ``PR`` and/or ``PV``, and on
6504how to ensure binary package revisioning remains linear.
6505
6506Working With a PR Service
6507~~~~~~~~~~~~~~~~~~~~~~~~~
6508
6509As mentioned, attempting to maintain revision numbers in the
6510:term:`Metadata` is error prone, inaccurate,
6511and causes problems for people submitting recipes. Conversely, the PR
6512Service automatically generates increasing numbers, particularly the
6513revision field, which removes the human element.
6514
6515.. note::
6516
6517 For additional information on using a PR Service, you can see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06006518 :yocto_wiki:`PR Service </PR_Service>` wiki page.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006519
6520The Yocto Project uses variables in order of decreasing priority to
6521facilitate revision numbering (i.e.
6522:term:`PE`,
6523:term:`PV`, and
6524:term:`PR` for epoch, version, and
6525revision, respectively). The values are highly dependent on the policies
6526and procedures of a given distribution and package feed.
6527
6528Because the OpenEmbedded build system uses
Andrew Geissler09209ee2020-12-13 08:44:15 -06006529":ref:`signatures <overview-manual/concepts:checksums (signatures)>`", which are
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006530unique to a given build, the build system knows when to rebuild
6531packages. All the inputs into a given task are represented by a
6532signature, which can trigger a rebuild when different. Thus, the build
6533system itself does not rely on the ``PR``, ``PV``, and ``PE`` numbers to
6534trigger a rebuild. The signatures, however, can be used to generate
6535these values.
6536
6537The PR Service works with both ``OEBasic`` and ``OEBasicHash``
6538generators. The value of ``PR`` bumps when the checksum changes and the
6539different generator mechanisms change signatures under different
6540circumstances.
6541
6542As implemented, the build system includes values from the PR Service
6543into the ``PR`` field as an addition using the form "``.x``" so ``r0``
6544becomes ``r0.1``, ``r0.2`` and so forth. This scheme allows existing
6545``PR`` values to be used for whatever reasons, which include manual
6546``PR`` bumps, should it be necessary.
6547
6548By default, the PR Service is not enabled or running. Thus, the packages
6549generated are just "self consistent". The build system adds and removes
6550packages and there are no guarantees about upgrade paths but images will
6551be consistent and correct with the latest changes.
6552
6553The simplest form for a PR Service is for it to exist for a single host
6554development system that builds the package feed (building system). For
6555this scenario, you can enable a local PR Service by setting
6556:term:`PRSERV_HOST` in your
6557``local.conf`` file in the :term:`Build Directory`:
6558::
6559
6560 PRSERV_HOST = "localhost:0"
6561
6562Once the service is started, packages will automatically
6563get increasing ``PR`` values and BitBake takes care of starting and
6564stopping the server.
6565
6566If you have a more complex setup where multiple host development systems
6567work against a common, shared package feed, you have a single PR Service
6568running and it is connected to each building system. For this scenario,
6569you need to start the PR Service using the ``bitbake-prserv`` command:
6570::
6571
6572 bitbake-prserv --host ip --port port --start
6573
6574In addition to
6575hand-starting the service, you need to update the ``local.conf`` file of
6576each building system as described earlier so each system points to the
6577server and port.
6578
6579It is also recommended you use build history, which adds some sanity
6580checks to binary package versions, in conjunction with the server that
6581is running the PR Service. To enable build history, add the following to
6582each building system's ``local.conf`` file:
6583::
6584
6585 # It is recommended to activate "buildhistory" for testing the PR service
6586 INHERIT += "buildhistory"
6587 BUILDHISTORY_COMMIT = "1"
6588
6589For information on build
6590history, see the "`Maintaining Build Output
6591Quality <#maintaining-build-output-quality>`__" section.
6592
6593.. note::
6594
6595 The OpenEmbedded build system does not maintain ``PR`` information as
6596 part of the shared state (sstate) packages. If you maintain an sstate
6597 feed, its expected that either all your building systems that
6598 contribute to the sstate feed use a shared PR Service, or you do not
6599 run a PR Service on any of your building systems. Having some systems
6600 use a PR Service while others do not leads to obvious problems.
6601
6602 For more information on shared state, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06006603 ":ref:`overview-manual/concepts:shared state cache`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006604 section in the Yocto Project Overview and Concepts Manual.
6605
6606Manually Bumping PR
6607~~~~~~~~~~~~~~~~~~~
6608
6609The alternative to setting up a PR Service is to manually "bump" the
6610:term:`PR` variable.
6611
6612If a committed change results in changing the package output, then the
6613value of the PR variable needs to be increased (or "bumped") as part of
6614that commit. For new recipes you should add the ``PR`` variable and set
6615its initial value equal to "r0", which is the default. Even though the
6616default value is "r0", the practice of adding it to a new recipe makes
6617it harder to forget to bump the variable when you make changes to the
6618recipe in future.
6619
6620If you are sharing a common ``.inc`` file with multiple recipes, you can
6621also use the ``INC_PR`` variable to ensure that the recipes sharing the
6622``.inc`` file are rebuilt when the ``.inc`` file itself is changed. The
6623``.inc`` file must set ``INC_PR`` (initially to "r0"), and all recipes
6624referring to it should set ``PR`` to "${INC_PR}.0" initially,
6625incrementing the last number when the recipe is changed. If the ``.inc``
6626file is changed then its ``INC_PR`` should be incremented.
6627
6628When upgrading the version of a binary package, assuming the ``PV``
6629changes, the ``PR`` variable should be reset to "r0" (or "${INC_PR}.0"
6630if you are using ``INC_PR``).
6631
6632Usually, version increases occur only to binary packages. However, if
6633for some reason ``PV`` changes but does not increase, you can increase
6634the ``PE`` variable (Package Epoch). The ``PE`` variable defaults to
6635"0".
6636
6637Binary package version numbering strives to follow the `Debian Version
6638Field Policy
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006639Guidelines <https://www.debian.org/doc/debian-policy/ch-controlfields.html>`__.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006640These guidelines define how versions are compared and what "increasing"
6641a version means.
6642
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006643Automatically Incrementing a Package Version Number
6644~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
6645
6646When fetching a repository, BitBake uses the
6647:term:`SRCREV` variable to determine
6648the specific source code revision from which to build. You set the
6649``SRCREV`` variable to
6650:term:`AUTOREV` to cause the
6651OpenEmbedded build system to automatically use the latest revision of
6652the software:
6653::
6654
6655 SRCREV = "${AUTOREV}"
6656
6657Furthermore, you need to reference ``SRCPV`` in ``PV`` in order to
6658automatically update the version whenever the revision of the source
6659code changes. Here is an example:
6660::
6661
6662 PV = "1.0+git${SRCPV}"
6663
6664The OpenEmbedded build system substitutes ``SRCPV`` with the following:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006665
6666.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006667
6668 AUTOINC+source_code_revision
6669
6670The build system replaces the ``AUTOINC``
6671with a number. The number used depends on the state of the PR Service:
6672
6673- If PR Service is enabled, the build system increments the number,
6674 which is similar to the behavior of
6675 :term:`PR`. This behavior results in
6676 linearly increasing package versions, which is desirable. Here is an
6677 example:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006678
6679 .. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006680
6681 hello-world-git_0.0+git0+b6558dd387-r0.0_armv7a-neon.ipk
6682 hello-world-git_0.0+git1+dd2f5c3565-r0.0_armv7a-neon.ipk
6683
6684- If PR Service is not enabled, the build system replaces the
6685 ``AUTOINC`` placeholder with zero (i.e. "0"). This results in
6686 changing the package version since the source revision is included.
6687 However, package versions are not increased linearly. Here is an
6688 example:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006689
6690 .. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006691
6692 hello-world-git_0.0+git0+b6558dd387-r0.0_armv7a-neon.ipk
6693 hello-world-git_0.0+git0+dd2f5c3565-r0.0_armv7a-neon.ipk
6694
6695In summary, the OpenEmbedded build system does not track the history of
6696binary package versions for this purpose. ``AUTOINC``, in this case, is
6697comparable to ``PR``. If PR server is not enabled, ``AUTOINC`` in the
6698package version is simply replaced by "0". If PR server is enabled, the
6699build system keeps track of the package versions and bumps the number
6700when the package revision changes.
6701
6702Handling Optional Module Packaging
6703----------------------------------
6704
6705Many pieces of software split functionality into optional modules (or
6706plugins) and the plugins that are built might depend on configuration
6707options. To avoid having to duplicate the logic that determines what
6708modules are available in your recipe or to avoid having to package each
6709module by hand, the OpenEmbedded build system provides functionality to
6710handle module packaging dynamically.
6711
6712To handle optional module packaging, you need to do two things:
6713
6714- Ensure the module packaging is actually done.
6715
6716- Ensure that any dependencies on optional modules from other recipes
6717 are satisfied by your recipe.
6718
6719Making Sure the Packaging is Done
6720~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
6721
6722To ensure the module packaging actually gets done, you use the
6723``do_split_packages`` function within the ``populate_packages`` Python
6724function in your recipe. The ``do_split_packages`` function searches for
6725a pattern of files or directories under a specified path and creates a
6726package for each one it finds by appending to the
6727:term:`PACKAGES` variable and
6728setting the appropriate values for ``FILES_packagename``,
6729``RDEPENDS_packagename``, ``DESCRIPTION_packagename``, and so forth.
6730Here is an example from the ``lighttpd`` recipe:
6731::
6732
6733 python populate_packages_prepend () {
6734 lighttpd_libdir = d.expand('${libdir}')
6735 do_split_packages(d, lighttpd_libdir, '^mod_(.*).so$',
6736 'lighttpd-module-%s', 'Lighttpd module for %s',
6737 extra_depends='')
6738 }
6739
6740The previous example specifies a number of things in the call to
6741``do_split_packages``.
6742
6743- A directory within the files installed by your recipe through
6744 ``do_install`` in which to search.
6745
6746- A regular expression used to match module files in that directory. In
6747 the example, note the parentheses () that mark the part of the
6748 expression from which the module name should be derived.
6749
6750- A pattern to use for the package names.
6751
6752- A description for each package.
6753
6754- An empty string for ``extra_depends``, which disables the default
6755 dependency on the main ``lighttpd`` package. Thus, if a file in
6756 ``${libdir}`` called ``mod_alias.so`` is found, a package called
6757 ``lighttpd-module-alias`` is created for it and the
6758 :term:`DESCRIPTION` is set to
6759 "Lighttpd module for alias".
6760
6761Often, packaging modules is as simple as the previous example. However,
6762more advanced options exist that you can use within
6763``do_split_packages`` to modify its behavior. And, if you need to, you
6764can add more logic by specifying a hook function that is called for each
6765package. It is also perfectly acceptable to call ``do_split_packages``
6766multiple times if you have more than one set of modules to package.
6767
6768For more examples that show how to use ``do_split_packages``, see the
6769``connman.inc`` file in the ``meta/recipes-connectivity/connman/``
Andrew Geissler09209ee2020-12-13 08:44:15 -06006770directory of the ``poky`` :ref:`source repository <overview-manual/development-environment:yocto project source repositories>`. You can
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006771also find examples in ``meta/classes/kernel.bbclass``.
6772
6773Following is a reference that shows ``do_split_packages`` mandatory and
6774optional arguments:
6775::
6776
6777 Mandatory arguments
6778
6779 root
6780 The path in which to search
6781 file_regex
6782 Regular expression to match searched files.
6783 Use parentheses () to mark the part of this
6784 expression that should be used to derive the
6785 module name (to be substituted where %s is
6786 used in other function arguments as noted below)
6787 output_pattern
6788 Pattern to use for the package names. Must
6789 include %s.
6790 description
6791 Description to set for each package. Must
6792 include %s.
6793
6794 Optional arguments
6795
6796 postinst
6797 Postinstall script to use for all packages
6798 (as a string)
6799 recursive
6800 True to perform a recursive search - default
6801 False
6802 hook
6803 A hook function to be called for every match.
6804 The function will be called with the following
6805 arguments (in the order listed):
6806
6807 f
6808 Full path to the file/directory match
6809 pkg
6810 The package name
6811 file_regex
6812 As above
6813 output_pattern
6814 As above
6815 modulename
6816 The module name derived using file_regex
Andrew Geissler4c19ea12020-10-27 13:52:24 -05006817 extra_depends
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006818 Extra runtime dependencies (RDEPENDS) to be
6819 set for all packages. The default value of None
6820 causes a dependency on the main package
6821 (${PN}) - if you do not want this, pass empty
6822 string '' for this parameter.
6823 aux_files_pattern
6824 Extra item(s) to be added to FILES for each
6825 package. Can be a single string item or a list
6826 of strings for multiple items. Must include %s.
6827 postrm
6828 postrm script to use for all packages (as a
6829 string)
6830 allow_dirs
6831 True to allow directories to be matched -
6832 default False
6833 prepend
6834 If True, prepend created packages to PACKAGES
6835 instead of the default False which appends them
6836 match_path
6837 match file_regex on the whole relative path to
6838 the root rather than just the file name
6839 aux_files_pattern_verbatim
6840 Extra item(s) to be added to FILES for each
6841 package, using the actual derived module name
6842 rather than converting it to something legal
6843 for a package name. Can be a single string item
6844 or a list of strings for multiple items. Must
6845 include %s.
6846 allow_links
6847 True to allow symlinks to be matched - default
6848 False
6849 summary
6850 Summary to set for each package. Must include %s;
6851 defaults to description if not set.
6852
6853
6854
6855Satisfying Dependencies
6856~~~~~~~~~~~~~~~~~~~~~~~
6857
6858The second part for handling optional module packaging is to ensure that
6859any dependencies on optional modules from other recipes are satisfied by
6860your recipe. You can be sure these dependencies are satisfied by using
6861the :term:`PACKAGES_DYNAMIC`
6862variable. Here is an example that continues with the ``lighttpd`` recipe
6863shown earlier:
6864::
6865
6866 PACKAGES_DYNAMIC = "lighttpd-module-.*"
6867
6868The name
6869specified in the regular expression can of course be anything. In this
6870example, it is ``lighttpd-module-`` and is specified as the prefix to
6871ensure that any :term:`RDEPENDS` and
6872:term:`RRECOMMENDS` on a package
6873name starting with the prefix are satisfied during build time. If you
6874are using ``do_split_packages`` as described in the previous section,
6875the value you put in ``PACKAGES_DYNAMIC`` should correspond to the name
6876pattern specified in the call to ``do_split_packages``.
6877
6878Using Runtime Package Management
6879--------------------------------
6880
6881During a build, BitBake always transforms a recipe into one or more
6882packages. For example, BitBake takes the ``bash`` recipe and produces a
6883number of packages (e.g. ``bash``, ``bash-bashbug``,
6884``bash-completion``, ``bash-completion-dbg``, ``bash-completion-dev``,
6885``bash-completion-extra``, ``bash-dbg``, and so forth). Not all
6886generated packages are included in an image.
6887
6888In several situations, you might need to update, add, remove, or query
6889the packages on a target device at runtime (i.e. without having to
6890generate a new image). Examples of such situations include:
6891
6892- You want to provide in-the-field updates to deployed devices (e.g.
6893 security updates).
6894
6895- You want to have a fast turn-around development cycle for one or more
6896 applications that run on your device.
6897
6898- You want to temporarily install the "debug" packages of various
6899 applications on your device so that debugging can be greatly improved
6900 by allowing access to symbols and source debugging.
6901
6902- You want to deploy a more minimal package selection of your device
6903 but allow in-the-field updates to add a larger selection for
6904 customization.
6905
6906In all these situations, you have something similar to a more
6907traditional Linux distribution in that in-field devices are able to
6908receive pre-compiled packages from a server for installation or update.
6909Being able to install these packages on a running, in-field device is
6910what is termed "runtime package management".
6911
6912In order to use runtime package management, you need a host or server
6913machine that serves up the pre-compiled packages plus the required
6914metadata. You also need package manipulation tools on the target. The
6915build machine is a likely candidate to act as the server. However, that
6916machine does not necessarily have to be the package server. The build
6917machine could push its artifacts to another machine that acts as the
6918server (e.g. Internet-facing). In fact, doing so is advantageous for a
6919production environment as getting the packages away from the development
6920system's build directory prevents accidental overwrites.
6921
6922A simple build that targets just one device produces more than one
6923package database. In other words, the packages produced by a build are
6924separated out into a couple of different package groupings based on
6925criteria such as the target's CPU architecture, the target board, or the
6926C library used on the target. For example, a build targeting the
6927``qemux86`` device produces the following three package databases:
6928``noarch``, ``i586``, and ``qemux86``. If you wanted your ``qemux86``
6929device to be aware of all the packages that were available to it, you
6930would need to point it to each of these databases individually. In a
6931similar way, a traditional Linux distribution usually is configured to
6932be aware of a number of software repositories from which it retrieves
6933packages.
6934
6935Using runtime package management is completely optional and not required
6936for a successful build or deployment in any way. But if you want to make
6937use of runtime package management, you need to do a couple things above
6938and beyond the basics. The remainder of this section describes what you
6939need to do.
6940
Andrew Geisslerc9f78652020-09-18 14:11:35 -05006941Build Considerations
6942~~~~~~~~~~~~~~~~~~~~
6943
6944This section describes build considerations of which you need to be
6945aware in order to provide support for runtime package management.
6946
6947When BitBake generates packages, it needs to know what format or formats
6948to use. In your configuration, you use the
6949:term:`PACKAGE_CLASSES`
6950variable to specify the format:
6951
69521. Open the ``local.conf`` file inside your
6953 :term:`Build Directory` (e.g.
6954 ``~/poky/build/conf/local.conf``).
6955
69562. Select the desired package format as follows:
6957 ::
6958
6959 PACKAGE_CLASSES ?= "package_packageformat"
6960
6961 where packageformat can be "ipk", "rpm",
6962 "deb", or "tar" which are the supported package formats.
6963
6964 .. note::
6965
6966 Because the Yocto Project supports four different package formats,
6967 you can set the variable with more than one argument. However, the
6968 OpenEmbedded build system only uses the first argument when
6969 creating an image or Software Development Kit (SDK).
6970
6971If you would like your image to start off with a basic package database
6972containing the packages in your current build as well as to have the
6973relevant tools available on the target for runtime package management,
6974you can include "package-management" in the
6975:term:`IMAGE_FEATURES`
6976variable. Including "package-management" in this configuration variable
6977ensures that when the image is assembled for your target, the image
6978includes the currently-known package databases as well as the
6979target-specific tools required for runtime package management to be
6980performed on the target. However, this is not strictly necessary. You
6981could start your image off without any databases but only include the
6982required on-target package tool(s). As an example, you could include
6983"opkg" in your
6984:term:`IMAGE_INSTALL` variable
6985if you are using the IPK package format. You can then initialize your
6986target's package database(s) later once your image is up and running.
6987
6988Whenever you perform any sort of build step that can potentially
6989generate a package or modify existing package, it is always a good idea
6990to re-generate the package index after the build by using the following
6991command:
6992::
6993
6994 $ bitbake package-index
6995
6996It might be tempting to build the
6997package and the package index at the same time with a command such as
6998the following:
6999::
7000
7001 $ bitbake some-package package-index
7002
7003Do not do this as
7004BitBake does not schedule the package index for after the completion of
7005the package you are building. Consequently, you cannot be sure of the
7006package index including information for the package you just built.
7007Thus, be sure to run the package update step separately after building
7008any packages.
7009
7010You can use the
7011:term:`PACKAGE_FEED_ARCHS`,
7012:term:`PACKAGE_FEED_BASE_PATHS`,
7013and
7014:term:`PACKAGE_FEED_URIS`
7015variables to pre-configure target images to use a package feed. If you
7016do not define these variables, then manual steps as described in the
7017subsequent sections are necessary to configure the target. You should
7018set these variables before building the image in order to produce a
7019correctly configured image.
7020
7021When your build is complete, your packages reside in the
7022``${TMPDIR}/deploy/packageformat`` directory. For example, if
7023``${``\ :term:`TMPDIR`\ ``}`` is
7024``tmp`` and your selected package type is RPM, then your RPM packages
7025are available in ``tmp/deploy/rpm``.
7026
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007027Host or Server Machine Setup
7028~~~~~~~~~~~~~~~~~~~~~~~~~~~~
7029
7030Although other protocols are possible, a server using HTTP typically
7031serves packages. If you want to use HTTP, then set up and configure a
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007032web server such as Apache 2, lighttpd, or Python web server on the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007033machine serving the packages.
7034
7035To keep things simple, this section describes how to set up a
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007036Python web server to share package feeds from the developer's
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007037machine. Although this server might not be the best for a production
7038environment, the setup is simple and straight forward. Should you want
7039to use a different server more suited for production (e.g. Apache 2,
7040Lighttpd, or Nginx), take the appropriate steps to do so.
7041
7042From within the build directory where you have built an image based on
7043your packaging choice (i.e. the
7044:term:`PACKAGE_CLASSES`
7045setting), simply start the server. The following example assumes a build
7046directory of ``~/poky/build/tmp/deploy/rpm`` and a ``PACKAGE_CLASSES``
7047setting of "package_rpm":
7048::
7049
7050 $ cd ~/poky/build/tmp/deploy/rpm
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007051 $ python3 -m http.server
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007052
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007053Target Setup
7054~~~~~~~~~~~~
7055
7056Setting up the target differs depending on the package management
7057system. This section provides information for RPM, IPK, and DEB.
7058
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007059Using RPM
7060^^^^^^^^^
7061
7062The `Dandified Packaging
7063Tool <https://en.wikipedia.org/wiki/DNF_(software)>`__ (DNF) performs
7064runtime package management of RPM packages. In order to use DNF for
7065runtime package management, you must perform an initial setup on the
7066target machine for cases where the ``PACKAGE_FEED_*`` variables were not
7067set as part of the image that is running on the target. This means if
7068you built your image and did not not use these variables as part of the
7069build and your image is now running on the target, you need to perform
7070the steps in this section if you want to use runtime package management.
7071
7072.. note::
7073
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007074 For information on the ``PACKAGE_FEED_*`` variables, see
7075 :term:`PACKAGE_FEED_ARCHS`, :term:`PACKAGE_FEED_BASE_PATHS`, and
7076 :term:`PACKAGE_FEED_URIS` in the Yocto Project Reference Manual variables
7077 glossary.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007078
7079On the target, you must inform DNF that package databases are available.
7080You do this by creating a file named
7081``/etc/yum.repos.d/oe-packages.repo`` and defining the ``oe-packages``.
7082
7083As an example, assume the target is able to use the following package
7084databases: ``all``, ``i586``, and ``qemux86`` from a server named
7085``my.server``. The specifics for setting up the web server are up to
7086you. The critical requirement is that the URIs in the target repository
7087configuration point to the correct remote location for the feeds.
7088
7089.. note::
7090
7091 For development purposes, you can point the web server to the build
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007092 system's ``deploy`` directory. However, for production use, it is better to
7093 copy the package directories to a location outside of the build area and use
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007094 that location. Doing so avoids situations where the build system
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007095 overwrites or changes the ``deploy`` directory.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007096
7097When telling DNF where to look for the package databases, you must
7098declare individual locations per architecture or a single location used
7099for all architectures. You cannot do both:
7100
7101- *Create an Explicit List of Architectures:* Define individual base
7102 URLs to identify where each package database is located:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007103
7104 .. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007105
7106 [oe-packages]
7107 baseurl=http://my.server/rpm/i586 http://my.server/rpm/qemux86 http://my.server/rpm/all
7108
7109 This example
7110 informs DNF about individual package databases for all three
7111 architectures.
7112
7113- *Create a Single (Full) Package Index:* Define a single base URL that
7114 identifies where a full package database is located:
7115 ::
7116
7117 [oe-packages]
7118 baseurl=http://my.server/rpm
7119
7120 This example informs DNF about a single
7121 package database that contains all the package index information for
7122 all supported architectures.
7123
7124Once you have informed DNF where to find the package databases, you need
7125to fetch them:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007126
7127.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007128
7129 # dnf makecache
7130
7131DNF is now able to find, install, and
7132upgrade packages from the specified repository or repositories.
7133
7134.. note::
7135
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007136 See the `DNF documentation <https://dnf.readthedocs.io/en/latest/>`__ for
7137 additional information.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007138
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007139Using IPK
7140^^^^^^^^^
7141
7142The ``opkg`` application performs runtime package management of IPK
7143packages. You must perform an initial setup for ``opkg`` on the target
7144machine if the
7145:term:`PACKAGE_FEED_ARCHS`,
7146:term:`PACKAGE_FEED_BASE_PATHS`,
7147and
7148:term:`PACKAGE_FEED_URIS`
7149variables have not been set or the target image was built before the
7150variables were set.
7151
7152The ``opkg`` application uses configuration files to find available
7153package databases. Thus, you need to create a configuration file inside
7154the ``/etc/opkg/`` direction, which informs ``opkg`` of any repository
7155you want to use.
7156
7157As an example, suppose you are serving packages from a ``ipk/``
7158directory containing the ``i586``, ``all``, and ``qemux86`` databases
7159through an HTTP server named ``my.server``. On the target, create a
7160configuration file (e.g. ``my_repo.conf``) inside the ``/etc/opkg/``
7161directory containing the following:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007162
7163.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007164
7165 src/gz all http://my.server/ipk/all
7166 src/gz i586 http://my.server/ipk/i586
7167 src/gz qemux86 http://my.server/ipk/qemux86
7168
7169Next, instruct ``opkg`` to fetch the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007170repository information:
7171
7172.. code-block:: none
7173
7174 # opkg update
7175
7176The ``opkg`` application is now able to find, install, and upgrade packages
7177from the specified repository.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007178
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007179Using DEB
7180^^^^^^^^^
7181
7182The ``apt`` application performs runtime package management of DEB
7183packages. This application uses a source list file to find available
7184package databases. You must perform an initial setup for ``apt`` on the
7185target machine if the
7186:term:`PACKAGE_FEED_ARCHS`,
7187:term:`PACKAGE_FEED_BASE_PATHS`,
7188and
7189:term:`PACKAGE_FEED_URIS`
7190variables have not been set or the target image was built before the
7191variables were set.
7192
7193To inform ``apt`` of the repository you want to use, you might create a
7194list file (e.g. ``my_repo.list``) inside the
7195``/etc/apt/sources.list.d/`` directory. As an example, suppose you are
7196serving packages from a ``deb/`` directory containing the ``i586``,
7197``all``, and ``qemux86`` databases through an HTTP server named
7198``my.server``. The list file should contain:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007199
7200.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007201
7202 deb http://my.server/deb/all ./
7203 deb http://my.server/deb/i586 ./
7204 deb http://my.server/deb/qemux86 ./
7205
7206Next, instruct the ``apt`` application
7207to fetch the repository information:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007208
7209.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007210
7211 # apt-get update
7212
7213After this step,
7214``apt`` is able to find, install, and upgrade packages from the
7215specified repository.
7216
7217Generating and Using Signed Packages
7218------------------------------------
7219
7220In order to add security to RPM packages used during a build, you can
7221take steps to securely sign them. Once a signature is verified, the
7222OpenEmbedded build system can use the package in the build. If security
7223fails for a signed package, the build system aborts the build.
7224
7225This section describes how to sign RPM packages during a build and how
7226to use signed package feeds (repositories) when doing a build.
7227
7228Signing RPM Packages
7229~~~~~~~~~~~~~~~~~~~~
7230
7231To enable signing RPM packages, you must set up the following
7232configurations in either your ``local.config`` or ``distro.config``
7233file:
7234::
7235
7236 # Inherit sign_rpm.bbclass to enable signing functionality
7237 INHERIT += " sign_rpm"
7238 # Define the GPG key that will be used for signing.
7239 RPM_GPG_NAME = "key_name"
7240 # Provide passphrase for the key
7241 RPM_GPG_PASSPHRASE = "passphrase"
7242
7243.. note::
7244
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007245 Be sure to supply appropriate values for both `key_name` and
7246 `passphrase`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007247
7248Aside from the ``RPM_GPG_NAME`` and ``RPM_GPG_PASSPHRASE`` variables in
7249the previous example, two optional variables related to signing exist:
7250
7251- *GPG_BIN:* Specifies a ``gpg`` binary/wrapper that is executed
7252 when the package is signed.
7253
7254- *GPG_PATH:* Specifies the ``gpg`` home directory used when the
7255 package is signed.
7256
7257Processing Package Feeds
7258~~~~~~~~~~~~~~~~~~~~~~~~
7259
7260In addition to being able to sign RPM packages, you can also enable
7261signed package feeds for IPK and RPM packages.
7262
7263The steps you need to take to enable signed package feed use are similar
7264to the steps used to sign RPM packages. You must define the following in
7265your ``local.config`` or ``distro.config`` file:
7266::
7267
7268 INHERIT += "sign_package_feed"
7269 PACKAGE_FEED_GPG_NAME = "key_name"
7270 PACKAGE_FEED_GPG_PASSPHRASE_FILE = "path_to_file_containing_passphrase"
7271
7272For signed package feeds, the passphrase must exist in a separate file,
7273which is pointed to by the ``PACKAGE_FEED_GPG_PASSPHRASE_FILE``
7274variable. Regarding security, keeping a plain text passphrase out of the
7275configuration is more secure.
7276
7277Aside from the ``PACKAGE_FEED_GPG_NAME`` and
7278``PACKAGE_FEED_GPG_PASSPHRASE_FILE`` variables, three optional variables
7279related to signed package feeds exist:
7280
7281- *GPG_BIN* Specifies a ``gpg`` binary/wrapper that is executed
7282 when the package is signed.
7283
7284- *GPG_PATH:* Specifies the ``gpg`` home directory used when the
7285 package is signed.
7286
7287- *PACKAGE_FEED_GPG_SIGNATURE_TYPE:* Specifies the type of ``gpg``
7288 signature. This variable applies only to RPM and IPK package feeds.
7289 Allowable values for the ``PACKAGE_FEED_GPG_SIGNATURE_TYPE`` are
7290 "ASC", which is the default and specifies ascii armored, and "BIN",
7291 which specifies binary.
7292
7293Testing Packages With ptest
7294---------------------------
7295
7296A Package Test (ptest) runs tests against packages built by the
7297OpenEmbedded build system on the target machine. A ptest contains at
7298least two items: the actual test, and a shell script (``run-ptest``)
7299that starts the test. The shell script that starts the test must not
7300contain the actual test - the script only starts the test. On the other
7301hand, the test can be anything from a simple shell script that runs a
7302binary and checks the output to an elaborate system of test binaries and
7303data files.
7304
7305The test generates output in the format used by Automake:
7306::
7307
7308 result: testname
7309
7310where the result can be ``PASS``, ``FAIL``, or ``SKIP``, and
7311the testname can be any identifying string.
7312
7313For a list of Yocto Project recipes that are already enabled with ptest,
Andrew Geissler09209ee2020-12-13 08:44:15 -06007314see the :yocto_wiki:`Ptest </Ptest>` wiki page.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007315
7316.. note::
7317
7318 A recipe is "ptest-enabled" if it inherits the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007319 :ref:`ptest <ref-classes-ptest>` class.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007320
7321Adding ptest to Your Build
7322~~~~~~~~~~~~~~~~~~~~~~~~~~
7323
7324To add package testing to your build, add the
7325:term:`DISTRO_FEATURES` and
7326:term:`EXTRA_IMAGE_FEATURES`
7327variables to your ``local.conf`` file, which is found in the
7328:term:`Build Directory`:
7329::
7330
7331 DISTRO_FEATURES_append = " ptest"
7332 EXTRA_IMAGE_FEATURES += "ptest-pkgs"
7333
7334Once your build is complete, the ptest files are installed into the
7335``/usr/lib/package/ptest`` directory within the image, where ``package``
7336is the name of the package.
7337
7338Running ptest
7339~~~~~~~~~~~~~
7340
7341The ``ptest-runner`` package installs a shell script that loops through
7342all installed ptest test suites and runs them in sequence. Consequently,
7343you might want to add this package to your image.
7344
7345Getting Your Package Ready
7346~~~~~~~~~~~~~~~~~~~~~~~~~~
7347
7348In order to enable a recipe to run installed ptests on target hardware,
7349you need to prepare the recipes that build the packages you want to
7350test. Here is what you have to do for each recipe:
7351
7352- *Be sure the recipe inherits
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007353 the* :ref:`ptest <ref-classes-ptest>` *class:*
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007354 Include the following line in each recipe:
7355 ::
7356
7357 inherit ptest
7358
7359- *Create run-ptest:* This script starts your test. Locate the
7360 script where you will refer to it using
7361 :term:`SRC_URI`. Here is an
7362 example that starts a test for ``dbus``:
7363 ::
7364
7365 #!/bin/sh
7366 cd test
7367 make -k runtest-TESTS
7368
7369- *Ensure dependencies are met:* If the test adds build or runtime
7370 dependencies that normally do not exist for the package (such as
7371 requiring "make" to run the test suite), use the
7372 :term:`DEPENDS` and
7373 :term:`RDEPENDS` variables in
7374 your recipe in order for the package to meet the dependencies. Here
7375 is an example where the package has a runtime dependency on "make":
7376 ::
7377
7378 RDEPENDS_${PN}-ptest += "make"
7379
7380- *Add a function to build the test suite:* Not many packages support
7381 cross-compilation of their test suites. Consequently, you usually
7382 need to add a cross-compilation function to the package.
7383
7384 Many packages based on Automake compile and run the test suite by
7385 using a single command such as ``make check``. However, the host
7386 ``make check`` builds and runs on the same computer, while
7387 cross-compiling requires that the package is built on the host but
7388 executed for the target architecture (though often, as in the case
7389 for ptest, the execution occurs on the host). The built version of
7390 Automake that ships with the Yocto Project includes a patch that
7391 separates building and execution. Consequently, packages that use the
7392 unaltered, patched version of ``make check`` automatically
7393 cross-compiles.
7394
7395 Regardless, you still must add a ``do_compile_ptest`` function to
7396 build the test suite. Add a function similar to the following to your
7397 recipe:
7398 ::
7399
7400 do_compile_ptest() {
7401 oe_runmake buildtest-TESTS
7402 }
7403
7404- *Ensure special configurations are set:* If the package requires
7405 special configurations prior to compiling the test code, you must
7406 insert a ``do_configure_ptest`` function into the recipe.
7407
7408- *Install the test suite:* The ``ptest`` class automatically copies
7409 the file ``run-ptest`` to the target and then runs make
7410 ``install-ptest`` to run the tests. If this is not enough, you need
7411 to create a ``do_install_ptest`` function and make sure it gets
7412 called after the "make install-ptest" completes.
7413
7414Creating Node Package Manager (NPM) Packages
7415--------------------------------------------
7416
7417`NPM <https://en.wikipedia.org/wiki/Npm_(software)>`__ is a package
7418manager for the JavaScript programming language. The Yocto Project
Andrew Geissler09209ee2020-12-13 08:44:15 -06007419supports the NPM :ref:`fetcher <bitbake:bitbake-user-manual/bitbake-user-manual-fetching:fetchers>`. You can
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007420use this fetcher in combination with
Andrew Geissler09209ee2020-12-13 08:44:15 -06007421:doc:`devtool </ref-manual/devtool-reference>` to create
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007422recipes that produce NPM packages.
7423
7424Two workflows exist that allow you to create NPM packages using
7425``devtool``: the NPM registry modules method and the NPM project code
7426method.
7427
7428.. note::
7429
7430 While it is possible to create NPM recipes manually, using
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007431 ``devtool`` is far simpler.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007432
7433Additionally, some requirements and caveats exist.
7434
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007435Requirements and Caveats
7436~~~~~~~~~~~~~~~~~~~~~~~~
7437
7438You need to be aware of the following before using ``devtool`` to create
7439NPM packages:
7440
7441- Of the two methods that you can use ``devtool`` to create NPM
7442 packages, the registry approach is slightly simpler. However, you
7443 might consider the project approach because you do not have to
7444 publish your module in the NPM registry
7445 (`npm-registry <https://docs.npmjs.com/misc/registry>`_), which
7446 is NPM's public registry.
7447
7448- Be familiar with
Andrew Geissler09209ee2020-12-13 08:44:15 -06007449 :doc:`devtool </ref-manual/devtool-reference>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007450
7451- The NPM host tools need the native ``nodejs-npm`` package, which is
7452 part of the OpenEmbedded environment. You need to get the package by
7453 cloning the https://github.com/openembedded/meta-openembedded
7454 repository out of GitHub. Be sure to add the path to your local copy
7455 to your ``bblayers.conf`` file.
7456
7457- ``devtool`` cannot detect native libraries in module dependencies.
7458 Consequently, you must manually add packages to your recipe.
7459
7460- While deploying NPM packages, ``devtool`` cannot determine which
7461 dependent packages are missing on the target (e.g. the node runtime
7462 ``nodejs``). Consequently, you need to find out what files are
7463 missing and be sure they are on the target.
7464
7465- Although you might not need NPM to run your node package, it is
7466 useful to have NPM on your target. The NPM package name is
7467 ``nodejs-npm``.
7468
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007469Using the Registry Modules Method
7470~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
7471
7472This section presents an example that uses the ``cute-files`` module,
7473which is a file browser web application.
7474
7475.. note::
7476
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007477 You must know the ``cute-files`` module version.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007478
7479The first thing you need to do is use ``devtool`` and the NPM fetcher to
7480create the recipe:
7481::
7482
7483 $ devtool add "npm://registry.npmjs.org;package=cute-files;version=1.0.2"
7484
7485The
7486``devtool add`` command runs ``recipetool create`` and uses the same
7487fetch URI to download each dependency and capture license details where
7488possible. The result is a generated recipe.
7489
7490The recipe file is fairly simple and contains every license that
7491``recipetool`` finds and includes the licenses in the recipe's
7492:term:`LIC_FILES_CHKSUM`
7493variables. You need to examine the variables and look for those with
7494"unknown" in the :term:`LICENSE`
7495field. You need to track down the license information for "unknown"
7496modules and manually add the information to the recipe.
7497
7498``recipetool`` creates a "shrinkwrap" file for your recipe. Shrinkwrap
7499files capture the version of all dependent modules. Many packages do not
7500provide shrinkwrap files. ``recipetool`` create a shrinkwrap file as it
7501runs.
7502
7503.. note::
7504
7505 A package is created for each sub-module. This policy is the only
7506 practical way to have the licenses for all of the dependencies
7507 represented in the license manifest of the image.
7508
7509The ``devtool edit-recipe`` command lets you take a look at the recipe:
7510::
7511
7512 $ devtool edit-recipe cute-files
7513 SUMMARY = "Turn any folder on your computer into a cute file browser, available on the local network."
7514 LICENSE = "MIT & ISC & Unknown"
7515 LIC_FILES_CHKSUM = "file://LICENSE;md5=71d98c0a1db42956787b1909c74a86ca \
7516 file://node_modules/toidentifier/LICENSE;md5=1a261071a044d02eb6f2bb47f51a3502 \
7517 file://node_modules/debug/LICENSE;md5=ddd815a475e7338b0be7a14d8ee35a99 \
7518 ...
7519 SRC_URI = " \
7520 npm://registry.npmjs.org/;package=cute-files;version=${PV} \
7521 npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \
7522 "
7523 S = "${WORKDIR}/npm"
7524 inherit npm LICENSE_${PN} = "MIT"
7525 LICENSE_${PN}-accepts = "MIT"
7526 LICENSE_${PN}-array-flatten = "MIT"
7527 ...
7528 LICENSE_${PN}-vary = "MIT"
7529
7530Three key points exist in the previous example:
7531
7532- :term:`SRC_URI` uses the NPM
7533 scheme so that the NPM fetcher is used.
7534
7535- ``recipetool`` collects all the license information. If a
7536 sub-module's license is unavailable, the sub-module's name appears in
7537 the comments.
7538
7539- The ``inherit npm`` statement causes the
7540 :ref:`npm <ref-classes-npm>` class to package
7541 up all the modules.
7542
7543You can run the following command to build the ``cute-files`` package:
7544::
7545
7546 $ devtool build cute-files
7547
7548Remember that ``nodejs`` must be installed on
7549the target before your package.
7550
7551Assuming 192.168.7.2 for the target's IP address, use the following
7552command to deploy your package:
7553::
7554
7555 $ devtool deploy-target -s cute-files root@192.168.7.2
7556
7557Once the package is installed on the target, you can
7558test the application:
7559
7560.. note::
7561
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007562 Because of a known issue, you cannot simply run ``cute-files`` as you would
7563 if you had run ``npm install``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007564
7565::
7566
7567 $ cd /usr/lib/node_modules/cute-files
7568 $ node cute-files.js
7569
7570On a browser,
7571go to ``http://192.168.7.2:3000`` and you see the following:
7572
7573.. image:: figures/cute-files-npm-example.png
7574 :align: center
7575
7576You can find the recipe in ``workspace/recipes/cute-files``. You can use
7577the recipe in any layer you choose.
7578
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007579Using the NPM Projects Code Method
7580~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
7581
7582Although it is useful to package modules already in the NPM registry,
7583adding ``node.js`` projects under development is a more common developer
7584use case.
7585
7586This section covers the NPM projects code method, which is very similar
7587to the "registry" approach described in the previous section. In the NPM
7588projects method, you provide ``devtool`` with an URL that points to the
7589source files.
7590
7591Replicating the same example, (i.e. ``cute-files``) use the following
7592command:
7593::
7594
7595 $ devtool add https://github.com/martinaglv/cute-files.git
7596
7597The
7598recipe this command generates is very similar to the recipe created in
7599the previous section. However, the ``SRC_URI`` looks like the following:
7600::
7601
7602 SRC_URI = " \
7603 git://github.com/martinaglv/cute-files.git;protocol=https \
7604 npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \
7605 "
7606
7607In this example,
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007608the main module is taken from the Git repository and dependencies are
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007609taken from the NPM registry. Other than those differences, the recipe is
7610basically the same between the two methods. You can build and deploy the
7611package exactly as described in the previous section that uses the
7612registry modules method.
7613
7614Adding custom metadata to packages
7615----------------------------------
7616
7617The variable
7618:term:`PACKAGE_ADD_METADATA`
7619can be used to add additional metadata to packages. This is reflected in
7620the package control/spec file. To take the ipk format for example, the
7621CONTROL file stored inside would contain the additional metadata as
7622additional lines.
7623
7624The variable can be used in multiple ways, including using suffixes to
7625set it for a specific package type and/or package. Note that the order
7626of precedence is the same as this list:
7627
7628- ``PACKAGE_ADD_METADATA_<PKGTYPE>_<PN>``
7629
7630- ``PACKAGE_ADD_METADATA_<PKGTYPE>``
7631
7632- ``PACKAGE_ADD_METADATA_<PN>``
7633
7634- ``PACKAGE_ADD_METADATA``
7635
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007636`<PKGTYPE>` is a parameter and expected to be a distinct name of specific
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007637package type:
7638
7639- IPK for .ipk packages
7640
7641- DEB for .deb packages
7642
7643- RPM for .rpm packages
7644
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007645`<PN>` is a parameter and expected to be a package name.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007646
7647The variable can contain multiple [one-line] metadata fields separated
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007648by the literal sequence '\\n'. The separator can be redefined using the
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007649variable flag ``separator``.
7650
7651The following is an example that adds two custom fields for ipk
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007652packages:
7653::
7654
7655 PACKAGE_ADD_METADATA_IPK = "Vendor: CustomIpk\nGroup:Applications/Spreadsheets"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007656
7657Efficiently Fetching Source Files During a Build
7658================================================
7659
7660The OpenEmbedded build system works with source files located through
7661the :term:`SRC_URI` variable. When
7662you build something using BitBake, a big part of the operation is
7663locating and downloading all the source tarballs. For images,
7664downloading all the source for various packages can take a significant
7665amount of time.
7666
7667This section shows you how you can use mirrors to speed up fetching
7668source files and how you can pre-fetch files all of which leads to more
7669efficient use of resources and time.
7670
7671Setting up Effective Mirrors
7672----------------------------
7673
7674A good deal that goes into a Yocto Project build is simply downloading
7675all of the source tarballs. Maybe you have been working with another
7676build system (OpenEmbedded or Angstrom) for which you have built up a
7677sizable directory of source tarballs. Or, perhaps someone else has such
7678a directory for which you have read access. If so, you can save time by
7679adding statements to your configuration file so that the build process
7680checks local directories first for existing tarballs before checking the
7681Internet.
7682
7683Here is an efficient way to set it up in your ``local.conf`` file:
7684::
7685
7686 SOURCE_MIRROR_URL ?= "file:///home/you/your-download-dir/"
7687 INHERIT += "own-mirrors"
7688 BB_GENERATE_MIRROR_TARBALLS = "1"
7689 # BB_NO_NETWORK = "1"
7690
7691In the previous example, the
7692:term:`BB_GENERATE_MIRROR_TARBALLS`
7693variable causes the OpenEmbedded build system to generate tarballs of
7694the Git repositories and store them in the
7695:term:`DL_DIR` directory. Due to
7696performance reasons, generating and storing these tarballs is not the
7697build system's default behavior.
7698
7699You can also use the
7700:term:`PREMIRRORS` variable. For
7701an example, see the variable's glossary entry in the Yocto Project
7702Reference Manual.
7703
7704Getting Source Files and Suppressing the Build
7705----------------------------------------------
7706
7707Another technique you can use to ready yourself for a successive string
7708of build operations, is to pre-fetch all the source files without
7709actually starting a build. This technique lets you work through any
7710download issues and ultimately gathers all the source files into your
7711download directory :ref:`structure-build-downloads`,
7712which is located with :term:`DL_DIR`.
7713
7714Use the following BitBake command form to fetch all the necessary
7715sources without starting the build:
7716::
7717
7718 $ bitbake target --runall=fetch
7719
7720This
7721variation of the BitBake command guarantees that you have all the
7722sources for that BitBake target should you disconnect from the Internet
7723and want to do the build later offline.
7724
7725Selecting an Initialization Manager
7726===================================
7727
7728By default, the Yocto Project uses SysVinit as the initialization
7729manager. However, support also exists for systemd, which is a full
7730replacement for init with parallel starting of services, reduced shell
7731overhead and other features that are used by many distributions.
7732
7733Within the system, SysVinit treats system components as services. These
7734services are maintained as shell scripts stored in the ``/etc/init.d/``
7735directory. Services organize into different run levels. This
7736organization is maintained by putting links to the services in the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007737``/etc/rcN.d/`` directories, where `N/` is one of the following options:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007738"S", "0", "1", "2", "3", "4", "5", or "6".
7739
7740.. note::
7741
7742 Each runlevel has a dependency on the previous runlevel. This
7743 dependency allows the services to work properly.
7744
7745In comparison, systemd treats components as units. Using units is a
7746broader concept as compared to using a service. A unit includes several
7747different types of entities. Service is one of the types of entities.
7748The runlevel concept in SysVinit corresponds to the concept of a target
7749in systemd, where target is also a type of supported unit.
7750
7751In a SysVinit-based system, services load sequentially (i.e. one by one)
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007752during init and parallelization is not supported. With systemd, services
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007753start in parallel. Needless to say, the method can have an impact on
7754system startup performance.
7755
7756If you want to use SysVinit, you do not have to do anything. But, if you
7757want to use systemd, you must take some steps as described in the
7758following sections.
7759
7760Using systemd Exclusively
7761-------------------------
7762
7763Set these variables in your distribution configuration file as follows:
7764::
7765
7766 DISTRO_FEATURES_append = " systemd"
7767 VIRTUAL-RUNTIME_init_manager = "systemd"
7768
7769You can also prevent the SysVinit distribution feature from
7770being automatically enabled as follows:
7771::
7772
7773 DISTRO_FEATURES_BACKFILL_CONSIDERED = "sysvinit"
7774
7775Doing so removes any
7776redundant SysVinit scripts.
7777
7778To remove initscripts from your image altogether, set this variable
7779also:
7780::
7781
7782 VIRTUAL-RUNTIME_initscripts = ""
7783
7784For information on the backfill variable, see
7785:term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`.
7786
7787Using systemd for the Main Image and Using SysVinit for the Rescue Image
7788------------------------------------------------------------------------
7789
7790Set these variables in your distribution configuration file as follows:
7791::
7792
7793 DISTRO_FEATURES_append = " systemd"
7794 VIRTUAL-RUNTIME_init_manager = "systemd"
7795
7796Doing so causes your main image to use the
7797``packagegroup-core-boot.bb`` recipe and systemd. The rescue/minimal
7798image cannot use this package group. However, it can install SysVinit
7799and the appropriate packages will have support for both systemd and
7800SysVinit.
7801
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007802Selecting a Device Manager
7803==========================
7804
7805The Yocto Project provides multiple ways to manage the device manager
7806(``/dev``):
7807
7808- Persistent and Pre-Populated\ ``/dev``: For this case, the ``/dev``
7809 directory is persistent and the required device nodes are created
7810 during the build.
7811
7812- Use ``devtmpfs`` with a Device Manager: For this case, the ``/dev``
7813 directory is provided by the kernel as an in-memory file system and
7814 is automatically populated by the kernel at runtime. Additional
7815 configuration of device nodes is done in user space by a device
7816 manager like ``udev`` or ``busybox-mdev``.
7817
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007818Using Persistent and Pre-Populated\ ``/dev``
7819--------------------------------------------
7820
7821To use the static method for device population, you need to set the
7822:term:`USE_DEVFS` variable to "0"
7823as follows:
7824::
7825
7826 USE_DEVFS = "0"
7827
7828The content of the resulting ``/dev`` directory is defined in a Device
7829Table file. The
7830:term:`IMAGE_DEVICE_TABLES`
7831variable defines the Device Table to use and should be set in the
7832machine or distro configuration file. Alternatively, you can set this
7833variable in your ``local.conf`` configuration file.
7834
7835If you do not define the ``IMAGE_DEVICE_TABLES`` variable, the default
7836``device_table-minimal.txt`` is used:
7837::
7838
7839 IMAGE_DEVICE_TABLES = "device_table-mymachine.txt"
7840
7841The population is handled by the ``makedevs`` utility during image
7842creation:
7843
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007844Using ``devtmpfs`` and a Device Manager
7845---------------------------------------
7846
7847To use the dynamic method for device population, you need to use (or be
7848sure to set) the :term:`USE_DEVFS`
7849variable to "1", which is the default:
7850::
7851
7852 USE_DEVFS = "1"
7853
7854With this
7855setting, the resulting ``/dev`` directory is populated by the kernel
7856using ``devtmpfs``. Make sure the corresponding kernel configuration
7857variable ``CONFIG_DEVTMPFS`` is set when building you build a Linux
7858kernel.
7859
7860All devices created by ``devtmpfs`` will be owned by ``root`` and have
7861permissions ``0600``.
7862
7863To have more control over the device nodes, you can use a device manager
7864like ``udev`` or ``busybox-mdev``. You choose the device manager by
7865defining the ``VIRTUAL-RUNTIME_dev_manager`` variable in your machine or
7866distro configuration file. Alternatively, you can set this variable in
7867your ``local.conf`` configuration file:
7868::
7869
7870 VIRTUAL-RUNTIME_dev_manager = "udev"
7871
7872 # Some alternative values
7873 # VIRTUAL-RUNTIME_dev_manager = "busybox-mdev"
7874 # VIRTUAL-RUNTIME_dev_manager = "systemd"
7875
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007876Using an External SCM
7877=====================
7878
7879If you're working on a recipe that pulls from an external Source Code
7880Manager (SCM), it is possible to have the OpenEmbedded build system
7881notice new recipe changes added to the SCM and then build the resulting
7882packages that depend on the new recipes by using the latest versions.
7883This only works for SCMs from which it is possible to get a sensible
7884revision number for changes. Currently, you can do this with Apache
7885Subversion (SVN), Git, and Bazaar (BZR) repositories.
7886
7887To enable this behavior, the :term:`PV` of
7888the recipe needs to reference
7889:term:`SRCPV`. Here is an example:
7890::
7891
7892 PV = "1.2.3+git${SRCPV}"
7893
7894Then, you can add the following to your
7895``local.conf``:
7896::
7897
7898 SRCREV_pn-PN = "${AUTOREV}"
7899
7900:term:`PN` is the name of the recipe for
7901which you want to enable automatic source revision updating.
7902
7903If you do not want to update your local configuration file, you can add
7904the following directly to the recipe to finish enabling the feature:
7905::
7906
7907 SRCREV = "${AUTOREV}"
7908
7909The Yocto Project provides a distribution named ``poky-bleeding``, whose
7910configuration file contains the line:
7911::
7912
7913 require conf/distro/include/poky-floating-revisions.inc
7914
7915This line pulls in the
7916listed include file that contains numerous lines of exactly that form:
7917::
7918
7919 #SRCREV_pn-opkg-native ?= "${AUTOREV}"
7920 #SRCREV_pn-opkg-sdk ?= "${AUTOREV}"
7921 #SRCREV_pn-opkg ?= "${AUTOREV}"
7922 #SRCREV_pn-opkg-utils-native ?= "${AUTOREV}"
7923 #SRCREV_pn-opkg-utils ?= "${AUTOREV}"
7924 SRCREV_pn-gconf-dbus ?= "${AUTOREV}"
7925 SRCREV_pn-matchbox-common ?= "${AUTOREV}"
7926 SRCREV_pn-matchbox-config-gtk ?= "${AUTOREV}"
7927 SRCREV_pn-matchbox-desktop ?= "${AUTOREV}"
7928 SRCREV_pn-matchbox-keyboard ?= "${AUTOREV}"
7929 SRCREV_pn-matchbox-panel-2 ?= "${AUTOREV}"
7930 SRCREV_pn-matchbox-themes-extra ?= "${AUTOREV}"
7931 SRCREV_pn-matchbox-terminal ?= "${AUTOREV}"
7932 SRCREV_pn-matchbox-wm ?= "${AUTOREV}"
7933 SRCREV_pn-settings-daemon ?= "${AUTOREV}"
7934 SRCREV_pn-screenshot ?= "${AUTOREV}"
7935 . . .
7936
7937These lines allow you to
7938experiment with building a distribution that tracks the latest
7939development source for numerous packages.
7940
7941.. note::
7942
Andrew Geissler4c19ea12020-10-27 13:52:24 -05007943 The ``poky-bleeding`` distribution is not tested on a regular basis. Keep
7944 this in mind if you use it.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007945
7946Creating a Read-Only Root Filesystem
7947====================================
7948
7949Suppose, for security reasons, you need to disable your target device's
7950root filesystem's write permissions (i.e. you need a read-only root
7951filesystem). Or, perhaps you are running the device's operating system
7952from a read-only storage device. For either case, you can customize your
7953image for that behavior.
7954
7955.. note::
7956
7957 Supporting a read-only root filesystem requires that the system and
7958 applications do not try to write to the root filesystem. You must
7959 configure all parts of the target system to write elsewhere, or to
7960 gracefully fail in the event of attempting to write to the root
7961 filesystem.
7962
7963Creating the Root Filesystem
7964----------------------------
7965
7966To create the read-only root filesystem, simply add the
7967"read-only-rootfs" feature to your image, normally in one of two ways.
7968The first way is to add the "read-only-rootfs" image feature in the
7969image's recipe file via the ``IMAGE_FEATURES`` variable:
7970::
7971
7972 IMAGE_FEATURES += "read-only-rootfs"
7973
7974As an alternative, you can add the same feature
7975from within your build directory's ``local.conf`` file with the
7976associated ``EXTRA_IMAGE_FEATURES`` variable, as in:
7977::
7978
7979 EXTRA_IMAGE_FEATURES = "read-only-rootfs"
7980
7981For more information on how to use these variables, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06007982":ref:`dev-manual/common-tasks:Customizing Images Using Custom \`\`IMAGE_FEATURES\`\` and \`\`EXTRA_IMAGE_FEATURES\`\``"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007983section. For information on the variables, see
7984:term:`IMAGE_FEATURES` and
7985:term:`EXTRA_IMAGE_FEATURES`.
7986
7987Post-Installation Scripts and Read-Only Root Filesystem
7988-------------------------------------------------------
7989
7990It is very important that you make sure all post-Installation
7991(``pkg_postinst``) scripts for packages that are installed into the
7992image can be run at the time when the root filesystem is created during
7993the build on the host system. These scripts cannot attempt to run during
7994first-boot on the target device. With the "read-only-rootfs" feature
7995enabled, the build system checks during root filesystem creation to make
7996sure all post-installation scripts succeed. If any of these scripts
7997still need to be run after the root filesystem is created, the build
7998immediately fails. These build-time checks ensure that the build fails
7999rather than the target device fails later during its initial boot
8000operation.
8001
8002Most of the common post-installation scripts generated by the build
8003system for the out-of-the-box Yocto Project are engineered so that they
8004can run during root filesystem creation (e.g. post-installation scripts
8005for caching fonts). However, if you create and add custom scripts, you
8006need to be sure they can be run during this file system creation.
8007
8008Here are some common problems that prevent post-installation scripts
8009from running during root filesystem creation:
8010
8011- *Not using $D in front of absolute paths:* The build system defines
8012 ``$``\ :term:`D` when the root
8013 filesystem is created. Furthermore, ``$D`` is blank when the script
8014 is run on the target device. This implies two purposes for ``$D``:
8015 ensuring paths are valid in both the host and target environments,
8016 and checking to determine which environment is being used as a method
8017 for taking appropriate actions.
8018
8019- *Attempting to run processes that are specific to or dependent on the
8020 target architecture:* You can work around these attempts by using
8021 native tools, which run on the host system, to accomplish the same
8022 tasks, or by alternatively running the processes under QEMU, which
8023 has the ``qemu_run_binary`` function. For more information, see the
8024 :ref:`qemu <ref-classes-qemu>` class.
8025
8026Areas With Write Access
8027-----------------------
8028
8029With the "read-only-rootfs" feature enabled, any attempt by the target
8030to write to the root filesystem at runtime fails. Consequently, you must
8031make sure that you configure processes and applications that attempt
8032these types of writes do so to directories with write access (e.g.
8033``/tmp`` or ``/var/run``).
8034
8035Maintaining Build Output Quality
8036================================
8037
8038Many factors can influence the quality of a build. For example, if you
8039upgrade a recipe to use a new version of an upstream software package or
8040you experiment with some new configuration options, subtle changes can
8041occur that you might not detect until later. Consider the case where
8042your recipe is using a newer version of an upstream package. In this
8043case, a new version of a piece of software might introduce an optional
8044dependency on another library, which is auto-detected. If that library
8045has already been built when the software is building, the software will
8046link to the built library and that library will be pulled into your
8047image along with the new software even if you did not want the library.
8048
8049The :ref:`buildhistory <ref-classes-buildhistory>`
8050class exists to help you maintain the quality of your build output. You
8051can use the class to highlight unexpected and possibly unwanted changes
8052in the build output. When you enable build history, it records
8053information about the contents of each package and image and then
8054commits that information to a local Git repository where you can examine
8055the information.
8056
8057The remainder of this section describes the following:
8058
Andrew Geissler09209ee2020-12-13 08:44:15 -06008059- :ref:`How you can enable and disable build history <dev-manual/common-tasks:enabling and disabling build history>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008060
Andrew Geissler09209ee2020-12-13 08:44:15 -06008061- :ref:`How to understand what the build history contains <dev-manual/common-tasks:understanding what the build history contains>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008062
Andrew Geissler09209ee2020-12-13 08:44:15 -06008063- :ref:`How to limit the information used for build history <dev-manual/common-tasks:using build history to gather image information only>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008064
Andrew Geissler09209ee2020-12-13 08:44:15 -06008065- :ref:`How to examine the build history from both a command-line and web interface <dev-manual/common-tasks:examining build history information>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008066
8067Enabling and Disabling Build History
8068------------------------------------
8069
8070Build history is disabled by default. To enable it, add the following
8071``INHERIT`` statement and set the
8072:term:`BUILDHISTORY_COMMIT`
8073variable to "1" at the end of your ``conf/local.conf`` file found in the
8074:term:`Build Directory`:
8075::
8076
8077 INHERIT += "buildhistory"
8078 BUILDHISTORY_COMMIT = "1"
8079
8080Enabling build history as
8081previously described causes the OpenEmbedded build system to collect
8082build output information and commit it as a single commit to a local
Andrew Geissler09209ee2020-12-13 08:44:15 -06008083:ref:`overview-manual/development-environment:git` repository.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008084
8085.. note::
8086
8087 Enabling build history increases your build times slightly,
8088 particularly for images, and increases the amount of disk space used
8089 during the build.
8090
8091You can disable build history by removing the previous statements from
8092your ``conf/local.conf`` file.
8093
8094Understanding What the Build History Contains
8095---------------------------------------------
8096
8097Build history information is kept in
8098``${``\ :term:`TOPDIR`\ ``}/buildhistory``
8099in the Build Directory as defined by the
8100:term:`BUILDHISTORY_DIR`
8101variable. The following is an example abbreviated listing:
8102
8103.. image:: figures/buildhistory.png
8104 :align: center
8105
8106At the top level, a ``metadata-revs`` file exists that lists the
8107revisions of the repositories for the enabled layers when the build was
8108produced. The rest of the data splits into separate ``packages``,
8109``images`` and ``sdk`` directories, the contents of which are described
8110as follows.
8111
8112Build History Package Information
8113~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8114
8115The history for each package contains a text file that has name-value
8116pairs with information about the package. For example,
8117``buildhistory/packages/i586-poky-linux/busybox/busybox/latest``
8118contains the following:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008119
8120.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008121
8122 PV = 1.22.1
8123 PR = r32
8124 RPROVIDES =
8125 RDEPENDS = glibc (>= 2.20) update-alternatives-opkg
8126 RRECOMMENDS = busybox-syslog busybox-udhcpc update-rc.d
8127 PKGSIZE = 540168
8128 FILES = /usr/bin/* /usr/sbin/* /usr/lib/busybox/* /usr/lib/lib*.so.* \
8129 /etc /com /var /bin/* /sbin/* /lib/*.so.* /lib/udev/rules.d \
8130 /usr/lib/udev/rules.d /usr/share/busybox /usr/lib/busybox/* \
8131 /usr/share/pixmaps /usr/share/applications /usr/share/idl \
8132 /usr/share/omf /usr/share/sounds /usr/lib/bonobo/servers
8133 FILELIST = /bin/busybox /bin/busybox.nosuid /bin/busybox.suid /bin/sh \
8134 /etc/busybox.links.nosuid /etc/busybox.links.suid
8135
8136Most of these
8137name-value pairs correspond to variables used to produce the package.
8138The exceptions are ``FILELIST``, which is the actual list of files in
8139the package, and ``PKGSIZE``, which is the total size of files in the
8140package in bytes.
8141
8142A file also exists that corresponds to the recipe from which the package
8143came (e.g. ``buildhistory/packages/i586-poky-linux/busybox/latest``):
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008144
8145.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008146
8147 PV = 1.22.1
8148 PR = r32
8149 DEPENDS = initscripts kern-tools-native update-rc.d-native \
8150 virtual/i586-poky-linux-compilerlibs virtual/i586-poky-linux-gcc \
8151 virtual/libc virtual/update-alternatives
8152 PACKAGES = busybox-ptest busybox-httpd busybox-udhcpd busybox-udhcpc \
8153 busybox-syslog busybox-mdev busybox-hwclock busybox-dbg \
8154 busybox-staticdev busybox-dev busybox-doc busybox-locale busybox
8155
8156Finally, for those recipes fetched from a version control system (e.g.,
8157Git), a file exists that lists source revisions that are specified in
8158the recipe and lists the actual revisions used during the build. Listed
8159and actual revisions might differ when
8160:term:`SRCREV` is set to
8161${:term:`AUTOREV`}. Here is an
8162example assuming
8163``buildhistory/packages/qemux86-poky-linux/linux-yocto/latest_srcrev``):
8164::
8165
8166 # SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
8167 SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
8168 # SRCREV_meta = "a227f20eff056e511d504b2e490f3774ab260d6f"
8169 SRCREV_meta ="a227f20eff056e511d504b2e490f3774ab260d6f"
8170
8171You can use the
8172``buildhistory-collect-srcrevs`` command with the ``-a`` option to
8173collect the stored ``SRCREV`` values from build history and report them
8174in a format suitable for use in global configuration (e.g.,
8175``local.conf`` or a distro include file) to override floating
8176``AUTOREV`` values to a fixed set of revisions. Here is some example
8177output from this command:
8178::
8179
8180 $ buildhistory-collect-srcrevs -a
8181 # i586-poky-linux
8182 SRCREV_pn-glibc = "b8079dd0d360648e4e8de48656c5c38972621072"
8183 SRCREV_pn-glibc-initial = "b8079dd0d360648e4e8de48656c5c38972621072"
8184 SRCREV_pn-opkg-utils = "53274f087565fd45d8452c5367997ba6a682a37a"
8185 SRCREV_pn-kmod = "fd56638aed3fe147015bfa10ed4a5f7491303cb4"
8186 # x86_64-linux
8187 SRCREV_pn-gtk-doc-stub-native = "1dea266593edb766d6d898c79451ef193eb17cfa"
8188 SRCREV_pn-dtc-native = "65cc4d2748a2c2e6f27f1cf39e07a5dbabd80ebf"
8189 SRCREV_pn-update-rc.d-native = "eca680ddf28d024954895f59a241a622dd575c11"
8190 SRCREV_glibc_pn-cross-localedef-native = "b8079dd0d360648e4e8de48656c5c38972621072"
8191 SRCREV_localedef_pn-cross-localedef-native = "c833367348d39dad7ba018990bfdaffaec8e9ed3"
8192 SRCREV_pn-prelink-native = "faa069deec99bf61418d0bab831c83d7c1b797ca"
8193 SRCREV_pn-opkg-utils-native = "53274f087565fd45d8452c5367997ba6a682a37a"
8194 SRCREV_pn-kern-tools-native = "23345b8846fe4bd167efdf1bd8a1224b2ba9a5ff"
8195 SRCREV_pn-kmod-native = "fd56638aed3fe147015bfa10ed4a5f7491303cb4"
8196 # qemux86-poky-linux
8197 SRCREV_machine_pn-linux-yocto = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
8198 SRCREV_meta_pn-linux-yocto = "a227f20eff056e511d504b2e490f3774ab260d6f"
8199 # all-poky-linux
8200 SRCREV_pn-update-rc.d = "eca680ddf28d024954895f59a241a622dd575c11"
8201
8202.. note::
8203
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008204 Here are some notes on using the ``buildhistory-collect-srcrevs`` command:
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008205
8206 - By default, only values where the ``SRCREV`` was not hardcoded
8207 (usually when ``AUTOREV`` is used) are reported. Use the ``-a``
8208 option to see all ``SRCREV`` values.
8209
8210 - The output statements might not have any effect if overrides are
8211 applied elsewhere in the build system configuration. Use the
8212 ``-f`` option to add the ``forcevariable`` override to each output
8213 line if you need to work around this restriction.
8214
8215 - The script does apply special handling when building for multiple
8216 machines. However, the script does place a comment before each set
8217 of values that specifies which triplet to which they belong as
8218 previously shown (e.g., ``i586-poky-linux``).
8219
8220Build History Image Information
8221~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8222
8223The files produced for each image are as follows:
8224
8225- ``image-files:`` A directory containing selected files from the root
8226 filesystem. The files are defined by
8227 :term:`BUILDHISTORY_IMAGE_FILES`.
8228
8229- ``build-id.txt:`` Human-readable information about the build
8230 configuration and metadata source revisions. This file contains the
8231 full build header as printed by BitBake.
8232
8233- ``*.dot:`` Dependency graphs for the image that are compatible with
8234 ``graphviz``.
8235
8236- ``files-in-image.txt:`` A list of files in the image with
8237 permissions, owner, group, size, and symlink information.
8238
8239- ``image-info.txt:`` A text file containing name-value pairs with
8240 information about the image. See the following listing example for
8241 more information.
8242
8243- ``installed-package-names.txt:`` A list of installed packages by name
8244 only.
8245
8246- ``installed-package-sizes.txt:`` A list of installed packages ordered
8247 by size.
8248
8249- ``installed-packages.txt:`` A list of installed packages with full
8250 package filenames.
8251
8252.. note::
8253
8254 Installed package information is able to be gathered and produced
8255 even if package management is disabled for the final image.
8256
8257Here is an example of ``image-info.txt``:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008258
8259.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008260
8261 DISTRO = poky
8262 DISTRO_VERSION = 1.7
8263 USER_CLASSES = buildstats image-mklibs image-prelink
8264 IMAGE_CLASSES = image_types
8265 IMAGE_FEATURES = debug-tweaks
8266 IMAGE_LINGUAS =
8267 IMAGE_INSTALL = packagegroup-core-boot run-postinsts
8268 BAD_RECOMMENDATIONS =
8269 NO_RECOMMENDATIONS =
8270 PACKAGE_EXCLUDE =
8271 ROOTFS_POSTPROCESS_COMMAND = write_package_manifest; license_create_manifest; \
8272 write_image_manifest ; buildhistory_list_installed_image ; \
8273 buildhistory_get_image_installed ; ssh_allow_empty_password; \
8274 postinst_enable_logging; rootfs_update_timestamp ; ssh_disable_dns_lookup ;
8275 IMAGE_POSTPROCESS_COMMAND = buildhistory_get_imageinfo ;
8276 IMAGESIZE = 6900
8277
8278Other than ``IMAGESIZE``,
8279which is the total size of the files in the image in Kbytes, the
8280name-value pairs are variables that may have influenced the content of
8281the image. This information is often useful when you are trying to
8282determine why a change in the package or file listings has occurred.
8283
8284Using Build History to Gather Image Information Only
8285~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8286
8287As you can see, build history produces image information, including
8288dependency graphs, so you can see why something was pulled into the
8289image. If you are just interested in this information and not interested
8290in collecting specific package or SDK information, you can enable
8291writing only image information without any history by adding the
8292following to your ``conf/local.conf`` file found in the
8293:term:`Build Directory`:
8294::
8295
8296 INHERIT += "buildhistory"
8297 BUILDHISTORY_COMMIT = "0"
8298 BUILDHISTORY_FEATURES = "image"
8299
8300Here, you set the
8301:term:`BUILDHISTORY_FEATURES`
8302variable to use the image feature only.
8303
8304Build History SDK Information
8305~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8306
8307Build history collects similar information on the contents of SDKs (e.g.
8308``bitbake -c populate_sdk imagename``) as compared to information it
8309collects for images. Furthermore, this information differs depending on
8310whether an extensible or standard SDK is being produced.
8311
8312The following list shows the files produced for SDKs:
8313
8314- ``files-in-sdk.txt:`` A list of files in the SDK with permissions,
8315 owner, group, size, and symlink information. This list includes both
8316 the host and target parts of the SDK.
8317
8318- ``sdk-info.txt:`` A text file containing name-value pairs with
8319 information about the SDK. See the following listing example for more
8320 information.
8321
8322- ``sstate-task-sizes.txt:`` A text file containing name-value pairs
8323 with information about task group sizes (e.g. ``do_populate_sysroot``
8324 tasks have a total size). The ``sstate-task-sizes.txt`` file exists
8325 only when an extensible SDK is created.
8326
8327- ``sstate-package-sizes.txt:`` A text file containing name-value pairs
8328 with information for the shared-state packages and sizes in the SDK.
8329 The ``sstate-package-sizes.txt`` file exists only when an extensible
8330 SDK is created.
8331
8332- ``sdk-files:`` A folder that contains copies of the files mentioned
8333 in ``BUILDHISTORY_SDK_FILES`` if the files are present in the output.
8334 Additionally, the default value of ``BUILDHISTORY_SDK_FILES`` is
8335 specific to the extensible SDK although you can set it differently if
8336 you would like to pull in specific files from the standard SDK.
8337
8338 The default files are ``conf/local.conf``, ``conf/bblayers.conf``,
8339 ``conf/auto.conf``, ``conf/locked-sigs.inc``, and
8340 ``conf/devtool.conf``. Thus, for an extensible SDK, these files get
8341 copied into the ``sdk-files`` directory.
8342
8343- The following information appears under each of the ``host`` and
8344 ``target`` directories for the portions of the SDK that run on the
8345 host and on the target, respectively:
8346
8347 .. note::
8348
8349 The following files for the most part are empty when producing an
8350 extensible SDK because this type of SDK is not constructed from
8351 packages as is the standard SDK.
8352
8353 - ``depends.dot:`` Dependency graph for the SDK that is compatible
8354 with ``graphviz``.
8355
8356 - ``installed-package-names.txt:`` A list of installed packages by
8357 name only.
8358
8359 - ``installed-package-sizes.txt:`` A list of installed packages
8360 ordered by size.
8361
8362 - ``installed-packages.txt:`` A list of installed packages with full
8363 package filenames.
8364
8365Here is an example of ``sdk-info.txt``:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008366
8367.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008368
8369 DISTRO = poky
8370 DISTRO_VERSION = 1.3+snapshot-20130327
8371 SDK_NAME = poky-glibc-i686-arm
8372 SDK_VERSION = 1.3+snapshot
8373 SDKMACHINE =
8374 SDKIMAGE_FEATURES = dev-pkgs dbg-pkgs
8375 BAD_RECOMMENDATIONS =
8376 SDKSIZE = 352712
8377
8378Other than ``SDKSIZE``, which is
8379the total size of the files in the SDK in Kbytes, the name-value pairs
8380are variables that might have influenced the content of the SDK. This
8381information is often useful when you are trying to determine why a
8382change in the package or file listings has occurred.
8383
8384Examining Build History Information
8385~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8386
8387You can examine build history output from the command line or from a web
8388interface.
8389
8390To see any changes that have occurred (assuming you have
8391:term:`BUILDHISTORY_COMMIT` = "1"),
8392you can simply use any Git command that allows you to view the history
8393of a repository. Here is one method:
8394::
8395
8396 $ git log -p
8397
8398You need to realize,
8399however, that this method does show changes that are not significant
8400(e.g. a package's size changing by a few bytes).
8401
8402A command-line tool called ``buildhistory-diff`` does exist, though,
8403that queries the Git repository and prints just the differences that
8404might be significant in human-readable form. Here is an example:
8405::
8406
8407 $ ~/poky/poky/scripts/buildhistory-diff . HEAD^
8408 Changes to images/qemux86_64/glibc/core-image-minimal (files-in-image.txt):
8409 /etc/anotherpkg.conf was added
8410 /sbin/anotherpkg was added
8411 * (installed-package-names.txt):
8412 * anotherpkg was added
8413 Changes to images/qemux86_64/glibc/core-image-minimal (installed-package-names.txt):
8414 anotherpkg was added
8415 packages/qemux86_64-poky-linux/v86d: PACKAGES: added "v86d-extras"
8416 * PR changed from "r0" to "r1"
8417 * PV changed from "0.1.10" to "0.1.12"
8418 packages/qemux86_64-poky-linux/v86d/v86d: PKGSIZE changed from 110579 to 144381 (+30%)
8419 * PR changed from "r0" to "r1"
8420 * PV changed from "0.1.10" to "0.1.12"
8421
8422.. note::
8423
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008424 The ``buildhistory-diff`` tool requires the ``GitPython``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008425 package. Be sure to install it using Pip3 as follows:
8426 ::
8427
8428 $ pip3 install GitPython --user
8429
8430
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008431 Alternatively, you can install ``python3-git`` using the appropriate
8432 distribution package manager (e.g. ``apt-get``, ``dnf``, or ``zipper``).
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008433
8434To see changes to the build history using a web interface, follow the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008435instruction in the ``README`` file
Andrew Geissler09209ee2020-12-13 08:44:15 -06008436:yocto_git:`here </buildhistory-web/>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008437
8438Here is a sample screenshot of the interface:
8439
8440.. image:: figures/buildhistory-web.png
8441 :align: center
8442
8443Performing Automated Runtime Testing
8444====================================
8445
8446The OpenEmbedded build system makes available a series of automated
8447tests for images to verify runtime functionality. You can run these
8448tests on either QEMU or actual target hardware. Tests are written in
8449Python making use of the ``unittest`` module, and the majority of them
8450run commands on the target system over SSH. This section describes how
8451you set up the environment to use these tests, run available tests, and
8452write and add your own tests.
8453
8454For information on the test and QA infrastructure available within the
Andrew Geissler09209ee2020-12-13 08:44:15 -06008455Yocto Project, see the ":ref:`ref-manual/release-process:testing and quality assurance`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008456section in the Yocto Project Reference Manual.
8457
8458Enabling Tests
8459--------------
8460
8461Depending on whether you are planning to run tests using QEMU or on the
8462hardware, you have to take different steps to enable the tests. See the
8463following subsections for information on how to enable both types of
8464tests.
8465
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008466Enabling Runtime Tests on QEMU
8467~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8468
8469In order to run tests, you need to do the following:
8470
8471- *Set up to avoid interaction with sudo for networking:* To
8472 accomplish this, you must do one of the following:
8473
8474 - Add ``NOPASSWD`` for your user in ``/etc/sudoers`` either for all
8475 commands or just for ``runqemu-ifup``. You must provide the full
8476 path as that can change if you are using multiple clones of the
8477 source repository.
8478
8479 .. note::
8480
8481 On some distributions, you also need to comment out "Defaults
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008482 requiretty" in ``/etc/sudoers``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008483
8484 - Manually configure a tap interface for your system.
8485
8486 - Run as root the script in ``scripts/runqemu-gen-tapdevs``, which
8487 should generate a list of tap devices. This is the option
8488 typically chosen for Autobuilder-type environments.
8489
8490 .. note::
8491
8492 - Be sure to use an absolute path when calling this script
8493 with sudo.
8494
8495 - The package recipe ``qemu-helper-native`` is required to run
8496 this script. Build the package using the following command:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008497 ::
8498
8499 $ bitbake qemu-helper-native
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008500
8501- *Set the DISPLAY variable:* You need to set this variable so that
8502 you have an X server available (e.g. start ``vncserver`` for a
8503 headless machine).
8504
8505- *Be sure your host's firewall accepts incoming connections from
8506 192.168.7.0/24:* Some of the tests (in particular DNF tests) start an
8507 HTTP server on a random high number port, which is used to serve
8508 files to the target. The DNF module serves
8509 ``${WORKDIR}/oe-rootfs-repo`` so it can run DNF channel commands.
8510 That means your host's firewall must accept incoming connections from
8511 192.168.7.0/24, which is the default IP range used for tap devices by
8512 ``runqemu``.
8513
8514- *Be sure your host has the correct packages installed:* Depending
8515 your host's distribution, you need to have the following packages
8516 installed:
8517
8518 - Ubuntu and Debian: ``sysstat`` and ``iproute2``
8519
8520 - OpenSUSE: ``sysstat`` and ``iproute2``
8521
8522 - Fedora: ``sysstat`` and ``iproute``
8523
8524 - CentOS: ``sysstat`` and ``iproute``
8525
8526Once you start running the tests, the following happens:
8527
85281. A copy of the root filesystem is written to ``${WORKDIR}/testimage``.
8529
85302. The image is booted under QEMU using the standard ``runqemu`` script.
8531
85323. A default timeout of 500 seconds occurs to allow for the boot process
8533 to reach the login prompt. You can change the timeout period by
8534 setting
8535 :term:`TEST_QEMUBOOT_TIMEOUT`
8536 in the ``local.conf`` file.
8537
85384. Once the boot process is reached and the login prompt appears, the
8539 tests run. The full boot log is written to
8540 ``${WORKDIR}/testimage/qemu_boot_log``.
8541
85425. Each test module loads in the order found in ``TEST_SUITES``. You can
8543 find the full output of the commands run over SSH in
8544 ``${WORKDIR}/testimgage/ssh_target_log``.
8545
85466. If no failures occur, the task running the tests ends successfully.
8547 You can find the output from the ``unittest`` in the task log at
8548 ``${WORKDIR}/temp/log.do_testimage``.
8549
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008550Enabling Runtime Tests on Hardware
8551~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8552
8553The OpenEmbedded build system can run tests on real hardware, and for
8554certain devices it can also deploy the image to be tested onto the
8555device beforehand.
8556
8557For automated deployment, a "master image" is installed onto the
8558hardware once as part of setup. Then, each time tests are to be run, the
8559following occurs:
8560
85611. The master image is booted into and used to write the image to be
8562 tested to a second partition.
8563
85642. The device is then rebooted using an external script that you need to
8565 provide.
8566
85673. The device boots into the image to be tested.
8568
8569When running tests (independent of whether the image has been deployed
8570automatically or not), the device is expected to be connected to a
8571network on a pre-determined IP address. You can either use static IP
8572addresses written into the image, or set the image to use DHCP and have
8573your DHCP server on the test network assign a known IP address based on
8574the MAC address of the device.
8575
8576In order to run tests on hardware, you need to set ``TEST_TARGET`` to an
8577appropriate value. For QEMU, you do not have to change anything, the
8578default value is "qemu". For running tests on hardware, the following
8579options exist:
8580
8581- *"simpleremote":* Choose "simpleremote" if you are going to run tests
8582 on a target system that is already running the image to be tested and
8583 is available on the network. You can use "simpleremote" in
8584 conjunction with either real hardware or an image running within a
8585 separately started QEMU or any other virtual machine manager.
8586
8587- *"SystemdbootTarget":* Choose "SystemdbootTarget" if your hardware is
8588 an EFI-based machine with ``systemd-boot`` as bootloader and
8589 ``core-image-testmaster`` (or something similar) is installed. Also,
8590 your hardware under test must be in a DHCP-enabled network that gives
8591 it the same IP address for each reboot.
8592
8593 If you choose "SystemdbootTarget", there are additional requirements
8594 and considerations. See the "`Selecting
8595 SystemdbootTarget <#selecting-systemdboottarget>`__" section, which
8596 follows, for more information.
8597
8598- *"BeagleBoneTarget":* Choose "BeagleBoneTarget" if you are deploying
8599 images and running tests on the BeagleBone "Black" or original
8600 "White" hardware. For information on how to use these tests, see the
8601 comments at the top of the BeagleBoneTarget
8602 ``meta-yocto-bsp/lib/oeqa/controllers/beaglebonetarget.py`` file.
8603
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008604- *"EdgeRouterTarget":* Choose "EdgeRouterTarget" if you are deploying
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008605 images and running tests on the Ubiquiti Networks EdgeRouter Lite.
8606 For information on how to use these tests, see the comments at the
8607 top of the EdgeRouterTarget
8608 ``meta-yocto-bsp/lib/oeqa/controllers/edgeroutertarget.py`` file.
8609
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008610- *"GrubTarget":* Choose "GrubTarget" if you are deploying images and running
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008611 tests on any generic PC that boots using GRUB. For information on how
8612 to use these tests, see the comments at the top of the GrubTarget
8613 ``meta-yocto-bsp/lib/oeqa/controllers/grubtarget.py`` file.
8614
8615- *"your-target":* Create your own custom target if you want to run
8616 tests when you are deploying images and running tests on a custom
8617 machine within your BSP layer. To do this, you need to add a Python
8618 unit that defines the target class under ``lib/oeqa/controllers/``
8619 within your layer. You must also provide an empty ``__init__.py``.
8620 For examples, see files in ``meta-yocto-bsp/lib/oeqa/controllers/``.
8621
8622Selecting SystemdbootTarget
8623~~~~~~~~~~~~~~~~~~~~~~~~~~~
8624
8625If you did not set ``TEST_TARGET`` to "SystemdbootTarget", then you do
8626not need any information in this section. You can skip down to the
8627"`Running Tests <#qemu-image-running-tests>`__" section.
8628
8629If you did set ``TEST_TARGET`` to "SystemdbootTarget", you also need to
8630perform a one-time setup of your master image by doing the following:
8631
86321. *Set EFI_PROVIDER:* Be sure that ``EFI_PROVIDER`` is as follows:
8633 ::
8634
8635 EFI_PROVIDER = "systemd-boot"
8636
86372. *Build the master image:* Build the ``core-image-testmaster`` image.
8638 The ``core-image-testmaster`` recipe is provided as an example for a
8639 "master" image and you can customize the image recipe as you would
8640 any other recipe.
8641
8642 Here are the image recipe requirements:
8643
8644 - Inherits ``core-image`` so that kernel modules are installed.
8645
8646 - Installs normal linux utilities not busybox ones (e.g. ``bash``,
8647 ``coreutils``, ``tar``, ``gzip``, and ``kmod``).
8648
8649 - Uses a custom Initial RAM Disk (initramfs) image with a custom
8650 installer. A normal image that you can install usually creates a
8651 single rootfs partition. This image uses another installer that
8652 creates a specific partition layout. Not all Board Support
8653 Packages (BSPs) can use an installer. For such cases, you need to
8654 manually create the following partition layout on the target:
8655
8656 - First partition mounted under ``/boot``, labeled "boot".
8657
8658 - The main rootfs partition where this image gets installed,
8659 which is mounted under ``/``.
8660
8661 - Another partition labeled "testrootfs" where test images get
8662 deployed.
8663
86643. *Install image:* Install the image that you just built on the target
8665 system.
8666
8667The final thing you need to do when setting ``TEST_TARGET`` to
8668"SystemdbootTarget" is to set up the test image:
8669
86701. *Set up your local.conf file:* Make sure you have the following
8671 statements in your ``local.conf`` file:
8672 ::
8673
8674 IMAGE_FSTYPES += "tar.gz"
8675 INHERIT += "testimage"
8676 TEST_TARGET = "SystemdbootTarget"
8677 TEST_TARGET_IP = "192.168.2.3"
8678
86792. *Build your test image:* Use BitBake to build the image:
8680 ::
8681
8682 $ bitbake core-image-sato
8683
8684Power Control
8685~~~~~~~~~~~~~
8686
8687For most hardware targets other than "simpleremote", you can control
8688power:
8689
8690- You can use ``TEST_POWERCONTROL_CMD`` together with
8691 ``TEST_POWERCONTROL_EXTRA_ARGS`` as a command that runs on the host
8692 and does power cycling. The test code passes one argument to that
8693 command: off, on or cycle (off then on). Here is an example that
8694 could appear in your ``local.conf`` file:
8695 ::
8696
8697 TEST_POWERCONTROL_CMD = "powercontrol.exp test 10.11.12.1 nuc1"
8698
8699 In this example, the expect
8700 script does the following:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008701
8702 .. code-block:: shell
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008703
8704 ssh test@10.11.12.1 "pyctl nuc1 arg"
8705
8706 It then runs a Python script that controls power for a label called
8707 ``nuc1``.
8708
8709 .. note::
8710
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008711 You need to customize ``TEST_POWERCONTROL_CMD`` and
8712 ``TEST_POWERCONTROL_EXTRA_ARGS`` for your own setup. The one requirement
8713 is that it accepts "on", "off", and "cycle" as the last argument.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008714
8715- When no command is defined, it connects to the device over SSH and
8716 uses the classic reboot command to reboot the device. Classic reboot
8717 is fine as long as the machine actually reboots (i.e. the SSH test
8718 has not failed). It is useful for scenarios where you have a simple
8719 setup, typically with a single board, and where some manual
8720 interaction is okay from time to time.
8721
8722If you have no hardware to automatically perform power control but still
8723wish to experiment with automated hardware testing, you can use the
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008724``dialog-power-control`` script that shows a dialog prompting you to perform
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008725the required power action. This script requires either KDialog or Zenity
8726to be installed. To use this script, set the
8727:term:`TEST_POWERCONTROL_CMD`
8728variable as follows:
8729::
8730
8731 TEST_POWERCONTROL_CMD = "${COREBASE}/scripts/contrib/dialog-power-control"
8732
8733Serial Console Connection
8734~~~~~~~~~~~~~~~~~~~~~~~~~
8735
8736For test target classes requiring a serial console to interact with the
8737bootloader (e.g. BeagleBoneTarget, EdgeRouterTarget, and GrubTarget),
8738you need to specify a command to use to connect to the serial console of
8739the target machine by using the
8740:term:`TEST_SERIALCONTROL_CMD`
8741variable and optionally the
8742:term:`TEST_SERIALCONTROL_EXTRA_ARGS`
8743variable.
8744
8745These cases could be a serial terminal program if the machine is
8746connected to a local serial port, or a ``telnet`` or ``ssh`` command
8747connecting to a remote console server. Regardless of the case, the
8748command simply needs to connect to the serial console and forward that
8749connection to standard input and output as any normal terminal program
8750does. For example, to use the picocom terminal program on serial device
8751``/dev/ttyUSB0`` at 115200bps, you would set the variable as follows:
8752::
8753
8754 TEST_SERIALCONTROL_CMD = "picocom /dev/ttyUSB0 -b 115200"
8755
8756For local
8757devices where the serial port device disappears when the device reboots,
8758an additional "serdevtry" wrapper script is provided. To use this
8759wrapper, simply prefix the terminal command with
8760``${COREBASE}/scripts/contrib/serdevtry``:
8761::
8762
8763 TEST_SERIALCONTROL_CMD = "${COREBASE}/scripts/contrib/serdevtry picocom -b 115200 /dev/ttyUSB0"
8764
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008765Running Tests
8766-------------
8767
8768You can start the tests automatically or manually:
8769
8770- *Automatically running tests:* To run the tests automatically after
8771 the OpenEmbedded build system successfully creates an image, first
8772 set the
8773 :term:`TESTIMAGE_AUTO`
8774 variable to "1" in your ``local.conf`` file in the
8775 :term:`Build Directory`:
8776 ::
8777
8778 TESTIMAGE_AUTO = "1"
8779
8780 Next, build your image. If the image successfully builds, the
8781 tests run:
8782 ::
8783
8784 bitbake core-image-sato
8785
8786- *Manually running tests:* To manually run the tests, first globally
8787 inherit the
8788 :ref:`testimage <ref-classes-testimage*>` class
8789 by editing your ``local.conf`` file:
8790 ::
8791
8792 INHERIT += "testimage"
8793
8794 Next, use BitBake to run the tests:
8795 ::
8796
8797 bitbake -c testimage image
8798
8799All test files reside in ``meta/lib/oeqa/runtime`` in the
8800:term:`Source Directory`. A test name maps
8801directly to a Python module. Each test module may contain a number of
8802individual tests. Tests are usually grouped together by the area tested
8803(e.g tests for systemd reside in ``meta/lib/oeqa/runtime/systemd.py``).
8804
8805You can add tests to any layer provided you place them in the proper
8806area and you extend :term:`BBPATH` in
8807the ``local.conf`` file as normal. Be sure that tests reside in
8808``layer/lib/oeqa/runtime``.
8809
8810.. note::
8811
8812 Be sure that module names do not collide with module names used in
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008813 the default set of test modules in ``meta/lib/oeqa/runtime``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008814
8815You can change the set of tests run by appending or overriding
8816:term:`TEST_SUITES` variable in
8817``local.conf``. Each name in ``TEST_SUITES`` represents a required test
8818for the image. Test modules named within ``TEST_SUITES`` cannot be
8819skipped even if a test is not suitable for an image (e.g. running the
8820RPM tests on an image without ``rpm``). Appending "auto" to
8821``TEST_SUITES`` causes the build system to try to run all tests that are
8822suitable for the image (i.e. each test module may elect to skip itself).
8823
8824The order you list tests in ``TEST_SUITES`` is important and influences
8825test dependencies. Consequently, tests that depend on other tests should
8826be added after the test on which they depend. For example, since the
8827``ssh`` test depends on the ``ping`` test, "ssh" needs to come after
8828"ping" in the list. The test class provides no re-ordering or dependency
8829handling.
8830
8831.. note::
8832
8833 Each module can have multiple classes with multiple test methods.
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008834 And, Python ``unittest`` rules apply.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008835
8836Here are some things to keep in mind when running tests:
8837
8838- The default tests for the image are defined as:
8839 ::
8840
8841 DEFAULT_TEST_SUITES_pn-image = "ping ssh df connman syslog xorg scp vnc date rpm dnf dmesg"
8842
8843- Add your own test to the list of the by using the following:
8844 ::
8845
8846 TEST_SUITES_append = " mytest"
8847
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008848- Run a specific list of tests as follows:
8849 ::
8850
8851 TEST_SUITES = "test1 test2 test3"
8852
8853 Remember, order is important. Be sure to place a test that is
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008854 dependent on another test later in the order.
8855
8856Exporting Tests
8857---------------
8858
8859You can export tests so that they can run independently of the build
8860system. Exporting tests is required if you want to be able to hand the
8861test execution off to a scheduler. You can only export tests that are
8862defined in :term:`TEST_SUITES`.
8863
8864If your image is already built, make sure the following are set in your
8865``local.conf`` file:
8866::
8867
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008868 INHERIT += "testexport"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008869 TEST_TARGET_IP = "IP-address-for-the-test-target"
8870 TEST_SERVER_IP = "IP-address-for-the-test-server"
8871
8872You can then export the tests with the
8873following BitBake command form:
8874::
8875
8876 $ bitbake image -c testexport
8877
8878Exporting the tests places them in the
8879:term:`Build Directory` in
8880``tmp/testexport/``\ image, which is controlled by the
8881``TEST_EXPORT_DIR`` variable.
8882
8883You can now run the tests outside of the build environment:
8884::
8885
8886 $ cd tmp/testexport/image
8887 $ ./runexported.py testdata.json
8888
8889Here is a complete example that shows IP addresses and uses the
8890``core-image-sato`` image:
8891::
8892
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008893 INHERIT += "testexport"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008894 TEST_TARGET_IP = "192.168.7.2"
8895 TEST_SERVER_IP = "192.168.7.1"
8896
8897Use BitBake to export the tests:
8898::
8899
8900 $ bitbake core-image-sato -c testexport
8901
8902Run the tests outside of
8903the build environment using the following:
8904::
8905
8906 $ cd tmp/testexport/core-image-sato
8907 $ ./runexported.py testdata.json
8908
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008909Writing New Tests
8910-----------------
8911
8912As mentioned previously, all new test files need to be in the proper
8913place for the build system to find them. New tests for additional
8914functionality outside of the core should be added to the layer that adds
8915the functionality, in ``layer/lib/oeqa/runtime`` (as long as
8916:term:`BBPATH` is extended in the
8917layer's ``layer.conf`` file as normal). Just remember the following:
8918
8919- Filenames need to map directly to test (module) names.
8920
8921- Do not use module names that collide with existing core tests.
8922
8923- Minimally, an empty ``__init__.py`` file must exist in the runtime
8924 directory.
8925
8926To create a new test, start by copying an existing module (e.g.
8927``syslog.py`` or ``gcc.py`` are good ones to use). Test modules can use
8928code from ``meta/lib/oeqa/utils``, which are helper classes.
8929
8930.. note::
8931
8932 Structure shell commands such that you rely on them and they return a
8933 single code for success. Be aware that sometimes you will need to
Andrew Geissler4c19ea12020-10-27 13:52:24 -05008934 parse the output. See the ``df.py`` and ``date.py`` modules for examples.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008935
8936You will notice that all test classes inherit ``oeRuntimeTest``, which
8937is found in ``meta/lib/oetest.py``. This base class offers some helper
8938attributes, which are described in the following sections:
8939
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008940Class Methods
8941~~~~~~~~~~~~~
8942
8943Class methods are as follows:
8944
8945- *hasPackage(pkg):* Returns "True" if ``pkg`` is in the installed
8946 package list of the image, which is based on the manifest file that
8947 is generated during the ``do_rootfs`` task.
8948
8949- *hasFeature(feature):* Returns "True" if the feature is in
8950 :term:`IMAGE_FEATURES` or
8951 :term:`DISTRO_FEATURES`.
8952
Andrew Geisslerc9f78652020-09-18 14:11:35 -05008953Class Attributes
8954~~~~~~~~~~~~~~~~
8955
8956Class attributes are as follows:
8957
8958- *pscmd:* Equals "ps -ef" if ``procps`` is installed in the image.
8959 Otherwise, ``pscmd`` equals "ps" (busybox).
8960
8961- *tc:* The called test context, which gives access to the
8962 following attributes:
8963
8964 - *d:* The BitBake datastore, which allows you to use stuff such
8965 as ``oeRuntimeTest.tc.d.getVar("VIRTUAL-RUNTIME_init_manager")``.
8966
8967 - *testslist and testsrequired:* Used internally. The tests
8968 do not need these.
8969
8970 - *filesdir:* The absolute path to
8971 ``meta/lib/oeqa/runtime/files``, which contains helper files for
8972 tests meant for copying on the target such as small files written
8973 in C for compilation.
8974
8975 - *target:* The target controller object used to deploy and
8976 start an image on a particular target (e.g. Qemu, SimpleRemote,
8977 and SystemdbootTarget). Tests usually use the following:
8978
8979 - *ip:* The target's IP address.
8980
8981 - *server_ip:* The host's IP address, which is usually used
8982 by the DNF test suite.
8983
8984 - *run(cmd, timeout=None):* The single, most used method.
8985 This command is a wrapper for: ``ssh root@host "cmd"``. The
8986 command returns a tuple: (status, output), which are what their
8987 names imply - the return code of "cmd" and whatever output it
8988 produces. The optional timeout argument represents the number
8989 of seconds the test should wait for "cmd" to return. If the
8990 argument is "None", the test uses the default instance's
8991 timeout period, which is 300 seconds. If the argument is "0",
8992 the test runs until the command returns.
8993
8994 - *copy_to(localpath, remotepath):*
8995 ``scp localpath root@ip:remotepath``.
8996
8997 - *copy_from(remotepath, localpath):*
8998 ``scp root@host:remotepath localpath``.
8999
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009000Instance Attributes
9001~~~~~~~~~~~~~~~~~~~
9002
9003A single instance attribute exists, which is ``target``. The ``target``
9004instance attribute is identical to the class attribute of the same name,
9005which is described in the previous section. This attribute exists as
9006both an instance and class attribute so tests can use
9007``self.target.run(cmd)`` in instance methods instead of
9008``oeRuntimeTest.tc.target.run(cmd)``.
9009
9010Installing Packages in the DUT Without the Package Manager
9011----------------------------------------------------------
9012
9013When a test requires a package built by BitBake, it is possible to
9014install that package. Installing the package does not require a package
9015manager be installed in the device under test (DUT). It does, however,
9016require an SSH connection and the target must be using the
9017``sshcontrol`` class.
9018
9019.. note::
9020
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009021 This method uses ``scp`` to copy files from the host to the target, which
9022 causes permissions and special attributes to be lost.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009023
9024A JSON file is used to define the packages needed by a test. This file
9025must be in the same path as the file used to define the tests.
9026Furthermore, the filename must map directly to the test module name with
9027a ``.json`` extension.
9028
9029The JSON file must include an object with the test name as keys of an
9030object or an array. This object (or array of objects) uses the following
9031data:
9032
9033- "pkg" - A mandatory string that is the name of the package to be
9034 installed.
9035
9036- "rm" - An optional boolean, which defaults to "false", that specifies
9037 to remove the package after the test.
9038
9039- "extract" - An optional boolean, which defaults to "false", that
9040 specifies if the package must be extracted from the package format.
9041 When set to "true", the package is not automatically installed into
9042 the DUT.
9043
9044Following is an example JSON file that handles test "foo" installing
9045package "bar" and test "foobar" installing packages "foo" and "bar".
9046Once the test is complete, the packages are removed from the DUT.
9047::
9048
9049 {
9050 "foo": {
9051 "pkg": "bar"
9052 },
9053 "foobar": [
9054 {
9055 "pkg": "foo",
9056 "rm": true
9057 },
9058 {
9059 "pkg": "bar",
9060 "rm": true
9061 }
9062 ]
9063 }
9064
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009065Debugging Tools and Techniques
9066==============================
9067
9068The exact method for debugging build failures depends on the nature of
9069the problem and on the system's area from which the bug originates.
9070Standard debugging practices such as comparison against the last known
9071working version with examination of the changes and the re-application
9072of steps to identify the one causing the problem are valid for the Yocto
9073Project just as they are for any other system. Even though it is
9074impossible to detail every possible potential failure, this section
9075provides some general tips to aid in debugging given a variety of
9076situations.
9077
9078.. note::
9079
9080 A useful feature for debugging is the error reporting tool.
9081 Configuring the Yocto Project to use this tool causes the
9082 OpenEmbedded build system to produce error reporting commands as part
9083 of the console output. You can enter the commands after the build
9084 completes to log error information into a common database, that can
9085 help you figure out what might be going wrong. For information on how
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009086 to enable and use this feature, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06009087 ":ref:`dev-manual/common-tasks:using the error reporting tool`"
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009088 section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009089
9090The following list shows the debugging topics in the remainder of this
9091section:
9092
9093- "`Viewing Logs from Failed
9094 Tasks <#dev-debugging-viewing-logs-from-failed-tasks>`__" describes
9095 how to find and view logs from tasks that failed during the build
9096 process.
9097
9098- "`Viewing Variable
9099 Values <#dev-debugging-viewing-variable-values>`__" describes how to
9100 use the BitBake ``-e`` option to examine variable values after a
9101 recipe has been parsed.
9102
Andrew Geissler09209ee2020-12-13 08:44:15 -06009103- ":ref:`dev-manual/common-tasks:viewing package information with \`\`oe-pkgdata-util\`\``"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009104 describes how to use the ``oe-pkgdata-util`` utility to query
9105 :term:`PKGDATA_DIR` and
9106 display package-related information for built packages.
9107
9108- "`Viewing Dependencies Between Recipes and
9109 Tasks <#dev-viewing-dependencies-between-recipes-and-tasks>`__"
9110 describes how to use the BitBake ``-g`` option to display recipe
9111 dependency information used during the build.
9112
9113- "`Viewing Task Variable
9114 Dependencies <#dev-viewing-task-variable-dependencies>`__" describes
9115 how to use the ``bitbake-dumpsig`` command in conjunction with key
9116 subdirectories in the
9117 :term:`Build Directory` to determine
9118 variable dependencies.
9119
9120- "`Running Specific Tasks <#dev-debugging-taskrunning>`__" describes
9121 how to use several BitBake options (e.g. ``-c``, ``-C``, and ``-f``)
9122 to run specific tasks in the build chain. It can be useful to run
9123 tasks "out-of-order" when trying isolate build issues.
9124
9125- "`General BitBake Problems <#dev-debugging-bitbake>`__" describes how
9126 to use BitBake's ``-D`` debug output option to reveal more about what
9127 BitBake is doing during the build.
9128
9129- "`Building with No Dependencies <#dev-debugging-buildfile>`__"
9130 describes how to use the BitBake ``-b`` option to build a recipe
9131 while ignoring dependencies.
9132
9133- "`Recipe Logging Mechanisms <#recipe-logging-mechanisms>`__"
9134 describes how to use the many recipe logging functions to produce
9135 debugging output and report errors and warnings.
9136
9137- "`Debugging Parallel Make Races <#debugging-parallel-make-races>`__"
9138 describes how to debug situations where the build consists of several
9139 parts that are run simultaneously and when the output or result of
9140 one part is not ready for use with a different part of the build that
9141 depends on that output.
9142
9143- "`Debugging With the GNU Project Debugger (GDB)
9144 Remotely <#platdev-gdb-remotedebug>`__" describes how to use GDB to
9145 allow you to examine running programs, which can help you fix
9146 problems.
9147
9148- "`Debugging with the GNU Project Debugger (GDB) on the
9149 Target <#debugging-with-the-gnu-project-debugger-gdb-on-the-target>`__"
9150 describes how to use GDB directly on target hardware for debugging.
9151
9152- "`Other Debugging Tips <#dev-other-debugging-others>`__" describes
9153 miscellaneous debugging tips that can be useful.
9154
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009155Viewing Logs from Failed Tasks
9156------------------------------
9157
9158You can find the log for a task in the file
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009159``${``\ :term:`WORKDIR`\ ``}/temp/log.do_``\ `taskname`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009160For example, the log for the
9161:ref:`ref-tasks-compile` task of the
9162QEMU minimal image for the x86 machine (``qemux86``) might be in
9163``tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/temp/log.do_compile``.
9164To see the commands :term:`BitBake` ran
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009165to generate a log, look at the corresponding ``run.do_``\ `taskname` file
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009166in the same directory.
9167
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009168``log.do_``\ `taskname` and ``run.do_``\ `taskname` are actually symbolic
9169links to ``log.do_``\ `taskname`\ ``.``\ `pid` and
9170``log.run_``\ `taskname`\ ``.``\ `pid`, where `pid` is the PID the task had
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009171when it ran. The symlinks always point to the files corresponding to the
9172most recent run.
9173
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009174Viewing Variable Values
9175-----------------------
9176
9177Sometimes you need to know the value of a variable as a result of
9178BitBake's parsing step. This could be because some unexpected behavior
9179occurred in your project. Perhaps an attempt to :ref:`modify a variable
9180<bitbake:bitbake-user-manual/bitbake-user-manual-metadata:modifying existing
9181variables>` did not work out as expected.
9182
9183BitBake's ``-e`` option is used to display variable values after
9184parsing. The following command displays the variable values after the
9185configuration files (i.e. ``local.conf``, ``bblayers.conf``,
9186``bitbake.conf`` and so forth) have been parsed:
9187::
9188
9189 $ bitbake -e
9190
9191The following command displays variable values after a specific recipe has
9192been parsed. The variables include those from the configuration as well:
9193::
9194
9195 $ bitbake -e recipename
9196
9197.. note::
9198
9199 Each recipe has its own private set of variables (datastore).
9200 Internally, after parsing the configuration, a copy of the resulting
9201 datastore is made prior to parsing each recipe. This copying implies
9202 that variables set in one recipe will not be visible to other
9203 recipes.
9204
9205 Likewise, each task within a recipe gets a private datastore based on
9206 the recipe datastore, which means that variables set within one task
9207 will not be visible to other tasks.
9208
9209In the output of ``bitbake -e``, each variable is preceded by a
9210description of how the variable got its value, including temporary
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009211values that were later overridden. This description also includes
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009212variable flags (varflags) set on the variable. The output can be very
9213helpful during debugging.
9214
9215Variables that are exported to the environment are preceded by
9216``export`` in the output of ``bitbake -e``. See the following example:
9217::
9218
9219 export CC="i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/ulf/poky/build/tmp/sysroots/qemux86"
9220
9221In addition to variable values, the output of the ``bitbake -e`` and
9222``bitbake -e`` recipe commands includes the following information:
9223
9224- The output starts with a tree listing all configuration files and
9225 classes included globally, recursively listing the files they include
9226 or inherit in turn. Much of the behavior of the OpenEmbedded build
Andrew Geissler09209ee2020-12-13 08:44:15 -06009227 system (including the behavior of the :ref:`ref-manual/tasks:normal recipe build tasks`) is
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009228 implemented in the
9229 :ref:`base <ref-classes-base>` class and the
9230 classes it inherits, rather than being built into BitBake itself.
9231
9232- After the variable values, all functions appear in the output. For
9233 shell functions, variables referenced within the function body are
9234 expanded. If a function has been modified using overrides or using
9235 override-style operators like ``_append`` and ``_prepend``, then the
9236 final assembled function body appears in the output.
9237
9238Viewing Package Information with ``oe-pkgdata-util``
9239----------------------------------------------------
9240
9241You can use the ``oe-pkgdata-util`` command-line utility to query
9242:term:`PKGDATA_DIR` and display
9243various package-related information. When you use the utility, you must
9244use it to view information on packages that have already been built.
9245
9246Following are a few of the available ``oe-pkgdata-util`` subcommands.
9247
9248.. note::
9249
9250 You can use the standard \* and ? globbing wildcards as part of
9251 package names and paths.
9252
9253- ``oe-pkgdata-util list-pkgs [pattern]``: Lists all packages
9254 that have been built, optionally limiting the match to packages that
9255 match pattern.
9256
9257- ``oe-pkgdata-util list-pkg-files package ...``: Lists the
9258 files and directories contained in the given packages.
9259
9260 .. note::
9261
9262 A different way to view the contents of a package is to look at
9263 the
9264 ``${``\ :term:`WORKDIR`\ ``}/packages-split``
9265 directory of the recipe that generates the package. This directory
9266 is created by the
9267 :ref:`ref-tasks-package` task
9268 and has one subdirectory for each package the recipe generates,
9269 which contains the files stored in that package.
9270
9271 If you want to inspect the ``${WORKDIR}/packages-split``
9272 directory, make sure that
9273 :ref:`rm_work <ref-classes-rm-work>` is not
9274 enabled when you build the recipe.
9275
9276- ``oe-pkgdata-util find-path path ...``: Lists the names of
9277 the packages that contain the given paths. For example, the following
9278 tells us that ``/usr/share/man/man1/make.1`` is contained in the
9279 ``make-doc`` package:
9280 ::
9281
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009282 $ oe-pkgdata-util find-path /usr/share/man/man1/make.1
9283 make-doc: /usr/share/man/man1/make.1
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009284
9285- ``oe-pkgdata-util lookup-recipe package ...``: Lists the name
9286 of the recipes that produce the given packages.
9287
9288For more information on the ``oe-pkgdata-util`` command, use the help
9289facility:
9290::
9291
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009292 $ oe-pkgdata-util --help
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009293 $ oe-pkgdata-util subcommand --help
9294
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009295Viewing Dependencies Between Recipes and Tasks
9296----------------------------------------------
9297
9298Sometimes it can be hard to see why BitBake wants to build other recipes
9299before the one you have specified. Dependency information can help you
9300understand why a recipe is built.
9301
9302To generate dependency information for a recipe, run the following
9303command:
9304::
9305
9306 $ bitbake -g recipename
9307
9308This command writes the following files in the current directory:
9309
9310- ``pn-buildlist``: A list of recipes/targets involved in building
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009311 `recipename`. "Involved" here means that at least one task from the
9312 recipe needs to run when building `recipename` from scratch. Targets
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009313 that are in
9314 :term:`ASSUME_PROVIDED`
9315 are not listed.
9316
9317- ``task-depends.dot``: A graph showing dependencies between tasks.
9318
9319The graphs are in
9320`DOT <https://en.wikipedia.org/wiki/DOT_%28graph_description_language%29>`__
9321format and can be converted to images (e.g. using the ``dot`` tool from
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009322`Graphviz <https://www.graphviz.org/>`__).
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009323
9324.. note::
9325
9326 - DOT files use a plain text format. The graphs generated using the
9327 ``bitbake -g`` command are often so large as to be difficult to
9328 read without special pruning (e.g. with Bitbake's ``-I`` option)
9329 and processing. Despite the form and size of the graphs, the
9330 corresponding ``.dot`` files can still be possible to read and
9331 provide useful information.
9332
9333 As an example, the ``task-depends.dot`` file contains lines such
9334 as the following:
9335 ::
9336
9337 "libxslt.do_configure" -> "libxml2.do_populate_sysroot"
9338
9339 The above example line reveals that the
9340 :ref:`ref-tasks-configure`
9341 task in ``libxslt`` depends on the
9342 :ref:`ref-tasks-populate_sysroot`
9343 task in ``libxml2``, which is a normal
9344 :term:`DEPENDS` dependency
9345 between the two recipes.
9346
9347 - For an example of how ``.dot`` files can be processed, see the
9348 ``scripts/contrib/graph-tool`` Python script, which finds and
9349 displays paths between graph nodes.
9350
9351You can use a different method to view dependency information by using
9352the following command:
9353::
9354
9355 $ bitbake -g -u taskexp recipename
9356
9357This command
9358displays a GUI window from which you can view build-time and runtime
9359dependencies for the recipes involved in building recipename.
9360
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009361Viewing Task Variable Dependencies
9362----------------------------------
9363
9364As mentioned in the
9365":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-execution:checksums (signatures)`" section of the BitBake
9366User Manual, BitBake tries to automatically determine what variables a
9367task depends on so that it can rerun the task if any values of the
9368variables change. This determination is usually reliable. However, if
9369you do things like construct variable names at runtime, then you might
9370have to manually declare dependencies on those variables using
9371``vardeps`` as described in the
9372":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags`" section of the BitBake
9373User Manual.
9374
9375If you are unsure whether a variable dependency is being picked up
9376automatically for a given task, you can list the variable dependencies
9377BitBake has determined by doing the following:
9378
93791. Build the recipe containing the task:
9380::
9381
9382 $ bitbake recipename
9383
93842. Inside the :term:`STAMPS_DIR`
9385 directory, find the signature data (``sigdata``) file that
9386 corresponds to the task. The ``sigdata`` files contain a pickled
9387 Python database of all the metadata that went into creating the input
9388 checksum for the task. As an example, for the
9389 :ref:`ref-tasks-fetch` task of the
9390 ``db`` recipe, the ``sigdata`` file might be found in the following
9391 location:
9392 ::
9393
9394 ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1
9395
9396 For tasks that are accelerated through the shared state
Andrew Geissler09209ee2020-12-13 08:44:15 -06009397 (:ref:`sstate <overview-manual/concepts:shared state cache>`) cache, an
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009398 additional ``siginfo`` file is written into
9399 :term:`SSTATE_DIR` along with
9400 the cached task output. The ``siginfo`` files contain exactly the
9401 same information as ``sigdata`` files.
9402
94033. Run ``bitbake-dumpsig`` on the ``sigdata`` or ``siginfo`` file. Here
9404 is an example:
9405 ::
9406
9407 $ bitbake-dumpsig ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1
9408
9409 In the output of the above command, you will find a line like the
9410 following, which lists all the (inferred) variable dependencies for
9411 the task. This list also includes indirect dependencies from
9412 variables depending on other variables, recursively.
9413 ::
9414
9415 Task dependencies: ['PV', 'SRCREV', 'SRC_URI', 'SRC_URI[md5sum]', 'SRC_URI[sha256sum]', 'base_do_fetch']
9416
9417 .. note::
9418
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009419 Functions (e.g. ``base_do_fetch``) also count as variable dependencies.
9420 These functions in turn depend on the variables they reference.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009421
9422 The output of ``bitbake-dumpsig`` also includes the value each
9423 variable had, a list of dependencies for each variable, and
9424 :term:`bitbake:BB_HASHBASE_WHITELIST`
9425 information.
9426
9427There is also a ``bitbake-diffsigs`` command for comparing two
9428``siginfo`` or ``sigdata`` files. This command can be helpful when
9429trying to figure out what changed between two versions of a task. If you
9430call ``bitbake-diffsigs`` with just one file, the command behaves like
9431``bitbake-dumpsig``.
9432
9433You can also use BitBake to dump out the signature construction
9434information without executing tasks by using either of the following
9435BitBake command-line options:
9436::
9437
9438 ‐‐dump-signatures=SIGNATURE_HANDLER
9439 -S SIGNATURE_HANDLER
9440
9441
9442.. note::
9443
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009444 Two common values for `SIGNATURE_HANDLER` are "none" and "printdiff", which
9445 dump only the signature or compare the dumped signature with the cached one,
9446 respectively.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009447
9448Using BitBake with either of these options causes BitBake to dump out
9449``sigdata`` files in the ``stamps`` directory for every task it would
9450have executed instead of building the specified target package.
9451
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009452Viewing Metadata Used to Create the Input Signature of a Shared State Task
9453--------------------------------------------------------------------------
9454
9455Seeing what metadata went into creating the input signature of a shared
9456state (sstate) task can be a useful debugging aid. This information is
9457available in signature information (``siginfo``) files in
9458:term:`SSTATE_DIR`. For
9459information on how to view and interpret information in ``siginfo``
9460files, see the "`Viewing Task Variable
9461Dependencies <#dev-viewing-task-variable-dependencies>`__" section.
9462
9463For conceptual information on shared state, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -06009464":ref:`overview-manual/concepts:shared state`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009465section in the Yocto Project Overview and Concepts Manual.
9466
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009467Invalidating Shared State to Force a Task to Run
9468------------------------------------------------
9469
9470The OpenEmbedded build system uses
Andrew Geissler09209ee2020-12-13 08:44:15 -06009471:ref:`checksums <overview-manual/concepts:checksums (signatures)>` and
9472:ref:`overview-manual/concepts:shared state` cache to avoid unnecessarily
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009473rebuilding tasks. Collectively, this scheme is known as "shared state
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009474code".
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009475
9476As with all schemes, this one has some drawbacks. It is possible that
9477you could make implicit changes to your code that the checksum
9478calculations do not take into account. These implicit changes affect a
9479task's output but do not trigger the shared state code into rebuilding a
9480recipe. Consider an example during which a tool changes its output.
9481Assume that the output of ``rpmdeps`` changes. The result of the change
9482should be that all the ``package`` and ``package_write_rpm`` shared
9483state cache items become invalid. However, because the change to the
9484output is external to the code and therefore implicit, the associated
9485shared state cache items do not become invalidated. In this case, the
9486build process uses the cached items rather than running the task again.
9487Obviously, these types of implicit changes can cause problems.
9488
9489To avoid these problems during the build, you need to understand the
9490effects of any changes you make. Realize that changes you make directly
9491to a function are automatically factored into the checksum calculation.
9492Thus, these explicit changes invalidate the associated area of shared
9493state cache. However, you need to be aware of any implicit changes that
9494are not obvious changes to the code and could affect the output of a
9495given task.
9496
9497When you identify an implicit change, you can easily take steps to
9498invalidate the cache and force the tasks to run. The steps you can take
9499are as simple as changing a function's comments in the source code. For
9500example, to invalidate package shared state files, change the comment
9501statements of
9502:ref:`ref-tasks-package` or the
9503comments of one of the functions it calls. Even though the change is
9504purely cosmetic, it causes the checksum to be recalculated and forces
9505the build system to run the task again.
9506
9507.. note::
9508
9509 For an example of a commit that makes a cosmetic change to invalidate
9510 shared state, see this
Andrew Geissler09209ee2020-12-13 08:44:15 -06009511 :yocto_git:`commit </poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009512
9513Running Specific Tasks
9514----------------------
9515
9516Any given recipe consists of a set of tasks. The standard BitBake
9517behavior in most cases is: ``do_fetch``, ``do_unpack``, ``do_patch``,
9518``do_configure``, ``do_compile``, ``do_install``, ``do_package``,
9519``do_package_write_*``, and ``do_build``. The default task is
9520``do_build`` and any tasks on which it depends build first. Some tasks,
9521such as ``do_devshell``, are not part of the default build chain. If you
9522wish to run a task that is not part of the default build chain, you can
9523use the ``-c`` option in BitBake. Here is an example:
9524::
9525
9526 $ bitbake matchbox-desktop -c devshell
9527
9528The ``-c`` option respects task dependencies, which means that all other
9529tasks (including tasks from other recipes) that the specified task
9530depends on will be run before the task. Even when you manually specify a
9531task to run with ``-c``, BitBake will only run the task if it considers
9532it "out of date". See the
Andrew Geissler09209ee2020-12-13 08:44:15 -06009533":ref:`overview-manual/concepts:stamp files and the rerunning of tasks`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009534section in the Yocto Project Overview and Concepts Manual for how
9535BitBake determines whether a task is "out of date".
9536
9537If you want to force an up-to-date task to be rerun (e.g. because you
9538made manual modifications to the recipe's
9539:term:`WORKDIR` that you want to try
9540out), then you can use the ``-f`` option.
9541
9542.. note::
9543
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009544 The reason ``-f`` is never required when running the
9545 :ref:`ref-tasks-devshell` task is because the
9546 [\ :ref:`nostamp <bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ]
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009547 variable flag is already set for the task.
9548
9549The following example shows one way you can use the ``-f`` option:
9550::
9551
9552 $ bitbake matchbox-desktop
9553 .
9554 .
9555 make some changes to the source code in the work directory
9556 .
9557 .
9558 $ bitbake matchbox-desktop -c compile -f
9559 $ bitbake matchbox-desktop
9560
9561This sequence first builds and then recompiles ``matchbox-desktop``. The
9562last command reruns all tasks (basically the packaging tasks) after the
9563compile. BitBake recognizes that the ``do_compile`` task was rerun and
9564therefore understands that the other tasks also need to be run again.
9565
9566Another, shorter way to rerun a task and all
Andrew Geissler09209ee2020-12-13 08:44:15 -06009567:ref:`ref-manual/tasks:normal recipe build tasks`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009568that depend on it is to use the ``-C`` option.
9569
9570.. note::
9571
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009572 This option is upper-cased and is separate from the ``-c``
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009573 option, which is lower-cased.
9574
9575Using this option invalidates the given task and then runs the
9576:ref:`ref-tasks-build` task, which is
9577the default task if no task is given, and the tasks on which it depends.
9578You could replace the final two commands in the previous example with
9579the following single command:
9580::
9581
9582 $ bitbake matchbox-desktop -C compile
9583
9584Internally, the ``-f`` and ``-C`` options work by tainting (modifying)
9585the input checksum of the specified task. This tainting indirectly
9586causes the task and its dependent tasks to be rerun through the normal
9587task dependency mechanisms.
9588
9589.. note::
9590
9591 BitBake explicitly keeps track of which tasks have been tainted in
9592 this fashion, and will print warnings such as the following for
9593 builds involving such tasks:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009594
9595 .. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009596
9597 WARNING: /home/ulf/poky/meta/recipes-sato/matchbox-desktop/matchbox-desktop_2.1.bb.do_compile is tainted from a forced run
9598
9599
9600 The purpose of the warning is to let you know that the work directory
9601 and build output might not be in the clean state they would be in for
9602 a "normal" build, depending on what actions you took. To get rid of
9603 such warnings, you can remove the work directory and rebuild the
9604 recipe, as follows:
9605 ::
9606
9607 $ bitbake matchbox-desktop -c clean
9608 $ bitbake matchbox-desktop
9609
9610
9611You can view a list of tasks in a given package by running the
9612``do_listtasks`` task as follows:
9613::
9614
9615 $ bitbake matchbox-desktop -c listtasks
9616
9617The results appear as output to the console and are also in
9618the file ``${WORKDIR}/temp/log.do_listtasks``.
9619
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009620General BitBake Problems
9621------------------------
9622
9623You can see debug output from BitBake by using the ``-D`` option. The
9624debug output gives more information about what BitBake is doing and the
9625reason behind it. Each ``-D`` option you use increases the logging
9626level. The most common usage is ``-DDD``.
9627
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009628The output from ``bitbake -DDD -v targetname`` can reveal why BitBake
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009629chose a certain version of a package or why BitBake picked a certain
9630provider. This command could also help you in a situation where you
9631think BitBake did something unexpected.
9632
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009633Building with No Dependencies
9634-----------------------------
9635
9636To build a specific recipe (``.bb`` file), you can use the following
9637command form:
9638::
9639
9640 $ bitbake -b somepath/somerecipe.bb
9641
9642This command form does
9643not check for dependencies. Consequently, you should use it only when
9644you know existing dependencies have been met.
9645
9646.. note::
9647
9648 You can also specify fragments of the filename. In this case, BitBake
9649 checks for a unique match.
9650
9651Recipe Logging Mechanisms
9652-------------------------
9653
9654The Yocto Project provides several logging functions for producing
9655debugging output and reporting errors and warnings. For Python
9656functions, the following logging functions exist. All of these functions
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009657log to ``${T}/log.do_``\ `task`, and can also log to standard output
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009658(stdout) with the right settings:
9659
9660- ``bb.plain(msg)``: Writes msg as is to the log while also
9661 logging to stdout.
9662
9663- ``bb.note(msg)``: Writes "NOTE: msg" to the log. Also logs to
9664 stdout if BitBake is called with "-v".
9665
9666- ``bb.debug(level, msg)``: Writes "DEBUG: msg" to the
9667 log. Also logs to stdout if the log level is greater than or equal to
9668 level. See the ":ref:`-D <bitbake:bitbake-user-manual/bitbake-user-manual-intro:usage and syntax>`" option
9669 in the BitBake User Manual for more information.
9670
9671- ``bb.warn(msg)``: Writes "WARNING: msg" to the log while also
9672 logging to stdout.
9673
9674- ``bb.error(msg)``: Writes "ERROR: msg" to the log while also
9675 logging to standard out (stdout).
9676
9677 .. note::
9678
9679 Calling this function does not cause the task to fail.
9680
9681- ``bb.fatal(``\ msg\ ``)``: This logging function is similar to
9682 ``bb.error(``\ msg\ ``)`` but also causes the calling task to fail.
9683
9684 .. note::
9685
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009686 ``bb.fatal()`` raises an exception, which means you do not need to put a
9687 "return" statement after the function.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009688
9689The same logging functions are also available in shell functions, under
9690the names ``bbplain``, ``bbnote``, ``bbdebug``, ``bbwarn``, ``bberror``,
9691and ``bbfatal``. The
9692:ref:`logging <ref-classes-logging>` class
9693implements these functions. See that class in the ``meta/classes``
9694folder of the :term:`Source Directory` for information.
9695
9696Logging With Python
9697~~~~~~~~~~~~~~~~~~~
9698
9699When creating recipes using Python and inserting code that handles build
9700logs, keep in mind the goal is to have informative logs while keeping
9701the console as "silent" as possible. Also, if you want status messages
9702in the log, use the "debug" loglevel.
9703
9704Following is an example written in Python. The code handles logging for
9705a function that determines the number of tasks needed to be run. See the
9706":ref:`ref-tasks-listtasks`"
9707section for additional information:
9708::
9709
9710 python do_listtasks() {
9711 bb.debug(2, "Starting to figure out the task list")
9712 if noteworthy_condition:
9713 bb.note("There are 47 tasks to run")
9714 bb.debug(2, "Got to point xyz")
9715 if warning_trigger:
9716 bb.warn("Detected warning_trigger, this might be a problem later.")
9717 if recoverable_error:
9718 bb.error("Hit recoverable_error, you really need to fix this!")
9719 if fatal_error:
9720 bb.fatal("fatal_error detected, unable to print the task list")
9721 bb.plain("The tasks present are abc")
9722 bb.debug(2, "Finished figuring out the tasklist")
9723 }
9724
9725Logging With Bash
9726~~~~~~~~~~~~~~~~~
9727
9728When creating recipes using Bash and inserting code that handles build
9729logs, you have the same goals - informative with minimal console output.
9730The syntax you use for recipes written in Bash is similar to that of
9731recipes written in Python described in the previous section.
9732
9733Following is an example written in Bash. The code logs the progress of
9734the ``do_my_function`` function.
9735::
9736
9737 do_my_function() {
9738 bbdebug 2 "Running do_my_function"
9739 if [ exceptional_condition ]; then
9740 bbnote "Hit exceptional_condition"
9741 fi
9742 bbdebug 2 "Got to point xyz"
9743 if [ warning_trigger ]; then
9744 bbwarn "Detected warning_trigger, this might cause a problem later."
9745 fi
9746 if [ recoverable_error ]; then
9747 bberror "Hit recoverable_error, correcting"
9748 fi
9749 if [ fatal_error ]; then
9750 bbfatal "fatal_error detected"
9751 fi
9752 bbdebug 2 "Completed do_my_function"
9753 }
9754
9755
9756Debugging Parallel Make Races
9757-----------------------------
9758
9759A parallel ``make`` race occurs when the build consists of several parts
9760that are run simultaneously and a situation occurs when the output or
9761result of one part is not ready for use with a different part of the
9762build that depends on that output. Parallel make races are annoying and
9763can sometimes be difficult to reproduce and fix. However, some simple
9764tips and tricks exist that can help you debug and fix them. This section
9765presents a real-world example of an error encountered on the Yocto
9766Project autobuilder and the process used to fix it.
9767
9768.. note::
9769
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009770 If you cannot properly fix a ``make`` race condition, you can work around it
9771 by clearing either the :term:`PARALLEL_MAKE` or :term:`PARALLEL_MAKEINST`
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009772 variables.
9773
9774The Failure
9775~~~~~~~~~~~
9776
9777For this example, assume that you are building an image that depends on
9778the "neard" package. And, during the build, BitBake runs into problems
9779and creates the following output.
9780
9781.. note::
9782
9783 This example log file has longer lines artificially broken to make
9784 the listing easier to read.
9785
9786If you examine the output or the log file, you see the failure during
9787``make``:
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009788
9789.. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009790
9791 | DEBUG: SITE files ['endian-little', 'bit-32', 'ix86-common', 'common-linux', 'common-glibc', 'i586-linux', 'common']
9792 | DEBUG: Executing shell function do_compile
9793 | NOTE: make -j 16
9794 | make --no-print-directory all-am
9795 | /bin/mkdir -p include/near
9796 | /bin/mkdir -p include/near
9797 | /bin/mkdir -p include/near
9798 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9799 0.14-r0/neard-0.14/include/types.h include/near/types.h
9800 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9801 0.14-r0/neard-0.14/include/log.h include/near/log.h
9802 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9803 0.14-r0/neard-0.14/include/plugin.h include/near/plugin.h
9804 | /bin/mkdir -p include/near
9805 | /bin/mkdir -p include/near
9806 | /bin/mkdir -p include/near
9807 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9808 0.14-r0/neard-0.14/include/tag.h include/near/tag.h
9809 | /bin/mkdir -p include/near
9810 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9811 0.14-r0/neard-0.14/include/adapter.h include/near/adapter.h
9812 | /bin/mkdir -p include/near
9813 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9814 0.14-r0/neard-0.14/include/ndef.h include/near/ndef.h
9815 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9816 0.14-r0/neard-0.14/include/tlv.h include/near/tlv.h
9817 | /bin/mkdir -p include/near
9818 | /bin/mkdir -p include/near
9819 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9820 0.14-r0/neard-0.14/include/setting.h include/near/setting.h
9821 | /bin/mkdir -p include/near
9822 | /bin/mkdir -p include/near
9823 | /bin/mkdir -p include/near
9824 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9825 0.14-r0/neard-0.14/include/device.h include/near/device.h
9826 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9827 0.14-r0/neard-0.14/include/nfc_copy.h include/near/nfc_copy.h
9828 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9829 0.14-r0/neard-0.14/include/snep.h include/near/snep.h
9830 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9831 0.14-r0/neard-0.14/include/version.h include/near/version.h
9832 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9833 0.14-r0/neard-0.14/include/dbus.h include/near/dbus.h
9834 | ./src/genbuiltin nfctype1 nfctype2 nfctype3 nfctype4 p2p > src/builtin.h
9835 | i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/
9836 build/build/tmp/sysroots/qemux86 -DHAVE_CONFIG_H -I. -I./include -I./src -I./gdbus -I/home/pokybuild/
9837 yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/sysroots/qemux86/usr/include/glib-2.0
9838 -I/home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/sysroots/qemux86/usr/
9839 lib/glib-2.0/include -I/home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/
9840 tmp/sysroots/qemux86/usr/include/dbus-1.0 -I/home/pokybuild/yocto-autobuilder/yocto-slave/
9841 nightly-x86/build/build/tmp/sysroots/qemux86/usr/lib/dbus-1.0/include -I/home/pokybuild/yocto-autobuilder/
9842 yocto-slave/nightly-x86/build/build/tmp/sysroots/qemux86/usr/include/libnl3
9843 -DNEAR_PLUGIN_BUILTIN -DPLUGINDIR=\""/usr/lib/near/plugins"\"
9844 -DCONFIGDIR=\""/etc/neard\"" -O2 -pipe -g -feliminate-unused-debug-types -c
9845 -o tools/snep-send.o tools/snep-send.c
9846 | In file included from tools/snep-send.c:16:0:
9847 | tools/../src/near.h:41:23: fatal error: near/dbus.h: No such file or directory
9848 | #include <near/dbus.h>
9849 | ^
9850 | compilation terminated.
9851 | make[1]: *** [tools/snep-send.o] Error 1
9852 | make[1]: *** Waiting for unfinished jobs....
9853 | make: *** [all] Error 2
9854 | ERROR: oe_runmake failed
9855
9856Reproducing the Error
9857~~~~~~~~~~~~~~~~~~~~~
9858
9859Because race conditions are intermittent, they do not manifest
9860themselves every time you do the build. In fact, most times the build
9861will complete without problems even though the potential race condition
9862exists. Thus, once the error surfaces, you need a way to reproduce it.
9863
9864In this example, compiling the "neard" package is causing the problem.
9865So the first thing to do is build "neard" locally. Before you start the
9866build, set the
9867:term:`PARALLEL_MAKE` variable
9868in your ``local.conf`` file to a high number (e.g. "-j 20"). Using a
9869high value for ``PARALLEL_MAKE`` increases the chances of the race
9870condition showing up:
9871::
9872
9873 $ bitbake neard
9874
9875Once the local build for "neard" completes, start a ``devshell`` build:
9876::
9877
9878 $ bitbake neard -c devshell
9879
9880For information on how to use a
9881``devshell``, see the "`Using a Development
9882Shell <#platdev-appdev-devshell>`__" section.
9883
9884In the ``devshell``, do the following:
9885::
9886
9887 $ make clean
9888 $ make tools/snep-send.o
9889
9890The ``devshell`` commands cause the failure to clearly
9891be visible. In this case, a missing dependency exists for the "neard"
9892Makefile target. Here is some abbreviated, sample output with the
9893missing dependency clearly visible at the end:
9894::
9895
9896 i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/scott-lenovo/......
9897 .
9898 .
9899 .
9900 tools/snep-send.c
9901 In file included from tools/snep-send.c:16:0:
9902 tools/../src/near.h:41:23: fatal error: near/dbus.h: No such file or directory
9903 #include <near/dbus.h>
9904 ^
9905 compilation terminated.
9906 make: *** [tools/snep-send.o] Error 1
9907 $
9908
9909
9910Creating a Patch for the Fix
9911~~~~~~~~~~~~~~~~~~~~~~~~~~~~
9912
9913Because there is a missing dependency for the Makefile target, you need
9914to patch the ``Makefile.am`` file, which is generated from
9915``Makefile.in``. You can use Quilt to create the patch:
9916::
9917
9918 $ quilt new parallelmake.patch
9919 Patch patches/parallelmake.patch is now on top
9920 $ quilt add Makefile.am
9921 File Makefile.am added to patch patches/parallelmake.patch
9922
9923For more information on using Quilt, see the
9924"`Using Quilt in Your Workflow <#using-a-quilt-workflow>`__" section.
9925
9926At this point you need to make the edits to ``Makefile.am`` to add the
9927missing dependency. For our example, you have to add the following line
9928to the file:
9929::
9930
9931 tools/snep-send.$(OBJEXT): include/near/dbus.h
9932
9933Once you have edited the file, use the ``refresh`` command to create the
9934patch:
9935::
9936
9937 $ quilt refresh
9938 Refreshed patch patches/parallelmake.patch
9939
9940Once
9941the patch file exists, you need to add it back to the originating recipe
9942folder. Here is an example assuming a top-level
9943:term:`Source Directory` named ``poky``:
9944::
9945
9946 $ cp patches/parallelmake.patch poky/meta/recipes-connectivity/neard/neard
9947
9948The final thing you need to do to implement the fix in the build is to
9949update the "neard" recipe (i.e. ``neard-0.14.bb``) so that the
9950:term:`SRC_URI` statement includes
9951the patch file. The recipe file is in the folder above the patch. Here
9952is what the edited ``SRC_URI`` statement would look like:
9953::
9954
9955 SRC_URI = "${KERNELORG_MIRROR}/linux/network/nfc/${BPN}-${PV}.tar.xz \
9956 file://neard.in \
9957 file://neard.service.in \
9958 file://parallelmake.patch \
9959 "
9960
9961With the patch complete and moved to the correct folder and the
9962``SRC_URI`` statement updated, you can exit the ``devshell``:
9963::
9964
9965 $ exit
9966
9967Testing the Build
9968~~~~~~~~~~~~~~~~~
9969
9970With everything in place, you can get back to trying the build again
9971locally:
9972::
9973
Andrew Geissler4c19ea12020-10-27 13:52:24 -05009974 $ bitbake neard
9975
9976This build should succeed.
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009977
9978Now you can open up a ``devshell`` again and repeat the clean and make
9979operations as follows:
9980::
9981
9982 $ bitbake neard -c devshell
9983 $ make clean
9984 $ make tools/snep-send.o
9985
9986The build should work without issue.
9987
9988As with all solved problems, if they originated upstream, you need to
9989submit the fix for the recipe in OE-Core and upstream so that the
9990problem is taken care of at its source. See the "`Submitting a Change to
9991the Yocto Project <#how-to-submit-a-change>`__" section for more
9992information.
9993
Andrew Geisslerc9f78652020-09-18 14:11:35 -05009994Debugging With the GNU Project Debugger (GDB) Remotely
9995------------------------------------------------------
9996
9997GDB allows you to examine running programs, which in turn helps you to
9998understand and fix problems. It also allows you to perform post-mortem
9999style analysis of program crashes. GDB is available as a package within
10000the Yocto Project and is installed in SDK images by default. See the
Andrew Geissler09209ee2020-12-13 08:44:15 -060010001":ref:`ref-manual/images:Images`" chapter in the Yocto
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010002Project Reference Manual for a description of these images. You can find
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010003information on GDB at https://sourceware.org/gdb/.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010004
10005.. note::
10006
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010007 For best results, install debug (``-dbg``) packages for the applications you
10008 are going to debug. Doing so makes extra debug symbols available that give
10009 you more meaningful output.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010010
10011Sometimes, due to memory or disk space constraints, it is not possible
10012to use GDB directly on the remote target to debug applications. These
10013constraints arise because GDB needs to load the debugging information
10014and the binaries of the process being debugged. Additionally, GDB needs
10015to perform many computations to locate information such as function
10016names, variable names and values, stack traces and so forth - even
10017before starting the debugging process. These extra computations place
10018more load on the target system and can alter the characteristics of the
10019program being debugged.
10020
10021To help get past the previously mentioned constraints, you can use
10022gdbserver, which runs on the remote target and does not load any
10023debugging information from the debugged process. Instead, a GDB instance
10024processes the debugging information that is run on a remote computer -
10025the host GDB. The host GDB then sends control commands to gdbserver to
10026make it stop or start the debugged program, as well as read or write
10027memory regions of that debugged program. All the debugging information
10028loaded and processed as well as all the heavy debugging is done by the
10029host GDB. Offloading these processes gives the gdbserver running on the
10030target a chance to remain small and fast.
10031
10032Because the host GDB is responsible for loading the debugging
10033information and for doing the necessary processing to make actual
10034debugging happen, you have to make sure the host can access the
10035unstripped binaries complete with their debugging information and also
10036be sure the target is compiled with no optimizations. The host GDB must
10037also have local access to all the libraries used by the debugged
10038program. Because gdbserver does not need any local debugging
10039information, the binaries on the remote target can remain stripped.
10040However, the binaries must also be compiled without optimization so they
10041match the host's binaries.
10042
10043To remain consistent with GDB documentation and terminology, the binary
10044being debugged on the remote target machine is referred to as the
10045"inferior" binary. For documentation on GDB see the `GDB
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010046site <https://sourceware.org/gdb/documentation/>`__.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010047
10048The following steps show you how to debug using the GNU project
10049debugger.
10050
100511. *Configure your build system to construct the companion debug
10052 filesystem:*
10053
10054 In your ``local.conf`` file, set the following:
10055 ::
10056
10057 IMAGE_GEN_DEBUGFS = "1"
10058 IMAGE_FSTYPES_DEBUGFS = "tar.bz2"
10059
10060 These options cause the
10061 OpenEmbedded build system to generate a special companion filesystem
10062 fragment, which contains the matching source and debug symbols to
10063 your deployable filesystem. The build system does this by looking at
10064 what is in the deployed filesystem, and pulling the corresponding
10065 ``-dbg`` packages.
10066
10067 The companion debug filesystem is not a complete filesystem, but only
10068 contains the debug fragments. This filesystem must be combined with
10069 the full filesystem for debugging. Subsequent steps in this procedure
10070 show how to combine the partial filesystem with the full filesystem.
10071
100722. *Configure the system to include gdbserver in the target filesystem:*
10073
10074 Make the following addition in either your ``local.conf`` file or in
10075 an image recipe:
10076 ::
10077
10078 IMAGE_INSTALL_append = " gdbserver"
10079
10080 The change makes
10081 sure the ``gdbserver`` package is included.
10082
100833. *Build the environment:*
10084
10085 Use the following command to construct the image and the companion
10086 Debug Filesystem:
10087 ::
10088
10089 $ bitbake image
10090
10091 Build the cross GDB component and
10092 make it available for debugging. Build the SDK that matches the
10093 image. Building the SDK is best for a production build that can be
10094 used later for debugging, especially during long term maintenance:
10095 ::
10096
10097 $ bitbake -c populate_sdk image
10098
10099 Alternatively, you can build the minimal toolchain components that
10100 match the target. Doing so creates a smaller than typical SDK and
10101 only contains a minimal set of components with which to build simple
10102 test applications, as well as run the debugger:
10103 ::
10104
10105 $ bitbake meta-toolchain
10106
10107 A final method is to build Gdb itself within the build system:
10108 ::
10109
10110 $ bitbake gdb-cross-<architecture>
10111
10112 Doing so produces a temporary copy of
10113 ``cross-gdb`` you can use for debugging during development. While
10114 this is the quickest approach, the two previous methods in this step
10115 are better when considering long-term maintenance strategies.
10116
10117 .. note::
10118
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010119 If you run ``bitbake gdb-cross``, the OpenEmbedded build system suggests
10120 the actual image (e.g. ``gdb-cross-i586``). The suggestion is usually the
10121 actual name you want to use.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010122
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500101234. *Set up the* ``debugfs``\ *:*
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010124
10125 Run the following commands to set up the ``debugfs``:
10126 ::
10127
10128 $ mkdir debugfs
10129 $ cd debugfs
10130 $ tar xvfj build-dir/tmp-glibc/deploy/images/machine/image.rootfs.tar.bz2
10131 $ tar xvfj build-dir/tmp-glibc/deploy/images/machine/image-dbg.rootfs.tar.bz2
10132
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500101335. *Set up GDB:*
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010134
10135 Install the SDK (if you built one) and then source the correct
10136 environment file. Sourcing the environment file puts the SDK in your
10137 ``PATH`` environment variable.
10138
10139 If you are using the build system, Gdb is located in
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010140 `build-dir`\ ``/tmp/sysroots/``\ `host`\ ``/usr/bin/``\ `architecture`\ ``/``\ `architecture`\ ``-gdb``
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010141
101426. *Boot the target:*
10143
10144 For information on how to run QEMU, see the `QEMU
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010145 Documentation <https://wiki.qemu.org/Documentation/GettingStartedDevelopers>`__.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010146
10147 .. note::
10148
10149 Be sure to verify that your host can access the target via TCP.
10150
101517. *Debug a program:*
10152
10153 Debugging a program involves running gdbserver on the target and then
10154 running Gdb on the host. The example in this step debugs ``gzip``:
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010155
10156 .. code-block:: shell
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010157
10158 root@qemux86:~# gdbserver localhost:1234 /bin/gzip —help
10159
10160 For
10161 additional gdbserver options, see the `GDB Server
10162 Documentation <https://www.gnu.org/software/gdb/documentation/>`__.
10163
10164 After running gdbserver on the target, you need to run Gdb on the
10165 host and configure it and connect to the target. Use these commands:
10166 ::
10167
10168 $ cd directory-holding-the-debugfs-directory
10169 $ arch-gdb
10170 (gdb) set sysroot debugfs
10171 (gdb) set substitute-path /usr/src/debug debugfs/usr/src/debug
10172 (gdb) target remote IP-of-target:1234
10173
10174 At this
10175 point, everything should automatically load (i.e. matching binaries,
10176 symbols and headers).
10177
10178 .. note::
10179
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010180 The Gdb ``set`` commands in the previous example can be placed into the
10181 users ``~/.gdbinit`` file. Upon starting, Gdb automatically runs whatever
10182 commands are in that file.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010183
101848. *Deploying without a full image rebuild:*
10185
10186 In many cases, during development you want a quick method to deploy a
10187 new binary to the target and debug it, without waiting for a full
10188 image build.
10189
10190 One approach to solving this situation is to just build the component
10191 you want to debug. Once you have built the component, copy the
10192 executable directly to both the target and the host ``debugfs``.
10193
10194 If the binary is processed through the debug splitting in
10195 OpenEmbedded, you should also copy the debug items (i.e. ``.debug``
10196 contents and corresponding ``/usr/src/debug`` files) from the work
10197 directory. Here is an example:
10198 ::
10199
10200 $ bitbake bash
10201 $ bitbake -c devshell bash
10202 $ cd ..
10203 $ scp packages-split/bash/bin/bash target:/bin/bash
10204 $ cp -a packages-split/bash-dbg/\* path/debugfs
10205
10206Debugging with the GNU Project Debugger (GDB) on the Target
10207-----------------------------------------------------------
10208
10209The previous section addressed using GDB remotely for debugging
10210purposes, which is the most usual case due to the inherent hardware
10211limitations on many embedded devices. However, debugging in the target
10212hardware itself is also possible with more powerful devices. This
10213section describes what you need to do in order to support using GDB to
10214debug on the target hardware.
10215
10216To support this kind of debugging, you need do the following:
10217
10218- Ensure that GDB is on the target. You can do this by adding "gdb" to
10219 :term:`IMAGE_INSTALL`:
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010220 ::
10221
10222 IMAGE_INSTALL_append = " gdb"
10223
10224 Alternatively, you can add "tools-debug" to :term:`IMAGE_FEATURES`:
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010225 ::
10226
10227 IMAGE_FEATURES_append = " tools-debug"
10228
10229- Ensure that debug symbols are present. You can make sure these
10230 symbols are present by installing ``-dbg``:
10231 ::
10232
10233 IMAGE_INSTALL_append = "packagename-dbg"
10234
10235 Alternatively, you can do the following to include
10236 all the debug symbols:
10237 ::
10238
10239 IMAGE_FEATURES_append = " dbg-pkgs"
10240
10241.. note::
10242
10243 To improve the debug information accuracy, you can reduce the level
10244 of optimization used by the compiler. For example, when adding the
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010245 following line to your ``local.conf`` file, you will reduce optimization
10246 from :term:`FULL_OPTIMIZATION` of "-O2" to :term:`DEBUG_OPTIMIZATION`
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010247 of "-O -fno-omit-frame-pointer":
10248 ::
10249
10250 DEBUG_BUILD = "1"
10251
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010252 Consider that this will reduce the application's performance and is
10253 recommended only for debugging purposes.
10254
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010255Other Debugging Tips
10256--------------------
10257
10258Here are some other tips that you might find useful:
10259
10260- When adding new packages, it is worth watching for undesirable items
10261 making their way into compiler command lines. For example, you do not
10262 want references to local system files like ``/usr/lib/`` or
10263 ``/usr/include/``.
10264
10265- If you want to remove the ``psplash`` boot splashscreen, add
10266 ``psplash=false`` to the kernel command line. Doing so prevents
10267 ``psplash`` from loading and thus allows you to see the console. It
10268 is also possible to switch out of the splashscreen by switching the
10269 virtual console (e.g. Fn+Left or Fn+Right on a Zaurus).
10270
10271- Removing :term:`TMPDIR` (usually
10272 ``tmp/``, within the
10273 :term:`Build Directory`) can often fix
10274 temporary build issues. Removing ``TMPDIR`` is usually a relatively
10275 cheap operation, because task output will be cached in
10276 :term:`SSTATE_DIR` (usually
10277 ``sstate-cache/``, which is also in the Build Directory).
10278
10279 .. note::
10280
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010281 Removing ``TMPDIR`` might be a workaround rather than a fix.
10282 Consequently, trying to determine the underlying cause of an issue before
10283 removing the directory is a good idea.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010284
10285- Understanding how a feature is used in practice within existing
10286 recipes can be very helpful. It is recommended that you configure
10287 some method that allows you to quickly search through files.
10288
10289 Using GNU Grep, you can use the following shell function to
10290 recursively search through common recipe-related files, skipping
10291 binary files, ``.git`` directories, and the Build Directory (assuming
10292 its name starts with "build"):
10293 ::
10294
10295 g() {
10296 grep -Ir \
10297 --exclude-dir=.git \
10298 --exclude-dir='build*' \
10299 --include='*.bb*' \
10300 --include='*.inc*' \
10301 --include='*.conf*' \
10302 --include='*.py*' \
10303 "$@"
10304 }
10305
10306 Following are some usage examples:
10307 ::
10308
10309 $ g FOO # Search recursively for "FOO"
10310 $ g -i foo # Search recursively for "foo", ignoring case
10311 $ g -w FOO # Search recursively for "FOO" as a word, ignoring e.g. "FOOBAR"
10312
10313 If figuring
10314 out how some feature works requires a lot of searching, it might
10315 indicate that the documentation should be extended or improved. In
10316 such cases, consider filing a documentation bug using the Yocto
10317 Project implementation of
10318 :yocto_bugs:`Bugzilla <>`. For information on
10319 how to submit a bug against the Yocto Project, see the Yocto Project
Andrew Geissler09209ee2020-12-13 08:44:15 -060010320 Bugzilla :yocto_wiki:`wiki page </Bugzilla_Configuration_and_Bug_Tracking>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010321 and the "`Submitting a Defect Against the Yocto
10322 Project <#submitting-a-defect-against-the-yocto-project>`__" section.
10323
10324 .. note::
10325
10326 The manuals might not be the right place to document variables
10327 that are purely internal and have a limited scope (e.g. internal
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010328 variables used to implement a single ``.bbclass`` file).
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010329
10330Making Changes to the Yocto Project
10331===================================
10332
10333Because the Yocto Project is an open-source, community-based project,
10334you can effect changes to the project. This section presents procedures
10335that show you how to submit a defect against the project and how to
10336submit a change.
10337
10338Submitting a Defect Against the Yocto Project
10339---------------------------------------------
10340
10341Use the Yocto Project implementation of
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010342`Bugzilla <https://www.bugzilla.org/about/>`__ to submit a defect (bug)
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010343against the Yocto Project. For additional information on this
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010344implementation of Bugzilla see the ":ref:`Yocto Project
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010345Bugzilla <resources-bugtracker>`" section in the
10346Yocto Project Reference Manual. For more detail on any of the following
10347steps, see the Yocto Project
Andrew Geissler09209ee2020-12-13 08:44:15 -060010348:yocto_wiki:`Bugzilla wiki page </Bugzilla_Configuration_and_Bug_Tracking>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010349
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010350Use the following general steps to submit a bug:
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010351
103521. Open the Yocto Project implementation of :yocto_bugs:`Bugzilla <>`.
10353
103542. Click "File a Bug" to enter a new bug.
10355
103563. Choose the appropriate "Classification", "Product", and "Component"
10357 for which the bug was found. Bugs for the Yocto Project fall into
10358 one of several classifications, which in turn break down into
10359 several products and components. For example, for a bug against the
10360 ``meta-intel`` layer, you would choose "Build System, Metadata &
10361 Runtime", "BSPs", and "bsps-meta-intel", respectively.
10362
103634. Choose the "Version" of the Yocto Project for which you found the
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010364 bug (e.g. &DISTRO;).
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010365
103665. Determine and select the "Severity" of the bug. The severity
10367 indicates how the bug impacted your work.
10368
103696. Choose the "Hardware" that the bug impacts.
10370
103717. Choose the "Architecture" that the bug impacts.
10372
103738. Choose a "Documentation change" item for the bug. Fixing a bug might
10374 or might not affect the Yocto Project documentation. If you are
10375 unsure of the impact to the documentation, select "Don't Know".
10376
103779. Provide a brief "Summary" of the bug. Try to limit your summary to
10378 just a line or two and be sure to capture the essence of the bug.
10379
1038010. Provide a detailed "Description" of the bug. You should provide as
10381 much detail as you can about the context, behavior, output, and so
10382 forth that surrounds the bug. You can even attach supporting files
10383 for output from logs by using the "Add an attachment" button.
10384
1038511. Click the "Submit Bug" button submit the bug. A new Bugzilla number
10386 is assigned to the bug and the defect is logged in the bug tracking
10387 system.
10388
10389Once you file a bug, the bug is processed by the Yocto Project Bug
10390Triage Team and further details concerning the bug are assigned (e.g.
10391priority and owner). You are the "Submitter" of the bug and any further
10392categorization, progress, or comments on the bug result in Bugzilla
10393sending you an automated email concerning the particular change or
10394progress to the bug.
10395
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010396Submitting a Change to the Yocto Project
10397----------------------------------------
10398
10399Contributions to the Yocto Project and OpenEmbedded are very welcome.
10400Because the system is extremely configurable and flexible, we recognize
10401that developers will want to extend, configure or optimize it for their
10402specific uses.
10403
10404The Yocto Project uses a mailing list and a patch-based workflow that is
10405similar to the Linux kernel but contains important differences. In
10406general, a mailing list exists through which you can submit patches. You
10407should send patches to the appropriate mailing list so that they can be
10408reviewed and merged by the appropriate maintainer. The specific mailing
10409list you need to use depends on the location of the code you are
10410changing. Each component (e.g. layer) should have a ``README`` file that
10411indicates where to send the changes and which process to follow.
10412
10413You can send the patch to the mailing list using whichever approach you
10414feel comfortable with to generate the patch. Once sent, the patch is
10415usually reviewed by the community at large. If somebody has concerns
10416with the patch, they will usually voice their concern over the mailing
10417list. If a patch does not receive any negative reviews, the maintainer
10418of the affected layer typically takes the patch, tests it, and then
10419based on successful testing, merges the patch.
10420
10421The "poky" repository, which is the Yocto Project's reference build
10422environment, is a hybrid repository that contains several individual
10423pieces (e.g. BitBake, Metadata, documentation, and so forth) built using
10424the combo-layer tool. The upstream location used for submitting changes
10425varies by component:
10426
10427- *Core Metadata:* Send your patch to the
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010428 :oe_lists:`openembedded-core </g/openembedded-core>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010429 mailing list. For example, a change to anything under the ``meta`` or
10430 ``scripts`` directories should be sent to this mailing list.
10431
10432- *BitBake:* For changes to BitBake (i.e. anything under the
10433 ``bitbake`` directory), send your patch to the
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010434 :oe_lists:`bitbake-devel </g/bitbake-devel>`
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010435 mailing list.
10436
Andrew Geisslerc3d88e42020-10-02 09:45:00 -050010437- *"meta-\*" trees:* These trees contain Metadata. Use the
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010438 :yocto_lists:`poky </g/poky>` mailing list.
Andrew Geisslerc3d88e42020-10-02 09:45:00 -050010439
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010440- *Documentation*: For changes to the Yocto Project documentation, use the
10441 :yocto_lists:`docs </g/docs>` mailing list.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010442
10443For changes to other layers hosted in the Yocto Project source
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010444repositories (i.e. ``yoctoproject.org``) and tools use the
10445:yocto_lists:`Yocto Project </g/yocto/>` general mailing list.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010446
10447.. note::
10448
10449 Sometimes a layer's documentation specifies to use a particular
10450 mailing list. If so, use that list.
10451
10452For additional recipes that do not fit into the core Metadata, you
10453should determine which layer the recipe should go into and submit the
10454change in the manner recommended by the documentation (e.g. the
10455``README`` file) supplied with the layer. If in doubt, please ask on the
10456Yocto general mailing list or on the openembedded-devel mailing list.
10457
10458You can also push a change upstream and request a maintainer to pull the
10459change into the component's upstream repository. You do this by pushing
Andrew Geissler09209ee2020-12-13 08:44:15 -060010460to a contribution repository that is upstream. See the
10461":ref:`overview-manual/development-environment:git workflows and the yocto project`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010462section in the Yocto Project Overview and Concepts Manual for additional
10463concepts on working in the Yocto Project development environment.
10464
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010465Maintainers commonly use ``-next`` branches to test submissions prior to
10466merging patches. Thus, you can get an idea of the status of a patch based on
10467whether the patch has been merged into one of these branches. The commonly
10468used testing branches for OpenEmbedded-Core are as follows:
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010469
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010470- *openembedded-core "master-next" branch:* This branch is part of the
10471 :oe_git:`openembedded-core </openembedded-core/>` repository and contains
10472 proposed changes to the core metadata.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010473
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010474- *poky "master-next" branch:* This branch is part of the
Andrew Geissler09209ee2020-12-13 08:44:15 -060010475 :yocto_git:`poky </poky/>` repository and combines proposed
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010476 changes to bitbake, the core metadata and the poky distro.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010477
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010478Similarly, stable branches maintained by the project may have corresponding
10479``-next`` branches which collect proposed changes. For example,
10480``&DISTRO_NAME_NO_CAP;-next`` and ``&DISTRO_NAME_NO_CAP_MINUS_ONE;-next``
10481branches in both the "openembdedded-core" and "poky" repositories.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010482
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010483Other layers may have similar testing branches but there is no formal
10484requirement or standard for these so please check the documentation for the
10485layers you are contributing to.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010486
10487The following sections provide procedures for submitting a change.
10488
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010489Preparing Changes for Submission
10490~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010491
104921. *Make Your Changes Locally:* Make your changes in your local Git
10493 repository. You should make small, controlled, isolated changes.
10494 Keeping changes small and isolated aids review, makes
10495 merging/rebasing easier and keeps the change history clean should
10496 anyone need to refer to it in future.
10497
104982. *Stage Your Changes:* Stage your changes by using the ``git add``
10499 command on each file you changed.
10500
105013. *Commit Your Changes:* Commit the change by using the ``git commit``
10502 command. Make sure your commit information follows standards by
10503 following these accepted conventions:
10504
10505 - Be sure to include a "Signed-off-by:" line in the same style as
10506 required by the Linux kernel. Adding this line signifies that you,
10507 the submitter, have agreed to the Developer's Certificate of
10508 Origin 1.1 as follows:
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010509
10510 .. code-block:: none
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010511
10512 Developer's Certificate of Origin 1.1
10513
10514 By making a contribution to this project, I certify that:
10515
10516 (a) The contribution was created in whole or in part by me and I
10517 have the right to submit it under the open source license
10518 indicated in the file; or
10519
10520 (b) The contribution is based upon previous work that, to the best
10521 of my knowledge, is covered under an appropriate open source
10522 license and I have the right under that license to submit that
10523 work with modifications, whether created in whole or in part
10524 by me, under the same open source license (unless I am
10525 permitted to submit under a different license), as indicated
10526 in the file; or
10527
10528 (c) The contribution was provided directly to me by some other
10529 person who certified (a), (b) or (c) and I have not modified
10530 it.
10531
10532 (d) I understand and agree that this project and the contribution
10533 are public and that a record of the contribution (including all
10534 personal information I submit with it, including my sign-off) is
10535 maintained indefinitely and may be redistributed consistent with
10536 this project or the open source license(s) involved.
10537
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010538 - Provide a single-line summary of the change and, if more
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010539 explanation is needed, provide more detail in the body of the
10540 commit. This summary is typically viewable in the "shortlist" of
10541 changes. Thus, providing something short and descriptive that
10542 gives the reader a summary of the change is useful when viewing a
10543 list of many commits. You should prefix this short description
10544 with the recipe name (if changing a recipe), or else with the
10545 short form path to the file being changed.
10546
10547 - For the body of the commit message, provide detailed information
10548 that describes what you changed, why you made the change, and the
10549 approach you used. It might also be helpful if you mention how you
10550 tested the change. Provide as much detail as you can in the body
10551 of the commit message.
10552
10553 .. note::
10554
10555 You do not need to provide a more detailed explanation of a
10556 change if the change is minor to the point of the single line
10557 summary providing all the information.
10558
10559 - If the change addresses a specific bug or issue that is associated
10560 with a bug-tracking ID, include a reference to that ID in your
10561 detailed description. For example, the Yocto Project uses a
10562 specific convention for bug references - any commit that addresses
10563 a specific bug should use the following form for the detailed
10564 description. Be sure to use the actual bug-tracking ID from
10565 Bugzilla for bug-id:
10566 ::
10567
10568 Fixes [YOCTO #bug-id]
10569
10570 detailed description of change
10571
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010572Using Email to Submit a Patch
10573~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10574
10575Depending on the components changed, you need to submit the email to a
10576specific mailing list. For some guidance on which mailing list to use,
10577see the `list <#figuring-out-the-mailing-list-to-use>`__ at the
10578beginning of this section. For a description of all the available
10579mailing lists, see the ":ref:`Mailing Lists <resources-mailinglist>`" section in the
10580Yocto Project Reference Manual.
10581
10582Here is the general procedure on how to submit a patch through email
10583without using the scripts once the steps in
Andrew Geissler09209ee2020-12-13 08:44:15 -060010584:ref:`dev-manual/common-tasks:preparing changes for submission` have been followed:
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010585
105861. *Format the Commit:* Format the commit into an email message. To
10587 format commits, use the ``git format-patch`` command. When you
10588 provide the command, you must include a revision list or a number of
10589 patches as part of the command. For example, either of these two
10590 commands takes your most recent single commit and formats it as an
10591 email message in the current directory:
10592 ::
10593
10594 $ git format-patch -1
10595
10596 or ::
10597
10598 $ git format-patch HEAD~
10599
10600 After the command is run, the current directory contains a numbered
10601 ``.patch`` file for the commit.
10602
10603 If you provide several commits as part of the command, the
10604 ``git format-patch`` command produces a series of numbered files in
10605 the current directory – one for each commit. If you have more than
10606 one patch, you should also use the ``--cover`` option with the
10607 command, which generates a cover letter as the first "patch" in the
10608 series. You can then edit the cover letter to provide a description
10609 for the series of patches. For information on the
10610 ``git format-patch`` command, see ``GIT_FORMAT_PATCH(1)`` displayed
10611 using the ``man git-format-patch`` command.
10612
10613 .. note::
10614
10615 If you are or will be a frequent contributor to the Yocto Project
10616 or to OpenEmbedded, you might consider requesting a contrib area
10617 and the necessary associated rights.
10618
106192. *Send the patches via email:* Send the patches to the recipients and
10620 relevant mailing lists by using the ``git send-email`` command.
10621
10622 .. note::
10623
10624 In order to use ``git send-email``, you must have the proper Git packages
10625 installed on your host.
10626 For Ubuntu, Debian, and Fedora the package is ``git-email``.
10627
10628 The ``git send-email`` command sends email by using a local or remote
10629 Mail Transport Agent (MTA) such as ``msmtp``, ``sendmail``, or
10630 through a direct ``smtp`` configuration in your Git ``~/.gitconfig``
10631 file. If you are submitting patches through email only, it is very
10632 important that you submit them without any whitespace or HTML
10633 formatting that either you or your mailer introduces. The maintainer
10634 that receives your patches needs to be able to save and apply them
10635 directly from your emails. A good way to verify that what you are
10636 sending will be applicable by the maintainer is to do a dry run and
10637 send them to yourself and then save and apply them as the maintainer
10638 would.
10639
10640 The ``git send-email`` command is the preferred method for sending
10641 your patches using email since there is no risk of compromising
10642 whitespace in the body of the message, which can occur when you use
10643 your own mail client. The command also has several options that let
10644 you specify recipients and perform further editing of the email
10645 message. For information on how to use the ``git send-email``
10646 command, see ``GIT-SEND-EMAIL(1)`` displayed using the
10647 ``man git-send-email`` command.
10648
10649The Yocto Project uses a `Patchwork instance <https://patchwork.openembedded.org/>`__
10650to track the status of patches submitted to the various mailing lists and to
10651support automated patch testing. Each submitted patch is checked for common
10652mistakes and deviations from the expected patch format and submitters are
10653notified by patchtest if such mistakes are found. This process helps to
10654reduce the burden of patch review on maintainers.
10655
10656.. note::
10657
10658 This system is imperfect and changes can sometimes get lost in the flow.
10659 Asking about the status of a patch or change is reasonable if the change
10660 has been idle for a while with no feedback.
10661
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010662Using Scripts to Push a Change Upstream and Request a Pull
10663~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10664
10665For larger patch series it is preferable to send a pull request which not
10666only includes the patch but also a pointer to a branch that can be pulled
10667from. This involves making a local branch for your changes, pushing this
10668branch to an accessible repository and then using the ``create-pull-request``
10669and ``send-pull-request`` scripts from openembedded-core to create and send a
10670patch series with a link to the branch for review.
10671
10672Follow this procedure to push a change to an upstream "contrib" Git
Andrew Geissler09209ee2020-12-13 08:44:15 -060010673repository once the steps in :ref:`dev-manual/common-tasks:preparing changes for submission` have
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010674been followed:
10675
10676.. note::
10677
10678 You can find general Git information on how to push a change upstream
10679 in the
10680 `Git Community Book <https://git-scm.com/book/en/v2/Distributed-Git-Distributed-Workflows>`__.
10681
106821. *Push Your Commits to a "Contrib" Upstream:* If you have arranged for
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010683 permissions to push to an upstream contrib repository, push the
10684 change to that repository:
10685 ::
10686
10687 $ git push upstream_remote_repo local_branch_name
10688
10689 For example, suppose you have permissions to push
10690 into the upstream ``meta-intel-contrib`` repository and you are
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010691 working in a local branch named `your_name`\ ``/README``. The following
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010692 command pushes your local commits to the ``meta-intel-contrib``
10693 upstream repository and puts the commit in a branch named
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010694 `your_name`\ ``/README``:
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010695 ::
10696
10697 $ git push meta-intel-contrib your_name/README
10698
Andrew Geissler6ce62a22020-11-30 19:58:47 -0600106992. *Determine Who to Notify:* Determine the maintainer or the mailing
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010700 list that you need to notify for the change.
10701
10702 Before submitting any change, you need to be sure who the maintainer
10703 is or what mailing list that you need to notify. Use either these
10704 methods to find out:
10705
10706 - *Maintenance File:* Examine the ``maintainers.inc`` file, which is
10707 located in the :term:`Source Directory` at
10708 ``meta/conf/distro/include``, to see who is responsible for code.
10709
Andrew Geissler09209ee2020-12-13 08:44:15 -060010710 - *Search by File:* Using :ref:`overview-manual/development-environment:git`, you can
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010711 enter the following command to bring up a short list of all
10712 commits against a specific file:
10713 ::
10714
10715 git shortlog -- filename
10716
10717 Just provide the name of the file for which you are interested. The
10718 information returned is not ordered by history but does include a
10719 list of everyone who has committed grouped by name. From the list,
10720 you can see who is responsible for the bulk of the changes against
10721 the file.
10722
10723 - *Examine the List of Mailing Lists:* For a list of the Yocto
10724 Project and related mailing lists, see the ":ref:`Mailing
10725 lists <resources-mailinglist>`" section in
10726 the Yocto Project Reference Manual.
10727
Andrew Geissler6ce62a22020-11-30 19:58:47 -0600107283. *Make a Pull Request:* Notify the maintainer or the mailing list that
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010729 you have pushed a change by making a pull request.
10730
10731 The Yocto Project provides two scripts that conveniently let you
10732 generate and send pull requests to the Yocto Project. These scripts
10733 are ``create-pull-request`` and ``send-pull-request``. You can find
10734 these scripts in the ``scripts`` directory within the
10735 :term:`Source Directory` (e.g.
10736 ``~/poky/scripts``).
10737
10738 Using these scripts correctly formats the requests without
10739 introducing any whitespace or HTML formatting. The maintainer that
10740 receives your patches either directly or through the mailing list
10741 needs to be able to save and apply them directly from your emails.
10742 Using these scripts is the preferred method for sending patches.
10743
10744 First, create the pull request. For example, the following command
10745 runs the script, specifies the upstream repository in the contrib
10746 directory into which you pushed the change, and provides a subject
10747 line in the created patch files:
10748 ::
10749
10750 $ ~/poky/scripts/create-pull-request -u meta-intel-contrib -s "Updated Manual Section Reference in README"
10751
10752 Running this script forms ``*.patch`` files in a folder named
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010753 ``pull-``\ `PID` in the current directory. One of the patch files is a
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010754 cover letter.
10755
10756 Before running the ``send-pull-request`` script, you must edit the
10757 cover letter patch to insert information about your change. After
10758 editing the cover letter, send the pull request. For example, the
10759 following command runs the script and specifies the patch directory
10760 and email address. In this example, the email address is a mailing
10761 list:
10762 ::
10763
10764 $ ~/poky/scripts/send-pull-request -p ~/meta-intel/pull-10565 -t meta-intel@yoctoproject.org
10765
10766 You need to follow the prompts as the script is interactive.
10767
10768 .. note::
10769
Andrew Geissler4c19ea12020-10-27 13:52:24 -050010770 For help on using these scripts, simply provide the ``-h``
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010771 argument as follows:
10772 ::
10773
10774 $ poky/scripts/create-pull-request -h
10775 $ poky/scripts/send-pull-request -h
10776
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010777Responding to Patch Review
10778~~~~~~~~~~~~~~~~~~~~~~~~~~
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010779
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010780You may get feedback on your submitted patches from other community members
10781or from the automated patchtest service. If issues are identified in your
10782patch then it is usually necessary to address these before the patch will be
10783accepted into the project. In this case you should amend the patch according
10784to the feedback and submit an updated version to the relevant mailing list,
10785copying in the reviewers who provided feedback to the previous version of the
10786patch.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010787
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010788The patch should be amended using ``git commit --amend`` or perhaps ``git
10789rebase`` for more expert git users. You should also modify the ``[PATCH]``
10790tag in the email subject line when sending the revised patch to mark the new
10791iteration as ``[PATCH v2]``, ``[PATCH v3]``, etc as appropriate. This can be
10792done by passing the ``-v`` argument to ``git format-patch`` with a version
10793number.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010794
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010795Lastly please ensure that you also test your revised changes. In particular
10796please don't just edit the patch file written out by ``git format-patch`` and
10797resend it.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010798
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010799Submitting Changes to Stable Release Branches
10800~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010801
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010802The process for proposing changes to a Yocto Project stable branch differs
10803from the steps described above. Changes to a stable branch must address
10804identified bugs or CVEs and should be made carefully in order to avoid the
10805risk of introducing new bugs or breaking backwards compatibility. Typically
10806bug fixes must already be accepted into the master branch before they can be
10807backported to a stable branch unless the bug in question does not affect the
10808master branch or the fix on the master branch is unsuitable for backporting.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010809
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010810The list of stable branches along with the status and maintainer for each
10811branch can be obtained from the
Andrew Geissler09209ee2020-12-13 08:44:15 -060010812:yocto_wiki:`Releases wiki page </Releases>`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010813
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010814.. note::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010815
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010816 Changes will not typically be accepted for branches which are marked as
10817 End-Of-Life (EOL).
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010818
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010819With this in mind, the steps to submit a change for a stable branch are as
10820follows:
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010821
Andrew Geissler6ce62a22020-11-30 19:58:47 -0600108221. *Identify the bug or CVE to be fixed:* This information should be
10823 collected so that it can be included in your submission.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010824
Andrew Geissler6ce62a22020-11-30 19:58:47 -0600108252. *Check if the fix is already present in the master branch:* This will
10826 result in the most straightforward path into the stable branch for the
10827 fix.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010828
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010829 a. *If the fix is present in the master branch - Submit a backport request
10830 by email:* You should send an email to the relevant stable branch
10831 maintainer and the mailing list with details of the bug or CVE to be
10832 fixed, the commit hash on the master branch that fixes the issue and
10833 the stable branches which you would like this fix to be backported to.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010834
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010835 b. *If the fix is not present in the master branch - Submit the fix to the
10836 master branch first:* This will ensure that the fix passes through the
10837 project's usual patch review and test processes before being accepted.
10838 It will also ensure that bugs are not left unresolved in the master
10839 branch itself. Once the fix is accepted in the master branch a backport
10840 request can be submitted as above.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010841
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010842 c. *If the fix is unsuitable for the master branch - Submit a patch
10843 directly for the stable branch:* This method should be considered as a
10844 last resort. It is typically necessary when the master branch is using
10845 a newer version of the software which includes an upstream fix for the
10846 issue or when the issue has been fixed on the master branch in a way
10847 that introduces backwards incompatible changes. In this case follow the
Andrew Geissler09209ee2020-12-13 08:44:15 -060010848 steps in :ref:`dev-manual/common-tasks:preparing changes for submission` and
10849 :ref:`dev-manual/common-tasks:using email to submit a patch` but modify the subject header of your patch
Andrew Geissler6ce62a22020-11-30 19:58:47 -060010850 email to include the name of the stable branch which you are
10851 targetting. This can be done using the ``--subject-prefix`` argument to
10852 ``git format-patch``, for example to submit a patch to the dunfell
10853 branch use
10854 ``git format-patch --subject-prefix='&DISTRO_NAME_NO_CAP_MINUS_ONE;][PATCH' ...``.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010855
10856Working With Licenses
10857=====================
10858
Andrew Geissler09209ee2020-12-13 08:44:15 -060010859As mentioned in the ":ref:`overview-manual/development-environment:licensing`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010860section in the Yocto Project Overview and Concepts Manual, open source
10861projects are open to the public and they consequently have different
10862licensing structures in place. This section describes the mechanism by
10863which the :term:`OpenEmbedded Build System`
10864tracks changes to
10865licensing text and covers how to maintain open source license compliance
10866during your project's lifecycle. The section also describes how to
10867enable commercially licensed recipes, which by default are disabled.
10868
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010869Tracking License Changes
10870------------------------
10871
10872The license of an upstream project might change in the future. In order
10873to prevent these changes going unnoticed, the
10874:term:`LIC_FILES_CHKSUM`
10875variable tracks changes to the license text. The checksums are validated
10876at the end of the configure step, and if the checksums do not match, the
10877build will fail.
10878
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010879Specifying the ``LIC_FILES_CHKSUM`` Variable
10880~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10881
10882The ``LIC_FILES_CHKSUM`` variable contains checksums of the license text
10883in the source code for the recipe. Following is an example of how to
10884specify ``LIC_FILES_CHKSUM``:
10885::
10886
10887 LIC_FILES_CHKSUM = "file://COPYING;md5=xxxx \
10888 file://licfile1.txt;beginline=5;endline=29;md5=yyyy \
10889 file://licfile2.txt;endline=50;md5=zzzz \
10890 ..."
10891
10892.. note::
10893
10894 - When using "beginline" and "endline", realize that line numbering
10895 begins with one and not zero. Also, the included lines are
10896 inclusive (i.e. lines five through and including 29 in the
10897 previous example for ``licfile1.txt``).
10898
10899 - When a license check fails, the selected license text is included
10900 as part of the QA message. Using this output, you can determine
10901 the exact start and finish for the needed license text.
10902
10903The build system uses the :term:`S`
10904variable as the default directory when searching files listed in
10905``LIC_FILES_CHKSUM``. The previous example employs the default
10906directory.
10907
10908Consider this next example:
10909::
10910
10911 LIC_FILES_CHKSUM = "file://src/ls.c;beginline=5;endline=16;\
10912 md5=bb14ed3c4cda583abc85401304b5cd4e"
10913 LIC_FILES_CHKSUM = "file://${WORKDIR}/license.html;md5=5c94767cedb5d6987c902ac850ded2c6"
10914
10915The first line locates a file in ``${S}/src/ls.c`` and isolates lines
10916five through 16 as license text. The second line refers to a file in
10917:term:`WORKDIR`.
10918
10919Note that ``LIC_FILES_CHKSUM`` variable is mandatory for all recipes,
10920unless the ``LICENSE`` variable is set to "CLOSED".
10921
Andrew Geisslerc9f78652020-09-18 14:11:35 -050010922Explanation of Syntax
10923~~~~~~~~~~~~~~~~~~~~~
10924
10925As mentioned in the previous section, the ``LIC_FILES_CHKSUM`` variable
10926lists all the important files that contain the license text for the
10927source code. It is possible to specify a checksum for an entire file, or
10928a specific section of a file (specified by beginning and ending line
10929numbers with the "beginline" and "endline" parameters, respectively).
10930The latter is useful for source files with a license notice header,
10931README documents, and so forth. If you do not use the "beginline"
10932parameter, then it is assumed that the text begins on the first line of
10933the file. Similarly, if you do not use the "endline" parameter, it is
10934assumed that the license text ends with the last line of the file.
10935
10936The "md5" parameter stores the md5 checksum of the license text. If the
10937license text changes in any way as compared to this parameter then a
10938mismatch occurs. This mismatch triggers a build failure and notifies the
10939developer. Notification allows the developer to review and address the
10940license text changes. Also note that if a mismatch occurs during the
10941build, the correct md5 checksum is placed in the build log and can be
10942easily copied to the recipe.
10943
10944There is no limit to how many files you can specify using the
10945``LIC_FILES_CHKSUM`` variable. Generally, however, every project
10946requires a few specifications for license tracking. Many projects have a
10947"COPYING" file that stores the license information for all the source
10948code files. This practice allows you to just track the "COPYING" file as
10949long as it is kept up to date.
10950
10951.. note::
10952
10953 - If you specify an empty or invalid "md5" parameter,
10954 :term:`BitBake` returns an md5
10955 mis-match error and displays the correct "md5" parameter value
10956 during the build. The correct parameter is also captured in the
10957 build log.
10958
10959 - If the whole file contains only license text, you do not need to
10960 use the "beginline" and "endline" parameters.
10961
10962Enabling Commercially Licensed Recipes
10963--------------------------------------
10964
10965By default, the OpenEmbedded build system disables components that have
10966commercial or other special licensing requirements. Such requirements
10967are defined on a recipe-by-recipe basis through the
10968:term:`LICENSE_FLAGS` variable
10969definition in the affected recipe. For instance, the
10970``poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly`` recipe
10971contains the following statement:
10972::
10973
10974 LICENSE_FLAGS = "commercial"
10975
10976Here is a
10977slightly more complicated example that contains both an explicit recipe
10978name and version (after variable expansion):
10979::
10980
10981 LICENSE_FLAGS = "license_${PN}_${PV}"
10982
10983In order for a component restricted by a
10984``LICENSE_FLAGS`` definition to be enabled and included in an image, it
10985needs to have a matching entry in the global
10986:term:`LICENSE_FLAGS_WHITELIST`
10987variable, which is a variable typically defined in your ``local.conf``
10988file. For example, to enable the
10989``poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly`` package, you
10990could add either the string "commercial_gst-plugins-ugly" or the more
10991general string "commercial" to ``LICENSE_FLAGS_WHITELIST``. See the
10992"`License Flag Matching <#license-flag-matching>`__" section for a full
10993explanation of how ``LICENSE_FLAGS`` matching works. Here is the
10994example:
10995::
10996
10997 LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly"
10998
10999Likewise, to additionally enable the package built from the recipe
11000containing ``LICENSE_FLAGS = "license_${PN}_${PV}"``, and assuming that
11001the actual recipe name was ``emgd_1.10.bb``, the following string would
11002enable that package as well as the original ``gst-plugins-ugly``
11003package:
11004::
11005
11006 LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly license_emgd_1.10"
11007
11008As a convenience, you do not need to specify the
11009complete license string in the whitelist for every package. You can use
11010an abbreviated form, which consists of just the first portion or
11011portions of the license string before the initial underscore character
11012or characters. A partial string will match any license that contains the
11013given string as the first portion of its license. For example, the
11014following whitelist string will also match both of the packages
11015previously mentioned as well as any other packages that have licenses
11016starting with "commercial" or "license".
11017::
11018
11019 LICENSE_FLAGS_WHITELIST = "commercial license"
11020
11021License Flag Matching
11022~~~~~~~~~~~~~~~~~~~~~
11023
11024License flag matching allows you to control what recipes the
11025OpenEmbedded build system includes in the build. Fundamentally, the
11026build system attempts to match ``LICENSE_FLAGS`` strings found in
11027recipes against ``LICENSE_FLAGS_WHITELIST`` strings found in the
11028whitelist. A match causes the build system to include a recipe in the
11029build, while failure to find a match causes the build system to exclude
11030a recipe.
11031
11032In general, license flag matching is simple. However, understanding some
11033concepts will help you correctly and effectively use matching.
11034
11035Before a flag defined by a particular recipe is tested against the
11036contents of the whitelist, the expanded string ``_${PN}`` is appended to
11037the flag. This expansion makes each ``LICENSE_FLAGS`` value
11038recipe-specific. After expansion, the string is then matched against the
11039whitelist. Thus, specifying ``LICENSE_FLAGS = "commercial"`` in recipe
11040"foo", for example, results in the string ``"commercial_foo"``. And, to
11041create a match, that string must appear in the whitelist.
11042
11043Judicious use of the ``LICENSE_FLAGS`` strings and the contents of the
11044``LICENSE_FLAGS_WHITELIST`` variable allows you a lot of flexibility for
11045including or excluding recipes based on licensing. For example, you can
11046broaden the matching capabilities by using license flags string subsets
11047in the whitelist.
11048
11049.. note::
11050
11051 When using a string subset, be sure to use the part of the expanded
11052 string that precedes the appended underscore character (e.g.
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011053 ``usethispart_1.3``, ``usethispart_1.4``, and so forth).
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011054
11055For example, simply specifying the string "commercial" in the whitelist
11056matches any expanded ``LICENSE_FLAGS`` definition that starts with the
11057string "commercial" such as "commercial_foo" and "commercial_bar", which
11058are the strings the build system automatically generates for
11059hypothetical recipes named "foo" and "bar" assuming those recipes simply
11060specify the following:
11061::
11062
11063 LICENSE_FLAGS = "commercial"
11064
11065Thus, you can choose
11066to exhaustively enumerate each license flag in the whitelist and allow
11067only specific recipes into the image, or you can use a string subset
11068that causes a broader range of matches to allow a range of recipes into
11069the image.
11070
11071This scheme works even if the ``LICENSE_FLAGS`` string already has
11072``_${PN}`` appended. For example, the build system turns the license
11073flag "commercial_1.2_foo" into "commercial_1.2_foo_foo" and would match
11074both the general "commercial" and the specific "commercial_1.2_foo"
11075strings found in the whitelist, as expected.
11076
11077Here are some other scenarios:
11078
11079- You can specify a versioned string in the recipe such as
11080 "commercial_foo_1.2" in a "foo" recipe. The build system expands this
11081 string to "commercial_foo_1.2_foo". Combine this license flag with a
11082 whitelist that has the string "commercial" and you match the flag
11083 along with any other flag that starts with the string "commercial".
11084
11085- Under the same circumstances, you can use "commercial_foo" in the
11086 whitelist and the build system not only matches "commercial_foo_1.2"
11087 but also matches any license flag with the string "commercial_foo",
11088 regardless of the version.
11089
11090- You can be very specific and use both the package and version parts
11091 in the whitelist (e.g. "commercial_foo_1.2") to specifically match a
11092 versioned recipe.
11093
11094Other Variables Related to Commercial Licenses
11095~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
11096
11097Other helpful variables related to commercial license handling exist and
11098are defined in the
11099``poky/meta/conf/distro/include/default-distrovars.inc`` file:
11100::
11101
11102 COMMERCIAL_AUDIO_PLUGINS ?= ""
11103 COMMERCIAL_VIDEO_PLUGINS ?= ""
11104
11105If you
11106want to enable these components, you can do so by making sure you have
11107statements similar to the following in your ``local.conf`` configuration
11108file:
11109::
11110
11111 COMMERCIAL_AUDIO_PLUGINS = "gst-plugins-ugly-mad \
11112 gst-plugins-ugly-mpegaudioparse"
11113 COMMERCIAL_VIDEO_PLUGINS = "gst-plugins-ugly-mpeg2dec \
11114 gst-plugins-ugly-mpegstream gst-plugins-bad-mpegvideoparse"
11115 LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly commercial_gst-plugins-bad commercial_qmmp"
11116
11117
11118Of course, you could also create a matching whitelist for those
11119components using the more general "commercial" in the whitelist, but
11120that would also enable all the other packages with ``LICENSE_FLAGS``
11121containing "commercial", which you may or may not want:
11122::
11123
11124 LICENSE_FLAGS_WHITELIST = "commercial"
11125
11126Specifying audio and video plugins as part of the
11127``COMMERCIAL_AUDIO_PLUGINS`` and ``COMMERCIAL_VIDEO_PLUGINS`` statements
11128(along with the enabling ``LICENSE_FLAGS_WHITELIST``) includes the
11129plugins or components into built images, thus adding support for media
11130formats or components.
11131
11132Maintaining Open Source License Compliance During Your Product's Lifecycle
11133--------------------------------------------------------------------------
11134
11135One of the concerns for a development organization using open source
11136software is how to maintain compliance with various open source
11137licensing during the lifecycle of the product. While this section does
11138not provide legal advice or comprehensively cover all scenarios, it does
11139present methods that you can use to assist you in meeting the compliance
11140requirements during a software release.
11141
11142With hundreds of different open source licenses that the Yocto Project
11143tracks, it is difficult to know the requirements of each and every
11144license. However, the requirements of the major FLOSS licenses can begin
11145to be covered by assuming that three main areas of concern exist:
11146
11147- Source code must be provided.
11148
11149- License text for the software must be provided.
11150
11151- Compilation scripts and modifications to the source code must be
11152 provided.
11153
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011154- spdx files can be provided.
11155
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011156There are other requirements beyond the scope of these three and the
11157methods described in this section (e.g. the mechanism through which
11158source code is distributed).
11159
11160As different organizations have different methods of complying with open
11161source licensing, this section is not meant to imply that there is only
11162one single way to meet your compliance obligations, but rather to
11163describe one method of achieving compliance. The remainder of this
11164section describes methods supported to meet the previously mentioned
11165three requirements. Once you take steps to meet these requirements, and
11166prior to releasing images, sources, and the build system, you should
11167audit all artifacts to ensure completeness.
11168
11169.. note::
11170
11171 The Yocto Project generates a license manifest during image creation
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011172 that is located in ``${DEPLOY_DIR}/licenses/``\ `image_name`\ ``-``\ `datestamp`
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011173 to assist with any audits.
11174
11175Providing the Source Code
11176~~~~~~~~~~~~~~~~~~~~~~~~~
11177
11178Compliance activities should begin before you generate the final image.
11179The first thing you should look at is the requirement that tops the list
11180for most compliance groups - providing the source. The Yocto Project has
11181a few ways of meeting this requirement.
11182
11183One of the easiest ways to meet this requirement is to provide the
11184entire :term:`DL_DIR` used by the
11185build. This method, however, has a few issues. The most obvious is the
11186size of the directory since it includes all sources used in the build
11187and not just the source used in the released image. It will include
11188toolchain source, and other artifacts, which you would not generally
11189release. However, the more serious issue for most companies is
11190accidental release of proprietary software. The Yocto Project provides
11191an :ref:`archiver <ref-classes-archiver>` class to
11192help avoid some of these concerns.
11193
11194Before you employ ``DL_DIR`` or the ``archiver`` class, you need to
11195decide how you choose to provide source. The source ``archiver`` class
11196can generate tarballs and SRPMs and can create them with various levels
11197of compliance in mind.
11198
11199One way of doing this (but certainly not the only way) is to release
11200just the source as a tarball. You can do this by adding the following to
11201the ``local.conf`` file found in the
11202:term:`Build Directory`:
11203::
11204
11205 INHERIT += "archiver"
11206 ARCHIVER_MODE[src] = "original"
11207
11208During the creation of your
11209image, the source from all recipes that deploy packages to the image is
11210placed within subdirectories of ``DEPLOY_DIR/sources`` based on the
11211:term:`LICENSE` for each recipe.
11212Releasing the entire directory enables you to comply with requirements
11213concerning providing the unmodified source. It is important to note that
11214the size of the directory can get large.
11215
11216A way to help mitigate the size issue is to only release tarballs for
11217licenses that require the release of source. Let us assume you are only
11218concerned with GPL code as identified by running the following script:
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011219
11220.. code-block:: shell
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011221
11222 # Script to archive a subset of packages matching specific license(s)
11223 # Source and license files are copied into sub folders of package folder
11224 # Must be run from build folder
11225 #!/bin/bash
11226 src_release_dir="source-release"
11227 mkdir -p $src_release_dir
11228 for a in tmp/deploy/sources/*; do
11229 for d in $a/*; do
11230 # Get package name from path
11231 p=`basename $d`
11232 p=${p%-*}
11233 p=${p%-*}
11234 # Only archive GPL packages (update *GPL* regex for your license check)
11235 numfiles=`ls tmp/deploy/licenses/$p/*GPL* 2> /dev/null | wc -l`
11236 if [ $numfiles -gt 1 ]; then
11237 echo Archiving $p
11238 mkdir -p $src_release_dir/$p/source
11239 cp $d/* $src_release_dir/$p/source 2> /dev/null
11240 mkdir -p $src_release_dir/$p/license
11241 cp tmp/deploy/licenses/$p/* $src_release_dir/$p/license 2> /dev/null
11242 fi
11243 done
11244 done
11245
11246At this point, you
11247could create a tarball from the ``gpl_source_release`` directory and
11248provide that to the end user. This method would be a step toward
11249achieving compliance with section 3a of GPLv2 and with section 6 of
11250GPLv3.
11251
11252Providing License Text
11253~~~~~~~~~~~~~~~~~~~~~~
11254
11255One requirement that is often overlooked is inclusion of license text.
11256This requirement also needs to be dealt with prior to generating the
11257final image. Some licenses require the license text to accompany the
11258binary. You can achieve this by adding the following to your
11259``local.conf`` file:
11260::
11261
11262 COPY_LIC_MANIFEST = "1"
11263 COPY_LIC_DIRS = "1"
11264 LICENSE_CREATE_PACKAGE = "1"
11265
11266Adding these statements to the
11267configuration file ensures that the licenses collected during package
11268generation are included on your image.
11269
11270.. note::
11271
11272 Setting all three variables to "1" results in the image having two
11273 copies of the same license file. One copy resides in
11274 ``/usr/share/common-licenses`` and the other resides in
11275 ``/usr/share/license``.
11276
11277 The reason for this behavior is because
11278 :term:`COPY_LIC_DIRS` and
11279 :term:`COPY_LIC_MANIFEST`
11280 add a copy of the license when the image is built but do not offer a
11281 path for adding licenses for newly installed packages to an image.
11282 :term:`LICENSE_CREATE_PACKAGE`
11283 adds a separate package and an upgrade path for adding licenses to an
11284 image.
11285
11286As the source ``archiver`` class has already archived the original
11287unmodified source that contains the license files, you would have
11288already met the requirements for inclusion of the license information
11289with source as defined by the GPL and other open source licenses.
11290
11291Providing Compilation Scripts and Source Code Modifications
11292~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
11293
11294At this point, we have addressed all we need to prior to generating the
11295image. The next two requirements are addressed during the final
11296packaging of the release.
11297
11298By releasing the version of the OpenEmbedded build system and the layers
11299used during the build, you will be providing both compilation scripts
11300and the source code modifications in one step.
11301
Andrew Geissler09209ee2020-12-13 08:44:15 -060011302If the deployment team has a :ref:`overview-manual/concepts:bsp layer`
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011303and a distro layer, and those
11304those layers are used to patch, compile, package, or modify (in any way)
11305any open source software included in your released images, you might be
11306required to release those layers under section 3 of GPLv2 or section 1
11307of GPLv3. One way of doing that is with a clean checkout of the version
11308of the Yocto Project and layers used during your build. Here is an
11309example:
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011310
11311.. code-block:: shell
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011312
11313 # We built using the dunfell branch of the poky repo
11314 $ git clone -b dunfell git://git.yoctoproject.org/poky
11315 $ cd poky
11316 # We built using the release_branch for our layers
11317 $ git clone -b release_branch git://git.mycompany.com/meta-my-bsp-layer
11318 $ git clone -b release_branch git://git.mycompany.com/meta-my-software-layer
11319 # clean up the .git repos
11320 $ find . -name ".git" -type d -exec rm -rf {} \;
11321
11322One
11323thing a development organization might want to consider for end-user
11324convenience is to modify ``meta-poky/conf/bblayers.conf.sample`` to
11325ensure that when the end user utilizes the released build system to
11326build an image, the development organization's layers are included in
11327the ``bblayers.conf`` file automatically:
11328::
11329
11330 # POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
11331 # changes incompatibly
11332 POKY_BBLAYERS_CONF_VERSION = "2"
11333
11334 BBPATH = "${TOPDIR}"
11335 BBFILES ?= ""
11336
11337 BBLAYERS ?= " \
11338 ##OEROOT##/meta \
11339 ##OEROOT##/meta-poky \
11340 ##OEROOT##/meta-yocto-bsp \
11341 ##OEROOT##/meta-mylayer \
11342 "
11343
11344Creating and
11345providing an archive of the :term:`Metadata`
11346layers (recipes, configuration files, and so forth) enables you to meet
11347your requirements to include the scripts to control compilation as well
11348as any modifications to the original source.
11349
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011350Providing spdx files
11351~~~~~~~~~~~~~~~~~~~~~~~~~
11352
11353The spdx module has been integrated to a layer named meta-spdxscanner.
11354meta-spdxscanner provides several kinds of scanner. If you want to enable
11355this function, you have to follow the following steps:
11356
113571. Add meta-spdxscanner layer into ``bblayers.conf``.
11358
113592. Refer to the README in meta-spdxscanner to setup the environment (e.g,
11360 setup a fossology server) needed for the scanner.
11361
113623. Meta-spdxscanner provides several methods within the bbclass to create spdx files.
11363 Please choose one that you want to use and enable the spdx task. You have to
11364 add some config options in ``local.conf`` file in your :term:`Build
11365 Directory`. The following is an example showing how to generate spdx files
11366 during bitbake using the fossology-python.bbclass::
11367
11368 # Select fossology-python.bbclass.
11369 INHERIT += "fossology-python"
11370 # For fossology-python.bbclass, TOKEN is necessary, so, after setup a
11371 # Fossology server, you have to create a token.
11372 TOKEN = "eyJ0eXAiO..."
11373 # The fossology server is necessary for fossology-python.bbclass.
11374 FOSSOLOGY_SERVER = "http://xx.xx.xx.xx:8081/repo"
11375 # If you want to upload the source code to a special folder:
11376 FOLDER_NAME = "xxxx" //Optional
11377 # If you don't want to put spdx files in tmp/deploy/spdx, you can enable:
11378 SPDX_DEPLOY_DIR = "${DEPLOY_DIR}" //Optional
11379
11380For more usage information refer to :yocto_git:`the meta-spdxscanner repository
Andrew Geissler09209ee2020-12-13 08:44:15 -060011381</meta-spdxscanner/>`.
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011382
11383
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011384Copying Licenses that Do Not Exist
11385----------------------------------
11386
11387Some packages, such as the linux-firmware package, have many licenses
11388that are not in any way common. You can avoid adding a lot of these
11389types of common license files, which are only applicable to a specific
11390package, by using the
11391:term:`NO_GENERIC_LICENSE`
11392variable. Using this variable also avoids QA errors when you use a
11393non-common, non-CLOSED license in a recipe.
11394
11395The following is an example that uses the ``LICENSE.Abilis.txt`` file as
11396the license from the fetched source:
11397::
11398
11399 NO_GENERIC_LICENSE[Firmware-Abilis] = "LICENSE.Abilis.txt"
11400
11401Using the Error Reporting Tool
11402==============================
11403
11404The error reporting tool allows you to submit errors encountered during
11405builds to a central database. Outside of the build environment, you can
11406use a web interface to browse errors, view statistics, and query for
11407errors. The tool works using a client-server system where the client
11408portion is integrated with the installed Yocto Project
11409:term:`Source Directory` (e.g. ``poky``).
11410The server receives the information collected and saves it in a
11411database.
11412
11413A live instance of the error reporting server exists at
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011414https://errors.yoctoproject.org. This server exists so that when
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011415you want to get help with build failures, you can submit all of the
11416information on the failure easily and then point to the URL in your bug
11417report or send an email to the mailing list.
11418
11419.. note::
11420
11421 If you send error reports to this server, the reports become publicly
11422 visible.
11423
11424Enabling and Using the Tool
11425---------------------------
11426
11427By default, the error reporting tool is disabled. You can enable it by
11428inheriting the
11429:ref:`report-error <ref-classes-report-error>`
11430class by adding the following statement to the end of your
11431``local.conf`` file in your
11432:term:`Build Directory`.
11433::
11434
11435 INHERIT += "report-error"
11436
11437By default, the error reporting feature stores information in
11438``${``\ :term:`LOG_DIR`\ ``}/error-report``.
11439However, you can specify a directory to use by adding the following to
11440your ``local.conf`` file:
11441::
11442
11443 ERR_REPORT_DIR = "path"
11444
11445Enabling error
11446reporting causes the build process to collect the errors and store them
11447in a file as previously described. When the build system encounters an
11448error, it includes a command as part of the console output. You can run
11449the command to send the error file to the server. For example, the
11450following command sends the errors to an upstream server:
11451::
11452
11453 $ send-error-report /home/brandusa/project/poky/build/tmp/log/error-report/error_report_201403141617.txt
11454
11455In the previous example, the errors are sent to a public database
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011456available at https://errors.yoctoproject.org, which is used by the
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011457entire community. If you specify a particular server, you can send the
11458errors to a different database. Use the following command for more
11459information on available options:
11460::
11461
11462 $ send-error-report --help
11463
11464When sending the error file, you are prompted to review the data being
11465sent as well as to provide a name and optional email address. Once you
11466satisfy these prompts, the command returns a link from the server that
11467corresponds to your entry in the database. For example, here is a
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011468typical link: https://errors.yoctoproject.org/Errors/Details/9522/
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011469
11470Following the link takes you to a web interface where you can browse,
11471query the errors, and view statistics.
11472
11473Disabling the Tool
11474------------------
11475
11476To disable the error reporting feature, simply remove or comment out the
11477following statement from the end of your ``local.conf`` file in your
11478:term:`Build Directory`.
11479::
11480
11481 INHERIT += "report-error"
11482
11483Setting Up Your Own Error Reporting Server
11484------------------------------------------
11485
11486If you want to set up your own error reporting server, you can obtain
Andrew Geissler09209ee2020-12-13 08:44:15 -060011487the code from the Git repository at :yocto_git:`/error-report-web/`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011488Instructions on how to set it up are in the README document.
11489
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011490Using Wayland and Weston
11491========================
11492
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011493`Wayland <https://en.wikipedia.org/wiki/Wayland_(display_server_protocol)>`__
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011494is a computer display server protocol that provides a method for
11495compositing window managers to communicate directly with applications
11496and video hardware and expects them to communicate with input hardware
11497using other libraries. Using Wayland with supporting targets can result
11498in better control over graphics frame rendering than an application
11499might otherwise achieve.
11500
11501The Yocto Project provides the Wayland protocol libraries and the
11502reference
Andrew Geissler4c19ea12020-10-27 13:52:24 -050011503`Weston <https://en.wikipedia.org/wiki/Wayland_(display_server_protocol)#Weston>`__
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011504compositor as part of its release. You can find the integrated packages
11505in the ``meta`` layer of the :term:`Source Directory`.
11506Specifically, you
11507can find the recipes that build both Wayland and Weston at
11508``meta/recipes-graphics/wayland``.
11509
11510You can build both the Wayland and Weston packages for use only with
11511targets that accept the `Mesa 3D and Direct Rendering
11512Infrastructure <https://en.wikipedia.org/wiki/Mesa_(computer_graphics)>`__,
11513which is also known as Mesa DRI. This implies that you cannot build and
11514use the packages if your target uses, for example, the Intel Embedded
11515Media and Graphics Driver (Intel EMGD) that overrides Mesa DRI.
11516
11517.. note::
11518
11519 Due to lack of EGL support, Weston 1.0.3 will not run directly on the
11520 emulated QEMU hardware. However, this version of Weston will run
11521 under X emulation without issues.
11522
11523This section describes what you need to do to implement Wayland and use
11524the Weston compositor when building an image for a supporting target.
11525
11526Enabling Wayland in an Image
11527----------------------------
11528
11529To enable Wayland, you need to enable it to be built and enable it to be
11530included (installed) in the image.
11531
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011532Building Wayland
11533~~~~~~~~~~~~~~~~
11534
11535To cause Mesa to build the ``wayland-egl`` platform and Weston to build
11536Wayland with Kernel Mode Setting
11537(`KMS <https://wiki.archlinux.org/index.php/Kernel_Mode_Setting>`__)
11538support, include the "wayland" flag in the
11539:term:`DISTRO_FEATURES`
11540statement in your ``local.conf`` file:
11541::
11542
11543 DISTRO_FEATURES_append = " wayland"
11544
11545.. note::
11546
11547 If X11 has been enabled elsewhere, Weston will build Wayland with X11
11548 support
11549
Andrew Geisslerc9f78652020-09-18 14:11:35 -050011550Installing Wayland and Weston
11551~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
11552
11553To install the Wayland feature into an image, you must include the
11554following
11555:term:`CORE_IMAGE_EXTRA_INSTALL`
11556statement in your ``local.conf`` file:
11557::
11558
11559 CORE_IMAGE_EXTRA_INSTALL += "wayland weston"
11560
11561Running Weston
11562--------------
11563
11564To run Weston inside X11, enabling it as described earlier and building
11565a Sato image is sufficient. If you are running your image under Sato, a
11566Weston Launcher appears in the "Utility" category.
11567
11568Alternatively, you can run Weston through the command-line interpretor
11569(CLI), which is better suited for development work. To run Weston under
11570the CLI, you need to do the following after your image is built:
11571
115721. Run these commands to export ``XDG_RUNTIME_DIR``:
11573 ::
11574
11575 mkdir -p /tmp/$USER-weston
11576 chmod 0700 /tmp/$USER-weston
11577 export XDG_RUNTIME_DIR=/tmp/$USER-weston
11578
115792. Launch Weston in the shell:
11580 ::
11581
11582 weston