blob: e5028fffdc825aeeec00e3ef256e1583211d5b02 [file] [log] [blame]
Andrew Geissler517393d2023-01-13 08:55:19 -06001.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Working with Packages
4*********************
5
6This section describes a few tasks that involve packages:
7
8- :ref:`dev-manual/packages:excluding packages from an image`
9
10- :ref:`dev-manual/packages:incrementing a package version`
11
12- :ref:`dev-manual/packages:handling optional module packaging`
13
14- :ref:`dev-manual/packages:using runtime package management`
15
16- :ref:`dev-manual/packages:generating and using signed packages`
17
18- :ref:`Setting up and running package test
19 (ptest) <dev-manual/packages:testing packages with ptest>`
20
21- :ref:`dev-manual/packages:creating node package manager (npm) packages`
22
23- :ref:`dev-manual/packages:adding custom metadata to packages`
24
25Excluding Packages from an Image
26================================
27
28You might find it necessary to prevent specific packages from being
29installed into an image. If so, you can use several variables to direct
30the build system to essentially ignore installing recommended packages
31or to not install a package at all.
32
33The following list introduces variables you can use to prevent packages
34from being installed into your image. Each of these variables only works
35with IPK and RPM package types, not for Debian packages.
36Also, you can use these variables from your ``local.conf`` file
37or attach them to a specific image recipe by using a recipe name
38override. For more detail on the variables, see the descriptions in the
39Yocto Project Reference Manual's glossary chapter.
40
41- :term:`BAD_RECOMMENDATIONS`:
42 Use this variable to specify "recommended-only" packages that you do
43 not want installed.
44
45- :term:`NO_RECOMMENDATIONS`:
46 Use this variable to prevent all "recommended-only" packages from
47 being installed.
48
49- :term:`PACKAGE_EXCLUDE`:
50 Use this variable to prevent specific packages from being installed
51 regardless of whether they are "recommended-only" or not. You need to
52 realize that the build process could fail with an error when you
53 prevent the installation of a package whose presence is required by
54 an installed package.
55
56Incrementing a Package Version
57==============================
58
59This section provides some background on how binary package versioning
60is accomplished and presents some of the services, variables, and
61terminology involved.
62
63In order to understand binary package versioning, you need to consider
64the following:
65
66- Binary Package: The binary package that is eventually built and
67 installed into an image.
68
69- Binary Package Version: The binary package version is composed of two
70 components --- a version and a revision.
71
72 .. note::
73
74 Technically, a third component, the "epoch" (i.e. :term:`PE`) is involved
75 but this discussion for the most part ignores :term:`PE`.
76
77 The version and revision are taken from the
78 :term:`PV` and
79 :term:`PR` variables, respectively.
80
81- :term:`PV`: The recipe version. :term:`PV` represents the version of the
82 software being packaged. Do not confuse :term:`PV` with the binary
83 package version.
84
85- :term:`PR`: The recipe revision.
86
87- :term:`SRCPV`: The OpenEmbedded
88 build system uses this string to help define the value of :term:`PV` when
89 the source code revision needs to be included in it.
90
91- :yocto_wiki:`PR Service </PR_Service>`: A
92 network-based service that helps automate keeping package feeds
93 compatible with existing package manager applications such as RPM,
94 APT, and OPKG.
95
96Whenever the binary package content changes, the binary package version
97must change. Changing the binary package version is accomplished by
98changing or "bumping" the :term:`PR` and/or :term:`PV` values. Increasing these
99values occurs one of two ways:
100
101- Automatically using a Package Revision Service (PR Service).
102
103- Manually incrementing the :term:`PR` and/or :term:`PV` variables.
104
105Given a primary challenge of any build system and its users is how to
106maintain a package feed that is compatible with existing package manager
107applications such as RPM, APT, and OPKG, using an automated system is
108much preferred over a manual system. In either system, the main
109requirement is that binary package version numbering increases in a
110linear fashion and that there is a number of version components that
111support that linear progression. For information on how to ensure
112package revisioning remains linear, see the
113":ref:`dev-manual/packages:automatically incrementing a package version number`"
114section.
115
116The following three sections provide related information on the PR
117Service, the manual method for "bumping" :term:`PR` and/or :term:`PV`, and on
118how to ensure binary package revisioning remains linear.
119
120Working With a PR Service
121-------------------------
122
123As mentioned, attempting to maintain revision numbers in the
124:term:`Metadata` is error prone, inaccurate,
125and causes problems for people submitting recipes. Conversely, the PR
126Service automatically generates increasing numbers, particularly the
127revision field, which removes the human element.
128
129.. note::
130
131 For additional information on using a PR Service, you can see the
132 :yocto_wiki:`PR Service </PR_Service>` wiki page.
133
134The Yocto Project uses variables in order of decreasing priority to
135facilitate revision numbering (i.e.
136:term:`PE`,
137:term:`PV`, and
138:term:`PR` for epoch, version, and
139revision, respectively). The values are highly dependent on the policies
140and procedures of a given distribution and package feed.
141
142Because the OpenEmbedded build system uses
143":ref:`signatures <overview-manual/concepts:checksums (signatures)>`", which are
144unique to a given build, the build system knows when to rebuild
145packages. All the inputs into a given task are represented by a
146signature, which can trigger a rebuild when different. Thus, the build
147system itself does not rely on the :term:`PR`, :term:`PV`, and :term:`PE` numbers to
148trigger a rebuild. The signatures, however, can be used to generate
149these values.
150
151The PR Service works with both ``OEBasic`` and ``OEBasicHash``
152generators. The value of :term:`PR` bumps when the checksum changes and the
153different generator mechanisms change signatures under different
154circumstances.
155
156As implemented, the build system includes values from the PR Service
157into the :term:`PR` field as an addition using the form "``.x``" so ``r0``
158becomes ``r0.1``, ``r0.2`` and so forth. This scheme allows existing
159:term:`PR` values to be used for whatever reasons, which include manual
160:term:`PR` bumps, should it be necessary.
161
162By default, the PR Service is not enabled or running. Thus, the packages
163generated are just "self consistent". The build system adds and removes
164packages and there are no guarantees about upgrade paths but images will
165be consistent and correct with the latest changes.
166
167The simplest form for a PR Service is for a single host development system
168that builds the package feed (building system). For this scenario, you can
169enable a local PR Service by setting :term:`PRSERV_HOST` in your
170``local.conf`` file in the :term:`Build Directory`::
171
172 PRSERV_HOST = "localhost:0"
173
174Once the service is started, packages will automatically
175get increasing :term:`PR` values and BitBake takes care of starting and
176stopping the server.
177
178If you have a more complex setup where multiple host development systems
179work against a common, shared package feed, you have a single PR Service
180running and it is connected to each building system. For this scenario,
181you need to start the PR Service using the ``bitbake-prserv`` command::
182
183 bitbake-prserv --host ip --port port --start
184
185In addition to
186hand-starting the service, you need to update the ``local.conf`` file of
187each building system as described earlier so each system points to the
188server and port.
189
190It is also recommended you use build history, which adds some sanity
191checks to binary package versions, in conjunction with the server that
192is running the PR Service. To enable build history, add the following to
193each building system's ``local.conf`` file::
194
195 # It is recommended to activate "buildhistory" for testing the PR service
196 INHERIT += "buildhistory"
197 BUILDHISTORY_COMMIT = "1"
198
199For information on build
200history, see the
201":ref:`dev-manual/build-quality:maintaining build output quality`" section.
202
203.. note::
204
205 The OpenEmbedded build system does not maintain :term:`PR` information as
206 part of the shared state (sstate) packages. If you maintain an sstate
207 feed, it's expected that either all your building systems that
Patrick Williams39653562024-03-01 08:54:02 -0600208 contribute to the sstate feed use a shared PR service, or you do not
Patrick Williamsb58112e2024-03-07 11:16:36 -0600209 run a PR service on any of your building systems.
Patrick Williams39653562024-03-01 08:54:02 -0600210
211 That's because if you had multiple machines sharing a PR service but
212 not their sstate feed, you could end up with "diverging" hashes for
213 the same output artefacts. When presented to the share PR service,
214 each would be considered as new and would increase the revision
215 number, causing many unnecessary package upgrades.
Andrew Geissler517393d2023-01-13 08:55:19 -0600216
217 For more information on shared state, see the
218 ":ref:`overview-manual/concepts:shared state cache`"
219 section in the Yocto Project Overview and Concepts Manual.
220
221Manually Bumping PR
222-------------------
223
224The alternative to setting up a PR Service is to manually "bump" the
225:term:`PR` variable.
226
227If a committed change results in changing the package output, then the
228value of the :term:`PR` variable needs to be increased (or "bumped") as part of
229that commit. For new recipes you should add the :term:`PR` variable and set
230its initial value equal to "r0", which is the default. Even though the
231default value is "r0", the practice of adding it to a new recipe makes
232it harder to forget to bump the variable when you make changes to the
233recipe in future.
234
Andrew Geissler517393d2023-01-13 08:55:19 -0600235Usually, version increases occur only to binary packages. However, if
236for some reason :term:`PV` changes but does not increase, you can increase
237the :term:`PE` variable (Package Epoch). The :term:`PE` variable defaults to
238"0".
239
240Binary package version numbering strives to follow the `Debian Version
241Field Policy
242Guidelines <https://www.debian.org/doc/debian-policy/ch-controlfields.html>`__.
243These guidelines define how versions are compared and what "increasing"
244a version means.
245
246Automatically Incrementing a Package Version Number
247---------------------------------------------------
248
249When fetching a repository, BitBake uses the
250:term:`SRCREV` variable to determine
251the specific source code revision from which to build. You set the
252:term:`SRCREV` variable to
253:term:`AUTOREV` to cause the
254OpenEmbedded build system to automatically use the latest revision of
255the software::
256
257 SRCREV = "${AUTOREV}"
258
259Furthermore, you need to reference :term:`SRCPV` in :term:`PV` in order to
260automatically update the version whenever the revision of the source
261code changes. Here is an example::
262
263 PV = "1.0+git${SRCPV}"
264
265The OpenEmbedded build system substitutes :term:`SRCPV` with the following:
266
267.. code-block:: none
268
269 AUTOINC+source_code_revision
270
271The build system replaces the ``AUTOINC``
272with a number. The number used depends on the state of the PR Service:
273
274- If PR Service is enabled, the build system increments the number,
275 which is similar to the behavior of
276 :term:`PR`. This behavior results in
277 linearly increasing package versions, which is desirable. Here is an
278 example:
279
280 .. code-block:: none
281
282 hello-world-git_0.0+git0+b6558dd387-r0.0_armv7a-neon.ipk
283 hello-world-git_0.0+git1+dd2f5c3565-r0.0_armv7a-neon.ipk
284
285- If PR Service is not enabled, the build system replaces the
286 ``AUTOINC`` placeholder with zero (i.e. "0"). This results in
287 changing the package version since the source revision is included.
288 However, package versions are not increased linearly. Here is an
289 example:
290
291 .. code-block:: none
292
293 hello-world-git_0.0+git0+b6558dd387-r0.0_armv7a-neon.ipk
294 hello-world-git_0.0+git0+dd2f5c3565-r0.0_armv7a-neon.ipk
295
296In summary, the OpenEmbedded build system does not track the history of
297binary package versions for this purpose. ``AUTOINC``, in this case, is
298comparable to :term:`PR`. If PR server is not enabled, ``AUTOINC`` in the
299package version is simply replaced by "0". If PR server is enabled, the
300build system keeps track of the package versions and bumps the number
301when the package revision changes.
302
303Handling Optional Module Packaging
304==================================
305
306Many pieces of software split functionality into optional modules (or
307plugins) and the plugins that are built might depend on configuration
308options. To avoid having to duplicate the logic that determines what
309modules are available in your recipe or to avoid having to package each
310module by hand, the OpenEmbedded build system provides functionality to
311handle module packaging dynamically.
312
313To handle optional module packaging, you need to do two things:
314
315- Ensure the module packaging is actually done.
316
317- Ensure that any dependencies on optional modules from other recipes
318 are satisfied by your recipe.
319
320Making Sure the Packaging is Done
321---------------------------------
322
323To ensure the module packaging actually gets done, you use the
324``do_split_packages`` function within the ``populate_packages`` Python
325function in your recipe. The ``do_split_packages`` function searches for
326a pattern of files or directories under a specified path and creates a
327package for each one it finds by appending to the
328:term:`PACKAGES` variable and
329setting the appropriate values for ``FILES:packagename``,
330``RDEPENDS:packagename``, ``DESCRIPTION:packagename``, and so forth.
331Here is an example from the ``lighttpd`` recipe::
332
333 python populate_packages:prepend () {
334 lighttpd_libdir = d.expand('${libdir}')
335 do_split_packages(d, lighttpd_libdir, '^mod_(.*).so$',
336 'lighttpd-module-%s', 'Lighttpd module for %s',
337 extra_depends='')
338 }
339
340The previous example specifies a number of things in the call to
341``do_split_packages``.
342
343- A directory within the files installed by your recipe through
344 :ref:`ref-tasks-install` in which to search.
345
346- A regular expression used to match module files in that directory. In
347 the example, note the parentheses () that mark the part of the
348 expression from which the module name should be derived.
349
350- A pattern to use for the package names.
351
352- A description for each package.
353
354- An empty string for ``extra_depends``, which disables the default
355 dependency on the main ``lighttpd`` package. Thus, if a file in
356 ``${libdir}`` called ``mod_alias.so`` is found, a package called
357 ``lighttpd-module-alias`` is created for it and the
358 :term:`DESCRIPTION` is set to
359 "Lighttpd module for alias".
360
361Often, packaging modules is as simple as the previous example. However,
362there are more advanced options that you can use within
363``do_split_packages`` to modify its behavior. And, if you need to, you
364can add more logic by specifying a hook function that is called for each
365package. It is also perfectly acceptable to call ``do_split_packages``
366multiple times if you have more than one set of modules to package.
367
368For more examples that show how to use ``do_split_packages``, see the
369``connman.inc`` file in the ``meta/recipes-connectivity/connman/``
370directory of the ``poky`` :ref:`source repository <overview-manual/development-environment:yocto project source repositories>`. You can
371also find examples in ``meta/classes-recipe/kernel.bbclass``.
372
Patrick Williams39653562024-03-01 08:54:02 -0600373Here is a reference that shows ``do_split_packages`` mandatory and
Andrew Geissler517393d2023-01-13 08:55:19 -0600374optional arguments::
375
376 Mandatory arguments
377
378 root
379 The path in which to search
380 file_regex
381 Regular expression to match searched files.
382 Use parentheses () to mark the part of this
383 expression that should be used to derive the
384 module name (to be substituted where %s is
385 used in other function arguments as noted below)
386 output_pattern
387 Pattern to use for the package names. Must
388 include %s.
389 description
390 Description to set for each package. Must
391 include %s.
392
393 Optional arguments
394
395 postinst
396 Postinstall script to use for all packages
397 (as a string)
398 recursive
399 True to perform a recursive search --- default
400 False
401 hook
402 A hook function to be called for every match.
403 The function will be called with the following
404 arguments (in the order listed):
405
406 f
407 Full path to the file/directory match
408 pkg
409 The package name
410 file_regex
411 As above
412 output_pattern
413 As above
414 modulename
415 The module name derived using file_regex
416 extra_depends
417 Extra runtime dependencies (RDEPENDS) to be
418 set for all packages. The default value of None
419 causes a dependency on the main package
420 (${PN}) --- if you do not want this, pass empty
421 string '' for this parameter.
422 aux_files_pattern
423 Extra item(s) to be added to FILES for each
424 package. Can be a single string item or a list
425 of strings for multiple items. Must include %s.
426 postrm
427 postrm script to use for all packages (as a
428 string)
429 allow_dirs
430 True to allow directories to be matched -
431 default False
432 prepend
433 If True, prepend created packages to PACKAGES
434 instead of the default False which appends them
435 match_path
436 match file_regex on the whole relative path to
437 the root rather than just the filename
438 aux_files_pattern_verbatim
439 Extra item(s) to be added to FILES for each
440 package, using the actual derived module name
441 rather than converting it to something legal
442 for a package name. Can be a single string item
443 or a list of strings for multiple items. Must
444 include %s.
445 allow_links
446 True to allow symlinks to be matched --- default
447 False
448 summary
449 Summary to set for each package. Must include %s;
450 defaults to description if not set.
451
452
453
454Satisfying Dependencies
455-----------------------
456
457The second part for handling optional module packaging is to ensure that
458any dependencies on optional modules from other recipes are satisfied by
459your recipe. You can be sure these dependencies are satisfied by using
460the :term:`PACKAGES_DYNAMIC`
461variable. Here is an example that continues with the ``lighttpd`` recipe
462shown earlier::
463
464 PACKAGES_DYNAMIC = "lighttpd-module-.*"
465
466The name
467specified in the regular expression can of course be anything. In this
468example, it is ``lighttpd-module-`` and is specified as the prefix to
469ensure that any :term:`RDEPENDS` and
470:term:`RRECOMMENDS` on a package
471name starting with the prefix are satisfied during build time. If you
472are using ``do_split_packages`` as described in the previous section,
473the value you put in :term:`PACKAGES_DYNAMIC` should correspond to the name
474pattern specified in the call to ``do_split_packages``.
475
476Using Runtime Package Management
477================================
478
479During a build, BitBake always transforms a recipe into one or more
480packages. For example, BitBake takes the ``bash`` recipe and produces a
481number of packages (e.g. ``bash``, ``bash-bashbug``,
482``bash-completion``, ``bash-completion-dbg``, ``bash-completion-dev``,
483``bash-completion-extra``, ``bash-dbg``, and so forth). Not all
484generated packages are included in an image.
485
486In several situations, you might need to update, add, remove, or query
487the packages on a target device at runtime (i.e. without having to
488generate a new image). Examples of such situations include:
489
490- You want to provide in-the-field updates to deployed devices (e.g.
491 security updates).
492
493- You want to have a fast turn-around development cycle for one or more
494 applications that run on your device.
495
496- You want to temporarily install the "debug" packages of various
497 applications on your device so that debugging can be greatly improved
498 by allowing access to symbols and source debugging.
499
500- You want to deploy a more minimal package selection of your device
501 but allow in-the-field updates to add a larger selection for
502 customization.
503
504In all these situations, you have something similar to a more
505traditional Linux distribution in that in-field devices are able to
506receive pre-compiled packages from a server for installation or update.
507Being able to install these packages on a running, in-field device is
508what is termed "runtime package management".
509
510In order to use runtime package management, you need a host or server
511machine that serves up the pre-compiled packages plus the required
512metadata. You also need package manipulation tools on the target. The
513build machine is a likely candidate to act as the server. However, that
514machine does not necessarily have to be the package server. The build
515machine could push its artifacts to another machine that acts as the
516server (e.g. Internet-facing). In fact, doing so is advantageous for a
517production environment as getting the packages away from the development
518system's :term:`Build Directory` prevents accidental overwrites.
519
520A simple build that targets just one device produces more than one
521package database. In other words, the packages produced by a build are
522separated out into a couple of different package groupings based on
523criteria such as the target's CPU architecture, the target board, or the
524C library used on the target. For example, a build targeting the
525``qemux86`` device produces the following three package databases:
526``noarch``, ``i586``, and ``qemux86``. If you wanted your ``qemux86``
527device to be aware of all the packages that were available to it, you
528would need to point it to each of these databases individually. In a
529similar way, a traditional Linux distribution usually is configured to
530be aware of a number of software repositories from which it retrieves
531packages.
532
533Using runtime package management is completely optional and not required
534for a successful build or deployment in any way. But if you want to make
535use of runtime package management, you need to do a couple things above
536and beyond the basics. The remainder of this section describes what you
537need to do.
538
539Build Considerations
540--------------------
541
542This section describes build considerations of which you need to be
543aware in order to provide support for runtime package management.
544
545When BitBake generates packages, it needs to know what format or formats
546to use. In your configuration, you use the
547:term:`PACKAGE_CLASSES`
548variable to specify the format:
549
550#. Open the ``local.conf`` file inside your :term:`Build Directory` (e.g.
551 ``poky/build/conf/local.conf``).
552
553#. Select the desired package format as follows::
554
555 PACKAGE_CLASSES ?= "package_packageformat"
556
557 where packageformat can be "ipk", "rpm",
558 "deb", or "tar" which are the supported package formats.
559
560 .. note::
561
562 Because the Yocto Project supports four different package formats,
563 you can set the variable with more than one argument. However, the
564 OpenEmbedded build system only uses the first argument when
565 creating an image or Software Development Kit (SDK).
566
567If you would like your image to start off with a basic package database
568containing the packages in your current build as well as to have the
569relevant tools available on the target for runtime package management,
570you can include "package-management" in the
571:term:`IMAGE_FEATURES`
572variable. Including "package-management" in this configuration variable
573ensures that when the image is assembled for your target, the image
574includes the currently-known package databases as well as the
575target-specific tools required for runtime package management to be
576performed on the target. However, this is not strictly necessary. You
577could start your image off without any databases but only include the
578required on-target package tool(s). As an example, you could include
579"opkg" in your
580:term:`IMAGE_INSTALL` variable
581if you are using the IPK package format. You can then initialize your
582target's package database(s) later once your image is up and running.
583
584Whenever you perform any sort of build step that can potentially
585generate a package or modify existing package, it is always a good idea
586to re-generate the package index after the build by using the following
587command::
588
589 $ bitbake package-index
590
591It might be tempting to build the
592package and the package index at the same time with a command such as
593the following::
594
595 $ bitbake some-package package-index
596
597Do not do this as
598BitBake does not schedule the package index for after the completion of
599the package you are building. Consequently, you cannot be sure of the
600package index including information for the package you just built.
601Thus, be sure to run the package update step separately after building
602any packages.
603
604You can use the
605:term:`PACKAGE_FEED_ARCHS`,
606:term:`PACKAGE_FEED_BASE_PATHS`,
607and
608:term:`PACKAGE_FEED_URIS`
609variables to pre-configure target images to use a package feed. If you
610do not define these variables, then manual steps as described in the
611subsequent sections are necessary to configure the target. You should
612set these variables before building the image in order to produce a
613correctly configured image.
614
Patrick Williams39653562024-03-01 08:54:02 -0600615.. note::
616
617 Your image will need enough free storage space to run package upgrades,
618 especially if many of them need to be downloaded at the same time.
619 You should make sure images are created with enough free space
620 by setting the :term:`IMAGE_ROOTFS_EXTRA_SPACE` variable.
621
Andrew Geissler517393d2023-01-13 08:55:19 -0600622When your build is complete, your packages reside in the
623``${TMPDIR}/deploy/packageformat`` directory. For example, if
624``${``\ :term:`TMPDIR`\ ``}`` is
625``tmp`` and your selected package type is RPM, then your RPM packages
626are available in ``tmp/deploy/rpm``.
627
628Host or Server Machine Setup
629----------------------------
630
631Although other protocols are possible, a server using HTTP typically
632serves packages. If you want to use HTTP, then set up and configure a
633web server such as Apache 2, lighttpd, or Python web server on the
634machine serving the packages.
635
636To keep things simple, this section describes how to set up a
637Python web server to share package feeds from the developer's
638machine. Although this server might not be the best for a production
639environment, the setup is simple and straight forward. Should you want
640to use a different server more suited for production (e.g. Apache 2,
641Lighttpd, or Nginx), take the appropriate steps to do so.
642
643From within the :term:`Build Directory` where you have built an image based on
644your packaging choice (i.e. the :term:`PACKAGE_CLASSES` setting), simply start
645the server. The following example assumes a :term:`Build Directory` of ``poky/build``
646and a :term:`PACKAGE_CLASSES` setting of ":ref:`ref-classes-package_rpm`"::
647
648 $ cd poky/build/tmp/deploy/rpm
649 $ python3 -m http.server
650
651Target Setup
652------------
653
654Setting up the target differs depending on the package management
655system. This section provides information for RPM, IPK, and DEB.
656
657Using RPM
658~~~~~~~~~
659
660The :wikipedia:`Dandified Packaging <DNF_(software)>` (DNF) performs
661runtime package management of RPM packages. In order to use DNF for
662runtime package management, you must perform an initial setup on the
663target machine for cases where the ``PACKAGE_FEED_*`` variables were not
664set as part of the image that is running on the target. This means if
665you built your image and did not use these variables as part of the
666build and your image is now running on the target, you need to perform
667the steps in this section if you want to use runtime package management.
668
669.. note::
670
671 For information on the ``PACKAGE_FEED_*`` variables, see
672 :term:`PACKAGE_FEED_ARCHS`, :term:`PACKAGE_FEED_BASE_PATHS`, and
673 :term:`PACKAGE_FEED_URIS` in the Yocto Project Reference Manual variables
674 glossary.
675
676On the target, you must inform DNF that package databases are available.
677You do this by creating a file named
678``/etc/yum.repos.d/oe-packages.repo`` and defining the ``oe-packages``.
679
680As an example, assume the target is able to use the following package
681databases: ``all``, ``i586``, and ``qemux86`` from a server named
682``my.server``. The specifics for setting up the web server are up to
683you. The critical requirement is that the URIs in the target repository
684configuration point to the correct remote location for the feeds.
685
686.. note::
687
688 For development purposes, you can point the web server to the build
689 system's ``deploy`` directory. However, for production use, it is better to
690 copy the package directories to a location outside of the build area and use
691 that location. Doing so avoids situations where the build system
692 overwrites or changes the ``deploy`` directory.
693
694When telling DNF where to look for the package databases, you must
695declare individual locations per architecture or a single location used
696for all architectures. You cannot do both:
697
698- *Create an Explicit List of Architectures:* Define individual base
699 URLs to identify where each package database is located:
700
701 .. code-block:: none
702
703 [oe-packages]
704 baseurl=http://my.server/rpm/i586 http://my.server/rpm/qemux86 http://my.server/rpm/all
705
706 This example
707 informs DNF about individual package databases for all three
708 architectures.
709
710- *Create a Single (Full) Package Index:* Define a single base URL that
711 identifies where a full package database is located::
712
713 [oe-packages]
714 baseurl=http://my.server/rpm
715
716 This example informs DNF about a single
717 package database that contains all the package index information for
718 all supported architectures.
719
720Once you have informed DNF where to find the package databases, you need
721to fetch them:
722
723.. code-block:: none
724
725 # dnf makecache
726
727DNF is now able to find, install, and
728upgrade packages from the specified repository or repositories.
729
730.. note::
731
732 See the `DNF documentation <https://dnf.readthedocs.io/en/latest/>`__ for
733 additional information.
734
735Using IPK
736~~~~~~~~~
737
738The ``opkg`` application performs runtime package management of IPK
739packages. You must perform an initial setup for ``opkg`` on the target
740machine if the
741:term:`PACKAGE_FEED_ARCHS`,
742:term:`PACKAGE_FEED_BASE_PATHS`,
743and
744:term:`PACKAGE_FEED_URIS`
745variables have not been set or the target image was built before the
746variables were set.
747
748The ``opkg`` application uses configuration files to find available
749package databases. Thus, you need to create a configuration file inside
750the ``/etc/opkg/`` directory, which informs ``opkg`` of any repository
751you want to use.
752
753As an example, suppose you are serving packages from a ``ipk/``
754directory containing the ``i586``, ``all``, and ``qemux86`` databases
755through an HTTP server named ``my.server``. On the target, create a
756configuration file (e.g. ``my_repo.conf``) inside the ``/etc/opkg/``
757directory containing the following:
758
759.. code-block:: none
760
761 src/gz all http://my.server/ipk/all
762 src/gz i586 http://my.server/ipk/i586
763 src/gz qemux86 http://my.server/ipk/qemux86
764
765Next, instruct ``opkg`` to fetch the
766repository information:
767
768.. code-block:: none
769
770 # opkg update
771
772The ``opkg`` application is now able to find, install, and upgrade packages
773from the specified repository.
774
775Using DEB
776~~~~~~~~~
777
778The ``apt`` application performs runtime package management of DEB
779packages. This application uses a source list file to find available
780package databases. You must perform an initial setup for ``apt`` on the
781target machine if the
782:term:`PACKAGE_FEED_ARCHS`,
783:term:`PACKAGE_FEED_BASE_PATHS`,
784and
785:term:`PACKAGE_FEED_URIS`
786variables have not been set or the target image was built before the
787variables were set.
788
789To inform ``apt`` of the repository you want to use, you might create a
790list file (e.g. ``my_repo.list``) inside the
791``/etc/apt/sources.list.d/`` directory. As an example, suppose you are
792serving packages from a ``deb/`` directory containing the ``i586``,
793``all``, and ``qemux86`` databases through an HTTP server named
794``my.server``. The list file should contain:
795
796.. code-block:: none
797
798 deb http://my.server/deb/all ./
799 deb http://my.server/deb/i586 ./
800 deb http://my.server/deb/qemux86 ./
801
802Next, instruct the ``apt`` application
803to fetch the repository information:
804
805.. code-block:: none
806
807 $ sudo apt update
808
809After this step,
810``apt`` is able to find, install, and upgrade packages from the
811specified repository.
812
813Generating and Using Signed Packages
814====================================
815
816In order to add security to RPM packages used during a build, you can
817take steps to securely sign them. Once a signature is verified, the
818OpenEmbedded build system can use the package in the build. If security
819fails for a signed package, the build system stops the build.
820
821This section describes how to sign RPM packages during a build and how
822to use signed package feeds (repositories) when doing a build.
823
824Signing RPM Packages
825--------------------
826
827To enable signing RPM packages, you must set up the following
828configurations in either your ``local.config`` or ``distro.config``
829file::
830
831 # Inherit sign_rpm.bbclass to enable signing functionality
832 INHERIT += " sign_rpm"
833 # Define the GPG key that will be used for signing.
834 RPM_GPG_NAME = "key_name"
835 # Provide passphrase for the key
836 RPM_GPG_PASSPHRASE = "passphrase"
837
838.. note::
839
840 Be sure to supply appropriate values for both `key_name` and
841 `passphrase`.
842
843Aside from the ``RPM_GPG_NAME`` and ``RPM_GPG_PASSPHRASE`` variables in
844the previous example, two optional variables related to signing are available:
845
846- *GPG_BIN:* Specifies a ``gpg`` binary/wrapper that is executed
847 when the package is signed.
848
849- *GPG_PATH:* Specifies the ``gpg`` home directory used when the
850 package is signed.
851
852Processing Package Feeds
853------------------------
854
855In addition to being able to sign RPM packages, you can also enable
856signed package feeds for IPK and RPM packages.
857
858The steps you need to take to enable signed package feed use are similar
859to the steps used to sign RPM packages. You must define the following in
860your ``local.config`` or ``distro.config`` file::
861
862 INHERIT += "sign_package_feed"
863 PACKAGE_FEED_GPG_NAME = "key_name"
864 PACKAGE_FEED_GPG_PASSPHRASE_FILE = "path_to_file_containing_passphrase"
865
866For signed package feeds, the passphrase must be specified in a separate file,
867which is pointed to by the ``PACKAGE_FEED_GPG_PASSPHRASE_FILE``
868variable. Regarding security, keeping a plain text passphrase out of the
869configuration is more secure.
870
871Aside from the ``PACKAGE_FEED_GPG_NAME`` and
872``PACKAGE_FEED_GPG_PASSPHRASE_FILE`` variables, three optional variables
873related to signed package feeds are available:
874
875- *GPG_BIN* Specifies a ``gpg`` binary/wrapper that is executed
876 when the package is signed.
877
878- *GPG_PATH:* Specifies the ``gpg`` home directory used when the
879 package is signed.
880
881- *PACKAGE_FEED_GPG_SIGNATURE_TYPE:* Specifies the type of ``gpg``
882 signature. This variable applies only to RPM and IPK package feeds.
883 Allowable values for the ``PACKAGE_FEED_GPG_SIGNATURE_TYPE`` are
884 "ASC", which is the default and specifies ascii armored, and "BIN",
885 which specifies binary.
886
887Testing Packages With ptest
888===========================
889
890A Package Test (ptest) runs tests against packages built by the
891OpenEmbedded build system on the target machine. A ptest contains at
892least two items: the actual test, and a shell script (``run-ptest``)
893that starts the test. The shell script that starts the test must not
894contain the actual test --- the script only starts the test. On the other
895hand, the test can be anything from a simple shell script that runs a
896binary and checks the output to an elaborate system of test binaries and
897data files.
898
899The test generates output in the format used by Automake::
900
901 result: testname
902
903where the result can be ``PASS``, ``FAIL``, or ``SKIP``, and
904the testname can be any identifying string.
905
906For a list of Yocto Project recipes that are already enabled with ptest,
907see the :yocto_wiki:`Ptest </Ptest>` wiki page.
908
909.. note::
910
911 A recipe is "ptest-enabled" if it inherits the :ref:`ref-classes-ptest`
912 class.
913
914Adding ptest to Your Build
915--------------------------
916
917To add package testing to your build, add the :term:`DISTRO_FEATURES` and
918:term:`EXTRA_IMAGE_FEATURES` variables to your ``local.conf`` file, which
919is found in the :term:`Build Directory`::
920
921 DISTRO_FEATURES:append = " ptest"
922 EXTRA_IMAGE_FEATURES += "ptest-pkgs"
923
924Once your build is complete, the ptest files are installed into the
925``/usr/lib/package/ptest`` directory within the image, where ``package``
926is the name of the package.
927
928Running ptest
929-------------
930
931The ``ptest-runner`` package installs a shell script that loops through
932all installed ptest test suites and runs them in sequence. Consequently,
933you might want to add this package to your image.
934
935Getting Your Package Ready
936--------------------------
937
938In order to enable a recipe to run installed ptests on target hardware,
939you need to prepare the recipes that build the packages you want to
940test. Here is what you have to do for each recipe:
941
942- *Be sure the recipe inherits the* :ref:`ref-classes-ptest` *class:*
943 Include the following line in each recipe::
944
945 inherit ptest
946
947- *Create run-ptest:* This script starts your test. Locate the
948 script where you will refer to it using
949 :term:`SRC_URI`. Here is an
950 example that starts a test for ``dbus``::
951
952 #!/bin/sh
953 cd test
954 make -k runtest-TESTS
955
956- *Ensure dependencies are met:* If the test adds build or runtime
957 dependencies that normally do not exist for the package (such as
958 requiring "make" to run the test suite), use the
959 :term:`DEPENDS` and
960 :term:`RDEPENDS` variables in
961 your recipe in order for the package to meet the dependencies. Here
962 is an example where the package has a runtime dependency on "make"::
963
964 RDEPENDS:${PN}-ptest += "make"
965
966- *Add a function to build the test suite:* Not many packages support
967 cross-compilation of their test suites. Consequently, you usually
968 need to add a cross-compilation function to the package.
969
970 Many packages based on Automake compile and run the test suite by
971 using a single command such as ``make check``. However, the host
972 ``make check`` builds and runs on the same computer, while
973 cross-compiling requires that the package is built on the host but
974 executed for the target architecture (though often, as in the case
975 for ptest, the execution occurs on the host). The built version of
976 Automake that ships with the Yocto Project includes a patch that
977 separates building and execution. Consequently, packages that use the
978 unaltered, patched version of ``make check`` automatically
979 cross-compiles.
980
981 Regardless, you still must add a ``do_compile_ptest`` function to
982 build the test suite. Add a function similar to the following to your
983 recipe::
984
985 do_compile_ptest() {
986 oe_runmake buildtest-TESTS
987 }
988
989- *Ensure special configurations are set:* If the package requires
990 special configurations prior to compiling the test code, you must
991 insert a ``do_configure_ptest`` function into the recipe.
992
993- *Install the test suite:* The :ref:`ref-classes-ptest` class
994 automatically copies the file ``run-ptest`` to the target and then runs make
995 ``install-ptest`` to run the tests. If this is not enough, you need
996 to create a ``do_install_ptest`` function and make sure it gets
997 called after the "make install-ptest" completes.
998
999Creating Node Package Manager (NPM) Packages
1000============================================
1001
Andrew Geisslerfc113ea2023-03-31 09:59:46 -05001002:wikipedia:`NPM <Npm_(software)>` is a package manager for the JavaScript
1003programming language. The Yocto Project supports the NPM
1004:ref:`fetcher <bitbake-user-manual/bitbake-user-manual-fetching:fetchers>`.
1005You can use this fetcher in combination with
1006:doc:`devtool </ref-manual/devtool-reference>` to create recipes that produce
1007NPM packages.
Andrew Geissler517393d2023-01-13 08:55:19 -06001008
1009There are two workflows that allow you to create NPM packages using
1010``devtool``: the NPM registry modules method and the NPM project code
1011method.
1012
1013.. note::
1014
1015 While it is possible to create NPM recipes manually, using
1016 ``devtool`` is far simpler.
1017
1018Additionally, some requirements and caveats exist.
1019
1020Requirements and Caveats
1021------------------------
1022
1023You need to be aware of the following before using ``devtool`` to create
1024NPM packages:
1025
1026- Of the two methods that you can use ``devtool`` to create NPM
1027 packages, the registry approach is slightly simpler. However, you
1028 might consider the project approach because you do not have to
1029 publish your module in the `NPM registry <https://docs.npmjs.com/misc/registry>`__,
1030 which is NPM's public registry.
1031
1032- Be familiar with
1033 :doc:`devtool </ref-manual/devtool-reference>`.
1034
1035- The NPM host tools need the native ``nodejs-npm`` package, which is
1036 part of the OpenEmbedded environment. You need to get the package by
1037 cloning the :oe_git:`meta-openembedded </meta-openembedded>`
1038 repository. Be sure to add the path to your local copy
1039 to your ``bblayers.conf`` file.
1040
1041- ``devtool`` cannot detect native libraries in module dependencies.
1042 Consequently, you must manually add packages to your recipe.
1043
1044- While deploying NPM packages, ``devtool`` cannot determine which
1045 dependent packages are missing on the target (e.g. the node runtime
1046 ``nodejs``). Consequently, you need to find out what files are
1047 missing and be sure they are on the target.
1048
1049- Although you might not need NPM to run your node package, it is
1050 useful to have NPM on your target. The NPM package name is
1051 ``nodejs-npm``.
1052
1053Using the Registry Modules Method
1054---------------------------------
1055
1056This section presents an example that uses the ``cute-files`` module,
1057which is a file browser web application.
1058
1059.. note::
1060
1061 You must know the ``cute-files`` module version.
1062
1063The first thing you need to do is use ``devtool`` and the NPM fetcher to
1064create the recipe::
1065
1066 $ devtool add "npm://registry.npmjs.org;package=cute-files;version=1.0.2"
1067
1068The
1069``devtool add`` command runs ``recipetool create`` and uses the same
1070fetch URI to download each dependency and capture license details where
1071possible. The result is a generated recipe.
1072
1073After running for quite a long time, in particular building the
1074``nodejs-native`` package, the command should end as follows::
1075
1076 INFO: Recipe /home/.../build/workspace/recipes/cute-files/cute-files_1.0.2.bb has been automatically created; further editing may be required to make it fully functional
1077
1078The recipe file is fairly simple and contains every license that
1079``recipetool`` finds and includes the licenses in the recipe's
1080:term:`LIC_FILES_CHKSUM`
1081variables. You need to examine the variables and look for those with
1082"unknown" in the :term:`LICENSE`
1083field. You need to track down the license information for "unknown"
1084modules and manually add the information to the recipe.
1085
1086``recipetool`` creates a "shrinkwrap" file for your recipe. Shrinkwrap
1087files capture the version of all dependent modules. Many packages do not
1088provide shrinkwrap files but ``recipetool`` will create a shrinkwrap file as it
1089runs.
1090
1091.. note::
1092
1093 A package is created for each sub-module. This policy is the only
1094 practical way to have the licenses for all of the dependencies
1095 represented in the license manifest of the image.
1096
1097The ``devtool edit-recipe`` command lets you take a look at the recipe::
1098
1099 $ devtool edit-recipe cute-files
1100 # Recipe created by recipetool
1101 # This is the basis of a recipe and may need further editing in order to be fully functional.
1102 # (Feel free to remove these comments when editing.)
1103
1104 SUMMARY = "Turn any folder on your computer into a cute file browser, available on the local network."
1105 # WARNING: the following LICENSE and LIC_FILES_CHKSUM values are best guesses - it is
1106 # your responsibility to verify that the values are complete and correct.
1107 #
1108 # NOTE: multiple licenses have been detected; they have been separated with &
1109 # in the LICENSE value for now since it is a reasonable assumption that all
1110 # of the licenses apply. If instead there is a choice between the multiple
1111 # licenses then you should change the value to separate the licenses with |
1112 # instead of &. If there is any doubt, check the accompanying documentation
1113 # to determine which situation is applicable.
1114
1115 SUMMARY = "Turn any folder on your computer into a cute file browser, available on the local network."
1116 LICENSE = "BSD-3-Clause & ISC & MIT"
1117 LIC_FILES_CHKSUM = "file://LICENSE;md5=71d98c0a1db42956787b1909c74a86ca \
1118 file://node_modules/accepts/LICENSE;md5=bf1f9ad1e2e1d507aef4883fff7103de \
1119 file://node_modules/array-flatten/LICENSE;md5=44088ba57cb871a58add36ce51b8de08 \
1120 ...
1121 file://node_modules/cookie-signature/Readme.md;md5=57ae8b42de3dd0c1f22d5f4cf191e15a"
1122
1123 SRC_URI = " \
1124 npm://registry.npmjs.org/;package=cute-files;version=${PV} \
1125 npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \
1126 "
1127
1128 S = "${WORKDIR}/npm"
1129
1130 inherit npm
1131
1132 LICENSE:${PN} = "MIT"
1133 LICENSE:${PN}-accepts = "MIT"
1134 LICENSE:${PN}-array-flatten = "MIT"
1135 ...
1136 LICENSE:${PN}-vary = "MIT"
1137
Patrick Williams39653562024-03-01 08:54:02 -06001138Three key points in the previous example are:
Andrew Geissler517393d2023-01-13 08:55:19 -06001139
1140- :term:`SRC_URI` uses the NPM
1141 scheme so that the NPM fetcher is used.
1142
1143- ``recipetool`` collects all the license information. If a
1144 sub-module's license is unavailable, the sub-module's name appears in
1145 the comments.
1146
1147- The ``inherit npm`` statement causes the :ref:`ref-classes-npm` class to
1148 package up all the modules.
1149
1150You can run the following command to build the ``cute-files`` package::
1151
1152 $ devtool build cute-files
1153
1154Remember that ``nodejs`` must be installed on
1155the target before your package.
1156
1157Assuming 192.168.7.2 for the target's IP address, use the following
1158command to deploy your package::
1159
1160 $ devtool deploy-target -s cute-files root@192.168.7.2
1161
1162Once the package is installed on the target, you can
1163test the application to show the contents of any directory::
1164
1165 $ cd /usr/lib/node_modules/cute-files
1166 $ cute-files
1167
1168On a browser,
1169go to ``http://192.168.7.2:3000`` and you see the following:
1170
1171.. image:: figures/cute-files-npm-example.png
1172 :width: 100%
1173
1174You can find the recipe in ``workspace/recipes/cute-files``. You can use
1175the recipe in any layer you choose.
1176
1177Using the NPM Projects Code Method
1178----------------------------------
1179
1180Although it is useful to package modules already in the NPM registry,
1181adding ``node.js`` projects under development is a more common developer
1182use case.
1183
1184This section covers the NPM projects code method, which is very similar
1185to the "registry" approach described in the previous section. In the NPM
1186projects method, you provide ``devtool`` with an URL that points to the
1187source files.
1188
1189Replicating the same example, (i.e. ``cute-files``) use the following
1190command::
1191
1192 $ devtool add https://github.com/martinaglv/cute-files.git
1193
1194The recipe this command generates is very similar to the recipe created in
1195the previous section. However, the :term:`SRC_URI` looks like the following::
1196
1197 SRC_URI = " \
1198 git://github.com/martinaglv/cute-files.git;protocol=https;branch=master \
1199 npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \
1200 "
1201
1202In this example,
1203the main module is taken from the Git repository and dependencies are
1204taken from the NPM registry. Other than those differences, the recipe is
1205basically the same between the two methods. You can build and deploy the
1206package exactly as described in the previous section that uses the
1207registry modules method.
1208
1209Adding custom metadata to packages
1210==================================
1211
1212The variable
1213:term:`PACKAGE_ADD_METADATA`
1214can be used to add additional metadata to packages. This is reflected in
1215the package control/spec file. To take the ipk format for example, the
1216CONTROL file stored inside would contain the additional metadata as
1217additional lines.
1218
1219The variable can be used in multiple ways, including using suffixes to
1220set it for a specific package type and/or package. Note that the order
1221of precedence is the same as this list:
1222
1223- ``PACKAGE_ADD_METADATA_<PKGTYPE>:<PN>``
1224
1225- ``PACKAGE_ADD_METADATA_<PKGTYPE>``
1226
1227- ``PACKAGE_ADD_METADATA:<PN>``
1228
1229- :term:`PACKAGE_ADD_METADATA`
1230
1231`<PKGTYPE>` is a parameter and expected to be a distinct name of specific
1232package type:
1233
1234- IPK for .ipk packages
1235
1236- DEB for .deb packages
1237
1238- RPM for .rpm packages
1239
1240`<PN>` is a parameter and expected to be a package name.
1241
1242The variable can contain multiple [one-line] metadata fields separated
1243by the literal sequence '\\n'. The separator can be redefined using the
1244variable flag ``separator``.
1245
1246Here is an example that adds two custom fields for ipk
1247packages::
1248
1249 PACKAGE_ADD_METADATA_IPK = "Vendor: CustomIpk\nGroup:Applications/Spreadsheets"
1250