blob: 6a2fe4bb0ba00d1d8b2e56d27f6683ffbbe86bfb [file] [log] [blame]
Andrew Geisslerf0343792020-11-18 10:42:21 -06001.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002
3************************
4Advanced Kernel Concepts
5************************
6
Andrew Geisslerc9f78652020-09-18 14:11:35 -05007Yocto Project Kernel Development and Maintenance
8================================================
9
10Kernels available through the Yocto Project (Yocto Linux kernels), like
11other kernels, are based off the Linux kernel releases from
Andrew Geissler4c19ea12020-10-27 13:52:24 -050012https://www.kernel.org. At the beginning of a major Linux kernel
Andrew Geisslerc9f78652020-09-18 14:11:35 -050013development cycle, the Yocto Project team chooses a Linux kernel based
14on factors such as release timing, the anticipated release timing of
15final upstream ``kernel.org`` versions, and Yocto Project feature
16requirements. Typically, the Linux kernel chosen is in the final stages
17of development by the Linux community. In other words, the Linux kernel
18is in the release candidate or "rc" phase and has yet to reach final
19release. But, by being in the final stages of external development, the
20team knows that the ``kernel.org`` final release will clearly be within
21the early stages of the Yocto Project development window.
22
23This balance allows the Yocto Project team to deliver the most
24up-to-date Yocto Linux kernel possible, while still ensuring that the
25team has a stable official release for the baseline Linux kernel
26version.
27
28As implied earlier, the ultimate source for Yocto Linux kernels are
29released kernels from ``kernel.org``. In addition to a foundational
30kernel from ``kernel.org``, the available Yocto Linux kernels contain a
31mix of important new mainline developments, non-mainline developments
32(when no alternative exists), Board Support Package (BSP) developments,
33and custom features. These additions result in a commercially released
34Yocto Project Linux kernel that caters to specific embedded designer
35needs for targeted hardware.
36
37You can find a web interface to the Yocto Linux kernels in the
Andrew Geissler09209ee2020-12-13 08:44:15 -060038:ref:`overview-manual/development-environment:yocto project source repositories`
Andrew Geisslerc9f78652020-09-18 14:11:35 -050039at :yocto_git:`/`. If you look at the interface, you will see to
40the left a grouping of Git repositories titled "Yocto Linux Kernel".
41Within this group, you will find several Linux Yocto kernels developed
42and included with Yocto Project releases:
43
44- *linux-yocto-4.1:* The stable Yocto Project kernel to use with
45 the Yocto Project Release 2.0. This kernel is based on the Linux 4.1
46 released kernel.
47
48- *linux-yocto-4.4:* The stable Yocto Project kernel to use with
49 the Yocto Project Release 2.1. This kernel is based on the Linux 4.4
50 released kernel.
51
52- *linux-yocto-4.6:* A temporary kernel that is not tied to any
53 Yocto Project release.
54
55- *linux-yocto-4.8:* The stable yocto Project kernel to use with
56 the Yocto Project Release 2.2.
57
58- *linux-yocto-4.9:* The stable Yocto Project kernel to use with
59 the Yocto Project Release 2.3. This kernel is based on the Linux 4.9
60 released kernel.
61
62- *linux-yocto-4.10:* The default stable Yocto Project kernel to
63 use with the Yocto Project Release 2.3. This kernel is based on the
64 Linux 4.10 released kernel.
65
66- *linux-yocto-4.12:* The default stable Yocto Project kernel to
67 use with the Yocto Project Release 2.4. This kernel is based on the
68 Linux 4.12 released kernel.
69
70- *yocto-kernel-cache:* The ``linux-yocto-cache`` contains patches
71 and configurations for the linux-yocto kernel tree. This repository
72 is useful when working on the linux-yocto kernel. For more
73 information on this "Advanced Kernel Metadata", see the
Andrew Geissler09209ee2020-12-13 08:44:15 -060074 ":doc:`/kernel-dev/advanced`" Chapter.
Andrew Geisslerc9f78652020-09-18 14:11:35 -050075
76- *linux-yocto-dev:* A development kernel based on the latest
77 upstream release candidate available.
78
79.. note::
80
81 Long Term Support Initiative (LTSI) for Yocto Linux kernels is as
82 follows:
83
84 - For Yocto Project releases 1.7, 1.8, and 2.0, the LTSI kernel is
85 ``linux-yocto-3.14``.
86
87 - For Yocto Project releases 2.1, 2.2, and 2.3, the LTSI kernel is
88 ``linux-yocto-4.1``.
89
90 - For Yocto Project release 2.4, the LTSI kernel is
91 ``linux-yocto-4.9``
92
93 - ``linux-yocto-4.4`` is an LTS kernel.
94
95Once a Yocto Linux kernel is officially released, the Yocto Project team
96goes into their next development cycle, or upward revision (uprev)
97cycle, while still continuing maintenance on the released kernel. It is
98important to note that the most sustainable and stable way to include
99feature development upstream is through a kernel uprev process.
100Back-porting hundreds of individual fixes and minor features from
101various kernel versions is not sustainable and can easily compromise
102quality.
103
104During the uprev cycle, the Yocto Project team uses an ongoing analysis
105of Linux kernel development, BSP support, and release timing to select
106the best possible ``kernel.org`` Linux kernel version on which to base
107subsequent Yocto Linux kernel development. The team continually monitors
108Linux community kernel development to look for significant features of
109interest. The team does consider back-porting large features if they
110have a significant advantage. User or community demand can also trigger
111a back-port or creation of new functionality in the Yocto Project
112baseline kernel during the uprev cycle.
113
114Generally speaking, every new Linux kernel both adds features and
115introduces new bugs. These consequences are the basic properties of
116upstream Linux kernel development and are managed by the Yocto Project
117team's Yocto Linux kernel development strategy. It is the Yocto Project
118team's policy to not back-port minor features to the released Yocto
119Linux kernel. They only consider back-porting significant technological
Andrew Geissler615f2f12022-07-15 14:00:58 -0500120jumps --- and, that is done after a complete gap analysis. The reason
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500121for this policy is that back-porting any small to medium sized change
122from an evolving Linux kernel can easily create mismatches,
123incompatibilities and very subtle errors.
124
125The policies described in this section result in both a stable and a
126cutting edge Yocto Linux kernel that mixes forward ports of existing
127Linux kernel features and significant and critical new functionality.
128Forward porting Linux kernel functionality into the Yocto Linux kernels
129available through the Yocto Project can be thought of as a "micro
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500130uprev". The many "micro uprevs" produce a Yocto Linux kernel version
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500131with a mix of important new mainline, non-mainline, BSP developments and
132feature integrations. This Yocto Linux kernel gives insight into new
133features and allows focused amounts of testing to be done on the kernel,
134which prevents surprises when selecting the next major uprev. The
135quality of these cutting edge Yocto Linux kernels is evolving and the
136kernels are used in leading edge feature and BSP development.
137
138Yocto Linux Kernel Architecture and Branching Strategies
139========================================================
140
141As mentioned earlier, a key goal of the Yocto Project is to present the
142developer with a kernel that has a clear and continuous history that is
143visible to the user. The architecture and mechanisms, in particular the
144branching strategies, used achieve that goal in a manner similar to
145upstream Linux kernel development in ``kernel.org``.
146
147You can think of a Yocto Linux kernel as consisting of a baseline Linux
148kernel with added features logically structured on top of the baseline.
149The features are tagged and organized by way of a branching strategy
150implemented by the Yocto Project team using the Source Code Manager
151(SCM) Git.
152
153.. note::
154
155 - Git is the obvious SCM for meeting the Yocto Linux kernel
156 organizational and structural goals described in this section. Not
157 only is Git the SCM for Linux kernel development in ``kernel.org``
158 but, Git continues to grow in popularity and supports many
159 different work flows, front-ends and management techniques.
160
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500161 - You can find documentation on Git at https://git-scm.com/doc. You can
162 also get an introduction to Git as it applies to the Yocto Project in the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600163 ":ref:`overview-manual/development-environment:git`" section in the Yocto Project
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500164 Overview and Concepts Manual. The latter reference provides an
165 overview of Git and presents a minimal set of Git commands that
166 allows you to be functional using Git. You can use as much, or as
167 little, of what Git has to offer to accomplish what you need for
168 your project. You do not have to be a "Git Expert" in order to use
169 it with the Yocto Project.
170
171Using Git's tagging and branching features, the Yocto Project team
172creates kernel branches at points where functionality is no longer
173shared and thus, needs to be isolated. For example, board-specific
174incompatibilities would require different functionality and would
175require a branch to separate the features. Likewise, for specific kernel
176features, the same branching strategy is used.
177
178This "tree-like" architecture results in a structure that has features
179organized to be specific for particular functionality, single kernel
180types, or a subset of kernel types. Thus, the user has the ability to
181see the added features and the commits that make up those features. In
182addition to being able to see added features, the user can also view the
183history of what made up the baseline Linux kernel.
184
185Another consequence of this strategy results in not having to store the
186same feature twice internally in the tree. Rather, the kernel team
187stores the unique differences required to apply the feature onto the
188kernel type in question.
189
190.. note::
191
192 The Yocto Project team strives to place features in the tree such
193 that features can be shared by all boards and kernel types where
194 possible. However, during development cycles or when large features
195 are merged, the team cannot always follow this practice. In those
196 cases, the team uses isolated branches to merge features.
197
198BSP-specific code additions are handled in a similar manner to
199kernel-specific additions. Some BSPs only make sense given certain
200kernel types. So, for these types, the team creates branches off the end
201of that kernel type for all of the BSPs that are supported on that
202kernel type. From the perspective of the tools that create the BSP
203branch, the BSP is really no different than a feature. Consequently, the
204same branching strategy applies to BSPs as it does to kernel features.
205So again, rather than store the BSP twice, the team only stores the
206unique differences for the BSP across the supported multiple kernels.
207
208While this strategy can result in a tree with a significant number of
209branches, it is important to realize that from the developer's point of
210view, there is a linear path that travels from the baseline
211``kernel.org``, through a select group of features and ends with their
212BSP-specific commits. In other words, the divisions of the kernel are
213transparent and are not relevant to the developer on a day-to-day basis.
Andrew Geissler595f6302022-01-24 19:11:47 +0000214From the developer's perspective, this path is the development branch.
215The developer does not need to be aware of the existence of
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700216any other branches at all. Of course, it can make sense to have these
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500217branches in the tree, should a person decide to explore them. For
218example, a comparison between two BSPs at either the commit level or at
219the line-by-line code ``diff`` level is now a trivial operation.
220
221The following illustration shows the conceptual Yocto Linux kernel.
222
223.. image:: figures/kernel-architecture-overview.png
Andrew Geisslerd5838332022-05-27 11:33:10 -0500224 :width: 100%
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500225
226In the illustration, the "Kernel.org Branch Point" marks the specific
227spot (or Linux kernel release) from which the Yocto Linux kernel is
228created. From this point forward in the tree, features and differences
229are organized and tagged.
230
231The "Yocto Project Baseline Kernel" contains functionality that is
232common to every kernel type and BSP that is organized further along in
233the tree. Placing these common features in the tree this way means
234features do not have to be duplicated along individual branches of the
235tree structure.
236
237From the "Yocto Project Baseline Kernel", branch points represent
238specific functionality for individual Board Support Packages (BSPs) as
239well as real-time kernels. The illustration represents this through
240three BSP-specific branches and a real-time kernel branch. Each branch
241represents some unique functionality for the BSP or for a real-time
242Yocto Linux kernel.
243
244In this example structure, the "Real-time (rt) Kernel" branch has common
245features for all real-time Yocto Linux kernels and contains more
246branches for individual BSP-specific real-time kernels. The illustration
247shows three branches as an example. Each branch points the way to
248specific, unique features for a respective real-time kernel as they
249apply to a given BSP.
250
251The resulting tree structure presents a clear path of markers (or
252branches) to the developer that, for all practical purposes, is the
253Yocto Linux kernel needed for any given set of requirements.
254
255.. note::
256
257 Keep in mind the figure does not take into account all the supported
258 Yocto Linux kernels, but rather shows a single generic kernel just
259 for conceptual purposes. Also keep in mind that this structure
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500260 represents the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600261 :ref:`overview-manual/development-environment:yocto project source repositories`
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500262 that are either pulled from during the build or established on the
263 host development system prior to the build by either cloning a
264 particular kernel's Git repository or by downloading and unpacking a
265 tarball.
266
267Working with the kernel as a structured tree follows recognized
268community best practices. In particular, the kernel as shipped with the
269product, should be considered an "upstream source" and viewed as a
270series of historical and documented modifications (commits). These
271modifications represent the development and stabilization done by the
272Yocto Project kernel development team.
273
274Because commits only change at significant release points in the product
275life cycle, developers can work on a branch created from the last
276relevant commit in the shipped Yocto Project Linux kernel. As mentioned
277previously, the structure is transparent to the developer because the
278kernel tree is left in this state after cloning and building the kernel.
279
280Kernel Build File Hierarchy
281===========================
282
283Upstream storage of all the available kernel source code is one thing,
284while representing and using the code on your host development system is
285another. Conceptually, you can think of the kernel source repositories
286as all the source files necessary for all the supported Yocto Linux
287kernels. As a developer, you are just interested in the source files for
288the kernel on which you are working. And, furthermore, you need them
289available on your host system.
290
291Kernel source code is available on your host system several different
292ways:
293
294- *Files Accessed While using devtool:* ``devtool``, which is
295 available with the Yocto Project, is the preferred method by which to
Andrew Geissler09209ee2020-12-13 08:44:15 -0600296 modify the kernel. See the ":ref:`kernel-dev/intro:kernel modification workflow`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500297
298- *Cloned Repository:* If you are working in the kernel all the time,
299 you probably would want to set up your own local Git repository of
300 the Yocto Linux kernel tree. For information on how to clone a Yocto
301 Linux kernel Git repository, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600302 ":ref:`kernel-dev/common:preparing the build host to work on the kernel`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500303 section.
304
305- *Temporary Source Files from a Build:* If you just need to make some
306 patches to the kernel using a traditional BitBake workflow (i.e. not
307 using the ``devtool``), you can access temporary kernel source files
308 that were extracted and used during a kernel build.
309
310The temporary kernel source files resulting from a build using BitBake
311have a particular hierarchy. When you build the kernel on your
312development system, all files needed for the build are taken from the
313source repositories pointed to by the
314:term:`SRC_URI` variable and gathered
315in a temporary work area where they are subsequently used to create the
316unique kernel. Thus, in a sense, the process constructs a local source
317tree specific to your kernel from which to generate the new kernel
318image.
319
320The following figure shows the temporary file structure created on your
Andrew Geisslerd5838332022-05-27 11:33:10 -0500321host system when you build the kernel using BitBake. This
Patrick Williams2390b1b2022-11-03 13:47:49 -0500322:term:`Build Directory` contains all the source files used during the build.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500323
324.. image:: figures/kernel-overview-2-generic.png
325 :align: center
Andrew Geisslerd5838332022-05-27 11:33:10 -0500326 :width: 70%
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500327
328Again, for additional information on the Yocto Project kernel's
329architecture and its branching strategy, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600330":ref:`kernel-dev/concepts-appx:yocto linux kernel architecture and branching strategies`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500331section. You can also reference the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600332":ref:`kernel-dev/common:using \`\`devtool\`\` to patch the kernel`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500333and
Andrew Geissler09209ee2020-12-13 08:44:15 -0600334":ref:`kernel-dev/common:using traditional kernel development to patch the kernel`"
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500335sections for detailed example that modifies the kernel.
336
337Determining Hardware and Non-Hardware Features for the Kernel Configuration Audit Phase
338=======================================================================================
339
340This section describes part of the kernel configuration audit phase that
341most developers can ignore. For general information on kernel
342configuration including ``menuconfig``, ``defconfig`` files, and
343configuration fragments, see the
Andrew Geissler09209ee2020-12-13 08:44:15 -0600344":ref:`kernel-dev/common:configuring the kernel`" section.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500345
346During this part of the audit phase, the contents of the final
347``.config`` file are compared against the fragments specified by the
348system. These fragments can be system fragments, distro fragments, or
349user-specified configuration elements. Regardless of their origin, the
350OpenEmbedded build system warns the user if a specific option is not
351included in the final kernel configuration.
352
353By default, in order to not overwhelm the user with configuration
354warnings, the system only reports missing "hardware" options as they
355could result in a boot failure or indicate that important hardware is
356not available.
357
358To determine whether or not a given option is "hardware" or
359"non-hardware", the kernel Metadata in ``yocto-kernel-cache`` contains
360files that classify individual or groups of options as either hardware
361or non-hardware. To better show this, consider a situation where the
Andrew Geisslerc926e172021-05-07 16:11:35 -0500362``yocto-kernel-cache`` contains the following files::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500363
364 yocto-kernel-cache/features/drm-psb/hardware.cfg
365 yocto-kernel-cache/features/kgdb/hardware.cfg
366 yocto-kernel-cache/ktypes/base/hardware.cfg
367 yocto-kernel-cache/bsp/mti-malta32/hardware.cfg
368 yocto-kernel-cache/bsp/qemu-ppc32/hardware.cfg
369 yocto-kernel-cache/bsp/qemuarma9/hardware.cfg
370 yocto-kernel-cache/bsp/mti-malta64/hardware.cfg
371 yocto-kernel-cache/bsp/arm-versatile-926ejs/hardware.cfg
372 yocto-kernel-cache/bsp/common-pc/hardware.cfg
373 yocto-kernel-cache/bsp/common-pc-64/hardware.cfg
374 yocto-kernel-cache/features/rfkill/non-hardware.cfg
375 yocto-kernel-cache/ktypes/base/non-hardware.cfg
376 yocto-kernel-cache/features/aufs/non-hardware.kcf
377 yocto-kernel-cache/features/ocf/non-hardware.kcf
378 yocto-kernel-cache/ktypes/base/non-hardware.kcf
379 yocto-kernel-cache/ktypes/base/hardware.kcf
380 yocto-kernel-cache/bsp/qemu-ppc32/hardware.kcf
381
William A. Kennington IIIac69b482021-06-02 12:28:27 -0700382Here are explanations for the various files:
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500383
384- ``hardware.kcf``: Specifies a list of kernel Kconfig files that
385 contain hardware options only.
386
387- ``non-hardware.kcf``: Specifies a list of kernel Kconfig files that
388 contain non-hardware options only.
389
390- ``hardware.cfg``: Specifies a list of kernel ``CONFIG_`` options that
391 are hardware, regardless of whether or not they are within a Kconfig
392 file specified by a hardware or non-hardware Kconfig file (i.e.
393 ``hardware.kcf`` or ``non-hardware.kcf``).
394
395- ``non-hardware.cfg``: Specifies a list of kernel ``CONFIG_`` options
396 that are not hardware, regardless of whether or not they are within a
397 Kconfig file specified by a hardware or non-hardware Kconfig file
398 (i.e. ``hardware.kcf`` or ``non-hardware.kcf``).
399
400Here is a specific example using the
Andrew Geisslerc926e172021-05-07 16:11:35 -0500401``kernel-cache/bsp/mti-malta32/hardware.cfg``::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500402
403 CONFIG_SERIAL_8250
404 CONFIG_SERIAL_8250_CONSOLE
405 CONFIG_SERIAL_8250_NR_UARTS
406 CONFIG_SERIAL_8250_PCI
407 CONFIG_SERIAL_CORE
408 CONFIG_SERIAL_CORE_CONSOLE
409 CONFIG_VGA_ARB
410
411The kernel configuration audit automatically detects
412these files (hence the names must be exactly the ones discussed here),
413and uses them as inputs when generating warnings about the final
414``.config`` file.
415
416A user-specified kernel Metadata repository, or recipe space feature,
417can use these same files to classify options that are found within its
418``.cfg`` files as hardware or non-hardware, to prevent the OpenEmbedded
419build system from producing an error or warning when an option is not in
420the final ``.config`` file.