blob: 681faee5229d8962186c607a4377a476721fe96d [file] [log] [blame]
Andrew Geisslerf0343792020-11-18 10:42:21 -06001.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002
3************************
4Advanced Kernel Concepts
5************************
6
7.. _kernel-big-picture:
8
9Yocto Project Kernel Development and Maintenance
10================================================
11
12Kernels available through the Yocto Project (Yocto Linux kernels), like
13other kernels, are based off the Linux kernel releases from
Andrew Geissler4c19ea12020-10-27 13:52:24 -050014https://www.kernel.org. At the beginning of a major Linux kernel
Andrew Geisslerc9f78652020-09-18 14:11:35 -050015development cycle, the Yocto Project team chooses a Linux kernel based
16on factors such as release timing, the anticipated release timing of
17final upstream ``kernel.org`` versions, and Yocto Project feature
18requirements. Typically, the Linux kernel chosen is in the final stages
19of development by the Linux community. In other words, the Linux kernel
20is in the release candidate or "rc" phase and has yet to reach final
21release. But, by being in the final stages of external development, the
22team knows that the ``kernel.org`` final release will clearly be within
23the early stages of the Yocto Project development window.
24
25This balance allows the Yocto Project team to deliver the most
26up-to-date Yocto Linux kernel possible, while still ensuring that the
27team has a stable official release for the baseline Linux kernel
28version.
29
30As implied earlier, the ultimate source for Yocto Linux kernels are
31released kernels from ``kernel.org``. In addition to a foundational
32kernel from ``kernel.org``, the available Yocto Linux kernels contain a
33mix of important new mainline developments, non-mainline developments
34(when no alternative exists), Board Support Package (BSP) developments,
35and custom features. These additions result in a commercially released
36Yocto Project Linux kernel that caters to specific embedded designer
37needs for targeted hardware.
38
39You can find a web interface to the Yocto Linux kernels in the
40:ref:`overview-manual/overview-manual-development-environment:yocto project source repositories`
41at :yocto_git:`/`. If you look at the interface, you will see to
42the left a grouping of Git repositories titled "Yocto Linux Kernel".
43Within this group, you will find several Linux Yocto kernels developed
44and included with Yocto Project releases:
45
46- *linux-yocto-4.1:* The stable Yocto Project kernel to use with
47 the Yocto Project Release 2.0. This kernel is based on the Linux 4.1
48 released kernel.
49
50- *linux-yocto-4.4:* The stable Yocto Project kernel to use with
51 the Yocto Project Release 2.1. This kernel is based on the Linux 4.4
52 released kernel.
53
54- *linux-yocto-4.6:* A temporary kernel that is not tied to any
55 Yocto Project release.
56
57- *linux-yocto-4.8:* The stable yocto Project kernel to use with
58 the Yocto Project Release 2.2.
59
60- *linux-yocto-4.9:* The stable Yocto Project kernel to use with
61 the Yocto Project Release 2.3. This kernel is based on the Linux 4.9
62 released kernel.
63
64- *linux-yocto-4.10:* The default stable Yocto Project kernel to
65 use with the Yocto Project Release 2.3. This kernel is based on the
66 Linux 4.10 released kernel.
67
68- *linux-yocto-4.12:* The default stable Yocto Project kernel to
69 use with the Yocto Project Release 2.4. This kernel is based on the
70 Linux 4.12 released kernel.
71
72- *yocto-kernel-cache:* The ``linux-yocto-cache`` contains patches
73 and configurations for the linux-yocto kernel tree. This repository
74 is useful when working on the linux-yocto kernel. For more
75 information on this "Advanced Kernel Metadata", see the
76 ":doc:`kernel-dev-advanced`" Chapter.
77
78- *linux-yocto-dev:* A development kernel based on the latest
79 upstream release candidate available.
80
81.. note::
82
83 Long Term Support Initiative (LTSI) for Yocto Linux kernels is as
84 follows:
85
86 - For Yocto Project releases 1.7, 1.8, and 2.0, the LTSI kernel is
87 ``linux-yocto-3.14``.
88
89 - For Yocto Project releases 2.1, 2.2, and 2.3, the LTSI kernel is
90 ``linux-yocto-4.1``.
91
92 - For Yocto Project release 2.4, the LTSI kernel is
93 ``linux-yocto-4.9``
94
95 - ``linux-yocto-4.4`` is an LTS kernel.
96
97Once a Yocto Linux kernel is officially released, the Yocto Project team
98goes into their next development cycle, or upward revision (uprev)
99cycle, while still continuing maintenance on the released kernel. It is
100important to note that the most sustainable and stable way to include
101feature development upstream is through a kernel uprev process.
102Back-porting hundreds of individual fixes and minor features from
103various kernel versions is not sustainable and can easily compromise
104quality.
105
106During the uprev cycle, the Yocto Project team uses an ongoing analysis
107of Linux kernel development, BSP support, and release timing to select
108the best possible ``kernel.org`` Linux kernel version on which to base
109subsequent Yocto Linux kernel development. The team continually monitors
110Linux community kernel development to look for significant features of
111interest. The team does consider back-porting large features if they
112have a significant advantage. User or community demand can also trigger
113a back-port or creation of new functionality in the Yocto Project
114baseline kernel during the uprev cycle.
115
116Generally speaking, every new Linux kernel both adds features and
117introduces new bugs. These consequences are the basic properties of
118upstream Linux kernel development and are managed by the Yocto Project
119team's Yocto Linux kernel development strategy. It is the Yocto Project
120team's policy to not back-port minor features to the released Yocto
121Linux kernel. They only consider back-porting significant technological
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500122jumps - and, that is done after a complete gap analysis. The reason
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500123for this policy is that back-porting any small to medium sized change
124from an evolving Linux kernel can easily create mismatches,
125incompatibilities and very subtle errors.
126
127The policies described in this section result in both a stable and a
128cutting edge Yocto Linux kernel that mixes forward ports of existing
129Linux kernel features and significant and critical new functionality.
130Forward porting Linux kernel functionality into the Yocto Linux kernels
131available through the Yocto Project can be thought of as a "micro
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500132uprev". The many "micro uprevs" produce a Yocto Linux kernel version
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500133with a mix of important new mainline, non-mainline, BSP developments and
134feature integrations. This Yocto Linux kernel gives insight into new
135features and allows focused amounts of testing to be done on the kernel,
136which prevents surprises when selecting the next major uprev. The
137quality of these cutting edge Yocto Linux kernels is evolving and the
138kernels are used in leading edge feature and BSP development.
139
140Yocto Linux Kernel Architecture and Branching Strategies
141========================================================
142
143As mentioned earlier, a key goal of the Yocto Project is to present the
144developer with a kernel that has a clear and continuous history that is
145visible to the user. The architecture and mechanisms, in particular the
146branching strategies, used achieve that goal in a manner similar to
147upstream Linux kernel development in ``kernel.org``.
148
149You can think of a Yocto Linux kernel as consisting of a baseline Linux
150kernel with added features logically structured on top of the baseline.
151The features are tagged and organized by way of a branching strategy
152implemented by the Yocto Project team using the Source Code Manager
153(SCM) Git.
154
155.. note::
156
157 - Git is the obvious SCM for meeting the Yocto Linux kernel
158 organizational and structural goals described in this section. Not
159 only is Git the SCM for Linux kernel development in ``kernel.org``
160 but, Git continues to grow in popularity and supports many
161 different work flows, front-ends and management techniques.
162
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500163 - You can find documentation on Git at https://git-scm.com/doc. You can
164 also get an introduction to Git as it applies to the Yocto Project in the
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500165 ":ref:`overview-manual/overview-manual-development-environment:git`" section in the Yocto Project
166 Overview and Concepts Manual. The latter reference provides an
167 overview of Git and presents a minimal set of Git commands that
168 allows you to be functional using Git. You can use as much, or as
169 little, of what Git has to offer to accomplish what you need for
170 your project. You do not have to be a "Git Expert" in order to use
171 it with the Yocto Project.
172
173Using Git's tagging and branching features, the Yocto Project team
174creates kernel branches at points where functionality is no longer
175shared and thus, needs to be isolated. For example, board-specific
176incompatibilities would require different functionality and would
177require a branch to separate the features. Likewise, for specific kernel
178features, the same branching strategy is used.
179
180This "tree-like" architecture results in a structure that has features
181organized to be specific for particular functionality, single kernel
182types, or a subset of kernel types. Thus, the user has the ability to
183see the added features and the commits that make up those features. In
184addition to being able to see added features, the user can also view the
185history of what made up the baseline Linux kernel.
186
187Another consequence of this strategy results in not having to store the
188same feature twice internally in the tree. Rather, the kernel team
189stores the unique differences required to apply the feature onto the
190kernel type in question.
191
192.. note::
193
194 The Yocto Project team strives to place features in the tree such
195 that features can be shared by all boards and kernel types where
196 possible. However, during development cycles or when large features
197 are merged, the team cannot always follow this practice. In those
198 cases, the team uses isolated branches to merge features.
199
200BSP-specific code additions are handled in a similar manner to
201kernel-specific additions. Some BSPs only make sense given certain
202kernel types. So, for these types, the team creates branches off the end
203of that kernel type for all of the BSPs that are supported on that
204kernel type. From the perspective of the tools that create the BSP
205branch, the BSP is really no different than a feature. Consequently, the
206same branching strategy applies to BSPs as it does to kernel features.
207So again, rather than store the BSP twice, the team only stores the
208unique differences for the BSP across the supported multiple kernels.
209
210While this strategy can result in a tree with a significant number of
211branches, it is important to realize that from the developer's point of
212view, there is a linear path that travels from the baseline
213``kernel.org``, through a select group of features and ends with their
214BSP-specific commits. In other words, the divisions of the kernel are
215transparent and are not relevant to the developer on a day-to-day basis.
216From the developer's perspective, this path is the "master" branch in
217Git terms. The developer does not need to be aware of the existence of
218any other branches at all. Of course, value exists in the having these
219branches in the tree, should a person decide to explore them. For
220example, a comparison between two BSPs at either the commit level or at
221the line-by-line code ``diff`` level is now a trivial operation.
222
223The following illustration shows the conceptual Yocto Linux kernel.
224
225.. image:: figures/kernel-architecture-overview.png
226 :align: center
227
228In the illustration, the "Kernel.org Branch Point" marks the specific
229spot (or Linux kernel release) from which the Yocto Linux kernel is
230created. From this point forward in the tree, features and differences
231are organized and tagged.
232
233The "Yocto Project Baseline Kernel" contains functionality that is
234common to every kernel type and BSP that is organized further along in
235the tree. Placing these common features in the tree this way means
236features do not have to be duplicated along individual branches of the
237tree structure.
238
239From the "Yocto Project Baseline Kernel", branch points represent
240specific functionality for individual Board Support Packages (BSPs) as
241well as real-time kernels. The illustration represents this through
242three BSP-specific branches and a real-time kernel branch. Each branch
243represents some unique functionality for the BSP or for a real-time
244Yocto Linux kernel.
245
246In this example structure, the "Real-time (rt) Kernel" branch has common
247features for all real-time Yocto Linux kernels and contains more
248branches for individual BSP-specific real-time kernels. The illustration
249shows three branches as an example. Each branch points the way to
250specific, unique features for a respective real-time kernel as they
251apply to a given BSP.
252
253The resulting tree structure presents a clear path of markers (or
254branches) to the developer that, for all practical purposes, is the
255Yocto Linux kernel needed for any given set of requirements.
256
257.. note::
258
259 Keep in mind the figure does not take into account all the supported
260 Yocto Linux kernels, but rather shows a single generic kernel just
261 for conceptual purposes. Also keep in mind that this structure
Andrew Geissler4c19ea12020-10-27 13:52:24 -0500262 represents the
263 :ref:`overview-manual/overview-manual-development-environment:yocto project source repositories`
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500264 that are either pulled from during the build or established on the
265 host development system prior to the build by either cloning a
266 particular kernel's Git repository or by downloading and unpacking a
267 tarball.
268
269Working with the kernel as a structured tree follows recognized
270community best practices. In particular, the kernel as shipped with the
271product, should be considered an "upstream source" and viewed as a
272series of historical and documented modifications (commits). These
273modifications represent the development and stabilization done by the
274Yocto Project kernel development team.
275
276Because commits only change at significant release points in the product
277life cycle, developers can work on a branch created from the last
278relevant commit in the shipped Yocto Project Linux kernel. As mentioned
279previously, the structure is transparent to the developer because the
280kernel tree is left in this state after cloning and building the kernel.
281
282Kernel Build File Hierarchy
283===========================
284
285Upstream storage of all the available kernel source code is one thing,
286while representing and using the code on your host development system is
287another. Conceptually, you can think of the kernel source repositories
288as all the source files necessary for all the supported Yocto Linux
289kernels. As a developer, you are just interested in the source files for
290the kernel on which you are working. And, furthermore, you need them
291available on your host system.
292
293Kernel source code is available on your host system several different
294ways:
295
296- *Files Accessed While using devtool:* ``devtool``, which is
297 available with the Yocto Project, is the preferred method by which to
298 modify the kernel. See the ":ref:`kernel-dev/kernel-dev-intro:kernel modification workflow`" section.
299
300- *Cloned Repository:* If you are working in the kernel all the time,
301 you probably would want to set up your own local Git repository of
302 the Yocto Linux kernel tree. For information on how to clone a Yocto
303 Linux kernel Git repository, see the
304 ":ref:`kernel-dev/kernel-dev-common:preparing the build host to work on the kernel`"
305 section.
306
307- *Temporary Source Files from a Build:* If you just need to make some
308 patches to the kernel using a traditional BitBake workflow (i.e. not
309 using the ``devtool``), you can access temporary kernel source files
310 that were extracted and used during a kernel build.
311
312The temporary kernel source files resulting from a build using BitBake
313have a particular hierarchy. When you build the kernel on your
314development system, all files needed for the build are taken from the
315source repositories pointed to by the
316:term:`SRC_URI` variable and gathered
317in a temporary work area where they are subsequently used to create the
318unique kernel. Thus, in a sense, the process constructs a local source
319tree specific to your kernel from which to generate the new kernel
320image.
321
322The following figure shows the temporary file structure created on your
323host system when you build the kernel using Bitbake. This
324:term:`Build Directory` contains all the
325source files used during the build.
326
327.. image:: figures/kernel-overview-2-generic.png
328 :align: center
329
330Again, for additional information on the Yocto Project kernel's
331architecture and its branching strategy, see the
332":ref:`kernel-dev/kernel-dev-concepts-appx:yocto linux kernel architecture and branching strategies`"
333section. You can also reference the
334":ref:`kernel-dev/kernel-dev-common:using \`\`devtool\`\` to patch the kernel`"
335and
336":ref:`kernel-dev/kernel-dev-common:using traditional kernel development to patch the kernel`"
337sections for detailed example that modifies the kernel.
338
339Determining Hardware and Non-Hardware Features for the Kernel Configuration Audit Phase
340=======================================================================================
341
342This section describes part of the kernel configuration audit phase that
343most developers can ignore. For general information on kernel
344configuration including ``menuconfig``, ``defconfig`` files, and
345configuration fragments, see the
346":ref:`kernel-dev/kernel-dev-common:configuring the kernel`" section.
347
348During this part of the audit phase, the contents of the final
349``.config`` file are compared against the fragments specified by the
350system. These fragments can be system fragments, distro fragments, or
351user-specified configuration elements. Regardless of their origin, the
352OpenEmbedded build system warns the user if a specific option is not
353included in the final kernel configuration.
354
355By default, in order to not overwhelm the user with configuration
356warnings, the system only reports missing "hardware" options as they
357could result in a boot failure or indicate that important hardware is
358not available.
359
360To determine whether or not a given option is "hardware" or
361"non-hardware", the kernel Metadata in ``yocto-kernel-cache`` contains
362files that classify individual or groups of options as either hardware
363or non-hardware. To better show this, consider a situation where the
364``yocto-kernel-cache`` contains the following files:
365::
366
367 yocto-kernel-cache/features/drm-psb/hardware.cfg
368 yocto-kernel-cache/features/kgdb/hardware.cfg
369 yocto-kernel-cache/ktypes/base/hardware.cfg
370 yocto-kernel-cache/bsp/mti-malta32/hardware.cfg
371 yocto-kernel-cache/bsp/qemu-ppc32/hardware.cfg
372 yocto-kernel-cache/bsp/qemuarma9/hardware.cfg
373 yocto-kernel-cache/bsp/mti-malta64/hardware.cfg
374 yocto-kernel-cache/bsp/arm-versatile-926ejs/hardware.cfg
375 yocto-kernel-cache/bsp/common-pc/hardware.cfg
376 yocto-kernel-cache/bsp/common-pc-64/hardware.cfg
377 yocto-kernel-cache/features/rfkill/non-hardware.cfg
378 yocto-kernel-cache/ktypes/base/non-hardware.cfg
379 yocto-kernel-cache/features/aufs/non-hardware.kcf
380 yocto-kernel-cache/features/ocf/non-hardware.kcf
381 yocto-kernel-cache/ktypes/base/non-hardware.kcf
382 yocto-kernel-cache/ktypes/base/hardware.kcf
383 yocto-kernel-cache/bsp/qemu-ppc32/hardware.kcf
384
385The following list
386provides explanations for the various files:
387
388- ``hardware.kcf``: Specifies a list of kernel Kconfig files that
389 contain hardware options only.
390
391- ``non-hardware.kcf``: Specifies a list of kernel Kconfig files that
392 contain non-hardware options only.
393
394- ``hardware.cfg``: Specifies a list of kernel ``CONFIG_`` options that
395 are hardware, regardless of whether or not they are within a Kconfig
396 file specified by a hardware or non-hardware Kconfig file (i.e.
397 ``hardware.kcf`` or ``non-hardware.kcf``).
398
399- ``non-hardware.cfg``: Specifies a list of kernel ``CONFIG_`` options
400 that are not hardware, regardless of whether or not they are within a
401 Kconfig file specified by a hardware or non-hardware Kconfig file
402 (i.e. ``hardware.kcf`` or ``non-hardware.kcf``).
403
404Here is a specific example using the
405``kernel-cache/bsp/mti-malta32/hardware.cfg``:
406::
407
408 CONFIG_SERIAL_8250
409 CONFIG_SERIAL_8250_CONSOLE
410 CONFIG_SERIAL_8250_NR_UARTS
411 CONFIG_SERIAL_8250_PCI
412 CONFIG_SERIAL_CORE
413 CONFIG_SERIAL_CORE_CONSOLE
414 CONFIG_VGA_ARB
415
416The kernel configuration audit automatically detects
417these files (hence the names must be exactly the ones discussed here),
418and uses them as inputs when generating warnings about the final
419``.config`` file.
420
421A user-specified kernel Metadata repository, or recipe space feature,
422can use these same files to classify options that are found within its
423``.cfg`` files as hardware or non-hardware, to prevent the OpenEmbedded
424build system from producing an error or warning when an option is not in
425the final ``.config`` file.