| <!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" |
| "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd" |
| [<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] > |
| <!--SPDX-License-Identifier: CC-BY-2.0-UK--> |
| |
| <appendix id='kernel-dev-concepts-appx'> |
| <title>Advanced Kernel Concepts</title> |
| |
| <section id='kernel-big-picture'> |
| <title>Yocto Project Kernel Development and Maintenance</title> |
| |
| <para> |
| Kernels available through the Yocto Project (Yocto Linux kernels), |
| like other kernels, are based off the Linux kernel releases from |
| <ulink url='http://www.kernel.org'></ulink>. |
| At the beginning of a major Linux kernel development cycle, the |
| Yocto Project team chooses a Linux kernel based on factors such as |
| release timing, the anticipated release timing of final upstream |
| <filename>kernel.org</filename> versions, and Yocto Project |
| feature requirements. |
| Typically, the Linux kernel chosen is in the final stages of |
| development by the Linux community. |
| In other words, the Linux kernel is in the release candidate |
| or "rc" phase and has yet to reach final release. |
| But, by being in the final stages of external development, the |
| team knows that the <filename>kernel.org</filename> final release |
| will clearly be within the early stages of the Yocto Project |
| development window. |
| </para> |
| |
| <para> |
| This balance allows the Yocto Project team to deliver the most |
| up-to-date Yocto Linux kernel possible, while still ensuring that |
| the team has a stable official release for the baseline Linux |
| kernel version. |
| </para> |
| |
| <para> |
| As implied earlier, the ultimate source for Yocto Linux kernels |
| are released kernels from <filename>kernel.org</filename>. |
| In addition to a foundational kernel from |
| <filename>kernel.org</filename>, the available Yocto Linux kernels |
| contain a mix of important new mainline developments, non-mainline |
| developments (when no alternative exists), Board Support Package |
| (BSP) developments, and custom features. |
| These additions result in a commercially released Yocto |
| Project Linux kernel that caters to specific embedded designer |
| needs for targeted hardware. |
| </para> |
| |
| <para> |
| You can find a web interface to the Yocto Linux kernels in the |
| <ulink url='&YOCTO_DOCS_OM_URL;#source-repositories'>Source Repositories</ulink> |
| at |
| <ulink url='&YOCTO_GIT_URL;'></ulink>. |
| If you look at the interface, you will see to the left a |
| grouping of Git repositories titled "Yocto Linux Kernel". |
| Within this group, you will find several Linux Yocto kernels |
| developed and included with Yocto Project releases: |
| <itemizedlist> |
| <listitem><para> |
| <emphasis><filename>linux-yocto-4.1</filename>:</emphasis> |
| The stable Yocto Project kernel to use with the Yocto |
| Project Release 2.0. |
| This kernel is based on the Linux 4.1 released kernel. |
| </para></listitem> |
| <listitem><para> |
| <emphasis><filename>linux-yocto-4.4</filename>:</emphasis> |
| The stable Yocto Project kernel to use with the Yocto |
| Project Release 2.1. |
| This kernel is based on the Linux 4.4 released kernel. |
| </para></listitem> |
| <listitem><para> |
| <emphasis><filename>linux-yocto-4.6</filename>:</emphasis> |
| A temporary kernel that is not tied to any Yocto Project |
| release. |
| </para></listitem> |
| <listitem><para> |
| <emphasis><filename>linux-yocto-4.8</filename>:</emphasis> |
| The stable yocto Project kernel to use with the Yocto |
| Project Release 2.2. |
| </para></listitem> |
| <listitem><para> |
| <emphasis><filename>linux-yocto-4.9</filename>:</emphasis> |
| The stable Yocto Project kernel to use with the Yocto |
| Project Release 2.3. |
| This kernel is based on the Linux 4.9 released kernel. |
| </para></listitem> |
| <listitem><para> |
| <emphasis><filename>linux-yocto-4.10</filename>:</emphasis> |
| The default stable Yocto Project kernel to use with the |
| Yocto Project Release 2.3. |
| This kernel is based on the Linux 4.10 released kernel. |
| </para></listitem> |
| <listitem><para> |
| <emphasis><filename>linux-yocto-4.12</filename>:</emphasis> |
| The default stable Yocto Project kernel to use with the |
| Yocto Project Release 2.4. |
| This kernel is based on the Linux 4.12 released kernel. |
| </para></listitem> |
| <listitem><para> |
| <emphasis><filename>yocto-kernel-cache</filename>:</emphasis> |
| The <filename>linux-yocto-cache</filename> contains |
| patches and configurations for the linux-yocto kernel |
| tree. |
| This repository is useful when working on the linux-yocto |
| kernel. |
| For more information on this "Advanced Kernel Metadata", |
| see the |
| "<link linkend='kernel-dev-advanced'>Working With Advanced Metadata (<filename>yocto-kernel-cache</filename>)</link>" |
| Chapter. |
| </para></listitem> |
| <listitem><para> |
| <emphasis><filename>linux-yocto-dev</filename>:</emphasis> |
| A development kernel based on the latest upstream release |
| candidate available. |
| </para></listitem> |
| </itemizedlist> |
| <note><title>Notes</title> |
| Long Term Support Initiative (LTSI) for Yocto Linux |
| kernels is as follows: |
| <itemizedlist> |
| <listitem><para> |
| For Yocto Project releases 1.7, 1.8, and 2.0, |
| the LTSI kernel is |
| <filename>linux-yocto-3.14</filename>. |
| </para></listitem> |
| <listitem><para> |
| For Yocto Project releases 2.1, 2.2, and 2.3, |
| the LTSI kernel is <filename>linux-yocto-4.1</filename>. |
| </para></listitem> |
| <listitem><para> |
| For Yocto Project release 2.4, the LTSI kernel is |
| <filename>linux-yocto-4.9</filename> |
| </para></listitem> |
| <listitem><para> |
| <filename>linux-yocto-4.4</filename> is an LTS |
| kernel. |
| </para></listitem> |
| </itemizedlist> |
| </note> |
| </para> |
| |
| <para> |
| Once a Yocto Linux kernel is officially released, the Yocto |
| Project team goes into their next development cycle, or upward |
| revision (uprev) cycle, while still continuing maintenance on the |
| released kernel. |
| It is important to note that the most sustainable and stable way |
| to include feature development upstream is through a kernel uprev |
| process. |
| Back-porting hundreds of individual fixes and minor features from |
| various kernel versions is not sustainable and can easily |
| compromise quality. |
| </para> |
| |
| <para> |
| During the uprev cycle, the Yocto Project team uses an ongoing |
| analysis of Linux kernel development, BSP support, and release |
| timing to select the best possible <filename>kernel.org</filename> |
| Linux kernel version on which to base subsequent Yocto Linux |
| kernel development. |
| The team continually monitors Linux community kernel development |
| to look for significant features of interest. |
| The team does consider back-porting large features if they have a |
| significant advantage. |
| User or community demand can also trigger a back-port or creation |
| of new functionality in the Yocto Project baseline kernel during |
| the uprev cycle. |
| </para> |
| |
| <para> |
| Generally speaking, every new Linux kernel both adds features and |
| introduces new bugs. |
| These consequences are the basic properties of upstream |
| Linux kernel development and are managed by the Yocto Project |
| team's Yocto Linux kernel development strategy. |
| It is the Yocto Project team's policy to not back-port minor |
| features to the released Yocto Linux kernel. |
| They only consider back-porting significant technological |
| jumps ‐ and, that is done after a complete gap analysis. |
| The reason for this policy is that back-porting any small to |
| medium sized change from an evolving Linux kernel can easily |
| create mismatches, incompatibilities and very subtle errors. |
| </para> |
| |
| <para> |
| The policies described in this section result in both a stable |
| and a cutting edge Yocto Linux kernel that mixes forward ports of |
| existing Linux kernel features and significant and critical new |
| functionality. |
| Forward porting Linux kernel functionality into the Yocto Linux |
| kernels available through the Yocto Project can be thought of as |
| a "micro uprev." |
| The many "micro uprevs" produce a Yocto Linux kernel version with |
| a mix of important new mainline, non-mainline, BSP developments |
| and feature integrations. |
| This Yocto Linux kernel gives insight into new features and |
| allows focused amounts of testing to be done on the kernel, |
| which prevents surprises when selecting the next major uprev. |
| The quality of these cutting edge Yocto Linux kernels is evolving |
| and the kernels are used in leading edge feature and BSP |
| development. |
| </para> |
| </section> |
| |
| <section id='yocto-linux-kernel-architecture-and-branching-strategies'> |
| <title>Yocto Linux Kernel Architecture and Branching Strategies</title> |
| |
| <para> |
| As mentioned earlier, a key goal of the Yocto Project is |
| to present the developer with a kernel that has a clear and |
| continuous history that is visible to the user. |
| The architecture and mechanisms, in particular the branching |
| strategies, used achieve that goal in a manner similar to |
| upstream Linux kernel development in |
| <filename>kernel.org</filename>. |
| </para> |
| |
| <para> |
| You can think of a Yocto Linux kernel as consisting of a |
| baseline Linux kernel with added features logically structured |
| on top of the baseline. |
| The features are tagged and organized by way of a branching |
| strategy implemented by the Yocto Project team using the |
| Source Code Manager (SCM) Git. |
| <note><title>Notes</title> |
| <itemizedlist> |
| <listitem><para> |
| Git is the obvious SCM for meeting the Yocto Linux |
| kernel organizational and structural goals described |
| in this section. |
| Not only is Git the SCM for Linux kernel development in |
| <filename>kernel.org</filename> but, Git continues to |
| grow in popularity and supports many different work |
| flows, front-ends and management techniques. |
| </para></listitem> |
| <listitem><para> |
| You can find documentation on Git at |
| <ulink url='http://git-scm.com/documentation'></ulink>. |
| You can also get an introduction to Git as it |
| applies to the Yocto Project in the |
| "<ulink url='&YOCTO_DOCS_OM_URL;#git'>Git</ulink>" |
| section in the Yocto Project Overview and Concepts |
| Manual. |
| The latter reference provides an overview of |
| Git and presents a minimal set of Git commands |
| that allows you to be functional using Git. |
| You can use as much, or as little, of what Git |
| has to offer to accomplish what you need for your |
| project. |
| You do not have to be a "Git Expert" in order to |
| use it with the Yocto Project. |
| </para></listitem> |
| </itemizedlist> |
| </note> |
| </para> |
| |
| <para> |
| Using Git's tagging and branching features, the Yocto Project |
| team creates kernel branches at points where functionality is |
| no longer shared and thus, needs to be isolated. |
| For example, board-specific incompatibilities would require |
| different functionality and would require a branch to |
| separate the features. |
| Likewise, for specific kernel features, the same branching |
| strategy is used. |
| </para> |
| |
| <para> |
| This "tree-like" architecture results in a structure that has |
| features organized to be specific for particular functionality, |
| single kernel types, or a subset of kernel types. |
| Thus, the user has the ability to see the added features and the |
| commits that make up those features. |
| In addition to being able to see added features, the user |
| can also view the history of what made up the baseline |
| Linux kernel. |
| </para> |
| |
| <para> |
| Another consequence of this strategy results in not having to |
| store the same feature twice internally in the tree. |
| Rather, the kernel team stores the unique differences required |
| to apply the feature onto the kernel type in question. |
| <note> |
| The Yocto Project team strives to place features in the tree |
| such that features can be shared by all boards and kernel |
| types where possible. |
| However, during development cycles or when large features |
| are merged, the team cannot always follow this practice. |
| In those cases, the team uses isolated branches to merge |
| features. |
| </note> |
| </para> |
| |
| <para> |
| BSP-specific code additions are handled in a similar manner to |
| kernel-specific additions. |
| Some BSPs only make sense given certain kernel types. |
| So, for these types, the team creates branches off the end |
| of that kernel type for all of the BSPs that are supported on |
| that kernel type. |
| From the perspective of the tools that create the BSP branch, |
| the BSP is really no different than a feature. |
| Consequently, the same branching strategy applies to BSPs as |
| it does to kernel features. |
| So again, rather than store the BSP twice, the team only |
| stores the unique differences for the BSP across the supported |
| multiple kernels. |
| </para> |
| |
| <para> |
| While this strategy can result in a tree with a significant number |
| of branches, it is important to realize that from the developer's |
| point of view, there is a linear path that travels from the |
| baseline <filename>kernel.org</filename>, through a select |
| group of features and ends with their BSP-specific commits. |
| In other words, the divisions of the kernel are transparent and |
| are not relevant to the developer on a day-to-day basis. |
| From the developer's perspective, this path is the "master" branch |
| in Git terms. |
| The developer does not need to be aware of the existence of any |
| other branches at all. |
| Of course, value exists in the having these branches in the tree, |
| should a person decide to explore them. |
| For example, a comparison between two BSPs at either the commit |
| level or at the line-by-line code <filename>diff</filename> level |
| is now a trivial operation. |
| </para> |
| |
| <para> |
| The following illustration shows the conceptual Yocto |
| Linux kernel. |
| <imagedata fileref="figures/kernel-architecture-overview.png" width="6in" depth="7in" align="center" scale="100" /> |
| </para> |
| |
| <para> |
| In the illustration, the "Kernel.org Branch Point" marks the |
| specific spot (or Linux kernel release) from which the |
| Yocto Linux kernel is created. |
| From this point forward in the tree, features and differences |
| are organized and tagged. |
| </para> |
| |
| <para> |
| The "Yocto Project Baseline Kernel" contains functionality that |
| is common to every kernel type and BSP that is organized |
| further along in the tree. |
| Placing these common features in the tree this way means |
| features do not have to be duplicated along individual |
| branches of the tree structure. |
| </para> |
| |
| <para> |
| From the "Yocto Project Baseline Kernel", branch points represent |
| specific functionality for individual Board Support Packages |
| (BSPs) as well as real-time kernels. |
| The illustration represents this through three BSP-specific |
| branches and a real-time kernel branch. |
| Each branch represents some unique functionality for the BSP |
| or for a real-time Yocto Linux kernel. |
| </para> |
| |
| <para> |
| In this example structure, the "Real-time (rt) Kernel" branch has |
| common features for all real-time Yocto Linux kernels and |
| contains more branches for individual BSP-specific real-time |
| kernels. |
| The illustration shows three branches as an example. |
| Each branch points the way to specific, unique features for a |
| respective real-time kernel as they apply to a given BSP. |
| </para> |
| |
| <para> |
| The resulting tree structure presents a clear path of markers |
| (or branches) to the developer that, for all practical |
| purposes, is the Yocto Linux kernel needed for any given set of |
| requirements. |
| <note> |
| Keep in mind the figure does not take into account all the |
| supported Yocto Linux kernels, but rather shows a single |
| generic kernel just for conceptual purposes. |
| Also keep in mind that this structure represents the Yocto |
| Project |
| <ulink url='&YOCTO_DOCS_OM_URL;#source-repositories'>Source Repositories</ulink> |
| that are either pulled from during the build or established |
| on the host development system prior to the build by either |
| cloning a particular kernel's Git repository or by |
| downloading and unpacking a tarball. |
| </note> |
| </para> |
| |
| <para> |
| Working with the kernel as a structured tree follows recognized |
| community best practices. |
| In particular, the kernel as shipped with the product, should be |
| considered an "upstream source" and viewed as a series of |
| historical and documented modifications (commits). |
| These modifications represent the development and stabilization |
| done by the Yocto Project kernel development team. |
| </para> |
| |
| <para> |
| Because commits only change at significant release points in the |
| product life cycle, developers can work on a branch created |
| from the last relevant commit in the shipped Yocto Project Linux |
| kernel. |
| As mentioned previously, the structure is transparent to the |
| developer because the kernel tree is left in this state after |
| cloning and building the kernel. |
| </para> |
| </section> |
| |
| <section id='kernel-build-file-hierarchy'> |
| <title>Kernel Build File Hierarchy</title> |
| |
| <para> |
| Upstream storage of all the available kernel source code is |
| one thing, while representing and using the code on your host |
| development system is another. |
| Conceptually, you can think of the kernel source repositories |
| as all the source files necessary for all the supported |
| Yocto Linux kernels. |
| As a developer, you are just interested in the source files |
| for the kernel on which you are working. |
| And, furthermore, you need them available on your host system. |
| </para> |
| |
| <para> |
| Kernel source code is available on your host system several |
| different ways: |
| <itemizedlist> |
| <listitem><para> |
| <emphasis>Files Accessed While using <filename>devtool</filename>:</emphasis> |
| <filename>devtool</filename>, which is available with the |
| Yocto Project, is the preferred method by which to |
| modify the kernel. |
| See the |
| "<link linkend='kernel-modification-workflow'>Kernel Modification Workflow</link>" |
| section. |
| </para></listitem> |
| <listitem><para> |
| <emphasis>Cloned Repository:</emphasis> |
| If you are working in the kernel all the time, you probably |
| would want to set up your own local Git repository of the |
| Yocto Linux kernel tree. |
| For information on how to clone a Yocto Linux kernel |
| Git repository, see the |
| "<link linkend='preparing-the-build-host-to-work-on-the-kernel'>Preparing the Build Host to Work on the Kernel</link>" |
| section. |
| </para></listitem> |
| <listitem><para> |
| <emphasis>Temporary Source Files from a Build:</emphasis> |
| If you just need to make some patches to the kernel using |
| a traditional BitBake workflow (i.e. not using the |
| <filename>devtool</filename>), you can access temporary |
| kernel source files that were extracted and used during |
| a kernel build. |
| </para></listitem> |
| </itemizedlist> |
| </para> |
| |
| <para> |
| The temporary kernel source files resulting from a build using |
| BitBake have a particular hierarchy. |
| When you build the kernel on your development system, all files |
| needed for the build are taken from the source repositories |
| pointed to by the |
| <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink> |
| variable and gathered in a temporary work area where they are |
| subsequently used to create the unique kernel. |
| Thus, in a sense, the process constructs a local source tree |
| specific to your kernel from which to generate the new kernel |
| image. |
| </para> |
| |
| <para> |
| The following figure shows the temporary file structure |
| created on your host system when you build the kernel using |
| Bitbake. |
| This |
| <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink> |
| contains all the source files used during the build. |
| <imagedata fileref="figures/kernel-overview-2-generic.png" |
| width="6in" depth="5in" align="center" scale="100" /> |
| </para> |
| |
| <para> |
| Again, for additional information on the Yocto Project kernel's |
| architecture and its branching strategy, see the |
| "<link linkend='yocto-linux-kernel-architecture-and-branching-strategies'>Yocto Linux Kernel Architecture and Branching Strategies</link>" |
| section. |
| You can also reference the |
| "<link linkend='using-devtool-to-patch-the-kernel'>Using <filename>devtool</filename> to Patch the Kernel</link>" |
| and |
| "<link linkend='using-traditional-kernel-development-to-patch-the-kernel'>Using Traditional Kernel Development to Patch the Kernel</link>" |
| sections for detailed example that modifies the kernel. |
| </para> |
| </section> |
| |
| <section id='determining-hardware-and-non-hardware-features-for-the-kernel-configuration-audit-phase'> |
| <title>Determining Hardware and Non-Hardware Features for the Kernel Configuration Audit Phase</title> |
| |
| <para> |
| This section describes part of the kernel configuration audit |
| phase that most developers can ignore. |
| For general information on kernel configuration including |
| <filename>menuconfig</filename>, <filename>defconfig</filename> |
| files, and configuration fragments, see the |
| "<link linkend='configuring-the-kernel'>Configuring the Kernel</link>" |
| section. |
| </para> |
| |
| <para> |
| During this part of the audit phase, the contents of the final |
| <filename>.config</filename> file are compared against the |
| fragments specified by the system. |
| These fragments can be system fragments, distro fragments, |
| or user-specified configuration elements. |
| Regardless of their origin, the OpenEmbedded build system |
| warns the user if a specific option is not included in the |
| final kernel configuration. |
| </para> |
| |
| <para> |
| By default, in order to not overwhelm the user with |
| configuration warnings, the system only reports missing |
| "hardware" options as they could result in a boot |
| failure or indicate that important hardware is not available. |
| </para> |
| |
| <para> |
| To determine whether or not a given option is "hardware" or |
| "non-hardware", the kernel Metadata in |
| <filename>yocto-kernel-cache</filename> contains files that |
| classify individual or groups of options as either hardware |
| or non-hardware. |
| To better show this, consider a situation where the |
| <filename>yocto-kernel-cache</filename> contains the following |
| files: |
| <literallayout class='monospaced'> |
| yocto-kernel-cache/features/drm-psb/hardware.cfg |
| yocto-kernel-cache/features/kgdb/hardware.cfg |
| yocto-kernel-cache/ktypes/base/hardware.cfg |
| yocto-kernel-cache/bsp/mti-malta32/hardware.cfg |
| yocto-kernel-cache/bsp/qemu-ppc32/hardware.cfg |
| yocto-kernel-cache/bsp/qemuarma9/hardware.cfg |
| yocto-kernel-cache/bsp/mti-malta64/hardware.cfg |
| yocto-kernel-cache/bsp/arm-versatile-926ejs/hardware.cfg |
| yocto-kernel-cache/bsp/common-pc/hardware.cfg |
| yocto-kernel-cache/bsp/common-pc-64/hardware.cfg |
| yocto-kernel-cache/features/rfkill/non-hardware.cfg |
| yocto-kernel-cache/ktypes/base/non-hardware.cfg |
| yocto-kernel-cache/features/aufs/non-hardware.kcf |
| yocto-kernel-cache/features/ocf/non-hardware.kcf |
| yocto-kernel-cache/ktypes/base/non-hardware.kcf |
| yocto-kernel-cache/ktypes/base/hardware.kcf |
| yocto-kernel-cache/bsp/qemu-ppc32/hardware.kcf |
| </literallayout> |
| The following list provides explanations for the various |
| files: |
| <itemizedlist> |
| <listitem><para> |
| <filename>hardware.kcf</filename>: |
| Specifies a list of kernel Kconfig files that contain |
| hardware options only. |
| </para></listitem> |
| <listitem><para> |
| <filename>non-hardware.kcf</filename>: |
| Specifies a list of kernel Kconfig files that contain |
| non-hardware options only. |
| </para></listitem> |
| <listitem><para> |
| <filename>hardware.cfg</filename>: |
| Specifies a list of kernel <filename>CONFIG_</filename> |
| options that are hardware, regardless of whether or not |
| they are within a Kconfig file specified by a hardware |
| or non-hardware Kconfig file (i.e. |
| <filename>hardware.kcf</filename> or |
| <filename>non-hardware.kcf</filename>). |
| </para></listitem> |
| <listitem><para> |
| <filename>non-hardware.cfg</filename>: |
| Specifies a list of kernel <filename>CONFIG_</filename> |
| options that are not hardware, regardless of whether or |
| not they are within a Kconfig file specified by a |
| hardware or non-hardware Kconfig file (i.e. |
| <filename>hardware.kcf</filename> or |
| <filename>non-hardware.kcf</filename>). |
| </para></listitem> |
| </itemizedlist> |
| Here is a specific example using the |
| <filename>kernel-cache/bsp/mti-malta32/hardware.cfg</filename>: |
| <literallayout class='monospaced'> |
| CONFIG_SERIAL_8250 |
| CONFIG_SERIAL_8250_CONSOLE |
| CONFIG_SERIAL_8250_NR_UARTS |
| CONFIG_SERIAL_8250_PCI |
| CONFIG_SERIAL_CORE |
| CONFIG_SERIAL_CORE_CONSOLE |
| CONFIG_VGA_ARB |
| </literallayout> |
| The kernel configuration audit automatically detects these |
| files (hence the names must be exactly the ones discussed here), |
| and uses them as inputs when generating warnings about the |
| final <filename>.config</filename> file. |
| </para> |
| |
| <para> |
| A user-specified kernel Metadata repository, or recipe space |
| feature, can use these same files to classify options that are |
| found within its <filename>.cfg</filename> files as hardware |
| or non-hardware, to prevent the OpenEmbedded build system from |
| producing an error or warning when an option is not in the |
| final <filename>.config</filename> file. |
| </para> |
| </section> |
| </appendix> |
| <!-- |
| vim: expandtab tw=80 ts=4 |
| --> |