Patrick Williams | d8c66bc | 2016-06-20 12:57:21 -0500 | [diff] [blame] | 1 | <!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" |
| 2 | "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd" |
| 3 | [<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] > |
| 4 | |
| 5 | <chapter id='sdk-intro'> |
| 6 | <title>Introduction</title> |
| 7 | |
| 8 | <section id='sdk-manual-intro'> |
| 9 | <title>Introduction</title> |
| 10 | |
| 11 | <para> |
| 12 | Welcome to the Yocto Project Software Development Kit (SDK) |
| 13 | Developer's Guide. |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 14 | This manual provides information that explains how to use both the |
| 15 | Yocto Project extensible and standard SDKs to develop |
| 16 | applications and images. |
Patrick Williams | d8c66bc | 2016-06-20 12:57:21 -0500 | [diff] [blame] | 17 | Additionally, the manual also provides information on how to use |
| 18 | the popular <trademark class='trade'>Eclipse</trademark> IDE as part |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 19 | of your application development workflow within the SDK environment. |
| 20 | <note> |
| 21 | Prior to the 2.0 Release of the Yocto Project, application |
| 22 | development was primarily accomplished through the use of the |
| 23 | Application Development Toolkit (ADT) and the availability |
| 24 | of stand-alone cross-development toolchains and other tools. |
| 25 | With the 2.1 Release of the Yocto Project, application development |
| 26 | has transitioned to within a tool-rich extensible SDK and the more |
| 27 | traditional standard SDK. |
| 28 | </note> |
Patrick Williams | d8c66bc | 2016-06-20 12:57:21 -0500 | [diff] [blame] | 29 | </para> |
| 30 | |
| 31 | <para> |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 32 | All SDKs consist of the following: |
| 33 | <itemizedlist> |
| 34 | <listitem><para><emphasis>Cross-Development Toolchain</emphasis>: |
| 35 | This toolchain contains a compiler, debugger, and various |
| 36 | miscellaneous tools. |
| 37 | </para></listitem> |
| 38 | <listitem><para><emphasis>Libraries, Headers, and Symbols</emphasis>: |
| 39 | The libraries, headers, and symbols are specific to the image |
| 40 | (i.e. they match the image). |
| 41 | </para></listitem> |
| 42 | <listitem><para><emphasis>Environment Setup Script</emphasis>: |
| 43 | This <filename>*.sh</filename> file, once run, sets up the |
| 44 | cross-development environment by defining variables and |
| 45 | preparing for SDK use. |
| 46 | </para></listitem> |
| 47 | </itemizedlist> |
Patrick Williams | d8c66bc | 2016-06-20 12:57:21 -0500 | [diff] [blame] | 48 | </para> |
| 49 | |
| 50 | <para> |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 51 | Additionally an extensible SDK has tools that allow you to easily add |
| 52 | new applications and libraries to an image, modify the source of an |
| 53 | existing component, test changes on the target hardware, and easily |
| 54 | integrate an application into the |
Patrick Williams | d8c66bc | 2016-06-20 12:57:21 -0500 | [diff] [blame] | 55 | <ulink url='&YOCTO_DOCS_DEV_URL;#build-system-term'>OpenEmbedded build system</ulink>. |
| 56 | </para> |
| 57 | |
| 58 | <para> |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 59 | You can use an SDK to independently develop and test code |
| 60 | that is destined to run on some target machine. |
Patrick Williams | d8c66bc | 2016-06-20 12:57:21 -0500 | [diff] [blame] | 61 | SDKs are completely self-contained. |
| 62 | The binaries are linked against their own copy of |
| 63 | <filename>libc</filename>, which results in no dependencies |
| 64 | on the target system. |
| 65 | To achieve this, the pointer to the dynamic loader is |
| 66 | configured at install time since that path cannot be dynamically |
| 67 | altered. |
| 68 | This is the reason for a wrapper around the |
| 69 | <filename>populate_sdk</filename> and |
| 70 | <filename>populate_sdk_ext</filename> archives. |
| 71 | </para> |
| 72 | |
| 73 | <para> |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 74 | Another feature for the SDKs is that only one set of cross-compiler |
| 75 | toolchain binaries are produced for any given architecture. |
Patrick Williams | d8c66bc | 2016-06-20 12:57:21 -0500 | [diff] [blame] | 76 | This feature takes advantage of the fact that the target hardware can |
| 77 | be passed to <filename>gcc</filename> as a set of compiler options. |
| 78 | Those options are set up by the environment script and contained in |
| 79 | variables such as |
| 80 | <ulink url='&YOCTO_DOCS_REF_URL;#var-CC'><filename>CC</filename></ulink> |
| 81 | and |
| 82 | <ulink url='&YOCTO_DOCS_REF_URL;#var-LD'><filename>LD</filename></ulink>. |
| 83 | This reduces the space needed for the tools. |
| 84 | Understand, however, that a sysroot is still needed for every target |
| 85 | since those binaries are target-specific. |
| 86 | </para> |
| 87 | |
| 88 | <para> |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 89 | The SDK development environment consists of the following: |
Patrick Williams | d8c66bc | 2016-06-20 12:57:21 -0500 | [diff] [blame] | 90 | <itemizedlist> |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 91 | <listitem><para>The self-contained SDK, which is an |
| 92 | architecture-specific cross-toolchain and |
Patrick Williams | d8c66bc | 2016-06-20 12:57:21 -0500 | [diff] [blame] | 93 | matching sysroots (target and native) all built by the |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 94 | OpenEmbedded build system (e.g. the SDK). |
Patrick Williams | d8c66bc | 2016-06-20 12:57:21 -0500 | [diff] [blame] | 95 | The toolchain and sysroots are based on a |
| 96 | <ulink url='&YOCTO_DOCS_DEV_URL;#metadata'>Metadata</ulink> |
| 97 | configuration and extensions, |
| 98 | which allows you to cross-develop on the host machine for the |
| 99 | target hardware. |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 100 | Additionally, the extensible SDK contains the |
| 101 | <filename>devtool</filename> functionality. |
Patrick Williams | d8c66bc | 2016-06-20 12:57:21 -0500 | [diff] [blame] | 102 | </para></listitem> |
| 103 | <listitem><para>The Quick EMUlator (QEMU), which lets you simulate |
| 104 | target hardware. |
| 105 | QEMU is not literally part of the SDK. |
| 106 | You must build and include this emulator separately. |
| 107 | However, QEMU plays an important role in the development |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 108 | process that revolves around use of the SDK. |
Patrick Williams | d8c66bc | 2016-06-20 12:57:21 -0500 | [diff] [blame] | 109 | </para></listitem> |
| 110 | <listitem><para>The Eclipse IDE Yocto Plug-in. |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 111 | This plug-in is available for you if you are an Eclipse |
Patrick Williams | d8c66bc | 2016-06-20 12:57:21 -0500 | [diff] [blame] | 112 | user. |
| 113 | In the same manner as QEMU, the plug-in is not literally part |
| 114 | of the SDK but is rather available for use as part of the |
| 115 | development process. |
| 116 | </para></listitem> |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 117 | <listitem><para>Various performance-related |
| 118 | <ulink url='http://www.eclipse.org/linuxtools/index.php'>tools</ulink> |
| 119 | that can enhance your development experience. |
Patrick Williams | d8c66bc | 2016-06-20 12:57:21 -0500 | [diff] [blame] | 120 | These tools are also separate from the actual SDK but can be |
| 121 | independently obtained and used in the development process. |
| 122 | </para></listitem> |
| 123 | </itemizedlist> |
| 124 | </para> |
| 125 | |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 126 | <para> |
| 127 | In summary, the extensible and standard SDK share many features. |
| 128 | However, the extensible SDK has powerful development tools to help you |
| 129 | more quickly develop applications. |
| 130 | Following is a table that summarizes the primary differences between |
| 131 | the standard and extensible SDK types when considering which to |
| 132 | build: |
| 133 | <informaltable frame='none'> |
| 134 | <tgroup cols='3' align='left' colsep='1' rowsep='1'> |
| 135 | <colspec colname='c1' colwidth='1*'/> |
| 136 | <colspec colname='c2' colwidth='1*'/> |
| 137 | <colspec colname='c3' colwidth='1*'/> |
| 138 | <thead> |
| 139 | <row> |
| 140 | <entry align="left"><emphasis>Feature</emphasis></entry> |
| 141 | <entry align="left"><emphasis>Standard SDK</emphasis></entry> |
| 142 | <entry align="left"><emphasis>Extensible SDK</emphasis></entry> |
| 143 | </row> |
| 144 | </thead> |
| 145 | <tbody> |
| 146 | <row> |
| 147 | <entry align="left">Toolchain</entry> |
| 148 | <entry align="left">Yes</entry> |
| 149 | <entry align="left">Yes*</entry> |
| 150 | </row> |
| 151 | <row> |
| 152 | <entry align="left">Debugger</entry> |
| 153 | <entry align="left">Yes</entry> |
| 154 | <entry align="left">Yes*</entry> |
| 155 | </row> |
| 156 | <row> |
| 157 | <entry align="left">Size</entry> |
| 158 | <entry align="left">100+ MBytes</entry> |
| 159 | <entry align="left">1+ GBytes (or 300+ MBytes for minimal w/toolchain)</entry> |
| 160 | </row> |
| 161 | <row> |
| 162 | <entry align="left"><filename>devtool</filename></entry> |
| 163 | <entry align="left">No</entry> |
| 164 | <entry align="left">Yes</entry> |
| 165 | </row> |
| 166 | <row> |
| 167 | <entry align="left">Build Images</entry> |
| 168 | <entry align="left">No</entry> |
| 169 | <entry align="left">Yes</entry> |
| 170 | </row> |
| 171 | <row> |
| 172 | <entry align="left">Updateable</entry> |
| 173 | <entry align="left">No</entry> |
| 174 | <entry align="left">Yes</entry> |
| 175 | </row> |
| 176 | <row> |
| 177 | <entry align="left">Managed Sysroot**</entry> |
| 178 | <entry align="left">No</entry> |
| 179 | <entry align="left">Yes</entry> |
| 180 | </row> |
| 181 | <row> |
| 182 | <entry align="left">Installed Packages</entry> |
| 183 | <entry align="left">No***</entry> |
| 184 | <entry align="left">Yes****</entry> |
| 185 | </row> |
| 186 | <row> |
| 187 | <entry align="left">Construction</entry> |
| 188 | <entry align="left">Packages</entry> |
| 189 | <entry align="left">Shared State</entry> |
| 190 | </row> |
| 191 | </tbody> |
| 192 | </tgroup> |
| 193 | </informaltable> |
| 194 | <literallayout class='monospaced'> |
| 195 | * Extensible SDK will contain the toolchain and debugger if <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_EXT_TYPE'><filename>SDK_EXT_TYPE</filename></ulink> is "full" or <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_INCLUDE_TOOLCHAIN'><filename>SDK_INCLUDE_TOOLCHAIN</filename></ulink> is "1", which is the default. |
| 196 | |
| 197 | ** Sysroot is managed through use of <filename>devtool</filename>. Thus, it is less likely that you will corrupt your SDK sysroot when you try to add additional libraries. |
| 198 | |
| 199 | *** Runtime package management can be added to the standard SDK but it is not supported by default. |
| 200 | |
| 201 | **** You must build and make the shared state available to extensible SDK users for "packages" you want to enable users to install. |
| 202 | </literallayout> |
| 203 | </para> |
| 204 | |
Patrick Williams | d8c66bc | 2016-06-20 12:57:21 -0500 | [diff] [blame] | 205 | <section id='the-cross-development-toolchain'> |
| 206 | <title>The Cross-Development Toolchain</title> |
| 207 | |
| 208 | <para> |
| 209 | The |
| 210 | <ulink url='&YOCTO_DOCS_DEV_URL;#cross-development-toolchain'>Cross-Development Toolchain</ulink> |
| 211 | consists of a cross-compiler, cross-linker, and cross-debugger |
| 212 | that are used to develop user-space applications for targeted |
| 213 | hardware. |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 214 | Additionally, for an extensible SDK, the toolchain also has |
| 215 | built-in <filename>devtool</filename> functionality. |
| 216 | This toolchain is created by running a SDK installer script |
Patrick Williams | d8c66bc | 2016-06-20 12:57:21 -0500 | [diff] [blame] | 217 | or through a |
| 218 | <ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink> |
| 219 | that is based on your Metadata configuration or extension for |
| 220 | your targeted device. |
| 221 | The cross-toolchain works with a matching target sysroot. |
| 222 | </para> |
| 223 | </section> |
| 224 | |
| 225 | <section id='sysroot'> |
| 226 | <title>Sysroots</title> |
| 227 | |
| 228 | <para> |
| 229 | The native and target sysroots contain needed headers and libraries |
| 230 | for generating binaries that run on the target architecture. |
| 231 | The target sysroot is based on the target root filesystem image |
| 232 | that is built by the OpenEmbedded build system and uses the same |
| 233 | Metadata configuration used to build the cross-toolchain. |
| 234 | </para> |
| 235 | </section> |
| 236 | |
| 237 | <section id='the-qemu-emulator'> |
| 238 | <title>The QEMU Emulator</title> |
| 239 | |
| 240 | <para> |
| 241 | The QEMU emulator allows you to simulate your hardware while |
| 242 | running your application or image. |
| 243 | QEMU is not part of the SDK but is made available a number of ways: |
| 244 | <itemizedlist> |
| 245 | <listitem><para> |
| 246 | If you have cloned the <filename>poky</filename> Git |
| 247 | repository to create a |
| 248 | <ulink url='&YOCTO_DOCS_DEV_URL;#source-directory'>Source Directory</ulink> |
| 249 | and you have sourced the environment setup script, QEMU is |
| 250 | installed and automatically available. |
| 251 | </para></listitem> |
| 252 | <listitem><para> |
| 253 | If you have downloaded a Yocto Project release and unpacked |
| 254 | it to create a |
| 255 | <ulink url='&YOCTO_DOCS_DEV_URL;#source-directory'>Source Directory</ulink> |
| 256 | and you have sourced the environment setup script, QEMU is |
| 257 | installed and automatically available. |
| 258 | </para></listitem> |
| 259 | <listitem><para> |
| 260 | If you have installed the cross-toolchain tarball and you |
| 261 | have sourced the toolchain's setup environment script, QEMU |
| 262 | is also installed and automatically available. |
| 263 | </para></listitem> |
| 264 | </itemizedlist> |
| 265 | </para> |
| 266 | </section> |
| 267 | |
| 268 | <section id='eclipse-overview'> |
| 269 | <title>Eclipse Yocto Plug-in</title> |
| 270 | |
| 271 | <para> |
| 272 | The Eclipse IDE is a popular development environment and it fully |
| 273 | supports development using the Yocto Project. |
| 274 | When you install and configure the Eclipse Yocto Project Plug-in |
| 275 | into the Eclipse IDE, you maximize your Yocto Project experience. |
| 276 | Installing and configuring the Plug-in results in an environment |
| 277 | that has extensions specifically designed to let you more easily |
| 278 | develop software. |
| 279 | These extensions allow for cross-compilation, deployment, and |
| 280 | execution of your output into a QEMU emulation session. |
| 281 | You can also perform cross-debugging and profiling. |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 282 | The environment also supports many performance-related |
| 283 | <ulink url='http://www.eclipse.org/linuxtools/index.php'>tools</ulink> |
| 284 | that enhance your development experience. |
| 285 | <note> |
| 286 | Previous releases of the Eclipse Yocto Plug-in supported |
| 287 | "user-space tools" (i.e. LatencyTOP, PowerTOP, Perf, SystemTap, |
| 288 | and Lttng-ust) that also added to the development experience. |
| 289 | These tools have been deprecated beginning with this release |
| 290 | of the plug-in. |
| 291 | </note> |
Patrick Williams | d8c66bc | 2016-06-20 12:57:21 -0500 | [diff] [blame] | 292 | </para> |
| 293 | |
| 294 | <para> |
| 295 | For information about the application development workflow that |
| 296 | uses the Eclipse IDE and for a detailed example of how to install |
| 297 | and configure the Eclipse Yocto Project Plug-in, see the |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 298 | "<link linkend='sdk-developing-applications-using-eclipse'>Developing Applications Using <trademark class='trade'>Eclipse</trademark></link>" |
Patrick Williams | d8c66bc | 2016-06-20 12:57:21 -0500 | [diff] [blame] | 299 | section. |
| 300 | </para> |
| 301 | </section> |
| 302 | |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 303 | <section id='performance-enhancing-tools'> |
| 304 | <title>Performance Enhancing Tools</title> |
Patrick Williams | d8c66bc | 2016-06-20 12:57:21 -0500 | [diff] [blame] | 305 | |
| 306 | <para> |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 307 | Supported performance enhancing tools are available that let you |
| 308 | profile, debug, and perform tracing on your projects developed |
| 309 | using Eclipse. |
| 310 | For information on these tools see |
| 311 | <ulink url='http://www.eclipse.org/linuxtools/'>http://www.eclipse.org/linuxtools/</ulink>. |
Patrick Williams | d8c66bc | 2016-06-20 12:57:21 -0500 | [diff] [blame] | 312 | </para> |
| 313 | </section> |
| 314 | </section> |
| 315 | |
| 316 | <section id='sdk-development-model'> |
| 317 | <title>SDK Development Model</title> |
| 318 | |
| 319 | <para> |
| 320 | Fundamentally, the SDK fits into the development process as follows: |
| 321 | <imagedata fileref="figures/sdk-environment.png" align="center" width="6in" depth="5in" scalefit="100" /> |
| 322 | The SDK is installed on any machine and can be used to develop |
| 323 | applications, images, and kernels. |
| 324 | An SDK can even be used by a QA Engineer or Release Engineer. |
| 325 | The fundamental concept is that the machine that has the SDK installed |
| 326 | does not have to be associated with the machine that has the |
| 327 | Yocto Project installed. |
| 328 | A developer can independently compile and test an object on their |
| 329 | machine and then, when the object is ready for integration into an |
| 330 | image, they can simply make it available to the machine that has the |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 331 | Yocto Project. |
Patrick Williams | d8c66bc | 2016-06-20 12:57:21 -0500 | [diff] [blame] | 332 | Once the object is available, the image can be rebuilt using the |
| 333 | Yocto Project to produce the modified image. |
| 334 | </para> |
| 335 | |
| 336 | <para> |
| 337 | You just need to follow these general steps: |
| 338 | <orderedlist> |
| 339 | <listitem><para><emphasis>Install the SDK for your target hardware:</emphasis> |
| 340 | For information on how to install the SDK, see the |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 341 | "<link linkend='sdk-installing-the-sdk'>Installing the SDK</link>" |
Patrick Williams | d8c66bc | 2016-06-20 12:57:21 -0500 | [diff] [blame] | 342 | section.</para></listitem> |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 343 | <listitem><para><emphasis>Download or Build the Target Image:</emphasis> |
Patrick Williams | d8c66bc | 2016-06-20 12:57:21 -0500 | [diff] [blame] | 344 | The Yocto Project supports several target architectures |
| 345 | and has many pre-built kernel images and root filesystem |
| 346 | images.</para> |
| 347 | <para>If you are going to develop your application on |
| 348 | hardware, go to the |
| 349 | <ulink url='&YOCTO_MACHINES_DL_URL;'><filename>machines</filename></ulink> |
| 350 | download area and choose a target machine area |
| 351 | from which to download the kernel image and root filesystem. |
| 352 | This download area could have several files in it that |
| 353 | support development using actual hardware. |
| 354 | For example, the area might contain |
| 355 | <filename>.hddimg</filename> files that combine the |
| 356 | kernel image with the filesystem, boot loaders, and |
| 357 | so forth. |
| 358 | Be sure to get the files you need for your particular |
| 359 | development process.</para> |
| 360 | <para>If you are going to develop your application and |
| 361 | then run and test it using the QEMU emulator, go to the |
| 362 | <ulink url='&YOCTO_QEMU_DL_URL;'><filename>machines/qemu</filename></ulink> |
| 363 | download area. |
| 364 | From this area, go down into the directory for your |
| 365 | target architecture (e.g. <filename>qemux86_64</filename> |
| 366 | for an <trademark class='registered'>Intel</trademark>-based |
| 367 | 64-bit architecture). |
| 368 | Download kernel, root filesystem, and any other files you |
| 369 | need for your process. |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 370 | <note> |
| 371 | To use the root filesystem in QEMU, you |
| 372 | need to extract it. |
| 373 | See the |
| 374 | "<link linkend='sdk-extracting-the-root-filesystem'>Extracting the Root Filesystem</link>" |
| 375 | section for information on how to extract the root |
| 376 | filesystem. |
| 377 | </note> |
| 378 | </para></listitem> |
Patrick Williams | d8c66bc | 2016-06-20 12:57:21 -0500 | [diff] [blame] | 379 | <listitem><para><emphasis>Develop and Test your |
| 380 | Application:</emphasis> At this point, you have the tools |
| 381 | to develop your application. |
| 382 | If you need to separately install and use the QEMU |
| 383 | emulator, you can go to |
| 384 | <ulink url='http://wiki.qemu.org/Main_Page'>QEMU Home Page</ulink> |
| 385 | to download and learn about the emulator. |
Patrick Williams | c0f7c04 | 2017-02-23 20:41:17 -0600 | [diff] [blame] | 386 | See the |
Patrick Williams | d8c66bc | 2016-06-20 12:57:21 -0500 | [diff] [blame] | 387 | "<ulink url='&YOCTO_DOCS_DEV_URL;#dev-manual-qemu'>Using the Quick EMUlator (QEMU)</ulink>" |
| 388 | chapter in the Yocto Project Development Manual |
| 389 | for information on using QEMU within the Yocto |
| 390 | Project.</para></listitem> |
| 391 | </orderedlist> |
| 392 | </para> |
| 393 | |
| 394 | <para> |
| 395 | The remainder of this manual describes how to use both the standard |
| 396 | SDK and the extensible SDK. |
| 397 | Information also exists in appendix form that describes how you can |
| 398 | build, install, and modify an SDK. |
| 399 | </para> |
| 400 | </section> |
| 401 | |
| 402 | </chapter> |
| 403 | <!-- |
| 404 | vim: expandtab tw=80 ts=4 |
| 405 | --> |