blob: e95536e052d2260353b5437da92926172c4ced21 [file] [log] [blame]
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001.. SPDX-License-Identifier: CC-BY-2.0-UK
2
3**********************
4Concepts and Reference
5**********************
6
7In order to configure and use Toaster, you should understand some
8concepts and have some basic command reference material available. This
9final chapter provides conceptual information on layer sources,
10releases, and JSON configuration files. Also provided is a quick look at
11some useful ``manage.py`` commands that are Toaster-specific.
12Information on ``manage.py`` commands does exist across the Web and the
13information in this manual by no means attempts to provide a command
14comprehensive reference.
15
16Layer Source
17============
18
19In general, a "layer source" is a source of information about existing
20layers. In particular, we are concerned with layers that you can use
21with the Yocto Project and Toaster. This chapter describes a particular
22type of layer source called a "layer index."
23
24A layer index is a web application that contains information about a set
25of custom layers. A good example of an existing layer index is the
26OpenEmbedded Layer Index. A public instance of this layer index exists
27at http://layers.openembedded.org. You can find the code for this
28layer index's web application at
29http://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/.
30
31When you tie a layer source into Toaster, it can query the layer source
32through a
33`REST <http://en.wikipedia.org/wiki/Representational_state_transfer>`__
34API, store the information about the layers in the Toaster database, and
35then show the information to users. Users are then able to view that
36information and build layers from Toaster itself without worrying about
37cloning or editing the BitBake layers configuration file
38``bblayers.conf``.
39
40Tying a layer source into Toaster is convenient when you have many
41custom layers that need to be built on a regular basis by a community of
42developers. In fact, Toaster comes pre-configured with the OpenEmbedded
43Metadata Index.
44
45.. note::
46
47 You do not have to use a layer source to use Toaster. Tying into a
48 layer source is optional.
49
50.. _layer-source-using-with-toaster:
51
52Setting Up and Using a Layer Source
53-----------------------------------
54
55To use your own layer source, you need to set up the layer source and
56then tie it into Toaster. This section describes how to tie into a layer
57index in a manner similar to the way Toaster ties into the OpenEmbedded
58Metadata Index.
59
60Understanding Your Layers
61~~~~~~~~~~~~~~~~~~~~~~~~~
62
63The obvious first step for using a layer index is to have several custom
64layers that developers build and access using the Yocto Project on a
65regular basis. This set of layers needs to exist and you need to be
66familiar with where they reside. You will need that information when you
67set up the code for the web application that "hooks" into your set of
68layers.
69
70For general information on layers, see the
71":ref:`overview-manual/overview-manual-yp-intro:the yocto project layer model`"
72section in the Yocto Project Overview and Concepts Manual. For information on how
73to create layers, see the ":ref:`dev-manual/dev-manual-common-tasks:understanding and creating layers`"
74section in the Yocto Project Development Tasks Manual.
75
76.. _configuring-toaster-to-hook-into-your-layer-source:
77
78Configuring Toaster to Hook Into Your Layer Index
79~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
80
81If you want Toaster to use your layer index, you must host the web
82application in a server to which Toaster can connect. You also need to
83give Toaster the information about your layer index. In other words, you
84have to configure Toaster to use your layer index. This section
85describes two methods by which you can configure and use your layer
86index.
87
88In the previous section, the code for the OpenEmbedded Metadata Index
89(i.e. http://layers.openembedded.org) was referenced. You can use
90this code, which is at
91http://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/, as a
92base to create your own layer index.
93
94Use the Administration Interface
95^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
96
97Access the administration interface through a browser by entering the
98URL of your Toaster instance and adding "``/admin``" to the end of the
99URL. As an example, if you are running Toaster locally, use the
100following URL::
101
102 http://127.0.0.1:8000/admin
103
104The administration interface has a "Layer sources" section that includes
105an "Add layer source" button. Click that button and provide the required
106information. Make sure you select "layerindex" as the layer source type.
107
108Use the Fixture Feature
109^^^^^^^^^^^^^^^^^^^^^^^
110
111The Django fixture feature overrides the default layer server when you
112use it to specify a custom URL. To use the fixture feature, create (or
113edit) the file ``bitbake/lib/toaster.orm/fixtures/custom.xml``, and then
114set the following Toaster setting to your custom URL:
115
116.. code-block:: xml
117
118 <?xml version="1.0" ?>
119 <django-objects version="1.0">
120 <object model="orm.toastersetting" pk="100">
121 <field name="name" type="CharField">CUSTOM_LAYERINDEX_SERVER</field>
122 <field name="value" type="CharField">https://layers.my_organization.org/layerindex/branch/master/layers/</field>
123 </object>
124 <django-objects>
125
126When you start Toaster for the first time, or
127if you delete the file ``toaster.sqlite`` and restart, the database will
128populate cleanly from this layer index server.
129
130Once the information has been updated, verify the new layer information
131is available by using the Toaster web interface. To do that, visit the
132"All compatible layers" page inside a Toaster project. The layers from
133your layer source should be listed there.
134
135If you change the information in your layer index server, refresh the
136Toaster database by running the following command:
137
138.. code-block:: shell
139
140 $ bitbake/lib/toaster/manage.py lsupdates
141
142
143If Toaster can reach the API URL, you should see a message telling you that
144Toaster is updating the layer source information.
145
146.. _toaster-releases:
147
148Releases
149========
150
151When you create a Toaster project using the web interface, you are asked
152to choose a "Release." In the context of Toaster, the term "Release"
153refers to a set of layers and a BitBake version the OpenEmbedded build
154system uses to build something. As shipped, Toaster is pre-configured
155with releases that correspond to Yocto Project release branches.
156However, you can modify, delete, and create new releases according to
157your needs. This section provides some background information on
158releases.
159
160.. _toaster-releases-supported:
161
162Pre-Configured Releases
163-----------------------
164
165As shipped, Toaster is configured to use a specific set of releases. Of
166course, you can always configure Toaster to use any release. For
167example, you might want your project to build against a specific commit
168of any of the "out-of-the-box" releases. Or, you might want your project
169to build against different revisions of OpenEmbedded and BitBake.
170
171As shipped, Toaster is configured to work with the following releases:
172
173- *Yocto Project &DISTRO; "&DISTRO_NAME;" or OpenEmbedded "&DISTRO_NAME;":*
174 This release causes your Toaster projects to build against the head
175 of the &DISTRO_NAME_NO_CAP; branch at
176 https://git.yoctoproject.org/cgit/cgit.cgi/poky/log/?h=&DISTRO_NAME_NO_CAP; or
177 http://git.openembedded.org/openembedded-core/commit/?h=&DISTRO_NAME_NO_CAP;.
178
179- *Yocto Project "Master" or OpenEmbedded "Master":* This release
180 causes your Toaster Projects to build against the head of the master
181 branch, which is where active development takes place, at
182 https://git.yoctoproject.org/cgit/cgit.cgi/poky/log/ or
183 http://git.openembedded.org/openembedded-core/log/.
184
185- *Local Yocto Project or Local OpenEmbedded:* This release causes your
186 Toaster Projects to build against the head of the ``poky`` or
187 ``openembedded-core`` clone you have local to the machine running
188 Toaster.
189
190Configuring Toaster
191===================
192
193In order to use Toaster, you must configure the database with the
194default content. The following subsections describe various aspects of
195Toaster configuration.
196
197Configuring the Workflow
198------------------------
199
200The ``bldcontrol/management/commands/checksettings.py`` file controls
201workflow configuration. The following steps outline the process to
202initially populate this database.
203
2041. The default project settings are set from
205 ``orm/fixtures/settings.xml``.
206
2072. The default project distro and layers are added from
208 ``orm/fixtures/poky.xml`` if poky is installed. If poky is not
209 installed, they are added from ``orm/fixtures/oe-core.xml``.
210
2113. If the ``orm/fixtures/custom.xml`` file exists, then its values are
212 added.
213
2144. The layer index is then scanned and added to the database.
215
216Once these steps complete, Toaster is set up and ready to use.
217
218Customizing Pre-Set Data
219------------------------
220
221The pre-set data for Toaster is easily customizable. You can create the
222``orm/fixtures/custom.xml`` file to customize the values that go into to
223the database. Customization is additive, and can either extend or
224completely replace the existing values.
225
226You use the ``orm/fixtures/custom.xml`` file to change the default
227project settings for the machine, distro, file images, and layers. When
228creating a new project, you can use the file to define the offered
229alternate project release selections. For example, you can add one or
230more additional selections that present custom layer sets or distros,
231and any other local or proprietary content.
232
233Additionally, you can completely disable the content from the
234``oe-core.xml`` and ``poky.xml`` files by defining the section shown
235below in the ``settings.xml`` file. For example, this option is
236particularly useful if your custom configuration defines fewer releases
237or layers than the default fixture files.
238
239The following example sets "name" to "CUSTOM_XML_ONLY" and its value to
240"True".
241
242.. code-block:: xml
243
244 <object model="orm.toastersetting" pk="99">
245 <field type="CharField" name="name">CUSTOM_XML_ONLY</field>
246 <field type="CharField" name="value">True</field>
247 </object>
248
249Understanding Fixture File Format
250---------------------------------
251
252The following is an overview of the file format used by the
253``oe-core.xml``, ``poky.xml``, and ``custom.xml`` files.
254
255The following subsections describe each of the sections in the fixture
256files, and outline an example section of the XML code. you can use to
257help understand this information and create a local ``custom.xml`` file.
258
259Defining the Default Distro and Other Values
260~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
261
262This section defines the default distro value for new projects. By
263default, it reserves the first Toaster Setting record "1". The following
264demonstrates how to set the project default value for
265:term:`DISTRO`:
266
267.. code-block:: xml
268
269 <!-- Set the project default value for DISTRO -->
270 <object model="orm.toastersetting" pk="1">
271 <field type="CharField" name="name">DEFCONF_DISTRO</field>
272 <field type="CharField" name="value">poky</field>
273 </object>
274
275You can override
276other default project values by adding additional Toaster Setting
277sections such as any of the settings coming from the ``settings.xml``
278file. Also, you can add custom values that are included in the BitBake
279environment. The "pk" values must be unique. By convention, values that
280set default project values have a "DEFCONF" prefix.
281
282Defining BitBake Version
283~~~~~~~~~~~~~~~~~~~~~~~~
284
285The following defines which version of BitBake is used for the following
286release selection:
287
288.. code-block:: xml
289
290 <!-- Bitbake versions which correspond to the metadata release -->
291 <object model="orm.bitbakeversion" pk="1">
292 <field type="CharField" name="name">&DISTRO_NAME_NO_CAP;</field>
293 <field type="CharField" name="giturl">git://git.yoctoproject.org/poky</field>
294 <field type="CharField" name="branch">&DISTRO_NAME_NO_CAP;</field>
295 <field type="CharField" name="dirpath">bitbake</field>
296 </object>
297
298.. _defining-releases:
299
300Defining Release
301~~~~~~~~~~~~~~~~
302
303The following defines the releases when you create a new project:
304
305.. code-block:: xml
306
307 <!-- Releases available -->
308 <object model="orm.release" pk="1">
309 <field type="CharField" name="name">&DISTRO_NAME_NO_CAP;</field>
310 <field type="CharField" name="description">Yocto Project &DISTRO; "&DISTRO_NAME;"</field>
311 <field rel="ManyToOneRel" to="orm.bitbakeversion" name="bitbake_version">1</field>
312 <field type="CharField" name="branch_name">&DISTRO_NAME_NO_CAP;</field>
313 <field type="TextField" name="helptext">Toaster will run your builds using the tip of the <a href="http://git.yoctoproject.org/cgit/cgit.cgi/poky/log/?h=&DISTRO_NAME_NO_CAP;">Yocto Project &DISTRO_NAME; branch</a>.</field>
314 </object>
315
316The "pk" value must match the above respective BitBake version record.
317
318Defining the Release Default Layer Names
319~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
320
321The following defines the default layers for each release:
322
323.. code-block:: xml
324
325 <!-- Default project layers for each release -->
326 <object model="orm.releasedefaultlayer" pk="1">
327 <field rel="ManyToOneRel" to="orm.release" name="release">1</field>
328 <field type="CharField" name="layer_name">openembedded-core</field>
329 </object>
330
331The 'pk' values in the example above should start at "1" and increment
332uniquely. You can use the same layer name in multiple releases.
333
334Defining Layer Definitions
335~~~~~~~~~~~~~~~~~~~~~~~~~~
336
337Layer definitions are the most complex. The following defines each of
338the layers, and then defines the exact layer version of the layer used
339for each respective release. You must have one ``orm.layer`` entry for
340each layer. Then, with each entry you need a set of
341``orm.layer_version`` entries that connects the layer with each release
342that includes the layer. In general all releases include the layer.
343
344.. code-block:: xml
345
346 <object model="orm.layer" pk="1">
347 <field type="CharField" name="name">openembedded-core</field>
348 <field type="CharField" name="layer_index_url"></field>
349 <field type="CharField" name="vcs_url">git://git.yoctoproject.org/poky</field>
350 <field type="CharField" name="vcs_web_url">http://git.yoctoproject.org/cgit/cgit.cgi/poky</field>
351 <field type="CharField" name="vcs_web_tree_base_url">http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/%path%?h=%branch%</field>
352 <field type="CharField" name="vcs_web_file_base_url">http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/%path%?h=%branch%</field>
353 </object>
354 <object model="orm.layer_version" pk="1">
355 <field rel="ManyToOneRel" to="orm.layer" name="layer">1</field>
356 <field type="IntegerField" name="layer_source">0</field>
357 <field rel="ManyToOneRel" to="orm.release" name="release">1</field>
358 <field type="CharField" name="branch">&DISTRO_NAME_NO_CAP;</field>
359 <field type="CharField" name="dirpath">meta</field>
360 </object> <object model="orm.layer_version" pk="2">
361 <field rel="ManyToOneRel" to="orm.layer" name="layer">1</field>
362 <field type="IntegerField" name="layer_source">0</field>
363 <field rel="ManyToOneRel" to="orm.release" name="release">2</field>
364 <field type="CharField" name="branch">HEAD</field>
365 <field type="CharField" name="commit">HEAD</field>
366 <field type="CharField" name="dirpath">meta</field>
367 </object>
368 <object model="orm.layer_version" pk="3">
369 <field rel="ManyToOneRel" to="orm.layer" name="layer">1</field>
370 <field type="IntegerField" name="layer_source">0</field>
371 <field rel="ManyToOneRel" to="orm.release" name="release">3</field>
372 <field type="CharField" name="branch">master</field>
373 <field type="CharField" name="dirpath">meta</field>
374 </object>
375
376The layer "pk" values above must be unique, and typically start at "1". The
377layer version "pk" values must also be unique across all layers, and typically
378start at "1".
379
380Remote Toaster Monitoring
381=========================
382
383Toaster has an API that allows remote management applications to
384directly query the state of the Toaster server and its builds in a
385machine-to-machine manner. This API uses the
386`REST <http://en.wikipedia.org/wiki/Representational_state_transfer>`__
387interface and the transfer of JSON files. For example, you might monitor
388a build inside a container through well supported known HTTP ports in
389order to easily access a Toaster server inside the container. In this
390example, when you use this direct JSON API, you avoid having web page
391parsing against the display the user sees.
392
393Checking Health
394---------------
395
396Before you use remote Toaster monitoring, you should do a health check.
397To do this, ping the Toaster server using the following call to see if
398it is still alive::
399
400 http://host:port/health
401
402Be sure to provide values for host and port. If the server is alive, you will
403get the response HTML:
404
405.. code-block:: html
406
407 <!DOCTYPE html>
408 <html lang="en">
409 <head><title>Toaster Health</title></head>
410 <body>Ok</body>
411 </html>
412
413Determining Status of Builds in Progress
414----------------------------------------
415
416Sometimes it is useful to determine the status of a build in progress.
417To get the status of pending builds, use the following call::
418
419 http://host:port/toastergui/api/building
420
421Be sure to provide values for host and port. The output is a JSON file that
422itemizes all builds in progress. This file includes the time in seconds since
423each respective build started as well as the progress of the cloning, parsing,
424and task execution. The following is sample output for a build in progress:
425
426.. code-block:: JSON
427
428 {"count": 1,
429 "building": [
430 {"machine": "beaglebone",
431 "seconds": "463.869",
432 "task": "927:2384",
433 "distro": "poky",
434 "clone": "1:1",
435 "id": 2,
436 "start": "2017-09-22T09:31:44.887Z",
437 "name": "20170922093200",
438 "parse": "818:818",
439 "project": "my_rocko",
440 "target": "core-image-minimal"
441 }]
442 }
443
444The JSON data for this query is returned in a
445single line. In the previous example the line has been artificially
446split for readability.
447
448Checking Status of Builds Completed
449-----------------------------------
450
451Once a build is completed, you get the status when you use the following
452call::
453
454 http://host:port/toastergui/api/builds
455
456Be sure to provide values for host and port. The output is a JSON file that
457itemizes all complete builds, and includes build summary information. The
458following is sample output for a completed build:
459
460.. code-block:: JSON
461
462 {"count": 1,
463 "builds": [
464 {"distro": "poky",
465 "errors": 0,
466 "machine": "beaglebone",
467 "project": "my_rocko",
468 "stop": "2017-09-22T09:26:36.017Z",
469 "target": "quilt-native",
470 "seconds": "78.193",
471 "outcome": "Succeeded",
472 "id": 1,
473 "start": "2017-09-22T09:25:17.824Z",
474 "warnings": 1,
475 "name": "20170922092618"
476 }]
477 }
478
479The JSON data for this query is returned in a single line. In the
480previous example the line has been artificially split for readability.
481
482Determining Status of a Specific Build
483--------------------------------------
484
485Sometimes it is useful to determine the status of a specific build. To
486get the status of a specific build, use the following call::
487
488 http://host:port/toastergui/api/build/ID
489
490Be sure to provide values for
491host, port, and ID. You can find the value for ID from the Builds
492Completed query. See the ":ref:`toaster-manual/toaster-manual-reference:checking status of builds completed`"
493section for more information.
494
495The output is a JSON file that itemizes the specific build and includes
496build summary information. The following is sample output for a specific
497build:
498
499.. code-block:: JSON
500
501 {"build":
502 {"distro": "poky",
503 "errors": 0,
504 "machine": "beaglebone",
505 "project": "my_rocko",
506 "stop": "2017-09-22T09:26:36.017Z",
507 "target": "quilt-native",
508 "seconds": "78.193",
509 "outcome": "Succeeded",
510 "id": 1,
511 "start": "2017-09-22T09:25:17.824Z",
512 "warnings": 1,
513 "name": "20170922092618",
514 "cooker_log": "/opt/user/poky/build-toaster-2/tmp/log/cooker/beaglebone/build_20170922_022607.991.log"
515 }
516 }
517
518The JSON data for this query is returned in a single line. In the
519previous example the line has been artificially split for readability.
520
521.. _toaster-useful-commands:
522
523Useful Commands
524===============
525
526In addition to the web user interface and the scripts that start and
527stop Toaster, command-line commands exist through the ``manage.py``
528management script. You can find general documentation on ``manage.py``
529at the
530`Django <https://docs.djangoproject.com/en/2.2/topics/settings/>`__
531site. However, several ``manage.py`` commands have been created that are
532specific to Toaster and are used to control configuration and back-end
533tasks. You can locate these commands in the
534:term:`Source Directory` (e.g. ``poky``) at
535``bitbake/lib/manage.py``. This section documents those commands.
536
537.. note::
538
539 - When using ``manage.py`` commands given a default configuration,
540 you must be sure that your working directory is set to the
541 :term:`Build Directory`. Using
542 ``manage.py`` commands from the Build Directory allows Toaster to
543 find the ``toaster.sqlite`` file, which is located in the Build
544 Directory.
545
546 - For non-default database configurations, it is possible that you
547 can use ``manage.py`` commands from a directory other than the
548 Build Directory. To do so, the ``toastermain/settings.py`` file
549 must be configured to point to the correct database backend.
550
551.. _toaster-command-buildslist:
552
553``buildslist``
554--------------
555
556The ``buildslist`` command lists all builds that Toaster has recorded.
557Access the command as follows:
558
559.. code-block:: shell
560
561 $ bitbake/lib/toaster/manage.py buildslist
562
563The command returns a list, which includes numeric
564identifications, of the builds that Toaster has recorded in the current
565database.
566
567You need to run the ``buildslist`` command first to identify existing
568builds in the database before using the
569:ref:`toaster-manual/toaster-manual-reference:\`\`builddelete\`\`` command. Here is an
570example that assumes default repository and build directory names:
571
572.. code-block:: shell
573
574 $ cd ~/poky/build
575 $ python ../bitbake/lib/toaster/manage.py buildslist
576
577If your Toaster database had only one build, the above
578:ref:`toaster-manual/toaster-manual-reference:\`\`buildslist\`\``
579command would return something like the following::
580
581 1: qemux86 poky core-image-minimal
582
583.. _toaster-command-builddelete:
584
585``builddelete``
586---------------
587
588The ``builddelete`` command deletes data associated with a build. Access
589the command as follows:
590
591.. code-block::
592
593 $ bitbake/lib/toaster/manage.py builddelete build_id
594
595The command deletes all the build data for the specified
596build_id. This command is useful for removing old and unused data from
597the database.
598
599Prior to running the ``builddelete`` command, you need to get the ID
600associated with builds by using the
601:ref:`toaster-manual/toaster-manual-reference:\`\`buildslist\`\`` command.
602
603.. _toaster-command-perf:
604
605``perf``
606--------
607
608The ``perf`` command measures Toaster performance. Access the command as
609follows:
610
611.. code-block:: shell
612
613 $ bitbake/lib/toaster/manage.py perf
614
615The command is a sanity check that returns page loading times in order to
616identify performance problems.
617
618.. _toaster-command-checksettings:
619
620``checksettings``
621-----------------
622
623The ``checksettings`` command verifies existing Toaster settings. Access
624the command as follows:
625
626.. code-block:: shell
627
628 $ bitbake/lib/toaster/manage.py checksettings
629
630Toaster uses settings that are based on the database to configure the
631building tasks. The ``checksettings`` command verifies that the database
632settings are valid in the sense that they have the minimal information
633needed to start a build.
634
635In order for the ``checksettings`` command to work, the database must be
636correctly set up and not have existing data. To be sure the database is
637ready, you can run the following:
638
639.. code-block:: shell
640
641 $ bitbake/lib/toaster/manage.py syncdb
642 $ bitbake/lib/toaster/manage.py migrate orm
643 $ bitbake/lib/toaster/manage.py migrate bldcontrol
644
645After running these commands, you can run the ``checksettings`` command.
646
647.. _toaster-command-runbuilds:
648
649``runbuilds``
650-------------
651
652The ``runbuilds`` command launches scheduled builds. Access the command
653as follows:
654
655.. code-block:: shell
656
657 $ bitbake/lib/toaster/manage.py runbuilds
658
659The ``runbuilds`` command checks if scheduled builds exist in the database
660and then launches them per schedule. The command returns after the builds
661start but before they complete. The Toaster Logging Interface records and
662updates the database when the builds complete.