blob: 2202d599f62402c0929b92fae0b944f689e9e3c9 [file] [log] [blame]
Andrew Geisslerf0343792020-11-18 10:42:21 -06001.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002
3**********************
4Concepts and Reference
5**********************
6
7In order to configure and use Toaster, you should understand some
8concepts and have some basic command reference material available. This
9final chapter provides conceptual information on layer sources,
10releases, and JSON configuration files. Also provided is a quick look at
11some useful ``manage.py`` commands that are Toaster-specific.
12Information on ``manage.py`` commands does exist across the Web and the
13information in this manual by no means attempts to provide a command
14comprehensive reference.
15
16Layer Source
17============
18
19In general, a "layer source" is a source of information about existing
20layers. In particular, we are concerned with layers that you can use
21with the Yocto Project and Toaster. This chapter describes a particular
22type of layer source called a "layer index."
23
24A layer index is a web application that contains information about a set
25of custom layers. A good example of an existing layer index is the
26OpenEmbedded Layer Index. A public instance of this layer index exists
27at http://layers.openembedded.org. You can find the code for this
28layer index's web application at
29http://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/.
30
31When you tie a layer source into Toaster, it can query the layer source
32through a
33`REST <http://en.wikipedia.org/wiki/Representational_state_transfer>`__
34API, store the information about the layers in the Toaster database, and
35then show the information to users. Users are then able to view that
36information and build layers from Toaster itself without worrying about
37cloning or editing the BitBake layers configuration file
38``bblayers.conf``.
39
40Tying a layer source into Toaster is convenient when you have many
41custom layers that need to be built on a regular basis by a community of
42developers. In fact, Toaster comes pre-configured with the OpenEmbedded
43Metadata Index.
44
45.. note::
46
47 You do not have to use a layer source to use Toaster. Tying into a
48 layer source is optional.
49
Andrew Geisslerc9f78652020-09-18 14:11:35 -050050Setting Up and Using a Layer Source
51-----------------------------------
52
53To use your own layer source, you need to set up the layer source and
54then tie it into Toaster. This section describes how to tie into a layer
55index in a manner similar to the way Toaster ties into the OpenEmbedded
56Metadata Index.
57
58Understanding Your Layers
59~~~~~~~~~~~~~~~~~~~~~~~~~
60
61The obvious first step for using a layer index is to have several custom
62layers that developers build and access using the Yocto Project on a
63regular basis. This set of layers needs to exist and you need to be
64familiar with where they reside. You will need that information when you
65set up the code for the web application that "hooks" into your set of
66layers.
67
68For general information on layers, see the
69":ref:`overview-manual/overview-manual-yp-intro:the yocto project layer model`"
70section in the Yocto Project Overview and Concepts Manual. For information on how
71to create layers, see the ":ref:`dev-manual/dev-manual-common-tasks:understanding and creating layers`"
72section in the Yocto Project Development Tasks Manual.
73
Andrew Geisslerc9f78652020-09-18 14:11:35 -050074Configuring Toaster to Hook Into Your Layer Index
75~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
76
77If you want Toaster to use your layer index, you must host the web
78application in a server to which Toaster can connect. You also need to
79give Toaster the information about your layer index. In other words, you
80have to configure Toaster to use your layer index. This section
81describes two methods by which you can configure and use your layer
82index.
83
84In the previous section, the code for the OpenEmbedded Metadata Index
85(i.e. http://layers.openembedded.org) was referenced. You can use
86this code, which is at
87http://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/, as a
88base to create your own layer index.
89
90Use the Administration Interface
91^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
92
93Access the administration interface through a browser by entering the
94URL of your Toaster instance and adding "``/admin``" to the end of the
95URL. As an example, if you are running Toaster locally, use the
96following URL::
97
98 http://127.0.0.1:8000/admin
99
100The administration interface has a "Layer sources" section that includes
101an "Add layer source" button. Click that button and provide the required
102information. Make sure you select "layerindex" as the layer source type.
103
104Use the Fixture Feature
105^^^^^^^^^^^^^^^^^^^^^^^
106
107The Django fixture feature overrides the default layer server when you
108use it to specify a custom URL. To use the fixture feature, create (or
109edit) the file ``bitbake/lib/toaster.orm/fixtures/custom.xml``, and then
110set the following Toaster setting to your custom URL:
111
112.. code-block:: xml
113
114 <?xml version="1.0" ?>
115 <django-objects version="1.0">
116 <object model="orm.toastersetting" pk="100">
117 <field name="name" type="CharField">CUSTOM_LAYERINDEX_SERVER</field>
118 <field name="value" type="CharField">https://layers.my_organization.org/layerindex/branch/master/layers/</field>
119 </object>
120 <django-objects>
121
122When you start Toaster for the first time, or
123if you delete the file ``toaster.sqlite`` and restart, the database will
124populate cleanly from this layer index server.
125
126Once the information has been updated, verify the new layer information
127is available by using the Toaster web interface. To do that, visit the
128"All compatible layers" page inside a Toaster project. The layers from
129your layer source should be listed there.
130
131If you change the information in your layer index server, refresh the
132Toaster database by running the following command:
133
134.. code-block:: shell
135
136 $ bitbake/lib/toaster/manage.py lsupdates
137
138
139If Toaster can reach the API URL, you should see a message telling you that
140Toaster is updating the layer source information.
141
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500142Releases
143========
144
145When you create a Toaster project using the web interface, you are asked
146to choose a "Release." In the context of Toaster, the term "Release"
147refers to a set of layers and a BitBake version the OpenEmbedded build
148system uses to build something. As shipped, Toaster is pre-configured
149with releases that correspond to Yocto Project release branches.
150However, you can modify, delete, and create new releases according to
151your needs. This section provides some background information on
152releases.
153
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500154Pre-Configured Releases
155-----------------------
156
157As shipped, Toaster is configured to use a specific set of releases. Of
158course, you can always configure Toaster to use any release. For
159example, you might want your project to build against a specific commit
160of any of the "out-of-the-box" releases. Or, you might want your project
161to build against different revisions of OpenEmbedded and BitBake.
162
163As shipped, Toaster is configured to work with the following releases:
164
165- *Yocto Project &DISTRO; "&DISTRO_NAME;" or OpenEmbedded "&DISTRO_NAME;":*
166 This release causes your Toaster projects to build against the head
167 of the &DISTRO_NAME_NO_CAP; branch at
168 https://git.yoctoproject.org/cgit/cgit.cgi/poky/log/?h=&DISTRO_NAME_NO_CAP; or
169 http://git.openembedded.org/openembedded-core/commit/?h=&DISTRO_NAME_NO_CAP;.
170
171- *Yocto Project "Master" or OpenEmbedded "Master":* This release
172 causes your Toaster Projects to build against the head of the master
173 branch, which is where active development takes place, at
174 https://git.yoctoproject.org/cgit/cgit.cgi/poky/log/ or
175 http://git.openembedded.org/openembedded-core/log/.
176
177- *Local Yocto Project or Local OpenEmbedded:* This release causes your
178 Toaster Projects to build against the head of the ``poky`` or
179 ``openembedded-core`` clone you have local to the machine running
180 Toaster.
181
182Configuring Toaster
183===================
184
185In order to use Toaster, you must configure the database with the
186default content. The following subsections describe various aspects of
187Toaster configuration.
188
189Configuring the Workflow
190------------------------
191
192The ``bldcontrol/management/commands/checksettings.py`` file controls
193workflow configuration. The following steps outline the process to
194initially populate this database.
195
1961. The default project settings are set from
197 ``orm/fixtures/settings.xml``.
198
1992. The default project distro and layers are added from
200 ``orm/fixtures/poky.xml`` if poky is installed. If poky is not
201 installed, they are added from ``orm/fixtures/oe-core.xml``.
202
2033. If the ``orm/fixtures/custom.xml`` file exists, then its values are
204 added.
205
2064. The layer index is then scanned and added to the database.
207
208Once these steps complete, Toaster is set up and ready to use.
209
210Customizing Pre-Set Data
211------------------------
212
213The pre-set data for Toaster is easily customizable. You can create the
214``orm/fixtures/custom.xml`` file to customize the values that go into to
215the database. Customization is additive, and can either extend or
216completely replace the existing values.
217
218You use the ``orm/fixtures/custom.xml`` file to change the default
219project settings for the machine, distro, file images, and layers. When
220creating a new project, you can use the file to define the offered
221alternate project release selections. For example, you can add one or
222more additional selections that present custom layer sets or distros,
223and any other local or proprietary content.
224
225Additionally, you can completely disable the content from the
226``oe-core.xml`` and ``poky.xml`` files by defining the section shown
227below in the ``settings.xml`` file. For example, this option is
228particularly useful if your custom configuration defines fewer releases
229or layers than the default fixture files.
230
231The following example sets "name" to "CUSTOM_XML_ONLY" and its value to
232"True".
233
234.. code-block:: xml
235
236 <object model="orm.toastersetting" pk="99">
237 <field type="CharField" name="name">CUSTOM_XML_ONLY</field>
238 <field type="CharField" name="value">True</field>
239 </object>
240
241Understanding Fixture File Format
242---------------------------------
243
244The following is an overview of the file format used by the
245``oe-core.xml``, ``poky.xml``, and ``custom.xml`` files.
246
247The following subsections describe each of the sections in the fixture
248files, and outline an example section of the XML code. you can use to
249help understand this information and create a local ``custom.xml`` file.
250
251Defining the Default Distro and Other Values
252~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
253
254This section defines the default distro value for new projects. By
255default, it reserves the first Toaster Setting record "1". The following
256demonstrates how to set the project default value for
257:term:`DISTRO`:
258
259.. code-block:: xml
260
261 <!-- Set the project default value for DISTRO -->
262 <object model="orm.toastersetting" pk="1">
263 <field type="CharField" name="name">DEFCONF_DISTRO</field>
264 <field type="CharField" name="value">poky</field>
265 </object>
266
267You can override
268other default project values by adding additional Toaster Setting
269sections such as any of the settings coming from the ``settings.xml``
270file. Also, you can add custom values that are included in the BitBake
271environment. The "pk" values must be unique. By convention, values that
272set default project values have a "DEFCONF" prefix.
273
274Defining BitBake Version
275~~~~~~~~~~~~~~~~~~~~~~~~
276
277The following defines which version of BitBake is used for the following
278release selection:
279
280.. code-block:: xml
281
282 <!-- Bitbake versions which correspond to the metadata release -->
283 <object model="orm.bitbakeversion" pk="1">
284 <field type="CharField" name="name">&DISTRO_NAME_NO_CAP;</field>
285 <field type="CharField" name="giturl">git://git.yoctoproject.org/poky</field>
286 <field type="CharField" name="branch">&DISTRO_NAME_NO_CAP;</field>
287 <field type="CharField" name="dirpath">bitbake</field>
288 </object>
289
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500290Defining Release
291~~~~~~~~~~~~~~~~
292
293The following defines the releases when you create a new project:
294
295.. code-block:: xml
296
297 <!-- Releases available -->
298 <object model="orm.release" pk="1">
299 <field type="CharField" name="name">&DISTRO_NAME_NO_CAP;</field>
300 <field type="CharField" name="description">Yocto Project &DISTRO; "&DISTRO_NAME;"</field>
301 <field rel="ManyToOneRel" to="orm.bitbakeversion" name="bitbake_version">1</field>
302 <field type="CharField" name="branch_name">&DISTRO_NAME_NO_CAP;</field>
303 <field type="TextField" name="helptext">Toaster will run your builds using the tip of the <a href="http://git.yoctoproject.org/cgit/cgit.cgi/poky/log/?h=&DISTRO_NAME_NO_CAP;">Yocto Project &DISTRO_NAME; branch</a>.</field>
304 </object>
305
306The "pk" value must match the above respective BitBake version record.
307
308Defining the Release Default Layer Names
309~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
310
311The following defines the default layers for each release:
312
313.. code-block:: xml
314
315 <!-- Default project layers for each release -->
316 <object model="orm.releasedefaultlayer" pk="1">
317 <field rel="ManyToOneRel" to="orm.release" name="release">1</field>
318 <field type="CharField" name="layer_name">openembedded-core</field>
319 </object>
320
321The 'pk' values in the example above should start at "1" and increment
322uniquely. You can use the same layer name in multiple releases.
323
324Defining Layer Definitions
325~~~~~~~~~~~~~~~~~~~~~~~~~~
326
327Layer definitions are the most complex. The following defines each of
328the layers, and then defines the exact layer version of the layer used
329for each respective release. You must have one ``orm.layer`` entry for
330each layer. Then, with each entry you need a set of
331``orm.layer_version`` entries that connects the layer with each release
332that includes the layer. In general all releases include the layer.
333
334.. code-block:: xml
335
336 <object model="orm.layer" pk="1">
337 <field type="CharField" name="name">openembedded-core</field>
338 <field type="CharField" name="layer_index_url"></field>
339 <field type="CharField" name="vcs_url">git://git.yoctoproject.org/poky</field>
340 <field type="CharField" name="vcs_web_url">http://git.yoctoproject.org/cgit/cgit.cgi/poky</field>
341 <field type="CharField" name="vcs_web_tree_base_url">http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/%path%?h=%branch%</field>
342 <field type="CharField" name="vcs_web_file_base_url">http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/%path%?h=%branch%</field>
343 </object>
344 <object model="orm.layer_version" pk="1">
345 <field rel="ManyToOneRel" to="orm.layer" name="layer">1</field>
346 <field type="IntegerField" name="layer_source">0</field>
347 <field rel="ManyToOneRel" to="orm.release" name="release">1</field>
348 <field type="CharField" name="branch">&DISTRO_NAME_NO_CAP;</field>
349 <field type="CharField" name="dirpath">meta</field>
350 </object> <object model="orm.layer_version" pk="2">
351 <field rel="ManyToOneRel" to="orm.layer" name="layer">1</field>
352 <field type="IntegerField" name="layer_source">0</field>
353 <field rel="ManyToOneRel" to="orm.release" name="release">2</field>
354 <field type="CharField" name="branch">HEAD</field>
355 <field type="CharField" name="commit">HEAD</field>
356 <field type="CharField" name="dirpath">meta</field>
357 </object>
358 <object model="orm.layer_version" pk="3">
359 <field rel="ManyToOneRel" to="orm.layer" name="layer">1</field>
360 <field type="IntegerField" name="layer_source">0</field>
361 <field rel="ManyToOneRel" to="orm.release" name="release">3</field>
362 <field type="CharField" name="branch">master</field>
363 <field type="CharField" name="dirpath">meta</field>
364 </object>
365
366The layer "pk" values above must be unique, and typically start at "1". The
367layer version "pk" values must also be unique across all layers, and typically
368start at "1".
369
370Remote Toaster Monitoring
371=========================
372
373Toaster has an API that allows remote management applications to
374directly query the state of the Toaster server and its builds in a
375machine-to-machine manner. This API uses the
376`REST <http://en.wikipedia.org/wiki/Representational_state_transfer>`__
377interface and the transfer of JSON files. For example, you might monitor
378a build inside a container through well supported known HTTP ports in
379order to easily access a Toaster server inside the container. In this
380example, when you use this direct JSON API, you avoid having web page
381parsing against the display the user sees.
382
383Checking Health
384---------------
385
386Before you use remote Toaster monitoring, you should do a health check.
387To do this, ping the Toaster server using the following call to see if
388it is still alive::
389
390 http://host:port/health
391
392Be sure to provide values for host and port. If the server is alive, you will
393get the response HTML:
394
395.. code-block:: html
396
397 <!DOCTYPE html>
398 <html lang="en">
399 <head><title>Toaster Health</title></head>
400 <body>Ok</body>
401 </html>
402
403Determining Status of Builds in Progress
404----------------------------------------
405
406Sometimes it is useful to determine the status of a build in progress.
407To get the status of pending builds, use the following call::
408
409 http://host:port/toastergui/api/building
410
411Be sure to provide values for host and port. The output is a JSON file that
412itemizes all builds in progress. This file includes the time in seconds since
413each respective build started as well as the progress of the cloning, parsing,
414and task execution. The following is sample output for a build in progress:
415
416.. code-block:: JSON
417
418 {"count": 1,
419 "building": [
420 {"machine": "beaglebone",
421 "seconds": "463.869",
422 "task": "927:2384",
423 "distro": "poky",
424 "clone": "1:1",
425 "id": 2,
426 "start": "2017-09-22T09:31:44.887Z",
427 "name": "20170922093200",
428 "parse": "818:818",
429 "project": "my_rocko",
430 "target": "core-image-minimal"
431 }]
432 }
433
434The JSON data for this query is returned in a
435single line. In the previous example the line has been artificially
436split for readability.
437
438Checking Status of Builds Completed
439-----------------------------------
440
441Once a build is completed, you get the status when you use the following
442call::
443
444 http://host:port/toastergui/api/builds
445
446Be sure to provide values for host and port. The output is a JSON file that
447itemizes all complete builds, and includes build summary information. The
448following is sample output for a completed build:
449
450.. code-block:: JSON
451
452 {"count": 1,
453 "builds": [
454 {"distro": "poky",
455 "errors": 0,
456 "machine": "beaglebone",
457 "project": "my_rocko",
458 "stop": "2017-09-22T09:26:36.017Z",
459 "target": "quilt-native",
460 "seconds": "78.193",
461 "outcome": "Succeeded",
462 "id": 1,
463 "start": "2017-09-22T09:25:17.824Z",
464 "warnings": 1,
465 "name": "20170922092618"
466 }]
467 }
468
469The JSON data for this query is returned in a single line. In the
470previous example the line has been artificially split for readability.
471
472Determining Status of a Specific Build
473--------------------------------------
474
475Sometimes it is useful to determine the status of a specific build. To
476get the status of a specific build, use the following call::
477
478 http://host:port/toastergui/api/build/ID
479
480Be sure to provide values for
481host, port, and ID. You can find the value for ID from the Builds
482Completed query. See the ":ref:`toaster-manual/toaster-manual-reference:checking status of builds completed`"
483section for more information.
484
485The output is a JSON file that itemizes the specific build and includes
486build summary information. The following is sample output for a specific
487build:
488
489.. code-block:: JSON
490
491 {"build":
492 {"distro": "poky",
493 "errors": 0,
494 "machine": "beaglebone",
495 "project": "my_rocko",
496 "stop": "2017-09-22T09:26:36.017Z",
497 "target": "quilt-native",
498 "seconds": "78.193",
499 "outcome": "Succeeded",
500 "id": 1,
501 "start": "2017-09-22T09:25:17.824Z",
502 "warnings": 1,
503 "name": "20170922092618",
504 "cooker_log": "/opt/user/poky/build-toaster-2/tmp/log/cooker/beaglebone/build_20170922_022607.991.log"
505 }
506 }
507
508The JSON data for this query is returned in a single line. In the
509previous example the line has been artificially split for readability.
510
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500511Useful Commands
512===============
513
514In addition to the web user interface and the scripts that start and
515stop Toaster, command-line commands exist through the ``manage.py``
516management script. You can find general documentation on ``manage.py``
517at the
518`Django <https://docs.djangoproject.com/en/2.2/topics/settings/>`__
519site. However, several ``manage.py`` commands have been created that are
520specific to Toaster and are used to control configuration and back-end
521tasks. You can locate these commands in the
522:term:`Source Directory` (e.g. ``poky``) at
523``bitbake/lib/manage.py``. This section documents those commands.
524
525.. note::
526
527 - When using ``manage.py`` commands given a default configuration,
528 you must be sure that your working directory is set to the
529 :term:`Build Directory`. Using
530 ``manage.py`` commands from the Build Directory allows Toaster to
531 find the ``toaster.sqlite`` file, which is located in the Build
532 Directory.
533
534 - For non-default database configurations, it is possible that you
535 can use ``manage.py`` commands from a directory other than the
536 Build Directory. To do so, the ``toastermain/settings.py`` file
537 must be configured to point to the correct database backend.
538
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500539``buildslist``
540--------------
541
542The ``buildslist`` command lists all builds that Toaster has recorded.
543Access the command as follows:
544
545.. code-block:: shell
546
547 $ bitbake/lib/toaster/manage.py buildslist
548
549The command returns a list, which includes numeric
550identifications, of the builds that Toaster has recorded in the current
551database.
552
553You need to run the ``buildslist`` command first to identify existing
554builds in the database before using the
555:ref:`toaster-manual/toaster-manual-reference:\`\`builddelete\`\`` command. Here is an
556example that assumes default repository and build directory names:
557
558.. code-block:: shell
559
560 $ cd ~/poky/build
561 $ python ../bitbake/lib/toaster/manage.py buildslist
562
563If your Toaster database had only one build, the above
564:ref:`toaster-manual/toaster-manual-reference:\`\`buildslist\`\``
565command would return something like the following::
566
567 1: qemux86 poky core-image-minimal
568
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500569``builddelete``
570---------------
571
572The ``builddelete`` command deletes data associated with a build. Access
573the command as follows:
574
575.. code-block::
576
577 $ bitbake/lib/toaster/manage.py builddelete build_id
578
579The command deletes all the build data for the specified
580build_id. This command is useful for removing old and unused data from
581the database.
582
583Prior to running the ``builddelete`` command, you need to get the ID
584associated with builds by using the
585:ref:`toaster-manual/toaster-manual-reference:\`\`buildslist\`\`` command.
586
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500587``perf``
588--------
589
590The ``perf`` command measures Toaster performance. Access the command as
591follows:
592
593.. code-block:: shell
594
595 $ bitbake/lib/toaster/manage.py perf
596
597The command is a sanity check that returns page loading times in order to
598identify performance problems.
599
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500600``checksettings``
601-----------------
602
603The ``checksettings`` command verifies existing Toaster settings. Access
604the command as follows:
605
606.. code-block:: shell
607
608 $ bitbake/lib/toaster/manage.py checksettings
609
610Toaster uses settings that are based on the database to configure the
611building tasks. The ``checksettings`` command verifies that the database
612settings are valid in the sense that they have the minimal information
613needed to start a build.
614
615In order for the ``checksettings`` command to work, the database must be
616correctly set up and not have existing data. To be sure the database is
617ready, you can run the following:
618
619.. code-block:: shell
620
621 $ bitbake/lib/toaster/manage.py syncdb
622 $ bitbake/lib/toaster/manage.py migrate orm
623 $ bitbake/lib/toaster/manage.py migrate bldcontrol
624
625After running these commands, you can run the ``checksettings`` command.
626
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500627``runbuilds``
628-------------
629
630The ``runbuilds`` command launches scheduled builds. Access the command
631as follows:
632
633.. code-block:: shell
634
635 $ bitbake/lib/toaster/manage.py runbuilds
636
637The ``runbuilds`` command checks if scheduled builds exist in the database
638and then launches them per schedule. The command returns after the builds
639start but before they complete. The Toaster Logging Interface records and
640updates the database when the builds complete.