blob: b0817b091ba3a8848486b69dfc278da01197eb01 [file] [log] [blame]
Andrew Geisslerf0343792020-11-18 10:42:21 -06001.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
Andrew Geisslerc9f78652020-09-18 14:11:35 -05002
3***********************************
4Project Testing and Release Process
5***********************************
6
7.. _test-daily-devel:
8
9Day to Day Development
10======================
11
12This section details how the project tests changes, through automation
13on the Autobuilder or with the assistance of QA teams, through to making
14releases.
15
16The project aims to test changes against our test matrix before those
17changes are merged into the master branch. As such, changes are queued
18up in batches either in the ``master-next`` branch in the main trees, or
19in user trees such as ``ross/mut`` in ``poky-contrib`` (Ross Burton
20helps review and test patches and this is his testing tree).
21
22We have two broad categories of test builds, including "full" and
23"quick". On the Autobuilder, these can be seen as "a-quick" and
24"a-full", simply for ease of sorting in the UI. Use our Autobuilder
25console view to see where me manage most test-related items, available
26at: :yocto_ab:`/typhoon/#/console`.
27
28Builds are triggered manually when the test branches are ready. The
29builds are monitored by the SWAT team. For additional information, see
30:yocto_wiki:`/wiki/Yocto_Build_Failure_Swat_Team`.
31If successful, the changes would usually be merged to the ``master``
32branch. If not successful, someone would respond to the changes on the
33mailing list explaining that there was a failure in testing. The choice
34of quick or full would depend on the type of changes and the speed with
35which the result was required.
36
37The Autobuilder does build the ``master`` branch once daily for several
38reasons, in particular, to ensure the current ``master`` branch does
39build, but also to keep ``yocto-testresults``
40(:yocto_git:`/cgit.cgi/yocto-testresults/`),
41buildhistory
42(:yocto_git:`/cgit.cgi/poky-buildhistory/`), and
43our sstate up to date. On the weekend, there is a master-next build
44instead to ensure the test results are updated for the less frequently
45run targets.
46
47Performance builds (buildperf-\* targets in the console) are triggered
48separately every six hours and automatically push their results to the
49buildstats repository at:
50:yocto_git:`/cgit.cgi/yocto-buildstats/`.
51
52The 'quick' targets have been selected to be the ones which catch the
53most failures or give the most valuable data. We run 'fast' ptests in
54this case for example but not the ones which take a long time. The quick
55target doesn't include \*-lsb builds for all architectures, some world
56builds and doesn't trigger performance tests or ltp testing. The full
57build includes all these things and is slower but more comprehensive.
58
59Release Builds
60==============
61
62The project typically has two major releases a year with a six month
63cadence in April and October. Between these there would be a number of
64milestone releases (usually four) with the final one being stablization
65only along with point releases of our stable branches.
66
67The build and release process for these project releases is similar to
68that in `Day to Day Development <#test-daily-devel>`__, in that the
69a-full target of the Autobuilder is used but in addition the form is
70configured to generate and publish artefacts and the milestone number,
71version, release candidate number and other information is entered. The
72box to "generate an email to QA"is also checked.
73
74When the build completes, an email is sent out using the send-qa-email
75script in the ``yocto-autobuilder-helper`` repository to the list of
76people configured for that release. Release builds are placed into a
77directory in https://autobuilder.yocto.io/pub/releases on the
78Autobuilder which is included in the email. The process from here is
79more manual and control is effectively passed to release engineering.
80The next steps include:
81
82- QA teams respond to the email saying which tests they plan to run and
83 when the results will be available.
84
85- QA teams run their tests and share their results in the yocto-
86 testresults-contrib repository, along with a summary of their
87 findings.
88
89- Release engineering prepare the release as per their process.
90
91- Test results from the QA teams are included into the release in
92 separate directories and also uploaded to the yocto-testresults
93 repository alongside the other test results for the given revision.
94
95- The QA report in the final release is regenerated using resulttool to
96 include the new test results and the test summaries from the teams
97 (as headers to the generated report).
98
99- The release is checked against the release checklist and release
100 readiness criteria.
101
102- A final decision on whether to release is made by the YP TSC who have
103 final oversight on release readiness.