Blame - poky/documentation/test-manual/test-manual-understand-autobuilder.xml - stefanberger/openbmc

Andrew Geissler

4873add

2020-11-02 18:44:49 -0600

[diff] [blame]

1

<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"

2

"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"

3

[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >

4

<title>Understanding the Yocto Project Autobuilder</title>

9

10

<title>Execution Flow within the Autobuilder</title>

11

<para>The "a-full" and "a-quick" targets are the usual entry points into the Autobuilder and

12

it makes sense to follow the process through the system starting there. This is best

13

visualised from the Autobuilder Console view (<link linkend=""

14

>https://autobuilder.yoctoproject.org/typhoon/#/console</link>). </para>

15

<para>Each item along the top of that view represents some "target build" and these targets

16

are all run in parallel. The 'full' build will trigger the majority of them, the "quick"

17

build will trigger some subset of them. The Autobuilder effectively runs whichever

18

configuration is defined for each of those targets on a seperate buildbot worker. To

19

understand the configuration, you need to look at the entry on

20

<filename>config.json</filename> file within the

21

<filename>yocto-autobuilder-helper</filename> repository. The targets are defined in

22

the ‘overrides' section, a quick example could be qemux86-64 which looks

23

like:<literallayout class="monospaced">

24

"qemux86-64" : {

25

"MACHINE" : "qemux86-64",

26

"TEMPLATE" : "arch-qemu",

27

"step1" : {

28

"extravars" : [

29

"IMAGE_FSTYPES_append = ' wic wic.bmap'"

]

}

},

</literallayout>And

to expand that, you need the "arch-qemu" entry from the "templates" section, which looks

35

like:<literallayout class="monospaced">

36

"arch-qemu" : {

37

"BUILDINFO" : true,

38

"BUILDHISTORY" : true,

39

"step1" : {

40

"BBTARGETS" : "core-image-sato core-image-sato-dev core-image-sato-sdk core-image-minimal core-image-minimal-dev core-image-sato:do_populate_sdk",

41

"SANITYTARGETS" : "core-image-minimal:do_testimage core-image-sato:do_testimage core-image-sato-sdk:do_testimage core-image-sato:do_testsdk"

42

},

43

"step2" : {

44

"SDKMACHINE" : "x86_64",

45

"BBTARGETS" : "core-image-sato:do_populate_sdk core-image-minimal:do_populate_sdk_ext core-image-sato:do_populate_sdk_ext",

46

"SANITYTARGETS" : "core-image-sato:do_testsdk core-image-minimal:do_testsdkext core-image-sato:do_testsdkext"

47

},

48

"step3" : {

49

"BUILDHISTORY" : false,

50

"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest ${HELPERSTMACHTARGS} -j 15"],

51

"ADDLAYER" : ["${BUILDDIR}/../meta-selftest"]

52

}

53

},

54

</literallayout>Combining

55

these two entries you can see that "qemux86-64" is a three step build where the

56

<filename>bitbake BBTARGETS</filename> would be run, then <filename>bitbake

57

SANITYTARGETS</filename> for each step; all for

58

<filename>MACHINE="qemx86-64"</filename> but with differing SDKMACHINE settings. In

59

step 1 an extra variable is added to the <filename>auto.conf</filename> file to enable

60

wic image generation.</para>

61

<para>While not every detail of this is covered here, you can see how the templating

62

mechanism allows quite complex configurations to be built up yet allows duplication and

63

repetition to be kept to a minimum.</para>

64

<para>The different build targets are designed to allow for parallelisation, so different

65

machines are usually built in parallel, operations using the same machine and metadata

66

are built sequentially, with the aim of trying to optimise build efficiency as much as

67

possible.</para>

68

<para>The <filename>config.json</filename> file is processed by the scripts in the Helper

69

repository in the <filename>scripts</filename> directory. The following section details

70

how this works.</para>

</section>

<title>Autobuilder Target Execution Overview</title>

75

76

<para>For each given target in a build, the Autobuilder executes several steps. These are

77

configured in <filename>yocto-autobuilder2/builders.py</filename> and roughly consist

78

of: <orderedlist>

79

80

<para><emphasis>Run <filename>clobberdir</filename></emphasis></para>

81

<para>This cleans out any previous build. Old builds are left around to allow

82

easier debugging of failed builds. For additional information, see <link

83

linkend="test-clobberdir"><filename>clobberdir</filename></link>.</para>

84

</listitem>

85

86

<para><emphasis>Obtain yocto-autobuilder-helper</emphasis></para>

87

<para>This step clones the <filename>yocto-autobuilder-helper</filename> git

88

repository. This is necessary to prevent the requirement to maintain all the

89

release or project-specific code within Buildbot. The branch chosen matches

90

the release being built so we can support older releases and still make

91

changes in newer ones.</para>

92

</listitem>

93

94

<para><emphasis>Write layerinfo.json</emphasis></para>

95

<para>This transfers data in the Buildbot UI when the build was configured to

the Helper.</para>

</listitem>

<para><emphasis>Call scripts/shared-repo-unpack</emphasis></para>

100

<para>This is a call into the Helper scripts to set up a checkout of all the

101

pieces this build might need. It might clone the BitBake repository and the

102

OpenEmbedded-Core repository. It may clone the Poky repository, as well as

103

additional layers. It will use the data from the

104

<filename>layerinfo.json</filename> file to help understand the

105

configuration. It will also use a local cache of repositories to speed up

106

the clone checkouts. For additional information, see <link

107

linkend="test-autobuilder-clone-cache">Autobuilder Clone

108

Cache</link>.</para>

109

<para>This step has two possible modes of operation. If the build is part of a

110

parent build, its possible that all the repositories needed may already be

111

available, ready in a pre-prepared directory. An "a-quick" or "a-full" build

112

would prepare this before starting the other sub-target builds. This is done

113

for two reasons:<itemizedlist>

114

115

<para>the upstream may change during a build, for example, from a

116

forced push and this ensures we have matching content for the

whole build</para>

</listitem>

<para>if 15 Workers all tried to pull the same data from the same

121

repos, we can hit resource limits on upstream servers as they

122

can think they are under some kind of network attack</para>

123

</listitem>

124

</itemizedlist>This pre-prepared directory is shared among the Workers over

125

NFS. If the build is an individual build and there is no "shared" directory

126

available, it would clone from the cache and the upstreams as necessary.

127

This is considered the fallback mode.</para>

128

</listitem>

129

130

<para><emphasis>Call scripts/run-config</emphasis></para>

131

<para>This is another call into the Helper scripts where its expected that the

132

main functionality of this target will be executed.</para>

133

</listitem>

134

</orderedlist></para>

135

</section>

136

137

<title>Autobuilder Technology</title>

138

<para>The Autobuilder has Yocto Project-specific functionality to allow builds to operate

139

with increased efficiency and speed.</para>

140

141

<title>clobberdir</title>

142

<para>When deleting files, the Autobuilder uses <filename>clobberdir</filename>, which

143

is a special script that moves files to a special location, rather than deleting

144

them. Files in this location are deleted by an <filename>rm</filename> command,

145

which is run under <filename>ionice -c 3</filename>. For example, the deletion only

146

happens when there is idle IO capacity on the Worker. The Autobuilder Worker Janitor

147

runs this deletion. See <link linkend="test-autobuilder-worker-janitor">Autobuilder

148

Worker Janitor</link>.</para>

149

</section>

150

151

<title>Autobuilder Clone Cache</title>

152

<para>Cloning repositories from scratch each time they are required was slow on the

153

Autobuilder. We therefore have a stash of commonly used repositories pre-cloned on

154

the Workers. Data is fetched from these during clones first, then "topped up" with

155

later revisions from any upstream when necesary. The cache is maintained by the

156

Autobuilder Worker Janitor. See <link linkend="test-autobuilder-worker-janitor"

157

>Autobuilder Worker Janitor</link>.</para>

158

</section>

159

160

<title>Autobuilder Worker Janitor</title>

161

<para>This is a process running on each Worker that performs two basic operations,

162

including background file deletion at IO idle (see <link

163

linkend="test-list-tgt-exec-clobberdir">Target Execution: clobberdir</link>) and

164

maintainenance of a cache of cloned repositories to improve the speed the system can

165

checkout repositories.</para>

166

</section>

167

168

<title>Shared DL_DIR</title>

169

<para>The Workers are all connected over NFS which allows DL_DIR to be shared between

170

them. This reduces network accesses from the system and allows the build to be sped

171

up. Usage of the directory within the build system is designed to be able to be

172

shared over NFS.</para>

173

</section>

174

175

<title>Shared SSTATE_DIR</title>

176

<para>The Workers are all connected over NFS which allows the

177

<filename>sstate</filename> directory to be shared between them. This means once

178

a Worker has built an artefact, all the others can benefit from it. Usage of the

179

directory within the directory is designed for sharing over NFS.</para>

180

</section>

181

182

<title>Resulttool</title>

183

<para>All of the different tests run as part of the build generate output into

184

<filename>testresults.json</filename> files. This allows us to determine which

185

tests ran in a given build and their status. Additional information, such as failure

186

logs or the time taken to run the tests, may also be included.</para>

187

<para>Resulttool is part of OpenEmbedded-Core and is used to manipulate these json

188

results files. It has the ability to merge files together, display reports of the

189

test results and compare different result files.</para>

190

<para>For details, see <link linkend=""

191

>https://wiki.yoctoproject.org/wiki/Resulttool</link>.</para>

</section>

</section>

<title>run-config Target Execution</title>

196

<para>The <filename>scripts/run-config</filename> execution is where most of the work within

197

the Autobuilder happens. It runs through a number of steps; the first are general setup

198

steps that are run once and include:<orderedlist>

199

200

<para>Set up any <filename>buildtools-tarball</filename> if configured.</para>

201

</listitem>

202

203

<para>Call "buildhistory-init" if buildhistory is configured.</para>

204

</listitem>

205

</orderedlist></para>

206

<para>For each step that is configured in <filename>config.json</filename>, it will perform

207

the following:</para>

208

<para>

209

<remark>## WRITER's question: What does "logging in as stepXa" and others refer to

below? ##</remark>

<para dir="ltr">Add any layers that are specified using the

214

<filename>bitbake-layers add-layer</filename> command (logging as

stepXa)</para>

</listitem>

<para dir="ltr">Call the <filename>scripts/setup-config</filename> script to

219

generate the necessary <filename>auto.conf</filename> configuration file for

the build</para>

</listitem>

<para dir="ltr">Run the <filename>bitbake BBTARGETS</filename> command (logging

as stepXb)</para>

</listitem>

<para dir="ltr">Run the <filename>bitbake SANITYTARGETS</filename> command

228

(logging as stepXc)</para>

229

</listitem>

230

231

<para dir="ltr">Run the <filename>EXTRACMDS</filename> command, which are run

232

within the BitBake build environment (logging as stepXd)</para>

233

</listitem>

234

235

<para dir="ltr">Run the <filename>EXTRAPLAINCMDS</filename> command(s), which

236

are run outside the BitBake build environment (logging as stepXd)</para>

237

</listitem>

238

239

<para dir="ltr">Remove any layers added in <link

240

linkend="test-run-config-add-layers-step">step 1</link> using the

241

<filename>bitbake-layers remove-layer</filename> command (logging as

stepXa)</para>

</listitem>

</orderedlist>

</para>

<para>Once the execution steps above complete, <filename>run-config</filename> executes a

247

set of post-build steps, including:<orderedlist>

248

249

<para dir="ltr">Call <filename>scripts/publish-artifacts</filename> to collect

250

any output which is to be saved from the build.</para>

251

</listitem>

252

253

<para dir="ltr">Call <filename>scripts/collect-results</filename> to collect any

254

test results to be saved from the build.</para>

255

</listitem>

256

257

<para dir="ltr">Call <filename>scripts/upload-error-reports</filename> to send

258

any error reports generated to the remote server.</para>

259

</listitem>

260

261

<para dir="ltr">Cleanup the build directory using <link

262

linkend="test-clobberdir"><filename>clobberdir</filename></link> if the

263

build was successful, else rename it to "build-renamed" for potential future

264

debugging.</para>

265

</listitem>

266

</orderedlist></para>

267

</section>

268

269

<title>Deploying Yocto Autobuilder</title>

270

<para>The most up to date information about how to setup and deploy your own Autbuilder can

271

be found in README.md in the <filename>yocto-autobuilder2</filename> repository.</para>

272

<para>We hope that people can use the <filename>yocto-autobuilder2</filename> code directly

273

but it is inevitable that users will end up needing to heavily customise the

274

<filename>yocto-autobuilder-helper</filename> repository, particularly the

275

<filename>config.json</filename> file as they will want to define their own test

276

matrix.</para>

277

<para>The Autobuilder supports wo customization options: <itemizedlist>

278

279

<para>variable substitution</para>

280

</listitem>

281

282

<para>overlaying configuration files</para>

283

</listitem>

284

</itemizedlist>The standard <filename>config.json</filename> minimally attempts to allow

285

substitution of the paths. The Helper script repository includes a

286

<filename>local-example.json</filename> file to show how you could override these

287

from a separate configuration file. Pass the following into the environment of the

288

Autobuilder:<literallayout class="monospaced">

289

$ ABHELPER_JSON="config.json local-example.json"

290

</literallayout>As

291

another example, you could also pass the following into the

292

environment:<literallayout class="monospaced">

293

$ ABHELPER_JSON="config.json <replaceable>/some/location/</replaceable>local.json"

294

</literallayout>One

295

issue users often run into is validation of the <filename>config.json</filename> files.

296

A tip for minimizing issues from invalid json files is to use a Git

297

<filename>pre-commit-hook.sh</filename> script to verify the JSON file before

298

committing it. Create a symbolic link as

299

follows:<literallayout class="monospaced">

300

$ ln -s ../../scripts/pre-commit-hook.sh .git/hooks/pre-commit

301

</literallayout></para>

</section>

</chapter>

<!--

vim: expandtab tw=80 ts=4

314

-->