blob: 3acd7c403ec0944dfb7c81e6c9969604ec161fc7 [file] [log] [blame]
Patrick Williamsc124f4f2015-09-15 14:41:29 -05001<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
2"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">
3
4<chapter>
5<title>File Download Support</title>
6
7 <para>
8 BitBake's fetch module is a standalone piece of library code
9 that deals with the intricacies of downloading source code
10 and files from remote systems.
11 Fetching source code is one of the cornerstones of building software.
12 As such, this module forms an important part of BitBake.
13 </para>
14
15 <para>
16 The current fetch module is called "fetch2" and refers to the
17 fact that it is the second major version of the API.
18 The original version is obsolete and has been removed from the codebase.
19 Thus, in all cases, "fetch" refers to "fetch2" in this
20 manual.
21 </para>
22
23 <section id='the-download-fetch'>
24 <title>The Download (Fetch)</title>
25
26 <para>
27 BitBake takes several steps when fetching source code or files.
28 The fetcher codebase deals with two distinct processes in order:
29 obtaining the files from somewhere (cached or otherwise)
30 and then unpacking those files into a specific location and
31 perhaps in a specific way.
32 Getting and unpacking the files is often optionally followed
33 by patching.
34 Patching, however, is not covered by this module.
35 </para>
36
37 <para>
38 The code to execute the first part of this process, a fetch,
39 looks something like the following:
40 <literallayout class='monospaced'>
Brad Bishop6e60e8b2018-02-01 10:27:11 -050041 src_uri = (d.getVar('SRC_URI') or "").split()
Patrick Williamsc124f4f2015-09-15 14:41:29 -050042 fetcher = bb.fetch2.Fetch(src_uri, d)
43 fetcher.download()
44 </literallayout>
45 This code sets up an instance of the fetch class.
46 The instance uses a space-separated list of URLs from the
Brad Bishop19323692019-04-05 15:28:33 -040047 <link linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link>
Patrick Williamsc124f4f2015-09-15 14:41:29 -050048 variable and then calls the <filename>download</filename>
49 method to download the files.
50 </para>
51
52 <para>
53 The instantiation of the fetch class is usually followed by:
54 <literallayout class='monospaced'>
Brad Bishop6e60e8b2018-02-01 10:27:11 -050055 rootdir = l.getVar('WORKDIR')
Patrick Williamsc124f4f2015-09-15 14:41:29 -050056 fetcher.unpack(rootdir)
57 </literallayout>
58 This code unpacks the downloaded files to the
59 specified by <filename>WORKDIR</filename>.
60 <note>
61 For convenience, the naming in these examples matches
62 the variables used by OpenEmbedded.
63 If you want to see the above code in action, examine
64 the OpenEmbedded class file <filename>base.bbclass</filename>.
65 </note>
66 The <filename>SRC_URI</filename> and <filename>WORKDIR</filename>
67 variables are not hardcoded into the fetcher, since those fetcher
68 methods can be (and are) called with different variable names.
69 In OpenEmbedded for example, the shared state (sstate) code uses
70 the fetch module to fetch the sstate files.
71 </para>
72
73 <para>
74 When the <filename>download()</filename> method is called,
75 BitBake tries to resolve the URLs by looking for source files
76 in a specific search order:
77 <itemizedlist>
78 <listitem><para><emphasis>Pre-mirror Sites:</emphasis>
79 BitBake first uses pre-mirrors to try and find source files.
80 These locations are defined using the
Brad Bishop19323692019-04-05 15:28:33 -040081 <link linkend='var-bb-PREMIRRORS'><filename>PREMIRRORS</filename></link>
Patrick Williamsc124f4f2015-09-15 14:41:29 -050082 variable.
83 </para></listitem>
84 <listitem><para><emphasis>Source URI:</emphasis>
85 If pre-mirrors fail, BitBake uses the original URL (e.g from
86 <filename>SRC_URI</filename>).
87 </para></listitem>
88 <listitem><para><emphasis>Mirror Sites:</emphasis>
89 If fetch failures occur, BitBake next uses mirror locations as
90 defined by the
Brad Bishop19323692019-04-05 15:28:33 -040091 <link linkend='var-bb-MIRRORS'><filename>MIRRORS</filename></link>
Patrick Williamsc124f4f2015-09-15 14:41:29 -050092 variable.
93 </para></listitem>
94 </itemizedlist>
95 </para>
96
97 <para>
98 For each URL passed to the fetcher, the fetcher
99 calls the submodule that handles that particular URL type.
100 This behavior can be the source of some confusion when you
101 are providing URLs for the <filename>SRC_URI</filename>
102 variable.
103 Consider the following two URLs:
104 <literallayout class='monospaced'>
105 http://git.yoctoproject.org/git/poky;protocol=git
106 git://git.yoctoproject.org/git/poky;protocol=http
107 </literallayout>
108 In the former case, the URL is passed to the
109 <filename>wget</filename> fetcher, which does not
110 understand "git".
111 Therefore, the latter case is the correct form since the
112 Git fetcher does know how to use HTTP as a transport.
113 </para>
114
115 <para>
116 Here are some examples that show commonly used mirror
117 definitions:
118 <literallayout class='monospaced'>
119 PREMIRRORS ?= "\
120 bzr://.*/.* http://somemirror.org/sources/ \n \
121 cvs://.*/.* http://somemirror.org/sources/ \n \
122 git://.*/.* http://somemirror.org/sources/ \n \
123 hg://.*/.* http://somemirror.org/sources/ \n \
124 osc://.*/.* http://somemirror.org/sources/ \n \
125 p4://.*/.* http://somemirror.org/sources/ \n \
126 svn://.*/.* http://somemirror.org/sources/ \n"
127
128 MIRRORS =+ "\
129 ftp://.*/.* http://somemirror.org/sources/ \n \
130 http://.*/.* http://somemirror.org/sources/ \n \
131 https://.*/.* http://somemirror.org/sources/ \n"
132 </literallayout>
133 It is useful to note that BitBake supports
134 cross-URLs.
135 It is possible to mirror a Git repository on an HTTP
136 server as a tarball.
137 This is what the <filename>git://</filename> mapping in
138 the previous example does.
139 </para>
140
141 <para>
142 Since network accesses are slow, Bitbake maintains a
143 cache of files downloaded from the network.
144 Any source files that are not local (i.e.
145 downloaded from the Internet) are placed into the download
146 directory, which is specified by the
Brad Bishop19323692019-04-05 15:28:33 -0400147 <link linkend='var-bb-DL_DIR'><filename>DL_DIR</filename></link>
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500148 variable.
149 </para>
150
151 <para>
152 File integrity is of key importance for reproducing builds.
153 For non-local archive downloads, the fetcher code can verify
154 SHA-256 and MD5 checksums to ensure the archives have been
155 downloaded correctly.
156 You can specify these checksums by using the
157 <filename>SRC_URI</filename> variable with the appropriate
158 varflags as follows:
159 <literallayout class='monospaced'>
160 SRC_URI[md5sum] = "<replaceable>value</replaceable>"
161 SRC_URI[sha256sum] = "<replaceable>value</replaceable>"
162 </literallayout>
163 You can also specify the checksums as parameters on the
164 <filename>SRC_URI</filename> as shown below:
165 <literallayout class='monospaced'>
166 SRC_URI = "http://example.com/foobar.tar.bz2;md5sum=4a8e0f237e961fd7785d19d07fdb994d"
167 </literallayout>
168 If multiple URIs exist, you can specify the checksums either
169 directly as in the previous example, or you can name the URLs.
170 The following syntax shows how you name the URIs:
171 <literallayout class='monospaced'>
172 SRC_URI = "http://example.com/foobar.tar.bz2;name=foo"
173 SRC_URI[foo.md5sum] = 4a8e0f237e961fd7785d19d07fdb994d
174 </literallayout>
175 After a file has been downloaded and has had its checksum checked,
176 a ".done" stamp is placed in <filename>DL_DIR</filename>.
177 BitBake uses this stamp during subsequent builds to avoid
178 downloading or comparing a checksum for the file again.
179 <note>
180 It is assumed that local storage is safe from data corruption.
181 If this were not the case, there would be bigger issues to worry about.
182 </note>
183 </para>
184
185 <para>
186 If
Brad Bishop19323692019-04-05 15:28:33 -0400187 <link linkend='var-bb-BB_STRICT_CHECKSUM'><filename>BB_STRICT_CHECKSUM</filename></link>
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500188 is set, any download without a checksum triggers an
189 error message.
190 The
Brad Bishop19323692019-04-05 15:28:33 -0400191 <link linkend='var-bb-BB_NO_NETWORK'><filename>BB_NO_NETWORK</filename></link>
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500192 variable can be used to make any attempted network access a fatal
193 error, which is useful for checking that mirrors are complete
194 as well as other things.
195 </para>
196 </section>
197
198 <section id='bb-the-unpack'>
199 <title>The Unpack</title>
200
201 <para>
202 The unpack process usually immediately follows the download.
203 For all URLs except Git URLs, BitBake uses the common
204 <filename>unpack</filename> method.
205 </para>
206
207 <para>
208 A number of parameters exist that you can specify within the
209 URL to govern the behavior of the unpack stage:
210 <itemizedlist>
211 <listitem><para><emphasis>unpack:</emphasis>
212 Controls whether the URL components are unpacked.
213 If set to "1", which is the default, the components
214 are unpacked.
215 If set to "0", the unpack stage leaves the file alone.
216 This parameter is useful when you want an archive to be
217 copied in and not be unpacked.
218 </para></listitem>
219 <listitem><para><emphasis>dos:</emphasis>
220 Applies to <filename>.zip</filename> and
221 <filename>.jar</filename> files and specifies whether to
222 use DOS line ending conversion on text files.
223 </para></listitem>
224 <listitem><para><emphasis>basepath:</emphasis>
225 Instructs the unpack stage to strip the specified
226 directories from the source path when unpacking.
227 </para></listitem>
228 <listitem><para><emphasis>subdir:</emphasis>
229 Unpacks the specific URL to the specified subdirectory
230 within the root directory.
231 </para></listitem>
232 </itemizedlist>
233 The unpack call automatically decompresses and extracts files
234 with ".Z", ".z", ".gz", ".xz", ".zip", ".jar", ".ipk", ".rpm".
235 ".srpm", ".deb" and ".bz2" extensions as well as various combinations
236 of tarball extensions.
237 </para>
238
239 <para>
240 As mentioned, the Git fetcher has its own unpack method that
241 is optimized to work with Git trees.
242 Basically, this method works by cloning the tree into the final
243 directory.
244 The process is completed using references so that there is
245 only one central copy of the Git metadata needed.
246 </para>
247 </section>
248
249 <section id='bb-fetchers'>
250 <title>Fetchers</title>
251
252 <para>
253 As mentioned earlier, the URL prefix determines which
254 fetcher submodule BitBake uses.
255 Each submodule can support different URL parameters,
256 which are described in the following sections.
257 </para>
258
259 <section id='local-file-fetcher'>
260 <title>Local file fetcher (<filename>file://</filename>)</title>
261
262 <para>
263 This submodule handles URLs that begin with
264 <filename>file://</filename>.
265 The filename you specify within the URL can be
266 either an absolute or relative path to a file.
267 If the filename is relative, the contents of the
Brad Bishop19323692019-04-05 15:28:33 -0400268 <link linkend='var-bb-FILESPATH'><filename>FILESPATH</filename></link>
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500269 variable is used in the same way
270 <filename>PATH</filename> is used to find executables.
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500271 If the file cannot be found, it is assumed that it is available in
Brad Bishop19323692019-04-05 15:28:33 -0400272 <link linkend='var-bb-DL_DIR'><filename>DL_DIR</filename></link>
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500273 by the time the <filename>download()</filename> method is called.
274 </para>
275
276 <para>
277 If you specify a directory, the entire directory is
278 unpacked.
279 </para>
280
281 <para>
282 Here are a couple of example URLs, the first relative and
283 the second absolute:
284 <literallayout class='monospaced'>
285 SRC_URI = "file://relativefile.patch"
286 SRC_URI = "file:///Users/ich/very_important_software"
287 </literallayout>
288 </para>
289 </section>
290
291 <section id='http-ftp-fetcher'>
292 <title>HTTP/FTP wget fetcher (<filename>http://</filename>, <filename>ftp://</filename>, <filename>https://</filename>)</title>
293
294 <para>
295 This fetcher obtains files from web and FTP servers.
296 Internally, the fetcher uses the wget utility.
297 </para>
298
299 <para>
300 The executable and parameters used are specified by the
301 <filename>FETCHCMD_wget</filename> variable, which defaults
302 to sensible values.
303 The fetcher supports a parameter "downloadfilename" that
304 allows the name of the downloaded file to be specified.
305 Specifying the name of the downloaded file is useful
306 for avoiding collisions in
Brad Bishop19323692019-04-05 15:28:33 -0400307 <link linkend='var-bb-DL_DIR'><filename>DL_DIR</filename></link>
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500308 when dealing with multiple files that have the same name.
309 </para>
310
311 <para>
312 Some example URLs are as follows:
313 <literallayout class='monospaced'>
314 SRC_URI = "http://oe.handhelds.org/not_there.aac"
315 SRC_URI = "ftp://oe.handhelds.org/not_there_as_well.aac"
316 SRC_URI = "ftp://you@oe.handhelds.org/home/you/secret.plan"
317 </literallayout>
318 </para>
319 <note>
320 Because URL parameters are delimited by semi-colons, this can
321 introduce ambiguity when parsing URLs that also contain semi-colons,
322 for example:
323 <literallayout class='monospaced'>
324 SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git;a=snapshot;h=a5dd47"
325 </literallayout>
326 Such URLs should should be modified by replacing semi-colons with '&amp;' characters:
327 <literallayout class='monospaced'>
328 SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&amp;a=snapshot&amp;h=a5dd47"
329 </literallayout>
330 In most cases this should work. Treating semi-colons and '&amp;' in queries
331 identically is recommended by the World Wide Web Consortium (W3C).
332 Note that due to the nature of the URL, you may have to specify the name
333 of the downloaded file as well:
334 <literallayout class='monospaced'>
335 SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&amp;a=snapshot&amp;h=a5dd47;downloadfilename=myfile.bz2"
336 </literallayout>
337 </note>
338 </section>
339
340 <section id='cvs-fetcher'>
341 <title>CVS fetcher (<filename>(cvs://</filename>)</title>
342
343 <para>
344 This submodule handles checking out files from the
345 CVS version control system.
346 You can configure it using a number of different variables:
347 <itemizedlist>
348 <listitem><para><emphasis><filename>FETCHCMD_cvs</filename>:</emphasis>
349 The name of the executable to use when running
350 the <filename>cvs</filename> command.
351 This name is usually "cvs".
352 </para></listitem>
353 <listitem><para><emphasis><filename>SRCDATE</filename>:</emphasis>
354 The date to use when fetching the CVS source code.
355 A special value of "now" causes the checkout to
356 be updated on every build.
357 </para></listitem>
Brad Bishop19323692019-04-05 15:28:33 -0400358 <listitem><para><emphasis><link linkend='var-bb-CVSDIR'><filename>CVSDIR</filename></link>:</emphasis>
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500359 Specifies where a temporary checkout is saved.
360 The location is often <filename>DL_DIR/cvs</filename>.
361 </para></listitem>
362 <listitem><para><emphasis><filename>CVS_PROXY_HOST</filename>:</emphasis>
363 The name to use as a "proxy=" parameter to the
364 <filename>cvs</filename> command.
365 </para></listitem>
366 <listitem><para><emphasis><filename>CVS_PROXY_PORT</filename>:</emphasis>
367 The port number to use as a "proxyport=" parameter to
368 the <filename>cvs</filename> command.
369 </para></listitem>
370 </itemizedlist>
371 As well as the standard username and password URL syntax,
372 you can also configure the fetcher with various URL parameters:
373 </para>
374
375 <para>
376 The supported parameters are as follows:
377 <itemizedlist>
378 <listitem><para><emphasis>"method":</emphasis>
Brad Bishop6e60e8b2018-02-01 10:27:11 -0500379 The protocol over which to communicate with the CVS
380 server.
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500381 By default, this protocol is "pserver".
382 If "method" is set to "ext", BitBake examines the
383 "rsh" parameter and sets <filename>CVS_RSH</filename>.
384 You can use "dir" for local directories.
385 </para></listitem>
386 <listitem><para><emphasis>"module":</emphasis>
387 Specifies the module to check out.
388 You must supply this parameter.
389 </para></listitem>
390 <listitem><para><emphasis>"tag":</emphasis>
391 Describes which CVS TAG should be used for
392 the checkout.
393 By default, the TAG is empty.
394 </para></listitem>
395 <listitem><para><emphasis>"date":</emphasis>
396 Specifies a date.
397 If no "date" is specified, the
Brad Bishop19323692019-04-05 15:28:33 -0400398 <link linkend='var-bb-SRCDATE'><filename>SRCDATE</filename></link>
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500399 of the configuration is used to checkout a specific date.
400 The special value of "now" causes the checkout to be
401 updated on every build.
402 </para></listitem>
403 <listitem><para><emphasis>"localdir":</emphasis>
404 Used to rename the module.
405 Effectively, you are renaming the output directory
406 to which the module is unpacked.
407 You are forcing the module into a special
408 directory relative to
Brad Bishop19323692019-04-05 15:28:33 -0400409 <link linkend='var-bb-CVSDIR'><filename>CVSDIR</filename></link>.
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500410 </para></listitem>
411 <listitem><para><emphasis>"rsh"</emphasis>
412 Used in conjunction with the "method" parameter.
413 </para></listitem>
414 <listitem><para><emphasis>"scmdata":</emphasis>
415 Causes the CVS metadata to be maintained in the tarball
416 the fetcher creates when set to "keep".
417 The tarball is expanded into the work directory.
418 By default, the CVS metadata is removed.
419 </para></listitem>
420 <listitem><para><emphasis>"fullpath":</emphasis>
421 Controls whether the resulting checkout is at the
422 module level, which is the default, or is at deeper
423 paths.
424 </para></listitem>
425 <listitem><para><emphasis>"norecurse":</emphasis>
426 Causes the fetcher to only checkout the specified
427 directory with no recurse into any subdirectories.
428 </para></listitem>
429 <listitem><para><emphasis>"port":</emphasis>
430 The port to which the CVS server connects.
431 </para></listitem>
432 </itemizedlist>
433 Some example URLs are as follows:
434 <literallayout class='monospaced'>
435 SRC_URI = "cvs://CVSROOT;module=mymodule;tag=some-version;method=ext"
436 SRC_URI = "cvs://CVSROOT;module=mymodule;date=20060126;localdir=usethat"
437 </literallayout>
438 </para>
439 </section>
440
441 <section id='svn-fetcher'>
442 <title>Subversion (SVN) Fetcher (<filename>svn://</filename>)</title>
443
444 <para>
445 This fetcher submodule fetches code from the
446 Subversion source control system.
447 The executable used is specified by
448 <filename>FETCHCMD_svn</filename>, which defaults
449 to "svn".
450 The fetcher's temporary working directory is set by
Brad Bishop19323692019-04-05 15:28:33 -0400451 <link linkend='var-bb-SVNDIR'><filename>SVNDIR</filename></link>,
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500452 which is usually <filename>DL_DIR/svn</filename>.
453 </para>
454
455 <para>
456 The supported parameters are as follows:
457 <itemizedlist>
458 <listitem><para><emphasis>"module":</emphasis>
459 The name of the svn module to checkout.
460 You must provide this parameter.
461 You can think of this parameter as the top-level
462 directory of the repository data you want.
463 </para></listitem>
Brad Bishop6e60e8b2018-02-01 10:27:11 -0500464 <listitem><para><emphasis>"path_spec":</emphasis>
465 A specific directory in which to checkout the
466 specified svn module.
467 </para></listitem>
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500468 <listitem><para><emphasis>"protocol":</emphasis>
469 The protocol to use, which defaults to "svn".
Brad Bishop6e60e8b2018-02-01 10:27:11 -0500470 If "protocol" is set to "svn+ssh", the "ssh"
471 parameter is also used.
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500472 </para></listitem>
473 <listitem><para><emphasis>"rev":</emphasis>
474 The revision of the source code to checkout.
475 </para></listitem>
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500476 <listitem><para><emphasis>"scmdata":</emphasis>
477 Causes the “.svn” directories to be available during
478 compile-time when set to "keep".
479 By default, these directories are removed.
480 </para></listitem>
Brad Bishop6e60e8b2018-02-01 10:27:11 -0500481 <listitem><para><emphasis>"ssh":</emphasis>
482 An optional parameter used when "protocol" is set
483 to "svn+ssh".
484 You can use this parameter to specify the ssh
485 program used by svn.
486 </para></listitem>
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500487 <listitem><para><emphasis>"transportuser":</emphasis>
488 When required, sets the username for the transport.
489 By default, this parameter is empty.
490 The transport username is different than the username
491 used in the main URL, which is passed to the subversion
492 command.
493 </para></listitem>
494 </itemizedlist>
Brad Bishop6e60e8b2018-02-01 10:27:11 -0500495 Following are three examples using svn:
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500496 <literallayout class='monospaced'>
Brad Bishop6e60e8b2018-02-01 10:27:11 -0500497 SRC_URI = "svn://myrepos/proj1;module=vip;protocol=http;rev=667"
498 SRC_URI = "svn://myrepos/proj1;module=opie;protocol=svn+ssh"
499 SRC_URI = "svn://myrepos/proj1;module=trunk;protocol=http;path_spec=${MY_DIR}/proj1"
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500500 </literallayout>
501 </para>
502 </section>
503
504 <section id='git-fetcher'>
505 <title>Git Fetcher (<filename>git://</filename>)</title>
506
507 <para>
508 This fetcher submodule fetches code from the Git
509 source control system.
510 The fetcher works by creating a bare clone of the
511 remote into
Brad Bishop19323692019-04-05 15:28:33 -0400512 <link linkend='var-bb-GITDIR'><filename>GITDIR</filename></link>,
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500513 which is usually <filename>DL_DIR/git2</filename>.
514 This bare clone is then cloned into the work directory during the
515 unpack stage when a specific tree is checked out.
516 This is done using alternates and by reference to
517 minimize the amount of duplicate data on the disk and
518 make the unpack process fast.
519 The executable used can be set with
520 <filename>FETCHCMD_git</filename>.
521 </para>
522
523 <para>
524 This fetcher supports the following parameters:
525 <itemizedlist>
526 <listitem><para><emphasis>"protocol":</emphasis>
527 The protocol used to fetch the files.
528 The default is "git" when a hostname is set.
529 If a hostname is not set, the Git protocol is "file".
530 You can also use "http", "https", "ssh" and "rsync".
531 </para></listitem>
532 <listitem><para><emphasis>"nocheckout":</emphasis>
533 Tells the fetcher to not checkout source code when
534 unpacking when set to "1".
535 Set this option for the URL where there is a custom
536 routine to checkout code.
537 The default is "0".
538 </para></listitem>
539 <listitem><para><emphasis>"rebaseable":</emphasis>
540 Indicates that the upstream Git repository can be rebased.
541 You should set this parameter to "1" if
542 revisions can become detached from branches.
543 In this case, the source mirror tarball is done per
544 revision, which has a loss of efficiency.
545 Rebasing the upstream Git repository could cause the
546 current revision to disappear from the upstream repository.
547 This option reminds the fetcher to preserve the local cache
548 carefully for future use.
549 The default value for this parameter is "0".
550 </para></listitem>
551 <listitem><para><emphasis>"nobranch":</emphasis>
552 Tells the fetcher to not check the SHA validation
553 for the branch when set to "1".
554 The default is "0".
555 Set this option for the recipe that refers to
556 the commit that is valid for a tag instead of
557 the branch.
558 </para></listitem>
559 <listitem><para><emphasis>"bareclone":</emphasis>
560 Tells the fetcher to clone a bare clone into the
561 destination directory without checking out a working tree.
562 Only the raw Git metadata is provided.
563 This parameter implies the "nocheckout" parameter as well.
564 </para></listitem>
565 <listitem><para><emphasis>"branch":</emphasis>
566 The branch(es) of the Git tree to clone.
567 If unset, this is assumed to be "master".
568 The number of branch parameters much match the number of
569 name parameters.
570 </para></listitem>
571 <listitem><para><emphasis>"rev":</emphasis>
572 The revision to use for the checkout.
573 The default is "master".
574 </para></listitem>
575 <listitem><para><emphasis>"tag":</emphasis>
576 Specifies a tag to use for the checkout.
577 To correctly resolve tags, BitBake must access the
578 network.
579 For that reason, tags are often not used.
580 As far as Git is concerned, the "tag" parameter behaves
581 effectively the same as the "rev" parameter.
582 </para></listitem>
583 <listitem><para><emphasis>"subpath":</emphasis>
584 Limits the checkout to a specific subpath of the tree.
585 By default, the whole tree is checked out.
586 </para></listitem>
587 <listitem><para><emphasis>"destsuffix":</emphasis>
588 The name of the path in which to place the checkout.
589 By default, the path is <filename>git/</filename>.
590 </para></listitem>
Yong, Jonathan068291f2018-12-10 10:46:03 -0800591 <listitem><para><emphasis>"usehead":</emphasis>
592 Enables local <filename>git://</filename> URLs to use the
593 current branch HEAD as the revision for use with
594 <filename>AUTOREV</filename>.
595 The "usehead" parameter implies no branch and only works
596 when the transfer protocol is
597 <filename>file://</filename>.
598 </para></listitem>
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500599 </itemizedlist>
600 Here are some example URLs:
601 <literallayout class='monospaced'>
602 SRC_URI = "git://git.oe.handhelds.org/git/vip.git;tag=version-1"
603 SRC_URI = "git://git.oe.handhelds.org/git/vip.git;protocol=http"
604 </literallayout>
605 </para>
606 </section>
607
608 <section id='gitsm-fetcher'>
609 <title>Git Submodule Fetcher (<filename>gitsm://</filename>)</title>
610
611 <para>
612 This fetcher submodule inherits from the
613 <link linkend='git-fetcher'>Git fetcher</link> and extends
614 that fetcher's behavior by fetching a repository's submodules.
Brad Bishop19323692019-04-05 15:28:33 -0400615 <link linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link>
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500616 is passed to the Git fetcher as described in the
617 "<link linkend='git-fetcher'>Git Fetcher (<filename>git://</filename>)</link>"
618 section.
619 <note>
620 <title>Notes and Warnings</title>
621 <para>
622 You must clean a recipe when switching between
623 '<filename>git://</filename>' and
624 '<filename>gitsm://</filename>' URLs.
625 </para>
626
627 <para>
628 The Git Submodules fetcher is not a complete fetcher
629 implementation.
630 The fetcher has known issues where it does not use the
Brad Bishopd7bf8c12018-02-25 22:55:05 -0500631 normal source mirroring infrastructure properly. Further,
632 the submodule sources it fetches are not visible to the
633 licensing and source archiving infrastructures.
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500634 </para>
635 </note>
636 </para>
637 </section>
638
639 <section id='clearcase-fetcher'>
640 <title>ClearCase Fetcher (<filename>ccrc://</filename>)</title>
641
642 <para>
643 This fetcher submodule fetches code from a
644 <ulink url='http://en.wikipedia.org/wiki/Rational_ClearCase'>ClearCase</ulink>
645 repository.
646 </para>
647
648 <para>
649 To use this fetcher, make sure your recipe has proper
Brad Bishop19323692019-04-05 15:28:33 -0400650 <link linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link>,
651 <link linkend='var-bb-SRCREV'><filename>SRCREV</filename></link>, and
652 <link linkend='var-bb-PV'><filename>PV</filename></link> settings.
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500653 Here is an example:
654 <literallayout class='monospaced'>
655 SRC_URI = "ccrc://cc.example.org/ccrc;vob=/example_vob;module=/example_module"
656 SRCREV = "EXAMPLE_CLEARCASE_TAG"
657 PV = "${@d.getVar("SRCREV", False).replace("/", "+")}"
658 </literallayout>
659 The fetcher uses the <filename>rcleartool</filename> or
660 <filename>cleartool</filename> remote client, depending on
661 which one is available.
662 </para>
663
664 <para>
665 Following are options for the <filename>SRC_URI</filename>
666 statement:
667 <itemizedlist>
668 <listitem><para><emphasis><filename>vob</filename></emphasis>:
669 The name, which must include the
670 prepending "/" character, of the ClearCase VOB.
671 This option is required.
672 </para></listitem>
673 <listitem><para><emphasis><filename>module</filename></emphasis>:
674 The module, which must include the
675 prepending "/" character, in the selected VOB.
676 <note>
677 The <filename>module</filename> and <filename>vob</filename>
678 options are combined to create the <filename>load</filename> rule in
679 the view config spec.
Patrick Williamsc0f7c042017-02-23 20:41:17 -0600680 As an example, consider the <filename>vob</filename> and
681 <filename>module</filename> values from the
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500682 <filename>SRC_URI</filename> statement at the start of this section.
683 Combining those values results in the following:
684 <literallayout class='monospaced'>
685 load /example_vob/example_module
686 </literallayout>
687 </note>
688 </para></listitem>
689 <listitem><para><emphasis><filename>proto</filename></emphasis>:
690 The protocol, which can be either <filename>http</filename> or
691 <filename>https</filename>.
692 </para></listitem>
693 </itemizedlist>
694 </para>
695
696 <para>
697 By default, the fetcher creates a configuration specification.
698 If you want this specification written to an area other than the default,
699 use the <filename>CCASE_CUSTOM_CONFIG_SPEC</filename> variable
700 in your recipe to define where the specification is written.
701 <note>
702 the <filename>SRCREV</filename> loses its functionality if you
703 specify this variable.
704 However, <filename>SRCREV</filename> is still used to label the
705 archive after a fetch even though it does not define what is
706 fetched.
707 </note>
708 </para>
709
710 <para>
711 Here are a couple of other behaviors worth mentioning:
712 <itemizedlist>
713 <listitem><para>
714 When using <filename>cleartool</filename>, the login of
715 <filename>cleartool</filename> is handled by the system.
716 The login require no special steps.
717 </para></listitem>
718 <listitem><para>
719 In order to use <filename>rcleartool</filename> with authenticated
720 users, an "rcleartool login" is necessary before using the fetcher.
721 </para></listitem>
722 </itemizedlist>
723 </para>
724 </section>
725
Patrick Williamsc0f7c042017-02-23 20:41:17 -0600726 <section id='perforce-fetcher'>
727 <title>Perforce Fetcher (<filename>p4://</filename>)</title>
728
729 <para>
730 This fetcher submodule fetches code from the
731 <ulink url='https://www.perforce.com/'>Perforce</ulink>
732 source control system.
733 The executable used is specified by
734 <filename>FETCHCMD_p4</filename>, which defaults
735 to "p4".
736 The fetcher's temporary working directory is set by
Brad Bishop19323692019-04-05 15:28:33 -0400737 <link linkend='var-bb-P4DIR'><filename>P4DIR</filename></link>,
Patrick Williamsc0f7c042017-02-23 20:41:17 -0600738 which defaults to "DL_DIR/p4".
739 </para>
740
741 <para>
742 To use this fetcher, make sure your recipe has proper
Brad Bishop19323692019-04-05 15:28:33 -0400743 <link linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link>,
744 <link linkend='var-bb-SRCREV'><filename>SRCREV</filename></link>, and
745 <link linkend='var-bb-PV'><filename>PV</filename></link> values.
Patrick Williamsc0f7c042017-02-23 20:41:17 -0600746 The p4 executable is able to use the config file defined by your
747 system's <filename>P4CONFIG</filename> environment variable in
748 order to define the Perforce server URL and port, username, and
749 password if you do not wish to keep those values in a recipe
750 itself.
751 If you choose not to use <filename>P4CONFIG</filename>,
752 or to explicitly set variables that <filename>P4CONFIG</filename>
753 can contain, you can specify the <filename>P4PORT</filename> value,
754 which is the server's URL and port number, and you can
755 specify a username and password directly in your recipe within
756 <filename>SRC_URI</filename>.
757 </para>
758
759 <para>
760 Here is an example that relies on <filename>P4CONFIG</filename>
761 to specify the server URL and port, username, and password, and
762 fetches the Head Revision:
763 <literallayout class='monospaced'>
764 SRC_URI = "p4://example-depot/main/source/..."
765 SRCREV = "${AUTOREV}"
766 PV = "p4-${SRCPV}"
767 S = "${WORKDIR}/p4"
768 </literallayout>
769 </para>
770
771 <para>
772 Here is an example that specifies the server URL and port,
773 username, and password, and fetches a Revision based on a Label:
774 <literallayout class='monospaced'>
775 P4PORT = "tcp:p4server.example.net:1666"
776 SRC_URI = "p4://user:passwd@example-depot/main/source/..."
777 SRCREV = "release-1.0"
778 PV = "p4-${SRCPV}"
779 S = "${WORKDIR}/p4"
780 </literallayout>
781 <note>
782 You should always set <filename>S</filename>
783 to <filename>"${WORKDIR}/p4"</filename> in your recipe.
784 </note>
785 </para>
786 </section>
787
Brad Bishop316dfdd2018-06-25 12:45:53 -0400788 <section id='repo-fetcher'>
789 <title>Repo Fetcher (<filename>repo://</filename>)</title>
790
791 <para>
792 This fetcher submodule fetches code from
793 <filename>google-repo</filename> source control system.
794 The fetcher works by initiating and syncing sources of the
795 repository into
Brad Bishop19323692019-04-05 15:28:33 -0400796 <link linkend='var-bb-REPODIR'><filename>REPODIR</filename></link>,
Brad Bishop316dfdd2018-06-25 12:45:53 -0400797 which is usually
Brad Bishop19323692019-04-05 15:28:33 -0400798 <link linkend='var-bb-DL_DIR'><filename>DL_DIR</filename></link><filename>/repo</filename>.
Brad Bishop316dfdd2018-06-25 12:45:53 -0400799 </para>
800
801 <para>
802 This fetcher supports the following parameters:
803 <itemizedlist>
804 <listitem><para>
805 <emphasis>"protocol":</emphasis>
806 Protocol to fetch the repository manifest (default: git).
807 </para></listitem>
808 <listitem><para>
809 <emphasis>"branch":</emphasis>
810 Branch or tag of repository to get (default: master).
811 </para></listitem>
812 <listitem><para>
813 <emphasis>"manifest":</emphasis>
814 Name of the manifest file (default: <filename>default.xml</filename>).
815 </para></listitem>
816 </itemizedlist>
817 Here are some example URLs:
818 <literallayout class='monospaced'>
819 SRC_URI = "repo://REPOROOT;protocol=git;branch=some_branch;manifest=my_manifest.xml"
820 SRC_URI = "repo://REPOROOT;protocol=file;branch=some_branch;manifest=my_manifest.xml"
821 </literallayout>
822 </para>
823 </section>
824
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500825 <section id='other-fetchers'>
826 <title>Other Fetchers</title>
827
828 <para>
829 Fetch submodules also exist for the following:
830 <itemizedlist>
831 <listitem><para>
832 Bazaar (<filename>bzr://</filename>)
833 </para></listitem>
834 <listitem><para>
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500835 Trees using Git Annex (<filename>gitannex://</filename>)
836 </para></listitem>
837 <listitem><para>
838 Secure FTP (<filename>sftp://</filename>)
839 </para></listitem>
840 <listitem><para>
841 Secure Shell (<filename>ssh://</filename>)
842 </para></listitem>
843 <listitem><para>
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500844 OSC (<filename>osc://</filename>)
845 </para></listitem>
846 <listitem><para>
847 Mercurial (<filename>hg://</filename>)
848 </para></listitem>
849 </itemizedlist>
850 No documentation currently exists for these lesser used
851 fetcher submodules.
852 However, you might find the code helpful and readable.
853 </para>
854 </section>
855 </section>
856
857 <section id='auto-revisions'>
858 <title>Auto Revisions</title>
859
860 <para>
861 We need to document <filename>AUTOREV</filename> and
862 <filename>SRCREV_FORMAT</filename> here.
863 </para>
864 </section>
865</chapter>