blob: 2a3340b39983bf28f5147c3ce8544b57bab0ef1c [file] [log] [blame]
Patrick Williamsc124f4f2015-09-15 14:41:29 -05001<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
2"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">
3
4<chapter>
5<title>File Download Support</title>
6
7 <para>
8 BitBake's fetch module is a standalone piece of library code
9 that deals with the intricacies of downloading source code
10 and files from remote systems.
11 Fetching source code is one of the cornerstones of building software.
12 As such, this module forms an important part of BitBake.
13 </para>
14
15 <para>
16 The current fetch module is called "fetch2" and refers to the
17 fact that it is the second major version of the API.
18 The original version is obsolete and has been removed from the codebase.
19 Thus, in all cases, "fetch" refers to "fetch2" in this
20 manual.
21 </para>
22
23 <section id='the-download-fetch'>
24 <title>The Download (Fetch)</title>
25
26 <para>
27 BitBake takes several steps when fetching source code or files.
28 The fetcher codebase deals with two distinct processes in order:
29 obtaining the files from somewhere (cached or otherwise)
30 and then unpacking those files into a specific location and
31 perhaps in a specific way.
32 Getting and unpacking the files is often optionally followed
33 by patching.
34 Patching, however, is not covered by this module.
35 </para>
36
37 <para>
38 The code to execute the first part of this process, a fetch,
39 looks something like the following:
40 <literallayout class='monospaced'>
41 src_uri = (d.getVar('SRC_URI', True) or "").split()
42 fetcher = bb.fetch2.Fetch(src_uri, d)
43 fetcher.download()
44 </literallayout>
45 This code sets up an instance of the fetch class.
46 The instance uses a space-separated list of URLs from the
47 <link linkend='var-SRC_URI'><filename>SRC_URI</filename></link>
48 variable and then calls the <filename>download</filename>
49 method to download the files.
50 </para>
51
52 <para>
53 The instantiation of the fetch class is usually followed by:
54 <literallayout class='monospaced'>
55 rootdir = l.getVar('WORKDIR', True)
56 fetcher.unpack(rootdir)
57 </literallayout>
58 This code unpacks the downloaded files to the
59 specified by <filename>WORKDIR</filename>.
60 <note>
61 For convenience, the naming in these examples matches
62 the variables used by OpenEmbedded.
63 If you want to see the above code in action, examine
64 the OpenEmbedded class file <filename>base.bbclass</filename>.
65 </note>
66 The <filename>SRC_URI</filename> and <filename>WORKDIR</filename>
67 variables are not hardcoded into the fetcher, since those fetcher
68 methods can be (and are) called with different variable names.
69 In OpenEmbedded for example, the shared state (sstate) code uses
70 the fetch module to fetch the sstate files.
71 </para>
72
73 <para>
74 When the <filename>download()</filename> method is called,
75 BitBake tries to resolve the URLs by looking for source files
76 in a specific search order:
77 <itemizedlist>
78 <listitem><para><emphasis>Pre-mirror Sites:</emphasis>
79 BitBake first uses pre-mirrors to try and find source files.
80 These locations are defined using the
81 <link linkend='var-PREMIRRORS'><filename>PREMIRRORS</filename></link>
82 variable.
83 </para></listitem>
84 <listitem><para><emphasis>Source URI:</emphasis>
85 If pre-mirrors fail, BitBake uses the original URL (e.g from
86 <filename>SRC_URI</filename>).
87 </para></listitem>
88 <listitem><para><emphasis>Mirror Sites:</emphasis>
89 If fetch failures occur, BitBake next uses mirror locations as
90 defined by the
91 <link linkend='var-MIRRORS'><filename>MIRRORS</filename></link>
92 variable.
93 </para></listitem>
94 </itemizedlist>
95 </para>
96
97 <para>
98 For each URL passed to the fetcher, the fetcher
99 calls the submodule that handles that particular URL type.
100 This behavior can be the source of some confusion when you
101 are providing URLs for the <filename>SRC_URI</filename>
102 variable.
103 Consider the following two URLs:
104 <literallayout class='monospaced'>
105 http://git.yoctoproject.org/git/poky;protocol=git
106 git://git.yoctoproject.org/git/poky;protocol=http
107 </literallayout>
108 In the former case, the URL is passed to the
109 <filename>wget</filename> fetcher, which does not
110 understand "git".
111 Therefore, the latter case is the correct form since the
112 Git fetcher does know how to use HTTP as a transport.
113 </para>
114
115 <para>
116 Here are some examples that show commonly used mirror
117 definitions:
118 <literallayout class='monospaced'>
119 PREMIRRORS ?= "\
120 bzr://.*/.* http://somemirror.org/sources/ \n \
121 cvs://.*/.* http://somemirror.org/sources/ \n \
122 git://.*/.* http://somemirror.org/sources/ \n \
123 hg://.*/.* http://somemirror.org/sources/ \n \
124 osc://.*/.* http://somemirror.org/sources/ \n \
125 p4://.*/.* http://somemirror.org/sources/ \n \
126 svn://.*/.* http://somemirror.org/sources/ \n"
127
128 MIRRORS =+ "\
129 ftp://.*/.* http://somemirror.org/sources/ \n \
130 http://.*/.* http://somemirror.org/sources/ \n \
131 https://.*/.* http://somemirror.org/sources/ \n"
132 </literallayout>
133 It is useful to note that BitBake supports
134 cross-URLs.
135 It is possible to mirror a Git repository on an HTTP
136 server as a tarball.
137 This is what the <filename>git://</filename> mapping in
138 the previous example does.
139 </para>
140
141 <para>
142 Since network accesses are slow, Bitbake maintains a
143 cache of files downloaded from the network.
144 Any source files that are not local (i.e.
145 downloaded from the Internet) are placed into the download
146 directory, which is specified by the
147 <link linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
148 variable.
149 </para>
150
151 <para>
152 File integrity is of key importance for reproducing builds.
153 For non-local archive downloads, the fetcher code can verify
154 SHA-256 and MD5 checksums to ensure the archives have been
155 downloaded correctly.
156 You can specify these checksums by using the
157 <filename>SRC_URI</filename> variable with the appropriate
158 varflags as follows:
159 <literallayout class='monospaced'>
160 SRC_URI[md5sum] = "<replaceable>value</replaceable>"
161 SRC_URI[sha256sum] = "<replaceable>value</replaceable>"
162 </literallayout>
163 You can also specify the checksums as parameters on the
164 <filename>SRC_URI</filename> as shown below:
165 <literallayout class='monospaced'>
166 SRC_URI = "http://example.com/foobar.tar.bz2;md5sum=4a8e0f237e961fd7785d19d07fdb994d"
167 </literallayout>
168 If multiple URIs exist, you can specify the checksums either
169 directly as in the previous example, or you can name the URLs.
170 The following syntax shows how you name the URIs:
171 <literallayout class='monospaced'>
172 SRC_URI = "http://example.com/foobar.tar.bz2;name=foo"
173 SRC_URI[foo.md5sum] = 4a8e0f237e961fd7785d19d07fdb994d
174 </literallayout>
175 After a file has been downloaded and has had its checksum checked,
176 a ".done" stamp is placed in <filename>DL_DIR</filename>.
177 BitBake uses this stamp during subsequent builds to avoid
178 downloading or comparing a checksum for the file again.
179 <note>
180 It is assumed that local storage is safe from data corruption.
181 If this were not the case, there would be bigger issues to worry about.
182 </note>
183 </para>
184
185 <para>
186 If
187 <link linkend='var-BB_STRICT_CHECKSUM'><filename>BB_STRICT_CHECKSUM</filename></link>
188 is set, any download without a checksum triggers an
189 error message.
190 The
191 <link linkend='var-BB_NO_NETWORK'><filename>BB_NO_NETWORK</filename></link>
192 variable can be used to make any attempted network access a fatal
193 error, which is useful for checking that mirrors are complete
194 as well as other things.
195 </para>
196 </section>
197
198 <section id='bb-the-unpack'>
199 <title>The Unpack</title>
200
201 <para>
202 The unpack process usually immediately follows the download.
203 For all URLs except Git URLs, BitBake uses the common
204 <filename>unpack</filename> method.
205 </para>
206
207 <para>
208 A number of parameters exist that you can specify within the
209 URL to govern the behavior of the unpack stage:
210 <itemizedlist>
211 <listitem><para><emphasis>unpack:</emphasis>
212 Controls whether the URL components are unpacked.
213 If set to "1", which is the default, the components
214 are unpacked.
215 If set to "0", the unpack stage leaves the file alone.
216 This parameter is useful when you want an archive to be
217 copied in and not be unpacked.
218 </para></listitem>
219 <listitem><para><emphasis>dos:</emphasis>
220 Applies to <filename>.zip</filename> and
221 <filename>.jar</filename> files and specifies whether to
222 use DOS line ending conversion on text files.
223 </para></listitem>
224 <listitem><para><emphasis>basepath:</emphasis>
225 Instructs the unpack stage to strip the specified
226 directories from the source path when unpacking.
227 </para></listitem>
228 <listitem><para><emphasis>subdir:</emphasis>
229 Unpacks the specific URL to the specified subdirectory
230 within the root directory.
231 </para></listitem>
232 </itemizedlist>
233 The unpack call automatically decompresses and extracts files
234 with ".Z", ".z", ".gz", ".xz", ".zip", ".jar", ".ipk", ".rpm".
235 ".srpm", ".deb" and ".bz2" extensions as well as various combinations
236 of tarball extensions.
237 </para>
238
239 <para>
240 As mentioned, the Git fetcher has its own unpack method that
241 is optimized to work with Git trees.
242 Basically, this method works by cloning the tree into the final
243 directory.
244 The process is completed using references so that there is
245 only one central copy of the Git metadata needed.
246 </para>
247 </section>
248
249 <section id='bb-fetchers'>
250 <title>Fetchers</title>
251
252 <para>
253 As mentioned earlier, the URL prefix determines which
254 fetcher submodule BitBake uses.
255 Each submodule can support different URL parameters,
256 which are described in the following sections.
257 </para>
258
259 <section id='local-file-fetcher'>
260 <title>Local file fetcher (<filename>file://</filename>)</title>
261
262 <para>
263 This submodule handles URLs that begin with
264 <filename>file://</filename>.
265 The filename you specify within the URL can be
266 either an absolute or relative path to a file.
267 If the filename is relative, the contents of the
268 <link linkend='var-FILESPATH'><filename>FILESPATH</filename></link>
269 variable is used in the same way
270 <filename>PATH</filename> is used to find executables.
271 Failing that,
272 <link linkend='var-FILESDIR'><filename>FILESDIR</filename></link>
273 is used to find the appropriate relative file.
274 <note>
275 <filename>FILESDIR</filename> is deprecated and can
276 be replaced with <filename>FILESPATH</filename>.
277 Because <filename>FILESDIR</filename> is likely to be
278 removed, you should not use this variable in any new code.
279 </note>
280 If the file cannot be found, it is assumed that it is available in
281 <link linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
282 by the time the <filename>download()</filename> method is called.
283 </para>
284
285 <para>
286 If you specify a directory, the entire directory is
287 unpacked.
288 </para>
289
290 <para>
291 Here are a couple of example URLs, the first relative and
292 the second absolute:
293 <literallayout class='monospaced'>
294 SRC_URI = "file://relativefile.patch"
295 SRC_URI = "file:///Users/ich/very_important_software"
296 </literallayout>
297 </para>
298 </section>
299
300 <section id='http-ftp-fetcher'>
301 <title>HTTP/FTP wget fetcher (<filename>http://</filename>, <filename>ftp://</filename>, <filename>https://</filename>)</title>
302
303 <para>
304 This fetcher obtains files from web and FTP servers.
305 Internally, the fetcher uses the wget utility.
306 </para>
307
308 <para>
309 The executable and parameters used are specified by the
310 <filename>FETCHCMD_wget</filename> variable, which defaults
311 to sensible values.
312 The fetcher supports a parameter "downloadfilename" that
313 allows the name of the downloaded file to be specified.
314 Specifying the name of the downloaded file is useful
315 for avoiding collisions in
316 <link linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
317 when dealing with multiple files that have the same name.
318 </para>
319
320 <para>
321 Some example URLs are as follows:
322 <literallayout class='monospaced'>
323 SRC_URI = "http://oe.handhelds.org/not_there.aac"
324 SRC_URI = "ftp://oe.handhelds.org/not_there_as_well.aac"
325 SRC_URI = "ftp://you@oe.handhelds.org/home/you/secret.plan"
326 </literallayout>
327 </para>
328 <note>
329 Because URL parameters are delimited by semi-colons, this can
330 introduce ambiguity when parsing URLs that also contain semi-colons,
331 for example:
332 <literallayout class='monospaced'>
333 SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git;a=snapshot;h=a5dd47"
334 </literallayout>
335 Such URLs should should be modified by replacing semi-colons with '&amp;' characters:
336 <literallayout class='monospaced'>
337 SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&amp;a=snapshot&amp;h=a5dd47"
338 </literallayout>
339 In most cases this should work. Treating semi-colons and '&amp;' in queries
340 identically is recommended by the World Wide Web Consortium (W3C).
341 Note that due to the nature of the URL, you may have to specify the name
342 of the downloaded file as well:
343 <literallayout class='monospaced'>
344 SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&amp;a=snapshot&amp;h=a5dd47;downloadfilename=myfile.bz2"
345 </literallayout>
346 </note>
347 </section>
348
349 <section id='cvs-fetcher'>
350 <title>CVS fetcher (<filename>(cvs://</filename>)</title>
351
352 <para>
353 This submodule handles checking out files from the
354 CVS version control system.
355 You can configure it using a number of different variables:
356 <itemizedlist>
357 <listitem><para><emphasis><filename>FETCHCMD_cvs</filename>:</emphasis>
358 The name of the executable to use when running
359 the <filename>cvs</filename> command.
360 This name is usually "cvs".
361 </para></listitem>
362 <listitem><para><emphasis><filename>SRCDATE</filename>:</emphasis>
363 The date to use when fetching the CVS source code.
364 A special value of "now" causes the checkout to
365 be updated on every build.
366 </para></listitem>
367 <listitem><para><emphasis><link linkend='var-CVSDIR'><filename>CVSDIR</filename></link>:</emphasis>
368 Specifies where a temporary checkout is saved.
369 The location is often <filename>DL_DIR/cvs</filename>.
370 </para></listitem>
371 <listitem><para><emphasis><filename>CVS_PROXY_HOST</filename>:</emphasis>
372 The name to use as a "proxy=" parameter to the
373 <filename>cvs</filename> command.
374 </para></listitem>
375 <listitem><para><emphasis><filename>CVS_PROXY_PORT</filename>:</emphasis>
376 The port number to use as a "proxyport=" parameter to
377 the <filename>cvs</filename> command.
378 </para></listitem>
379 </itemizedlist>
380 As well as the standard username and password URL syntax,
381 you can also configure the fetcher with various URL parameters:
382 </para>
383
384 <para>
385 The supported parameters are as follows:
386 <itemizedlist>
387 <listitem><para><emphasis>"method":</emphasis>
388 The protocol over which to communicate with the CVS server.
389 By default, this protocol is "pserver".
390 If "method" is set to "ext", BitBake examines the
391 "rsh" parameter and sets <filename>CVS_RSH</filename>.
392 You can use "dir" for local directories.
393 </para></listitem>
394 <listitem><para><emphasis>"module":</emphasis>
395 Specifies the module to check out.
396 You must supply this parameter.
397 </para></listitem>
398 <listitem><para><emphasis>"tag":</emphasis>
399 Describes which CVS TAG should be used for
400 the checkout.
401 By default, the TAG is empty.
402 </para></listitem>
403 <listitem><para><emphasis>"date":</emphasis>
404 Specifies a date.
405 If no "date" is specified, the
406 <link linkend='var-SRCDATE'><filename>SRCDATE</filename></link>
407 of the configuration is used to checkout a specific date.
408 The special value of "now" causes the checkout to be
409 updated on every build.
410 </para></listitem>
411 <listitem><para><emphasis>"localdir":</emphasis>
412 Used to rename the module.
413 Effectively, you are renaming the output directory
414 to which the module is unpacked.
415 You are forcing the module into a special
416 directory relative to
417 <link linkend='var-CVSDIR'><filename>CVSDIR</filename></link>.
418 </para></listitem>
419 <listitem><para><emphasis>"rsh"</emphasis>
420 Used in conjunction with the "method" parameter.
421 </para></listitem>
422 <listitem><para><emphasis>"scmdata":</emphasis>
423 Causes the CVS metadata to be maintained in the tarball
424 the fetcher creates when set to "keep".
425 The tarball is expanded into the work directory.
426 By default, the CVS metadata is removed.
427 </para></listitem>
428 <listitem><para><emphasis>"fullpath":</emphasis>
429 Controls whether the resulting checkout is at the
430 module level, which is the default, or is at deeper
431 paths.
432 </para></listitem>
433 <listitem><para><emphasis>"norecurse":</emphasis>
434 Causes the fetcher to only checkout the specified
435 directory with no recurse into any subdirectories.
436 </para></listitem>
437 <listitem><para><emphasis>"port":</emphasis>
438 The port to which the CVS server connects.
439 </para></listitem>
440 </itemizedlist>
441 Some example URLs are as follows:
442 <literallayout class='monospaced'>
443 SRC_URI = "cvs://CVSROOT;module=mymodule;tag=some-version;method=ext"
444 SRC_URI = "cvs://CVSROOT;module=mymodule;date=20060126;localdir=usethat"
445 </literallayout>
446 </para>
447 </section>
448
449 <section id='svn-fetcher'>
450 <title>Subversion (SVN) Fetcher (<filename>svn://</filename>)</title>
451
452 <para>
453 This fetcher submodule fetches code from the
454 Subversion source control system.
455 The executable used is specified by
456 <filename>FETCHCMD_svn</filename>, which defaults
457 to "svn".
458 The fetcher's temporary working directory is set by
459 <link linkend='var-SVNDIR'><filename>SVNDIR</filename></link>,
460 which is usually <filename>DL_DIR/svn</filename>.
461 </para>
462
463 <para>
464 The supported parameters are as follows:
465 <itemizedlist>
466 <listitem><para><emphasis>"module":</emphasis>
467 The name of the svn module to checkout.
468 You must provide this parameter.
469 You can think of this parameter as the top-level
470 directory of the repository data you want.
471 </para></listitem>
472 <listitem><para><emphasis>"protocol":</emphasis>
473 The protocol to use, which defaults to "svn".
474 Other options are "svn+ssh" and "rsh".
475 For "rsh", the "rsh" parameter is also used.
476 </para></listitem>
477 <listitem><para><emphasis>"rev":</emphasis>
478 The revision of the source code to checkout.
479 </para></listitem>
480 <listitem><para><emphasis>"date":</emphasis>
481 The date of the source code to checkout.
482 Specific revisions are generally much safer to checkout
483 rather than by date as they do not involve timezones
484 (e.g. they are much more deterministic).
485 </para></listitem>
486 <listitem><para><emphasis>"scmdata":</emphasis>
487 Causes the “.svn” directories to be available during
488 compile-time when set to "keep".
489 By default, these directories are removed.
490 </para></listitem>
491 <listitem><para><emphasis>"transportuser":</emphasis>
492 When required, sets the username for the transport.
493 By default, this parameter is empty.
494 The transport username is different than the username
495 used in the main URL, which is passed to the subversion
496 command.
497 </para></listitem>
498 </itemizedlist>
499 Following are two examples using svn:
500 <literallayout class='monospaced'>
501 SRC_URI = "svn://svn.oe.handhelds.org/svn;module=vip;proto=http;rev=667"
502 SRC_URI = "svn://svn.oe.handhelds.org/svn/;module=opie;proto=svn+ssh;date=20060126"
503 </literallayout>
504 </para>
505 </section>
506
507 <section id='git-fetcher'>
508 <title>Git Fetcher (<filename>git://</filename>)</title>
509
510 <para>
511 This fetcher submodule fetches code from the Git
512 source control system.
513 The fetcher works by creating a bare clone of the
514 remote into
515 <link linkend='var-GITDIR'><filename>GITDIR</filename></link>,
516 which is usually <filename>DL_DIR/git2</filename>.
517 This bare clone is then cloned into the work directory during the
518 unpack stage when a specific tree is checked out.
519 This is done using alternates and by reference to
520 minimize the amount of duplicate data on the disk and
521 make the unpack process fast.
522 The executable used can be set with
523 <filename>FETCHCMD_git</filename>.
524 </para>
525
526 <para>
527 This fetcher supports the following parameters:
528 <itemizedlist>
529 <listitem><para><emphasis>"protocol":</emphasis>
530 The protocol used to fetch the files.
531 The default is "git" when a hostname is set.
532 If a hostname is not set, the Git protocol is "file".
533 You can also use "http", "https", "ssh" and "rsync".
534 </para></listitem>
535 <listitem><para><emphasis>"nocheckout":</emphasis>
536 Tells the fetcher to not checkout source code when
537 unpacking when set to "1".
538 Set this option for the URL where there is a custom
539 routine to checkout code.
540 The default is "0".
541 </para></listitem>
542 <listitem><para><emphasis>"rebaseable":</emphasis>
543 Indicates that the upstream Git repository can be rebased.
544 You should set this parameter to "1" if
545 revisions can become detached from branches.
546 In this case, the source mirror tarball is done per
547 revision, which has a loss of efficiency.
548 Rebasing the upstream Git repository could cause the
549 current revision to disappear from the upstream repository.
550 This option reminds the fetcher to preserve the local cache
551 carefully for future use.
552 The default value for this parameter is "0".
553 </para></listitem>
554 <listitem><para><emphasis>"nobranch":</emphasis>
555 Tells the fetcher to not check the SHA validation
556 for the branch when set to "1".
557 The default is "0".
558 Set this option for the recipe that refers to
559 the commit that is valid for a tag instead of
560 the branch.
561 </para></listitem>
562 <listitem><para><emphasis>"bareclone":</emphasis>
563 Tells the fetcher to clone a bare clone into the
564 destination directory without checking out a working tree.
565 Only the raw Git metadata is provided.
566 This parameter implies the "nocheckout" parameter as well.
567 </para></listitem>
568 <listitem><para><emphasis>"branch":</emphasis>
569 The branch(es) of the Git tree to clone.
570 If unset, this is assumed to be "master".
571 The number of branch parameters much match the number of
572 name parameters.
573 </para></listitem>
574 <listitem><para><emphasis>"rev":</emphasis>
575 The revision to use for the checkout.
576 The default is "master".
577 </para></listitem>
578 <listitem><para><emphasis>"tag":</emphasis>
579 Specifies a tag to use for the checkout.
580 To correctly resolve tags, BitBake must access the
581 network.
582 For that reason, tags are often not used.
583 As far as Git is concerned, the "tag" parameter behaves
584 effectively the same as the "rev" parameter.
585 </para></listitem>
586 <listitem><para><emphasis>"subpath":</emphasis>
587 Limits the checkout to a specific subpath of the tree.
588 By default, the whole tree is checked out.
589 </para></listitem>
590 <listitem><para><emphasis>"destsuffix":</emphasis>
591 The name of the path in which to place the checkout.
592 By default, the path is <filename>git/</filename>.
593 </para></listitem>
594 </itemizedlist>
595 Here are some example URLs:
596 <literallayout class='monospaced'>
597 SRC_URI = "git://git.oe.handhelds.org/git/vip.git;tag=version-1"
598 SRC_URI = "git://git.oe.handhelds.org/git/vip.git;protocol=http"
599 </literallayout>
600 </para>
601 </section>
602
603 <section id='gitsm-fetcher'>
604 <title>Git Submodule Fetcher (<filename>gitsm://</filename>)</title>
605
606 <para>
607 This fetcher submodule inherits from the
608 <link linkend='git-fetcher'>Git fetcher</link> and extends
609 that fetcher's behavior by fetching a repository's submodules.
610 <link linkend='var-SRC_URI'><filename>SRC_URI</filename></link>
611 is passed to the Git fetcher as described in the
612 "<link linkend='git-fetcher'>Git Fetcher (<filename>git://</filename>)</link>"
613 section.
614 <note>
615 <title>Notes and Warnings</title>
616 <para>
617 You must clean a recipe when switching between
618 '<filename>git://</filename>' and
619 '<filename>gitsm://</filename>' URLs.
620 </para>
621
622 <para>
623 The Git Submodules fetcher is not a complete fetcher
624 implementation.
625 The fetcher has known issues where it does not use the
626 normal source mirroring infrastructure properly.
627 </para>
628 </note>
629 </para>
630 </section>
631
632 <section id='clearcase-fetcher'>
633 <title>ClearCase Fetcher (<filename>ccrc://</filename>)</title>
634
635 <para>
636 This fetcher submodule fetches code from a
637 <ulink url='http://en.wikipedia.org/wiki/Rational_ClearCase'>ClearCase</ulink>
638 repository.
639 </para>
640
641 <para>
642 To use this fetcher, make sure your recipe has proper
643 <link linkend='var-SRC_URI'><filename>SRC_URI</filename></link>,
644 <link linkend='var-SRCREV'><filename>SRCREV</filename></link>, and
645 <link linkend='var-PV'><filename>PV</filename></link> settings.
646 Here is an example:
647 <literallayout class='monospaced'>
648 SRC_URI = "ccrc://cc.example.org/ccrc;vob=/example_vob;module=/example_module"
649 SRCREV = "EXAMPLE_CLEARCASE_TAG"
650 PV = "${@d.getVar("SRCREV", False).replace("/", "+")}"
651 </literallayout>
652 The fetcher uses the <filename>rcleartool</filename> or
653 <filename>cleartool</filename> remote client, depending on
654 which one is available.
655 </para>
656
657 <para>
658 Following are options for the <filename>SRC_URI</filename>
659 statement:
660 <itemizedlist>
661 <listitem><para><emphasis><filename>vob</filename></emphasis>:
662 The name, which must include the
663 prepending "/" character, of the ClearCase VOB.
664 This option is required.
665 </para></listitem>
666 <listitem><para><emphasis><filename>module</filename></emphasis>:
667 The module, which must include the
668 prepending "/" character, in the selected VOB.
669 <note>
670 The <filename>module</filename> and <filename>vob</filename>
671 options are combined to create the <filename>load</filename> rule in
672 the view config spec.
Patrick Williamsc0f7c042017-02-23 20:41:17 -0600673 As an example, consider the <filename>vob</filename> and
674 <filename>module</filename> values from the
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500675 <filename>SRC_URI</filename> statement at the start of this section.
676 Combining those values results in the following:
677 <literallayout class='monospaced'>
678 load /example_vob/example_module
679 </literallayout>
680 </note>
681 </para></listitem>
682 <listitem><para><emphasis><filename>proto</filename></emphasis>:
683 The protocol, which can be either <filename>http</filename> or
684 <filename>https</filename>.
685 </para></listitem>
686 </itemizedlist>
687 </para>
688
689 <para>
690 By default, the fetcher creates a configuration specification.
691 If you want this specification written to an area other than the default,
692 use the <filename>CCASE_CUSTOM_CONFIG_SPEC</filename> variable
693 in your recipe to define where the specification is written.
694 <note>
695 the <filename>SRCREV</filename> loses its functionality if you
696 specify this variable.
697 However, <filename>SRCREV</filename> is still used to label the
698 archive after a fetch even though it does not define what is
699 fetched.
700 </note>
701 </para>
702
703 <para>
704 Here are a couple of other behaviors worth mentioning:
705 <itemizedlist>
706 <listitem><para>
707 When using <filename>cleartool</filename>, the login of
708 <filename>cleartool</filename> is handled by the system.
709 The login require no special steps.
710 </para></listitem>
711 <listitem><para>
712 In order to use <filename>rcleartool</filename> with authenticated
713 users, an "rcleartool login" is necessary before using the fetcher.
714 </para></listitem>
715 </itemizedlist>
716 </para>
717 </section>
718
Patrick Williamsc0f7c042017-02-23 20:41:17 -0600719 <section id='perforce-fetcher'>
720 <title>Perforce Fetcher (<filename>p4://</filename>)</title>
721
722 <para>
723 This fetcher submodule fetches code from the
724 <ulink url='https://www.perforce.com/'>Perforce</ulink>
725 source control system.
726 The executable used is specified by
727 <filename>FETCHCMD_p4</filename>, which defaults
728 to "p4".
729 The fetcher's temporary working directory is set by
730 <link linkend='var-P4DIR'><filename>P4DIR</filename></link>,
731 which defaults to "DL_DIR/p4".
732 </para>
733
734 <para>
735 To use this fetcher, make sure your recipe has proper
736 <link linkend='var-SRC_URI'><filename>SRC_URI</filename></link>,
737 <link linkend='var-SRCREV'><filename>SRCREV</filename></link>, and
738 <link linkend='var-PV'><filename>PV</filename></link> values.
739 The p4 executable is able to use the config file defined by your
740 system's <filename>P4CONFIG</filename> environment variable in
741 order to define the Perforce server URL and port, username, and
742 password if you do not wish to keep those values in a recipe
743 itself.
744 If you choose not to use <filename>P4CONFIG</filename>,
745 or to explicitly set variables that <filename>P4CONFIG</filename>
746 can contain, you can specify the <filename>P4PORT</filename> value,
747 which is the server's URL and port number, and you can
748 specify a username and password directly in your recipe within
749 <filename>SRC_URI</filename>.
750 </para>
751
752 <para>
753 Here is an example that relies on <filename>P4CONFIG</filename>
754 to specify the server URL and port, username, and password, and
755 fetches the Head Revision:
756 <literallayout class='monospaced'>
757 SRC_URI = "p4://example-depot/main/source/..."
758 SRCREV = "${AUTOREV}"
759 PV = "p4-${SRCPV}"
760 S = "${WORKDIR}/p4"
761 </literallayout>
762 </para>
763
764 <para>
765 Here is an example that specifies the server URL and port,
766 username, and password, and fetches a Revision based on a Label:
767 <literallayout class='monospaced'>
768 P4PORT = "tcp:p4server.example.net:1666"
769 SRC_URI = "p4://user:passwd@example-depot/main/source/..."
770 SRCREV = "release-1.0"
771 PV = "p4-${SRCPV}"
772 S = "${WORKDIR}/p4"
773 </literallayout>
774 <note>
775 You should always set <filename>S</filename>
776 to <filename>"${WORKDIR}/p4"</filename> in your recipe.
777 </note>
778 </para>
779 </section>
780
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500781 <section id='other-fetchers'>
782 <title>Other Fetchers</title>
783
784 <para>
785 Fetch submodules also exist for the following:
786 <itemizedlist>
787 <listitem><para>
788 Bazaar (<filename>bzr://</filename>)
789 </para></listitem>
790 <listitem><para>
Patrick Williamsc124f4f2015-09-15 14:41:29 -0500791 Trees using Git Annex (<filename>gitannex://</filename>)
792 </para></listitem>
793 <listitem><para>
794 Secure FTP (<filename>sftp://</filename>)
795 </para></listitem>
796 <listitem><para>
797 Secure Shell (<filename>ssh://</filename>)
798 </para></listitem>
799 <listitem><para>
800 Repo (<filename>repo://</filename>)
801 </para></listitem>
802 <listitem><para>
803 OSC (<filename>osc://</filename>)
804 </para></listitem>
805 <listitem><para>
806 Mercurial (<filename>hg://</filename>)
807 </para></listitem>
808 </itemizedlist>
809 No documentation currently exists for these lesser used
810 fetcher submodules.
811 However, you might find the code helpful and readable.
812 </para>
813 </section>
814 </section>
815
816 <section id='auto-revisions'>
817 <title>Auto Revisions</title>
818
819 <para>
820 We need to document <filename>AUTOREV</filename> and
821 <filename>SRCREV_FORMAT</filename> here.
822 </para>
823 </section>
824</chapter>