blob: f9d9e617f30f0ec344cd99ccf778d22498ef62c4 [file] [log] [blame]
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001.. SPDX-License-Identifier: CC-BY-2.5
2
3=====================
4File Download Support
5=====================
6
7|
8
9BitBake's fetch module is a standalone piece of library code that deals
10with the intricacies of downloading source code and files from remote
11systems. Fetching source code is one of the cornerstones of building
12software. As such, this module forms an important part of BitBake.
13
14The current fetch module is called "fetch2" and refers to the fact that
15it is the second major version of the API. The original version is
16obsolete and has been removed from the codebase. Thus, in all cases,
17"fetch" refers to "fetch2" in this manual.
18
19The Download (Fetch)
20====================
21
22BitBake takes several steps when fetching source code or files. The
23fetcher codebase deals with two distinct processes in order: obtaining
24the files from somewhere (cached or otherwise) and then unpacking those
25files into a specific location and perhaps in a specific way. Getting
26and unpacking the files is often optionally followed by patching.
27Patching, however, is not covered by this module.
28
29The code to execute the first part of this process, a fetch, looks
Andrew Geisslerc926e172021-05-07 16:11:35 -050030something like the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050031
32 src_uri = (d.getVar('SRC_URI') or "").split()
33 fetcher = bb.fetch2.Fetch(src_uri, d)
34 fetcher.download()
35
36This code sets up an instance of the fetch class. The instance uses a
37space-separated list of URLs from the :term:`SRC_URI`
38variable and then calls the ``download`` method to download the files.
39
Andrew Geisslerc926e172021-05-07 16:11:35 -050040The instantiation of the fetch class is usually followed by::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050041
42 rootdir = l.getVar('WORKDIR')
43 fetcher.unpack(rootdir)
44
45This code unpacks the downloaded files to the specified by ``WORKDIR``.
46
47.. note::
48
49 For convenience, the naming in these examples matches the variables
50 used by OpenEmbedded. If you want to see the above code in action,
51 examine the OpenEmbedded class file ``base.bbclass``
52 .
53
Patrick Williams213cb262021-08-07 19:21:33 -050054The :term:`SRC_URI` and ``WORKDIR`` variables are not hardcoded into the
Andrew Geisslerc9f78652020-09-18 14:11:35 -050055fetcher, since those fetcher methods can be (and are) called with
56different variable names. In OpenEmbedded for example, the shared state
57(sstate) code uses the fetch module to fetch the sstate files.
58
59When the ``download()`` method is called, BitBake tries to resolve the
60URLs by looking for source files in a specific search order:
61
62- *Pre-mirror Sites:* BitBake first uses pre-mirrors to try and find
63 source files. These locations are defined using the
64 :term:`PREMIRRORS` variable.
65
66- *Source URI:* If pre-mirrors fail, BitBake uses the original URL (e.g
Patrick Williams213cb262021-08-07 19:21:33 -050067 from :term:`SRC_URI`).
Andrew Geisslerc9f78652020-09-18 14:11:35 -050068
69- *Mirror Sites:* If fetch failures occur, BitBake next uses mirror
70 locations as defined by the :term:`MIRRORS` variable.
71
72For each URL passed to the fetcher, the fetcher calls the submodule that
73handles that particular URL type. This behavior can be the source of
Patrick Williams213cb262021-08-07 19:21:33 -050074some confusion when you are providing URLs for the :term:`SRC_URI` variable.
Andrew Geisslerc926e172021-05-07 16:11:35 -050075Consider the following two URLs::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050076
Andrew Geisslereff27472021-10-29 15:35:00 -050077 https://git.yoctoproject.org/git/poky;protocol=git
Andrew Geisslerc9f78652020-09-18 14:11:35 -050078 git://git.yoctoproject.org/git/poky;protocol=http
79
80In the former case, the URL is passed to the ``wget`` fetcher, which does not
81understand "git". Therefore, the latter case is the correct form since the Git
82fetcher does know how to use HTTP as a transport.
83
Andrew Geisslerc926e172021-05-07 16:11:35 -050084Here are some examples that show commonly used mirror definitions::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050085
86 PREMIRRORS ?= "\
Andrew Geissler595f6302022-01-24 19:11:47 +000087 bzr://.*/.\* http://somemirror.org/sources/ \
88 cvs://.*/.\* http://somemirror.org/sources/ \
89 git://.*/.\* http://somemirror.org/sources/ \
90 hg://.*/.\* http://somemirror.org/sources/ \
91 osc://.*/.\* http://somemirror.org/sources/ \
92 p4://.*/.\* http://somemirror.org/sources/ \
93 svn://.*/.\* http://somemirror.org/sources/"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050094
95 MIRRORS =+ "\
Andrew Geissler595f6302022-01-24 19:11:47 +000096 ftp://.*/.\* http://somemirror.org/sources/ \
97 http://.*/.\* http://somemirror.org/sources/ \
98 https://.*/.\* http://somemirror.org/sources/"
Andrew Geisslerc9f78652020-09-18 14:11:35 -050099
100It is useful to note that BitBake
101supports cross-URLs. It is possible to mirror a Git repository on an
102HTTP server as a tarball. This is what the ``git://`` mapping in the
103previous example does.
104
105Since network accesses are slow, BitBake maintains a cache of files
106downloaded from the network. Any source files that are not local (i.e.
107downloaded from the Internet) are placed into the download directory,
108which is specified by the :term:`DL_DIR` variable.
109
110File integrity is of key importance for reproducing builds. For
111non-local archive downloads, the fetcher code can verify SHA-256 and MD5
112checksums to ensure the archives have been downloaded correctly. You can
Patrick Williams213cb262021-08-07 19:21:33 -0500113specify these checksums by using the :term:`SRC_URI` variable with the
Andrew Geisslerc926e172021-05-07 16:11:35 -0500114appropriate varflags as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500115
116 SRC_URI[md5sum] = "value"
117 SRC_URI[sha256sum] = "value"
118
119You can also specify the checksums as
Patrick Williams213cb262021-08-07 19:21:33 -0500120parameters on the :term:`SRC_URI` as shown below::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500121
122 SRC_URI = "http://example.com/foobar.tar.bz2;md5sum=4a8e0f237e961fd7785d19d07fdb994d"
123
124If multiple URIs exist, you can specify the checksums either directly as
125in the previous example, or you can name the URLs. The following syntax
Andrew Geisslerc926e172021-05-07 16:11:35 -0500126shows how you name the URIs::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500127
128 SRC_URI = "http://example.com/foobar.tar.bz2;name=foo"
129 SRC_URI[foo.md5sum] = 4a8e0f237e961fd7785d19d07fdb994d
130
131After a file has been downloaded and
Patrick Williams213cb262021-08-07 19:21:33 -0500132has had its checksum checked, a ".done" stamp is placed in :term:`DL_DIR`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500133BitBake uses this stamp during subsequent builds to avoid downloading or
134comparing a checksum for the file again.
135
136.. note::
137
138 It is assumed that local storage is safe from data corruption. If
139 this were not the case, there would be bigger issues to worry about.
140
141If :term:`BB_STRICT_CHECKSUM` is set, any
142download without a checksum triggers an error message. The
143:term:`BB_NO_NETWORK` variable can be used to
144make any attempted network access a fatal error, which is useful for
145checking that mirrors are complete as well as other things.
146
Patrick Williams0ca19cc2021-08-16 14:03:13 -0500147If :term:`BB_CHECK_SSL_CERTS` is set to ``0`` then SSL certificate checking will
148be disabled. This variable defaults to ``1`` so SSL certificates are normally
149checked.
150
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500151.. _bb-the-unpack:
152
153The Unpack
154==========
155
156The unpack process usually immediately follows the download. For all
157URLs except Git URLs, BitBake uses the common ``unpack`` method.
158
159A number of parameters exist that you can specify within the URL to
160govern the behavior of the unpack stage:
161
162- *unpack:* Controls whether the URL components are unpacked. If set to
163 "1", which is the default, the components are unpacked. If set to
164 "0", the unpack stage leaves the file alone. This parameter is useful
165 when you want an archive to be copied in and not be unpacked.
166
167- *dos:* Applies to ``.zip`` and ``.jar`` files and specifies whether
168 to use DOS line ending conversion on text files.
169
Andrew Geissler595f6302022-01-24 19:11:47 +0000170- *striplevel:* Strip specified number of leading components (levels)
171 from file names on extraction
172
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500173- *subdir:* Unpacks the specific URL to the specified subdirectory
174 within the root directory.
175
176The unpack call automatically decompresses and extracts files with ".Z",
177".z", ".gz", ".xz", ".zip", ".jar", ".ipk", ".rpm". ".srpm", ".deb" and
178".bz2" extensions as well as various combinations of tarball extensions.
179
180As mentioned, the Git fetcher has its own unpack method that is
181optimized to work with Git trees. Basically, this method works by
182cloning the tree into the final directory. The process is completed
183using references so that there is only one central copy of the Git
184metadata needed.
185
186.. _bb-fetchers:
187
188Fetchers
189========
190
191As mentioned earlier, the URL prefix determines which fetcher submodule
192BitBake uses. Each submodule can support different URL parameters, which
193are described in the following sections.
194
195.. _local-file-fetcher:
196
197Local file fetcher (``file://``)
198--------------------------------
199
200This submodule handles URLs that begin with ``file://``. The filename
201you specify within the URL can be either an absolute or relative path to
202a file. If the filename is relative, the contents of the
203:term:`FILESPATH` variable is used in the same way
204``PATH`` is used to find executables. If the file cannot be found, it is
205assumed that it is available in :term:`DL_DIR` by the
206time the ``download()`` method is called.
207
208If you specify a directory, the entire directory is unpacked.
209
210Here are a couple of example URLs, the first relative and the second
Andrew Geisslerc926e172021-05-07 16:11:35 -0500211absolute::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500212
213 SRC_URI = "file://relativefile.patch"
214 SRC_URI = "file:///Users/ich/very_important_software"
215
216.. _http-ftp-fetcher:
217
218HTTP/FTP wget fetcher (``http://``, ``ftp://``, ``https://``)
219-------------------------------------------------------------
220
221This fetcher obtains files from web and FTP servers. Internally, the
222fetcher uses the wget utility.
223
224The executable and parameters used are specified by the
225``FETCHCMD_wget`` variable, which defaults to sensible values. The
226fetcher supports a parameter "downloadfilename" that allows the name of
227the downloaded file to be specified. Specifying the name of the
228downloaded file is useful for avoiding collisions in
229:term:`DL_DIR` when dealing with multiple files that
230have the same name.
231
Andrew Geissler595f6302022-01-24 19:11:47 +0000232If a username and password are specified in the ``SRC_URI``, a Basic
233Authorization header will be added to each request, including across redirects.
234To instead limit the Authorization header to the first request, add
235"redirectauth=0" to the list of parameters.
236
Andrew Geisslerc926e172021-05-07 16:11:35 -0500237Some example URLs are as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500238
239 SRC_URI = "http://oe.handhelds.org/not_there.aac"
240 SRC_URI = "ftp://oe.handhelds.org/not_there_as_well.aac"
241 SRC_URI = "ftp://you@oe.handhelds.org/home/you/secret.plan"
242
243.. note::
244
245 Because URL parameters are delimited by semi-colons, this can
246 introduce ambiguity when parsing URLs that also contain semi-colons,
Andrew Geisslerc926e172021-05-07 16:11:35 -0500247 for example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500248
249 SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git;a=snapshot;h=a5dd47"
250
251
252 Such URLs should should be modified by replacing semi-colons with '&'
Andrew Geisslerc926e172021-05-07 16:11:35 -0500253 characters::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500254
255 SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47"
256
257
258 In most cases this should work. Treating semi-colons and '&' in
259 queries identically is recommended by the World Wide Web Consortium
260 (W3C). Note that due to the nature of the URL, you may have to
Andrew Geisslerc926e172021-05-07 16:11:35 -0500261 specify the name of the downloaded file as well::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500262
263 SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47;downloadfilename=myfile.bz2"
264
265
266.. _cvs-fetcher:
267
268CVS fetcher (``(cvs://``)
269-------------------------
270
271This submodule handles checking out files from the CVS version control
272system. You can configure it using a number of different variables:
273
274- :term:`FETCHCMD_cvs <FETCHCMD>`: The name of the executable to use when running
275 the ``cvs`` command. This name is usually "cvs".
276
277- :term:`SRCDATE`: The date to use when fetching the CVS source code. A
278 special value of "now" causes the checkout to be updated on every
279 build.
280
281- :term:`CVSDIR`: Specifies where a temporary
282 checkout is saved. The location is often ``DL_DIR/cvs``.
283
284- CVS_PROXY_HOST: The name to use as a "proxy=" parameter to the
285 ``cvs`` command.
286
287- CVS_PROXY_PORT: The port number to use as a "proxyport="
288 parameter to the ``cvs`` command.
289
290As well as the standard username and password URL syntax, you can also
291configure the fetcher with various URL parameters:
292
293The supported parameters are as follows:
294
295- *"method":* The protocol over which to communicate with the CVS
296 server. By default, this protocol is "pserver". If "method" is set to
297 "ext", BitBake examines the "rsh" parameter and sets ``CVS_RSH``. You
298 can use "dir" for local directories.
299
300- *"module":* Specifies the module to check out. You must supply this
301 parameter.
302
303- *"tag":* Describes which CVS TAG should be used for the checkout. By
304 default, the TAG is empty.
305
306- *"date":* Specifies a date. If no "date" is specified, the
307 :term:`SRCDATE` of the configuration is used to
308 checkout a specific date. The special value of "now" causes the
309 checkout to be updated on every build.
310
311- *"localdir":* Used to rename the module. Effectively, you are
312 renaming the output directory to which the module is unpacked. You
313 are forcing the module into a special directory relative to
314 :term:`CVSDIR`.
315
316- *"rsh":* Used in conjunction with the "method" parameter.
317
318- *"scmdata":* Causes the CVS metadata to be maintained in the tarball
319 the fetcher creates when set to "keep". The tarball is expanded into
320 the work directory. By default, the CVS metadata is removed.
321
322- *"fullpath":* Controls whether the resulting checkout is at the
323 module level, which is the default, or is at deeper paths.
324
325- *"norecurse":* Causes the fetcher to only checkout the specified
326 directory with no recurse into any subdirectories.
327
328- *"port":* The port to which the CVS server connects.
329
Andrew Geisslerc926e172021-05-07 16:11:35 -0500330Some example URLs are as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500331
332 SRC_URI = "cvs://CVSROOT;module=mymodule;tag=some-version;method=ext"
333 SRC_URI = "cvs://CVSROOT;module=mymodule;date=20060126;localdir=usethat"
334
335.. _svn-fetcher:
336
337Subversion (SVN) Fetcher (``svn://``)
338-------------------------------------
339
340This fetcher submodule fetches code from the Subversion source control
341system. The executable used is specified by ``FETCHCMD_svn``, which
342defaults to "svn". The fetcher's temporary working directory is set by
343:term:`SVNDIR`, which is usually ``DL_DIR/svn``.
344
345The supported parameters are as follows:
346
347- *"module":* The name of the svn module to checkout. You must provide
348 this parameter. You can think of this parameter as the top-level
349 directory of the repository data you want.
350
351- *"path_spec":* A specific directory in which to checkout the
352 specified svn module.
353
354- *"protocol":* The protocol to use, which defaults to "svn". If
355 "protocol" is set to "svn+ssh", the "ssh" parameter is also used.
356
357- *"rev":* The revision of the source code to checkout.
358
Andrew Geisslerf0343792020-11-18 10:42:21 -0600359- *"scmdata":* Causes the ".svn" directories to be available during
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500360 compile-time when set to "keep". By default, these directories are
361 removed.
362
363- *"ssh":* An optional parameter used when "protocol" is set to
364 "svn+ssh". You can use this parameter to specify the ssh program used
365 by svn.
366
367- *"transportuser":* When required, sets the username for the
368 transport. By default, this parameter is empty. The transport
369 username is different than the username used in the main URL, which
370 is passed to the subversion command.
371
Andrew Geisslerc926e172021-05-07 16:11:35 -0500372Following are three examples using svn::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500373
374 SRC_URI = "svn://myrepos/proj1;module=vip;protocol=http;rev=667"
375 SRC_URI = "svn://myrepos/proj1;module=opie;protocol=svn+ssh"
376 SRC_URI = "svn://myrepos/proj1;module=trunk;protocol=http;path_spec=${MY_DIR}/proj1"
377
378.. _git-fetcher:
379
380Git Fetcher (``git://``)
381------------------------
382
383This fetcher submodule fetches code from the Git source control system.
384The fetcher works by creating a bare clone of the remote into
385:term:`GITDIR`, which is usually ``DL_DIR/git2``. This
386bare clone is then cloned into the work directory during the unpack
387stage when a specific tree is checked out. This is done using alternates
388and by reference to minimize the amount of duplicate data on the disk
389and make the unpack process fast. The executable used can be set with
390``FETCHCMD_git``.
391
392This fetcher supports the following parameters:
393
394- *"protocol":* The protocol used to fetch the files. The default is
395 "git" when a hostname is set. If a hostname is not set, the Git
396 protocol is "file". You can also use "http", "https", "ssh" and
397 "rsync".
398
Andrew Geissler595f6302022-01-24 19:11:47 +0000399 .. note::
400
401 When ``protocol`` is "ssh", the URL expected in :term:`SRC_URI` differs
402 from the one that is typically passed to ``git clone`` command and provided
403 by the Git server to fetch from. For example, the URL returned by GitLab
404 server for ``mesa`` when cloning over SSH is
405 ``git@gitlab.freedesktop.org:mesa/mesa.git``, however the expected URL in
406 :term:`SRC_URI` is the following::
407
Andrew Geissler9aee5002022-03-30 16:27:02 +0000408 SRC_URI = "git://git@gitlab.freedesktop.org/mesa/mesa.git;branch=main;protocol=ssh;..."
Andrew Geissler595f6302022-01-24 19:11:47 +0000409
410 Note the ``:`` character changed for a ``/`` before the path to the project.
411
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500412- *"nocheckout":* Tells the fetcher to not checkout source code when
413 unpacking when set to "1". Set this option for the URL where there is
414 a custom routine to checkout code. The default is "0".
415
416- *"rebaseable":* Indicates that the upstream Git repository can be
417 rebased. You should set this parameter to "1" if revisions can become
418 detached from branches. In this case, the source mirror tarball is
419 done per revision, which has a loss of efficiency. Rebasing the
420 upstream Git repository could cause the current revision to disappear
421 from the upstream repository. This option reminds the fetcher to
422 preserve the local cache carefully for future use. The default value
423 for this parameter is "0".
424
425- *"nobranch":* Tells the fetcher to not check the SHA validation for
426 the branch when set to "1". The default is "0". Set this option for
427 the recipe that refers to the commit that is valid for a tag instead
428 of the branch.
429
430- *"bareclone":* Tells the fetcher to clone a bare clone into the
431 destination directory without checking out a working tree. Only the
432 raw Git metadata is provided. This parameter implies the "nocheckout"
433 parameter as well.
434
Andrew Geissler9aee5002022-03-30 16:27:02 +0000435- *"branch":* The branch(es) of the Git tree to clone. Unless
436 "nobranch" is set to "1", this is a mandatory parameter. The number of
437 branch parameters must match the number of name parameters.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500438
439- *"rev":* The revision to use for the checkout. The default is
440 "master".
441
442- *"tag":* Specifies a tag to use for the checkout. To correctly
443 resolve tags, BitBake must access the network. For that reason, tags
444 are often not used. As far as Git is concerned, the "tag" parameter
445 behaves effectively the same as the "rev" parameter.
446
447- *"subpath":* Limits the checkout to a specific subpath of the tree.
448 By default, the whole tree is checked out.
449
450- *"destsuffix":* The name of the path in which to place the checkout.
451 By default, the path is ``git/``.
452
453- *"usehead":* Enables local ``git://`` URLs to use the current branch
454 HEAD as the revision for use with ``AUTOREV``. The "usehead"
455 parameter implies no branch and only works when the transfer protocol
456 is ``file://``.
457
Andrew Geisslerc926e172021-05-07 16:11:35 -0500458Here are some example URLs::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500459
Andrew Geissler9aee5002022-03-30 16:27:02 +0000460 SRC_URI = "git://github.com/fronteed/icheck.git;protocol=https;branch=${PV};tag=${PV}"
461 SRC_URI = "git://github.com/asciidoc/asciidoc-py;protocol=https;branch=main"
462 SRC_URI = "git://git@gitlab.freedesktop.org/mesa/mesa.git;branch=main;protocol=ssh;..."
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500463
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500464.. note::
465
Andrew Geissler5199d832021-09-24 16:47:35 -0500466 When using ``git`` as the fetcher of the main source code of your software,
467 ``S`` should be set accordingly::
468
469 S = "${WORKDIR}/git"
470
471.. note::
472
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500473 Specifying passwords directly in ``git://`` urls is not supported.
Patrick Williams213cb262021-08-07 19:21:33 -0500474 There are several reasons: :term:`SRC_URI` is often written out to logs and
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500475 other places, and that could easily leak passwords; it is also all too
476 easy to share metadata without removing passwords. SSH keys, ``~/.netrc``
477 and ``~/.ssh/config`` files can be used as alternatives.
478
479
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500480.. _gitsm-fetcher:
481
482Git Submodule Fetcher (``gitsm://``)
483------------------------------------
484
485This fetcher submodule inherits from the :ref:`Git
486fetcher<bitbake-user-manual/bitbake-user-manual-fetching:git fetcher
487(\`\`git://\`\`)>` and extends that fetcher's behavior by fetching a
488repository's submodules. :term:`SRC_URI` is passed to the Git fetcher as
489described in the :ref:`bitbake-user-manual/bitbake-user-manual-fetching:git
490fetcher (\`\`git://\`\`)` section.
491
492.. note::
493
494 You must clean a recipe when switching between '``git://``' and
495 '``gitsm://``' URLs.
496
497 The Git Submodules fetcher is not a complete fetcher implementation.
498 The fetcher has known issues where it does not use the normal source
499 mirroring infrastructure properly. Further, the submodule sources it
500 fetches are not visible to the licensing and source archiving
501 infrastructures.
502
503.. _clearcase-fetcher:
504
505ClearCase Fetcher (``ccrc://``)
506-------------------------------
507
508This fetcher submodule fetches code from a
509`ClearCase <http://en.wikipedia.org/wiki/Rational_ClearCase>`__
510repository.
511
512To use this fetcher, make sure your recipe has proper
513:term:`SRC_URI`, :term:`SRCREV`, and
Andrew Geisslerc926e172021-05-07 16:11:35 -0500514:term:`PV` settings. Here is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500515
516 SRC_URI = "ccrc://cc.example.org/ccrc;vob=/example_vob;module=/example_module"
517 SRCREV = "EXAMPLE_CLEARCASE_TAG"
518 PV = "${@d.getVar("SRCREV", False).replace("/", "+")}"
519
520The fetcher uses the ``rcleartool`` or
521``cleartool`` remote client, depending on which one is available.
522
Patrick Williams213cb262021-08-07 19:21:33 -0500523Following are options for the :term:`SRC_URI` statement:
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500524
525- *vob*: The name, which must include the prepending "/" character,
526 of the ClearCase VOB. This option is required.
527
528- *module*: The module, which must include the prepending "/"
529 character, in the selected VOB.
530
531 .. note::
532
533 The module and vob options are combined to create the load rule in the
534 view config spec. As an example, consider the vob and module values from
535 the SRC_URI statement at the start of this section. Combining those values
Andrew Geisslerc926e172021-05-07 16:11:35 -0500536 results in the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500537
538 load /example_vob/example_module
539
540- *proto*: The protocol, which can be either ``http`` or ``https``.
541
542By default, the fetcher creates a configuration specification. If you
543want this specification written to an area other than the default, use
544the ``CCASE_CUSTOM_CONFIG_SPEC`` variable in your recipe to define where
545the specification is written.
546
547.. note::
548
549 the SRCREV loses its functionality if you specify this variable. However,
550 SRCREV is still used to label the archive after a fetch even though it does
551 not define what is fetched.
552
553Here are a couple of other behaviors worth mentioning:
554
555- When using ``cleartool``, the login of ``cleartool`` is handled by
556 the system. The login require no special steps.
557
558- In order to use ``rcleartool`` with authenticated users, an
559 "rcleartool login" is necessary before using the fetcher.
560
561.. _perforce-fetcher:
562
563Perforce Fetcher (``p4://``)
564----------------------------
565
566This fetcher submodule fetches code from the
567`Perforce <https://www.perforce.com/>`__ source control system. The
568executable used is specified by ``FETCHCMD_p4``, which defaults to "p4".
569The fetcher's temporary working directory is set by
570:term:`P4DIR`, which defaults to "DL_DIR/p4".
571The fetcher does not make use of a perforce client, instead it
572relies on ``p4 files`` to retrieve a list of
573files and ``p4 print`` to transfer the content
574of those files locally.
575
576To use this fetcher, make sure your recipe has proper
577:term:`SRC_URI`, :term:`SRCREV`, and
578:term:`PV` values. The p4 executable is able to use the
579config file defined by your system's ``P4CONFIG`` environment variable
580in order to define the Perforce server URL and port, username, and
581password if you do not wish to keep those values in a recipe itself. If
582you choose not to use ``P4CONFIG``, or to explicitly set variables that
583``P4CONFIG`` can contain, you can specify the ``P4PORT`` value, which is
584the server's URL and port number, and you can specify a username and
Patrick Williams213cb262021-08-07 19:21:33 -0500585password directly in your recipe within :term:`SRC_URI`.
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500586
587Here is an example that relies on ``P4CONFIG`` to specify the server URL
Andrew Geisslerc926e172021-05-07 16:11:35 -0500588and port, username, and password, and fetches the Head Revision::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500589
590 SRC_URI = "p4://example-depot/main/source/..."
591 SRCREV = "${AUTOREV}"
592 PV = "p4-${SRCPV}"
593 S = "${WORKDIR}/p4"
594
595Here is an example that specifies the server URL and port, username, and
Andrew Geisslerc926e172021-05-07 16:11:35 -0500596password, and fetches a Revision based on a Label::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500597
598 P4PORT = "tcp:p4server.example.net:1666"
599 SRC_URI = "p4://user:passwd@example-depot/main/source/..."
600 SRCREV = "release-1.0"
601 PV = "p4-${SRCPV}"
602 S = "${WORKDIR}/p4"
603
604.. note::
605
606 You should always set S to "${WORKDIR}/p4" in your recipe.
607
608By default, the fetcher strips the depot location from the local file paths. In
609the above example, the content of ``example-depot/main/source/`` will be placed
610in ``${WORKDIR}/p4``. For situations where preserving parts of the remote depot
611paths locally is desirable, the fetcher supports two parameters:
612
613- *"module":*
614 The top-level depot location or directory to fetch. The value of this
615 parameter can also point to a single file within the depot, in which case
616 the local file path will include the module path.
617- *"remotepath":*
618 When used with the value "``keep``", the fetcher will mirror the full depot
619 paths locally for the specified location, even in combination with the
620 ``module`` parameter.
621
Andrew Geisslerc926e172021-05-07 16:11:35 -0500622Here is an example use of the the ``module`` parameter::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500623
624 SRC_URI = "p4://user:passwd@example-depot/main;module=source/..."
625
626In this case, the content of the top-level directory ``source/`` will be fetched
627to ``${P4DIR}``, including the directory itself. The top-level directory will
628be accesible at ``${P4DIR}/source/``.
629
Andrew Geisslerc926e172021-05-07 16:11:35 -0500630Here is an example use of the the ``remotepath`` parameter::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500631
632 SRC_URI = "p4://user:passwd@example-depot/main;module=source/...;remotepath=keep"
633
634In this case, the content of the top-level directory ``source/`` will be fetched
635to ``${P4DIR}``, but the complete depot paths will be mirrored locally. The
636top-level directory will be accessible at
637``${P4DIR}/example-depot/main/source/``.
638
639.. _repo-fetcher:
640
641Repo Fetcher (``repo://``)
642--------------------------
643
644This fetcher submodule fetches code from ``google-repo`` source control
645system. The fetcher works by initiating and syncing sources of the
646repository into :term:`REPODIR`, which is usually
647``${DL_DIR}/repo``.
648
649This fetcher supports the following parameters:
650
651- *"protocol":* Protocol to fetch the repository manifest (default:
652 git).
653
654- *"branch":* Branch or tag of repository to get (default: master).
655
656- *"manifest":* Name of the manifest file (default: ``default.xml``).
657
Andrew Geisslerc926e172021-05-07 16:11:35 -0500658Here are some example URLs::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500659
660 SRC_URI = "repo://REPOROOT;protocol=git;branch=some_branch;manifest=my_manifest.xml"
661 SRC_URI = "repo://REPOROOT;protocol=file;branch=some_branch;manifest=my_manifest.xml"
662
Andrew Geissler95ac1b82021-03-31 14:34:31 -0500663.. _az-fetcher:
664
665Az Fetcher (``az://``)
666--------------------------
667
668This submodule fetches data from an
669`Azure Storage account <https://docs.microsoft.com/en-us/azure/storage/>`__ ,
670it inherits its functionality from the HTTP wget fetcher, but modifies its
671behavior to accomodate the usage of a
672`Shared Access Signature (SAS) <https://docs.microsoft.com/en-us/azure/storage/common/storage-sas-overview>`__
673for non-public data.
674
675Such functionality is set by the variable:
676
677- :term:`AZ_SAS`: The Azure Storage Shared Access Signature provides secure
678 delegate access to resources, if this variable is set, the Az Fetcher will
679 use it when fetching artifacts from the cloud.
680
Andrew Geisslerc926e172021-05-07 16:11:35 -0500681You can specify the AZ_SAS variable as shown below::
Andrew Geissler95ac1b82021-03-31 14:34:31 -0500682
683 AZ_SAS = "se=2021-01-01&sp=r&sv=2018-11-09&sr=c&skoid=<skoid>&sig=<signature>"
684
Andrew Geisslerc926e172021-05-07 16:11:35 -0500685Here is an example URL::
Andrew Geissler95ac1b82021-03-31 14:34:31 -0500686
687 SRC_URI = "az://<azure-storage-account>.blob.core.windows.net/<foo_container>/<bar_file>"
688
689It can also be used when setting mirrors definitions using the :term:`PREMIRRORS` variable.
690
Andrew Geissler9aee5002022-03-30 16:27:02 +0000691Crate Fetcher (``crate://``)
692----------------------------
693
694This submodule fetches code for
695`Rust language "crates" <https://doc.rust-lang.org/reference/glossary.html?highlight=crate#crate>`__
696corresponding to Rust libraries and programs to compile. Such crates are typically shared
697on https://crates.io/ but this fetcher supports other crate registries too.
698
699The format for the :term:`SRC_URI` setting must be::
700
701 SRC_URI = "crate://REGISTRY/NAME/VERSION"
702
703Here is an example URL::
704
705 SRC_URI = "crate://crates.io/glob/0.2.11"
706
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500707Other Fetchers
708--------------
709
710Fetch submodules also exist for the following:
711
712- Bazaar (``bzr://``)
713
714- Mercurial (``hg://``)
715
716- npm (``npm://``)
717
718- OSC (``osc://``)
719
720- Secure FTP (``sftp://``)
721
722- Secure Shell (``ssh://``)
723
724- Trees using Git Annex (``gitannex://``)
725
726No documentation currently exists for these lesser used fetcher
727submodules. However, you might find the code helpful and readable.
728
729Auto Revisions
730==============
731
Patrick Williams213cb262021-08-07 19:21:33 -0500732We need to document ``AUTOREV`` and :term:`SRCREV_FORMAT` here.