blob: bd1fb4fc74e67ffad61653535fba5555245ba4b2 [file] [log] [blame]
Andrew Geisslerc9f78652020-09-18 14:11:35 -05001.. SPDX-License-Identifier: CC-BY-2.5
2
3=====================
4File Download Support
5=====================
6
7|
8
9BitBake's fetch module is a standalone piece of library code that deals
10with the intricacies of downloading source code and files from remote
11systems. Fetching source code is one of the cornerstones of building
12software. As such, this module forms an important part of BitBake.
13
14The current fetch module is called "fetch2" and refers to the fact that
15it is the second major version of the API. The original version is
16obsolete and has been removed from the codebase. Thus, in all cases,
17"fetch" refers to "fetch2" in this manual.
18
19The Download (Fetch)
20====================
21
22BitBake takes several steps when fetching source code or files. The
23fetcher codebase deals with two distinct processes in order: obtaining
24the files from somewhere (cached or otherwise) and then unpacking those
25files into a specific location and perhaps in a specific way. Getting
26and unpacking the files is often optionally followed by patching.
27Patching, however, is not covered by this module.
28
29The code to execute the first part of this process, a fetch, looks
Andrew Geisslerc926e172021-05-07 16:11:35 -050030something like the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050031
32 src_uri = (d.getVar('SRC_URI') or "").split()
33 fetcher = bb.fetch2.Fetch(src_uri, d)
34 fetcher.download()
35
36This code sets up an instance of the fetch class. The instance uses a
37space-separated list of URLs from the :term:`SRC_URI`
38variable and then calls the ``download`` method to download the files.
39
Andrew Geisslerc926e172021-05-07 16:11:35 -050040The instantiation of the fetch class is usually followed by::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050041
42 rootdir = l.getVar('WORKDIR')
43 fetcher.unpack(rootdir)
44
45This code unpacks the downloaded files to the specified by ``WORKDIR``.
46
47.. note::
48
49 For convenience, the naming in these examples matches the variables
50 used by OpenEmbedded. If you want to see the above code in action,
51 examine the OpenEmbedded class file ``base.bbclass``
52 .
53
54The ``SRC_URI`` and ``WORKDIR`` variables are not hardcoded into the
55fetcher, since those fetcher methods can be (and are) called with
56different variable names. In OpenEmbedded for example, the shared state
57(sstate) code uses the fetch module to fetch the sstate files.
58
59When the ``download()`` method is called, BitBake tries to resolve the
60URLs by looking for source files in a specific search order:
61
62- *Pre-mirror Sites:* BitBake first uses pre-mirrors to try and find
63 source files. These locations are defined using the
64 :term:`PREMIRRORS` variable.
65
66- *Source URI:* If pre-mirrors fail, BitBake uses the original URL (e.g
67 from ``SRC_URI``).
68
69- *Mirror Sites:* If fetch failures occur, BitBake next uses mirror
70 locations as defined by the :term:`MIRRORS` variable.
71
72For each URL passed to the fetcher, the fetcher calls the submodule that
73handles that particular URL type. This behavior can be the source of
74some confusion when you are providing URLs for the ``SRC_URI`` variable.
Andrew Geisslerc926e172021-05-07 16:11:35 -050075Consider the following two URLs::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050076
77 http://git.yoctoproject.org/git/poky;protocol=git
78 git://git.yoctoproject.org/git/poky;protocol=http
79
80In the former case, the URL is passed to the ``wget`` fetcher, which does not
81understand "git". Therefore, the latter case is the correct form since the Git
82fetcher does know how to use HTTP as a transport.
83
Andrew Geisslerc926e172021-05-07 16:11:35 -050084Here are some examples that show commonly used mirror definitions::
Andrew Geisslerc9f78652020-09-18 14:11:35 -050085
86 PREMIRRORS ?= "\
87 bzr://.*/.\* http://somemirror.org/sources/ \\n \
88 cvs://.*/.\* http://somemirror.org/sources/ \\n \
89 git://.*/.\* http://somemirror.org/sources/ \\n \
90 hg://.*/.\* http://somemirror.org/sources/ \\n \
91 osc://.*/.\* http://somemirror.org/sources/ \\n \
92 p4://.*/.\* http://somemirror.org/sources/ \\n \
93 svn://.*/.\* http://somemirror.org/sources/ \\n"
94
95 MIRRORS =+ "\
96 ftp://.*/.\* http://somemirror.org/sources/ \\n \
97 http://.*/.\* http://somemirror.org/sources/ \\n \
98 https://.*/.\* http://somemirror.org/sources/ \\n"
99
100It is useful to note that BitBake
101supports cross-URLs. It is possible to mirror a Git repository on an
102HTTP server as a tarball. This is what the ``git://`` mapping in the
103previous example does.
104
105Since network accesses are slow, BitBake maintains a cache of files
106downloaded from the network. Any source files that are not local (i.e.
107downloaded from the Internet) are placed into the download directory,
108which is specified by the :term:`DL_DIR` variable.
109
110File integrity is of key importance for reproducing builds. For
111non-local archive downloads, the fetcher code can verify SHA-256 and MD5
112checksums to ensure the archives have been downloaded correctly. You can
113specify these checksums by using the ``SRC_URI`` variable with the
Andrew Geisslerc926e172021-05-07 16:11:35 -0500114appropriate varflags as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500115
116 SRC_URI[md5sum] = "value"
117 SRC_URI[sha256sum] = "value"
118
119You can also specify the checksums as
Andrew Geisslerc926e172021-05-07 16:11:35 -0500120parameters on the ``SRC_URI`` as shown below::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500121
122 SRC_URI = "http://example.com/foobar.tar.bz2;md5sum=4a8e0f237e961fd7785d19d07fdb994d"
123
124If multiple URIs exist, you can specify the checksums either directly as
125in the previous example, or you can name the URLs. The following syntax
Andrew Geisslerc926e172021-05-07 16:11:35 -0500126shows how you name the URIs::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500127
128 SRC_URI = "http://example.com/foobar.tar.bz2;name=foo"
129 SRC_URI[foo.md5sum] = 4a8e0f237e961fd7785d19d07fdb994d
130
131After a file has been downloaded and
132has had its checksum checked, a ".done" stamp is placed in ``DL_DIR``.
133BitBake uses this stamp during subsequent builds to avoid downloading or
134comparing a checksum for the file again.
135
136.. note::
137
138 It is assumed that local storage is safe from data corruption. If
139 this were not the case, there would be bigger issues to worry about.
140
141If :term:`BB_STRICT_CHECKSUM` is set, any
142download without a checksum triggers an error message. The
143:term:`BB_NO_NETWORK` variable can be used to
144make any attempted network access a fatal error, which is useful for
145checking that mirrors are complete as well as other things.
146
147.. _bb-the-unpack:
148
149The Unpack
150==========
151
152The unpack process usually immediately follows the download. For all
153URLs except Git URLs, BitBake uses the common ``unpack`` method.
154
155A number of parameters exist that you can specify within the URL to
156govern the behavior of the unpack stage:
157
158- *unpack:* Controls whether the URL components are unpacked. If set to
159 "1", which is the default, the components are unpacked. If set to
160 "0", the unpack stage leaves the file alone. This parameter is useful
161 when you want an archive to be copied in and not be unpacked.
162
163- *dos:* Applies to ``.zip`` and ``.jar`` files and specifies whether
164 to use DOS line ending conversion on text files.
165
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500166- *subdir:* Unpacks the specific URL to the specified subdirectory
167 within the root directory.
168
169The unpack call automatically decompresses and extracts files with ".Z",
170".z", ".gz", ".xz", ".zip", ".jar", ".ipk", ".rpm". ".srpm", ".deb" and
171".bz2" extensions as well as various combinations of tarball extensions.
172
173As mentioned, the Git fetcher has its own unpack method that is
174optimized to work with Git trees. Basically, this method works by
175cloning the tree into the final directory. The process is completed
176using references so that there is only one central copy of the Git
177metadata needed.
178
179.. _bb-fetchers:
180
181Fetchers
182========
183
184As mentioned earlier, the URL prefix determines which fetcher submodule
185BitBake uses. Each submodule can support different URL parameters, which
186are described in the following sections.
187
188.. _local-file-fetcher:
189
190Local file fetcher (``file://``)
191--------------------------------
192
193This submodule handles URLs that begin with ``file://``. The filename
194you specify within the URL can be either an absolute or relative path to
195a file. If the filename is relative, the contents of the
196:term:`FILESPATH` variable is used in the same way
197``PATH`` is used to find executables. If the file cannot be found, it is
198assumed that it is available in :term:`DL_DIR` by the
199time the ``download()`` method is called.
200
201If you specify a directory, the entire directory is unpacked.
202
203Here are a couple of example URLs, the first relative and the second
Andrew Geisslerc926e172021-05-07 16:11:35 -0500204absolute::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500205
206 SRC_URI = "file://relativefile.patch"
207 SRC_URI = "file:///Users/ich/very_important_software"
208
209.. _http-ftp-fetcher:
210
211HTTP/FTP wget fetcher (``http://``, ``ftp://``, ``https://``)
212-------------------------------------------------------------
213
214This fetcher obtains files from web and FTP servers. Internally, the
215fetcher uses the wget utility.
216
217The executable and parameters used are specified by the
218``FETCHCMD_wget`` variable, which defaults to sensible values. The
219fetcher supports a parameter "downloadfilename" that allows the name of
220the downloaded file to be specified. Specifying the name of the
221downloaded file is useful for avoiding collisions in
222:term:`DL_DIR` when dealing with multiple files that
223have the same name.
224
Andrew Geisslerc926e172021-05-07 16:11:35 -0500225Some example URLs are as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500226
227 SRC_URI = "http://oe.handhelds.org/not_there.aac"
228 SRC_URI = "ftp://oe.handhelds.org/not_there_as_well.aac"
229 SRC_URI = "ftp://you@oe.handhelds.org/home/you/secret.plan"
230
231.. note::
232
233 Because URL parameters are delimited by semi-colons, this can
234 introduce ambiguity when parsing URLs that also contain semi-colons,
Andrew Geisslerc926e172021-05-07 16:11:35 -0500235 for example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500236
237 SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git;a=snapshot;h=a5dd47"
238
239
240 Such URLs should should be modified by replacing semi-colons with '&'
Andrew Geisslerc926e172021-05-07 16:11:35 -0500241 characters::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500242
243 SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47"
244
245
246 In most cases this should work. Treating semi-colons and '&' in
247 queries identically is recommended by the World Wide Web Consortium
248 (W3C). Note that due to the nature of the URL, you may have to
Andrew Geisslerc926e172021-05-07 16:11:35 -0500249 specify the name of the downloaded file as well::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500250
251 SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47;downloadfilename=myfile.bz2"
252
253
254.. _cvs-fetcher:
255
256CVS fetcher (``(cvs://``)
257-------------------------
258
259This submodule handles checking out files from the CVS version control
260system. You can configure it using a number of different variables:
261
262- :term:`FETCHCMD_cvs <FETCHCMD>`: The name of the executable to use when running
263 the ``cvs`` command. This name is usually "cvs".
264
265- :term:`SRCDATE`: The date to use when fetching the CVS source code. A
266 special value of "now" causes the checkout to be updated on every
267 build.
268
269- :term:`CVSDIR`: Specifies where a temporary
270 checkout is saved. The location is often ``DL_DIR/cvs``.
271
272- CVS_PROXY_HOST: The name to use as a "proxy=" parameter to the
273 ``cvs`` command.
274
275- CVS_PROXY_PORT: The port number to use as a "proxyport="
276 parameter to the ``cvs`` command.
277
278As well as the standard username and password URL syntax, you can also
279configure the fetcher with various URL parameters:
280
281The supported parameters are as follows:
282
283- *"method":* The protocol over which to communicate with the CVS
284 server. By default, this protocol is "pserver". If "method" is set to
285 "ext", BitBake examines the "rsh" parameter and sets ``CVS_RSH``. You
286 can use "dir" for local directories.
287
288- *"module":* Specifies the module to check out. You must supply this
289 parameter.
290
291- *"tag":* Describes which CVS TAG should be used for the checkout. By
292 default, the TAG is empty.
293
294- *"date":* Specifies a date. If no "date" is specified, the
295 :term:`SRCDATE` of the configuration is used to
296 checkout a specific date. The special value of "now" causes the
297 checkout to be updated on every build.
298
299- *"localdir":* Used to rename the module. Effectively, you are
300 renaming the output directory to which the module is unpacked. You
301 are forcing the module into a special directory relative to
302 :term:`CVSDIR`.
303
304- *"rsh":* Used in conjunction with the "method" parameter.
305
306- *"scmdata":* Causes the CVS metadata to be maintained in the tarball
307 the fetcher creates when set to "keep". The tarball is expanded into
308 the work directory. By default, the CVS metadata is removed.
309
310- *"fullpath":* Controls whether the resulting checkout is at the
311 module level, which is the default, or is at deeper paths.
312
313- *"norecurse":* Causes the fetcher to only checkout the specified
314 directory with no recurse into any subdirectories.
315
316- *"port":* The port to which the CVS server connects.
317
Andrew Geisslerc926e172021-05-07 16:11:35 -0500318Some example URLs are as follows::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500319
320 SRC_URI = "cvs://CVSROOT;module=mymodule;tag=some-version;method=ext"
321 SRC_URI = "cvs://CVSROOT;module=mymodule;date=20060126;localdir=usethat"
322
323.. _svn-fetcher:
324
325Subversion (SVN) Fetcher (``svn://``)
326-------------------------------------
327
328This fetcher submodule fetches code from the Subversion source control
329system. The executable used is specified by ``FETCHCMD_svn``, which
330defaults to "svn". The fetcher's temporary working directory is set by
331:term:`SVNDIR`, which is usually ``DL_DIR/svn``.
332
333The supported parameters are as follows:
334
335- *"module":* The name of the svn module to checkout. You must provide
336 this parameter. You can think of this parameter as the top-level
337 directory of the repository data you want.
338
339- *"path_spec":* A specific directory in which to checkout the
340 specified svn module.
341
342- *"protocol":* The protocol to use, which defaults to "svn". If
343 "protocol" is set to "svn+ssh", the "ssh" parameter is also used.
344
345- *"rev":* The revision of the source code to checkout.
346
Andrew Geisslerf0343792020-11-18 10:42:21 -0600347- *"scmdata":* Causes the ".svn" directories to be available during
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500348 compile-time when set to "keep". By default, these directories are
349 removed.
350
351- *"ssh":* An optional parameter used when "protocol" is set to
352 "svn+ssh". You can use this parameter to specify the ssh program used
353 by svn.
354
355- *"transportuser":* When required, sets the username for the
356 transport. By default, this parameter is empty. The transport
357 username is different than the username used in the main URL, which
358 is passed to the subversion command.
359
Andrew Geisslerc926e172021-05-07 16:11:35 -0500360Following are three examples using svn::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500361
362 SRC_URI = "svn://myrepos/proj1;module=vip;protocol=http;rev=667"
363 SRC_URI = "svn://myrepos/proj1;module=opie;protocol=svn+ssh"
364 SRC_URI = "svn://myrepos/proj1;module=trunk;protocol=http;path_spec=${MY_DIR}/proj1"
365
366.. _git-fetcher:
367
368Git Fetcher (``git://``)
369------------------------
370
371This fetcher submodule fetches code from the Git source control system.
372The fetcher works by creating a bare clone of the remote into
373:term:`GITDIR`, which is usually ``DL_DIR/git2``. This
374bare clone is then cloned into the work directory during the unpack
375stage when a specific tree is checked out. This is done using alternates
376and by reference to minimize the amount of duplicate data on the disk
377and make the unpack process fast. The executable used can be set with
378``FETCHCMD_git``.
379
380This fetcher supports the following parameters:
381
382- *"protocol":* The protocol used to fetch the files. The default is
383 "git" when a hostname is set. If a hostname is not set, the Git
384 protocol is "file". You can also use "http", "https", "ssh" and
385 "rsync".
386
387- *"nocheckout":* Tells the fetcher to not checkout source code when
388 unpacking when set to "1". Set this option for the URL where there is
389 a custom routine to checkout code. The default is "0".
390
391- *"rebaseable":* Indicates that the upstream Git repository can be
392 rebased. You should set this parameter to "1" if revisions can become
393 detached from branches. In this case, the source mirror tarball is
394 done per revision, which has a loss of efficiency. Rebasing the
395 upstream Git repository could cause the current revision to disappear
396 from the upstream repository. This option reminds the fetcher to
397 preserve the local cache carefully for future use. The default value
398 for this parameter is "0".
399
400- *"nobranch":* Tells the fetcher to not check the SHA validation for
401 the branch when set to "1". The default is "0". Set this option for
402 the recipe that refers to the commit that is valid for a tag instead
403 of the branch.
404
405- *"bareclone":* Tells the fetcher to clone a bare clone into the
406 destination directory without checking out a working tree. Only the
407 raw Git metadata is provided. This parameter implies the "nocheckout"
408 parameter as well.
409
410- *"branch":* The branch(es) of the Git tree to clone. If unset, this
411 is assumed to be "master". The number of branch parameters much match
412 the number of name parameters.
413
414- *"rev":* The revision to use for the checkout. The default is
415 "master".
416
417- *"tag":* Specifies a tag to use for the checkout. To correctly
418 resolve tags, BitBake must access the network. For that reason, tags
419 are often not used. As far as Git is concerned, the "tag" parameter
420 behaves effectively the same as the "rev" parameter.
421
422- *"subpath":* Limits the checkout to a specific subpath of the tree.
423 By default, the whole tree is checked out.
424
425- *"destsuffix":* The name of the path in which to place the checkout.
426 By default, the path is ``git/``.
427
428- *"usehead":* Enables local ``git://`` URLs to use the current branch
429 HEAD as the revision for use with ``AUTOREV``. The "usehead"
430 parameter implies no branch and only works when the transfer protocol
431 is ``file://``.
432
Andrew Geisslerc926e172021-05-07 16:11:35 -0500433Here are some example URLs::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500434
435 SRC_URI = "git://git.oe.handhelds.org/git/vip.git;tag=version-1"
436 SRC_URI = "git://git.oe.handhelds.org/git/vip.git;protocol=http"
437
Andrew Geissler3b8a17c2021-04-15 15:55:55 -0500438.. note::
439
440 Specifying passwords directly in ``git://`` urls is not supported.
441 There are several reasons: ``SRC_URI`` is often written out to logs and
442 other places, and that could easily leak passwords; it is also all too
443 easy to share metadata without removing passwords. SSH keys, ``~/.netrc``
444 and ``~/.ssh/config`` files can be used as alternatives.
445
446
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500447.. _gitsm-fetcher:
448
449Git Submodule Fetcher (``gitsm://``)
450------------------------------------
451
452This fetcher submodule inherits from the :ref:`Git
453fetcher<bitbake-user-manual/bitbake-user-manual-fetching:git fetcher
454(\`\`git://\`\`)>` and extends that fetcher's behavior by fetching a
455repository's submodules. :term:`SRC_URI` is passed to the Git fetcher as
456described in the :ref:`bitbake-user-manual/bitbake-user-manual-fetching:git
457fetcher (\`\`git://\`\`)` section.
458
459.. note::
460
461 You must clean a recipe when switching between '``git://``' and
462 '``gitsm://``' URLs.
463
464 The Git Submodules fetcher is not a complete fetcher implementation.
465 The fetcher has known issues where it does not use the normal source
466 mirroring infrastructure properly. Further, the submodule sources it
467 fetches are not visible to the licensing and source archiving
468 infrastructures.
469
470.. _clearcase-fetcher:
471
472ClearCase Fetcher (``ccrc://``)
473-------------------------------
474
475This fetcher submodule fetches code from a
476`ClearCase <http://en.wikipedia.org/wiki/Rational_ClearCase>`__
477repository.
478
479To use this fetcher, make sure your recipe has proper
480:term:`SRC_URI`, :term:`SRCREV`, and
Andrew Geisslerc926e172021-05-07 16:11:35 -0500481:term:`PV` settings. Here is an example::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500482
483 SRC_URI = "ccrc://cc.example.org/ccrc;vob=/example_vob;module=/example_module"
484 SRCREV = "EXAMPLE_CLEARCASE_TAG"
485 PV = "${@d.getVar("SRCREV", False).replace("/", "+")}"
486
487The fetcher uses the ``rcleartool`` or
488``cleartool`` remote client, depending on which one is available.
489
490Following are options for the ``SRC_URI`` statement:
491
492- *vob*: The name, which must include the prepending "/" character,
493 of the ClearCase VOB. This option is required.
494
495- *module*: The module, which must include the prepending "/"
496 character, in the selected VOB.
497
498 .. note::
499
500 The module and vob options are combined to create the load rule in the
501 view config spec. As an example, consider the vob and module values from
502 the SRC_URI statement at the start of this section. Combining those values
Andrew Geisslerc926e172021-05-07 16:11:35 -0500503 results in the following::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500504
505 load /example_vob/example_module
506
507- *proto*: The protocol, which can be either ``http`` or ``https``.
508
509By default, the fetcher creates a configuration specification. If you
510want this specification written to an area other than the default, use
511the ``CCASE_CUSTOM_CONFIG_SPEC`` variable in your recipe to define where
512the specification is written.
513
514.. note::
515
516 the SRCREV loses its functionality if you specify this variable. However,
517 SRCREV is still used to label the archive after a fetch even though it does
518 not define what is fetched.
519
520Here are a couple of other behaviors worth mentioning:
521
522- When using ``cleartool``, the login of ``cleartool`` is handled by
523 the system. The login require no special steps.
524
525- In order to use ``rcleartool`` with authenticated users, an
526 "rcleartool login" is necessary before using the fetcher.
527
528.. _perforce-fetcher:
529
530Perforce Fetcher (``p4://``)
531----------------------------
532
533This fetcher submodule fetches code from the
534`Perforce <https://www.perforce.com/>`__ source control system. The
535executable used is specified by ``FETCHCMD_p4``, which defaults to "p4".
536The fetcher's temporary working directory is set by
537:term:`P4DIR`, which defaults to "DL_DIR/p4".
538The fetcher does not make use of a perforce client, instead it
539relies on ``p4 files`` to retrieve a list of
540files and ``p4 print`` to transfer the content
541of those files locally.
542
543To use this fetcher, make sure your recipe has proper
544:term:`SRC_URI`, :term:`SRCREV`, and
545:term:`PV` values. The p4 executable is able to use the
546config file defined by your system's ``P4CONFIG`` environment variable
547in order to define the Perforce server URL and port, username, and
548password if you do not wish to keep those values in a recipe itself. If
549you choose not to use ``P4CONFIG``, or to explicitly set variables that
550``P4CONFIG`` can contain, you can specify the ``P4PORT`` value, which is
551the server's URL and port number, and you can specify a username and
552password directly in your recipe within ``SRC_URI``.
553
554Here is an example that relies on ``P4CONFIG`` to specify the server URL
Andrew Geisslerc926e172021-05-07 16:11:35 -0500555and port, username, and password, and fetches the Head Revision::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500556
557 SRC_URI = "p4://example-depot/main/source/..."
558 SRCREV = "${AUTOREV}"
559 PV = "p4-${SRCPV}"
560 S = "${WORKDIR}/p4"
561
562Here is an example that specifies the server URL and port, username, and
Andrew Geisslerc926e172021-05-07 16:11:35 -0500563password, and fetches a Revision based on a Label::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500564
565 P4PORT = "tcp:p4server.example.net:1666"
566 SRC_URI = "p4://user:passwd@example-depot/main/source/..."
567 SRCREV = "release-1.0"
568 PV = "p4-${SRCPV}"
569 S = "${WORKDIR}/p4"
570
571.. note::
572
573 You should always set S to "${WORKDIR}/p4" in your recipe.
574
575By default, the fetcher strips the depot location from the local file paths. In
576the above example, the content of ``example-depot/main/source/`` will be placed
577in ``${WORKDIR}/p4``. For situations where preserving parts of the remote depot
578paths locally is desirable, the fetcher supports two parameters:
579
580- *"module":*
581 The top-level depot location or directory to fetch. The value of this
582 parameter can also point to a single file within the depot, in which case
583 the local file path will include the module path.
584- *"remotepath":*
585 When used with the value "``keep``", the fetcher will mirror the full depot
586 paths locally for the specified location, even in combination with the
587 ``module`` parameter.
588
Andrew Geisslerc926e172021-05-07 16:11:35 -0500589Here is an example use of the the ``module`` parameter::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500590
591 SRC_URI = "p4://user:passwd@example-depot/main;module=source/..."
592
593In this case, the content of the top-level directory ``source/`` will be fetched
594to ``${P4DIR}``, including the directory itself. The top-level directory will
595be accesible at ``${P4DIR}/source/``.
596
Andrew Geisslerc926e172021-05-07 16:11:35 -0500597Here is an example use of the the ``remotepath`` parameter::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500598
599 SRC_URI = "p4://user:passwd@example-depot/main;module=source/...;remotepath=keep"
600
601In this case, the content of the top-level directory ``source/`` will be fetched
602to ``${P4DIR}``, but the complete depot paths will be mirrored locally. The
603top-level directory will be accessible at
604``${P4DIR}/example-depot/main/source/``.
605
606.. _repo-fetcher:
607
608Repo Fetcher (``repo://``)
609--------------------------
610
611This fetcher submodule fetches code from ``google-repo`` source control
612system. The fetcher works by initiating and syncing sources of the
613repository into :term:`REPODIR`, which is usually
614``${DL_DIR}/repo``.
615
616This fetcher supports the following parameters:
617
618- *"protocol":* Protocol to fetch the repository manifest (default:
619 git).
620
621- *"branch":* Branch or tag of repository to get (default: master).
622
623- *"manifest":* Name of the manifest file (default: ``default.xml``).
624
Andrew Geisslerc926e172021-05-07 16:11:35 -0500625Here are some example URLs::
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500626
627 SRC_URI = "repo://REPOROOT;protocol=git;branch=some_branch;manifest=my_manifest.xml"
628 SRC_URI = "repo://REPOROOT;protocol=file;branch=some_branch;manifest=my_manifest.xml"
629
Andrew Geissler95ac1b82021-03-31 14:34:31 -0500630.. _az-fetcher:
631
632Az Fetcher (``az://``)
633--------------------------
634
635This submodule fetches data from an
636`Azure Storage account <https://docs.microsoft.com/en-us/azure/storage/>`__ ,
637it inherits its functionality from the HTTP wget fetcher, but modifies its
638behavior to accomodate the usage of a
639`Shared Access Signature (SAS) <https://docs.microsoft.com/en-us/azure/storage/common/storage-sas-overview>`__
640for non-public data.
641
642Such functionality is set by the variable:
643
644- :term:`AZ_SAS`: The Azure Storage Shared Access Signature provides secure
645 delegate access to resources, if this variable is set, the Az Fetcher will
646 use it when fetching artifacts from the cloud.
647
Andrew Geisslerc926e172021-05-07 16:11:35 -0500648You can specify the AZ_SAS variable as shown below::
Andrew Geissler95ac1b82021-03-31 14:34:31 -0500649
650 AZ_SAS = "se=2021-01-01&sp=r&sv=2018-11-09&sr=c&skoid=<skoid>&sig=<signature>"
651
Andrew Geisslerc926e172021-05-07 16:11:35 -0500652Here is an example URL::
Andrew Geissler95ac1b82021-03-31 14:34:31 -0500653
654 SRC_URI = "az://<azure-storage-account>.blob.core.windows.net/<foo_container>/<bar_file>"
655
656It can also be used when setting mirrors definitions using the :term:`PREMIRRORS` variable.
657
Andrew Geisslerc9f78652020-09-18 14:11:35 -0500658Other Fetchers
659--------------
660
661Fetch submodules also exist for the following:
662
663- Bazaar (``bzr://``)
664
665- Mercurial (``hg://``)
666
667- npm (``npm://``)
668
669- OSC (``osc://``)
670
671- Secure FTP (``sftp://``)
672
673- Secure Shell (``ssh://``)
674
675- Trees using Git Annex (``gitannex://``)
676
677No documentation currently exists for these lesser used fetcher
678submodules. However, you might find the code helpful and readable.
679
680Auto Revisions
681==============
682
683We need to document ``AUTOREV`` and ``SRCREV_FORMAT`` here.