Revert "poky: subtree update:b23aa6b753..ad30a6d470"
This reverts commit af5e4ef732faedf66c6dc1756432e9de2ac72988.
This commit introduced openbmc/openbmc#3720 and no solution has been
forthcoming. Revert until we can get to the bottom of this.
Change-Id: I2fb0d81eb26cf3dadb2f2abdd1a1bb7a95eaf03c
diff --git a/poky/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml b/poky/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml
new file mode 100644
index 0000000..fe4372a
--- /dev/null
+++ b/poky/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml
@@ -0,0 +1,928 @@
+<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
+"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">
+
+<chapter>
+<title>File Download Support</title>
+
+ <para>
+ BitBake's fetch module is a standalone piece of library code
+ that deals with the intricacies of downloading source code
+ and files from remote systems.
+ Fetching source code is one of the cornerstones of building software.
+ As such, this module forms an important part of BitBake.
+ </para>
+
+ <para>
+ The current fetch module is called "fetch2" and refers to the
+ fact that it is the second major version of the API.
+ The original version is obsolete and has been removed from the codebase.
+ Thus, in all cases, "fetch" refers to "fetch2" in this
+ manual.
+ </para>
+
+ <section id='the-download-fetch'>
+ <title>The Download (Fetch)</title>
+
+ <para>
+ BitBake takes several steps when fetching source code or files.
+ The fetcher codebase deals with two distinct processes in order:
+ obtaining the files from somewhere (cached or otherwise)
+ and then unpacking those files into a specific location and
+ perhaps in a specific way.
+ Getting and unpacking the files is often optionally followed
+ by patching.
+ Patching, however, is not covered by this module.
+ </para>
+
+ <para>
+ The code to execute the first part of this process, a fetch,
+ looks something like the following:
+ <literallayout class='monospaced'>
+ src_uri = (d.getVar('SRC_URI') or "").split()
+ fetcher = bb.fetch2.Fetch(src_uri, d)
+ fetcher.download()
+ </literallayout>
+ This code sets up an instance of the fetch class.
+ The instance uses a space-separated list of URLs from the
+ <link linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link>
+ variable and then calls the <filename>download</filename>
+ method to download the files.
+ </para>
+
+ <para>
+ The instantiation of the fetch class is usually followed by:
+ <literallayout class='monospaced'>
+ rootdir = l.getVar('WORKDIR')
+ fetcher.unpack(rootdir)
+ </literallayout>
+ This code unpacks the downloaded files to the
+ specified by <filename>WORKDIR</filename>.
+ <note>
+ For convenience, the naming in these examples matches
+ the variables used by OpenEmbedded.
+ If you want to see the above code in action, examine
+ the OpenEmbedded class file <filename>base.bbclass</filename>.
+ </note>
+ The <filename>SRC_URI</filename> and <filename>WORKDIR</filename>
+ variables are not hardcoded into the fetcher, since those fetcher
+ methods can be (and are) called with different variable names.
+ In OpenEmbedded for example, the shared state (sstate) code uses
+ the fetch module to fetch the sstate files.
+ </para>
+
+ <para>
+ When the <filename>download()</filename> method is called,
+ BitBake tries to resolve the URLs by looking for source files
+ in a specific search order:
+ <itemizedlist>
+ <listitem><para><emphasis>Pre-mirror Sites:</emphasis>
+ BitBake first uses pre-mirrors to try and find source files.
+ These locations are defined using the
+ <link linkend='var-bb-PREMIRRORS'><filename>PREMIRRORS</filename></link>
+ variable.
+ </para></listitem>
+ <listitem><para><emphasis>Source URI:</emphasis>
+ If pre-mirrors fail, BitBake uses the original URL (e.g from
+ <filename>SRC_URI</filename>).
+ </para></listitem>
+ <listitem><para><emphasis>Mirror Sites:</emphasis>
+ If fetch failures occur, BitBake next uses mirror locations as
+ defined by the
+ <link linkend='var-bb-MIRRORS'><filename>MIRRORS</filename></link>
+ variable.
+ </para></listitem>
+ </itemizedlist>
+ </para>
+
+ <para>
+ For each URL passed to the fetcher, the fetcher
+ calls the submodule that handles that particular URL type.
+ This behavior can be the source of some confusion when you
+ are providing URLs for the <filename>SRC_URI</filename>
+ variable.
+ Consider the following two URLs:
+ <literallayout class='monospaced'>
+ http://git.yoctoproject.org/git/poky;protocol=git
+ git://git.yoctoproject.org/git/poky;protocol=http
+ </literallayout>
+ In the former case, the URL is passed to the
+ <filename>wget</filename> fetcher, which does not
+ understand "git".
+ Therefore, the latter case is the correct form since the
+ Git fetcher does know how to use HTTP as a transport.
+ </para>
+
+ <para>
+ Here are some examples that show commonly used mirror
+ definitions:
+ <literallayout class='monospaced'>
+ PREMIRRORS ?= "\
+ bzr://.*/.* http://somemirror.org/sources/ \n \
+ cvs://.*/.* http://somemirror.org/sources/ \n \
+ git://.*/.* http://somemirror.org/sources/ \n \
+ hg://.*/.* http://somemirror.org/sources/ \n \
+ osc://.*/.* http://somemirror.org/sources/ \n \
+ p4://.*/.* http://somemirror.org/sources/ \n \
+ svn://.*/.* http://somemirror.org/sources/ \n"
+
+ MIRRORS =+ "\
+ ftp://.*/.* http://somemirror.org/sources/ \n \
+ http://.*/.* http://somemirror.org/sources/ \n \
+ https://.*/.* http://somemirror.org/sources/ \n"
+ </literallayout>
+ It is useful to note that BitBake supports
+ cross-URLs.
+ It is possible to mirror a Git repository on an HTTP
+ server as a tarball.
+ This is what the <filename>git://</filename> mapping in
+ the previous example does.
+ </para>
+
+ <para>
+ Since network accesses are slow, BitBake maintains a
+ cache of files downloaded from the network.
+ Any source files that are not local (i.e.
+ downloaded from the Internet) are placed into the download
+ directory, which is specified by the
+ <link linkend='var-bb-DL_DIR'><filename>DL_DIR</filename></link>
+ variable.
+ </para>
+
+ <para>
+ File integrity is of key importance for reproducing builds.
+ For non-local archive downloads, the fetcher code can verify
+ SHA-256 and MD5 checksums to ensure the archives have been
+ downloaded correctly.
+ You can specify these checksums by using the
+ <filename>SRC_URI</filename> variable with the appropriate
+ varflags as follows:
+ <literallayout class='monospaced'>
+ SRC_URI[md5sum] = "<replaceable>value</replaceable>"
+ SRC_URI[sha256sum] = "<replaceable>value</replaceable>"
+ </literallayout>
+ You can also specify the checksums as parameters on the
+ <filename>SRC_URI</filename> as shown below:
+ <literallayout class='monospaced'>
+ SRC_URI = "http://example.com/foobar.tar.bz2;md5sum=4a8e0f237e961fd7785d19d07fdb994d"
+ </literallayout>
+ If multiple URIs exist, you can specify the checksums either
+ directly as in the previous example, or you can name the URLs.
+ The following syntax shows how you name the URIs:
+ <literallayout class='monospaced'>
+ SRC_URI = "http://example.com/foobar.tar.bz2;name=foo"
+ SRC_URI[foo.md5sum] = 4a8e0f237e961fd7785d19d07fdb994d
+ </literallayout>
+ After a file has been downloaded and has had its checksum checked,
+ a ".done" stamp is placed in <filename>DL_DIR</filename>.
+ BitBake uses this stamp during subsequent builds to avoid
+ downloading or comparing a checksum for the file again.
+ <note>
+ It is assumed that local storage is safe from data corruption.
+ If this were not the case, there would be bigger issues to worry about.
+ </note>
+ </para>
+
+ <para>
+ If
+ <link linkend='var-bb-BB_STRICT_CHECKSUM'><filename>BB_STRICT_CHECKSUM</filename></link>
+ is set, any download without a checksum triggers an
+ error message.
+ The
+ <link linkend='var-bb-BB_NO_NETWORK'><filename>BB_NO_NETWORK</filename></link>
+ variable can be used to make any attempted network access a fatal
+ error, which is useful for checking that mirrors are complete
+ as well as other things.
+ </para>
+ </section>
+
+ <section id='bb-the-unpack'>
+ <title>The Unpack</title>
+
+ <para>
+ The unpack process usually immediately follows the download.
+ For all URLs except Git URLs, BitBake uses the common
+ <filename>unpack</filename> method.
+ </para>
+
+ <para>
+ A number of parameters exist that you can specify within the
+ URL to govern the behavior of the unpack stage:
+ <itemizedlist>
+ <listitem><para><emphasis>unpack:</emphasis>
+ Controls whether the URL components are unpacked.
+ If set to "1", which is the default, the components
+ are unpacked.
+ If set to "0", the unpack stage leaves the file alone.
+ This parameter is useful when you want an archive to be
+ copied in and not be unpacked.
+ </para></listitem>
+ <listitem><para><emphasis>dos:</emphasis>
+ Applies to <filename>.zip</filename> and
+ <filename>.jar</filename> files and specifies whether to
+ use DOS line ending conversion on text files.
+ </para></listitem>
+ <listitem><para><emphasis>basepath:</emphasis>
+ Instructs the unpack stage to strip the specified
+ directories from the source path when unpacking.
+ </para></listitem>
+ <listitem><para><emphasis>subdir:</emphasis>
+ Unpacks the specific URL to the specified subdirectory
+ within the root directory.
+ </para></listitem>
+ </itemizedlist>
+ The unpack call automatically decompresses and extracts files
+ with ".Z", ".z", ".gz", ".xz", ".zip", ".jar", ".ipk", ".rpm".
+ ".srpm", ".deb" and ".bz2" extensions as well as various combinations
+ of tarball extensions.
+ </para>
+
+ <para>
+ As mentioned, the Git fetcher has its own unpack method that
+ is optimized to work with Git trees.
+ Basically, this method works by cloning the tree into the final
+ directory.
+ The process is completed using references so that there is
+ only one central copy of the Git metadata needed.
+ </para>
+ </section>
+
+ <section id='bb-fetchers'>
+ <title>Fetchers</title>
+
+ <para>
+ As mentioned earlier, the URL prefix determines which
+ fetcher submodule BitBake uses.
+ Each submodule can support different URL parameters,
+ which are described in the following sections.
+ </para>
+
+ <section id='local-file-fetcher'>
+ <title>Local file fetcher (<filename>file://</filename>)</title>
+
+ <para>
+ This submodule handles URLs that begin with
+ <filename>file://</filename>.
+ The filename you specify within the URL can be
+ either an absolute or relative path to a file.
+ If the filename is relative, the contents of the
+ <link linkend='var-bb-FILESPATH'><filename>FILESPATH</filename></link>
+ variable is used in the same way
+ <filename>PATH</filename> is used to find executables.
+ If the file cannot be found, it is assumed that it is available in
+ <link linkend='var-bb-DL_DIR'><filename>DL_DIR</filename></link>
+ by the time the <filename>download()</filename> method is called.
+ </para>
+
+ <para>
+ If you specify a directory, the entire directory is
+ unpacked.
+ </para>
+
+ <para>
+ Here are a couple of example URLs, the first relative and
+ the second absolute:
+ <literallayout class='monospaced'>
+ SRC_URI = "file://relativefile.patch"
+ SRC_URI = "file:///Users/ich/very_important_software"
+ </literallayout>
+ </para>
+ </section>
+
+ <section id='http-ftp-fetcher'>
+ <title>HTTP/FTP wget fetcher (<filename>http://</filename>, <filename>ftp://</filename>, <filename>https://</filename>)</title>
+
+ <para>
+ This fetcher obtains files from web and FTP servers.
+ Internally, the fetcher uses the wget utility.
+ </para>
+
+ <para>
+ The executable and parameters used are specified by the
+ <filename>FETCHCMD_wget</filename> variable, which defaults
+ to sensible values.
+ The fetcher supports a parameter "downloadfilename" that
+ allows the name of the downloaded file to be specified.
+ Specifying the name of the downloaded file is useful
+ for avoiding collisions in
+ <link linkend='var-bb-DL_DIR'><filename>DL_DIR</filename></link>
+ when dealing with multiple files that have the same name.
+ </para>
+
+ <para>
+ Some example URLs are as follows:
+ <literallayout class='monospaced'>
+ SRC_URI = "http://oe.handhelds.org/not_there.aac"
+ SRC_URI = "ftp://oe.handhelds.org/not_there_as_well.aac"
+ SRC_URI = "ftp://you@oe.handhelds.org/home/you/secret.plan"
+ </literallayout>
+ </para>
+ <note>
+ Because URL parameters are delimited by semi-colons, this can
+ introduce ambiguity when parsing URLs that also contain semi-colons,
+ for example:
+ <literallayout class='monospaced'>
+ SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git;a=snapshot;h=a5dd47"
+ </literallayout>
+ Such URLs should should be modified by replacing semi-colons with '&' characters:
+ <literallayout class='monospaced'>
+ SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47"
+ </literallayout>
+ In most cases this should work. Treating semi-colons and '&' in queries
+ identically is recommended by the World Wide Web Consortium (W3C).
+ Note that due to the nature of the URL, you may have to specify the name
+ of the downloaded file as well:
+ <literallayout class='monospaced'>
+ SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47;downloadfilename=myfile.bz2"
+ </literallayout>
+ </note>
+ </section>
+
+ <section id='cvs-fetcher'>
+ <title>CVS fetcher (<filename>(cvs://</filename>)</title>
+
+ <para>
+ This submodule handles checking out files from the
+ CVS version control system.
+ You can configure it using a number of different variables:
+ <itemizedlist>
+ <listitem><para><emphasis><filename>FETCHCMD_cvs</filename>:</emphasis>
+ The name of the executable to use when running
+ the <filename>cvs</filename> command.
+ This name is usually "cvs".
+ </para></listitem>
+ <listitem><para><emphasis><filename>SRCDATE</filename>:</emphasis>
+ The date to use when fetching the CVS source code.
+ A special value of "now" causes the checkout to
+ be updated on every build.
+ </para></listitem>
+ <listitem><para><emphasis><link linkend='var-bb-CVSDIR'><filename>CVSDIR</filename></link>:</emphasis>
+ Specifies where a temporary checkout is saved.
+ The location is often <filename>DL_DIR/cvs</filename>.
+ </para></listitem>
+ <listitem><para><emphasis><filename>CVS_PROXY_HOST</filename>:</emphasis>
+ The name to use as a "proxy=" parameter to the
+ <filename>cvs</filename> command.
+ </para></listitem>
+ <listitem><para><emphasis><filename>CVS_PROXY_PORT</filename>:</emphasis>
+ The port number to use as a "proxyport=" parameter to
+ the <filename>cvs</filename> command.
+ </para></listitem>
+ </itemizedlist>
+ As well as the standard username and password URL syntax,
+ you can also configure the fetcher with various URL parameters:
+ </para>
+
+ <para>
+ The supported parameters are as follows:
+ <itemizedlist>
+ <listitem><para><emphasis>"method":</emphasis>
+ The protocol over which to communicate with the CVS
+ server.
+ By default, this protocol is "pserver".
+ If "method" is set to "ext", BitBake examines the
+ "rsh" parameter and sets <filename>CVS_RSH</filename>.
+ You can use "dir" for local directories.
+ </para></listitem>
+ <listitem><para><emphasis>"module":</emphasis>
+ Specifies the module to check out.
+ You must supply this parameter.
+ </para></listitem>
+ <listitem><para><emphasis>"tag":</emphasis>
+ Describes which CVS TAG should be used for
+ the checkout.
+ By default, the TAG is empty.
+ </para></listitem>
+ <listitem><para><emphasis>"date":</emphasis>
+ Specifies a date.
+ If no "date" is specified, the
+ <link linkend='var-bb-SRCDATE'><filename>SRCDATE</filename></link>
+ of the configuration is used to checkout a specific date.
+ The special value of "now" causes the checkout to be
+ updated on every build.
+ </para></listitem>
+ <listitem><para><emphasis>"localdir":</emphasis>
+ Used to rename the module.
+ Effectively, you are renaming the output directory
+ to which the module is unpacked.
+ You are forcing the module into a special
+ directory relative to
+ <link linkend='var-bb-CVSDIR'><filename>CVSDIR</filename></link>.
+ </para></listitem>
+ <listitem><para><emphasis>"rsh"</emphasis>
+ Used in conjunction with the "method" parameter.
+ </para></listitem>
+ <listitem><para><emphasis>"scmdata":</emphasis>
+ Causes the CVS metadata to be maintained in the tarball
+ the fetcher creates when set to "keep".
+ The tarball is expanded into the work directory.
+ By default, the CVS metadata is removed.
+ </para></listitem>
+ <listitem><para><emphasis>"fullpath":</emphasis>
+ Controls whether the resulting checkout is at the
+ module level, which is the default, or is at deeper
+ paths.
+ </para></listitem>
+ <listitem><para><emphasis>"norecurse":</emphasis>
+ Causes the fetcher to only checkout the specified
+ directory with no recurse into any subdirectories.
+ </para></listitem>
+ <listitem><para><emphasis>"port":</emphasis>
+ The port to which the CVS server connects.
+ </para></listitem>
+ </itemizedlist>
+ Some example URLs are as follows:
+ <literallayout class='monospaced'>
+ SRC_URI = "cvs://CVSROOT;module=mymodule;tag=some-version;method=ext"
+ SRC_URI = "cvs://CVSROOT;module=mymodule;date=20060126;localdir=usethat"
+ </literallayout>
+ </para>
+ </section>
+
+ <section id='svn-fetcher'>
+ <title>Subversion (SVN) Fetcher (<filename>svn://</filename>)</title>
+
+ <para>
+ This fetcher submodule fetches code from the
+ Subversion source control system.
+ The executable used is specified by
+ <filename>FETCHCMD_svn</filename>, which defaults
+ to "svn".
+ The fetcher's temporary working directory is set by
+ <link linkend='var-bb-SVNDIR'><filename>SVNDIR</filename></link>,
+ which is usually <filename>DL_DIR/svn</filename>.
+ </para>
+
+ <para>
+ The supported parameters are as follows:
+ <itemizedlist>
+ <listitem><para><emphasis>"module":</emphasis>
+ The name of the svn module to checkout.
+ You must provide this parameter.
+ You can think of this parameter as the top-level
+ directory of the repository data you want.
+ </para></listitem>
+ <listitem><para><emphasis>"path_spec":</emphasis>
+ A specific directory in which to checkout the
+ specified svn module.
+ </para></listitem>
+ <listitem><para><emphasis>"protocol":</emphasis>
+ The protocol to use, which defaults to "svn".
+ If "protocol" is set to "svn+ssh", the "ssh"
+ parameter is also used.
+ </para></listitem>
+ <listitem><para><emphasis>"rev":</emphasis>
+ The revision of the source code to checkout.
+ </para></listitem>
+ <listitem><para><emphasis>"scmdata":</emphasis>
+ Causes the “.svn” directories to be available during
+ compile-time when set to "keep".
+ By default, these directories are removed.
+ </para></listitem>
+ <listitem><para><emphasis>"ssh":</emphasis>
+ An optional parameter used when "protocol" is set
+ to "svn+ssh".
+ You can use this parameter to specify the ssh
+ program used by svn.
+ </para></listitem>
+ <listitem><para><emphasis>"transportuser":</emphasis>
+ When required, sets the username for the transport.
+ By default, this parameter is empty.
+ The transport username is different than the username
+ used in the main URL, which is passed to the subversion
+ command.
+ </para></listitem>
+ </itemizedlist>
+ Following are three examples using svn:
+ <literallayout class='monospaced'>
+ SRC_URI = "svn://myrepos/proj1;module=vip;protocol=http;rev=667"
+ SRC_URI = "svn://myrepos/proj1;module=opie;protocol=svn+ssh"
+ SRC_URI = "svn://myrepos/proj1;module=trunk;protocol=http;path_spec=${MY_DIR}/proj1"
+ </literallayout>
+ </para>
+ </section>
+
+ <section id='git-fetcher'>
+ <title>Git Fetcher (<filename>git://</filename>)</title>
+
+ <para>
+ This fetcher submodule fetches code from the Git
+ source control system.
+ The fetcher works by creating a bare clone of the
+ remote into
+ <link linkend='var-bb-GITDIR'><filename>GITDIR</filename></link>,
+ which is usually <filename>DL_DIR/git2</filename>.
+ This bare clone is then cloned into the work directory during the
+ unpack stage when a specific tree is checked out.
+ This is done using alternates and by reference to
+ minimize the amount of duplicate data on the disk and
+ make the unpack process fast.
+ The executable used can be set with
+ <filename>FETCHCMD_git</filename>.
+ </para>
+
+ <para>
+ This fetcher supports the following parameters:
+ <itemizedlist>
+ <listitem><para><emphasis>"protocol":</emphasis>
+ The protocol used to fetch the files.
+ The default is "git" when a hostname is set.
+ If a hostname is not set, the Git protocol is "file".
+ You can also use "http", "https", "ssh" and "rsync".
+ </para></listitem>
+ <listitem><para><emphasis>"nocheckout":</emphasis>
+ Tells the fetcher to not checkout source code when
+ unpacking when set to "1".
+ Set this option for the URL where there is a custom
+ routine to checkout code.
+ The default is "0".
+ </para></listitem>
+ <listitem><para><emphasis>"rebaseable":</emphasis>
+ Indicates that the upstream Git repository can be rebased.
+ You should set this parameter to "1" if
+ revisions can become detached from branches.
+ In this case, the source mirror tarball is done per
+ revision, which has a loss of efficiency.
+ Rebasing the upstream Git repository could cause the
+ current revision to disappear from the upstream repository.
+ This option reminds the fetcher to preserve the local cache
+ carefully for future use.
+ The default value for this parameter is "0".
+ </para></listitem>
+ <listitem><para><emphasis>"nobranch":</emphasis>
+ Tells the fetcher to not check the SHA validation
+ for the branch when set to "1".
+ The default is "0".
+ Set this option for the recipe that refers to
+ the commit that is valid for a tag instead of
+ the branch.
+ </para></listitem>
+ <listitem><para><emphasis>"bareclone":</emphasis>
+ Tells the fetcher to clone a bare clone into the
+ destination directory without checking out a working tree.
+ Only the raw Git metadata is provided.
+ This parameter implies the "nocheckout" parameter as well.
+ </para></listitem>
+ <listitem><para><emphasis>"branch":</emphasis>
+ The branch(es) of the Git tree to clone.
+ If unset, this is assumed to be "master".
+ The number of branch parameters much match the number of
+ name parameters.
+ </para></listitem>
+ <listitem><para><emphasis>"rev":</emphasis>
+ The revision to use for the checkout.
+ The default is "master".
+ </para></listitem>
+ <listitem><para><emphasis>"tag":</emphasis>
+ Specifies a tag to use for the checkout.
+ To correctly resolve tags, BitBake must access the
+ network.
+ For that reason, tags are often not used.
+ As far as Git is concerned, the "tag" parameter behaves
+ effectively the same as the "rev" parameter.
+ </para></listitem>
+ <listitem><para><emphasis>"subpath":</emphasis>
+ Limits the checkout to a specific subpath of the tree.
+ By default, the whole tree is checked out.
+ </para></listitem>
+ <listitem><para><emphasis>"destsuffix":</emphasis>
+ The name of the path in which to place the checkout.
+ By default, the path is <filename>git/</filename>.
+ </para></listitem>
+ <listitem><para><emphasis>"usehead":</emphasis>
+ Enables local <filename>git://</filename> URLs to use the
+ current branch HEAD as the revision for use with
+ <filename>AUTOREV</filename>.
+ The "usehead" parameter implies no branch and only works
+ when the transfer protocol is
+ <filename>file://</filename>.
+ </para></listitem>
+ </itemizedlist>
+ Here are some example URLs:
+ <literallayout class='monospaced'>
+ SRC_URI = "git://git.oe.handhelds.org/git/vip.git;tag=version-1"
+ SRC_URI = "git://git.oe.handhelds.org/git/vip.git;protocol=http"
+ </literallayout>
+ </para>
+ </section>
+
+ <section id='gitsm-fetcher'>
+ <title>Git Submodule Fetcher (<filename>gitsm://</filename>)</title>
+
+ <para>
+ This fetcher submodule inherits from the
+ <link linkend='git-fetcher'>Git fetcher</link> and extends
+ that fetcher's behavior by fetching a repository's submodules.
+ <link linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link>
+ is passed to the Git fetcher as described in the
+ "<link linkend='git-fetcher'>Git Fetcher (<filename>git://</filename>)</link>"
+ section.
+ <note>
+ <title>Notes and Warnings</title>
+ <para>
+ You must clean a recipe when switching between
+ '<filename>git://</filename>' and
+ '<filename>gitsm://</filename>' URLs.
+ </para>
+
+ <para>
+ The Git Submodules fetcher is not a complete fetcher
+ implementation.
+ The fetcher has known issues where it does not use the
+ normal source mirroring infrastructure properly. Further,
+ the submodule sources it fetches are not visible to the
+ licensing and source archiving infrastructures.
+ </para>
+ </note>
+ </para>
+ </section>
+
+ <section id='clearcase-fetcher'>
+ <title>ClearCase Fetcher (<filename>ccrc://</filename>)</title>
+
+ <para>
+ This fetcher submodule fetches code from a
+ <ulink url='http://en.wikipedia.org/wiki/Rational_ClearCase'>ClearCase</ulink>
+ repository.
+ </para>
+
+ <para>
+ To use this fetcher, make sure your recipe has proper
+ <link linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link>,
+ <link linkend='var-bb-SRCREV'><filename>SRCREV</filename></link>, and
+ <link linkend='var-bb-PV'><filename>PV</filename></link> settings.
+ Here is an example:
+ <literallayout class='monospaced'>
+ SRC_URI = "ccrc://cc.example.org/ccrc;vob=/example_vob;module=/example_module"
+ SRCREV = "EXAMPLE_CLEARCASE_TAG"
+ PV = "${@d.getVar("SRCREV", False).replace("/", "+")}"
+ </literallayout>
+ The fetcher uses the <filename>rcleartool</filename> or
+ <filename>cleartool</filename> remote client, depending on
+ which one is available.
+ </para>
+
+ <para>
+ Following are options for the <filename>SRC_URI</filename>
+ statement:
+ <itemizedlist>
+ <listitem><para><emphasis><filename>vob</filename></emphasis>:
+ The name, which must include the
+ prepending "/" character, of the ClearCase VOB.
+ This option is required.
+ </para></listitem>
+ <listitem><para><emphasis><filename>module</filename></emphasis>:
+ The module, which must include the
+ prepending "/" character, in the selected VOB.
+ <note>
+ The <filename>module</filename> and <filename>vob</filename>
+ options are combined to create the <filename>load</filename> rule in
+ the view config spec.
+ As an example, consider the <filename>vob</filename> and
+ <filename>module</filename> values from the
+ <filename>SRC_URI</filename> statement at the start of this section.
+ Combining those values results in the following:
+ <literallayout class='monospaced'>
+ load /example_vob/example_module
+ </literallayout>
+ </note>
+ </para></listitem>
+ <listitem><para><emphasis><filename>proto</filename></emphasis>:
+ The protocol, which can be either <filename>http</filename> or
+ <filename>https</filename>.
+ </para></listitem>
+ </itemizedlist>
+ </para>
+
+ <para>
+ By default, the fetcher creates a configuration specification.
+ If you want this specification written to an area other than the default,
+ use the <filename>CCASE_CUSTOM_CONFIG_SPEC</filename> variable
+ in your recipe to define where the specification is written.
+ <note>
+ the <filename>SRCREV</filename> loses its functionality if you
+ specify this variable.
+ However, <filename>SRCREV</filename> is still used to label the
+ archive after a fetch even though it does not define what is
+ fetched.
+ </note>
+ </para>
+
+ <para>
+ Here are a couple of other behaviors worth mentioning:
+ <itemizedlist>
+ <listitem><para>
+ When using <filename>cleartool</filename>, the login of
+ <filename>cleartool</filename> is handled by the system.
+ The login require no special steps.
+ </para></listitem>
+ <listitem><para>
+ In order to use <filename>rcleartool</filename> with authenticated
+ users, an "rcleartool login" is necessary before using the fetcher.
+ </para></listitem>
+ </itemizedlist>
+ </para>
+ </section>
+
+ <section id='perforce-fetcher'>
+ <title>Perforce Fetcher (<filename>p4://</filename>)</title>
+
+ <para>
+ This fetcher submodule fetches code from the
+ <ulink url='https://www.perforce.com/'>Perforce</ulink>
+ source control system.
+ The executable used is specified by
+ <filename>FETCHCMD_p4</filename>, which defaults
+ to "p4".
+ The fetcher's temporary working directory is set by
+ <link linkend='var-bb-P4DIR'><filename>P4DIR</filename></link>,
+ which defaults to "DL_DIR/p4".
+ The fetcher does not make use of a perforce client, instead it
+ relies on <filename>p4 files</filename> to retrieve a list of
+ files and <filename>p4 print</filename> to transfer the content
+ of those files locally.
+ </para>
+
+ <para>
+ To use this fetcher, make sure your recipe has proper
+ <link linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link>,
+ <link linkend='var-bb-SRCREV'><filename>SRCREV</filename></link>, and
+ <link linkend='var-bb-PV'><filename>PV</filename></link> values.
+ The p4 executable is able to use the config file defined by your
+ system's <filename>P4CONFIG</filename> environment variable in
+ order to define the Perforce server URL and port, username, and
+ password if you do not wish to keep those values in a recipe
+ itself.
+ If you choose not to use <filename>P4CONFIG</filename>,
+ or to explicitly set variables that <filename>P4CONFIG</filename>
+ can contain, you can specify the <filename>P4PORT</filename> value,
+ which is the server's URL and port number, and you can
+ specify a username and password directly in your recipe within
+ <filename>SRC_URI</filename>.
+ </para>
+
+ <para>
+ Here is an example that relies on <filename>P4CONFIG</filename>
+ to specify the server URL and port, username, and password, and
+ fetches the Head Revision:
+ <literallayout class='monospaced'>
+ SRC_URI = "p4://example-depot/main/source/..."
+ SRCREV = "${AUTOREV}"
+ PV = "p4-${SRCPV}"
+ S = "${WORKDIR}/p4"
+ </literallayout>
+ </para>
+
+ <para>
+ Here is an example that specifies the server URL and port,
+ username, and password, and fetches a Revision based on a Label:
+ <literallayout class='monospaced'>
+ P4PORT = "tcp:p4server.example.net:1666"
+ SRC_URI = "p4://user:passwd@example-depot/main/source/..."
+ SRCREV = "release-1.0"
+ PV = "p4-${SRCPV}"
+ S = "${WORKDIR}/p4"
+ </literallayout>
+ <note>
+ You should always set <filename>S</filename>
+ to <filename>"${WORKDIR}/p4"</filename> in your recipe.
+ </note>
+ </para>
+
+ <para>
+ By default, the fetcher strips the depot location from the
+ local file paths. In the above example, the content of
+ <filename>example-depot/main/source/</filename>
+ will be placed in <filename>${WORKDIR}/p4</filename>.
+ For situations where preserving parts of the remote depot paths
+ locally is desirable, the fetcher supports two parameters:
+
+ <itemizedlist>
+ <listitem><para>
+ <emphasis>"module":</emphasis>
+ The top-level depot location or directory to fetch. The
+ value of this parameter can also point to a single file
+ within the depot, in which case the local file path will
+ include the module path.
+ </para></listitem>
+ <listitem><para>
+ <emphasis>"remotepath":</emphasis>
+ When used with the value "<filename>keep</filename>",
+ the fetcher will mirror the full depot paths locally
+ for the specified location, even in combination with
+ the <filename>module</filename> parameter.
+ </para></listitem>
+ </itemizedlist>
+ </para>
+
+ <para>
+ Here is an example use of the the <filename>module</filename>
+ parameter:
+
+ <literallayout class='monospaced'>
+ SRC_URI = "p4://user:passwd@example-depot/main;module=source/..."
+ </literallayout>
+
+ In this case, the content of the top-level directory
+ <filename>source/</filename> will be fetched to
+ <filename>${P4DIR}</filename>, including the directory itself.
+ The top-level directory will be accesible at
+ <filename>${P4DIR}/source/</filename>.
+ </para>
+
+ <para>
+ Here is an example use of the the <filename>remotepath</filename>
+ parameter:
+
+ <literallayout class='monospaced'>
+ SRC_URI = "p4://user:passwd@example-depot/main;module=source/...;remotepath=keep"
+ </literallayout>
+
+ In this case, the content of the top-level directory
+ <filename>source/</filename> will be fetched to
+ <filename>${P4DIR}</filename>, but the complete depot paths will
+ be mirrored locally. The top-level directory will be accessible
+ at <filename>${P4DIR}/example-depot/main/source/</filename>.
+ </para>
+ </section>
+
+ <section id='repo-fetcher'>
+ <title>Repo Fetcher (<filename>repo://</filename>)</title>
+
+ <para>
+ This fetcher submodule fetches code from
+ <filename>google-repo</filename> source control system.
+ The fetcher works by initiating and syncing sources of the
+ repository into
+ <link linkend='var-bb-REPODIR'><filename>REPODIR</filename></link>,
+ which is usually
+ <link linkend='var-bb-DL_DIR'><filename>DL_DIR</filename></link><filename>/repo</filename>.
+ </para>
+
+ <para>
+ This fetcher supports the following parameters:
+ <itemizedlist>
+ <listitem><para>
+ <emphasis>"protocol":</emphasis>
+ Protocol to fetch the repository manifest (default: git).
+ </para></listitem>
+ <listitem><para>
+ <emphasis>"branch":</emphasis>
+ Branch or tag of repository to get (default: master).
+ </para></listitem>
+ <listitem><para>
+ <emphasis>"manifest":</emphasis>
+ Name of the manifest file (default: <filename>default.xml</filename>).
+ </para></listitem>
+ </itemizedlist>
+ Here are some example URLs:
+ <literallayout class='monospaced'>
+ SRC_URI = "repo://REPOROOT;protocol=git;branch=some_branch;manifest=my_manifest.xml"
+ SRC_URI = "repo://REPOROOT;protocol=file;branch=some_branch;manifest=my_manifest.xml"
+ </literallayout>
+ </para>
+ </section>
+
+ <section id='other-fetchers'>
+ <title>Other Fetchers</title>
+
+ <para>
+ Fetch submodules also exist for the following:
+ <itemizedlist>
+ <listitem><para>
+ Bazaar (<filename>bzr://</filename>)
+ </para></listitem>
+ <listitem><para>
+ Mercurial (<filename>hg://</filename>)
+ </para></listitem>
+ <listitem><para>
+ npm (<filename>npm://</filename>)
+ </para></listitem>
+ <listitem><para>
+ OSC (<filename>osc://</filename>)
+ </para></listitem>
+ <listitem><para>
+ Secure FTP (<filename>sftp://</filename>)
+ </para></listitem>
+ <listitem><para>
+ Secure Shell (<filename>ssh://</filename>)
+ </para></listitem>
+ <listitem><para>
+ Trees using Git Annex (<filename>gitannex://</filename>)
+ </para></listitem>
+ </itemizedlist>
+ No documentation currently exists for these lesser used
+ fetcher submodules.
+ However, you might find the code helpful and readable.
+ </para>
+ </section>
+ </section>
+
+ <section id='auto-revisions'>
+ <title>Auto Revisions</title>
+
+ <para>
+ We need to document <filename>AUTOREV</filename> and
+ <filename>SRCREV_FORMAT</filename> here.
+ </para>
+ </section>
+</chapter>