Blame - poky/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml - openbmc/openbmc

Andrew Geissler

4873add

2020-11-02 18:44:49 -0600

[diff] [blame^]

1

<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"

2

"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">

3

4

5

<title>File Download Support</title>

6

7

<para>

8

BitBake's fetch module is a standalone piece of library code

9

that deals with the intricacies of downloading source code

10

and files from remote systems.

11

Fetching source code is one of the cornerstones of building software.

12

As such, this module forms an important part of BitBake.

</para>

<para>

The current fetch module is called "fetch2" and refers to the

17

fact that it is the second major version of the API.

18

The original version is obsolete and has been removed from the codebase.

19

Thus, in all cases, "fetch" refers to "fetch2" in this

manual.

</para>

<title>The Download (Fetch)</title>

25

26

<para>

27

BitBake takes several steps when fetching source code or files.

28

The fetcher codebase deals with two distinct processes in order:

29

obtaining the files from somewhere (cached or otherwise)

30

and then unpacking those files into a specific location and

31

perhaps in a specific way.

32

Getting and unpacking the files is often optionally followed

33

by patching.

34

Patching, however, is not covered by this module.

</para>

<para>

The code to execute the first part of this process, a fetch,

39

looks something like the following:

40

41

src_uri = (d.getVar('SRC_URI') or "").split()

42

fetcher = bb.fetch2.Fetch(src_uri, d)

43

fetcher.download()

44

</literallayout>

45

This code sets up an instance of the fetch class.

46

The instance uses a space-separated list of URLs from the

47

48

variable and then calls the <filename>download</filename>

49

method to download the files.

</para>

<para>

The instantiation of the fetch class is usually followed by:

54

55

rootdir = l.getVar('WORKDIR')

56

fetcher.unpack(rootdir)

57

</literallayout>

58

This code unpacks the downloaded files to the

59

specified by <filename>WORKDIR</filename>.

60

<note>

61

For convenience, the naming in these examples matches

62

the variables used by OpenEmbedded.

63

If you want to see the above code in action, examine

64

the OpenEmbedded class file <filename>base.bbclass</filename>.

65

</note>

66

The <filename>SRC_URI</filename> and <filename>WORKDIR</filename>

67

variables are not hardcoded into the fetcher, since those fetcher

68

methods can be (and are) called with different variable names.

69

In OpenEmbedded for example, the shared state (sstate) code uses

70

the fetch module to fetch the sstate files.

</para>

<para>

When the <filename>download()</filename> method is called,

75

BitBake tries to resolve the URLs by looking for source files

76

in a specific search order:

77

78

<listitem><para><emphasis>Pre-mirror Sites:</emphasis>

79

BitBake first uses pre-mirrors to try and find source files.

80

These locations are defined using the

variable.

</para></listitem>

<listitem><para><emphasis>Source URI:</emphasis>

85

If pre-mirrors fail, BitBake uses the original URL (e.g from

86

<filename>SRC_URI</filename>).

87

</para></listitem>

88

<listitem><para><emphasis>Mirror Sites:</emphasis>

89

If fetch failures occur, BitBake next uses mirror locations as

defined by the

variable.

</para></listitem>

</itemizedlist>

</para>

<para>

For each URL passed to the fetcher, the fetcher

99

calls the submodule that handles that particular URL type.

100

This behavior can be the source of some confusion when you

101

are providing URLs for the <filename>SRC_URI</filename>

102

variable.

103

Consider the following two URLs:

104

105

http://git.yoctoproject.org/git/poky;protocol=git

106

git://git.yoctoproject.org/git/poky;protocol=http

107

</literallayout>

108

In the former case, the URL is passed to the

109

<filename>wget</filename> fetcher, which does not

110

understand "git".

111

Therefore, the latter case is the correct form since the

112

Git fetcher does know how to use HTTP as a transport.

</para>

<para>

Here are some examples that show commonly used mirror

definitions:

PREMIRRORS ?= "\

bzr://.*/.* http://somemirror.org/sources/ \n \

121

cvs://.*/.* http://somemirror.org/sources/ \n \

122

git://.*/.* http://somemirror.org/sources/ \n \

123

hg://.*/.* http://somemirror.org/sources/ \n \

124

osc://.*/.* http://somemirror.org/sources/ \n \

125

p4://.*/.* http://somemirror.org/sources/ \n \

126

svn://.*/.* http://somemirror.org/sources/ \n"

127

128

MIRRORS =+ "\

129

ftp://.*/.* http://somemirror.org/sources/ \n \

130

http://.*/.* http://somemirror.org/sources/ \n \

131

https://.*/.* http://somemirror.org/sources/ \n"

132

</literallayout>

133

It is useful to note that BitBake supports

134

cross-URLs.

135

It is possible to mirror a Git repository on an HTTP

136

server as a tarball.

137

This is what the <filename>git://</filename> mapping in

138

the previous example does.

</para>

<para>

Since network accesses are slow, BitBake maintains a

143

cache of files downloaded from the network.

144

Any source files that are not local (i.e.

145

downloaded from the Internet) are placed into the download

146

directory, which is specified by the

variable.

</para>

<para>

File integrity is of key importance for reproducing builds.

153

For non-local archive downloads, the fetcher code can verify

154

SHA-256 and MD5 checksums to ensure the archives have been

155

downloaded correctly.

156

You can specify these checksums by using the

157

<filename>SRC_URI</filename> variable with the appropriate

158

varflags as follows:

159

160

SRC_URI[md5sum] = "<replaceable>value</replaceable>"

161

SRC_URI[sha256sum] = "<replaceable>value</replaceable>"

162

</literallayout>

163

You can also specify the checksums as parameters on the

164

<filename>SRC_URI</filename> as shown below:

165

166

SRC_URI = "http://example.com/foobar.tar.bz2;md5sum=4a8e0f237e961fd7785d19d07fdb994d"

167

</literallayout>

168

If multiple URIs exist, you can specify the checksums either

169

directly as in the previous example, or you can name the URLs.

170

The following syntax shows how you name the URIs:

171

172

SRC_URI = "http://example.com/foobar.tar.bz2;name=foo"

173

SRC_URI[foo.md5sum] = 4a8e0f237e961fd7785d19d07fdb994d

174

</literallayout>

175

After a file has been downloaded and has had its checksum checked,

176

a ".done" stamp is placed in <filename>DL_DIR</filename>.

177

BitBake uses this stamp during subsequent builds to avoid

178

downloading or comparing a checksum for the file again.

179

<note>

180

It is assumed that local storage is safe from data corruption.

181

If this were not the case, there would be bigger issues to worry about.

</note>

</para>

<para>

If

is set, any download without a checksum triggers an

error message.

The

variable can be used to make any attempted network access a fatal

193

error, which is useful for checking that mirrors are complete

194

as well as other things.

</para>

</section>

<title>The Unpack</title>

200

201

<para>

202

The unpack process usually immediately follows the download.

203

For all URLs except Git URLs, BitBake uses the common

204

<filename>unpack</filename> method.

</para>

<para>

A number of parameters exist that you can specify within the

209

URL to govern the behavior of the unpack stage:

210

211

<listitem><para><emphasis>unpack:</emphasis>

212

Controls whether the URL components are unpacked.

213

If set to "1", which is the default, the components

214

are unpacked.

215

If set to "0", the unpack stage leaves the file alone.

216

This parameter is useful when you want an archive to be

217

copied in and not be unpacked.

218

</para></listitem>

219

220

Applies to <filename>.zip</filename> and

221

<filename>.jar</filename> files and specifies whether to

222

use DOS line ending conversion on text files.

223

</para></listitem>

224

<listitem><para><emphasis>basepath:</emphasis>

225

Instructs the unpack stage to strip the specified

226

directories from the source path when unpacking.

227

</para></listitem>

228

<listitem><para><emphasis>subdir:</emphasis>

229

Unpacks the specific URL to the specified subdirectory

230

within the root directory.

231

</para></listitem>

232

</itemizedlist>

233

The unpack call automatically decompresses and extracts files

234

with ".Z", ".z", ".gz", ".xz", ".zip", ".jar", ".ipk", ".rpm".

235

".srpm", ".deb" and ".bz2" extensions as well as various combinations

236

of tarball extensions.

</para>

<para>

As mentioned, the Git fetcher has its own unpack method that

241

is optimized to work with Git trees.

242

Basically, this method works by cloning the tree into the final

243

directory.

244

The process is completed using references so that there is

245

only one central copy of the Git metadata needed.

</para>

</section>

<title>Fetchers</title>

251

252

<para>

253

As mentioned earlier, the URL prefix determines which

254

fetcher submodule BitBake uses.

255

Each submodule can support different URL parameters,

256

which are described in the following sections.

</para>

<title>Local file fetcher (<filename>file://</filename>)</title>

261

262

<para>

263

This submodule handles URLs that begin with

264

<filename>file://</filename>.

265

The filename you specify within the URL can be

266

either an absolute or relative path to a file.

267

If the filename is relative, the contents of the

268

269

variable is used in the same way

270

<filename>PATH</filename> is used to find executables.

271

If the file cannot be found, it is assumed that it is available in

272

273

by the time the <filename>download()</filename> method is called.

</para>

<para>

If you specify a directory, the entire directory is

unpacked.

</para>

<para>

Here are a couple of example URLs, the first relative and

283

the second absolute:

284

285

SRC_URI = "file://relativefile.patch"

286

SRC_URI = "file:///Users/ich/very_important_software"

</literallayout>

</para>

</section>

<title>HTTP/FTP wget fetcher (<filename>http://</filename>, <filename>ftp://</filename>, <filename>https://</filename>)</title>

293

294

<para>

295

This fetcher obtains files from web and FTP servers.

296

Internally, the fetcher uses the wget utility.

</para>

<para>

The executable and parameters used are specified by the

301

<filename>FETCHCMD_wget</filename> variable, which defaults

302

to sensible values.

303

The fetcher supports a parameter "downloadfilename" that

304

allows the name of the downloaded file to be specified.

305

Specifying the name of the downloaded file is useful

306

for avoiding collisions in

307

308

when dealing with multiple files that have the same name.

</para>

<para>

Some example URLs are as follows:

313

314

SRC_URI = "http://oe.handhelds.org/not_there.aac"

315

SRC_URI = "ftp://oe.handhelds.org/not_there_as_well.aac"

316

SRC_URI = "ftp://you@oe.handhelds.org/home/you/secret.plan"

</literallayout>

</para>

<note>

Because URL parameters are delimited by semi-colons, this can

321

introduce ambiguity when parsing URLs that also contain semi-colons,

322

for example:

323

324

SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git;a=snapshot;h=a5dd47"

325

</literallayout>

326

Such URLs should should be modified by replacing semi-colons with '&' characters:

327

328

SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47"

329

</literallayout>

330

In most cases this should work. Treating semi-colons and '&' in queries

331

identically is recommended by the World Wide Web Consortium (W3C).

332

Note that due to the nature of the URL, you may have to specify the name

333

of the downloaded file as well:

334

335

SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47;downloadfilename=myfile.bz2"

</literallayout>

</note>

</section>

<title>CVS fetcher (<filename>(cvs://</filename>)</title>

342

343

<para>

344

This submodule handles checking out files from the

345

CVS version control system.

346

You can configure it using a number of different variables:

347

348

<listitem><para><emphasis><filename>FETCHCMD_cvs</filename>:</emphasis>

349

The name of the executable to use when running

350

the <filename>cvs</filename> command.

351

This name is usually "cvs".

352

</para></listitem>

353

<listitem><para><emphasis><filename>SRCDATE</filename>:</emphasis>

354

The date to use when fetching the CVS source code.

355

A special value of "now" causes the checkout to

356

be updated on every build.

357

</para></listitem>

358

<listitem><para><emphasis><link linkend='var-bb-CVSDIR'><filename>CVSDIR</filename></link>:</emphasis>

359

Specifies where a temporary checkout is saved.

360

The location is often <filename>DL_DIR/cvs</filename>.

361

</para></listitem>

362

<listitem><para><emphasis><filename>CVS_PROXY_HOST</filename>:</emphasis>

363

The name to use as a "proxy=" parameter to the

364

<filename>cvs</filename> command.

365

</para></listitem>

366

<listitem><para><emphasis><filename>CVS_PROXY_PORT</filename>:</emphasis>

367

The port number to use as a "proxyport=" parameter to

368

the <filename>cvs</filename> command.

369

</para></listitem>

370

</itemizedlist>

371

As well as the standard username and password URL syntax,

372

you can also configure the fetcher with various URL parameters:

</para>

<para>

The supported parameters are as follows:

377

378

<listitem><para><emphasis>"method":</emphasis>

379

The protocol over which to communicate with the CVS

380

server.

381

By default, this protocol is "pserver".

382

If "method" is set to "ext", BitBake examines the

383

"rsh" parameter and sets <filename>CVS_RSH</filename>.

384

You can use "dir" for local directories.

385

</para></listitem>

386

<listitem><para><emphasis>"module":</emphasis>

387

Specifies the module to check out.

388

You must supply this parameter.

389

</para></listitem>

390

391

Describes which CVS TAG should be used for

392

the checkout.

393

By default, the TAG is empty.

</para></listitem>

Specifies a date.

If no "date" is specified, the

398

399

of the configuration is used to checkout a specific date.

400

The special value of "now" causes the checkout to be

401

updated on every build.

402

</para></listitem>

403

<listitem><para><emphasis>"localdir":</emphasis>

404

Used to rename the module.

405

Effectively, you are renaming the output directory

406

to which the module is unpacked.

407

You are forcing the module into a special

408

directory relative to

</para></listitem>

Used in conjunction with the "method" parameter.

413

</para></listitem>

414

<listitem><para><emphasis>"scmdata":</emphasis>

415

Causes the CVS metadata to be maintained in the tarball

416

the fetcher creates when set to "keep".

417

The tarball is expanded into the work directory.

418

By default, the CVS metadata is removed.

419

</para></listitem>

420

<listitem><para><emphasis>"fullpath":</emphasis>

421

Controls whether the resulting checkout is at the

422

module level, which is the default, or is at deeper

423

paths.

424

</para></listitem>

425

<listitem><para><emphasis>"norecurse":</emphasis>

426

Causes the fetcher to only checkout the specified

427

directory with no recurse into any subdirectories.

428

</para></listitem>

429

430

The port to which the CVS server connects.

431

</para></listitem>

432

</itemizedlist>

433

Some example URLs are as follows:

434

435

SRC_URI = "cvs://CVSROOT;module=mymodule;tag=some-version;method=ext"

436

SRC_URI = "cvs://CVSROOT;module=mymodule;date=20060126;localdir=usethat"

</literallayout>

</para>

</section>

<title>Subversion (SVN) Fetcher (<filename>svn://</filename>)</title>

443

444

<para>

445

This fetcher submodule fetches code from the

446

Subversion source control system.

447

The executable used is specified by

448

<filename>FETCHCMD_svn</filename>, which defaults

449

to "svn".

450

The fetcher's temporary working directory is set by

451

452

which is usually <filename>DL_DIR/svn</filename>.

</para>

<para>

The supported parameters are as follows:

457

458

<listitem><para><emphasis>"module":</emphasis>

459

The name of the svn module to checkout.

460

You must provide this parameter.

461

You can think of this parameter as the top-level

462

directory of the repository data you want.

463

</para></listitem>

464

465

A specific directory in which to checkout the

466

specified svn module.

467

</para></listitem>

468

<listitem><para><emphasis>"protocol":</emphasis>

469

The protocol to use, which defaults to "svn".

470

If "protocol" is set to "svn+ssh", the "ssh"

471

parameter is also used.

472

</para></listitem>

473

474

The revision of the source code to checkout.

475

</para></listitem>

476

<listitem><para><emphasis>"scmdata":</emphasis>

477

Causes the “.svn” directories to be available during

478

compile-time when set to "keep".

479

By default, these directories are removed.

480

</para></listitem>

481

482

An optional parameter used when "protocol" is set

483

to "svn+ssh".

484

You can use this parameter to specify the ssh

485

program used by svn.

486

</para></listitem>

487

<listitem><para><emphasis>"transportuser":</emphasis>

488

When required, sets the username for the transport.

489

By default, this parameter is empty.

490

The transport username is different than the username

491

used in the main URL, which is passed to the subversion

command.

</para></listitem>

</itemizedlist>

Following are three examples using svn:

496

497

SRC_URI = "svn://myrepos/proj1;module=vip;protocol=http;rev=667"

498

SRC_URI = "svn://myrepos/proj1;module=opie;protocol=svn+ssh"

499

SRC_URI = "svn://myrepos/proj1;module=trunk;protocol=http;path_spec=${MY_DIR}/proj1"

</literallayout>

</para>

</section>

<title>Git Fetcher (<filename>git://</filename>)</title>

506

507

<para>

508

This fetcher submodule fetches code from the Git

509

source control system.

510

The fetcher works by creating a bare clone of the

511

remote into

512

513

which is usually <filename>DL_DIR/git2</filename>.

514

This bare clone is then cloned into the work directory during the

515

unpack stage when a specific tree is checked out.

516

This is done using alternates and by reference to

517

minimize the amount of duplicate data on the disk and

518

make the unpack process fast.

519

The executable used can be set with

520

<filename>FETCHCMD_git</filename>.

</para>

<para>

This fetcher supports the following parameters:

525

526

<listitem><para><emphasis>"protocol":</emphasis>

527

The protocol used to fetch the files.

528

The default is "git" when a hostname is set.

529

If a hostname is not set, the Git protocol is "file".

530

You can also use "http", "https", "ssh" and "rsync".

531

</para></listitem>

532

<listitem><para><emphasis>"nocheckout":</emphasis>

533

Tells the fetcher to not checkout source code when

534

unpacking when set to "1".

535

Set this option for the URL where there is a custom

536

routine to checkout code.

537

The default is "0".

538

</para></listitem>

539

<listitem><para><emphasis>"rebaseable":</emphasis>

540

Indicates that the upstream Git repository can be rebased.

541

You should set this parameter to "1" if

542

revisions can become detached from branches.

543

In this case, the source mirror tarball is done per

544

revision, which has a loss of efficiency.

545

Rebasing the upstream Git repository could cause the

546

current revision to disappear from the upstream repository.

547

This option reminds the fetcher to preserve the local cache

548

carefully for future use.

549

The default value for this parameter is "0".

550

</para></listitem>

551

<listitem><para><emphasis>"nobranch":</emphasis>

552

Tells the fetcher to not check the SHA validation

553

for the branch when set to "1".

554

The default is "0".

555

Set this option for the recipe that refers to

556

the commit that is valid for a tag instead of

557

the branch.

558

</para></listitem>

559

<listitem><para><emphasis>"bareclone":</emphasis>

560

Tells the fetcher to clone a bare clone into the

561

destination directory without checking out a working tree.

562

Only the raw Git metadata is provided.

563

This parameter implies the "nocheckout" parameter as well.

564

</para></listitem>

565

<listitem><para><emphasis>"branch":</emphasis>

566

The branch(es) of the Git tree to clone.

567

If unset, this is assumed to be "master".

568

The number of branch parameters much match the number of

name parameters.

</para></listitem>

The revision to use for the checkout.

573

The default is "master".

574

</para></listitem>

575

576

Specifies a tag to use for the checkout.

577

To correctly resolve tags, BitBake must access the

578

network.

579

For that reason, tags are often not used.

580

As far as Git is concerned, the "tag" parameter behaves

581

effectively the same as the "rev" parameter.

582

</para></listitem>

583

<listitem><para><emphasis>"subpath":</emphasis>

584

Limits the checkout to a specific subpath of the tree.

585

By default, the whole tree is checked out.

586

</para></listitem>

587

<listitem><para><emphasis>"destsuffix":</emphasis>

588

The name of the path in which to place the checkout.

589

By default, the path is <filename>git/</filename>.

590

</para></listitem>

591

<listitem><para><emphasis>"usehead":</emphasis>

592

Enables local <filename>git://</filename> URLs to use the

593

current branch HEAD as the revision for use with

594

<filename>AUTOREV</filename>.

595

The "usehead" parameter implies no branch and only works

596

when the transfer protocol is

597

<filename>file://</filename>.

598

</para></listitem>

599

</itemizedlist>

600

Here are some example URLs:

601

602

SRC_URI = "git://git.oe.handhelds.org/git/vip.git;tag=version-1"

603

SRC_URI = "git://git.oe.handhelds.org/git/vip.git;protocol=http"

</literallayout>

</para>

</section>

<title>Git Submodule Fetcher (<filename>gitsm://</filename>)</title>

610

611

<para>

612

This fetcher submodule inherits from the

613

614

that fetcher's behavior by fetching a repository's submodules.

615

616

is passed to the Git fetcher as described in the

617

"<link linkend='git-fetcher'>Git Fetcher (<filename>git://</filename>)</link>"

618

section.

619

<note>

620

<title>Notes and Warnings</title>

621

<para>

622

You must clean a recipe when switching between

623

'<filename>git://</filename>' and

624

'<filename>gitsm://</filename>' URLs.

</para>

<para>

The Git Submodules fetcher is not a complete fetcher

629

implementation.

630

The fetcher has known issues where it does not use the

631

normal source mirroring infrastructure properly. Further,

632

the submodule sources it fetches are not visible to the

633

licensing and source archiving infrastructures.

</para>

</note>

</para>

</section>

<title>ClearCase Fetcher (<filename>ccrc://</filename>)</title>

641

642

<para>

643

This fetcher submodule fetches code from a

644

<ulink url='http://en.wikipedia.org/wiki/Rational_ClearCase'>ClearCase</ulink>

repository.

</para>

<para>

To use this fetcher, make sure your recipe has proper

Here is an example:

SRC_URI = "ccrc://cc.example.org/ccrc;vob=/example_vob;module=/example_module"

656

SRCREV = "EXAMPLE_CLEARCASE_TAG"

657

PV = "${@d.getVar("SRCREV", False).replace("/", "+")}"

658

</literallayout>

659

The fetcher uses the <filename>rcleartool</filename> or

660

<filename>cleartool</filename> remote client, depending on

661

which one is available.

</para>

<para>

Following are options for the <filename>SRC_URI</filename>

666

statement:

667

668

<listitem><para><emphasis><filename>vob</filename></emphasis>:

669

The name, which must include the

670

prepending "/" character, of the ClearCase VOB.

671

This option is required.

672

</para></listitem>

673

<listitem><para><emphasis><filename>module</filename></emphasis>:

674

The module, which must include the

675

prepending "/" character, in the selected VOB.

676

<note>

677

The <filename>module</filename> and <filename>vob</filename>

678

options are combined to create the <filename>load</filename> rule in

679

the view config spec.

680

As an example, consider the <filename>vob</filename> and

681

<filename>module</filename> values from the

682

<filename>SRC_URI</filename> statement at the start of this section.

683

Combining those values results in the following:

684

685

load /example_vob/example_module

</literallayout>

</note>

</para></listitem>

<listitem><para><emphasis><filename>proto</filename></emphasis>:

690

The protocol, which can be either <filename>http</filename> or

691

<filename>https</filename>.

</para></listitem>

</itemizedlist>

</para>

<para>

By default, the fetcher creates a configuration specification.

698

If you want this specification written to an area other than the default,

699

use the <filename>CCASE_CUSTOM_CONFIG_SPEC</filename> variable

700

in your recipe to define where the specification is written.

701

<note>

702

the <filename>SRCREV</filename> loses its functionality if you

703

specify this variable.

704

However, <filename>SRCREV</filename> is still used to label the

705

archive after a fetch even though it does not define what is

fetched.

</note>

</para>

<para>

Here are a couple of other behaviors worth mentioning:

712

713

714

When using <filename>cleartool</filename>, the login of

715

<filename>cleartool</filename> is handled by the system.

716

The login require no special steps.

717

</para></listitem>

718

719

In order to use <filename>rcleartool</filename> with authenticated

720

users, an "rcleartool login" is necessary before using the fetcher.

</para></listitem>

</itemizedlist>

</para>

</section>

<title>Perforce Fetcher (<filename>p4://</filename>)</title>

728

729

<para>

730

This fetcher submodule fetches code from the

731

<ulink url='https://www.perforce.com/'>Perforce</ulink>

732

source control system.

733

The executable used is specified by

734

<filename>FETCHCMD_p4</filename>, which defaults

735

to "p4".

736

The fetcher's temporary working directory is set by

737

738

which defaults to "DL_DIR/p4".

739

The fetcher does not make use of a perforce client, instead it

740

relies on <filename>p4 files</filename> to retrieve a list of

741

files and <filename>p4 print</filename> to transfer the content

742

of those files locally.

</para>

<para>

To use this fetcher, make sure your recipe has proper

The p4 executable is able to use the config file defined by your

751

system's <filename>P4CONFIG</filename> environment variable in

752

order to define the Perforce server URL and port, username, and

753

password if you do not wish to keep those values in a recipe

754

itself.

755

If you choose not to use <filename>P4CONFIG</filename>,

756

or to explicitly set variables that <filename>P4CONFIG</filename>

757

can contain, you can specify the <filename>P4PORT</filename> value,

758

which is the server's URL and port number, and you can

759

specify a username and password directly in your recipe within

760

<filename>SRC_URI</filename>.

</para>

<para>

Here is an example that relies on <filename>P4CONFIG</filename>

765

to specify the server URL and port, username, and password, and

766

fetches the Head Revision:

767

768

SRC_URI = "p4://example-depot/main/source/..."

769

SRCREV = "${AUTOREV}"

PV = "p4-${SRCPV}"

S = "${WORKDIR}/p4"

</literallayout>

</para>

<para>

Here is an example that specifies the server URL and port,

777

username, and password, and fetches a Revision based on a Label:

778

779

P4PORT = "tcp:p4server.example.net:1666"

780

SRC_URI = "p4://user:passwd@example-depot/main/source/..."

781

SRCREV = "release-1.0"

PV = "p4-${SRCPV}"

S = "${WORKDIR}/p4"

</literallayout>

<note>

You should always set <filename>S</filename>

787

to <filename>"${WORKDIR}/p4"</filename> in your recipe.

</note>

</para>

<para>

By default, the fetcher strips the depot location from the

793

local file paths. In the above example, the content of

794

<filename>example-depot/main/source/</filename>

795

will be placed in <filename>${WORKDIR}/p4</filename>.

796

For situations where preserving parts of the remote depot paths

797

locally is desirable, the fetcher supports two parameters:

<emphasis>"module":</emphasis>

802

The top-level depot location or directory to fetch. The

803

value of this parameter can also point to a single file

804

within the depot, in which case the local file path will

805

include the module path.

806

</para></listitem>

807

808

<emphasis>"remotepath":</emphasis>

809

When used with the value "<filename>keep</filename>",

810

the fetcher will mirror the full depot paths locally

811

for the specified location, even in combination with

812

the <filename>module</filename> parameter.

</para></listitem>

</itemizedlist>

</para>

<para>

Here is an example use of the the <filename>module</filename>

parameter:

SRC_URI = "p4://user:passwd@example-depot/main;module=source/..."

823

</literallayout>

824

825

In this case, the content of the top-level directory

826

<filename>source/</filename> will be fetched to

827

<filename>${P4DIR}</filename>, including the directory itself.

828

The top-level directory will be accesible at

829

<filename>${P4DIR}/source/</filename>.

</para>

<para>

Here is an example use of the the <filename>remotepath</filename>

parameter:

SRC_URI = "p4://user:passwd@example-depot/main;module=source/...;remotepath=keep"

838

</literallayout>

839

840

In this case, the content of the top-level directory

841

<filename>source/</filename> will be fetched to

842

<filename>${P4DIR}</filename>, but the complete depot paths will

843

be mirrored locally. The top-level directory will be accessible

844

at <filename>${P4DIR}/example-depot/main/source/</filename>.

</para>

</section>

<title>Repo Fetcher (<filename>repo://</filename>)</title>

850

851

<para>

852

This fetcher submodule fetches code from

853

<filename>google-repo</filename> source control system.

854

The fetcher works by initiating and syncing sources of the

repository into

which is usually

</para>

<para>

This fetcher supports the following parameters:

863

864

865

<emphasis>"protocol":</emphasis>

866

Protocol to fetch the repository manifest (default: git).

867

</para></listitem>

868

869

<emphasis>"branch":</emphasis>

870

Branch or tag of repository to get (default: master).

871

</para></listitem>

872

873

<emphasis>"manifest":</emphasis>

874

Name of the manifest file (default: <filename>default.xml</filename>).

875

</para></listitem>

876

</itemizedlist>

877

Here are some example URLs:

878

879

SRC_URI = "repo://REPOROOT;protocol=git;branch=some_branch;manifest=my_manifest.xml"

880

SRC_URI = "repo://REPOROOT;protocol=file;branch=some_branch;manifest=my_manifest.xml"

</literallayout>

</para>

</section>

<title>Other Fetchers</title>

887

888

<para>

889

Fetch submodules also exist for the following:

890

891

892

Bazaar (<filename>bzr://</filename>)

893

</para></listitem>

894

895

Mercurial (<filename>hg://</filename>)

896

</para></listitem>

897

898

npm (<filename>npm://</filename>)

899

</para></listitem>

900

901

OSC (<filename>osc://</filename>)

902

</para></listitem>

903

904

Secure FTP (<filename>sftp://</filename>)

905

</para></listitem>

906

907

Secure Shell (<filename>ssh://</filename>)

908

</para></listitem>

909

910

Trees using Git Annex (<filename>gitannex://</filename>)

911

</para></listitem>

912

</itemizedlist>

913

No documentation currently exists for these lesser used

914

fetcher submodules.

915

However, you might find the code helpful and readable.

</para>

</section>

</section>

<title>Auto Revisions</title>

922

923

<para>

924

We need to document <filename>AUTOREV</filename> and

925

<filename>SRCREV_FORMAT</filename> here.

926

</para>

927

</section>

928

</chapter>