Blame - import-layers/yocto-poky/documentation/ref-manual/usingpoky.xml - stefanberger/openbmc

2018-02-25 22:55:05 -0500

[diff] [blame]

212

Patrick Williams

2017-02-23 20:41:17 -0600

[diff] [blame]

213

to generate a log, look at the corresponding

214

<filename>run.do_</filename><replaceable>taskname</replaceable>

215

file in the same directory.

Patrick Williams

2015-09-15 14:41:29 -0500

[diff] [blame]

216

</para>

217

218

<para>

Patrick Williams

2017-02-23 20:41:17 -0600

[diff] [blame]

219

<filename>log.do_</filename><replaceable>taskname</replaceable> and

220

<filename>run.do_</filename><replaceable>taskname</replaceable>

221

are actually symbolic links to

222

<filename>log.do_</filename><replaceable>taskname</replaceable><filename>.</filename><replaceable>pid</replaceable>

223

and

224

<filename>log.run_</filename><replaceable>taskname</replaceable><filename>.</filename><replaceable>pid</replaceable>,

225

where <replaceable>pid</replaceable> is the PID the task had when

226

it ran.

227

The symlinks always point to the files corresponding to the most

recent run.

</para>

</section>

<title>Viewing Variable Values</title>

234

<para>

235

BitBake's <filename>-e</filename> option is used to display

236

variable values after parsing.

237

The following command displays the variable values after the

238

configuration files (i.e. <filename>local.conf</filename>,

239

<filename>bblayers.conf</filename>,

240

<filename>bitbake.conf</filename> and so forth) have been

parsed:

$ bitbake -e

</literallayout>

The following command displays variable values after a specific

246

recipe has been parsed.

247

The variables include those from the configuration as well:

248

249

$ bitbake -e recipename

250

</literallayout>

251

252

Each recipe has its own private set of variables (datastore).

253

Internally, after parsing the configuration, a copy of the

254

resulting datastore is made prior to parsing each recipe.

255

This copying implies that variables set in one recipe will

256

not be visible to other recipes.</para>

257

258

<para>Likewise, each task within a recipe gets a private

259

datastore based on the recipe datastore, which means that

260

variables set within one task will not be visible to

other tasks.</para>

</note>

</para>

<para>

In the output of <filename>bitbake -e</filename>, each variable is

267

preceded by a description of how the variable got its value,

268

including temporary values that were later overriden.

269

This description also includes variable flags (varflags) set on

270

the variable.

271

The output can be very helpful during debugging.

</para>

<para>

Variables that are exported to the environment are preceded by

276

<filename>export</filename> in the output of

277

<filename>bitbake -e</filename>.

278

See the following example:

279

280

export CC="i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/ulf/poky/build/tmp/sysroots/qemux86"

</literallayout>

</para>

<para>

In addition to variable values, the output of the

286

<filename>bitbake -e</filename> and

287

<filename>bitbake -e</filename> <replaceable>recipe</replaceable>

288

commands includes the following information:

289

290

291

The output starts with a tree listing all configuration

292

files and classes included globally, recursively listing

293

the files they include or inherit in turn.

294

Much of the behavior of the OpenEmbedded build system

295

(including the behavior of the

296

297

is implemented in the

298

299

class and the classes it inherits, rather than being built

into BitBake itself.

</para></listitem>

After the variable values, all functions appear in the

304

output.

305

For shell functions, variables referenced within the

306

function body are expanded.

307

If a function has been modified using overrides or

308

using override-style operators like

309

<filename>_append</filename> and

310

<filename>_prepend</filename>, then the final assembled

311

function body appears in the output.

</para></listitem>

</itemizedlist>

</para>

</section>

<title>Viewing Package Information with <filename>oe-pkgdata-util</filename></title>

319

320

<para>

321

You can use the <filename>oe-pkgdata-util</filename> command-line

322

utility to query

323

324

and display various package-related information.

325

When you use the utility, you must use it to view information

326

on packages that have already been built.

</para>

<para>

Following are a few of the available

331

<filename>oe-pkgdata-util</filename> subcommands.

332

<note>

333

You can use the standard * and ? globbing wildcards as part of

334

package names and paths.

</note>

<filename>oe-pkgdata-util list-pkgs [</filename><replaceable>pattern</replaceable><filename>]</filename>:

339

Lists all packages that have been built, optionally

340

limiting the match to packages that match

341

<replaceable>pattern</replaceable>.

342

</para></listitem>

343

344

<filename>oe-pkgdata-util list-pkg-files </filename><replaceable>package</replaceable><filename> ...</filename>:

345

Lists the files and directories contained in the given

packages.

<note>

<para>

A different way to view the contents of a package is

350

to look at the

351

<filename>${</filename><link linkend='var-WORKDIR'><filename>WORKDIR</filename></link><filename>}/packages-split</filename>

352

directory of the recipe that generates the

353

package.

354

This directory is created by the

355

356

task and has one subdirectory for each package the

357

recipe generates, which contains the files stored in

358

that package.</para>

359

<para>

360

If you want to inspect the

361

<filename>${WORKDIR}/packages-split</filename>

362

directory, make sure that

363

364

is not enabled when you build the recipe.

</para>

</note>

</para></listitem>

<filename>oe-pkgdata-util find-path </filename><replaceable>path</replaceable><filename> ...</filename>:

370

Lists the names of the packages that contain the given

371

paths.

372

For example, the following tells us that

373

<filename>/usr/share/man/man1/make.1</filename>

374

is contained in the <filename>make-doc</filename>

375

package:

376

377

$ oe-pkgdata-util find-path /usr/share/man/man1/make.1

378

make-doc: /usr/share/man/man1/make.1

</literallayout>

</para></listitem>

<filename>oe-pkgdata-util lookup-recipe </filename><replaceable>package</replaceable><filename> ...</filename>:

383

Lists the name of the recipes that

384

produce the given packages.

</para></listitem>

</itemizedlist>

</para>

<para>

For more information on the <filename>oe-pkgdata-util</filename>

391

command, use the help facility:

392

393

$ oe-pkgdata-util &dash;&dash;help

394

$ oe-pkgdata-util <replaceable>subcommand</replaceable> --help

</literallayout>

</para>

</section>

<title>Viewing Dependencies Between Recipes and Tasks</title>

401

402

<para>

403

Sometimes it can be hard to see why BitBake wants to build other

404

recipes before the one you have specified.

405

Dependency information can help you understand why a recipe is

built.

</para>

<para>

To generate dependency information for a recipe, run the following

411

command:

412

413

$ bitbake -g <replaceable>recipename</replaceable>

414

</literallayout>

415

This command writes the following files in the current directory:

416

417

418

<filename>pn-buildlist</filename>: A list of

419

recipes/targets involved in building

420

<replaceable>recipename</replaceable>.

421

"Involved" here means that at least one task from the

422

recipe needs to run when building

423

<replaceable>recipename</replaceable> from scratch.

Targets that are in

are not listed.

</para></listitem>

Patrick Williams

2017-02-23 20:41:17 -0600

[diff] [blame]

429

<filename>task-depends.dot</filename>: A graph showing

430

dependencies between tasks.

</para></listitem>

</itemizedlist>

</para>

<para>

The graphs are in

format and can be converted to images (e.g. using the

439

<filename>dot</filename> tool from

440

<ulink url='http://www.graphviz.org/'>Graphviz</ulink>).

441

<note><title>Notes</title>

442

443

444

DOT files use a plain text format.

445

The graphs generated using the

446

<filename>bitbake -g</filename> command are often so

447

large as to be difficult to read without special

448

pruning (e.g. with Bitbake's

449

<filename>-I</filename> option) and processing.

450

Despite the form and size of the graphs, the

451

corresponding <filename>.dot</filename> files can still

452

be possible to read and provide useful information.

453

</para>

454

455

<para>As an example, the

456

<filename>task-depends.dot</filename> file contains

457

lines such as the following:

458

459

"libxslt.do_configure" -> "libxml2.do_populate_sysroot"

460

</literallayout>

461

The above example line reveals that the

462

463

task in <filename>libxslt</filename> depends on the

464

465

task in <filename>libxml2</filename>, which is a normal

466

467

dependency between the two recipes.

468

</para></listitem>

469

470

For an example of how <filename>.dot</filename> files

471

can be processed, see the

472

<filename>scripts/contrib/graph-tool</filename> Python

473

script, which finds and displays paths between graph

nodes.

</para></listitem>

</itemizedlist>

</note>

</para>

<para>

You can use a different method to view dependency information

482

by using the following command:

483

Brad Bishop

6e60e8b

2018-02-01 10:27:11 -0500

[diff] [blame]

484

$ bitbake -g -u taskexp <replaceable>recipename</replaceable>

Patrick Williams

2017-02-23 20:41:17 -0600

[diff] [blame]

485

</literallayout>

486

This command displays a GUI window from which you can view

487

build-time and runtime dependencies for the recipes involved in

488

building <replaceable>recipename</replaceable>.

</para>

</section>

<title>Viewing Task Variable Dependencies</title>

<para>

As mentioned in the

"<ulink url='&YOCTO_DOCS_BB_URL;#checksums'>Checksums (Signatures)</ulink>"

498

section of the BitBake User Manual, BitBake tries to automatically

499

determine what variables a task depends on so that it can rerun

500

the task if any values of the variables change.

501

This determination is usually reliable.

502

However, if you do things like construct variable names at runtime,

503

then you might have to manually declare dependencies on those

504

variables using <filename>vardeps</filename> as described in the

505

"<ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'>Variable Flags</ulink>"

506

section of the BitBake User Manual.

</para>

<para>

If you are unsure whether a variable dependency is being picked up

511

automatically for a given task, you can list the variable

512

dependencies BitBake has determined by doing the following:

513

514

515

Build the recipe containing the task:

516

517

$ bitbake <replaceable>recipename</replaceable>

</literallayout>

</para></listitem>

Inside the

directory, find the signature data

524

(<filename>sigdata</filename>) file that corresponds to the

525

task.

526

The <filename>sigdata</filename> files contain a pickled

527

Python database of all the metadata that went into creating

528

the input checksum for the task.

529

As an example, for the

530

531

task of the <filename>db</filename> recipe, the

532

<filename>sigdata</filename> file might be found in the

533

following location:

534

535

${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1

536

</literallayout>

537

For tasks that are accelerated through the shared state

538

(<link linkend='shared-state-cache'>sstate</link>)

539

cache, an additional <filename>siginfo</filename> file is

540

written into

541

542

along with the cached task output.

543

The <filename>siginfo</filename> files contain exactly the

544

same information as <filename>sigdata</filename> files.

545

</para></listitem>

546

547

Run <filename>bitbake-dumpsig</filename> on the

548

<filename>sigdata</filename> or

549

<filename>siginfo</filename> file.

550

Here is an example:

551

552

$ bitbake-dumpsig ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1

553

</literallayout>

554

In the output of the above command, you will find a line

555

like the following, which lists all the (inferred) variable

556

dependencies for the task.

557

This list also includes indirect dependencies from

558

variables depending on other variables, recursively.

559

560

Task dependencies: ['PV', 'SRCREV', 'SRC_URI', 'SRC_URI[md5sum]', 'SRC_URI[sha256sum]', 'base_do_fetch']

561

</literallayout>

562

<note>

563

Functions (e.g. <filename>base_do_fetch</filename>)

564

also count as variable dependencies.

565

These functions in turn depend on the variables they

566

reference.

567

</note>

568

The output of <filename>bitbake-dumpsig</filename> also includes

569

the value each variable had, a list of dependencies for each

570

variable, and

571

<ulink url='&YOCTO_DOCS_BB_URL;#var-BB_HASHBASE_WHITELIST'><filename>BB_HASHBASE_WHITELIST</filename></ulink>

information.

</para></listitem>

</orderedlist>

</para>

<para>

There is also a <filename>bitbake-diffsigs</filename> command for

579

comparing two <filename>siginfo</filename> or

580

<filename>sigdata</filename> files.

581

This command can be helpful when trying to figure out what changed

582

between two versions of a task.

583

If you call <filename>bitbake-diffsigs</filename> with just one

584

file, the command behaves like

585

<filename>bitbake-dumpsig</filename>.

</para>

<para>

You can also use BitBake to dump out the signature construction

590

information without executing tasks by using either of the

591

following BitBake command-line options:

592

593

&dash;&dash;dump-signatures=<replaceable>SIGNATURE_HANDLER</replaceable>

594

-S <replaceable>SIGNATURE_HANDLER</replaceable>

595

</literallayout>

596

<note>

597

Two common values for

598

<replaceable>SIGNATURE_HANDLER</replaceable> are "none" and

599

"printdiff", which dump only the signature or compare the

600

dumped signature with the cached one, respectively.

601

</note>

602

Using BitBake with either of these options causes BitBake to dump

603

out <filename>sigdata</filename> files in the

604

<filename>stamps</filename> directory for every task it would have

605

executed instead of building the specified target package.

Patrick Williams

2015-09-15 14:41:29 -0500

[diff] [blame]

</para>

</section>

<title>Running Specific Tasks</title>

611

612

<para>

Patrick Williams

2017-02-23 20:41:17 -0600

[diff] [blame]

613

Any given recipe consists of a set of tasks.

Patrick Williams

2015-09-15 14:41:29 -0500

[diff] [blame]

614

The standard BitBake behavior in most cases is:

615

<filename>do_fetch</filename>,

616

<filename>do_unpack</filename>,

617

<filename>do_patch</filename>, <filename>do_configure</filename>,

618

<filename>do_compile</filename>, <filename>do_install</filename>,

619

<filename>do_package</filename>,

620

<filename>do_package_write_*</filename>, and

621

<filename>do_build</filename>.

622

The default task is <filename>do_build</filename> and any tasks

623

on which it depends build first.

624

Some tasks, such as <filename>do_devshell</filename>, are not part

625

of the default build chain.

626

If you wish to run a task that is not part of the default build

627

chain, you can use the <filename>-c</filename> option in BitBake.

628

Here is an example:

629

630

$ bitbake matchbox-desktop -c devshell

</literallayout>

</para>

<para>

Patrick Williams

2017-02-23 20:41:17 -0600

[diff] [blame]

635

The <filename>-c</filename> option respects task dependencies,

636

which means that all other tasks (including tasks from other

637

recipes) that the specified task depends on will be run before the

638

task.

639

Even when you manually specify a task to run with

640

<filename>-c</filename>, BitBake will only run the task if it

641

considers it "out of date".

642

See the

643

"<link linkend='stamp-files-and-the-rerunning-of-tasks'>Stamp Files and the Rerunning of Tasks</link>"

644

section for how BitBake determines whether a task is "out of date".

</para>

<para>

If you want to force an up-to-date task to be rerun (e.g.

649

because you made manual modifications to the recipe's

650

651

that you want to try out), then you can use the

652

<filename>-f</filename> option.

653

<note>

654

The reason <filename>-f</filename> is never required when

running the

task is because the

<filename>[</filename><ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'><filename>nostamp</filename></ulink><filename>]</filename>

659

variable flag is already set for the task.

660

</note>

661

The following example shows one way you can use the

662

<filename>-f</filename> option:

Patrick Williams

2015-09-15 14:41:29 -0500

[diff] [blame]

663

664

$ bitbake matchbox-desktop

665

.

666

.

Patrick Williams

2017-02-23 20:41:17 -0600

[diff] [blame]

667

make some changes to the source code in the work directory

Patrick Williams

2015-09-15 14:41:29 -0500

[diff] [blame]

668

.

669

.

670

$ bitbake matchbox-desktop -c compile -f

671

$ bitbake matchbox-desktop

</literallayout>

</para>

<para>

This sequence first builds and then recompiles

677

<filename>matchbox-desktop</filename>.

678

The last command reruns all tasks (basically the packaging tasks)

679

after the compile.

680

BitBake recognizes that the <filename>do_compile</filename>

681

task was rerun and therefore understands that the other tasks

682

also need to be run again.

683

</para>

684

685

<para>

Patrick Williams

2017-02-23 20:41:17 -0600

[diff] [blame]

686

Another, shorter way to rerun a task and all

687

688

that depend on it is to use the <filename>-C</filename>

689

option.

690

<note>

691

This option is upper-cased and is separate from the

692

<filename>-c</filename> option, which is lower-cased.

693

</note>

694

Using this option invalidates the given task and then runs the

695

696

task, which is the default task if no task is given, and the

697

tasks on which it depends.

698

You could replace the final two commands in the previous example

699

with the following single command:

700

701

$ bitbake matchbox-desktop -C compile

702

</literallayout>

703

Internally, the <filename>-f</filename> and

704

<filename>-C</filename> options work by tainting (modifying) the

705

input checksum of the specified task.

706

This tainting indirectly causes the task and its

707

dependent tasks to be rerun through the normal task dependency

708

mechanisms.

709

<note>

710

BitBake explicitly keeps track of which tasks have been

711

tainted in this fashion, and will print warnings such as the

712

following for builds involving such tasks:

713

714

WARNING: /home/ulf/poky/meta/recipes-sato/matchbox-desktop/matchbox-desktop_2.1.bb.do_compile is tainted from a forced run

715

</literallayout>

716

The purpose of the warning is to let you know that the work

717

directory and build output might not be in the clean state they

718

would be in for a "normal" build, depending on what actions

719

you took.

720

To get rid of such warnings, you can remove the work directory

721

and rebuild the recipe, as follows:

722

723

$ bitbake matchbox-desktop -c clean

724

$ bitbake matchbox-desktop

</literallayout>

</note>

</para>

<para>

Patrick Williams

2015-09-15 14:41:29 -0500

[diff] [blame]

730

You can view a list of tasks in a given package by running the

731

<filename>do_listtasks</filename> task as follows:

732

733

$ bitbake matchbox-desktop -c listtasks

734

</literallayout>

735

The results appear as output to the console and are also in the

736

file <filename>${WORKDIR}/temp/log.do_listtasks</filename>.

</para>

</section>

Patrick Williams

2015-09-15 14:41:29 -0500

[diff] [blame]

740

741

<title>General BitBake Problems</title>

742

743

<para>

744

You can see debug output from BitBake by using the <filename>-D</filename> option.

745

The debug output gives more information about what BitBake

746

is doing and the reason behind it.

747

Each <filename>-D</filename> option you use increases the logging level.

748

The most common usage is <filename>-DDD</filename>.

</para>

<para>

The output from <filename>bitbake -DDD -v</filename> <replaceable>targetname</replaceable> can reveal why

753

BitBake chose a certain version of a package or why BitBake

754

picked a certain provider.

755

This command could also help you in a situation where you think BitBake did something

unexpected.

</para>

</section>

<title>Development Host System Issues</title>

762

763

<para>

764

Sometimes issues on the host development system can cause your

765

build to fail.

766

Following are known, host-specific problems.

767

Be sure to always consult the

768

<ulink url='&YOCTO_RELEASE_NOTES;'>Release Notes</ulink>

769

for a look at all release-related issues.

770

771

<listitem><para><emphasis><filename>glibc-initial</filename> fails to build</emphasis>:

772

If your development host system has the unpatched

773

<filename>GNU Make 3.82</filename>,

774

the

775

776

task fails for <filename>glibc-initial</filename> during

777

the build.</para>

778

<para>Typically, every distribution that ships

779

<filename>GNU Make 3.82</filename> as

780

the default already has the patched version.

781

However, some distributions, such as Debian, have

782

<filename>GNU Make 3.82</filename> as an option, which

783

is unpatched.

784

You will see this error on these types of distributions.

785

Switch to <filename>GNU Make 3.81</filename> or patch

786

your <filename>make</filename> to solve the problem.

</para></listitem>

</itemizedlist>

</para>

</section>

<title>Building with No Dependencies</title>

794

<para>

795

To build a specific recipe (<filename>.bb</filename> file),

796

you can use the following command form:

797

798

$ bitbake -b <replaceable>somepath</replaceable>/<replaceable>somerecipe</replaceable>.bb

799

</literallayout>

800

This command form does not check for dependencies.

801

Consequently, you should use it

802

only when you know existing dependencies have been met.

803

<note>

804

You can also specify fragments of the filename.

805

In this case, BitBake checks for a unique match.

</note>

</para>

</section>

Patrick Williams

2015-09-15 14:41:29 -0500

[diff] [blame]

810

811

<title>Recipe Logging Mechanisms</title>

812

<para>

Patrick Williams

2017-02-23 20:41:17 -0600

[diff] [blame]

813

The Yocto Project provides several logging functions for producing

814

debugging output and reporting errors and warnings.

815

For Python functions, the following logging functions exist.

816

All of these functions log to

817

<filename>${T}/log.do_</filename><replaceable>task</replaceable>,

818

and can also log to standard output (stdout) with the right

819

settings:

Patrick Williams

2015-09-15 14:41:29 -0500

[diff] [blame]

820

Patrick Williams

2017-02-23 20:41:17 -0600

[diff] [blame]

821

822

<filename>bb.plain(</filename><replaceable>msg</replaceable><filename>)</filename>:

823

Writes <replaceable>msg</replaceable> as is to the log while

824

also logging to stdout.

825

</para></listitem>

826

827

<filename>bb.note(</filename><replaceable>msg</replaceable><filename>)</filename>:

828

Writes "NOTE: <replaceable>msg</replaceable>" to the log.

829

Also logs to stdout if BitBake is called with "-v".

830

</para></listitem>

831

832

<filename>bb.debug(</filename><replaceable>level</replaceable><filename>, </filename><replaceable>msg</replaceable><filename>)</filename>:

833

Writes "DEBUG: <replaceable>msg</replaceable>" to the log.

834

Also logs to stdout if the log level is greater than or

835

equal to <replaceable>level</replaceable>.

836

See the

837

"<ulink url='&YOCTO_DOCS_BB_URL;#usage-and-syntax'>-D</ulink>"

838

option in the BitBake User Manual for more information.

839

</para></listitem>

840

841

<filename>bb.warn(</filename><replaceable>msg</replaceable><filename>)</filename>:

842

Writes "WARNING: <replaceable>msg</replaceable>" to the log

843

while also logging to stdout.

844

</para></listitem>

845

846

<filename>bb.error(</filename><replaceable>msg</replaceable><filename>)</filename>:

847

Writes "ERROR: <replaceable>msg</replaceable>" to the log

848

while also logging to stdout.

849

<note>

850

Calling this function does not cause the task to fail.

</note>

</para></listitem>

<filename>bb.fatal(</filename><replaceable>msg</replaceable><filename>)</filename>:

855

This logging function is similar to

856

<filename>bb.error(</filename><replaceable>msg</replaceable><filename>)</filename>

857

but also causes the calling task to fail.

858

<note>

859

<filename>bb.fatal()</filename> raises an exception,

860

which means you do not need to put a "return"

861

statement after the function.

862

</note>

863

</para></listitem>

Patrick Williams

2015-09-15 14:41:29 -0500

[diff] [blame]

</itemizedlist>

</para>

<para>

Patrick Williams

2017-02-23 20:41:17 -0600

[diff] [blame]

868

The same logging functions are also available in shell functions,

869

under the names

870

<filename>bbplain</filename>, <filename>bbnote</filename>,

871

<filename>bbdebug</filename>, <filename>bbwarn</filename>,

872

<filename>bberror</filename>, and <filename>bbfatal</filename>.

873

The

874

875

class implements these functions.

876

See that class in the

Patrick Williams

2015-09-15 14:41:29 -0500

[diff] [blame]

877

<filename>meta/classes</filename> folder of the

Brad Bishop

2018-02-25 22:55:05 -0500

[diff] [blame]

878

Patrick Williams

2017-02-23 20:41:17 -0600

[diff] [blame]

879

for information.

Patrick Williams

2015-09-15 14:41:29 -0500

[diff] [blame]

</para>

<title>Logging With Python</title>

884

<para>

885

When creating recipes using Python and inserting code that handles build logs,

886

keep in mind the goal is to have informative logs while keeping the console as

887

"silent" as possible.

888

Also, if you want status messages in the log, use the "debug" loglevel.

</para>

<para>

Following is an example written in Python.

893

The code handles logging for a function that determines the

894

number of tasks needed to be run.

895

See the

896

"<link linkend='ref-tasks-listtasks'><filename>do_listtasks</filename></link>"

897

section for additional information:

898

899

python do_listtasks() {

900

bb.debug(2, "Starting to figure out the task list")

901

if noteworthy_condition:

902

bb.note("There are 47 tasks to run")

903

bb.debug(2, "Got to point xyz")

904

if warning_trigger:

905

bb.warn("Detected warning_trigger, this might be a problem later.")

906

if recoverable_error:

907

bb.error("Hit recoverable_error, you really need to fix this!")

908

if fatal_error:

909

bb.fatal("fatal_error detected, unable to print the task list")

910

bb.plain("The tasks present are abc")

911

bb.debug(2, "Finished figuring out the tasklist")

}

</literallayout>

</para>

</section>

<title>Logging With Bash</title>

919

<para>

920

When creating recipes using Bash and inserting code that handles build

921

logs, you have the same goals - informative with minimal console output.

922

The syntax you use for recipes written in Bash is similar to that of

923

recipes written in Python described in the previous section.

</para>

<para>

Following is an example written in Bash.

928

The code logs the progress of the <filename>do_my_function</filename> function.

929

930

do_my_function() {

931

bbdebug 2 "Running do_my_function"

932

if [ exceptional_condition ]; then

933

bbnote "Hit exceptional_condition"

934

fi

935

bbdebug 2 "Got to point xyz"

936

if [ warning_trigger ]; then

937

bbwarn "Detected warning_trigger, this might cause a problem later."

938

fi

939

if [ recoverable_error ]; then

940

bberror "Hit recoverable_error, correcting"

941

fi

942

if [ fatal_error ]; then

943

bbfatal "fatal_error detected"

944

fi

945

bbdebug 2 "Completed do_my_function"

}

</literallayout>

</para>

</section>

</section>

<title>Other Tips</title>

954

955

<para>

956

Here are some other tips that you might find useful:

957

Patrick Williams

2017-02-23 20:41:17 -0600

[diff] [blame]

958

959

When adding new packages, it is worth watching for

960

undesirable items making their way into compiler command

961

lines.

962

For example, you do not want references to local system

963

files like

964

<filename>/usr/lib/</filename> or

965

<filename>/usr/include/</filename>.

Patrick Williams

2015-09-15 14:41:29 -0500

[diff] [blame]

966

</para></listitem>

Patrick Williams

2017-02-23 20:41:17 -0600

[diff] [blame]

967

968

If you want to remove the <filename>psplash</filename>

Patrick Williams

2015-09-15 14:41:29 -0500

[diff] [blame]

969

boot splashscreen,

Patrick Williams

2017-02-23 20:41:17 -0600

[diff] [blame]

970

add <filename>psplash=false</filename> to the kernel

971

command line.

Patrick Williams

2015-09-15 14:41:29 -0500

[diff] [blame]

972

Doing so prevents <filename>psplash</filename> from loading

973

and thus allows you to see the console.

974

It is also possible to switch out of the splashscreen by

Patrick Williams

2017-02-23 20:41:17 -0600

[diff] [blame]

975

switching the virtual console (e.g. Fn+Left or Fn+Right

on a Zaurus).

</para></listitem>

Removing

(usually <filename>tmp/</filename>, within the

Brad Bishop

2018-02-25 22:55:05 -0500

[diff] [blame]

982

Patrick Williams

2017-02-23 20:41:17 -0600

[diff] [blame]

983

can often fix temporary build issues.

984

Removing <filename>TMPDIR</filename> is usually a

985

relatively cheap operation, because task output will be

986

cached in

987

988

(usually <filename>sstate-cache/</filename>, which is

989

also in the Build Directory).

990

<note>

991

Removing <filename>TMPDIR</filename> might be a

992

workaround rather than a fix.

993

Consequently, trying to determine the underlying cause

994

of an issue before removing the directory is a good

idea.

</note>

</para></listitem>

Understanding how a feature is used in practice within

1000

existing recipes can be very helpful.

1001

It is recommended that you configure some method that

1002

allows you to quickly search through files.</para>

1003

1004

<para>Using GNU Grep, you can use the following shell

1005

function to recursively search through common

1006

recipe-related files, skipping binary files,

1007

<filename>.git</filename> directories, and the

1008

Build Directory (assuming its name starts with

"build"):

g() {

grep -Ir \

--exclude-dir=.git \

--exclude-dir='build*' \

1015

--include='*.bb*' \

1016

--include='*.inc*' \

1017

--include='*.conf*' \

--include='*.py*' \

"$@"

}

</literallayout>

Following are some usage examples:

1023

1024

$ g FOO # Search recursively for "FOO"

1025

$ g -i foo # Search recursively for "foo", ignoring case

1026

$ g -w FOO # Search recursively for "FOO" as a word, ignoring e.g. "FOOBAR"

1027

</literallayout>

1028

If figuring out how some feature works requires a lot of

1029

searching, it might indicate that the documentation should

1030

be extended or improved.

1031

In such cases, consider filing a documentation bug using

1032

the Yocto Project implementation of

1033

<ulink url='https://bugzilla.yoctoproject.org/'>Bugzilla</ulink>.

1034

For general information on how to submit a bug against

Brad Bishop

2018-02-25 22:55:05 -0500

[diff] [blame]

1035

the Yocto Project, see the Yocto Project Bugzilla

1036

<ulink url='&YOCTO_WIKI_URL;/wiki/Bugzilla_Configuration_and_Bug_Tracking'>wiki page</ulink>"

1037

or the

1038

<ulink url='&YOCTO_DOCS_DEV_URL;#submitting-a-defect-against-the-yocto-project'>Submitting a Defect Against the Yocto Project</ulink>"

1039

section, which is in the Yocto Project Development Tasks

1040

Manual.

Patrick Williams

2017-02-23 20:41:17 -0600

[diff] [blame]

1041

<note>

1042

The manuals might not be the right place to document

1043

variables that are purely internal and have a limited

1044

scope (e.g. internal variables used to implement a

1045

single <filename>.bbclass</filename> file).

1046

</note>

Patrick Williams

2015-09-15 14:41:29 -0500

[diff] [blame]

</para></listitem>

</itemizedlist>

</para>

</section>

</section>

Brad Bishop

2018-02-25 22:55:05 -0500

[diff] [blame]

1053

1054

<title>Quick EMUlator (QEMU)</title>

1055

1056

<para>

1057

The Yocto Project uses an implementation of the Quick EMUlator (QEMU)

1058

Open Source project as part of the Yocto Project development "tool

set".

</para>

<para>

Within the context of the Yocto Project, QEMU is an

1064

emulator and virtualization machine that allows you to run a complete

1065

image you have built using the Yocto Project as just another task

1066

on your build system.

1067

QEMU is useful for running and testing images and applications on

1068

supported Yocto Project architectures without having actual hardware.

1069

Among other things, the Yocto Project uses QEMU to run automated

1070

Quality Assurance (QA) tests on final images shipped with each

1071

release.

1072

<note>

1073

This implementation is not the same as QEMU in general.

1074

</note>

1075

This section provides a brief reference for the Yocto Project

1076

implementation of QEMU.

</para>

<para>

For official information and documentation on QEMU in general, see the

1081

following references:

1082

1083

1084

<emphasis><ulink url='http://wiki.qemu.org/Main_Page'>QEMU Website</ulink>:</emphasis>

1085

The official website for the QEMU Open Source project.

1086

</para></listitem>

1087

1088

<emphasis><ulink url='http://wiki.qemu.org/Manual'>Documentation</ulink>:</emphasis>

1089

The QEMU user manual.

</para></listitem>

</itemizedlist>

</para>

<para>

For information on how to use the Yocto Project implementation of

1096

QEMU, see the

1097

"<ulink url='&YOCTO_DOCS_DEV_URL;#dev-manual-qemu'>Using the Quick EMUlator (QEMU)</ulink>"

1098

chapter in the Yocto Project Development Tasks Manual.

</para>

<title>QEMU Availability</title>

1103

1104

<para>

1105

QEMU is made available with the Yocto Project a number of ways.

1106

One method is to install a Software Development Kit (SDK).

1107

For more information on how to make sure you have

1108

QEMU available, see

1109

"<ulink url='&YOCTO_DOCS_SDK_URL;#the-qemu-emulator'>The QEMU Emulator</ulink>"

1110

section in the Yocto Project Application Development and the

1111

Extensible Software Development Kit (eSDK) manual.

</para>

</section>

<title>QEMU Performance</title>

1117

1118

<para>

1119

Using QEMU to emulate your hardware can result in speed issues

1120

depending on the target and host architecture mix.

1121

For example, using the <filename>qemux86</filename> image in the

1122

emulator on an Intel-based 32-bit (x86) host machine is fast

1123

because the target and host architectures match.

1124

On the other hand, using the <filename>qemuarm</filename> image

1125

on the same Intel-based host can be slower.

1126

But, you still achieve faithful emulation of ARM-specific issues.

</para>

<para>

To speed things up, the QEMU images support using

1131

<filename>distcc</filename> to call a cross-compiler outside the

1132

emulated system.

1133

If you used <filename>runqemu</filename> to start QEMU, and the

1134

<filename>distccd</filename> application is present on the host

1135

system, any BitBake cross-compiling toolchain available from the

1136

build system is automatically used from within QEMU simply by

1137

calling <filename>distcc</filename>.

1138

You can accomplish this by defining the cross-compiler variable

1139

(e.g. <filename>export CC="distcc"</filename>).

1140

Alternatively, if you are using a suitable SDK image or the

1141

appropriate stand-alone toolchain is present, the toolchain is

1142

also automatically used.

</para>

<note>

Several mechanisms exist that let you connect to the system

1147

running on the QEMU emulator:

1148

1149

1150

QEMU provides a framebuffer interface that makes standard

consoles available.

</para></listitem>

Generally, headless embedded devices have a serial port.

1155

If so, you can configure the operating system of the

1156

running image to use that port to run a console.

1157

The connection uses standard IP networking.

1158

</para></listitem>

1159

1160

SSH servers exist in some QEMU images.

1161

The <filename>core-image-sato</filename> QEMU image has a

1162

Dropbear secure shell (SSH) server that runs with the root

1163

password disabled.

1164

The <filename>core-image-full-cmdline</filename> and

1165

<filename>core-image-lsb</filename> QEMU images

1166

have OpenSSH instead of Dropbear.

1167

Including these SSH servers allow you to use standard

1168

1169

commands.

1170

The <filename>core-image-minimal</filename> QEMU image,

1171

however, contains no SSH server.

1172

</para></listitem>

1173

1174

You can use a provided, user-space NFS server to boot

1175

the QEMU session using a local copy of the root

1176

filesystem on the host.

1177

In order to make this connection, you must extract a

1178

root filesystem tarball by using the

1179

<filename>runqemu-extract-sdk</filename> command.

1180

After running the command, you must then point the

1181

<filename>runqemu</filename>

1182

script to the extracted directory instead of a root

1183

filesystem image file.

1184

See the

1185

"<ulink url='&YOCTO_DOCS_DEV_URL;#qemu-running-under-a-network-file-system-nfs-server'>Running Under a Network File System (NFS) Server</ulink>"

1186

section in the Yocto Project Development Tasks Manual for

more information.

</para></listitem>

</itemizedlist>

</note>

</section>

<title>QEMU Command-Line Syntax</title>

1195

1196

<para>

1197

The basic <filename>runqemu</filename> command syntax is as

1198

follows:

1199

1200

$ runqemu [<replaceable>option</replaceable> ] [...]

1201

</literallayout>

1202

Based on what you provide on the command line,

1203

<filename>runqemu</filename> does a good job of figuring out what

1204

you are trying to do.

1205

For example, by default, QEMU looks for the most recently built

1206

image according to the timestamp when it needs to look for an

1207

image.

1208

Minimally, through the use of options, you must provide either

1209

a machine name, a virtual machine image

1210

(<filename>*wic.vmdk</filename>), or a kernel image

1211

(<filename>*.bin</filename>).

</para>

<para>

Following is the command-line help output for the

1216

<filename>runqemu</filename> command:

$ runqemu --help

Usage: you can run this script with any valid combination

1221

of the following environment variables (in any order):

1222

KERNEL - the kernel image file to use

1223

ROOTFS - the rootfs image file or nfsroot directory to use

1224

MACHINE - the machine name (optional, autodetected from KERNEL filename if unspecified)

1225

Simplified QEMU command-line options can be passed with:

1226

nographic - disable video console

1227

serial - enable a serial console on /dev/ttyS0

1228

slirp - enable user networking, no root privileges is required

1229

kvm - enable KVM when running x86/x86_64 (VT-capable CPU required)

1230

kvm-vhost - enable KVM with vhost when running x86/x86_64 (VT-capable CPU required)

1231

publicvnc - enable a VNC server open to all hosts

1232

audio - enable audio

1233

[*/]ovmf* - OVMF firmware file or base name for booting with UEFI

1234

tcpserial=<port> - specify tcp serial port number

1235

biosdir=<dir> - specify custom bios dir

1236

biosfilename=<filename> - specify bios filename

1237

qemuparams=<xyz> - specify custom parameters to QEMU

1238

bootparams=<xyz> - specify custom kernel parameters during boot

1239

help, -h, --help: print this text

Examples:

runqemu

runqemu qemuarm

runqemu tmp/deploy/images/qemuarm

1245

runqemu tmp/deploy/images/qemux86/<qemuboot.conf>

1246

runqemu qemux86-64 core-image-sato ext4

1247

runqemu qemux86-64 wic-image-minimal wic

1248

runqemu path/to/bzImage-qemux86.bin path/to/nfsrootdir/ serial

1249

runqemu qemux86 iso/hddimg/wic.vmdk/wic.qcow2/wic.vdi/ramfs/cpio.gz...

1250

runqemu qemux86 qemuparams="-m 256"

1251

runqemu qemux86 bootparams="psplash=false"

1252

runqemu path/to/<image>-<machine>.wic

1253

runqemu path/to/<image>-<machine>.wic.vmdk

</literallayout>

</para>

</section>

<title><filename>runqemu</filename> Command-Line Options</title>

1260

1261

<para>

1262

Following is a description of <filename>runqemu</filename>

1263

options you can provide on the command line:

1264

1265

If you do provide some "illegal" option combination or perhaps

1266

you do not provide enough in the way of options,

1267

<filename>runqemu</filename> provides appropriate error

1268

messaging to help you correct the problem.

</note>

<replaceable>QEMUARCH</replaceable>:

1273

The QEMU machine architecture, which must be "qemuarm",

1274

"qemuarm64", "qemumips", "qemumips64", "qemuppc",

1275

"qemux86", or "qemux86-64".

1276

</para></listitem>

1277

1278

<filename><replaceable>VM</replaceable></filename>:

1279

The virtual machine image, which must be a

1280

<filename>.wic.vmdk</filename> file.

1281

Use this option when you want to boot a

1282

<filename>.wic.vmdk</filename> image.

1283

The image filename you provide must contain one of the

1284

following strings: "qemux86-64", "qemux86", "qemuarm",

1285

"qemumips64", "qemumips", "qemuppc", or "qemush4".

1286

</para></listitem>

1287

1288

<replaceable>ROOTFS</replaceable>:

1289

A root filesystem that has one of the following

1290

filetype extensions: "ext2", "ext3", "ext4", "jffs2",

1291

"nfs", or "btrfs".

1292

If the filename you provide for this option uses “nfs”, it

1293

must provide an explicit root filesystem path.

1294

</para></listitem>

1295

1296

<replaceable>KERNEL</replaceable>:

1297

A kernel image, which is a <filename>.bin</filename> file.

1298

When you provide a <filename>.bin</filename> file,

1299

<filename>runqemu</filename> detects it and assumes the

1300

file is a kernel image.

1301

</para></listitem>

1302

1303

<replaceable>MACHINE</replaceable>:

1304

The architecture of the QEMU machine, which must be one

1305

of the following: "qemux86", "qemux86-64", "qemuarm",

1306

"qemuarm64", "qemumips", “qemumips64", or "qemuppc".

1307

The <replaceable>MACHINE</replaceable> and

1308

<replaceable>QEMUARCH</replaceable> options are basically

1309

identical.

1310

If you do not provide a <replaceable>MACHINE</replaceable>

1311

option, <filename>runqemu</filename> tries to determine

1312

it based on other options.

1313

</para></listitem>

1314

1315

<filename>ramfs</filename>:

1316

Indicates you are booting an initial RAM disk (initramfs)

1317

image, which means the <filename>FSTYPE</filename> is

1318

<filename>cpio.gz</filename>.

1319

</para></listitem>

1320

1321

<filename>iso</filename>:

1322

Indicates you are booting an ISO image, which means the

1323

<filename>FSTYPE</filename> is

1324

<filename>.iso</filename>.

1325

</para></listitem>

1326

1327

<filename>nographic</filename>:

1328

Disables the video console, which sets the console to

"ttys0".

</para></listitem>

<filename>serial</filename>:

1333

Enables a serial console on

1334

<filename>/dev/ttyS0</filename>.

1335

</para></listitem>

1336

1337

<filename>biosdir</filename>:

1338

Establishes a custom directory for BIOS, VGA BIOS and

keymaps.

</para></listitem>

<filename>biosfilename</filename>:

1343

Establishes a custom BIOS name.

1344

</para></listitem>

1345

1346

<filename>qemuparams=\"<replaceable>xyz</replaceable>\"</filename>:

1347

Specifies custom QEMU parameters.

1348

Use this option to pass options other than the simple

1349

"kvm" and "serial" options.

1350

</para></listitem>

1351

<listitem><para><filename>bootparams=\"<replaceable>xyz</replaceable>\"</filename>:

1352

Specifies custom boot parameters for the kernel.

1353

</para></listitem>

1354

1355

<filename>audio</filename>:

1356

Enables audio in QEMU.

1357

The <replaceable>MACHINE</replaceable> option must be

1358

either "qemux86" or "qemux86-64" in order for audio to be

1359

enabled.

1360

Additionally, the <filename>snd_intel8x0</filename>

1361

or <filename>snd_ens1370</filename> driver must be

1362

installed in linux guest.

1363

</para></listitem>

1364

1365

<filename>slirp</filename>:

1366

Enables "slirp" networking, which is a different way

1367

of networking that does not need root access

1368

but also is not as easy to use or comprehensive

as the default.

</para></listitem>

<filename>kvm</filename>:

1373

Enables KVM when running "qemux86" or "qemux86-64"

1374

QEMU architectures.

1375

For KVM to work, all the following conditions must be met:

1376

1377

1378

Your <replaceable>MACHINE</replaceable> must be either

1379

qemux86" or "qemux86-64".

1380

</para></listitem>

1381

1382

Your build host has to have the KVM modules

1383

installed, which are

1384

<filename>/dev/kvm</filename>.

1385

</para></listitem>

1386

1387

The build host <filename>/dev/kvm</filename>

1388

directory has to be both writable and readable.

</para></listitem>

</itemizedlist>

</para></listitem>

<filename>kvm-vhost</filename>:

1394

Enables KVM with VHOST support when running "qemux86"

1395

or "qemux86-64" QEMU architectures.

1396

For KVM with VHOST to work, the following conditions must

be met:

conditions must be met.

1402

</para></listitem>

1403

1404

Your build host has to have virtio net device, which

1405

are <filename>/dev/vhost-net</filename>.

1406

</para></listitem>

1407

1408

The build host <filename>/dev/vhost-net</filename>

1409

directory has to be either readable or writable

and “slirp-enabled”.

</para></listitem>

</itemizedlist>

</para></listitem>

<filename>publicvnc</filename>:

1416

Enables a VNC server open to all hosts.

</para></listitem>

</itemizedlist>

</para>

</section>

</section>

Patrick Williams

2015-09-15 14:41:29 -0500

[diff] [blame]

1423

1424

<title>Maintaining Build Output Quality</title>

1425

1426

<para>

1427

Many factors can influence the quality of a build.

1428

For example, if you upgrade a recipe to use a new version of an upstream software

1429

package or you experiment with some new configuration options, subtle changes

1430

can occur that you might not detect until later.

1431

Consider the case where your recipe is using a newer version of an upstream package.

1432

In this case, a new version of a piece of software might introduce an optional

1433

dependency on another library, which is auto-detected.

1434

If that library has already been built when the software is building,

1435

the software will link to the built library and that library will be pulled

1436

into your image along with the new software even if you did not want the

library.

</para>

<para>

The

class exists to help you maintain

1444

the quality of your build output.

1445

You can use the class to highlight unexpected and possibly unwanted

1446

changes in the build output.

1447

When you enable build history, it records information about the contents of

1448

each package and image and then commits that information to a local Git

1449

repository where you can examine the information.

</para>

<para>

The remainder of this section describes the following:

1454

1455

<listitem><para>How you can enable and disable

1456

build history</para></listitem>

1457

<listitem><para>How to understand what the build history contains

1458

</para></listitem>

1459

<listitem><para>How to limit the information used for build history

1460

</para></listitem>

1461

<listitem><para>How to examine the build history from both a

1462

command-line and web interface</para></listitem>

</itemizedlist>

</para>

<title>Enabling and Disabling Build History</title>

1468

1469

<para>

1470

Build history is disabled by default.

1471

To enable it, add the following <filename>INHERIT</filename>

1472

statement and set the

1473

1474

variable to "1" at the end of your

1475

<filename>conf/local.conf</filename> file found in the

Brad Bishop

2018-02-25 22:55:05 -0500

[diff] [blame]

1476

Patrick Williams

2015-09-15 14:41:29 -0500

[diff] [blame]

1477

1478

INHERIT += "buildhistory"

1479

BUILDHISTORY_COMMIT = "1"

1480

</literallayout>

Brad Bishop

2018-02-25 22:55:05 -0500

[diff] [blame]

1481

Enabling build history as previously described causes the

1482

OpenEmbedded build system to collect build output information and

1483

commit it as a single commit to a local

1484

Patrick Williams

2015-09-15 14:41:29 -0500

[diff] [blame]

1485

<note>

1486

Enabling build history increases your build times slightly,

1487

particularly for images, and increases the amount of disk

1488

space used during the build.

</note>

</para>

<para>

You can disable build history by removing the previous statements

1494

from your <filename>conf/local.conf</filename> file.

</para>

</section>

<title>Understanding What the Build History Contains</title>

1500

1501

<para>

1502

Build history information is kept in

1503

<filename>${</filename><link linkend='var-TOPDIR'><filename>TOPDIR</filename></link><filename>}/buildhistory</filename>

1504

in the Build Directory as defined by the

1505

1506

variable.

1507

The following is an example abbreviated listing:

</para>

<para>

At the top level, there is a <filename>metadata-revs</filename> file

1513

that lists the revisions of the repositories for the layers enabled

1514

when the build was produced.

1515

The rest of the data splits into separate

1516

<filename>packages</filename>, <filename>images</filename> and

1517

<filename>sdk</filename> directories, the contents of which are

described below.

</para>

<title>Build History Package Information</title>

1523

1524

<para>

1525

The history for each package contains a text file that has

1526

name-value pairs with information about the package.

1527

For example, <filename>buildhistory/packages/i586-poky-linux/busybox/busybox/latest</filename>

1528

contains the following:

PV = 1.22.1

PR = r32

RPROVIDES =

RDEPENDS = glibc (>= 2.20) update-alternatives-opkg

1534

RRECOMMENDS = busybox-syslog busybox-udhcpc update-rc.d

1535

PKGSIZE = 540168

1536

FILES = /usr/bin/* /usr/sbin/* /usr/lib/busybox/* /usr/lib/lib*.so.* \

1537

/etc /com /var /bin/* /sbin/* /lib/*.so.* /lib/udev/rules.d \

1538

/usr/lib/udev/rules.d /usr/share/busybox /usr/lib/busybox/* \

1539

/usr/share/pixmaps /usr/share/applications /usr/share/idl \

1540

/usr/share/omf /usr/share/sounds /usr/lib/bonobo/servers

1541

FILELIST = /bin/busybox /bin/busybox.nosuid /bin/busybox.suid /bin/sh \

1542

/etc/busybox.links.nosuid /etc/busybox.links.suid

1543

</literallayout>

1544

Most of these name-value pairs correspond to variables used

1545

to produce the package.

1546

The exceptions are <filename>FILELIST</filename>, which is the

1547

actual list of files in the package, and

1548

<filename>PKGSIZE</filename>, which is the total size of files

1549

in the package in bytes.

</para>

<para>

There is also a file corresponding to the recipe from which the

1554

package came (e.g.

1555

<filename>buildhistory/packages/i586-poky-linux/busybox/latest</filename>):

PV = 1.22.1

PR = r32

DEPENDS = initscripts kern-tools-native update-rc.d-native \

1560

virtual/i586-poky-linux-compilerlibs virtual/i586-poky-linux-gcc \

1561

virtual/libc virtual/update-alternatives

1562

PACKAGES = busybox-ptest busybox-httpd busybox-udhcpd busybox-udhcpc \

1563

busybox-syslog busybox-mdev busybox-hwclock busybox-dbg \

1564

busybox-staticdev busybox-dev busybox-doc busybox-locale busybox

</literallayout>

</para>

<para>

Finally, for those recipes fetched from a version control

1570

system (e.g., Git), a file exists that lists source revisions

1571

that are specified in the recipe and lists the actual revisions

1572

used during the build.

1573

Listed and actual revisions might differ when

1574

1575

is set to

1576

<filename>${<link linkend='var-AUTOREV'>AUTOREV</link>}</filename>.

1577

Here is an example assuming

1578

<filename>buildhistory/packages/qemux86-poky-linux/linux-yocto/latest_srcrev</filename>):

1579

1580

# SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"

1581

SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"

1582

# SRCREV_meta = "a227f20eff056e511d504b2e490f3774ab260d6f"

1583

SRCREV_meta = "a227f20eff056e511d504b2e490f3774ab260d6f"

1584

</literallayout>

1585

You can use the <filename>buildhistory-collect-srcrevs</filename>

1586

command with the <filename>-a</filename> option to

1587

collect the stored <filename>SRCREV</filename> values

1588

from build history and report them in a format suitable for

1589

use in global configuration (e.g.,

1590

<filename>local.conf</filename> or a distro include file) to

1591

override floating <filename>AUTOREV</filename> values to a

1592

fixed set of revisions.

1593

Here is some example output from this command:

1594

1595

$ buildhistory-collect-srcrevs -a

1596

# i586-poky-linux

1597

SRCREV_pn-glibc = "b8079dd0d360648e4e8de48656c5c38972621072"

1598

SRCREV_pn-glibc-initial = "b8079dd0d360648e4e8de48656c5c38972621072"

1599

SRCREV_pn-opkg-utils = "53274f087565fd45d8452c5367997ba6a682a37a"

1600

SRCREV_pn-kmod = "fd56638aed3fe147015bfa10ed4a5f7491303cb4"

1601

# x86_64-linux

1602

SRCREV_pn-gtk-doc-stub-native = "1dea266593edb766d6d898c79451ef193eb17cfa"

1603

SRCREV_pn-dtc-native = "65cc4d2748a2c2e6f27f1cf39e07a5dbabd80ebf"

1604

SRCREV_pn-update-rc.d-native = "eca680ddf28d024954895f59a241a622dd575c11"

1605

SRCREV_glibc_pn-cross-localedef-native = "b8079dd0d360648e4e8de48656c5c38972621072"

1606

SRCREV_localedef_pn-cross-localedef-native = "c833367348d39dad7ba018990bfdaffaec8e9ed3"

1607

SRCREV_pn-prelink-native = "faa069deec99bf61418d0bab831c83d7c1b797ca"

1608

SRCREV_pn-opkg-utils-native = "53274f087565fd45d8452c5367997ba6a682a37a"

1609

SRCREV_pn-kern-tools-native = "23345b8846fe4bd167efdf1bd8a1224b2ba9a5ff"

1610

SRCREV_pn-kmod-native = "fd56638aed3fe147015bfa10ed4a5f7491303cb4"

1611

# qemux86-poky-linux

1612

SRCREV_machine_pn-linux-yocto = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"

1613

SRCREV_meta_pn-linux-yocto = "a227f20eff056e511d504b2e490f3774ab260d6f"

1614

# all-poky-linux

1615

SRCREV_pn-update-rc.d = "eca680ddf28d024954895f59a241a622dd575c11"

1616

</literallayout>

1617

<note>

1618

Here are some notes on using the

1619

<filename>buildhistory-collect-srcrevs</filename> command:

1620

1621

<listitem><para>By default, only values where the

1622

<filename>SRCREV</filename> was

1623

not hardcoded (usually when <filename>AUTOREV</filename>

1624

was used) are reported.

1625

Use the <filename>-a</filename> option to see all

1626

<filename>SRCREV</filename> values.

1627

</para></listitem>

1628

<listitem><para>The output statements might not have any effect

1629

if overrides are applied elsewhere in the build system

1630

configuration.

1631

Use the <filename>-f</filename> option to add the

1632

<filename>forcevariable</filename> override to each output line

1633

if you need to work around this restriction.

1634

</para></listitem>

1635

<listitem><para>The script does apply special handling when

1636

building for multiple machines.

1637

However, the script does place a

1638

comment before each set of values that specifies

1639

which triplet to which they belong as shown above

1640

(e.g., <filename>i586-poky-linux</filename>).

</para></listitem>

</itemizedlist>

</note>

</para>

</section>

<title>Build History Image Information</title>

1649

1650

<para>

1651

The files produced for each image are as follows:

1652

1653

<listitem><para><filename>image-files:</filename>

1654

A directory containing selected files from the root

1655

filesystem.

1656

The files are defined by

1657

1658

</para></listitem>

1659

<listitem><para><filename>build-id.txt:</filename>

1660

Human-readable information about the build configuration

1661

and metadata source revisions.

1662

This file contains the full build header as printed

1663

by BitBake.</para></listitem>

1664

1665

Dependency graphs for the image that are

1666

compatible with <filename>graphviz</filename>.

1667

</para></listitem>

1668

<listitem><para><filename>files-in-image.txt:</filename>

1669

A list of files in the image with permissions,

1670

owner, group, size, and symlink information.

1671

</para></listitem>

1672

<listitem><para><filename>image-info.txt:</filename>

1673

A text file containing name-value pairs with information

1674

about the image.

1675

See the following listing example for more information.

1676

</para></listitem>

1677

<listitem><para><filename>installed-package-names.txt:</filename>

1678

A list of installed packages by name only.</para></listitem>

1679

<listitem><para><filename>installed-package-sizes.txt:</filename>

1680

A list of installed packages ordered by size.

1681

</para></listitem>

1682

<listitem><para><filename>installed-packages.txt:</filename>

1683

A list of installed packages with full package

1684

filenames.</para></listitem>

1685

</itemizedlist>

1686

<note>

1687

Installed package information is able to be gathered and

1688

produced even if package management is disabled for the final

image.

</note>

</para>

<para>

Here is an example of <filename>image-info.txt</filename>:

DISTRO = poky

DISTRO_VERSION = 1.7

USER_CLASSES = buildstats image-mklibs image-prelink

1699

IMAGE_CLASSES = image_types

1700

IMAGE_FEATURES = debug-tweaks

1701

IMAGE_LINGUAS =

1702

IMAGE_INSTALL = packagegroup-core-boot run-postinsts

1703

BAD_RECOMMENDATIONS =

1704

NO_RECOMMENDATIONS =

1705

PACKAGE_EXCLUDE =

1706

ROOTFS_POSTPROCESS_COMMAND = write_package_manifest; license_create_manifest; \

1707

write_image_manifest ; buildhistory_list_installed_image ; \

1708

buildhistory_get_image_installed ; ssh_allow_empty_password; \

1709

postinst_enable_logging; rootfs_update_timestamp ; ssh_disable_dns_lookup ;

1710

IMAGE_POSTPROCESS_COMMAND = buildhistory_get_imageinfo ;

1711

IMAGESIZE = 6900

1712

</literallayout>

1713

Other than <filename>IMAGESIZE</filename>, which is the

1714

total size of the files in the image in Kbytes, the

1715

name-value pairs are variables that may have influenced the

1716

content of the image.

1717

This information is often useful when you are trying to determine

1718

why a change in the package or file listings has occurred.

</para>

</section>

<title>Using Build History to Gather Image Information Only</title>

1724

1725

<para>

1726

As you can see, build history produces image information,

1727

including dependency graphs, so you can see why something

1728

was pulled into the image.

1729

If you are just interested in this information and not

1730

interested in collecting specific package or SDK information,

1731

you can enable writing only image information without

1732

any history by adding the following to your

1733

<filename>conf/local.conf</filename> file found in the

Brad Bishop

2018-02-25 22:55:05 -0500

[diff] [blame]

1734

Patrick Williams

2015-09-15 14:41:29 -0500

[diff] [blame]

1735

1736

INHERIT += "buildhistory"

1737

BUILDHISTORY_COMMIT = "0"

1738

BUILDHISTORY_FEATURES = "image"

</literallayout>

Here, you set the

variable to use the image feature only.

</para>

</section>

<title>Build History SDK Information</title>

Patrick Williams

2016-06-20 12:57:21 -0500

[diff] [blame]

1748

Patrick Williams

2015-09-15 14:41:29 -0500

[diff] [blame]

1749

<para>

1750

Build history collects similar information on the contents

Patrick Williams

2016-06-20 12:57:21 -0500

[diff] [blame]

1751

of SDKs

1752

(e.g. <filename>bitbake -c populate_sdk imagename</filename>)

Patrick Williams

2015-09-15 14:41:29 -0500

[diff] [blame]

1753

as compared to information it collects for images.

Patrick Williams

2016-06-20 12:57:21 -0500

[diff] [blame]

1754

Furthermore, this information differs depending on whether an

1755

extensible or standard SDK is being produced.

</para>

<para>

The following list shows the files produced for SDKs:

Patrick Williams

2015-09-15 14:41:29 -0500

[diff] [blame]

1760

1761

<listitem><para><filename>files-in-sdk.txt:</filename>

1762

A list of files in the SDK with permissions,

1763

owner, group, size, and symlink information.

1764

This list includes both the host and target parts

of the SDK.

</para></listitem>

A text file containing name-value pairs with information

1769

about the SDK.

1770

See the following listing example for more information.

1771

</para></listitem>

Patrick Williams

2016-06-20 12:57:21 -0500

[diff] [blame]

1772

<listitem><para><filename>sstate-task-sizes.txt:</filename>

1773

A text file containing name-value pairs with information

1774

about task group sizes

1775

(e.g. <filename>do_populate_sysroot</filename> tasks

1776

have a total size).

1777

The <filename>sstate-task-sizes.txt</filename> file

1778

exists only when an extensible SDK is created.

1779

</para></listitem>

1780

<listitem><para><filename>sstate-package-sizes.txt:</filename>

1781

A text file containing name-value pairs with information

1782

for the shared-state packages and sizes in the SDK.

1783

The <filename>sstate-package-sizes.txt</filename> file

1784

exists only when an extensible SDK is created.

1785

</para></listitem>

1786

<listitem><para><filename>sdk-files:</filename>

1787

A folder that contains copies of the files mentioned in

1788

<filename>BUILDHISTORY_SDK_FILES</filename> if the

1789

files are present in the output.

1790

Additionally, the default value of

1791

<filename>BUILDHISTORY_SDK_FILES</filename> is specific

1792

to the extensible SDK although you can set it

1793

differently if you would like to pull in specific files

1794

from the standard SDK.</para>

1795

<para>The default files are

1796

<filename>conf/local.conf</filename>,

1797

<filename>conf/bblayers.conf</filename>,

1798

<filename>conf/auto.conf</filename>,

1799

<filename>conf/locked-sigs.inc</filename>, and

1800

<filename>conf/devtool.conf</filename>.

1801

Thus, for an extensible SDK, these files get copied

1802

into the <filename>sdk-files</filename> directory.

1803

</para></listitem>

Patrick Williams

2015-09-15 14:41:29 -0500

[diff] [blame]

1804

<listitem><para>The following information appears under

1805

each of the <filename>host</filename>

1806

and <filename>target</filename> directories

1807

for the portions of the SDK that run on the host and

1808

on the target, respectively:

Patrick Williams

2016-06-20 12:57:21 -0500

[diff] [blame]

1809

<note>

1810

The following files for the most part are empty

1811

when producing an extensible SDK because this

1812

type of SDK is not constructed from packages as is

1813

the standard SDK.

1814

</note>

Patrick Williams

2015-09-15 14:41:29 -0500

[diff] [blame]

1815

1816

<listitem><para><filename>depends.dot:</filename>

1817

Dependency graph for the SDK that is

1818

compatible with <filename>graphviz</filename>.

1819

</para></listitem>

1820

<listitem><para><filename>installed-package-names.txt:</filename>

1821

A list of installed packages by name only.

1822

</para></listitem>

1823

<listitem><para><filename>installed-package-sizes.txt:</filename>

1824

A list of installed packages ordered by size.

1825

</para></listitem>

1826

<listitem><para><filename>installed-packages.txt:</filename>

1827

A list of installed packages with full package

1828

filenames.</para></listitem>

</itemizedlist>

</para></listitem>

</itemizedlist>

</para>

<para>

Here is an example of <filename>sdk-info.txt</filename>:

1836

1837

DISTRO = poky

1838

DISTRO_VERSION = 1.3+snapshot-20130327

1839

SDK_NAME = poky-glibc-i686-arm

1840

SDK_VERSION = 1.3+snapshot

1841

SDKMACHINE =

1842

SDKIMAGE_FEATURES = dev-pkgs dbg-pkgs

1843

BAD_RECOMMENDATIONS =

1844

SDKSIZE = 352712

1845

</literallayout>

1846

Other than <filename>SDKSIZE</filename>, which is the

1847

total size of the files in the SDK in Kbytes, the

1848

name-value pairs are variables that might have influenced the

1849

content of the SDK.

1850

This information is often useful when you are trying to

1851

determine why a change in the package or file listings

has occurred.

</para>

</section>

<title>Examining Build History Information</title>

1858

1859

<para>

1860

You can examine build history output from the command line or

1861

from a web interface.

</para>

<para>

To see any changes that have occurred (assuming you have

1866

1867

you can simply

1868

use any Git command that allows you to view the history of

a repository.

Here is one method:

$ git log -p

</literallayout>

You need to realize, however, that this method does show

1875

changes that are not significant (e.g. a package's size

1876

changing by a few bytes).

</para>

<para>

A command-line tool called <filename>buildhistory-diff</filename>

1881

does exist, though, that queries the Git repository and prints just

1882

the differences that might be significant in human-readable form.

1883

Here is an example:

1884

1885

$ ~/poky/poky/scripts/buildhistory-diff . HEAD^

1886

Changes to images/qemux86_64/glibc/core-image-minimal (files-in-image.txt):

1887

/etc/anotherpkg.conf was added

1888

/sbin/anotherpkg was added

1889

* (installed-package-names.txt):

1890

* anotherpkg was added

1891

Changes to images/qemux86_64/glibc/core-image-minimal (installed-package-names.txt):

1892

anotherpkg was added

1893

packages/qemux86_64-poky-linux/v86d: PACKAGES: added "v86d-extras"

1894

* PR changed from "r0" to "r1"

1895

* PV changed from "0.1.10" to "0.1.12"

1896

packages/qemux86_64-poky-linux/v86d/v86d: PKGSIZE changed from 110579 to 144381 (+30%)

1897

* PR changed from "r0" to "r1"

1898

* PV changed from "0.1.10" to "0.1.12"

1899

</literallayout>

Patrick Williams

2017-02-23 20:41:17 -0600

[diff] [blame]

1900

<note>

1901

The <filename>buildhistory-diff</filename> tool requires

1902

the <filename>GitPython</filename> package.

1903

Be sure to install it using Pip3 as follows:

1904

1905

$ pip3 install GitPython --user

1906

</literallayout>

1907

Alternatively, you can install

1908

<filename>python3-git</filename> using the appropriate

1909

distribution package manager (e.g.

1910

<filename>apt-get</filename>, <filename>dnf</filename>, or

1911

<filename>zipper</filename>).

1912

</note>

Patrick Williams

2015-09-15 14:41:29 -0500

[diff] [blame]

</para>

<para>

To see changes to the build history using a web interface, follow

1917

the instruction in the <filename>README</filename> file here.

1918

<ulink url='http://git.yoctoproject.org/cgit/cgit.cgi/buildhistory-web/'></ulink>.

</para>

<para>

Here is a sample screenshot of the interface:

</para>

</section>

</section>

</section>

<title>Speeding Up the Build</title>

1931

1932

<para>

1933

Build time can be an issue.

1934

By default, the build system uses simple controls to try and maximize

1935

build efficiency.

1936

In general, the default settings for all the following variables

1937

result in the most efficient build times when dealing with single

1938

socket systems (i.e. a single CPU).

1939

If you have multiple CPUs, you might try increasing the default

1940

values to gain more speed.

1941

See the descriptions in the glossary for each variable for more

information:

The maximum number of threads BitBake simultaneously executes.

1947

</para></listitem>

1948

1949

<ulink url='&YOCTO_DOCS_BB_URL;#var-BB_NUMBER_PARSE_THREADS'><filename>BB_NUMBER_PARSE_THREADS</filename>:</ulink>

1950

The number of threads BitBake uses during parsing.

</para></listitem>

Extra options passed to the <filename>make</filename> command

1955

during the

1956

1957

task in order to specify parallel compilation on the

local build host.

</para></listitem>

Extra options passed to the <filename>make</filename> command

1963

during the

1964

1965

task in order to specify parallel installation on the

local build host.

</para></listitem>

</itemizedlist>

As mentioned, these variables all scale to the number of processor

1970

cores available on the build system.

1971

For single socket systems, this auto-scaling ensures that the build

1972

system fundamentally takes advantage of potential parallel operations

1973

during the build based on the build machine's capabilities.

</para>

<para>

Following are additional factors that can affect build speed:

File system type:

The file system type that the build is being performed on can

1982

also influence performance.

1983

Using <filename>ext4</filename> is recommended as compared

1984

to <filename>ext2</filename> and <filename>ext3</filename>

1985

due to <filename>ext4</filename> improved features

such as extents.

</para></listitem>

Disabling the updating of access time using

1990

<filename>noatime</filename>:

1991

The <filename>noatime</filename> mount option prevents the

1992

build system from updating file and directory access times.

1993

</para></listitem>

1994

1995

Setting a longer commit:

1996

Using the "commit=" mount option increases the interval

1997

in seconds between disk cache writes.

1998

Changing this interval from the five second default to

1999

something longer increases the risk of data loss but decreases

2000

the need to write to the disk, thus increasing the build

performance.

</para></listitem>

Choosing the packaging backend:

2005

Of the available packaging backends, IPK is the fastest.

2006

Additionally, selecting a singular packaging backend also

helps.

</para></listitem>

Using <filename>tmpfs</filename> for

2011

2012

as a temporary file system:

2013

While this can help speed up the build, the benefits are

2014

limited due to the compiler using

2015

<filename>-pipe</filename>.

2016

The build system goes to some lengths to avoid

2017

<filename>sync()</filename> calls into the

2018

file system on the principle that if there was a significant

2019

failure, the

Brad Bishop

2018-02-25 22:55:05 -0500

[diff] [blame]

2020

Patrick Williams