Blame - import-layers/yocto-poky/documentation/ref-manual/technical-details.xml - mdmillerii/openbmc

2015-09-15 14:41:29 -0500

[diff] [blame]

section.

</note>

<para>

The rest of this section goes into detail about the overall incremental build

441

architecture, the checksums (signatures), shared state, and some tips and tricks.

</para>

<title>Overall Architecture</title>

446

447

<para>

448

When determining what parts of the system need to be built, BitBake

449

works on a per-task basis rather than a per-recipe basis.

450

You might wonder why using a per-task basis is preferred over a per-recipe basis.

451

To help explain, consider having the IPK packaging backend enabled and then switching to DEB.

In this case, the

and

task outputs are still valid.

457

However, with a per-recipe approach, the build would not include the

458

<filename>.deb</filename> files.

459

Consequently, you would have to invalidate the whole build and rerun it.

460

Rerunning everything is not the best solution.

461

Also, in this case, the core must be "taught" much about specific tasks.

462

This methodology does not scale well and does not allow users to easily add new tasks

463

in layers or as external recipes without touching the packaged-staging core.

</para>

</section>

<title>Checksums (Signatures)</title>

469

470

<para>

471

The shared state code uses a checksum, which is a unique signature of a task's

472

inputs, to determine if a task needs to be run again.

473

Because it is a change in a task's inputs that triggers a rerun, the process

474

needs to detect all the inputs to a given task.

475

For shell tasks, this turns out to be fairly easy because

476

the build process generates a "run" shell script for each task and

477

it is possible to create a checksum that gives you a good idea of when

478

the task's data changes.

479

</para>

480

481

<para>

Patrick Williams

2016-06-20 12:57:21 -0500

[diff] [blame]

482

To complicate the problem, there are things that should not be

483

included in the checksum.

Patrick Williams

2015-09-15 14:41:29 -0500

[diff] [blame]

484

First, there is the actual specific build path of a given task -

485

the <link linkend='var-WORKDIR'><filename>WORKDIR</filename></link>.

Patrick Williams

2016-06-20 12:57:21 -0500

[diff] [blame]

486

It does not matter if the work directory changes because it should

487

not affect the output for target packages.

488

Also, the build process has the objective of making native

489

or cross packages relocatable.

490

<note>

491

Both native and cross packages run on the build host.

492

However, cross packages generate output for the target

493

architecture.

494

</note>

495

The checksum therefore needs to exclude

496

<filename>WORKDIR</filename>.

Patrick Williams

2015-09-15 14:41:29 -0500

[diff] [blame]

497

The simplistic approach for excluding the work directory is to set

Patrick Williams

2016-06-20 12:57:21 -0500

[diff] [blame]

498

<filename>WORKDIR</filename> to some fixed value and create the

499

checksum for the "run" script.

Patrick Williams

2015-09-15 14:41:29 -0500

[diff] [blame]

</para>

<para>

Another problem results from the "run" scripts containing functions that

504

might or might not get called.

505

The incremental build solution contains code that figures out dependencies

506

between shell functions.

507

This code is used to prune the "run" scripts down to the minimum set,

508

thereby alleviating this problem and making the "run" scripts much more

readable as a bonus.

</para>

<para>

So far we have solutions for shell scripts.

514

What about Python tasks?

515

The same approach applies even though these tasks are more difficult.

516

The process needs to figure out what variables a Python function accesses

517

and what functions it calls.

518

Again, the incremental build solution contains code that first figures out

519

the variable and function dependencies, and then creates a checksum for the data

520

used as the input to the task.

</para>

<para>

Like the <filename>WORKDIR</filename> case, situations exist where dependencies

525

should be ignored.

526

For these cases, you can instruct the build process to ignore a dependency

527

by using a line like the following:

528

529

PACKAGE_ARCHS[vardepsexclude] = "MACHINE"

530

</literallayout>

531

This example ensures that the

532

533

variable does not

534

depend on the value of

535

536

even if it does reference it.

</para>

<para>

Equally, there are cases where we need to add dependencies BitBake is not able to find.

541

You can accomplish this by using a line like the following:

542

543

PACKAGE_ARCHS[vardeps] = "MACHINE"

544

</literallayout>

545

This example explicitly adds the <filename>MACHINE</filename> variable as a

546

dependency for <filename>PACKAGE_ARCHS</filename>.

</para>

<para>

Consider a case with in-line Python, for example, where BitBake is not

551

able to figure out dependencies.

552

When running in debug mode (i.e. using <filename>-DDD</filename>), BitBake

553

produces output when it discovers something for which it cannot figure out

554

dependencies.

555

The Yocto Project team has currently not managed to cover those dependencies

556

in detail and is aware of the need to fix this situation.

</para>

<para>

Thus far, this section has limited discussion to the direct inputs into a task.

561

Information based on direct inputs is referred to as the "basehash" in the

562

code.

563

However, there is still the question of a task's indirect inputs - the

564

things that were already built and present in the

565

<ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink>.

566

The checksum (or signature) for a particular task needs to add the hashes

567

of all the tasks on which the particular task depends.

568

Choosing which dependencies to add is a policy decision.

569

However, the effect is to generate a master checksum that combines the basehash

570

and the hashes of the task's dependencies.

</para>

<para>

At the code level, there are a variety of ways both the basehash and the

575

dependent task hashes can be influenced.

576

Within the BitBake configuration file, we can give BitBake some extra information

577

to help it construct the basehash.

578

The following statement effectively results in a list of global variable

579

dependency excludes - variables never included in any checksum:

580

581

BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \

582

SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \

583

USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \

584

PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \

585

CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX"

586

</literallayout>

587

The previous example excludes

588

589

since that variable is actually constructed as a path within

the whitelist.

</para>

<para>

The rules for deciding which hashes of dependent tasks to include through

596

dependency chains are more complex and are generally accomplished with a

597

Python function.

598

The code in <filename>meta/lib/oe/sstatesig.py</filename> shows two examples

599

of this and also illustrates how you can insert your own policy into the system

600

if so desired.

601

This file defines the two basic signature generators <filename>OE-Core</filename>

602

uses: "OEBasic" and "OEBasicHash".

603

By default, there is a dummy "noop" signature handler enabled in BitBake.

604

This means that behavior is unchanged from previous versions.

605

<filename>OE-Core</filename> uses the "OEBasicHash" signature handler by default

606

through this setting in the <filename>bitbake.conf</filename> file:

607

608

BB_SIGNATURE_HANDLER ?= "OEBasicHash"

609

</literallayout>

610

The "OEBasicHash" <filename>BB_SIGNATURE_HANDLER</filename> is the same as the

611

"OEBasic" version but adds the task hash to the stamp files.

612

This results in any

613

<ulink url='&YOCTO_DOCS_DEV_URL;#metadata'>Metadata</ulink>

614

change that changes the task hash, automatically

615

causing the task to be run again.

616

This removes the need to bump <link linkend='var-PR'><filename>PR</filename></link>

617

values, and changes to Metadata automatically ripple across the build.

</para>

<para>

It is also worth noting that the end result of these signature generators is to

622

make some dependency and hash information available to the build.

623

This information includes:

624

625

<listitem><para><filename>BB_BASEHASH_task-</filename><replaceable>taskname</replaceable>:

626

The base hashes for each task in the recipe.

627

</para></listitem>

628

<listitem><para><filename>BB_BASEHASH_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>:

629

The base hashes for each dependent task.

630

</para></listitem>

631

<listitem><para><filename>BBHASHDEPS_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>:

632

The task dependencies for each task.

633

</para></listitem>

634

<listitem><para><filename>BB_TASKHASH</filename>:

635

The hash of the currently running task.

</para></listitem>

</itemizedlist>

</para>

</section>

<title>Shared State</title>

643

644

<para>

645

Checksums and dependencies, as discussed in the previous section, solve half the

646

problem of supporting a shared state.

647

The other part of the problem is being able to use checksum information during the build

648

and being able to reuse or rebuild specific components.

</para>

<para>

The

class is a relatively generic implementation of how to "capture"

655

a snapshot of a given task.

656

The idea is that the build process does not care about the source of a task's output.

657

Output could be freshly built or it could be downloaded and unpacked from

658

somewhere - the build process does not need to worry about its origin.

</para>

<para>

There are two types of output, one is just about creating a directory

663

in <link linkend='var-WORKDIR'><filename>WORKDIR</filename></link>.

664

A good example is the output of either

or

The other type of output occurs when a set of data is merged into a shared directory

669

tree such as the sysroot.

</para>

<para>

The Yocto Project team has tried to keep the details of the

674

implementation hidden in <filename>sstate</filename> class.

675

From a user's perspective, adding shared state wrapping to a task

676

is as simple as this

677

678

example taken from the

class:

DEPLOYDIR = "${WORKDIR}/deploy-${PN}"

683

SSTATETASKS += "do_deploy"

Patrick Williams

2015-09-15 14:41:29 -0500

[diff] [blame]

684

do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"

685

do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"

686

687

python do_deploy_setscene () {

688

sstate_setscene(d)

689

}

690

addtask do_deploy_setscene

691

do_deploy[dirs] = "${DEPLOYDIR} ${B}"

692

</literallayout>

Patrick Williams

2017-02-23 20:41:17 -0600

[diff] [blame]

693

The following list explains the previous example:

694

695

696

Adding "do_deploy" to <filename>SSTATETASKS</filename>

697

adds some required sstate-related processing, which is

698

implemented in the

699

700

class, to before and after the

task.

</para></listitem>

The

<filename>do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"</filename>

707

declares that <filename>do_deploy</filename> places its

708

output in <filename>${DEPLOYDIR}</filename> when run

709

normally (i.e. when not using the sstate cache).

710

This output becomes the input to the shared state cache.

</para></listitem>

The

<filename>do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"</filename>

715

line causes the contents of the shared state cache to be

716

copied to <filename>${DEPLOY_DIR_IMAGE}</filename>.

717

<note>

718

If <filename>do_deploy</filename> is not already in

719

the shared state cache or if its input checksum

720

(signature) has changed from when the output was

721

cached, the task will be run to populate the shared

722

state cache, after which the contents of the shared

723

state cache is copied to

724

<filename>${DEPLOY_DIR_IMAGE}</filename>.

725

If <filename>do_deploy</filename> is in the shared

726

state cache and its signature indicates that the

727

cached output is still valid (i.e. if no

728

relevant task inputs have changed), then the contents

729

of the shared state cache will be copied directly to

730

<filename>${DEPLOY_DIR_IMAGE}</filename> by the

731

<filename>do_deploy_setscene</filename> task instead,

732

skipping the <filename>do_deploy</filename> task.

</note>

</para></listitem>

The following task definition is glue logic needed to make

737

the previous settings effective:

738

739

python do_deploy_setscene () {

740

sstate_setscene(d)

741

}

742

addtask do_deploy_setscene

743

</literallayout>

744

<filename>sstate_setscene()</filename> takes the flags

745

above as input and accelerates the

746

<filename>do_deploy</filename> task through the

747

shared state cache if possible.

748

If the task was accelerated,

749

<filename>sstate_setscene()</filename> returns True.

750

Otherwise, it returns False, and the normal

751

<filename>do_deploy</filename> task runs.

752

For more information, see the

753

"<ulink url='&YOCTO_DOCS_BB_URL;#setscene'>setscene</ulink>"

754

section in the BitBake User Manual.

755

</para></listitem>

756

757

The <filename>do_deploy[dirs] = "${DEPLOYDIR} ${B}"</filename>

758

line creates <filename>${DEPLOYDIR}</filename> and

759

<filename>${B}</filename> before the

760

<filename>do_deploy</filename> task runs, and also sets

761

the current working directory of

762

<filename>do_deploy</filename> to

763

<filename>${B}</filename>.

764

For more information, see the

765

"<ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'>Variable Flags</ulink>"

766

section in the BitBake User Manual.

767

<note>

768

In cases where

769

<filename>sstate-inputdirs</filename> and

770

<filename>sstate-outputdirs</filename> would be the

771

same, you can use

772

<filename>sstate-plaindirs</filename>.

773

For example, to preserve the

774

<filename>${PKGD}</filename> and

775

<filename>${PKGDEST}</filename> output from the

776

777

task, use the following:

778

779

do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}"

</literallayout>

</note>

</para></listitem>

<filename>sstate-inputdirs</filename> and

785

<filename>sstate-outputdirs</filename> can also be used

786

with multiple directories.

787

For example, the following declares

788

<filename>PKGDESTWORK</filename> and

789

<filename>SHLIBWORK</filename> as shared state

790

input directories, which populates the shared state

791

cache, and <filename>PKGDATA_DIR</filename> and

792

<filename>SHLIBSDIR</filename> as the corresponding

793

shared state output directories:

794

795

do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}"

796

do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}"

</literallayout>

</para></listitem>

These methods also include the ability to take a lockfile

801

when manipulating shared state directory structures,

802

for cases where file additions or removals are sensitive:

803

804

do_package[sstate-lockfile] = "${PACKAGELOCK}"

</literallayout>

</para></listitem>

</itemizedlist>

</para>

<!--

<para>

Patrick Williams

2015-09-15 14:41:29 -0500

[diff] [blame]

812

In this example, we add some extra flags to the task, a name field ("deploy"), an

813

input directory where the task sends data, and the output

814

directory where the data from the task should eventually be copied.

815

We also add a <filename>_setscene</filename> variant of the task and add the task

816

name to the <filename>SSTATETASKS</filename> list.

</para>

<para>

If you have a directory whose contents you need to preserve, you can do this with

821

a line like the following:

822

823

do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}"

824

</literallayout>

825

This method, as well as the following example, also works for multiple directories.

826

827

do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}"

828

do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}"

829

do_package[sstate-lockfile] = "${PACKAGELOCK}"

830

</literallayout>

831

These methods also include the ability to take a lockfile when manipulating

832

shared state directory structures since some cases are sensitive to file

833

additions or removals.

834

</para>

Patrick Williams

2017-02-23 20:41:17 -0600

[diff] [blame]

835

-->

Patrick Williams

2015-09-15 14:41:29 -0500

[diff] [blame]

836

837

<para>

838

Behind the scenes, the shared state code works by looking in

839

840

841

for shared state files.

842

Here is an example:

843

844

SSTATE_MIRRORS ?= "\

Patrick Williams

2017-02-23 20:41:17 -0600

[diff] [blame]

845

file://.* http://someserver.tld/share/sstate/PATH;downloadfilename=PATH \n \

Patrick Williams

2015-09-15 14:41:29 -0500

[diff] [blame]

846

file://.* file:///some/local/dir/sstate/PATH"

847

</literallayout>

848

<note>

849

The shared state directory (<filename>SSTATE_DIR</filename>) is

850

organized into two-character subdirectories, where the subdirectory

851

names are based on the first two characters of the hash.

852

If the shared state directory structure for a mirror has the

853

same structure as <filename>SSTATE_DIR</filename>, you must

854

specify "PATH" as part of the URI to enable the build system

855

to map to the appropriate subdirectory.

</note>

</para>

<para>

The shared state package validity can be detected just by looking at the

861

filename since the filename contains the task checksum (or signature) as

862

described earlier in this section.

863

If a valid shared state package is found, the build process downloads it

864

and uses it to accelerate the task.

</para>

<para>

The build processes use the <filename>*_setscene</filename> tasks

869

for the task acceleration phase.

870

BitBake goes through this phase before the main execution code and tries

871

to accelerate any tasks for which it can find shared state packages.

872

If a shared state package for a task is available, the shared state

873

package is used.

874

This means the task and any tasks on which it is dependent are not

executed.

</para>

<para>

As a real world example, the aim is when building an IPK-based image,

880

only the

881

882

tasks would have their

883

shared state packages fetched and extracted.

884

Since the sysroot is not used, it would never get extracted.

885

This is another reason why a task-based approach is preferred over a

886

recipe-based approach, which would have to install the output from every task.

</para>

</section>

<title>Tips and Tricks</title>

892

893

<para>

894

The code in the build system that supports incremental builds is not

895

simple code.

896

This section presents some tips and tricks that help you work around

897

issues related to shared state code.

</para>

<title>Debugging</title>

902

903

<para>

Patrick Williams

2017-02-23 20:41:17 -0600

[diff] [blame]

904

Seeing what metadata went into creating the input signature

905

of a shared state (sstate) task can be a useful debugging aid.

906

This information is available in signature information

907

(<filename>siginfo</filename>) files in

908

909

For information on how to view and interpret information in

910

<filename>siginfo</filename> files, see the

911

"<link linkend='usingpoky-viewing-task-variable-dependencies'>Viewing Task Variable Dependencies</link>"

912

section.

Patrick Williams

2015-09-15 14:41:29 -0500

[diff] [blame]

</para>

</section>

<title>Invalidating Shared State</title>

918

919

<para>

920

The OpenEmbedded build system uses checksums and shared state

921

cache to avoid unnecessarily rebuilding tasks.

922

Collectively, this scheme is known as "shared state code."

</para>

<para>

As with all schemes, this one has some drawbacks.

927

It is possible that you could make implicit changes to your

928

code that the checksum calculations do not take into

929

account.

930

These implicit changes affect a task's output but do not trigger

931

the shared state code into rebuilding a recipe.

932

Consider an example during which a tool changes its output.

933

Assume that the output of <filename>rpmdeps</filename> changes.

934

The result of the change should be that all the

935

<filename>package</filename> and

936

<filename>package_write_rpm</filename> shared state cache

937

items become invalid.

938

However, because the change to the output is

939

external to the code and therefore implicit,

940

the associated shared state cache items do not become

941

invalidated.

942

In this case, the build process uses the cached items rather

943

than running the task again.

944

Obviously, these types of implicit changes can cause problems.

</para>

<para>

To avoid these problems during the build, you need to

949

understand the effects of any changes you make.

950

Realize that changes you make directly to a function

951

are automatically factored into the checksum calculation.

952

Thus, these explicit changes invalidate the associated area of

953

shared state cache.

954

However, you need to be aware of any implicit changes that

955

are not obvious changes to the code and could affect the output

of a given task.

</para>

<para>

When you identify an implicit change, you can easily take steps

961

to invalidate the cache and force the tasks to run.

962

The steps you can take are as simple as changing a function's

963

comments in the source code.

964

For example, to invalidate package shared state files, change

965

the comment statements of

966

967

or the comments of one of the functions it calls.

968

Even though the change is purely cosmetic, it causes the

969

checksum to be recalculated and forces the OpenEmbedded build

970

system to run the task again.

</para>

<note>

For an example of a commit that makes a cosmetic change to

975

invalidate shared state, see this

976

<ulink url='&YOCTO_GIT_URL;/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54'>commit</ulink>.

</note>

</section>

</section>

</section>

Patrick Williams

2017-02-23 20:41:17 -0600

[diff] [blame]

982

983

<title>Automatically Added Runtime Dependencies</title>

984

985

<para>

986

The OpenEmbedded build system automatically adds common types of

987

runtime dependencies between packages, which means that you do not

988

need to explicitly declare the packages using

989

990

Three automatic mechanisms exist (<filename>shlibdeps</filename>,

991

<filename>pcdeps</filename>, and <filename>depchains</filename>) that

992

handle shared libraries, package configuration (pkg-config) modules,

993

and <filename>-dev</filename> and <filename>-dbg</filename> packages,

994

respectively.

995

For other types of runtime dependencies, you must manually declare

the dependencies.

<filename>shlibdeps</filename>:

1000

During the

1001

1002

task of each recipe, all shared libraries installed by the

1003

recipe are located.

1004

For each shared library, the package that contains the shared

1005

library is registered as providing the shared library.

1006

More specifically, the package is registered as providing the

1007

<ulink url='https://en.wikipedia.org/wiki/Soname'>soname</ulink>

1008

of the library.

1009

The resulting shared-library-to-package mapping

is saved globally in

by the

task.</para>

<para>Simultaneously, all executables and shared libraries

1017

installed by the recipe are inspected to see what shared

1018

libraries they link against.

1019

For each shared library dependency that is found,

1020

<filename>PKGDATA_DIR</filename> is queried to

1021

see if some package (likely from a different recipe) contains

1022

the shared library.

1023

If such a package is found, a runtime dependency is added from

1024

the package that depends on the shared library to the package

1025

that contains the library.</para>

1026

1027

<para>The automatically added runtime dependency also includes

1028

a version restriction.

1029

This version restriction specifies that at least the current

1030

version of the package that provides the shared library must be

1031

used, as if

1032

"<replaceable>package</replaceable> (>= <replaceable>version</replaceable>)"

1033

had been added to

1034

1035

This forces an upgrade of the package containing the shared

1036

library when installing the package that depends on the

1037

library, if needed.</para>

1038

1039

<para>If you want to avoid a package being registered as

1040

providing a particular shared library (e.g. because the library

1041

is for internal use only), then add the library to

1042

1043

inside the package's recipe.

1044

</para></listitem>

1045

1046

<filename>pcdeps</filename>:

1047

During the

1048

1049

task of each recipe, all pkg-config modules

1050

(<filename>*.pc</filename> files) installed by the recipe are

1051

located.

1052

For each module, the package that contains the module is

1053

registered as providing the module.

1054

The resulting module-to-package mapping is saved globally in

by the

task.</para>

<para>Simultaneously, all pkg-config modules installed by the

1061

recipe are inspected to see what other pkg-config modules they

1062

depend on.

1063

A module is seen as depending on another module if it contains

1064

a "Requires:" line that specifies the other module.

1065

For each module dependency,

1066

<filename>PKGDATA_DIR</filename> is queried to see if some

1067

package contains the module.

1068

If such a package is found, a runtime dependency is added from

1069

the package that depends on the module to the package that

1070

contains the module.

1071

<note>

1072

The <filename>pcdeps</filename> mechanism most often infers

1073

dependencies between <filename>-dev</filename> packages.

</note>

</para></listitem>

<filename>depchains</filename>:

1078

If a package <filename>foo</filename> depends on a package

1079

1080

and <filename>foo-dbg</filename> are also made to depend on

1081

<filename>bar-dev</filename> and <filename>bar-dbg</filename>,

1082

respectively.

1083

Taking the <filename>-dev</filename> packages as an example,

1084

the <filename>bar-dev</filename> package might provide

1085

headers and shared library symlinks needed by

1086

<filename>foo-dev</filename>, which shows the need

1087

for a dependency between the packages.</para>

1088

1089

<para>The dependencies added by <filename>depchains</filename>

are in the form of

<note>

By default, <filename>foo-dev</filename> also has an

1094

<filename>RDEPENDS</filename>-style dependency on

1095

<filename>foo</filename>, because the default value of

1096

<filename>RDEPENDS_${PN}-dev</filename> (set in

1097

<filename>bitbake.conf</filename>) includes

"${PN}".

</note></para>

<para>To ensure that the dependency chain is never broken,

1102

1103

packages are always generated by default, even if the packages

1104

turn out to be empty.

1105

See the

1106

1107

variable for more information.

</para></listitem>

</itemizedlist>

</para>

<para>

The <filename>do_package</filename> task depends on the

1114

1115

task of each recipe in

1116

1117

through use of a

1118

<filename>[</filename><ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'><filename>deptask</filename></ulink><filename>]</filename>

1119

declaration, which guarantees that the required

1120

shared-library/module-to-package mapping information will be available

1121

when needed as long as <filename>DEPENDS</filename> has been

correctly set.

</para>

</section>

<title>Fakeroot and Pseudo</title>

1128

1129

<para>

1130

Some tasks are easier to implement when allowed to perform certain

1131

operations that are normally reserved for the root user.

1132

For example, the

1133

1134

task benefits from being able to set the UID and GID of installed files

to arbitrary values.

</para>

<para>

One approach to allowing tasks to perform root-only operations

1140

would be to require BitBake to run as root.

1141

However, this method is cumbersome and has security issues.

1142

The approach that is actually used is to run tasks that benefit from

1143

root privileges in a "fake" root environment.

1144

Within this environment, the task and its child processes believe that

1145

they are running as the root user, and see an internally consistent

1146

view of the filesystem.

1147

As long as generating the final output (e.g. a package or an image)

1148

does not require root privileges, the fact that some earlier steps ran

1149

in a fake root environment does not cause problems.

</para>

<para>

The capability to run tasks in a fake root environment is known as

1154

"fakeroot", which is derived from the BitBake keyword/variable

1155

flag that requests a fake root environment for a task.

1156

In current versions of the OpenEmbedded build system,

1157

the program that implements fakeroot is known as Pseudo.

</para>

<para>

Pseudo overrides system calls through the

1162

<filename>LD_PRELOAD</filename> mechanism to give the

1163

illusion of running as root.

1164

To keep track of "fake" file ownership and permissions resulting from

1165

operations that require root permissions, an sqlite3

1166

database is used.

1167

This database is stored in

1168

<filename>${</filename><link linkend='var-WORKDIR'><filename>WORKDIR</filename></link><filename>}/pseudo/files.db</filename>

1169

for individual recipes.

1170

Storing the database in a file as opposed to in memory

1171

gives persistence between tasks, and even between builds.

1172

<note><title>Caution</title>

1173

If you add your own task that manipulates the same files or

1174

directories as a fakeroot task, then that task should also run

1175

under fakeroot.

1176

Otherwise, the task will not be able to run root-only operations,

1177

and will not see the fake file ownership and permissions set by the

1178

other task.

1179

You should also add a dependency on

1180

<filename>virtual/fakeroot-native:do_populate_sysroot</filename>,

1181

giving the following:

1182

1183

fakeroot do_mytask () {

1184

...

1185

}

1186

do_mytask[depends] += "virtual/fakeroot-native:do_populate_sysroot"

1187

</literallayout>

1188

</note>

1189

For more information, see the

1190

<ulink url='&YOCTO_DOCS_BB_URL;#var-FAKEROOT'><filename>FAKEROOT*</filename></ulink>

1191

variables in the BitBake User Manual.

1192

You can also reference this

1193

<ulink url='http://www.ibm.com/developerworks/opensource/library/os-aapseudo1/index.html'>Pseudo</ulink>

article.

</para>

</section>

Patrick Williams

2015-09-15 14:41:29 -0500

[diff] [blame]

<para>

x32 is a processor-specific Application Binary Interface (psABI) for x86_64.

1203

An ABI defines the calling conventions between functions in a processing environment.

1204

The interface determines what registers are used and what the sizes are for various C data types.

</para>

<para>

Some processing environments prefer using 32-bit applications even when running

1209

on Intel 64-bit platforms.

1210

Consider the i386 psABI, which is a very old 32-bit ABI for Intel 64-bit platforms.

1211

The i386 psABI does not provide efficient use and access of the Intel 64-bit processor resources,

1212

leaving the system underutilized.

1213

Now consider the x86_64 psABI.

1214

This ABI is newer and uses 64-bits for data sizes and program pointers.

1215

The extra bits increase the footprint size of the programs, libraries,

1216

and also increases the memory and file system size requirements.

1217

Executing under the x32 psABI enables user programs to utilize CPU and system resources

1218

more efficiently while keeping the memory footprint of the applications low.

1219

Extra bits are used for registers but not for addressing mechanisms.

</para>

<title>Support</title>

1224

1225

<para>

1226

This Yocto Project release supports the final specifications of x32

1227

psABI.

1228

Support for x32 psABI exists as follows:

1229

1230

<listitem><para>You can create packages and images in x32 psABI format on x86_64 architecture targets.

1231

</para></listitem>

1232

<listitem><para>You can successfully build many recipes with the x32 toolchain.</para></listitem>

1233

<listitem><para>You can create and boot <filename>core-image-minimal</filename> and

1234

<filename>core-image-sato</filename> images.</para></listitem>

</itemizedlist>

</para>

</section>

<title>Completing x32</title>

1241

1242

<para>

1243

Future Plans for the x32 psABI in the Yocto Project include the following:

1244

1245

<listitem><para>Enhance and fix the few remaining recipes so they

1246

work with and support x32 toolchains.</para></listitem>

1247

<listitem><para>Enhance RPM Package Manager (RPM) support for x32 binaries.</para></listitem>

1248

<listitem><para>Support larger images.</para></listitem>

</itemizedlist>

</para>

</section>

<title>Using x32 Right Now</title>

1255

1256

<para>

1257

Follow these steps to use the x32 spABI:

1258

1259

<listitem><para>Enable the x32 psABI tuning file for <filename>x86_64</filename>

1260

machines by editing the <filename>conf/local.conf</filename> like this:

1261

1262

MACHINE = "qemux86-64"

1263

DEFAULTTUNE = "x86-64-x32"

1264

baselib = "${@d.getVar('BASE_LIB_tune-' + (d.getVar('DEFAULTTUNE', True) \

1265

or 'INVALID'), True) or 'lib'}"

1266

#MACHINE = "genericx86"

1267

#DEFAULTTUNE = "core2-64-x32"

1268

</literallayout></para></listitem>

1269

<listitem><para>As usual, use BitBake to build an image that supports the x32 psABI.

1270

Here is an example:

1271

1272

$ bitbake core-image-sato

1273

</literallayout></para></listitem>

1274

<listitem><para>As usual, run your image using QEMU:

1275

1276

$ runqemu qemux86-64 core-image-sato

1277

</literallayout></para></listitem>

</itemizedlist>

</para>

</section>

</section>

<title>Wayland</title>

1285

1286

<para>

1287

<ulink url='http://en.wikipedia.org/wiki/Wayland_(display_server_protocol)'>Wayland</ulink>

1288

is a computer display server protocol that

1289

provides a method for compositing window managers to communicate

1290

directly with applications and video hardware and expects them to

1291

communicate with input hardware using other libraries.

1292

Using Wayland with supporting targets can result in better control

1293

over graphics frame rendering than an application might otherwise

achieve.

</para>

<para>

The Yocto Project provides the Wayland protocol libraries and the

1299

reference

1300

<ulink url='http://en.wikipedia.org/wiki/Wayland_(display_server_protocol)#Weston'>Weston</ulink>

1301

compositor as part of its release.

1302

This section describes what you need to do to implement Wayland and

1303

use the compositor when building an image for a supporting target.

</para>

<title>Support</title>

1308

1309

<para>

1310

The Wayland protocol libraries and the reference Weston compositor

1311

ship as integrated packages in the <filename>meta</filename> layer

1312

of the

1313

<ulink url='&YOCTO_DOCS_DEV_URL;#source-directory'>Source Directory</ulink>.

1314

Specifically, you can find the recipes that build both Wayland

1315

and Weston at <filename>meta/recipes-graphics/wayland</filename>.

</para>

<para>

You can build both the Wayland and Weston packages for use only

1320

with targets that accept the

1321

<ulink url='http://dri.freedesktop.org/wiki/'>Mesa 3D and Direct Rendering Infrastructure</ulink>,

1322

which is also known as Mesa DRI.

1323

This implies that you cannot build and use the packages if your

1324

target uses, for example, the

1325

<trademark class='registered'>Intel</trademark> Embedded Media and

1326

Graphics Driver (<trademark class='registered'>Intel</trademark>

1327

EMGD) that overrides Mesa DRI.

</para>

<note>

Due to lack of EGL support, Weston 1.0.3 will not run directly on

1332

the emulated QEMU hardware.

1333

However, this version of Weston will run under X emulation without

issues.

</note>

</section>

<title>Enabling Wayland in an Image</title>

1340

1341

<para>

1342

To enable Wayland, you need to enable it to be built and enable

1343

it to be included in the image.

</para>

<title>Building</title>

1348

1349

<para>

1350

To cause Mesa to build the <filename>wayland-egl</filename>

1351

platform and Weston to build Wayland with Kernel Mode

1352

Setting

1353

(<ulink url='https://wiki.archlinux.org/index.php/Kernel_Mode_Setting'>KMS</ulink>)

1354

support, include the "wayland" flag in the

1355

1356

statement in your <filename>local.conf</filename> file:

1357

1358

DISTRO_FEATURES_append = " wayland"

</literallayout>

</para>

<note>

If X11 has been enabled elsewhere, Weston will build Wayland

with X11 support

</note>

</section>

<title>Installing</title>

1370

1371

<para>

1372

To install the Wayland feature into an image, you must

1373

include the following

1374

1375

statement in your <filename>local.conf</filename> file:

1376

1377

CORE_IMAGE_EXTRA_INSTALL += "wayland weston"

</literallayout>

</para>

</section>

</section>

<title>Running Weston</title>

1385

1386

<para>

1387

To run Weston inside X11, enabling it as described earlier and

1388

building a Sato image is sufficient.

1389

If you are running your image under Sato, a Weston Launcher appears

1390

in the "Utility" category.

</para>

<para>

Alternatively, you can run Weston through the command-line

1395

interpretor (CLI), which is better suited for development work.

1396

To run Weston under the CLI, you need to do the following after

1397

your image is built:

1398

1399

<listitem><para>Run these commands to export

1400

<filename>XDG_RUNTIME_DIR</filename>:

1401

1402

mkdir -p /tmp/$USER-weston

1403

chmod 0700 /tmp/$USER-weston

1404

export XDG_RUNTIME_DIR=/tmp/$USER-weston

1405

</literallayout></para></listitem>

1406

<listitem><para>Launch Weston in the shell:

1407

1408

weston

1409

</literallayout></para></listitem>

</orderedlist>

</para>

</section>

</section>

<title>Licenses</title>

1417

1418

<para>

1419

This section describes the mechanism by which the OpenEmbedded build system

1420

tracks changes to licensing text.

1421

The section also describes how to enable commercially licensed recipes,

1422

which by default are disabled.

</para>

<para>

For information that can help you maintain compliance with various open

1427

source licensing during the lifecycle of the product, see the

1428

"<ulink url='&YOCTO_DOCS_DEV_URL;#maintaining-open-source-license-compliance-during-your-products-lifecycle'>Maintaining Open Source License Compliance During Your Project's Lifecycle</ulink>" section

1429

in the Yocto Project Development Manual.

</para>

<title>Tracking License Changes</title>

1434

1435

<para>

1436

The license of an upstream project might change in the future.

1437

In order to prevent these changes going unnoticed, the

1438

<filename><link linkend='var-LIC_FILES_CHKSUM'>LIC_FILES_CHKSUM</link></filename>

1439

variable tracks changes to the license text. The checksums are validated at the end of the

1440

configure step, and if the checksums do not match, the build will fail.

</para>

<title>Specifying the <filename>LIC_FILES_CHKSUM</filename> Variable</title>

1445

1446

<para>

1447

The <filename>LIC_FILES_CHKSUM</filename>

Brad Bishop

6e60e8b

2018-02-01 10:27:11 -0500

[diff] [blame]

1448

variable contains checksums of the license text in the source

1449

code for the recipe.

1450

Following is an example of how to specify

1451

<filename>LIC_FILES_CHKSUM</filename>:

Patrick Williams

2015-09-15 14:41:29 -0500

[diff] [blame]

1452

1453

LIC_FILES_CHKSUM = "file://COPYING;md5=xxxx \

1454

file://licfile1.txt;beginline=5;endline=29;md5=yyyy \

1455

file://licfile2.txt;endline=50;md5=zzzz \

1456

..."

1457

</literallayout>

Brad Bishop

6e60e8b

2018-02-01 10:27:11 -0500

[diff] [blame]

1458

<note><title>Notes</title>

1459

1460

1461

When using "beginline" and "endline", realize that

1462

line numbering begins with one and not zero.

1463

Also, the included lines are inclusive (i.e. lines

1464

five through and including 29 in the previous

1465

example for <filename>licfile1.txt</filename>).

1466

</para></listitem>

1467

1468

When a license check fails, the selected license

1469

text is included as part of the QA message.

1470

Using this output, you can determine the exact

1471

start and finish for the needed license text.

1472

</para></listitem>

1473

</itemizedlist>

1474

</note>

Patrick Williams

2015-09-15 14:41:29 -0500

[diff] [blame]

</para>

<para>

The build system uses the

1479

<filename><link linkend='var-S'>S</link></filename> variable as

1480

the default directory when searching files listed in

1481

<filename>LIC_FILES_CHKSUM</filename>.

1482

The previous example employs the default directory.

</para>

<para>

Consider this next example:

1487

1488

LIC_FILES_CHKSUM = "file://src/ls.c;beginline=5;endline=16;\

1489

md5=bb14ed3c4cda583abc85401304b5cd4e"

1490

LIC_FILES_CHKSUM = "file://${WORKDIR}/license.html;md5=5c94767cedb5d6987c902ac850ded2c6"

</literallayout>

</para>

<para>

The first line locates a file in

Brad Bishop

6e60e8b

2018-02-01 10:27:11 -0500

[diff] [blame]

1496

<filename>${S}/src/ls.c</filename> and isolates lines five

1497

through 16 as license text.

Patrick Williams

2015-09-15 14:41:29 -0500

[diff] [blame]

1498

The second line refers to a file in

1499

<filename><link linkend='var-WORKDIR'>WORKDIR</link></filename>.

1500

</para>

1501

<para>

1502

Note that <filename>LIC_FILES_CHKSUM</filename> variable is

1503

mandatory for all recipes, unless the

1504

<filename>LICENSE</filename> variable is set to "CLOSED".

</para>

</section>

<title>Explanation of Syntax</title>

1510

<para>

1511

As mentioned in the previous section, the

1512

<filename>LIC_FILES_CHKSUM</filename> variable lists all the

1513

important files that contain the license text for the source code.

1514

It is possible to specify a checksum for an entire file, or a specific section of a

1515

file (specified by beginning and ending line numbers with the "beginline" and "endline"

1516

parameters, respectively).

1517

The latter is useful for source files with a license notice header,

1518

README documents, and so forth.

1519

If you do not use the "beginline" parameter, then it is assumed that the text begins on the

1520

first line of the file.

1521

Similarly, if you do not use the "endline" parameter, it is assumed that the license text

1522

ends with the last line of the file.

</para>

<para>

The "md5" parameter stores the md5 checksum of the license text.

1527

If the license text changes in any way as compared to this parameter

1528

then a mismatch occurs.

1529

This mismatch triggers a build failure and notifies the developer.

1530

Notification allows the developer to review and address the license text changes.

1531

Also note that if a mismatch occurs during the build, the correct md5

1532

checksum is placed in the build log and can be easily copied to the recipe.

</para>

<para>

There is no limit to how many files you can specify using the

1537

<filename>LIC_FILES_CHKSUM</filename> variable.

1538

Generally, however, every project requires a few specifications for license tracking.

1539

Many projects have a "COPYING" file that stores the license information for all the source

1540

code files.

1541

This practice allows you to just track the "COPYING" file as long as it is kept up to date.

</para>

<tip>

If you specify an empty or invalid "md5" parameter, BitBake returns an md5 mis-match

1546

error and displays the correct "md5" parameter value during the build.

1547

The correct parameter is also captured in the build log.

</tip>

<tip>

If the whole file contains only license text, you do not need to use the "beginline" and

1552

"endline" parameters.

</tip>

</section>

</section>

<title>Enabling Commercially Licensed Recipes</title>

1559

1560

<para>

1561

By default, the OpenEmbedded build system disables

1562

components that have commercial or other special licensing

1563

requirements.

1564

Such requirements are defined on a

1565

recipe-by-recipe basis through the

1566

1567

variable definition in the affected recipe.

1568

For instance, the

1569

<filename>poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly</filename>

1570

recipe contains the following statement:

1571

1572

LICENSE_FLAGS = "commercial"

1573

</literallayout>

1574

Here is a slightly more complicated example that contains both an

1575

explicit recipe name and version (after variable expansion):

1576

1577

LICENSE_FLAGS = "license_${PN}_${PV}"

1578

</literallayout>

1579

In order for a component restricted by a <filename>LICENSE_FLAGS</filename>

1580

definition to be enabled and included in an image, it

1581

needs to have a matching entry in the global

1582

1583

variable, which is a variable

1584

typically defined in your <filename>local.conf</filename> file.

1585

For example, to enable

1586

the <filename>poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly</filename>

1587

package, you could add either the string

1588

"commercial_gst-plugins-ugly" or the more general string

1589

"commercial" to <filename>LICENSE_FLAGS_WHITELIST</filename>.

1590

See the

1591

"<link linkend='license-flag-matching'>License Flag Matching</link>" section

1592

for a full explanation of how <filename>LICENSE_FLAGS</filename> matching works.

1593

Here is the example:

1594

1595

LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly"

1596

</literallayout>

1597

Likewise, to additionally enable the package built from the recipe containing

1598

<filename>LICENSE_FLAGS = "license_${PN}_${PV}"</filename>, and assuming

1599

that the actual recipe name was <filename>emgd_1.10.bb</filename>,

1600

the following string would enable that package as well as

1601

the original <filename>gst-plugins-ugly</filename> package:

1602

1603

LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly license_emgd_1.10"

1604

</literallayout>

1605

As a convenience, you do not need to specify the complete license string

1606

in the whitelist for every package.

1607

You can use an abbreviated form, which consists

1608

of just the first portion or portions of the license string before

1609

the initial underscore character or characters.

1610

A partial string will match

1611

any license that contains the given string as the first

1612

portion of its license.

1613

For example, the following

1614

whitelist string will also match both of the packages

1615

previously mentioned as well as any other packages that have

1616

licenses starting with "commercial" or "license".

1617

1618

LICENSE_FLAGS_WHITELIST = "commercial license"

</literallayout>

</para>

<title>License Flag Matching</title>

1624

1625

<para>

1626

License flag matching allows you to control what recipes the

1627

OpenEmbedded build system includes in the build.

1628

Fundamentally, the build system attempts to match

1629

1630

strings found in recipes against

1631

1632

strings found in the whitelist.

1633

A match causes the build system to include a recipe in the

1634

build, while failure to find a match causes the build system to

exclude a recipe.

</para>

<para>

In general, license flag matching is simple.

1640

However, understanding some concepts will help you

1641

correctly and effectively use matching.

</para>

<para>

Before a flag

defined by a particular recipe is tested against the

1647

contents of the whitelist, the expanded string

1648

<filename>_${PN}</filename> is appended to the flag.

1649

This expansion makes each <filename>LICENSE_FLAGS</filename>

1650

value recipe-specific.

1651

After expansion, the string is then matched against the

1652

whitelist.

1653

Thus, specifying

1654

<filename>LICENSE_FLAGS = "commercial"</filename>

1655

in recipe "foo", for example, results in the string

1656

<filename>"commercial_foo"</filename>.

1657

And, to create a match, that string must appear in the

whitelist.

</para>

<para>

Judicious use of the <filename>LICENSE_FLAGS</filename>

1663

strings and the contents of the

1664

<filename>LICENSE_FLAGS_WHITELIST</filename> variable

1665

allows you a lot of flexibility for including or excluding

1666

recipes based on licensing.

1667

For example, you can broaden the matching capabilities by

1668

using license flags string subsets in the whitelist.

1669

<note>When using a string subset, be sure to use the part of

1670

the expanded string that precedes the appended underscore

1671

character (e.g. <filename>usethispart_1.3</filename>,

1672

<filename>usethispart_1.4</filename>, and so forth).

1673

</note>

1674

For example, simply specifying the string "commercial" in

1675

the whitelist matches any expanded

1676

<filename>LICENSE_FLAGS</filename> definition that starts with

1677

the string "commercial" such as "commercial_foo" and

1678

"commercial_bar", which are the strings the build system

1679

automatically generates for hypothetical recipes named

1680

"foo" and "bar" assuming those recipes simply specify the

1681

following:

1682

1683

LICENSE_FLAGS = "commercial"

1684

</literallayout>

1685

Thus, you can choose to exhaustively

1686

enumerate each license flag in the whitelist and

1687

allow only specific recipes into the image, or

1688

you can use a string subset that causes a broader range of

1689

matches to allow a range of recipes into the image.

</para>

<para>

This scheme works even if the

1694

<filename>LICENSE_FLAGS</filename> string already

1695

has <filename>_${PN}</filename> appended.

1696

For example, the build system turns the license flag

1697

"commercial_1.2_foo" into "commercial_1.2_foo_foo" and would

1698

match both the general "commercial" and the specific

1699

"commercial_1.2_foo" strings found in the whitelist, as

expected.

</para>

<para>

Here are some other scenarios:

1705

1706

<listitem><para>You can specify a versioned string in the

1707

recipe such as "commercial_foo_1.2" in a "foo" recipe.

1708

The build system expands this string to

1709

"commercial_foo_1.2_foo".

1710

Combine this license flag with a whitelist that has

1711

the string "commercial" and you match the flag along

1712

with any other flag that starts with the string

1713

"commercial".</para></listitem>

1714

<listitem><para>Under the same circumstances, you can

1715

use "commercial_foo" in the whitelist and the

1716

build system not only matches "commercial_foo_1.2" but

1717

also matches any license flag with the string

1718

"commercial_foo", regardless of the version.

1719

</para></listitem>

1720

<listitem><para>You can be very specific and use both the

1721

package and version parts in the whitelist (e.g.

1722

"commercial_foo_1.2") to specifically match a

1723

versioned recipe.</para></listitem>

</itemizedlist>

</para>

</section>

<title>Other Variables Related to Commercial Licenses</title>

1730

1731

<para>

1732

Other helpful variables related to commercial

1733

license handling exist and are defined in the

1734

<filename>poky/meta/conf/distro/include/default-distrovars.inc</filename> file:

1735

1736

COMMERCIAL_AUDIO_PLUGINS ?= ""

1737

COMMERCIAL_VIDEO_PLUGINS ?= ""

Patrick Williams

2015-09-15 14:41:29 -0500

[diff] [blame]

1738

</literallayout>

1739

If you want to enable these components, you can do so by making sure you have

1740

statements similar to the following

1741

in your <filename>local.conf</filename> configuration file:

1742

1743

COMMERCIAL_AUDIO_PLUGINS = "gst-plugins-ugly-mad \

1744

gst-plugins-ugly-mpegaudioparse"

1745

COMMERCIAL_VIDEO_PLUGINS = "gst-plugins-ugly-mpeg2dec \

1746

gst-plugins-ugly-mpegstream gst-plugins-bad-mpegvideoparse"

Patrick Williams

2015-09-15 14:41:29 -0500

[diff] [blame]

1747

LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly commercial_gst-plugins-bad commercial_qmmp"

1748

</literallayout>

1749

Of course, you could also create a matching whitelist

1750

for those components using the more general "commercial"

1751

in the whitelist, but that would also enable all the

1752

other packages with

1753

1754

containing "commercial", which you may or may not want:

1755

1756

LICENSE_FLAGS_WHITELIST = "commercial"

</literallayout>

</para>

<para>

Specifying audio and video plug-ins as part of the

1762

<filename>COMMERCIAL_AUDIO_PLUGINS</filename> and

1763

<filename>COMMERCIAL_VIDEO_PLUGINS</filename> statements

Patrick Williams

2016-06-20 12:57:21 -0500

[diff] [blame]

1764

(along with the enabling

1765

<filename>LICENSE_FLAGS_WHITELIST</filename>) includes the

Patrick Williams