Blame - poky/documentation/test-manual/test-manual-intro.xml - openbmc/openbmc

>http://downloads.yoctoproject.org/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.html</link>

370

and <link linkend=""

371

>http://downloads.yoctoproject.org/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.txt</link>.</para></listitem>

372

373

<para>The tests are contained in

374

<filename>lib/oeqa/buildperf/test_basic.py</filename>.</para>

</listitem>

</itemizedlist>

</para></listitem>

</itemizedlist>

</para>

</section>

<title>Test Examples</title>

388

389

<para>This section provides example tests for each of the tests listed in the <link

390

linkend="test-test-mapping">How Tests Map to Areas of Code</link> section. </para>

391

<para>For oeqa tests, testcases for each area reside in the main test directory at

392

<filename>meta/lib/oeqa/selftest/cases</filename> directory.</para>

393

<para>For oe-selftest. bitbake testcases reside in the <filename>lib/bb/tests/</filename>

directory. </para>

<title><filename>bitbake-selftest</filename></title>

398

399

<para>A simple test example from <filename>lib/bb/tests/data.py</filename> is:

400

401

class DataExpansions(unittest.TestCase):

402

def setUp(self):

403

self.d = bb.data.init()

404

self.d["foo"] = "value_of_foo"

405

self.d["bar"] = "value_of_bar"

406

self.d["value_of_foo"] = "value_of_'value_of_foo'"

407

408

def test_one_var(self):

409

val = self.d.expand("${foo}")

410

self.assertEqual(str(val), "value_of_foo")

411

</literallayout>

412

</para>

413

<para>In this example, a <ulink url=""><filename>DataExpansions</filename></ulink> class

414

of tests is created, derived from standard python unittest. The class has a common

415

<filename>setUp</filename> function which is shared by all the tests in the

416

class. A simple test is then added to test that when a variable is expanded, the

417

correct value is found.</para>

418

<para>Bitbake selftests are straightforward python unittest. Refer to the Python

419

unittest documentation for additional information on writing these tests at: <link

420

linkend="">https://docs.python.org/3/library/unittest.html</link>.</para>

</section>

<title><filename>oe-selftest</filename></title>

425

426

<para>These tests are more complex due to the setup required behind the scenes for full

427

builds. Rather than directly using Python's unittest, the code wraps most of the

428

standard objects. The tests can be simple, such as testing a command from within the

429

OE build environment using the following

430

example:<literallayout class="monospaced">

431

class BitbakeLayers(OESelftestTestCase):

432

def test_bitbakelayers_showcrossdepends(self):

433

result = runCmd('bitbake-layers show-cross-depends')

434

self.assertTrue('aspell' in result.output, msg = "No dependencies

435

were shown. bitbake-layers show-cross-depends output:

436

%s"% result.output)

437

</literallayout></para>

438

<para>This example, taken from

439

<filename>meta/lib/oeqa/selftest/cases/bblayers.py</filename>, creates a

440

testcase from the <ulink url=""><filename>OESelftestTestCase</filename></ulink>

441

class, derived from <filename>unittest.TestCase</filename>, which runs the

442

<filename>bitbake-layers</filename> command and checks the output to ensure it

443

contains something we know should be here.</para>

444

<para>The <filename>oeqa.utils.commands</filename> module contains Helpers which can

445

assist with common tasks, including:<itemizedlist>

446

447

<para><emphasis>Obtaining the value of a bitbake variable:</emphasis> Use

448

<filename>oeqa.utils.commands.get_bb_var()</filename> or use

449

<filename>oeqa.utils.commands.get_bb_vars()</filename> for more than

one variable</para>

</listitem>

<para><emphasis>Running a bitbake invocation for a build:</emphasis> Use

454

<filename>oeqa.utils.commands.bitbake()</filename></para>

455

</listitem>

456

457

<para><emphasis>Running a command:</emphasis> Use

458

<filename>oeqa.utils.commandsrunCmd()</filename></para>

459

</listitem>

460

</itemizedlist></para>

461

<para>There is also a <filename>oeqa.utils.commands.runqemu()</filename> function for

462

launching the <filename>runqemu</filename> command for testing things within a

463

running, virtualized image.</para>

464

<para>You can run these tests in parallel. Parallelism works per test class, so tests

465

within a given test class should always run in the same build, while tests in

466

different classes or modules may be split into different builds. There is no data

467

store available for these tests since the tests launch the

468

<filename>bitbake</filename> command and exist outside of its context. As a

469

result, common bitbake library functions (bb.*) are also unavailable.</para>

</section>

<title><filename>testimage</filename></title>

474

475

<para>These tests are run once an image is up and running, either on target hardware or

476

under QEMU. As a result, they are assumed to be running in a target image

477

environment, as opposed to a host build environment. A simple example from

478

<filename>meta/lib/oeqa/runtime/cases/python.py</filename> contains the

479

following:<literallayout class="monospaced">

480

class PythonTest(OERuntimeTestCase):

481

@OETestDepends(['ssh.SSHTest.test_ssh'])

482

@OEHasPackage(['python3-core'])

483

def test_python3(self):

484

cmd = "python3 -c \"import codecs; print(codecs.encode('Uryyb,

485

jbeyq', 'rot13'))\""

486

status, output = self.target.run(cmd)

487

msg = 'Exit status was not 0. Output: %s' % output

488

self.assertEqual(status, 0, msg=msg)

489

</literallayout></para>

490

<para>In this example, the <ulink url=""><filename>OERuntimeTestCase</filename></ulink>

491

class wraps <filename>unittest.TestCase</filename>. Within the test,

492

<filename>self.target</filename> represents the target system, where commands

493

can be run on it using the <filename>run()</filename> method. </para>

494

<para>To ensure certain test or package dependencies are met, you can use the

495

<filename>OETestDepends</filename> and <filename>OEHasPackage</filename>

496

decorators. For example, the test in this example would only make sense if

497

python3-core is installed in the image.</para>

</section>

<title><filename>testsdk_ext</filename></title>

502

503

<para>These tests are run against built extensible SDKs (eSDKs). The tests can assume

504

that the eSDK environment has already been setup. An example from

505

<filename>meta/lib/oeqa/sdk/cases/devtool.py</filename> contains the

506

following:<literallayout class="monospaced">

507

class DevtoolTest(OESDKExtTestCase):

508

@classmethod

509

def setUpClass(cls):

510

myapp_src = os.path.join(cls.tc.esdk_files_dir, "myapp")

511

cls.myapp_dst = os.path.join(cls.tc.sdk_dir, "myapp")

512

shutil.copytree(myapp_src, cls.myapp_dst)

513

subprocess.check_output(['git', 'init', '.'], cwd=cls.myapp_dst)

514

subprocess.check_output(['git', 'add', '.'], cwd=cls.myapp_dst)

515

subprocess.check_output(['git', 'commit', '-m', "'test commit'"], cwd=cls.myapp_dst)

516

517

@classmethod

518

def tearDownClass(cls):

519

shutil.rmtree(cls.myapp_dst)

520

def _test_devtool_build(self, directory):

521

self._run('devtool add myapp %s' % directory)

522

try:

523

self._run('devtool build myapp')

524

finally:

525

self._run('devtool reset myapp')

526

def test_devtool_build_make(self):

527

self._test_devtool_build(self.myapp_dst)

528

</literallayout>In

529

this example, the <filename>devtool</filename> command is tested to see whether a

530

sample application can be built with the <filename>devtool build</filename> command

531

within the eSDK.</para>

</section>

<title><filename>testsdk</filename></title>

536

537

<para>These tests are run against built SDKs. The tests can assume that an SDK has

538

already been extracted and its environment file has been sourced. A simple example

539

from <filename>meta/lib/oeqa/sdk/cases/python2.py</filename> contains the

540

following:<literallayout class="monospaced">

541

class Python3Test(OESDKTestCase):

542

def setUp(self):

543

if not (self.tc.hasHostPackage("nativesdk-python3-core") or

544

self.tc.hasHostPackage("python3-core-native")):

545

raise unittest.SkipTest("No python3 package in the SDK")

546

547

def test_python3(self):

548

cmd = "python3 -c \"import codecs; print(codecs.encode('Uryyb, jbeyq', 'rot13'))\""

549

output = self._run(cmd)

550

self.assertEqual(output, "Hello, world\n")

551

</literallayout>In

552

this example, if nativesdk-python3-core has been installed into the SDK, the code

553

runs the python3 interpreter with a basic command to check it is working correctly.

554

The test would only run if python3 is installed in the SDK.</para>

</section>

<title><filename>oe-build-perf-test</filename></title>

559

560

<para>The performance tests usually measure how long operations take and the resource

561

utilisation as that happens. An example from

562

<filename>meta/lib/oeqa/buildperf/test_basic.py</filename> contains the

563

following:<literallayout class="monospaced">

564

class Test3(BuildPerfTestCase):

565

566

def test3(self):

567

"""Bitbake parsing (bitbake -p)"""

568

# Drop all caches and parse

569

self.rm_cache()

570

oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)

571

self.measure_cmd_resources(['bitbake', '-p'], 'parse_1',

572

'bitbake -p (no caches)')

573

# Drop tmp/cache

574

oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)

575

self.measure_cmd_resources(['bitbake', '-p'], 'parse_2',

576

'bitbake -p (no tmp/cache)')

577

# Parse with fully cached data

578

self.measure_cmd_resources(['bitbake', '-p'], 'parse_3',

579

'bitbake -p (cached)')

580

</literallayout>This

581

example shows how three specific parsing timings are measured, with and without

Andrew Geissler

c9f7865

2020-09-18 14:11:35 -0500

[diff] [blame]

582

various caches, to show how BitBake's parsing performance trends over time.</para>

Andrew Geissler

d25ed32

2020-06-27 00:28:28 -0500

[diff] [blame]

</section>

</section>

<title>Considerations When Writing Tests</title>

587

<para>When writing good tests, there are several things to keep in mind. Since things

588

running on the Autobuilder are accessed concurrently by multiple workers, consider the

589

following:</para>

590

591

<title>Running "cleanall" is not permitted</title>

592

<para>This can delete files from DL_DIR which would potentially break other builds

593

running in parallel. If this is required, DL_DIR must be set to an isolated

directory.</para>

</formalpara>

<title>Running "cleansstate" is not permitted</title>

598

<para>This can delete files from SSTATE_DIR which would potentially break other builds

599

running in parallel. If this is required, SSTATE_DIR must be set to an isolated

600

directory. Alternatively, you can use the "-f" option with the

601

<filename>bitbake</filename> command to "taint" tasks by changing the sstate

602

checksums to ensure sstate cache items will not be reused.</para>

603

</formalpara>

604

605

<title>Tests should not change the metadata</title>

606

<para>This is particularly true for oe-selftests since these can run in parallel and

607

changing metadata leads to changing checksums, which confuses BitBake while running

608

in parallel. If this is necessary, copy layers to a temporary location and modify

609

them. Some tests need to change metadata, such as the devtool tests. To prevent the

610

metadate from changes, set up temporary copies of that data first.</para>

</formalpara>

</section>

</chapter>

<!--

vim: expandtab tw=80 ts=4

624

-->