blob: 76b5fcdb6dcb596f1cb7ecc173add4b716197b07 [file] [log] [blame]
Patrick Williamsac13d5f2023-11-24 18:59:46 -06001# Patchtest
2
3## Introduction
4
5Patchtest is a test framework for community patches based on the standard
6unittest python module. As input, it needs tree elements to work properly:
7a patch in mbox format (either created with `git format-patch` or fetched
8from 'patchwork'), a test suite and a target repository.
9
10The first test suite intended to be used with patchtest is found in the
11openembedded-core repository [1] targeted for patches that get into the
12openembedded-core mailing list [2]. This suite is also intended as a
13baseline for development of similar suites for other layers as needed.
14
15Patchtest can either run on a host or a guest machine, depending on which
16environment the execution needs to be done. If you plan to test your own patches
17(a good practice before these are sent to the mailing list), the easiest way is
18to install and execute on your local host; in the other hand, if automatic
19testing is intended, the guest method is strongly recommended. The guest
20method requires the use of the patchtest layer, in addition to the tools
21available in oe-core: https://git.yoctoproject.org/patchtest/
22
23## Installation
24
25As a tool for use with the Yocto Project, the [quick start guide](https://docs.yoctoproject.org/brief-yoctoprojectqs/index.html)
26contains the necessary prerequisites for a basic project. In addition,
27patchtest relies on the following Python modules:
28
29- boto3 (for sending automated results emails only)
30- git-pw>=2.5.0
31- jinja2
32- pylint
33- pyparsing>=3.0.9
34- unidiff
35
36These can be installed by running `pip install -r
37meta/lib/patchtest/requirements.txt`. Note that git-pw is not
38automatically added to the user's PATH; by default, it is installed at
39~/.local/bin/git-pw.
40
41For git-pw (and therefore scripts such as patchtest-get--series) to work, you need
42to provide a Patchwork instance in your user's .gitconfig, like so (the project
43can be specified using the --project argument):
44
45 git config --global pw.server "https://patchwork.yoctoproject.org/api/1.2/"
46
47To work with patchtest, you should have the following repositories cloned:
48
491. https://git.openembedded.org/openembedded-core/ (or https://git.yoctoproject.org/poky/)
502. https://git.openembedded.org/bitbake/ (if not using poky)
513. https://git.yoctoproject.org/patchtest (if using guest mode)
52
53## Usage
54
55### Obtaining Patches
56
57Patch files can be obtained directly from cloned repositories using `git
58format-patch -N` (where N is the number of patches starting from HEAD to
59generate). git-pw can also be used with filters for users, patch/series IDs,
60and timeboxes if specific patches are desired. For more information, see the
61git-pw [documentation](https://patchwork.readthedocs.io/projects/git-pw/en/latest/).
62
63Alternatively, `scripts/patchtest-get-series` can be used to pull mbox files from
64the Patchwork instance configured previously in .gitconfig. It uses a log file
65called ".series_test.log" to store and compare series IDs so that the same
66versions of a patch are not tested multiple times unintentionally. By default,
67it will pull up to five patch series from the last 30 minutes using oe-core as
68the target project, but these parameters can be configured using the `--limit`,
69`--interval`, and `--project` arguments respectively. For more information, run
70`patchtest-get-series -h`.
71
72### Host Mode
73
74To run patchtest on the host, do the following:
75
761. In openembedded-core/poky, do `source oe-init-build-env`
772. Generate patch files from the target repository by doing `git-format patch -N`,
78 where N is the number of patches starting at HEAD, or by using git-pw
79 or patchtest-get-series
803. Run patchtest on a patch file by doing the following:
81
82 patchtest --patch /path/to/patch/file
83
84 or, if you have stored the patch files in a directory, do:
85
86 patchtest --directory /path/to/patch/directory
87
88 For example, to test `master-gcc-Fix--fstack-protector-issue-on-aarch64.patch` against the oe-core test suite:
89
90 patchtest --patch master-gcc-Fix--fstack-protector-issue-on-aarch64.patch
91
92 If you want to use a different test suite or target repository, you can use the --testdir and --repodir flags:
93
94 patchtest --patch /path/to/patch/file --repodir /path/to/repo --testdir /path/to/test/dir
95
96### Guest Mode
97
98Patchtest's guest mode has been refactored to more closely mirror the
99typical Yocto Project image build workflow, but there are still some key
100differences to keep in mind. The primary objective is to provide a level
101of isolation from the host when testing patches pulled automatically
102from the mailing lists. When executed this way, the test process is
103essentially running random code from the internet and could be
104catastrophic if malicious bits or even poorly-handled edge cases aren't
105protected against. In order to use this mode, the
106https://git.yoctoproject.org/patchtest/ repository must be cloned and
107the meta-patchtest layer added to bblayers.conf.
108
109The general flow of guest mode is:
110
1111. Run patchtest-setup-sharedir --directory <dirname> to create a
112 directory for mounting
1132. Collect patches via patchtest-get-series (or other manual step) into the
114 <dirname>/mboxes path
1153. Ensure that a user with ID 1200 has appropriate read/write
116 permissions to <dirname> and <dirname>/mboxes, so that the
117 "patchtest" user in the core-image-patchtest image can function
1184. Build the core-image-patchtest image
1195. Run the core-image-patchtest image with the mounted sharedir, like
120 so:
121 `runqemu kvm nographic qemuparams="-snapshot -fsdev
122 local,id=test_mount,path=/workspace/yocto/poky/build/patchtestdir,security_model=mapped
123 -device virtio-9p-pci,fsdev=test_mount,mount_tag=test_mount -smp 4 -m
124 2048"`
125
126Patchtest runs as an initscript for the core-image-patchtest image and
127shuts down after completion, so there is no input required from a user
128during operation. Unlike in host mode, the guest is designed to
129automatically generate test result files, in the same directory as the
130targeted patch files but with .testresult as an extension. These contain
131the entire output of the patchtest run for each respective pass,
132including the PASS, FAIL, and SKIP indicators for each test run.
133
134## Contributing
135
Patrick Williams73bd93f2024-02-20 08:07:48 -0600136The yocto mailing list (openembedded-core@lists.openembedded.org) is used for questions,
Patrick Williamsac13d5f2023-11-24 18:59:46 -0600137comments and patch review. It is subscriber only, so please register before
138posting.
139
Patrick Williamsac13d5f2023-11-24 18:59:46 -0600140When sending single patches, please use something like:
141
Patrick Williams73bd93f2024-02-20 08:07:48 -0600142 git send-email -M -1 --to=openembedded-core@lists.openembedded.org --subject-prefix=OE-core][PATCH
Patrick Williamsac13d5f2023-11-24 18:59:46 -0600143
144## Maintenance
145-----------
146
147Maintainers:
148 Trevor Gamblin <tgamblin@baylibre.com>
149
150## Links
151-----
152[1] https://git.openembedded.org/openembedded-core/
153[2] https://www.yoctoproject.org/community/mailing-lists/