| Age | Commit message (Collapse) | Author | Files |
|---|
|
Add an option for random MACHINE into oe-selftest:
--machine [random/all]
1. random: will set a random MACHINE for each test
2. all: will run tests for all machines
Custom machine sets only weak default values (??=) for MACHINE in machine.inc.
This let test cases that require a specific MACHINE to be able to
override it, using (?= or =).
e.g.:
oe-selftest --run-tests signing --machine random -->
will run all tests switching MACHINE randomly for each test
oe-selftest --run-tests signing --machine all -->
for each machine will run all tests
oe-selftest --run-all-tests --machine random
Also update oeqa/selftest/base.py to accomodate this feature.
Fix for [YOCTO #5880].
Signed-off-by: Daniel Istrate <daniel.alexandrux.istrate@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
This option will list all available tests in a comprehensive manner.
Fix for [YOCTO #8868]
Signed-off-by: Daniel Istrate <daniel.alexandrux.istrate@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
--list-classes does a weak validation when determining test names:
(if method.startswith("test_") which could report any class attribute
that starts with 'test_' as a valid test case.
This fix checks that the class attribute that starts with 'test_'
is also callable (is a method).
Fix for [YOCTO #8862]
Signed-off-by: Daniel Istrate <daniel.alexandrux.istrate@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Coverage data tracking initiates too early, causing coverage data from the
oe-selftest environment setting to be added to each run. Even when no tests are run
oe-selftest reports around 24% of coverage due to this extra data.
Change the custom resultclass used by the TextTestRunner to one generated from the
command arguments. The generated class processes coverage when needed, running
coverage setup just before the first testcase is run and reporting after the last
one finished.
[Yocto #8846]
Signed-off-by: Humberto Ibarra <humberto.ibarra.lopez@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Each time oe-selftest runs, the oe-selftest.log file is overwritten.
This patch solves it by adding time stamp to each selftest log file
and doing a symlink named as oe-selftest.log to the last one created.
Signed-off-by: Costin Constantin <costin.c.constantin@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
For scripts that use Python's standard argparse module to parse
command-line arguments, create a subclass which will show the usage
the usage information when a command-line parsing error occurs. The most
common case would be when the script is run with no arguments; at least
then the user immediately gets to see what arguments they might need to
pass instead of just an error message.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
[YOCTO #8750] Allow oe-selftest to run custom test suites based on different criteria
1. Can run custom lists of tests based on different criteria:
--run-tests-by <name|class|module|id|tag> <list of tests|classes|modules|ids|tags>
eg: --run-tests-by module imagefeatures signing recipetool
--run-tests-by id 1377 1273 935
--run-tests-by tag wic sstate bitbake
2. Can list tests based on different criteria:
--list-tests-by <name|class|module|id|tag> <list of tests|classes|modules|ids|tags>
eg: --list-tests-by module imagefeatures signing recipetool
--list-tests-by id 1377 1273 935
--list-tests-by tag wic sstate bitbake
3. Can list all tags that have been set to test cases:
--list-tags
The list of tags should be kept as minimal as possible.
This helps preview the tags used so far.
To take advantage of the 'tag' feature:
- add @tag(feature=<>) to testcases
eg: @tag(feature='signing') for a single tag
@tag(feature=(('signing', 'sstate')) or
@tag(feature=['signing', 'sstate']) for multiple tags
Signed-off-by: Daniel Istrate <daniel.alexandrux.istrate@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Enable code coverage through the library 'python coverage'. In case the environment
variable COVERAGE_PROCESS_START is present (one of the requisites for measuring
sub-processes; the second one is including some coverage statements into the
python sitecustomize.py file) it will be taken into account, otherwise it is
exported with value '.coveragerc'. The latter value is a configuration file
(also automatically created) with some default settings. Once tests are
executed, a coverage report is shown on the log and the coverage output data is stored
with name '.coverage.<args>' where '<args>' is the name of the unit tests executed
or 'all_tests' when running with --run-all-tests. This output data can be latter used
for better reporting using the same tool (coverage).
As briefly indicate before, measuring sub-process implies setting the env variable
COVERAGE_PROCESS_START (done automatically by the oe-selftest code with this patch if
not already set) and creating a sitecustomize.py as explained on [1].
If either one of these is missing, complete coverage will be incomplete.
Current measurements for 'oe-selftest --run-all-tests' indicate that current coverage
is around 42 % taking into account BBLAYERS, bitbake and scripts folders. More details
on [2], indicating the coverage per file/module.
This tasks has been done together with Humberto Ibarra <humberto.ibarra.lopez@linux.intel.com>
[YOCTO #8679]
[1] http://coverage.readthedocs.org/en/latest/subprocess.html
[2] https://bugzilla.yoctoproject.org/attachment.cgi?id=2854
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
In particular, this allows us to use code from bitbake's bb module
(such as tinfoil).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This ensures that oeqa.selftest.* from layers are found.
[YOCTO #7625]
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
This ensures that all paths that hold selftest tests will be checked
(oeqa.selftest is a namespace package).
[YOCTO #7625]
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
When oe-selftest starts it includes bblayers.inc into bblayers.conf
When oe-selftest ends it deletes bblayers.inc and the included line
from bblayers.conf
Signed-off-by: Daniel Istrate <daniel.alexandrux.istrate@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
The selftest can take a couple of hours to run, so add a custom result class to
timestamp the output to make it easy to spot any slow tests.
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
While trying to discover what tests are available, I felt the
need to be able to list all individual tests so I can run specific
tests.
This patch adds the "--list-classes" command line option that
lists the unit test classes and methods available.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Fixing a backtrace dump that happens if the script is started
without sourcing the oe-init-build-env first.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
[YOCTO #6453]
Signed-off-by: Corneliu Stoicescu <corneliux.stoicescu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Regardless if the tests passed or not the script returned 0,
which isn't what one would expect.
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
complete cleanup at the end
The script should clean-up all the .inc files that might
have been created by tests regardless of the outcome or if
the script is interrupted. (currently the
last test will leave a conf/selftest.inc around, even
if it's not included anywhere)
Also fix delete_recipeinc to actually delete what's supposed to.
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
various scripts
The purpose of oe-selftest is to run unittest modules added from meta/lib/oeqa/selftest,
which are tests against bitbake tools.
Right now the script it's useful for simple tests like:
- "bitbake --someoption, change some metadata, bitbake X, check something" type scenarios (PR service, error output, etc)
- or "bitbake-layers <...>" type scripts and yocto-bsp tools.
This commit also adds some helper modules that the tests will use and a base class.
Also, most of the tests will have a dependency on a meta-selftest layer
which contains specially modified recipes/bbappends/include files for the purpose of the tests.
The tests themselves will usually write to ".inc" files from the layer or in conf/selftest.inc
(which is added as an include in local.conf at the start and removed at the end)
It's a simple matter or sourcing the enviroment, adding the meta-selftest layer to bblayers.conf
and running: oe-selftest to get some results. It would finish faster if at least a core-image-minimal
was built before.
[ YOCTO #4740 ]
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
