| Age | Commit message (Collapse) | Author | Files |
|---|
|
Splits the SDK test implementation into configurable Python classes. The
classes used for the normal and extensible SDKs are
${TESTSDK_CLASS_NAME} and ${TESTSDKEXT_CLASS_NAME} respectively.
This allows SDK machines to override the classes used to implement the
tests. For the traditional SDK, a common "run()" function is provided by
the class (oeqa.sdk.testsdk.TestSDK), with several hook member functions
that can be overridden in child classes, making it easier to have
consistent behavior. The extensible SDK class
(oeqa.sdkext.testsdk.TestSDKEXT) also has a common "run()" function, but
no hooks have yet been added as there is not currently a known use case
for create derived classes.
These changes should be purely organizational; no functional changes
have been made to either the standard SDK or extensible SDK tests.
[YOCTO #13020]
(From OE-Core rev: a06d53928b22d5f88276023c4d57b206db2f27f9)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
|
|
warnings
Use our own lsb function instead as used elsewhere by the codebase.
(From OE-Core rev: acac45a6fd604d28ef7c23d67482af3d7e8bcfe3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Tweak the preceeding commit to:
* Add STARTTIME to the identifier to make it unique
* Add MACHINE to the identifier
* Use LOG_DIR
* Store the layer config in a more natural json format
* Drop '_' function prefixes
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
As part of the solution to replace Testopia to store testresult,
OEQA sdk and sdkext need to output testresult into json files, where
these json testresult files will be stored into git repository
by the future test-case-management tools.
By default, json testresult file will be written to "oeqa"
directory under the "WORKDIR" directory.
To configure multiple instances of bitbake to write json testresult
to a single testresult file at custom directory, user will define
the variable "OEQA_JSON_RESULT_DIR" with the custom directory for
json testresult.
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The testtools or subunit is not part of python's standard library, so check them
before use.
Fixed when they are not installed on host:
$ bitbake core-image-minimal -ctestsdk
Exception: ImportError: No module named 'testtools'
Now it can run with a warning:
WARNING: core-image-minimal-1.0-r0 do_testsdk: Failed to import testtools or subunit, the testcases will run serially
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
This uses the new concurrenttest code to enable parallel test execution
if specified.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We have several options for parallel processing in oeqa, parallel
execution of modules, threading and mulitple processes for the runners.
After much experimentation is appears the most scalable and least
invasive approach is multiple processes using concurrenttestsuite
from testtools. This means we can drop the current threading code
which is only used by the sdk test execution.
oeqa/decorator/depends: Remove threading code
Revert "oeqa/sdk: Enable usage of OEQA thread mode"
This reverts commit adc434c0636b7dea2ef70c8d2c8e61cdb5c703b1.
Revert "oeqa/core/tests: Add tests of OEQA Threaded mode"
This reverts commit a4eef558c9933eb32413b61ff80a11b999951b40.
Revert "oeqa/core/decorator/oetimeout: Add support for OEQA threaded mode"
This reverts commit d3d4ba902dee8b19fa1054330cffdf73f9b81fe7.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently if we do a testsdkext task for the image, it would take
very long time to finish. The time is mostly spent fetching kernel
source via network.
We have done some configuration in auto.conf, trying to make use
of own-mirrors.bbclass to avoid fetching kernel via network.
However, the solution normally does not work. Below is some log
from log.do_fetch.
DEBUG: Fetcher failure: Unable to find file \
file:///path/to/downloads/git2_git.yoctoproject.org.linux-yocto.git.tar.gz \
anywhere.
The tar.gz file is not available. It is generated only if
BB_GENERATE_MIRROR_TARBALLS is set to "1". The default value of
BB_GENERATE_MIRROR_TARBALLS is "0", and according to the manual, users
need choose to set it to "1" only if they are trying to make a source
mirror. So generally, this var's value is "0".
Anyway, we do need to avoid fetching kernel source from network when
doing testsdkext. So set PREMIRRORS in auto.conf to achieve this.
After this change, the time reduces from 4209.131s to 1399.436s on
my local machine.
[YOCTO #12729]
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The TEST_IMAGE interface has never particularly worked and image testing currently
gets configured manually.
This reworks the interface to better serve the needs of its users, replacing it
with TESTIMAGE_AUTO.
This does away with the need for the separate class due to better bitbake APIs for
changing tasks.
TESTIMAGE_AUTO will automatically boot under qemu any image that is built. It
also adds in dependencies so that any SDK for which testing is requested will
automatically be built first.
The code in bitbake.conf was error prone (e.g. testsdk wasn't considered), this
improves it to standardise on IMAGE_CLASSES as the standard configuration mechanism.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Otherwise it would generate lines like the following when multilib:
INHERIT += "own-mirrors"SSTATE_MIRRORS += " \n file://.* file:///path/to/../share/sstate-cache/PATH"
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This is very useful for debugging. The similar to testsdk.bbclass.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Is better to log the summary at end to see in an easy way
the actual result of the test run.
[YOCTO #11622]
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
oeqa/sdk/context.py - Use OETestContextThreaded.
classes/testsdk.bbclass - Enable bb event thread mode to avoid
corrupt the PIPE when multiple threads writes.
[YOCTO #11450]
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Those methods are used to write in the log the results so
it makes sense to have defined at OETestResult because
is a format of the result itself.
[YOCTO #11450]
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
In the previous lock removal patch I mixed up the two tasks and broke sdk
testing. Fix this by removing locks for both sdk and extsdk.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We should be able to test sdks in parallel. As far as I can tell, this
restriction crept in for no good reason, based on locks that testimage used
for the non-qemu case.
Therefore remove the lock and allow sdks to test in parallel.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The local.conf file can get overritten and changed by the SDK so (ab)use
auto.conf instead to ensure our configuration tweaks get applied.
Also ensure we append to any other configured SSTATE_MIRROR variable
that already exists.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Use the finally: block to always to cleanup.
Now that the test harness in testsdk.bbclass has monkey-patched
CalledProcessException to display the output we don't need to do that in the
test case.
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When using a minimal eSDK, testing currently fails as the sdk isn't
populated. We therefore setup the eSDK under test to point at local
sstate and execute a command to ensure the toolchain is populated
since most of the tests depend on this being present.
At the same time, add in a link to DL_DIR through own-mirrors so
that tests which fetch source (e.g. the kernel module one) can
use the local stash. This cuts test execution of the kernel module
test from 2000s to 120s.
We did try using DL_DIR directly but that causes uninative issues
requiring other workarounds so own-mirrors is neater.
Together these fixes unbreak eSDK testing on the autobuilder.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This is a complementary fix to commit 7c552996:
[ meta: remove True option to getVar calls ]
it intended to remove all True option to getVar calls, but there are
still some remaining.
Search made with the following regex: getVar ?\((.*), True\)
Signed-off-by: Ming Liu <peter.x.liu@external.atlascopco.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
|
|
The TEST_LOG_DIR was used for store sdk_target_log this log
contains the output of the run of build commands now that information
could be found also on log.do_testsdk under WORKDIR.
The log will continue to store into SDK_DIR instead of TEST_LOG_DIR.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
|
|
The functionality provided is the same with imporvements on code
reuse and better interfaces.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
|
|
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.
Search made with the following regex: getVar ?\(( ?[^,()]*), True\)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
This is helpful when debug.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Make it clear that SDK testing can use any valid image.
Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
This sets a good example and avoids unnecessarily contributing to
perceived complexity and cargo culting.
Motivating quote below:
< kergoth> the *original* intent was for the function/task to error via
whatever appropriate means, bb.fatal, whatever, and
funcfailed was what you'd catch if you were calling
exec_func/exec_task. that is, it's what those functions
raise, not what metadata functions should be raising
< kergoth> it didn't end up being used that way
< kergoth> but there's really never a reason to raise it yourself
FuncFailed.__init__ takes a 'name' argument rather than a 'msg'
argument, which also shows that the original purpose got lost.
Signed-off-by: Ulf Magnusson <ulfalizer@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Use the new BB_SETSCENE_ENFORCE functionality to avoid having to run
bitbake twice on installing the extensible SDK - we can now do it all in
one invocation which not only takes less time, but we should also get
more meaningful errors for some types of failure, in particular where
downloading from an sstate mirror fails.
One result of this change is that you get the errors printed on the
console during normal output rather than this going to the
preparing_build_system.log file first. In OE-Core revision
227d2cbf9e0b8c35fa6644e3d72e0699db9607fa, we changed to always print the
contents of preparing_build_system.log on failure, but now at least the
error contents of that log is duplicated. Besides, I intentionally
didn't print out the contents of that log during normal usage because
it's quite verbose - the bug that we were attempting to fix was about
not getting this information when seeing failures in the automated
tests, thus I've moved printing the log to the test handling code
instead.
Part of the implementation of [YOCTO #9367].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
With Python 3 we get a bytes object from the command output and not a
string, which gives some ugly formatting for error messages unless you
decode it first.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
tcname is needed for eSDK update testcase will be used for
publish it and then try to update
Signed-off-by: Aníbal Limón <limon.anibal@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
The removal of bitbake and scripts PATH is only needed by eSDK tests
so move to eSDK context only.
This also it's a support for eSDK update test because it needs to
execute oe-publish-sdk from scripts.
Signed-off-by: Aníbal Limón <limon.anibal@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Due to the neeed to use in other modules.
Signed-off-by: Aníbal Limón <limon.anibal@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
The inclusion of STAGING_DIR/BASE_WORKDIR in PATH is contaminating
the environment, i.e. when try to sanity check perl (check_perl_modules)
it takes perl from STAGING_DIR causing eSDK install to fail.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Aníbal Limón <limon.anibal@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
When run SDK tests over eSDK we need to use SDKExtTestContext
instead of SDKTestContext because if we use SDKTestContext search
for SDK manifest and depends on the SDK manifest generation so
populate_sdk needs to be executed.
Adds a compatibility mode flag to SDKExtTestContext for search tests
over sdk module instead of sdkext module and change testsdk calls
to comply with this new param.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
oeqa/sdkext: Add module and __init__.py will contain eSDK tests.
classes/testsdk: Add support for run eSDK tests.
oeqa/oetest: Create oeSDKExtTest for now only inherit oeSDKTest,
modified SDKExtTestContext now inherit SDKTestContext
and set sdkext filesdir for store data fixtures.
Signed-off-by: Aníbal Limón <limon.anibal@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Extensible SDK is compatible with SDK test suite so it need
to execute the same tests over it.
Signed-off-by: Aníbal Limón <limon.anibal@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
This helper functions will be serve as well to run extensible
sdk tests so generalize it to get function context as arg.
Signed-off-by: Aníbal Limón <limon.anibal@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Method's for loadTests and runTests make sense to define
inside TestContext because it can be different around
Image, SDK, SDKExt.
Signed-off-by: Aníbal Limón <limon.anibal@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
In order to provide better abstraction move functions to get the test
suite inside the TestContext.
Signed-off-by: Aníbal Limón <limon.anibal@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Move anonymous duplicated class TestContext from testimage/testsdk to
oeqa/oetest now we have two new classes ImageTestContext and
SDKTestContext with common code in TestContext class.
Signed-off-by: Aníbal Limón <limon.anibal@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Extensible SDK needs to use network and some networks requires
proxies then export it.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Add task for test extensible sdk for now only install the SDK.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Moves all the testsdk code from testimage in order to have it's own
class because new tests will be added for extensible SDK.
The old paths for store logs "${WORKDIR}/testimage" and sdk
"${WORKDIR}/testimage-sdk" was maintained for compatibility may be
change to point testsdk after review the codebase.
The dependency of QEMU was removed because isn't needed.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
