summaryrefslogtreecommitdiff
path: root/meta/lib/oeqa/oetest.py
AgeCommit message (Collapse)AuthorFiles
2017-02-23oeqa: fix typoHumberto Ibarra1
The word 'uninstall' was spelled incorrectly in various parts of oeqa files. (From OE-Core rev: af365e025030436f83b233fa51ecc9c58c58ce3c) Signed-off-by: Humberto Ibarra <humberto.ibarra.lopez@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23oeqa: Fix files handling on runtime tests.Aníbal Limón1
Common files was move to oeqa/files from oeqa/runtime/files because the same files are used across Runtime,SDK,eSDK tests. Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2017-01-23oeqa/sdkext: Adds case and context modules.Aníbal Limón1
The extensible sdk context and case modules extends the sdk ones, this means that the tests from sdk are run also the sdkext tests. Enables support in context for use oe-test esdk command for run the test suites, the same options of sdk are required for run esdk tests. Removes old related to case and context inside oetest.py. [YOCTO #10599] Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2017-01-23oeqa/sdk: Add case and context modules for the SDK componentAníbal Limón1
Adds case and context modules for SDK based on oetest.py old code. Enables SDK Test component usage with oe-test, the SDK Test component adds command line options for specify sdk installed dir, sdk environment and target/hosts maniftest. [YOCTO #10599] Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
2016-12-16meta: remove True option to getVar callsJoshua Lock1
getVar() now defaults to expanding by default, thus remove the True option from getVar() calls with a regex search and replace. Search made with the following regex: getVar ?\(( ?[^,()]*), True\) Signed-off-by: Joshua Lock <joshua.g.lock@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-09-15oeqa/oetest: show stderr when running commandsRoss Burton1
To help debug failures, redirect stderr to stdout in oeSDKTest.run() and oeSDKExtTest.run(). Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-17oeqa/oetest: Improve subprocess error reportingRichard Purdie1
Without this, we get to know the command failed and the exit code but have no idea how the command failed since we don't get the output by default. This makes it much easier to see what went wrong and stand a chance of fixing it. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-04oeqa/oetest.py: Allow to export packages using symlinksMariano Lopez1
Currently packages that contains symlinks can't be extracted and exported. This allows to export extracted such packages. A nice side effect is improved readability. [YOCTO #9932] Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-08-04oetest.py: Remove check for __init__.pyMariano Lopez1
When using "auto" in TEST_SUITES there is a check for __init__.py in the <layer>/lib/oeqa/runtime/ directory in all the layers in BBLAYERS. This check was needed because the way that python 2 import the modules, now that bitbake uses python 3 there is no need these __init__.py files, moreover these files won't allow to import tests from other layers. This patch removes the check. [YOCTO #9996] Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-08-04oetest.py: Add command line parameter support for tag in testexportMariano Lopez1
This allows to use a command line argument to change the tag used to filter test instead of rebuilding the tests. [YOCTO #8532] Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-07-20oetest.py: Fix problem when there is more than one json file for package ↵Mariano Lopez1
extraction This allow to have more than one test to have json file in order to install packages in the DUT without using a package manager. [YOCTO #9926] Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-06-23oeqa: fix hasPackage, add hasPackageMatchRoss Burton1
hasPackage() was looking for the string provided as an RE substring in the manifest, which resulted in a large number of false positives (i.e. libgtkfoo would match "gtk+"). Rewrite the manifest loader to parse the files into a proper data structure, change hasPackage to do full string matches, and add hasPackageMatch which does RE substring matches. Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-06-06oetest.py: Add install/uninstall functionality for DUTsMariano Lopez1
Add the functionality to install/unistall packages in the DUTs without the use of the package manager. This is possible with the extraction introduced in package manager class. testimage and testexport bbclasses has been modified in order to support this new feature. [YOCTO #8694] Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-06lib/oeqa/otest.py: Fix import tests from other layers with python3Mariano Lopez1
In python3 the functionality to import modules has been changed and this broke the capability to add runtime tests from other layers. This commit returns this capability to testimage and testexport. [YOCTO #9705] Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-02classes/lib: Complete transition to python3Richard Purdie1
This patch contains all the other misc pieces of the transition to python3 which didn't make sense to be broken into individual patches. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-02classes/lib: Update to use python3 command pipeline decodingRichard Purdie1
In python3, strings are unicode by default. We need to encode/decode from command pipelines and other places where we interface with the real world using the correct locales. This patch updates various call sites to use the correct encoding/decodings. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-30oetest.py: Add support to copy unextracted packages for runtime testingMariano Lopez1
Sometimes is needed to have a package without extraction when running a test. This patch adds the functionality. [YOCTO #8536] Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-05-30oetest.py: Add extract_packages() to RuntimeTestContext classMariano Lopez1
This new method extracts the content of package (RPM, DEB, or IPK) to a directory inside of WORKDIR. The extraction is needed for later install in the DUTs without using a package manager. [YOCTO #8694] Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-05-30oetest.py: Add json file support to specify packages needed in runtime testsMariano Lopez1
This adds the functionality to use a json file to specify the packages needed for a particular test. The content of the json file is a dictionary with dictionaries inside, using the test name as the hash. The json file must have the same name as the class module name and must be in the same path. [YOCTO #7850] Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-05-30oetest.py: Move getTests() outside loadTests() methodMariano Lopez1
The method getTests() can be useful to all the class, not just to loadTests(). [YOCTO #8694] Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-05-18oetest.py: Use the real ExportTestContext in exported testsMariano Lopez1
There are parts of the fuctionallity missing when using the dummy ExportTestContext class in runexported.py. This changes the use of ExportTestContext dummy class from runexported.py to the real class in oetest.py. Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-18oetest.py: Add default pscmd to oeTestMariano Lopez1
pscmd is used by some tests to get the process running on the target. If the test are exported there won't be any pscmd attibute in the oeTest. This adds "ps" as default pscmd. Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-18oeqa/oetest.py: Add class ExportTestContextMariano Lopez1
Adding the class is needed to separate the exported test from the test image; both test run under different conditions, i.e. an exported test doesn't require to change the signal handling. This change adds clasess ExportTestContext and ImageTestContext, both of them inherits from RuntimeTestContext. Also refactors RuntimeTestContext class, to keep the code common in this class. Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-18oetest: make console output more verboseEd Bartosh1
Added output of test runner to bitbake console output. bitbake <image> -c testimage now reports test progress to the console: NOTE: Executing RunQueue Tasks ... NOTE: recipe core-image-lsb-1.0-r0: task do_testimage: Started core-image-lsb-1.0-r0 do_testimage: test_ping (oeqa.runtime.ping.PingTest) ... ok core-image-lsb-1.0-r0 do_testimage: test_ssh (oeqa.runtime.ssh.SshTest) ... ok ... ---------------------------------------------------------------------- core-image-lsb-1.0-r0 do_testimage: Ran 38 tests in 785.100s core-image-lsb-1.0-r0 do_testimage: core-image-lsb-1.0-r0 do_testimage: OK (skipped=1) Adding more output to the console should also prevent autobuilder to kill long running tests as ab assumes that test is stuck if no console output produced by it for a long time. [YOCTO #8238] Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-04-12oeqa/runexported.py: Fix exported testMariano Lopez1
With the changes introduced to test the eSDK the runexported test failed during the execution. This change fix runexported test in the least invasive way, because of the release cycle. Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-28classes/testsdk: Pass tcname to SDK and SDKExt contextsAníbal Limón1
tcname is needed for eSDK update testcase will be used for publish it and then try to update Signed-off-by: Aníbal Limón <limon.anibal@gmail.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-02-28classes/testsdk: Move the removal of bitbake PATH to eSDK context onlyAníbal Limón1
The removal of bitbake and scripts PATH is only needed by eSDK tests so move to eSDK context only. This also it's a support for eSDK update test because it needs to execute oe-publish-sdk from scripts. Signed-off-by: Aníbal Limón <limon.anibal@gmail.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-02-10oeqa/oetest: Fix compatibility SDK tests using eSDK.Aníbal Limón1
When run SDK tests over eSDK we need to use SDKExtTestContext instead of SDKTestContext because if we use SDKTestContext search for SDK manifest and depends on the SDK manifest generation so populate_sdk needs to be executed. Adds a compatibility mode flag to SDKExtTestContext for search tests over sdk module instead of sdkext module and change testsdk calls to comply with this new param. Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-02-10classes/populate_sdk_ext: Add SDK_EXT_TARGET_MANIFEST and SDK_EXT_HOST_MANIFESTAníbal Limón1
Extensible SDK needs to point to the correct manifest so add SDK_EXT_TARGET_MANIFEST and SDK_EXT_HOST_MANIFEST variables. oeqa/oetest.py: Fix SDKExtTestContext for load the correct manifests. Signed-off-by: Aníbal Limón <limon.anibal@gmail.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-02-10testsdkext: Add skeleton for support Extensible SDK tests.Aníbal Limón1
oeqa/sdkext: Add module and __init__.py will contain eSDK tests. classes/testsdk: Add support for run eSDK tests. oeqa/oetest: Create oeSDKExtTest for now only inherit oeSDKTest, modified SDKExtTestContext now inherit SDKTestContext and set sdkext filesdir for store data fixtures. Signed-off-by: Aníbal Limón <limon.anibal@gmail.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-02-10oeqa/oetest: oeSDKTest when run a command redirect env output to nullAníbal Limón1
Some tests are failing because the eSDK env load script shows a help message on the output so redirect this to /dev/null. There was a discussion with Paul Eggleton [1], he don't want to have env variables to change the behaviour of what the env script needs to print so redirect the output in oeSDK.run() method. I didn't agree because it can hides another messages like error ones and with the variable you can control what prints or not. [1] http://lists.openembedded.org/pipermail/openembedded-core/2016-February/116744.html Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-02-10oetest.py/TestContext: Move loadTests and runTests inside it.Aníbal Limón1
Method's for loadTests and runTests make sense to define inside TestContext because it can be different around Image, SDK, SDKExt. Signed-off-by: Aníbal Limón <limon.anibal@gmail.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-02-10testimage/testsdk: Move get test suites routine inside TestContext.Aníbal Limón1
In order to provide better abstraction move functions to get the test suite inside the TestContext. Signed-off-by: Aníbal Limón <limon.anibal@gmail.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-02-10testimage/testsdk: Modularize TestContext.Aníbal Limón1
Move anonymous duplicated class TestContext from testimage/testsdk to oeqa/oetest now we have two new classes ImageTestContext and SDKTestContext with common code in TestContext class. Signed-off-by: Aníbal Limón <limon.anibal@gmail.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-02-10get_test_suites: Add sdkext type for load test suites.Aníbal Limón1
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-02-10testimage: Modularize helper functions for get test lists.Aníbal Limón1
Test lists functions can be used in other parts so modularize it and move to oeqa/oetest.py library. Testimage class was updated to meet the new sign of the functions. Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2015-10-19oetest: Add tearDownLocal classMariano Lopez1
In the current state there are some runtime test that don't run the tearDown method fromm oeRuntimeTest class because the tearDown class is overwritten in the child class. This change adds tearDownLocal method in oeRuntimeTest class that will run after tearDown. This method can be overwritten in the child classes to implement specific test functionality. [YOCTO #8465] Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2015-10-01oetest: Change logic of a failed testMariano Lopez1
Currently the logic to check if a test failed was to check for an exception in the thread, but some decorators used in the syslog runtime test would generate and handle exceptions; this will mess with the current check logic and will dump the host and the target as if the test failed. This patch changes the check logic to verify if the test that just happend is in the failure test list and dump the host and target accordingly. [YOCTO #8406] Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-26oeqa/oetest: Fix SDK command executionRichard Purdie1
The SDK environment wasn't being sourced when running these test commands, meaning in some cases the cross compiler was being tested, not the SDK tools. This is clearly not the intent so fix this by ensuring the SDK environment is present. This fixes test failures in multilib SDKs. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-23oeqa/oetest: Remove bb as requirement for oetest.Lucian Musat1
In order for the test export runner to work oetest needs to be separated from bitbake environment. There is no need to use bb import here so we can use a logger instead. Signed-off-by: Lucian Musat <george.l.musat@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-21oeqa/testimage: Enhance -v switch in testimageLucian Musat1
When testimage is run with -v switch now individual test progress can be seen directly in bitbake console. [YOCTO #6841] Signed-off-by: Lucian Musat <george.l.musat@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2015-09-15oeqa/decorators: Added decorator to restart the DUT in case of test hang.Lucian Musat1
Once the DUT is hanged during testing, currently all the following test cases have to wait for default timeout to exit. Using this decorator the user can choose a timeout at case by case basis and what happens when the timeout is reached by overwriting the self.target.restart method. [YOCTO #7853] Signed-off-by: Lucian Musat <george.l.musat@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-07testimage: handle SIGTERM to conclude runqemuMariano Lopez1
In the current state if a SIGTERM is sent to the testimage worker, the worker will exit but runqemu and qemu won't exit and the processes need to be killed manually to free the bitbake lock. This allows to catch the SIGTERM signal in testimage, this way it is possible to stop runqemu and qemu and allow to free the bitbake lock. Also this allows to skip the rest of the tests when running the tests in qemu or real hardware. This also solves minimal breaks in the setup of the runtime test when checking if qemu is alive. [YOCTO #8239] Signed-off-by: Benjamin Esquivel <benjamin.esquivel@linux.intel.com> Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-05oeqa: Test failure/cleanup improvementsRichard Purdie1
Currently, if qemu segfaults, the tests merrily continue trying to execute which takes time for them to timeout and is a bit silly. Worse, no logs about the segfault are shown to the user, its silent! This patch tries to unravel the tangled web of issues and ensures that we: * install a SIGCHLD handler which tells the user qemu exited * check if qemu is running, if it isn't fail the test outright * don't leave processes behind in sshcontrol which would hold bitbake.lock and block shutdown Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-03testimage: filter proper test cases by tagszjh1
If a test case is decorate by oeqa.utils.decorators.tag, this case will by add a tag, testrunner will filter these tags by TEST_SUITES_TAGS [YOCTO #7849] Signed-off-by: zjh <junhuix.zhang@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-01runexported: Make it compatible with host dumpMariano Lopez1
Currently it is not possible to run a exported test, but this patch will allow to use the HosDumper class when running a exported test, otherwise the HostDumper class will break runexpored test. [YOCTO #8118] Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-01qemurunner: Added host dumps when there are errorsMariano Lopez1
This adds an instance of HostDumper to qemurunner, with this instance now is possible to get dumps from the host when there is an error. This adds dump points in the next cases: - runqemu exits before seeing qemu pid - Fail to get qemu process arguments - Not reach login banner before timeout - qemu pid never appears This also modifies the constructors of BaseDumper, HostDumper and TargetDumper, they don't require the datastore anymore, but the feature to replace datastore variables has been lost (never used) [YOCTO #8118] Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-08-30oetest: Fix regresion when testing real hardwareMariano Lopez1
This fix the regresion introduced in commit 9c72c1a5aa0b49d3895bbefee7a264adfcc6f4ca when testing with real hardware. This regression happens when a test in real hardware fails. [YOCTO #8203] Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
2015-08-30dump: Created new classes for dump host and targetMariano Lopez1
It makes sense to separate the dump commands from the oeRuntimeTest class, this way it can be used in all the test context. These are the changes included in this patch: - Created classes: BaseDumper, HostDumper, TargetDumper - Create an instance of HostDumper in imagetest.bbclass and add it to TestContext class, this way any class that have access to the TestContext would be able to dump logs from the host - Create an instance of TargetDumper in QemuTarget class after get the runner, this way it is accessible during the tests. [YOCTO #8118] Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
2015-08-30testimage: Run commands in target and host when test failsMariano Lopez1
This patch modify three files altought two of them are minimal modifications. This version includes the changes proposed by Paul. testimage.bbclass: Create new vars for easy modification of the dump directory and commands to be run on host and target when a test fails TESTIMAGE_DUMP_DIR: Directory to save the dumps testimage_dump_target: Commands to run on target testimage_dump_host: Commands to run on host oetest.py: - Allow to use the vars defined in testimage class - Now able to run commands in the host and dump the results - Fix an issue with the condition where to run the dump commands (Before it run the commands every test after a failure, now it runs the commands only in tests that failed) - Fix the output to stdout [YOCTO #8118] (From OE-Core rev: 26fe645457633f90bb5ddbb12f5f7b9ca4a06cc5) Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3 (git 2.25.1) at 2025-05-04 17:49:16 +0000