summaryrefslogtreecommitdiff
path: root/scripts/oe-build-perf-test
AgeCommit message (Collapse)AuthorFiles
2017-03-26oe-build-perf-test: pack all buildstat in one fileMarkus Lehtonen1
Write out all buildstats into one big json file, instead of using multiple per-measurement files. Individual buildstats will be indexed using "<test_name>.<measurement_name>" as the key. Also, changes the per-testcase working directories into temporary directories that will be removed after test execution as there are no more per-testcase data files to store permanently. [YOCTO #10582] Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-15oe-build-perf-test: drop --commit-resultsMarkus Lehtonen1
Drop support for committing results into Git repository. The functionality was not directly related to testing and feels unnecessary complication of the script. The functionality has been moved into a separate oe-git-archive script. [YOCTO #10582] Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-01-23oe-build-perf-test: remove unused imports and fix indentMarkus Lehtonen1
[YOCTO #10590] Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-01-23oe-build-perf-test: save test metadata in a separate fileMarkus Lehtonen1
The patch introduces a new metadata (.json or .xml) file in the output directory. All test meta data, e.g. git revision information and tester host information is now stored there. The JSON report format is slightly changed as the metadata is not present in results.json anymore. [YOCTO #10590] Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-01-23oe-build-perf-test: enable xml reportingMarkus Lehtonen1
Add --xml command line option to oe-build-perf-test script for producing a test report in JUnit XML format instead of JSON. [YOCTO #10590] Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-09-30oe-build-perf-test: return 2 if some tests failedMarkus Lehtonen1
Add a new return value '2' that indicates that some tests failed but there were no fatal errors (i.e. configuration mistakes or bugs in the tests themselves). Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-09-06oe-build-perf-test: fix log file pathMarkus Lehtonen1
The --log-file command line argument was slightly broken as {out_dir} string replacement was not working as expected. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03oe-build-perf-test: rename log file and implement --log-fileMarkus Lehtonen1
Rename the (main) log file of the oe-build-perf-test script from 'output.log' to 'oe-build-perf-test.log'. Also, add a new command line option --log-file which makes it possible to use an alternative log file name/path, if needed. Note that the file name/path is relative to the output directory. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03oe-build-perf-test: update globalres and git even if tests failedMarkus Lehtonen1
Write globalres log file and commit results to Git even if some tests failed. Now that tests do not depend on each other there should be no risk of bogus results caused by test failures. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-25oe-build-perf-test: add {git_commit_count} keyword for --commit-results-tagMarkus Lehtonen1
Makes it possible to create easily sortable tags. Also, the default tag format is updated to use the new keyword. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-08-25oe-build-perf-test: new {tag_num} keyword for --commit-results-tagMarkus Lehtonen1
This makes it possible to create numbered tags, where the "basename" of the tag is the same and the only difference is an (automatically) increasing index number. This is useful if you do multiple test runs on the same commit. For example, using: --commit-results-tag {tester_host}/{git_commit}/{tag_num} would give you tags something like: myhost/decb3119dffd3fd38b800bebc1e510f9217a152e/0 myhost/decb3119dffd3fd38b800bebc1e510f9217a152e/1 ... The default tag format is updated to use this new keyword in order to prevent unintentional tag name clashes. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-08-25oe-build-perf-test: tag results committed to GitMarkus Lehtonen1
Create a Git tag when committing results to a Git repository. This patch also implements --commit-results-tag command line option for controlling the tag name. The value is a format string where the following fields may be used: - {git_branch} - target branch being tested - {git_commit} - target commit being tested - {tester_host} - hostname of the tester machine Tagging can be disabled by giving an empty string to --commit-results-tag. The option has no effect if --commit-results is not defined. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-08-25oe-build-perf-test: pre-check Git repo when using --commit-resultsMarkus Lehtonen1
Do a pre-check on the path that is specified with --commit-results before running any tests. The script will create and/or initialize a fresh Git repository if the given directory does not exist or if it is an empty directory. It fails if it finds a non-empty directory that is not a Git repository. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-08-25oe-build-perf-test: implement --commit-results-branchMarkus Lehtonen1
A new command line option for defining the branch where results are commited. The value is actually a format string accepting two field names: - {git_branch} expands to the name of the target branch being tested - {tester_host} expands to the hostname of the tester machine The option has no effect if --commit-results is not used. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-08-25oe-build-perf-test: support committing results data to GitMarkus Lehtonen1
Implement a new command line option '--commit-results' which commits the test results data into a Git repository. The given path must be an existing initialized local Git repository. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-08-25oe-build-perf-test: use absolute paths in cmdline argsMarkus Lehtonen1
This is safer as the current working directory may change. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-08-25oe-build-perf-test: implement --run-tests optionMarkus Lehtonen1
Makes it possible to run only a subset of tests. NOTE: The tests currently have (unwritten) dependencies on each other so use this option with care. Mainly for debugging. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-08-20oe-build-perf-test: align log message format with testrunner outputMarkus Lehtonen1
The previous attempt on this was a bit erroneous, dropping time stamps completely although only the timestamp format should've been changed. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-17oe-build-perf-test: simplify stderr log formatMarkus Lehtonen1
Remove timestamps from the stderr log in order to make the console output more readable, i.e. more in line with the output from unittest runner. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-08-17oe-build-perf-test: set-up file logging as early as possibleMarkus Lehtonen1
So that the log file would not miss any records. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-08-17oe-build-perf-test: suppress logger output when tests are being runMarkus Lehtonen1
Prevent logger from writing to stderr when the tests are being run by the TestRunner. During this time the logger output is only written to the log file. This way the console output from the script is cleaner and not mixed with possible logger records. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-08-17oe-build-perf-test: use new unittest based frameworkMarkus Lehtonen1
Convert scripts/oe-build-perf-test to be compatible with the new Python unittest based buildperf test framework. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-08-17oeqa.buildperf: add BuildPerfTestResult classMarkus Lehtonen1
The new class is derived from unittest.TextTestResult class. It is actually implemented by modifying the old BuildPerfTestRunner class which, in turn, is replaced by a totally new simple implementation derived from unittest.TestRunner. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-07-01oe-build-perf-test: add --out-dir command line argumentMarkus Lehtonen1
The new option defines the output directory for the test result data. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-07-01oe-build-perf-test: enable lockingMarkus Lehtonen1
Makes it possible to guard that multiple tests are not run in parallel. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-07-01oe-build-perf-test: implement --globalres-file optionMarkus Lehtonen1
Using this option the script appends test results into a 'global results file'. A CSV-formatted output of the results. This option is to provide compatibility with the old build-perf-test.sh. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-07-01oe-build-perf-test: enable logging into fileMarkus Lehtonen1
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-07-01oeqa.buildperf: implement BuildPerfTestRunner classMarkus Lehtonen1
The new class is responsible for actually running the tests and processing their results. This commit also adds a decorator function for adding new tests. No automatic test discovery, at least yet. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-07-01oeqa.buildperf: functionality to drop kernel cachesMarkus Lehtonen1
Add a new utility class for dropping Linux kernel caches. It uses sudo and tee to write to the drop_caches file. Checking if the user has the permissions to drop caches (without a password) is done by trying to writing an invalid value to the drop_caches file. This way, we will find if writing (with tee) is possible but not really dropping caches, yet. User can avoid giving the password by adding something like: <user> ALL = NOPASSWD: /usr/bin/tee /proc/sys/vm/drop_caches to the system sudoers file. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-07-01oe-build-perf-test: add pre-run sanity checkMarkus Lehtonen1
The script will be required to be run in an initialized bitbake build environment. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-07-01scripts: introduce oe-build-perf-testMarkus Lehtonen1
Initial wireframe for re-writing build-perf-test.sh in Python. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>