summaryrefslogtreecommitdiff
path: root/meta/lib/oeqa/buildperf
AgeCommit message (Collapse)AuthorFiles
2017-03-26oeqa.buildperf: limit the length of error outputMarkus Lehtonen1
Limit the length of error logs to 40 lines. We don't need to show/archive thousands of lines of bitbake logs if an error occurs. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-26oe-build-perf-test: sum rusage in buildstatsMarkus Lehtonen1
Instead of separate rusage and child rusage values, only store their sum value in buildstats. This is a big reduction in data footprint without really losing any interesting data. Also, utilize OrderedDict to order data more logically. [YOCTO #10582] Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-26oe-build-perf-test: pack all buildstat in one fileMarkus Lehtonen2
Write out all buildstats into one big json file, instead of using multiple per-measurement files. Individual buildstats will be indexed using "<test_name>.<measurement_name>" as the key. Also, changes the per-testcase working directories into temporary directories that will be removed after test execution as there are no more per-testcase data files to store permanently. [YOCTO #10582] Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-15oeqa/buildperf: don't archive stdout/stderr of commandsMarkus Lehtonen2
Stop capturing output of the shell commands into <test>/commands.log. Redirecting output into a file prevented the unittest framework from capturing it, causing useless errors (with empty output) like: oeqa.utils.CommandError: Command '['bitbake', 'core-image-sato']' returned non-zero exit status 1 with output: In general, the console output of commands is only interesting when something fails. Also, dropping the commands.log file is a huge saving in disk space, and thus, repository size when results are archived in Git. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-02-05oeqa.buildperf: reword test descriptionsMarkus Lehtonen1
In an attempt to make them more compact and coherent. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-01-23oeqa.buildperf: store measurements as a dict (object) in the JSON reportMarkus Lehtonen1
Store measurements as a dict, instead of an array, in the JSON report. This change makes traversing of the report much easier. The change also disallows identically named measurements under one test, as a sanity check for the test cases. [YOCTO #10590] Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-01-23oeqa.buildperf: change sorting in json reportMarkus Lehtonen1
Use OrderedDict() instead of sort_keys=True (of json.dump()). Makes for more logical sorting of the values in the report. [YOCTO #10590] Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-01-23oe-build-perf-test: remove unused imports and fix indentMarkus Lehtonen1
[YOCTO #10590] Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-01-23oe-build-perf-test: save test metadata in a separate fileMarkus Lehtonen1
The patch introduces a new metadata (.json or .xml) file in the output directory. All test meta data, e.g. git revision information and tester host information is now stored there. The JSON report format is slightly changed as the metadata is not present in results.json anymore. [YOCTO #10590] Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-01-23oeqa.buildperf: report results in chronological orderMarkus Lehtonen1
Write results in the report file in chronological order, instead of random order dependent on test statuses. [YOCTO #10590] Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-01-23oeqa.buildperf: extend xml report format with test descriptionMarkus Lehtonen1
Add test description as an attribute to the <testcase> element. [YOCTO #10590] Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-01-23oeqa.buildperf: extend xml format to contain measurement dataMarkus Lehtonen1
Make the xml report format slightly non-standard by incorporating measurement data into it. [YOCTO #10590] Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-01-23oe-build-perf-test: enable xml reportingMarkus Lehtonen1
Add --xml command line option to oe-build-perf-test script for producing a test report in JUnit XML format instead of JSON. [YOCTO #10590] Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-01-23oeqa.buildperf: include error details in json reportMarkus Lehtonen1
This will typically mean assert message and exception type plus a traceback. In case of skipped tests the reason (i.e. skip message) is included. [YOCTO #10590] Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-01-23oeqa.buildperf: sync test status names with JUnitMarkus Lehtonen1
Use 'failure' instead of 'fail'. Also, use 'expected' instead of 'exp'. [YOCTO #10590] Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-01-23oeqa.buildperf: prevent a crash on unexpected successMarkus Lehtonen1
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-10-05oeqa.buildperf: measure apparent size instead of real disk usageMarkus Lehtonen2
This change aligns disk usage measurements of the eSDK test with the old build-perf-test.sh script. And thus, also makes the results between the old and the new script comparable. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-13oeqa.buildperf: another fix for splitting 'nevr' stringMarkus Lehtonen1
When processing buildstats we determine recipe name, epoch, version and revision from the per-recipe buildstat directory name. One previous patch made an assumption that package version starts with a number. That might not be true because of a packaging mistake or whatever reason. Thus, if a version starting with a number is not found, fall back to the "old" method of just taking the second-last dash-delimited part (the one before revision). Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-09-08oeqa.buildperf: be sure to use the latest buildstatsMarkus Lehtonen1
Be sure to take the latest buildstats if multiple buildstats are found. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-09-08oeqa.buildperf: try harder when splitting 'nevr' stringMarkus Lehtonen1
Try to be more intelligent when splitting out recipe name, epoch, version and revision from the buildstat directory name. Previous assumption was that package versions never contain a dash but obviously that is not necessarily true. The new assumption is that the package version starts with a number. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-09-06oeqa.buildperf: correct globalres time formatMarkus Lehtonen1
Always use two digits for (integer part of) seconds, i.e. show '1:02.34' instead of '1:2.34'. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03oeqa.buildperf: include commands log file name in results.jsonMarkus Lehtonen1
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03oeqa.buildperf: include buildstats file name in results.jsonMarkus Lehtonen2
No need to do lsdir magic for finding buildstats when reading results. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03oeqa.buildperf: show skipped tests in results, tooMarkus Lehtonen1
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03oeqa.buildperf: convert buildstats into json formatMarkus Lehtonen1
Instead of archiving buildstats in raw text file format convert all buildstats into one json-formatted file. Some redundant information, i.e. 'Event:', 'utime:', 'stime:', 'cutime:' and 'cstime:' fields, are dropped. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03oeqa.buildperf: measure io statMarkus Lehtonen1
Add data from /proc/<pid>/io to system resource measurements. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03oeqa.buildperf: don't use Gnu timeMarkus Lehtonen1
Use Python standard library functionality instead of the time utility for measuring elapsed (wall clock) time of commands. The time.* log files are also ditched. However, the same detailed resource usage data, previously found in time.* logs is now provided in results.json file. This data is collected through the resource module of Python. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03oeqa.buildperf: rename buildstats directoriesMarkus Lehtonen1
Change directory name from 'buildstats-<test_name>' to just 'buildstats'. However, this patch adds the possibility to label buildstats directory name with a postfix which makes it possible to save multiple buildstats per test, for example. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03oeqa.buildperf: separate output dir for each testMarkus Lehtonen1
Store the output data of each test in an individual subdirectory instead of storing everything in the root output directory. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03oeqa.buildperf: strip date from buildstats directory pathMarkus Lehtonen1
Archive buildstats in a directory like 'buildstats' instead of something like 'buildstats/20160513120000'. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03oeqa.buildperf: enable json-formatted resultsMarkus Lehtonen1
Automatically create a json.formatted file (results.json) in the results directory that contains results from all tests. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03oeqa.buildperf: add 'product' to test result dataMarkus Lehtonen1
This defaults to 'oe-core' but can be defined using the OE_BUILDPERF_PRODUCT environment variable. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03oeqa.buildperf: treat failed measurements as errorsMarkus Lehtonen1
Now failed measurements correctly cause a test failure (recorded as an error). There should be no need to continue the test if one step fails, especially now that the tests don't depend on each other. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03oeqa.buildperf: make tests independentMarkus Lehtonen2
Add test set-up functionality so that the individual tests do not depend on each other. This should make sure that a failure in one test does not affect the results of another test. The patch also makes it reasonable to run only a subset of the tests by using the --run-tests option. The increase in total execution time of the full suite - caused by the additional set-up steps - is insignificant because normally no additional tasks need to be run. The previous test has already done all set-up work. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03oeqa.buildperf: fix checking of invalid resultsMarkus Lehtonen1
The test status check done when writing globalres log was incorrect. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-25oe-build-perf-test: add {git_commit_count} keyword for --commit-results-tagMarkus Lehtonen1
Makes it possible to create easily sortable tags. Also, the default tag format is updated to use the new keyword. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-08-25oeqa.buildperf: add git commit count to result dataMarkus Lehtonen1
This number represents the number of commits since the beginning of git history until the tested revision. This helps e.g. in ordering results. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-08-25oe-build-perf-test: new {tag_num} keyword for --commit-results-tagMarkus Lehtonen1
This makes it possible to create numbered tags, where the "basename" of the tag is the same and the only difference is an (automatically) increasing index number. This is useful if you do multiple test runs on the same commit. For example, using: --commit-results-tag {tester_host}/{git_commit}/{tag_num} would give you tags something like: myhost/decb3119dffd3fd38b800bebc1e510f9217a152e/0 myhost/decb3119dffd3fd38b800bebc1e510f9217a152e/1 ... The default tag format is updated to use this new keyword in order to prevent unintentional tag name clashes. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-08-25oe-build-perf-test: tag results committed to GitMarkus Lehtonen1
Create a Git tag when committing results to a Git repository. This patch also implements --commit-results-tag command line option for controlling the tag name. The value is a format string where the following fields may be used: - {git_branch} - target branch being tested - {git_commit} - target commit being tested - {tester_host} - hostname of the tester machine Tagging can be disabled by giving an empty string to --commit-results-tag. The option has no effect if --commit-results is not defined. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-08-25oe-build-perf-test: implement --commit-results-branchMarkus Lehtonen1
A new command line option for defining the branch where results are commited. The value is actually a format string accepting two field names: - {git_branch} expands to the name of the target branch being tested - {tester_host} expands to the hostname of the tester machine The option has no effect if --commit-results is not used. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-08-25oe-build-perf-test: support committing results data to GitMarkus Lehtonen1
Implement a new command line option '--commit-results' which commits the test results data into a Git repository. The given path must be an existing initialized local Git repository. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-08-25oeqa.buildperf: use term commit instead of revisionMarkus Lehtonen1
This is basically a internal change, at this point. Term 'commit' better represents the data we actually have. Term 'revision' is more vague and could be understood to point to a tag object, for example. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-08-25oeqa.utils.git: implement GitRepo.get_current_branch()Markus Lehtonen1
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-08-25oeqa.utils.git: introduce GitRepo.rev_parse()Markus Lehtonen1
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-08-20oeqa.buildperf: fix crash when creating globalres.logMarkus Lehtonen1
Fix a bug that was introduced when converting to unittest framework. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-17oeqa.buildperf: use oe.path.remove()Markus Lehtonen2
Drop the self-baked force_rm() method. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-08-17oeqa.buildperf: be more verbose about failed commandsMarkus Lehtonen1
Log failures of commands whose output is stored. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-08-17oeqa.buildperf: introduce runCmd2()Markus Lehtonen2
Special runCmd() for build perf tests which doesn't raise an AssertionError when the command fails. This causes command failures to be detected as test errors instead of test failures. This way "failed" state of tests is reserved for future making it possible to set e.g. thresholds for certain measurement results. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-08-17oeqa.buildperf: convert test cases to unittestMarkus Lehtonen1
This commit converts the actual tests to be compatible with the new Python unittest based framework. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-08-17oeqa.buildperf: add BuildPerfTestResult classMarkus Lehtonen2
The new class is derived from unittest.TextTestResult class. It is actually implemented by modifying the old BuildPerfTestRunner class which, in turn, is replaced by a totally new simple implementation derived from unittest.TestRunner. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
generated by cgit v1.2.3 (git 2.25.1) at 2025-02-13 21:08:04 +0000