<feed xmlns='http://www.w3.org/2005/Atom'>
<title>openembedded-core.git/meta/lib/oeqa/files, branch thud</title>
<subtitle>Mirror of openembedded-core</subtitle>
<link rel='alternate' type='text/html' href='https://git.multitech.net/cgit/openembedded-core.git/'/>
<entry>
<title>resulttool: enable merge, store, report and regression analysis</title>
<updated>2019-03-24T16:52:56+00:00</updated>
<author>
<name>Yeoh Ee Peng</name>
<email>ee.peng.yeoh@intel.com</email>
</author>
<published>2019-02-14T05:50:37+00:00</published>
<link rel='alternate' type='text/html' href='https://git.multitech.net/cgit/openembedded-core.git/commit/?id=39cda3498ec68b91a671ead256231324d74f7d4c'/>
<id>39cda3498ec68b91a671ead256231324d74f7d4c</id>
<content type='text'>
OEQA outputs test results into json files and these files were
archived by Autobuilder during QA releases. Example: each oe-selftest
run by Autobuilder for different host distro generate a
testresults.json file.

These scripts were developed as a test result tools to manage
these testresults.json file.

Using the "store" operation, user can store multiple testresults.json
files as well as the pre-configured directories used to hold those files.

Using the "merge" operation, user can merge multiple testresults.json
files to a target file.

Using the "report" operation, user can view the test result summary
for all available testresults.json files inside a ordinary directory
or a git repository.

Using the "regression-file" operation, user can perform regression
analysis on testresults.json files specified. Using the "regression-dir"
and "regression-git" operations, user can perform regression analysis
on directory and git accordingly.

These resulttool operations expect the testresults.json file to use
the json format below.
{
    "&lt;testresult_1&gt;": {
        "configuration": {
            "&lt;config_name_1&gt;": "&lt;config_value_1&gt;",
            "&lt;config_name_2&gt;": "&lt;config_value_2&gt;",
            ...
            "&lt;config_name_n&gt;": "&lt;config_value_n&gt;",
        },
        "result": {
            "&lt;testcase_namespace_1&gt;": {
                "status": "&lt;PASSED or FAILED or ERROR or SKIPPED&gt;",
                "log": "&lt;failure or error logging&gt;"
            },
            "&lt;testcase_namespace_2&gt;": {
                "status": "&lt;PASSED or FAILED or ERROR or SKIPPED&gt;",
                "log": "&lt;failure or error logging&gt;"
            },
            ...
            "&lt;testcase_namespace_n&gt;": {
                "status": "&lt;PASSED or FAILED or ERROR or SKIPPED&gt;",
                "log": "&lt;failure or error logging&gt;"
            },
        }
    },
    ...
    "&lt;testresult_n&gt;": {
        "configuration": {
            "&lt;config_name_1&gt;": "&lt;config_value_1&gt;",
            "&lt;config_name_2&gt;": "&lt;config_value_2&gt;",
            ...
            "&lt;config_name_n&gt;": "&lt;config_value_n&gt;",
        },
        "result": {
            "&lt;testcase_namespace_1&gt;": {
                "status": "&lt;PASSED or FAILED or ERROR or SKIPPED&gt;",
                "log": "&lt;failure or error logging&gt;"
            },
            "&lt;testcase_namespace_2&gt;": {
                "status": "&lt;PASSED or FAILED or ERROR or SKIPPED&gt;",
                "log": "&lt;failure or error logging&gt;"
            },
            ...
            "&lt;testcase_namespace_n&gt;": {
                "status": "&lt;PASSED or FAILED or ERROR or SKIPPED&gt;",
                "log": "&lt;failure or error logging&gt;"
            },
        }
    },
}

To use these scripts, first source oe environment, then run the
entry point script to look for help.
    $ resulttool

To store test result from oeqa automated tests, execute the below
    $ resulttool store &lt;source_dir&gt; &lt;git_branch&gt;

To merge multiple testresults.json files, execute the below
    $ resulttool merge &lt;base_result_file&gt; &lt;target_result_file&gt;

To report test report, execute the below
    $ resulttool report &lt;source_dir&gt;

To perform regression file analysis, execute the below
    $ resulttool regression-file &lt;base_result_file&gt; &lt;target_result_file&gt;

To perform regression dir analysis, execute the below
    $ resulttool regression-dir &lt;base_result_dir&gt; &lt;target_result_dir&gt;

To perform regression git analysis, execute the below
    $ resulttool regression-git &lt;source_dir&gt; &lt;base_branch&gt; &lt;target_branch&gt;

[YOCTO# 13012]
[YOCTO# 12654]

Signed-off-by: Yeoh Ee Peng &lt;ee.peng.yeoh@intel.com&gt;
Signed-off-by: Richard Purdie &lt;richard.purdie@linuxfoundation.org&gt;
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
OEQA outputs test results into json files and these files were
archived by Autobuilder during QA releases. Example: each oe-selftest
run by Autobuilder for different host distro generate a
testresults.json file.

These scripts were developed as a test result tools to manage
these testresults.json file.

Using the "store" operation, user can store multiple testresults.json
files as well as the pre-configured directories used to hold those files.

Using the "merge" operation, user can merge multiple testresults.json
files to a target file.

Using the "report" operation, user can view the test result summary
for all available testresults.json files inside a ordinary directory
or a git repository.

Using the "regression-file" operation, user can perform regression
analysis on testresults.json files specified. Using the "regression-dir"
and "regression-git" operations, user can perform regression analysis
on directory and git accordingly.

These resulttool operations expect the testresults.json file to use
the json format below.
{
    "&lt;testresult_1&gt;": {
        "configuration": {
            "&lt;config_name_1&gt;": "&lt;config_value_1&gt;",
            "&lt;config_name_2&gt;": "&lt;config_value_2&gt;",
            ...
            "&lt;config_name_n&gt;": "&lt;config_value_n&gt;",
        },
        "result": {
            "&lt;testcase_namespace_1&gt;": {
                "status": "&lt;PASSED or FAILED or ERROR or SKIPPED&gt;",
                "log": "&lt;failure or error logging&gt;"
            },
            "&lt;testcase_namespace_2&gt;": {
                "status": "&lt;PASSED or FAILED or ERROR or SKIPPED&gt;",
                "log": "&lt;failure or error logging&gt;"
            },
            ...
            "&lt;testcase_namespace_n&gt;": {
                "status": "&lt;PASSED or FAILED or ERROR or SKIPPED&gt;",
                "log": "&lt;failure or error logging&gt;"
            },
        }
    },
    ...
    "&lt;testresult_n&gt;": {
        "configuration": {
            "&lt;config_name_1&gt;": "&lt;config_value_1&gt;",
            "&lt;config_name_2&gt;": "&lt;config_value_2&gt;",
            ...
            "&lt;config_name_n&gt;": "&lt;config_value_n&gt;",
        },
        "result": {
            "&lt;testcase_namespace_1&gt;": {
                "status": "&lt;PASSED or FAILED or ERROR or SKIPPED&gt;",
                "log": "&lt;failure or error logging&gt;"
            },
            "&lt;testcase_namespace_2&gt;": {
                "status": "&lt;PASSED or FAILED or ERROR or SKIPPED&gt;",
                "log": "&lt;failure or error logging&gt;"
            },
            ...
            "&lt;testcase_namespace_n&gt;": {
                "status": "&lt;PASSED or FAILED or ERROR or SKIPPED&gt;",
                "log": "&lt;failure or error logging&gt;"
            },
        }
    },
}

To use these scripts, first source oe environment, then run the
entry point script to look for help.
    $ resulttool

To store test result from oeqa automated tests, execute the below
    $ resulttool store &lt;source_dir&gt; &lt;git_branch&gt;

To merge multiple testresults.json files, execute the below
    $ resulttool merge &lt;base_result_file&gt; &lt;target_result_file&gt;

To report test report, execute the below
    $ resulttool report &lt;source_dir&gt;

To perform regression file analysis, execute the below
    $ resulttool regression-file &lt;base_result_file&gt; &lt;target_result_file&gt;

To perform regression dir analysis, execute the below
    $ resulttool regression-dir &lt;base_result_dir&gt; &lt;target_result_dir&gt;

To perform regression git analysis, execute the below
    $ resulttool regression-git &lt;source_dir&gt; &lt;base_branch&gt; &lt;target_branch&gt;

[YOCTO# 13012]
[YOCTO# 12654]

Signed-off-by: Yeoh Ee Peng &lt;ee.peng.yeoh@intel.com&gt;
Signed-off-by: Richard Purdie &lt;richard.purdie@linuxfoundation.org&gt;
</pre>
</div>
</content>
</entry>
<entry>
<title>oeqa: rationalise Perl tests</title>
<updated>2018-07-26T12:16:31+00:00</updated>
<author>
<name>Ross Burton</name>
<email>ross.burton@intel.com</email>
</author>
<published>2018-07-19T16:12:40+00:00</published>
<link rel='alternate' type='text/html' href='https://git.multitech.net/cgit/openembedded-core.git/commit/?id=c6eef46747fe58bb2310be4f06d2fa9b67901d72'/>
<id>c6eef46747fe58bb2310be4f06d2fa9b67901d72</id>
<content type='text'>
As with the Python test, this can be both better and faster.  No need to copy a
file, just run a one-liner.

Signed-off-by: Ross Burton &lt;ross.burton@intel.com&gt;
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
As with the Python test, this can be both better and faster.  No need to copy a
file, just run a one-liner.

Signed-off-by: Ross Burton &lt;ross.burton@intel.com&gt;
</pre>
</div>
</content>
</entry>
<entry>
<title>oeqa/sdk/python: clean up Python test</title>
<updated>2018-07-18T09:13:30+00:00</updated>
<author>
<name>Ross Burton</name>
<email>ross.burton@intel.com</email>
</author>
<published>2018-07-16T12:54:42+00:00</published>
<link rel='alternate' type='text/html' href='https://git.multitech.net/cgit/openembedded-core.git/commit/?id=bead742a3ffc0a53162fb0c36610d74a1422e7b3'/>
<id>bead742a3ffc0a53162fb0c36610d74a1422e7b3</id>
<content type='text'>
For the same reasons as the runtime Python test, clean up the SDK test.

Also port from Python 2 to Python 3, as that's what is supported now.

Signed-off-by: Ross Burton &lt;ross.burton@intel.com&gt;
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
For the same reasons as the runtime Python test, clean up the SDK test.

Also port from Python 2 to Python 3, as that's what is supported now.

Signed-off-by: Ross Burton &lt;ross.burton@intel.com&gt;
</pre>
</div>
</content>
</entry>
<entry>
<title>oeqa: Move common files to oeqa/files instead of runtime only</title>
<updated>2017-01-23T12:03:54+00:00</updated>
<author>
<name>Aníbal Limón</name>
<email>anibal.limon@linux.intel.com</email>
</author>
<published>2016-11-09T20:57:40+00:00</published>
<link rel='alternate' type='text/html' href='https://git.multitech.net/cgit/openembedded-core.git/commit/?id=ec73e8a3d3149f3866b7bfc06f169c6e05e2d338'/>
<id>ec73e8a3d3149f3866b7bfc06f169c6e05e2d338</id>
<content type='text'>
Those files are used by runtime and sdk test cases, so move to
base directory of oeqa module.

[YOCTO #10599]

Signed-off-by: Aníbal Limón &lt;anibal.limon@linux.intel.com&gt;
Signed-off-by: Mariano Lopez &lt;mariano.lopez@linux.intel.com&gt;
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Those files are used by runtime and sdk test cases, so move to
base directory of oeqa module.

[YOCTO #10599]

Signed-off-by: Aníbal Limón &lt;anibal.limon@linux.intel.com&gt;
Signed-off-by: Mariano Lopez &lt;mariano.lopez@linux.intel.com&gt;
</pre>
</div>
</content>
</entry>
</feed>
