oe-selftest: Enable code coverage on unit tests

Enable code coverage through the library 'python coverage'. In case the environment
variable COVERAGE_PROCESS_START is present (one of the requisites for measuring
sub-processes; the second one is including some coverage statements into the
python sitecustomize.py file) it will be taken into account, otherwise it is
exported with value '.coveragerc'. The latter value is a configuration file
(also automatically created) with some default settings. Once tests are
executed, a coverage report is shown on the log and the coverage output data is stored
with name '.coverage.<args>' where '<args>' is the name of the  unit tests executed
or 'all_tests' when running with --run-all-tests. This output data can be latter used
for better reporting using the same tool (coverage).

As briefly indicate before, measuring sub-process implies setting the env variable
COVERAGE_PROCESS_START (done automatically by the oe-selftest code with this patch if
not already set) and creating a sitecustomize.py as explained on [1].
If either one of these is missing, complete coverage will be incomplete.

Current measurements for 'oe-selftest --run-all-tests' indicate that current coverage
is around 42 % taking into account BBLAYERS, bitbake and scripts folders. More details
on [2], indicating the coverage per file/module.

This tasks has been done together with Humberto Ibarra <humberto.ibarra.lopez@linux.intel.com>

[YOCTO #8679]

[1] http://coverage.readthedocs.org/en/latest/subprocess.html
[2] https://bugzilla.yoctoproject.org/attachment.cgi?id=2854

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>

1 files changed, 61 insertions, 0 deletions

diff --git a/scripts/oe-selftest b/scripts/oe-selftest
index 91e2dd2824..9679962ec0 100755
--- a/scripts/oe-selftest
+++ b/scripts/oe-selftest
@@ -30,6 +30,7 @@ import sys
 import unittest
 import logging
 import argparse
+import subprocess
 
 sys.path.insert(0, os.path.dirname(os.path.realpath(__file__)) + '/lib')
 import scriptpath
@@ -70,6 +71,7 @@ def get_args_parser():
     group.add_argument('--run-all-tests', required=False, action="store_true", dest="run_all_tests", default=False, help='Run all (unhidden) tests')
     group.add_argument('--list-modules', required=False, action="store_true", dest="list_modules", default=False, help='List all available test modules.')
     group.add_argument('--list-classes', required=False, action="store_true", dest="list_allclasses", default=False, help='List all available test classes.')
+    parser.add_argument('--coverage', action="store_true", help="Run code coverage when testing")
     return parser
 
 
@@ -197,6 +199,42 @@ def main():
         if not preflight_check():
             return 1
 
+        if args.coverage:
+            try:
+                # check if user can do coverage
+                import coverage
+                log.info("Coverage is enabled")
+            except:
+                log.warn(("python coverage is not installed\n",
+                          "Make sure you are also coverage takes into account sub-process\n",
+                          "More info on https://pypi.python.org/pypi/coverage\n"))
+
+            # In case the user has not set the variable COVERAGE_PROCESS_START,
+            # create a default one and export it. The COVERAGE_PROCESS_START
+            # value indicates where the coverage configuration file resides
+            # More info on https://pypi.python.org/pypi/coverage
+            coverage_process_start = os.environ.get('COVERAGE_PROCESS_START')
+            if not coverage_process_start:
+                builddir = os.environ.get("BUILDDIR")
+                coveragerc = "%s/.coveragerc" % builddir
+                data_file = "%s/.coverage." % builddir
+                data_file += ((args.run_tests and ".".join(args.run_tests)) or
+                                  (args.run_all_tests and ".all_tests") or '')
+                if os.path.isfile(data_file):
+                    os.remove(data_file)
+                with open(coveragerc, 'w') as cps:
+                    cps.write("[run]\n")
+                    cps.write("data_file = %s\n" % data_file)
+                    cps.write("branch = True\n")
+                    # Measure just BBLAYERS, scripts and bitbake folders
+                    cps.write("source = \n")
+                    for layer in get_bb_var('BBLAYERS').split():
+                        cps.write("    %s\n" % layer)
+                    cps.write("    %s\n" % os.path.dirname(os.path.realpath(__file__)))
+                    cps.write("    %s\n" % os.path.join(os.path.dirname(os.path.dirname(os.path.realpath(__file__))),'bitbake'))
+
+                coverage_process_start = os.environ["COVERAGE_PROCESS_START"] = coveragerc
+
         testslist = get_tests(exclusive_modules=(args.run_tests or []), include_hidden=False)
         suite = unittest.TestSuite()
         loader = unittest.TestLoader()
@@ -216,6 +254,29 @@ def main():
         add_include()
         result = runner.run(suite)
         log.info("Finished")
+
+        if args.coverage:
+            with open(coverage_process_start) as ccf:
+                log.info("Coverage configuration file (%s)" % coverage_process_start)
+                log.info("===========================")
+                log.info("\n%s" % "".join(ccf.readlines()))
+
+            try:
+                # depending on the version, coverage command is named 'python-coverage' or 'coverage',
+                # where the latter is for newer versions
+                coverage_cmd = "python-coverage"
+                subprocess.check_call(coverage_cmd, stderr=subprocess.PIPE, shell=True)
+            except subprocess.CalledProcessError:
+                coverage_cmd = "coverage"
+                pass
+
+            log.info("Coverage Report")
+            log.info("===============")
+            p = subprocess.Popen("%s report" % coverage_cmd, shell=True,
+                                 stdout=subprocess.PIPE, stderr=subprocess.STDOUT, stdin=subprocess.PIPE)
+            cov_output, cov_err = p.communicate()
+            log.info("\n%s" % cov_output)
+
         if result.wasSuccessful():
             return 0
         else: