openembedded-core.git/meta/lib/oeqa/core, branch thud

oeqa/target/ssh: Replace suggogatepass with ignoring errors

2019-05-18T04:31:01+00:00

We continued to see encoding problems with ssh commands run in oeqa. After much
research the conclusion was we should use ignore the errors since some occasional bad
locale encoding is better than the unicode decoding issues we were seeing which crashed
large parts of tests.

Signed-off-by: Richard Purdie 
Signed-off-by: Armin Kuster

oeqa/concurrenttest: Patch subunit module to handle classSetup failures

2019-05-18T04:30:39+00:00

Currently setupClass errors were not being mapped back to the failing tests
and they were hence being marked as UNKNOWN and the test statistics were
inaccurate.

This is because whilst the errors were being encoded into the test results
stream, the decoder doesn't cope with an error outside a testStart event.

We patch in an addError handler to the outsideTest parser so that this
does get handled in a way similar to the non-concurrent case.

It would be nice if we didn't have to do this but there doesn't seem
to be any other way to fix this other than forking subunit.

We also make a minor change so another of our changes can cope with
tests without a start time.

Signed-off-by: Richard Purdie 
Signed-off-by: Armin Kuster

oeqa/runner: Fix subunit setupClass/setupModule failure handling

2019-05-18T04:30:26+00:00

The string format for subunit setupClass/setupModule failures is slightly
different, tweak the regex to correctly handle both cases.

Signed-off-by: Richard Purdie 
Signed-off-by: Armin Kuster

oeqa/core/runner: Handle unexpectedSucesses

2019-05-18T04:30:06+00:00

Instead of showing:
RESULTS - ptest.PtestRunnerTest.test_ptestrunner - Testcase 1600: UNKNOWN (32.30s)

map unexpectedSuccesses to PASSED and improve the way they're displayed. We
expect/allow ptest runner to fail but if it passes we should handle it correctly.

Signed-off-by: Richard Purdie 
Signed-off-by: Armin Kuster

oeqa/ssh: Avoid unicode decode exceptions

2019-05-18T04:19:25+00:00

This code really needs to be rewritten to not split potential
multibyte characters, for now work around it to avoid exceptions like:

  File "/home/pokybuild/yocto-worker/qa-extras2/build/meta/lib/oeqa/core/target/ssh.py", line 211, in run
    data = reader.read(1024, 4096)
  File "/usr/lib64/python3.6/codecs.py", line 503, in read
    newchars, decodedbytes = self.decode(data, self.errors)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 0: invalid start byte

Signed-off-by: Richard Purdie 
Signed-off-by: Armin Kuster

oeqa/core/runner: dump stdout and stderr of each test case

2019-05-18T04:19:10+00:00

Some CI pipelines might perform further processing of the test output
(for instance, to plot some metrics into a chart). However, Since `thud`
we switched away from the XML-based jUnit reporting, and at the same
time we lost the ability of collecting the stdout and stderr of the
various tests.

We now restore this functionality by adding `stdout` and `stderr` keys
to the JSON reports. This behavior is off by default; in order to enable
it, one must set the `TESTREPORT_FULLLOGS` variable in the bitbake
configuration.

Signed-off-by: Alberto Mardegan 
Signed-off-by: Richard Purdie 
Signed-off-by: Armin Kuster

oeqa/qemu & runtime: qemu do not need ip input from external

2019-01-28T16:56:32+00:00

Qemu do not use the ip input from external. It will
retrieve ip from QemuRunner instance and assign
ip value.

(From OE-Core rev: 14d99dc6c39c963ba3e0d9a30274846bd5369210)

Signed-off-by: Yeoh Ee Peng 
Signed-off-by: Richard Purdie 
Signed-off-by: Armin Kuster

oeqa/concurrencytest: Avoid unclosed file warnings

2018-12-06T10:22:24+00:00

Avoid an unclosed file per thread warning when running selftests concurrently
by closing the result stream.

(From OE-Core rev: 33a4a076e8aa72a872807332501e7f5ae1cee0e2)

Signed-off-by: Richard Purdie

oeqa/core/runner: Correctly markup regexs

2018-12-06T10:21:31+00:00

Avoid the warning "DeprecationWarning: invalid escape sequence \(" by marking
the regexs correctly.

(From OE-Core rev: cb49980fa4a158d5529902df731dec61a8c9b3d4)

Signed-off-by: Richard Purdie

oeqa/loader: Fix deprecation warning

2018-12-06T10:21:20+00:00

Clean up the warning:
meta/lib/oeqa/core/loader.py:27: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
  _failed_test_args = inspect.getargspec(unittest.loader._make_failed_test).args

(From OE-Core rev: d2deb66830be2d44532fea3d5db763b57778252a)

Signed-off-by: Richard Purdie