automake-1.16: Log files generation and test results recording
15.3.3.2 Log files generation and test results recording
........................................................
The test driver must correctly generate the files specified by the
‘--log-file’ and ‘--trs-file’ option (even when the tested program fails
or crashes).
The ‘.log’ file should ideally contain all the output produced by the
tested program, plus optionally other information that might facilitate
debugging or analysis of bug reports. Apart from that, its format is
basically free.
The ‘.trs’ file is used to register some metadata through the use of
custom reStructuredText fields. This metadata is expected to be
employed in various ways by the parallel test harness; for example, to
count the test results when printing the testsuite summary, or to decide
which tests to re-run upon ‘make recheck’. Unrecognized metadata in a
‘.trs’ file is currently ignored by the harness, but this might change
in the future. The list of currently recognized metadata follows.
‘:test-result:’
The test driver must use this field to register the results of
_each_ test case run by a test script file. Several
‘:test-result:’ fields can be present in the same ‘.trs’ file; this
is done in order to support test protocols that allow a single test
script to run more test cases.
The only recognized test results are currently ‘PASS’, ‘XFAIL’,
‘SKIP’, ‘FAIL’, ‘XPASS’ and ‘ERROR’. These results, when declared
with ‘:test-result:’, can be optionally followed by text holding
the name and/or a brief description of the corresponding test; the
harness will ignore such extra text when generating
‘test-suite.log’ and preparing the testsuite summary.
‘:recheck:’
If this field is present and defined to ‘no’, then the
corresponding test script will _not_ be run upon a ‘make recheck’.
What happens when two or more ‘:recheck:’ fields are present in the
same ‘.trs’ file is undefined behaviour.
‘:copy-in-global-log:’
If this field is present and defined to ‘no’, then the content of
the ‘.log’ file will _not_ be copied into the global
‘test-suite.log’. We allow to forsake such copying because, while
it can be useful in debugging and analysis of bug report, it can
also be just a waste of space in normal situations, e.g., when a
test script is successful. What happens when two or more
‘:copy-in-global-log:’ fields are present in the same ‘.trs’ file
is undefined behaviour.
‘:test-global-result:’
This is used to declare the "global result" of the script.
Currently, the value of this field is needed only to be reported
(more or less verbatim) in the generated global log file
‘$(TEST_SUITE_LOG)’, so it’s quite free-form. For example, a test
script which runs 10 test cases, 6 of which pass and 4 of which are
skipped, could reasonably have a ‘PASS/SKIP’ value for this field,
while a test script which runs 19 successful tests and one failed
test could have an ‘ALMOST PASSED’ value. What happens when two or
more ‘:test-global-result:’ fields are present in the same ‘.trs’
file is undefined behaviour.
Let’s see a small example. Assume a ‘.trs’ file contains the following
lines:
:test-result: PASS server starts
:global-log-copy: no
:test-result: PASS HTTP/1.1 request
:test-result: FAIL HTTP/1.0 request
:recheck: yes
:test-result: SKIP HTTPS request (TLS library wasn't available)
:test-result: PASS server stops
Then the corresponding test script will be re-run by ‘make check’, will
contribute with _five_ test results to the testsuite summary (three of
these tests being successful, one failed, and one skipped), and the
content of the corresponding ‘.log’ file will _not_ be copied into the
global log file ‘test-suite.log’.
Info Catalog
automake-1.16: Command-line arguments for test drivers
automake-1.16: API for Custom Test Drivers
automake-1.16: Testsuite progress output