Info Catalog automake-1.16: Custom Test Drivers automake-1.16: Declaring Custom Test Drivers

automake-1.16: Overview of Custom Test Drivers Support

 
 15.3.1 Overview of Custom Test Drivers Support
 ----------------------------------------------
 
 Starting from Automake version 1.12, the parallel test harness allows
 the package authors to use third-party custom test drivers, in case the
 default ones are inadequate for their purposes, or do not support their
 testing protocol of choice.
 
    A custom test driver is expected to properly run the test programs
 passed to it (including the command-line arguments passed to those
 programs, if any), to analyze their execution and outcome, to create the
 ‘.log’ and ‘.trs’ files associated to these test runs, and to display
 the test results on the console.  It is responsibility of the author of
 the test driver to ensure that it implements all the above steps
 meaningfully and correctly; Automake isn’t and can’t be of any help
 here.  On the other hand, the Automake-provided code for testsuite
 summary generation offers support for test drivers allowing several test
 results per test script, if they take care to register such results
 properly (⇒Log files generation and test results recording).
 
    The exact details of how test scripts’ results are to be determined
 and analyzed is left to the individual drivers.  Some drivers might only
 consider the test script exit status (this is done for example by the
 default test driver used by the parallel test harness, described in the
 previous section).  Other drivers might implement more complex and
 advanced test protocols, which might require them to parse and interpret
 the output emitted by the test script they’re running (examples of such
 protocols are TAP and SubUnit).
 
    It’s very important to note that, even when using custom test
 drivers, most of the infrastructure described in the previous section
 about the parallel harness remains in place; this includes:
 
    • list of test scripts defined in ‘TESTS’, and overridable at runtime
      through the redefinition of ‘TESTS’ or ‘TEST_LOGS’;
    • concurrency through the use of ‘make’’s option ‘-j’;
    • per-test ‘.log’ and ‘.trs’ files, and generation of a summary
      ‘.log’ file from them;
    • ‘recheck’ target, ‘RECHECK_LOGS’ variable, and lazy reruns of
      tests;
    • inter-test dependencies;
    • support for ‘check_*’ variables (‘check_PROGRAMS’,
      ‘check_LIBRARIES’, ...);
    • use of ‘VERBOSE’ environment variable to get verbose output on
      testsuite failures;
    • definition and honoring of ‘TESTS_ENVIRONMENT’,
      ‘AM_TESTS_ENVIRONMENT’ and ‘AM_TESTS_FD_REDIRECT’ variables;
    • definition of generic and extension-specific ‘LOG_COMPILER’ and
      ‘LOG_FLAGS’ variables.
 
 On the other hand, the exact semantics of how (and if) testsuite output
 colorization, ‘XFAIL_TESTS’, and hard errors are supported and handled
 is left to the individual test drivers.
 

Info Catalog automake-1.16: Custom Test Drivers automake-1.16: Declaring Custom Test Drivers