commit - 12d156c595a1f9ba542596d76b57b71726cc7fe9
commit + fcb7466a5ce845e0ca67043a78ba4dd6df61fbac
blob - 3d574c64ef90eaee44add774faa8de4e357fe7ca
blob + 2af5617838c1ec5531c634697a93a6c1b98a3460
--- README
+++ README
exercise - simple but flexible test runner
SYNOPSIS
- exercise [path]
+ exercise [-q] [-p k=pattern] [path ...]
DESCRIPTION
The exercise utility searches for and executes software (unit) tests in
If given a directory argument, exercise searches for tests therein. If
given a file argument, it executes it as a single test (or test suite).
+ To be recognized as tests, files must match the pattern *.t. To be run
+ as test suites (see below), files must match the pattern *.tests.
- Tests are represented as executable files that are expected to exit 0 on
- success. Any output to standard error and non-zero exits are interpreted
- as negative results (test failures). Standard output is ignored. As a
- special case, exit status 127 is interpreted as an error during the test,
- implying an inconclusive result.
+ The options are as follows:
- To be searched for tests, subdirectories must match the pattern *tests.
- To be recognized as tests, files must match the pattern *.test. To be
- run as test suites (see below), files must match the pattern *.testsuite.
+ -p d=pattern
+ When searching for tests, only descend into directories that
+ match the given pattern. Defaults to `*' (all directories).
+ -p s=pattern
+ Process (only) files matching the given pattern as test suites.
+ Defaults to `*.tests'.
+
+ -p t=pattern
+ Process (only) files matching the given pattern as tests.
+ Defaults to `*.t'.
+
+ -q Do not print status counters to standard output.
+
+ Test Files
+ Tests are represented simply as executable files that are expected to
+ exit 0 on success. Non-zero exits as well as any output to standard
+ error are interpreted as negative results (failure). Standard output is
+ ignored.
+
+ As a special case, exit status 127 is interpreted as an error that
+ precluded the proper execution of the test, implying an inconclusive
+ result.
+
Test Suites
Multiple tests can be combined into one executable. A test suite should
pass error messages from individual tests unchanged to its standard
- error. Its exit status should indicate whether all tests passed. A test
- suite should only exit 127 if there was an error executing the suite
- itself. Errors within individual tests should be reported normally.
+ error. Its exit status should indicate whether all tests passed.
- The standard output from a "test suite" executable should consist of
- lines of the form "run N ok N fail N error N", representing running
+ A test suite should only exit 127 if there was a fatal issue in executing
+ the suite itself. Errors among the individual tests should be reported
+ through the status counters as detailed below.
+
+ The standard output from a test suite executable should consist of lines
+ of the form `total o run n ok m fail l error k', representing running
counts as printed by exercise itself. Consequently, the exercise utility
can be used as a (nested) test suite runner if desired.
+ After running a test suite, its final report is checked for consistency.
+ If any of the conditions (listed below) are not satisfied, the entire
+ test suite is discarded and counted as a single error, as if it had
+ exited with status 127.
+
+ Status Counters
+ Unless given the -q option, exercise prints the following running
+ counters to its standard output:
+
+ total The number of tests that are planned to execute. This is first
+ set to the number of test files found, including test suites.
+ The latter are initially counted as single tests because the
+ number of tests they contain is not known beforehand. The total
+ counter will increase at run time as test suites report their
+ totals.
+
+ run The number of tests that have been started so far. This is
+ normally the sum of ok, fail, and error plus any tests that are
+ currently in progress. Test suites may increment this counter
+ before or after a test finishes, so it may or may not include all
+ tests in progress.
+
+ ok The number of tests that have passed successfully, i.e. exited 0
+ without any output to standard error.
+
+ fail The number of tests that have yielded a negative result, i.e.
+ exited with a non-zero status (except 127) or printed anything to
+ standard error.
+
+ error The number of tests that could not be executed despite the
+ attempt or that exited with the special status 127.
+
+ The following conditions should hold at the end of a run:
+
+ 1. run <= total
+
+ 2. run = ok + fail + error
+
+ The second condition is relaxed to an inequality at run time to account
+ for tests in progress, as explained above. The run count is permitted to
+ ultimately remain short of total as the result of test suites or exercise
+ terminating early.
+
DEPENDENCIES
The exercise utility requires Tcl 8.4 or above.
AUTHORS
Sven M. Hallberg <pesco@khjk.org>
- May 11, 2025
+ May 25, 2025
