commit fcb7466a5ce845e0ca67043a78ba4dd6df61fbac from: Sven M. Hallberg date: Sun May 25 14:56:51 2025 UTC regen README commit - 12d156c595a1f9ba542596d76b57b71726cc7fe9 commit + fcb7466a5ce845e0ca67043a78ba4dd6df61fbac blob - 3d574c64ef90eaee44add774faa8de4e357fe7ca blob + 2af5617838c1ec5531c634697a93a6c1b98a3460 --- README +++ README @@ -4,7 +4,7 @@ NAME exercise - simple but flexible test runner SYNOPSIS - exercise [path] + exercise [-q] [-p k=pattern] [path ...] DESCRIPTION The exercise utility searches for and executes software (unit) tests in @@ -13,29 +13,92 @@ DESCRIPTION If given a directory argument, exercise searches for tests therein. If given a file argument, it executes it as a single test (or test suite). + To be recognized as tests, files must match the pattern *.t. To be run + as test suites (see below), files must match the pattern *.tests. - Tests are represented as executable files that are expected to exit 0 on - success. Any output to standard error and non-zero exits are interpreted - as negative results (test failures). Standard output is ignored. As a - special case, exit status 127 is interpreted as an error during the test, - implying an inconclusive result. + The options are as follows: - To be searched for tests, subdirectories must match the pattern *tests. - To be recognized as tests, files must match the pattern *.test. To be - run as test suites (see below), files must match the pattern *.testsuite. + -p d=pattern + When searching for tests, only descend into directories that + match the given pattern. Defaults to `*' (all directories). + -p s=pattern + Process (only) files matching the given pattern as test suites. + Defaults to `*.tests'. + + -p t=pattern + Process (only) files matching the given pattern as tests. + Defaults to `*.t'. + + -q Do not print status counters to standard output. + + Test Files + Tests are represented simply as executable files that are expected to + exit 0 on success. Non-zero exits as well as any output to standard + error are interpreted as negative results (failure). Standard output is + ignored. + + As a special case, exit status 127 is interpreted as an error that + precluded the proper execution of the test, implying an inconclusive + result. + Test Suites Multiple tests can be combined into one executable. A test suite should pass error messages from individual tests unchanged to its standard - error. Its exit status should indicate whether all tests passed. A test - suite should only exit 127 if there was an error executing the suite - itself. Errors within individual tests should be reported normally. + error. Its exit status should indicate whether all tests passed. - The standard output from a "test suite" executable should consist of - lines of the form "run N ok N fail N error N", representing running + A test suite should only exit 127 if there was a fatal issue in executing + the suite itself. Errors among the individual tests should be reported + through the status counters as detailed below. + + The standard output from a test suite executable should consist of lines + of the form `total o run n ok m fail l error k', representing running counts as printed by exercise itself. Consequently, the exercise utility can be used as a (nested) test suite runner if desired. + After running a test suite, its final report is checked for consistency. + If any of the conditions (listed below) are not satisfied, the entire + test suite is discarded and counted as a single error, as if it had + exited with status 127. + + Status Counters + Unless given the -q option, exercise prints the following running + counters to its standard output: + + total The number of tests that are planned to execute. This is first + set to the number of test files found, including test suites. + The latter are initially counted as single tests because the + number of tests they contain is not known beforehand. The total + counter will increase at run time as test suites report their + totals. + + run The number of tests that have been started so far. This is + normally the sum of ok, fail, and error plus any tests that are + currently in progress. Test suites may increment this counter + before or after a test finishes, so it may or may not include all + tests in progress. + + ok The number of tests that have passed successfully, i.e. exited 0 + without any output to standard error. + + fail The number of tests that have yielded a negative result, i.e. + exited with a non-zero status (except 127) or printed anything to + standard error. + + error The number of tests that could not be executed despite the + attempt or that exited with the special status 127. + + The following conditions should hold at the end of a run: + + 1. run <= total + + 2. run = ok + fail + error + + The second condition is relaxed to an inequality at run time to account + for tests in progress, as explained above. The run count is permitted to + ultimately remain short of total as the result of test suites or exercise + terminating early. + DEPENDENCIES The exercise utility requires Tcl 8.4 or above. @@ -55,4 +118,4 @@ SEE ALSO AUTHORS Sven M. Hallberg - May 11, 2025 + May 25, 2025