- Description:
- simple but flexible test runner
- Owner:
- pesco
- Last Change:
- Clone URL:
https://code.khjk.org/exercise.git
Commit Briefs
test.h: rename check() macro to expect() (main)
I can't remember where else I've seen this naming, but I like it because it seems to match up more neatly with assert().
put a TAB into "test() skipped" to fit regular output
This version matches project sc2 (git): commit fb675b5f860edf6266885e42d9c09c2802cf9416 from: Sven M. Hallberg <pesco@khjk.org> date: Sat Dec 28 23:46:43 2024 UTC
test.h: remove colons from output
As a side effect, this makes the test output clean TSV. This version matches project sc2 (git): commit 7e054f3208ac703dda195f3d0c49bf53e781ebc2 from: Sven M. Hallberg <pesco@khjk.org> date: Sat Dec 28 16:17:19 2024 UTC The change was initially recorded in project rx (darcs): patch 31afcfda494ef1937d8019360c9cc1ba1c4ef3c2 Author: pesco@khjk.org Date: Sat Dec 28 16:15:49 UTC 2024
include assert.h from test.h
This tweak came silently with the initial commit of project sc2 (git): commit a398e14898e509e7df5eb1648b4a28408e92eec0 from: Sven M. Hallberg <pesco@khjk.org> date: Sun Dec 8 13:37:38 2024 UTC open arrays and uinit tests
add lang/c/test.h
This is the first (2020) version, from project rx (darcs): patch e23a5d20646a06606d11f4ab711e123b4cd98ba6 Author: pesco@khjk.org Date: Mon Apr 12 15:47:51 CEST 2021 * prototype
separate run and total counters
Total is the number of tests we intend to execute. It is semantically the counter that was called 'run' before. It may increase while running test suites but is otherwise set up-front. Run is now the number of tests we have so far (at least tried) to execute and increments as we go. It will normally be the sum of ok, fail, and error. If and when we add parallel execution of tests, the sum might lag behind because 'run' gets incremented before the test is finished. Counter consistency gains the requirement that run not exceed total. At the same time, it is relaxed such that run may end lower than total. Thus test suites (or exercise itself) may abort early (after the first failure, say) without such a run being considered invalid. This commit includes two new tests and fixes some bugs they exposed.
assert-exercise: remove tempfiles via an exit trap
This ensures it happens also on a premature exit (127).
Branches
Tree
Makefile | commits | blame |
README | commits | blame |
exercise* | commits | blame |
exercise.1 | commits | blame |
lang/ | |
tests/ |
README
EXERCISE(1) General Commands Manual EXERCISE(1) NAME exercise - simple but flexible test runner SYNOPSIS exercise [-q] [-p k=pattern] [path ...] DESCRIPTION The exercise utility searches for and executes software (unit) tests in or below the current directory. It displays a running status and passes any error messages to standard error. If given a directory argument, exercise searches for tests therein. If given a file argument, it executes it as a single test (or test suite). To be recognized as tests, files must match the pattern *.t. To be run as test suites (see below), files must match the pattern *.tests. The options are as follows: -p d=pattern When searching for tests, only descend into directories that match the given pattern. Defaults to `*' (all directories). -p s=pattern Process (only) files matching the given pattern as test suites. Defaults to `*.tests'. -p t=pattern Process (only) files matching the given pattern as tests. Defaults to `*.t'. -q Do not print status counters to standard output. Test Files Tests are represented simply as executable files that are expected to exit 0 on success. Non-zero exits as well as any output to standard error are interpreted as negative results (failure). Standard output is ignored. As a special case, exit status 127 is interpreted as an error that precluded the proper execution of the test, implying an inconclusive result. Test Suites Multiple tests can be combined into one executable. A test suite should pass error messages from individual tests unchanged to its standard error. Its exit status should indicate whether all tests passed. A test suite should only exit 127 if there was a fatal issue in executing the suite itself. Errors among the individual tests should be reported through the status counters as detailed below. The standard output from a test suite executable should consist of lines of the form `total o run n ok m fail l error k', representing running counts as printed by exercise itself. Consequently, the exercise utility can be used as a (nested) test suite runner if desired. After running a test suite, its final report is checked for consistency. If any of the conditions (listed below) are not satisfied, the entire test suite is discarded and counted as a single error, as if it had exited with status 127. Status Counters Unless given the -q option, exercise prints the following running counters to its standard output: total The number of tests that are planned to execute. This is first set to the number of test files found, including test suites. The latter are initially counted as single tests because the number of tests they contain is not known beforehand. The total counter will increase at run time as test suites report their totals. run The number of tests that have been started so far. This is normally the sum of ok, fail, and error plus any tests that are currently in progress. Test suites may increment this counter before or after a test finishes, so it may or may not include all tests in progress. ok The number of tests that have passed successfully, i.e. exited 0 without any output to standard error. fail The number of tests that have yielded a negative result, i.e. exited with a non-zero status (except 127) or printed anything to standard error. error The number of tests that could not be executed despite the attempt or that exited with the special status 127. The following conditions should hold at the end of a run: 1. run <= total 2. run = ok + fail + error The second condition is relaxed to an inequality at run time to account for tests in progress, as explained above. The run count is permitted to ultimately remain short of total as the result of test suites or exercise terminating early. DEPENDENCIES The exercise utility requires Tcl 8.4 or above. EXIT STATUS The exercise utility exits 0 if all tests passed, 1 if some tests failed or encountered errors, and 127 if an error occured outside of the tests. EXAMPLES See the tests/ directory distributed with the program. It shows a useful idiom where a number of test (shell) scripts make use of a helper, assert-exercise. The helper runs exercise with some test (shell) script to execute and compares the outcome to expectations. SEE ALSO sh(1) AUTHORS Sven M. Hallberg <pesco@khjk.org> May 25, 2025