Commit Briefs

b01ca349cc Sven M. Hallberg

test.h: rename check() macro to expect() (main)

I can't remember where else I've seen this naming, but I like it because it seems to match up more neatly with assert().


0f3b75642b Sven M. Hallberg

put a TAB into "test() skipped" to fit regular output

This version matches project sc2 (git): commit fb675b5f860edf6266885e42d9c09c2802cf9416 from: Sven M. Hallberg <pesco@khjk.org> date: Sat Dec 28 23:46:43 2024 UTC


b638044ea2 Sven M. Hallberg

test.h: remove colons from output

As a side effect, this makes the test output clean TSV. This version matches project sc2 (git): commit 7e054f3208ac703dda195f3d0c49bf53e781ebc2 from: Sven M. Hallberg <pesco@khjk.org> date: Sat Dec 28 16:17:19 2024 UTC The change was initially recorded in project rx (darcs): patch 31afcfda494ef1937d8019360c9cc1ba1c4ef3c2 Author: pesco@khjk.org Date: Sat Dec 28 16:15:49 UTC 2024


5299567952 Sven M. Hallberg

include assert.h from test.h

This tweak came silently with the initial commit of project sc2 (git): commit a398e14898e509e7df5eb1648b4a28408e92eec0 from: Sven M. Hallberg <pesco@khjk.org> date: Sun Dec 8 13:37:38 2024 UTC open arrays and uinit tests


b0aa2229df Sven M. Hallberg

add lang/c/test.h

This is the first (2020) version, from project rx (darcs): patch e23a5d20646a06606d11f4ab711e123b4cd98ba6 Author: pesco@khjk.org Date: Mon Apr 12 15:47:51 CEST 2021 * prototype


fcb7466a5c Sven M. Hallberg

regen README


12d156c595 Sven M. Hallberg

manpage fixes


42d07c50c8 Sven M. Hallberg

add counters section to manpage


9c8a0916ad Sven M. Hallberg

separate run and total counters

Total is the number of tests we intend to execute. It is semantically the counter that was called 'run' before. It may increase while running test suites but is otherwise set up-front. Run is now the number of tests we have so far (at least tried) to execute and increments as we go. It will normally be the sum of ok, fail, and error. If and when we add parallel execution of tests, the sum might lag behind because 'run' gets incremented before the test is finished. Counter consistency gains the requirement that run not exceed total. At the same time, it is relaxed such that run may end lower than total. Thus test suites (or exercise itself) may abort early (after the first failure, say) without such a run being considered invalid. This commit includes two new tests and fixes some bugs they exposed.


d020afe4a9 Sven M. Hallberg

assert-exercise: remove tempfiles via an exit trap

This ensures it happens also on a premature exit (127).


Branches

Tags

This repository contains no tags

Tree

Makefilecommits | blame
READMEcommits | blame
exercise*commits | blame
exercise.1commits | blame
lang/
tests/

README

EXERCISE(1)                 General Commands Manual                EXERCISE(1)

NAME
     exercise - simple but flexible test runner

SYNOPSIS
     exercise [-q] [-p k=pattern] [path ...]

DESCRIPTION
     The exercise utility searches for and executes software (unit) tests in
     or below the current directory.  It displays a running status and passes
     any error messages to standard error.

     If given a directory argument, exercise searches for tests therein.  If
     given a file argument, it executes it as a single test (or test suite).
     To be recognized as tests, files must match the pattern *.t.  To be run
     as test suites (see below), files must match the pattern *.tests.

     The options are as follows:

     -p d=pattern
             When searching for tests, only descend into directories that
             match the given pattern.  Defaults to `*' (all directories).

     -p s=pattern
             Process (only) files matching the given pattern as test suites.
             Defaults to `*.tests'.

     -p t=pattern
             Process (only) files matching the given pattern as tests.
             Defaults to `*.t'.

     -q      Do not print status counters to standard output.

   Test Files
     Tests are represented simply as executable files that are expected to
     exit 0 on success.  Non-zero exits as well as any output to standard
     error are interpreted as negative results (failure).  Standard output is
     ignored.

     As a special case, exit status 127 is interpreted as an error that
     precluded the proper execution of the test, implying an inconclusive
     result.

   Test Suites
     Multiple tests can be combined into one executable.  A test suite should
     pass error messages from individual tests unchanged to its standard
     error.  Its exit status should indicate whether all tests passed.

     A test suite should only exit 127 if there was a fatal issue in executing
     the suite itself.  Errors among the individual tests should be reported
     through the status counters as detailed below.

     The standard output from a test suite executable should consist of lines
     of the form `total o run n ok m fail l error k', representing running
     counts as printed by exercise itself.  Consequently, the exercise utility
     can be used as a (nested) test suite runner if desired.

     After running a test suite, its final report is checked for consistency.
     If any of the conditions (listed below) are not satisfied, the entire
     test suite is discarded and counted as a single error, as if it had
     exited with status 127.

   Status Counters
     Unless given the -q option, exercise prints the following running
     counters to its standard output:

     total   The number of tests that are planned to execute.  This is first
             set to the number of test files found, including test suites.
             The latter are initially counted as single tests because the
             number of tests they contain is not known beforehand.  The total
             counter will increase at run time as test suites report their
             totals.

     run     The number of tests that have been started so far.  This is
             normally the sum of ok, fail, and error plus any tests that are
             currently in progress.  Test suites may increment this counter
             before or after a test finishes, so it may or may not include all
             tests in progress.

     ok      The number of tests that have passed successfully, i.e. exited 0
             without any output to standard error.

     fail    The number of tests that have yielded a negative result, i.e.
             exited with a non-zero status (except 127) or printed anything to
             standard error.

     error   The number of tests that could not be executed despite the
             attempt or that exited with the special status 127.

     The following conditions should hold at the end of a run:

           1.   run <= total

           2.   run = ok + fail + error

     The second condition is relaxed to an inequality at run time to account
     for tests in progress, as explained above.  The run count is permitted to
     ultimately remain short of total as the result of test suites or exercise
     terminating early.

DEPENDENCIES
     The exercise utility requires Tcl 8.4 or above.

EXIT STATUS
     The exercise utility exits 0 if all tests passed, 1 if some tests failed
     or encountered errors, and 127 if an error occured outside of the tests.

EXAMPLES
     See the tests/ directory distributed with the program.  It shows a useful
     idiom where a number of test (shell) scripts make use of a helper,
     assert-exercise.  The helper runs exercise with some test (shell) script
     to execute and compares the outcome to expectations.

SEE ALSO
     sh(1)

AUTHORS
     Sven M. Hallberg <pesco@khjk.org>

                                 May 25, 2025