RFC: New Autotest Suite for CVS

[Derek Robert Price]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hey all,

I finally have a decent prototype for the Autotest suite.  I've only
converted four tests (init, version, basica, & basicb), but I wanted to
get some opinions before I converted any more tests since at my current
rate of about 25 lines an hour, it will take me awhile to convert the
rest of the tests.

The suite is currently on the `ccvs-autotest-2' branch.  I've documented
the new suite in TESTS, but you can run it via `make check'.  Things to
notice are:

~    1.  The output is more verbose.  Individual test groups get "ok",
"failed", or "skipped".  If you have Autoconf 2.58 installed, go ahead
and force some failures by tweaking the expected result of tests -
otherwise, just run the test suite in verbose mode and check the output log.

~    2.  Tests are run in ./suite.dir from wherever you run the tests.

~    3.  All failed tests leave their data and their snippet of the log
in ./suite.dir/<test number>.  I was originally turned off by this,
thinking only one failure was useful, per sanity.sh, but the logging is
very verbose and I found that, like a C compiler, sometimes a list of
errors would just be a cascade, but sometimes I could fix multiple
problems after a single run of the tests.

~    4.  Lists of tests can be run as well as disjoint ranges.  Subsets
of the tests can be specified via keywords using the --keywords argument
to the tests suite.  --keywords=remote is a synonym for the old -r, but
without -r, all tests are run in both local and remote modes.

~    5.  Logging is very verbose and includes versions and paths to
executables.  This should be very useful for debugging users' reports of
test failures.

~    6.  Cleanup of tests is automatic, aided by the fact that they each
get their own directory and cvsroot.  Tests are even shielded from
changes in the environment of earlier tests.

~    That's enough for now.  Let me know what you think, especially
Larry and Mark.  I'm thinking that if you approve, I might move this
back to feature soon, with the Makefile tweaked to run both sanity.sh
and the new suite on a `make check', then convert tests as I have time
since it would mean less merging.

Derek

- --
~                *8^)

Email: derek@ximbiot.com

Get CVS support at <http://ximbiot.com>!
- --
Life in a vacuum sucks.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.7 (GNU/Linux)
Comment: Using GnuPG with Netscape - http://enigmail.mozdev.org

iD8DBQE/sFRRLD1OTBfyMaQRAlyvAKC4bgZeJ0gT+lfX4sFk6oDS1wgKHgCePW9O
whFBBKxKeZEBRgWgp0UYmtc=
=Qd6n
-----END PGP SIGNATURE-----