17-fyi-autotest-doc.patch

[Akim Demaille]

This is only a first stab, a lot of crucial information is missing
(e.g., relationship with M4sh, the fact that *anything* outside a test
group is completely ignored etc.).

But it's a starting point.  Of course, brave souls are most welcome to
complete this documentation, and the other souls may, if they really
want to, comment this first draft ;)


Index: ChangeLog
from  Akim Demaille  <address@hidden>

        * Makefile.am: Remove dead code and dead comments.
        (pdf, html): New targets.
        * doc/autoconf.texi (Using Autotest): New chapter.
        * doc/Makefile.am (pdf): New targets.
        (CLEANFILES): Adjust.

Index: Makefile.am
--- Makefile.am Wed, 01 Aug 2001 23:34:52 +0200 akim
+++ Makefile.am Mon, 13 Aug 2001 10:25:36 +0200 akim
@@ -24,17 +24,20 @@

 ## There is currently no means with Automake not to run aclocal.
 ACLOCAL_AMFLAGS = --version >/dev/null && touch aclocal.m4
-WGET = wget
-
-# FIXME:
-# s/distpackageDATA/dist_pkgdata_DATA/
-# s/nodistpackageDATA/nodist_pkgdata_DATA/
-# and adapt dependencies once we use a more recent Automake

 EXTRA_DIST = ChangeLog.0 ChangeLog.1 ChangeLog.2 \
              BUGS INSTALL.txt \
              GNUmakefile Makefile.maint

+
+## -------------------- ##
+## Forwarding targets.  ##
+## -------------------- ##
+
+html pdf:
+       cd doc && $(MAKE) $(AM_MAKEFLAGS) $@
+
+.PHONY: html pdf

 ## ------------------ ##
 ## Maintainer rules.  ##
Index: doc/Makefile.am
--- doc/Makefile.am Sun, 17 Jun 2001 20:12:25 +0200 akim
+++ doc/Makefile.am Mon, 13 Aug 2001 12:51:28 +0200 akim
@@ -20,6 +20,7 @@

 MAKEINFO = @MAKEINFO@ --no-split
 TEXI2HTML = texi2html
+TEXI2HTML_FLAGS = -split_chapter

 info_TEXINFOS = autoconf.texi standards.texi
 autoconf_TEXINFOS = install.texi
@@ -27,15 +28,34 @@

 # Files from texi2dvi that should be removed, but which Automake does
 # not know.
-CLEANFILES = autoconf.cvs autoconf.ev autoconf.evs autoconf.ma autoconf.mas \
-             autoconf.ov autoconf.ovs autoconf.ms autoconf.mss autoconf.tmp
+CLEANFILES = autoconf.cvs \
+            autoconf.ev autoconf.evs autoconf.ma autoconf.mas \
+             autoconf.ov autoconf.ovs autoconf.ms autoconf.mss \
+            autoconf.at autoconf.ats \
+            autoconf.tmp \
+            autoconf*.html standards*.html \
+            autoconf*.pdf standards*.pdf

-# The documentation
+
+## ----------------------------- ##
+## Other documentation formats.  ##
+## ----------------------------- ##

 html: autoconf_1.html standards_1.html

 autoconf_1.html: autoconf.texi install.texi
-       $(TEXI2HTML) -split_chapter $(srcdir)/autoconf.texi
+       $(TEXI2HTML) $(TEXI2HTML_FLAGS) $(srcdir)/autoconf.texi

 standards_1.html: standards.texi make-stds.texi
-       $(TEXI2HTML) -split_chapter $(srcdir)/standards.texi
+       $(TEXI2HTML) $(TEXI2HTML_FLAGS) $(srcdir)/standards.texi
+
+
+pdf: autoconf.pdf standards.pdf
+
+autoconf.pdf: autoconf.texi install.texi
+       $(TEXI2DVI) --pdf --batch $(srcdir)/autoconf.texi
+
+standards.pdf: standards.texi make-stds.texi
+       $(TEXI2DVI) --pdf --batch $(srcdir)/standards.texi
+
+.PHONY: html pdf
Index: doc/autoconf.texi
--- doc/autoconf.texi Sat, 04 Aug 2001 19:23:09 +0200 akim
+++ doc/autoconf.texi Mon, 13 Aug 2001 13:24:42 +0200 akim
@@ -46,6 +46,8 @@
                                 Configuring a package
 * config.status: (autoconf)config.status Invocation.
                                 Recreating a configuration
+* testsuite: (autoconf)testsuite invocation
+                                Running an Autotest test suite
 @end direntry

 @ifinfo
@@ -116,6 +118,8 @@ Autoconf: Creating Automatic Configurati
 @defcodeindex cv
 @c Define an Autoconf macro index that @defmac doesn't write to.
 @defcodeindex ma
address@hidden Define an Autotest macro index that @defmac doesn't write to.
address@hidden at
 @c Define an M4sugar macro index that @defmac doesn't write to.
 @defcodeindex ms

@@ -148,6 +152,7 @@ @node Top, Introduction, (dir), (dir)
 * Running configure scripts::   How to use the Autoconf output
 * config.status Invocation::    Recreating a configuration
 * Obsolete Constructs::         Kept for backward compatibility
+* Using Autotest::              Creating portable test suites
 * Questions::                   Questions about Autoconf, with answers
 * History::                     History of Autoconf
 * Environment Variable Index::  Index of environment variables used
@@ -155,6 +160,7 @@ @node Top, Introduction, (dir), (dir)
 * Preprocessor Symbol Index::   Index of C preprocessor symbols defined
 * Autoconf Macro Index::        Index of Autoconf macros
 * M4 Macro Index::              Index of M4, M4sugar, and M4sh macros
+* Autotest Macro Index::        Index of Autotest macros
 * Concept Index::               General index

 @detailmenu --- The Detailed Node Listing ---
@@ -393,6 +399,13 @@ @node Top, Introduction, (dir), (dir)
 * Changed Quotation::           Broken code which used to work
 * New Macros::                  Interaction with foreign macros

+Generating Test Suites with Autotest
+
+* testsuite Scripts::           The concepts of Autotest
+* Writing testsuite.at::        Autotest macros
+* testsuite invocation::        Running @command{testsuite} scripts
+* Making testsuite Scripts::    Using autom4te to create @command{testsuite}
+
 Questions About Autoconf

 * Distributing::                Distributing @code{configure} scripts
@@ -6536,6 +6549,8 @@ @node Forbidden Patterns,  , Redefined M
 @code{m4_pattern_forbid} pattern.
 @end defmac

+
+
 @c=================================================== Writing Autoconf Macros.

 @node Writing Autoconf Macros, Portable Shell, Programming in M4, Top
@@ -9681,7 +9696,7 @@ Makefile: Makefile.in config.status

 @c =================================================== Obsolete Constructs

address@hidden Obsolete Constructs, Questions, config.status Invocation, Top
address@hidden Obsolete Constructs, Using Autotest, config.status Invocation, 
Top
 @chapter Obsolete Constructs

 Autoconf changes, and throughout the years some constructs are obsoleted.
@@ -10967,9 +10982,344 @@ autoupdate: `configure.in' is updated
 $
 @end example

+
address@hidden ============================= Generating Test Suites with 
Autotest
+
address@hidden Using Autotest, Questions, Obsolete Constructs, Top
address@hidden Generating Test Suites with Autotest
+
address@hidden Autotest
+
address@hidden
address@hidden: This section describes an experimental features which will
+officially part of Autoconf in a forthcoming release.  Although we
+believe Autotest is stabilizing, the documentation below describes an
+interface which might change in the future: do not depend upon Autotest
+without subscribing to the Autoconf mailings lists.}
address@hidden display
+
+It is paradoxical that portable projects depend on nonportable tools to
+run their test suite.  Autoconf by itself is the paragon of this
+problem: although it aims at perfectly portability, up to 2.13, its test
+suite was using DejaGNU, a rich and complex testing framework, but which
+is far from being standard on Unix systems.  Worse yet, it was likely to
+be missing on the most fragile platforms, the very platforms that are
+most likely to torture Autoconf and exhibit deficiencies.
+
+To circumvent this problem many package maintainers have developed their
+own testing framework, based on simple shell scripts whose sole output
+are their exit status: the test succeeded, or failed.  In addition, most
+of these tests share some common patterns, what results in lots of
+duplicated code, tedious maintenance etc.
+
+Following exactly the same reasoning that yielded to the inception of
+Autoconf, Autotest provides a test suite generation frame work, based on
+M4 macros, building a portable shell script.  The suite itself is
+equipped with automatic logging and tracing facilities which greatly
+diminish the interaction with bug reporters, and simple timing reports.
+
+Autoconf itself has been using Autotest for years, and we do attest that
+it has considerably improved the strength of the test suite, and the
+quality of bug reports.  Other projects are known to use some generation
+of Autotest, such as Bison, Free Recode, Free Wdiff, GNU Tar, each of
+them having different needs, what slowly polishes Autotest as a general
+testing framework.
+
+Nonetheless, compared to DejaGNU, Autotest is inadequate for interactive
+tool testing, which is probably its main limitation.
+
address@hidden
+* testsuite Scripts::           The concepts of Autotest
+* Writing testsuite.at::        Autotest macros
+* testsuite invocation::        Running @command{testsuite} scripts
+* Making testsuite Scripts::    Using autom4te to create @command{testsuite}
address@hidden menu
+
address@hidden testsuite Scripts, Writing testsuite.at, Using Autotest, Using 
Autotest
address@hidden @command{testsuite} Scripts
+
address@hidden @command{testsuite}
+
+Generating testing or validation suites using Autotest is rather easy.
+The whole validation suite is held in a file to be processed through
address@hidden@address@hidden
address@hidden
address@hidden @command{autom4te}
+It is on purpose that we don't document @command{autom4te} which is yet
+another forthcoming Autoconf feature: it is being developed and
+polished.  It will be documented when ready for wide spread use.  Do not
+use it without at least being member of the Autoconf mailing lists.
address@hidden
+}, itself using GNU @code{m4} under the scene, to produce a stand-alone
+Bourne shell script which then gets distributed.  Neither
address@hidden nor GNU @code{m4} are not needed anymore at the
+installer end.
+
address@hidden test group
+Each test of the validation suite should be part of some test group.  A
address@hidden group} is a sequence of interwoven tests that ought to be
+executed together, usually because one test in the group creates data
+files than a later test in the same group needs to read.  Complex test
+groups make later debugging more tedious.  It is much better keeping
+keep only a few tests per test group, and if you can put only one test
+per test group, this is just ideal.
+
+For all but the simplest packages, some file such as @file{testsuite.at}
+does not fully hold all test sources, as these are often easier to
+maintain in separate files.  Each of these separate files holds a single
+test group, or a sequence of test groups all addressing some common
+functionality in the package.  In such cases, file @file{testsuite.at}
+only initializes the whole validation suite, and sometimes do elementary
+health checking, before listing include statements for all other test
+files.
+
+The validation scripts that Autotest produces are by convention called
address@hidden  When run, @command{testsuite} executes each test
+group in turn, producing only one summary line per test to say if that
+particular test succeeded or failed.  At end of all tests, summarizing
+counters get printed.  If any test failed, one debugging script gets
+automatically generated for each test group which failed.  These
+debugging scripts are named @address@hidden, using for @var{nn}
+the sequence number of the test group.  In the ideal situation, none of
+the tests tests fail, and consequently, no debugging script is generated
+out of validation.
+
+The automatic generation of debugging scripts for failed test has the
+purpose of easing the chase for bugs.
+
address@hidden FIXME: This is not accurate today: we have a single atconfig.in.
+It often happens in practice that individual tests in the validation
+suite need to get information coming out of the configuration process.
+Some of this information, common for all validation suites, is provided
+through the file @file{atconfig.in}, which your package should distribute
+verbatim, unchanged.  For configuration informations which your testing
+environment specifically needs, you might prepare an optionnal file named
address@hidden  The configuration process produces @file{atconfig}
+and @file{atlocal} out of these two input files, and these two produced
+files are automatically read by the @file{testsuite} script.
+
+Here is a diagram showing the relationship between files.
+
address@hidden
+Files used in preparing a software package for distribution:
+
address@hidden
+testsuite.at --> autom4te* --> testsuite
address@hidden example
+
address@hidden
+Files used in configuring a software package:
+
address@hidden
+atconfig.in                                        testsuite.log
+    |                                                   ^
+    v                    .-> atconfig --.               |
+    +--> config.status* -+              +-> testsuite* -+
+    ^                    `-> [atlocal] -'               |
+    |                                                   v
+[atlocal.in]                                       address@hidden
address@hidden example
+
address@hidden Writing testsuite.at, testsuite invocation, testsuite Scripts, 
Using Autotest
address@hidden Writing @file{testsuite.at}
+
+The @file{testsuite.at} is a Bourne shell script making use of special
+Autotest M4 macros.  It often contains a call to @code{AT_INIT} nears
+its beginning followed by one call to @code{m4_include} per source file
+for tests.  Each such included file, or the remainder of
address@hidden if include files are not used, contain a sequence of
+test groups.  Each test group begins with one call to @code{AT_SETUP},
+it contains an arbitrary number of shell commands or calls to
address@hidden, and it completes with one call to @code{AT_CLEANUP}.
+
address@hidden AT_INIT (@var{program})
address@hidden INIT
address@hidden FIXME: Not clear, plus duplication of the information.
+Initialize Autotest. This macro accepts a single argument, which is the
+file name of the executable program to use for checking version.  At the
+time the validation suite gets executed, the program search path should
+be already preset so the proper executable will be selected.  This is
+usually guaranteed through some @file{Makefile} magic.
+
address@hidden is then used to check that the test suite is ready to be
+run (@var{program} was found), and corresponds to the current version of
+the package (@address@hidden --version} is run and checked against
+the package version).
address@hidden defmac
+
address@hidden AT_SETUP (@var{test-group-name})
address@hidden SETUP
+This macro starts a group of related tests, all to be executed in the same
+subshell.  It accepts a single argument, which holds a few words (no more
+than about 30 or 40 characters) quickly describing the purpose of the test
+group being started.
address@hidden defmac
+
address@hidden AT_CLEANUP (@ovar{generated-files})
address@hidden CLEANUP
+This macro completes a group of related tests.  It accepts a single argument,
+which is a white separated list of files which have been created within the
+test group.  It has the effect of recursively removing those listed files.
+There is no need to list @file{stdout}, @file{stderr}, nor files created
+with @code{AT_DATA}.
address@hidden defmac
+
address@hidden AT_DATA (@var{file}, @var{contents})
address@hidden DATA
+Initialize an input data @var{file} with given @var{contents}.  Of
+course, @var{contents} might have to be properly quoted between square
+brackets to protect against included commas or spurious @code{m4}
+expansion.  The contents ought to end with an end of line.
address@hidden defmac
+
address@hidden AT_CHECK (@var{commands}, @ovar{status = 0}, @ovar{stdout}, 
@ovar{stderr})
address@hidden CHECK
+This macro has up to four arguments: @var{commands}, @var{status},
address@hidden and @var{stderr}.  The @var{commands} argument is
+mandatory, and @code{m4} quoting is often useful.
+
+The macro executes a test by performing given shell @var{commands}.
+These commands should normally exit with @var{status}, while producing
+expected @var{stdout} and @var{stderr} contents.  Unless empty,
address@hidden is a decimal integer; exit status is not checked if
address@hidden is empty.  The special word @code{expout} for @var{stdout}
+means that file @file{expout} contents has been preset to the expected
+standard output.  The special word @code{experr} for @var{stderr} means
+that file @file{experr} contents has been preset to the expected
+standard error output.
address@hidden defmac
+
+
address@hidden testsuite invocation, Making testsuite Scripts, Writing 
testsuite.at, Using Autotest
address@hidden Running @command{testsuite} Scripts
address@hidden @command{testsuite}
+
+The @command{testsuite} script, as well as all generated debugging
+scripts, accept the following options.
+
address@hidden FIXME: Completely obsolete now...
+
address@hidden @samp
+
address@hidden --help
+This option inhibits normal execution of the script, and merely gives a
+summary of available options instead.
+
address@hidden --version
+This option inhibits normal execution of the script, and gives the package
+name and version numbers to be tested.
+
address@hidden -e
+This option is only meaningful for the full testsuite, it is not really
+useful for generated debugging scripts.  If any test fails in the suite,
+immediately abort testing and inhibit normal clean up.  The default action
+is to clean up and proceed with the following tests, as the usual goal of
+the whole validation suite is to produce result counts and debugging scripts.
+
address@hidden -n
+For checking the contents of standard output and standard error output, their
+contents are normally redirected into special files named @file{stdout}
+and @file{stderr}.  This option prevents the redirection to occur, so
+standard error output and standard error output appear @emph{normally}.
+This option causes any testing of the contents of standard output or
+standard error output to be bypassed.
+
address@hidden -s
+When the full test suite is generating debugging scripts, or when the
+generated debugging scripts get executed, many informative details are given
+about what is being done.  This option inhibits those informative details.
+
address@hidden -v
+This option is only meaningful for the full testsuite, as it is automatically
+selected in the generated debugging scripts (unless option @samp{-s} is
+also used).  It has the purpose of forcing more verbosity in the detailed
+output of what is being done.
+
address@hidden -x
+This option is transmitted to the shell at appropriate places, and asks for
+a trace of command execution.  This option also implies option @samp{-n}.
address@hidden table
+
+
address@hidden Making testsuite Scripts,  , testsuite invocation, Using Autotest
address@hidden Making @command{testsuite} Scripts
+
+For putting Autotest into movement, you need some configuration and
+Makefile machinery.  We recommend, at least if your package uses deep or
+shallow hierarchies, that you use @file{tests/} as the name of the
+directory holding all your tests and their @file{Makefile}.  Here is a
+check list of things to do.
+
address@hidden @minus
+
address@hidden
+Ensure that configuration defines @code{PACKAGE} and @code{VERSION}.
+This is already guaranteed if you are using Automake through the
address@hidden call, but with straight Autoconf, it means that
+you should at least use @code{AC_SUBST} for these two variables, after
+having initialized them to proper values.
+
address@hidden
address@hidden FIXME: This macro should become part of Autoconf.  
AC_AUTOTEST_PATH?
+Use the @code{AT_CONFIG} macro from within file @file{configure.ac}.
+This macro accepts one argument, which is the directory, relative to the
+test directory, where the executables are prepared.
+
address@hidden
+Still within @file{configure.ac}, ensure that some
address@hidden command includes substitution for
address@hidden/atconfig} and also, as appropriate, @file{tests/atlocal}.
+
address@hidden
+The @file{tests/Makefile.in} should be modified so the validation in
+your package is triggered by @samp{make check}.  An example is provided
+below.
+
address@hidden itemize
+
+With Automake, here is a minimal example about how to link @samp{make check}
+with a validation suite.
+
address@hidden
+EXTRA_DIST = testsuite.at testsuite
+
+all-local: atconfig testsuite
+
+check-local: atconfig testsuite
+        $(SHELL) $(srcdir)/testsuite
+
+AUTOM4TE = autom4te -I /usr/local/share/autoconf/lib
+$(srcdir)/testsuite: $(srcdir)/testsuite.at
+        $(AUTOM4TE) -I $(srcdir) autotest/general.m4 address@hidden -o 
address@hidden
+        chmod +x address@hidden
+        mv address@hidden $@
address@hidden example
+assuming Autoconf was installed with @code{prefix} being
address@hidden/usr/local}.  You might want to list explicitly the dependencies,
+i.e., the list of the files @file{testsuite.at} includes.
+
+With strict Autoconf, you might need to add lines inspired from the
+following:
+
address@hidden
+subdir = tests
+
+check: check-local
+
+atconfig: $(top_builddir)/config.status atconfig.in
+       cd $(top_builddir) && \
+           $(SHELL) ./config.status --file$(subdir)/$@
address@hidden example
+
address@hidden
+and manage to have @file{atconfig.in} and @code{$(EXTRA_DIST)}
+distributed.
+
+
+
 @c ================================================ Questions About Autoconf.

address@hidden Questions, History, Obsolete Constructs, Top
address@hidden Questions, History, Using Autotest, Top
 @chapter Questions About Autoconf

 Several questions about Autoconf come up occasionally.  Here some of them
@@ -11400,7 +11750,7 @@ @node Autoconf Macro Index, M4 Macro Ind

 @printindex ma

address@hidden M4 Macro Index, Concept Index, Autoconf Macro Index, Top
address@hidden M4 Macro Index, Autotest Macro Index, Autoconf Macro Index, Top
 @unnumbered M4 Macro Index

 This is an alphabetical list of the M4, M4sugar, and M4sh macros.  To
@@ -11409,7 +11759,15 @@ @node M4 Macro Index, Concept Index, Aut

 @printindex ms

address@hidden Concept Index,  , M4 Macro Index, Top
address@hidden Autotest Macro Index, Concept Index, M4 Macro Index, Top
address@hidden Autotest Macro Index
+
+This is an alphabetical list of the Autotest macros.  To make the list
+easier to use, the macros are listed without their preceding @samp{AT_}.
+
address@hidden at
+
address@hidden Concept Index,  , Autotest Macro Index, Top
 @unnumbered Concept Index

 This is an alphabetical list of the files, tools, and concepts