test suite external loop (was: patch-1: testsuite automake version requirements)

[Ralf Wildenhues]

Hi Gary,

* Gary V. Vaughan wrote on Wed, Dec 14, 2005 at 11:06:16AM CET:
> 
> I'm back in the UK for a few weeks :-D

Cool.

First of all, let me thank you for all your reviews.
I'll apply everything soon, but commenting on one issue for now only:

> Ralf Wildenhues wrote:
> >
> >Not really sure about how to handle old Automake versions.

> >But this also means that we have very little actual test exposure
> >for versions <1.6.
> >
> >Comments?
> 
> Patch looks good by inspection.
> 
> Maybe we could put the bulk of the tests from testsuite.at inside a
> loop (oversimplified for explanatory purposes):
> 
>   for AMMINOR in 10 9 8 7 6 5 4; do
>     AUTOMAKE=automake-1.$AMMINOR
>     test -x $AUTOMAKE || continue
> 
>     echo "*** Running testsuite with $AUTOMAKE ***"
>     echo
>     m4_include(*.at)

This doesn't work (because of all the diversions autotest does).
I believe you knew that, and I get your point.

>     echo
>   done
> 
> Then any problems with compatibility with older automake APIs should
> come out in the wash...

Actually, I don't think it makes too much sense to do this on every
users' system.  They are most likely to use plain `automake' anyway,
and it's most important to get those cases right that are used the
most..

But.  Since you asked.

I have this own little script I've been working on a while to make
libtool prerelease testing more efficient.  Basically just a loop over
some configuration or environment settings, and shells, and whatnot (for
major bugs not already exposed I hope to add another configuration).

And systems (I have access to).   The nice thing is that it starts them
in parallel (one per system) and collects the output.  :)
As such, both human work involved and test latency scale only very lowly
with the number of tested configurations.  The major work is actually
reading the test results, and tuning the configurations.

The non-system specific part of the script is attached, for the curious
reader.

And yes, I've done above for all of automake-1.4 through 1.9 and 1.9a.
Next thing is Debian autoconf 2.13/2.59+wrapper.

Cheers,
Ralf

runthem.template
Description: Text document