Re: tests updates

[Ralf Wildenhues]

* Stefano Lattarini wrote on Thu, Nov 04, 2010 at 01:03:17PM CET:
> On Wednesday 03 November 2010, Ralf Wildenhues wrote:
> > * Stefano Lattarini wrote on Wed, Nov 03, 2010 at 02:27:47PM CET:
> > > Just one question: what about the already-existing "tests-init" branch?
> > > Should I try to bring it to a point where it can be easily merged into
> > > master, and then forget about it, or should it continue to live parallel
> > > to master?
> > 
> > Right now the branch looks like its changes would be fairly safe to
> > merge to master, but also, it would mean that sharing testsuite patches
> > between branch-1.11 and master may become a bit more cumbersome.  So I'd
> > say it depends a bit on where you want to go with this in the long run.
> Well, I'd like to:
> 
>  1. reorganize the code in "tests/defs.in" in a more rational way
>     (basically just a reordering of the code);

Seems like a good idea.

>  2. (re)introduce the `run_command' function and use it throughout;

I like it but before it can be merged it needs test exposure on
Tru64/OSF, IRIX, and Solaris at least, ideally also AIX and HP-UX.
You should be able to test on these systems in the near future,
hopefully.

>  3. make it easier for the user to override the aclocal "search path"
>     in the testsuite (a new pending patch from Paolo Bonzini could
>     help with this);

Why would the user need to mess with our testsuite?
Oh, is this so Libtool testing works with distcheck and with unusual
setups?

>  4. improve requirements' declaration in Automake test scripts, and
>     make it easier to run the Automake testsuite using different C,
>     C++ and Fortran compilers, without leaning too much towards the
>     GNU compilers as it does now.

This is a noble goal; I support it in a way that tests which are aimed
at showing off Automake API which is supposed to be portable to
different compilers, should also work with different compilers.  Tests
which are aimed at verifying more internal things do not need to be made
artificially portable to external tools.  Also, we should strive to not
make ourselves too dependent on bugs in Autoconf or Libtool; while that
can be helpful to uncover bugs in those tools, it also means that we
make ourselves liable to timely fixes, or version tests and all.

> These major changes would be interspesed with minor code cleanups
> and refactoring, such as (to list a few) avoiding to leak temporary
> variables from `tests/defs' into the test scripts, renaming `$curdir'
> to (say) `$testbuilddir' for better clarity, and improve the messages
> from skipped tests.

The minor cleanups sound ok, with the caveat that they have good chances
of breaking merges, or test suite additions coming from the stable
branch.  It is thus a good idea not to change things gratuitously here,
and it may be prudent to do defs API changes in a branch off of maint,
so it can be merged into other active branches before they are merged
into master.

> > We can easily move to a strategy of having new testsuite stuff added to
> > master only by default, and only care about branch-1.11 for fixing
> > existing regressions and serious bugs, which would probably make the
> > situation easier.
> Yes, I'd like to go this way.

OK then.

> > Of course there are other active branches too, that shouldn't have a
> > harder-than-necessary time.

> Ideally, all the tests that work with current `tests/defs' should
> continue to work with the new one; the only problem might be that
> some new tests committed to another branch might cause *maintcheck*
> failures when merged to master, but that is not a big deal IMHO,
> and easily fixed anyway.

OK.

> > We could treat merges similarly to patches in announcing the intention
> > to merge 72h before the fact, in those cases where there is doubt over
> > the merge (or the precise point at which a merge should be done).
> This won't be necessary if we stick to master for tests extensions and
> refactoring.

Well, don't let the above paragraph of mine be incentive to use less
branches.  That's not what I intended.  Doing things in separate
branches *when appropriate* is good!

> > I must admit that I don't have all that much experience with merges of
> > longer-term branches done by more than one person; one alternative would
> > be that you suggest the merge and I do it (if that works out).  Testing
> > such merges usually requires a bit of additional work, in that both
> > branches need to be checked for global changes that need adjusting in
> > the respective other branch.  Let's see how this works out.
> 
> In the end, are you OK with having me to merge "test-init" to master right
> away, and do future testsuite work on master only?

Yes.

Thanks,
Ralf