Re: parallel-tests: redo lazy checking: recheck and RECHECK_LOGS.

[Ralf Wildenhues]

* Akim Demaille wrote on Mon, Mar 30, 2009 at 09:50:59PM CEST:
> Le 29 mars 09 à 00:04, Ralf Wildenhues a écrit :
>
>>> I often use
>>>
>>>     STRICT_TEST_LOGS = $(shell $(LIST_FAILED_TEST_LOGS))
>>>
>>> which makes all failing test strict.  In other words, successful
>>> tests are not rerun by "make check", but failing tests are.  This is
>>> because our test suite sometimes hits the limit of the machine, and
>>> tests timeout for no sound reason.  So there is no reason to rerun
>>> successful tests, but failing test might pass if the machine has a
>>> lesser load.
>>
>> This is what the 'recheck' target does, right?
>
> Not exactly.  make recheck runs only the tests that failed, while here, 
> in addition, the tests whose dependencies have changed will be run again. 
>  Not to mention test that where not run at all.

I don't think your interpretation is right.  Those tests whose
dependencies have changed will be rerun in any case, with your
recheck, with my recheck, with above.  You cannot prevent it unless
you remove those dependency statements from the makefile.

Or maybe I have just completely misunderstood you again.

>> One thing that one needs to look out for is, when overriding variables
>> in recursive `make' instances is that non-GNU make don't override by
>> default unless you use `make -e' and pass via the environment.
>> Requiring our users to do so if they want to override variables is ok
>> IMVHO, because then they are warned that their environment needs to be
>> clean.  OTOH using -e internally is not ok at all, as the users may  
>> not be aware, and their environment could cause subtle breakage.  So
>> consequently we can only transport overrides one recursion deep and  
>> not any further, unless each lower re-sets the variables on the make  
>> command line.

> I have not understood exactly what you are referring too, but why don't 
> we just pass TESTS=$(TESTS) explicitly?

Because it can cause command line length issues.  Because variable
overriding is inherently fragile: it can subtly change the meaning
of the underlying makefile.

When another indirection for globbing were added, we'd have to pass
$(TESTS) through more than one $(MAKE).  And we'd have to pass
$(TEST_LOGS) as well, if we were to allow globbing on that.  And
I would like to avoid passing both, that halves our available length
limit.

>  Or ask the users to add his "precious macros" to AM_MAKEFLAGS?

We could do that.  In fact, I intend to add a note to this extent to the
Autoconf manual section about `make' portability, but AM_MAKEFLAGS
should be mentioned in automake.texi too I guess.  What I don't know is
whether it was intended as user variable or as makefile.am author realm.

>> This detail was BTW the reason that I have backed off of allowing  
>> shell
>> globbing for TESTS or TEST_LOGS.  It is just too complicated to get
>> right portably, causes other problems (e.g., I do not want to have a
>> rule that has both TESTS and TEST_LOGS expanded in one shell command:
>> it might overrun command line length limits), and can be emulated by  
>> the
>> user on the command line or with a script (or, for GNU make, a simple
>> wrapper target).  Thus this patch documents such an example.
>
> Can't we have it when using GNU Make?  Can't we have some sort of #ifdef 
> HAVE_GNU_MAKE sections in the output Makefile?  I'm not referring to 
> AM_CONDITIONAL, but to some nice run-time trick.

Produce a GNUmakefile in the build directory.  Let this GNUmakefile
include the Makefile.  Put all your nice goodies in GNUmakefile.

An example of this is what the gnulib module gnumakefile does.
For example, it allows you to
  make -j3 clean all check distcheck

and it will work!  In a git repo, and with the git .version strategy
used by several gnulib-using packages, it will autoreconf so the tarball
will have an up to date version string.

> Maybe on purpose, maybe not, you dropped recheck-html.

I've fixed this in the git next tree which is online.  Thanks for the
reminder, though!

I should apologize for not having posted any patches which I applied
yesterday.  I will post them in the next days.

Cheers,
Ralf