Re: [PATCH 00/17] De-recursion for the 'tests' subdirectory.

[Stefano Lattarini]

Hi Jim.

On 09/04/2012 12:13 PM, Jim Meyering wrote:
> Jim Meyering wrote:
> 
>> Stefano Lattarini wrote:
>>> Plus some other preparatory, follow-up or "while-at-it" changes.
>>>
>>> After this, "make syntax-check" and "make distcheck" still pass on
>>> my Debian box.
>>>
>>> This completes the first important part of the "de-recursion" of the
>>> coreutils build system: the compilation of the programs, the creation
>>> of Texinfo documentation and manpages, and the testsuite are now all
>>> controlled by one top-level Makefile.
>>>
>>> The next step would be de-recursing the gnulib-imported 'lib/' and
>>> 'gnulib-tests/' subdirectories, but I'll leave that to someone else ;-)
>>
>> Thanks a lot.
>> I've applied these and ensured that distcheck works for me, too.
>> (BTW, in coreutils, distcheck includes "make syntax-check" -- at least
>> it used to)
>>
>> I hope to actually review the changes later today.
>>
>> The only missing pieces:
>>   - a timing comparison: does this result in a performance improvement?
>>   - a NEWS entry: every 70+-change-set deserves one ;-)
>>     One incidental improvement I've noticed: it fixes at least one bug
>>     that would cause an occasional parallel "make syntax-check" failure.
> 
> Hi Stefano,
> 
> I still haven't reviewed your changes, but in using them I did notice
> a missing dependency:
> 
> I ran this command to run the single test:
> 
>     $ make tests/rm/v-slash.log
>     PASS: tests/rm/v-slash.log
> 
> then modified remove.c and recompiled src/rm and went to run the test again:
> 
>     $ make tests/rm/v-slash.log
>     make: `tests/rm/v-slash.log' is up to date.
> 
> Oops.  It didn't run.  The individual test scripts (the .log witness files,
> actually) do not depend on the programs they test.
>
The above usage is not actually supported Automake interface (for one thing,
it won't honour $(check_PROGRAMS), nor $(BUILT_SOURCES)).  To run a subset
of tests, you should do this:

    $ make check TESTS="tests/rm/v-slash"

And to run them lazily (i.e., only if one of their prerequisites has
changed), you should do:

    $ make check TESTS="tests/rm/v-slash" RECHECK_LOGS=

And if you want to re-run a test without overwriting the pre-existing
test-suite.log, here is what you should do:

    $ make check TESTS="tests/rm/v-slash" TEST_SUITE_LOGS=partial.log

All of this should be explained in the Automake manual, subsection
15.2.3, "Parallel Test Harness":

    
<http://www.gnu.org/software/automake/manual/automake.html#Parallel-Test-Harness>

> From 45bf1ab13682f8842b2b9ed81bb9069f6e5b01b3 Mon Sep 17 00:00:00 2001
> From: Jim Meyering <address@hidden>
> Date: Tue, 4 Sep 2012 12:12:19 +0200
> Subject: [PATCH] tests: depend on the programs we're testing
> 
> * tests/local.mk ($(TEST_LOGS)): Depend on $(PROGRAMS),
>
Beware that $(PROGRAMS) is not a documented Automake interface either.

> so that tests are rerun when any program is rebuilt.
>
Apart from my nits above, I believe this is a good idea, because it will
cause the documented interface for "lazy re-runs":

    $ make check TESTS="tests/rm/v-slash" RECHECK_LOGS=

to DTRT.

Your idiom won't be completely precise though, since $(TEST_LOGS) can be
overridden at make runtime by the user (either directly, or as a consequence
of his overriding of $(TESTS)); but being 100% correct would require you to
duplicate some of messy logic Automake applies to $(TESTS), to apply it to
$(all_tests) instead:

    all_test_logs1 = $(all_tests:=.log)
    all_test_logs2 = $(all_test_logs1:.sh.log=.log)
    all_test_logs3 = $(all_test_logs2:.pl.log=.log)
    all_test_logs = $(all_test_logs3:.xpl.log=.log)
    $(all_test_logs): $(PROGRAMS)

but that's not worth the hassle IMHO.

> Technically, we could
> specify precisely which few programs are dependents of each test,
> but that can come later, if deemed worth the trouble and maintenance
> burden.  Also, there is the issue of the primary program(s) being
> tested (i.e., those itemized via print_ver_) versus those that are
> tested incidentally: for example, nearly every test exercises "rm"
> when its clean-up code removes files.
> ---
>  tests/local.mk | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/tests/local.mk b/tests/local.mk
> index b129475..2440667 100644
> --- a/tests/local.mk
> +++ b/tests/local.mk
> @@ -784,3 +784,5 @@ pr_data =                                 \
>    tests/pr/ttb3-0FF                          \
>    tests/pr/ttb3-FF                           \
>    tests/pr/w72l24f-ll
> +
> +$(TEST_LOGS): $(PROGRAMS)
> --
> 1.7.12.176.g3fc0e4c

Regards,
  Stefano