Re: Intermittent test failures

[Ralf Wildenhues]

Finally continuing this:

* Ludovic Courtès wrote on Wed, Aug 25, 2010 at 02:48:31PM CEST:
> Ralf Wildenhues writes:
> >     Fix timestamp issues by ensuring configure takes at least a second.
> >     
> >     * m4/sanity.m4 (AM_SANITY_CHECK): If we didn't sleep here,
> >     start a sleep in the background and wait for it to finish
> >     before creating config.status, hopefully fixing all spurious
> >     testsuite failures involving botched time stamps.
> >     * NEWS: Update.
> >     Reports by Ludovic Courtès, Peter Breitenlohner, and others.
> 
> I applied the patch yesterday for the Hydra builder, so builds done with
> the tarball from build #600499 or later have the patch:
> 
>   http://hydra.nixos.org/jobset/gnu/automake-master/all
> 
> I’ll let you analyze the results.  ;-)

aclocal9.test, acloca10.test, aclocal18.test failures:
Interesting condition I had not though about: the configure does not use
AC_OUTPUT, thus never builds config.status, thus AC_CONFIG_COMMANDS_PRE
commands are never evaluated.  My bug.  I'm adding a bunch of $sleep to
the test which hopefully fixes this, otherwise we need to rethink
caching semantics.

I noted a failure in aclocal8.test back then, but can't see it now.
Let's ignore that until we see it again.

> Besides, python11.test fails, whereas it should just be skipped, I think
> (Python isn’t available in the build environment.)

python11.test is in the same boat as the rest.  This one can do with
--force though, we are not testing aclocal-specific semantics.

In addition to the previously posted patch, I'm thus applyin the one
below, and merging both into master.  master should be regression-free
now, the testsuite should hopefully pass consistently.  I'm interested
in any remaining failures, spurious or not.

Thanks,
Ralf

    tests: avoid running into timing issues due to sanity change.
    
    * tests/acloca10.test, tests/acloca18.test, tests/aclocal9.test:
    Insert strategic sleep before aclocal reruns, to ensure files
    are newer.
    * tests/python11.test: Use --force for repeated autotools runs.
    Reports from the NixOS Hydra build daemon via Ludovic Courtès.

diff --git a/tests/acloca10.test b/tests/acloca10.test
index 7716f2a..c89e80c 100755
--- a/tests/acloca10.test
+++ b/tests/acloca10.test
@@ -1,5 +1,5 @@
 #! /bin/sh
-# Copyright (C) 2003, 2005  Free Software Foundation, Inc.
+# Copyright (C) 2003, 2005, 2010 Free Software Foundation, Inc.
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -55,6 +55,7 @@ grep MACRO3 aclocal.m4
 test ! -f m4_1/macro.m4
 test ! -f m4_2/macro.m4
 
+$sleep
 $ACLOCAL -I m4_2 -I m4_1
 $AUTOCONF
 ./configure
@@ -65,6 +66,7 @@ grep MACRO3 aclocal.m4
 test ! -f m4_1/macro.m4
 test ! -f m4_2/macro.m4
 
+$sleep
 ACLOCAL_TESTSUITE_FLAGS='-I m4_1 -I m4_2'
 $ACLOCAL --install
 $AUTOCONF
@@ -77,6 +79,7 @@ test -f m4_1/macro.m4
 test ! -f m4_2/macro.m4
 cp aclocal.m4 copy.m4
 
+$sleep
 echo '#GREPME' >>dirlist-test/macro.m4
 $ACLOCAL --install
 $AUTOCONF
diff --git a/tests/acloca18.test b/tests/acloca18.test
index 22feb23..4acf063 100755
--- a/tests/acloca18.test
+++ b/tests/acloca18.test
@@ -1,5 +1,5 @@
 #! /bin/sh
-# Copyright (C) 2005, 2006, 2008  Free Software Foundation, Inc.
+# Copyright (C) 2005, 2006, 2008, 2010 Free Software Foundation, Inc.
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -62,6 +62,7 @@ $AUTOCONF
 grep macro11 foo
 grep macro21 foo
 
+$sleep
 rm -f foo
 $ACLOCAL --install
 $AUTOCONF
@@ -69,6 +70,7 @@ $AUTOCONF
 grep macro12 foo
 grep macro23 foo
 
+$sleep
 ACLOCAL_TESTSUITE_FLAGS='-I 4 -I 1 -I 2 -I 3'
 rm -f foo
 $ACLOCAL --install --dry-run
@@ -77,6 +79,7 @@ $AUTOCONF
 grep macro12 foo
 grep macro23 foo
 
+$sleep
 rm -f foo
 $ACLOCAL --install
 $AUTOCONF
@@ -84,11 +87,13 @@ $AUTOCONF
 grep macro14 foo
 grep macro23 foo
 
+$sleep
 ACLOCAL_TESTSUITE_FLAGS='-I 4 -I 1 -I 2'
 rm -f foo
 $ACLOCAL --install 2>stderr && { cat stderr >&2; Exit 1; }
 grep AM_MACRO2 stderr
 
+$sleep
 ACLOCAL_TESTSUITE_FLAGS='-I 4 -I 1'
 rm -f foo
 $ACLOCAL --install
@@ -98,6 +103,7 @@ grep macro14 foo
 grep macro21 foo
 
 
+$sleep
 mkdir dirlist-test
 cat >dirlist-test/m1.m4 <<EOF
 #serial 456
diff --git a/tests/aclocal9.test b/tests/aclocal9.test
index 325f6ae..a9657fc 100755
--- a/tests/aclocal9.test
+++ b/tests/aclocal9.test
@@ -1,5 +1,5 @@
 #! /bin/sh
-# Copyright (C) 2003  Free Software Foundation, Inc.
+# Copyright (C) 2003, 2010 Free Software Foundation, Inc.
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -42,6 +42,8 @@ $AUTOCONF
 grep macro11 foo
 grep macro21 foo
 
+$sleep
+
 $ACLOCAL -I m4_2 -I m4_1
 $AUTOCONF
 ./configure
diff --git a/tests/python11.test b/tests/python11.test
index 0f15f37..9163aac 100755
--- a/tests/python11.test
+++ b/tests/python11.test
@@ -1,5 +1,5 @@
 #! /bin/sh
-# Copyright (C) 2004, 2006, 2008  Free Software Foundation, Inc.
+# Copyright (C) 2004, 2006, 2008, 2010 Free Software Foundation, Inc.
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -44,8 +44,8 @@ grep 'no suitable Python interpreter found' stderr
 
 sed 's/AM_PATH_PYTHON/AM_PATH_PYTHON(,,:)/' configure.in >configure.int
 mv -f configure.int configure.in
-$ACLOCAL
-$AUTOCONF
+$ACLOCAL --force
+$AUTOCONF --force
 # This one should define PYTHON as : and exit succesfully
 ./configure