Re: running tests in parallel

[Akim Demaille]

>>> "DF" == David Fang <address@hidden> writes:

 >> > It would be nice if it were possible to run tests in parallel using
 >> > automake.  Waiting for tests to complete on a 4-way box while they
 >> > are run
 >> > on a single CPU is annoying.  As most 'make' implementations already
 >> > support running in parallel (-j), automake could just utilize this
 >> > functionality to runt tests in parallel.
 >>
 >> I very much agree.

 > Hi all,

 > I use a simple recipe for parallelizing tests without any modification to
 > automake, but it may require some semi-automatic dependence tracking to
 > get right.

That's also also true for my proposal.  But in order to support
multiple test suites in a single directory, in order to support a
wider set of possible extensions for tests (in particular the
troublesome $(EXEEXT) that might be empty, which is a problem to write
traditional-style generic make rules), it would be nicer to have some
assistance from automake.pl.

The following patch shows that there can be (almost) no impact on
automake.pl: I moved Automake's test suite to my check.mk.  The test
now run in parallel, and generate in a single run both test-suite.log
and test-suite.html.  To get a really nice test-suite.html, it's
better to use functions such as run_AUTOMAKE instead of relying on set
-x and set -e.  But that's to polish the result, the test-suite.log is
perfect as it is.  I also changed the clean rules, because I think it
is handy to introduce the check-clean family (CHECK_CLEANFILES,
check-clean-local) in addition to the *-clean families.

BTW, I forgot to emphasize that since all the tests have a log
generated while they run, one can read/send the logs individually,
even before the test suite finishes.

For some reason, make check fails, was this known?  The failures are
most bizarre...  I don't feel like investigating, should I?  Is this
an Autoconf/M4 version problem?


Here are the changes to Automake.

Index: ChangeLog
from  Akim Demaille  <address@hidden>

        * tests/defs.in (VERBOSE): Remove its hanlding: we now are always
        verbose.
        (tmpSubDir): Now a variable depending on $me to avoid collision in
        parallel checks.
        (subsection, tab, pre): New.
        (run_AUTOMAKE): Show how using rst function can improve the HTML logs.
        * tests/Makefile.am: Include check.mk.
        (distclean-local): Rename to...
        (check-clean-local): Adjust to $tmpSubDir.

Index: tests/Makefile.am
===================================================================
RCS file: /cvs/automake/automake/tests/Makefile.am,v
retrieving revision 1.617
diff -u -u -r1.617 Makefile.am
--- tests/Makefile.am 16 Oct 2006 05:24:17 -0000 1.617
+++ tests/Makefile.am 9 Feb 2007 14:57:46 -0000
@@ -617,5 +617,8 @@
 # Each test case depends on defs, aclocal, and automake.
 check_SCRIPTS = defs aclocal-$(APIVERSION) automake-$(APIVERSION)
 
-distclean-local:
-       -rm -rf testSubDir
+check-clean-local:
+       -chmod -R a+rwx $(TESTS:.test=.dir) >/dev/null 2>&1
+       -rm -rf $(TESTS:.test=.dir) >/dev/null 2>&1
+
+include check.mk
Index: tests/defs.in
===================================================================
RCS file: /cvs/automake/automake/tests/defs.in,v
retrieving revision 1.42
diff -u -u -r1.42 defs.in
--- tests/defs.in 10 Jan 2007 17:57:24 -0000 1.42
+++ tests/defs.in 9 Feb 2007 14:57:46 -0000
@@ -46,7 +46,6 @@
 
 # If srcdir is not set, then we are not running from `make check', be verbose.
 if test -z "$srcdir"; then
-   VERBOSE=x
    # compute $srcdir.
    srcdir=`echo "$0" | sed -e 's,/[^\\/]*$,,'`
    test $srcdir = $0 && srcdir=.
@@ -60,12 +59,32 @@
 
 me=`echo "$0" | sed -e 's,.*[\\/],,;s/\.test$//'`
 
-# See how redirections should work.  User can set VERBOSE to see all
-# output.
-test -z "$VERBOSE" && {
-   exec > /dev/null 2>&1
+
+## --------------- ##
+## Rst functions.  ##
+## --------------- ##
+
+subsection ()
+{
+  echo "$@" | sed 'p;s/./-/g;p;g'
+}
+
+tab ()
+{
+  sed -e 's/^/ /' "$@"
 }
 
+pre ()
+{
+  if test -s "$2"; then
+    echo "$1::"
+    echo
+    tab "$2"
+    echo
+  fi
+}
+
+
 # Make sure we override the user shell.
 SHELL='@SHELL@'
 export SHELL
@@ -228,16 +247,17 @@
     ;;
 esac
 
-chmod -R a+rwx testSubDir > /dev/null 2>&1
-rm -rf testSubDir > /dev/null 2>&1
-mkdir testSubDir
+testSubDir=$me.dir
+chmod -R a+rwx $testSubDir > /dev/null 2>&1
+rm -rf $testSubDir > /dev/null 2>&1
+mkdir $testSubDir
 
 # Copy in some files we need.
 for file in install-sh missing depcomp; do
-   cp $srcdir/../lib/$file testSubDir/$file || exit 1
+   cp $srcdir/../lib/$file $testSubDir/$file || exit 1
 done
 
-cd ./testSubDir
+cd ./$testSubDir
 
 # Build appropriate environment in test directory.  Eg create
 # configure.in, touch all necessary files, etc.
@@ -342,9 +362,10 @@
   expected_exitcode=$1
   shift
   exitcode=0
+  subsection "Running $AUTOMAKE $@"
   $AUTOMAKE ${1+"$@"} >stdout 2>stderr || exitcode=$?
-  cat stderr
-  cat stdout
+  pre "Automake's stderr" stderr
+  pre "Automake's stderr" stderr
   test $exitcode = $expected_exitcode || exit 1
 }
 
@@ -357,10 +378,5 @@
   AUTOMAKE_run 1 ${1+"$@"}
 }
 
-# Turn on shell traces when VERBOSE is set.
-if test -n "$VERBOSE"; then
-  set -x
-else
-  :
-fi
+set -x
 pwd


Here is check.mk.

check.mk
Description: Binary data

Here is test-suite.log.

==============================================
   GNU Automake 1.10a: tests/test-suite.log   
==============================================

5 of 608 tests failed.  (43 tests were not run).  

.. contents:: :depth: 2


FAIL: acloca10.test (exit: 1)
=============================

/home/akim/src/am/_build/tests:/home/akim/bin:/usr/local/bin:/usr/local/vcs/bin:/usr/local/SmartEiffel/bin:/lrde/beyrouth/admin/lrde/usr/bin:/lrde/beyrouth/admin/lrde/usr/sge/bin/lx24-x86:/usr/bin:/bin:/usr/bin/X11:/usr/games:/usr/X11R6/bin
=== Running test ./../../tests/acloca10.test
++ pwd
/home/akim/src/am/_build/tests/acloca10.dir
+ set -e
+ cat
+ mkdir m4_1 m4_2 dirlist-test
+ cat
+ cat
+ cat
+ aclocal-1.10a -Werror -I m4_1 -I m4_2
+ autoconf
+ ./configure
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
+ grep macro11 foo
macro11
+ grep macro21 foo
macro21
+ grep macro3 foo
macro3
+ grep MACRO3 aclocal.m4
AC_DEFUN([MACRO3], [echo macro3 >> foo])
+ test '!' -f m4_1/macro.m4
+ test '!' -f m4_2/macro.m4
+ aclocal-1.10a -Werror -I m4_2 -I m4_1
+ autoconf
+ ./configure
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
+ grep macro12 foo
macro12
+ grep macro21 foo
macro21
+ grep macro3 foo
macro3
+ grep MACRO3 aclocal.m4
AC_DEFUN([MACRO3], [echo macro3 >> foo])
+ test '!' -f m4_1/macro.m4
+ test '!' -f m4_2/macro.m4
+ ACLOCAL_TESTSUITE_FLAGS='-I m4_1 -I m4_2'
+ aclocal-1.10a -Werror --install
aclocal: installing `m4_1/macro.m4' from `dirlist-test/macro.m4'
aclocal: installing `m4_1/amversion.m4' from 
`/home/akim/src/am/_build/../m4/amversion.m4'
aclocal: installing `m4_1/auxdir.m4' from 
`/home/akim/src/am/_build/../m4/auxdir.m4'
aclocal: installing `m4_1/init.m4' from `/home/akim/src/am/_build/../m4/init.m4'
aclocal: installing `m4_1/install-sh.m4' from 
`/home/akim/src/am/_build/../m4/install-sh.m4'
aclocal: installing `m4_1/lead-dot.m4' from 
`/home/akim/src/am/_build/../m4/lead-dot.m4'
aclocal: installing `m4_1/missing.m4' from 
`/home/akim/src/am/_build/../m4/missing.m4'
aclocal: installing `m4_1/mkdirp.m4' from 
`/home/akim/src/am/_build/../m4/mkdirp.m4'
aclocal: installing `m4_1/options.m4' from 
`/home/akim/src/am/_build/../m4/options.m4'
aclocal: installing `m4_1/sanity.m4' from 
`/home/akim/src/am/_build/../m4/sanity.m4'
aclocal: installing `m4_1/strip.m4' from 
`/home/akim/src/am/_build/../m4/strip.m4'
aclocal: installing `m4_1/substnot.m4' from 
`/home/akim/src/am/_build/../m4/substnot.m4'
aclocal: installing `m4_1/tar.m4' from `/home/akim/src/am/_build/../m4/tar.m4'
+ autoconf
autoconf: Undefined macros:
configure.in:3:AC_CONFIG_FILES([Makefile])

FAIL: acloca18.test (exit: 1)
=============================

/home/akim/src/am/_build/tests:/home/akim/bin:/usr/local/bin:/usr/local/vcs/bin:/usr/local/SmartEiffel/bin:/lrde/beyrouth/admin/lrde/usr/bin:/lrde/beyrouth/admin/lrde/usr/sge/bin/lx24-x86:/usr/bin:/bin:/usr/bin/X11:/usr/games:/usr/X11R6/bin
=== Running test ./../../tests/acloca18.test
++ pwd
/home/akim/src/am/_build/tests/acloca18.dir
+ set -e
+ cat
+ mkdir 1 2 3 4
+ cat
+ cat
+ cat
+ cat
+ cat
+ ACLOCAL_TESTSUITE_FLAGS='-I 1 -I 2 -I 3 -I 4'
+ aclocal-1.10a -Werror
+ autoconf
+ ./configure
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
+ grep macro11 foo
macro11
+ grep macro21 foo
macro21
+ rm -f foo
+ aclocal-1.10a -Werror --install
aclocal: installing `1/amversion.m4' from 
`/home/akim/src/am/_build/../m4/amversion.m4'
aclocal: installing `1/auxdir.m4' from 
`/home/akim/src/am/_build/../m4/auxdir.m4'
aclocal: installing `1/init.m4' from `/home/akim/src/am/_build/../m4/init.m4'
aclocal: installing `1/install-sh.m4' from 
`/home/akim/src/am/_build/../m4/install-sh.m4'
aclocal: installing `1/lead-dot.m4' from 
`/home/akim/src/am/_build/../m4/lead-dot.m4'
aclocal: installing `1/missing.m4' from 
`/home/akim/src/am/_build/../m4/missing.m4'
aclocal: installing `1/mkdirp.m4' from 
`/home/akim/src/am/_build/../m4/mkdirp.m4'
aclocal: installing `1/options.m4' from 
`/home/akim/src/am/_build/../m4/options.m4'
aclocal: installing `1/sanity.m4' from 
`/home/akim/src/am/_build/../m4/sanity.m4'
aclocal: installing `1/strip.m4' from `/home/akim/src/am/_build/../m4/strip.m4'
aclocal: installing `1/substnot.m4' from 
`/home/akim/src/am/_build/../m4/substnot.m4'
aclocal: installing `1/tar.m4' from `/home/akim/src/am/_build/../m4/tar.m4'
+ autoconf
autoconf: Undefined macros:
configure.in:3:AC_CONFIG_FILES([Makefile])

FAIL: acloca20.test (exit: 1)
=============================

/home/akim/src/am/_build/tests:/home/akim/bin:/usr/local/bin:/usr/local/vcs/bin:/usr/local/SmartEiffel/bin:/lrde/beyrouth/admin/lrde/usr/bin:/lrde/beyrouth/admin/lrde/usr/sge/bin/lx24-x86:/usr/bin:/bin:/usr/bin/X11:/usr/games:/usr/X11R6/bin
=== Running test ./../../tests/acloca20.test
++ pwd
/home/akim/src/am/_build/tests/acloca20.dir
+ set -e
+ cat
+ cat
+ cat
+ autoconf
autoconf: Undefined macros:
configure.in:4:AC_CONFIG_FILES([doit], [chmod +x doit])

FAIL: missing5.test (exit: 1)
=============================

/home/akim/src/am/_build/tests:/home/akim/bin:/usr/local/bin:/usr/local/vcs/bin:/usr/local/SmartEiffel/bin:/lrde/beyrouth/admin/lrde/usr/bin:/lrde/beyrouth/admin/lrde/usr/sge/bin/lx24-x86:/usr/bin:/bin:/usr/bin/X11:/usr/games:/usr/X11R6/bin
=== Running test ./../../tests/missing5.test
++ pwd
/home/akim/src/am/_build/tests/missing5.dir
+ set -e
+ needed_tools='chmod find sed test touch'
++ echo chmod find sed test touch
++ sed 's/ /, /g'
+ needed_tools_csep='chmod, find, sed, test, touch'
+ cat
+ for tool in '$needed_tools'
+ cat
+ for tool in '$needed_tools'
+ cat
+ for tool in '$needed_tools'
+ cat
+ for tool in '$needed_tools'
+ cat
+ for tool in '$needed_tools'
+ cat
+ autoconf
autoconf: Undefined macros:
configure.in:3:    [AC_PATH_PROG(tool, tool, [false])
configure.in:4:     AC_CONFIG_FILES(tool, chmod +x tool)

FAIL: obsolete.test (exit: 1)
=============================

/home/akim/src/am/_build/tests:/home/akim/bin:/usr/local/bin:/usr/local/vcs/bin:/usr/local/SmartEiffel/bin:/lrde/beyrouth/admin/lrde/usr/bin:/lrde/beyrouth/admin/lrde/usr/sge/bin/lx24-x86:/usr/bin:/bin:/usr/bin/X11:/usr/games:/usr/X11R6/bin
=== Running test ./../../tests/obsolete.test
++ pwd
/home/akim/src/am/_build/tests/obsolete.dir
+ cat
+ /usr/bin/perl -ne '/AU_DEFUN\(\[(\w+)\]/ && print "$1\n"' 
/home/akim/src/am/tests/../m4/obsolete.m4
+ cat obs
+ /usr/bin/perl -ne 'chomp; print "grep $_ stderr || exit 1\n"; ' obs
+ /usr/bin/perl -ne 'chomp; print "grep $_ configure.in && exit 1\n"; ' obs
++ cat configure.in
++ wc -l
+ test 14 -gt 1
+ aclocal-1.10a -Werror
+ autoconf -Wobsolete
+ . ./obs.1
++ grep AC_FEATURE_CTYPE stderr
configure.in:2: warning: The macro `AC_FEATURE_CTYPE' is obsolete.
aclocal.m4:30: AC_FEATURE_CTYPE is expanded from...
++ grep AC_FEATURE_ERRNO stderr
configure.in:3: warning: The macro `AC_FEATURE_ERRNO' is obsolete.
aclocal.m4:31: AC_FEATURE_ERRNO is expanded from...
++ grep AM_CYGWIN32 stderr
configure.in:4: warning: The macro `AM_CYGWIN32' is obsolete.
aclocal.m4:32: AM_CYGWIN32 is expanded from...
++ grep AM_EXEEXT stderr
configure.in:5: warning: The macro `AM_EXEEXT' is obsolete.
aclocal.m4:33: AM_EXEEXT is expanded from...
++ grep AM_FUNC_MKTIME stderr
configure.in:6: warning: The macro `AM_FUNC_MKTIME' is obsolete.
aclocal.m4:34: AM_FUNC_MKTIME is expanded from...
++ grep AM_HEADER_TIOCGWINSZ_NEEDS_SYS_IOCTL stderr
configure.in:7: warning: The macro `AM_HEADER_TIOCGWINSZ_NEEDS_SYS_IOCTL' is 
obsolete.
aclocal.m4:35: AM_HEADER_TIOCGWINSZ_NEEDS_SYS_IOCTL is expanded from...
++ grep AM_MINGW32 stderr
configure.in:8: warning: The macro `AM_MINGW32' is obsolete.
aclocal.m4:37: AM_MINGW32 is expanded from...
++ grep AM_PROG_INSTALL stderr
configure.in:9: warning: The macro `AM_PROG_INSTALL' is obsolete.
aclocal.m4:38: AM_PROG_INSTALL is expanded from...
++ grep AM_SANITY_CHECK_CC stderr
configure.in:10: warning: The macro `AM_SANITY_CHECK_CC' is obsolete.
aclocal.m4:39: AM_SANITY_CHECK_CC is expanded from...
++ grep AM_SYS_POSIX_TERMIOS stderr
configure.in:11: warning: The macro `AM_SYS_POSIX_TERMIOS' is obsolete.
aclocal.m4:40: AM_SYS_POSIX_TERMIOS is expanded from...
++ grep fp_FUNC_FNMATCH stderr
configure.in:12: warning: The macro `fp_FUNC_FNMATCH' is obsolete.
aclocal.m4:41: fp_FUNC_FNMATCH is expanded from...
++ grep fp_PROG_INSTALL stderr
configure.in:13: warning: The macro `fp_PROG_INSTALL' is obsolete.
aclocal.m4:42: fp_PROG_INSTALL is expanded from...
++ grep md_TYPE_PTRDIFF_T stderr
configure.in:14: warning: The macro `md_TYPE_PTRDIFF_T' is obsolete.
aclocal.m4:43: md_TYPE_PTRDIFF_T is expanded from...
+ autoupdate
configure.in:4: warning: AC_CYGWIN is obsolete: use AC_CANONICAL_HOST and check 
if $host_os
matches *cygwin*
configure.in:8: warning: AC_MINGW32 is obsolete: use AC_CANONICAL_HOST and 
check if $host_os
matches *mingw32*
+ . ./obs.2
++ grep AC_FEATURE_CTYPE configure.in
++ grep AC_FEATURE_ERRNO configure.in
++ grep AM_CYGWIN32 configure.in
++ grep AM_EXEEXT configure.in
++ grep AM_FUNC_MKTIME configure.in
++ grep AM_HEADER_TIOCGWINSZ_NEEDS_SYS_IOCTL configure.in
++ grep AM_MINGW32 configure.in
++ grep AM_PROG_INSTALL configure.in
++ grep AM_SANITY_CHECK_CC configure.in
++ grep AM_SYS_POSIX_TERMIOS configure.in
++ grep fp_FUNC_FNMATCH configure.in
++ grep fp_PROG_INSTALL configure.in
++ grep md_TYPE_PTRDIFF_T configure.in
+ autoconf
configure.in:24: AC_PROG_CPP was called before AC_PROG_CC
autoconf: Undefined macros:
configure.in:13:AC_FUNC_MKTIME
configure.in:14:AC_HEADER_TIOCGWINSZ
configure.in:15:AC_DIAGNOSE([obsolete],[AC_MINGW32 is obsolete: use 
AC_CANONICAL_HOST and check if $host_os
configure.in:17:AC_CANONICAL_HOST
configure.in:25:AC_SYS_POSIX_TERMIOS
configure.in:28:AC_CHECK_TYPES([ptrdiff_t])
configure.in:4:AC_DIAGNOSE([obsolete],[AC_CYGWIN is obsolete: use 
AC_CANONICAL_HOST and check if $host_os
configure.in:6:AC_CANONICAL_HOST
+ exit 1


And here is a snapshot of my screen at the end of make check -j10.

failures.png
Description: PNG image