Re: automake -vs- huge projects (1st patch)

[Alexandre Duret-Lutz]

>>> "adl" == Alexandre Duret-Lutz <address@hidden> writes:

[...]
 adl> 1999-11-22  Tom Tromey  <address@hidden>
 adl> 
 adl>         * automake.in (handle_single_transform_list): Generate explicit
 adl>         rule for subdir objects.  Fixes new addition to subobj.test.
[...]
 adl> The other side of the coin is that dependency tracking will not
 adl> work anymore, because the dependency stuff for subdir/X.o should
 adl> go into subdir/.deps/X.Po but the default suffix rule will put
 adl> it in ./.deps/subdir/X.Po.
[...]
 Tom> Hmm, maybe this is the issue from way back.

 adl> I'm not sure about this.  In the past dependencies were put in
 adl> .deps/subdir/X.Po (i.e. where the suffix rule--at least
 adl> today--would put them).  You moved them to subdir/.deps/X.Po on
 adl> 2002-01-20 for PR/224.

I've just noticed that the above change of 1999-11-22 occurs
just after the merge of user-dep-gen-branch.  So perhaps you
were right to think the issue was related to dependency
tracking...

Anyway, here is a first patch that disables explicit rules for
subdir-objects without per-target flags.

Random notes:
  - folding the depfile&tmpdepfile computation into depcomp shorten
    the "screen garbage" resulting from the invocation of depcomp
    by one line.  This applies to all non-fastdep compilation,
    not only subdir-objects.
  - computing $depbase on-the-fly in fastdep rules adds one line
    of output, so I've arranged things so that it is only used
    when subdir-objects is set.  
  - the test suite passes with GNU Make, and I've run the 14 tests
    that uses subdir-objects with BSD Make (will start a full
    check momently).  Apart from this I have not yet tested this
    on a real project.  I'm hoping you can test this on libgcj.
  - If you omit the NEWS chunk, the patch should also apply to
    automake-1.8 vanilla.

-- 
Alexandre Duret-Lutz

2003-12-28  Alexandre Duret-Lutz  <address@hidden>

        * automake.in (handle_languages): Define %DEPBASE% conditionally
        on subdir-objects.  Define SUBDIROBJ.  Do not clean *_.c files
        here ...
        (lang_c_finish): ... do it here.
        (handle_single_transform_list): Do not output specific rules for
        subdir-objects files which are not renamed.  This should reduce
        the size of Makefiles with lots of subdirectory sources.
        * lib/depcomp: Use $DEPDIR to compute dependency directories.
        * lib/am/depend2.am (%EXT%.o, %EXT%.obj, %EXT%.lo): Adjust call
        to depcomp.  Compute depbase on-the-fly in generic fastdep rules
        for subdir-objects.
        * tests/ansi9.test: Do not grep for an explicit rule that
        we no longer expect.  Really run $MAKE to make sure the chain
        of rules works.
        * tests/yacc5.test: Do not grep for an explicit rule that
        we no longer expect.  Adjust to use set -e.

Index: NEWS
===================================================================
RCS file: /cvs/automake/automake/NEWS,v
retrieving revision 1.258
diff -u -r1.258 NEWS
--- NEWS        26 Dec 2003 03:57:59 -0000      1.258
+++ NEWS        28 Dec 2003 01:54:04 -0000
@@ -1,6 +1,9 @@
 New in 1.8a:
 
-* Nothing yet.
+* Inference rules are used to compile sources in subdirectories when the
+  `subdir-objects' option is used and no per-target flags are used.  This
+  should reduce the size of some projects a lot, because Automake used to
+  output an explicit rule for each such object in the past.
 
 New in 1.8:
 
Index: automake.in
===================================================================
RCS file: /cvs/automake/automake/automake.in,v
retrieving revision 1.1528
diff -u -r1.1528 automake.in
--- automake.in 27 Dec 2003 15:39:45 -0000      1.1528
+++ automake.in 28 Dec 2003 01:54:09 -0000
@@ -1087,6 +1087,27 @@
            # This is not used by depend2.am.
            my $der_ext = (&{$lang->output_extensions} ($ext))[0];
 
+           # When we output an inference rule like `.c.o:' we
+           # have two cases to consider: either subdir-objects
+           # is used, or it is not.
+           #
+           # In the latter case the rule is used to build objects
+           # in the current directory, and dependencies always
+           # go into `./$(DEPDIR)/'.  We can hard-code this value.
+           #
+           # In the former case the rule can be used to build
+           # objects in sub-directories too.  Dependencies should
+           # go into the appropriate sub-directories, e.g.,
+           # `sub/$(DEPDIR)/'.  The value of this directory
+           # need the be computed on-the-fly.
+           #
+           # DEPBASE holds the name of this directory, plus the
+           # basename part of the object file (extensions Po, TPo,
+           # Plo, TPlo will be added later as appropriate).  It is
+           # either hardcoded, or a shell variable (`$depbase') that
+           # will be computed by the rule.
+           my $depbase =
+             option ('subdir-objects') ? '$$depbase' : '$(DEPDIR)/$*';
            $output_rules .=
              file_contents ($rule_file,
                             new Automake::Location,
@@ -1095,11 +1116,7 @@
 
                             'DERIVED-EXT' => $der_ext,
 
-                            # In this situation we know that the
-                            # object is in this directory, so
-                            # $(DEPDIR) is the correct location for
-                            # dependencies.
-                            DEPBASE   => '$(DEPDIR)/$*',
+                            DEPBASE   => $depbase,
                             BASE      => '$*',
                             SOURCE    => '$<',
                             OBJ       => '$@',
@@ -1108,7 +1125,8 @@
 
                             COMPILE   => '$(' . $lang->compiler . ')',
                             LTCOMPILE => '$(LT' . $lang->compiler . ')',
-                            -o        => $output_flag);
+                            -o        => $output_flag,
+                            SUBDIROBJ => !! option 'subdir-objects');
        }
 
        # Now include code for each specially handled object with this
@@ -1209,11 +1227,6 @@
                  if $source !~ /\$U/;
 
                (my $source_ = $source) =~ s/\$U/_/g;
-               # Explicitly clean the _.c files if they are in
-               # a subdirectory. (In the current directory they get
-               # erased by a `rm -f *_.c' rule.)
-               $clean_files{$source_} = MOSTLY_CLEAN
-                 if $objdir ne '.';
                # Output an additional rule if _.c and .c are not in
                # the same directory.  (_.c is always in $objdir.)
                if ($objdir ne $srcdir)
@@ -1531,13 +1544,37 @@
                 $object = $directory . '/' . $object;
             }
 
-            # If doing dependency tracking, then we can't print
-            # the rule.  If we have a subdir object, we need to
-            # generate an explicit rule.  Actually, in any case
-            # where the object is not in `.' we need a special
-            # rule.  The per-object rules in this case are
-            # generated later, by handle_languages.
-            if ($renamed || $directory ne '')
+           # If the object file has been renamed (because per-target
+           # flags are used) we cannot compile the file with an
+           # inference rule: we need an explicit rule.
+           #
+           # If the source is in a subdirectory and the object is in
+           # the current directory, we also need an explicit rule.
+           #
+           # If both source and object files are in a subdirectory
+           # (this happens when the subdir-objects option is used),
+           # then the inference will work.
+           #
+           # The latter case deserves a historical note.  When the
+           # subdir-objects option was added on 1999-04-11 it was
+           # thought that inferences rules would work for
+           # subdirectory objects too.  Later, on 1999-11-22,
+           # automake was changed to output explicit rules even for
+           # subdir-objects.  Nobody remembers why, but this occured
+           # soon after the merge of the user-dep-gen-branch so it
+           # might be related.  In late 2003 people complained about
+           # the size of the generated Makefile.ins (libgcj, with
+           # 2200+ subdir objects was reported to have a 9MB
+           # Makefile), so we now rely on inference rules again.
+           # Maybe we'll run across the same issue as in the past,
+           # but at least this time we can document it.  However since
+           # dependency tracking has evolved it is possible that
+           # our old problem no longer exists.
+           # Using inference rules for subdir-objects has been tested
+           # with GNU make, Solaris make, Ultrix make, BSD make,
+           # HP-UX make, and OSF1 make successfully.
+            if ($renamed ||
+               ($directory ne '' && ! option 'subdir-objects'))
             {
                 my $obj_sans_ext = substr ($object, 0,
                                           - length ($this_obj_ext));
@@ -4935,6 +4972,12 @@
            push (@objects, $base . '_.$(OBJEXT)');
            push (@objects, $base . '_.lo')
              if var ('LIBTOOL');
+
+           # Explicitly clean the _.c files if they are in a
+           # subdirectory. (In the current directory they get erased
+           # by a `rm -f *_.c' rule.)
+           $clean_files{$base . "_.c"} = MOSTLY_CLEAN
+             if dirname ($base) ne '.';
        }
 
        # Make all _.o (and _.lo) files depend on ansi2knr.
Index: lib/depcomp
===================================================================
RCS file: /cvs/automake/automake/lib/depcomp,v
retrieving revision 1.47
diff -u -r1.47 depcomp
--- lib/depcomp 9 Nov 2003 00:10:50 -0000       1.47
+++ lib/depcomp 28 Dec 2003 01:54:09 -0000
@@ -1,7 +1,7 @@
 #! /bin/sh
 # depcomp - compile a program generating dependencies as side-effects
 
-scriptversion=2003-11-08.23
+scriptversion=2003-12-27.19
 
 # Copyright (C) 1999, 2000, 2003 Free Software Foundation, Inc.
 
@@ -43,6 +43,7 @@
   depmode     Dependency tracking mode.
   source      Source file read by `PROGRAMS ARGS'.
   object      Object file output by `PROGRAMS ARGS'.
+  DEPDIR      directory where to store dependencies.
   depfile     Dependency file to output.
   tmpdepfile  Temporary file to use when outputing dependencies.
   libtool     Whether libtool is used (yes/no).
@@ -61,7 +62,6 @@
   echo "depcomp: Variables source, object and depmode must be set" 1>&2
   exit 1
 fi
-# `libtool' can also be set to `yes' or `no'.
 
 if test -z "$depfile"; then
    base=`echo "$object" | sed -e 's,^.*/,,' -e 's,\.\([^.]*\)$,.P\1,'`
@@ -69,8 +69,7 @@
    if test "$dir" = "$object"; then
       dir=
    fi
-   # FIXME: should be _deps on DOS.
-   depfile="$dir.deps/$base"
+   depfile="$dir$DEPDIR/$base"
 fi
 
 tmpdepfile=${tmpdepfile-`echo "$depfile" | sed 's/\.\([^.]*\)$/.T\1/'`}
Index: lib/am/depend2.am
===================================================================
RCS file: /cvs/automake/automake/lib/am/depend2.am,v
retrieving revision 1.56
diff -u -r1.56 depend2.am
--- lib/am/depend2.am   8 Nov 2003 14:03:35 -0000       1.56
+++ lib/am/depend2.am   28 Dec 2003 01:54:09 -0000
@@ -56,7 +56,8 @@
 ## compile rules so that they are output on a single line (instead of 5)
 ## would be a good compromise.  Actually we use two line rather than one,
 ## because this way %SOURCE% is always located at the end of the first
-## line and is therefore easier to spot.
+## line and is therefore easier to spot.  (We need an extra line when
+## depbase is used.)
 
 ?GENERIC?%EXT%.o:
 ?!GENERIC?%OBJ%: %SOURCE%
@@ -64,13 +65,13 @@
 ## In fast-dep mode, we can always use -o.
 ## For non-suffix rules, we must emulate a VPATH search on %SOURCE%.
 ?!GENERIC?     if %COMPILE% -MT %OBJ% -MD -MP -MF "%DEPBASE%.Tpo" %-c% -o 
%OBJ% `test -f '%SOURCE%' || echo '$(srcdir)/'`%SOURCE%; \
+?SUBDIROBJ??GENERIC?   depbase=`echo %OBJ% | sed 
's|[^/]*$$|$(DEPDIR)/&|;s|\.o$$||'`; \
 ?GENERIC?      if %COMPILE% -MT %OBJ% -MD -MP -MF "%DEPBASE%.Tpo" %-c% -o 
%OBJ% %SOURCE%; \
        then mv -f "%DEPBASE%.Tpo" "%DEPBASE%.Po"; else rm -f "%DEPBASE%.Tpo"; 
exit 1; fi
 else !%FASTDEP%
 if %AMDEP%
        source='%SOURCE%' object='%OBJ%' libtool=no @AMDEPBACKSLASH@
-       depfile='%DEPBASE%.Po' tmpdepfile='%DEPBASE%.TPo' @AMDEPBACKSLASH@
-       $(%FPFX%DEPMODE) $(depcomp) @AMDEPBACKSLASH@
+       DEPDIR=$(DEPDIR) $(%FPFX%DEPMODE) $(depcomp) @AMDEPBACKSLASH@
 endif %AMDEP%
 if %?GENERIC%
 ?-o?   %COMPILE% %-c% %-o% %OBJ% %SOURCE%
@@ -88,13 +89,13 @@
 ## In fast-dep mode, we can always use -o.
 ## For non-suffix rules, we must emulate a VPATH search on %SOURCE%.
 ?!GENERIC?     if %COMPILE% -MT %OBJOBJ% -MD -MP -MF "%DEPBASE%.Tpo" %-c% -o 
%OBJOBJ% `if test -f '%SOURCE%'; then $(CYGPATH_W) '%SOURCE%'; else 
$(CYGPATH_W) '$(srcdir)/%SOURCE%'; fi`; \
+?SUBDIROBJ??GENERIC?   depbase=`echo %OBJ% | sed 
's|[^/]*$$|$(DEPDIR)/&|;s|\.obj$$||'`; \
 ?GENERIC?      if %COMPILE% -MT %OBJOBJ% -MD -MP -MF "%DEPBASE%.Tpo" %-c% -o 
%OBJOBJ% `$(CYGPATH_W) '%SOURCE%'`; \
        then mv -f "%DEPBASE%.Tpo" "%DEPBASE%.Po"; else rm -f "%DEPBASE%.Tpo"; 
exit 1; fi
 else !%FASTDEP%
 if %AMDEP%
        source='%SOURCE%' object='%OBJOBJ%' libtool=no @AMDEPBACKSLASH@
-       depfile='%DEPBASE%.Po' tmpdepfile='%DEPBASE%.TPo' @AMDEPBACKSLASH@
-       $(%FPFX%DEPMODE) $(depcomp) @AMDEPBACKSLASH@
+       DEPDIR=$(DEPDIR) $(%FPFX%DEPMODE) $(depcomp) @AMDEPBACKSLASH@
 endif %AMDEP%
 if %?GENERIC%
 ?-o?   %COMPILE% %-c% %-o% %OBJOBJ% `$(CYGPATH_W) '%SOURCE%'`
@@ -112,15 +113,14 @@
 if %FASTDEP%
 ## In fast-dep mode, we can always use -o.
 ## For non-suffix rules, we must emulate a VPATH search on %SOURCE%.
-?GENERIC?      if %LTCOMPILE% -MT %LTOBJ% -MD -MP -MF "%DEPBASE%.Tpo" %-c% -o 
%LTOBJ% %SOURCE%; \
-## For non-suffix rules, we must emulate a VPATH search on %SOURCE%.
 ?!GENERIC?     if %LTCOMPILE% -MT %LTOBJ% -MD -MP -MF "%DEPBASE%.Tpo" %-c% -o 
%LTOBJ% `test -f '%SOURCE%' || echo '$(srcdir)/'`%SOURCE%; \
+?SUBDIROBJ??GENERIC?   depbase=`echo %OBJ% | sed 
's|[^/]*$$|$(DEPDIR)/&|;s|\.lo$$||'`; \
+?GENERIC?      if %LTCOMPILE% -MT %LTOBJ% -MD -MP -MF "%DEPBASE%.Tpo" %-c% -o 
%LTOBJ% %SOURCE%; \
        then mv -f "%DEPBASE%.Tpo" "%DEPBASE%.Plo"; else rm -f "%DEPBASE%.Tpo"; 
exit 1; fi
 else !%FASTDEP%
 if %AMDEP%
        source='%SOURCE%' object='%LTOBJ%' libtool=yes @AMDEPBACKSLASH@
-       depfile='%DEPBASE%.Plo' tmpdepfile='%DEPBASE%.TPlo' @AMDEPBACKSLASH@
-       $(%FPFX%DEPMODE) $(depcomp) @AMDEPBACKSLASH@
+       DEPDIR=$(DEPDIR) $(%FPFX%DEPMODE) $(depcomp) @AMDEPBACKSLASH@
 endif %AMDEP%
 ## We can always use `-o' with Libtool.
 ?GENERIC?      %LTCOMPILE% %-c% -o %LTOBJ% %SOURCE%
Index: tests/ansi9.test
===================================================================
RCS file: /cvs/automake/automake/tests/ansi9.test,v
retrieving revision 1.2
diff -u -r1.2 ansi9.test
--- tests/ansi9.test    14 Nov 2003 21:25:58 -0000      1.2
+++ tests/ansi9.test    28 Dec 2003 01:54:09 -0000
@@ -1,5 +1,5 @@
 #! /bin/sh
-# Copyright (C) 2002  Free Software Foundation, Inc.
+# Copyright (C) 2002, 2003  Free Software Foundation, Inc.
 #
 # This file is part of GNU Automake.
 #
@@ -22,6 +22,7 @@
 # can still be deansified.
 # Report from Paul D. Smith.
 
+required=gcc
 . ./defs || exit 1
 
 set -e
@@ -30,6 +31,7 @@
 AC_PROG_CC
 AM_PROG_CC_C_O
 AM_C_PROTOTYPES
+AC_OUTPUT
 EOF
 
 cat > Makefile.am << 'END'
@@ -38,10 +40,29 @@
 loadavg_SOURCES = loadavg.c
 loadavg_CFLAGS = -DTEST
 sub_sub_SOURCES = sub/sub.c
+# Force ansi2knr's use, regardless of the compiler.
+U=_
+ANSI2KNR=./ansi2knr
 END
 
+cat > loadavg.c << 'END'
+int
+main ()
+{
+   return 0;
+}
+END
+
+mkdir sub
+cp loadavg.c sub/sub.c
+
 $ACLOCAL
-$AUTOMAKE --add-missing
+$AUTOCONF
+$AUTOMAKE --add-missing -Wno-override
 
 $FGREP 'loadavg-loadavg$U.o: loadavg$U.c' Makefile.in
-$FGREP 'sub/sub$U.o: sub/sub$U.c' Makefile.in
+# The following rule should not exists, because the
+# default .o.c: inference rule is enough.
+$FGREP 'sub/sub$U.o: sub/sub$U.c' Makefile.in && exit 1
+./configure
+$MAKE sub/sub_.c
Index: tests/yacc5.test
===================================================================
RCS file: /cvs/automake/automake/tests/yacc5.test,v
retrieving revision 1.6
diff -u -r1.6 yacc5.test
--- tests/yacc5.test    14 Nov 2003 21:26:01 -0000      1.6
+++ tests/yacc5.test    28 Dec 2003 01:54:09 -0000
@@ -1,5 +1,5 @@
 #! /bin/sh
-# Copyright (C) 2001, 2002  Free Software Foundation, Inc.
+# Copyright (C) 2001, 2002, 2003  Free Software Foundation, Inc.
 #
 # This file is part of GNU Automake.
 #
@@ -23,6 +23,8 @@
 
 . ./defs || exit 1
 
+set -e
+
 cat > configure.in << 'END'
 AC_INIT
 AM_INIT_AUTOMAKE(nonesuch, nonesuch)
@@ -41,10 +43,10 @@
 
 : > sub/maude.y
 
-$ACLOCAL || exit 1
-$AUTOMAKE -a || exit 1
+$ACLOCAL
+$AUTOMAKE -a
 
-grep '^maude\.c:' Makefile.in || exit 1
+grep '^maude\.c:' Makefile.in
 
 
 ## Try again with subdir-objects.
@@ -55,10 +57,11 @@
 maude_SOURCES = sub/maude.y
 END
 
-$ACLOCAL || exit 1
-$AUTOMAKE -a || exit 1
+$ACLOCAL
+$AUTOMAKE -a
 
-grep '^sub/maude\.c:' Makefile.in || exit 1
+# No rule needed, the default .y.c: inference rule is enough.
+grep '^sub/maude\.c:' Makefile.in && exit 1
 
 
 ## Try again with per-exe flags.
@@ -80,6 +83,4 @@
 grep 'maudec' Makefile.in && exit 1
 
 # Make sure the .o file is required.
-grep '^am_maude_OBJECTS.*maude' Makefile.in || exit 1
-
-exit 0
+grep '^am_maude_OBJECTS.*maude' Makefile.in