Re: [bug-libunistring] Improve parallel build.

[Ralf Wildenhues]

Hi Bruno,

* Bruno Haible wrote on Sun, May 03, 2009 at 06:20:58PM CEST:
> > The missing dependency of config.h upon exported.sh is a bug
> 
> Certainly not. exported.sh gets rebuilt every time './config.status'
> is run, and virtually never changes (because exported.sh.in is so tiny
> that it's not going to be modified in several years). Such a
> dependency would have the effect that each time './config.status'
> is run, config.h is regenerated - which takes 30 seconds of time.

Ah, ok, that makes sense; I wasn't aware of the fact that this was done
on purpose.

> > Timings on a two-way system:
> > [sec]           before  after the patch
> > serial          86      84
> > parallel        63      53
> 
> I have just added a patch that saves even more build time: Not a 20%
> speedup, but a 34% speedup. It's an option --disable-namespacing that
> disables the entire config.h compilations. Additionally, --disable-static
> gives an additional 38% speed increase:
> 
> Times of "make" in wall-clock time:
> [sec]           before    --disable-namespacing    + --disable-static
> serial           128               84                      50
> parallel (-j 4)   91               47                      29
> 
> To make it clear: The namespace cleanliness of the resulting library
> is an *essential* feature for someone who takes a released tarball and
> installs it. It an *optional* feature for someone who modifies the
> source code and builds the library 10 times in a row.

Agreed.

> >     Build intermediate objects using a parallelizable sub-make
> >     invocation.
> 
> Unfortunately, I find this sub-make invocations horrible from a
> maintainability point of view. In CLN, the main Makefile was operating
> through recursive "make" invocations in the same directory, and the new
> maintainer was completely unable to understand and fix problems with it.

I agree that recursive 'make' invocations can be hard to understand,
especially if they are not used for simple directory recursion.
However, the underlying problem here, as I see it, is that the
libunistring build does not have a direct acyclic graph structure,
in that one file name is used for two different purposes: for example,
memcmp2.o is both used as object to be added to the libunistring
library, as well as intermediate file used to find out the names of
exported identifiers.

Without this complication, it would be possible to write clean rules
without the hacks which are used in the current config.h rule.

> In the case of your patch, I additionally fear that some command-line length
> thresholds are exceeded on weird systems (IRIX, Tru64), either in the
> backquote, or as arguments to 'make', or as arguments to exported.sh and
> 'nm'.

Good point.  The current list is probably ok, but it might grow to be a
problem in the future.  The arguments to 'make' could be avoided by
having another custom target

> > GNU make will execute rules containing the string '$(MAKE)'
> > even when in dry-run mode
> 
> Eeek. I did not know about this misfeature. One more reason to avoid
> sub-make invocations other than into subdirectories.

Well, the idea behind this feature is a good one IMHO.  It's only the
fact that several conceptually independent semantics are connected with
this (execution even in dry-run mode, transport of internal make flags).

> > Another nice side effect of this patch is that it is more visible what
> > is going on.
> 
> Good point. Increased transparency is welcome. I add an 'echo' of the
> commands being executed.

Thanks!

> > (This will be even more visible when used in combination 
> > with Automake-1.10c's silent-rules option.)
> 
> Well, I don't see why I should use this silent-rules option.

You certainly don't have to; nobody will make you use it.  But try it
for a while.  Just as more information can sometimes hide information,
reduced information can sometimes reveal bits of information otherwise
hidden from plain view.  I've come to use both methods judiciously, and
have found additional issues that way.

> It provides
> for "prettier" output, but actually reduces transparency. If a user sees
> 
>     CC foobar.c
> 
> followed by an error message of the compiler, how is he supposed to
> know how to produce the preprocessed output? I found out that I could do
> "make -n foobar.o", then it would print me the command line.
> 
> Is that silent-rules option more for the maintainer for the package,
> or more for the user who installs the package without being familiar with
> it? It seems to me that the silent-rules option is equally bad for either
> type of user.

Quoting bits of the Automake manual (git master):

     To enable less verbose build rules, both the developer and the user
     of the package have to take a number of steps.  The developer needs
     to do either of the following:

        * Add the `silent-rules' option as argument to
          `AM_INIT_AUTOMAKE'.

        * Call the `AM_SILENT_RULES' macro from within the
          `configure.ac' file.

     If the developer has done either of the above, then the user of the
     package may influence the verbosity at `configure' run time as
     well as at `make' run time:

        * Passing `--enable-silent-rules' to `configure' will cause
          build rules to be less verbose; the option
          `--disable-silent-rules' is the default and will cause normal
          verbose output.

        * At `make' run time, the default chosen at `configure' time
          may be overridden: `make V=1' will produce verbose output,
          `make V=0' less verbose output.

     For portability to different `make' implementations, package
     authors are advised to not set the variable `V' inside the
     `Makefile.am' file, to allow the user to override the value for
     subdirectories as well.

     The current implementation of this feature relies on a non-POSIX,
     but in practice rather widely supported `Makefile' construct of
     nested variable expansion `$(VAR1$(V))'.  Do not use the
     `silent-rules' option if your package needs to build with `make'
     implementations that do not support it.  The `silent-rules' option
     turns off warnings about recursive variable expansion, which are
     in turn enabled by `-Wportability' (*note Invoking Automake::).


Cheers,
Ralf