Re: Recursive Make Considered Harmful [Long Post]

[Johan Bezem]

Hi,

I've taken some time to thoroughly read the paper by Peter Miller, and
I happen to disagree with him in several points, and since you wanted
'contrary experience', here's the 'short' form. I refer to the original
PDF document available on http://aegis.sourceforge.net/auug97.pdf

gk wrote:
> 
> At 01:57 AM 10/20/2002 -0700, Philip Guenther wrote:
> >I don't know what Paul was referring to in the manual, but I will
> >suggest that you read, if you haven't already, the "Recursive Make
> >Considered Harmful" paper by Peter Miller:
> >
> >http://owlsoft.ne.client2.attbi.com/build/rmch/recu-make-cons-harm.html
> 
<SNIP>
> If anyone has contrary experience of the validity of Miller's arguments I'd
> love to hear them.
> 
> Greg Keraunen <address@hidden>

Section 2 lists the problems PM sees:
- The first one is a problem itself, but you can specify dependencies
between _directories_ (not files!) in GNUMake like this:

  module1/ : module2/

So, if you need 'module2' to be built before 'module1', say so. I did
this and tested on various platforms. Mind the slashes.

- The second one I do not understand. Yes, if you provide the wrong
input, this might seem a solution. However, I see no inherent necessity.
If someone does, please explain.

- The third and fourth item are IMHO examples of wrong solutions,
not a problem by itself.

- Numbers five and six are consequences of taking the wrong track,
again not problems in themselves.

So IMHO we have but one remediable problem left, as well as several
problems derived from the first.

Now, from the sample he uses in 3.2, I can see where a real problem
is: It's in the decomposition of a large project into modules.
Again IMHO the problem between 'ant' and 'bee' is the lack of 
symmetry between two modules at the same level in the hierarchy:
- 'ant' produces an *.o file to be used in a linker run.
- 'bee' also produces such an *.o file, but it _also_ performs
the linker-stage.

In the makefile structure I've created, modules on _any_ level
are considered to be libraries (whether or not the libraries are
effectively created is not important). The linking stage for
a collection of modules (all on the same level of the hierarchy!)
is one level up the hierarchy (towards the root).

If a close interaction between two modules as in the sample in
3.2 is necessary, the two modules belong in one: The decomposition
is wrong.

Now, the argument in 4.1 is true IMHO.
But 4.2, if you remove the necessity for 'tweaks' the first
paragraph becomes void. In the second paragraph I fail to see the
benefit, it's equivalent to me.

4.3 and 4.4 go OK from my side; writing makfiles for large
projects is never trivial.
4.5.1 confirms that it's not the recursiveness that takes a
long time, but more the complexities (both inherent _and_
superfluous), and the avoidable "wrong solutions".

The second paragraph of 4.5.2 contains an interesting item:
"By consistently using C include files which contain accurate
interface definitions ..." This is relevant to overall
make-performance, not in the sense of overhead necessary for
make, but more in the sense of overhead during compilation.
Using exactly the right includes at the right spot under the right
circumstances can reduce compilation times massively. OTOH,
consistently using whatever includefile necessary to make the
current syntax error go away, will slow down the whole process
enormously...
Refer to the book by John Lakos, "Large Scale C++ Software Design"
http://www.amazon.com/exec/obidos/ASIN/0201633620 Also contains many
things useful in C as well.
The rest of 4.5.2 is OK of course, as is 4.6!

About 4.7:
Issue one: If the decomposition of a program into modules is done
with the 'library mindset', the responsibility of maintaining
such interdependencies is reduced to module-interdependencies, 
usually a responsibility of a senior developer in charge of the
"super-module" one level up the hierachy, the same person responsible
for resolving integration conflicts, if any, between the modules.
Issue two: Again, an appropriate decomposition into modules does
wonders. Each module provides its own makefile, the corresponding
interface-includes are the only interface between modules during
compilation, and linking is done one level up. So, if
interdependencies exist, they will be resolved in the makefile
one level up.
WARNING: Circular dependencies between modules cannot exist,
neither in the one-makefile approach, nor recursively. However,
in the one-makefile approach they may go unnoticed for a long time,
whereas in the recursive structure, using directory-based
dependencies, make can and _will_ tell you.
Issue three can be countered with a correct handling of search
paths for include files, so you'll never need to specify anything
other than #include "ant.h". Let make and the compiler worry about
where to find the correct file.
Even: Section 6 (using VPATH) contradicts the use of "../ant/ant.h"
since it may be the case that the "../ant/ant.h" is not available in
the sandbox, but only in the project master source. However, VPATH is
only for make, and the compiler doesn't know about VPATH...

OK, section 5 is all OK, these are issues with make.
6 is OK too, however, in practice, VPATH alone for providing sandboxing
is a little dangerous. For me, in practice, it's unusable, since most
of the time changes in the project master sources at some time broke
my compilation of an urgent fix, so I had to fix dependencies in the
master source before producing my fix.
I prefer to have a full local copy as a sandbox, and before checking
in my files, I check out everything new and do a make run.

Section 7 looks OK for intermediate programs, but IMHO it doesn't scale
too well. If you, like me, are talking about 1500 _modules_ producing
more than 30 separate libraries as well as 6 executables, on multiple
platforms (Windows <whatever>, Solaris 2.6-2.9, AIX, Linux in various
variants, HP-UX, possibly more) you'll need something more powerful
(which is not to say that section 7 is wrong!)

For multiple platform builds I've also used the techniques discussed
by Paul D. Smith at http://make.paulandlesley.org/ especially the 
multi-architecture article.

I've been working on my system quite some time now, including
documentation, and I'm considering writing up everything into
an article of my own, discussing the reasons behind the approach, as
well as the approach itself.

If anyone's interested, I'll announce its availability here.

Regards,

Johan Bezem
CSK Software AG