|
From: | Jan Schiefer |
Subject: | Re: [Discuss-gnuradio] Make -j<N> should now be working |
Date: | Fri, 11 Aug 2006 10:03:49 -0700 |
User-agent: | Thunderbird 1.5.0.4 (Windows/20060516) |
With all due respect, I don't think the current approach of recursive makefiles is going to get us to a point where you can get a consistent and repeatable build with arbitrary numbers of processors or build machines (in the case of a distributed build), with multiple developers, over long periods of time. It is really really hard to do this without the top-level make being aware of all the dependencies throughout the whole code base. It is so easy to subtly break a build by introducing a dependency in a lower-level Makefile that the top-level make isn't aware of. Then multi-process builds will sometimes fail and sometimes succeed, depending on the execution order, which is completely arbitrary. Spotting and debugging the resulting inconsistencies is usually pretty painful, as most of you will know.
I know I might be starting a holy war here, but I believe that a better way to do this reliably is for the top-level Makefile to include the lower-level ones, which just state the dependencies for the particular module. All the actual work is controlled by a single instance of make, which has the whole picture of the dependencies and can schedule accordingly. That doesn't preclude building individual modules, as they can be stated as sub-targets and built individually.
A reference to the landmark paper "Recursive make considered harmful" may be in order (http://www.pcug.org.au/~millerp/rmch/recu-make-cons-harm.html). This is almost 10 years old, but still well worth reading.
I just wondered whether this was on anybody else's radar. Cheers, Jan Johnathan Corgan wrote:
Ticket #13 was reporting failure with parallel makes. Invoking (after bootstrap & configure) make with the -j option causes it to run as many instances in parallel as possible, dependencies allowing, in order to take advantage of multiple processors or cores, or to have a second instance run while the first was waiting for I/O, etc. Normally you put -j2 or -j3 to limit the maximum, as beyond that there is little benefit. This was failing in gnuradio-core, but was fixed back in r3222. Those of you tracking the new svn trunk--could you test this and confirm? Incidentally, make -j doesn't break anything, and at one point had 1100 processes running in parallel to build various parts of the whole tree. So when we get to 1024 core processors, we know we'll be able to take advantage of them. :-) -Johnathan _______________________________________________ Discuss-gnuradio mailing list address@hidden http://lists.gnu.org/mailman/listinfo/discuss-gnuradio
[Prev in Thread] | Current Thread | [Next in Thread] |