Re: [RFC]serialize the output of parallel make?

[Chiheng Xu]

On Fri, Jul 30, 2010 at 1:26 PM, Paul Smith <address@hidden> wrote:
> On Fri, 2010-07-30 at 09:59 +0800, Chiheng Xu wrote:
>> As parallel make are becoming more and more popular,  can make
>> serialize the output of parallel make?
>>
>> Make can redirect every parallelly issued shell's output to an
>> temporary file,  and output the stored output serially, as if in a
>> serial make.
>
> This would be a good thing, but as always the details are not quite so
> trivial.
>
> We have to ensure that these temporary files are cleaned up properly,
> even in the face of users ^C'ing their make invocations.  We also need
> to verify that whatever methods we use will work properly on Windows and
> VMS and other operating systems make supports (where are their "/tmp"
> equivalents?)
>

My suggestion is that you can implement it as an optional command line
option(like -j), and on one or two primary platforms(Linux/Windows),
instead of on all platforms.


> And, what about stdout vs. stderr?  Should we write both to the same
> file?  Then we lose the ability to do things like "make -j4 2>/dev/null"
> since all output will be written to stdout (presumably).  Or should we
> keep two files per command, one for stdout and one for stderr?  But
> that's even worse since then when we printed it we'd have to print all
> the stdout first then all the stderr, which could lose important
> context.
>


The scenario like "make -j4 2>/dev/null" may be very rare, but
scenario like "make -j4  2>&1 |  tee output.txt" may be common.


I think using make to parallelly build or test large software on
multicore system is by far the most normal situation. User of make
most need is : 1. utilize the computing power of multicore system
using parallel make(-j) ; 2. easily analyze the output of the build or
test, to precisely find the problem.

When user of make provide the optional output-serializing command line
option, he known what he need, he is responsible for the mess
situation.




> Then there's the possibility that some commands will behave differently
> if they are writing to a TTY, then they will if they're writing to a
> file.  Do we not care about that, or do we try to do something crazy
> with PTYs or similar (ugh!)
>
> And of course we have to have a guaranteed unique naming strategy in
> case multiple instances of make are running on the same system at the
> same time, maybe running the same makefiles and even building the same
> targets.  On POSIX systems we can use tmpfile() or mkstemp() or
> something but other systems might need other measures.
>
> These are just some things I thought of off the top of my head.
>
> It certainly does not mean that it would not be a good thing to have
> this ability though.
>
> --
> -------------------------------------------------------------------------------
>  Paul D. Smith <address@hidden>          Find some GNU make tips at:
>  http://www.gnu.org                      http://make.mad-scientist.net
>  "Please remain calm...I may be mad, but I am a professional." --Mad Scientist
>
>



-- 
Chiheng Xu
Wuhan,China