Re: [bug #33138] .PARLLELSYNC enhancement with patch

[Paul Smith]

On Tue, 2013-04-23 at 21:40 +0300, Eli Zaretskii wrote:
> > Date: Tue, 23 Apr 2013 11:29:35 -0700
> > From: David Boyce <address@hidden>
> > Cc: Frank Heckenbach <address@hidden>, bug-make <address@hidden>
> > 
> > Since you asked basic questions I'm going to start this at a basic level.
> > Apologies if it covers some stuff you already know or if I misinterpreted
> > the questions. Note that I haven't actually looked at the patch that went
> > in so this is generally wrt the original.
> > [...]
> 
> Thanks, I will dwell on this.

When thinking about this, remember that it's not enough to consider how
a single make invocation will work.  If you run with a single make
instance under -j, then redirecting each job's output to a temp file and
then when make reaps each job, copying the contents of that temp file to
stdout, is a sufficient solution.  You just need to be able to redirect
stdout/stderr of a given job to temporary files.  In UNIX of course this
is done by dup()'ing the file descriptors after the fork and before the
exec.  Presumably on Windows there's some other way to do this.

However, in a situation where we have recursive make instances all
running at the same time under -j then each one of those make instances
is also running one or more jobs in parallel.  In this case it's not
good enough for each make to synchronize its own jobs' output.

So in addition to the temp file change above, you ALSO need a way to
synchronize the use of the single resource (stdout) that is being shared
by all instances of recursive make.  On UNIX we have chosen to use an
advisory lock on the stdout file descriptor: it's handy, and it's the
resource being contended for, so it makes sense.

You asked, what if someone redirected the stdout of a sub-make.  In that
case things still work: that sub-make will not participate in the same
locking as the other sub-makes, it's true, but that's OK because the
output is going to a different location from the other sub-makes so
there's no need to synchronize them.  Meanwhile any sub-sub-makes using
the same output file will still be synchronous with each other.


I'm not sure if the lock locks the FD (so that if you dup'd the FD but
it still pointed to the same output, you could take exclusive locks on
both), or if it locks the underlying resource.  If the former I guess
it's possible to break the synchrony if you worked at it.