Re: [bug #33138] .PARLLELSYNC enhancement with patch

[Paul Smith]

On Tue, 2013-04-23 at 23:16 +0300, Eli Zaretskii wrote:
> > All it requires is inheriting the redirected stdout/stderr to child
> > processes. This was already possible under Dos (with the exception
> > that since there was no fork, you had to redirect in the parent
> > process, call the child, then undirect in the parent, IIRC).
> 
> Inheritance is not the problem; _finding_ the inherited object or its
> handle is.  With stdout, its name is global, but not so with a handle
> to some mutex I can create or its name (unless we indeed make the name
> fixed and thus system-global, which I don't like, if only because it's
> not how Make works on Unix).

You're right: we cannot use a fixed name.  That would allow only one
(top-level) invocation of make per system to use -O.  That's not
acceptable.

> > They can have their own stdout, in particular with the
> > "--output-sync=make" option. But that's actually the harmless case:
> > Each sub-make runs with its stdout already redirected to a temp file
> > by the main make. In turn, it redirects the stdout of its children
> > to separate temp files, and when they are done, collects the data to
> > its stdout, i.e. the temp file from the main make. When the sub make
> > is finished, the main make collects its output to the original
> > stdout. So unless I'm mistaken, no locking is actually required in
> > this case.
> 
> But we still acquire the semaphore in this case, don't we?  Perhaps we
> shouldn't.

We do acquire it but it's on a different resource.  We still do need the
semaphore because we want the output directed to the other file to be
synchronized as well, at least within itself.  If you run "make -O" and
your output to stdout is synchronized by you run "make -O >out" and the
contents of the "out" file are not synchronized, that would be bogus.

However, I'm not worried about losing this ability to change the "lock
domain" for sub-makes based on redirection.  If all the sub-makes were
synchronized together even though some redirected to a different file
that is fine.

First, I sincerely doubt there are many makefiles which run sub-makes
with IO redirected like that.  And second, the slight loss in
parallelization is not worth a huge increase in complexity.  It's a nice
thing that we get it for free with stdout but I wouldn't expend days
trying to implement it if we didn't... in fact without stdout I just
don't see that it's even possible.  I guess if the child could somehow
detect that it was using a different stdout it could be done, but that
would require the parent passing information about stdout to the
child... ugh.  Not worth it.

> IOW, the top-level design is indeed quite general, but the low-level
> algorithmic details are not, and therefore just replacing these
> functions will not necessarily do the job.

Right.  The locking is the part that's not portable, and we need a lock
that can be shared between all the make instances for one "top-level"
make, but not shared with another "top-level" make.

Without knowing what kind of resource Windows can take locks on, we
can't really know how to help with that.  There must be precedent for
this kind of "shared between a subset of processes" lock on Windows.
What handle do we need to know, and how can that handle be communicated?
We can pass a filename through the environment, we can add a
command-line flag to the sub-makes, we can do whatever the equivalent of
inheriting an open file descriptor is on Windows...

> If we really want to make this reasonably portable (and without that,
> I cannot see how Paul's dream about less ifdef's is going to
> materialize), this code needs IMO to be refactored to make the
> algorithm know less about the implementation details.

Personally I've never had any luck trying to create portable code from
scratch, at least not unless I'm very familiar with all the platforms
which is certainly not the case here.

Once we see the second implementation it will be a lot more obvious what
needs to be done to make this more portable.