Re: patch to support output synchronization under -j

[Tim Murphy]

On 3 May 2011 17:39, Eli Zaretskii <address@hidden> wrote:

> That was exactly the scenario I had in mind when I wrote my message.
> Recursive Makefiles are the rule nowadays, at least with GNU software,
> and the top-level Makefile does little more than launch a "make all"
> job in each subdirectory.  GCC or GDB might be extreme examples, but I
> do build them from time to time, and there are less extreme examples
> which nevertheless take considerable time to run each sub-Make.
>
> I think we must find some solution to put this under user control, or
> else this feature will be useful only for small projects.

I wrote a similar "descrambler" (Talon) and use it on absolutely
unbelievably enormous builds but they do only the most minimal amount
of recursion.  This is actually a problem because it takes so long to
read in the makefiles that one wastes 10s of minutes on a single CPU
before the cluster can start working.  On the other hand the build
part is pretty efficient :-).

Anyhow I am just saying that this feature is useful huge builds and
more so because the huger they are the less one tends to sit there
watching the output.

Even in smaller ones the boon is that you can be highly parallel, thus
not taking too long to get the entire makefile.   So you can't watch
it with 0 latency while it is working but you do get the complete set
of information sooner.

We needed more though - we needed the output to be wrapped up and
classified and so on so I made up a way of setting variables that
controlled this output.  In David's implementation one might have the
luxury of using target specific variables or something like that.

e.g. Talon has a "FORCESUCCESS" option to tell the shell to output the
correct exit code of a command but tell make that it really succeeded.
 Another option allowed us to get detailed debugging output as the job
communicated with the shell so that we could see more precisely when
and how some crash happened but without turning on this feature for
all recipes in the whole 20 hour build.

We use macros but it is like this underneath:

mytarget:
   
|TALON_DEBUG=1;FORCESUCCESS=1;TALON_BUILDID=1342;TALON_DESCRAMBLE=1;TALON_TIMEOUT=10000;|armcc
-o ekern.exe xxxx.o yyyy.o

Notice the "DESCRAMBLE" which allows me to turn off synchronisation.

Now with Talon we are using a systemwide semaphore whose name is based
on the TALON_BUILDID.  So if we were running a submake we could turn
off synchronisation for that recipe and as long as the submake used
the same "build id" it would still be deconflicted.

In this case target specific variables might be a neater way to do
something similar:

mytarget: PARALLELSYNC=
mytarget:
       $(MAKE) -f submake.mk


Then submake.mk carries on as normal.  The problem is that David is
not using semaphores so I haven't thought how to spin it for his case.

Regards,

Tim




-- 
You could help some brave and decent people to have access to
uncensored news by making a donation at:

http://www.thezimbabwean.co.uk/