From: Paul Smith <address@hidden>
To: Jim Michaels <address@hidden>
Cc: "address@hidden" <address@hidden>
Sent: Tuesday, April 30, 2013 11:23 PM
Subject: Re: feature request: parallel builds feature
On Tue, 2013-04-30 at 17:20 -0700, Jim Michaels wrote:
> I wasn't digressing. I was explaining the point. the concept I am
> trying to present as a solution to the problem of making parallel
> stdin for --jobs in gnu make (which currenty doesn't work and is I
> guess single-threaded) is to make a separate terminal or command shell
> for each job, such as via a generated batch file or shell script.
>
> this is as simple as I can make it.
You need to give a concrete example of the problem you're trying to
solve. When the manual discusses stdin
it means that only one job at a
time can read from the make program's stdin.
Multithreading won't help because there is only one input to read from,
regardless of how many threads do the reading.
What this limitation is discussing is if there were a makefile like
this:
read: read1 read2
read1 read2: ; @echo $@: enter a word: ; read word ; echo $@: $$word
Both of these two targets read from stdin. If you run them serially, it
works:
$ make
read1: enter a word:
fooblatz
read1: fooblatz
read2: enter a word:
barflatz
read2: barflatz
If you run in parallel then both the read1 and read2 targets run at the
same time and both want input from stdin, at the same time. There's no
way this can work: when you typed a word how could you know or specify
which read operation got
it? So make arbitrarily chooses one of the
jobs to get the input and the others have their stdin closed.
But if course, this doesn't impact in any way rules like this:
read: read1 read2
read1 read2: ; @word=`cat address@hidden; echo $@: $$word
Now if you have files like read1-input and read2-input, those will be
read inside these rules and behave properly.
> I have learned that on a machine with 12 threads and 64GB of memory,
> you can have 50+ jobs running.
This depends very much on what those jobs are doing. Obviously you CAN
run as many jobs as you want. However I've never heard of being able to
get more than #CPUs plus a few jobs running at the same time without
making the build _slower_. At some point the kernel will simply thrash
trying to keep all those jobs running at the same time, if they
seriously outnumber the cores
available to run them on.