Re: EAGAIN in "read jobs pipe"

[Howard Chu]

address@hidden wrote:
> > Noticed something else.
> >
> > Something else completely pointless!
> >
> > <ridiculous> must not be > 16385
> >     
>
> True, if you have 16 KB pipes...
>
>   
> >  gmake -j 16386; # or above hang
> >  gmake -j 16385; # or below fire along rapidly
> >     
>
> Because you're trying to write more into the job tokens pipe than it can
> hold...
>
> You appearantly have 16K pipes (ulimit -a should tell you)... I get the
> same on x86 Linux if I go beyond 4K jobs...
>
> This code (from main.c) assumes that the pipe will be big enough to hold
> all tokens:
>
>       while (--job_slots)
>         {
>           int r;
>
>           EINTRLOOP (r, write (job_fds[1], &c, 1));
>           if (r != 1)
>             pfatal_with_name (_("init jobserver pipe"));
>         }
>
> A bug?

An essential design choice. This stuff relies on reads and writes of the 
job_fd being atomic and the writes never blocking. POSIX guarantees a 4K 
buffer for pipes. Perhaps the code should check the resource limit and 
complain if the -j argument exceeds the resource limit: "Error: why 
don't you just use '-j' ??"

>  Perhaps... On the other hand, if you're using -j 65K, why not
> just -j? Does you build even have 65K jobs?
>   

Very good question.
> I could be wrong of course, but I in my experience you don't gain any
> real benefit from going beyond 3-4 jobs per (virtual) core... What's the
> difference in build time from, say, -j 128 and -j 65385 for you?
>   

I usually count on 1.5 jobs per core, but obviously the right balance 
depends on your disk speeds relative to your CPUs...
> The fact that your machine stays responsive during a build does not
> necessarily mean that you will gain any benefit from increasing the -j
> number...
>
>   
When I see that total CPU idle time is about 3% or less I figure that's 
enough jobs.

--
 -- Howard Chu
 Chief Architect, Symas Corp.  http://www.symas.com
 Director, Highland Sun        http://highlandsun.com/hyc
 OpenLDAP Core Team            http://www.openldap.org/project/