Re: Fix for slow process output processing (please test).

[David Kastrup]

address@hidden (Kim F. Storm) writes:

> David Kastrup and others have reported very slow processing of process
> output, particularly with the Linux kernel's scheduler giving emacs
> too much attention while starving the processing producing the output.
> 
> Below is a patch which introduces a small delay on reading output from
> such processes, without using global delays in processing other events
> (this is done by temporarily removing the process' file descriptor from
> the call to select and use a short timeout on the select instead.

I have glanced over that patch.  There is one problem I see with it:
if indeed the generating process is able to produce lots of data with
a single write call, Emacs will be throttling its input without a
purpose.

This situation is quite more probable when we have a multiprocessor
machine or when the input is generated by some network process (so
again, on a different CPU).

The best behavior would be
a) don't bother reading from the pipe for a while if it is not yet
   full.
b) process it immediately once it is completely full.

The select call does not offer any possibility to say "wake up
immediately if there is such an amount of data".  And we need the
select call to poll other input, so we can't just use blocking I/O
with a timeout set (which would probably be the best solution if we
had only to worry about a single source of input).

`Enterprise level' solutions would probably be:
a) use blocking I/O with a timeout set in a separate thread.  When
the timeout hits or the read completes, whatever has accumulated in
the corresponding input buffer is marked for processing and made
available to the main Emacs.
b) use aio_read to request a full pipe worth of data.  If no signal
announces the availability of the data, some timeout may be used to
cancel the aio_read if there is data available, but not a full pipe
worth of it.

Of course, those solutions are more operating system specific than the
current one.  It may well be that there are few applications that will
turn out worse by throttling Emacs to something like 8kB/20ms or so.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum