bug#44007: 26.3; Strange shell mode performance

[Herman]

I've debugged the issue a little bit, and the cause of it is 
process-adaptive-read-buffering indeed. I misused this variable before: 
I tried to set it to nil before executing "cat", but it needs to be set 
before "M-x shell".

Setting it to nil completely fixes the issue. If its value is "t", my 
"cat test.txt" runs in 18 sec. Setting it to "nil", cat runs in ~2 sec.

Looking at the logic which does adaptive reading, I see why it can be 
suboptimal. If a process sends less than 256 bytes at first, then delay 
is increased, and it will be decreased only if the process can send 
bytes which completely fills the buffer. But this may never be true 
(because some other part of the system already throttles the process, 
because emacs didn't read enough from it - I suppose this can depend on 
the value read-process-output-max).

The issue can be made less severe by modifying emacs_intr_read to read 
as much as data fit into the buffer (execute read() until it reports 
failure/EOF). It solves the issue completely for me, if 
read-process-output-max is 4096. If it's 1MB, this modification makes 
the issue less severe: "cat test.txt" runs in 6 sec.

Here's the modification I did to emacs_intr_read, put this after the 
do/while loop:

  if (result > 0) {
      for (;;) {
          ssize_t r = read(fd, buf+result, nbyte-result);
          if (r<=0) break;
          if (interruptible) maybe_quit();
          result += r;
      }
  }

(I'm not saying that this is the correct solution, I put this here so 
you can repro my results)

The reason that it's needed that read() returns 4095 for some reason, 
even though nbyte is 4096. An additional read() call will provide the 
missing 1 byte. As far as I know, this is a valid and allowed, yet 
strange behavior. I'm not sure what causes it.

I think that the logic around decreasing the delay is not good. The 
current "decrease delay when buffer is full" is a fragile condition. To 
me, it seems logical to clear/decrease delay if more than 256 bytes 
arrived. It may not be 100% optimal, but it's near to it. On the other 
hand, the current behavior can cause severe performance problems for 
fast streams. Having read-process-output-max a large value is essential: 
"cat test.txt" runs in 0.3 sec, when this value is 1MB (adaptive is 
turned off). If read-process-output-max is 4K, it runs in 1.4 sec 
(adaptive is turned off).