bug#61350: Eglot over Tramp freezes with large project

[João Távora]

On Tue, Mar 14, 2023 at 3:00 PM Michael Albinus <michael.albinus@gmx.de> wrote:

> Achhhhhh :-(

Pech!

> I will investigate. IMHO, this is the final indication that we shouldn't
> allow a nil JUST-THIS-ONE in accept-process-output.

I don't see this failure as an indication of that, at all!!

First, you didn't do that -- put nil in JUST-THIS-ONE --  did you?
At most this means that you are doing more or less the same
as setting JUST-THIS-ONE to nil, but in a much more convoluted
way, so that you may as well start setting JUST-THIS-ONE to nil...

...and bite the bullet, continuing this analysis.  We're making
progress...

So, I reach a completely different conclusion.

For me, this means that those auto-revert and symlinks-remote tests,
whatever they are, must be investigated. It's good that you have these
tests! Do they fail in all Emacsen or just some? Always/sometimes?
Can I reproduce this here?

After we get a consistent failure, we should ask ourselves what exactly
is the reentrancy in question here. Perhaps you already have a conjecture
of the chain of events that leads to it.

FWIW I've recently solved a reentrancy bug in jsonrpc.el.
I identified the conditions (here, i was ignoring that process-send-string
when given enough input to send, will cause output acceptance, and
filters to run), and fixed it.

João

commit 8bf4cdcf79bc3254a9169f28f68922ab83bd4e78
Author: João Távora Date: Thu Dec 15 15:26:13 2022 +0000
Avoid recursive process filters in lisp/jsonrpc.el (bug#60088)

jsonrpc.el may lose JSON-RPC messages because of recursive process
filters. The problem happens in jsonrpc.el's jsonrpc--process-filter.

The code of the process filter didn't expect to be called recursively
 and fails in that case.

But that can happen if the three conditions are verified.

1. the client code invoked by its jsonrpc--connection-receive inside
   the process filter callee immediately sends follow-up input to
   process within the same Lisp stack. This is a common scenario,
   especially during LSP initialiation sequence used by Eglot, a
   jsonrpc.el client.

2. that follow-up message is large enough for process-send-string to
   send the input in bunches (output from processes can arrive in
   between bunches).

3. the process happens to have already some more output ready

The fix in this commit detects recursive invocations and immediately
re-schedules them as non-recursive calls to the
jsonrpc--process-filter (but started from timers).

* lisp/jsonrpc.el (jsonrpc--process-filter): Rework.
(Version): Bump to 1.0.16.