Re: User interaction from multiple threads

[Eli Zaretskii]

> From: Michael Welsh Duggan <address@hidden>
> Cc: Michael Welsh Duggan <address@hidden>,  address@hidden
> Date: Thu, 16 Aug 2018 22:59:57 -0400

Let me start responding from the end:

> No problem.  As my employer makes it unfeasibly difficult to contribute
> code to the FSF (would have to have a separate disclaimer for every
> feature), participating in discussions like this is the least I can do.

Does your contract allow you to share ideas?  How about share ideas
that are later attributed to you when those ideas are implemented in
code?  If you can do that, then describing an idea of the
implementation in sufficient detail will allow someone else to write
the code relatively easily, and AFAIU won't be copyrightable or
subject to IP restrictions.  (But IANAL.)

> >> #1 If "waiting for input" means in read-from-minibuffer or something
> >>    similar, I believe that input should go to the the thread.  The other
> >>    thread will have to wait.  If "waiting for input" means idle, it
> >>    should go the the other thread.
> >
> > "Waiting for input" means waiting for user to type something.  That is
> > normal Emacs situation when the user does nothing, so unconditionally
> > sending input to the main thread and letting the other thread wait
> > would mean that other thread might wait indefinitely.
> 
> To clarify, I believe the command-loop (main-thread), aside from a large
> amount of maintainance, is fundamentally a loop around
> read-key-sequence, lookup-key, and command-execute.  *This*
> read-key-sequence needs to be interruptable by other threads as long as
> no keys have yet been entered into the sequence.  This is the point I
> considered to be "idling."  If any other thread, or even the main
> thread, calls read-key-sequence outside of the main command-loop, that
> one should get the input mutex immediately.

I deliberately avoided saying anything about being "idle", because in
the context of this discussion, the meaning is overloaded several
times, and is likely to cause confusion.  So let me now describe the
low-level details which are IMO pertinent to the issue at hand.  (I
suggest to review the relevant code parts: thread_select and the
functions it calls, and wait_reading_process_output.)

When any thread in Emacs is about to become "idle", i.e. has nothing
else to do except wait for input (in particular, but not necessarily,
because it needs the user to respond to some prompt), it releases the
global lock and calls pselect.  The descriptors on which pselect will
wait are computed by compute_input_wait_mask, which skips any
descriptors already being waited for by some other thread; this has
the side effect of having only the main thread wait on keyboard input,
at least in most cases.

The command loop causes pselect to be called with a very large timeout
(unless there are active timers, in which case the timeout is computed
to end when the first timer expires).  So the "idling" thread will be
parked inside the pselect call until that timeout expires.  And since
the global lock is released and up for grabs, another thread may take
the lock and begin/continue running during that "idle wait".  When
that other thread attempts to read from the minibuffer (a.k.a. "become
idle"), it will also release the global lock and call pselect.  What
will happen then depends on whether or not the main thread still
waits:

   . If the main thread still waits, the non-main thread will NOT wait
     on keyboard input, and we now have 2 threads parked inside
     pselect; user input will go to the main thread.
   . If the main thread's timeout expired, it is waiting to take the
     global lock, so it will now take it and loop back to the next
     pselect call.  We therefore have a race between the main and
     non-main thread: there's a small probability that the non-main
     thread will call compute_input_wait_mask first, in which case it
     will wait on the keyboard input.  (This situation could be
     triggered with timers that fire at high frequency, I think.)

Therefore, your idea of "interrupting" an "idling" main thread would
need to somehow cause it exit the pselect call, before the non-main
thread is about to prompt the user, and also prevent it from calling
another read_key_sequence until the non-main thread finishes its
interaction with the user.  Interrupting the pselect call could be
emulated by using a very short timeout, but that is unfriendly to
laptops when Emacs is _really_ idle.  Another way to interrupt pselect
is with some signal and setjmp/longjmp.  We then need a mutex to
prevent the read_key_sequence call during user interaction, so that
the interacting thread could arrange for waiting on the keyboard
descriptor.

Does this make sense, given the details in the related code?

> You probably understand what I mean at this point, but to clarify, in
> case: Another thread should not be blocked for input because the main
> thread is waiting for an event, unless the main thread is waiting for
> that event because it explicitly called read-key-sequence or some other
> higher-level input function (such as read-from-minibuffer) from anything
> other than the command loop.  But the instant the command loop's
> read-key-sequence reads a single key, it must have the input mutex until
> it has finished reading a key sequence (or is quit).

The last part will be non-trivial to translate into code, because I'm
not sure the related code is aware of where we are in the process of
reading a key sequence, and there are many different paths through
that code.  But I may be wrong.

> Now, another command -- say, read-from-minibuffer -- could end up
> calling the command loop, which will look for events.  That command
> should have set the input mutex beforehand, and, to be clear, this
> should be a recursive mutex, so it won't be interrupted in this context.

We will need to enhance the mutexes we use, because currently I think
we use non-recursive mutexes on Posix-ish platforms, and recursive
ones on MS-Windows.

> A call to recursive-edit, however, would not (I think) grab the input
> mutex, and as such would still yield to an input request from another
> thread.

Not sure about that, we need to review the various uses of
recursive-edit first.  For example, read-from-minibuffer uses
recursive-edit, and I think we do need that to lock out input from
other threads.

> I have no idea what would be caused by a non-main thread calling
> recursive-edit.  That way lies madness.  But someone should consider
> what it might mean.

Well, read-from-minibuffer calls (a subroutine of) recursive-edit, so
we should definitely support that madness...

> >> #2 You should neveer break a key sequence.  We do not preempt the main
> >>    thread.  The other thread will have to wait.
> >
> > Do we need to tell the user the other thread wants to prompt?
> 
> No.  Just wait for the current input sequence to finish, then allow the
> prompt.  I suppose there could be indicators in the mode line, or
> something like that, but I don't think we need that initialy, and I have
> no idea whether that would actually be desirable anyway.

I think an indication is extremely desirable, but I agree it could be
tricky to produce, even on the mode line.

Thanks.