Re: textconv.c

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> Please move the code which manipulates buffer text and the gap to
> insdel.c.  And I don't really understand why you needed to add
> copy_buffer, since we already have insert_from_buffer and copy_text.
>
> The rest (textconv_query) should go to some existing file(fns.c,
> perhaps?).  There's no justification for a new file for such a little
> code.
>
> Thanks.

That code is going to be significantly expanded in the feature/android
branch, so I expect it will become much larger (and definitely worth its
own file.)

`insert_from_buffer' inserts text into a buffer, instead of copying out
of a buffer, and `copy_text' seems to not handle the gap nor work on
character positions.  The work on the feature/android branch will later
be useful for PGTK as well.

I implemented this feature on X in order to have an implementation of
text conversion on a free operating system.

> P.S. I wonder why you couldn't ask about this before doing this stuff.
> This list is supposed to be used for such design discussions.

I didn't think the small amount of code that was installed amounted to a
significant design change.

There will be many more changes on the feature/android branch, since
Android input methods more or less can't work without the ability to
perform arbitrary modifications to buffer text, and the ability to
obtain buffer contents and the position of the point in the buffer being
displayed.

My idea is that on each redisplay after the point in a frame's selected
window changes, and each change to buffer contents in the selected
window around point, 600 characters (which is the absolute minimum
required to satisfy the input method-side buffer content analysis)
around point will be transferred to the thread which runs the Android
GUI and be made available to input methods, and any editing requests
from input methods will be sent back to Emacs, translated into ``text
conversion'' requests, and then performed in the selected window's
buffer.

textconv.c will contain the code that figures out what text to report to
the input method, and how to carry out its editing requests
(i.e. whether or not to convert text insertion and deletion into key
events, or to insert that text directly into the buffer.)

I have a sinking suspicion that this is going to be non trivial.

I tried five or six variations of using a standalone buffer for
character compositions from Android input methods before concluding that
such an approach can not work, because input methods demand text they
insert to later appear inside a query for the contents of the buffer,
and to be able to later delete the text themselves.  Basically, the
problem is this: if you type

  h e l l o SPC

and the input method inserts ``hello '', it will remember that it
inserted ``hello '' at the current position of point.  When you later
press backspace, instead of sending a delete key event, the input method
just asks you to delete the text that was where it thinks ``hello '' is.

In effect, this means that when you type DEL, Emacs actually receives:

  ``replace text from 0 to 6 with the preconversion text hello''

and if you keep pressing DEL after that, Emacs doesn't get any delete
key events, but rather:

  ``replace the preconversion text with hell''
  ``replace the preconversion text with hel''
  ``replace the preconversion text he''
  ``replace the preconversion text h''
  ``delete the preconversion text''

until the word ``hello'' disappears from the buffer.  Then, instead of
sending more delete key events, the input method asks for 600 characters
before the current position of the cursor, analyzes the text there, and
performs the same process on each character, until it cannot find any
more text, at which point it stops sending any events altogether.

Likewise, when inserting text, the input method does not send any key
events.  Instead, when you type ``hello'' and finally SPC, the input
method simply tells Emacs to:

  ``tell me where the point is''

suppose Emacs responds with the number ``42''.  The input method will
then ask:

  ``give me 100 before 42, and then 5 characters after 42''

at which point it performs analysis on both segments of extracted text
in order to determine whether or not to punctuate the word, or even to
insert any text at all.  Finally, it asks:

  ``insert the text hello after 42''

which Emacs is supposed to obey.

The same applies in a lesser degree to newer input methods on Wayland
and X11 systems.  The current support we have there is quite buggy and
does not work with input methods designed for handhelds, and input
methods which perform any kind of analysis on text surrounding a buffer
did not work at all before the change I installed.

I stayed up very late last night trying several different approaches to
the problem before concluding that for input methods to work on Android
(and probably Wayland based systems after several years), Emacs must
provide a 100% by-the-book implementation of what input methods expect.
Here are two examples of what approaches I tried, and how it blew up.

If you provide an editing buffer that is always completely empty, the
input method refuses to send any events at all.  The input method which
comes with the Android emulator complains loudly about Emacs
``misbehaving'' and not reporting back changes made by the input method
itself.  The third party input method AnySoftKeyboard doesn't do that,
but insteads assumes Emacs is buggy and proceeds to operate on the text
it thinks it inserted, so when you hit DEL it ends up sending the word
you previously inserted with a single letter chopped off the end,
instead of a DEL key event.

If you provide an editing buffer that contains any preconversion text,
but clears itself and sends its contents to Emacs every time
preconversion ends, then AnySoftKeyboard simply ignores the buffer being
cleared, while the Android emulator's input method locks up for a while
to re-initialize itself.  The emulator's input method always inserts
spaces after the word itself, and without signalling any kind of
preconversion, and it does not insert spaces after it locks up, in
effect making it impossible to insert any spaces at all.

I tried more approaches, but I can't remember what exactly they were or
why they didn't work.