Re: term, utf-8 and cooked mode, combining characters

[Roland McGrath]

> I am kinda excited about the possibility to get all applications
> transparently use libreadline if they use cooked mode.  I might be
> pipe-dreaming...

That has been a stock pipe-dream from the beginning of the Hurd.  But
that's different from saying it's the right solution just to handle
the problem of multibyte characters properly.

> It is big, but most code is for the configuration file parsing, key
> binding handling, emacs/vi cod, etc.

Yes, and you need all that code unless you write something to replace it.

> With regards to readline's depenency on termios, that is really not that
> much.

It certainly doesn't need to be much tied in principle, but that is
different from actually teasing it out from the code we have now.

> What I am more concerned about is organizing the input correctly.

I really meant all of that i/o model too when I said "termios".

> I think I am not aware of what configuration you mean.  I agree that it is
> probably not reasonable to have filename tab completion by default in term's
> cooked mode, but the default settings should be good enough for normal
> interactive mode.

Hardwired key bindings are unacceptable.  If you don't add any kind of
configuration interface, then the only one you get is stty, which lets you
bind characters to the things c_cc represents but not other editting commands.

Another reason using readline is good is that people already use it and
configure it how they like it for bash and whatnot using their .inputrc.
So the desireable thing is to allow configuration of the terminal in a way
that can be fed from .inputrc.  For example, fsysopts options that take
lines in .inputrc syntax would be a minimally adequate interface.  But
you'd want that configuration to act more like stty configuration than like
fsysopts usually does, i.e. reset to defaults by a revoke or something.

> In fact, I don't know how to implement this given the iconv interface.

I know even less I can assure you.

> libreadline uses the mb* functions which work on the local encoding only
> and intimately know it.  I guess we could make console run in this local
> encoding, but this will be a pain if we want to support local encodings
> per virtual console (which is easy to do right now).  (This would of
> course also be a problem with a libreadline using libtermserver in the
> console).

In glibc 2.3 you can use uselocale to change the encoding used by mb* (and
everything else) in a single thread, so you can switch a thread to the
appropriate locale for each console's local encoding while using it.
(The uselocale call to switch is cheap, it just sets some pointer variables.)

> Also, I am not sure I agree it is the right thing.  Sure, if you put term
> into the console, it doesn't make no difference from the outside.  But that
> doesn't solve the problem that you have if you want to use the stand alone
> term server in any multi-byte encoding locale.  So, although that would
> do the job here, it doesn't seem to be the generic solution.

True.  I had meant to bring this up (ptys, serial, etc) in the first
message but ran out of steam typing.  I don't have a good answer, I am just
not happy with a bloated term.  I meant to ask what exactly is being done
or considered in Linux, since you mentioned something was happening.  The
latest Linux 2.5 kernel sources I am aware of don't appear to do anything
about it.  If the extent of the support there will just be a bit to assume
UTF8, then that is probably good enough for us too in the simple term.
If only the snazzy readline-enabled term handles multibyte characters
optimally, we will live.

> Interesting, although I am not sure that would be easy to make convenient to
> use in the GUI.  libreadline in term seems to be more natural for me, as
> this will make it behave exactly like bash etc.  

You can have whatever UI you want, I am talking about how the protocols are
structured so that part of that UI could be in a different layer.  The
closer to the user the UI is implemented, there more versatile it can be.
Perhaps I would like an interface where the cooked mode input is processed
with key bindings just like readline, but the display in my console client
pops up an enlarged text box to show me what I'm editting.  For non-visual
interfaces, there might be many useful possibilities that are hard to do
otherwise.  The point is that many things are possible if the client knows
more about what kind of interaction is happening.

This is not really part of the same point I just argued, but related.
(This idea applies to any fancy cooked-mode interface, whether in term or
console or whereever.)  We should hack libreadline with a special mode for
interaction with a fancy cooked mode driver, so that bash et al running on
such a terminal don't actually behave as they do now but use cooked mode.
This could be enabled by a special TERM setting or something like that.
The advantage of this is that you can configure your key bindings for your
terminal once (rather than having each program read your .inputrc), and you
can have history across commands if you want it (I have wished for that
feature many times myself--when I was foolish enough not to be running all
my programs inside Emacs buffers).  The special interaction beyond just
using cooked mode (really we mean ICANON, and not necessarily ISIG) would
be so that things like completion can be implemented in the driver while
getting information from the programs.  The concept is the same as how gdb
runs under Emacs.  What I have in mind off hand is something like having a
command byte or two at the beginning of the "line" returned by the
ICANON+special-mode read that says "this is a line" or "this is a request
for completions of the prefix" or suchlike.  For feeding the completions
back to the driver, it could be a write to the terminal prefixed with a
magic escape sequence, or some kludge like that.  You get the idea.