Re: curses should not suppress unassigned codepoints

[Thomas Dickey]

On Sun, Jan 10, 2016 at 05:51:47PM +0100, Benno Schulenberg wrote:
> 
> On Sun, Jan 10, 2016, at 00:37, Thomas Dickey wrote:
> > > On Thu, Jan 07, 2016 at 03:18:59PM +0100, Benno Schulenberg wrote:
> > > > Unicode codepoints that don't yet have a character assigned
> > > > to them, are somehow suppressed by ncurses.
> > 
> > On second thought: unassigned codepoints don't have a defined width
> > and representation.
> 
> True.  But the representation doesn't matter to ncurses, just the
> width.  Why not default to 1 for the width?  Or to 3, to accommodate
> what will actually happen on a terminal emulator?

actually the latter is unspecified (it depends upon the terminal emulator).

Just to refresh my memory, I checked on urxvt, pterm, mlterm, konsole,
and gnome-terminal in my Debian/testing machine.  Some displayed double-width
characters, some displayed single-width characters, and there were 3 flavors
of _what_ they displayed.  None matched xterm.

If you use an older Debian or (almost) any other platform, the results
_will_ differ.
 
> > ncurses has always displayed a blank for those cases.
> 
> But that is deception.  It makes it seem as if nothing is there,
> but the application is trying to show something.  What is the
> rationale for ncurses to censor that?

The width is unknown, and the characters aren't valid combining characters.

I looked at the output of vim - it seems that vim just assumes they ought
to be combining characters - and it's definitely confused about their width.
xterm displays the replacement character (for vim).  Some other terminal
may display something else -- or nothing at all.

In the section of code we're discussing, there are some special cases to
handle non-Unicode data, which no one's asked about for a while.  Adding
special cases to out-guess the locale tables is a losing proposition.

(ncurses also doesn't display control characters, but interprets form-feed,
carriage return and newline).

> If the application wanted to show blank space, it would send
> blanks.  So ncurses certainly should not show blanks instead
> of the unassigned codepoints.  And why replace them with
> something else when the terminal itself is perfectly able
> to either show the codepoint or put in a replacement?

There's no way to tell if the terminal _can_ display the characters,
because the locale runtime is the "reliable" method, and it says "no".

If you happen to use some program which ignores the locale and just
sends it to the terminal and it _happens_ to work, then that sounds
like a problem with the locale information.

-- 
Thomas E. Dickey <address@hidden>
http://invisible-island.net
ftp://invisible-island.net

signature.asc
Description: Digital signature