Re: Unicode confusables and reordering characters considered harmful, a simple solution

[Eli Zaretskii]

> From: Daniel Brooks <db48x@db48x.net>
> Cc: cpitclaudel@gmail.com,  emacs-devel@gnu.org,  stefan@marxist.se,
>   monnier@iro.umontreal.ca,  yuri.v.khan@gmail.com
> Date: Wed, 03 Nov 2021 23:00:28 -0700
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Daniel Brooks <db48x@db48x.net>
> >> Cc: Yuri Khan <yuri.v.khan@gmail.com>,  cpitclaudel@gmail.com,
> >>   stefan@marxist.se,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> >> Date: Wed, 03 Nov 2021 12:54:31 -0700
> >> 
> >> > Do you read Hebrew?  Those characters look like line noise there,
> >> > whereas the text with the default display is perfectly readable, and
> >> > most people won't even know these controls are there (as intended).
> >> 
> >> My suggestion is to only enable it by default in _programming modes_. It
> >> should remain disabled in ordinary prose like a TUTORIAL file.
> >
> > What about comments and strings?  Are we going to pretend that RTL
> > scripts aren't used in those?
> 
> Of course it will show them in the comments and strings.

Then this visual noise will get in the way of people's reading those
comments and strings, and, for strings, will make it very hard to
understand what will be presented to the user when those strings are
output in some UI.

> That’s where the problem is.

No, the problem is elsewhere entirely: it's in the punctuation
characters unrelated to strings and comments whose directionality is
overridden, and which thus display in places that cause incorrect
visual interpretation of the program during a casual read.

> > You are welcome to make such customizations in your Emacs.  My point
> > is that for a useful feature that doesn't get in the way when those
> > controls are used for legitimate purposes, and only highlights _text_
> > (NOT the controls!) whose appearance may have been altered by them for
> > questionable or suspicious reasons -- for such a useful feature what
> > you propose is not enough for having it in Emacs for everyone.  It is
> > a blunt weapon that I would be ashamed to install.
> 
> Ok, it is helpful to know your thoughts on the matter.
> 
> However, your suggestion of highlighting the text affected by the bidi
> override characters while not actually showing those characters visibly
> is not something that I would care to use. It shows that there may be a
> problem without showing what the cause is. The cause is the presense of
> certain characters, and I must be able to see those characters in order
> to fix the problem, or even to judge whether there is a problem at
> all.

You misunderstand the cause.  The mere presence of these characters is
NOT the root cause.  These characters are legitimate and helpful when
used as intended.  See TUTORIAL.he for a pertinent example.

The real cause is that these characters are used with the explicit
intent of changing the visual presentation of some code fragment or an
identifier in source code or in a URL.  The challenge, therefore, is
not to make these characters stand out wherever they happen, because
that would flag also their legitimate uses for no good reason.  the
challenge is to flag only those suspicious or malicious uses of these
characters.  And that cannot be done by just changing the visual
appearance of those characters, because their legitimate uses are by
far more frequent than their malicious uses.  To flag only the
suspicious cases, the code which does that needs to examine the
details of the text whose directionality was overridden and detect
those cases where such overriding is suspicious.  For example, when a
character with a strong left-to-right directionality has its
directionality overridden to behave like right-to-left character, that
is highly suspicious, because it makes no sense to do that in 99.99%
of valid use cases.

> Anything short of that is useless to me, and I suspect to many
> others as well. Do you hide the tags when you write HTML? Do you hide
> the parentheses when you write Lisp? Or the semicolons when you write C?
> This is no different.

This is VERY different, for the reasons I explained above.  What you
suggest will have a very low signal-to-noise ratio, so having such a
feature in Emacs in general is a bad idea.  And people who for some
reason still want to have that noise in their face can simply
customize glyphless-char-display-control to show those characters as
their acronyms in a small box.

> Furthermore, I have not suggested that showing the characters needs to
> preclude any other form of highlighting. If you wish to develop some
> additional way of warning the developer, please do so.

We are talking about what should be in Emacs.  What you suggest
shouldn't.

> However, I suspect that the compilers for most languages currently in
> active development will develop their own warnings and error messages as
> well. We have plenty of ways for those messages to show up inside Emacs
> as highlights.

That's a tangent.  We are discussing what Emacs should do as a
programmer's editor to flag such suspicious code.  That shouldn't need
a compiler if we can do the job ourselves.  And we can.

> Rust, for example, has already done so. Here’s an example:
> 
>     error: unicode codepoint changing visible direction of text present in 
> comment
>       --> src/pathmap/path.rs:10:5
>        |
>     10 |     /* } if is_admin  begin admins only */
>        |     ^^-^^-^^^^^^^^^^--^^^^^^^^^^^^^^^^^^^^
>        |     | |  |          ||
>        |     | |  |          |'\u{2066}'
>        |     | |  |          '\u{2069}'
>        |     | |  '\u{2066}'
>        |     | '\u{202e}'
>        |     this comment contains invisible unicode text flow control 
> codepoints
>        |
>        = note: `#[deny(text_direction_codepoint_in_comment)]` on by default
>        = note: these kind of unicode codepoints change the way text flows on 
> applications that support them, but can cause confusion because they change 
> the order of characters on the screen
>        = help: if their presence wasn't intentional, you can remove them

Since the Rust compiler evidently does this when it finds these
characters inside comments (and probably also inside strings), IMNSHO
this is a terrible misfeature, because it means code that uses those
controls in legitimate ways cannot be compiled without tweaking
non-default options.  That's a cop-out, not the way to flag the
problematic cases.

> Naturally that already shows up inside of Emacs just fine; see the
> attached image.

I think this is terrible.  At best, it only tells you that something
non-trivial goes on here (but what exactly?).  At worst, it looks like
corruption of the source.  And while in the malicious case treating
that as corruption is not such a bad idea, all the valid uses of these
characters will also look like corruption.  Which means the cure is
probably worse than the disease, because the malicious cases are a
tiny fraction of the valid ones.

It's the same kind of "solution" like the airport security after 9/11:
because there was a bunch of terrorists, we are all now suspect as
potential terrorists, and for that reason we are constantly delayed
for hours and humiliated by endless frisking.