Re: bidi-display-reordering is now non-nil by default

[Stephen J. Turnbull]

Eli Zaretskii writes:
 > > From: Lars Magne Ingebrigtsen <address@hidden>

 > > As I said earlier, I think adding characters to the text to control pure
 > > layout issues is a sub-optimal design decision.
 > 
 > You are wrong.  LRM, RLM, etc. exist _precisely_ for these purposes:
 > to arrange bidirectional text for display as the display designer
 > wishes, where the implicit reordering mandated by the UBA does not
 > give satisfactory results.  These _explicit_ directional control
 > characters exist for when the _implicit_ reordering fails.  It's the
 > functional equivalent of the corresponding HTML directives -- you
 > won't say that HTML is "suboptimal" when it dictates to the browser
 > how to render bidirectional text, would you?

Eli, you seem to forget that *Unicode is a wire protocol*, an inter-
application communication tool.  It is not intended to be a
specification, or even recommendation, of how applications handle text
internally.

More specifically, everything you write applies to *streams of Unicode
(or HTML) text communicated between applications* (eg, files or HTTP
streams), as Lars points out.  I don't know enough about the details
to have an opinion on what *should* be done internally by Emacs, but
your argument above is bogus.  Lars's example of ANSI color control
shows why.  Lars's arguments about the deficiencies of including these
*control* characters in the text are precisely the same as the
arguments for deprecating the "Plane 14 language tags."  (Yes, I
understand why directional marks will *not* be deprecated in that
way by the Unicode standard.  Nevertheless, the arguments are valid
and should be given some weight for *internal* implementation.)

Of course on writing a stream to the outside world, Emacs will need to
use directional marks.  Surely Lars does not deny that!  However,
internally, text properties could in theory suffice, just as they do
for ANSI coloring.

 > Because (a) text properties are specific to Emacs, and (b) they cannot
 > overlap (for the same property).  By contrast, to force certain visual
 > order, one must sometimes force some direction on a portion of text
 > and then the opposite direction on an inner substring of that very
 > text.  Text properties won't grok that.

Huh?  Of course text properties nest.  If for some subtle reason, they
don't quite nest correctly for this purpose, overlays most likely will.

I really don't understand your arguments here.

 > What is the difference between aligning HELLO and aligning a summary
 > buffer?  They are both plain text, and they both are arranged to align
 > nicely.

HELLO arrives as an external plain text stream, and therefore is
governed by the Unicode standard.  The summary buffer is constructed
by Emacs and it is not plain text (it has a *lot* of structure, being
mousable etc), and therefore is not governed by the standard for its
*implementation*.  It *is* governed by the standard for the *display*,
because that is I/O.

IOW, you may be right, but please don't use incorrect arguments like
the ones you have used in this post.  It's just going to confuse
people, and bidi is hard enough already.

 > Using LRM/RLM _is_ TRT, Lars.  It may sound weird to you, but it's
 > like democracy: the worst political system, except all the rest. 

Well, you're the expert, which counts for a lot (and IIRC Mohsen seems
to agree with you, which adds even more credibility).  Nevertheless,
Lars (and the Gnus crews in general) are probably the leading experts
on Emacs edge cases.  You would do well to consult his (their)
expertise on issues like copy/paste and wacky behavior of special text
properties and characters, etc.

 > I wrote the entire Hebrew tutorial with these characters, and didn't see
 > any problems.

Sure, because you started with the intention of producing a plain text
stream conforming to Unicode while implementing correct direction of
exceptional intervals.  But ....  How much copy/pasting did you do?
How many directional marks are needed in the Hebrew TUTORIAL, given
the full BIDI algorithm implementation, and how many are redundant?
Have you copied portions of the TUTORIAL with embedded marks into
email headers and gotten appropriate results?

I bet that, as Lars implies, Emacs is going to need
`yank-dropping-directional-marks' in some applications.  Maybe even
`yank-dropping-directional-marks-at-interval-boundaries' or similar.