Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Kenichi Handa <address@hidden>
> Cc: address@hidden, address@hidden, address@hidden,
>         address@hidden
> Date: Thu, 04 Aug 2011 15:55:20 +0900
> 
>   The algorithm reorders text only within a paragraph;
>   characters in one paragraph have no effect on characters
>   in a different paragraph. Paragraphs are divided by the
>   Paragraph Separator or appropriate Newline Function (for
>   guidelines on the handling of CR, LF, and CRLF, see
>   Section 4.4, Directionality, and Section 5.8, Newline
>   Guidelines of [Unicode]). Paragraphs may also be
>   determined by higher-level protocols: for example, the
>   text in two different cells of a table will be in
>   different paragraphs.
> 
> The case of summary line belongs to the last sentence above.

That is one possible interpretation, yes.  But it isn't the only one.
The goal is to have correct display of the summary lines, not to
decide whether we are dealing with cells or not.  Directional control
character allow to render summary lines for display correctly even
without "cells".

> In a summary line, we surely have multiple cells, and
> UAX#9's algorithm should be applied to each cell
> independently.
> 
> So what we need is a way to tell the display engine about
> cells themselves or about cell-separators

It's possible, but it will need changes of a different kind in Gnus
(and in other modes that produce similar display).  See below.

> (or artificial paragraph separators).

This alternative, I don't recommend, for the reasons explained below.

> To display something like summarly line, we always use
> separating character(s) (tab, space, |, ...).  How about
> putting the text property `paragraph-separator' to them, and
> making handle_stop (in xdisp.c) to pay attention to that
> property too.

Such a text property is not needed: the UBA already provides a
character whose function is "segment separator": the TAB character.
Here's a demonstration of it in action in Emacs, a single line with 5
fields or "cells" (you need Emacs 24 with bidi-display-reordering
turned on to see it rendered correctly):

    abcde       אבגדה   12345   עבודה   9876

To see how this is rendered by default, without segment separators,
remove the TAB characters between the different fields, leaving just
the blanks:

    abcde אבגדה 12345 עבודה 9876

So we can have such "cells" already, but that requires to add TAB
characters to separate fields in the summary buffers, and I already
hear Lars telling me that this is "a hack a-la VT color escape
sequences".  After all, a TAB is another control character...

TABs also have a disadvantage that the tab stops can be customized,
and the visual appearance depends on the indentation of the first
character in the line.  Whether this is better than LRM, I don't know.

As for "artificial paragraph separators": I don't want to bother
people with too many technicalities (interested readers can read
relevant portions of bidi.c), but supporting "paragraphs" that have no
relation to newlines, i.e. begin and end not on line boundaries, would
mean a significant surgery of the current reordering engine.  So even
if this is deemed as the optimal solution (and I'm not at all
convinced it is), it won't make it in time for Emacs 24.1, at the very
least.

Actually, any solution based on special text properties has a similar
problem: bidi.c works below handle_stop, and ignores text properties
entirely (with the single exception of `display' properties that
replace the underlying text with something else, like images or
display strings).  So any such solution will also be out of the
question for Emacs 24.1.

So eventually, this discussion boils down to this: do we want Gnus and
other Emacs-based MUAs to have reasonable support in their summary
buffers for R2L scripts in Emacs 24.1, or don't we?  If we do (and I
surely hope so), then we must use only the facilities available now,
which include (1) LRM characters and (2) TABs to separate the "cells".
Anything else will simply not work in Emacs 24.1.