bug#27525: 25.1; Line wrapping of bidi paragraphs

[Itai Berli]

And in case you counter that Emacs takes advantage of the "higher-level protocol" clause, this clause doesn't apply to paragraph 3.4, which Emacs violates.

But don't take my word for it. I contacted Mr. Aharon Lanin in the matter. Mr. Lanin is a senior software engineer at Google Tel Aviv as well as one of the three editors of the Unicode Bidirectional Algorithm from version 6.3.0 till the latest one, v. 10.0.0. I presented him with the following screenshot of a bidi paragraph in Emacs (it's the same screenshot as I posted originally in the present ticket): http://imgur.com/Bckn7zP

I list below an excerp from our conversation, which Mr. Lanin has given me permission to quote.

--- EXCERPT BEGIN ---

***Me***: Just to be clear, are the following statements correct for the Unicode Standard v. 8.0.0 and above? (I'm mentioning v. 8.0.0 because this is the version that Emacs claims conformance to.)

1. The way Emacs handles line wrapping of bidi paragraphs does not satisfy section 3.4 of the Unicode Bidirectional Algorithm. There are no provisions for higher-level protocol interpretation of this section.

2. If a candidate implementation of the Unicode Bidirectional Algorithm doesn't satisfy section 3.4, it does not conform to the Unicode Bidirectional Algorithm.

3. If a candidate implementation of the Unicode Standard does not conform to the Unicode Bidirectional Algorithm (of the same version), it does not conform to the Unicode Standard.

***Lanin*** I think so, but I am a programmer, not a lawyer :-)

--- EXCERPT END ---

The Emacs manual and all official Emacs publications should make it clear that Emacs does not conform to the Unicode Standard. Anything else is simply not true, and is a deliberate misleading.

On Wed, Jul 19, 2017 at 11:50 AM, Itai Berli <itai.berli@gmail.com> wrote:

Eli, in different bug report, namely 27526, I recently wrote the following remark:

> the line-wrapping bug is still a major annoyance, at best, and until it is fixed, Emacs cannot claim to be Unicode compliant.

to which you replied:

> I disagree, as I already said many times.

You do agree, though, that Emacs does not conform to the Unicode Bidirectional Algorithm as specified in the Unicode Standard Annex #9. After all, the following paragraph appears in the bidi code itself (http://git.savannah.gnu.org/cgit/emacs.git/tree/src/bidi.c):

   Note that, because reordering is implemented below the level in
   xdisp.c that breaks glyphs into screen lines, we are violating
   paragraph 3.4 of UAX#9. which mandates that line breaking shall be
   done before reordering each screen line separately.

So the only thing you disagree with me is that non-conformance to the Unicode Bidirectional Algorithm is tantamount to non-conformance to the Unicode Standard. Well, this disagreement is easily settled by reading article C12 'Bidirectional Text' of section 3.2 'Conformance Requirements' of the Unicode Standard:

A process that displays text containing supported right-to-left characters or embedding codes shall display all visible representations of characters (excluding format characters) in the same order as if the Bidirectional Algorithm had been applied to the text, unless tailored by a higher-level protocol as permitted by the specification.

* The Bidirectional Algorithm is specified in Unicode Standard Annex #9, “Uni- code Bidirectional Algorithm.”

On Sun, Jul 9, 2017 at 9:30 PM, Eli Zaretskii <eliz@gnu.org> wrote:

> From: Benjamin Riefenstahl <b.riefenstahl@turtle-trading.net>
> Cc: 27525@debbugs.gnu.org,  itai.berli@gmail.com
> Date: Sun, 09 Jul 2017 20:17:41 +0200
>
> >> From: Richard Stallman <rms@gnu.org>
> >> It might be good if Emacs could refill lines automatically the way
> >> some other ediors do.
>
> Eli Zaretskii writes:
> > We already have that: "M-x visual-line-mode RET".
>
> JFTR, even that does not help in this case.  With visual-line-mode the
> order of the lines is still wrong with the text that the OP gave.

Of course.  It isn't supposed to help.  From the POV of the display
engine, visual-line-mode is just a fancy kind of producing
continuation lines, so all the problems you see with continued lines
will still be there in visual-line-mode.