Re: [emacs-bidi] reordering based on implicit levels

[Eli Zaretskii]

> From: Alex Schroeder <address@hidden>
> Date: Sun, 11 Nov 2001 05:47:41 +0100
> 
> I finished the code for the reordering function.  If you are
> interested in the code, send me mail.  It's about 30k right now.  You
> need at least one table.  The testing table is around 5k, the real
> (and still incomplete) one is 545k.  :)

Please send me the code and the testing table.

> I'd be interested in a list of test cases -- obviously only test cases
> which are based on implicit levels.

You can find a few test cases in the FriBidi library distribution.  A
word of caution: it is important that you use the same mapping they
use from upper-case to Hebrew or Arabic characters (for example,
letters A-F are AL, IIRC), otherwise the results might be incorrect.
See the table in the file fribidi_char_sets_caps_rtl.c for the full
story.

> I'm not sure what the next steps
> are.  Things which I might want to add is refilling of such thrings,
> working on paragraphs, adding explicit levels (should another
> application put the chars in visual order into the clipboard and leave
> the explicit direction codes in there -- don't know wether such an app
> actually exists), trying to do the right things if being fed chunks of
> bidi text (such as comint buffers).
> 
> Manager?  :)

That's me ;-)

I'd prefer to get the code you have to work on paragraphs, because
that's by far the most frequent use it will get (I think).

Note that UAX#9 rules that every line is a separate paragraph (because
\n is defined as a NEUTRAL_B character), but Emacs cannot behave like
this because we use newlines for filling text.  This is not much of a
complication for you, and perhaps you could even ignore that subtlety
altogether, since visual order doesn't have embedding levels, but
please be aware of this, just in case.

As for explicit formatting codes in visual-order text, I don't think
there is such a thing.  (If someone have seen this, please speak
up.)  UAX#9 clearly says that formatting codes are either removed or
ignored when reordering from logical to visual.

> Here's some code that works.
> 
> (when bidi-testing
>   (assert (string=
>          (bidi-reorder-string
>           "FOO is a word for FOOLS."
>           (bidi-resolve-implicit-levels
>            (bidi-resolve-weak-types
>             (bidi-get-types "FOO is a word for FOOLS."))))
>          "OOF is a word for SLOOF."))

Is this a left-to-right paragraph or a right-to-left one?  The result
is correct for the former.

> The following don't work.  I'm not sure yet, but I think this cannot
> be done using implicit levels only.
> 
> (string=
>  (bidi-reorder-string
>   "I SAW A f14 FLYING BY."
>   (bidi-resolve-implicit-levels
>    (bidi-resolve-weak-types
>     (bidi-get-types "I SAW A f14 FLYING BY."))))
>  ".YB GNIYLF f14 A WAS I")

Again, if this is a left-to-right paragraph, the result of converting
"I SAW A f14 FLYING BY." from visual to logical should be this:

  A WAS I f14 YB GNIYLF.

> (string=
>  (bidi-reorder-string
>   "he said \"I AM GOING TO london TODAY\"."
>   (bidi-resolve-implicit-levels
>    (bidi-resolve-weak-types
>     (bidi-get-types "he said \"I AM GOING TO london TODAY\"."))))
>  "he said \".YADOT london OT GNIOG MA I\"")

And the result of converting this should be

   he said "OT GNIOG MA I london YADOT."

Note that I assumed that bidi-reorder-string performs visual to
logical conversion, not the other way around.