Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Chong Yidong <address@hidden>
> Cc: address@hidden, address@hidden
> Date: Tue, 16 Aug 2011 18:24:24 -0400
> 
> Eli Zaretskii <address@hidden> writes:
> 
> > Won't a newline and/or LRM/RLM (the latter inside the string) fix
> > that?  In general, all reordering information is tossed at every
> > newline and restarted anew for the next line.  So judicious placement
> > of newlines should do the trick in most cases.
> 
> Having to juggle newlines when editing source code is no good.

It's an annoyance, I agree.  We should lift this limitation as soon as
we can.

> The ideal long-term solution, I think, is to make the bidi display code
> aware of text properties, as several people have already suggested.

I'm afraid making the reordering engine aware of all text properties
will considerably slow down redisplay, due to the need to check
character properties very frequently.  It also runs a high risk of
completely blending the reordering code with the display engine, which
will make them both very hard to maintain; currently, they are clearly
separated.

In addition, the current design of bidi.c does not support starting
and stopping reordering on arbitrary character positions; it only does
that at BEGV and ZV.  If possible, I'd like to avoid changing that.

The idea I currently favor is to use the `display' properties for this
purpose.  bidi.c already takes notice of those `display' properties
which replace the underlying text with something else, like an image
or another text.  The text covered by such properties is reordered as
a single long character, and if a property is a string, it is then
reordered separately by pushing the iterator object onto a stack and
restarting the iteration, including reordering, anew for the display
string.

So suppose we define a new display spec, something like

 'display '(bidi-reorder BASE-DIRECTION)

where BASE-DIRECTION, either left-to-right or right-to-left, is the
"paragraph direction" to be in effect while reordering the text
covered by the display spec.  Then we add code to the display engine
to implement this spec by pushing the iterator, reinitializing the
iteration for the span of this display property, then popping it when
it's done.  In effect, we create a "display string" out of a chunk of
buffer text, and then reorder it separately.

With this feature, a major mode can put such display properties on
whichever parts of buffer text it wants.  This will work regardless of
whether the buffer itself has bidi-display-reordering set to nil or t,
because the implementation of such display spec will force reordering
for the covered text regardless of the parent buffer.

> Then it should be easy to exploit font-lock to give reasonably correct
> bidi segmentation, e.g. by treating font-lock-comment-face and
> font-lock-string-face boundaries as bidi segmentation boundaries.

We should be very careful with reusing font-lock as basis for
reordering, because the user has too much knobs to control font-lock.
For example, few of the font-lock features speed up redisplay by
deferring fontification to a later time.  With font-lock, this just
displays text in the default face; with reordering, it will flush
incorrectly rendered text for a perceptible amount of time.  I'm not
sure it's a good idea.

There are also people who disable font-lock completely.

We could use font-lock faces, when they exist, as a cue for placing
the `display' properties described above, though.

> For now, I don't feel strongly about the idea of turning off bidi
> display in prog modes.  But if we don't, we should at least document
> some of the pitfalls discussed in this thread, and maybe state that
> having RTL script in source code buffers is currently a bad idea.

I'll add this to my todo and revisit that during the pretest.