Re: [emacs-bidi] Supporting non-plain-text buffers

[Eli Zaretskii]

> From: Beni Cherniavsky-Paskin <address@hidden>
> Date: Fri, 2 Jul 2010 13:43:33 +0300
> Cc: "Martin J. Dürst" <address@hidden>, address@hidden
> 
> I'm second-guessing Martin here (correct me if I'm wrong!) but I suspect
> this proposal is based on:
> 
> (A) HTML bidi experience, where CSS already supports before/after-string,
>     and browsers respect bidi control codes appearing there (?)
> 
> (B) a presumption that it's the easiest for you because you planned to
>     implement it anyway.

I'm not following, probably because I don't know enough about CSS.

> In an ideal world, we would have a separate "bidi-lock" engine,
> hand-tuned for each syntax in the world, parsing it and endowing
> characters with whatever overlays/properties the display engine
> needs to show it in perfect order.

This isn't possible, because overlays and text properties cannot be
used for storing reordering information.  I explained in my previous
message why.

With the current implementation, bidi reordering happens only when a
portion of text is being delivered to the glass.  Then the information
about reordering is thrown away (and recomputed as needed each time
the same portion of text is about to be displayed again).  Therefore,
it would be useless to try to "bidi-lock" invisible portions of the
buffer -- you will have no place to record the results of this
reordering anyway, and no way of reusing that information even if
recorded somewhere.

> A much more realistic (even if less perfect) approach
> is to harness existing polished font-lock engine, defining bidi
> in terms of it output (e.g. "characters with font-lock-string-face
> should be treated as embeddings and not mix with outside text").

Maybe.  But note that this would mean we cannot properly display
comments and strings if font-lock mode is turned off in the buffer (or
globally).

> So currently there is no way whatsoever by which lisp code (even the
> hypothetical perfect bidi-lock) can affect reordering?

No.  If we think this would be needed, a useful first step would be to
identify the use-cases which require that.

> (Except for actually inserting characters into the buffer, which we
> agree is no-no)

It is a no-no for display purposes.  But for other purposes,
e.g. forcing paragraph direction on specific paragraphs, it's actually
the way to go, because you do want that direction to be in force even
outside Emacs, in any other conforming application.

> 1. What bidi metadata exactly do we want?
>    Emacs has many kinds of metadata (face info, invisibility,
>    local vars/keymap, hooks...) but we'll need new one(s).
>    - Virtual control characters?
>    - An increment to embedding level?
>    - Overridding character's unicode bidi cathegory?
>    ...
> 
>    It's an interesting discussion, but I don't have much to say now.
> 
> 2. How will it be attached to the text?
>    It seems that Emacs has exactly two ways to attach metadata to text:
>    - Text properties
>    - Overlays (implemented by pairs of markers?)
> 
>    Almost all metadata can be attached by both, although a few (e.g.
>    before/after-string) only work on overlays.
> 
>    Also, there there are several indirection mechanisms, notably faces:
>    a lot of metadata (such as font size) is not directly attached to
>    text/overlays but to a face object (which is a property of text/overlay).
> 
> Let's concentrate on question 2 here.

Again, I think we should start with identifying the use-cases which
would need these features.

> From the implementation POV:
> How would you like bidi information to be attached so the bidi engine
> could easily inspect it?  Is there a shortcut at all, or would *any*
> support require changing the architecture to do full-blown resolution
> of text properties before bidi?

Text properties are resolved before bidi reordering anyway, for 2
reasons:

  1) Emacs enters redisplay only when it is idle, i.e. all the buffers
     are already modified according to all the available input events.

  2) The display engine examines the text properties as part of its
     work (that's how it can take into account faces, invisibility,
     etc.), so these properties need to be known by the time we
     display the text.

So I don't see what architecture changes would be necessary, or why.

In principle, the way to affect bidi reordering is to attach some
bidi-specific information to a portion of text.  The kinds of
information I could think of is:

  . forcing specific base direction or embedding level of a paragraph

  . changing the bidi properties of specific characters

  . limiting reordering to specific portions of the buffer

None of this is implemented at present.  Some of them would be easy to
implement, others hard.  For example, the first one introduces some
really nasty side effects, like what happens if characters are
prepended in front of the paragraph.

To justify each one of these, I would need to see use-cases that
cannot be resolved within the current implementation.

Whether to use text properties or overlays for attaching this
information to portions of the buffer depends on whether we want the
information go with the text when it is cut/pasted elsewhere.  Again,
such kind of decisions is impossible without specific use-cases.

> AFAIU, font-lock only sets faces as direct text properties (not overlays),
> right?

Right.

> So, it would be convenient to indirect bidi metadata through faces -
> although in general it seems a weird place to put it.

Why faces?  Text properties are not limited to faces.  They can be any
symbol/property list.