bug#35721: 27.0.50; Strange Arabic shaping behavior

[Eli Zaretskii]

> From: "Basil L. Contovounesios" <contovob@tcd.ie>
> Date: Mon, 13 May 2019 23:09:06 +0100
> 
> I see the following on the master, harfbuzz, and emacs-26 branches
> (precise versions follow my signature), but I'm not sure how much of
> this is expected or due to e.g. my font.
> 
> 0. emacs -Q
> 1. C-x 8 RET 0634 RET
> 
> The "tail" of the sheen is truncated by the fringe:

After looking at the code and thinking about this, I think this is a
feature (as strange as it may sound); see below for why I think so.
And yes, it definitely depends on the font, in this case DejaVu Sans
Mono.  I don't see this with any other fixed-pitch font I have.

I'm not 100% sure I'm right here, so I CC Behdad and Kenichi in the
hope that they will comment on this.  Behdad, I see a very similar
issue with hb-view, when it renders this character using DejaVu Sans
Mono, so it isn't just an Emacs issue (and seeing what hb-view
produces actually made me think my opinion is correct about this).

> 3. SPC
> 
> The "tail" of the sheen becomes visible, but falls outside of the box
> cursor:

Yes, this particular font's glyph for sheen has a negative value of
left bearing.  Which AFAIU means it extends beyond the box dimensions
to the left.

> 4. C-x 8 RET 0643 RET
> 
> The kaf is correctly shaped in its initial form:
> 
> 5. C-SPC
> 
> The kaf changes to its isolated form:

This is different problem, related to how we redraw portions of the
buffer inside the region (more generally, those which have colors
different from the default face).

The problem is that we only pass to the shaping engine stretches of
text that have the same face.  The basic reason for that is that a
different face can use a different font, and we can only handle
character composition for characters supported by the same font.
Another fundamental reason is that the display engine processes text
in chunks that have the same face.  So when the active region, or some
other Emacs feature, paints portions of text in some non-default face,
we redraw the display, and pass to the shaping engine only the portion
that has that different face.  If that portion is a single character,
you will see that it loses its correct shape and is rendered in its
isolated form.  And if the colors change between two characters that
need to be shaped together, the shaping will break.

You can easily see this effect if you display HELLO, and then
shift-select portions of the Arabic greeting (or any other script that
is a heavy user of character compositions).

To fix this, we need some mechanism that will pass larger chunks of
text to the shaper in these cases, which will need some changes in how
the display engine iterates through buffer/string text when it
prepares them for display: we currently stop at every change of face.

Patches to fix this are most welcome.

> I occasionally see this happen even without typing anything, as if by a
> timer, but I'm not sure how to reproduce it.  I think, without being
> 100% certain, that it's only happened while using the 'arabic' input
> method.

Maybe, but given my description above, I'm not surprised, because it's
enough that Emacs decides, for some reason, to redraw just that one
character.

Now to the original problem.  Let me turn the table and ask you: what
did you expect to happen instead?  This is a fixed-pitch font, so how
can Emacs display a character that extends to the left from its box,
at the left-most window coordinate?  It has no choice but consider its
extension be off-screen, as if the window was hscrolled.

The "normal" case for this character is to be part of R2L text, which
begins at the right window margin, and flows to the left.  In that
case, the extension will overlap the character cell of the next (in
the logical order) character.

So I think we have no bug here, we behave as expected.  If not, I'm
sure Behdad and Kenichi will correct me.