bug#11094: Wrong cursor positioning with display+invisible

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Mon, 26 Mar 2012 00:07:32 -0400
> 
> Extract the test.cpio and cpio-mode.el files attached, and then try:
> 
>    src/emacs -Q -l .../cpio-mode.el .../test.cpio

Ouch!

(Btw, test.cpio is an invalid archive, according to 2 different
implementations of the cpio command.  The one from GNU/Linux says
"cpio: premature end of file".  Not that it matters for the purposes
of this discussion.)

> First, you'll note that the cursor is displayed at end-of-line (whereas
> in reality point as it BOB), then move forward with C-f and you'll see
> the cursor jumping in odd ways (often being pushed to end-of-line even
> when we're still in the middle of the line).

Right.  The end-of-line placement of the cursor usually means that the
display engine knows the cursor should be on this line, but it didn't
find a proper place to put it.  And in this case, for a good reason,
see below.

> In Emacs-23, the cursor jumps correctly over the various display and
> invisible fields

For some value of "correctly".  E.g., position the cursor over the
".", which is the first file name in the archive, and type "C-x =".
You will see "63", which is a lie: the actual value of point is 111.
This lie becomes more evident if you position the cursor on the first
's' of "scripts", the file name on the second line: "C-x =" says "175"
(whereas the truth is 223), but if you now type "C-f C-x =", you will
see "224".  Huh?

Anyway, that this "works" in Emacs 23 and before is more by accident
than by anything else.  Your cpio-mode.el almost completely conceals
the buffer contents from the display engine, either by 'invisible'
property or behind display strings.  The cursor-positioning code is
therefore completely helpless when it comes to decide where to put the
cursor, because all it has is (a) the value of point, and (b) the
glyphs generated for each display line (a.k.a. "glyph row").  With
most of the buffer positions covered by 'invisible' and 'display'
properties, no glyphs to place the cursor on can be found for most
values of point, since there are no glyphs which "tell" the display
engine that they came from those buffer positions.

The old, unidirectional display engine could get away (albeit by the
skin of its teeth) with such code because it relied on buffer
positions to increase monotonically with screen positions.  So once it
found a glyph from buffer position N that is greater or equal to the
value of point, it could place the cursor there, because it "knew" all
the later glyphs _must_ correspond to positions beyond N.  That is why
it "works" in Emacs 23.

A case in point is the position of cursor at BOB.  The value of point
is 1, whereas the first glyph on the first screen lines comes from
buffer position 15(!).  The old display code could put the cursor
there _only_ because that is the first glyph whose buffer position is
greater or equal to 1.

However, the bidirectional display engine can no longer make such
assumptions, so it is helpless without at least _some_ clue about the
buffer positions corresponding to the glyphs.  And your code gives
almost no such clues.

I think we should either (1) from now on discourage (i.e., maintain
that we don't support) code that covers too much of buffer text by
display strings, or (2) completely redesign 'struct glyph', so that it
keeps more information about the buffer positions "associated" with
the glyph, and then refactor the code which needs that information.
(The second alternative is, of course, not for Emacs 24.)

Alternatively, may I suggest that you use the 'cursor' property to
provide some rope for the display engine to do its job even in this
situation?  The attached variant of your code demonstrates how to do
that; it works equally well with Emacs 23 and with the current trunk.

Bottom line, to resolve this bug for Emacs 24.1, I suggest a judicial
use of the 'cursor' property as the only safe (and IMO the only sane)
alternative.  If you still think we should support your original code,
we should schedule some post-24.1 redesign and refactoring.  Let me
know what you think.

Here's a modified version of cpio-mode.el.  Note that it removes most
of the 'invisible' properties, because they are not really needed, and
because they have an unpleasant side effect of reporting incorrect
value of point due to "point adjustments".  I suspect that all the
other 'invisible' properties should be removed as well, but I didn't
test the situations where they would come into play.

cpio-mode1.el
Description: application/emacs-lisp