bugs and documentation issues: cursors, text/overlay properties (display, before-string, after-string, invisible, intangible), column numbers

[Joe Wells]

Here are some assorted bugs and documentation issues.  They affect
cursors, column numbers, and these text/overlay properties: display,
before-string, after-string, invisible, intangible.

Basically I went through a bug report I sent 5 years ago for Emacs
21.2 and checked what the current status is.  It seems some of the
issues were fixed, but the fixes were not documented, so for these I
am reporting the documentation issues.  Some bugs are still the same.
In some cases, the behavior is better now, but new anomalous behavior
has appeared.

Now here are the issues:

ISSUE 1:  The "Cursor Display" Emacs manual node says this:

     Normally, the cursor appears in non-selected windows in the "off"
  state, with the same appearance as when the blinking cursor blinks
  "off."  For a box cursor, this is a hollow box; for a bar cursor, this
  is a thinner bar.  To turn off cursors in non-selected windows,
  customize the variable `cursor-in-non-selected-windows' and assign it a
  `nil' value.

Contradicting the above documentation, if the cursor-type
variable is set to bar, then a thinner bar is not used when the
window is not selected:

  (progn
    (delete-other-windows)
    (kill-buffer (get-buffer-create "test"))
    (set-buffer (get-buffer-create "test"))
    (setq cursor-type 'bar)
    (display-buffer (current-buffer)))

To get the documented effect, it is necessary to also set the
variable cursor-in-non-selected-windows, like this:

  (progn
    (delete-other-windows)
    (kill-buffer (get-buffer-create "test"))
    (set-buffer (get-buffer-create "test"))
    (setq cursor-type 'bar)
    ;; The fix is to use this new variable which should be documented
    ;; at cursor-type but is not yet.
    (setq cursor-in-non-selected-windows 'bar)
    (display-buffer (current-buffer)))

ISSUE 2:  The documentation string for
cursor-in-non-selected-windows does not mention any distinctions
other than nil vs. non-nil:

  *Non-nil means show a hollow box cursor in non-selected windows.
  If nil, don't show a cursor except in the selected window.
  Use Custom to set this variable to get the display updated.

Similarly, the only mention of cursor-in-non-selected-windows in
the Emacs manual is the one quoted above in issue 1, which also
does not mention the other possible meaningful values of
cursor-in-non-selected-windows.  It seems that the only place
where cursor-in-non-selected-windows is fully documented is in
the etc/NEWS file.

ISSUE 3:  When the display property is given a display
specification which replaces the text, then point is not allowed
to be interactively placed inside the replaced text.  While this
is reasonable behavior, it is not documented in the Emacs Lisp
manual section on "The `display' Property".

ISSUE 4:  A before-string property on a range of characters causes
cursor problems.  If point is just before all of the characters
with the before-string property, the cursor is displayed _after_
the string which is the value of the before-string
property.  (This issue was first reported in 2002.)

(progn
  (delete-other-windows)
  (kill-buffer (get-buffer-create "test"))
  (set-buffer (get-buffer-create "test"))
  (insert "AB")
  (let ((o (make-overlay 2 3))) ;; covering the B
    (overlay-put o 'before-string "YZ")
    ;;(overlay-put o 'display "C")
    )
  (goto-char 2) ;; point now between A and B
  (display-buffer (current-buffer))
  )

In the above example, because the characters "YZ" are logically
part of the display of "B", and because point is between "A"
and "B", then logically the cursor should be presented before
the "YZ".

ISSUE 5:  An after-string and invisible property on a range of
characters causes cursor problems.  If point is just before all
of the characters with the after-string and non-nil invisible
property, the cursor is displayed _after_ the string which is the
value of the after-string property.  (This issue was first
reported in 2002.)

(progn
  (kill-buffer (get-buffer-create "test"))
  (set-buffer (get-buffer-create "test"))
  (insert "AB")
  (let ((o (make-overlay 2 3))) ;; covering the B
    (overlay-put o 'after-string "YZ")
    (overlay-put o 'invisible t))
  (goto-char 2) ;; between A and B
  (display-buffer (current-buffer))
  (what-cursor-position) ;; shows point is _before_ the B
  )

For the same reasons mentioned for the previous issue, the cursor
should logically be displayed just after the "A" in this example.

ISSUE 6:  The previous-line and next-line commands skip over lines
that have legal spots to move to.

Here is a test case that shows the problem with previous-line.

Warning:  These test cases do not always produce the same
behavior (despite the fact that they destroy and recreate all
windows and buffers involved!).  When invoked 2 or more times in
a row, the test case always produces stable behavior on the
invocations after the first.

(progn
  (delete-other-windows)
  (kill-buffer (get-buffer-create "test"))
  (set-buffer (get-buffer-create "test"))
  (insert "A\nB\nC\nD\nE")
  (let ((o (make-overlay 3 8))) ;; covering the "B\nC\nD"
    (overlay-put o 'intangible t))
  (goto-char 9)
  (previous-line 1)
  (display-buffer (current-buffer))
  ;; shows point has skipped all the way to the line with the B, while
  ;; instead it should be just after the D:
  (what-cursor-position)
  )

Similarly, here is a test case that shows the problem with next-line:

(progn
  (delete-other-windows)
  (kill-buffer (get-buffer-create "test"))
  (set-buffer (get-buffer-create "test"))
  (insert "A\nB\nC\nD\nE")
  (let ((o (make-overlay 3 8))) ;; covering the "B\nC\nD"
    (overlay-put o 'intangible t))
  (goto-char 2)
  (next-line 1)
  (display-buffer (current-buffer))
  ;; shows point has skipped all the way to the line with the D, while
  ;; instead it should be just before the B:
  (what-cursor-position)
  )

ISSUE 7:  Column numbers as used by previous-line and next-line
have surprising (to the naive user) effects when a display
property (overlay or text property) is used which causes
additional characters to be displayed or prevents characters in
the buffer from being displayed.  Strictly speaking, this may not
be a bug, but it is highly undesirable behavior.  There should be
a way to have motion to a particular column based on what is
displayed rather than on what is in the buffer.

First, observe that column numbers are handled nicely when the
invisible property is used:

(progn
  (delete-other-windows)
  (kill-buffer (get-buffer-create "test"))
  (set-buffer (get-buffer-create "test"))
  (insert "ABCDEFGHI\nABCDEFGHI")
  (let ((o (make-overlay 3 5))) ;; covering the "CD"
    (overlay-put o 'invisible t))
  (goto-char 7) ;; now between the "F" and the "G"
  (next-line 1)
  (display-buffer (current-buffer))
  ;; now between the "D" and "E" (as desired)
  )

Then, observe that column numbers are _not_ handled nicely when the
display property is used instead:

(progn
  (delete-other-windows)
  (kill-buffer (get-buffer-create "test"))
  (set-buffer (get-buffer-create "test"))
  (insert "ABCDEFGHI\nABCDEFGHI")
  (let ((o (make-overlay 3 5))) ;; covering the "CD"
    (overlay-put o 'display ""))
  (goto-char 7) ;; now between the "F" and the "G"
  (next-line 1)
  (display-buffer (current-buffer))
  ;; now between the "F" and "G" (undesired)
  )

For this purpose, a display property of "" should behave the same as a
non-nil invisible property, but it does not.

Warning:  The test cases above are unstable in the same way as the
test cases for issue 6 above.  They do not always produce the
same behavior, but do seem to produce the same behavior on the
second and subsequent uses when repeated.  You can see some very
strange behavior by alternating invocations of the two test cases
in this issue.

I hope this report is useful.

Joe

======================================================================
In GNU Emacs 22.1.1 (i686-pc-linux-gnu, GTK+ Version 2.8.20)
 of 2007-06-27 on artemis
Windowing system distributor `The X.Org Foundation', version 11.0.70000000
configured using `configure  '--prefix=/home/jbw/local2' '--enable-debug' 
'--disable-nls' '--with-x-toolkit=gtk' 'CFLAGS=-O0 -g3 -ggdb''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: en_US.UTF-8
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: jbw
  value of $LANG: nil
  locale-coding-system: utf-8
  default-enable-multibyte-characters: t

Minor modes in effect:
  tooltip-mode: t
  tool-bar-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  unify-8859-on-encoding-mode: t
  utf-translate-cjk-mode: t
  auto-compression-mode: t
  line-number-mode: t