Bug: Org sentence commands and emphasis markers [9.3 (release_9.3 @ /usr/local/share/emacs/27.1/lisp/org/)]

[Gustavo Barros]

Hi All,

There is some adverse interaction between Emacs sentence related 
commands and Org emphasis markers, when whole sentences are emphasized. 
This report describe some cases of this interaction, for your 
consideration.  But, as a general rule, if a whole sentence is 
emphasized, sentence movement and killing, and paragraph filling won't 
work as expected.

Let's take the case of a small paragraph, with sentences ended with 
double spaces, and with `sentence-end-double-space' having its default 
value of t.  And let's have the whole second sentence emphasized with 
bold markers.  "!" represents point position.

#+begin_src org
Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy 
eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam 
voluptua.  *At vero eos et accusam et justo duo dolores et ea rebum.* 
Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor 
sit amet.  Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed 
diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam 
erat, sed diam voluptua.  At vero eos et accusam et justo duo dolores et 
ea rebum.  Stet clita kasd gubergren, no sea takimata sanctus est Lorem 
ipsum dolor sit amet.
#+end_src

1 - Sentence movement

If we call on this paragraph the regular Org sentence movement commands 
`org-backward-sentence' and `org-forward-sentence', the second sentence 
will no longer be recognized as such.  Starting from the beginning of 
the paragraph and calling `org-forward-sentence' will result in:

#+begin_src org
!Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam 
nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, 
sed diam voluptua.  *At vero eos et accusam et justo duo dolores et ea 
rebum.*  Stet clita kasd gubergren, no sea takimata sanctus est Lorem 
ipsum dolor sit amet.  Lorem ipsum dolor sit amet, consetetur 
sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et 
dolore magna aliquyam erat, sed diam voluptua.  At vero eos et accusam 
et justo duo dolores et ea rebum.  Stet clita kasd gubergren, no sea 
takimata sanctus est Lorem ipsum dolor sit amet.
#+end_src

#+begin_src org
Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy 
eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam 
voluptua.!  *At vero eos et accusam et justo duo dolores et ea rebum.* 
Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor 
sit amet.  Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed 
diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam 
erat, sed diam voluptua.  At vero eos et accusam et justo duo dolores et 
ea rebum.  Stet clita kasd gubergren, no sea takimata sanctus est Lorem 
ipsum dolor sit amet.
#+end_src

#+begin_src org
Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy 
eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam 
voluptua.  *At vero eos et accusam et justo duo dolores et ea rebum.* 
Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor 
sit amet.!  Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed 
diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam 
erat, sed diam voluptua.  At vero eos et accusam et justo duo dolores et 
ea rebum.  Stet clita kasd gubergren, no sea takimata sanctus est Lorem 
ipsum dolor sit amet.
#+end_src

That is, the end of the second (emphasized) sentence was missed.  A 
similar problem occurs when going backwards with 
`org-backward-sentence'.

2 - Sentence killing

With point at the end of the first sentence, as in;

#+begin_src org
Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy 
eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam 
voluptua.!  *At vero eos et accusam et justo duo dolores et ea rebum.* 
Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor 
sit amet.  Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed 
diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam 
erat, sed diam voluptua.  At vero eos et accusam et justo duo dolores et 
ea rebum.  Stet clita kasd gubergren, no sea takimata sanctus est Lorem 
ipsum dolor sit amet.
#+end_src

If we call `kill-sentence' we will get:

#+begin_src org
Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy 
eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam 
voluptua.!  Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed 
diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam 
erat, sed diam voluptua.  At vero eos et accusam et justo duo dolores et 
ea rebum.  Stet clita kasd gubergren, no sea takimata sanctus est Lorem 
ipsum dolor sit amet.
#+end_src

That is, the second and third sentences were killed.  A similar problem 
occurs with point somewhere on the third sentence and 
`backward-kill-sentence'.

3 - Paragraph filling

If we `org-fill-paragraph' the following:

#+begin_src org
Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy 
eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam 
voluptua.  *At vero eos et accusam et justo duo dolores et ea rebum.* 
Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor 
sit amet.  Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed 
diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam 
erat, sed diam voluptua.  At vero eos et accusam et justo duo dolores et 
ea rebum.  Stet clita kasd gubergren, no sea takimata sanctus est Lorem 
ipsum dolor sit amet.
#+end_src

We get:

#+begin_src org
Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy
eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam
voluptua.  *At vero eos et accusam et justo duo dolores et ea rebum.* 
Stet clita
kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. 
Lorem
ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy 
eirmod
tempor invidunt ut labore et dolore magna aliquyam erat, sed diam 
voluptua.
At vero eos et accusam et justo duo dolores et ea rebum.  Stet clita 
kasd
gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.
#+end_src

That is, the double space at the end of the second (emphasized) sentence 
is gone.


Different values of `org-hide-emphasis-markers' have no influence over 
the issue.  Setting `sentence-end-double-space' to nil will make the 
filling issue irrelevant, but otherwise things are the same.

The examples were tested with "GNU Emacs 27.1 (build 1, 
x86_64-pc-linux-gnu, GTK+ Version 3.24.20, cairo version 1.16.0) of 
2020-08-01" and "Org mode version 9.3.7 (9.3.7-15-gc9abb4-elpaplus @ 
/home/gustavo/.emacs.d/elpa/org-plus-contrib-20200727/)", and "emacs 
-Q".


Best,
Gustavo.



Emacs  : GNU Emacs 27.1 (build 1, x86_64-pc-linux-gnu, GTK+ Version 
3.24.20, cairo version 1.16.0)
of 2020-08-01
Package: Org mode version 9.3 (release_9.3 @ 
/usr/local/share/emacs/27.1/lisp/org/)

current state:
==============
(setq
org-src-mode-hook '(org-src-babel-configure-edit-buffer 
org-src-mode-configure-edit-buffer)
org-link-shell-confirm-function 'yes-or-no-p
org-metadown-hook '(org-babel-pop-to-session-maybe)
org-clock-out-hook '(org-clock-remove-empty-clock-drawer)
org-reveal-start-hook '(org-decrypt-entry)
org-mode-hook '(#[0 "\300\301\302\303\304$\207" [add-hook 
change-major-mode-hook org-show-all append local] 5]
                 #[0 "\300\301\302\303\304$\207"
		   [add-hook change-major-mode-hook 
		   org-babel-show-result-all append local] 5]
                 org-babel-result-hide-spec org-babel-hide-all-hashes)
org-archive-hook '(org-attach-archive-delete-maybe)
org-confirm-elisp-link-function 'yes-or-no-p
org-agenda-before-write-hook '(org-agenda-add-entry-text)
org-metaup-hook '(org-babel-load-in-session-maybe)
org-bibtex-headline-format-function #[257 "\300\236A\207" [:title] 3 
"\n\n(fn ENTRY)"]
org-babel-pre-tangle-hook '(save-buffer)
org-tab-first-hook '(org-babel-hide-result-toggle-maybe 
org-babel-header-arg-expand)
org-occur-hook '(org-first-headline-recenter)
org-cycle-hook '(org-cycle-hide-archived-subtrees 
org-cycle-show-empty-lines
                  org-optimize-window-after-visibility-change)
org-speed-command-hook '(org-speed-command-activate 
org-babel-speed-command-activate)
org-confirm-shell-link-function 'yes-or-no-p
org-link-parameters '(("attachment" :follow org-attach-open-link 
:export org-attach-export-link :complete
                        org-attach-complete-link)
		       ("id" :follow org-id-open) ("eww" :follow eww 
		       :store org-eww-store-link)
		       ("rmail" :follow org-rmail-open :store 
		       org-rmail-store-link)
		       ("mhe" :follow org-mhe-open :store 
		       org-mhe-store-link)
		       ("irc" :follow org-irc-visit :store 
		       org-irc-store-link :export org-irc-export)
		       ("info" :follow org-info-open :export 
		       org-info-export :store org-info-store-link)
		       ("gnus" :follow org-gnus-open :store 
		       org-gnus-store-link)
		       ("docview" :follow org-docview-open :export 
		       org-docview-export :store
                        org-docview-store-link)
		       ("bibtex" :follow org-bibtex-open :store 
		       org-bibtex-store-link)
		       ("bbdb" :follow org-bbdb-open :export 
		       org-bbdb-export :complete org-bbdb-complete-link
                        :store org-bbdb-store-link)
		       ("w3m" :store org-w3m-store-link) ("file+sys") 
		       ("file+emacs")
                       ("shell" :follow org-link--open-shell)
		       ("news" :follow #[257 "\301\300\302Q!\207" 
		       ["news" browse-url ":"] 5 "\n\n(fn URL)"])
		       ("mailto" :follow #[257 "\301\300\302Q!\207" 
		       ["mailto" browse-url ":"] 5 "\n\n(fn URL)"])
		       ("https" :follow #[257 "\301\300\302Q!\207" 
		       ["https" browse-url ":"] 5 "\n\n(fn URL)"])
		       ("http" :follow #[257 "\301\300\302Q!\207" 
		       ["http" browse-url ":"] 5 "\n\n(fn URL)"])
		       ("ftp" :follow #[257 "\301\300\302Q!\207" ["ftp" 
		       browse-url ":"] 5 "\n\n(fn URL)"])
		       ("help" :follow org-link--open-help) ("file" 
		       :complete org-link-complete-file)
		       ("elisp" :follow org-link--open-elisp) ("doi" 
		       :follow org-link--open-doi))
org-link-elisp-confirm-function 'yes-or-no-p
)