Bug: refiling gobbles a newline and absorbs the next heading [9.1.9 (release_9.1.9-65-g5e4542 @ /Applications/Emacs.app/Contents/Resources/lisp/org/)]

[Miguel Morin]

Hello,

TL;DR: starting a capture item and deleting the new lines at the end of the 
capture buffer gobbles the next heading, and refiling moves the gobbled heading 
along with the new capture item.

I asked on emacs.stackexchange and found no solution:

https://emacs.stackexchange.com/questions/53526/org-mode-refiling-gobbles-a-newline-and-absorbs-the-next-heading

In the course of fixing this, I found another bug in using the hook 
`org-capture-before-finalize-hook`, which adds text at the beginning and end of 
the org buffer and not in the beggining and end of the capture item.

Full details, copied from the StackExchange thread:

I often have this problem in Org mode: I write an item with org-capture, refile 
it, and days later I see a result like this:

<pre>
*** TODO some task
some notes** StackExchange
</pre>

I believe that the problem is that org capture inserts the text into the 
buffer, and if the item does not end with a newline, it gobbles the delimiter 
to the next heading and both get refiled together.

I already added two newlines to my capture template to minimize this problem. 
My capture template is:

``` lisp
(setq org-capture-templates
      '(("t" "Todo" entry (file+headline "~/org/gtd.org" "Reminders")
         "* TODO %?\n\n"))
```

If I try a new item with `C-c c t` and type something, I see it appear in the 
main buffer. If I delete the newline at the end, the newline is deleted in the 
buffer too and gobbles the next line. Deleting further characters in the 
capture buffer does not delete them from the main buffer.

I think I could add a hook such as `(add-hook 'org-capture-before-finalize-hook 
...)` but I'm not sure how to include a newline and how to do it before 
refiling, to avoid the next item being refiled with the captured item.

How can I fix this problem?

**Update**: Following @Aquaactress's suggestions, I tried two methods. Both 
fail and the versions may be important:

<pre>
Org mode version 9.1.9 (release_9.1.9-65-g5e4542 @ 
/Applications/Emacs.app/Contents/Resources/lisp/org/)
GNU Emacs 26.3 (build 1, x86_64-apple-darwin18.2.0, NS appkit-1671.20 Version 
10.14.3 (Build 18D109)) of 2019-09-02
</pre>

In the first method, I added `:empty-lines` in the org-capture template:

```lisp
(setq org-capture-before-finalize-hook nil)

(setq org-capture-templates
      '(("t" "Todo" entry (file+headline "~/test.org" "heading 1")
         "* TODO %?\n" :empty-lines 1)))
```

The result in screenshots is below: I add a capture item, delete newlines from 
the end of the org-capture buffer, and refile. The heading next to the refiled 
item is gobbled:

[![Empty lines method, step 1 - create an org-capture item][1]][1]

[![Empty lines method, step 2 - delete lines from end of org-capture 
buffer][2]][2]

[![Empty lines method, step 3 - refile][3]][3]

In the second method, I added a function that inserts new lines at the 
beginning and end of the org capture buffer (for exposition, I also add `hello 
world!` to show that these insertions go at the beginning and end of the whole 
org buffer, not the org-capture buffer):

```lisp
(setq org-capture-before-finalize-hook nil)

(defun +org|insert-newlines ()
  (beginning-of-buffer) (insert "\nhello world!\n")
  (end-of-buffer) (insert "\nhello world!\n"))

(add-hook 'org-capture-before-finalize-hook #'+org|insert-newlines)

(setq org-capture-templates
      '(("t" "Todo" entry (file+headline "~/test.org" "heading 1")
         "* TODO %?\n" :empty-lines 1)))
```

The result in screenshots is below: I add a capture item, delete newlines from 
the end of the org-capture buffer, and refile. The heading next to the refiled 
item is gobbled and the lines inserted go at the beginning and end of the 
buffer, instead of the org-capture buffer.

[![Insert lines method, step 1 - add org-capture item][4]][4]

[![Insert lines method, step 2 - delete newlines from end of org-capture 
buffer][5]][5]

[![Insert lines method, step 3 - refile][6]][6]

  [1]: https://i.stack.imgur.com/6cUlo.png
  [2]: https://i.stack.imgur.com/GPwg8.png
  [3]: https://i.stack.imgur.com/tzAJI.png
  [4]: https://i.stack.imgur.com/xcKva.png
  [5]: https://i.stack.imgur.com/g5sIT.png
  [6]: https://i.stack.imgur.com/D1H8l.png
  [7]: https://i.stack.imgur.com/l6a55.png

------------------------------------------------------------------------



Emacs  : GNU Emacs 26.3 (build 1, x86_64-apple-darwin18.2.0, NS appkit-1671.20 
Version 10.14.3 (Build 18D109))
 of 2019-09-02
Package: Org mode version 9.1.9 (release_9.1.9-65-g5e4542 @ 
/Applications/Emacs.app/Contents/Resources/lisp/org/)

current state:
==============
(setq
 org-src-mode-hook '(org-src-babel-configure-edit-buffer
                     org-src-mode-configure-edit-buffer)
 org-after-todo-state-change-hook '(org-clock-out-if-current)
 org-metadown-hook '(org-babel-pop-to-session-maybe)
 org-clock-out-hook '(org-clock-remove-empty-clock-drawer)
 org-mode-hook '(#[0 "\300\301\302\303\304$\207"
                   [add-hook change-major-mode-hook org-show-block-all append
                    local]
                   5]
                 #[0 "\300\301\302\303\304$\207"
                   [add-hook change-major-mode-hook org-babel-show-result-all
                    append local]
                   5]
                 org-babel-result-hide-spec org-babel-hide-all-hashes)
 org-archive-hook '(org-attach-archive-delete-maybe)
 org-confirm-elisp-link-function 'yes-or-no-p
 org-agenda-before-write-hook '(org-agenda-add-entry-text)
 org-metaup-hook '(org-babel-load-in-session-maybe)
 org-bibtex-headline-format-function #[257 "\300
\236A\207" [:title] 3 "\n\n(fn 
ENTRY)"]
 org-babel-pre-tangle-hook '(save-buffer)
 org-tab-first-hook '(org-babel-hide-result-toggle-maybe
                      org-babel-header-arg-expand)
 org-occur-hook '(org-first-headline-recenter)
 org-cycle-hook '(org-cycle-hide-archived-subtrees org-cycle-hide-drawers
                  org-cycle-show-empty-lines
                  org-optimize-window-after-visibility-change)
 org-speed-command-hook '(org-speed-command-activate
                          org-babel-speed-command-activate)
 org-confirm-shell-link-function 'yes-or-no-p
 org-link-parameters '(("id" :follow org-id-open)
                       ("rmail" :follow org-rmail-open :store
                        org-rmail-store-link)
                       ("mhe" :follow org-mhe-open :store org-mhe-store-link)
                       ("irc" :follow org-irc-visit :store org-irc-store-link)
                       ("info" :follow org-info-open :export org-info-export
                        :store org-info-store-link)
                       ("gnus" :follow org-gnus-open :store
                        org-gnus-store-link)
                       ("docview" :follow org-docview-open :export
                        org-docview-export :store org-docview-store-link)
                       ("bibtex" :follow org-bibtex-open :store
                        org-bibtex-store-link)
                       ("bbdb" :follow org-bbdb-open :export org-bbdb-export
                        :complete org-bbdb-complete-link :store
                        org-bbdb-store-link)
                       ("w3m" :store org-w3m-store-link) ("file+sys")
                       ("file+emacs") ("doi" :follow org--open-doi-link)
                       ("elisp" :follow org--open-elisp-link)
                       ("file" :complete org-file-complete-link)
                       ("ftp" :follow
                        (lambda (path) (browse-url (concat "ftp:" path))))
                       ("help" :follow org--open-help-link)
                       ("http" :follow
                        (lambda (path) (browse-url (concat "http:" path))))
                       ("https" :follow
                        (lambda (path) (browse-url (concat "https:" path))))
                       ("mailto" :follow
                        (lambda (path) (browse-url (concat "mailto:"; path))))
                       ("news" :follow
                        (lambda (path) (browse-url (concat "news:"; path))))
                       ("shell" :follow org--open-shell-link))
 )