bug#33139: 12.1.1; Running syntax-ppss breaks font-latex

[Ikumi Keita]

Dear Frank,

We are very sorry for taking such a long period to respond. We couldn't
reproduce what you described until recently, but at last managed to
develop a prescription to reproduce the symptom reliably, in the course
of chase of another bug.

Now we have just fixed this bug. Please try the latest development
version in the git repository or the ELPA release which is to be appear
soon.

Thank you very much for providing the sample hw.tex file and the
analysis that `syntax-ppss' is involved in the core of this bug. They
were valuable in the course of tackling down this bug.

Best regards,
Ikumi Keita

P.S. For record, I write down the prescription to reproduce this bug
reliably before this bug is closed. It seems that the size of the emacs
frame is an important factor whether and where this bug appears:
1. Save the following init.el under ~/.emacs.d/. (After backing up the
   original init.el, of course)
----------------------------------------------------------------------
(defun call-syntax-ppss ()
  (syntax-ppss))
(add-hook 'pre-command-hook 'call-syntax-ppss)
(load "auctex.el" nil t t)
----------------------------------------------------------------------
2. Start a new emacs session by
emacs -geometry 140x35 &
3. Open the example file hw.tex and type C-v three times.
4. Then the several paragraphs beginning with "Recall that $b$ is
   algebraic ..." get fontified wrongly, where normal texts are given
   math face and math expressions are without particular face.

>>>>> Frank Dai <frankdai320@gmail.com> writes:
> Hello AUCTeX maintainers,
> Currently using smartparens with AUCTex will break font-latex non
> deterministically. Attached is a screenshot of what it looks like when
> broken. After some debugging, I concluded that the issue with
> smartparens was that a 'yntax-ppss call was put into pre-command-hook.
> The issue is reproductible with the following .emacs file (plus the
> necessary AUCTeX importing) as well as an attached .tex:
> (defun call-syntax-ppss ()
>   (syntax-ppss))
> (add-hook 'pre-command-hook 'call-syntax-ppss)

> syntax-ppss should be stateless, so calling it should not affect
> font-latex at all. The relevant changes that syntax-ppss makes is
> modification of syntax-ppss-narrow (through syntax-ppss--data) in
> lines 609-610 (in emacs-26.1) of syntax.el. I once found a case where
> setting syntax-ppss--directly broke fontification, without even
> calling syntax-ppss at all (but as it is nondeterministic, it no
> longer works).

> I have found a workaround, by putting the following line in .emacs
> (add-hook 'LaTeX-mode-hook (lambda () (add-to-list
> 'font-latex-extend-region-functions  (lambda (min max) (point-min)))))
> The idea behind this function was just to always extend the region to
> the beginning of the buffer when fontifying. It seems to work, even
> with syntax-ppss in 'pre-command-hook. From some cursory debugging,
> none of the current functions in 'font-latex-extend-region-functions
> check for presence of an unclosed $ or $$, which make it seem to me
> that merely adding a function to check for an unclosed $ or $$ (or
> modifying an existing one to do so) would fix this issue.