bug#59477: python-tests fail without tree-sitter

[Yuan Fu]

> On Nov 25, 2022, at 8:50 AM, Mattias Engdegård <mattias.engdegard@gmail.com> 
> wrote:
> 
>> But if I run this in a buffer manually I get the ending newline. I’m not 
>> sure what’s the cause of that. Bisecting give 
>> 7c5d4348330b206aff1f8e5bc4fd241d6a6dc0b5, but that commit doesn’t change 
>> anything filling-related. 
> 
> No idea really, but it might have something to do with the fact that the 
> changes move the assignments
> 
>  (setq-local font-lock-defaults
>              `(,python-font-lock-keywords
>                nil nil nil nil
>                (font-lock-syntactic-face-function
>                 . python-font-lock-syntactic-face-function)))
>  (setq-local syntax-propertize-function
>              python-syntax-propertize-function)
> 
> so that they are executed after
> 
>  (when python-indent-guess-indent-offset
>    (python-indent-guess-indent-offset))
> 
> instead of before. `python-indent-guess-indent-offset` has the side-effect of 
> setting syntax properties, in particular for the string terminator 
> (triple-quote in the test).
> 
> This is important, because python-fill-string (called as part of 
> fill-paragraph in the test) assumes this having already been done and if not, 
> str-end-pos isn't computed correctly and things take a turn for the worse 
> after that.
> 
> Stefan probably knows better how this is supposed to work, but presumably 
> python-fill-string should take measures to ensure accurate syntax properties 
> before doing things like
> 
>  (re-search-forward (rx (syntax string-delimiter)) nil t)
> 
> and so on. Sorry about not being of much help here.
> 

Thanks, that’s a very useful information. And I can only blame myself for 
breaking the tests :-)

While still unable to find the culprit. I have the following observations:

1. Setting require-final-newline to t doesn’t work
2. If I change with-temp-buffer to with-current-buffer (get-buffer-create 
"*test*”), the problem disappears, the newlines is not dropped
3. I edebugged fill-paragraph, the newlines in the temp buffer disappears at 
line 865 in fill.el, where the recursive call returns. Before (funcall function 
justify) returns (`function` is fill-paragraph itself), the newline still 
exists, but after we return to the caller at line 865, the newline disappears.

I checked for newline by hitting e and evaluating (char-before (point-max))

Yuan