bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Yuan Fu]

> On Mar 22, 2023, at 6:03 PM, Dmitry Gutov <dgutov@yandex.ru> wrote:
> 
> On 21/03/2023 16:00, Wilhelm Kirschbaum wrote:
>> With the following text in a buffer with elixir-ts-mode:
>> def foo() do
>> end
>> When I delete the `)` and re-insert it, the tree-sitter syntax tree does
>> not fully recover and the `do end` part is absent when observing using
>> treesit-explore-mode.
>> Before removing the `)`:
>> (source
>> (call target: (identifier)
>>  (arguments
>>   (call target: (identifier)
>>    (arguments ( ))))
>>  (do_block do end)))
>> after re-inserting `)`:
>> (source
>> (call target: (identifier)
>>  (arguments
>>   (call target: (identifier)
>>    (arguments ( ))))))
>> I can see the same behaviour using ruby-ts-mode when I remove the
>> following expression:
>> (setq-local syntax-propertize-function #'ruby-ts--syntax-propertize)
>> The ruby syntax I tested with:
>> def foo()
>> end
>> When I add a local `syntax-propertize-function' function to elixir-ts-mode
>> it also solves the problem of not fully recovering the tree-sitter
>> syntax tree.  It does not seem to matter what the function achieves
>> thought, so am a bit confused.
> 
> I can repro both scenarios. Except with ruby-ts-mode, the difference in 
> printed AST is the added wrapping in (program ...). But the visible result is 
> similar: "end" not being highlighted.

The problem seems to be caused by narrowing during fontification: when parsing, 
tree-sitter sees that BUF_ZV_BYTE is smaller than it’s visible_end, and removed 
the part after the parenthesis from the parse tree. I suspect this has to do 
with the narrowed fontification we added for dealing with long lines. 

CC’ed Eli because I don’t really know anything about the locked narrow thing 
besides that it exists. In particular, how does it determine when and where to 
narrow?


To see it in action, apply this patch, and repeat Wilhelm’s experiment: remove 
the “)” and reinsert it. You should see

start: 9, oend: 10, nend: 9
start: 9, oend: 10, nend: 9
start: 9, oend: 9, nend: 10
start: 9, oend: 9, nend: 10
2.5 start: 10, oend: 18, nend: 10 —> culprit

(Make sure to turn off electric-pair-mode to reduce noise.)

Yuan

print.diff
Description: Binary data