Re: treesitter local parser: huge slowdown and memory usage in a long file

[Yuan Fu]

> On May 12, 2024, at 4:44 PM, Dmitry Gutov <dmitry@gutov.dev> wrote:
> 
> On 09/05/2024 03:16, Yuan Fu wrote:
> 
>>> Is it possible that there would occur multiple changes and reparses between 
>>> some firings of the above hooks? For example, some new feature might go 
>>> over the buffer's text with an automated multi-step transformation, calling 
>>> the parser (but not syntax-ppss) on each step.
>>> In such a scenario it seems treesit--pre-redisplay might miss intermediate 
>>> range updates. Would that be okay?
>> I think you’re right. The chance of it actually go wrong will be slim, but 
>> anything that’s possible to go wrong will eventually go wrong.
> 
> Thanks for confirming the concern.
> 
>> The remaining question is how. I’m thinking of keeping a history of updated 
>> ranges, each marked with the parser timestamp. The parser timestamp is 
>> already there, it’s incremented every time the parser reparses. And 
>> treesit-parser-changed-ranges will return the timestamp along with the 
>> updated ranges. Then in the next iteration, the consumer can pass the last 
>> timestamp to treesit-parser-changed-ranges, which tells it to return all the 
>> changed ranges since that timestamp.
>> The only problem is to decide how long a history of updated ranges do we 
>> keep for each parser. The 100% correct approach is to maintain a separate 
>> history for each consumer, and never throw away old ranges until the 
>> consumer consumes them. But then you risk wasting memory if some consumer 
>> never consumes the ranges. To handle that we can add a hard limit. But then 
>> this hard limit might be too low for some edge case… We can make this hard 
>> limit configurable, and if we ever encountered a case where this hard limit 
>> is not enough and there’s no way around it (unlikely), we can instruct users 
>> or lisp program to increase it.
> 
> That could work. Although it's hard for me to imagine how far back the 
> history would have to be stored, and would that have any practical 
> consequences for Emacs's memory use. Maybe not.
> 
> The approach I was thinking of is in different direction: we take a step 
> back, remove (or stop using at least) the new function, and go back to the 
> idea of subscribing to parsers' after-change notifiers. The improvement in 
> commit f62c1b4cd00 seems to stem from relying foremost on changes ranges in 
> the primary parser. Okay - we re-add the listener for the primary parser only.
> 
> This listener would be specific for a particular consumer. In our case, we'd 
> have a listener which would populate - and then update - the variable used by 
> treesit--pre-redisplay. That variable would store the "up to date" list of 
> updated ranges. The listener, on every call, would "merge" its current value 
> one with the new list of ranges (*). treesit--pre-redisplay would use the 
> data in that data structure instead of calling treesit-parser-changed-ranges, 
> and set the value to nil to "reset" it for the next update.
> 
> (*) So real "merging" would only need to be performed when listener fires 2+ 
> times between the two adjacent treesit--pre-redisplay calls. Otherwise the 
> current value is nil, so the the new list is simply assigned to the variable. 
> Anyway, the merging logic seems to be the trickiest part in this scheme 
> (managing and interpreting offsets), but it should be very similar in both 
> approaches.

I agree. The usefulness of treesit-parser-changed-ranges aren’t really 
justified at this point (well, except that it makes the caller’s code much 
easier to follow). Let me implement what you described and let’s see how it 
goes. I think we don’t even need to merge the ranges (which will be prone to 
bugs if I were to write it ;-), we can just push the new ranges to a list and 
later process them one by one.

Yuan