bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 05.08.2022 17:09, Gregory Heytings wrote:
> > > But if you're so annoyed by mis-fontification, why don't you just 
> > > turn font-lock mode off?
> >
> > "If you're annoyed by Emacs's performance with large files, why don't 
> > you just never open them?"
>
> That's just wrong: in one case we're talking about your personal 
> feelings/preferences, in the other one about Emacs' capabilities for all 
> its users.

Being able to edit a medium-size file with correct syntax highlighting 
without redisplay-related stuttering is also a capability.

> > I like font-lock and the visual cues that come with it. Only 
> > font-locking the first 1 MB of a large file seems like a good 
> > compromise: show correct highlighting where we can with reasonable 
> > performance, and omit it in the rest of the file.
>
> So what you prefer IIUC would be to call fontification-functions with a 
> locked narrowing to 1 MB if point is before that threshold, and to not 
> call fontification-functions at all after that threshold?  That might be 
> another doable approach.

If we have to support huge files with max responsiveness, then that 
would be my preference, yes.

I don't see the point of using a "locked" narrowing for this, though. 
Maybe not even a narrowing at all: just avoid calling 
fontification-functions with START > value_of(large_file_fontification_max).

Or even implement that limitation in font-lock itself (in Lisp). Not 
sure which place is better. Depends on how we want 
'font-lock-fontify-region' to behave: would it still fontify a region 
when asked, or would it abort when BEG is too high. I'd prefer the 
former approach since it gives more power to programmers, if we don't 
find any significant pitfalls with it.

> > > Also, why did you not protest vehemently when Stefan added 
> > > syntax-wholeline-max, which also causes occasional mis-fontification?
> >
> > I have replied to this exact question in an earlier email. We can 
> > continue this line of inquiry in that subthread.
>
> Sorry, I missed that part of your earlier post:
>
> > syntax-wholelines-max indeed can potentially cause problems too, but 
> > in a much narrow range of situations (and only with major modes which 
> > have non-trivial syntax-propertize-function).
>
> You forget to metion that syntax-wholelines-max can in fact make things 
> much worse, see the recipes I sent you a few days ago.  So it doesn't 
> seem like it's the right approach.

These recipes, I guess:

> emacs -Q
> M-: (setq long-line-threshold nil syntax-wholeline-max most-positive-
> fixnum) RET
> C-x C-f dictionary.json RET y ;; takes 160 seconds
> C-e ;; takes 200 seconds
>
> emacs -Q
> M-: (setq long-line-threshold nil) RET
> C-x C-f dictionary.json RET y ;; immediate
> C-e ;; not finished after 1200 seconds (20 minutes), I killed Emacs

I get what you're saying, but the approach does seem "right" to me 
because it works, letting me view and edit dictionary.json with good 
performance and behavior. And because it fits the overall 
font-lock/syntax-propertize/syntax-ppss scheme.

I'm not considering it without the improvements that are enabled with 
non-nil long-line-threshold, though. As you show it above, Emacs is 
simply not usable with those numbers. 200 seconds might as well be 
"never finished". It took like 10-20 seconds on my machine (and with the 
latest patch I sent to this thread), but that's still a lot. Hadn't 
managed to see the second scenario finish either.

Why the default value of syntax-wholeline-max makes things worse in this 
scenario, is a good question, but is it a useful one? Perhaps if you 
could show a similar comparison with a non-nil value of 
long-line-threshold (so that Emacs actually works rather than grind to a 
halt), we could debug it and produce the answer.

Or, if the scenario requires long-line-threshold to be nil, I guess the 
example file needs to be smaller so that the second scenario can finish 
in a bearable amount of time, and Emacs is still usable.

Either way, I believe the change is at the right level of abstraction, 
and if it has bugs, they should be solvable without major redesign.