bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 05.08.2022 10:43, Eli Zaretskii wrote:
> > Date: Fri, 5 Aug 2022 05:03:39 +0300
> > Cc: 56682@debbugs.gnu.org, Eli Zaretskii <eliz@gnu.org>,
> >   monnier@iro.umontreal.ca
> > From: Dmitry Gutov <dgutov@yandex.ru>
> >
> > That was about a 1 GB buffer, right?
> >
> > Let's take care of buffers with more reasonable sizes first, and then we
> > can consider extremes.
>
> We want to solve all of them.

I didn't say we don't. But different issues call for different solutions.

> > A separate threshold for syntax-ppss to avoid parsing the whole
> > buffer might fit the bill.
>
> Don't we already have such a threshold?

Not exactly: the buffer is still fully parsed by parse-partial-sexp 
(once). AFAICT, the variable makes the application of 
syntax-propertize-rules more lax, but at least it keeps counting the 
simple parens/quotes from the beginning of the buffer. That's why the 
fontification remained correct in both examples that I posted.

The time it takes to (parse-partial-sexp 1 (point-max)) accounts for the 
whole fontification delay at the end of dictionary.json.

Now, if nobody manages to speed up parse-partial-sexp itself further, we 
can add an additional tweak/size threshold, after which syntax-ppss 
won't parse the whole buffer anymore. But if we do that in Lisp, we can 
later improve that bit of logic so that the result is not entirely 
arbitrary, like it is now on master with dictionary.json.

> But again, if you are just saying that the current balance between
> response time and font-lock correctness is sub-optimal, and should be
> made better by changing the values of the thresholds, that's a
> different discussion.  For that discussion, we'd need a representative
> enough sample of real-life files with long lines and in as many major
> modes as possible, to make our balance really close to optimal.  I,
> for one, will welcome more examples of such files, especially if they
> use major modes we didn't consider until now.

We really have different problems and thus need different solutions for 
them. Not just one blunt instrument.