bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Fri, 5 Aug 2022 15:08:22 +0300
> Cc: gregory@heytings.org, 56682@debbugs.gnu.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > The current opinion of both the head maintainers and of Gregory is
> > that these are all parts of the same problem, and a single class of
> > solutions can solve most of them.
> 
> This kind of approach fails to optimize for the behavior in medium-sized 
> files, like the downloadify.js I showed previously.
> 
> Simply customizing long-line-threshold to a much higher value will bring 
> back redisplay stutters on C-n/C-p/etc, which *were* a real problem that 
> some of Gregory's changes solved.

You want a separate threshold for calling font-lock from redisplay?
We could discuss that.

> > The problem being that many
> > portions of Emacs code involved in navigation and redisplay don't
> > expect lines to be too long, and therefore employ algorithms that
> > don't scale well with line length.
> 
> As I demonstrated, font-lock itself doesn't have that issue.

What do you mean by "font-lock itself"?

> Furthermore, the performance problem with syntax-ppss which we are 
> talking about now doesn't have anything to do long lines.

The facts are that it has problems in large files, but it also has
problems in not-so-large files with long lines, due to whole-line
approach.

> Go ahead and pretty-print dictionary.json (you can use 'M-x 
> json-pretty-print', write the buffer to a new file, then re-visit it). 
> There won't be any long lines in the resulting file, but 'M->' will 
> still make you wait a few seconds the first time.

No one said that the changes being discussed solve all the problems in
Emacs.

> > I get it that you disagree, but I haven't seen any real data behind
> > your dissenting opinions, and thus I don't yet see any reason to
> > reconsider changing the direction of development in this regard.
> 
> I don't understand why you dismiss the more subtle approach which still 
> seems to reach the stated goals.
> 
> Gregory's changes, along with my suggested tweak, indeed bring work 
> "surprisingly well" already. All without breaking font-lock in the 
> common case.

I don't think I understand what "tweak" you are suggesting.  can you
show a patch relative to the current master?

> Like, we're going from a 255 (?) second delay to 2 second delay already 
> without breaking fontification. And yet you're eager to go from 2 
> seconds down to ~0 and sacrifice highlighting correctness?

Yes, because 2 sec (in an optimized build) is a very long time.