bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 15.08.2022 19:05, Eli Zaretskii wrote:
> > Date: Mon, 15 Aug 2022 18:52:41 +0300
> > Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
> > From: Dmitry Gutov <dgutov@yandex.ru>
> >
> > On 15.08.2022 17:03, Eli Zaretskii wrote:
> >
> > > I tested this branch with an unoptimized build and a 416MB file made
> > > of 23 copies of dictionary.json, after disabling show-paren-mode.
> > >
> > > Observations:
> > >
> > >     . With the default value of font-lock-large-files, which allows
> > >       font-lock to widen as it pleases, the initial M-> in takes 5 min
> > >       here.  That's way too long.  On master, this is instantaneous.
> > >     . Thereafter jumping into various random places inside the file is
> > >       much faster, but still takes between 1 min and 2.25 min -- again
> > >       too long, even if I divide by 10 to get an approximation to your
> > >       faster machine.  On master this takes maybe 1 sec, maybe less.
> > >     . Horizontal and vertical motion commands are as fast (or as slow)
> > >       as on master, which is not surprising.
> > >
> > > Bottom line: with the default value of font-lock-large-files, this
> > > scales much worse than the master branch -- which again is no
> > > surprise.
> >
> > So we change the default, then?
> >
> >     (setq font-lock-large-files '(narrow . 5000))
>
> No, I want a simple variable that just gives the size of the narrowed
> region, with nil meaning don't narrow at all.

OK.

> > > My conclusion is that we do need to prevent uncontrolled
> > > widening in fontification-functions, if we want a scalable solution.
> >
> > Do you mean the widening inside font-lock-fontify-region that's
> > controlled by font-lock-dont-widen?
>
> No, I mean _any_ widening in _any_ function placed on the
> fontification-functions hook.
>
> > > I honestly don't see any reason to explore more sophisticated
> > > alternatives for where and how to narrow, until and unless major modes
> > > will on their side develop capabilities which will make such
> > > complications (both in code, but mainly on the user part) justified.
> >
> > I'm pretty sure that doing all that in Lisp is a good thing, and would
> > be conducive to whatever improvements we might want to add later...
>
> No, because quite a few packages use jit-lock-register to run stuff
> that is not font-lock at all, and I don't want that to slow down
> redisplay when long lines are involved.  And trying various values of
> a variable exposed to Lisp is easy enough.

It should also be possible to move the whole invocation of the 
fontification-functions up to Lisp, and apply the narrowing in the new 
function that does that.

Another reason I chose font-lock-fontify-region, though, is that it's 
called from different places. It's the common entry point for font-lock.

Like, from 'font-lock-ensure', for example (font-lock-turn-on-thing-lock 
sets font-lock-ensure-function to #'jit-lock-fontify-now).

It's called from Org (see all references to org-font-lock-ensure), for 
instance. mmm-mode calls font-lock-fontify-region directly (sometime 
we'll change it to use font-lock-ensure instead). Also see 
htmlfontify-buffer, occur-engine-line, describe-variable, 
diff-syntax-fontify-props.

It's also used in after-change-functions when font-lock-support-mode is 
not `jit-lock', but that's probably not important.

Some of these definitely can be dealing with long lines and/or large 
files. If font-lock-fontify-region decides the narrowing bounds, it can 
apply them correctly it each case.

I get what you're saying about alternative fontification-functions, 
though: we have multiple similar hooks: it, then jit-lock-functions, and 
font-lock-keywords are more or less used for the same purpose.

I've never seen fontification-functions contain anything but 
jit-lock-function, but jit-lock-functions is used by nlinum, for 
instance. And bug-reference-mode, and goto-address-mode. Perhaps we 
should ask Stefan's opinion on this.

As a practical matter, though, neither goto-addr, nor bug-reference 
should call syntax-ppss or do anything similarly costly in on long 
lines/in large files.

And nlinum might not appreciate being narrowed. It seems to be working 
okay without that in my 20 MB XML file. And in 200 MB one too.