bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

> Where does one obtain dictionary.json from?

From here, for example: https://www.heytings.org/data/dictionary.json .

> > You'll see that Emacs needs about five seconds (on my laptop) to 
> > display the end of the buffer.
>
> Does that come from the long lines, or solely from the size of the 
> buffer?

You're replying to a week old email.  IIRC that delay comes solely from 
font locking.

> > Now compare that with the feature branch, with which the end of the 
> > buffer is displayed instantaneously.  That five seconds delay is caused 
> > by fontification-functions.
>
> At some point we should accept that visiting a huge file might take some 
> time (5 seconds doesn't sound terrible, depending on the context). 
> Because the alternative is mis-fontifications and broken display.

If you dislike mis-fontification, turn font-lock mode off.

> Here I'm attaching a version of downloadify.js we can use for comparison 
> (please rename the extension from .sj to .js locally; Gmail was not 
> letting it through otherwise). It's not a huge file, just about 88K.

It's a tiny file, not in any way representative of the ones we're dealing 
with.  But amusingly, even with that tiny file, you can see the problem at 
hand.  Do M-: (setq long-line-threshold nil) RET, and open it in a large 
enough window (e.g. 160 characters).  Type M->, and try to move point 
there with C-p or C-n.  You'll see that Emacs is already sluggish.

> I'm also attaching a screenshot of another problem: suddenly the bottom 
> several screens of the buffer are mis-highlighted as if starting inside 
> a string. That very much look like a result of breaking syntax-ppss's 
> visibility of the buffer.
>
> So the buffer scrolls quickly but looks bad.

If you dislike mis-fontification, turn font-lock mode off.  It's as easy 
as that.  Mis-fontification is expected in such cases.  The docstring of 
syntax-wholeline-max also mentions that "misfontification may then occur". 
Why did you not protest at that time?

> Branch feature/long-lines-and-font-locking, revision cd41ce8c6c1079 from 
> July 25. That branch is not there anymore, so let me know if I should 
> re-test this with some later version of your work.

That branch doesn't exist anymore, it has been merged in master.