Re: cc-mode fontification feels random

[Ergus]

On Sat, Jun 12, 2021 at 01:52:12AM +0200, �scar Fuentes wrote:
> Ergus <spacibba@aol.com> writes:
>
> > Going a bit more into this. And reconsidering tree-sitter.
> >
> > As there is already a tree-sitter module package with some interesting
> > functionalities. (I know Eli didn't like some details in it's
> > implementation)
> >
> > But maybe it is a good time to try to disable the cc-mode font-locking
> > (I don't actually know if it is possible to do that), and repeat the
> > scrolling benchmark only with the tree-sitter-mode and
> > tree-sitter-hl-mode enabled?
> >
> > Just to see how it compares and how much of that approach is useful?
>
> More easily, you can use some of the editors that already use
> tree-sitter for fontification of C/C++ and do the PgDn test 

We have all the lisp machine overhead in the middle. So doing this will
be like comparing apples with pears.

> (which looks
> like a rather silly test to me, because who navigates large files by
> holding PgDn and why Emacs should support that terrible use case well?)

The scrolling test is because during scrolling we call re-display,
font-look and some hooks. So it is the easiest way to measure all the
syntax highlighting in action.

> This would provide a valuable comparison point for little effort.

I don't think so. The tree-sitter mode does not require special effort
to install. And comparing emacs vs emacs is more realistic and useful
IMHO (neovim redisplay is ridiculously fast).

But any way just to start: tree-sitter parses all the text in xdisp.c,
(in my machine), in 0.12 seconds from scratch and re-parses it (reusing
the tree) 10 times faster; in 0.008 ~ 0.01 seconds.

If we count that we don't need to re-parse the file (buffer), but only
the modified regions (that is possible to specify with the api). Then
the times are ridiculous small.

In this case the parse is mostly already done, so scrolling won't need
to parse the text to add the highlighting... so maybe we need something
else to measure the impact (maybe something that modifies the text)

BTW: Eli was concerned about the extra copy of the buffer text to send
it to tree-sitter. In this case the time to memcopy an array with all
xdisp text is ~0.00085 seconds.

Any way if we don't want the copy we can use
ts_parser_set_included_ranges to exclude the gap and pass the text
pointer directly without any copy.

> Although I'm more interested on accuracy, but it seems that the thread
> was effectively and hopelessly hijacked :-/
To improve accuracy we need to improve the parsing OR add more work to
cc-mode. So that's why we are looking for alternatives.

There are already some interesting information to see tree sitter in
action:

https://www.youtube.com/watch?v=ZwibVdNtFjs
https://www.youtube.com/watch?v=oSrXK8ovBfQ

Where you can actually see that the accuracy should also improve (and
probably some navigation commands)