Re: cc-mode fontification feels random

[Eli Zaretskii]

> From: Daniel Colascione <dancol@dancol.org>
> CC: <monnier@iro.umontreal.ca>, <joaotavora@gmail.com>, <emacs-devel@gnu.org>
> Date: Fri, 04 Jun 2021 12:33:25 -0700
> 
> > What do you think tree-sitter does with the fast copy you hand to it?
> > doesn't it walk it one character at a time?
> >
> > And if you studied the tree-sitter's internals, and it uses
> > get_buffer_char as a means of copying text into its own buffer, then
> > perhaps we could ask tree-sitter developers to avoid the copy and use
> > the text directly.
> 
> Teaching TS to use a generic cursor interface would be great.

I don't remember if I looked at how it does it now, but are you sure
it doesn't already know how to do that?  Sounds like a natural thing
to me, but maybe I'm missing something.

> > buffer-substring is not just a copy of a chunk of text, it's much
> > more.
> 
> The variant without text properties doesn't do much.

It allocates memory!  For a large buffer (think xdisp.c) that is best
avoided.  I hope if we need to memcpy, we could at least use a pointer
to a buffer allocated by the parser library, so we won't need to.

> > Even if eventually we need to use a memory copy, that'll run
> > circles around buffer-substring, and will avoid triggering GC.
> 
> Sure. I'm not opposed to adding an API that's basically a more efficient 
> buffer substring for C callers. I'm just pointing out that the idea of 
> giving TS "direct access" to a buffer without any copy at all doesn't make 
> a lot of sense.

If it can use that wisely, I don't see why it wouldn't make sense.  If
it cannot, then I agree.  But still, I'd rather not give up from the
get-go and use buffer-substring just because it's there, I'd try
looking for something more scalable and less Lisp-consing.

Also, I hope we could arrange the copying to be driven by the display
engine through the JIT font-lock machinery, rather than sending the
entire buffer or its large parts.

> >> Because any kind of "access" to the buffer that doesn't expose the gap is
> >> going to be a copy anyway.
> >
> > The regexp routines aren't.
> 
> The regexp routines have Emacs specific knowledge.

I mean the way regexp routines use the buffer text as a C string (as 2
C strings, actually).  That doesn't use any Emacs specific knowledge
except the gap, and even the latter is largely solved by the caller.