Re: How to add pseudo vector types

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Thu, 15 Jul 2021 14:23:02 -0400
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,
>  emacs-devel@gnu.org
> 
> >> Say you have a buffer with some content and scrolled through it, so 
> >> tree-sitter has parsed the whole buffer. Then some elisp edited some text 
> >> outside the visible portion. Redisplay doesn’t happen, we don’t tell this 
> >> edit to tree-sitter. Then I scroll to the place that has been edited. What 
> >> now?
> > 
> > Now you call tree-sitter passing it the part of the buffer that needs
> > to be parsed (e.g., the chunk that is about to be displayed).  If
> > tree-sitter needs to look back, it will.
> > 
> >> I’ve lost the change information, and tree-sitter’s tree is out-dated.
> > 
> > No information is lost because the updated buffer text is available.
> > 
> >> We can fontify on-demand, but we can’t parse on-demand.
> > 
> > Sorry, I don't believe this is true.  tree-sitter _must_ be able to
> > deal with these situations, because it must be able to deal with
> > incomplete text that cannot be parsed without parse errors.
> > 
> I think my assertion was too strong. By “can’t parse on-demand” I mean we 
> can’t easily pass tree-sitter a random chunk of text and not letting it to 
> parse from BOB. 

You must start from BOB only in languages that require that; not every
language does.

And even with languages that require starting from BOB, you could do
that only once, the first time a buffer needs parsing; thereafter, you
can only pass to tree-sitter the parts that were changed since the
last time.  Emacs records that information for the display engine, see
BEG_UNCHANGED and END_UNCHANGED.  If that is not enough, we could
record more information about changes to buffer text.

The main issue here is to pass the buffer text to tree-sitter lazily,
only when and as much as needed.

> I understand. I want to point out that parsing is separated from 
> fontification, and syntax-pass flushes its cache in before-change-hook. I was 
> hoping to use the parse tree for more than fontification, e.g., motion 
> commands like sexp-forward/backward or structural editing commands like 
> expand-region. Another scenario: some elisp edited some text before the 
> visible portion, the tree is not updated, now I want to select the node at 
> point (like expand-region), I look for the leave node that contains the byte 
> position of point. However, because the tree is out-dated, the byte position 
> of point will not correspond to the node I want.

Each command/feature that needs an updated TS tree will take care of
updating TS with the relevant information.  We should record whatever
we need for that as side effect of primitives that change buffer text
(in insdel.c), and use the recorded info to update TS.  But the actual
passing of text to TS should happen lazily, when we actually need its
re-parsing, not when the changes to buffer text are done.