bug#62951: 29.0.90; c-ts-mode: Incorrect fontification due to FOR_EACH_TAIL_SAFE

[Yuan Fu]

> On Apr 22, 2023, at 11:25 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Yuan Fu <casouri@gmail.com>
>> Date: Sat, 22 Apr 2023 17:28:25 -0700
>> Cc: 62951@debbugs.gnu.org
>> 
>>>> I’m aware of this issue, but the truth is there isn’t a good solution to
>>>> it. We need to recognize FOR_EACH_TAIL_SAFE (not hard) and fix arbitrary
>>>> code after it (hard). In this case it’s a if statement, with macro calls
>>>> and AND operation in it’s condition, it’s already three things we need
>>>> to recognize and somehow handle. It can also be a for loop, a switch
>>>> case, a function call, a while loop. If we want to fix FOR_EACH_TAIL we
>>>> would need to handle every possible thing, at that point we might as
>>>> well have wrote a parser :-)
>>> 
>>> Sorry, I don't understand the difficulties.  The body of FOR_EACH_TAIL
>>> and a few similar macros we use could be on of the following:
>>> 
>>> . a single simple statement
>>> . an 'if' clause
>>> . a 'while' loop
>>> . a 'do' loop
>>> . a 'for' loop
>>> . a brace-delimited block (this one already works, AFAICS, so we
>>>   perhaps need not anything about it)
>>> 
>>> (In practice, only the first 2 and the last one are used, AFAICS.)
>>> 
>>> Doesn't tree-sitter tell us enough to figure out which of the above is
>>> in the body?  If so, why would we need to write a full parser?
>> 
>> Ok, the full parser part is a bit exaggeration :-) But my point is it’s a 
>> lot of dirty code for a subpar result. Take
>> 
>> if (NILP (XCDR (tail)) && STRINGP (XCAR (tail)))
>> 
>> for example, it’s parsed as a function definition and all the tokens in the 
>> condition are parsed as weird things like macro declarator, error, function 
>> declarator, type, etc. And if the condition changes to something else, say 
>> it has another layer of function call, it’ll parse differently.
> 
> But the top-level construct is 'if', no?  Isn't that enough?
> 
> Also, can we detect the FOR_EACH_TAIL etc. macros themselves, and then
> treat their body specially?
> 
>>> Please understand: good support for editing Emacs C sources is from my
>>> POV imperative for c-ts-mode to gain traction.  One of my gripes about
>>> CC Mode was insufficient support for our macro system and for various
>>> GCC attributes; that improved recently to some extend, but not enough,
>>> and at a price of introducing ugly lists of strings that cause trouble
>>> when used in file-local variables.  We must do better in c-ts-mode!
>>> 
>>> So please make an effort of providing reasonable built-in solutions
>>> for these idiosyncrasies of the Emacs C sources, conditioned on the
>>> new variable c-ts-mode-emacs-sources-support, at least for those of
>>> them that are used widely enough.  It is very important.
>> 
>> What do you think of extending the parser to support these macros instead? 
>> (So we fork tree-sitter-c.)
> 
> This goes against the purpose of using tree-sitter and its grammars.
> I don't think we should maintain and develop language grammars,
> especially since the production of the C sources from them needs
> non-trivial system resources and additional tools.
> 
> Maybe coming up with a way of extending the C grammar in some more
> general way, and then submitting the changes to the tree-sitter-c
> developers for inclusion would be OK.
> 
> But I very much hope that we could support these macros at a lower
> cost, by some tailored Lisp.  Please give it a try.  Even if the
> result works only for the cases we actually use in our sources, it
> might be "good enough" for us.

Fair enough, I’ll try :-)

Yuan