Re: "Font-lock is limited to text matching" is a myth

[Eric M. Ludlam]

On Tue, 2009-08-11 at 00:19 +0200, Lennart Borgman wrote:
> On Tue, Aug 11, 2009 at 12:06 AM, Eric M. Ludlam<address@hidden> wrote:
> 
> Hi Eric,
> 
> >  The concept of using the Semantic parser/generator framework for
> > improving font-locking accuracy has come up many times.  No-one to my
> > knowledge has attempted to mix the two.
> 
> 
> Maybe that can easier be done if Semantic parser use
> font-lock/JIT-lock timers and marking to keep track of what need to be
> reparsed? (It is just a wild idea perhaps.)

I'm not certain of how the font/jit lock works.  Semantic works by
tracking edits (after-change-functions) and then on it's own timer, it
coalesces the changes into parsable units.  It then reparses those
units.

Font lock can refontify based on fairly small subsections of a buffer,
such as a single code line, or a comment section.  Semantic's
subsections are the size of functions, variables, and datatypes (ie, the
tags it creates.)

> >  The CONS are that everything in Semantic is set up to parse the entire
> > buffer in one pass, and to parse logical sub-sections only after a full
> > parse has been done.
> 
> 
> So you do a first pass with coarse parsing and then you look in the
> sub-sections for details? Is this strictly necessary? I guess you are
> looking for top level definitions in the first pass?
> 
> Could that pass have its own state and continue upon demand (when an
> item is not recognized) or is such a logic impossible?

It could, but I have not done so.  Tagging information is not generally
needed right away, so just waiting for the user to either ask for it, or
sit idle for a while works pretty well.  The overhead of such an
incremental parser isn't really needed.

> (I guess font-lock/JIT-lock could be improved to help with keeping
> track of what parts of the buffer that have been parsed/maybe
> fontified.)

The needs between the tagging parser and the font-lock parser are
different.  Font lock needs to colorize arbitrary blocks of text, and a
tagging parser needs to parse everything, but only needs the data
periodically.

Converting a tagging parser to a colorizing parser would be challenging
because of these different uses.

> >  I would imagine that the parsing engine in Semantic, if it is deemed
> > critical by the maintainers, will get faster if key components are
> > integrated into the C code.
> 
> Is that part stable?

Yes.  Not much is going on there.

> >  Lastly, as David Engster stated, CEDET has decoration tools that
> > decorate entire tags in some way, such as putting a line on top of
> > functions.  This is a separate decoration activity not related to font
> > lock, and something font lock would not be able to do reliably.
> 
> Why not if it asks the parser?

Font lock runs long before the parser bothers trying to catch up.  Font
lock would needs hooks for after the parser runs.
problems.

While font lock and semantic share a need for a parsing infrastructure,
the where/when of the parsing is quite different.  It is possible to
conceptually mix and match the parsers vs the schedulers.  In practice,
the two tools have their own lengthy histories that will make that
challenging.  Before tackling such a project, it would be wise to take
multi-mode (or similar tool) into account.

Eric