Re: "Font-lock is limited to text matching" is a myth

[Stefan Monnier]

> I. Asynchronous parsing

BTW, I'm interested in adding core-Emacs support for such parsing, so if
you have any ideas about it, please share them.  The way I see it, there
should be a table-based parsing engine written in C running in
a spearate thread, so the question are "what should the tables look
like?", "what should the output look like?", "what should the engine
look like?", "how should the asynchronous code synchronize with the rest
of Emacs?".  Any help along this effort would be welcome.

> There is a relatively simple alternative that might appease Daniel:
> I could have js2-mode simply not do any highlighting by default,
> except for errors and warnings.  We'd use whatever highlighting is
> provided by espresso-mode, and users would be able to choose between
> espresso-highlighting and js2-mode highlighting.  With the former,
> they'd get "instantaneous" font-locking, albeit not as rich as what
> js2-mode can provide.

Bringing espresso-mode and js2-mode closer together would be good
(e.g. by merging them into a single mode with customization options
allowing to choose between different ways to do highlighting, imenu,
etc...).
Especially on the indentation side since it seems they ahre some of
their history.

> Errors and warnings would still need to be asynchronous (if they're
> enabled).  So, too, would the imenu outline and my in-progress
> buffer-based outline, which is somewhat nicer than the IMenu one.

Please try and make sure that the imenu-style info you provide is
provided in a compatible way (e.g. so that which-func-mode and other
users of imenu info work).  "Make sure" might mean "report any
obstruction along the way", of course.

> Note: diagnostic messages in js2-mode are highlighted using overlays.
> I tried using overlays for all highlighting but it was unacceptably
> slow and had a tendency to crash Emacs.

Overlays should be used parsimoniously (their algorithmic complexity is
poor).  They're basically somewhere between O(N) and O(N^2), whereas
text-properties are O(log N) as god intended.  So for the few expected
errors, overlays are OK, but for font-lock-style highlighting, it'll
kill performance in very short order.

> I took a close look at Eclipse and IntelliJ, and even asked some
> of their users to characterize the highlighting behavior of the IDE.
> Without exception, the IDE users had internalized a ~1000 ms delay
> in highlighting and error reporting as part of their normal workflow,
> and they uniformly described it as "instant{aneous}" until I made
> them time it.

;-)
There used to be a defer-mode for font-lock (I actually hacked one
myself before the official one came out).

> I've been an Emacs user for 20+ years now, and like many I found
> the idea of a parsing delay to be somewhere between "undesirable"
> and "sickening".  But the majority of programmers today have
> apparently learned not to notice delays of ~1sec as long as it
> never interferes with their typing or indentation (see IV below).

Yes, you mostly notice it when you previously used a system without such
a delay.  But I agree that a 1s delay is not problematic (it even
sometimes avoids intermediate states that are undesirable).

> It is based on the indentation in Karl Langstrom's mode, which does a
> better job for JavaScript than any indenter based on cc-engine, but
> that doesn't mean it's a good job.  And it's essentially unconfigurable.

> espresso-mode shares this problem, which means that for this
> important use case it is not an improvement over js2-mode.

In this case, if espresso-mode gets installed, I'll ask you guys to get
together and merge your two indentation codes (it can be as simple as
"pick one", I don't care; I just don't want to have two diverging
copies of the same code base and feature).

> Daniel's objections to js2-mode's non-interaction with font-lock
> apply equally to the non-interaction with cc-engine's indentation
> configuration system.  The indent configuration for JavaScript should
> share as many settings as practical with cc-mode.

I'm not too fond of cc-mode's indentation code and configuration,
actually, so I don't mind if js2-mode doesn't share anything with it
(tho I won't oppose a change in this respect either).

>   3) indentation in "normal" Emacs modes also runs synchronously as
>      the user types.  Waiting 500-800 msec or more for the parse to
>      finish is (I think) not acceptable for indentation.  For small
>      files the parse time is acceptable, but it would not be generally
>      scalable.

Agreed.  Do you happen to know who other IDEs do about it?

> There seems to be a common misconception flitting about to the
> effect that font-lock is perfect and will never need to change.

I can guarantee you that those misconceptions are not shared by the
authors and maintainers of font-lock.

> Va) Inadequate/insufficient style names

[ Putting on my functional programmer hat here. ]
All you're saying here is that your languages have too many concepts.

[ Putting on my Emacs maintainer hat again. ]
highlighting should be about helping the user understand his code:
highlighting every character with a different color is the way to
get there.  You may want to help him find structure (e.g. make
function/method declaration stand out), you may want to help him not get
confused (highlight strings and comments differently), you may want to
attract his attention to weird things (undeclared variables, ...), but
I highly doubt that highlighting function parameters differently from
local variables will help her in any way.

This said, the set of default faces deserves a rethink as well as some
additions, yes.

> languages, not the intersection.  There should, for instance, be a
> font-lock-symbol-face for languages with distinguished symbols such
> as Lisp, Scheme and Ruby.

What for?

> Vf) No font-lock interface for setting exact style runs
[...]
> The problem is that I need a way, in a given font-lock redisplay, to
> say "highlight the region from X to Y with text properties {Z}".

I'm not sure I understand the problem.  What's wrong with
put-text-property?

> When I assert that it's not possible, I understand that it's
> _theoretically_ possible.  Given a JavaScript file with 2500 style
> runs, assuming I had that information available at font-lock time, I
> could return a matcher that contains 2500 regular expressions, each
> one of which is tailored to match one and exactly one region in the
> buffer.

Just place in font-lock-keywords a MATCHER that is a function whose code
walks the list of your "runs" (checking which of your runs are within
(point) and LIMIT) and uses add-text-properties on them; and finally
returns nil.

> In practice, however, I am not aware of a way to do this that is
> either clean or efficient.

That seems about as efficient as it can get within the constraints
of Elisp.  And it looks pretty clean to me.

> Vg) Lack of differentiation between mode- and minor-mode styles
[...]
> As far as I can tell, the officially supported mechanism for
> adding additional font-lock patterns is `font-lock-add-keywords'.
> This either appends or prepends the keywords to the defaults.

Yes, this sucks.  It should be replaced by a more declarative interface.

> But it would be nice to have more direct support for modes like mine.

We're pretty far from it, I think, but the intention is definitely there.


        Stefan