Re: Last steps for pretesting (font-lock-extend-region-function)

[Alan Mackenzie]

Good morning, Stefan!

On Thu, 20 Apr 2006, Stefan Monnier wrote:

>>> IIRC part (i) needs more than an after-change-function: it also needs
>>> a before-change-function.

>> That would depend on the major mode, I think.  AWK mode needs a
>> before-change function.  The CC Mode replacement for (setq
>> font-lock-lines-before 2) will not need a b-c function.

>Simulating font-lock-lines-before only needs an `extend-region' hook in
>font-lock-default-fontify-region, a subject about which we already agree.

We do.  :-)

>> - or else it will require the major mode to maintain the f-l-multiline
>> property permanently.

>The font-lock-multiline property is removed by font-lock-unfontify-region
>and it is expected that it gets re-added (if needed) every time you fontify
>a piece of text.  It's never seemed to be a big deal.
>No before-change-function in sight.

That's why I think it won't work in AWK mode.  For Font locking triggered
by a change, you need the details of the _change_, not merely how the
buffer looks after it.

>>> And (again IIRC) it's the part that can already be solved by placing
>>> (from one of the font-lock-keywords rules) a font-lock-multiline
>>> property.

>> I'm not sure.  Can a font-lock-keyword rule get access to after-change's
>> OLD-LEN in an reasonable fashion?

>What for?  The font-lock-multiline it needs to add is for use by the next
>font-lock round.  It says: "next time you fontify this area, make sure you
>refontify this piece atomically".

The font locking is triggered by the Change.  Therefore it needs details
of that change  to determine what region of text needs refontifying.  It
is up to the major mode maintainer, whether she needs those details or
not.  AWK Mode, for example, does need them.

>> If a change consists of yanking 20k of text into a buffer, the Major Mode
>> must not splat font-lock-multiline onto 20k + 20 bytes, since this would
>> defeat the purpose of JIT lock (fontifying a bit at a time).

>The font-lock-multiline property is added not to the whole fontified text,
>but just to a particular region of text that matched a pattern (a
>"keyword").  I think you're absolutely right: I have to document it because
>you seem to completely misunderstand it.

:-)  Thanks!

>> Could you please document the font-lock-multiline mechanism
>> immediately, including details on how best to access the
>> after-change-functions's OLD-LEN, and details of how and where to hook
>> in the "(put-text-property a b 'font-lock-multiline t)" call.  Either
>> that, or tell me where to find an example of this text property being
>> used, preferably one which uses OLD-LEN.

>AFAIK the OLD-LEN is never needed.

It is most definitely needed.  I don't think you have understood what I
have been trying to say.  :-(

>See font-lock.el for an example of code that sets font-lock-multiline
>property: if a regexp matches several lines, it puts the
>font-lock-multiline property on the wholse matched text (this is only
>done if the font-lock-multiline variable is set).

Perhaps I have truly misunderstood you.  I thought you were telling me
that some code contained within the major mode would do the
(put-text-property ... 'font-lock-multiline ...).  Are you really saying
that the major mode merely has to set up its font-lock-keywords so that
font-lock.el finds the right places to apply the text property?

>> Or, perhaps you could recode the following AWK Mode fragment into an
>> equivalent which uses f-l-multiline instead of f-l-expand-region
>> function.  PLEASE!

>I'll try to, although I'd need to see the font-lock-keywords part (where
>I'll have to add the code).

It's right at the end of cc-fonts.el.

>> #########################################################################
>> (defvar c-awk-old-EOLL 0)
>> (make-variable-buffer-local 'c-awk-old-EOLL)
>> ;; End of logical line following the region which is about to be changed.  
>> Set
>> ;; in c-awk-before-change and used in c-awk-font-lock-extend-region.

>> (defun c-awk-before-change (beg end)
>> ;; This function is called exclusively from the before-change-functions hook.
>> ;; It does two things: Finds the end of the (logical) line on which END lies,
>> ;; and clears c-awk-NL-prop text properties from this point onwards.
>>   (save-restriction
>>     (save-excursion
>>       (setq c-awk-old-EOLL (c-awk-end-of-logical-line end))
>>       (c-save-buffer-state nil
>>        (c-awk-clear-NL-props end (point-max))))))
>> (add-hook 'before-change-functions c-awk-before-change nil t)

>> (defun c-awk-end-of-change-region (beg end old-len)
>>   ;; Find the end of the region which needs to be font-locked after a change.
>>   ;; This is the end of the logical line on which the change happened, either
>>   ;; as it was before the change, or as it is now, whichever is later.
>>   ;; N.B. point is left undefined.
>>   (max (+ (- c-awk-old-EOLL old-len) (- end beg))
>>        (c-awk-end-of-logical-line end)))

>> (defun c-awk-font-lock-extend-region (beg end old-len)
>>   (cons (c-awk-beginning-of-logical-line beg)
>>         (if old-len
>>             ;; From an after-change-function
>>             (c-awk-end-of-change-region beg end old-len)
>>           ;; From a fontifying routine
>>           (c-awk-end-of-logical-line end))))
>> (setq font-lock-extend-region-function 'c-awk-font-lock-extend-region)
>> #########################################################################

>IIUC, something like the following should do:

>(defun c-awk-font-lock-extend-region (beg end)
>  (cons (c-awk-beginning-of-logical-line beg)
>        (c-awk-end-of-logical-line end))) <==================================
>(setq font-lock-extend-region-function 'c-awk-font-lock-extend-region)

That, quite demonstrably, WON'T do, since that will fail to extend the
region to what was the end of the logical line before the change.

>(defconst c-awk-font-lock-keywords
>   '(...
>     (".*\\\\\n.*"
>      (0 (progn (put-text-property (match-beginning 0) (match-end 0)
>                                   'font-lock-multiline t)
>                nil)))
>     ...))

OK, I think I'm beginning to get the idea.



>>> Here is the problems I see with your proposal:
>>> - an `extend-region' hook in font-lock-fontify-region is needed

>> It is indeed.  It is a simple patch, and I am ready and willing to write
>> it, both for font-core.el and modes.texi.  I do first ask that it's
>> accepted in principle, though.

>It's completely accepted.  I just hope we can call it
>"font-lock-extend-region-function".

It has been accepted (by Richard, no less), as part of the
font-lock-after-change-function/jit-lock-after-change.  I'm proposing to
enhance it to do the necessary region extension from within
f-l-default-fontify-region/j-l-fontify-now too.

>>> - the font-lock-multiline property should be enough in all cases to make
>>> it unnecessary to use an after-change-function hook.

>> f-l-extend-region-function also makes a (separate) after-change-f hook
>> unnecessary.

>Which f-l-extend-region-function?  The one called from after-change-f?

Yes.

>Then it's what I meant by "an after-change-function hook".  The fact that
>it's not called directly from a-c-f but via f-t-a-c-f is not very
>important for this discussion.

I think it's important.  It saves the major mode maintainer from having
to install his code as a separate a-c-function, and juggling them around
to make sure they get called in the correct order.

>>> - an after-change-function is expensive, and even more when it comes
>>> bundled with a before-change-function, and even more so when compared
>>> to code that's only run during font-locking.
>> Huh?  Font Locking _is_ expensive.

>Yup, but (thanks to jit-lock) it's not executed after each and every
>buffer modification.

OK.  I think your point is that the f-l-extend-region-function called
from f-l-a-f-f/j-l-a-c is expensive because it is called after _every_
change.  That is potentially true, and it is up to the major mode writer
to ensure that this isn't a problem.  In C Mode, for example,
c-extend-region-function (as it will be in CC Mode 5.32/Emacs 23.1) will
be an extensive thing, involving sophisticated caching, sneaky
heuristics, and so on, so as it will run fast enough.  In AWK Mode, the
function is trivial enough that it's already not a problem.

>> A before-change function need not be expensive (e.g.  c-awk-before-change,
>> see above), and calling it is not expensive.

>The code will have probably the same cost whether it's called from a-c-f or
>from font-lock, but in one case it'll be called (much) more often.

Yes.  But that calling from a-c-f is essential to correct font locking.

>> I suspect that using the f-l-multiline property will absolutely
>> mandate an extra after-change function - how else can it access
>> OLD-LEN?

>Why would it need OLD-LEN?

Stefan, I've tried to answer this question several times already, but
I've clearly not made a good job of it.  I'm getting very frustrated with
not being able to get my point across to you.

PLEASE go through this canonical example in all its gory detail and tell
me you've understood it.  Please tell me that you understand what I'm
doing in the c-awk-before-change, and what I'm doing with OLD-LEN.
Please tell me you see why the way you suggested (89 lines higher up)
wouldn't work.

If you can then tell me a way of achieving the same effect WITHOUT the
use of before-change and/or OLD-LEN, I'd be happy to hear about it.

OK, here goes!

#########################################################################
In AWK Mode:

Point is at EOL 3, and we delete the backslash there.

1. "string \
2. over \
3. several \       <========= point is at EOL 3, about to delete the \
4. #lines."

1. In c-awk-before-change (see above for the code), we calculate
c-awk-old-EOLL ("end of old logical line").  This give 36, (EOL 4).

2. The \ gets deleted, and the text looks like this:

1. "string \
2. over \
3. several 
4. #lines."

3. jit-lock-after-change gets called as (jit-lock-after-change 26 26 1),
which in its turn calls (c-awk-font-lock-extend-region 26 26 1).  This
"1" is the OLD-LEN, of course.

4. c-awk-f-l-e-r calls (c-awk-end-of-change-region 26 26 1).  This does
the following:
(i) It determines the current position of the PREVIOUS end of logical
line: (+ (- c-awk-old-EOLL old-len) (- end beg)),
      => (+ (- 36 1) (- 26 26))
      => 35
NOTE THE USE OF old-len.

(ii) It determines the current end of logical line:
    (c-awk-end-of-logical 26)
     => 26.  This is EOL 3

(iii) It selects the greater of these two positions:
    (max (+ (- c-awk-old-EOLL old-len) (- end beg))
         (c-awk-end-of-logical-line end))
    => (max 35 26)
    => 35.
This 35 is the current EOL 4.

(iv) 35 is returned to jit-lock-after-change in the cons (1 . 35), the
extended fontification region.
#########################################################################

>        Stefan

-- 
Alan.