Changing the lookahead token

[David Kastrup]

Hi,

I have a question regarding the possibility of messing with the compiler
in the area of lexical tie-ins.

Basically, I am working on LilyPond, a language with a rather free-form
syntax.  The problem area I am with is the area of function calls: a
function call token is recognized in the lexer, and the lexer looks at
the required argument list and types and produces appropriate tokens
causing the parser to collect the function arguments.  Now since the
language is basically free-form, quite a few arguments are undelimited
and are recognized as being complete once the lookahead token can no
longer become part of the argument.

Now here is the problem: after the argument list is collected, I might
have a lookahead token already.  The function is called, and only after
the call the _syntactic_ category of the call is determinable, so I need
to inject an appropriate token into the stream.  The question now is at
what point of time it is too late to swap out the lookahead token
because its state is irretrievably part of the parser state?  Can I
still swap it out as long as it has not been shifted and has only caused
reductions?  Or may that be too late?

It would seem to me that the basic definition of LALR(1) which makes the
lookahead token not be part in the reduction rules itself would make it
somewhat plausible that an algorithm working on LALR(1) efficiently
might fully reconsider the lookahead token after every reduction.

But that's just guessing.  Are there any hard or soft criteria about
when it may or may not be allowed to pull the lookahead token out from
under Bison and put something else there?

Thanks

-- 
David Kastrup