Re: Changing the lookahead token

[David Kastrup]

Hans Aberg <address@hidden> writes:

> On 17 Sep 2012, at 11:15, David Kastrup wrote:
>
>> Hans Aberg <address@hidden> writes:
>> 
>>> Hi David!
>>> 
>>> On 16 Sep 2012, at 10:51, David Kastrup wrote:
>>> 
>>>> But that's just guessing.  Are there any hard or soft criteria about
>>>> when it may or may not be allowed to pull the lookahead token out from
>>>> under Bison and put something else there?
>>> 
>>> You might look at the push parser, which allows one to call it
>>> whenever a token is available. Then LALR(1) compacts the states so
>>> that additional reductions may be called before an error is issued;
>>> possibly something similar may happen in your case.
>> 
>> I am not saying that anything is happening in my case: I am _asking_
>> what will be happening.  I am not keen on investing weeks of work
>> reworking the grammar and then figuring out that my approach was doomed
>> from the start.  It is clear that the lookahead token does not creep
>> into the stack.  It is also clear that it is responsible for causing
>> reductions, and the reductions have actions which I can use to swap the
>> lookahead token.  
>
> Looking back at your OP, the LALR(1) collects total information about
> the grammar and computes, via compaction, a series of states. From the
> user perspective, it is hard to know when the lookahead is used and
> not. The simplest way seems to be to do some experimentation, which I
> think is what the documentation possibly refers to, but that might be
> too volatile for your application.
>
>>> One way around it is to try to rewrite the grammar, so that the tokens
>>> are not needed.
>> 
>> Not possible since the respective arguments are not delimited and may
>> continue indefinitely.
>> 
>> c   c-.   c-.-^
>> 
>> are all valid music arguments.  So I need the lookahead token to decide
>> where the argument may end, and the lexical class of the result is only
>> decided upon executing user-definable code.  So I basically need to push
>> information before the lookahead token.  I have a stack for that in the
>> lexer and already use it in cases where I don't have a lookahead token.
>> The question is whether I can push the lookahead token there when it has
>> already triggered reductions.
>
> As for this last question, I think it will screw up the parsing in
> rather unexpected ways.
>
> Have you considered the GLR parser? If the correct parse depends on a
> future lookahead, it will split and join when the correct parse is
> used.

But the correct parse does not depend on a future lookahead, but on a
future _action_, namely actively calling the function and looking at the
(dynamic) type of its result.  I can't combine this kind of lexical
tie-in with GLR parsing since GLR parsing will not even _execute_ the
action _until_ the ambiguity has been resolved.

>>> Another might be to use LR(1) which makes sure actions are applied as
>>> dictated by the tokens.
>> 
>> I want to avoid going there.  
>
> One just sets a %define - see the manual.

Correct me if I am wrong, but LR(1) will do exactly the _opposite_ of
what I want here by keeping states _separate_ depending on the lookahead
token.  As far as I can tell it is _exactly_ the state-merging nature of
LALR(1) that gives me at least a chance to exchange the lookahead token
without punishment in the first place.

Well, short of any more definite information, I don't see that I have a
chance to do better than try and debug.  And/or analyze the workings of
the skeletons.

-- 
David Kastrup