Re: [Grammatica-users] Tokenizer problem

[Per Cederberg]

Just adding on to the previous answer:

The problem is that the Tokenizer always returns the longest
matching token. So for the input "GO" the COMMAND token will
always be returned, as the LETTER token is only one character
long.

There are at least two more alternative solutions to this:

2) Create a new token for words:

   COMMAND = "GO"
   WORD = <<[A-Z]+>>

   This will always return the WORD token except for the input
   "GO" (so "GO GO" will still cause parse errors). Please note
   that the ordering of the tokens is important in that case.

3) Modify the INPUT production to handle "GO" tokens:

   INPUT = COMMAND SPACE DETAILS+ ;

   DETAILS = LETTER
           | "GO" ;

Hope this helps! (And thanks to Anant for helping out answering 
questions on this list!)

/Per

On fri, 2005-07-01 at 10:48 +0200, HECKHAUSEN Ralf wrote:
> I have a problem with the parser, or better the tokenizer. Below is a
> simplified example.
> 
> %header%
> 
> GRAMMARTYPE = "LL"
> 
> %tokens%
> LETTER = <<[A-Z]>>
> COMMAND = "GO"
> SPACE = " "
> 
> %productions%
> 
> INPUT = COMMAND SPACE LETTER+;
> 
> 
> The input "GO SOUTH" is parsed correctly, but with "GO SOGO" I get a
> parse error, because "GO" is not recognized as two letters anymore. It
> seems that whenever letters are defined as tokens, they may not occur
> in other context. But I am sure there is a way around this problem,
> can someone help me?
>  
> Cheers,
> Ralf