Re: Which lexer do people use?

[Akim Demaille]

> Le 6 juil. 2020 à 14:00, Maury Markowitz <maury.markowitz@gmail.com> a écrit :
> 
>> On Jul 4, 2020, at 6:46 AM, Christian Schoenebeck 
>> <schoenebeck@crudebyte.com> wrote:
>> The only thing that people are missing once in a while on scanner side is 
>> unicode support
> 
> I am missing the capability to support grammars with optional spaces between 
> elements. In BASIC:
> 
>   FORI=1TO5
> 
> is equivalent to:
> 
>   FOR I=1 TO 5
> 
> I have yet to see an elegant way to model this in Flex. This topic comes up 
> time to time here, so I don't think Unicode is the "only" thing.

I believe this is a perfect place to use Flex's / operator I mentioned
(similar to what other engines call "lookahead", and write 'r(?=s)').

'r/s'
     an 'r' but only if it is followed by an 's'.  The text matched by
     's' is included when determining whether this rule is the longest
     match, but is then returned to the input before the action is
     executed.  So the action only sees the text matched by 'r'.  This
     type of pattern is called "trailing context".  (There are some
     combinations of 'r/s' that flex cannot match correctly.  *Note
     Limitations::, regarding dangerous trailing context.)