Re: Dynamic token kinds

[Hans Åberg]

> On 16 Dec 2018, at 15:48, Frank Heckenbach <address@hidden> wrote:
> 
> Hans Åberg wrote:
> 
>> The idea would be that rather than returning the token value,
>> something that does not need translation. I don't know if that
>> helps up the static approach, though.
> 
> Not sure what you mean here. Please elaborate.

I couldn't see the details when I looked at it. I don't use the typed parser, 
but might if it is safe.

>>>> Personally, I am already at C++17, so I have no reason for using a
>>>> simpler variant. Having the type stored just adds a size_t, and
>>>> that is used a lot in other circumstances, so no overhead to worry
>>>> about.
>>> 
>>> Well, we had this discussion recently (as far as Bison is
>>> concerned).
>> 
>> Indeed, but that was where it seemed not requiring the type being
>> stored in the variant. This situation might be different in that
>> respect.
>> 
>> Here, not making sure the type is properly returned may bomb the
>> program, so preventing that seems higher than a rather small
>> overhead.
> 
> I think Akim made it clear enough that he doesn't like the overhead.
> (I don't mind so much, as I used std::variant in my implementation,
> but I only have a few parsers to care about.)

In that case, my impression was that he thought he could do without it.

> One might validly say that preventing it is the job of the lexer
> (and my lexer does so), not Bison's, just like other kinds of
> undefined or wrong behaviour outside of the generated code, also
> dynamic token types are a somewhat special usage anyway, so Bison
> can just do nothing about it, that's fine.

I use the same thing, returning the token value found on a lookup table, but I 
would not want to use the typed parser if I would have to add translations for 
every possibility. The information about it is in Bison, therefore it should 
not be put on the writing of the lexer.

> I also suggested an approach in my previous mail with a few more
> generated functions that help runtime checking. I'd prefer Bison to
> add them, and then we'd have runtime checking as good as we'd have
> with std::variant in this respect.

Maybe an option. Akim perhaps haven't used this dynamic token lookup. Those 
that do might prefer not risking the program to bomb.