Accessing token type with .token()

[Felix Nieuwenhuizen]

Hi all,

I'm using bison with "Complete Symbols" with %define api.token.constructor,
and yylex is defined to return symbol_type with YY_DECL.

I have some testcases that test basic lexing.
Basically, I parse/lex some synthetic example files, and make sure the
right tokens and values are returned.
Here's a snippet:
for (auto id : ids) {
  auto symbol = lexer->yylex(*driver);
  ASSERT_EQ(symbol.token(), token::T_BASIC_IDENTIFIER);
  ASSERT_EQ(symbol.value.as<std::string>(), id);
}
(Where BASIC_IDENTIFIER is one of the tokens defined in my .yy file and
T_BASIC_IDENTIFIER is in the generated token_type / yytokentype enum)

This was working with Bison 3.4.x, but stopped working with Bison 3.5:
error: no member named 'token' in '...::Parser::symbol_type'
    ASSERT_EQ(symbol.token(), token::T_BASIC_IDENTIFIER);

token() was removed in commit 8c87a623, so the error is understandable.
Unfortunatly, I don't understand the distinction between internal and
external token numbers, so I'm not sure how to proceed.
I did try using .type / .type_get() instead of .token(), but as I
understand it these return the internal token numbers instead of the
external token numbers.
And since the yytokentype enum (e.g. T_BASIC_IDENTIFIER) uses the
external numbers, they don't match.

Maybe I'm doing something unorthodox, but what's the best way of
accessing/checking which token type is returned by yylex()?

Cheers, Felix