Re: Dynamic token kinds

[Akim Demaille]

Hi Frank, Hi all,

> Le 16 déc. 2018 à 10:02, Frank Heckenbach <address@hidden> a écrit :
> 
> So to make it safe, we might need something like this:
> 
>  static inline
>  symbol_type
>  make_symbol (token_type type, b4_locations_if([const location_type& l, ])T&& 
> v);
> 
> auto-generated for each semantic type T of any token (plus one
> without the "v" parameter for untyped tokens) that checks (at
> runtime) the "type" parameter against the (statically known) valid
> token types for T.

I like this idea.  I have a draft for it in my repo, as "make-symbol".
Please, try it and report about it.

There are a few issues:
- make_symbol will collide if the user has a token named symbol
  Any idea of a better name?  
- In the signature of make_symbol, I've had to use int for the
  token type, instead of the enum token_type, and then
  to convert the int into token_type.  I don't like that, but I
  probably don't have much of a choice.  (A template would be
  overkill IMHO).
- I'm not sure I should emit the assert only when parse.assert
  was set, maybe we should always do it.  After all, if the user
  does not want the check, he can pass -DNDEBUG.  parse.assert
  was made to check what Bison generates, more than to check what
  the user does.

It's used like this:

yy::parser::symbol_type yylex ()
{
  static char const input[] = "12";
  static size_t toknum = 0;
  int res = input[toknum++];
  if (res == '1')
    return yy::parser::make_symbol (res, std::make_unique<int> (10));
  else if (res == '2')
    return yy::parser::make_symbol (res, std::make_pair<int, int> (21, 22));
  else
    return yy::parser::make_symbol (res);
}

The generated code looks like this (I had a token named
m4_define to make sure everything was properly quoted):

Input:
%token <pair<int, int>> FOO '+' BAR "BAR"  m4_define
%nterm <pair<int, int>> exp exp1

Output:
  inline
  parser::symbol_type
  parser::make_symbol (int tok)
  {
    assert (tok == 0);
    return symbol_type (token_type (tok));
  }

  inline
  parser::symbol_type
  parser::make_symbol (int tok, pair<int, int> v)
  {
    assert (tok == token::FOO || tok == 43 || tok == token::BAR || tok == 
token::m4_define);
    return symbol_type (token_type (tok), std::move (v));
  }


commit 1f86c28ea2ff69f68a569568ad6890dee3ac9684
Author: Akim Demaille <address@hidden>
Date:   Tue Dec 18 07:16:55 2018 +0100

    c++: provide symbol constructors per type
    
    On
    
        %token <int> FOO BAR
    
    we currently generate make_FOO(int) and make_bar(int).  However, in
    order to factor their scanners, some users would also like to have
    make_symbol(tok, int), where tok is FOO or BAR.  To ensure type
    safety, add assertions that do check that value type and token type
    match.  Bind this assertion to the parse.assert %define variable.
    
    Suggested by Frank Heckenbach.
    http://lists.gnu.org/archive/html/bug-bison/2018-12/msg00034.html
    Should also match expectations from Аскар Сафин.
    http://lists.gnu.org/archive/html/bug-bison/2018-12/msg00023.html
    
    * data/variant.hh: Use b4_token_visible_if where applicable.
    (_b4_type_constructor_declare, _b4_type_constructor_define): New.
    Use them.

diff --git a/data/variant.hh b/data/variant.hh
index 9696cf7b..75811b0d 100644
--- a/data/variant.hh
+++ b/data/variant.hh
@@ -335,6 +335,16 @@ m4_define([b4_symbol_value_template],
 ## ------------- ##
 
 
+# _b4_includes_tokens(SYMBOL-NUM...)
+# ----------------------------------
+# Expands to non-empty iff one of the SYMBOL-NUM denotes
+# a token.
+m4_define([_b4_is_token],
+          [b4_symbol_if([$1], [is_token], [1])])
+m4_define([_b4_includes_tokens],
+          [m4_map([_b4_is_token], address@hidden)])
+
+
 # _b4_symbol_constructor_declare(SYMBOL-NUM)
 # ------------------------------------------
 # Declare make_SYMBOL for SYMBOL-NUM.  Use at class-level.
@@ -358,12 +368,38 @@ b4_join(b4_symbol_if([$1], [has_type],
 ])])
 
 
+# _b4_type_constructor_declare(SYMBOL-NUM...)
+# -------------------------------------------
+# Declare a unique make_symbol for all the SYMBOL-NUM (they
+# have the same type).  Use at class-level.
+m4_define([_b4_type_constructor_declare],
+[m4_ifval(_b4_includes_tokens($@),
+[#if 201103L <= YY_CPLUSPLUS
+    static
+    symbol_type
+    make_symbol (dnl
+b4_join([int tok],
+        b4_symbol_if([$1], [has_type],
+                     [b4_symbol([$1], [type]) v]),
+        b4_locations_if([location_type l])));
+#else
+    static
+    symbol_type
+    make_symbol (dnl
+b4_join([int tok],
+        b4_symbol_if([$1], [has_type],
+                     [const b4_symbol([$1], [type])& v]),
+        b4_locations_if([const location_type& l])));
+#endif
+])])
+
+
 # b4_symbol_constructor_declare
 # -----------------------------
-# Declare symbol constructors for all the value types.
-# Use at class-level.
+# Declare symbol constructors.  Use at class-level.
 m4_define([b4_symbol_constructor_declare],
 [    // Symbol constructors declarations.
+b4_type_foreach([_b4_type_constructor_declare])
 b4_symbol_foreach([_b4_symbol_constructor_declare])])
 
 
@@ -401,6 +437,50 @@ b4_join(b4_symbol_if([$1], [has_type],
 ])])
 
 
+# _b4_type_constructor_define(SYMBOL-NUM...)
+# ------------------------------------------
+# Declare a unique make_symbol for all the SYMBOL-NUM (they
+# have the same type).  Use at class-level.
+m4_define([_b4_type_clause],
+[b4_symbol_if([$1], [is_token],
+              [b4_symbol_if([$1], [has_id],
+                            [tok == token::b4_symbol([$1], [id])],
+                            [tok == b4_symbol([$1], [user_number])])])])
+
+m4_define([_b4_type_constructor_define],
+[m4_ifval(_b4_includes_tokens($@),
+[#if 201103L <= YY_CPLUSPLUS
+  inline
+  b4_parser_class_name::symbol_type
+  b4_parser_class_name::make_symbol (dnl
+b4_join([int tok],
+        b4_symbol_if([$1], [has_type],
+                     [b4_symbol([$1], [type]) v]),
+        b4_locations_if([location_type l])))
+  {b4_parse_assert_if([
+    assert (m4_join([ || ], m4_map_sep([_b4_type_clause], [, ], 
address@hidden)));])[
+    return symbol_type (]b4_join([token_type (tok)],
+                                b4_symbol_if([$1], [has_type], [std::move 
(v)]),
+                                b4_locations_if([std::move (l)])));
+  }
+#else
+  inline
+  b4_parser_class_name::symbol_type
+  b4_parser_class_name::make_symbol (dnl
+b4_join([int tok],
+        b4_symbol_if([$1], [has_type],
+                     [const b4_symbol([$1], [type])& v]),
+        b4_locations_if([const location_type& l])))
+  {b4_parse_assert_if([
+    assert (m4_join([ || ], m4_map_sep([_b4_type_clause], [, ], 
address@hidden)));])[
+    return symbol_type (]b4_join([token_type (tok)],
+                                b4_symbol_if([$1], [has_type], [v]),
+                                b4_locations_if([l])));
+  }
+#endif
+])])
+
+
 # b4_basic_symbol_constructor_declare(SYMBOL-NUM)
 # -----------------------------------------------
 # Generate a constructor declaration for basic_symbol from given type.
@@ -451,4 +531,5 @@ m4_define([b4_basic_symbol_constructor_define],
 # Define the overloaded versions of make_symbol for all the value types.
 m4_define([b4_symbol_constructor_define],
 [  // Implementation of make_symbol for each symbol type.
+b4_type_foreach([_b4_type_constructor_define])
 b4_symbol_foreach([_b4_symbol_constructor_define])])
diff --git a/tests/types.at b/tests/types.at
index 2289f751..adc8ec6a 100644
--- a/tests/types.at
+++ b/tests/types.at
@@ -327,6 +327,26 @@ m4_foreach([b4_skel], [[yacc.c], [glr.c], [lalr1.cc], 
[glr.cc]],
             [10, 21, 22],
             [AT_REQUIRE_CXX_STD(14, [echo "$at_std not supported"; continue])])
 
+    # Type-based token constructors on move-only types, and types with commas.
+    AT_TEST([%skeleton "]b4_skel["
+             %code requires { #include <memory> }
+             %define api.value.type variant
+             %define api.token.constructor],
+            [[%token <std::unique_ptr<int>> '1';
+             %token <std::pair<int, int>> '2';]],
+            ['1' '2' { std::cout << *$1 << ", "
+                                 << $2.first << ", "
+                                 << $2.second << '\n'; }],
+            ["12"],
+            [[if (res == '1')
+               return yy::parser::make_symbol ('1', std::make_unique<int> 
(10));
+             else if (res == '2')
+               return yy::parser::make_symbol ('2', std::make_pair<int, int> 
(21, 22));
+             else
+               return yy::parser::make_symbol (0)]],
+            [10, 21, 22],
+            [AT_REQUIRE_CXX_STD(14, [echo "$at_std not supported"; continue])])
+
   ])
 ])