Re: FTR: fix the interface of yyexpected_tokens

[Akim Demaille]

> Le 30 avr. 2020 à 06:55, Akim Demaille <address@hidden> a écrit :
> 
> Hi!
> 
>> Le 29 avr. 2020 à 07:58, Akim Demaille <address@hidden> a écrit :
>> 
>> If the user wants 5 tokens at most, and would like to know if there
>> are actually more than 5, or really 0, then let her ask for 6, and
>> not use the 6th one.
> 
> This is stupid, and of course does not work.  If there are 10 expected
> tokens, asking for 6 will return 0 again.  So I guess I should install
> that change anyway.

I'm installing this.

commit d9a9b054ae1821e0f01fa876180961f5b2fa05bd
Author: Akim Demaille <address@hidden>
Date:   Wed Apr 29 07:58:51 2020 +0200

    all: fix the interface of yyexpected_tokens
    
    The user gives yyexpected_tokens a limit: the max number of tokens she
    wants to hear about.  That's because an error message that reports a
    bazillion of possible tokens is useless.
    
    In that case yyexpected_tokens returned 0, so the user would not know
    if there are too many expected tokens or none (yes, that's possible).
    
    There are several ways to tell the user in which situation she's in:
    
    - return some E2MANY, a negative value.  Then it makes the pattern
    
        int argsize = yypcontext_expected_tokens (ctx, arg, ARGS_MAX);
        if (argsize < 0)
          return argsize;
    
      no longer valid, as for E2MANY (i) the user must generate the error
      message anyway, and (ii) she should not return E2MANY
    
    - return ARGS_MAX + 1.  Then it makes it dangerous for the user, as
      she has to iterate update `min (ARGS_MAX, argsize)`.
    
    Returning 0 is definitely simpler and safer for the user, as it tells
    her "this is not an error, just generate your message without a list
    of expecting tokens".  So let's still return 0, but set arg[0] to the
    empty token when the list is really empty.
    
    * data/skeletons/glr.c, data/skeletons/lalr1.cc, data/skeletons/lalr1.java
    * data/skeletons/yacc.c (yyexpected_tokens): Put the empty symbol
    first if there are no possible tokens at all.
    * examples/c/bistromathic/parse.y: Demonstrate how to use that.

diff --git a/data/skeletons/glr.c b/data/skeletons/glr.c
index ab3a0718..c5074a3e 100644
--- a/data/skeletons/glr.c
+++ b/data/skeletons/glr.c
@@ -2119,6 +2119,8 @@ yypcontext_expected_tokens (const yyGLRStack* yystackp,
               yyarg[yycount++] = YY_CAST (yysymbol_kind_t, yyx);
           }
     }
+  if (yyarg && yycount == 0 && 0 < yyargn)
+    yyarg[0] = ]b4_symbol(-2, kind)[;
   return yycount;
 }]])[
 
diff --git a/data/skeletons/lalr1.cc b/data/skeletons/lalr1.cc
index 2b27088f..e6da4d81 100644
--- a/data/skeletons/lalr1.cc
+++ b/data/skeletons/lalr1.cc
@@ -1326,6 +1326,8 @@ b4_dollar_popdef])[]dnl
             }
       }
 ]])[
+    if (yyarg && yycount == 0 && 0 < yyargn)
+      yyarg[0] = symbol_kind::]b4_symbol(-2, kind)[;
     return yycount;
   }
 
diff --git a/data/skeletons/lalr1.java b/data/skeletons/lalr1.java
index 1baa497d..ce29a3f3 100644
--- a/data/skeletons/lalr1.java
+++ b/data/skeletons/lalr1.java
@@ -944,6 +944,8 @@ b4_dollar_popdef[]dnl
                   yyarg[yycount++] = SymbolKind.get (yyx);
               }
         }
+      if (yyarg != null && yycount == yyoffset && yyoffset < yyargn)
+        yyarg[yycount] = null;
       return yycount - yyoffset;
     }
   }
diff --git a/data/skeletons/yacc.c b/data/skeletons/yacc.c
index ef65287f..a790f6fa 100644
--- a/data/skeletons/yacc.c
+++ b/data/skeletons/yacc.c
@@ -1158,6 +1158,8 @@ yypcontext_expected_tokens (const yypcontext_t *yyctx,
               yyarg[yycount++] = YY_CAST (yysymbol_kind_t, yyx);
           }
     }]])[
+  if (yyarg && yycount == 0 && 0 < yyargn)
+    yyarg[0] = ]b4_symbol(-2, kind)[;
   return yycount;
 }
 
diff --git a/doc/bison.texi b/doc/bison.texi
index aeb176a4..fd497d2d 100644
--- a/doc/bison.texi
+++ b/doc/bison.texi
@@ -7551,9 +7551,12 @@ Fill @var{argv} with the expected tokens, which never 
includes
 @code{YYSYMBOL_YYUNDEF}.
 
 Never put more than @var{argc} elements into @var{argv}, and on success
-return the effective number of tokens stored in @var{argv}.  Return 0 if
-there are more than @var{argc} expected tokens, yet fill @var{argv} up to
-@var{argc}.  When LAC is enabled, may return a negative number on errors,
+return the number of tokens stored in @var{argv}.  If there are more
+expected tokens than @var{argc}, fill @var{argv} up to @var{argc} and return
+0.  If there are no expected tokens, also return 0, but set @code{argv[0]}
+to @code{YYSYMBOL_YYEMPTY}.
+
+When LAC is enabled, may return a negative number on errors,
 such as @code{YYENOMEM} on memory exhaustion.
 
 If @var{argv} is null, return the size needed to store all the possible
@@ -12137,9 +12140,10 @@ Fill @var{argv} with the expected tokens, which never 
includes
 @code{symbol_kind::S_YYUNDEF}.
 
 Never put more than @var{argc} elements into @var{argv}, and on success
-return the effective number of tokens stored in @var{argv}.  Return 0 if
-there are more than @var{argc} expected tokens, yet fill @var{argv} up to
-@var{argc}.
+return the number of tokens stored in @var{argv}.  If there are more
+expected tokens than @var{argc}, fill @var{argv} up to @var{argc} and return
+0.  If there are no expected tokens, also return 0, but set @code{argv[0]}
+to @code{symbol_kind::S_YYEMPTY}.
 
 If @var{argv} is null, return the size needed to store all the possible
 values, which is always less than @code{YYNTOKENS}.
@@ -13298,9 +13302,10 @@ Fill @var{argv} with the expected tokens, which never 
includes
 @code{SymbolKind.S_YYERROR}, or @code{SymbolKind.S_YYUNDEF}.
 
 Never put more than @var{argc} elements into @var{argv}, and on success
-return the effective number of tokens stored in @var{argv}.  Return 0 if
-there are more than @var{argc} expected tokens, yet fill @var{argv} up to
-@var{argc}.
+return the number of tokens stored in @var{argv}.  If there are more
+expected tokens than @var{argc}, fill @var{argv} up to @var{argc} and return
+0.  If there are no expected tokens, also return 0, but set @code{argv[0]}
+to @code{null}.
 
 If @var{argv} is null, return the size needed to store all the possible
 values, which is always less than @code{YYNTOKENS}.
diff --git a/examples/c/bistromathic/bistromathic.test 
b/examples/c/bistromathic/bistromathic.test
index 3bb83b77..495146bd 100755
--- a/examples/c/bistromathic/bistromathic.test
+++ b/examples/c/bistromathic/bistromathic.test
@@ -96,7 +96,7 @@ cat >input <<EOF
 EOF
 run 0 '> *
 > ''
-err: 1.1: syntax error: expected end of file or - or ( or exit or number or 
function or variable before *'
+err: 1.1: syntax error: expected end of file or - or ( or exit or number or 
function etc., before *'
 
 cat >input <<EOF
 1 + 2 * * 3
diff --git a/examples/c/bistromathic/parse.y b/examples/c/bistromathic/parse.y
index 38fedb46..c2ddb124 100644
--- a/examples/c/bistromathic/parse.y
+++ b/examples/c/bistromathic/parse.y
@@ -333,7 +333,7 @@ error_format_string (int argc)
     case 5: return _("%@: syntax error: expected %0e or %1e or %2e or %3e 
before %u");
     case 6: return _("%@: syntax error: expected %0e or %1e or %2e or %3e or 
%4e before %u");
     case 7: return _("%@: syntax error: expected %0e or %1e or %2e or %3e or 
%4e or %5e before %u");
-    case 8: return _("%@: syntax error: expected %0e or %1e or %2e or %3e or 
%4e or %5e or %6e before %u");
+    case 8: return _("%@: syntax error: expected %0e or %1e or %2e or %3e or 
%4e or %5e etc., before %u");
     }
 }
 
@@ -341,12 +341,15 @@ error_format_string (int argc)
 int
 yyreport_syntax_error (const yypcontext_t *ctx)
 {
-  enum { ARGS_MAX = 7 };
+  enum { ARGS_MAX = 6 };
   yysymbol_kind_t arg[ARGS_MAX];
   int argsize = yypcontext_expected_tokens (ctx, arg, ARGS_MAX);
   if (argsize < 0)
     return argsize;
-  const char *format = error_format_string (1 + argsize);
+  const int too_many_expected_tokens = argsize == 0 && arg[0] != 
YYSYMBOL_YYEMPTY;
+  if (too_many_expected_tokens)
+    argsize = ARGS_MAX;
+  const char *format = error_format_string (1 + argsize + 
too_many_expected_tokens);
 
   while (*format)
     // %@: location.