Re: --yacc and token #define's

[Joel E. Denny]

On Sat, 10 Jun 2006, Joel E. Denny wrote:

> The idea of removing token #define's in the absence of --yacc has come up 
> a few times over the past year.

There are a few unresolved questions in this thread, but I figure we can 
fix those later if necessary.  I installed this.

2006-06-11  Joel E. Denny  <address@hidden>

        For associating token numbers with token names for "yacc.c", don't use
        #define statements unless `--yacc' is specified; always use enum
        yytokentype.  Most important discussions start at:
        <http://lists.gnu.org/archive/html/bison-patches/2005-09/msg00053.html>,
        <http://lists.gnu.org/archive/html/bison-patches/2005-12/msg00052.html>,
        and
        <http://lists.gnu.org/archive/html/bison-patches/2006-06/msg00043.html>.
        * NEWS (2.3+): Mention.
        * data/c.m4 (b4_yacc_if): New.
        (b4_token_enums_defines): Use b4_yacc_if to decide whether to add the
        token #define's.
        * doc/bison.texinfo (Bison Options): Describe the effect of `--yacc'
        on token name definitions.
        * src/getargs.c (usage): Capitalize `Yacc' in English.
        * src/output.c (prepare): Define b4_yacc_flag.
        * tests/regression.at (Early token definitions): Test that tokens names
        are defined before the pre-prologue not just before the post-prologue.
        Remove this test case and copy to...
        (Early token definitions with --yacc): ... this to test #define's.
        (Early token definitions without --yacc): ... and this to test enums.

Index: NEWS
===================================================================
RCS file: /sources/bison/bison/NEWS,v
retrieving revision 1.150
diff -p -u -r1.150 NEWS
--- NEWS        11 Jun 2006 08:16:04 -0000      1.150
+++ NEWS        11 Jun 2006 18:15:55 -0000
@@ -3,6 +3,12 @@ Bison News
 
 Changes in version 2.3+:
 
+* Except for LALR(1) parsers in C with POSIX Yacc emulation enabled (with `-y',
+  `--yacc', or `%yacc'), Bison no longer generates #define statements for
+  associating token numbers with token names.  Removing the #define statements
+  helps to sanitize the global namespace during preprocessing, but POSIX Yacc
+  requires them.  Bison still generates an enum for token names in all cases.
+
 * The option `--report=look-ahead' has been changed to `--report=lookahead'.
   The old spelling still works, but is not documented and may be removed
   in a future release.
Index: data/c.m4
===================================================================
RCS file: /sources/bison/bison/data/c.m4,v
retrieving revision 1.56
diff -p -u -r1.56 c.m4
--- data/c.m4   21 May 2006 04:48:47 -0000      1.56
+++ data/c.m4   11 Jun 2006 18:15:56 -0000
@@ -207,6 +207,7 @@ b4_define_flag_if([defines])        # Wh
 b4_define_flag_if([error_verbose])  # Wheter error are verbose.
 b4_define_flag_if([locations])      # Whether locations are tracked.
 b4_define_flag_if([pure])           # Whether the interface is pure.
+b4_define_flag_if([yacc])           # Whether POSIX Yacc is emulated.
 
 
 
@@ -260,9 +261,10 @@ m4_map_sep([     b4_token_enum], [,
 
 # b4_token_enums_defines(LIST-OF-PAIRS-TOKEN-NAME-TOKEN-NUMBER)
 # -------------------------------------------------------------
-# Output the definition of the tokens (if there are) as enums and #defines.
+# Output the definition of the tokens (if there are any) as enums and, if POSIX
+# Yacc is enabled, as #defines.
 m4_define([b4_token_enums_defines],
-[b4_token_enums($@)b4_token_defines($@)
+[b4_token_enums($@)b4_yacc_if([b4_token_defines($@)], [])
 ])
 
 
Index: doc/bison.texinfo
===================================================================
RCS file: /sources/bison/bison/doc/bison.texinfo,v
retrieving revision 1.192
diff -p -u -r1.192 bison.texinfo
--- doc/bison.texinfo   10 Jun 2006 03:02:23 -0000      1.192
+++ doc/bison.texinfo   11 Jun 2006 18:16:00 -0000
@@ -6830,9 +6830,12 @@ different diagnostics to be generated, a
 other minor ways.  Most importantly, imitate Yacc's output
 file name conventions, so that the parser output file is called
 @file{y.tab.c}, and the other outputs are called @file{y.output} and
address@hidden  Thus, the following shell script can substitute
-for Yacc, and the Bison distribution contains such a script for
-compatibility with @acronym{POSIX}:
address@hidden
+Also, if generating an @acronym{LALR}(1) parser in C, generate @code{#define}
+statements in addition to an @code{enum} to associate token numbers with token
+names.
+Thus, the following shell script can substitute for Yacc, and the Bison
+distribution contains such a script for compatibility with @acronym{POSIX}:
 
 @example
 #! /bin/sh
Index: src/getargs.c
===================================================================
RCS file: /sources/bison/bison/src/getargs.c,v
retrieving revision 1.73
diff -p -u -r1.73 getargs.c
--- src/getargs.c       10 Jun 2006 03:02:23 -0000      1.73
+++ src/getargs.c       11 Jun 2006 18:16:00 -0000
@@ -217,7 +217,7 @@ Operation modes:\n\
   -h, --help                 display this help and exit\n\
   -V, --version              output version information and exit\n\
       --print-localedir      output directory containing locale-dependent 
data\n\
-  -y, --yacc                 emulate POSIX yacc\n"), stdout);
+  -y, --yacc                 emulate POSIX Yacc\n"), stdout);
       putc ('\n', stdout);
 
       fputs (_("\
Index: src/output.c
===================================================================
RCS file: /sources/bison/bison/src/output.c,v
retrieving revision 1.249
diff -p -u -r1.249 output.c
--- src/output.c        10 Jun 2006 03:02:23 -0000      1.249
+++ src/output.c        11 Jun 2006 18:16:00 -0000
@@ -590,6 +590,7 @@ prepare (void)
   MUSCLE_INSERT_BOOL ("locations_flag", locations_flag);
   MUSCLE_INSERT_BOOL ("pure_flag", pure_parser);
   MUSCLE_INSERT_BOOL ("synclines_flag", !no_lines_flag);
+  MUSCLE_INSERT_BOOL ("yacc_flag", yacc_flag);
 
   /* File names.  */
   MUSCLE_INSERT_STRING ("prefix", spec_name_prefix ? spec_name_prefix : "yy");
Index: tests/regression.at
===================================================================
RCS file: /sources/bison/bison/tests/regression.at,v
retrieving revision 1.99
diff -p -u -r1.99 regression.at
--- tests/regression.at 6 Jun 2006 16:40:06 -0000       1.99
+++ tests/regression.at 11 Jun 2006 18:16:00 -0000
@@ -49,12 +49,12 @@ AT_CLEANUP
 
 
 
-## ------------------------- ##
-## Early token definitions.  ##
-## ------------------------- ##
+## ------------------------------------- ##
+## Early token definitions with --yacc.  ##
+## ------------------------------------- ##
 
 
-AT_SETUP([Early token definitions])
+AT_SETUP([Early token definitions with --yacc])
 
 # Found in GCJ: they expect the tokens to be defined before the user
 # prologue, so that they can use the token definitions in it.
@@ -63,17 +63,56 @@ AT_DATA_GRAMMAR([input.y],
 [[%{
 void yyerror (const char *s);
 int yylex (void);
+#ifndef MY_TOKEN
+# error "MY_TOKEN not defined."
+#endif
 %}
 
 %union
 {
   int val;
 };
-%{
-#ifndef MY_TOKEN
-# error "MY_TOKEN not defined."
-#endif
+%token MY_TOKEN
+%%
+exp: MY_TOKEN;
+%%
+]])
+
+AT_CHECK([bison -y -o input.c input.y])
+AT_COMPILE([input.o], [-c input.c])
+
+AT_CLEANUP
+
+
+
+## ---------------------------------------- ##
+## Early token definitions without --yacc.  ##
+## ---------------------------------------- ##
+
+
+AT_SETUP([Early token definitions without --yacc])
+
+# Found in GCJ: they expect the tokens to be defined before the user
+# prologue, so that they can use the token definitions in it.
+
+AT_DATA_GRAMMAR([input.y],
+[[%{
+#include <stdio.h>
+void yyerror (const char *s);
+int yylex (void);
+void print_my_token (void);
+void
+print_my_token (void)
+{
+  enum yytokentype my_token = MY_TOKEN;
+  printf ("%d\n", my_token);
+}
 %}
+
+%union
+{
+  int val;
+};
 %token MY_TOKEN
 %%
 exp: MY_TOKEN;