[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: java push parser diffs yet again...
|
From: |
Akim Demaille |
|
Subject: |
Re: java push parser diffs yet again... |
|
Date: |
Thu, 7 Feb 2013 11:45:55 +0100 |
Hi Dennis,
Please, stay in the same thread, so that the archives
can be easy to read. Also, I think that Paolo is not
a subscriber of this list.
Sorry for the delays too.
I have installed your patch in the java-push branch on
gnu's git repository. I have put two patches on top
of yours:
commit 7fa0a8b5f857ce6eb8aa6b7c2a303bd47c09924b
Author: Akim Demaille <address@hidden>
Date: Wed Feb 6 10:03:00 2013 +0100
squashme: spaces fixes
because you left plenty of trailing spaces, and even used
DOS like end of lines.
commit 273691f88b5871b07b3778f4552a1a294b6c12b6
Author: Akim Demaille <address@hidden>
Date: Wed Feb 6 10:06:40 2013 +0100
squashme: restore the full test suite
* tests/testsuite.at: here.
because testsuite.at was reduced to only your tests. If
you want to run some tests only, use
make check TESTSUITEFLAGS='1 23 234'
etc.
When the stack of patches is mature enough, I will squash
them all in a single commit, so that the history of the
master branch remains clean.
So *do work from this branch*, and no longer from master.
Thanks in advance.
Le 2 févr. 2013 à 00:40, Dennis Heimbigner <address@hidden> a écrit :
> Next attempt to include java push
> parsing functionality.
> I think this version includes
> all requested changes to date.
>
> =Dennis Heimbigner
> From 7a85bd3193b288aee8cfdbb6d3f2667b6d17d455 Mon Sep 17 00:00:00 2001
> From: Dennis Heimbigner <address@hidden>
> Date: Fri, 1 Feb 2013 16:33:40 -0700
> Subject: [PATCH 1/1] skeletons: add java push parsing
>
> Added java push parsing support
>
> * data/lalr1.java
>
> 1. capture the declarations as m4 macros. This was done to
> avoid duplication. When push parsing, the declarations
> occur at the class instance level rather than within the
> parse() function.
>
> 2. Initialization of the declarations occurs in a function
> called push_parse_initialize() that is called on the first
> invocation of push_parse().
>
> 3. The body of the parse loop is modified to return values at
> appropriate points when doing push parsing. In order to
> make push parsing work, it was necessary to divide
> YYNEWSTATE into two states: YYNEWSTATE and YYGETTOKEN. On
> the first call to push_parse, the state is YYNEWSTATE. On
> all later entries, the state is set to YYGETTOKEN. The
> YYNEWSTATE switch arm falls through into
> YYGETTOKEN. YYGETTOKEN indicates that a new token is
> potentially needed. Normally, with a pull parser, this new
> token would be obtained by calling yylex(). In the push
> parser, the value YYMORE is returned to the caller. On the
> next call to push_parse(), the parser will return to the
> YYGETTOKEN state and continue operation.
>
> * tests/javapush.at: new test file for java push parsing
>
> * tests/testsuite.at: add invocation of javapush.at
>
> * tests/local.mk: add javapush.at to distribution
>
> * NEWS: add note about Java push parsing support
>
> * doc/bison.texi: add description of how to use Java push parsing
Please, prepare a rewording of the commit message so that
it looks like the other commit messages. In particular,
start sentences with an upper case, and end with a period.
> ---
> NEWS | 3 +
> data/lalr1.java | 239 +++-
> doc/bison.texi | 73 +-
> tests/javapush.at | 4278 ++++++++++++++++++++++++++++++++++++++++++++++++++++
> tests/local.mk | 1 +
> tests/testsuite.at | 77 +-
> 6 files changed, 4552 insertions(+), 119 deletions(-)
> create mode 100755 tests/javapush.at
>
> diff --git a/NEWS b/NEWS
> index 20c3100..8fdcbbd 100644
> --- a/NEWS
> +++ b/NEWS
> @@ -221,6 +221,9 @@ GNU Bison NEWS
> is possible to add code to the parser's constructors using "%code init"
> and "%define init_throws".
>
> + Contributed by Dennis Heimbigner
> + The java skeleton, data/lalr1.java, now supports push parsing.
> +
> ** C++ skeletons improvements
>
> *** The parser header is no longer mandatory (lalr1.cc, glr.cc)
> diff --git a/data/lalr1.java b/data/lalr1.java
> index 187580a..b11b90b 100644
> --- a/data/lalr1.java
> +++ b/data/lalr1.java
> @@ -29,6 +29,52 @@ m4_define([b4_symbol_no_destructor_assert],
> [b4_skeleton],
> [b4_symbol_action_location([$1], [destructor])])])])
> b4_symbol_foreach([b4_symbol_no_destructor_assert])
> +b4_percent_define_default([[api.push-pull]], [[pull]])dnl
> +b4_percent_define_check_values([[[[api.push-pull]],
> + [[pull]], [[push]], [[both]]]])dnl
> +b4_define_flag_if([pull]) m4_define([b4_pull_flag], [[1]])dnl
> +b4_define_flag_if([push]) m4_define([b4_push_flag], [[1]])dnl
> +m4_case(b4_percent_define_get([[api.push-pull]]),
> + [pull], [m4_define([b4_push_flag], [[0]])],
> + [push], [m4_define([b4_pull_flag], [[0]])])dnl
If this piece of code is alike something that is already available
in C, consider factoring all this in the c-like.m4 file, which is
common to C, C++ and Java.
> +
> +# Handle BISON_USE_PUSH_FOR_PULL for the test suite. So that push parsing
> +# tests function as written, do not let BISON_USE_PUSH_FOR_PULL modify the
> +# behavior of Bison at all when push parsing is already requested.
> +b4_define_flag_if([use_push_for_pull])dnl
> +b4_use_push_for_pull_if([
> + b4_push_if([m4_define([b4_use_push_for_pull_flag], [[0]])],
> + [m4_define([b4_push_flag], [[1]])])])
It's not just the test suite: users will be able to move
gradually from pull to push.
> +
> +# Define a macro to be true when api.push-pull has the value "both"
> +m4_define([b4_both_if],[b4_push_if([b4_pull_if([$1],[$2])],[$2])])dnl
> +
> +# Define a macro to encapsulate the parse state variables.
> +# This allows them to be defined either in parse() when doing
> +# pull parsing, or as class instance variable when doing push parsing.
> +m4_define([b4_define_state],[[
> + /// Lookahead and lookahead in internal form.
> + int yychar = yyempty_;
> + int yytoken = 0;
> +
> + /* State. */
Could you please convert to a uniform style of comments?
> + int yyn = 0;
> + int yylen = 0;
> + int yystate = 0;
> + YYStack yystack = new YYStack ();
> + int label = YYNEWSTATE;
> +
> + /* Error handling. */
> + int yynerrs_ = 0;
> + ]b4_locations_if([/// The location where the error started.
> + b4_location_type yyerrloc = null;
> +
> + /// b4_location_type of the lookahead.
"Location" seems good enough (yes, I can see this was not
introduced by your patch, you just moved it).
> + b4_location_type yylloc = new b4_location_type (null, null);])[
> +
> + /// Semantic value of the lookahead.
> + ]b4_yystype[ yylval = null;
> +]])
>
> b4_output_begin([b4_parser_file_name])
> b4_copyright([Skeleton implementation for Bison LALR(1) parsers in Java],
> @@ -344,6 +390,12 @@ b4_lexer_if([[
> * return failure (<tt>false</tt>). */
> public static final int YYABORT = 1;
>
> +]b4_push_if([
> + /**
> + * Returned by a Bison action in order to request a new token.
> + */
> + public static final int YYMORE = 4;])[
> +
> /**
> * Returned by a Bison action in order to start error recovery without
> * printing an error message. */
> @@ -357,9 +409,12 @@ b4_lexer_if([[
> private static final int YYREDUCE = 6;
> private static final int YYERRLAB1 = 7;
> private static final int YYRETURN = 8;
> +]b4_push_if([ private static final int YYGETTOKEN = 9;])[
Maybe a line of comment?
>
> private int yyerrstatus_ = 0;
>
> +]b4_push_if([dnl
> +b4_define_state])[
> /**
> * Return whether error recovery is being done. In this state, the parser
> * reads token until it reaches a known state, and then restarts normal
> @@ -463,6 +518,7 @@ b4_lexer_if([[
> + (yyvaluep == null ? "(null)" : yyvaluep.toString ()) + ")");
> }
>
> +]b4_push_if([],[
It should be opening with two [, that's why below you removed
levels of quotation.
> /**
> * Parse input from the scanner that was specified at object construction
> * time. Return whether the end of the input was reached successfully.
> @@ -470,46 +526,53 @@ b4_lexer_if([[
> * @@return <tt>true</tt> if the parsing succeeds. Note that this does not
> * imply that there were no syntax errors.
> */
> - public boolean parse () ]b4_maybe_throws([b4_list2([b4_lex_throws],
> [b4_throws])])[
> + public boolean parse () b4_maybe_throws([b4_list2([b4_lex_throws],
> [b4_throws])])])[
> +]b4_push_if([
> + /**
> + * Push Parse input from external lexer
> + *
> + * @@param yylextoken current token
> + * @@param yylexval current lval
> +b4_locations_if([ * @@param yylexloc current position])
> + *
> + * @@return <tt>YYACCEPT, YYABORT, YYMORE</tt>
> + */
> + public int push_parse (int yylextoken, b4_yystype
> yylexval[]b4_locations_if([, b4_location_type yylexloc]))
> + b4_maybe_throws([b4_list2([b4_lex_throws], [b4_throws])])])[
> {
> - /// Lookahead and lookahead in internal form.
> - int yychar = yyempty_;
> - int yytoken = 0;
> -
> - /* State. */
> - int yyn = 0;
> - int yylen = 0;
> - int yystate = 0;
> -
> - YYStack yystack = new YYStack ();
> -
> - /* Error handling. */
> - int yynerrs_ = 0;
> - ]b4_locations_if([/// The location where the error started.
> - ]b4_location_type[ yyerrloc = null;
> -
> - /// ]b4_location_type[ of the lookahead.
> - ]b4_location_type[ yylloc = new ]b4_location_type[ (null, null);
> -
> - /// @@$.
> - ]b4_location_type[ yyloc;])
> -
> - /// Semantic value of the lookahead.
> - b4_yystype[ yylval = null;
> -
> + ]b4_locations_if([/// @@$.
> + b4_location_type yyloc;])[
> +]b4_push_if([],[[
> +]b4_define_state[
> yycdebug ("Starting parse\n");
> yyerrstatus_ = 0;
>
> + /* Initialize the stack. */
> + yystack.push (yystate, yylval ]b4_locations_if([, yylloc])[);
> ]m4_ifdef([b4_initial_action], [
> b4_dollar_pushdef([yylval], [], [yylloc])dnl
> /* User initialization code. */
> b4_user_initial_action
> -b4_dollar_popdef])[]dnl
> -
> - [ /* Initialize the stack. */
> - yystack.push (yystate, yylval]b4_locations_if([, yylloc])[);
> +b4_dollar_popdef[]dnl
> +])[
> +]])[
> +]b4_push_if([[
> + boolean push_token_consumed = true;
>
> - int label = YYNEWSTATE;
> + if (!push_parse_initialized)
> + {
> + push_parse_initialize ();
> + yycdebug ("Starting parse\n");
> + yyerrstatus_ = 0;
> + } else
> + label = YYGETTOKEN;
> +]m4_ifdef([b4_initial_action], [
> +b4_dollar_pushdef([yylval], [], [yylloc])dnl
> + /* User initialization code. */
> + b4_user_initial_action
Have you checked how your skeleton behaves with %initial-action?
It looks like it's going to run the initial-action at each call
to parse(), but I might be wrong.
> +b4_dollar_popdef[]dnl
> +])[
> +]])[
> for (;;)
> switch (label)
> {
> @@ -522,7 +585,7 @@ b4_dollar_popdef])[]dnl
>
> /* Accept? */
> if (yystate == yyfinal_)
> - return true;
> + ]b4_push_if([{label = YYACCEPT; break;}],[return true;])[
Please, align both '[' of the b4_push_if arguments.
>
> /* Take a decision. First try without lookahead. */
> yyn = yypact_[yystate];
> @@ -531,16 +594,28 @@ b4_dollar_popdef])[]dnl
> label = YYDEFAULT;
> break;
> }
> +]b4_push_if([ /* Fall Through */
>
> + case YYGETTOKEN:])[
> /* Read a lookahead token. */
> if (yychar == yyempty_)
> {
> +]b4_push_if([[
> + if (!push_token_consumed)
> + return YYMORE;
> yycdebug ("Reading a token: ");
> - yychar = yylexer.yylex ();]
> - b4_locations_if([[
> - yylloc = new ]b4_location_type[(yylexer.getStartPos (),
> - yylexer.getEndPos ());]])
> - yylval = yylexer.getLVal ();[
> + yychar = yylextoken;
> + yylval = yylexval;
> + ]b4_locations_if([yylloc = yylexloc;])[
> + push_token_consumed = false;]])[
> +]b4_push_if([],[[
It looks like this is a single b4_push_if, but with two branches
(if you fuse it with the previous b4_push_if).
> + yycdebug ("Reading a token: ");
> + yychar = yylexer.yylex ();
> + yylval = yylexer.getLVal ();
> + ]b4_locations_if([dnl
> + yylloc = new b4_location_type (yylexer.getStartPos (),
> + yylexer.getEndPos ());])[
> +]])[
> }
>
> /* Convert token to internal form. */
> @@ -637,10 +712,10 @@ b4_dollar_popdef])[]dnl
> {
> /* Return failure if at end of input. */
> if (yychar == Lexer.EOF)
> - return false;
> + ]b4_push_if([{label = YYABORT; break;}],[return false;])[
Likewise.
> }
> else
> - yychar = yyempty_;
> + yychar = yyempty_;
> }
>
> /* Else will try to reuse lookahead token after shifting the error
> @@ -684,7 +759,7 @@ b4_dollar_popdef])[]dnl
>
> /* Pop the current state because it cannot handle the error
> token. */
> if (yystack.height == 0)
> - return false;
> + ]b4_push_if([{label = YYABORT; break;}],[return false;])[
Likewise.
>
> ]b4_locations_if([yyerrloc = yystack.locationAt (0);])[
> yystack.pop ();
> @@ -693,6 +768,10 @@ b4_dollar_popdef])[]dnl
> yystack.print (yyDebugStream);
> }
>
> + if (label == YYABORT)
> + /* leave the switch */
"Leave the switch."
> + break;
> +
> ]b4_locations_if([
> /* Muck with the stack to setup for yylloc. */
> yystack.push (0, null, yylloc);
> @@ -711,13 +790,88 @@ b4_dollar_popdef])[]dnl
>
> /* Accept. */
> case YYACCEPT:
> - return true;
> + ]b4_push_if([push_parse_initialized = false; return YYACCEPT;],
> + [return true;])[
>
> /* Abort. */
> case YYABORT:
> - return false;
> + ]b4_push_if([push_parse_initialized = false; return YYABORT;],
> + [return false;])[
> }
> +}
> +]b4_push_if([[
> + boolean push_parse_initialized = false;
> +
> + public void push_parse_initialize()
> + {
> + // (Re-)Initialize the state
Please add a period.
> + /// Lookahead and lookahead in internal form.
> + this.yychar = yyempty_;
> + this.yytoken = 0;
> +
> + /* State. */
> + this.yyn = 0;
> + this.yylen = 0;
> + this.yystate = 0;
> + this.yystack = new YYStack ();
> + this.label = YYNEWSTATE;
> +
> + /* Error handling. */
> + this.yynerrs_ = 0;
> + ]b4_locations_if([/// The location where the error started.
> + this.yyerrloc = null;
> + this.yylloc = new b4_location_type (null, null);])[
> +
> + /// Semantic value of the lookahead.
> + this.yylval = null;
> +
> + yystack.push (this.yystate, this.yylval]b4_locations_if([,
> this.yylloc])[);
> + push_parse_initialized = true;
> }
> +]b4_locations_if([
> + /**
> + * Push Parse input from external lexer
> + *
> + * @@param yylextoken current token
> + * @@param yylexval current lval
> + * @@param yyylexpos current position
> + *
> + * @@return <tt>YYACCEPT, YYABORT, YYMORE</tt>
> + */
> + public int push_parse (int yylextoken, b4_yystype yylexval,
> b4_position_type yylexpos)
> + b4_maybe_throws([b4_list2([b4_lex_throws], [b4_throws])])
> + {
> + return push_parse (yylextoken, yylexval, new b4_location_type
> (yylexpos));
> + }
> +])[]])
> +
> +b4_both_if([[
> + /**
> + * Parse input from the scanner that was specified at object construction
> + * time. Return whether the end of the input was reached successfully.
> + * This version of parse () is defined only when api.push-push=both
> + *
> + * @@return <tt>true</tt> if the parsing succeeds. Note that this does not
> + * imply that there were no syntax errors.
> + */
> + public boolean parse () ]b4_maybe_throws([b4_list2([b4_lex_throws],
> [b4_throws])])[
> + {
> + if (yylexer == null)
> + throw new NullPointerException("Null Lexer");
> + int status;
> + do {
> + int token = yylexer.yylex();
> + ]b4_yystype[ lval = yylexer.getLVal();
> +]b4_locations_if([dnl
> + b4_location_type yyloc = new b4_location_type (yylexer.getStartPos
> (),
> + yylexer.getEndPos ());])[
> + this.yyerrstatus_ = 0;
> + ]b4_locations_if([status = push_parse(token,lval,yyloc);],[
> + status = push_parse(token,lval);])[
> + } while (status == YYMORE);
> + return (status == YYACCEPT);
> + }
> +]])[
>
> // Generate an error message.
> private String yysyntax_error (int yystate, int tok)
> @@ -825,6 +979,7 @@ b4_dollar_popdef])[]dnl
> ]b4_integral_parser_table_define([rline], [b4_rline],
> [[YYRLINE[YYN] -- Source line where rule number YYN was defined.]])[
>
> +
> // Report on the debug stream that the rule yyrule is going to be reduced.
> private void yy_reduce_print (int yyrule, YYStack yystack)
> {
> diff --git a/doc/bison.texi b/doc/bison.texi
> index 1218b58..ca09ae0 100644
> --- a/doc/bison.texi
> +++ b/doc/bison.texi
> @@ -358,6 +358,7 @@ Java Parsers
> * Java Parser Interface:: Instantiating and running the parser
> * Java Scanner Interface:: Specifying the scanner for the parser
> * Java Action Features:: Special features for use in actions
> +* Java Push Parser Interface:: Instantiating and running the a push parser
> * Java Differences:: Differences between C/C++ and Java Grammars
> * Java Declarations Summary:: List of Bison declarations used with Java
>
> @@ -11228,6 +11229,7 @@ main (int argc, char *argv[])
> * Java Parser Interface:: Instantiating and running the parser
> * Java Scanner Interface:: Specifying the scanner for the parser
> * Java Action Features:: Special features for use in actions
> +* Java Push Parser Interface:: Instantiating and running the a push parser
> * Java Differences:: Differences between C/C++ and Java Grammars
> * Java Declarations Summary:: List of Bison declarations used with Java
> @end menu
> @@ -11539,7 +11541,6 @@ The return type can be changed using @samp{%define
> api.value.type
> "@var{class-name}".}
> @end deftypemethod
>
> -
> @node Java Action Features
> @subsection Special Features for Use in Java Actions
>
> @@ -11618,6 +11619,76 @@ instance in use. The @code{Location} and
> @code{Position} parameters are
> available only if location tracking is active.
> @end deftypefn
>
> address@hidden Java Push Parser Interface
> address@hidden Java Push Parser Interface
> address@hidden - define push_parse
> address@hidden %define api.push-pull
> +
> +(The current push parsing interface is experimental and may evolve. More
> user feedback will help to stabilize it.)
Please, wrap your line so that they are not longer than 76 columns.
> +
> +Normally, Bison generates a pull parser for Java.
> +The following Bison declaration says that you want the parser to be a push
> +parser (@pxref{%define Summary,,api.push-pull}):
> +
> address@hidden
> +%define api.push-pull push
> address@hidden example
> +
> +Most of the discussion about the
> +Java pull Parser Interface,
> +(@pxref{Java Parser Interface})
> +applies to the push parser interface as well.
> +
> +When generating a push parser, the method @code{push_parse} is
> +created with the following signature (depending on if locations are
> +enabled).
> +
> address@hidden {YYParser} {void} push_parse ({int} @var{token}, {Object}
> @var{yylval})
> address@hidden {YYParser} {void} push_parse ({int} @var{token}, {Object}
> @var{yylval}, {Location} @var{yyloc})
> address@hidden {YYParser} {void} push_parse ({int} @var{token}, {Object}
> @var{yylval}, {Position} @var{yypos})
You accept lone positions? So far we have either nothing, or a
full location.
> address@hidden deftypemethod
> +
> +The primary difference with respect to
> +a pull parser is that the parser method
> address@hidden is invoked repeatedly to parse each token.
> +This function is available if either the
> +"%define api.push-pull push" or "%define api.push-pull both"
replace "…" by @samp{…}.
> +declaration is used (@pxref{%define Summary,,api.push-pull}).
I'm not sure this use of @pxref{} is really conform to the
Texinfo specification. Have you checked what it looks like
in PDF and HTML?
> +The @code{Location} and @code{Position} parameters are
> +available only if location tracking is active.
> +
> +The value returned by the @code{push_parse} method
> +is one of the following four constants:
> address@hidden, @code{YYACCEPT}, @code{YYERROR}, or @code{YYMORE}.
> +This new value,
> address@hidden, may be returned if more input is required to finish
> +parsing the grammar.
> +
> +If api.push-pull
If @code{api.push-pull}
> is declared as @code{both}, then the generated parser
> +class will also implement the @code{parse} method. This method's
> +body is a loop that repeatedly invokes the scanner and then
> +passes the values obtained from the scanner to the @code{push_parse}
> +method.
> +
> +There is one additional complication.
> +Technically, the push parser does not need to know about the scanner
> +(i.e. an object implementing the @code{YYParser.Lexer} interface),
> +but it does need access to the @code{yyerror} method.
> +The current approach (and subject to change) is to require
> +the @code{YYParser} constructor to be given an object implementing
> +the @code{YYParser.Lexer} interface. This object need only
> +implement the @code{yyerror} method; the other methods can be stubbed
> +since they will never be invoked.
I don't get it here. Are you saying that yyerror is provided
by the scanner?
> +The simplest way to do this is to add code like this to your
> +.y file.
"to your @file{.y} file", or "to your grammar file".
> +
> address@hidden
> +%code lexer @{
> +public Object getLVal() @{return null;@}
> +public int yylex() @{return 0;@}
> +public void yyerror(String s) @{System.err.println(s);@}
Please, "{ … }" instead of "{…}".
> address@hidden
> address@hidden example
>
> @node Java Differences
> @subsection Differences between C/C++ and Java Grammars
> diff --git a/tests/javapush.at b/tests/javapush.at
> new file mode 100755
> index 0000000..80f7bef
> --- /dev/null
> +++ b/tests/javapush.at
> @@ -0,0 +1,4278 @@
> +# Checking Java Push Parsing. -*- Autotest -*-
> +
> +# Copyright © 2007, 2009-2013 Free Software Foundation, Inc.
2013
> +
> +# This program is free software: you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation, either version 3 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program. If not, see <http://www.gnu.org/licenses/>.
> +
> +# The Java push parser tests are intended primarily
> +# to verify that the sequence of states that the parser
> +# traverses is the same as a pull parser would traverse.
> +
> +##################################################
> +# Provide a way to generate data with and without push parsing
> +# so we can capture the output for comparison (except test 1).
> +# Use both rather than push so we can also set it to pull to
use "both" … "push" … "pull"
> +# get the experr data.
> +
> +m4_define([PUSHPULLFLAGS],[-Dapi.push-pull=both])
> +
> +# Bring in the grep tester from java.at
> +# AT_CHECK_JAVA_GREP([PARSERFILE], [LINE], [COUNT=1])
> +# -------------------------------------
> +# Check that PARSERFILEcontains exactly COUNT lines matching ^LINE$
> +# with grep.
> +
> +# Modify to ignore status so we can test for zero matching lines
> +m4_define([AT_CHECK_JAVA_GREP],
> + [AT_CHECK([grep -c '^$2$' $1], [ignore], [m4_default([$3], [1])
> +])
> +])
You don't need to do that: if java.at is loaded before, the
definition is still available. It was probably needed because
you had removed all the other files from testsuite.at.
> +
> +##################################################
> +
> +AT_BANNER([[Java Push Parsing Tests]])
> +
> +# Define a single copy of the trivial parser
> +m4_define([AT_TRIVIAL_PARSER],[
> +%define parser_class_name "YYParser"
> +%error-verbose
> +
> +%code imports {
> +import java.io.*;
> +import java.util.*;
> +}
> +
> +%%
> +
> +start: 'a' 'b' 'c' ;
> +
> +%%
> +
> +public class Main
> +{
> +
> + static class YYerror implements YYParser.Lexer
> + {
> + public Object getLVal() {return null;}
> + public int yylex () throws java.io.IOException {return 0;}
> + public void yyerror (String msg)
> + {
> + System.err.println(msg);
> + }
> + }
> +
> +
> + static public void main (String[[]] argv) throws IOException
> + {
> + YYerror yyerror = new YYerror();
> + YYParser parser;
> + int state = -1;
> +
> + parser = new YYParser(yyerror);
> + parser.setDebugLevel(1);
> +
> + state = parser.push_parse('a', null);
> + check(state,YYParser.YYMORE,"push_parse('a', null)");
> +
> + parser = new YYParser(yyerror);
> +
> + state = parser.push_parse('a', null);
> + check(state,YYParser.YYMORE,"push_parse('a', null)");
> + state = parser.push_parse('b', null);
> + check(state,YYParser.YYMORE,"push_parse('b', null)");
> + state = parser.push_parse('c', null);
> + check(state,YYParser.YYMORE,"push_parse('c', null)");
> + state = parser.push_parse('\0', null);
> + check(state,YYParser.YYACCEPT,"push_parse('\\0', null)");
> +
> + /* Reuse the parser instance and cause a failure */
> + state = parser.push_parse('b', null);
> + check(state,YYParser.YYABORT,"push_parse('b', null)");
> + System.exit(0);
> + }
> +
> + static String[[]] statename = new
> String[[]]{"YYACCEPT","YYABORT","YYERROR","UNKNOWN","YYMORE"};
> +
> + static void check(int state, int expected, String msg)
> + {
> + System.err.println("state="+statename[[state]]+";
> expected="+statename[[expected]]);
Too wide, please wrap.
> + if (state == expected)
> + return;
> + System.err.println("unexpected state: "+msg);
> + System.exit(1);
> + }
> +
> +}
> +])
> +
> +AT_SETUP([Trivial Push Parser with api.push-pull verification])
> +AT_BISON_OPTION_PUSHDEFS
> +AT_DATA([[input.y]],[[%language "Java"
> +]AT_TRIVIAL_PARSER[
> +]])
> +AT_BISON_OPTION_POPDEFS
This test looks great!
> +
> +# Verify that the proper procedure(s) are generated for each case
> +AT_BISON_CHECK([[-Dapi.push-pull=pull -o Main.java input.y]])
> +AT_CHECK_JAVA_GREP([[Main.java]],[[.*public boolean parse ().*]],[1])
> +AT_CHECK_JAVA_GREP([[Main.java]],[[.*public int push_parse (int yylextoken,
> Object yylexval).*]],[0])
> +AT_BISON_CHECK([[-Dapi.push-pull=both -o Main.java input.y]])
> +AT_CHECK_JAVA_GREP([[Main.java]],[[.*public boolean parse ().*]],[1])
> +AT_CHECK_JAVA_GREP([[Main.java]],[[.*public int push_parse (int yylextoken,
> Object yylexval).*]],[1])
> +AT_BISON_CHECK([[-Dapi.push-pull=push -o Main.java input.y]])
> +AT_CHECK_JAVA_GREP([[Main.java]],[[.*public boolean parse ().*]],[0])
> +AT_CHECK_JAVA_GREP([[Main.java]],[[.*public int push_parse (int yylextoken,
> Object yylexval).*]],[1])
Please, wrap your lines. Something like:
AT_CHECK_JAVA_GREP([[Main.java]],
[[.*public int push_parse (int yylextoken, Object yylexval).*]],
[1])
for instance.
> +
> +AT_JAVA_COMPILE([[Main.java]])
> +AT_JAVA_PARSER_CHECK([Main], 0, [], [stderr-nolog])
> +AT_CLEANUP
> +
> +AT_SETUP([Trivial Push Parser with %initial-action])
> +AT_BISON_OPTION_PUSHDEFS
> +AT_DATA([[input.y]],[[%language "Java"
> +%initial-action {
> +final int INITIAL_ACTION_DECLARED = 1;
Could you use a counter increment here, to make sure it is
run only once? Or some other technique.
> +}
> +]AT_TRIVIAL_PARSER[
> +]])
> +AT_BISON_OPTION_POPDEFS
> +AT_BISON_CHECK([[-Dapi.push-pull=push -o Main.java input.y]])
> +AT_CHECK_JAVA_GREP([[Main.java]],[[final int INITIAL_ACTION_DECLARED = 1;]])
> +AT_JAVA_COMPILE([[Main.java]])
> +AT_JAVA_PARSER_CHECK([Main], 0, [], [stderr-nolog])
> +AT_CLEANUP
> +
> +# Define a single copy of the Calculator
> +m4_define([AT_CALC_BODY1],[
> +/* Bison Declarations */
> +%token <Integer> NUM "number"
> +%type <Integer> exp
> +
> +%nonassoc '=' /* comparison */
> +%left '-' '+'
> +%left '*' '/'
> +%left NEG /* negation--unary minus */
> +%right '^' /* exponentiation */
> +
> +/* Grammar follows */
> +%%
> +input:
> + line
> +| input line
> +;
> +
> +line:
> + '\n'
> +| exp '\n'
> +| error '\n'
> +;
> +
> +exp:
> + NUM { $[]$ = $[]1;}
> +| exp '=' exp
> + {
> + if ($[]1.intValue() != $[]3.intValue())
> + yyerror (]AT_LOCATION_IF(address@hidden,]])[ "calc: error: " + $[]1 +
> " != " + $[]3);
> + }
> +| exp '+' exp { $[]$ = new Integer ($[]1.intValue () + $[]3.intValue
> ()); }
> +| exp '-' exp { $[]$ = new Integer ($[]1.intValue () - $[]3.intValue
> ()); }
> +| exp '*' exp { $[]$ = new Integer ($[]1.intValue () * $[]3.intValue
> ()); }
> +| exp '/' exp { $[]$ = new Integer ($[]1.intValue () / $[]3.intValue
> ()); }
> +| '-' exp %prec NEG { $[]$ = new Integer (-$[]2.intValue ());
> }
> +| exp '^' exp { $[]$ = new Integer ((int)
> + Math.pow ($[]1.intValue (),
> + $[]3.intValue ()));
> }
> +| '(' exp ')' { $[]$ = $[]2;
> }
> +| '(' error ')' { $[]$ = new Integer (1111);}
> +| '!' { $[]$ = new Integer (0); return YYERROR;}
> +| '-' error { $[]$ = new Integer (0); return YYERROR;}
> +;
> +])
Is there a means to factor this calculator from another test?
> +
> +AT_SETUP([Calc parser with api.push-pull both])
> +AT_BISON_OPTION_PUSHDEFS
> +AT_DATA([Calc.y],[[/* Infix notation calculator--calc */
> +%language "Java"
> +%name-prefix "Calc"
> +%define parser_class_name "Calc"
> +
> +%code imports {
> + import java.io.*;
> +}
> +
> +%code {
> +
> + public static void main (String[] argv)
> + throws IOException
> + {
> + StringReader reader = getinput(argv[0]);
> + UserLexer lexer = new UserLexer(reader);
> + Calc calc = new Calc(lexer);
> + calc.setDebugLevel(1);
> + calc.parse();
> + }
> +
> +
> + static StringReader
> + getinput(String filename) throws IOException
> + {
> + StringBuilder buf = new StringBuilder();
> + FileReader file = new FileReader(filename);
> + int c;
> + while((c=file.read()) > 0) {
> + buf.append((char)c);
> + }
> + file.close();
> + return new StringReader(buf.toString());
> + }
> +
> +static class UserLexer implements Calc.Lexer
> +{
> + StreamTokenizer st;
> + StringReader rdr;
> +
> + public UserLexer(StringReader reader)
> + {
> + rdr = reader;
> + st = new StreamTokenizer(rdr);
> + st.resetSyntax();
> + st.eolIsSignificant(true);
> + st.whitespaceChars(9, 9);
> + st.whitespaceChars(32, 32);
> + st.wordChars(48, 57);
> + }
> +
> + Integer yylval;
> +
> + public Object getLVal() {
> + return yylval;
> + }
> +
> + public void yyerror(String msg)
> + {
> + System.err.println(msg);
> + }
> +
> + public int yylex () throws IOException {
{ on the next line.
> + int ttype = st.nextToken ();
> + if (ttype == st.TT_EOF)
> + return EOF;
> +
> + else if (ttype == st.TT_EOL)
> + {
> + return (int) '\n';
> + }
> +
> + else if (ttype == st.TT_WORD)
> + {
> + yylval = new Integer (st.sval);
> + return NUM;
> + }
> +
> + else
> + return st.ttype;
> + }
> +}
> +
> +}
> +
> +]AT_CALC_BODY1[
> +
> +]])
> +
> +AT_DATA([[input]],[[1 + 2 * 3 = 7
> +1 + 2 * -3 = -5
> +
> +-1^2 = -1
> +(-1)^2 = 1
> +
> +---1 = -1
> +
> +1 - 2 - 3 = -4
> +1 - (2 - 3) = 2
> +
> +2^2^3 = 256
> +(2^2)^3 = 64
> +]])
> +
> +AT_DATA([[expout]],[[Starting parse
> +
> +Entering state 0
> +
> +Stack now 0
> +Reading a token: @&t@
> +Next token is token "number" (1)
> +Shifting token "number" (1)
> +Entering state 2
This log is way too long, I doubt it is really that useful
to store it as it. Why did you include it? What kind of
failure are you trying to capture? Can't this be done by
filtering a few well selected line of the trace, instead of
everything?
> +Stack now 0 7 15
> +]])
> +
> +AT_BISON_OPTION_POPDEFS
> +
> +AT_BISON_CHECK([PUSHPULLFLAGS [-o Calc.java Calc.y]])
> +AT_JAVA_COMPILE([[Calc.java]])
> +# Verify that this is a push parser
> +AT_CHECK_JAVA_GREP([[Calc.java]],[[.*public void push_parse_initialize().*]])
> +# Capture the output so we can edit out the (line nnnn) occurrences
> +AT_JAVA_PARSER_CHECK([Calc input], 0, [], [stderr-nolog])
> +AT_CHECK([[sed -e 's/(line[ ][0-9][0-9]*)[,]/,/' stderr]],[ignore],[expout])
You don't seem to be checking anything at all here.
> +
> +AT_CLEANUP
> +
> +AT_SETUP([Calc parser with %locations %code lexer and api.push-pull both])
> +AT_BISON_OPTION_PUSHDEFS
> +AT_DATA([[Calc.y]],[[/* Infix notation calculator--calc */
> +%language "Java"
> +%name-prefix "Calc"
> +%define parser_class_name "Calc"
> +%lex-param { Reader rdr}
> +%locations
> +
> +%code imports {
> + import java.io.*;
> +}
> +
> +%code lexer {
> + StreamTokenizer st;
> +
> + public YYLexer(Reader rdr)
> + {
> + st = new StreamTokenizer(rdr);
> + st.resetSyntax();
> + st.eolIsSignificant(true);
> + st.whitespaceChars(9, 9);
> + st.whitespaceChars(32, 32);
> + st.wordChars(48, 57);
> + }
> +
> + Position yypos = new Position (1, 0);
> +
> + public Position getStartPos() {
> + return yypos;
> + }
> +
> + public Position getEndPos() {
> + return yypos;
> + }
> +
> + Integer yylval;
> +
> + public Object getLVal() {
> + return yylval;
> + }
> +
> + public void yyerror(Location loc, String msg)
> + {
> + System.err.println(loc+":"+msg);
> + }
> +
> + public int yylex () throws IOException {
{ alone
> + int ttype = st.nextToken ();
> + yypos = new Position (yypos.lineno (),yypos.token () + 1);
> + if (ttype == st.TT_EOF)
> + return EOF;
> +
> + else if (ttype == st.TT_EOL)
> + {
> + yypos = new Position (yypos.lineno () + 1, 0);
> + return (int) '\n';
> + }
> +
> + else if (ttype == st.TT_WORD)
> + {
> + yylval = new Integer (st.sval);
> + return NUM;
> + }
> +
> + else
> + return st.ttype;
There seems to be quite some duplication of test code here.
Can you factor it?
> + }
> +}
> +
> +%code {
> +class Position {
> + public int line;
> + public int token;
> +
> + public Position ()
> + {
> + line = 0;
> + token = 0;
> + }
> +
> + public Position (int l, int t)
> + {
> + line = l;
> + token = t;
> + }
> +
> + public boolean equals (Position l)
> + {
> + return l.line == line && l.token == token;
> + }
> +
> + public String toString ()
> + {
> + return Integer.toString(line) + "." + Integer.toString(token);
> + }
> +
> + public int lineno ()
> + {
> + return line;
> + }
> +
> + public int token ()
> + {
> + return token;
> + }
> +}
> +
> +public static void main (String[] argv)
> + throws IOException
> +{
> + StringReader reader = getinput(argv[0]);
> + Calc calc = new Calc(reader);
> + calc.setDebugLevel(1);
> + calc.parse();
> +}
> +
> +static StringReader
> +getinput(String filename)
> + throws IOException
> +{
> + StringBuilder buf = new StringBuilder();
> + FileReader file = new FileReader(filename);
> + int c;
> + while((c=file.read()) > 0) {
> + buf.append((char)c);
> + }
Space after while, { alone (or none in this case).
> + file.close();
> + return new StringReader(buf.toString());
> +}
> +}
> +
> +]AT_CALC_BODY1[
> +
> +]])
> +AT_BISON_OPTION_POPDEFS
> +
> +AT_DATA([[input]],[[1 + 2 * 3 = 7
> +1 + 2 * -3 = -5
> +
> +-1^2 = -1
> +(-1)^2 = 1
> +
> +---1 = -1
> +
> +1 - 2 - 3 = -4
> +1 - (2 - 3) = 2
> +
> +2^2^3 = 256
> +(2^2)^3 = 64
> +]])
> +
> +AT_DATA([[expout]],[[Starting parse
> +
> +Entering state 0
> +
> +Stack now 0
> +Reading a token: @&t@
> +Next token is token "number" (1.1: 1)
> +Shifting token "number" (1.1: 1)
> +Entering state 2
Again, this is way too long.
> +-> $$ = nterm input (1.1-14.0: 7)
> +Entering state 7
> +
> +Stack now 0 7
> +Reading a token: @&t@
> +Now at end of input.
> +
> +Shifting token $end (14.1: 64)
> +Entering state 15
> +
> +Stack now 0 7 15
> +]])
> +
> +AT_BISON_CHECK([PUSHPULLFLAGS [-o Calc.java Calc.y]])
> +AT_JAVA_COMPILE([[Calc.java]])
> +# Verify that this is a push parser
> +AT_CHECK_JAVA_GREP([[Calc.java]],[[.*public void push_parse_initialize().*]])
> +# Capture the output so we can edit out the (line nnnn) occurrences
> +AT_JAVA_PARSER_CHECK([Calc input], 0, [], [stderr-nolog])
> +AT_CHECK([[sed -e 's/(line[ ][0-9][0-9]*)[,]/,/' stderr]],[ignore],[expout])
> +AT_CLEANUP
> +
> +AT_SETUP([Calc parser with api.push-pull both and %define extends])
> +AT_BISON_OPTION_PUSHDEFS
> +AT_DATA([[Calc.y]],[[/* Infix notation calculator--calc */
> +%language "Java"
> +%name-prefix "Calc"
> +%define parser_class_name "Calc"
> +%define extends "CalcActions"
> +
> +%code init {
> + super();
> +}
> +
> +%code imports {
> + import java.io.*;
> +}
> +
> +%code {
> +
> + /* Make yylexer visible */
> + Lexer getLexer() {return yylexer;}
> +
> + public static void main (String[] argv)
> + throws IOException
> + {
> + StringReader reader = getinput(argv[0]);
> + UserLexer lexer = new UserLexer(reader);
> + Calc calc = new Calc(lexer);
> + calc.setDebugLevel(1);
> + calc.parse();
> + }
> +
> +
> + static StringReader
> + getinput(String filename) throws IOException
> + {
> + StringBuilder buf = new StringBuilder();
> + FileReader file = new FileReader(filename);
> + int c;
> + while((c=file.read()) > 0) {
> + buf.append((char)c);
> + }
> + file.close();
> + return new StringReader(buf.toString());
> + }
> +}
> +
> +/* Bison Declarations */
> +%token <Integer> NUM "number"
> +%type <Integer> exp
> +
> +%nonassoc '=' /* comparison */
> +%left '-' '+'
> +%left '*' '/'
> +%left NEG /* negation--unary minus */
> +%right '^' /* exponentiation */
> +
> +/* Grammar follows */
> +%%
> +input:
> + line
> +| input line
> +;
> +
> +line:
> + '\n'
> +| exp '\n'
> +| error '\n'
> +;
> +
> +exp:
> + NUM { $$ = $1;}
> +| exp '=' exp {$$ = action_eq($1,$3);}
> +| exp '+' exp {$$ = action_add($1,$3);}
> +| exp '-' exp {$$ = action_sub($1,$3);}
> +| exp '*' exp {$$ = action_mult($1,$3);}
> +| exp '/' exp {$$ = action_div($1,$3);}
> +| '-' exp %prec NEG {$$ = action_minus($2);}
> +| exp '^' exp {$$ = action_exp($1,$3);}
> +| '(' exp ')' {$$ = action_ident($2);}
> +| '(' error ')' {$$ = action_error1();}
> +| '!' {$$ = action_fail(); return YYERROR;}
> +| '-' error {$$ = action_error2(); return YYERROR;}
> +;
> +
> +%%
> +
> +class UserLexer implements Calc.Lexer
> +{
> + StreamTokenizer st;
> + StringReader rdr;
> +
> + public UserLexer(StringReader reader)
> + {
> + rdr = reader;
> + st = new StreamTokenizer(rdr);
> + st.resetSyntax();
> + st.eolIsSignificant(true);
> + st.whitespaceChars(9, 9);
> + st.whitespaceChars(32, 32);
> + st.wordChars(48, 57);
> + }
> +
> + Integer yylval;
> +
> + public Object getLVal() {return yylval;}
> +
> + public void yyerror(String msg) {System.err.println(msg);}
> +
> + public int yylex () throws IOException
> + {
> + int ttype = st.nextToken ();
> + if (ttype == st.TT_EOF) {return EOF;}
Please improve the indentation. How about using a switch?
> + else if (ttype == st.TT_EOL) {
> + return (int) '\n';
> + } else if (ttype == st.TT_WORD) {
> + yylval = new Integer (st.sval);
> + return NUM;
> + } else
> + return st.ttype;
> + }
> +}
> +
> +abstract class CalcActions
> +{
> + CalcActions() {};
> +
> + abstract Calc.Lexer getLexer();
> +
> + Integer
> + action_eq(Integer i1, Integer i2)
> + {
> + if (i1.intValue() != i2.intValue())
> + getLexer().yyerror (]AT_LOCATION_IF(address@hidden,])[ "calc: error: "
> + i1 + " != " + i2);
> + return i1; /* default */
> + }
> +
> + Integer
> + action_sub(Integer i1, Integer i2)
> + {return new Integer (i1.intValue () - i2.intValue ());}
> +
> + Integer
> + action_add(Integer i1, Integer i2)
> + {return new Integer (i1.intValue () + i2.intValue ());}
> +
> + Integer
> + action_mult(Integer i1, Integer i2)
> + {return new Integer (i1.intValue () * i2.intValue ());}
> +
> + Integer
> + action_div(Integer i1, Integer i2)
> + {return new Integer (i1.intValue () / i2.intValue ());}
> +
> + Integer
> + action_minus(Integer i1)
> + {return new Integer (- i1.intValue ());}
> +
> + Integer
> + action_exp(Integer i1, Integer i2)
> + {return new Integer ((int)Math.pow(i1.intValue(),i2.intValue ()));}
> +
> + Integer
> + action_ident(Integer i1)
> + {return i1;}
> +
> + Integer
> + action_error1()
> + {return new Integer(1111);}
> +
> + Integer
> + action_fail()
> + {return new Integer(0);}
> +
> + Integer
> + action_error2()
> + {return new Integer(0);}
> +}
> +]])
> +
> +AT_BISON_OPTION_POPDEFS
> +
> +AT_DATA([[input]],[[1 + 2 * 3 = 7
> +1 + 2 * -3 = -5
> +
> +-1^2 = -1
> +(-1)^2 = 1
> +
> +---1 = -1
> +
> +1 - 2 - 3 = -4
> +1 - (2 - 3) = 2
> +
> +2^2^3 = 256
> +(2^2)^3 = 64
> +]])
> +
> +AT_DATA([[expout]],[[Starting parse
> +
> +Entering state 0
> +
> +Stack now 0
> +Reading a token: @&t@
again :)
> +Entering state 15
> +
> +Stack now 0 7 15
> +]])
> +
> +AT_BISON_CHECK([PUSHPULLFLAGS [-o Calc.java Calc.y]])
> +AT_JAVA_COMPILE([[Calc.java]])
> +# Verify that this is a push parser
> +AT_CHECK_JAVA_GREP([[Calc.java]],[[.*public void push_parse_initialize().*]])
> +# Capture the output so we can edit out the (line nnnn) occurrences
> +AT_JAVA_PARSER_CHECK([Calc input], 0, [], [stderr-nolog])
> +AT_CHECK([[sed -e 's/(line[ ][0-9][0-9]*)[,]/,/' stderr]],[ignore],[expout])
These bits seem useless.
> +AT_CLEANUP
> diff --git a/tests/local.mk b/tests/local.mk
> index 7bc8b78..b27b96c 100644
> --- a/tests/local.mk
> +++ b/tests/local.mk
> @@ -53,6 +53,7 @@ TESTSUITE_AT = \
> tests/headers.at \
> tests/input.at \
> tests/java.at \
> + tests/javapush.at \
> tests/local.at \
> tests/named-refs.at \
> tests/output.at \
> diff --git a/tests/testsuite.at b/tests/testsuite.at
> index f11866b..7f56578 100644
> --- a/tests/testsuite.at
> +++ b/tests/testsuite.at
> @@ -1,76 +1 @@
> -# Test suite for GNU Bison. -*- Autotest -*-
> -
> -# Copyright (C) 2000-2004, 2006-2007, 2009-2013 Free Software
> -# Foundation, Inc.
> -
I have fixed these.
Thanks Dennis!