Re: [Flex-help] How to debug bison/flex program? usage of yyerror()

[Marcel Laverdet]

You don't change the BNF. The idea is that it's an unexpected token and
should never come up. If it does come up you've got a problem.

Really you're best off making your scanner be able to handle ALL input no
matter what.

On Wed, 30 Dec 2009 21:52:46 -0600, Peng Yu <address@hidden> wrote:
> On Wed, Dec 30, 2009 at 1:05 PM, Marcel Laverdet <address@hidden>
> wrote:
>>
>>
>>> Do you actually mean '.' matches non newline?
>>
>> Yes, apologies :)
>>
>>> I made some corrections. Now yylval_string.l becomes the following.
You
>> said yyerror should not be used in the flex file. I'm wondering what I
>> should use to replace the line '.       { yyerror("mystery character
>> %c\n",
>> *yytext); }' in order to catch errors.
>>
>> Really, it's up to you. You can replace it with whatever you want. You
>> could replace it with assert(false) if you want or fprintf(stderr, "oh
>> no!"), I don't know what you want it's your program. It's just a matter
>> of
>> what you want to do when something unexpected happens. Bison will
>> generate
>> error messages for you when something bad happens, but Flex is not in a
>> position to generate error messages. In my scanners I usually do one of
>> two
>> things. I either memoize an error message in my yyextra (or just as a
>> global if you're not using "%option reentrant"), which I can then raise
>> to
>> the client of my scanner/parser. Or, I do something like "return
>> UNEXPECTED_CHARACTER;" (you would have to define this token). Then,
Bison
>> would generate an error such as "Error: Unexpected token
>> UNEXPECTED_CHARACTER on line 5".
> 
> If I return UNEXPECTED_CHARACTER in yylval_string.l, how to change the
> BNF in yylval_string.y?
> 
>>yylval_string.l
> %option nodefault
> %{
> # include "yylval_string.tab.h"
> %}
> %%
> [0-9]+ { yylval=atoi(yytext); return NUMBER; }
> [[:space:]] { /*SPACE*/ }
> .     { return UNEXPECED_CHARACTER; }
> %%
>>yylval_string.y
> %{
> #  include <stdio.h>
> %}
> %token NUMBER UNEXPECED_CHARACTER
> %%
> numbers:
>        NUMBER { printf("NUMBER = %d\n", $1); }
>        | numbers NUMBER { printf("NUMBER = %d\n", $2); }
>  ;
> %%
> main()
> {
>   yyparse();
> }
> 
> yyerror(char *s)
> {
>   fprintf(stderr, "error: %s\n", s);
> }
> 
> 
>> On Wed, 30 Dec 2009 09:39:30 -0600, Peng Yu <address@hidden>
wrote:
>>> On Wed, Dec 30, 2009 at 12:54 AM, Marcel Laverdet
<address@hidden>
>>> wrote:
>>>>
>>>>
>>>> 1) [:space:] is a character class expression. If you want one or more
>>>> spaces you would do [[:space:]]+. What your scanner is looking for
>> right
>>>> now is one of either ":, s, p, a, c, or e". Does that make sense?
Just
>>>> wrap
>>>> it in another set of []'s
>>>>
>>>> 2) . only matches newline, the documentation is not lying :). The
>> message
>>>> you're seeing is from the spaces, since your [:space:] does not match
>>>> them
>>>> correctly. Additionally, you're hitting the default rule, whose
default
>>>> action is to print the matching character. Pretend that this always
>>>> exists
>>>> at the bottom of your file:
>>>
>>> Do you actually mean '.' matches non newline?
>>>
>>>> <*>.|\n fprintf(strerr, "%c", *yytext);
>>>>
>>>> You can (and should) override this using "%option nodefault" which
will
>>>> make it so that your scanner fails to build if you don't handle all
>>>> inputs.
>>>> This is a much more reasonable behavior than just spitting out to
>> strerr.
>>>>
>>>> 3) yyerror is a macro defined by bison, not flex. There should be no
>>>> yyerror in your lex file. bison's definition of yyerror is not ("%s",
>>>> ...)
>>>> it's something different (which depends on your options to bison).
>>>
>>> I made some corrections. Now yylval_string.l becomes the following.
>>> You said yyerror should not be used in the flex file. I'm wondering
>>> what I should use to replace the line '.      { yyerror("mystery
>>> character
>>> %c\n", *yytext); }' in order to catch errors.
>>>
>>> %option nodefault
>>> %{
>>> # include "yylval_string.tab.h"
>>> %}
>>> %%
>>> [0-9]+ { yylval=atoi(yytext); return NUMBER; }
>>> [[:space:]] { /*SPACE*/ }
>>> .     { yyerror("mystery character %c\n", *yytext); }
>>> %%
>>>
>>>
>>>
>>>> On Tue, 29 Dec 2009 23:02:33 -0600, Peng Yu <address@hidden>
>> wrote:
>>>>> I have the source files listed at the end of the message. I
basically
>>>>> want to parse a file with only numbers (separated by spaces) and
print
>>>>> the numbers out. It is an overkill to use bison/flex. But I just
want
>>>>> to try how to use bison/flex.
>>>>>
>>>>> I need to understand how to debug the program. Could somebody help
me
>>>>> with the following three questions?
>>>>>
>>>>> 1. I don't understand why the error message is printed. Shouldn't
the
>>>>> regexes [0-9]+ and [:space:] match all the strings in 'test.txt'.
>>>>>
>>>>> 2. I suspect that '.' is matched to EOF. I'm not sure if I'm
correct.
>>>>> But it seem that EOF can not be printed (is it why '%c' is printed
>>>>> literally?).
>>>>>
>>>>> 3. Why yyerror() in the .l file has two arguments but it has one
>>>>> argument in the .y file? Are they the same function or two different
>>>>> functions?
>>>>>
>>>>> $ make
>>>>> bison -d yylval_string.y
>>>>> flex yylval_string.l
>>>>> cc -o yylval_string yylval_string.tab.c lex.yy.c -lfl
>>>>> $./yylval_string< test.txt
>>>>> NUMBER = 133
>>>>> error: mystery character %c
>>>>>
>>>>> NUMBER = 7
>>>>> error: mystery character %c
>>>>>
>>>>> NUMBER = 33
>>>>> error: mystery character %c
>>>>>
>>>>> NUMBER = 76
>>>>>
>>>>> NUMBER = 35
>>>>>
>>>>>
>>>>> --------------------------------source files listed
>>>>> below---------------------------
>>>>>
>>>>> $cat yylval_string.y %{
>>>>> #  include <stdio.h>
>>>>> %}
>>>>>
>>>>> %token NUMBER
>>>>>
>>>>> %%
>>>>>
>>>>> numbers:
>>>>>        NUMBER { printf("NUMBER = %d\n", $1); }
>>>>>        | numbers NUMBER { printf("NUMBER = %d\n", $2); }
>>>>>  ;
>>>>>
>>>>> %%
>>>>> main()
>>>>> {
>>>>>   yyparse();
>>>>> }
>>>>>
>>>>> yyerror(char *s)
>>>>> {
>>>>>   fprintf(stderr, "error: %s\n", s);
>>>>> }
>>>>>
>>>>>
>>>>> $cat yylval_string.l
>>>>> %{
>>>>> # include "yylval_string.tab.h"
>>>>> %}
>>>>>
>>>>> %%
>>>>> [0-9]+ { yylval=atoi(yytext); return NUMBER; }
>>>>> [:space:] { /*SPACE*/ }
>>>>> .     { yyerror("mystery character %c\n", *yytext); }
>>>>> %%
>>>>>
>>>>> $cat Makefile
>>>>>  .PHONY: all
>>>>>
>>>>> all: yylval_string
>>>>>
>>>>> yylval_string:        yylval_string.l yylval_string.y
>>>>>       bison -d yylval_string.y
>>>>>       flex yylval_string.l
>>>>>       cc -o $@ yylval_string.tab.c lex.yy.c -lfl
>>>>>
>>>>> $cat test.txt
>>>>> 133 7 33 76
>>>>> 35
>>>>>
>>>>>
>>>>
>>
------------------------------------------------------------------------------
>>>>> This SF.Net email is sponsored by the Verizon Developer Community
>>>>> Take advantage of Verizon's best-in-class app development support
>>>>> A streamlined, 14 day to market process makes app distribution fast
>> and
>>>>> easy
>>>>> Join now and get one step closer to millions of Verizon customers
>>>>> http://p.sf.net/sfu/verizon-dev2dev
>>>>> _______________________________________________
>>>>> Flex-help mailing list
>>>>> address@hidden
>>>>> https://lists.sourceforge.net/lists/listinfo/flex-help
>>>>
>>
> 
>
------------------------------------------------------------------------------
> This SF.Net email is sponsored by the Verizon Developer Community
> Take advantage of Verizon's best-in-class app development support
> A streamlined, 14 day to market process makes app distribution fast and
> easy
> Join now and get one step closer to millions of Verizon customers
> http://p.sf.net/sfu/verizon-dev2dev 
> _______________________________________________
> Flex-help mailing list
> address@hidden
> https://lists.sourceforge.net/lists/listinfo/flex-help