Re: Weird issue with dotted-list-notation

[Urs Liska]

Hi David,

thank you for taking the time looking into this.

Am 23.03.2015 um 10:19 schrieb David Kastrup:
> Urs Liska <address@hidden> writes:
>
> > Well, I do have a minimal example outlining what I want to do, but
> > "unfortunately" this works exactly as expected:
> >
> > \version  "2.19.16"
> >
> > testDotList =
> > #(define-void-function (parser location opts path)
> >     ((ly:context-mod?) list?)
> >     (for-each
> >      (lambda  (p)
> >        (display  p)
> >        (newline))
> >      path))
> >
> >
> > \testDotList  path.to.whatever
> >
> > \testDotList  \with  {
> >    optionOne = ##t
> >    optionTwo = ##f
> > }
> > path.to.something.else
> Any string that could be a notename will not be parsed as string.  Try
> quoting "path.to.something.else".  Then the predicate "list?" is a
> recipe for trouble particularly in connection with optional arguments.
> Try "symbol-list?" instead.

OK, I see and update this instance and others.
Although - as you write at the end - this doesn't help with the issue at 
hand.

> > As it is the file compiles fine, but when I comment out line 13 (the
> > pointless "ly:message") the compilation results in errors starting
> > with
> >
> > <included string>:1:1 <0>: error: unknown escaped string: `\include'
> That's rather curious.  \include is recognized in the lexer with rule
>
> <INITIAL,chords,lyrics,figures,notes>\\include           {
>          yy_push_state (incl);
> }
>
> It is not even a "reserved word" but hardwired.  So it would appear that
> \\include manages to seep into the lexer in a state different from the
> above list.  Either that, or your "\include" is not "\include" but
> contains some funny Unicode characters, like a non-breaking non-space
> (or byte order mark) or something else.  I rather suspect the state,
> however.
>
> > If I prepend the music expression with a \relative clause I get yet
> > other errors, starting with
> >
> > openlilylib/ly/scholarly/diplomatic-line-breaks.ily:3:1 <0>: error:
> > syntax error, unexpected \header, expecting (backed-up?)
> (backed-up?) is an internal token used by the parser.  This message
> means that you managed to confuse the parser to a degree where its
> internals leak out.
>
> So you managed confusing _both_ the parser _and_ the lexer beyond their
> expected scope.
>
> > So the function behaves (breaks) differently depending on what comes
> > after it. If I uncomment the ly:message line or insert e.g.  a
> > #(define foo 'bar) between \useModule and the music expression or the
> > markup everything works as expected again.
> Try calling LilyPond with -ddebug-parser and redirecting the rather
> voluminous output somewhere.  It's probably the last few screens before
> the error occurs that are relevant.

At the end I attache the relevant (?) portions of a run that fails and a 
run that works. I can see the difference when the lookahead touches the 
\markup (= failing) and the #(display ... (= working) but I can't make 
any sense of it because I can't really read this debug output.


Debug output from working run:
===================

Stack now 0
Entering state 2
Now at end of input.
Shifting token "end of input" (: )
Entering state 5
Stack now 0 2 5
Cleanup: popping token "end of input" (: )
Cleanup: popping nterm start_symbol (: )
-> $$ = nterm scm_function_call (: )
Stack now 0 1 4
Entering state 109
Reducing stack by rule 36 (line 513):
   $1 = nterm scm_function_call (: )
-> $$ = nterm embedded_scm (: )
Stack now 0 1 4
Entering state 108
Reading a token: Now at end of input.
Reducing stack by rule 72 (line 681):
   $1 = nterm embedded_scm (: )
-> $$ = nterm identifier_init_nonumber (: )
Stack now 0 1 4
Entering state 112
Reducing stack by rule 45 (line 559):
   $1 = nterm identifier_init_nonumber (: )
-> $$ = nterm embedded_lilypond (: )
Stack now 0 1 4
Entering state 111
Reducing stack by rule 3 (line 372):
   $1 = token "#{" (: )
   $2 = nterm address@hidden (: )
   $3 = nterm embedded_lilypond (: )
-> $$ = nterm start_symbol (: )
Stack now 0
Entering state 2
Now at end of input.
Shifting token "end of input" (: )
Entering state 5
Stack now 0 2 5
Cleanup: popping token "end of input" (: )
Cleanup: popping nterm start_symbol (: )
-> $$ = nterm scm_function_call (: )
Stack now 0 1 4
Entering state 109
Reducing stack by rule 36 (line 513):
   $1 = nterm scm_function_call (: )
-> $$ = nterm embedded_scm (: )
Stack now 0 1 4
Entering state 108
Now at end of input.
Reducing stack by rule 72 (line 681):
   $1 = nterm embedded_scm (: )
-> $$ = nterm identifier_init_nonumber (: )
Stack now 0 1 4
Entering state 112
Reducing stack by rule 45 (line 559):
   $1 = nterm identifier_init_nonumber (: )
-> $$ = nterm embedded_lilypond (: )
Stack now 0 1 4
Entering state 111
Reducing stack by rule 3 (line 372):
   $1 = token "#{" (: )
   $2 = nterm address@hidden (: )
   $3 = nterm embedded_lilypond (: )
-> $$ = nterm start_symbol (: )
Stack now 0
Entering state 2
Now at end of input.
Shifting token "end of input" (: )
Entering state 5
Stack now 0 2 5
Cleanup: popping token "end of input" (: )
Cleanup: popping nterm start_symbol (: )

\include 
"/home/uliska/git/openlilylib/openlilylib/ly/scholarly/diplomatic-line-breaks.ily"-> 
$$ = nterm scm_function_call (: )
Stack now 0 3
Entering state 43
Reducing stack by rule 24 (line 490):
   $1 = nterm scm_function_call (: )
-> $$ = nterm embedded_scm_active (: )
Stack now 0 3
Entering state 42
Reducing stack by rule 19 (line 436):
   $1 = nterm embedded_scm_active (: )
-> $$ = nterm toplevel_expression (: )
Stack now 0 3
Entering state 40
Reducing stack by rule 5 (line 382):
   $1 = nterm lilypond (: )
   $2 = nterm toplevel_expression (: )
-> $$ = nterm lilypond (: )
Stack now 0
Entering state 3
Next token is token SCM_TOKEN (: #'(ly:message "loaded"))
Shifting token SCM_TOKEN (: #'(ly:message "loaded"))
Entering state 37
Reducing stack by rule 18 (line 432):
   $1 = token SCM_TOKEN (: #'(ly:message "loaded"))

loaded-> $$ = nterm toplevel_expression (: )
Stack now 0 3
Entering state 40
Reducing stack by rule 5 (line 382):
   $1 = nterm lilypond (: )
   $2 = nterm toplevel_expression (: )
-> $$ = nterm lilypond (: )
Stack now 0
Entering state 3
Reading a token: Next token is token "\\header" (: )
Reducing stack by rule 9 (line 396):
-> $$ = nterm address@hidden (: )
Stack now 0 3
Entering state 41



Debug output from failing run:
==================

Stack now 0
Entering state 2
Now at end of input.
Shifting token "end of input" (: )
Entering state 5
Stack now 0 2 5
Cleanup: popping token "end of input" (: )
Cleanup: popping nterm start_symbol (: )
-> $$ = nterm scm_function_call (: )
Stack now 0 1 4
Entering state 109
Reducing stack by rule 36 (line 513):
   $1 = nterm scm_function_call (: )
-> $$ = nterm embedded_scm (: )
Stack now 0 1 4
Entering state 108
Now at end of input.
Reducing stack by rule 72 (line 681):
   $1 = nterm embedded_scm (: )
-> $$ = nterm identifier_init_nonumber (: )
Stack now 0 1 4
Entering state 112
Reducing stack by rule 45 (line 559):
   $1 = nterm identifier_init_nonumber (: )
-> $$ = nterm embedded_lilypond (: )
Stack now 0 1 4
Entering state 111
Reducing stack by rule 3 (line 372):
   $1 = token "#{" (: )
   $2 = nterm address@hidden (: )
   $3 = nterm embedded_lilypond (: )
-> $$ = nterm start_symbol (: )
Stack now 0
Entering state 2
Now at end of input.
Shifting token "end of input" (: )
Entering state 5
Stack now 0 2 5
Cleanup: popping token "end of input" (: )
Cleanup: popping nterm start_symbol (: )

\include 
"/home/uliska/git/openlilylib/openlilylib/ly/scholarly/diplomatic-line-breaks.ily"-> 
$$ = nterm scm_function_call (: )
Stack now 0 3
Entering state 43
Reducing stack by rule 24 (line 490):
   $1 = nterm scm_function_call (: )
-> $$ = nterm embedded_scm_active (: )
Stack now 0 3
Entering state 42
Reducing stack by rule 19 (line 436):
   $1 = nterm embedded_scm_active (: )
-> $$ = nterm toplevel_expression (: )
Stack now 0 3
Entering state 40
Reducing stack by rule 5 (line 382):
   $1 = nterm lilypond (: )
   $2 = nterm toplevel_expression (: )
-> $$ = nterm lilypond (: )
Stack now 0
Entering state 3
Next token is token "\\markup" (: )
Shifting token "\\markup" (: )
Entering state 22
Reducing stack by rule 485 (line 3512):
-> $$ = nterm address@hidden (: )
Stack now 0 3 22
Entering state 166
Reading a token:
<included string>:1:1: error: unknown escaped string: `\include'

\include 
"/home/uliska/git/openlilylib/openlilylib/ly/scholarly/diplomatic-line-breaks.ily"
Next token is token STRING (: #"include")
Shifting token STRING (: #"include")
Entering state 330
Reducing stack by rule 513 (line 3646):
   $1 = token STRING (: #"include")
-> $$ = nterm simple_markup (: )
Stack now 0 3 22 166
Entering state 342
Reducing stack by rule 489 (line 3528):
   $1 = nterm simple_markup (: )
-> $$ = nterm markup_top (: )
Stack now 0 3 22 166
Entering state 333
Reducing stack by rule 486 (line 3511):
   $1 = token "\\markup" (: )
   $2 = nterm address@hidden (: )
   $3 = nterm markup_top (: )
-> $$ = nterm full_markup (: )
Stack now 0 3
Entering state 68
Reducing stack by rule 16 (line 424):
   $1 = nterm full_markup (: )
-> $$ = nterm toplevel_expression (: )
Stack now 0 3
Entering state 40
Reducing stack by rule 5 (line 382):
   $1 = nterm lilypond (: )
   $2 = nterm toplevel_expression (: )
-> $$ = nterm lilypond (: )
Stack now 0
Entering state 3
Reading a token: Next token is token STRING (: 
#"/home/uliska/git/openlilylib/openlilylib/ly/scholarly/diplomatic-line-breaks.ily")
Shifting token STRING (: 
#"/home/uliska/git/openlilylib/openlilylib/ly/scholarly/diplomatic-line-breaks.ily")
Entering state 38
Reducing stack by rule 56 (line 623):
   $1 = token STRING (: 
#"/home/uliska/git/openlilylib/openlilylib/ly/scholarly/diplomatic-line-breaks.ily")
-> $$ = nterm assignment_id (: )
Stack now 0 3
Entering state 44
Reading a token: 
/home/uliska/git/openlilylib/openlilylib/ly/scholarly/usage-examples/diplomatic-line-breaks.ly:15:9: 
error: unknown escaped string: `\vspace'
\markup
        \vspace #1
Next token is token STRING (: #"vspace")
Shifting token STRING (: #"vspace")
Entering state 185
Reducing stack by rule 203 (line 1596):
   $1 = token STRING (: #"vspace")
-> $$ = nterm symbol_list_element (: )
Stack now 0 3 44
Entering state 191
Reducing stack by rule 202 (line 1582):
   $1 = nterm symbol_list_element (: )
-> $$ = nterm symbol_list_part (: )
Stack now 0 3 44
Entering state 190
Reducing stack by rule 200 (line 1572):
   $1 = nterm symbol_list_part (: )
-> $$ = nterm symbol_list_rev (: )
Stack now 0 3 44
Entering state 189
Reading a token: Next token is token SCM_TOKEN (: #1)
Shifting token SCM_TOKEN (: #1)
Entering state 103
Reducing stack by rule 21 (line 481):
   $1 = token SCM_TOKEN (: #1)
-> $$ = nterm embedded_scm_bare (: )
Stack now 0 3 44 189
Entering state 188
Reducing stack by rule 204 (line 1597):
   $1 = nterm embedded_scm_bare (: )
-> $$ = nterm symbol_list_element (: )
Stack now 0 3 44 189
Entering state 191
Reducing stack by rule 202 (line 1582):
   $1 = nterm symbol_list_element (: )
/home/uliska/git/openlilylib/openlilylib/ly/scholarly/usage-examples/diplomatic-line-breaks.ly:15:17: 
error: not a symbol
\markup \vspace
                #1
-> $$ = nterm symbol_list_part (: )
Stack now 0 3 44 189
Entering state 190
Reducing stack by rule 200 (line 1572):
   $1 = nterm symbol_list_part (: )
-> $$ = nterm symbol_list_rev (: )
Stack now 0 3 44 189
Entering state 189
Reading a token: Next token is token '{' (: )
Reducing stack by rule 291 (line 2320):
   $1 = nterm symbol_list_rev (: )
-> $$ = nterm property_path (: )
Stack now 0 3 44 189