Re: Changes to lexer invalidate documentation

[John W. Eaton]

On 13-Apr-2009, Rik wrote:

| What I imagined is that the sequence of inputs was parsed and as each
| logical group of commands was divided the result was placed in a queue. 
| After parsing a simple while (!empty) loop grabs the head item from the
| queue and passes it to the evaluation engine.  This architecture has a
| strict division between parsing, which occurs at one time, and
| evaluation, which occurs at a different time.  I was imagining that it
| would be simple to intervene when a single block of commands was
| available at the head of the queue.  Instead of finishing all of the
| parsing the engine would evaluate the first group 'A' and only if that
| succeeded would it move to parse B.
| 
| The more I think about it the more pathological the example in the
| documentation becomes.  First, I find it really rare to enter long
| multiple commands on a single line.  I would normally code a function on
| the command line like so:
| 
| octave:1> functon y = f (x)<CR>
| > y = x^2;<CR>
| > endfunction<CR>
| 
| In this case the first line WOULD be evaluated by itself and produce an
| error before I ever got to lines 2 and 3.  Secondly, it has to be just
| the right sort of parsing error to pass initial parsing checks but fail
| a subsequent one.  Most of the time I have a much simpler typo which the
| parser will catch.  As an example, the current parser correctly catches
| a failure to close a parenthesis and indicates where.
| 
| a = sin(pi; b = cos(pi)<CR>
| parse error:
| 
|   syntax error
| 
| >>> a = sin(pi; b = cos(pi)
|               ^

Yes, but this is a case where the syntax error occurs in the first
expression in the list.

| In view of the rarity of the documentation example (and the fact that
| I'm changing it to something simpler) it is probably not worth bothering
| about.
| > I don't see how we can change that behavior.  Imagine typing
| >
| >   missing_function (); x+++
| >
| > and having your typing interrupted by an error message about the
| > missing function as you started to type "x+++", or typing
| >
| >   y = rand (10), x = 1
| >
| > and getting output (the display of Y) before you finished typing "x = 1".
| >   
| I wasn't thinking of a just-in-time parser that was processing
| character-by-character.  The current parser is engaged by the carriage
| return character and then works on the entire line.  There is no
| intermediate output from one command obscuring another because the
| semicolon is specifically there for silencing output.

But semicolons are not the only command separator.

| The current parser already behaves pretty much the way I was envisioning it.
| 
| x = 0; x++; missing_function()
| 
| will produce an error about missing_function being undefined but the
| value of x will be 1.

Yes, because there are no syntax errors here, so there are no errors
generated by the parser.  The error occurs when the parse tree is
evaluated, and your example has the error coming from the last
expression in the list.

| x = 0; missing_function(); x++
| 
| will also produce an error about missing_function but x will be 0.  So
| it appears as if the commands are coming out of a queue and that
| processing stops as soon as an error is encountered.

Yes, that's essentially correct.

| My only suggestion
| was to limit the queue size between the parsing and evaluation engines.

I don't see how to do that.  Imagine if you have

  x = 0, for i = 1:10
    ...
  endfor

Currently the parser sees no input until the first newline character
(and given that we have command-line editing, I don't think this is
something we want to attempt to change).  At that point, do you expect
to start evaluating the preceding "x = 0" expression and printing the
result as soon as it is available?  I think doing so would cause great
amounts of confusion.

jwe