Re: how to get left hand side symbol in action

[Christian Schoenebeck]

On Mittwoch, 10. Juni 2020 07:29:33 CEST Akim Demaille wrote:
> > So my code looks at the individual grammar rules to decide
> > whether the corresponding symbol should be interpreted as terminal or non-
> > terminal accordingly.
> > 
> > Then the other thing is that my auto completion code actually auto
> > completes as much as possible, not just the very next token. So if you
> > have a highly redundant language (like it is often the case with human
> > readable protocols), then it would auto complete several tokens up to the
> > point where an actual input decision would have to be made. For instance,
> > say you were writing an> 
> > SQL parser and the user typed ('^' illustrating the cursor position):
> >     CREATE TABLE foo (id bigint UNIQUE U^
> > 
> > then it would auto complete it to:
> >     CREATE TABLE foo (id bigint UNIQUE USING INDEX TABLESPACE ^
> 
> Ain't the two features the same one?  I mean, the completion of U as USING
> could be a repeated concatenation of S I N G.

Depends. If you just look at this like the user clicks a button and it fills 
the command line, then yes, you are right.

In practice though an application most certainly also wants to know more 
details, i.e. what's a non-terminal, what's a terminal, and where does one 
particular terminal start and stop from raw character sequence point of view.

Sticking to the auto completion example:

a) In a modern UI you probably want to show a dropdown list with the next 
tokens in real-time e.g. while typing, probably also with a different visual 
markup for terminals vs. non-terminals in that list, and the app probably also 
wants to filter out some of the non-terminals like <newline>, <space>, <eof> 
or <string> from that list.

b) When you are showing the next possible options to the user in real-time, 
unlike the actual final auto completion action; you probably don't want to 
concatenate each option in that option list all to their *full* possible 
extent but instead e.g. just limit them to only one next token each (depending 
on the situation), as the number of next options can be quite large and things 
can become very slow in this case (due to potential polynomial complexity).

c) for language keywords the application probably wants to automatically 
handle obvious auto corrections like, e.g:

        select * from foo;  ->  SELECT * FROM foo;
        Select * From foo;  ->  SELECT * FROM foo;
        ...

such that e.g. a developer does not have to worry about case sensitivity 
aspects, hence avoiding to bloat grammar definitions. Or yet other things to 
correct automatically like:

        SELECT * FROM Main Table;  ->   SELECT * FROM 'Main Table';

> > And BTW this was also the 1st compilation issue with a recent Bison
> > version in more than 6 years. That's quite good I would say, so also
> > thanks for taking care of not breaking things! :)
> 
> Nothing *public* is expected to break :)

Mhm, with many other projects it is actually quite normal that things are 
broken in every n-th public release. So having such a long API stability 
(where I'm even using not so 'public' parts of the skeleton) is definitely not 
standard, and I appreciate that!

> >> I'm curious: why don't you require a modern Bison, *and ship the
> >> generated
> >> files*?  Waiting for the end users to have installed recent versions of
> >> the
> >> generators does not buy you a lot of freedom, and forces you to uglify
> >> your
> >> parser.
> > 
> > Reason for still supporting Bison 2: the license. Remember I also use this
> > for commercial projects.
> 
> Given that the GPL does not apply to the generated parser, I don't see what
> worries you hear.

First of all, it does not reflect my personal legal opinion. If you you are in 
the supply chain of other companies you simply have to adapt to their 
requirements.

AFAICS there are two issues: There are use cases where the language needs to 
be adjustable / extensible, which requires to ship a parser generator along.

Probably the most relevant issue though: Many companies simply have a strict 
policy that GPL3 components must not be used. Period. Now you might say, this 
does not make sense in this or another case, but who would evaluate that? 
Individual developers themselves, probably including apprentices or 
freelancers? People from the legal department which mostly cannot read code at 
all? And when would they do that, for every commit?

> > About shipping pregenerated parsers: I already do, for release tarballs
> > that is (not for development versions though, which these reports were
> > about).
> > 
> > Actually most reports about parser related compilation errors were always
> > about users compiling a pregenerated one (i.e. release tarballs),
> 
> I don't understand this.  If you released a pregenerated parser, it
> obviously works, you wouldn't have released otherwise.  So what kind of
> failure can users find that would be fixed by regenerating?
> 
> I can see one scenario: the tarball is old and newer compilers generate
> new warnings.  In which case regenerating with a more recent Bison would
> probably address the issue.
> 
> But I sense you are referring to something different.

I really meant compiler errors, not warnings. I could not tell you the details 
of these issues right now, as I often don't get detailed information about the 
precise circumstances that lead to these compilation errors (e.g. precise 
versions of compiler, flex, bison, ... probably any source changes who knows).

I usually also don't ask though, because the fix is so easy: regenerate it.

Best regards,
Christian Schoenebeck