Re: api.header.include and backward-compatible .y files

[Akim Demaille]

Kaz,

> Le 5 sept. 2020 à 17:58, Kaz Kylheku <kaz@kylheku.com> a écrit :
> 
> On 2020-08-30 05:21, Akim Demaille wrote:
>> Hi Adam,
>>> Le 22 août 2020 à 02:33, Adam Novak <anovak@soe.ucsc.edu> a écrit :
>>> Hello,
>>> I'm maintaining a .y file at
>>> https://github.com/vgteam/raptor/blob/master/src/turtle_parser.y that
>>> needs to be backward-compatible with the Bison available in Ubuntu
>>> 18.04 (3.0.4), but also work on the latest Bison that our project's
>>> Mac users get supplied from Homebrew (3.7.1).
>> Back in the days, people were *shipping* the generated files.  That
>> was awesome, since then maintainers are free from such constraints:
>> they use whatever version of their favorite generator is, and are
>> free from requiring anything from the user; users don't even need
>> to have the generator (Bison in the present case).
>> It's a pity today we lost this wisdom.
> 
> Back in the day, people retained the generated files because
> the C language had started to become portable, whereas to get
> C from a Yacc grammar, they still had to upload their code
> to a Unix box to run the proprietary Yacc program.
> 
> Even the person who wrote the program didn't necessarily have
> consistent Unix access, not to mention anyone friends to whom
> that person might give the code.
> 
> People would upload just their .y file to a Unix system, run
> yacc and then download the y.tab.[ch] files.
> 
> The only valid reasons for having any generated files in version
> control or distribution is unavailability of the tool.

You are vastly simplifying things.  In particular, you completely
discard the problems with evolutions here.


> Bison's user is whoever runs Bison. Bison's user is not that one
> who runs the program built with Bison; that is the user's user.
> The user's user is not your user.
> 
> You cannot assume that your user is just a middleman in
> a delivery chain, who can deal with any nuisance that lands his way,
> because it's his job. That user may be a free software developer,
> like you.

You are misunderstanding my point.  My point is that back in the
days people were shipping releases, and releases are self-contained,
they protect the end user from any non standard dependency such
as Autoconf, Automake, Bison, Flex, Gperf, Libtool, Gettext, just
to name a few of them close the GNU project.  Installing a release
was super easy, because you hardly had any dependency.

Maintainers and contributors had a way more complex task: setting
up a *developer*  environment with all the required versions of the
required tools.  And they had to keep their environment fresh.  On
occasions it meant using non released versions of these tools.  But
that was not a problem, because it was only on the shoulders of a
few experienced people.

Way too often today people no longer make self-contained releases,
and releases are hardly different from a git snapshot.  That is
wrong.

This is wrong because now end users need to install tons of tools.
And most of them don't want to install recent versions of these tools
(and I don't blame them), they just want to use the one provided by
their distro.

So today, some maintainers locked themselves into not being able
to use tools that are no widespread enough.  Not to mention that they
might even have to deal with different behaviors from different
versions of the tool.  Then they find convenient to blame the
evolution of the tool.

But the problem is rather their use of the generator.  *They* are
in charge of generating say Bison parsers, and to pass them in
their releases.  That's a mild effort, but with a huge ROI: you
no longer, ever, have to face the nightmare of having to support
very different versions of the tool, and you also can *immediately*
benefit from new features.

I know of several projects, some very important ones, that are
stuck with old versions of Bison although they could benefit from
newer features, features that have sometimes been written *for them*,
to simplify *their* problems.  But they still have the old hackish
code because the recent releases of Bison are not available "yet"
in Ubuntu 18.04...  Gee.


> Bison is a programming language.

True, but irrelevant. The point here is that the output is universal,
independent of the environment on the end user.

> The consumers of programming languages
> are programmers. Yet, we broadly value stability of programming
> languages. Multiple implementations that adhere to common standards
> are also a boon.

True, but moot.  There's one Bison.

> Can you imagine some GCC maintainer suggesting that you ship a .o
> file built with a specific GCC version because of some problem?

Again, completely irrelevant.


> These build problems are not even issues with the "core" of Bison,
> which works great. They are "fluff". But the fluff is a pain in the butt.

Oh, I know.  I am also an author of software who sometimes have to
fight changes, so I know how painful it can be on occasion, so I
pay strict attention to backward compatibility on the published
contract.

But I also know that we must go forward, and sometimes decisions
made years ago, in a given context, need to be reconsidered when
the context changed enough.


> I think the problem is that Bison has a good test suite for the "meat".
> But the tests are, by and large, single-file programs that do not
> provide coverage of possible program structures and build steps
> that surround a grammar file "in the wild".

The test suite can definitely be improved, but it will never be
as good as having users that test the betas, and report their
results.  Every release of Bison had announced betas, and some
users did provide useful feedback.



> This is a problem because Bison is a generator that copy and pastes
> pieces of the user's code into multiple generated files, and
> code is subject to issues of ordering, naming and scope.
> 
> Bison "slices and dices" code like a giant, unhygienic macro,
> whose expansion strategy is not completely documented in every detail.
> 
> Any changes which alter which piece you stick where and in what
> order can potentially break things downstream.

Do not rely on undefined (read "undocumented") behavior.  Don't expect 
undefined behavior to be preserved.

The right approach is rather to see how your need is part of general
pattern, and how that need can be fulfilled in a clean way.

But don't, say, happily sed the generated output, and expect it to
work forever.

Cheers!