Re: outdated m4sugar

[Akim Demaille]

Le 10 juil. 08 à 22:43, Eric Blake a écrit :

Hi Eric,

> Right now, the latest bison sources are incompatible with the latest  
> m4
> sources.  When I built bison with the latest m4.git (will become m4  
> 1.6),

Yay!

> the
> change from LIFO to FIFO m4wrap (done in order to make m4 obey POSIX)

I have not understood why they made that choice :(

> trips up
> a fresh build of bison:
>
> $ m4 --version | head -n1
> m4 (GNU M4) 1.5.89a.15-21859
> $ sed -n '1,4 { s/<.*>/<...>/; p }' ChangeLog
> 2008-05-27  Joel E. Denny  <...>
>
>         * NEWS (2.3b+): New section, empty for now.
>         * configure.ac (AC_INIT): 2.3b -> 2.3b+.
> $ make
> ...
> make[2]: Entering directory `/home/eblake/bison/examples/calc++'
> ../../tests/bison -d -ra -S lalr1.cc -o ./calc++-parser.cc \
>           ./calc++-parser.yy
> stdin:1: error: m4_init: unbalanced m4_divert_push:
> stdin:1: m4_divert_push: KILL
> stdin:1: the top level
> /home/eblake/bison/src/bison: subsidiary program `/usr/local/bin/m4'  
> failed
> (exit status 1)
>
> I tracked it to the fact that bison's m4sugar.m4 is three years old  
> (it is
> newer than the m4sugar shipped with autoconf 2.59, but older than  
> 2.59c), which
> means it relies on non-POSIX LIFO m4wrap semantics (autoconf 2.60  
> was the first
> non-beta release that works regardless of LIFO vs. FIFO m4wrap).   
> ChangeLog-
> wise, bison's version looks like it forked from autoconf commit
> http://git.savannah.gnu.org/gitweb/?p=autoconf.git;a=commitdiff;h=cf3b3ede 
> ,
> then added the macro m4_prepend (which autoconf should also consider  
> adding).

It would be very nice, indeed, that m4sugar be unified again.

> In addition to Autoconf 2.62's m4sugar already working around the  
> m4wrap issue,
> recent autoconf also has a number of speed optimizations and bug  
> fixes in
> various m4sugar macros, which make autoconf execution noticeably  
> faster than
> what it was with autoconf 2.61 and earlier.

Bison is paying its performance penalty too to m4, so any speed
improvement will be warmly accepted!

>  The latest autoconf m4sugar
> provides a speed increase even when using GNU m4 1.4.5, independent  
> from the
> speedups also possible by upgrading to m4 1.6.  However, in  
> improving m4sugar
> for autoconf 2.62, I also made some semantic changes on macros that  
> were
> previously undocumented.  I'm not sure whether blindly replacing
> data/m4sugar/m4sugar.m4 with the latest version from autoconf,  
> modulo the
> addition of m4_prepend and the change of one line regarding the  
> sourcing of
> m4sugar/version.m4, will break anything in bison, since I'm not sure  
> how many
> of the undocumented m4sugar macros were in use by bison.

The test suite is reasonably strong, and I would be very surprised
if semantic incompatibilities would be unnoticed.

>  But something needs
> to be done to ensure that bison will continue to work after m4 1.6  
> is released,
> and to make it easier to keep m4sugar in sync between bison and  
> autoconf (or
> perhaps even move m4sugar into GNU m4 as the master).

Indeed, putting it in GNU m4 would be really great!  And actually,
autom4te was also meant to be part of m4.  I suppose that aclocal
is still part of Automake, and not of Autoconf, right?  Moving
things to the right place might not be so easy :)

>  At any rate, I'm
> debating about making bison a show-stopper for the release of m4 1.6  
> (ie. I'm
> reluctant to release m4 1.6 today, since it would mean the latest  
> released
> versions of bison and m4 would be incompatible).

Thank you for your attention to Bison!

>  Autoconf has already proven
> that m4sugar can be made to work across the m4 1.4.x/1.6 transition,  
> so it
> would be nice to have a released version of bison that does  
> likewise, allowing
> a user to upgrade to m4 1.6 without breaking bison (and so that if a  
> user
> happens across a bison/m4 pair that fail to work together, the  
> answer on either
> mailing list can be a simple "known issue; upgrade to the latest  
> stable
> version").

Bison 2.4 is basically ready, betas were sent and AFAIK there
are no bug reports.  Maybe 2.4 could wait for the completed
conversion to a newer GNU M4?  Or would you advise to publish
2.4 as-is, and roll up a 2.5 for never M4s?

> Also, I'm considering making m4 1.6 treat the -d command-line option  
> in a
> position-dependent manner.

Really?  I have been bitten by positional handling of the arguments
before.  What's the motivation?

>  That means that bison's current use of 'm4
> m4sugar/m4sugar.m4 - bison.m4 skeleton.m4 -dV" in output_skeleton
> (src/output.c) will fail to turn on the desired tracing, since the - 
> dV occurs
> too late in the command line.  If you want full debugging for  
> anything passed
> into the pipe to m4, the -dV should precede the "-" argument  
> (although it can
> probably occur after the "m4sugar/m4sugar.m4" argument, since  
> tracing the
> loading of m4sugar is not as interesting).  Autoconf was immune to  
> this issue,
> since autom4te uses -d prior to any files.

OK, thanks for the prompt.

> Meanwhile, bison uses a check for m4 1.3's ability to use frozen files
> (m4/m4.m4, also borrowed from autoconf), even though bison itself  
> does not
> appear to use frozen files.  I don't know how well the current BSD  
> m4 would
> fare with bison's m4 usage patterns, so I suspect that the test is  
> bogus
> (feature-based tests to require GNU m4 over any other version should  
> only be
> necessary if you actually rely on that feature).  On the other hand,  
> I know
> that the BSD folks have already taken pains to implement all GNU  
> extensions
> that were in use by autoconf 2.59 except for frozen files, such that  
> the BSD
> packaging of autoconf intentionally bypasses the m4/m4.m4 check to  
> favor the
> BSD m4 by making a one-line tweak to autom4te to avoid the need for  
> frozen
> files (at the expense of slower operation when using BSD m4 instead  
> of GNU
> m4).

The non-use of frozen files is probably just the result of laziness:
it's too hard to set up and use.  If we had autom4te instead of m4
directly, we would probably use frozen files.  I have no idea what
the speed-up would be though.

>  I also know that while updating autoconf's version of m4sugar for
> autoconf 2.62, I took some liberties to use GNU-specific constructs  
> that
> require GNU m4 1.4.5 or newer, but which will fail with the current  
> release of
> BSD m4.  Thus, upgrading to a newer m4sugar may slightly hurt  
> portability to
> BSD systems if the BSD m4 could previously be used to perform the  
> processing
> needed by bison's m4sugar, and where GNU m4 has not been installed.

That's not a problem, that's theirs.  We *require* GNU M4, and if
packagers want to do otherwise, that's none of our business.

>  But you
> can also argue that one of the features that will be introduced by  
> GNU m4 1.6
> (linear instead of quadratic execution speed on recursive macro  
> expansion)

Wow :)  Where can we read about this?

> means that introducing a hard dependency on GNU m4 is in the user's  
> best
> interest, since BSD m4 already suffers from slower performance, and  
> the
> performance gap between GNU and BSD only widens with the newer GNU  
> m4 release.
>
> I am willing to file copyright papers and help in the effort of  
> updating
> bison's usage of m4sugar, if you would like, since I maintain GNU m4  
> and made
> the bulk of the autoconf m4sugar improvements over the past 3 years.

Please, do!  You are most welcome!