RFC: FAQ chapter for the manual (LONG)

[Alexandre Duret-Lutz]

I people,

Here is new chapter I would like to append to the Automake manual.

As English is obviously not my mother tongue, and because I know
some people have strong opinions about some of the issues
discussed, I'd appreciate any comments.

I've tried to stay neutral, but I'm pretty sure some people will
think I'm unfair in some places.  Let me know.

Cheerio,

Alexandre Duret-Lutz

--- snip --- snip --- snip ---

Frequently Asked Questions about Automake
*****************************************

   This chapters covers some questions that often come up on the mailing
lists.

* Menu:

* CVS::                         CVS and generated files
* maintainer-mode::             missing and AM_MAINTAINER_MODE
* wildcards::                   Why doesn't Automake support wildcards?
* distcleancheck::              Files left in build directory after distclean

File: automake.info,  Node: CVS,  Next: maintainer-mode,  Prev: FAQ,  Up: FAQ

CVS and generated files
=======================

Background: distributed generated files
---------------------------------------

   Packages made with Autoconf and Automake ship with some generated
files like `configure' or `Makefile.in'.  These files were generated on
the developer's host and are distributed so that end-users do not have
to install the maintainer tools required to rebuild them.  Other
generated files like Lex scanners, Yacc parsers, or Info documentation,
are usually distributed on similar grounds.

   Automake output rules in `Makefile's to rebuild these files.  For
instance `make' will run `autoconf' to rebuild `configure' whenever
`configure.in' is changed.  This makes development safer by ensuring a
`configure' is never out-of-date with respect to `configure.in'.

   As generated files shipped in packages are up-to-date, and because
`tar' preserves timestamps, these rebuild rules are not triggered when
a user unpacks and builds a package.

Background: CVS and timestamps
------------------------------

   CVS does not preserve timestamps.  The modification time of a
CVS-controlled file is its checkout date.  Because CVS processes files
in alphabetical order, `configure.in' will appear older than
`configure' after a fresh checkout, even if `configure' was up-to-date
when it was checked in.

   In other words, `make' relies on timestamps, CVS messes up
timestamps, so these two tools do not work well together.

Living with CVS in Autoconfiscated projects
-------------------------------------------

   There are basically two clans amongst maintainers: those who keep all
distributed files under CVS, including generated files, and those who
keep generated files _out_ of CVS.

All files in CVS
................

   * The CVS repository contains all distributed files so you know
     exactly what is distributed, and you can checkout any prior
     version entirely (except for timestamps).

   * Maintainers can see how generated files evolve (for instance you
     can see what happens to your `Makefile.in's when you upgrade
     Automake and make sure they look OK).

   * Users do not need the autotools to build a checkout of the
     project, it works almost like a released tarball.  Timestamps are
     inaccurate, hence some rebuild rules will be triggered and attempt
     to run developer tools such as `autoconf' or `automake'.
   
     Actually, calls to such tools are all wrapped into a call to the
     `missing' script discussed later (*note maintainer-mode::).
     `missing' will take care of fixing the timestamps when these tools
     are not installed, so that the build can continue.
   
   * In distributed development, developers are likely to have different
     version of the maintainer tools installed.  In this case rebuilds
     triggered by timestamp lossage will lead to spurious changes to
     generated files.  There are several solutions to this:

        * All developers should use the same versions, so that the
          rebuilt files are identical to files in CVS.  (This starts to
          be difficult when each project you work on uses different
          versions.)

        * Or people use a script to fix the timestamp after a checkout
          (the GCC folks have such a script).
   
        * Or `configure.in' uses `AM_MAINTAINER_MODE', which will
          disable all these rebuild rules by default.  This is further
          discussed in *Note maintainer-mode::.

   * Although we focused on spurious rebuilds, the converse can also
     happen.  CVS's timestamp handling can also let you think an
     out-of-date file is up-to-date.

     For instance, suppose a developer has modified `Makefile.am' and
     rebuilt `Makefile.in', and then decide to do a last-minute change
     to `Makefile.am' right before checking in both files (without
     rebuilding `Makefile.in' to account for the change).

     This last change to `Makefile.am' make the copy of `Makefile.in'
     out-of-date.  Since CVS processes files alphabetically, when
     another developer update his or her tree, `Makefile.in' will
     happen to be newer than `Makefile.am'.  This other developer will
     not see `Makefile.in' is out-of-date.


Generated files out of CVS
..........................
   
   One way to get CVS and `make' working peacefully is to never store
generated files in CVS, i.e., do not CVS-control files which are
`Makefile' targets (or _derived_ files in Make terminology).
   
   This way developers are not annoyed by changes to generated files.
It does not matter if they all have different versions (assuming they
are compatible, of course).  And finally, timestamps are not lost,
changes to sources files can't be missed as in the
`Makefile.am'/`Makefile.in' example discussed earlier.
   
   The drawback is that the CVS repository is not an exact copy of what
is distributed, and that users now need to install various development
tools (maybe even specific versions) before they can build a checkout.
   
   Allowing developers to use different versions of their tools can also
hide bugs during distributed development.  Indeed, developers will be
using (hence testing) their own generated files, instead of the
generated files that will be released actually.  The developer who
prepares the tarball might be using a version of the tool that produce
bogus output (for instance a non-portable C file), something other
developers could have noticed if they weren't using there own versions
of this tool.

Third-party files
-----------------
   
   Another class of files not discussed here (because they dont cause
timestamp issues) are files which are shipped with a package, but
maintained elsewhere.  For instance tools like `gettextize' and
`autopoint' (from Gettext) or `libtoolize' (from Libtool), will install
or update files in your package.
   
   These files, whether they are kept under CVS or not, raise similar
concerns about version mismatch between developers' tools.  The Gettext
manual has a section about this, see *Note CVS Issues: (gettext)CVS
Issues.

File: automake.info,  Node: maintainer-mode,  Next: wildcards,  Prev: CVS,  
Up:\ FAQ

`missing' and `AM_MAINTAINER_MODE'
==================================

`missing'
---------

   The `missing' script is a wrapper around several maintainer tools,
designed to warn users if a maintainer tool is required but missing.
Typical maintainer tools are `autoconf', `automake', `bison', etc.
Because file generated by these tools are shipped with the other
sources of a package, these tools shouldn't be required during a user
build and they are not checked for in `configure'.
   
   However, if for some reason a rebuild rule is triggered and involves
a missing tool, `missing' will notice it and warn the user.  Besides
the warning, when a tool is missing, `missing' will attempt to fix
timestamps in a way which allow the build to continue.  For instance
`missing' will touch `configure' if `autoconf' is not installed.  When
all distributed files are kept under CVS, this feature of `missing'
allows user _with no maintainer tools_ to build a package off CVS,
bypassing timestamp lossage (*note CVS::).

   If the required tool is installed, `missing' will run it and won't
attempt to continue after failures.  This is correct during
development: developers love fixing failures.  However, users with
wrong versions of maintainer tools may get an error when the rebuild
rule is spuriously triggered, halting the build.  This failure to let
the build continue is one of the arguments of the `AM_MAINTAINER_MODE'
advocators.

`AM_MAINTAINER_MODE'
--------------------

   `AM_MAINTAINER_MODE' disables the so called "rebuild rules" by
default.  If you have `AM_MAINTAINER_MODE' in `configure.ac', and run
`./configure && make', then `make' will *never* attempt to rebuilt
`configure', `Makefile.in's, Lex or Yacc outputs, etc.  I.e., this
disables build rules for files which are usually distributed and that
users should normally not have to update.

   If you run `./configure --enable-maintainer-mode', then these
rebuild rules will be active.

   People use `AM_MAINTAINER_MODE' either because they do want their
users (or themselves) annoyed by timestamps lossage (*note CVS::), or
because they simply can't stand the rebuild rules and prefer running
maintainer tools explicitly.

   `AM_MAINTAINER_MODE' also allows you to disable some of your custom
build rules conditionally, and some people use that to disable any rule
that uses some exotic tool users may not have.

   Several years ago François Pinard pointed out several arguments
against `AM_MAINTAINER_MODE'.  Most of them relate to insecurity.  By
removing dependencies you get non-dependable builds: change to sources
files can have no-effect on generated files and this can be very
confusing when unnoticed.  He adds that security shouldn't be reserved
to maintainers (what `--enable-maintainer-mode' suggests), on the
contrary.  If one user has to modify a `Makefile.am', then either
`Makefile.in' should be updated or a warning should be output (this is
what Automake uses `missing' for) but the last thing you want is that
nothing happens and the user doesn't notice it (this is what happens
when rebuild rules are disabled by `AM_MAINTAINER_MODE').

   Jim Meyering, the inventor of the `AM_MAINTAINER_MODE' macro was
swayed by François's arguments, and got rid of `AM_MAINTAINER_MODE' in
all of his packages.

   Still many people continue to use `AM_MAINTAINER_MODE', because it
helps them working on projects where all files are kept under CVS, and
because `missing' isn't enough if you have the wrong version of the
tools.

File: automake.info,  Node: wildcards,  Next: distcleancheck,  Prev: 
maintainer\-mode,  Up: FAQ

Why doesn't Automake support wildcards?
=======================================

   Developers are lazy.  People often they would like to use wildcards
in `Makefile.am's so they don't need to remember they have to update
`Makefile.am's every time they add, delete, or rename a file.

   There are several objections to this:
   * When using CVS (or similar) developers need to remember they have
     to run `cvs add' or `cvs rm' anyway.  Updating `Makefile.am'
     accordingly quickly becomes a reflex.

     Conversely, if your application doesn't compile because you forgot
     to add a file in `Makefile.am', it will help you remember to `cvs
     add' it.

   * Using wildcards makes easy to distribute files by mistake.  For
     instance some code a developer is experimenting with (a test case,
     say) but which should not be part of the distribution.
   
   * Using wildcards it's easy to omit some files by mistake.  For
     instance one developer creates a new file, uses it at many places,
     but forget to commit it.  Another developer then checkout the
     incomplete project and is able to run `make dist' successfully,
     even though a file is missing.

   * Listing files, you control *exactly* what you distribute.  If some
     file that should be distributed is missing from your tree, `make
     dist' will complain.  Besides, you don't distribute more than what
     you listed.

   * Finally it's really hard to `forget' adding a file to
     `Makefile.am', because if you don't add it, it doesn't get
     compiled nor installed, so you can't even test it.

   Portability is another issue.  Although `$(wildcard ...)' works with
GNU `make', it is not portable to other `make' implementations.

   The only way we could support `$(wildcard ...)' is by expending
`$(wildcard ...)' when `automake' is run.  This way resulting
`Makefile.in's would be portable: they would list all files and not use
`$(wildcard ...)'.  However that means developers need to remember they
must run `automake' each time they add, delete, or rename files.
Compared to editing `Makefile.am', this really little win, and you can
get the same effect by writing a script that generate your
`Makefile.am'.  It probably explains why nobody bothered writing a
patch to support this.

File: automake.info,  Node: distcleancheck,  Prev: wildcards,  Up: FAQ

Files left in build directory after distclean
=============================================

   This is a diagnostic you might encounter while running `make
distcheck'.

   As explained in *Note Dist::, `make distcheck' attempts to build and
check your package for errors like this one.

   `make distcheck' will perform a `VPATH' build of your package, and
then call `make distclean'.  Files left in the build directory after
`make distclean' has run are listed after this error.

   This diagnostic really covers two kinds of errors.

   * files that are forgotten by distclean;

   * distributed files that are erroneously rebuilt.

   The former left-over files are not distributed, so the fix is to mark
them for cleaning (*note Clean::), this is obvious and doesn't deserve
more explanations.

   The latter bug is not always easy to understand and fix, so let's
proceed with an example.  Suppose our package contains a program for
which we want to build a man page using `help2man'.  GNU `help2man'
produces simple manual pages from the `--help' and `--version' output
of other commands (*note Overview: (help2man)Top.).  Because we don't
want our users to install `help2man', we decide to distribute the
generated man page using the following setup.

     # This Makefile.am is bogus.
     bin_PROGRAMS = foo
     foo_SOURCES = foo.c
     dist_man_MANS = foo.1
     
     foo.1: foo$(EXEEXT)
        help2man --output=foo.1 ./foo$(EXEEXT)

   This will effectively distribute the man page.  However, `make
distcheck' will fail with:

     ERROR: files left in build directory after distclean:
     ./foo.1

   Why was `foo.1' rebuilt?  Because although distributed, `foo.1'
depends on a non-distributed built file: `foo$(EXEEXT)'.
`foo$(EXEEXT)' is built by the user, so it will always appear to be
newer than the distributed `foo.1'.

   `make distcheck' caught an inconsistency in our package.  Our intent
was to distribute `foo.1' so users do not need installing `help2man',
however since this our rule causes this file to be always rebuilt,
users _do_ need `help2man'.  Either we should ensure that `foo.1' is
not rebuilt by users, or there is no point in distributing `foo.1'.

   More generally, the rule is that distributed files should never
depend on non-distributed built files.  If you distribute something
generated, distribute its sources.

   One way to fix the above example, while still distributing `foo.1'
is to not depend on `foo$(EXEEXT)'.  For instance, assuming `foo
--version' and `foo --help' do not change unless `foo.c' or
`configure.ac' change, we could write the following `Makefile.am':

     bin_PROGRAMS = foo
     foo_SOURCES = foo.c
     dist_man_MANS = foo.1
     
     foo.1: foo.c $(top_srcdir)/configure.ac
             $(MAKE) $(AM_MAKEFLAGS) foo$(EXEEXT)
        help2man --output=foo.1 ./foo$(EXEEXT)

   This way, `foo.1' will not get rebuilt every time `foo$(EXEEXT)'
changes.  The `make' call makes sure `foo$(EXEEXT)' is up-to-date
before `help2man'.  Another way to ensure this would be to use separate
directories for binaries and man pages, and set `SUBDIRS' so that
binaries are built before man pages.