RFE: configure -> dependency list on exit.

[Hugh Sasse Staff Elec Eng]

A few weeks ago I wrote to the Gnu coding standards people, with a
suggestion that there should be a DEPENDENCIES file, so that
pre-requisites are listed in one place, and being a separate file
it may encourage people to include this information.  The suggestion
was received positively by RMS, and it was proposed that I put it to
gnu-prog-discuss.  The results are detailed below, with the original
message below that.

This message is a request to implement the suggestion that configure
produce a listing, when it finishes, of the packages needed to build
this package, rather than failing immediately something is missing.

This would overcome most of the problem of having to get "just one more
package", but several times, in fact, when trying to build something.

        Thank you,
        Hugh


Summary of responses to my dependencies suggestion
(hoping I haven't miscounted) and sorted reverse-numerically.

7 people raised duplication of effort since the information is in
  README, INSTALL or autoconf
5 people considered XML NOT to be Human Readable
5 people recommended that configure produce a dependency list at the
  end of operations.
2 poeple specified the problam of dependencies being too general
  like the C compiler, or specific like developer versions.
2 people pointed out that dependencies show up at build time, so are
  not static
2 people believed that this information should be held by the Free
  Software Directory
2 people believed that pkg-config could help manage this kind of
  information
2 people pointed out problems with pkg-config as it stands
1 person raised the problem of dependency loops
1 person raised the problem of making machine readable languages
  human-readable
1 person mentioned Debian 'build-essential' and 'build-depend'
1 person beleived that a program to parse this format should be
  part of the GNU project
1 person believed we should specify platform information in the
  file. 
1 person believed that GNOWSYS may be able to support this
  information

Meta-summary:
   Poeple hate extra work and duplication
   People hate XML
   Configure should produce a summary of dependencies
   The information could be usefully held by the Free Software
   Directory, possibly using GNOWSYS.
   Presently, pkg-config is a "curate's egg" for this application

[The original proposal follows:]

I would like to suggest an addition to the coding standards for GNU
software.

In '7.3 Making Releases' I'd like to see a recommendation like this:

     We recommend that the distribution include a file called
     DEPENDENCIES which should contain a list of packages which are
     prerequisites for building this package.

What is the purpose of this file?
=================================

As packages become more complex, and the need to reuse code
increases, a given package can depend on a number of others.  (For
example, when building GTK I found I needed GTKlib, atk, pkgconfig,
freetype, fontconfig, pango and expat.)  This file is not intended
to replace or even be used by configure, it is intended to allow the
person installing an application access to information up front
about which version(s) of what packages they need to build the
application.

The normal build process will not be altered by its presence or
absence, so it is backwards compatible with existing arrangements.
It should be considered as an advisory document, whose utility would
improve the more widely it were adopted.

This might mean that it takes the place of, for example,  the GCC
build reports page, in some senses: people would need to submit
reports of success and failures to keep this information accurate.
This is done automatically for some projects' test harnesses.

Whom does this help?
====================

It helps users, because they will know which prerequisites they
need, and thus be able to have some idea how long an installation
will take.  It will enable them to do all the downloading up front,
(and maybe copy all the tarballs to some unconnected machine) and have
a successful build, rather than having several successive build
attempts fail.

It will help maintainers, because they will get less enquiries about
how to build the software, freeing up time for more useful work.  It
will enable those working on the code to have an environment which
is a solid base on which to build.  It will ease the transition from
neophyte to contributor, because someone contributing will have more
of an idea of what to test when testing their code, and be more
confident about whether their system is correctly built.

It will help the community, because the installation requirements for
a system will be clearer, thus improving the reputation for ease of
installation for packages.  People will be more confident about what
will build.


What should the file contain?
=============================

The file should contain structured text in a human readable form
describing the dependencies for the various versions of the package.

The important aspects of the format are that it must be:

  * Human readable.  People need to be able to make sense of what
    packages, libraries, etc the application needs, in the absence
    of any tools to support browsing this information.  It should be
    easy to edit these files without introducing errors.

  * Machine Readable.  The possibilities for automatic processing of
    the file include at least:
     * Syntax checking
     * checking sufficient detail is supplied
     * checking for conflicting information about prerequisites
     * automated browsing of remote dependency files, so any
       dependencies introduced by a required package can be found.
     * graphical transformations on the tree to facilitate
       understanding (hiding details, etc, in a GUI)
     * transformation by means of tsort() to explain what order to
       install the packages.
     * It may be possible to create tools to help with maintenance,
       as patch was created to help with handling diff files.  For
       example, it might be possible to derive information from
       the existing configure.ac file, based on --with-package and
       --without-package options.

  * be able to represent the semantics of any dependency
    information, by means of syntactic elements.

Dependencies are usually hierarchical: A depends on B or C; C
depends on D; B depends on E or F......  So a syntax capable of
representing trees would seem to be useful.  However, this
flexibility needs to be balanced by the needs of maintainers.
So it is suggested that a paragraph like:

    Generally, it's best to list only the direct dependencies of
    your package.  The packages for those dependencies should
    include their own DEPENDENCIES file with their information.
    However, you may include sub-dependencies at your discretion.

be added to the document also.

I don't want to dwell on the syntax yet, because it depends on the
semantics, but I believe this requirement could be met with any of
XML, YAML, Lisp, Lua tables, and plenty of other formats.

For now, we can sketch out the ideas in something like XML, bearing
in mind that this is not the only way to represent this...

Semantic Elements
=================

This is the difficult area.  At the moment I think information about
a package or application should include:

  * Its name
  * The versions under consideration, possibly expressed as a range.
    Versions to avoid should be describable as well.
    Any version not mentioned explicitly, may work or may not
  * Where to get information about the package.  Clearly this is
    subject to change, but the file will be maintained, somehow.
  * For any given version(s), the dependencies, which may be AND-ed
    or OR-ed together.
  * Optional extras such as readline, which make the package more
    useable if they are present, but don't break it if they are not.
    This allows people to be aware that it can be useful to get
    related material.


A Fictional Example
===================

So, for this fictionalized example, only using real package names to
save drowning in alphabet soup (package A, B,...). We also assume
that we deliberately intend to specify dependencies of dependencies,
on the basis that we have some good reason for doing so. The file
may look like:

<dependencies>
   <package name="Firefox" version="1.0" avoid-version="0.9"
      site="http://www.mozilla.org/products/firefox/";>
     <or>
       <!-- or groups alternatives, AND is assumed otherwise -->
       <package name="gtk" version="1.1">
         <!-- -->
       </package>
       <package name="gtk" min-version="2.6.1" max-version="2.7">
         <package name=glib" version="2.6.1">
           <package name="pkgconfig" version="0.15.0">
             <!-- -->
           </package>
         </package>
       </package>
     </or>
     <optional>
       <!-- list of non-essential tools it would be nice to have -->
     </optional>
   </package>
</dependencies>

The version information above might be better expressed in a format
like this:

   version="(2.6.1)-(2.8.1),(!3.2)-(!3.4),(!3.1.415)"

the avoidance being declared by '!', but this is exactly the sort of
syntactic detail that must be avoided at this stage of discussion.
I raise it to show there are other possible forms of syntax.

As RMS wrote: 
> Thus, while I don't object to using XML as the syntax, I think that
> more work is needed on the semantics of these relationships before we
> specify a representation for them.


Other Issues
============

I expect there to be aspects which I don't have sufficient
experience to predict.  I have not used anything like the number of
systems supported by GNU software.  I'd imagine that people who
"live and breathe" automake and autoconf could provide useful input
on what is practical for a first shot at this.

There are also issues about network and server loading for the cases
where automated queries are made to obtain recursive dependency
information from servers.  In such a case there may be potential
for denial of service attacks.  A finished design should attempt to
address such issues.

         Hugh