Re: Multiply distributed files

[Ralf Wildenhues]

Hello Akim, Peter,

revisiting an old thread:
<http://thread.gmane.org/gmane.comp.sysutils.automake.bugs/3592/focus=3613>
that deals with essentially the same issue as PR 533,
<http://thread.gmane.org/gmane.comp.sysutils.automake.patches/3439>

* Akim Demaille wrote on Tue, May 30, 2006 at 03:54:48PM CEST:
> >>> "adl" == Alexandre Duret-Lutz <address@hidden> writes:
>  > Removing these rights ensures this.  Restoring them would hide some
>  > bugs, like people modifying source files during "make dist".
> 
>  > Maybe a less invasive change is to restore the rights only on the
>  > directories that have been copied with `cp -p'? (I think this can be
>  > done immediately after the `cp -p'.)
> 
> That wouldn't do it, since you don't know if the file or the directory
> will be installed first.

This comment of yours caused me a lot of head-scratching.  I applied
something quite like this approach in order to fix PR 533, see above.

Then I stumbled over this comment of yours again.  What is the exact
setup you were thinking of here?  Can you modify the code below (save
as tests/distdir2.test to try it out) so that it exposes the issue you
were thinking of?

More to the point, I don't think the issue you describe can happen in
practice (but in addition it requires to also use `cp -f' with -pR to
copy over directories).  Here's why:  (assume `.' is the builddir, D the
directory to be copied, F a file below D, F and D listed in DISTFILES):

Case 1: D listed before F in DISTFILES:
- if exists, $(distdir)/D and subdirs will be made writable/executable,
- $(srcdir)/D will be `cp -fpR'ed to $(distdir)/D,
- $(distdir)/D and subdirs will be made writable/executable,
- ./D will be `cp -fpR'ed to $(distdir)/D.

- neither ./F nor $(srcdir)/F will not be copied: a copy already exists
  at $(distdir)/F (and is equal to ./F, or if that is not present,
  $(srcdir)/F).

Case 2: F listed before D in DISTFILES:
- ./F or $(srcdir)/F will be copied to $(distdir)/F, still read-only.

- $(distdir)/D and subdirs will be made writable/executable,
- $(srcdir)/D will be `cp -fpR'ed to $(distdir)/D, possibly overwriting
  F,
- $(distdir)/D and subdirs will be made writable/executable,
- ./D will be `cp -fpR'ed to $(distdir)/D, possibly overwriting F,
  Afterward, $(distdir)/F will be the right version, because if ./F
  exists, it will be copied over in the last step.

Where is the problem?

You can play the same scenario with two directories in DISTFILES, that
either are the same, or one a subdirectory of the other.  I don't see
any problematic case.

> Actually, maybe we should use install
> instead of cp, and set the rights while we set up the tarball?

That would be one possibility, iff my reasoning turns out to be false.
However, that would make `distdir' quite a bit slower, so let's not go
there unless absolutely necessary.

> What is the contract between Automake and its users wrt permissions of
> files within the tarballs?  An exact copy of what is the source?  In
> this case, it defeats the point you rose above, since in some
> circumstances distdir uses permissions which are not those chosen by
> the user.

Here's how I see it: all stuff within the tarball itself should be very
permissive; the user extracting it should steer restrictions with umask
(incidentally, this is very similar to the way git handles permissions).
This gives `distdir' the flexibility to make all directories below
$(distdir) permissive.  However, files are problematic: they could be
hard- or symlinked to files outside of $(distdir) (created through some
dist-hook or so), and we should not touch any files outside of
$(distdir) if we can prevent it at all.

>  > IMHO EXTRA_DISTing directories should be only used for directories
>  > that are generated automatically (e.g., Doxygen documentation).  If
>  > the directory contains hand-maintained files (and hence backups, etc.),
>  > EXTRA_DISTing it is too error-prone.

FWIW I tend to agree; but with complex setups, it may not be uncommon
that directories are even listed twice by some automatic means.

Another common problematic setup is that users EXTRA_DIST a m4/
subdirectory, which then fails because the m4/*.m4 files included by
aclocal.m4 are distributed automatically.  Yes, removing m4 from
EXTRA_DIST is one possibility, but the smart developer may even have
macro files there that are needed with some combination of Autoconf and
Automake only.  One can work around it by adding stub always-required
macros to those files (thus introducing arbitrary differences with the
upstream of those macros) just to get them distributed anyway, is IMVHO
a lot more ugly than just EXTRA_DISTing m4.

> I agree, and I know and promoted this approach.  But for the case at
> hand, I still believe there is a bug from Automake, highlighted by
> this discouraged use of EXTRA_DIST.


Thanks,
Ralf


#! /bin/sh
# Copyright (C) 2009  Free Software Foundation, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 3, or (at your option)
# any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.

# Test to make sure we properly prefer distributing generated files
# over files from the source tree, also when DIST_FILES contains both
# a directory as well as file in it.
# This test exists to ensure we do not fix PR 533 in the wrong way.

. ./defs || Exit 1

set -e

echo AC_OUTPUT >>configure.in

cat > Makefile.am << 'END'
EXTRA_DIST = sub/file sub

all-local:
        $(MKDIR_P) sub
        echo "generated file" > sub/file
CLEANFILES = sub/file
dist-hook:
        grep "generated file" $(distdir)/sub/file
END

$ACLOCAL
$AUTOCONF
$AUTOMAKE

mkdir sub
echo "wrong file" > sub/file

mkdir build
cd build
../configure
$MAKE
$MAKE distcheck

Exit 0