Re: improved Automake test for file names with funny characters

[Paul Eggert]

Alexandre Duret-Lutz <address@hidden> writes:

> I'm reluctant to chop $destdir using sed in a loop, but at least
> that sounds less clumsy than going into an empty subdirectory to
> execute set :) Any other idea ?

Here's a fix for that problem.  The basic idea is to drop the use
of IFS, except on ancient platforms that don't support mkdir -p.

Also, this fix documents Automake's current limitations in this area.
And it tunes install-sh a bit.

By the way, I notice that we're now using install-sh even when GNU
install is available.  Is there a reason for that?  There are some
security issues involved here, I'm afraid, which would make it better
to use GNU install if available.

2005-07-20  Paul Eggert  <address@hidden>

        * doc/automake.texi (limitations on file names): New section.
        * lib/install-sh: Rewrite to support '*' in file names.
        Also, tune so that we don't invoke so many commands in the usual case.
        * tests/instspc.test: The "*" test is now fixed.

Index: doc/automake.texi
===================================================================
RCS file: /cvs/automake/automake/doc/automake.texi,v
retrieving revision 1.119
diff -p -c -r1.119 automake.texi
*** doc/automake.texi   9 Jul 2005 09:30:31 -0000       1.119
--- doc/automake.texi   20 Jul 2005 20:54:54 -0000
*************** Frequently Asked Questions about Automak
*** 253,258 ****
--- 253,259 ----
  * CVS::                         CVS and generated files
  * maintainer-mode::             missing and AM_MAINTAINER_MODE
  * wildcards::                   Why doesn't Automake support wildcards?
+ * limitations on file names::   Limitations on source and installed file names
  * distcleancheck::              Files left in build directory after distclean
  * Flag Variables Ordering::     CFLAGS vs.@: AM_CFLAGS vs.@: mumble_CFLAGS
  * renamed objects::             Why are object files sometimes renamed?
*************** lists.
*** 8016,8021 ****
--- 8017,8023 ----
  * CVS::                         CVS and generated files
  * maintainer-mode::             missing and AM_MAINTAINER_MODE
  * wildcards::                   Why doesn't Automake support wildcards?
+ * limitations on file names::   Limitations on source and installed file names
  * distcleancheck::              Files left in build directory after distclean
  * Flag Variables Ordering::     CFLAGS vs.@: AM_CFLAGS vs.@: mumble_CFLAGS
  * renamed objects::             Why are object files sometimes renamed?
*************** variables as far Automake is concerned.
*** 8342,8347 ****
--- 8344,8412 ----
  You can get warnings about @samp{$(wildcard ...}) constructs using the
  @option{-Wportability} flag.
  
+ @node limitations on file names
+ @section Limitations on file names
+ @cindex file names, limitations on
+ 
+ Automake attempts to support all kinds of file names, even those that
+ contain unusual characters or are unusually long.  However, some
+ limitations are imposed by the underlying operating system and tools.
+ 
+ Most operating systems prohibit the use of the null byte in file
+ names, and reserve @samp{/} as a directory separator.  Also, they
+ require that file names are properly encoded for the user's locale.
+ Automake is subject to these limits.
+ 
+ Portable packages should limit themselves to @acronym{POSIX} file
+ names.  These can contain @acronym{ASCII} letters and digits,
+ @samp{_}, @samp{.}, and @samp{-}.  File names consist of components
+ separated by @samp{/}.  File name components cannot begin with
+ @samp{-}.
+ 
+ Portable POSIX file names cannot contain components that exceed a
+ 14-byte limit, but nowadays it's normally safe to assume the
+ more-generous @acronym{XOPEN} limit of 255 bytes.  @acronym{POSIX}
+ limits file names to 255 bytes (@acronym{XOPEN} allows 1023 bytes),
+ but you may want to limit a source tarball to file names to 99 bytes
+ to avoid interoperability problems with old versions of @command{tar}.
+ 
+ If you depart from these rules (e.g., by using address@hidden
+ characters in file names, or by using length file names), your
+ installers may have problems for reasons unrelated to Automake.
+ However, if this does not concern you, you should know about the
+ limitations imposed by Automake itself.  These limitations are
+ undesirable, but some of them seem to be inherent to underlying tools
+ like Autoconf, Make, M4, and the shell.  They fall into three
+ categories: install directories, build directories, and file names.
+ 
+ The following characters:
+ 
+ @example
+ @r{newline} " # $ ' `
+ @end example
+ 
+ should not appear in the names of install directories.  For example,
+ the operand of @command{configure}'s @option{--prefix} option should
+ not contain these characters.
+ 
+ Build directories suffer the same limitations as install directories,
+ and in addition should not contain the following characters:
+ 
+ @example
+ & @@ \
+ @end example
+ 
+ For example, the full name of the directory containing the source
+ files should not contain these characters.
+ 
+ Source and installation file names like @file{main.c} are limited even
+ further: they should conform to the @acronym{POSIX}/@acronym{XOPEN}
+ rules described above.  In addition, if you plan to port to
+ address@hidden environments, you should avoid file names that
+ differ only in case (e.g., @file{makefile} and @file{Makefile}).
+ Nowadays it is no longer worth worrying about the 8.3 limits of
+ @acronym{DOS} file systems.
+ 
  @node distcleancheck
  @section Files left in build directory after distclean
  @cindex @code{distclean}, diagnostic
Index: lib/install-sh
===================================================================
RCS file: /cvs/automake/automake/lib/install-sh,v
retrieving revision 1.25
diff -p -c -r1.25 install-sh
*** lib/install-sh      9 Jul 2005 10:21:12 -0000       1.25
--- lib/install-sh      20 Jul 2005 20:54:54 -0000
*************** stripprog="${STRIPPROG-strip}"
*** 58,64 ****
  rmprog="${RMPROG-rm}"
  mkdirprog="${MKDIRPROG-mkdir}"
  
! chmodcmd="$chmodprog 0755"
  chowncmd=
  chgrpcmd=
  stripcmd=
--- 58,67 ----
  rmprog="${RMPROG-rm}"
  mkdirprog="${MKDIRPROG-mkdir}"
  
! posix_mkdir=
! test_mode=u=rwx,g=rx,o=rx,u+wx
! mode=0755
! chmodcmd=$chmodprog
  chowncmd=
  chgrpcmd=
  stripcmd=
*************** while test -n "$1"; do
*** 111,117 ****
  
      --help) echo "$usage"; exit $?;;
  
!     -m) chmodcmd="$chmodprog $2"
          shift
          shift
          continue;;
--- 114,120 ----
  
      --help) echo "$usage"; exit $?;;
  
!     -m) mode=$2
          shift
          shift
          continue;;
*************** if test -z "$1"; then
*** 164,169 ****
--- 167,174 ----
    exit 0
  fi
  
+ test -n "$dir_arg" || trap '(exit $?); exit' 1 2 13 15
+ 
  for src
  do
    # Protect names starting with `-'.
*************** do
*** 173,187 ****
  
    if test -n "$dir_arg"; then
      dst=$src
!     src=
! 
!     if test -d "$dst"; then
!       mkdircmd=:
!       chmodcmd=
!     else
!       mkdircmd=$mkdirprog
!     fi
    else
      # Waiting for this to be detected by the "$cpprog $src $dsttmp" command
      # might cause directories to be created, which would be especially bad
      # if $src (and thus $dsttmp) contains '*'.
--- 178,188 ----
  
    if test -n "$dir_arg"; then
      dst=$src
!     dstdir=$dst
!     test -d "$dstdir"
!     dstdir_status=$?
    else
+ 
      # Waiting for this to be detected by the "$cpprog $src $dsttmp" command
      # might cause directories to be created, which would be especially bad
      # if $src (and thus $dsttmp) contains '*'.
*************** do
*** 208,260 ****
        echo "$0: $dstarg: Is a directory" >&2
        exit 1
        fi
!       dst=$dst/`basename "$src"`
      fi
    fi
  
!   # This sed command emulates the dirname command.
!   dstdir=`echo "$dst" | sed -e 's,/*$,,;s,[^/]*$,,;s,/*$,,;s,^$,.,'`
  
!   # Make sure that the destination directory exists.
  
!   # Skip lots of stat calls in the usual case.
!   if test ! -d "$dstdir"; then
!     case $dstdir in
!       /*) pathcomp=/ ;;
!       -*) pathcomp=./ ;;
!       *)  pathcomp= ;;
      esac
-     oIFS=$IFS
-     IFS=/
-     set fnord $dstdir
-     shift
-     IFS=$oIFS
- 
-     for d
-     do
-       test "x$d" = x && continue
- 
-       pathcomp=$pathcomp$d
-       if test ! -d "$pathcomp"; then
-         $mkdirprog "$pathcomp"
-       # mkdir can fail with a `File exist' error in case several
-       # install-sh are creating the directory concurrently.  This
-       # is OK.
-       test -d "$pathcomp" || exit 1
-       fi
-       pathcomp=$pathcomp/
-     done
    fi
  
    if test -n "$dir_arg"; then
!     $doit $mkdircmd "$dst" \
!       && { test -z "$chowncmd" || $doit $chowncmd "$dst"; } \
!       && { test -z "$chgrpcmd" || $doit $chgrpcmd "$dst"; } \
!       && { test -z "$stripcmd" || $doit $stripcmd "$dst"; } \
!       && { test -z "$chmodcmd" || $doit $chmodcmd "$dst"; }
! 
    else
-     dstfile=`basename "$dst"`
  
      # Make a couple of temp file names in the proper directory.
      dsttmp=$dstdir/_inst.$$_
--- 209,307 ----
        echo "$0: $dstarg: Is a directory" >&2
        exit 1
        fi
!       dstdir=$dst
!       dst=$dstdir/`basename "$src"`
!       dstdir_status=0
!     else
!       # Prefer dirname, but fall back on a substitute if dirname fails.
!       dstdir=`
!       (dirname "$dst") 2>/dev/null ||
!       expr X"$dst" : 'X\(.*[^/]\)//*[^/][^/]*/*$' \| \
!            X"$dst" : 'X\(//\)[^/]' \| \
!            X"$dst" : 'X\(//\)$' \| \
!            X"$dst" : 'X\(/\)' \| \
!            .       : '\(.\)' 2>/dev/null ||
!       echo X"$dst" |
!           sed '/^X\(.*[^/]\)\/\/*[^/][^/]*\/*$/{ s//\1/; q; }
!                 /^X\(\/\/\)[^/].*/{ s//\1/; q; }
!                 /^X\(\/\/\)$/{ s//\1/; q; }
!                 /^X\(\/\).*/{ s//\1/; q; }
!                 s/.*/./; q'
!       `
! 
!       test -d "$dstdir"
!       dstdir_status=$?
      fi
    fi
  
!   obsolete_mkdir_used=false
! 
!   if test $dstdir_status != 0; then
!     case $posix_mkdir in
!       '')
!       posix_mkdir=false
!       if $mkdirprog -m $test_mode -p -- . >/dev/null 2>&1; then
!         posix_mkdir=true
!       else
!         # Remove any dirs left behind by ancient mkdir implementations.
!         rmdir ./-m "$test_mode" ./-p ./-- 2>/dev/null
!       fi ;
!     esac
! 
!     case $posix_mkdir in
!       true)
!       if test -n "$dir_arg"; then
!         mkdir_mode=$mode
!       else
!         case $intermediate_mode in
!           '')
!             if umask_S=`(umask -S) 2>/dev/null`; then
!               intermediate_mode=$umask_S,u+wx
!             else
!               intermediate_mode=$test_mode
!             fi ;;
!         esac
!         mkdir_mode=$intermediate_mode
!       fi
! 
!       $mkdirprog -m "$mkdir_mode" -p -- "$dstdir" ||
!       # Don't fail if two instances are running concurrently.
!       { sleep 1; test -d "$dstdir"; } || exit 1 ;;
! 
!       false)
!       case $dstdir in
!         /*) pathcomp=/ ;;
!         -*) pathcomp=./ ;;
!         *)  pathcomp= ;;
!       esac
!       oIFS=$IFS
!       IFS=/
!       set fnord $dstdir
!       shift
!       IFS=$oIFS
  
!       for d
!       do
!         test "x$d" = x && continue
  
!         pathcomp=$pathcomp$d
!         if test ! -d "$pathcomp"; then
!           $mkdirprog "$pathcomp"
!           # Don't fail if two instances are running concurrently.
!           test -d "$pathcomp" || exit 1
!         fi
!         pathcomp=$pathcomp/
!       done
!       obsolete_mkdir_used=true;;
      esac
    fi
  
    if test -n "$dir_arg"; then
!     { test -z "$chowncmd" || $doit $chowncmd "$dst"; } &&
!     { test -z "$chgrpcmd" || $doit $chgrpcmd "$dst"; } &&
!     { test "$obsolete_mkdir_used$chowncmd$chgrpcmd" = false ||
!       test -z "$chmodcmd" || $doit $chmodcmd "$mode" "$dst"; } || exit 1
    else
  
      # Make a couple of temp file names in the proper directory.
      dsttmp=$dstdir/_inst.$$_
*************** do
*** 262,268 ****
  
      # Trap to clean up those temp files at exit.
      trap 'ret=$?; rm -f "$dsttmp" "$rmtmp" && exit $ret' 0
-     trap '(exit $?); exit' 1 2 13 15
  
      # Copy the file name to the temp name.
      $doit $cpprog "$src" "$dsttmp" &&
--- 309,314 ----
*************** do
*** 276,285 ****
      { test -z "$chowncmd" || $doit $chowncmd "$dsttmp"; } \
        && { test -z "$chgrpcmd" || $doit $chgrpcmd "$dsttmp"; } \
        && { test -z "$stripcmd" || $doit $stripcmd "$dsttmp"; } \
!       && { test -z "$chmodcmd" || $doit $chmodcmd "$dsttmp"; } &&
  
      # Now rename the file to the real destination.
!     { $doit $mvcmd -f "$dsttmp" "$dstdir/$dstfile" 2>/dev/null \
        || {
           # The rename failed, perhaps because mv can't rename something else
           # to itself, or perhaps because mv is so ancient that it does not
--- 322,331 ----
      { test -z "$chowncmd" || $doit $chowncmd "$dsttmp"; } \
        && { test -z "$chgrpcmd" || $doit $chgrpcmd "$dsttmp"; } \
        && { test -z "$stripcmd" || $doit $stripcmd "$dsttmp"; } \
!       && { test -z "$chmodcmd" || $doit $chmodcmd "$mode" "$dsttmp"; } &&
  
      # Now rename the file to the real destination.
!     { $doit $mvcmd -f "$dsttmp" "$dst" 2>/dev/null \
        || {
           # The rename failed, perhaps because mv can't rename something else
           # to itself, or perhaps because mv is so ancient that it does not
*************** do
*** 291,301 ****
           # reasons.  In this case, the final cleanup might fail but the new
           # file should still install successfully.
           {
!            if test -f "$dstdir/$dstfile"; then
!              $doit $rmcmd -f "$dstdir/$dstfile" 2>/dev/null \
!              || $doit $mvcmd -f "$dstdir/$dstfile" "$rmtmp" 2>/dev/null \
               || {
!                echo "$0: cannot unlink or rename $dstdir/$dstfile" >&2
                 (exit 1); exit 1
               }
             else
--- 337,348 ----
           # reasons.  In this case, the final cleanup might fail but the new
           # file should still install successfully.
           {
!            if test -f "$dst"; then
!              $doit $rmcmd -f "$dst" 2>/dev/null \
!              || { $doit $mvcmd -f "$dst" "$rmtmp" 2>/dev/null \
!                    && { $doit $rmcmd -f "$rmtmp" 2>/dev/null; :; }; }\
               || {
!                echo "$0: cannot unlink or rename $dst" >&2
                 (exit 1); exit 1
               }
             else
*************** do
*** 304,319 ****
           } &&
  
           # Now rename the file to the real destination.
!          $doit $mvcmd "$dsttmp" "$dstdir/$dstfile"
         }
!     }
!   fi || { (exit 1); exit 1; }
! done
  
! # The final little trick to "correctly" pass the exit status to the exit trap.
! {
!   (exit 0); exit 0
! }
  
  # Local variables:
  # eval: (add-hook 'write-file-hooks 'time-stamp)
--- 351,363 ----
           } &&
  
           # Now rename the file to the real destination.
!          $doit $mvcmd "$dsttmp" "$dst"
         }
!     } || exit 1
  
!     trap - 0
!   fi
! done
  
  # Local variables:
  # eval: (add-hook 'write-file-hooks 'time-stamp)
Index: tests/instspc.test
===================================================================
RCS file: /cvs/automake/automake/tests/instspc.test,v
retrieving revision 1.6
diff -p -c -r1.6 instspc.test
*** tests/instspc.test  19 Jul 2005 20:08:40 -0000      1.6
--- tests/instspc.test  20 Jul 2005 20:54:54 -0000
*************** done
*** 152,158 ****
  
  # The list of the above file names that cannot be used as a build directory
  # on a POSIX host.  This list should be empty, but is not due to limitations
! # in Autoconf, Automake, Make, or M4.
  expected_build_failures='
  "
  #
--- 152,158 ----
  
  # The list of the above file names that cannot be used as a build directory
  # on a POSIX host.  This list should be empty, but is not due to limitations
! # in Autoconf, Automake, Make, M4, or the shell.
  expected_build_failures='
  "
  #
*************** expected_install_failures='
*** 172,178 ****
  #
  $
  '\''
- *
  `
  '"$lf"'
  a'"${lf}"'b'
--- 172,177 ----