Re: sed on binary files

[Eric Blake]

Eric Blake <ebb9 <at> byu.net> writes:

> Here's what I'm committing on the master branch,
> which actually skips the test if $SED (defaulting to the one learned from
> configure.ac) can't handle NUL.

It helps if I actually attach it for review...

From: Eric Blake <address@hidden>
Date: Wed, 8 Oct 2008 22:00:39 -0600
Subject: [PATCH] Clean up testsuite invocation.

* tests/atlocal.in: Fix copyright notice.
* Makefile.am (TESTS_ENVIRONMENT): Delete; it wasn't being used.
(check-local, installcheck-local): Adjust clients.  Add missing
dependency.

Signed-off-by: Eric Blake <address@hidden>
---
 ChangeLog        |    8 ++++++++
 Makefile.am      |   21 ++++++---------------
 tests/atlocal.in |   17 ++++++++++++++++-
 3 files changed, 30 insertions(+), 16 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index 229b057..9f3f75c 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,11 @@
+2008-10-09  Eric Blake  <address@hidden>
+
+       Clean up testsuite invocation.
+       * tests/atlocal.in: Fix copyright notice.
+       * Makefile.am (TESTS_ENVIRONMENT): Delete; it wasn't being used.
+       (check-local, installcheck-local): Adjust clients.  Add missing
+       dependency.
+
 2008-09-25  Eric Blake  <address@hidden>
 
        Tweak error message on command line failure.
diff --git a/Makefile.am b/Makefile.am
index f6a2b68..35b555b 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -372,16 +372,6 @@ EXTRA_DIST     += $(TESTSUITE) $(TESTSUITE_AT) 
$(OTHER_FILES) \
                  tests/generate.awk \
                  $(srcdir)/tests/package.m4
 
-# Be sure to reexport important environment variables:
-TESTS_ENVIRONMENT = MAKE="$(MAKE)" CC="$(CC)" CFLAGS="$(CFLAGS)" \
-       CPP="$(CPP)" CPPFLAGS="$(CPPFLAGS)" LD="$(LD)" LDFLAGS="$(LDFLAGS)" \
-       LIBS="$(LIBS)" LN_S="$(LN_S)" NM="$(NM)" RANLIB="$(RANLIB)" \
-       OBJEXT="$(OBJEXT)" EXEEXT="$(EXEEXT)" CONFIG_SHELL="$(SHELL)" \
-       abs_top_builddir="$(abs_top_builddir)" \
-       CXX="$(CXX)" CXXFLAGS="$(CXXFLAGS)" CXXCPP="$(CXXCPP)" \
-       F77="$(F77)" FFLAGS="$(FFLAGS)" \
-       GCJ="$(GCJ)" GCJFLAGS="$(GCJFLAGS)"
-
 # Using variables so that this snippet is not too wide and can
 # be used as is in Texinfo @example/@end example.
 m4_texinfo  = $(srcdir)/doc/m4.texinfo
@@ -413,15 +403,16 @@ tests/atconfig: config.status
 DISTCLEANFILES += tests/atconfig
 
 # Hook the test suite into the check rule
-check-local: tests/atconfig tests/m4 $(TESTSUITE) $(check_LTLIBRARIES)
-       $(SHELL) '$(srcdir)/tests/testsuite' -C tests \
-         $(TESTS_ENVIRONMENT) $(TESTSUITEFLAGS)
+check-local: tests/atconfig tests/atlocal tests/m4 $(TESTSUITE) \
+               $(check_LTLIBRARIES)
+       $(SHELL) '$(srcdir)/tests/testsuite' -C tests $(TESTSUITEFLAGS)
 
 # Run the test suite on the *installed* tree, including any renames
 # the user requested.
-installcheck-local: tests/atconfig $(TESTSUITE) $(check_LTLIBRARIES)
+installcheck-local: tests/atconfig tests/atlocal $(TESTSUITE) \
+               $(check_LTLIBRARIES)
        $(SHELL) '$(srcdir)/tests/testsuite' -C tests \
-         $(TESTS_ENVIRONMENT) AUTOTEST_PATH="$(bindir)" \
+         AUTOTEST_PATH="$(bindir)" \
          M4="`echo m4 | sed '$(program_transform_name)'`" $(TESTSUITEFLAGS)
 
 # Enough users install GNU M4 as gm4 that we make sure 'make installcheck'
diff --git a/tests/atlocal.in b/tests/atlocal.in
index 1462e8d..6974987 100644
--- a/tests/atlocal.in
+++ b/tests/atlocal.in
@@ -1,7 +1,22 @@
 # -*- shell-script -*-
 # @configure_input@
 # Configurable variable values for M4 test suite.
-# Copyright (C) 2000, 2001, 2006 Free Software Foundation, Inc.
+# Copyright (C) 2000, 2001, 2006, 2008 Free Software Foundation, Inc.
+
+# This file is part of GNU M4.
+#
+# GNU M4 is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# GNU M4 is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 # Some tests cannot be performed with all the configurations.
 address@hidden@
-- 
1.6.0.2


>From e21aa9fbd79d2080084ce108b8d5e3569f19756f Mon Sep 17 00:00:00 2001
From: Eric Blake <address@hidden>
Date: Thu, 9 Oct 2008 17:08:01 -0600
Subject: [PATCH] Allow user to choose which sed to use in testsuite.

* tests/atlocal.in (SED): Inherit from configure results.
* tests/builtins.at (__m4_@&address@hidden, divert, esyscmd)
(mkstemp, syscmd): Use $SED.
* tests/generate.awk (new_test): Likewise.
* tests/options.at (deprecated options, unknown option)
(--debugmode, --help and --version): Likewise.
* tests/testsuite.at (AT_CHECK_M4): Likewise.
* tests/others.at (directory, stderr closed, stdin seekable)
(stdout closed): Likewise.
(nul character): Likewise.  Also, skip test if sed can't handle
NUL bytes.

Signed-off-by: Eric Blake <address@hidden>
---
 ChangeLog          |   13 +++++++++++++
 tests/atlocal.in   |    4 ++++
 tests/builtins.at  |   15 ++++++++-------
 tests/generate.awk |    4 ++--
 tests/options.at   |   14 +++++++-------
 tests/others.at    |   30 +++++++++++++++++-------------
 tests/testsuite.at |    2 +-
 7 files changed, 52 insertions(+), 30 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index 9f3f75c..38a37b4 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,5 +1,18 @@
 2008-10-09  Eric Blake  <address@hidden>
 
+       Allow user to choose which sed to use in testsuite.
+       * tests/atlocal.in (SED): Inherit from configure results.
+       * tests/builtins.at (__m4_@&address@hidden, divert, esyscmd)
+       (mkstemp, syscmd): Use $SED.
+       * tests/generate.awk (new_test): Likewise.
+       * tests/options.at (deprecated options, unknown option)
+       (--debugmode, --help and --version): Likewise.
+       * tests/testsuite.at (AT_CHECK_M4): Likewise.
+       * tests/others.at (directory, stderr closed, stdin seekable)
+       (stdout closed): Likewise.
+       (nul character): Likewise.  Also, skip test if sed can't handle
+       NUL bytes.
+
        Clean up testsuite invocation.
        * tests/atlocal.in: Fix copyright notice.
        * Makefile.am (TESTS_ENVIRONMENT): Delete; it wasn't being used.
diff --git a/tests/atlocal.in b/tests/atlocal.in
index 6974987..a6120a3 100644
--- a/tests/atlocal.in
+++ b/tests/atlocal.in
@@ -21,3 +21,7 @@
 # Some tests cannot be performed with all the configurations.
 address@hidden@
 address@hidden@
+
+# Prefer tools learned during configure.
+SED='@SED@'
+export SED
diff --git a/tests/builtins.at b/tests/builtins.at
index fe7b0ca..83810fa 100644
--- a/tests/builtins.at
+++ b/tests/builtins.at
@@ -80,7 +80,7 @@ AT_SETUP([__m4_@&address@hidden)
 AT_DATA([in], [[defn(`__m4_version__')
 ]])
 AT_CHECK_M4([--version], [0], [stdout])
-AT_CHECK([[sed -e 's/.*(GNU M4\(.*\)) \([^ ]*\).*/\2\1/;q' < stdout]],
+AT_CHECK([[$SED -e 's/.*(GNU M4\(.*\)) \([^ ]*\).*/\2\1/;q' < stdout]],
 [0], [stdout])
 mv stdout expout
 AT_CHECK_M4([in], [0], [expout])
@@ -313,8 +313,9 @@ hello world
 ]])
 
 dnl Test large diversions, which were broken in m4 1.4.8-1.4.10.
+dnl Hopefully $SED doesn't choke on the over-long second line.
 AT_CHECK([echo 'divert(1)hi
-format(%1000000d, 1)' | $M4 | sed -n 1p], [0], [[hi
+format(%1000000d, 1)' | $M4 | $SED -n 1p], [0], [[hi
 ]])
 
 AT_DATA([in.m4], [M4_ONE_MEG_DEFN[divert(`2')f`'dnl
@@ -467,7 +468,7 @@ esyscmd(`echo hi >&3')divert
 hello
 ]])
 AT_CHECK_M4([3>&-], [0], [stdout], [experr], [in.m4])
-AT_CHECK([sed -ne '/./p' stdout], [0], [[hello
+AT_CHECK([$SED -ne '/./p' stdout], [0], [[hello
 world
 ]])
 
@@ -763,9 +764,9 @@ pid=`cat pid`
 cat >expout <<EOF
 
 X
-X`sed -e 's/.*\(.\)$/\1/' pid`
-X`echo "$pid" | sed -e "s/.*/00000000000000000000&/" -e 's/.*\(.\{20\}$\)/\1/'`
-no_such_dir/`echo "$pid" | sed -e "s/.*/000000&/" -e 's/.*\(.\{6\}$\)/\1/'`
+X`$SED -e 's/.*\(.\)$/\1/' pid`
+X`echo "$pid" | $SED -e "s/.*/00000000000000000000&/" -e 's/.*\(.\{20\}$\)/
\1/'`
+no_such_dir/`echo "$pid" | $SED -e "s/.*/000000&/" -e 's/.*\(.\{6\}$\)/\1/'`
 EOF
 AT_CHECK([cat stdout], [0], [expout])
 
@@ -1136,7 +1137,7 @@ syscmd(`echo hi >&3')divert
 hello
 ]])
 AT_CHECK_M4([3>&-], [0], [stdout], [experr], [in.m4])
-AT_CHECK([sed -ne '/./p' stdout], [0], [[hello
+AT_CHECK([$SED -ne '/./p' stdout], [0], [[hello
 world
 ]])
 
diff --git a/tests/generate.awk b/tests/generate.awk
index 692e21b..538dc09 100755
--- a/tests/generate.awk
+++ b/tests/generate.awk
@@ -179,12 +179,12 @@ function new_test(input, status, output, error, options, 
xfail, examples) {
   if (examples == 1)
     {
       printf ("AT_DATA([expout1],\n[[%s]])\n", output);
-      printf ("sed -e \"s|examples|$abs_top_srcdir/examples|g\" \\\n");
+      printf ("$SED -e \"s|examples|$abs_top_srcdir/examples|g\" \\\n");
       printf ("  < expout1 > expout\n\n");
       if (error)
        {
          printf ("AT_DATA([experr1],\n[[%s]])\n", error);
-         printf ("sed \"s|examples|$abs_top_srcdir/examples|g\" \\\n");
+         printf ("$SED \"s|examples|$abs_top_srcdir/examples|g\" \\\n");
          printf ("  < experr1 > experr\n\n");
        }
       options = options " -I\"$abs_top_srcdir/examples\"";
diff --git a/tests/options.at b/tests/options.at
index 31a1ced..9331a21 100644
--- a/tests/options.at
+++ b/tests/options.at
@@ -27,7 +27,7 @@ AT_SETUP([deprecated options])
 
 dnl -N/--diversions was removed after 1.4.x
 AT_CHECK_M4([--diversions=1], [1], [], [stderr])
-AT_CHECK([sed 's/Try.*help/Try `m4 --help/' stderr], [0],
+AT_CHECK([$SED 's/Try.*help/Try `m4 --help/' stderr], [0],
 [[m4: unrecognized option `--diversions=1'
 Try `m4 --help' for more information.
 ]])
@@ -256,7 +256,7 @@ AT_CLEANUP
 AT_SETUP([unknown option])
 
 AT_CHECK_M4([--unknown], [1], [], [stderr])
-AT_CHECK([sed 's/Try.*help/Try `m4 --help/' stderr], [0],
+AT_CHECK([$SED 's/Try.*help/Try `m4 --help/' stderr], [0],
 [[m4: unrecognized option `--unknown'
 Try `m4 --help' for more information.
 ]])
@@ -458,7 +458,7 @@ m4trace: -1- id 6: divnum
 
 dnl Test that shorter prefix is ambiguous.
 AT_CHECK_M4([--debu], [1], [], [stderr])
-AT_CHECK([sed -e 's/Try.*--help/Try `m4 --help/' stderr], [0],
+AT_CHECK([$SED -e 's/Try.*--help/Try `m4 --help/' stderr], [0],
 [[m4: option `--debu' is ambiguous
 Try `m4 --help' for more information.
 ]])
@@ -543,23 +543,23 @@ AT_CLEANUP
 AT_SETUP([--help and --version])
 
 AT_CHECK_M4([--help], [0], [stdout])
-AT_CHECK([[sed -n -e 's|Usage:.*\[OPTION\]... \[FILE\]...|success|p' stdout]],
+AT_CHECK([[$SED -n -e 's|Usage:.*\[OPTION\]... \[FILE\]...|success|p' stdout]],
 [0], [success
 ])
 
 AT_CHECK_M4([--version], [0], [stdout])
-AT_CHECK([[sed -n -e 's|There is NO WARRANTY.*|success|p' stdout]],
+AT_CHECK([[$SED -n -e 's|There is NO WARRANTY.*|success|p' stdout]],
 [0], [success
 ])
 
 dnl make sure option specified first takes precedence
 AT_CHECK_M4([--help --version], [0], [stdout])
-AT_CHECK([[sed -n -e 's|Usage:.*\[OPTION\]... \[FILE\]...|success|p' stdout]],
+AT_CHECK([[$SED -n -e 's|Usage:.*\[OPTION\]... \[FILE\]...|success|p' stdout]],
 [0], [success
 ])
 
 AT_CHECK_M4([--version --help], [0], [stdout])
-AT_CHECK([[sed -n -e 's|There is NO WARRANTY.*|success|p' stdout]],
+AT_CHECK([[$SED -n -e 's|There is NO WARRANTY.*|success|p' stdout]],
 [0], [success
 ])
 
diff --git a/tests/others.at b/tests/others.at
index 3a23d19..6776ee3 100644
--- a/tests/others.at
+++ b/tests/others.at
@@ -143,14 +143,14 @@ AT_DATA([in3.m4],
 
 AT_CHECK_M4([in1.m4/], [1], [], [stderr])
 dnl mingw fails with EINVAL rather than the expected ENOTDIR
-AT_CHECK([sed 's/Invalid argument/Not a directory/' stderr], [0],
+AT_CHECK([$SED 's/Invalid argument/Not a directory/' stderr], [0],
 [[m4: cannot open `in1.m4/': Not a directory
 ]])
 
 AT_CHECK_M4([in1.m4], [1], [[
 ]], [stderr])
 dnl mingw fails with EINVAL rather than the expected ENOTDIR
-AT_CHECK([sed 's/Invalid argument/Not a directory/' stderr], [0],
+AT_CHECK([$SED 's/Invalid argument/Not a directory/' stderr], [0],
 [[m4:in1.m4:1: include: cannot open `in2.m4/': Not a directory
 ]])
 
@@ -161,7 +161,7 @@ AT_CHECK_M4([in2.m4], [0], [[
 AT_CHECK_M4([in3.m4], [1], [[
 ]], [stderr])
 dnl mingw fails with EACCES rather than the expected EISDIR
-AT_CHECK([sed 's/Permission denied/Is a directory/' stderr], [0],
+AT_CHECK([$SED 's/Permission denied/Is a directory/' stderr], [0],
 [[m4:in3.m4:1: include: cannot open `.': Is a directory
 ]])
 
@@ -419,10 +419,14 @@ AT_CLEANUP
 
 AT_SETUP([nul character])
 
+# Operating on a binary file is a GNU sed extension.
+AT_CHECK([test `printf 'a\0b\n' | $SED s/a.b/abc/ | wc -c` = 4 dnl
+|| { echo "skipping: $SED can't handle NUL"; exit 77; }])
+
 # We don't embed null.* in here, since it is harder to guarantee the
 # behavior of NUL through autom4te.
-sed "s|null.m4|$abs_srcdir/null.m4|" < "$abs_srcdir/null.out" > expout
-sed "s|null.m4|$abs_srcdir/null.m4|" < "$abs_srcdir/null.err" > experr
+$SED "s|null.m4|$abs_srcdir/null.m4|" < "$abs_srcdir/null.out" > expout
+$SED "s|null.m4|$abs_srcdir/null.m4|" < "$abs_srcdir/null.err" > experr
 
 dnl all but m4exit
 AT_CHECK_M4([-Dm4exit -I "$abs_srcdir" null.m4], [0], [expout], [experr])
@@ -562,7 +566,7 @@ undivert
 goodbye
 ]])
 AT_CHECK_M4([2>&-], [1], [stdout], [], [in.m4])
-AT_CHECK([sed -ne '/./p' stdout], [0],
+AT_CHECK([$SED -ne '/./p' stdout], [0],
 [[hello
 and
 goodbye
@@ -688,19 +692,19 @@ AT_DATA([in.m4], [[dnl(
 ]])
 AT_CHECK([($M4 -EE; cat) < in.m4], [0], [[trailing data
 ]], [stderr])
-AT_CHECK([[sed 's/^[^:]*[lt-]*m4[.ex]*:/m4:/' stderr]], [0],
+AT_CHECK([[$SED 's/^[^:]*[lt-]*m4[.ex]*:/m4:/' stderr]], [0],
 [[m4:stdin:1: Warning: dnl: extra arguments ignored: 1 > 0
 ]])
 
 dnl Not all sed and libc combinations get the remaining tests right (for
 dnl example, sed 4.1.4 on glibc, or cygwin 1.5.22 and earlier).
-AT_CHECK([(sed -ne 1q; cat) < in.m4], [0], [stdout])
+AT_CHECK([($SED -ne 1q; cat) < in.m4], [0], [stdout])
 AT_CHECK([test "x`cat stdout`" = "x0)trailing data" || \
-  { echo "skipping: sed is too greedy on seekable stdin"; exit 77; }])
+  { echo "skipping: $SED is too greedy on seekable stdin"; exit 77; }])
 
 dnl Ensure that esyscmd resumes parsing where the child process left off.
 AT_DATA([in.m4], [[define(`foo', `FOO')m4 foo
-esyscmd(`sed -e "s/foo/bar/;q"')sed foo
+esyscmd(`$SED -e "s/foo/bar/;q"')sed foo
 m4 foo
 ]])
 AT_CHECK_M4([], [0], [[m4 FOO
@@ -710,7 +714,7 @@ m4 FOO
 
 dnl Ensure that syscmd resumes parsing where the child process left off.
 AT_DATA([in.m4], [[define(`foo', `FOO')m4 foo
-syscmd(`sed -e "s/foo/bar/;q"')sed foo
+syscmd(`$SED -e "s/foo/bar/;q"')sed foo
 m4 foo
 ]])
 AT_CHECK_M4([], [0], [[m4 FOO
@@ -783,7 +787,7 @@ AT_DATA([hi], [[hi
 ]])
 AT_CHECK([cat hi >&- && { echo "skipping: can't detect closed stdout"; exit 
77; }],
 [1], [], [stderr])
-AT_CHECK([sed 's/Bad file number/Bad file descriptor/' < stderr > experr])
+AT_CHECK([$SED 's/Bad file number/Bad file descriptor/' < stderr > experr])
 AT_DATA([in.m4], [[syscmd(`cat hi')dnl
 ]])
 AT_CHECK_M4([>&-], [0], [], [experr], [in.m4])
@@ -805,7 +809,7 @@ AT_DATA([in.m4], [[syscmd(`cat <&1 >&2')dnl
 dnl this line should not be read by cat
 ]])
 AT_CHECK_M4([in.m4 >&-], [0], [], [stderr])
-AT_CHECK([[sed -e 's/.*[Bb]\(ad file descriptor\)$/B\1/' stderr]], [0],
+AT_CHECK([[$SED -e 's/.*[Bb]\(ad file descriptor\)$/B\1/' stderr]], [0],
 [[Bad file descriptor
 ]])
 
diff --git a/tests/testsuite.at b/tests/testsuite.at
index b510e96..9372e92 100644
--- a/tests/testsuite.at
+++ b/tests/testsuite.at
@@ -75,7 +75,7 @@ m4_define([AT_CHECK_M4],
                                            [< m4_default([$5], [/dev/null])]),
           [$2], [$3], m4_case([$4], [], [], [ignore], [ignore], [stderr]))
 m4_case([$4], [], [], [ignore], [],
-[AT_CHECK([[sed 's/^[^:]*[lt-]*m4[.ex]*:/m4:/
+[AT_CHECK([[$SED 's/^[^:]*[lt-]*m4[.ex]*:/m4:/
        /^m4debug: module/s/opening file.*/opening file/
        s/\(cannot open module [^:]*\):.*/\1/
        s/Bad file number/Bad file descriptor/
-- 
1.6.0.2