Re: sed on binary files

[Eric Blake]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

According to Gary V. Vaughan on 10/1/2008 10:09 PM:
>> I'm working on making m4 1.6 transparently handle NUL,
> 
> Excellent!  I made an attempt to do that myself on the 2.0 branch some
> years ago, but it didn't go well so I never committed...

The argv_ref branch already does this; it is just a matter of finishing
porting it to branch-1.6 and master (and in the process of that porting, I
discovered that the tests I wrote worked fine under GNU sed but died under
Solaris 10).

>> or can have false positives (if both
>> stderr and expected error are normalized, then regressions involving
>> added
>> or missing NUL are not detected).  I don't want to require perl for just
>> this one test; m4 seems fundamental enough to keep the testsuite
>> restricted to the GNU coding standards set of tools.
> 
> I'd be inclined to do that in C.  A few lines should be sufficient to
> write a minimal filter that writes '\' '0' or '^' '@' to output whenever
> a NUL byte arrives?

Actually, I'm a bit lazy - I guess I'm okay with false positives on
Solaris when using deficient sed, so long as we can also run on Solaris
with GNU sed.  So I'm installing this patch, which lets the user select
the right sed, as well as passing both files through sed (a no-op for GNU
sed, but strips NUL bytes equally for Solaris sed).  (At any rate, it was
easier to code than searching for a tr that handles NUL).

Should I also modify configure.ac to call AC_PROG_SED, and feed that as
the default for $SED in the check script?  (The master branch is currently
the only branch that uses $SED, thanks to libtool.)

- --
Don't work too hard, make some time for fun as well!

Eric Blake             address@hidden
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Public key at home.comcast.net/~ericblake/eblake.gpg
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkjku5AACgkQ84KuGfSFAYAaEQCfUDVbbk+qjhQqwn5f43A/t9lq
bIAAoMPzPj1DL0G9jjZgOFao+vVC/S5G
=fyRw
-----END PGP SIGNATURE-----

>From dc967b18e45bed9591a508ed16bbf4cfa3b0e6b9 Mon Sep 17 00:00:00 2001
From: Eric Blake <address@hidden>
Date: Thu, 2 Oct 2008 06:11:51 -0600
Subject: [PATCH] Work around Solaris' sed inability to process NUL.

* checks/check-them (SED): Allow user to override.
(err): Postprocess with sed, just like xerr, to avoid differences
on platforms where sed strips NUL.

Signed-off-by: Eric Blake <address@hidden>
---
 ChangeLog         |    7 +++++++
 checks/check-them |   33 ++++++++++++++++++++-------------
 2 files changed, 27 insertions(+), 13 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index 31219e4..780b635 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,10 @@
+2008-10-02  Eric Blake  <address@hidden>
+
+       Work around Solaris' sed inability to process NUL.
+       * checks/check-them (SED): Allow user to override.
+       (err): Postprocess with sed, just like xerr, to avoid differences
+       on platforms where sed strips NUL.
+
 2008-09-25  Eric Blake  <address@hidden>
 
        Tweak error message on command line failure.
diff --git a/checks/check-them b/checks/check-them
index 89f8b41..017e13b 100755
--- a/checks/check-them
+++ b/checks/check-them
@@ -41,6 +41,9 @@ skipped=
 strip_needed=false
 diffopts=-c
 
+# Allow user to select sed
+: ${SED=sed}
+
 # Find out where the examples live.
 examples=.
 if test "x$1" = x-I ; then
@@ -56,7 +59,7 @@ if test "x$1" = x-m ; then
 fi
 
 # Find out how the executable prints argv[0]
-m4name=`"$m4" --help | sed -e 's/Usage: \(.*\) \[OPTION.*/\1/' \
+m4name=`"$m4" --help | ${SED} -e 's/Usage: \(.*\) \[OPTION.*/\1/' \
   -e 's/\\\\/\\\\\\\\/g' -e 1q`
 
 # Find out if we should strip \r in the output
@@ -70,7 +73,7 @@ else
 fi
 
 # Find out what version the executable claims to be
-m4version=`sed -n 's/.* //p;q' $out`
+m4version=`${SED} -n 's/.* //p;q' $out`
 
 # Find out if diff supports useful options.
 if diff -u /dev/null /dev/null 2>/dev/null ; then
@@ -88,12 +91,12 @@ do
     continue
   }
   echo "Checking $file"
-  options=`sed -ne '3s/^dnl @ extra options: //p;3q' "$file"`
-  sed -e '/^dnl @/d' -e '/^\^D$/q' "$file" \
+  options=`${SED} -ne '3s/^dnl @ extra options: //p;3q' "$file"`
+  ${SED} -e '/^dnl @/d' -e '/^\^D$/q' "$file" \
     | LC_MESSAGES=C M4PATH=$examples "$m4" -d $options - >$out 2>$err
   stat=$?
 
-  xstat=`sed -ne '2s/^dnl @ expected status: //p;2q' "$file"`
+  xstat=`${SED} -ne '2s/^dnl @ expected status: //p;2q' "$file"`
   case $stat in
     77)
       skipped="$skipped $file"
@@ -103,29 +106,33 @@ do
     $xstat) ;;
     *)
       failed="$failed $file:status"
-      echo `sed -e 's/^dnl //' -e 1q $file`
+      echo `${SED} -e 's/^dnl //' -e 1q $file`
       echo "$file: status was $stat, expected $xstat"
       ;;
   esac
 
-  xoutfile=`sed -n 's/^dnl @ expected output: //p' "$file"`
+  xoutfile=`${SED} -n 's/^dnl @ expected output: //p' "$file"`
   if test -z "$xoutfile" ; then
-    sed -e '/^dnl @result{}/!d' -e 's///' -e "s|examples/|$examples/|" \
+    ${SED} -e '/^dnl @result{}/!d' -e 's///' -e "s|examples/|$examples/|" \
       -e "s|@value{VERSION}|$m4version|" "$file" > $xout
   else
     cp "$examples/$xoutfile" $xout
   fi
 
-  xerrfile=`sed -n 's/^dnl @ expected error: //p' "$file"`
+  xerrfile=`${SED} -n 's/^dnl @ expected error: //p' "$file"`
   case $xerrfile in
     ignore)
       cp $err $xerr ;;
     '')
-      sed '/^dnl @error{}/!d
+      ${SED} '/^dnl @error{}/!d
           s///; '"s|^m4:|$m4name:|; s|examples/|$examples/|" \
         "$file" > $xerr ;;
     *)
-      sed "s|^m4:|$m4name:|; s|examples/|$examples/|" \
+      # This is a no-op for GNU sed, but is important for Solaris sed
+      # which strips NUL bytes from input.
+      ${SED} 's,x,x,' $err > $err.1
+      mv $err.1 $err
+      ${SED} "s|^m4:|$m4name:|; s|examples/|$examples/|" \
         "$examples/$xerrfile" > $xerr ;;
   esac
 
@@ -145,7 +152,7 @@ do
     :
   else
     failed="$failed $file:out"
-    echo `sed -e 's/^dnl //' -e 1q $file`
+    echo `${SED} -e 's/^dnl //' -e 1q $file`
     echo "$file: stdout mismatch"
     diff $diffopts $xout $out
   fi
@@ -154,7 +161,7 @@ do
     :
   else
     failed="$failed $file:err"
-    echo `sed -e 's/^dnl //' -e 1q $file`
+    echo `${SED} -e 's/^dnl //' -e 1q $file`
     echo "$file: stderr mismatch"
     diff $diffopts $xerr $err
   fi
-- 
1.6.0.2