Re: Implement "uniq -z", like "sort -z"

[Jim Meyering]

"James Youngman" <address@hidden> wrote:
> This patch (against current coreutils anon-CVS) implements a -z option
> to uniq.
>
> It is like sort's -z option.   It has the same long form,
> --zero-terminated, as the -z option to sort.    The implementation
> relies on the linebuffer module offering readlinebuffer_delim.  I
> submitted an implementation for that function to bug-gnulib this
> morning.
> 2007-05-12  James Youngman  <address@hidden>
>
>       Add -z option to uniq.  This was originally proposed by
>       Egmont Koblinger.
>       * NEWS: Mention that uniq has gained a new option,
>       --zero-terminated (-z).
>       * src/uniq.c (longopts, check_file, main): Added new option
>       --zero-terminated (-z).  This makes uniq consume and produce
>       NUL-terminated lines rather than newline-terminated lines, which
>       is the default.   Pass the delimiter as a function argument from
>       main into check_file.
>       * doc/uniq.texi (uniq invocation): Describe the new option
>       --zero-terminated (-z).
>       * tests/uniq/Test.pm (@tv): add a number of new tests for the uniq
>       option -z (and its synonym, --zero-terminated).
>       * tests/uniq/Makefile.am (run_gen, maint_gen): add the new test
>       files generated by the updated Test.pm file.

Thank you!
I've checked that in with some changes:

  - in usage, there must be two spaces between the option name and
    the following description.  Otherwise, help2man generates an
    improperly formatted man page.  I added the required space.
    BTW, running "make syntax-check" caught this.

  - in uniq.c, some corner cases were not handled with -z.  E.g.,
      printf 'a\0a\0b\0c\0c\0' |/cu/src/uniq -z --all-re=separate|cat -A
    would still output a newline (rather than a NUL) between the two groups.
    So I made this additional change:

diff --git a/src/uniq.c b/src/uniq.c
index ac0840b..36e2ea3 100644
--- a/src/uniq.c
+++ b/src/uniq.c
@@ -363,7 +365,7 @@ check_file (const char *infile, const char *outfile)
                  if ((delimit_groups == DM_PREPEND)
                      || (delimit_groups == DM_SEPARATE
                          && !first_delimiter))
-                   putchar ('\n');
+                   putchar (delimiter);
                }
            }

    and updated the documentation accordingly.

  - there was too much repetition in the 'z'-suffixed tests, so I
    changed the script to automatically add a '-z'-testing variant of
    each regular test case, when possible.

----------------------------------------------------------------
Here are the two change-sets I've committed locally:

2007-05-13  Jim Meyering  <address@hidden>

        Test uniq's new --zero-terminated (-z) option.
        * tests/uniq/Test.pm: When possible, create a "-z"-testing variant
        of each existing test.
        (2z, 3z, 4z, 5z, 20z, 122, 123): New tests from James Youngman.

2007-05-12  James Youngman  <address@hidden>

        Add -z option to uniq.  Originally proposed by Egmont Koblinger.
        * NEWS: Mention uniq's new option: --zero-terminated (-z).
        * src/uniq.c: Add new option, --zero-terminated (-z), to make
        uniq use the NUL byte as separator/delimiter rather than newline.
        (check_file): Add a parameter: delimiter.  Update caller.
        Use readlinebuffer_delim in place of readlinebuffer everywhere.
        (main): Handle the new option.
        (usage): Describe new option the same way sort does.
        * doc/coreutils.texi (uniq invocation): Describe the new option.

----------------------------------------------------------------
 ChangeLog          |   12 ++++++++++++
 NEWS               |    4 ++++
 doc/coreutils.texi |   19 ++++++++++++++++++-
 src/uniq.c         |   23 +++++++++++++++--------
 4 files changed, 49 insertions(+), 9 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index d9950c6..ed23b65 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,15 @@
+2007-05-12  James Youngman  <address@hidden>
+
+       Add -z option to uniq.  Originally proposed by Egmont Koblinger.
+       * NEWS: Mention uniq's new option: --zero-terminated (-z).
+       * src/uniq.c: Add new option, --zero-terminated (-z), to make
+       uniq use the NUL byte as separator/delimiter rather than newline.
+       (check_file): Add a parameter: delimiter.  Update caller.
+       Use readlinebuffer_delim in place of readlinebuffer everywhere.
+       (main): Handle the new option.
+       (usage): Describe new option the same way sort does.
+       * doc/coreutils.texi (uniq invocation): Describe the new option.
+
 2007-05-07  Jim Meyering  <address@hidden>

        * NEWS: Mention that last week's tr bug dates back to 1992.
diff --git a/NEWS b/NEWS
index ea3aafb..2a23108 100644
--- a/NEWS
+++ b/NEWS
@@ -6,6 +6,10 @@ GNU coreutils NEWS                                    -*- 
outline -*-

   Add SELinux support (FIXME: add details here)

+  uniq accepts a new option: --zero-terminated (-z).  As with the sort
+  option of the same name, this makes uniq consume and produce
+  NUL-terminated lines rather than newline-terminated lines.
+
 ** Bug fixes

   ls -x DIR would sometimes output the wrong string in place of the
diff --git a/doc/coreutils.texi b/doc/coreutils.texi
index 6fc72ec..cc4d0b8 100644
--- a/doc/coreutils.texi
+++ b/doc/coreutils.texi
@@ -4229,11 +4229,15 @@ This is equivalent to @option{--all-repeated} 
(@option{-D}).

 @item prepend
 Output a newline before each group of repeated lines.
+With @option{--zero-terminated} (@option{-z}), use
+an @acronym{ASCII} @sc{nul} (zero) byte instead of a newline.

 @item separate
 Separate groups of repeated lines with a single newline.
+With @option{--zero-terminated} (@option{-z}), use
+an @acronym{ASCII} @sc{nul} (zero) byte instead of a newline.
 This is the same as using @samp{prepend}, except that
-there is no newline before the first group, and hence
+no delimiter is inserted before the first group, and hence
 may be better suited for output direct to users.
 @end table

@@ -4261,6 +4265,19 @@ Compare at most @var{n} characters on each line (after 
skipping any specified
 fields and characters).  By default the entire rest of the lines are
 compared.

address@hidden -z
address@hidden --zero-terminated
address@hidden -z
address@hidden --zero-terminated
address@hidden sort zero-terminated lines
+Treat the input as a set of lines, each terminated by a null character
+(@acronym{ASCII} @sc{nul}) instead of a line feed
+(@acronym{ASCII} @sc{lf}).
+This option can be useful in conjunction with @samp{sort -z}, @samp{perl -0} or
address@hidden -print0} and @samp{xargs -0} which do the same in order to
+reliably handle arbitrary file names (even those containing blanks
+or other special characters).
+
 @end table

 @exitstatus
diff --git a/src/uniq.c b/src/uniq.c
index ac0840b..36e2ea3 100644
--- a/src/uniq.c
+++ b/src/uniq.c
@@ -1,5 +1,5 @@
 /* uniq -- remove duplicate lines from a sorted file
-   Copyright (C) 86, 91, 1995-2006 Free Software Foundation, Inc.
+   Copyright (C) 86, 91, 1995-2007 Free Software Foundation, Inc.

    This program is free software; you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
@@ -119,6 +119,7 @@ static struct option const longopts[] =
   {"skip-fields", required_argument, NULL, 'f'},
   {"skip-chars", required_argument, NULL, 's'},
   {"check-chars", required_argument, NULL, 'w'},
+  {"zero-terminated", no_argument, NULL, 'z'},
   {GETOPT_HELP_OPTION_DECL},
   {GETOPT_VERSION_OPTION_DECL},
   {NULL, 0, NULL, 0}
@@ -156,6 +157,7 @@ Mandatory arguments to long options are mandatory for short 
options too.\n\
   -i, --ignore-case     ignore differences in case when comparing\n\
   -s, --skip-chars=N    avoid comparing the first N characters\n\
   -u, --unique          only print unique lines\n\
+  -z, --zero-terminated  end lines with 0 byte, not newline\n\
 "), stdout);
      fputs (_("\
   -w, --check-chars=N   compare no more than N characters in lines\n\
@@ -268,7 +270,7 @@ writeline (struct linebuffer const *line,
    If either is "-", use the standard I/O stream for it instead. */

 static void
-check_file (const char *infile, const char *outfile)
+check_file (const char *infile, const char *outfile, char delimiter)
 {
   struct linebuffer lb1, lb2;
   struct linebuffer *thisline, *prevline;
@@ -300,7 +302,7 @@ check_file (const char *infile, const char *outfile)
        {
          char *thisfield;
          size_t thislen;
-         if (readlinebuffer (thisline, stdin) == 0)
+         if (readlinebuffer_delim (thisline, stdin, delimiter) == 0)
            break;
          thisfield = find_field (thisline);
          thislen = thisline->length - 1 - (thisfield - thisline->buffer);
@@ -323,7 +325,7 @@ check_file (const char *infile, const char *outfile)
       uintmax_t match_count = 0;
       bool first_delimiter = true;

-      if (readlinebuffer (prevline, stdin) == 0)
+      if (readlinebuffer_delim (prevline, stdin, delimiter) == 0)
        goto closefiles;
       prevfield = find_field (prevline);
       prevlen = prevline->length - 1 - (prevfield - prevline->buffer);
@@ -333,7 +335,7 @@ check_file (const char *infile, const char *outfile)
          bool match;
          char *thisfield;
          size_t thislen;
-         if (readlinebuffer (thisline, stdin) == 0)
+         if (readlinebuffer_delim (thisline, stdin, delimiter) == 0)
            {
              if (ferror (stdin))
                goto closefiles;
@@ -363,7 +365,7 @@ check_file (const char *infile, const char *outfile)
                  if ((delimit_groups == DM_PREPEND)
                      || (delimit_groups == DM_SEPARATE
                          && !first_delimiter))
-                   putchar ('\n');
+                   putchar (delimiter);
                }
            }

@@ -406,6 +408,7 @@ main (int argc, char **argv)
   enum Skip_field_option_type skip_field_option_type = SFO_NONE;
   int nfiles = 0;
   char const *file[2];
+  char delimiter = '\n';       /* change with --zero-terminated, -z */

   file[0] = file[1] = "-";
   initialize_main (&argc, &argv);
@@ -434,7 +437,7 @@ main (int argc, char **argv)
       if (optc == -1
          || (posixly_correct && nfiles != 0)
          || ((optc = getopt_long (argc, argv,
-                                  "-0123456789Dcdf:is:uw:", longopts, NULL))
+                                  "-0123456789Dcdf:is:uw:z", longopts, NULL))
              == -1))
        {
          if (argc <= optind)
@@ -530,6 +533,10 @@ main (int argc, char **argv)
                                  N_("invalid number of bytes to compare"));
          break;

+       case 'z':
+         delimiter = '\0';
+         break;
+
        case_GETOPT_HELP_CHAR;

        case_GETOPT_VERSION_CHAR (PROGRAM_NAME, AUTHORS);
@@ -546,7 +553,7 @@ main (int argc, char **argv)
       usage (EXIT_FAILURE);
     }

-  check_file (file[0], file[1]);
+  check_file (file[0], file[1], delimiter);

   exit (EXIT_SUCCESS);
 }


----------------------------------------------------------------
 ChangeLog          |    7 +++++++
 tests/uniq/Test.pm |   38 +++++++++++++++++++++++++++++++++++---
 2 files changed, 42 insertions(+), 3 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index ed23b65..bbbf99f 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,10 @@
+2007-05-13  Jim Meyering  <address@hidden>
+
+       Test uniq's new --zero-terminated (-z) option.
+       * tests/uniq/Test.pm: When possible, create a "-z"-testing variant
+       of each existing test.
+       (2z, 3z, 4z, 5z, 20z, 122, 123): New tests from James Youngman.
+
 2007-05-12  James Youngman  <address@hidden>

        Add -z option to uniq.  Originally proposed by Egmont Koblinger.
diff --git a/tests/uniq/Test.pm b/tests/uniq/Test.pm
index 1eda7e2..ddc67ac 100644
--- a/tests/uniq/Test.pm
+++ b/tests/uniq/Test.pm
@@ -1,7 +1,6 @@
 # Test "uniq".

-# Copyright (C) 1998, 1999, 2001, 2002, 2003, 2004, 2005, 2006 Free
-# Software Foundation, Inc.
+# Copyright (C) 1998, 1999, 2001-2007 Free Software Foundation, Inc.

 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -34,6 +33,14 @@ my @tv = (
 ['5',  '',    "a\na\nb",         "a\nb\n",          0],
 ['6',  '',    "b\na\na\n",       "b\na\n",          0],
 ['7',  '',    "a\nb\nc\n",       "a\nb\nc\n",       0],
+
+# Ensure that newlines are not interpreted with -z.
+['2z', '-z',  "a\na\n",          "a\na\n\0",        0],
+['3z', '-z',  "a\na",            "a\na\0",          0],
+['4z', '-z',  "a\nb",            "a\nb\0",          0],
+['5z', '-z',  "a\na\nb",         "a\na\nb\0",       0],
+['20z','-dz', "a\na\n",          "",                0],
+
 # Make sure that eight bit characters work
 ['8',  '',    "ö\nv\n",          "ö\nv\n",          0],
 # Test output of -u option; only unique lines
@@ -107,10 +114,15 @@ my @tv = (
 ['120', '-d -u', "a\na\n\b",        "",                         0],
 ['121', '-d -u -w340282366920938463463374607431768211456',
                 "a\na\n\b",        "",                         0],
+# Check that --zero-terminated is synonymous with -z.
+['122', '--zero-terminated',  "a\na\nb",         "a\na\nb\0",       0],
+['123', '--zero-terminated',  "a\0a\0b",         "a\0b\0",          0],
 );

 sub test_vector
 {
+  my @new;
+
   my $t;
   foreach $t (@tv)
     {
@@ -122,9 +134,29 @@ sub test_vector

       $ret
        and $Test::input_via{$test_name} = {REDIR => 0};
+
+      push @new, $t;
+
+      ###########################################################
+      # When possible, create a "-z"-testing variant of each test.
+
+      # skip any test whose input or output already contains a NUL byte
+      $in =~ /\0/ || $exp =~ /\0/
+       and next;
+      # skip any test that uses the -z option
+      $flags =~ /z/
+       and next;
+      # skip the obsolete-syntax tests
+      $test_name =~ /^obs-plus/
+       and next;
+
+      (my $inz = $in) =~ tr/\n/\0/;
+      (my $expz = $exp) =~ tr/\n/\0/;
+      my $t2 = ["$test_name-z", "-z $flags", $inz, $expz, $ret];
+      push @new, $t2;
     }

-  return @tv;
+  return @new;
 }

 1;