[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [GSoC] preliminary tests on TAP support
|
From: |
Stefano Lattarini |
|
Subject: |
Re: [GSoC] preliminary tests on TAP support |
|
Date: |
Fri, 15 Jul 2011 00:33:48 +0200 |
|
User-agent: |
KMail/1.13.3 (Linux/2.6.30-2-686; KDE/4.4.4; i686; ; ) |
On Monday 11 July 2011, Stefano Lattarini wrote:
> On Saturday 09 July 2011, Stefano Lattarini wrote:
> > On Thursday 07 July 2011, Stefano Lattarini wrote:
> > > Hello Automakers.
> > >
> > > I've created a nice battery of tests on TAP support, and some sketchy
> > > documentation. Some tests are still incomplete, but I think it's better
> > > to start getting some early feedback anyway. Thus I've already pushed
> > > them to the temporary branch 'GSoC/experimental/test-results-work' (note
> > > that, since that branch is temporary and rewindable, the patch can still
> > > be amended freely). Attached is what I've pushed.
> > >
> > > As usual, reviews, suggestions and criticism are welcome. I should be
> > > able to finish the incomplete tests by tomorrow evening.
> > >
> > > Regards,
> > > Stefano
> > >
> > I've now squashed in the attached patch. Notice that some tests are
> > still incomplete or buggy, and there are even some incompatibilities
> > between different tests; but these issues can be sorted out once we
> > have the experimental/exploratory implementation ready and rolling.
> >
> > Regards,
> > Stefano
> >
> And now here it is the first "working" version of the TAP driver (it is still
> incomplete and slightly buggy, but makes lots of the new tests pass). Already
> pushed to the temporary rewindable branch
> 'GSoC/experimental/test-results-work'.
> Comments welcome!
>
> Regards,
> Stefano
>
And attached is the final squash-in that IMHO makes the patch applicable
to a non-temporary branch (even if the new TAP driver is not 100% complete,
as it still lacks support for colorized output and generally is quite rough
around the edges). Since the final amended patch still depends on some
changes that have been so far applied only to the temporary branch
'GSoC/experimental/test-results-work', I'll re-post the final version
of this patch and of those preliminary patches to a new thread, with
the hope that they can end up in the non-rewindable branch 'test-protocols'
soon(ish).
Regards,
Stefano
diff --git a/ChangeLog b/ChangeLog
index 4a9d9c3..e81d0a4 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,18 +1,17 @@
-2011-07-11 Stefano Lattarini <address@hidden>
+2011-07-15 Stefano Lattarini <address@hidden>
tap: add experimental TAP-aware driver
* doc/automake.texi (Using the TAP test protocol): New section.
(Overview of Custom Test Drivers Support): Minor updates.
- * lib/tap-driver: New script, TAP-aware test driver for Automake.
- Implemented in perl and based on TAP::Parser.
+ * lib/tap-driver: New script, TAP-aware test driver for Automake;
+ implemented in perl and based on TAP::Parser.
* lib/Makefile.am (dist_script_DATA): Add it.
* tests/tap-autonumber.test: New test.
- * tests/tap-bad-prog.test: Likewise.
* tests/tap-bailout.test: Likewise.
* tests/tap-basic.test: Likewise.
- * tests/tap-color.test: Likewise.
* tests/tap-deps.test: Likewise.
* tests/tap-diagnostic.test: Likewise.
+ * tests/tap-empty.test: Likewise.
* tests/tap-empty-diagnostic.test: Likewise.
* tests/tap-escape-directive.test: Likewise.
* tests/tap-exit.test: Likewise.
@@ -31,19 +30,28 @@
* tests/tap-numeric-description.test: Likewise.
* tests/tap-out-of-order.test: Likewise.
* tests/tap-passthrough.test: Likewise.
- * tests/tap-plan-errors.test: Likewise.
* tests/tap-plan.test: Likewise.
+ * tests/tap-plan-errors.test: Likewise.
+ * tests/tap-plan-corner.test: Likewise.
* tests/tap-realtime.test: Likewise.
* tests/tap-recheck-logs.test: Likewise.
* tests/tap-recheck.test: Likewise.
* tests/tap-skip-whole.test: Likewise.
* tests/tap-summary.test: Likewise.
- * tests/tap-todo-skip-together.test: Likewise.
* tests/tap-todo-skip.test: Likewise.
+ * tests/tap-todo-skip-together.test: Likewise.
+ * tests/tap-todo-skip-whitespace.test: Likewise.
+ * tests/tap-skipall-whitespace.test: Likewise.
+ * tests/tap-unplanned.test: Likewise.
* tests/tap-whitespace-normalization.test: Likewise.
* tests/tap-with-and-without-number.test: Likewise.
* tests/tap-xfail-tests.test: Likewise.
- * tests/Makefile.am (TESTS): Update.
+ * tests/tap-bad-prog.test: New xfailing test.
+ * tests/tap-color.test: Likewise.
+ * tests/tap-plan-corner2.test: Likewise.
+ * tests/tap-message-0.test: Likewise.
+ * tests/tap-signal.test: Likewise.
+ * tests/Makefile.am (TESTS, XFAIL_TESTS): Update.
2011-07-07 Stefano Lattarini <address@hidden>
diff --git a/lib/tap-driver b/lib/tap-driver
index f71536d..acfcd59 100755
--- a/lib/tap-driver
+++ b/lib/tap-driver
@@ -11,9 +11,9 @@ use strict;
use Getopt::Long ();
use TAP::Parser;
-my $me = "tap-driver";
+my $ME = "tap-driver";
-my $usage = <<'END';
+my $USAGE = <<'END';
Usage:
tap-driver [--help|--version] --test-name=NAME --log-file=PATH
[--expect-failure={yes|no}] [--color-tests={yes|no}]
@@ -22,8 +22,19 @@ Usage:
The `--test-name' and `--log-file' options are mandatory.
END
-my $help = "$me: TAP-aware test driver for Automake testsuite harness." .
- "\n" . $usage;
+my $HELP = "$ME: TAP-aware test driver for Automake testsuite harness." .
+ "\n" . $USAGE;
+
+my $VERSION = '(experimental version)';
+
+# ------------------- #
+# Global variables. #
+# ------------------- #
+
+my $lineno = 0; # Number of input lines seen so far.
+my $testno = 0; # Number of test results seen so far.
+my $plan_seen = 0; # Whether the TAP plan has been seen or not.
+my $parser; # TAP parser object (will be initialized later).
# ----------------- #
# Option parsing. #
@@ -42,7 +53,8 @@ my $test_script_name = undef;
my $log_file = undef;
Getopt::Long::GetOptions (
- 'help' => sub { print $help; exit 0; },
+ 'help' => sub { print $HELP; exit 0; },
+ 'version' => sub { print "$ME $VERSION"; exit 0; },
'test-name=s' => \$test_script_name,
'log-file=s' => \$log_file,
'color-tests=s' => \&bool_opt,
@@ -52,25 +64,52 @@ Getopt::Long::GetOptions (
'no-comments' => sub { $cfg{"comments"} = 0; },
'merge' => sub { $cfg{"merge"} = 1; },
'no-merge' => sub { $cfg{"merge"} = 0; },
- 'ignore-exit' => sub { $cfg{"ignore-exit"} = 0; },
+ 'ignore-exit' => sub { $cfg{"ignore-exit"} = 1; },
) or exit 1;
-# --------------------------------------------- #
-# Prototypes, global variables, and main code. #
-# --------------------------------------------- #
+# -------------- #
+# Subroutines. #
+# -------------- #
+
+sub bool_opt ($$)
+{
+ my ($opt, $val) = @_;
+ if ($val =~ /^(?:y|yes)\z/i)
+ {
+ $cfg{$opt} = 1;
+ }
+ elsif ($val =~ /^(?:n|no)\z/i)
+ {
+ $cfg{$opt} = 0;
+ }
+ else
+ {
+ die "invalid argument '$val' for option '$opt'\n";
+ }
+}
-sub start (@);
-sub finish ();
-sub main (@);
+TAP_PEEKING :
+{
+ my @tap_lines = ();
-my $parser;
+ sub get_tap_line ()
+ {
+ @tap_lines > 0 ? pop @tap_lines : $parser->next;
+ }
-main @ARGV;
+ sub unget_tap_line ($)
+ {
+ push @tap_lines, @_;
+ }
-# -------------- #
-# Subroutines. #
-# -------------- #
+ sub peek_tap_line ()
+ {
+ my $res = get_tap_line;
+ unget_tap_line ($res);
+ return $res;
+ }
+}
TEST_RESULTS :
{
@@ -101,29 +140,65 @@ TEST_RESULTS :
}
-sub bool_opt ($$)
+sub start (@)
{
- my ($opt, $val) = @_;
- if ($val =~ /^(?:y|yes)\z/i)
- {
- $cfg{opt} = 1;
- }
- elsif ($val =~ /^(?:n|no)\z/i)
- {
- $cfg{opt} = 1;
- }
- else
- {
- die "invalid argument '$val' for option '$opt'\n";
- }
+ # Redirect stderr and stdout to a temporary log file. Save the
+ # original stdout stream, since we need it to print testsuite
+ # progress output.
+ open OLDOUT, ">&STDOUT" or die "duplicating stdout: $!\n";
+ open TMP, ">$log_file-t" or die "opening $log_file-t: $!\n";
+ open STDOUT, ">&TMP" or die "redirecting stdout: $!\n";
+ open STDERR, ">&TMP" or die "redirecting stderr: $!\n";
+ $parser = TAP::Parser->new ({ exec => address@hidden, merge => $cfg{merge}
});
+ $parser->ignore_exit(1) if $cfg{"ignore-exit"};
+}
+
+sub finish ()
+{
+ open LOG, ">", $log_file or die "opening $log_file: $!\n";
+
+ # We need to declare a global test result in order to have
+ # "make recheck" working.
+ my $global_result = get_global_test_result;
+ my $global_result_line = "$global_result: $test_script_name";
+ print LOG "$global_result_line\n";
+ print LOG "=" x length ($global_result_line) . "\n";
+ print LOG "\n";
+
+ foreach (get_test_results)
+ {
+ print LOG ":test-result: $_\n";
+ # Add extra trailing empty lines to support reStructuredText -> HTML
+ # conversion.
+ print LOG "\n";
+ }
+
+ # So that the output from the test script won't be parsed for test
+ # results (which would potentially expose us to false positives).
+ print LOG ":test-result: END\n";
+ print LOG "\n";
+
+ close TMP or die "closing $log_file-t: $!\n";
+ # FIXME: remove this hack!
+ my $test_output = `cat $log_file-t && rm -f $log_file-t`;
+ print LOG $test_output;
+ close LOG or die "closing $log_file: $!\n";
+
+ exit 0;
}
sub stringify_test_result ($)
{
my $result = shift;
- if (!$result->directive)
+ my $PASS = $cfg{"expect-failure"} ? "XPASS": "PASS";
+ my $FAIL = $cfg{"expect-failure"} ? "XFAIL": "FAIL";
+ if ($result->is_unplanned || $result->number != $testno)
+ {
+ return "ERROR";
+ }
+ elsif (!$result->directive)
{
- return $result->is_ok ? "PASS": "FAIL";
+ return $result->is_ok ? $PASS: $FAIL;
}
elsif ($result->has_todo)
{
@@ -131,34 +206,48 @@ sub stringify_test_result ($)
}
elsif ($result->has_skip)
{
- return $result->is_ok ? "SKIP" : "FAIL";
+ return $result->is_ok ? "SKIP" : $FAIL;
}
die "INTERNAL ERROR"; # NOTREACHED
}
sub console_output (@)
{
- print OLDOUT join ("\n", @_) . "\n" if @_ > 0;
+ return unless @_ > 0;
+ my $msg = join ("\n", @_) . "\n";
+ print OLDOUT $msg;
+ # Log the result in the log file too, to help debugging (this is
+ # especially true when said result is a TAP error or "Bail out!").
+ print $msg;
}
-sub all_callback ($)
+sub testuite_error ($)
{
- # Verbatim Copy any line of input into the log file.
- print TMP $_[0]->raw . "\n";
+ add_test_result "ERROR";
+ console_output "ERROR: $test_script_name - $_[0]";
}
-sub test_callback ($)
+sub handle_tap_test ($)
{
+ $testno++;
my $test = shift;
my $test_result = stringify_test_result $test;
-
my $string = "$test_result: $test_script_name " . $test->number;
+
if (my $description = $test->description)
{
$string .= " $description";
}
- if (my $directive = $test->directive)
+ if ($test->is_unplanned)
+ {
+ $string .= " # UNPLANNED";
+ }
+ elsif ($test->number != $testno)
+ {
+ $string .= " # OUT-OF-ORDER (expecting $testno)";
+ }
+ elsif (my $directive = $test->directive)
{
$string .= " # $directive";
if (my $explanation = $test->explanation)
@@ -166,16 +255,24 @@ sub test_callback ($)
$string .= " $explanation";
}
}
-
- add_test_result $test_result;
+ add_test_result $test_result;
console_output $string;
}
-sub plan_callback ()
+sub handle_tap_plan ($)
{
my $plan = shift;
+ testuite_error "multiple test plans" if $plan_seen;
+ $plan_seen = 1;
+ # TAP plan must be either in the first or in the last line.
+ if ($lineno > 1 && peek_tap_line)
+ {
+ testuite_error "test plan in middle of output";
+ return;
+ }
+ # Nothing more to do, unless the plan contains a SKIP directive.
return
- unless defined $plan->directive && length ($plan->directive) > 0;
+ if not defined $plan->directive && length ($plan->directive) > 0;
my $string = "SKIP: $test_script_name";
if (my $explanation = $plan->explanation)
{
@@ -183,25 +280,18 @@ sub plan_callback ()
}
add_test_result "SKIP";
console_output $string;
- all_callback ($plan);
finish;
}
-sub bailout_callback ($)
+sub handle_tap_bailout ($)
{
- my $bailout = shift;
- my $string = "ERROR: $test_script_name - Bail out!";
- if (my $explanation = $bailout->explanation)
- {
- $string .= " $explanation";
- }
- add_test_result "ERROR";
- console_output $string;
- all_callback ($bailout);
+ my ($bailout, $msg) = ($_[0], "Bail out!");
+ $msg .= " " . $bailout->explanation if $bailout->explanation;
+ testuite_error $msg;
finish;
}
-sub comment_callback ($)
+sub handle_tap_comment ($)
{
return unless $cfg{comments};
my $comment = $_[0]->comment;
@@ -209,71 +299,65 @@ sub comment_callback ($)
if length $comment;
}
-sub start (@)
-{
- # Redirect stderr and stdout to a temporary log file. Save the original
- # stdout stream, since we need it to print testsuite progress output.
- open OLDOUT, ">&STDOUT" or die "duplicating stdout: $!\n";
- open TMP, ">$log_file-t" or die "opening $log_file-t: $!\n";
- open STDOUT, ">&TMP" or die "redirecting stdout: $!\n";
- open STDERR, ">&TMP" or die "redirecting stderr: $!\n";
-
- my %callbacks = (
- test => \&test_callback,
- plan => \&plan_callback,
- comment => \&comment_callback,
- bailout => \&bailout_callback,
- unknown => sub {},
- ALL => \&all_callback,
- );
-
- $parser = TAP::Parser->new ({
- exec => address@hidden,
- callbacks => \%callbacks,
- merge => $cfg{merge},
- });
-}
-
-sub finish ()
+sub main (@)
{
- open LOG, ">", $log_file or die "opening $log_file: $!\n";
-
- # We need to declare a global test result in order to have
- # "make recheck" working.
- # FIXME: this should be "SKIP" in case the whole script has
- # been skipped.
- my $global_result = get_global_test_result;
- my $global_result_line = "$global_result: $test_script_name";
- print LOG "$global_result_line\n";
- print LOG "=" x length ($global_result_line) . "\n";
- print LOG "\n";
+ start @_;
- foreach (get_test_results)
+ while (defined (my $cur = get_tap_line))
+ {
+ # Verbatim copy any input line into the log file.
+ print $cur->raw . "\n";
+ $lineno++;
+ if ($cur->is_plan)
+ {
+ handle_tap_plan ($cur);
+ }
+ elsif ($cur->is_test)
+ {
+ handle_tap_test ($cur);
+ }
+ elsif ($cur->is_comment)
+ {
+ handle_tap_comment ($cur);
+ }
+ elsif ($cur->is_bailout)
+ {
+ handle_tap_bailout ($cur);
+ }
+ }
+ if (!$plan_seen)
+ {
+ testuite_error "missing test plan";
+ }
+ elsif ($parser->tests_planned != $parser->tests_run)
+ {
+ my ($planned, $run) = ($parser->tests_planned, $parser->tests_run);
+ my $bad_amount = $run > $planned ? "many" : "few";
+ testuite_error (sprintf "too %s tests run (expected %d, got %d)",
+ $bad_amount, $planned, $run);
+ }
+ # TODO: we should probably use $parser->wait here, to catch signals too
+ if ($parser->exit != 0)
{
- print LOG ":test-result: $_\n";
- # Add extra trailing empty lines to support reStructuredText -> HTML
- # conversion.
- print LOG "\n";
+ my $msg = sprintf "exited with status %d", $parser->exit;
+ if ($cfg{"ignore-exit"})
+ {
+ # Log the exit status of the script anyway, even if it is not
+ # considered to be an error, to help debugging.
+ print "INFO: $test_script_name - $msg\n";
+ }
+ else
+ {
+ testuite_error $msg;;
+ }
}
-
- # So that the output from the test script won't be parsed for test
- # results (which would potentially expose us to false positives).
- print LOG ":test-result: END\n";
- print LOG "\n";
-
- close TMP or die "closing $log_file-t: $!\n";
- my $test_output = `cat $log_file-t && rm -f $log_file-t`;
- print LOG $test_output;
- close LOG or die "closing $log_file: $!\n";
-
- exit 0;
-}
-
-sub main (@)
-{
- start @_;
- $parser->run;
finish;
}
+# ----------- #
+# Main code. #
+# ----------- #
+
+main @ARGV;
+
# vim: ft=perl ts=4 sw=4 et
diff --git a/tests/Makefile.am b/tests/Makefile.am
index da41369..2ef4d70 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -27,6 +27,11 @@ gcj6.test \
override-conditional-2.test \
pr8365-remake-timing.test \
yacc-dist-nobuild-subdir.test \
+tap-bad-prog.test \
+tap-color.test \
+tap-plan-corner2.test \
+tap-message-0.test \
+tap-signal.test \
txinfo5.test
@@ -987,17 +992,20 @@ tap-basic.test \
tap-color.test \
tap-deps.test \
tap-diagnostic.test \
+tap-empty.test \
tap-empty-diagnostic.test \
tap-escape-directive.test \
tap-exit.test \
+tap-signal.test \
tap-fancy.test \
tap-fancy2.test \
tap-global-result.test \
tap-html.test \
tap-log.test \
-tap-merge-stdout-stderr.test \
tap-more.test \
tap-more2.test \
+tap-message-0.test \
+tap-merge-stdout-stderr.test \
tap-no-merge-stdout-stderr.test \
tap-no-spurious-summary.test \
tap-no-spurious.test \
@@ -1005,15 +1013,21 @@ tap-not-ok-skip.test \
tap-numeric-description.test \
tap-out-of-order.test \
tap-passthrough.test \
-tap-plan-errors.test \
+tap-passthrough-exit.test \
tap-plan.test \
+tap-plan-errors.test \
+tap-plan-corner.test \
+tap-plan-corner2.test \
tap-realtime.test \
tap-recheck-logs.test \
tap-recheck.test \
-tap-skip-whole.test \
tap-summary.test \
-tap-todo-skip-together.test \
+tap-skip-whole.test \
+tap-skip-whole-whitespace.test \
tap-todo-skip.test \
+tap-todo-skip-whitespace.test \
+tap-todo-skip-together.test \
+tap-unplanned.test \
tap-whitespace-normalization.test \
tap-with-and-without-number.test \
tap-xfail-tests.test \
diff --git a/tests/Makefile.in b/tests/Makefile.in
index 1ad2ca0..c5e1009 100644
--- a/tests/Makefile.in
+++ b/tests/Makefile.in
@@ -273,8 +273,9 @@ EXTRA_DIST = ChangeLog-old gen-parallel-tests
instspc-tests.sh \
extract-testsuite-summary
XFAIL_TESTS = all.test auxdir2.test cond17.test gcj6.test \
override-conditional-2.test pr8365-remake-timing.test \
- yacc-dist-nobuild-subdir.test txinfo5.test \
- $(instspc_xfail_tests)
+ yacc-dist-nobuild-subdir.test tap-bad-prog.test tap-color.test \
+ tap-plan-corner2.test tap-message-0.test tap-signal.test \
+ txinfo5.test $(instspc_xfail_tests)
parallel_tests = backcompat5-p.test check-exported-srcdir-p.test \
check-tests-in-builddir-p.test check-tests_environment-p.test \
check-p.test check11-p.test check12-p.test check2-p.test \
@@ -1234,17 +1235,20 @@ tap-basic.test \
tap-color.test \
tap-deps.test \
tap-diagnostic.test \
+tap-empty.test \
tap-empty-diagnostic.test \
tap-escape-directive.test \
tap-exit.test \
+tap-signal.test \
tap-fancy.test \
tap-fancy2.test \
tap-global-result.test \
tap-html.test \
tap-log.test \
-tap-merge-stdout-stderr.test \
tap-more.test \
tap-more2.test \
+tap-message-0.test \
+tap-merge-stdout-stderr.test \
tap-no-merge-stdout-stderr.test \
tap-no-spurious-summary.test \
tap-no-spurious.test \
@@ -1252,15 +1256,21 @@ tap-not-ok-skip.test \
tap-numeric-description.test \
tap-out-of-order.test \
tap-passthrough.test \
-tap-plan-errors.test \
+tap-passthrough-exit.test \
tap-plan.test \
+tap-plan-errors.test \
+tap-plan-corner.test \
+tap-plan-corner2.test \
tap-realtime.test \
tap-recheck-logs.test \
tap-recheck.test \
-tap-skip-whole.test \
tap-summary.test \
-tap-todo-skip-together.test \
+tap-skip-whole.test \
+tap-skip-whole-whitespace.test \
tap-todo-skip.test \
+tap-todo-skip-whitespace.test \
+tap-todo-skip-together.test \
+tap-unplanned.test \
tap-whitespace-normalization.test \
tap-with-and-without-number.test \
tap-xfail-tests.test \
diff --git a/tests/tap-autonumber.test b/tests/tap-autonumber.test
index 7a72aec..745a639 100755
--- a/tests/tap-autonumber.test
+++ b/tests/tap-autonumber.test
@@ -39,7 +39,7 @@ EXTRA_DIST = $(TESTS)
END
cat > all.test <<'END'
-1..16
+1..14
ok
ok foo
ok - foo2
diff --git a/tests/tap-diagnostic.test b/tests/tap-diagnostic.test
index b70b88c..caef651 100755
--- a/tests/tap-diagnostic.test
+++ b/tests/tap-diagnostic.test
@@ -73,7 +73,7 @@ $AUTOMAKE
$MAKE check >stdout || { cat stdout; Exit 1; }
cat stdout
-grep '#.*all\.test' stdout && Exit 1
+$EGREP -i "#.*all\\.test|a comment|(Tests|Shell) " stdout && Exit 1
count_test_results total=4 pass=2 fail=0 xpass=0 xfail=1 skip=1 error=0
echo 'AM_TEST_LOG_DRIVER_FLAGS = --comments' >> Makefile
@@ -88,7 +88,7 @@ count_test_results total=4 pass=2 fail=0 xpass=0 xfail=1
skip=1 error=0
TEST_LOG_DRIVER_FLAGS="--no-comments" $MAKE -e check >stdout \
|| { cat stdout; Exit 1; }
cat stdout
-grep '#.*all\.test' stdout && Exit 1
+$EGREP -i "#.*all\\.test|a comment|(Tests|Shell) " stdout && Exit 1
count_test_results total=4 pass=2 fail=0 xpass=0 xfail=1 skip=1 error=0
# The "#"-prepended lines here shouldn't be parsed as test results.
@@ -111,9 +111,9 @@ $MAKE check >stdout || { cat stdout; Exit 1; }
cat stdout
count_test_results total=1 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=0
-# Diagnostic wihout leading whitespace, or with extra leading whitespace,
+# Diagnostic without leading whitespace, or with extra leading whitespace,
# is ok. Be laxer in the grepping checks, to allow for whitespace
-# normalization.
+# normalization by the TAP driver.
ws="[ $tab]"
ws0p="${ws}*"
diff --git a/tests/tap-empty.test b/tests/tap-empty.test
new file mode 100755
index 0000000..565c134
--- /dev/null
+++ b/tests/tap-empty.test
@@ -0,0 +1,73 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support: the following situations should be flagged as errors:
+# - empty TAP input
+# - blank TAP input
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+# Empty TAP input.
+: > empty.test
+
+# Blank TAP input (one empty line).
+echo > blank.test
+
+# Blank TAP input (one whitespace-only line).
+cat > white.test <<END
+ ${tab}
+END
+
+# Blank TAP input (few blank and whitespace-only lines).
+cat > white2.test <<END
+
+
+ ${tab} ${tab}${tab}
+
+${tab}
+
+END
+
+for input in empty blank white white2; do
+ cp $input.test all.test
+ $MAKE check >stdout && { cat stdout; Exit 1; }
+ cat stdout
+ count_test_results total=1 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=1
+ grep '^ERROR: all\.test - missing test plan$' stdout
+done
+
+:
diff --git a/tests/tap-exit.test b/tests/tap-exit.test
index be464a5..07209d1 100755
--- a/tests/tap-exit.test
+++ b/tests/tap-exit.test
@@ -32,24 +32,17 @@ END
cat > Makefile.am << 'END'
TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
-TESTS = signal.test
-END
-
-cat > signal.test <<'END'
-#! /bin/sh
-echo 1..1
-echo ok 1
-kill $$
+TESTS =
END
for st in 1 2 77 99; do
- unindent > exit$i.test <<END
+ unindent > exit${st}.test <<END
#!/bin/sh
echo 1..1
echo ok 1
exit $st
END
- echo TESTS += exit$i.test >> Makefile.am
+ echo TESTS += exit${st}.test >> Makefile.am
done
chmod a+x *.test
@@ -58,19 +51,20 @@ $ACLOCAL
$AUTOCONF
$AUTOMAKE
+./configure
+
$MAKE check >stdout && { cat stdout; Exit 1; }
cat stdout
-count_test_results total=10 pass=5 fail=0 xpass=0 xfail=0 skip=0 error=5
+count_test_results total=8 pass=4 fail=0 xpass=0 xfail=0 skip=0 error=4
grep '^ERROR: exit1\.test - exited with status 1$' stdout
grep '^ERROR: exit2\.test - exited with status 2$' stdout
grep '^ERROR: exit77\.test - exited with status 77$' stdout
grep '^ERROR: exit99\.test - exited with status 99$' stdout
-grep '^ERROR: signal\.test - terminated by signal.*' stdout
echo TEST_LOG_DRIVER_FLAGS = --ignore-exit >> Makefile
-$MAKE check >stdout && { cat stdout; Exit 1; }
+$MAKE check >stdout || { cat stdout; Exit 1; }
cat stdout
-count_test_results total=5 pass=5 fail=0 xpass=0 xfail=0 skip=0 error=0
+count_test_results total=4 pass=4 fail=0 xpass=0 xfail=0 skip=0 error=0
:
diff --git a/tests/tap-message-0.test b/tests/tap-message-0.test
new file mode 100755
index 0000000..9f16171
--- /dev/null
+++ b/tests/tap-message-0.test
@@ -0,0 +1,84 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - having "0" as a test description (or TODO or SKIP message) should
+# be supported
+# Note that a bug in some versions of TAP::Parser causes this not to be
+# generally true!
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+END
+
+cat > all.test << 'END'
+1..10
+ok 1 0
+ok - 0
+not ok 3 0 # TODO
+not ok - 0 # TODO
+ok 5 0 # SKIP
+ok - 0 # SKIP
+not ok 7 # TODO 0
+ok 8 # SKIP 0
+not ok 9 0 # TODO 0
+ok 10 0 # SKIP 0
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=10 pass=2 fail=0 xpass=0 xfail=4 skip=4 error=0
+
+cat > exp << 'END'
+PASS: all.test 1 0
+PASS: all.test 2 - 0
+XFAIL: all.test 3 0 # TODO
+XFAIL: all.test 4 - 0 # TODO
+SKIP: all.test 5 0 # SKIP
+SKIP: all.test 6 - 0 # SKIP
+XFAIL: all.test 7 # TODO 0
+SKIP: all.test 8 # SKIP 0
+XFAIL: all.test 9 0 # TODO 0
+SKIP: all.test 10 0 # SKIP 0
+END
+
+$FGREP ': all.test' stdout > got
+
+cat exp
+cat got
+diff exp got
+
+:
diff --git a/tests/tap-out-of-order.test b/tests/tap-out-of-order.test
index b1821cd..2639e2c 100755
--- a/tests/tap-out-of-order.test
+++ b/tests/tap-out-of-order.test
@@ -45,8 +45,8 @@ cat > b.test <<END
1..5
ok 1
ok 2
-ok 4 foo # SKIP
-not ok 2 # TODO
+ok 4 foo # SKIP mu
+not ok 2 bar # TODO um
ok 5
END
@@ -54,7 +54,7 @@ cat > c.test <<END
1..4
ok 1
ok foo
-ok 4 - bar
+ok 4 - zardoz
not ok 3
END
@@ -79,16 +79,20 @@ $AUTOMAKE
$MAKE check >stdout && { cat stdout; Exit 1; }
cat stdout
-count_test_results total=18 pass=11 fail=0 xpass=0 xfail=1 skip=0 error=6
+count_test_results total=17 pass=8 fail=0 xpass=0 xfail=1 skip=0 error=8
+
test `$FGREP -c ': a.test' stdout` -eq 3
test `$FGREP -c ': b.test' stdout` -eq 5
test `$FGREP -c ': c.test' stdout` -eq 4
test `$FGREP -c ': d.test' stdout` -eq 5
-grep '^ERROR: a\.test - test out of order (expecting 3, got 71)$'
-grep '^ERROR: b\.test - test out of order (expecting 3, got 4)$'
-grep '^ERROR: b\.test - test out of order (expecting 4, got 2)$'
-grep '^ERROR: c\.test - test out of order (expecting 3, got 4)$'
-grep '^ERROR: d\.test - test out of order (expecting 3, got 15)$'
+grep '^ERROR: a\.test 71 # OUT-OF-ORDER (expecting 3)$' stdout
+grep '^ERROR: b\.test 4 foo # OUT-OF-ORDER (expecting 3)$' stdout
+grep '^ERROR: b\.test 2 bar # OUT-OF-ORDER (expecting 4)$' stdout
+grep '^ERROR: c\.test 4 - zardoz # OUT-OF-ORDER (expecting 3)$' stdout
+grep '^ERROR: c\.test 3 # OUT-OF-ORDER (expecting 4)$' stdout
+grep '^ERROR: d\.test 15 # OUT-OF-ORDER (expecting 3)$' stdout
+grep '^ERROR: d\.test 16 # OUT-OF-ORDER (expecting 4)$' stdout
+grep '^ERROR: d\.test 17 # OUT-OF-ORDER (expecting 5)$' stdout
:
diff --git a/tests/tap-passthrough-exit.test b/tests/tap-passthrough-exit.test
new file mode 100755
index 0000000..98213a3
--- /dev/null
+++ b/tests/tap-passthrough-exit.test
@@ -0,0 +1,79 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - non-success exit status of a test script is reported in the
+# log file
+# See also related test 'tap-passthrough.test'.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = $(SHELL)
+TESTS =
+END
+
+exit_statuses='1 2 77 99'
+
+for e in $exit_statuses; do
+ unindent > exit-$e.test <<END
+ #!/bin/sh
+ echo 1..1
+ echo ok 1
+ exit $e
+END
+ echo TESTS += exit-$e.test >> Makefile.am
+done
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+st=0
+$MAKE check || st=$?
+for e in $exit_statuses; do cat exit-$e.log; done
+cat test-suite.log
+test $st -gt 0 || Exit 1
+
+for e in $exit_statuses; do
+ for log in exit-$e.log test-suite.log; do
+ grep "^ERROR: exit-$e\\.test - exited with status $e$" $log
+ done
+done
+
+st=0
+env TEST_LOG_DRIVER_FLAGS='--ignore-exit' $MAKE -e check || st=$?
+for e in $exit_statuses; do cat exit-$e.log; done
+cat test-suite.log
+test $st -eq 0 || Exit 1
+
+for e in $exit_statuses; do
+ grep "exit-$e\\.test - exited with status $e$" exit-$e.log
+done
+
+:
diff --git a/tests/tap-passthrough.test b/tests/tap-passthrough.test
index 75df5aa..ab9b3e5 100755
--- a/tests/tap-passthrough.test
+++ b/tests/tap-passthrough.test
@@ -17,7 +17,8 @@
# TAP support:
# - all input (valid TAP lines, invalid TAP lines, non-TAP lines)
# are passed through in the log file
-# - TAP parse errors are reported in the log file too.
+# - TAP errors are reported in the log file too
+# See also related test 'tap-passthrough-exit.test'.
parallel_tests=yes
. ./defs || Exit 1
@@ -48,6 +49,8 @@ $AUTOMAKE
# Only successful tests.
#
+# The whitespace in this test might be normalized in the testsuite
+# progress output, but should be copied verbatim in the log files.
cat > ok.test <<END
1..6
TAP plan in the previous line.
@@ -56,7 +59,7 @@ ok 2
ok - foo
ok 4 - x
This is not a TAP line, but should still be copied in the log file!
-# diagnostic
+# some diagnostic${tab}
not ok # TODO low priority
ok # SKIP who cares?
This is not in test-suite.log, since ok.test does not fail.
@@ -77,7 +80,7 @@ for rx in \
'ok - foo' \
'ok 4 - x' \
' This is not a TAP line, but should still be copied in the log file!' \
- '# diagnostic' \
+ "# some diagnostic${tab}" \
'not ok # TODO low priority' \
'ok # SKIP who cares?' \
'This is not in test-suite\.log, since ok\.test does not fail\.' \
@@ -93,6 +96,11 @@ $FGREP 'not in test-suite.log' test-suite.log && Exit 1
# Mixed failing/successful tests.
#
+cat > tiny.test <<END
+1..1
+ok
+END
+
cat > ok.test <<END
1..1
ok
@@ -126,7 +134,7 @@ ok 1
Invalid test count
ok 23
Misplaced plan
-1..3
+1..13
ok
Extra test
ok
@@ -134,7 +142,9 @@ Last line
END
st=0
-TESTS='ok.test ko.test bail.test skip.test err.test' $MAKE -e check || st=$?
+env TESTS='tiny.test ok.test ko.test bail.test skip.test err.test' \
+ $MAKE -e check || st=$?
+cat tiny.log
cat ok.log
cat ko.log
cat bail.log
@@ -143,6 +153,9 @@ cat err.log
cat test-suite.log
test $st -gt 0 || Exit 1
+grep '^1\.\.1$' tiny.log
+grep '^ok$' tiny.log
+
grep '^local only$' ok.log
grep 'local only' test-suite.log && Exit 1
@@ -171,18 +184,19 @@ for log in skip.log test-suite.log; do
done
for rx in \
+ '^1\.\.3$' \
'^Invalid test count$' \
+ '^ok 23$' \
'^Misplaced plan$' \
- '^Extra test$' \
+ '^1\.\.13$' \
+ '^ERROR:.* test plan in middle of output' \
+ '^Extra test$' \
'^Last line$' \
- '^ok 23$' \
- '[tT]oo many tests run.*expected 3, got 4' \
- '[oO]ut of order.*expecting 2, got 23'
+ '^ERROR:.* [tT]oo many tests run.*expected 3, got 4' \
+ '^ERROR:.* err\.test 23 .*OUT[ -]OF[ -]ORDER.*expecting 2' \
; do
grep "$rx" err.log
grep "$rx" test-suite.log
done
-test `grep -c '^1\.\.3$' err.log` -eq 2
-test `grep -c '^1\.\.3$' test-suite.log` -eq 2
:
diff --git a/tests/tap-plan-corner.test b/tests/tap-plan-corner.test
new file mode 100755
index 0000000..7f46263
--- /dev/null
+++ b/tests/tap-plan-corner.test
@@ -0,0 +1,81 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - some corner cases for TAP plan
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+cat > leading-repeated-1.test <<END
+1..1
+1..1
+ok 1
+END
+
+cat > trailing-repeated-1.test <<END
+ok 1
+1..1
+1..1
+END
+
+cat > leading-repeated-2.test <<END
+1..2
+ok 1
+1..2
+ok 2
+END
+
+cat > trailing-repeated-2.test <<END
+ok 1
+1..2
+ok 2
+1..2
+END
+
+for i in 1 2; do
+ for kind in leading trailing; do
+ cp $kind-repeated-$i.test all.test
+ $MAKE check >stdout && { cat stdout; Exit 1; }
+ cat stdout
+ count_test_results total=`expr $i + 2` pass=$i fail=0 \
+ xpass=0 xfail=0 skip=0 error=2
+ grep '^ERROR: all\.test - test plan in middle of output$' stdout
+ grep '^ERROR: all\.test - multiple test plans$' stdout
+ done
+done
+
+:
diff --git a/tests/tap-plan-corner2.test b/tests/tap-plan-corner2.test
new file mode 100755
index 0000000..155cf85
--- /dev/null
+++ b/tests/tap-plan-corner2.test
@@ -0,0 +1,67 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - more corner cases for TAP plan
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+# The leading blank line is meant.
+cat > leading-blank.test <<END
+
+1..2
+ok 1
+ok 2
+END
+
+# The trailing blank line is meant.
+cat > trailing-blank.test <<END
+ok 1
+ok 2
+1..2
+
+END
+
+for kind in leading trailing; do
+ cp $kind-blank.test all.test
+ $MAKE check >stdout && { cat stdout; Exit 1; }
+ cat stdout
+ count_test_results total=3 pass=2 fail=0 xpass=0 xfail=0 skip=0 error=1
+ grep '^ERROR: all\.test - test plan in middle of output$' stdout
+done
+
+:
diff --git a/tests/tap-plan-errors.test b/tests/tap-plan-errors.test
index e8d0935..539cb8f 100755
--- a/tests/tap-plan-errors.test
+++ b/tests/tap-plan-errors.test
@@ -14,11 +14,14 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
-# TAP support: the following are flagged as errors:
-# - unmatched test plan
+# TAP support: the following situations should be flagged as errors:
+# - unmatched test plan (too few tests run)
# - misplaced test plan
# - multiple test plans
# - missing test plan
+# Checks about unplanned tests are performed in 'tap-unplanned.test'.
+# More checks about corner-cases in TAP plans are performed in
+# 'tap-plan-corner.test' and 'tap-plan-corner2.test'.
parallel_tests=yes
. ./defs || Exit 1
@@ -46,123 +49,62 @@ $AUTOMAKE
my_check ()
{
cat > foo.test
+ test -n "$err" || fatal_ "\$err not set before calling my_check"
+ cat foo.test # For debugging.
$MAKE check >stdout && { cat stdout; Exit 1; }
cat stdout
count_test_results "$@"
+ grep "^ERROR: foo\\.test - $err$" stdout
+ unset err
}
-my_check pass=1 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
+err='too few tests run (expected 2, got 1)'
+my_check total=2 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
1..2
ok 1
END
-grep '^ERROR: foo\.test - too few tests run (expected 2, got 1)$' stdout
-my_check pass=2 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
+err='too few tests run (expected 12, got 3)'
+my_check total=4 pass=2 fail=0 xpass=0 xfail=1 skip=0 error=1 <<END
ok 1
ok 2
not ok 3 # TODO
1..12
END
-grep '^ERROR: foo\.test - too few tests run (expected 12, got 3)$' stdout
-my_check pass=0 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
+err='too few tests run (expected 1, got 0)'
+my_check total=1 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
1..1
END
-grep '^ERROR: foo\.test - too few tests run (expected 1, got 0)$' stdout
-
-my_check pass=2 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
-1..1
-ok 1
-ok 2
-END
-grep '^ERROR: foo\.test - too many tests run (expected 1, got 2)$' stdout
-
-my_check pass=4 fail=0 xpass=0 xfail=0 skip=5 error=1 <<END
-1..4
-ok 1
-ok 2
-ok 3
-ok 4
-ok 5
-ok 6
-ok 7
-END
-grep '^ERROR: foo\.test - too many tests run (expected 4, got 7)$' stdout
-
-my_check pass=2 fail=0 xpass=0 xfail=1 skip=0 error=1 <<END
-ok 1
-ok 2
-not ok 3 # TODO
-1..2
-END
-grep '^ERROR: foo\.test - too many tests run (expected 2, got 3)$' stdout
-
-# Leading blank line is meant.
-my_check pass=2 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
+err='test plan in middle of output'
+my_check total=3 pass=2 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
+oops
1..2
ok 1
ok 2
END
-grep '^ERROR: foo\.test - test plan in middle of output$' stdout
-# Trailing blank line is meant.
-my_check pass=2 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
+err='test plan in middle of output'
+my_check total=3 pass=2 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
ok 1
ok 2
1..2
-
+oops
END
-grep '^ERROR: foo\.test - test plan in middle of output$' stdout
-
-# In the next three checks, the two plans are equal -- this is deliberate.
-my_check pass=2 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
-1..1
-1..1
-ok 1
-END
-grep '^ERROR: foo\.test - multiple test plans$' stdout
-
-my_check pass=2 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
-ok 1
-1..1
-1..1
-END
-grep '^ERROR: foo\.test - multiple test plans$' stdout
-
-my_check pass=2 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
+# The two test plans here are deliberately equal.
+err='multiple test plans'
+my_check total=3 pass=2 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
1..2
ok 1
ok 2
1..2
END
-grep '^ERROR: foo\.test - multiple test plans$' stdout
-
-my_check pass=1 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
-ok 1
-END
-grep '^ERROR: foo\.test - missing test plans$' stdout
-
-my_check pass=0 fail=0 xpass=0 xfail=0 skip=0 error=1 </dev/null
-grep '^ERROR: foo\.test - missing test plans$' stdout
-
-# Check error message in corner cases (1).
-my_check pass=2 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
-1..2
-ok 1
-1..2
-ok 2
-END
-grep '^ERROR: foo\.test - test plan in middle of output$' stdout
-# Check error message in corner cases (2).
-my_check pass=2 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
+err='missing test plan'
+my_check total=2 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
ok 1
-1..2
-ok 2
-1..2
END
-grep '^ERROR: foo\.test - test plan in middle of output$' stdout
:
diff --git a/tests/tap-realtime.test b/tests/tap-realtime.test
index f5b9496..0f43e9e 100755
--- a/tests/tap-realtime.test
+++ b/tests/tap-realtime.test
@@ -18,14 +18,16 @@
# - testsuite progress on console should happen mostly "in real time";
# i.e., it's not acceptable for the driver to wait the end of the
# script to start displaying results from it.
+# FIXME: this test uses expect(1) to ensure line buffering from make and
+# children, and is pretty hacky and complex; is there a better way to
+# accomplish the checks done here?
parallel_tests=yes
. ./defs || Exit 1
-# Check that we have a working expect program.
cat >expect-check <<'END'
-eval spawn $env(SHELL) -c "echo ok"
-expect "^ok$"
+eval spawn $env(SHELL) -c ":"
+expect eof
END
expect -f expect-check || {
echo "$me: failed to find a working expect program" >&2
@@ -49,13 +51,23 @@ END
cat > all.test <<'END'
#! /bin/sh
echo 1..3
-echo ok 1
-: > milestone-1
-sleep '10' # FIXME: quoting required to placate maintainer-check
-echo ok 2
-: > milestone-2
-sleep '10' # FIXME: quoting required to placate maintainer-check
-echo ok 3
+
+# FIXME: creative quoting to placate maintainer-check
+sleep="sleep "3
+
+echo ok 1 - foo
+$sleep
+test -f ok-1 || { echo 'Bail out!'; exit 1; }
+
+echo ok 2 - bar
+$sleep
+test -f ok-2 || { echo 'Bail out!'; exit 1; }
+
+echo ok 3 - baz
+$sleep
+test -f ok-3 || { echo 'Bail out!'; exit 1; }
+
+: > all-is-well
END
chmod a+x all.test
@@ -66,31 +78,38 @@ $AUTOMAKE
./configure
-cat >expect-make <<'END'
-eval spawn $env(MAKE) -e check
-expect eof
+cat > expect-make <<'END'
+eval spawn $env(MAKE) check
+expect {
+ "PASS: all.test 1 - foo" {
+ open "ok-1" "w"
+ exp_continue
+ }
+ "PASS: all.test 2 - bar" {
+ open "ok-2" "w"
+ exp_continue
+ }
+ "PASS: all.test 3 - baz" {
+ open "ok-3" "w"
+ exp_continue
+ }
+ "Testsuite summary" {
+ exit 0
+ }
+ timeout {
+ puts "expect timed out"
+ exit 1
+ }
+ default {
+ puts "expect error"
+ exit 1
+ }
+}
END
# Expect should simulate a tty as stdout, which should ensure a
# line-buffered output.
-MAKE=$MAKE expect -f expect-make > stdout &
-
-while test ! -f milestone-1; do sleep '1'; done
-cat stdout
-grep '^PASS: all\.test 1$' stdout
-test `$FGREP -c ': all.test'` -eq 1
-
-while test ! -f milestone-2; do sleep '1'; done
-cat stdout
-grep '^PASS: all\.test 1$' stdout
-grep '^PASS: all\.test 2$' stdout
-test `$FGREP -c ': all.test'` -eq 2
-
-wait
-cat stdout
-grep '^PASS: all\.test 1$' stdout
-grep '^PASS: all\.test 2$' stdout
-grep '^PASS: all\.test 3$' stdout
-test `$FGREP -c ': all.test'` -eq 2
+MAKE=$MAKE expect -f expect-make
+test -f all-is-well
:
diff --git a/tests/tap-recheck.test b/tests/tap-recheck.test
index 27893c4..14b02ad 100755
--- a/tests/tap-recheck.test
+++ b/tests/tap-recheck.test
@@ -55,7 +55,7 @@ END
cat > c.test << 'END'
#! /bin/sh
-echo 1..2
+echo 1..3
if test -f c.pass; then
echo 'ok - c is ok :-)'
else
@@ -72,6 +72,8 @@ END
cat > d.test << 'END'
#! /bin/sh
+test -f ./d.count && . ./d.count
+echo 1..${test_count-2}
echo ok 1 '# SKIP: who cares ...'
(. ./d.extra) || echo 'not ok 2 - d.extra failed'
: > d.run
@@ -129,7 +131,6 @@ for vpath in : false; do
test -f c.run
test -f d.run
count_test_results total=9 pass=3 fail=2 xpass=1 xfail=1 skip=1 error=1
- head -1 *.log
: Let us make b.test pass.
echo OK > b.ok
@@ -171,6 +172,7 @@ for vpath in : false; do
done
: Let us make d.test run more testcases, and experience _more_ failures.
+ echo 'test_count=9' > d.count
unindent > d.extra <<'END'
echo 'ok # SKIP s'
echo 'not ok - f 1'
@@ -179,9 +181,8 @@ for vpath in : false; do
echo 'ok # TODO xp'
echo 'not ok - f 3'
echo 'not ok - f 4'
- # Misplaced plan
- echo '1..4'
echo 'ok - p 2'
+ echo 'ok' # Extra test.
echo 'Bail out!'
END
do_recheck --fail
@@ -192,6 +193,7 @@ END
count_test_results total=11 pass=2 fail=4 xpass=1 xfail=0 skip=2 error=2
: Let us finally make d.test pass.
+ echo 'test_count=1' > d.count
echo : > d.extra
do_recheck --pass
test ! -r a.run
diff --git a/tests/tap-signal.test b/tests/tap-signal.test
new file mode 100755
index 0000000..b1ac2df
--- /dev/null
+++ b/tests/tap-signal.test
@@ -0,0 +1,65 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - a test script terminated by a signal causes an hard error
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TESTS =
+END
+
+for sig in 1 2 13 15; do
+ unindent > signal-$sig.test <<END
+ #!/bin/sh
+ echo 1..1
+ echo ok 1
+ kill -$sig \$\$
+ echo "Bail out! \$0 not killed?"
+END
+ echo TESTS += signal-$sig.test >> Makefile.am
+done
+
+chmod a+x *.test
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+for append in '' 'TEST_LOG_DRIVER_FLAGS = --ignore-exit'; do
+ echo "$append" >> Makefile
+ $MAKE check >stdout && { cat stdout; Exit 1; }
+ cat stdout
+ count_test_results total=8 pass=4 fail=0 xpass=0 xfail=0 skip=0 error=4
+ for sig in 1 2 13 15; do
+ grep "^ERROR: signal-$sig\\.test - terminated by signal.*" stdout
+ done
+done
+
+:
diff --git a/tests/tap-skip-whole-whitespace.test
b/tests/tap-skip-whole-whitespace.test
new file mode 100755
index 0000000..2a19d4c
--- /dev/null
+++ b/tests/tap-skip-whole-whitespace.test
@@ -0,0 +1,68 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - normalization of whitespace in console testsuite progress associated
+# with a SKIP directive in the TAP plan
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+sp=' '
+tab=' '
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = foo.test bar.test baz.test
+END
+
+cat > foo.test <<END
+1..0${tab}${tab} #${tab}SKIP ${tab}Strip leading & trailing ${tab}${tab}
+END
+
+cat > bar.test <<END
+1..0 #SKIP Preserve ${tab} middle${tab}${tab}space
+END
+
+cat > baz.test <<END
+1..0 # SKIP${tab} Strip${tab}external preserve ${tab}middle ${tab}${sp}
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check > stdout || { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=3 pass=0 fail=0 error=0 xpass=0 xfail=0 skip=3
+
+grep "SKIP: foo\\.test - Strip leading & trailing$" stdout
+grep "SKIP: bar\\.test - Preserve ${tab} middle${tab}${tab}space" stdout
+grep "SKIP: baz\\.test - Strip${tab}external preserve ${tab}middle" stdout
+
+:
diff --git a/tests/tap-todo-skip-whitespace.test
b/tests/tap-todo-skip-whitespace.test
new file mode 100755
index 0000000..68fbbc1
--- /dev/null
+++ b/tests/tap-todo-skip-whitespace.test
@@ -0,0 +1,118 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - normalization of whitespace in console testsuite progress associated
+# with TODO and SKIP directives
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = all.test
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+cat > stub.tap <<END
+1 # TODO
+2 # TODO foo?
+3 # TODO: bar!
+4 aa # TODO
+5 bb # TODO fnord 5
+6 cc # TODO:${tab}fnord 6
+7 - x # TODO
+8 - y # TODO fnord $tab 8
+9 - z # TODO: fnord 9 $tab
+10# TODO x0
+11$tab# TODO x1
+12 $tab$tab # TODO x2
+13 asd# TODO x3
+14 sad$tab# TODO x4
+15 das$tab$tab # TODO x5
+END
+
+cat > stub.exp <<END
+1 # TODO
+2 # TODO foo?
+3 # TODO: bar!
+4 aa # TODO
+5 bb # TODO fnord 5
+6 cc # TODO:${tab}fnord 6
+7 - x # TODO
+8 - y # TODO fnord $tab 8
+9 - z # TODO: fnord 9
+10 # TODO x0
+11 # TODO x1
+12 # TODO x2
+13 asd # TODO x3
+14 sad # TODO x4
+15 das # TODO x5
+END
+
+plan=1..15
+
+my_make_check ()
+{
+ xpass=0 xfail=0 skip=0
+ case $1 in
+ xpass|xfail|skip) eval $1=15;;
+ *) fatal_ "bad argument '$1' for my_make_check";;
+ esac
+ cat all.test
+ $MAKE check >stdout || : # Don't care about the exit status in this test.
+ cat stdout
+ count_test_results total=15 pass=0 fail=0 error=0 \
+ xpass=$xpass xfail=$xfail skip=$skip
+ # Don't be too strict w.r.t. possible normalization of "TODO: foo" into
+ # "TODO : foo" (as is done by, e.g., the 'TAP::Parser' perl module).
+ LC_ALL=C grep '^[A-Z][A-Z]*:' stdout \
+ | sed -e 's/# \(TODO\|SKIP\) *:/# \1:/' > got
+ cat exp
+ cat got
+ diff exp got
+}
+
+# For "TODO" directives leading to XPASS results.
+(echo $plan && sed -e 's/^/ok /' stub.tap) > all.test
+sed -e 's/^/XPASS: all.test /' stub.exp > exp
+my_make_check xpass
+
+# For "TODO" directives leading to XFAIL results.
+(echo $plan && sed -e 's/^/not ok /' stub.tap) > all.test
+sed -e 's/^/XFAIL: all.test /' stub.exp > exp
+my_make_check xfail
+
+# For "SKIP" directives.
+(echo $plan && sed -e 's/^/ok /' -e 's/TODO/SKIP/' stub.tap) > all.test
+sed -e 's/TODO/SKIP/' -e 's/^/SKIP: all.test /' stub.exp > exp
+my_make_check skip
+
+:
diff --git a/tests/tap-todo-skip.test b/tests/tap-todo-skip.test
index f82da7b..c37b631 100755
--- a/tests/tap-todo-skip.test
+++ b/tests/tap-todo-skip.test
@@ -106,72 +106,4 @@ cat stdout
count_test_results total=9 pass=9 fail=0 xpass=0 xfail=0 skip=0 error=0
-# --------------------------------------------------------------------- #
-# Some more ad-hoc checks, also verifying that the testsuite progress #
-# output on console behaves as expected. #
-# --------------------------------------------------------------------- #
-
-cat > stub <<END
-1 # TODO
-2 # TODO foo?
-3 # TODO: bar!
-4 aa # TODO
-5 bb # TODO fnord 5
-6 cc # TODO:${tab}fnord 6
-7 - x # TODO
-8 - y # TODO fnord $tab 8
-9 - z # TODO: fnord 9 $tab
-10# TODO 0
-11$tab# TODO 1
-12 $tab$tab # TODO 2
-13 asd# TODO 3
-14 sad$tab# TODO 4
-15 das$tab$tab # TODO 5
-END
-
-# For "TODO" directives leading to XPASS results.
-
-sed -e 's/^/ok /' stub > all.test
-sed -e 's/^/XPASS: all.test /' stub > exp
-
-cat all.test
-$MAKE check >stdout && { cat stdout; Exit 1; }
-cat stdout
-count_test_results total=15 pass=0 fail=0 xpass=15 xfail=0 skip=0 error=0
-
-LC_ALL=C grep '^[A-Z][A-Z]*:' stdout > got
-cat exp
-cat got
-#diff exp got #XXX
-
-# For "TODO" directives leading to XFAIL results.
-
-sed -e 's/^/not ok /' stub > all.test
-sed -e 's/^/XFAIL: all.test /' stub > exp
-
-cat all.test
-$MAKE check >stdout || { cat stdout; Exit 1; }
-cat stdout
-count_test_results total=15 pass=0 fail=0 xpass=0 xfail=15 skip=0 error=0
-
-LC_ALL=C grep '^[A-Z][A-Z]*:' stdout > got
-cat exp
-cat got
-#diff exp got # XXX
-
-# For "SKIP" directives.
-
-sed -e 's/^/ok /' -e 's/TODO/SKIP/' stub > all.test
-sed 's/^/SKIP: all.test /' stub > exp
-
-cat all.test
-$MAKE check >stdout || { cat stdout; Exit 1; }
-cat stdout
-count_test_results total=15 pass=0 fail=0 xpass=0 xfail=0 skip=15 error=0
-
-LC_ALL=C grep '^[A-Z][A-Z]*:' stdout > got
-cat exp
-cat got
-#diff exp got # XXX
-
:
diff --git a/tests/tap-unplanned.test b/tests/tap-unplanned.test
new file mode 100755
index 0000000..696f73b
--- /dev/null
+++ b/tests/tap-unplanned.test
@@ -0,0 +1,177 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - unplanned tests are properly reported as errors
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+TESTS = foo.test
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+cat > foo.test <<END
+1..1
+ok 1
+ok 2
+END
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=3 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=2
+grep '^ERROR: foo\.test - too many tests run (expected 1, got 2)$' stdout
+grep '^ERROR: foo\.test 2 # UNPLANNED$' stdout
+
+cat > foo.test <<END
+1..2
+ok 1
+ok 2
+ok 3
+END
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=4 pass=2 fail=0 xpass=0 xfail=0 skip=0 error=2
+grep '^ERROR: foo\.test - too many tests run (expected 2, got 3)$' stdout
+grep '^ERROR: foo\.test 3 # UNPLANNED$' stdout
+
+# Interaction with XFAIL_TESTS.
+cat > foo.test <<END
+1..2
+not ok 1
+ok 2 # SKIP
+ok 3
+not ok 4
+ok 5 # SKIP
+END
+XFAIL_TESTS=foo.test $MAKE -e check >stdout && { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=6 pass=0 fail=0 xpass=0 xfail=1 skip=1 error=4
+grep '^ERROR: foo\.test - too many tests run (expected 2, got 5)$' stdout
+grep '^ERROR: foo\.test 3 # UNPLANNED$' stdout
+grep '^ERROR: foo\.test 4 # UNPLANNED$' stdout
+grep '^ERROR: foo\.test 5 # UNPLANNED$' stdout
+
+cat > foo.test <<END
+1..1
+ok 1
+
+ok
+ok 3
+ok foo
+ok 5 - bar bar
+
+not ok
+not ok 7
+not ok foo
+not ok 9 - bar bar
+
+ok # TODO
+ok 11 # TODO
+ok foo # TODO
+ok 13 - bar bar # TODO
+
+not ok # TODO
+not ok 15 # TODO
+not ok foo # TODO
+not ok 17 - bar bar # TODO
+
+ok # SKIP
+ok 19 # SKIP
+ok foo # SKIP
+ok 21 - bar bar # SKIP
+END
+
+cat > t <<END
+
+2
+3
+4 foo
+5 - bar bar
+
+6
+7
+8 foo
+9 - bar bar
+
+10
+11
+12 foo
+13 - bar bar
+
+14
+15
+16 foo
+17 - bar bar
+
+18
+19
+20 foo
+21 - bar bar
+
+END
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=22 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=21
+
+echo 'PASS: foo.test 1' > exp
+sed -e '/^$/d' -e 's/.*/ERROR: foo.test & # UNPLANNED/' t >> exp
+echo 'ERROR: foo.test - too many tests run (expected 1, got 21)' >> exp
+
+$FGREP ': foo.test' stdout > got
+
+cat exp
+cat got
+diff exp got
+
+# Note that, if the TAP input has a trailing plan, it is not possible
+# to flag unplanned tests as such, since we do not know they're unplanned
+# until the plan is reached; still, we should give at least an error
+# message about the unmatched number of tests once we've got the plan.
+
+for x in 'ok' 'ok 3' 'not ok' 'not ok # TODO' 'ok # TODO' 'ok # SKIP'; do
+ unindent > foo.test <<END
+ ok 1
+ ok 2 # SKIP
+ $x
+ 1..2
+END
+ $MAKE check >stdout && { cat stdout; Exit 1; }
+ cat stdout
+ test `$FGREP -c ': foo.test' stdout` -eq 4
+ $EGREP '^PASS: foo\.test 1($| )' stdout
+ $EGREP '^SKIP: foo\.test 2($| )' stdout
+ $EGREP ': foo\.test 3($| )' stdout
+ grep '^ERROR: foo\.test - too many tests run (expected 2, got 3)$' stdout
+done
+
+:
diff --git a/tests/tap-whitespace-normalization.test
b/tests/tap-whitespace-normalization.test
index 4a99e15..16e6801 100755
--- a/tests/tap-whitespace-normalization.test
+++ b/tests/tap-whitespace-normalization.test
@@ -18,6 +18,7 @@
# progress output on console. We keep all these checks here in a single
# script so that a potential cosmetic change in the output format won't
# force us to tweak dozens of other tests (hopefully).
+# See also related test 'tap-todo-skip-whitespace.test'.
parallel_tests=yes
. ./defs || Exit 1
@@ -38,21 +39,60 @@ END
: > exp
-# Just to be sure.
sp=' '
tab=' '
+spaces_a=${sp}${tab}${tab}${sp}${sp}${tab}
+spaces_b=${tab}${tab}${sp}${tab}${sp}${sp}${sp}
+
#-----------------------------------------------------------------------
+echo TESTS += numbers.test >> Makefile.am
+
cat > numbers.test <<END
-1..20
-ok${tab}1
-ok${sp}${sp}2
-ok${sp}${sp}${tab}${sp}${tab}${tab}3
-ok 4${sp}
-ok 5${tab}
-ok 6${sp}${sp}
-ok ${tab} ${tab}${tab}3
+1..6
+ok${spaces_a}1
+ok 2${spaces_b}
+ok${spaces_a}3${spaces_b}
+not ok${spaces_b}4
+not ok 5${spaces_a}
+not ok${spaces_b}6${spaces_a}
+END
+
+cat >> exp <<END
+PASS: numbers.test 1
+PASS: numbers.test 2
+PASS: numbers.test 3
+FAIL: numbers.test 4
+FAIL: numbers.test 5
+FAIL: numbers.test 6
+END
+
+#-----------------------------------------------------------------------
+
+echo TESTS += description.test >> Makefile.am
+
+cat > description.test <<END
+1..8
+ok${spaces_a}+foo
+ok +bar${spaces_b}
+ok${spaces_a}+baz${spaces_b}
+not ok${spaces_b}-foo
+not ok -bar${spaces_a}
+not ok${spaces_b}-baz${spaces_a}
+ok u${spaces_b}v${spaces_a}w${sp}
+not ok${spaces_a}x${spaces_a}y${tab}z${tab}
+END
+
+cat >> exp <<END
+PASS: description.test 1 +foo
+PASS: description.test 2 +bar
+PASS: description.test 3 +baz
+FAIL: description.test 4 -foo
+FAIL: description.test 5 -bar
+FAIL: description.test 6 -baz
+PASS: description.test 7 u${spaces_b}v${spaces_a}w
+FAIL: description.test 8 x${spaces_a}y${tab}z
END
#-----------------------------------------------------------------------
@@ -63,7 +103,7 @@ echo TESTS += bailout.test >> Makefile.am
cat > bailout.test <<END
1..1
-Bail out!${tab} ${tab}We're out of disk space.
+Bail out!${tab}${sp}${sp}${tab}We're out of disk space.
ok 1
END
@@ -111,48 +151,12 @@ END
#-----------------------------------------------------------------------
-# Skip whole test (from TAP plan).
-
-echo TESTS += skipall.test >> Makefile.am
-echo TESTS += skipall2.test >> Makefile.am
-
-cat > skipall.test <<END
-1..0${tab}${tab} #${tab} ${tab}Strip trailing & leading ${tab}${tab}
-ok 1
-END
-
-cat > mu.test <<END
-1..0 #foo $weirdchars
-# TAP errors in here shouldn't be seen, as the test is skipped.
-1..2
-ok 1
-ok 22
-not ok 333
-1..7
-END
-
-chmod a+x all.test
-
-$ACLOCAL
-$AUTOCONF
-$AUTOMAKE
-
-./configure
-
-$MAKE check >stdout || { cat stdout; Exit 1; }
-cat stdout
-
-count_test_results total=4 pass=0 fail=0 xpass=0 xfail=0 skip=4 error=0
-
-grep '^SKIP: foo\.test$' stdout
-grep '^SKIP: bar\.test$' stdout
-grep '^SKIP: wget\.test - wget(1) not installed$' stdout
-grep "^SKIP: curl\\.test #${tab} ${tab}Can't connect to gnu\\.org!
${tab}${tab}$" stdout
-grep "^SKIP: mu\\.test - foo $weirdchars$" stdout
-test `grep -c ': .*\.test' stdout` -eq 5
+# TODO: we should have more checks here ... (but let's not over-do FTM).
#-----------------------------------------------------------------------
+chmod a+x *.test
+
$ACLOCAL
$AUTOCONF
$AUTOMAKE
@@ -174,6 +178,4 @@ cat exp
cat got
diff exp got
-#Exit 1 # TODO finish it
-
:
diff --git a/tests/tap-with-and-without-number.test
b/tests/tap-with-and-without-number.test
index fac10dc..a1f7a4b 100755
--- a/tests/tap-with-and-without-number.test
+++ b/tests/tap-with-and-without-number.test
@@ -16,6 +16,8 @@
# TAP support:
# - numbered and unnumbered test results can coexist
+# - tests without explicit number get automatically numbered in the
+# testsuite progress output on console
parallel_tests=yes
. ./defs || Exit 1
@@ -40,8 +42,8 @@ ok 1 foo
ok 2 - foo2
ok - bar
not ok baz # TODO
-not ok 4 - asd # TODO
-ok 5 # SKIP
+not ok 5 - asd # TODO
+ok 6 # SKIP
ok zardoz
END
@@ -55,4 +57,12 @@ $MAKE check >stdout || { cat stdout; Exit 1; }
cat stdout
count_test_results total=7 pass=4 fail=0 xpass=0 xfail=2 skip=1 error=0
+grep '^PASS: all\.test 1 foo$' stdout
+grep '^PASS: all\.test 2 - foo2$' stdout
+grep '^PASS: all\.test 3 - bar$' stdout
+grep '^XFAIL: all\.test 4 baz # TODO$' stdout
+grep '^XFAIL: all\.test 5 - asd # TODO$' stdout
+grep '^SKIP: all\.test 6 # SKIP$' stdout
+grep '^PASS: all\.test 7 zardoz$' stdout
+
:
diff --git a/tests/tap-xfail-tests.test b/tests/tap-xfail-tests.test
index 82a39e9..f159e8a 100755
--- a/tests/tap-xfail-tests.test
+++ b/tests/tap-xfail-tests.test
@@ -67,8 +67,6 @@ grep '^SKIP: the\.test 5 # SKIP' stdout
grep '^XFAIL: the\.test 6 # SKIP' stdout
grep '^ERROR: the\.test - Bail out!' stdout
-test `$FGREP -c ': the.test' stdout` -eq 7
-
# Check that the exit status of the testsuite is indeed 0 when we
# would expect success.