autotest AT_CHECK variant with pattern awareness

[Luke Mewburn]

Hi folks,

Summary:

I propose a new macro (currently named AX_AT_CHECK_PATTERN()) which is
like AT_CHECK() except that the expected stdout and stderr arguments may
contain awk regular expressions, to allow for output where certain parts
are expected to change each run.

Any thoughts or comments about this idea?

If it was considered "useful" I wouldn't mind if it was incorporated into
the standard suite of autotest macros.

I've attached a tar file containing the macro and a test harness.


Details:

I've been using autotest for a few years.  Not long after I started
using it I had a need to be able to use AT_CHECK where certain parts of
the output were expected to change each run (e.g, timestamps).

I had experimented with saving the output of AT_CHECK (with [stdout])
and post-processing the output (e.g, using grep to filter out the lines
that we expected to change), but found that to be a bit unwieldy.

The solution used for a few years was a custom python script which
line-by-line compares the saved output against an expected pattern
file, and only failing if there was a pattern match.  This generally
sufficed, but it has the following limitations:
 - depends upon python (might not be available everywhere)
 - because of the line-by-line nature, the addition of an extra line
   of output resulted in a lot of false positive matching because
   the output and pattern checking got "out of sync".

Recently I had the idea to rework the implementation to postprocess
the output of diff, and elide from the output "change" blocks where the
left-hand output and the right-hand pattern lines matched up.
This results in much less output to wade through in testsuite.log
if there's only a couple of lines added or removed (a la the traditional
AT_CHECK output).

E.g, given file 'pattern':
        1
        2 a?
        2 [b-e]+
        3

and file 'output':
        1
        2 a
        2 b
        3
        4

then 'diff pattern output' is:
        2,3c2,3
        <       2 a?
        <       2 [b-e]+
        ---
        >       2 a
        >       2 b
        4a5
        >       4

but my diff post-processing just results in:
        4a5
        >       4

As part of the rework, I wrote the diff-postprocessing code in awk
only because I felt that it might be more available than python (or
perl), and there's other macros in autoconf and autotest that depend
upon awk.

What I've ended up with is a macro - AX_AT_CHECK_PATTERN() - which
takes arguments as per AT_CHECK() except that the expected stdout and
stderr arguments may contain awk regular expressions.
This macro depends upon a shell function ax_at_diff_pattern which
prerequisite macro defines.

The macro works by abusing the implementation of AT_CHECK() by
temporarily redefining the variable $at_diff to use my shell function
ax_at_diff_pattern().


There's still some outstanding issues:

1. The required quoting is a bit ugly if the expected regex contains
square brackets, because m4 wants to use those.

2. I'm not 100% happy with the way I temporarily change $at_diff.
Arguably the _AT_CHECK() helper could optionally take an extra argument
and do this for us instead; I'm not fussed.


regards,
Luke.

at_check_pattern-1.tar.gz
Description: GNU Zip compressed data

pgpnFiw1rlw8g.pgp
Description: PGP signature