Perl code coverage

[Ralf Wildenhues]

Code coverage can help find under-tested problem spots.  Devel::Cover
is a nice solution for Perl code.  In fact, it shows that the Autoconf
test suite can be improved quite a bit (but see the respective message
for Automake, it exercises some of the shared files better,
<http://thread.gmane.org/gmane.comp.sysutils.automake.general/11088>).

----------------------------------- ------ ------ ------ ------------  ------ 
------
File                                  stmt   bran   cond    sub   pod    time  
total
----------------------------------- ------ ------ ------ ------------  ------ 
------
...f/autoconf/lib/Autom4te/C4che.pm  100.0   75.0   66.7  100.0 100.0     0.0   
95.2
...conf/lib/Autom4te/ChannelDefs.pm   57.8   54.5    n/a   73.3 100.0     0.3   
59.4
...utoconf/lib/Autom4te/Channels.pm   66.3   53.2   57.1   71.1 100.0     0.1   
63.6
...onf/lib/Autom4te/Configure_ac.pm   96.4   87.5  100.0  100.0   0.0     0.0   
95.6
...toconf/lib/Autom4te/FileUtils.pm   84.6   59.4   90.9   84.0 100.0    54.3   
77.8
...autoconf/lib/Autom4te/General.pm   82.5   45.8   50.0   90.0 100.0     1.2   
77.3
...autoconf/lib/Autom4te/Request.pm   78.3    0.0    n/a   83.3   0.0     0.0   
74.2
.../autoconf/lib/Autom4te/Struct.pm   74.4   43.9   33.3   69.6  12.5     0.1   
63.1
...f/autoconf/lib/Autom4te/XFile.pm   82.6   45.8   14.3  100.0  55.6    24.2   
70.4
...vs/autoconf/build/bin/autoheader   92.9   78.9   53.8  100.0   n/a     0.1   
87.4
.../cvs/autoconf/build/bin/autom4te   96.2   74.2   69.0   95.7   n/a    10.9   
88.3
...vs/autoconf/build/bin/autoreconf   78.8   55.5   47.5  100.0   n/a     5.6   
69.3
.../cvs/autoconf/build/bin/autoscan   78.0   66.2   54.9   95.5   n/a     0.0   
73.6
...vs/autoconf/build/bin/autoupdate   95.7   65.8   50.0  100.0   n/a     0.2   
87.8
...d/cvs/autoconf/build/bin/ifnames  100.0   83.3   80.0  100.0   n/a     0.0   
96.1
[... automake-related files elided ...]
----------------------------------- ------ ------ ------ ------ ------ ------ 
------


Obvious followup question: how can we examine Autoconf macro coverage?

Bash/Zsh have some facilities that allow tracing shell scripts, but
that's more useful for mostly static scripts like 'libtool', and even
there, it is not simple to use effectively.  Add the fact that configure
and libtool have lots of system-dependent parts, so only a merge of the
coverage over various systems will provide complete data.

Can M4 tracing be instrumented to show coverage of expanded macros?
(I guess this ought to be fairly easy).

Can M4/M4sugar be instrumented to map shell coverage information back
from some random configure script to our input macro files?  That would
be cool, but I'd have no idea how to realize it.

You might need this additional diff for Devel::Cover to work right:

| --- a/lib/Autom4te/General.pm
| +++ b/lib/Autom4te/General.pm
| @@ -197,7 +197,7 @@ sub END
|  
|    # This is required if the code might send any output to stdout
|    # E.g., even --version or --help.  So it's best to do it unconditionally.
| -  if (! close STDOUT)
| +  if (! STDOUT->flush)
|      {
|        print STDERR "$me: closing standard output: $!\n";
|        $? = 1;


WDYT?  Please note that IMHO you shouldn't delay 2.65 for any of this.

I was going to hook cleanup to maintainer-clean, but Devel::Cover
invalidates traces anyway when the files are newer, so that wouldn't
help.

Thanks,
Ralf

    Perl coverage convenience targets.
    
    * Makefile.am (PERL_COVERAGE_DB, PERL_COVERAGE_FLAGS, PERL_COVER):
    New variables.
    (check-coverage, check-coverage-run, check-coverage-report)
    (clean-coverage): New phony targets.
    (clean-local): Depend on clean-coverage.

diff --git a/Makefile.am b/Makefile.am
index 3c90657..165eaf2 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -74,3 +74,32 @@ dist-hook:
 # TODO remove this once automake vs. AC_CONFIG_LINKS issue is fixed.
 distclean-local:
        if test x"$(VPATH)" != x ; then rm -f GNUmakefile ; fi
+
+# Perl coverage statistics.
+PERL_COVERAGE_DB = $(abs_top_builddir)/cover_db
+PERL_COVERAGE_FLAGS = 
-MDevel::Cover=-db,$(PERL_COVERAGE_DB),-silent,on,-summary,off
+PERL_COVER = cover
+
+check-coverage-run: all
+       $(mkinstalldirs) "$(PERL_COVERAGE_DB)"
+       PERL5OPT="$(PERL_COVERAGE_FLAGS)"; export PERL5OPT; \
+       $(MAKE) $(AM_MAKEFLAGS) check
+
+check-coverage-report:
+       @if test ! -d "$(PERL_COVERAGE_DB)"; then \
+         echo "No coverage database found in \`$(PERL_COVERAGE_DB)'." >&2; \
+         echo "Please run \`make check-coverage' first" >&2; \
+         exit 1; \
+       fi
+       $(PERL_COVER) $(PERL_COVER_FLAGS) "$(PERL_COVERAGE_DB)"
+
+# We don't use direct dependencies here because we'd like to be able
+# to invoke the report even after interrupted check-coverage.
+check-coverage: check-coverage-run
+       $(MAKE) $(AM_MAKEFLAGS) check-coverage-report
+
+clean-local: clean-coverage
+clean-coverage:
+       rm -rf "$(PERL_COVERAGE_DB)"
+
+.PHONY: check-coverage check-coverage-run check-coverage-report clean-coverage