[Axiom-developer] (no subject)

[daly]

Martin, Waldek,

I've attached the regress.lisp.pamphlet file. 

  notangle -t8 regress.lisp.pamphlet >regress.lisp

should extract the lisp code. If you add this file to your system
and add the following stanzas to your Makefile you can install the
regression test files. I've been converting the standard input files
to regression test format. The details are in the regress.dvi file.

The regression test files are in the src/input directory in the
trunk on sourceforge SVN.



The related Makefile stanzas are:

%.input: ${IN}/%.input.pamphlet
        @ echo generic 16 making ${MID}/$*.input from ${IN}/$*.input.pamphlet
        @(cd ${MID} ; \
          ${TANGLE} ${IN}/$*.input.pamphlet >$*.input )

%.regress: %.input
        @ echo regression testing $*
        @ rm -f $*.output
        @ echo ')read $*.input' | ${TESTSYS} 
        @ rm $*.input
        @ echo ')lisp (regress "$*.output")' | ${TESTSYS} \
                | egrep -v '(Timestamp|Version)' | tee $*.regress


The first stanza extracts the input file.
The second stanza runs the files listed in the REGRES variable
of the form 

REGRESS=foo.regress

which expects a foo.input, which is generated from the first stanza.
The current list of regression test files is:



REGRES= algaggr.regress algbrbf.regress  algfacob.regress alist.regress  \
    allfact.regress   antoine.regress \
    arith.regress     array1.regress   array2.regress \
    arrows.regress    assign.regress   atansqrt.regress \
    bags.regress      bbtree.regress \
    binary.regress    bop.regress      bstree.regress   bouquet.regress \
    bug10069.regress \
    bugs.regress      bug10312.regress bug6357.regress  bug9057.regress \
    calculus2.regress calculus.regress cardinal.regress card.regress \
    carten.regress    cclass.regress   char.regress     ch.regress \
    clifford.regress  clif.regress     coercels.regress collect.regress \
    complex.regress   conformal.regress \
    constant.regress  contfrac.regress contfrc.regress \
    curl.regress      cycles1.regress  cycles.regress   cyfactor.regress \
    danzwill.regress  decimal.regress  defintef.regress defintrf.regress \
    derham.regress    dfloat.regress   dhtri.regress    divisor.regress \
    dmp.regress       dpol.regress     easter.regress   efi.regress \
    eigen.regress     elemfun.regress  elemnum.regress  elfuts.regress \
    elt.regress       eq.regress       eqtbl.regress    equation2.regress \
    equation.regress  evalex.regress   eval.regress     exdiff.regress \
    exint.regress     exit.regress     exlap.regress    exlimit.regress \
    expexpan.regress  explim.regress   expr1.regress    exprode.regress \
    expr.regress      exseries.regress exsum.regress    exprpoly.regress \
    farray.regress    ffdemo.regress   fferr.regress    ffx72.regress \
    fib.regress       file.regress     float1.regress   float2.regress \
    float.regress     fname1.regress   fname.regress    fnla.regress \
    fns.regress       fparfrac.regress fparfrc.regress  fr1.regress \
    fr2.regress       frac.regress     fr.regress       galois.regress \
    gbf.regress       genups.regress   gonshor.regress  grpthry.regress \
    gstbl.regress     heap.regress     heat.regress     help.regress \
    herm.regress      hexadec.regress  ico.regress      ideal.regress \
    ifact.regress     infprod.regress  intaf.regress    intdeq.regress \
    intef2.regress    intef.regress    intg0.regress    intheory.regress \
    intmix2.regress   intmix.regress   int.regress      intrf.regress \
    ipftest.regress   is.regress       isprime.regress  kafile.regress \
    kernel.regress    knot2.regress    kovacic.regress  kuipers.regress \
    laplace.regress   leg.regress      limit.regress    lindep.regress \
    list.regress      lode.regress     lodesys.regress  lodo1.regress \
    lodo2.regress     lodo3.regress    lodof.regress    lodo.regress \
    lpoly.regress     lupfact.regress  lword.regress    macbug.regress \
    macros.regress    magma.regress    mapleok.regress  mappkg1.regress \
    matbug.regress    matrix1.regress  matrix22.regress matrix.regress \
    mfinfact.regress  mkfunc.regress   mpoly.regress    mset2.regress \
    mset.regress      multfact.regress multiple.regress ndftip.regress \
    negfloats.regress nepip.regress    newlodo.regress  newton.regress \
    nlode.regress     none.regress     noonburg.regress noptip.regress \
    nqip.regress      nsfip.regress    numbers.regress  octonion.regress \
    oct.regress       ode.regress      odpol.regress    op1.regress \
    opalg.regress     operator.regress op.regress       ovar.regress \
    padic.regress     parabola.regress pascal1.regress  pascal.regress \
    patmatch.regress  pat.regress      perman.regress   perm.regress \
    pfr1.regress      pfr.regress      page.regress     pmint.regress \
    poly1.regress     polycoer.regress poly.regress     psgenfcn.regress \
    quat1.regress     quat.regress     r20abugs.regress r20bugs.regress \
    r21bugsbig.regress r21bugs.regress radff.regress    radix.regress \
    realclos.regress  reclos.regress   repa6.regress    robidoux.regress \
    roman.regress     roots.regress    ruleset.regress  rules.regress \
    scherk.regress    scope.regress    segbind.regress  seg.regress \
    series2.regress   series.regress   sersolve.regress set.regress \
    sinCosEx.regress  sint.regress     skew.regress     slowint.regress \
    solvetra.regress  space3.regress   sqmatrix.regress sregset.regress \
    stbl.regress      stream2.regress  stream.regress   streams.regress \
    string.regress    strtbl.regress   symbol.regress   t111293.regress \
    table.regress     tanatan.regress  textfile.regress torus.regress \
    triglim.regress   tsetcatvermeer.regress            tutchap1.regress \
    void.regress      uniseg.regress



Tim

==========================================================================
src/interp/regress.lisp.pamphlet
==========================================================================
\documentclass{article}
\usepackage{axiom}
\begin{document}
\title{\$SPAD/src/interp regress.lisp}
\author{Timothy Daly}
\maketitle
\begin{abstract}
This is the regression test mechanism. The input files have been
rewritten to have a standard structure. This fixed format identifies
the tests within a file. Each test is run and any mismatch between
the actual and expected results is reported.
\end{abstract}
\eject
\tableofcontents
\eject
\section{Regress}
In order to regression test axiom results we created a standard
file format. This format has 3 kinds of markers:
\begin{itemize}
\item ``--S'' marker which must have a integer test number
\item ``--R'' marker lines, one per expected output from axiom
\item ``--E'' marker which has an integer matching the preceeding ``--S''
\item ``--I'' marker ignores the line, useful for gensyms and random
\end{itemize}
Because these markers use Axiom's standard comment prefix they
are valid lines in input files and are ignored by the ``)read''
command. They are simply copied to the output file. This allows
us to include the expected output in the output file so we can
compare what Axiom computes with what it should compute.

To create these regression files all you need to do is create an
input file and run it through Axiom. Then, for each test case in
the file you mark it up by putting a ``--S number'' {\bf before}
the Axiom input line. You put ``--R'' prefixes on each line of
Axiom output, including the blank lines. Then you put a ``--E number''
line after the last output line, usually the {\tt Type:} line.
This newly marked-up input file is now a regression test.

To actually run the regression test you simply include the
marked up the input file in the {\tt src/input} subdirectory.
This file will automatically be run at build time and any failing
tests will be marked. This code will ignore any input that does
not contain proper regression markups.

Ideally the regression test files should be pamphlet files that
explain the content and purpose of each regression test case.

Thus you run the marked-up input file {\tt foo.input}
and spool the result to {\tt foo.output} and then run the 
lisp function\\ 
{\tt (regress ``foo.output'')}

If the file does not contain proper regression markups it is
ignored. Comments or any other commands in the file that are not
surrounded by ``--S'' and ``--E'' boundaries are ignored.

\subsection{defun regress}
This function takes an output file which has been created by the
Axiom {\tt )spool} command and looks for regression test markups.
Each regression test is checked against the actual result and any
failures are marked. 
<<*>>=
(in-package "BOOT")

(defun regress (infile)
 (let (name comment test (count 0) (passed 0) (failed 0))
  (with-open-file (stream infile :direction :input)
   (setq name (getspoolname stream)) 
   (when name
    (format t "testing ~a~%" name)
    (loop
     (multiple-value-setq (comment test) (findnexttest stream))
     (unless comment (return))
     (setq count (+ count 1))
     (if (testpassed test)
      (progn
       (setq passed (+ passed 1))
       (format t "passed ~a ~a~%" name comment))
      (progn
       (setq failed (+ failed 1))
       (format t "FAILED ~a ~a~%" name comment))))
   (if (= failed 0)
    (format t "regression result passed ~a of ~a stanzas ~Tfile ~a~%"
     passed count name)
    (format t "regression result FAILED ~a of ~a stanzas ~Tfile ~a~%"
     failed count name))))))

@
\subsection{defun getspoolname}
We need to parse out the name of the test. The ``)spool'' command
writes a line into the output file containing the name of the test.
We parse out the name of the test from this line.
<<*>>=
(defun getspoolname (stream)
 (let (line point)
  (setq line (read-line stream))
  (setq point (position #\. line))
  (if (or (null point) 
      (< (length line) 30)
      (not (string= (subseq line (+ point 1) (+ point 7)) "output")))
    nil
  (subseq line 20 point))))

@
\subsection{defun findnexttest}
We need to break the file into separate test cases. This routine
looks for the ``--S'' line which indicates a test is starting. It
collects up input lines until it encounters the ``--E'' line marking
the end of the test case. These lines are returned as a list of strings.
<<*>>=
(defun findnexttest (stream)
 (let (teststart result)
  (do ((line (read-line stream nil 'done) (read-line stream nil 'done)))
      ((or (eq line 'done) (endedp line))
        (values (if line teststart) result))
   (if teststart 
    (push line result) 
    (setq teststart (testnumberp line))))))

@
\subsection{defun testnumberp}
The ``--S'' line has a test number on the line. We parse out the
test number for printing.
<<*>>=
(defun testnumberp (oneline)
 (when (startp oneline) (subseq oneline 3)))
;  (parse-integer (string-trim '(#\Space) (subseq oneline 3)))))

@
\subsection{defun testpassed}
This routine takes the test input, passes it to split to clean up
and break into two lists, and then compares the resulting lists
element by element, complaining about any mismatches. The result
is either true if everything passes or false if a mismatch occurs.

A test line can also be considered at passing if the expected line
is the string ``ignore''.
<<*>>=
(defun testpassed (test)
 (let (answer expected (passed t))
  (multiple-value-setq (answer expected) (split test))
  (dotimes (i (length answer))
   (unless 
    (or (string= (first expected) "ignore")
        (string= (first expected) (first answer)))
    (setq passed nil)
    (format t "MISMATCH~%expected:~s~%     got:~s~%" 
     (first expected) (first answer)))
   (pop answer)
   (pop expected))
  passed))

@
\subsection{defun split}
We have a list containing all of the lines in a test. The input is of
the form:
\begin{verbatim}
("--R                                         Type: List Integer"
 "--R   (1)  [1,4,2,- 6,0,3,5,4,2,3]" 
 "--R" 
 "--R "
 "                                         Type: List Integer"
 "   (1)  [1,4,2,- 6,0,3,5,4,2,3]" 
 "" 
 " "
 "l := [1,4,2,-6,0,3,5,4,2,3]")
\end{verbatim}
It removes the ``--R'' prefix from the result strings
and generates two hopefully equal-length lists, thus:
\begin{verbatim}
("                                         Type: List Integer"
 "   (1)  [1,4,2,- 6,0,3,5,4,2,3]" 
 "" 
 " ")
("                                         Type: List Integer"
 "   (1)  [1,4,2,- 6,0,3,5,4,2,3]" 
 "" 
 " "))
\end{verbatim}
Thus the first line is the start line, the second line is the Axiom 
input line, followed by the Axiom output. Then we have the lines marked
``--R'' which are the expected result. We split these into two separate
lists and throw way the lines that are the start and end lines. 

Once we
have classified all of the lines we need to throw away the input lines.
By assumption there will be more answer lines than expected lines because
the input lines are included. And given the way we process the file these
input lines are on the top of the answer stack. Since the number of
answer lines should equal the number of expected lines we pop the
stack until the numbers are equal.

Each element of the answer list should
be {\tt string=} to the corresponding element of the result list.

If the input line starts with ``--I'' we push the string ``ignore''.
This is useful for handling random results or gensym symbols.
<<*>>=
(defun split (test)
 (let (answer (acnt 0) expected (ecnt 0))
  (dolist (oneline test)
   (cond
    ((startp oneline))
    ((endedp oneline))
    ((ignorep oneline)
      (setq ecnt (+ ecnt 1))
      (push "ignore" expected))
    ((resultp oneline)
      (setq ecnt (+ ecnt 1))
      (push (subseq oneline 3) expected))
    (t 
      (setq acnt (+ acnt 1))
      (push oneline answer))))
  (dotimes (i (- acnt ecnt)) (pop answer))
  (values (nreverse answer) (nreverse expected))))

@
\subsection{defun startp}
This test returns true if we have a ``start'' line. That is, a line
with a ``--S'' prefix.
<<*>>=
(defun startp (oneline)
 (and (>= (length oneline) 3) (string= (subseq oneline 0 3) "--S")))

@
\subsection{defun endedp}
This test returns true if we have a ``ended'' line. That is, a line
with a ``--E'' prefix.
<<*>>=
(defun endedp (oneline)
 (and (>= (length oneline) 3) (string= (subseq oneline 0 3) "--E")))

@
\subsection{defun resultp}
This test returns true if we have a ``results'' line. That is, a line
with a ``--R'' prefix.
<<*>>=
(defun resultp (oneline)
 (and (>= (length oneline) 3) (string= (subseq oneline 0 3) "--R")))

@
\subsection{defun ignorep}
This test returns true if we have an ``ignore'' line. That is, a line
with a ``--I'' prefix.
<<*>>=
(defun ignorep (oneline)
 (and (>= (length oneline) 3) (string= (subseq oneline 0 3) "--I")))

@
\eject
\begin{thebibliography}{99}
\bibitem{1} nothing
\end{thebibliography}
\end{document}