Re: [Axiom-developer] unit testing

[Waldek Hebisch]

> Dear all,
> 
> I'm discussing with Christian Aistleitner unit testing in Axiom, and I
> wondered, how the regression tests are currently run. I only found the .input
> files, but cannot find any files containing the corresponding "should be"
> output.
> 
> Furthermore, how is "should be" and "is" currently compared?
> 

The procedure I use for wh-sandbox is as follows.  First, when a change
is developed I use ad hoc methods to verify that the change is having
intended effect typically going trough many iterations.  When I am
satisfied with change I do a full build and compare test output and
generated .pht pages.  For recent changes to algebra bootstrap and to
interpreter (switch from boot to Shoe) I also compared generated Lisp.

I do not consider any version as "should be" -- I simply monitor changes.
I compare with previous versions, typically with the preceeding one,
but sometimes also with older versions.

I compare .pht pages just using diff.  For .output files I use a script
that I recently posted.  For lisp files I have a special Lisp comparator
program (also recently posted).

I tried to use random expressions for testing but there are few problems
- my first attempt generated a lot of invalid expressions (like divisions
  by zero or type errors)
- the expressions have bad distribution, many are quite trivial but
  the other quickly get too complicated for Axiom to handle
- test runs tend to get trapped in infinite or semi-infinite computations
  or run out of memory (timeouts could help, but some examples _do_ get
  correct answer only after hours of computations)
- big problem is how to find correct answer to compare with test result.
  For indefinite integrals we can just integrate derivatives.  For
  definite integrals (and possibly some other expressions) we may
  compare numerical evaluation of symbolic result with result of 
  numerical integration.  Another possibility is are multi-CAS
  competitions:  compare outputs of different systems.

-- 
                              Waldek Hebisch
address@hidden