test scripts: alternative syntax

[Paul Kienzle]

Here is another proposal for syntax within test scripts.  It uses line-based
blocking rather than parenthesis based blocking, which virtually
eliminates block-type errors.  Plus bonus, the counterintuitive:
     #<< disp('execute block even though it seems to be commented out')>>
is no longer possible.

I'm still not happy with it, but I'm throwing it out here to trigger
your creative juices and try to come up with something nicer.
Things to think about are the feasibility of emacs support and the
readability/obviousness of what is being tested.

I'm particularly unhappy with the assert syntax.  Logically, what I
want is:
   arbitrary code;
   if conditions fail
        signal assertion error;
        display value;
        if comparison condition, display what value should have been
type checking example:  
   x=f(y); 
   msg = [ "getting\n" disp(x) ];
   if !strcmp(typeof(x), type), 
      error("%stype mismatch: %s instead of %s", msg, typeof(x), type);
   endif
numeric comparison example:
   x = toeplitz([1,2,3],[1,4]);
   answer = [1,4;2,1;3,2];
   msg = [ "expecting\n" disp(answer) "getting\n" disp(x) ];
   if !strcmp(typeof(x),typeof(answer))
      error("%stype mismatch: %s instead of %s", 
            msg, typeof(x), typeof(answer));
   elseif any(sizeof(x) != sizeof(answer)
      error("%ssize mismatch: [%d,%d] instead of [%d,%d]", 
            msg, size(answer), size(x));
   elseif any(any(x!=answer))
      error("%svalue mismatch: error=%f\n", msg,norm(x-answer));
   endif
   ## comparison could also use tolerances: abs(x-answer)>tolerance
   ## or relative tolerances: abs((x-answer)./answer)>tolerance
   ## and the displayed error could use norm p=1,2,Inf,"fro",etc.
string comparison example:
   x=sprintf("%s %s", "hello", "world");
   answer = "hello world"
   msg = ["expected: " disp(x) "but got:" disp(answer)];
   if !strcmp(typeof(x),typeof(answer))
      error("%stype mismatch: %s instead of %s", msg,
            typeof(x),typeof(answer))
   elseif !strcmp(x,answer)
      error("%svalue mismatch", msg);
   endif
shape checking example:
   x=toeplitz(rand(6,1),rand(5,2));
   if !all(size(x)==[6,5])
      error("expected size [6,5] but got size [%d,%d]", rows(x),columns(x))
   endif
Are there some that I haven't thought of?

Since the primary tests for most functions will be some form of assert, it
makes sense to get it right syntactically and semantically so that the
conditions are easy to test.  As you can see from the pseudocode above,
there is far to much work required to do the tests manually each time.

Leaving aside the assert business, is the following example any more
pleasing than the one already present in test.m?  Note: %! is a continuation
line for the current block.  Can you think of a better keyword than "block"
for arbitrary code blocks which are executed even in quiet mode, but
assuming we get assert right, won't actually have to do any tests?


%usage  test                    # no args, generates usage()
%usage  test(1,2,3,4)           # too many args, generates usage()
%error  test("test", 'bogus');  # incorrect args, generates error()

%shared a                # create a shared variable
%block  a=3;             # assign to a shared variable
%assert a::3;            # variable should equal 3    
%shared b,c              # replace shared variables
%assert !exist("a")      # a no longer exists
%assert isempty(b)       # variables start off empty
%shared a,b,c            # recreate a shared variable
%assert isempty(a)       # value is empty even if it had a previous value
%block  a=1; b=2; c=3;   # give them all values
%assert [a b c]::[1 2 3] # test all of them together
%demo   # multiline demo block
%! disp("here are the values for a b and c:")
%! a, b, c               # shows their values interactively
%block  c=6;             # update a value
%demo   c                # single line demo blocks work, too
%assert [a b c]::[1 2 6] # show that the update sticks

%# this is a comment block. it can contain anything.
%#
%! it is the "#" after the block open that makes it a comment
%! and it stays as a comment even through continuation lines

%# failure tests.
%block  error("---------Failure tests.  Use test('test','verbose',1)");
%assert [a b c]::[1 3 6]   # variables have wrong values
%bogus                     # unknown block type
%usage  toeplitz([1,2,3])  # too many arguments
%block  syntax errors)     # syntax errors fail properly
%shared garbage in and garbage out
%assert 5::5::5            # wrong format for assert block
%usage  garbage            # usage on nonexistent function 
%error  syntax++error      # error test fails on syntax errors
%error "succeeds.";        # error test fails if code succeeds
%demo   with syntax error  # syntax errors in demo fail properly
%error  
%! test('/etc/passwd');
%! test("nonexistent file");
%! ## These don't signal an error, so the test for an error fails. Note 
%! ## that the call doesn't reference the current fid (it is unavailable),
%! ## so of course the informational message is not printed in the log.>>

%# an extended demo
%demo 
%! ## Speech spectrogram
%! [x Fs] = auload(file_in_path(LOADPATH,"sample.wav")); # audio file
%! step = fix(5*Fs/1000);     # one spectral slice every 5 ms
%! window = fix(40*Fs/1000);  # 40 ms data window
%! fftn = 2^nextpow2(window); # next highest power of 2
%! [S f t] = specgram(x, fftn, Fs, window, window-step);
%! S = abs(S(2:fftn*4000/Fs,:)); # magnitude in range 0<f<=4000 Hz.
%! S = S/max(max(S));         # normalize magnitude so that max is 0 dB.
%! S = max(S, 10^(-40/10));   # clip below -40 dB.
%! S = min(S, 10^(-3/10));    # clip above -3 dB.
%! imagesc(flipud(20*log10(S)), 1);
%! disp("and finally, xa speech spectrogram");