|
From: | Oliver Heimlich |
Subject: | Re: @example's as doctests |
Date: | Fri, 27 Mar 2015 22:00:42 +0100 |
User-agent: | Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Icedove/31.5.0 |
Am 27.03.2015 um 10:39 schrieb Colin Macdonald:
Oliver (and octave-maintainers), I've hacked doctest-for-matlab enough [1] to run it on the infsup class from your Interval package. Interval uses the standard Octave @result{} convention: in my patched version, I split on @result{} and some other heuristics. It could be improved, see last bullet below. [1] https://github.com/catch22/doctest-for-matlab/pull/10 I attach a log: most tests pass. Oliver: I'm impressed!
Colin,thanks for sharing the results. The interval package probably is an easy target for your project. Most examples are syntactically very similar and not very complicated. Any output from this package should be system independent. Additionally the @example blocks in the package did not have a lot of time to age and become wrong. ;-)
I noticed: * there are a few actual errors.
I found only one error with ldivide. The documentation string calls mldivide by accident (bug 1) and your test happens to reveal a problem in mldivide (bug 2). Lucky strike!
* there are some cases of @example used as a LaTeX alternative (is there an @verbatim instead?)
Yes, the @example blocks have been used for ASCII art and do not contain actual examples. The @verbatim should be a valid replacement (without indentation). I am going to change the documentation.
* there a few minor (cosmetic?) errors: [-1, 5] to [-1, +5]
I would consider these actual errors and am going to fix the documentation. A very early version of the package had a different format for the output of signs.
* there are a couple cases were my current splitting input/output based on @result{} fails. (Personally I prefer marking input with ">>" but the doctest code could instead be refactored further.)
I found several problems with doctest:intervaltotext: The @example block contains 3 consecutive inputs and outputs. Without the missing “>>” for inputs there is a problem with seperating them. Would it be sufficient to use three @group blocks in the @example block?
eq, nextout: The output is boolean, but the documentation string shows the console output as ans = 0 (or 1). I can change the documentation to @result{} False (or True).
mulrev: The function returns two return values and the syntax used in the @example is probably wrong for that purpose.
Finally. I checked several functions in core: spdiags and svd are fairly close to testable. qr, chol not really written with this sort of tool in mind. IIRC, others warned me this was true of core Octave. Colin
I plan to improve the performance of console output for my package. I can put your work into great use during regression testing for that change.
Following our discussion in February, I have implemented all @example blocks in the package as %!test blocks. They check the return value semantically and can be run by any user. However, doctest also checks the @example blocks syntactically, which provides additional value. And as the ldivide example tought us: The %!test blocks are not guaranteed to be perfectly synchronized with their @example blocks!
Oliver
On 02/02/15 18:23, Oliver Heimlich wrote:Am 02.02.2015 um 18:42 schrieb Colin Macdonald:I've recently learned how to use address@hidden to put the expected result into the texinfo docs of my functions. (Thanks to "interval" pkg dev!) Is there any existing way to run these as tests? Like doctest in Python? I.e., extract them, run them, make sure the output matches the address@hidden The documentation refers to "example" but that seems to pull in `%!demo` blocks instead.The @result{} is just a macro that produces the ⇒ symbol in the documentation. I find it useful for short examples in the documentation. I copy the console output into the @example, replacing “$variable = ” with the macro. As a package maintainer I prefer to have these examples as test cases, just to make sure that the documentation is correct. That's why I like your proposal. The simplest solution IMHO is to add a %!test block for examples used in the documentation manually. It is complicated to check the @examples automatically, you must consider: - The @example blocks must not follow any particular syntax, nor must they contain useful code, - the author of the example blocks may have abreviated commands or console output, - if you wanted to check the console output of the examples, you would have to intercept the stdout, - some results may be system-dependent, - …I expect the answer will be "no, but please write such a thing"... :(I'd find it useful to code simple %!test blocks, which will be used to generate additional @example blocks at the end of the documentation. Perhaps introduce a new %!example block?! However, automating this is probably not worth it, since this only makes sense for short code snippets and the examples in the documentation usually do not change often.
[Prev in Thread] | Current Thread | [Next in Thread] |