[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: run BISTs for all installed packages

From: Andrew Janke
Subject: Re: run BISTs for all installed packages
Date: Tue, 5 Mar 2019 21:39:48 -0500
User-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:60.0) Gecko/20100101 Thunderbird/60.5.1

On 3/5/19 7:18 PM, Mike Miller wrote:
On Tue, Mar 05, 2019 at 10:59:01 -0500, Andrew Janke wrote:
I've put together a list of references to bugs and Octave maintainers
mailing list threads related to enhancing the test suite: 

Any major ones I'm missing?

Looks good, thank you for collecting all of these topics together.

I agree with your project goals.


I also hope you'll preserve some of the
current test suite features I think are important, including being able
to run the test suite pre-install or post-install, read-only, optionally
print full test failures in auto build or CI environments.

I'm gonna preserve ALL of these features, *ALL* of them, ha ha ha!

Though not necessarily with the same invocation syntax. :)

More seriously, my intention for Testify is to be a:
a) more-or-less strict functional superset of existing BIST functionality, with
b) nicer output and more convenient calling forms, and
c) maybe more structured control over how the BISTs are invoked and interact with each other and are collected together in summary objects

This Testify project isn't some big idea to make a whole new testing layer for Octave[1]. I mean, originally all I wanted to do was provide a concise summary output of failed tests at the end of the __run_test_suite__() output so you didn't have to scroll back and copy & paste multiple times to get a list of your failed tests (https://savannah.gnu.org/bugs/index.php?55522). But all this code is intertwingled, and one thing led to another, so...

[1] ...although, maybe, I'll throw in a port of the xUnit testing framework, given some spare time and user interest...

If I have my own list of annoyances / wishlist things I'd like to see
improved, would you rather see them reported on savannah or on your

I'd prefer them reported on my project (one issue report per annoyance/wishlist item, please, and no annoyance too small), because a) GitHub has nicer issue/task-tracking tools than Savannah IMHO, and it'll be easier to grind them down in to separate pieces of work there, b) I'd probably consider these in-scope for this particular project, and could iterate on them in GitHub independent of the Octave project management because Testify is an independent project that's not in Octave Forge yet, and c) I think in terms of tooling, it's actually easier for us to work on an independent octave project using git/GitHub and `pkg install` tooling? Once you get to multiple commits, forks and branches are easier to manage than Hg changesets stored behind bug attachment URLs.

You should view Testify as "Andrew's wacky idea of where Octave BIST support should go in the next release or two", and if you submit a gripe/wishlist item there, I'm likely to pick it up and have an Idea and do something about it. And then hopefully eventually we can get back together with core Octave developers, find a way that a compromise version of Testify can be merged back in to Octave core, do that, and everybody benefits.

Are you aware of the 'doctest' package [1] to run tests from Octave help
strings? I think it's mostly orthogonal to your goals, just wanted to
make sure you knew it was out there.
> [1]: https://github.com/catch22/octave-doctest

Uhh, no! Thanks for the heads-up. I'll look in to it.

At first glance, this actually doesn't look *that* different from BISTs, it just has fancy ways of specifying the strings it's using as "expected" values in its assert() logic...

I also maintain a separate shell-based test suite [2] to test Octave's
command interface, command line options, exit status, and so on, also
unrelated to your project goals.

This is interesting. Looks like you're mostly concerned with the interface between the `octave` command or runtime and the external system. This is probably something I should learn about some day. :)

But at some point I'd like to see these two supported by Octave to some

[2]: https://gitlab.com/mtmiller/octave-test-suite

reply via email to

[Prev in Thread] Current Thread [Next in Thread]