[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [PATCH 1/5] {test-protocols} parallel-tests: make parsing of test re
From: |
Stefano Lattarini |
Subject: |
Re: [PATCH 1/5] {test-protocols} parallel-tests: make parsing of test results safer |
Date: |
Thu, 21 Jul 2011 20:17:57 +0200 |
User-agent: |
KMail/1.13.3 (Linux/2.6.30-2-686; KDE/4.4.4; i686; ; ) |
On Wednesday 20 July 2011, Ralf Wildenhues wrote:
> * Stefano Lattarini wrote on Wed, Jul 20, 2011 at 10:05:14PM CEST:
> > Hi Ralf, thanks for the tips.
>
> Rather, 5 minutes of searching the web. But sure, anytime. ;-)
>
> > On Tuesday 19 July 2011, Ralf Wildenhues wrote:
> > > Test suite environments:
> > > any that use the above
> > >
> > I see now that prove(1), the default command line interface to TAP::*
> > modules, has a `--state' option that allow it to store the results of
> > the previous testsuite run. But the exact way this is done seems to
> > be an internal detail; should I look into this nonetheless?
>
> Why do you need to ask? Wasn't one of the first parts of this project
> looking into how other test environments work?
>
Yes, but had overlooked and/or glissed over this feature of prove because
it didn't seem relevant at the time. And similarly for other features too
BTW, such as `--archive', `--shuffle', `--recurse', passing arguments
to tests using `::' ... And similarly for other programs and testing
environments.
> > > dejagnu
> > >
> > After having read the blog post you link below, I'd rather not look into
> > this at all ...
>
> Learning from real-world software, including mistakes, can be a lot more
> enlightening than from shiny but unused software.
>
> > > qmtest
> > >
> > Just as an aside: the documentation of this tool seem to employ the same
> > terminology we are using in automake w.r.t. test results (in particular,
> > it uses "FAIL" and "ERROR" in the same way we do, making clear distinction
> > between the two); so we've probably managed to avoid NIH in this respect.
>
> No. Avoiding NIH means not doing your own thing in the first place,
> but doing the research first, and then acting intelligently afterwards.
>
I think I've already pointed out that, for the distinction between FAILs
and ERRORs, I took inspiration from the Python and xUnitw worlds (and to
a lesser degree from TAP).
> > > http://en.wikipedia.org/wiki/Portal:Software_Testing lists several
>
> There are probably a dozen more in this web page alone. I don't think
> it is asked too much to at least look around in them in search for
> valuable data for all kinds of things (future features you don't want to
> prevent being possible by wrong design decisions now). Even much more
> generally than just for the result format.
>
> Please take this as an opportunity, not as a burden.
>
I'll take it as both ;-)
> Sometimes copying
> ideas is just the right, the best thing. And learning from others
> almost always is.
>
The tone of you reply until here suggests that I've failed to make one
important thing about my previous answer clear: it was *absolutely not*
meant to be a definitive answer! Of course I'll look at the pointers
you've provided (or at least some of them), and act accordingly to the
findings. But I still think my patch should be applied, and then
improved or reworked later to better integrate with such "research
findings".
> > > Autotest
> > >
> > Hmmm... the recheck target of autotest appears to work through some
> > sort of "screen scraping" applied to the testsuite logs (not an example
> > I'm eager to follow I must say), and the format of the generated
> > `testsuite.log' itself seems quite ad-hoc, "grown" rather than designed.
>
> Sure, hacked that in a few hours, and I'm not proud of it, but it
> works reliably enough and was fast enough.
>
> Cheers,
> Ralf
>
Thanks,
Stefano
[PATCH 2/5] {test-protocols} parallel-tests: new recognized test result 'ERROR', Stefano Lattarini, 2011/07/14
[PATCH 3/5] {test-protocols} parallel-tests: simplify testsuite summary, Stefano Lattarini, 2011/07/14
[PATCH 4/5] {test-protocols} tests defs: new auxiliary function 'count_test_results', Stefano Lattarini, 2011/07/14