automake-patches
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [PATCH 1/5] {test-protocols} parallel-tests: make parsing of test re


From: Stefano Lattarini
Subject: Re: [PATCH 1/5] {test-protocols} parallel-tests: make parsing of test results safer
Date: Thu, 21 Jul 2011 20:17:57 +0200
User-agent: KMail/1.13.3 (Linux/2.6.30-2-686; KDE/4.4.4; i686; ; )

On Wednesday 20 July 2011, Ralf Wildenhues wrote:
> * Stefano Lattarini wrote on Wed, Jul 20, 2011 at 10:05:14PM CEST:
> > Hi Ralf, thanks for the tips.
> 
> Rather, 5 minutes of searching the web.  But sure, anytime.  ;-)
> 
> > On Tuesday 19 July 2011, Ralf Wildenhues wrote:
> > > Test suite environments:
> > > any that use the above
> > >
> > I see now that prove(1), the default command line interface to TAP::*
> > modules, has a `--state' option that allow it to store the results of
> > the previous testsuite run.  But the exact way this is done seems to
> > be an internal detail; should I look into this nonetheless?
> 
> Why do you need to ask?  Wasn't one of the first parts of this project
> looking into how other test environments work?
>
Yes, but had overlooked and/or glissed over this feature of prove because
it didn't seem relevant at the time.  And similarly for other features too
BTW, such as `--archive', `--shuffle', `--recurse', passing arguments
to tests using `::' ...  And similarly for other programs and testing
environments.

> > > dejagnu
> > >
> > After having read the blog post you link below, I'd rather not look into
> > this at all ...
> 
> Learning from real-world software, including mistakes, can be a lot more
> enlightening than from shiny but unused software.
>

> > > qmtest
> > >
> > Just as an aside: the documentation of this tool seem to employ the same
> > terminology we are using in automake w.r.t. test results (in particular,
> > it uses "FAIL" and "ERROR" in the same way we do, making clear distinction
> > between the two); so we've probably managed to avoid NIH in this respect.
> 
> No.  Avoiding NIH means not doing your own thing in the first place,
> but doing the research first, and then acting intelligently afterwards.
>
I think I've already pointed out that, for the distinction between FAILs
and ERRORs, I took inspiration from the Python and xUnitw worlds (and to
a lesser degree from TAP).

> > > http://en.wikipedia.org/wiki/Portal:Software_Testing lists several
> 
> There are probably a dozen more in this web page alone.  I don't think
> it is asked too much to at least look around in them in search for
> valuable data for all kinds of things (future features you don't want to
> prevent being possible by wrong design decisions now).  Even much more
> generally than just for the result format.
> 
> Please take this as an opportunity, not as a burden.
>
I'll take it as both ;-)

> Sometimes copying
> ideas is just the right, the best thing.  And learning from others
> almost always is.
>

The tone of you reply until here suggests that I've failed to make one
important thing about my previous answer clear: it was *absolutely not*
meant to be a definitive answer!  Of course I'll look at the pointers
you've provided (or at least some of them), and act accordingly to the
findings.  But I still think my patch should be applied, and then
improved or reworked later to better integrate with such "research
findings".

> > > Autotest
> > >
> > Hmmm... the recheck target of autotest appears to work through some
> > sort of "screen scraping" applied to the testsuite logs (not an example
> > I'm eager to follow I must say), and the format of the generated
> > `testsuite.log' itself seems quite ad-hoc, "grown" rather than designed.
> 
> Sure, hacked that in a few hours, and I'm not proud of it, but it
> works reliably enough and was fast enough.
>
> Cheers,
> Ralf
> 

Thanks,
  Stefano



reply via email to

[Prev in Thread] Current Thread [Next in Thread]