RE: [lwip-devel] testing methods

[Taranowski, Thomas \(SWCOE\)]

Thanks for the replies.  I'm about where you folks are at, testing wise.
I first ping as a first-pass smoke test, if you will.  I then do some
big FTP  downloads while performing a UDP transmit stress test.  This
seems to work pretty well at capturing problems with my port, however,
we need to produce a more formal test setup for a customer, and have got
6 man-months allocated for this project.  The customer uses the BSD api,
so the tests will be on that level.  We will not be testing the RFC
compliance of the stack.  

However, one of the goals I want for this test project is to make sure
that the framework we produce can be extended to be of more general use
to the community, with the end result hopefully being some sort of test
suite that we can run against a release so we can call it good.  Making
this sort of thing happen is problematic, though, in that it's got to be
compelling and easy enough to use, otherwise everyone gets bored with it
and it just withers.  But oh well, what the hey, I'm going to try
anyway.  So, if anyone has ideas on what this should look like, I'm all
ears.  

I've done some scanning of various commercial stack testing software,
but none of them really did what we wanted.  


> -----Original Message-----
> From: address@hidden
> [mailto:address@hidden
On
> Behalf Of Kieran Mansley
> Sent: Wednesday, March 14, 2007 2:25 AM
> To: lwip-devel
> Subject: Re: [lwip-devel] testing methods
> 
> On Tue, 2007-03-13 at 17:01 -0500, Taranowski, Thomas (SWCOE) wrote:
> > At the risk of being drug into the street and mauled by angry
> > developers, I'd like to know if we have any sort of informal test
> > framework for the lwIP stack.
> 
> We have none.  It is a shambles and embarrassing, but despite that
lwIP
> seems to be pretty stable for most users.
> 
> >
> > Now, I'm currently working on developing a test suite for my own
> > needs, which will be fairly comprehensive, with respect to the BSD
> > socket layer.  I'd like to propose to my management that we release
> > this to the lwIP community for further enhancement and support.
> 
> My doubts here would be that:
>  - we already know that the lwIP sockets layer is not particularly
> compliant with the standard, and doesn't support a lot of standard
> operations.
>  - it's hard to exercise large portions of the code using the sockets
> API alone.
> 
> A comprehensive socket API tester is also many man years of work!
> 
> > The primary reason for doing this is because I think it will result
in
> > a better product, but another reason, perhaps more compelling for
> > some, is because we're all doing our own testing thing, and it would
> > be more efficient if we could collaborate on a shared test
> > framework.
> >
> > Before I work up my proposal, I'd like to know if we already have
> > something, and if anyone else thinks this would be a useful addition
> > to our lwIP project.
> 
> Thanks for taking the time to involve lwIP - it would clearly be to
our
> benefit.
> 
> There are some existing commercial test suites for TCP stacks.  ANVL
is
> one I've used in the past (which although useful, has a number of bugs
> in its tests).  They don't come cheap though.
> 
> Kieran
> 
> 
> 
> _______________________________________________
> lwip-devel mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/lwip-devel