On 2003-10-15 22:58:38 +0200 Lars Sonchocky-Helldorf
<Lars.Sonchocky-Helldorf@bbdo-interone.de> wrote:
On 15.10.2003 20:58:37 Stefan Urbanek wrote:
<snip>
Simplier solution: What about lots of users as a 'testing framework and
quality
controll'? Advantage of 'user based testing framework' is that it
tests continualy. That 'framework' should also contain users, that
were marked
as
'kind of users that gnustep does not need' in a gworskpace thread few
days ago
on this list. More users means more testers and it does not matter what
kind of
users. At this time and state, gnustep needs any kind of users. More
eyes see
more.
I think, that if at least half of the effort devoted to writing test
suites (i
am not against them) is also devoted to make gnustep more attractive, it
can
also help to remove bugs and have a quality controll.
That's a circular dependency: less bugs mean more users, more users
mean less bugs. But the flipside is also true: lots of bugs mean less
users, less users mean lots of bugs :-(. Currently the situation tends
more to the latter...
Yes, and we need to get out of it.
The problem with a "unorganised user based testing" approach is that
it needs committed, disciplined users to work. That's something you
can't take for granted. My experience tells me instead that far most
users simply turn away if things don't work out of the box: You even
can be thankful if about one percent fire their mail client up and
send you just complaints, the users that are able to give you a
qualified bug report you can count with the fingers of both hands.
Another result introduced by the nature of your approach is that
you'll get many reports for easy bugs you already know, but virtually
nobody is willing to invest a lot of time to track down a bug which
a.) occurs irregulary or b.) involves a lot of work. In the end it is
like in any chaotic multi agent system: the path of least resistance
is chosen. The already long history of GNUstep is another proof for this.
True, users go away if it does not work. So what? We should at least
have the first-user-experience parts bug-free, to let them go in and
start play with the other stuff. Like a trap :-) You need a trap, hook
or attraction to get users. Is there any in GNUstep? "The great
framework desing with no alternative? Yes, but ... it does not work (as
expected), so what?" You get the point.
About the discipline and organisation: you need that and you do not.
Point is, that currently there is no need to fix bugs. Why? Because
noone wants them to be fixed, or there are just a few people that want
them to be fixed. So why waste an energy? More users means more
pressure, and perhaps a chance that there is someone who will likely
want to fix it. This is something about that famous "it works on my
machine, so why bother?". True, why to bother, if there are just one or
two users with this behaviour and noone else complains?
You make a mistake when you think that the "effort devoted to writing
test suites" is basically lost time and done just for respectability.
Let me tell you: it is different. Any amount of time invested in a
proper test pays off. For a long time I also was a opponent of unit
testing, I just saw no benefit from it, I resisted doing it and left
it to the other guys here at work. But one day I *had* to do it and it
came out to be much less complicated then I had feared. In fact I
realized that I was so reluctant towards it because it was unknown to
me. Today I have the habbit to test early and often. Unit-testing just
helps you to make rapid progress while coding. I can tell you that I
discovered lots of bugs, even 'thinkos' in my code doing testing
(often borderline cases where some special treatment for certain
values was needed) that would have gone undiscovered and would have
caused bugs hard to track down later on (if you build on a faulty
foundation you building will most likely be faulty to). I wouldn't
start a project (even a private) without unit testing today.
No, I haven't said that the effort devoted to writing test is lost time,
just that at least half of the time should be devoted to gaining more
users. This is not a commercial software for particular requestor where
you already have a customer, this is free software where "customers"
are not certain. And if you write a SW that is really perfect (according
to tests), why is it good for if it is for noone just those who
developed it? Then the SW will die. I'm saying that testing by users can
be complimentary to testing with a testing framework. GNUstep needs
both. Now a question is, what is easier to get at this time?
Btw., to paraphrase your last question ... Would you start a project
without users (even potential) today? Are there going to be any future
users with this state and attitude of GNUstep?
Moreover, there are functionalities that can be tested only by users, so
saying that users are not good for testing is not a way to go.
To sum it up:
- user testing is complimentary to unit testing
- more users will result in greather pressure/need of bug fixing
- to get more users, there should be a trap/hook/attraction - is there any?
- without users, project dies with its developers
Have a nice day,
Stefan Urbanek