Re: unit-testing Automake (was: [PATCH] Modernize, improve and/or extend tests `colon*.test.)

[Stefano Lattarini]

At Thursday 22 July 2010, Ralf Wildenhues wrote:
> * Stefano Lattarini wrote on Thu, Jul 22, 2010 at 01:04:03AM CEST:
> > At Wednesday 21 July 2010, Ralf Wildenhues wrote:
> > > Well, this is a common dilemma in testing: should the tester be
> > > allowed to use insider information or not?  [...]
> > > 
> > > Getting the right balance here is the most non-automatic part
> > > of testsuite work.  Ideally, we want the testsuite to be as
> > > strict as possible without ever peeking in details which do
> > > not matter for the documented API semantics.  So that any
> > > change we make to automake.in produces no testsuite failure if
> > > the change still provides the right user API.
> > 
> > Agreed.  This would be more feasible and natural if we had unit
> > tests for automake internals...  but, alas, ATM I don't see any
> > way to introduce such tests without an *extensive*
> > reorganization and refactoring of Automake code, which would
> > probably (surely?) be way too much dangerous and time-consuming.
> >  Oh well.
> 
> I'm not sure I follow.  We have a handful of unit tests in the
> lib/Automake/tests directory.
True, but they only test the code in auxiliary modules lib/Automake/*.pm,
not in scripts automake.in or aclocal.in.  And these last two scripts
don't seem written with unit-testability in mind.
> What's keeping us from adding more?
For lib/Automake/*.pm, nothing; for automake.in and aclocal.in, the
fact that their code should IMHO be first reorganized and refactored
in order to make them unit-testable.  And this would be the "dangerous"
part.
> Also, iff we have good API coverage in the first place, then a
> reorganization shouldn't be dangerous, because the testsuite will
> tell us if we have broken anything.
True.  What I was saying is that, when proper and complete unit tests
are missing, it's quite tempting to insert checks about internal details
in tests that should only do API coverage.
> Time-consuming, well, yes.

> > > For a related example of this theoretizing: your various
> > > patches introduce several greps for error message strings.  I
> > > have acked them, mostly because I think it is a good idea,
> > > generally.  Now, a while ago I started a patch series to add
> > > i18n to Automake.
> > 
> > Ouch.  I really hoped that Automake could have remained i18n-free
> > (yes, I have a strong dislike for i18n, esp. w.r.t. command-line
> > tools: new added complexity to gain so little -- and to lose
> > greppability of error and warning messages in the meantime...
> > sigh).
> 
> Actually, the added complexity is fairly low in this case ... at
> least for the messages produced by perl code.  The gain for some
> users is (hopefully) not little, while I agree that for others,
> there isn't much gain.
But automake users' are expected to be software developers, and from
them a knowledge of technical english is to be expected (a basic and
passive knowledge at least).  Moreover, the major help sources for
Automake users (the Automake official manual, the Autotools tutorial
at <http://www.lrde.epita.fr/~adl/autotools.html>, and the
address@hidden mailing list) are all in english.

Regards,
   Stefano