[ft-devel] source control, defect tracking and unit tests

[Graham Asher]

Dear FreeTypers,

I support the use of unit tests, bug tracking and the introduction of a
source control system that cooperates with them.

I suggest using Subversion for source control. It provides version numbers
covering the whole source tree, not just a single file (rather like Perforce
changelist numbers; but Perforce is expensive and Subversion is free and
open-source). This allows us to say that 'bug N was fixed by change M'; and
to synchronise all files to any state of the source tree very easily. In
other respects it works rather like CVS. It has another huge advantage: it
tracks changes to the file structure (addition, deletion, moving and
renaming of directories) as well as files, something I have wanted for a
long time.

Subversion has a very good Windows visual client called TortoiseSVN, also
free and open-source.

I already use FogBugz for bug tracking - it's not perfect but it's by far
the best bug tracking system I've used (others have been custom systems on
Lotus Notes, TeamTrack, etc.), its main advantage to me being that it makes
it extremely easy to enter a defect - the only compulsory field is the
defect title. As David says

"I don't want a ton of stupid fields to create a single issue".

Subversion and FogBugz cannot at present be integrated automatically (I
believe). However, all you need do is, when you commit to Subversion, note
in the Subversion log entry the FogBugz defect number(s), if any, that the
change fixes; and when you close a FogBugz case, note the Subversion version
number of the fix. I do this in my work and it is very simple and makes
fixing regression bugs relatively easy.

Subversion handles branching, of course.

Unit tests are obviously controversial: David says:

"
If I had time, I'd wrote an essay to explain why we don't need unit tests
for
FreeType at the moment (basically, 99.9% of the bugs uncovered in the
library wouldn't have been caught by these kinds of tests, we have tons of
"testers", and we're certainly not paid enough to code the tests :-)
"

I disagree very strongly, and I'd very much like to hear David's detailed
arguments. In the meantime here are a few counter-arguments:

1. I don't really buy the assertion that "99.9% of the bugs uncovered in the
library wouldn't have been caught by these kinds of tests" - that sounds a
very high proportion. But even if it were true, a very good way to add to
the stock of unit tests is to (carefully, and with added generalisation)
write new tests after bugs are discovered.

2. In my experience even a few unit tests are better than none. The way to
do it is to start off with a small number of childishly simple tests that do
things as fundamental as checking whether the main API functions work in
their simplest possible forms, and the data structures can be created
successfully (e.g., whether FreeType can be initialised). New tests are
added steadily in response to bugs and when new features are added. Test
files and 'difficult' fonts that expose problems are added to the test suite
and put into source control.

3. David says "we have tons of  "testers"". True, but their testing is
sporadic, uncoordinated and inconsistent. Again and again new versions of
FreeType have been released with major defects, so much so that I am
sticking to a relatively old version - 2.1.4 - and refusing to upgrade until
things settle down. So it looks as if the tons of testers aren't producing
the right result.

4. Unit tests allow the design aims of FreeType to be clarified, if not
analytically, then at least pragmatically. Part of the specification can be
that FreeType must behave in such-and-such a way when presented with
such-and-such input; and that can be tested by running the unit tests. Knuth
used this approach successfully with TeX and METAFONT (he did of course
define these languages analytically as well).

In summary, I urge the creation of unit tests. It requires very little work
to start off: decide where to put unit tests and their data files; write one
or two very simple tests; write a simple script to run them. We then have a
'stub' unit test system to which new tests can be added by volunteers.

Unit tests cannot do any harm, but they can do a lot of good.

Now for the apology in advance. I am committed to working six days per week
at the moment, and, although I acknowledge that I have gained immeasurable
benefits from FreeType over the last six years, I can't repay this debt of
honour at the moment; so I can't offer to write unit tests myself. I hope it
doesn't sound too hypocritical to urge that they be written.

Best wishes,

Graham Asher