Re: [ft-devel] source control, defect tracking and unit tests

[David Turner]

Hi everyone,

I knew I shouldn't have mentioned unit tests. Now I'll have to write 
this essay
anyway :-)

Graham Asher a écrit :

> I suggest using Subversion for source control. [...]
>
> Subversion has a very good Windows visual client called TortoiseSVN, also
> free and open-source.
>  
I agree completely. I've been using Subversion for my pet personal 
projects for more
than a year now, and I find it really great. What's really awesome is 
how the Subversion team
could implement something that is very similar to CVS but still easier 
to use and understand.

I'd love to be able to switch to Subversion for FreeType, but we've 
chosen to settle on Savannah,
which doesn't provide SVN repositories for the moment, so we're stuck 
with CVS until then.

Same goes for all other kinds of services that won't be ressurected 
(e.g. rsync mirrors for the
web and ftp sites), or won't see the day before long (e.g. a wiki). 
However, given that we have
so little free time, I'd rather spend it on development rather than 
server administration. We're
thus absolutely thankful to the people at Savannah, as well as to 
Stéfane, our former sysadmin,
for the time they can devote to the machines.

Now, for the really meaty pieces :-)

> Unit tests are obviously controversial: David says:
>
> "
> If I had time, I'd wrote an essay to explain why we don't need unit tests
> for
> FreeType at the moment (basically, 99.9% of the bugs uncovered in the
> library wouldn't have been caught by these kinds of tests, we have tons of
> "testers", and we're certainly not paid enough to code the tests :-)
> "
>
> I disagree very strongly, and I'd very much like to hear David's detailed
> arguments. In the meantime here are a few counter-arguments:
>
>  
OK, first, please understand that I make a very strong difference 
between "unit tests" and
"regression tests". I do think that writing regression tests for 
FreeType would be a very useful
thing, however:

- we really have little free time to write them (and I'll be a father 
again at the end of April :-)
- we must run the tests against third-party fonts which we cannot 
generally redistribute,
 which makes automated regression testing problematic.

A regression test is simply a test that check that certain things work 
as expected. You generally
don't expect exact rendering, but you can check your ability to open and 
read fonts, parse various
data, not crashing, etc... You generally write a regression test _after_ 
a bug was found and corrected,
to be certain that it will never occur again in the future. Hence the name.

A unit test is different, you generally write such a test to assert that 
a given 'unit', which can be a
class, module or source file, works as expected _before_ you release it 
in production code. Unit tests
differ from regression tests in several ways:

- you generally write them _before_ the tested code itself, or maybe at 
the same time.
 Most of us balk at the idea of writing unit tests for code that is 
more older than a few
 days, though we're ready to make the effort when we start afresh. I 
guess that's
 human nature.

- just like documentation, it's generally always better to put the unit 
test in the same
 source file(s) than the unit being tested. Otherwise, synchronisation 
problems always
 arise much too early, and you generally end up with out-dated junk 
that nobody wants to
 maintain anymore, or fight with wathever toolchain you setup to manage 
the mess.
 Use of conditionnal compilation (#ifdef  UNITTEST) is thus a plus

- you can use unit tests to document behaviour at edge cases. For 
instance, if you expect
 that your re-implementation of "strlen" accepts NULL as an alias for 
an empty string
 without crashing, you write the corresponding call in the unit test to 
verify that it
 doesn't crash, and return the expected value (in this case 0)

- the tests must not depend on other units ! otherwise, cascading bugs 
could prevent
 your tests to really detect failures. This is a _major_ point for unit 
tests. If you don't
 think it happens easily, I can sell you a really nice tower in Paris 
pretty cheap ;-). And
 there's nothing more disturbing when you start debugging (been there, 
done that,
 got the t-shirt...)

The problem is: how do we generate the input data for each test, and how 
do we
compare the results with what we expect. Remember, we'd better write the 
unit
tests in the same source file than the functions we want to test.

There are functions in the library that can be easily unit-tested, 
because doing the
above is trivial (e.g. the fixed-point computation routines, many Type1 
of stream
field parsing routines,...).

However, the most important parts of the library generally expect 
_highly_ structured
input data, and can generate results in a variety of formats that aren't 
all easily testable
with a '==', 'memcmp', 'strcmp', etc...

If your unit test depends on a specific font file that cannot be 
redistributed, most
people won't be able to run it, which is simply stupid. You could craft 
a new font
file for each new test, but this is serious work, and you must be 
extremely cautious
that the binary file you generated isn't buggy itself in ways that 
wouldn't be easily
checked by your code, and which wouldn't cause some false positives. Fun 
for the
whole family :-)

Another option is to hard-code test input data into constant variables 
in the unit test,
but this generally is a _big_pain_ to write. And the validation code can 
be very long,
sometimes even longer than the function being checked.

When I say that we don't "need" unit tests in FreeType at the moment, 
I'm only being a
pragmatist. What I think is that:

- most of the "easy" units test aren't exactly necessary, if we assume 
that the corresponding
 code has been tested to death through this little thing called "real 
users". I know this is
 false, but I'll fake that it doesn't matter.

- the "medium" and "difficult" unit tests are either a pain to write, or 
a legal encumberance;
 we can't afford to deal with them in our context. Heck, we just do 
that for the love of it.
 It's not like we're being paid to do that :-)

I will not respond to the rest of your post, because I believe that what 
you're describing are
really what I call "regression tests" instead. As said earlier, I think 
they're a good thing, but
we haven't been very organized to write them, and there are still 
problems with redistribution
of the corresponding data. And don't use Knuth/TeX as an example, he 
designed the fonts
from scratch himself, for god's sake :-)

I'm also not saying that having lots of testers is going to fix all of 
our problems. We happen
to introduce new bugs when implementing "improvements", or performing 
incorrect "fixes"
to existing problems, and this is our own responsability (no excuses :-)

Hope this helps,

- David

> 1. I don't really buy the assertion that "99.9% of the bugs uncovered in the
> library wouldn't have been caught by these kinds of tests" - that sounds a
> very high proportion. But even if it were true, a very good way to add to
> the stock of unit tests is to (carefully, and with added generalisation)
> write new tests after bugs are discovered.
>
>  
>
> 2. In my experience even a few unit tests are better than none. The way to
> do it is to start off with a small number of childishly simple tests that do
> things as fundamental as checking whether the main API functions work in
> their simplest possible forms, and the data structures can be created
> successfully (e.g., whether FreeType can be initialised). New tests are
> added steadily in response to bugs and when new features are added. Test
> files and 'difficult' fonts that expose problems are added to the test suite
> and put into source control.
>
> 3. David says "we have tons of  "testers"". True, but their testing is
> sporadic, uncoordinated and inconsistent. Again and again new versions of
> FreeType have been released with major defects, so much so that I am
> sticking to a relatively old version - 2.1.4 - and refusing to upgrade until
> things settle down. So it looks as if the tons of testers aren't producing
> the right result.
>
> 4. Unit tests allow the design aims of FreeType to be clarified, if not
> analytically, then at least pragmatically. Part of the specification can be
> that FreeType must behave in such-and-such a way when presented with
> such-and-such input; and that can be tested by running the unit tests. Knuth
> used this approach successfully with TeX and METAFONT (he did of course
> define these languages analytically as well).
>
> In summary, I urge the creation of unit tests. It requires very little work
> to start off: decide where to put unit tests and their data files; write one
> or two very simple tests; write a simple script to run them. We then have a
> 'stub' unit test system to which new tests can be added by volunteers.
>
> Unit tests cannot do any harm, but they can do a lot of good.
>
> Now for the apology in advance. I am committed to working six days per week
> at the moment, and, although I acknowledge that I have gained immeasurable
> benefits from FreeType over the last six years, I can't repay this debt of
> honour at the moment; so I can't offer to write unit tests myself. I hope it
> doesn't sound too hypocritical to urge that they be written.
>
> Best wishes,
>
> Graham Asher
>
>
>
>
>
> _______________________________________________
> Freetype-devel mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/freetype-devel
>
>