Re: Regression tests

[Carl Sorensen]

On 6/6/10 8:01 AM, "Graham Percival" <address@hidden> wrote:

> On Sun, Jun 06, 2010 at 07:29:03AM -0600, Carl Sorensen wrote:
>> 
>> I'm moving all of the detail out of 3.6.3 Post-compilation options into 8
>> Regression tests.
> 
> Great!
> 
>>  You can learn more about testing in the
>> documentation."  (I can't put a link here, because this is part of the
>> COMPILING file).
> 
> I think you mean INSTALL.txt.

Yep, that's what I meant.

> We currently *do* have such links,
> such as in the very first text paragraph of that file.
> ... 
> With that in mind, please add a link.  Since it's in
> Documentation/included/compile.itexi, you need to use @rcontrib{}
> even though you're linking within the CG.

OK, I'll add a link.
> 
> 
>> I've reordered CG 8.  Tentative organization is:
>> 
>> Introduction to regression tests
>> Precompiled regression tests
>>     Regression test output
>>     Regression test comparison
>> Compiling  regression tests
>> Identifying code regressions
>> Other tests
>>     Checking memory usage
>>     Checking code coverage
>> MusicXML tests
> 
> I don't like "other"... what about "Performance tests"?  no wait,
> that doesn't work for code coverage.  Hmm.

I'll think about other words.  "Other" was a default that came out with very
little thought.

> 
> Separate topic: is the distance-output script going to be
> discussed in Compiling regression tests?  In my mind, there's a
> difference between compiling the regtests (i.e. "do any errors
> break compilation?") and examining the before+after changes for a
> patch.

Yes.  Compiling regression tests uses make test, which only compiles the
tests (That's why I asked the question about getting usable output from
collated-files.html.  Right now, it's hard to see we get anything from make
test except the lack of an error.)

Identifying code regressions talks about using make test-baseline and make
check, which creates a useful diff output.

These two sections, Compiling regression tests and Identifying Code
regressions, parallel the two subsections in Precompiled regression tests.
Regression test output talks about the output of the complete regression
test suite, which is available for every stable version and the current
development version.  Regression test comparison talks about the differences
between different versions, which are available starting at 2.11.x.  So we
have already introduced the idea of two different uses for regression tests.
> 
> 
>> I haven't yet figured out where the information on what you need to do after
>> modifying fonts should go.  My recent experience has shown me that you don't
>> need to do a make clean after modifying fonts.  It's sufficient to do rm
>> mf/out/*; that triggers a font rebuild.
> 
> *shrug*
> I wouldn't expect that to make a huge difference in time, although
> of course if you're doing heavy font work, I guess that saving 3-4
> minutes for each compile *would* help a lot.

Actually, on my machine, it's more like 20 minutes.  Doing make clean takes
a *long* time (much more than 3-4 minutes); doing rm mf/out/* takes a
fraction of a second.  And then the build process seems to go faster, too
(although I haven't timed the difference).

> For now, I'd dump it somewhere in the Programming chapter;
> eventually we might split off a Font chapter from that.  But I'd
> really caution against trying to juggle too many balls at once --
> dump it somewhere and then focus on the Regressions chapter.
> 

OK, I can do that.

> 
> I'm seeing too much juggling.  Focus on the Regressions -- dump
> all other info into the CG (anywhere that makes remote sense,
> other than CG 1, 2, and 7).  Don't bother with a patch-review for
> that info-dump.  Then work carefully on Regressions, post a patch,
> etc.

Will do.

Carl