Re: 2.13.40 regtests

[Graham Percival]

On Fri, Nov 26, 2010 at 12:14:17AM +0000, Neil Puttock wrote:
> On 26 November 2010 00:00, Graham Percival <address@hidden> wrote:
> 
> > Hmm.  It shouldn't take a huge amount of time to compare each pair
> > of regtest images -- they're named, so you'd be comparing
> > something like 500 pairs of .png images.  (Neil: were you thinking
> > of something else?)
> 
> I think this would be very slow unless it does something similar to
> ouput-distance.py to reduce the number: you don't want to be comparing
> every single pair unless absolutely necessary.

Hmm.  Average png is, what, 400x50 pixels?  20kb?  So the program
needs to load 40kb from disk, then do 20,000 comparisons between
pixels.  We have roughly 500 regtests, so that gives us 10 million
pixel comparisons.

I wouldn't have thought that this would be a problem for a modern
computer, but I must admit that I've never done any graphics
processing.  But if the pixels are in RGB format, then my first
guess would be to compare each color format, and see if the
numbers differ by more than a certain amount.  Something like
  int changed = 0;
  for (int c=0; c<2; c++) {
    if (abs(first.color[c] - second.color[c]) > 0.0001) {
      changed = 1;
    }
  }

Running that code 30 million times shouldn't take too long.  I
think?  I might be off by an order of magnitude, but my violin
physical modeling can generate 30 seconds of audio (that's 1.3e6
samples) in less than 2 seconds of wall-clock, and computing each
audio sample is a lot more complicated than the above code.
Granted, the violin stuff is C++ code, so I'd expect python or C#
to be slower.

My main concern is still with false positives.

> While test bugfixes, I've occasionally produced regression checks
> which have affected lots of tests, and the time for `make check' in
> such cases rises drastically (from about five minutes to over half an
> hour), so unless Phil's pixel-by-pixel comparison is much faster than
> `compare', it's still going to be too slow.

Oh, I was only thinking about this being used for special cases
and in the release-build process.  I don't think we need to add
this to the usual "make check".

Cheers,
- Graham