Re: search for better regtest comparison algorithm

[Werner LEMBERG]

> The original example that you came up with was a false negative,
> namely a missing object that stayed unnoticed.  Now we're discussing
> all kinds of complicated algorithms to reduce the probability of
> false negatives, while also trying to avoid false positives.  My
> question is: Do we really have a problem with false positives?

Yes, it can happen that a code change suddenly make notes visible that
were suppressed before.

> I had a quick glance on the last MRs that had their artifacts still
> available and found no example except !2391 that had scores "below
> threshold".  Does this happen frequently?

'Below threshold' happens quite often, yes.

> If not, wouldn't it suffice to improve on the sensitivity of the
> comparison process and not introduce stuff that tries to
> discriminate between "good" and "bad" changes?  We could render
> tests with unclear results a second time with higher resolution,
> e.g.

In almost all cases, a higher resolution doesn't help with the problem
at hand, namely the appearance or disappearance of objects in
regression tests that accidentally go unnoticed.

I ask you and others to not complicate the issue.  What's really
needed (and nothing more) is an improved algorithm that can be either
used as an addition or as a replacement for the current one: In
comparison to what we currently have, it should yield smaller demerits
for slightly shifted images and/or objects and much larger demerits
for appearing or disappearing objects.

It doesn't make any sense right now to muse about completely changing
the process.


    Werner