Re: Treating tests as special case

[Pjotr Prins]

On Thu, Apr 05, 2018 at 04:26:50PM -0400, Mark H Weaver wrote:
> Tests on different hardware/kernel/kernel-config/file-system
> combinations are quite useful for those who care about reliability of
> their systems.  I, for one, would like to keep running test suites on my
> own systems.

Sure. And it is a great example why to test scenarios. But why force
it down everyone's throat? I don't want to test Scipy or ldc over and
over again. Note that I can work around it, but we are forcing our
methods here on others. If I do not like it, others won't. I am just
looking at running test billion times uselessly around the planet.
Does that not matter? We need to be green.

Ludo is correct that provisioning binary substitutes is one solution.
But not cheap.  Can we guarantee keeping all substitutes? At least the
ones with long running tests ;). I don't know how we remove
substitutes now, but it would make sense to me to base that on
download metrics and size. How about ranking downloads in the last 3
months times the time to build? And trim from the end. That may be
interesting.

Even so, with my idea of test substitutes you don't have to opt out of
testing.  And you would still have found that bug. Those who care can
test all they please. 

Anyway, that is enough. I made my point and I am certain that we will
change our ways at some point. The laborious solution is to remove all
meaningless tests. And I am sure over 90% are pretty damn meaningless
for our purposes. Like the glut in binaries, we will trim it down over
time.

One suggestion: let's also look at tests that are *not* about
integration or hardware/kernel configuration and allow for running them
optionally. Stupidly running all tests that people come up with is not
a great idea. We just run what authors decide that should be run.

Pj.