Re: [Chicken-users] Nursery sizing considered stupid

[John Cowan]

Brandon J. Van Every scripsit:

> Frighteningly, having examined the testing logic, it is correct.  It's 
> actually reporting on 2 different runs of the 128K sample size.  Once 
> when testing the default, and once when looping through the increasing 
> sample sizes.

That's exactly what I assumed in the first place, and in fact until
reading this very posting I didn't realize that you thought I meant there
was a bug in your code.  No, it's the whole idea of sampling that is
"stupid" and [ironically] "brilliant".  Your code merely rubbed my nose
in that.

And for that, Felix *is* responsible.

> This means with 10 samples, the default of 128K shows a
> variance of 15% !!  That means variance on your machine severely sucks, 
> even worse than on mine.  10 samples is a bit random on my machine, but 
> with a 5% performance threshold the results are stable.  Apparently, 
> there's no reason to assume the sampling results will be stable on any 
> other machine.

Exactly so.

> So, I'm bumping the samples to 100, and implementing a low-high variance 
> test.  The low-high variance test won't take any more time, and will 
> demonstrate how random the sampling actually is.  100 samples is kinda 
> slow on my box.  I was hoping to make due with fewer samples and have 
> faster build times.  But this apparently won't work in the general 
> case.  If 100 samples isn't good enough to create testing stability for 
> machines in general, then nsample is junk.  If proven to be junk, then 
> we will need to either improve it or scrap it.

Well, we can only hope.  I'm rather pessimistic, as you can see.

-- 
John Cowan  address@hidden   http://ccil.org/~cowan
It's the old, old story.  Droid meets droid.  Droid becomes chameleon.
Droid loses chameleon, chameleon becomes blob, droid gets blob back
again.  It's a classic tale.  --Kryten, Red Dwarf