Re: Robustness Check

[Theodore C. Belding]

So far we've been focusing on how to choose seeds for a set of runs with a
single set of parameter values. In that case it would be nice, but not    
essential in practice, to guarantee that the sequence of random numbers is 
completely different for each run.

But there are situations where we would actually like to use exactly the
same sequence of random numbers for different runs. One such situation is
when we simply want to replicate a run. Here's another example:

What we modelers usually do is vary the parameters of a model and do a set
of runs for each combination of parameter values. In other words, changing
the parameters is a treatment in the statistical sense, and we'd like to
see what effect the treatment has on the model's behavior, while holding
everything else constant.  "Everything else" in this case includes the
sequence of random numbers, so ideally we should use the *same* set of
seeds (and the same sequences of random numbers) for each combination of
parameter values.

We can then use something like a paired Student's t test, where each run
in the pair uses a different set of parameter values but uses the same
sequence of random numbers. This reduces the variance caused by the random
numbers and increases the power of the test, just as paired comparison
experiments do in general; in principle it reduces the sample size (the
number of runs) required for a statistically significant result.

This is known as variance reduction and is discussed in:
Bratley, P., B. L. Fox, and E. L. Schrage. (1983). A Guide to Simulation.
Springer.

In practice, this variance reduction is very difficult to achieve, since
you need to guarantee that the same random numbers are used for the same
purpose in each run. In many simulations, the number of calls made to the
random number generator depends on the course of the simulation; any
slight deviation in the course of a run will change the random number that
is used in a particular spot in the program.

Also, you can generally get statistically results without using variance
reduction, so it usually isn't worth the trouble. But it might be useful
in some situations where you need to extract statistically significant
results from a small set of data. 
-Ted

--
Ted Belding                              address@hidden
University of Michigan Center for the Study of Complex Systems
Homepage: http://www-personal.umich.edu/~streak/
PGP key:  http://www-personal.umich.edu/~streak/pgp-key.html



                  ==================================
   Swarm-Support is for discussion of the technical details of the day
   to day usage of Swarm.  For list administration needs (esp.
   [un]subscribing), please send a message to <address@hidden>
   with "help" in the body of the message.