Re: Formal methods for comparing model runs

[Robert Bell]

"M. Lang / S. Railsback" wrote:

> I'm trying to write up some comparisons of model scenarios; for example,
> comparing the distributions of water depth and velocity used by fish
> when we make two alternative assumptions about how they select habitat.
>
> At first it seemed obvious to compare the scenarios with a
> Kolmogorov-Smirnov test for differences between distributions. Now,
> though, it seems just as obvious that hypothesis-testing statistics like
> these are not useful for comparing agent-based model runs. First, the
> basic purpose of hypothesis-testing methods is to distinguish
> "information" from "noise". Our model's habitat selection methods,
> though, are completely mechanistic and don't have any noise. Second,
> with a model, the sample size can be arbitrarily large. If I want, I can
> compare model runs with 800 fish each observed on each of 10 days
> (sample size: 8000), or observed on each of 20 days (sample size:
> 16,000). Because the test results depend on sample size, I'm finding
> that even the tiniest differences in results between scenarios are
> statistically significant.

I have been having the same questions....though I do have a stochastic
component, I am pounding my head on the wall attempting to separate
computing artifact from biological significance. Then again, I think this is
the nature of the beast.

> So the consequence is that statistically significant results clearly are
> not biologically significant. (People that count real fish would be very
> jealous.)

I am confused here...what led you to this conclusion?  Are the results
with the different samplings so out of whack with reality as to conclude
they are not possible?  What kind of variance was there between
samplings of the same size?

> Consequently, my current plan is to show the distributions
> resulting from each model scenario and talk about the potential
> biological significance of the differences in distributions.

That's the usual way out...present the possibilities and let the audience
draw their own conclusions. But I would also like to see more
discussion of this.

Bob





                  ==================================
   Swarm-Modelling is for discussion of Simulation and Modelling techniques
   esp. using Swarm.  For list administration needs (esp. [un]subscribing),
   please send a message to <address@hidden> with "help" in the
   body of the message.
                  ==================================