Formal methods for comparing model runs

[M. Lang / S. Railsback]

I'm trying to write up some comparisons of model scenarios; for example,
comparing the distributions of water depth and velocity used by fish
when we make two alternative assumptions about how they select habitat.

At first it seemed obvious to compare the scenarios with a
Kolmogorov-Smirnov test for differences between distributions. Now,
though, it seems just as obvious that hypothesis-testing statistics like
these are not useful for comparing agent-based model runs. First, the
basic purpose of hypothesis-testing methods is to distinguish
"information" from "noise". Our model's habitat selection methods,
though, are completely mechanistic and don't have any noise. Second,
with a model, the sample size can be arbitrarily large. If I want, I can
compare model runs with 800 fish each observed on each of 10 days
(sample size: 8000), or observed on each of 20 days (sample size:
16,000). Because the test results depend on sample size, I'm finding
that even the tiniest differences in results between scenarios are
statistically significant. 

So the consequence is that statistically significant results clearly are
not biologically significant. (People that count real fish would be very
jealous.) Consequently, my current plan is to show the distributions
resulting from each model scenario and talk about the potential
biological significance of the differences in distributions.

Anybody know where this issue is discussed in the literature,
specifically for agent-based models? 

Steve
-- 
address@hidden
Lang, Railsback & Assoc.
250 California Ave., Arcata CA 95521
707-822-0453; Fax 822-1868


                  ==================================
   Swarm-Modelling is for discussion of Simulation and Modelling techniques
   esp. using Swarm.  For list administration needs (esp. [un]subscribing),
   please send a message to <address@hidden> with "help" in the
   body of the message.
                  ==================================