Re: More Questions about analysing ABMs

[M Lang / S Railsback]

address@hidden wrote:
> 
>  4)  So this is my project:  I built an ABM of the inflammatory response with
> cells as the agent level and used existing wet-lab data to formulate the rule
> systems for the cells.  The only independent variable is initial injury.
> There are pseudo-random numbers in the movement and pattern distribution of
> injury and initial cells, but after that the system is purely deterministic
> (these agents do not adapt).  I used as a measure of outcome a global damage
> measure and the number of remaining infectious agents.  After getting a
> distribution of outcomes over a range of initial injuries with the RNGs
> seeded, I picked regions of interest and ran multiple (~500) interations of
> the model with the RNGs turned back on.  Question:  Can I now use standard
> statistical techniques like distribution, standard deviation etc to compare
> the model to existing clinical data?  

I always avoid comparing the distribution of results from multiple model
replicates (varying only in random number seed) to distributions of
observed data- for this comparison to be valid, your clinical data would
have to be collected from very very similar injuries. When you compare
distributions of model results to data, you have to make sure the same
process was causing the variation in both cases. Rarely in nature do
"experiments" have the same initial conditions like replicate model runs
do.

I would be more comfortable comparing model runs that vary by the
initial injuries to your clinical data.

> Furthermore, I programmed in an
> intervention that was tested in a clinical trial, ran multiple iterations
> with the RNGs on.  Can I compare this data set to the baseline with standard
> statistical methods for significant difference?  The only conclusion I want
> to draw is that there is a qualitative similarity between the behavior of the
> model and the real world; no direct one to one mapping is intended.

OK, now you are comparing two sets of model runs statistically- we do
that all the time. Some things to worry about are that the statistical
significance of the difference between scenarios (with, without
intervention) depends on the number of model runs. Even if your
intervention has only a very small effect, it will be a statistically
significant one if you compare results of 1000 model runs. So be sure to
examine the medical significance as well as the statistical
significance. According to Bret Harvey, the statistically savvy fish
biologist I work with, the appropriate test for comparing two scenarios,
using data from multiple replicates of each, is one-way ANOVA followed
by pairwise comparisons using Bonferroni t-tests. (Don't ask me why, or
how to do it.) We just arbitrarily chose n=10 simulations of each
scenario, then used the t-tests to compare results among scenarios.

By the way, the above advice applies when comparing the final results of
each model run; we considered trying to compare, among scenarios,
statistical distributions of the states of the agents. Then the problem
with high sample size is severe- n=the number of agents in the model so
any difference is statistically significant. So we ignored statistics
and just talked about biological significance.

Steve
-- 
address@hidden
Lang, Railsback & Assoc.
250 California Ave., Arcata CA 95521
707-822-0453; Fax 822-1868


                  ==================================
   Swarm-Modelling is for discussion of Simulation and Modelling techniques
   esp. using Swarm.  For list administration needs (esp. [un]subscribing),
   please send a message to <address@hidden> with "help" in the
   body of the message.
                  ==================================