Re: Serialization using HDF5Archiver

[Marcus G. Daniels]

>>>>> "NEN" == Norberto Eiji Nawa <address@hidden> writes:

NEN> I want to output streams of data from my model using the
NEN> HDF5Archiver for posterior processing using R. These streams are
NEN> time series, i.e. at every time step the model is sampled, the
NEN> correspondent value of a certain parameter is obtained, and in
NEN> the end of the simulation I want to have the complete time series
NEN> curve. 

There are two approaches.  One is to put your `tmpNode' objects into
a List and then call putShallow:object: on that. 

The default shallow serialization code will do the job if you hide the
variables which are objects in a heatbug by not making them public.
Specifically, here are the required changes:

Three changes to Heatbugs:

  1) Change `world', `world{XY}Size', `heat', and `scratchHeatCell'
     to not be public.  This will make the default serialization code
     ignore those variables.

  2) A new method `writeHeatbugs' that saves the heatbugs list.
     It writes the list at time step 10 using the key "heatbugs10".   
     If you wanted to write snapshots, you could print into a key
     and save it every timestep.
 
  3) An additional action in modelActions that calls #2

Then, in R, loading is like so:

  $ R
  > library(hdf5)
  > hdf5load("jheatbugs.hdf")
  
To see the unhappiness vector, you could say:

  > heatbugs10$unhappiness

Or to see the distribution of ideal temperatures, you could say:

  > hist(heatbugs10$idealTemperature)

I'm using today's snapshot with a new hdf5_0.6.tar.gz.  (hdf5_0.6.tar.gz has
modifications for the development version of R 1.2.)

[see end of message for jheatbugs patch]

The advantage of this approach is that it is simple on the R end of
things: when you load the HDF5 file, there will be a variable set that
is a `data frame' that represents the agent state across time.  The
downside is that all those records go in virtual memory during the
simulation, so you might run into trouble if you have millions of
output records.

Another approach is to provide an unique key for each tmpNode object.
To do that, you'd use an Objective C idiom like below.  In this case
the saving is done in the Agent's step method (it saves itself).  `sn'
is a unique serial number for the Agent that is provided when the
Agent is born.  The record is also keyed on the timestamp.  So you'll
get many independent records that represent an agent at different
times.

   {
     char key[1 + DSIZE (sn) + 1 + DSIZE (timeval_t) + 1];
 
     sprintf (key, "h%02u.%lu", sn, getCurrentTime ());
     [hdf5AppArchiver putShallow: key object: self];
   }

The downside of this approach is that you have to combine all these
values in R, which is a bit of a hassle.  Happily, R is fully
programmable.  For example, functions like this will combine the first
100 records for a given variable and timestep:

$ cat test.R
varlist <- function (varname,timestep) { 
 # create a zero-padded vector of strings from 0 to 00
 seq <- formatC(0:99,width=2,flag="0",format="d")

 # prefix all the numbers with "h" and suffix them with .TIME$VARNAME
 paste("h",seq,".",timestep,"$",varname, sep="")
}

getvalues <- function (vl) {
 # combine the variable name vector into a single string delimited by ","
 vStr <- paste(vl,collapse=",")
 # wrap it with the `combine' function
 vExprStr <- paste("c(",vStr,")")
 # convert the string into an express and evaluate it
 eval(parse(text=vExprStr))
}

To get the `unhappiness' values for all the agents at timestep one, and
assign it to variable `v', you'd say:

$ R
> library(hdf5)
> hdf5load("agents.hdf")
> source("test.R")
> v <- getvalues(varlist("unhappiness",1))  

Then you could use it for whatever:

> mean(v) 
> plot(v)

--------------------
Patches to Java Heatbugs for saving heatbug state in a List:

Index: Heatbug.java
===================================================================
RCS file: /cvs/hive/Swarm/jheatbugs/Heatbug.java,v
retrieving revision 1.12
diff -c -r1.12 Heatbug.java
*** Heatbug.java        2000/02/29 00:57:40     1.12
--- Heatbug.java        2000/08/20 00:39:50
***************
*** 26,41 ****
    /** chance of moving randomly */
    public double randomMoveProbability;                          
    /** the world I live in */
!   public Grid2d world;                                  
    /** how big that world is */
    public int worldXSize, worldYSize;                    
    /** the heat for the world */
!   public HeatSpace heat;      
    /** my colour (display) */
    public byte bugColor;               
  
    /* Scratch cell for extracting return values */
!   public HeatCell scratchHeatCell;
  
    /** 
     * these methods are used to initialize the object's state. First,
--- 26,41 ----
    /** chance of moving randomly */
    public double randomMoveProbability;                          
    /** the world I live in */
!   Grid2d world;                                 
    /** how big that world is */
    public int worldXSize, worldYSize;                    
    /** the heat for the world */
!   HeatSpace heat;     
    /** my colour (display) */
    public byte bugColor;               
  
    /* Scratch cell for extracting return values */
!   HeatCell scratchHeatCell;
  
    /** 
     * these methods are used to initialize the object's state. First,
Index: HeatbugModelSwarm.java
===================================================================
RCS file: /cvs/hive/Swarm/jheatbugs/HeatbugModelSwarm.java,v
retrieving revision 1.12
diff -c -r1.12 HeatbugModelSwarm.java
*** HeatbugModelSwarm.java      2000/02/29 00:57:40     1.12
--- HeatbugModelSwarm.java      2000/08/20 00:39:52
***************
*** 245,250 ****
--- 245,257 ----
      return this;
    }
  
+   public void writeHeatbugs () {
+     if (Globals.env.getCurrentTime () == 10) {
+       Globals.env.hdf5AppArchiver.putShallow$object ("heatbugs10", 
heatbugList);
+       Globals.env.hdf5AppArchiver.sync ();
+     }
+   }
+ 
    /**
       * Here is where the model schedule is built, the data structures
       * that define the simulation of time in the mode. The core is an
***************
*** 305,310 ****
--- 312,319 ----
      try {
        modelActions.createActionTo$message 
          (heat, new Selector (heat.getClass (), "updateLattice", false));
+       modelActions.createActionTo$message
+         (this, new Selector (getClass (), "writeHeatbugs", false));
      } catch (Exception e) {
        System.err.println("Exception updateLattice: " + e.getMessage ());
      }


                  ==================================
   Swarm-Support is for discussion of the technical details of the day
   to day usage of Swarm.  For list administration needs (esp.
   [un]subscribing), please send a message to <address@hidden>
   with "help" in the body of the message.