Re: Lisp Serialization: GZ support?

[Marcus G. Daniels]

>>>>> "PJ" == Paul Johnson <address@hidden> writes:

PJ> I need an equivalent of "lispOutVars:" to shallow archive the ints
PJ> and doubles in the Data object. I made some guesses, none
PJ> satisfactory.

[super hdf5OutDeep:] should work, and does work for me.  Remember that
the default serialization will bypass pointers in either deep or
shallow mode.  This is the only change I made to your example.
Because of the way you wrote your example, it does require the enhancement
to Swarm in today's snapshot.

PJ> Reload crashes Swarm

Today's snapshot adds support for automatic allocation of memory for
ivars that are pointer (as was just recently added for the Lisp archiver).

  ftp://ftp.swarm.org/pub/swarm/src/testing/swarm-2002-03-18.tar.gz

PJ> I doubt I'm doing the rest of this correctly. The hdf5 output
PJ> dataset is about 12 times bigger than the size of the *.scm file
PJ> from the Lisp archiver and I can't get the hdf5 version of the
PJ> program to re-load the saved-arrays.

HDF5 has a small amount of overhead which you will overtake within the
first couple thousand observations (in your example). 
Think of HDF5 like a compressed filesystem on a disk and the Lisp
serialization like a raw tape drive.  Yes, it costs a little bit of
space to keep track of things, but it is well worth it.  

You will also observe that Lisp serialization gets slower as the data
gets bigger at a rate significantly higher than HDF5.  That's because
the data has to be converted back and forth between parse trees, and
text format for the numeric values.  HDF5 reads and writes raw binary
data in chunks.

                  ==================================
   Swarm-Support is for discussion of the technical details of the day
   to day usage of Swarm.  For list administration needs (esp.
   [un]subscribing), please send a message to <address@hidden>
   with "help" in the body of the message.