Re: R, HDF5, Swarm output serialization - Part II

[Marcus G. Daniels]

>>>>> "NEN" == Norberto Eiji Nawa <address@hidden> writes:

NEN> Hello: A while ago (*), I reported on problems I am having to
NEN> load several HDF5 files in R, to process data generated in my
NEN> models. I am not able to load more than 10 (of 50) 1.5-2MB files
NEN> without almost hanging the computer.

You're having better luck than me.  When I load single 8MB HDF5 in to
R, I routinely get segfaults.  I haven't seen this hanging though. 
I'm using machines with a gig or more of RAM, though.

NEN> Each time the model outputs data, it builds a "compound
NEN> object", with one field to each variable that is saved, and
NEN> pushes it into an HDF5 archiver. 

Have you looked at the recent extensions to EZGraph?  It can now
extend datastreams without introducing new objects.  That way you
don't have to get bigger and bigger lists in virtual memory.  (Lists
that you'd need to break up into `epochs' in order to drop them from
time to time.)

I'll take a look at your test programs as soon as I have a chance.

                  ==================================
   Swarm-Support is for discussion of the technical details of the day
   to day usage of Swarm.  For list administration needs (esp.
   [un]subscribing), please send a message to <address@hidden>
   with "help" in the body of the message.