R, HDF5, Swarm output serialization - Part II

[Norberto Eiji Nawa]

Hello:

A while ago (*), I reported on problems I am having to load several HDF5
files in R, to process data generated in my models. I am not able to
load more than 10 (of 50) 1.5-2MB files without almost hanging the
computer.

I wrote another small program to check whether this was a R-dependent
problem; it is a self-sufficient piece of code that directly calls the
HDF5 library functions to load the 50 files in a row and store the
data in some internal variables. Again, I got the same symptoms; the
program is unable to load more than 6 or 7 files without it starts
swapping badly.

For those who are interested, the files (app, Makefile, sample data)
can be found at http://www.isd.atr.co.jp/~eiji/swarm/loadingHDF5/
Please, let me know if you get the same problems or not.

Since the very beginning, I suspected that the cause of the problem is
actually the way I am serializing data from my model. Each time the
model outputs data, it builds a "compound object", with one field to
each variable that is saved, and pushes it into an HDF5 archiver. For
2000 iterations, there are 2000 objects, each one of them containing 7
to 10 float variables. This was actually a suggestion from Marcus,
which I properly spoiled (**) and made it into a problem to myself :-)
My guess is that it should be working anyway, and now I decided to
blame the computer. 

If anyone happens to run these simple test programs fine, please, let
me know, so I can look for the problem in my platform.

Thanks a lot!

Eiji

(*) http://www.santafe.edu/projects/swarm//archive/list-archive.0103/0068.html

(**) http://www.santafe.edu/projects/swarm/archive/list-archive.0009/0142.html


                  ==================================
   Swarm-Support is for discussion of the technical details of the day
   to day usage of Swarm.  For list administration needs (esp.
   [un]subscribing), please send a message to <address@hidden>
   with "help" in the body of the message.