Re: [h5md-user] Variable-size particle groups

[Peter Colberg]

On Tue, May 29, 2012 at 05:32:48PM +0200, Olaf Lenz wrote:
> Hi!
> 
> On 05/29/2012 05:17 PM, Peter Colberg wrote:
> > At any given timestep, each processor owns a defined set of 
> > particles of varying number, which shall be written to an HDF5 
> > dataset. The task that needs to be solved is to calculate for each 
> > process the offset from the beginning of the “range” for the
> > current time-step.
> > 
> > This could be done in parallel using a prefix sum [1].
> 
> Nah, the problem is not just knowing where to write in the file, but
> that it may not change throughout the process.
> The idea of parallel IO (with a distributed file system) is that a
> single file is sliced up over several hard disks, so that each task just
> has to write to its local disk. Otherwise, parallel IO wouldn't really
> help, as the bottleneck is the IO speed of the hard disks.

Oh, now I see what you mean with parallel I/O.

I was assuming a different definition of parallel I/O. At SciNet in
Toronto, they provide a parallel filesystem (GPFS), which allows
writing data in parallel from thousands of processes to a *single*
file. I believe this is the scenario that parallel HDF5 implements.


If each process writes to a separate file on a local disk, than the
range dataset would nevertheless work fine, would it not? Now you
would actually make use of both entries of range [first, last), and
ranges would be local to the process. After the simulation, the
files need to be pieced together manually, if desired.

Maybe I am missing the point, and there is a third definition ;-).

Peter