[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [h5md-user] Variable-size particle groups
From: |
Peter Colberg |
Subject: |
Re: [h5md-user] Variable-size particle groups |
Date: |
Tue, 29 May 2012 11:43:17 -0400 |
User-agent: |
Mutt/1.5.21 (2010-09-15) |
On Tue, May 29, 2012 at 05:32:48PM +0200, Olaf Lenz wrote:
> Hi!
>
> On 05/29/2012 05:17 PM, Peter Colberg wrote:
> > At any given timestep, each processor owns a defined set of
> > particles of varying number, which shall be written to an HDF5
> > dataset. The task that needs to be solved is to calculate for each
> > process the offset from the beginning of the “range” for the
> > current time-step.
> >
> > This could be done in parallel using a prefix sum [1].
>
> Nah, the problem is not just knowing where to write in the file, but
> that it may not change throughout the process.
> The idea of parallel IO (with a distributed file system) is that a
> single file is sliced up over several hard disks, so that each task just
> has to write to its local disk. Otherwise, parallel IO wouldn't really
> help, as the bottleneck is the IO speed of the hard disks.
Oh, now I see what you mean with parallel I/O.
I was assuming a different definition of parallel I/O. At SciNet in
Toronto, they provide a parallel filesystem (GPFS), which allows
writing data in parallel from thousands of processes to a *single*
file. I believe this is the scenario that parallel HDF5 implements.
If each process writes to a separate file on a local disk, than the
range dataset would nevertheless work fine, would it not? Now you
would actually make use of both entries of range [first, last), and
ranges would be local to the process. After the simulation, the
files need to be pieced together manually, if desired.
Maybe I am missing the point, and there is a third definition ;-).
Peter
- [h5md-user] Variable-size particle groups, Peter Colberg, 2012/05/27
- Re: [h5md-user] Variable-size particle groups, Felix Höfling, 2012/05/29
- Re: [h5md-user] Variable-size particle groups, Olaf Lenz, 2012/05/29
- Re: [h5md-user] Variable-size particle groups, Olaf Lenz, 2012/05/29
- Re: [h5md-user] Variable-size particle groups, Peter Colberg, 2012/05/29
- Re: [h5md-user] Variable-size particle groups, Olaf Lenz, 2012/05/29
- Re: [h5md-user] Variable-size particle groups,
Peter Colberg <=
- Re: [h5md-user] Variable-size particle groups, Olaf Lenz, 2012/05/29
- Re: [h5md-user] Variable-size particle groups, Peter Colberg, 2012/05/29
- Re: [h5md-user] Variable-size particle groups, Pierre de Buyl, 2012/05/29
- Re: [h5md-user] Variable-size particle groups, Pierre de Buyl, 2012/05/29