[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [h5md-user] Dataset layouts

From: Pierre de Buyl
Subject: Re: [h5md-user] Dataset layouts
Date: Tue, 9 Jun 2015 22:30:31 +0200
User-agent: Mutt/1.5.23 (2014-03-12)

Hi Peter,

On Tue, Jun 09, 2015 at 04:15:06PM -0400, Peter Colberg wrote:
> While working on a truly large-scale scientific application using
> the HDF5 MPI I/O driver, I noticed it would be useful to mandate that
> “meta-datasets” are always stored with compact layout. This would
> apply to proposal 100 (scalar step/time), and `box/edges` for the
> fixed-geometry case.
> When using parallel HDF5, most C API calls are collective, which means
> that a function must be called by all processes in the communicator
> used to access a file. Compact datasets are metadata and therefore
> must be written/read collectively. This goes nicely with the other
> metadata stored in attributes.

I understand the motivation. Did you notice a difference in performance or is
there some other motivation?

> In similar spirit, the specification states that time-dependent
> datasets have an infinite first dimension. I suggest we specify
> explicitly that these datasets must have a chunked layout.

This should not be controversial :-) One cannot make an extensible dataset
without chunking.



reply via email to

[Prev in Thread] Current Thread [Next in Thread]