Re: [h5md-user] Dataset layouts

[Pierre de Buyl]

Hi Peter,

On Tue, Jun 09, 2015 at 04:15:06PM -0400, Peter Colberg wrote:
> While working on a truly large-scale scientific application using
> the HDF5 MPI I/O driver, I noticed it would be useful to mandate that
> “meta-datasets” are always stored with compact layout. This would
> apply to proposal 100 (scalar step/time), and `box/edges` for the
> fixed-geometry case.
> 
> When using parallel HDF5, most C API calls are collective, which means
> that a function must be called by all processes in the communicator
> used to access a file. Compact datasets are metadata and therefore
> must be written/read collectively. This goes nicely with the other
> metadata stored in attributes.

I understand the motivation. Did you notice a difference in performance or is
there some other motivation?

> In similar spirit, the specification states that time-dependent
> datasets have an infinite first dimension. I suggest we specify
> explicitly that these datasets must have a chunked layout.

This should not be controversial :-) One cannot make an extensible dataset
without chunking.

Cheers,

Pierre