Re: [h5md-user] Strings again / parallel issue

[Felix Höfling]

Am 04.02.2014, 16:05 Uhr, schrieb Pierre de Buyl  
<address@hidden>:

> Hi all,
>
> I just discovered a serious annoyance with HDF5 variable-length types.  
> You
> cannot use them in parallel [1].
>
> This may have slipped through because nobody is doing parallel H5MD  
> right now,
> however for large simulations in parallel (with existing code like  
> ESPResSo or
> ESPResSo++) this will become a problem really fast.
>
> I became interested in this because I was toying with topologies (not  
> yet part
> of H5MD so no problem for now) but all the strings in H5MD are  
> variable-length
> currently. I thus started to check the behaviour for our string data.
>
> Unless someone has a solution to this problem (because I misunderstood
> something), we probably need to change all VL strings to fixed-length  
> strings.
> The only real annoyance is for arrays of strings (such as the box  
> boundary that
> may hold strings of different lengths) were the maximum length must be  
> used for
> the string type.
>
> To put the timeframe into context, our paper [2] (me, peter and felix)  
> is accepted
> now and we are waiting for the proofs. So this should be set within the  
> next few
> days.
>
> I attach parallel_string_attribute.f90 (fixed length attribute, works in
> parallel) and parallel_vl_string_attribute.f90 (vl strings, doesn't work  
> as far
> as I can tell). They require parallel hdf5, indeed.
>
> Best,
>
> Pierre
>
> [1] http://www.hdfgroup.org/hdf5-quest.html#pvl
> [2] http://arxiv.org/abs/1308.6382

I always felt uncomfortable with specialising to a certain String type. I  
suggest to drop "VL" and allow _any_ kind of HDF5 String type for the  
writer. The reader can easily handle the different cases by querying  
H5Tis_variable_str
http://www.hdfgroup.org/HDF5/doc/RM/RM_H5T.html#Datatype-IsVariableString

For reading a string attribute, you may have a look at the code snippet  
below.

Regards,

Felix

std::string read_string_attribute(hid_t attr_id)
{
    std::string value;
    hid_t type_id = H5Aget_type(attr_id));
    htri_t is_varlen_str =  H5Tis_variable_str(type_id));
    if (is_varlen_str > 0) {
        // read fixed-size string, allocate space in advance and let the  
HDF5
        // library take care about NULLTERM and NULLPAD strings
        hsize_t size = H5Tget_size(type_id);
        hid_t mem_type_id = H5Tcopy(H5T_C_S1);
        H5Tset_size(mem_type_id, size);
        value.resize(size, std::string::value_type());
        H5Aread(attr_id, mem_type_id, &*value.begin());
    }  else {
        // read variable-length string, memory will be allocated by HDF5 C
        // library and must be freed by us
        char *c_str;
        if (H5Aread(attr_id, H5T_C_S1, &c_str) >= 0) {
            value = c_str;  // copy '\0'-terminated string
            free(c_str);
        }
    }
    H5Tclose(type_id);
    H5Aclose(attr_id);
    return value;
}