Re: New undocumented mat-file format!!!!

[David Bateman]

John W. Eaton wrote:
> On 12-Apr-2007, David Bateman wrote:
>
> | While struggling to reverse engineer matlab's undocumented function
> | handle saving/loading code, I noticed the statement below in matlab's
> | release notes
> | 
> | <quote>
> | New MAT-File Format Based on HDF5
> | 
> | In Version 7.3 (R2006b), you can save MAT-files in a format based on
> | HDF5. Unlike earlier MAT-file formats, the HDF5-based format is capable
> | of saving variables that occupy more than 2 GB of storage, including
> | large arrays created on 64-bit systems.
> | 
> | To save a MAT-file in the HDF5-based format, use the -v7.3 option to the
> | MATLAB save function or the "w7.3" mode argument to the C or Fortran
> | matOpen function. The default MAT-file format is the same as that in
> | Version 7.2 (R2006a).
> | </quote>
> | 
> | This certainly going to turn around and bite us in the future, though as
> | its not the default format at the moment we should be ok for a little
> | long. We'll certainly be incompatible with their HDF5 format, though at
> | this point I can't find any documentation on exactly what this format is.
>
> Isn't there a program that dumps the structure of an HDF5 file so at
> least it might not be too hard to figure out what is going on
> (compared to an arbitrary proprietary binary format).
>   
Yes h5dump. However, if matlab did the same thing for the hdf5 format as
they did with their compressed file format, then I fear it will be
encapsulated in a standard mat-file, this might cause issues using a
dumping utility.

Ok, I just did

>> a = [1,3;2,4];
>> save -v7.3 test.mat a
>> quit
% h5dump test.mat

and got

HDF5 "test.mat" {
GROUP "/" {
   DATASET "a" {
      DATATYPE  H5T_IEEE_F64LE
      DATASPACE  SIMPLE { ( 2, 2 ) / ( 2, 2 ) }
      DATA {
      (0,0): 1, 2,
      (1,0): 3, 4
      }
      ATTRIBUTE "MATLAB_class" {
         DATATYPE  H5T_STRING {
               STRSIZE 6;
               STRPAD H5T_STR_NULLTERM;
               CSET H5T_CSET_ASCII;
               CTYPE H5T_C_S1;
            }
         DATASPACE  SCALAR
         DATA {
         (0): "double"
         }
      }
   }
}
}

> At some point after 3.0 I think we should consider
>
>   * implementing a thin wrapper around the HDF5 library
>   
The problem with a thin wrapper is it can't load arbitrary types. The
format of the existing hdf5 formats perhaps should be reconsidered in
that the choice of making each value a group with a type and and value
might be better to have the type as an hdf5 attribute of the value and
get rid of the encapsulating hdf5 group. This might make the importing
of octave files into non octave programs easier, with care it can also
be made almost identical to the matlab file format.
The reverse case of externally created hdf5 into octave is already
pretty good.

>   * writing load/save for the new Matlab HDF5 format using the
>     wrapper, so it would be in .m files and easy to modify
>   
I'd tend to want and try and make the octave hdf5 format identical to
the matlab one if possible

>   * dropping (at least some of) the old formats we have (we could
>     provide a standalone conversion program that could take any of the
>     old formats and rewrite them to use the new HDF5 format
>   
Agreed.

> | In any case this is just a heads up of an up coming issue.
>
> OK, thanks.
>
> jwe
>
>   
D.


-- 
David Bateman                                address@hidden
Motorola Labs - Paris                        +33 1 69 35 48 04 (Ph) 
Parc Les Algorithmes, Commune de St Aubin    +33 6 72 01 06 33 (Mob) 
91193 Gif-Sur-Yvette FRANCE                  +33 1 69 35 77 01 (Fax) 

The information contained in this communication has been classified as: 

[x] General Business Information 
[ ] Motorola Internal Use Only 
[ ] Motorola Confidential Proprietary