Re: problem loading hdf5 file

[David Bateman]

Ok, the problem is simple to state, but a bit harder to fix... Octave
only know two compund datatype. Complex and range, with the forms

  Type:      struct {
                   "real"             +0    native double
                   "imag"             +8    native double
               } 16 bytes

  Type:      struct {
                   "base"             +0    native double
                   "limit"            +8    native double
                   "increment"        +16   native double
               } 24 bytes

Your files have compound datatypes, containing arbitrary values
representing a structure... Octave assumes that such data will be
passed to it as an HDF5 group, rather than a compound type of a
dataset. "h5ls -v <file>" identifies the following compound types
for your file

    Type:      struct {
                   "instrumentModeId" +16   32-bit big-endian integer
                   "julianTimetag"    +0    IEEE 64-bit big-endian float
                   "version"          +12   32-bit big-endian integer
                   "minWavelength"    +24   IEEE 64-bit big-endian float
                   "maxWavelength"    +32   IEEE 64-bit big-endian float
                   "irradiance"       +40   IEEE 64-bit big-endian float
                   "irradianceUncertainty" +48   IEEE 64-bit big-endian float
                   "quality"          +56   IEEE 64-bit big-endian float
               } 64 bytes

    Type:      struct {
                   "julianTimetag"    +0    IEEE 64-bit big-endian float
                   "parameterName"    +32   39-byte null-terminated ASCII string
                   "value"            +16   IEEE 64-bit big-endian float
                   "uncertainty"      +24   IEEE 64-bit big-endian float
               } 72 bytes



    Type:      struct {
                   "julianTimetag"    +0    IEEE 64-bit big-endian float
                   "averageJulianTime" +16   IEEE 64-bit big-endian float
                   "averageJulianTimeStdev" +24   IEEE 64-bit big-endian float
                   "correctedIrradiance" +32   IEEE 64-bit big-endian float
                   "correctedIrradianceUncertainty" +40   IEEE 64-bit 
big-endian float
                   "correctedIrradianceStdev" +48   IEEE 64-bit big-endian float
                   "trueEarthIrradiance" +56   IEEE 64-bit big-endian float
                   "trueEarthIrradianceUncertainty" +64   IEEE 64-bit 
big-endian float
                   "trueEarthIrradianceStdev" +72   IEEE 64-bit big-endian float
               } 80 bytes
    Type:      struct {
                   "julianTimetag"    +0    IEEE 64-bit big-endian float
                   "averageJulianTimetag" +8    IEEE 64-bit big-endian float
                   "averageJulianTimetagStDev" +16   IEEE 64-bit big-endian 
float
                   "timeSpanInHours"  +24   8-bit integer
                   "diodeNumber"      +28   32-bit big-endian integer
                   "version"          +32   16-bit big-endian integer
                   "minWavelengthInBandpass" +36   IEEE 32-bit big-endian float
                   "maxWavelengthInBandpass" +40   IEEE 32-bit big-endian float
                   "medianIrradiance" +48   IEEE 64-bit big-endian float
                   "averageIrradiance" +56   IEEE 64-bit big-endian float
                   "absoluteUncertainty" +64   IEEE 64-bit big-endian float
                   "measurementPrecision" +72   IEEE 64-bit big-endian float
                   "calculationPrecision" +80   IEEE 64-bit big-endian float
                   "degradationModel" +88   8-bit integer
                   "numberOfPoints"   +92   32-bit big-endian integer
               } 96 bytes

>From which you creat a vector of these compound types. So there are two 
choices, can your software save these as groups?  If so then the easiest
would be to save the data as group, where the names of each subgroup
takes on the names of the variables in your compound type. Each of
these could be a normal HDF5 matrix of float or integers. Octave will
then interpret this as a structure with the array values you'd expect.
Your HDF5 should have "h5ls -r -v" give something like (description
written by hand, so its a fake example).

a                        Group
    Location:  0:1:0:1576
    Links:     1
a/julianTimetag          Dataset {1/1, 10/10}
    Location:  0:1:0:9960
    Links:     1
    Modified:  2004-05-03 15:07:39 CEST
    Storage:   80 logical bytes, 80 allocated bytes, 100.00% utilization
    Type:      native double
a/parameterName          Dataset {1/1, 10/10}
    Location:  0:1:0:1952
    Links:     1
    Modified:  2004-05-03 15:07:39 CEST
    Storage:   390 logical bytes, 390 allocated bytes, 100.00% utilization
    Type:      39-byte null-terminated ASCII string
a/uncertainity           Dataset {1/1, 10/10}
    Location:  0:1:0:7072
    Links:     1
    Modified:  2004-05-03 15:07:39 CEST
    Storage:   80 logical bytes, 80 allocated bytes, 100.00% utilization
    Type:      native double
a/value                  Dataset {1/1, 10/10}
    Location:  0:1:0:11480
    Links:     1
    Modified:  2004-05-03 15:07:39 CEST
    Storage:   80 logical bytes, 80 allocated bytes, 100.00% utilization
    Type:      native double

The second option of modifiying octave to read your datatype is not
that easy, in that how do we create the correct compound datatype,
since we have to consider the datatype as arbitrary?  What I see might
be possible is to use H5Tget_nmembers to find the number of member
elements in the compound type, and then H5Tget_member_name and
H5Tget_member_type, etc to bit by bit reconstruct a compatiable
compound type to hold your data to be imported as a structure. 

This is a non-trivial task, and is probably a few days work..  I'm
willing to give some pointers if you want to try and write the code
yourself, but I don't have to time to do this for you....

Regards
David

According to E. Joshua Rigler <address@hidden> (on 05/01/04):
> I'm using 2.1.57 and the latest octave-forge (not CVS).  The data *is*
> somewhat complex as it contains solar spectral measurements at many
> wavelengths, and from different instruments, with corresponding
> uncertainties, etc.  I've attached the one test file I have, but here's
> the output you requested anyway:
> 
> h5ls:
>  Solar\ Spectral\ Irradiance Dataset {1083}
>  Solar\ and\ Geophysical\ Parameters Dataset {4}
>  Total\ Solar\ Irradiance Dataset {1}
>  XPS\ EUV\ Solar\ Spectrum Dataset {11}
> 
> h5ls -r:
>  /Solar\ Spectral\ Irradiance Dataset {1083} 
>  /Solar\ and\ Geophysical\ Parameters Dataset {4} 
>  /Total\ Solar\ Irradiance Dataset {1}
>  /XPS\ EUV\ Solar\ Spectrum Dataset {11}
> 
> 
> Also, you might want to read this page for a detailed description of the
> different data types (there's quite the variety of different floats,
> integers, and strings in each file):
> 
>  http://daac.gsfc.nasa.gov/upperatm/sorce/level3_ssi.html
> 
> Thanks for looking into this with me.
> 
> -EJR
> 
> 
> On Fri, 2004-04-30 at 17:09, David Bateman wrote:
> > Can you send me the files, as the ftp site says it can't set guest
> > privileges.  Also what does "h5ls <file>" and "h5ls -r <file>" show?
> > 
> > It is is relatively simple data, there is no reason the file shouldn't
> > be able to be loaded in octave. BTW, what version of octave are you
> > using? There were major changes in the HDF code around 2.1.53, so I
> > suggest you use a new version of octave.
> > 
> > Regards
> > David
> 

-- 
David Bateman                                address@hidden
Motorola CRM                                 +33 1 69 35 48 04 (Ph) 
Parc Les Algorithmes, Commune de St Aubin    +33 1 69 35 77 01 (Fax) 
91193 Gif-Sur-Yvette FRANCE

The information contained in this communication has been classified as: 

[x] General Business Information 
[ ] Motorola Internal Use Only 
[ ] Motorola Confidential Proprietary



-------------------------------------------------------------
Octave is freely available under the terms of the GNU GPL.

Octave's home on the web:  http://www.octave.org
How to fund new projects:  http://www.octave.org/funding.html
Subscription information:  http://www.octave.org/archive.html
-------------------------------------------------------------