[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: problem loading hdf5 file
From: |
David Bateman |
Subject: |
Re: problem loading hdf5 file |
Date: |
Mon, 3 May 2004 15:19:38 +0200 |
User-agent: |
Mutt/1.4.1i |
Ok, the problem is simple to state, but a bit harder to fix... Octave
only know two compund datatype. Complex and range, with the forms
Type: struct {
"real" +0 native double
"imag" +8 native double
} 16 bytes
Type: struct {
"base" +0 native double
"limit" +8 native double
"increment" +16 native double
} 24 bytes
Your files have compound datatypes, containing arbitrary values
representing a structure... Octave assumes that such data will be
passed to it as an HDF5 group, rather than a compound type of a
dataset. "h5ls -v <file>" identifies the following compound types
for your file
Type: struct {
"instrumentModeId" +16 32-bit big-endian integer
"julianTimetag" +0 IEEE 64-bit big-endian float
"version" +12 32-bit big-endian integer
"minWavelength" +24 IEEE 64-bit big-endian float
"maxWavelength" +32 IEEE 64-bit big-endian float
"irradiance" +40 IEEE 64-bit big-endian float
"irradianceUncertainty" +48 IEEE 64-bit big-endian float
"quality" +56 IEEE 64-bit big-endian float
} 64 bytes
Type: struct {
"julianTimetag" +0 IEEE 64-bit big-endian float
"parameterName" +32 39-byte null-terminated ASCII string
"value" +16 IEEE 64-bit big-endian float
"uncertainty" +24 IEEE 64-bit big-endian float
} 72 bytes
Type: struct {
"julianTimetag" +0 IEEE 64-bit big-endian float
"averageJulianTime" +16 IEEE 64-bit big-endian float
"averageJulianTimeStdev" +24 IEEE 64-bit big-endian float
"correctedIrradiance" +32 IEEE 64-bit big-endian float
"correctedIrradianceUncertainty" +40 IEEE 64-bit
big-endian float
"correctedIrradianceStdev" +48 IEEE 64-bit big-endian float
"trueEarthIrradiance" +56 IEEE 64-bit big-endian float
"trueEarthIrradianceUncertainty" +64 IEEE 64-bit
big-endian float
"trueEarthIrradianceStdev" +72 IEEE 64-bit big-endian float
} 80 bytes
Type: struct {
"julianTimetag" +0 IEEE 64-bit big-endian float
"averageJulianTimetag" +8 IEEE 64-bit big-endian float
"averageJulianTimetagStDev" +16 IEEE 64-bit big-endian
float
"timeSpanInHours" +24 8-bit integer
"diodeNumber" +28 32-bit big-endian integer
"version" +32 16-bit big-endian integer
"minWavelengthInBandpass" +36 IEEE 32-bit big-endian float
"maxWavelengthInBandpass" +40 IEEE 32-bit big-endian float
"medianIrradiance" +48 IEEE 64-bit big-endian float
"averageIrradiance" +56 IEEE 64-bit big-endian float
"absoluteUncertainty" +64 IEEE 64-bit big-endian float
"measurementPrecision" +72 IEEE 64-bit big-endian float
"calculationPrecision" +80 IEEE 64-bit big-endian float
"degradationModel" +88 8-bit integer
"numberOfPoints" +92 32-bit big-endian integer
} 96 bytes
>From which you creat a vector of these compound types. So there are two
choices, can your software save these as groups? If so then the easiest
would be to save the data as group, where the names of each subgroup
takes on the names of the variables in your compound type. Each of
these could be a normal HDF5 matrix of float or integers. Octave will
then interpret this as a structure with the array values you'd expect.
Your HDF5 should have "h5ls -r -v" give something like (description
written by hand, so its a fake example).
a Group
Location: 0:1:0:1576
Links: 1
a/julianTimetag Dataset {1/1, 10/10}
Location: 0:1:0:9960
Links: 1
Modified: 2004-05-03 15:07:39 CEST
Storage: 80 logical bytes, 80 allocated bytes, 100.00% utilization
Type: native double
a/parameterName Dataset {1/1, 10/10}
Location: 0:1:0:1952
Links: 1
Modified: 2004-05-03 15:07:39 CEST
Storage: 390 logical bytes, 390 allocated bytes, 100.00% utilization
Type: 39-byte null-terminated ASCII string
a/uncertainity Dataset {1/1, 10/10}
Location: 0:1:0:7072
Links: 1
Modified: 2004-05-03 15:07:39 CEST
Storage: 80 logical bytes, 80 allocated bytes, 100.00% utilization
Type: native double
a/value Dataset {1/1, 10/10}
Location: 0:1:0:11480
Links: 1
Modified: 2004-05-03 15:07:39 CEST
Storage: 80 logical bytes, 80 allocated bytes, 100.00% utilization
Type: native double
The second option of modifiying octave to read your datatype is not
that easy, in that how do we create the correct compound datatype,
since we have to consider the datatype as arbitrary? What I see might
be possible is to use H5Tget_nmembers to find the number of member
elements in the compound type, and then H5Tget_member_name and
H5Tget_member_type, etc to bit by bit reconstruct a compatiable
compound type to hold your data to be imported as a structure.
This is a non-trivial task, and is probably a few days work.. I'm
willing to give some pointers if you want to try and write the code
yourself, but I don't have to time to do this for you....
Regards
David
According to E. Joshua Rigler <address@hidden> (on 05/01/04):
> I'm using 2.1.57 and the latest octave-forge (not CVS). The data *is*
> somewhat complex as it contains solar spectral measurements at many
> wavelengths, and from different instruments, with corresponding
> uncertainties, etc. I've attached the one test file I have, but here's
> the output you requested anyway:
>
> h5ls:
> Solar\ Spectral\ Irradiance Dataset {1083}
> Solar\ and\ Geophysical\ Parameters Dataset {4}
> Total\ Solar\ Irradiance Dataset {1}
> XPS\ EUV\ Solar\ Spectrum Dataset {11}
>
> h5ls -r:
> /Solar\ Spectral\ Irradiance Dataset {1083}
> /Solar\ and\ Geophysical\ Parameters Dataset {4}
> /Total\ Solar\ Irradiance Dataset {1}
> /XPS\ EUV\ Solar\ Spectrum Dataset {11}
>
>
> Also, you might want to read this page for a detailed description of the
> different data types (there's quite the variety of different floats,
> integers, and strings in each file):
>
> http://daac.gsfc.nasa.gov/upperatm/sorce/level3_ssi.html
>
> Thanks for looking into this with me.
>
> -EJR
>
>
> On Fri, 2004-04-30 at 17:09, David Bateman wrote:
> > Can you send me the files, as the ftp site says it can't set guest
> > privileges. Also what does "h5ls <file>" and "h5ls -r <file>" show?
> >
> > It is is relatively simple data, there is no reason the file shouldn't
> > be able to be loaded in octave. BTW, what version of octave are you
> > using? There were major changes in the HDF code around 2.1.53, so I
> > suggest you use a new version of octave.
> >
> > Regards
> > David
>
--
David Bateman address@hidden
Motorola CRM +33 1 69 35 48 04 (Ph)
Parc Les Algorithmes, Commune de St Aubin +33 1 69 35 77 01 (Fax)
91193 Gif-Sur-Yvette FRANCE
The information contained in this communication has been classified as:
[x] General Business Information
[ ] Motorola Internal Use Only
[ ] Motorola Confidential Proprietary
-------------------------------------------------------------
Octave is freely available under the terms of the GNU GPL.
Octave's home on the web: http://www.octave.org
How to fund new projects: http://www.octave.org/funding.html
Subscription information: http://www.octave.org/archive.html
-------------------------------------------------------------
- Re: problem loading hdf5 file,
David Bateman <=