[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
RE: [Openexr-devel] RAW images in OpenEXR?
|
From: |
Alex Forsythe |
|
Subject: |
RE: [Openexr-devel] RAW images in OpenEXR? |
|
Date: |
Wed, 18 Jun 2008 13:15:07 -0700 |
Chris,
I don't want to get too off the EXR topic here ...
I'll just say, I admit that I have not done any testing using the adobe
approach, but I just strikes me as an issue that opens the door to debate.
Is there a metadata tag that would be appropriate for storage of RGB to XYZ
matrices for additional illuminants in order to avoid the issue?
Alex
_____________________________________________
Alexander Forsythe
Senior Imaging Engineer
Academy of Motion Picture Arts and Sciences
Address: 1313 Vine Street
Hollywood, CA 90028
Phone: 310-247-3000 x310
Fax: 310-247-3611
>>> "Chris Cox" <address@hidden> 6/18/2008 12:14 PM >>>
Alex;
In all testing so far, interpolating the matrices has given very, very good
results.
You can ask for more details on the DNG/ACR user forums.
Common light sources don't have that wide a variety of spectra. Cheap
fluorescent lamps and sodium lamps are the exceptions (and difficult to correct
for anyway).
No, nothing about the file format or the calibrations dictates how subsequent
image processing occurs, nor does anything else about the file format.
DNG is already in widespread use, with very few complaints about the format
(although there have been a few misunderstandings).
Chris
-----Original Message-----
From: Alexander Forsythe [mailto:address@hidden
Sent: Wed 6/18/2008 9:22 AM
To: Chris Cox; Florian Kainz; address@hidden
Subject: Re: [Openexr-devel] RAW images in OpenEXR?
Chris,
One issue with DNG I've always had is the storage and usage of Camera
RGB to XYZ matrices.
Correct me if I'm wrong, but DNG only allows for the storage of two
RGB to XYX matrices and it's suggested that they be built using
sources with the chromaticities of D65 and A.
I could see many situations where it'd be highly preferable to store
more than two RGB to XYZ matrices in metadata. The proposed OpenEXR
also seems to suffer from this lack of support for multiple RGB to XYZ
transforms.
My biggest concern with DNG is, and again correct me if I'm wrong
here, how DNG goes about computing matrices for RGB to XYZ transforms
when the illuminant is not either D65, or A.
From what I understand, the 9 matrix coefficients are interpolated to
derive a new matrix.
Now, I'll first suggest this isn't a very good method of determining
the RGB to XYZ matrix for a light source which may have a very
different spectral power distribution from the light sources already
accounted for. However, regardless of how well it may or may not work
in practice my real concern is more philosophical in nature. I don't
think any RAW file format should dictate how subsequent image
processing occurs. This 2+interpolation method does just that.
On another note -
Florian,
B44 and B44A are wonderful algorithms for traditional uses of OpenEXR,
but you correctly pointed out that Lossy compression methods (B44,
B44A) would introduce crosstalk between the channels. This is highly
undesirable and exactly what the Red camera folks are now taking a lot
of heat for. I'd suggest limiting compression to lossless compression
methods for RAW data storage.
Thanks
Alex
______________________________________
Alexander Forsythe
Senior Imaging Engineer
Academy of Motion Picture Arts and Sciences
Science and Technology Council
email - address@hidden
address - 1313 Vine Street
Hollywood, CA 90028
phone - 310-247-3000 x310
On May 6, 2008, at 3:10 PM, Chris Cox wrote:
> Why would you want something un-EXR like in EXR?
>
> Why not use existing open standards for camera RAW images?
>
> See http://www.adobe.com/products/dng/index.html
> and http://www.adobe.com/support/downloads/dng/dng_sdk.html
>
> Chris
>
>
>
> -----Original Message-----
> From: address@hidden on behalf of
> Florian Kainz
> Sent: Tue 5/6/2008 1:19 PM
> To: address@hidden
> Subject: [Openexr-devel] RAW images in OpenEXR?
>
>
> Recently several people have asked whether OpenEXR would be suitable
> for storing RAW images from cameras with color filter array sensors.
> The proposal below describes a method to do that. I would be
> interested
> in feedback from OpenEXR users.
>
> Florian
>
>
> OpenEXR RAW Images
> ------------------
>
> CFA Image Sensors And RAW Images
>
> Digital image file formats such as OpenEXR or JPEG usually
> represent
> images as red-green-blue (RGB) data. Conceptually, each pixel
> in an
> image file has a red, a green and a blue value. Image files may
> be
> compressed, and compression often involves transforming the RGB
> pixels to an alternate format before the data are stored in a
> file,
> but the original RGB data can be recovered from the file - at
> least
> approximately - by reversing this transformation.
>
> The image sensors in most modern electronic cameras do not record
> full RGB data for every pixel. Cameras typically use sensors that
> are equipped with color filter arrays. Each pixel in such a
> sensor
> is covered with a red, green or blue color filter. The filters
> are
> arranged in a regular pattern, for example, like this:
>
> G R G R G R
> B G B G B G
> G R G R G R
> B G B G B G
> G R G R G R
> B G B G B G
>
> To reconstruct a full-color picture from an image that has been
> recorded by such a color filter array sensor (CFA sensor), the
> image is first split into a red, a green and a blue channel:
>
> . R . R . R G . G . G . . . . . . .
> . . . . . . . G . G . G B . B . B .
> . R . R . R G . G . G . . . . . . .
> . . . . . . . G . G . G B . B . B .
> . R . R . R G . G . G . . . . . . .
> . . . . . . . G . G . G B . B . B .
>
> Some of the pixels in each channel contain no data (indicated
> by a period). Before combining the red, green and blue channels
> into a an RGB image, values for the empty pixels in each channel
> must be interpolated from neighboring pixels that do contain data.
>
> Not all CFA sensors use red, green and blue filters. For example,
> some cameras use green, magenta, yellow and cyan filters:
>
> G Y G Y G Y
> C M C M C M
> G Y G Y G Y
> C M C M C M
> G Y G Y G Y
> C M C M C M
>
> In another variation, the pixel grid in some image sensors is
> rotated 45 degrees with respect to the edges of the image:
>
> G G G G G
> B R B R B R
> G G G G G
> R B R B R B
> G G G G G
> B R B R B R
>
> Most electronic cameras automatically convert raw CFA sensor
> data to
> RGB images. The camera outputs RGB images and discards the raw
> data.
> However, some users prefer to use their cameras in "raw mode,"
> where
> the camera directly outputs the more ore less unaltered CFA sensor
> data. Reconstruction of RGB images is deferred to an offline
> process.
> Saving raw data can be desirable for two reasons:
>
> - An offline process that does not have to work in real time and
> within the often limited computing resources available in the
> camera may be able to reconstruct better looking RGB images.
>
> - Since raw sensor data contain only one value per pixel instead
> of
> three, a raw image occupies only a third as much space as an RGB
> image with the same bit depth and compression.
>
> Image files that contain raw CFA sensor data are often called
> "RAW files" or "camera RAW files."
>
> Storing RAW Images in OpenEXR Files
>
> It would be possible to store the output of a CFA image sensor
> directly in a single-channel OpenEXR image file. Additional
> information such as the colors and locations of the filters
> could be stored in an attribute in the file header. The need
> for image compression makes this approach undesirable. Every
> pixel in such a single-channel image is surrounded by pixels
> with different color filters. Existing compression methods in
> OpenEXR are not aware of this interleaving of image channels.
> Lossy compression methods (B44, B44A) would introduce crosstalk
> between the channels. Lossless compression methods (PIZ, ZIP)
> would preserve the image exactly, but the compression rate
> would suffer.
>
> Another way to store raw CFA sensor data is to split the image
> into multiple channels with one channel per filter color.
> OpenEXR's sub-sampled image channels provide an efficient way to
> represent the resulting sparsely populated channels. Since each
> filter color is stored in its own channel, existing compression
> methods work well. Lossy compression does not introduce crosstalk
> between filter colors, and lossless compression achieve nearly
> the same compression rates as for regular RGB images.
>
> Every channel in an OpenEXR image has an x and a y sampling rate.
> A channel contains data only for pixel locations whose x and y
> coordinates are evenly divisible by the x and y sampling rates:
>
> (x % xSampling == 0) && (y % ySampling == 0)
>
> For a CFA image sensor with RGB filters, we use the following
> sampling rates:
>
> channel xSampling ySampling
>
> R 2 2
> G 2 1
> B 2 2
>
> Now our OpenEXR file contains one R, two G and one B sample for
> every four pixels, just as in the sensor. However, the spatial
> arrangement of the samples differs:
>
> sensor file
>
> G R G R G R RGB . RGB . RGB .
> B G B G B G G . G . G .
> G R G R G R RGB . RGB . RGB .
> B G B G B G G . G . G .
> G R G R G R RGB . RGB . RGB .
> B G B G B G G . G . G .
>
> We must augment the file by describing the arrangement of the
> pixels in the sensor.
>
> The color filters in front of the pixels in the sensor are
> arranged
> in a regular pattern; the sensor is covered with repetitions of a
> two-by-two pixel tile:
>
> G R
> B G
>
> We can describe this pattern by adding a new CfaTile attribute to
> the OpenEXR file header:
>
> struct CfaPixel
> {
> string channelName;
> int xOffset;
> int yOffset;
> V3f XYZ;
> };
>
> class CfaTile
> {
> public:
>
> int xSize () const;
> int ySize () const;
> const CfaPixel & pixel (int x, int y) const;
> CfaPixel & pixel (int x, int y);
>
> ...
> };
>
> A CfaPixel, p, at location (x, y) in CfaTile t defines the
> following:
>
> * Channel p.channelName in the OpenEXR file has values for
> all pixels whose coordinates (px, py) are of the form
>
> px = x + n * t.xSize
> py = y + m * t.ySize
>
> In the file, the value for pixel (px, py) is stored at
> location
>
> (px + p.xOffset, py + p.offset)
>
> * p.XYZ is a set of weights for reconstructing CIE XYZ colors
> from the CFA sensor data. After all channels have been fully
> populated by interpolation, the XYZ color of each pixel
> computed as a weighted sum of all the channels:
>
> XYZpixels[py][px] = V3f (0, 0, 0);
>
> for (...)
> XYZpixels[py][px] += channel(p.channelName)[py][px]
> * p.XYZ;
>
> Once the XYZ color of a pixel is known, the color can be
> converted to any desired RGB space.
>
> * As a special case, if p.channelName is an empty string, then
> the file contains no data for this pixel.
>
> For example, the two-by-two-pixel CfaTile for our RGB CFA sensor
> would look like this:
>
> x y channelName xOffset yOffset XYZ
>
> 0 0 G 0 0 (0.3576, 0.7152, 0.1192)
> 1 0 R -1 0 (0.4124, 0.2126, 0.0193)
> 0 1 B 0 -1 (0.1805, 0.0722, 0.9505)
> 1 1 G -1 0 (0.3576, 0.7152, 0.1192)
>
> Using sub-sampled channels and a CfaTile attribute, we can also
> handle sensors with green, magenta, yellow and cyan filters:
>
> sensor file
>
> G Y G Y G Y GYCM . GYCM . GYCM .
> C M C M C M . . . . . .
> G Y G Y G Y GYCM . GYCM . GYCM .
> C M C M C M . . . . . .
> G Y G Y G Y GYCM . GYCM . GYCM .
> C M C M C M . . . . . .
>
> channels
>
> name xSampling ySampling
> G 2 2
> Y 2 2
> C 2 2
> M 2 2
>
> CfaTile (2x2)
>
> x y channelName xOffset yOffset XYZ
>
> 0 0 G 0 0 (...)
> 1 0 Y -1 0 (...)
> 0 1 C 0 -1 (...)
> 1 1 M -1 -1 (...)
>
> The same representation can also handle sensor pixel grids that
> are rotated by 45 degrees:
>
> sensor file
>
> G G G G G RGB . G RGB . G .
> B R B R B R . . . . . . .
> G G G G G RGB . G RGB . G .
> R B R B R B . . . . . . .
> G G G G G RGB . G RGB . G .
> B R B R B R . . . . . . .
>
> channels
>
> name xSampling ySampling
>
> R 4 2
> G 2 2
> B 4 2
>
> CfaTile (4x4)
>
> x y channelName xOffset yOffset XYZ
>
> 0 0 (empty) -1 0 (...)
> 1 0 G
> 2 0 (empty)
> 3 0 G -1 0 (...)
>
> 0 1 B 0 -1 (...)
> 1 1 (empty)
> 2 1 R -2 -1 (...)
> 3 1 (empty)
>
> 0 2 (empty)
> 1 2 G -1 0 (...)
> 2 2 (empty)
> 3 2 G -1 0 (...)
>
> 0 3 R 0 -1 (...)
> 1 3 (empty)
> 2 3 B -2 -1 (...)
> 3 3 (empty)
>
> In this last case both the OpenEXR image channels and the CfaTile
> pixel grid are rather sparsely populated. The corresponding
> interpolated RGB image will have a rather high resolution, but
> it will not contain fine detail. The interpolated image should
> probably be scaled down, either by a factor of sqrt(2) (resulting
> in the same number of R, G and B sensor samples per RGB pixel as
> for a non-rotated grid) or by a factor of 2 (resulting in one
> green sample per RGB pixel). This scale factor should perhaps
> be included in the CfaTile attribute.
>
> Integer or Floating-Point?
>
> Representing raw CFA sensor data with sub-sampled channels and
> a CfaTile attribute would work with either floating-point or
> integer channels. With floating-point channels, the pixel data
> would probably be scaled such that middle gray falls somewhere
> close to 0.18. With integer channels, middle gray might be
> represented as a value close to 9% of the maximum, for example,
> 1475 for a sensor that outputs 14-bit data with a maximum of
> 16383 (effectively mapping the maximum value to 2.0).
>
> The XYZ scale factors of the CfaPixels would compensate for the
> different scale factors of floating-point versus integer pixel
> data.
>
> Integers would be "more raw" than floating-point numbers; the
> pixels could represent the exact bit patterns produced by the
> analog-to-digital converter in the camera's sensor system.
>
> 16-bit floating-point numbers would introduce a mild form of
> lossy data compression. With 14-bit sensor output, numbers
> close to the maximum (16383) have a relative quantization step
> of about 0.006% while the quantization step of 16-bit floating-
> point numbers is 0.1%, so the conversion to floating-point is
> not lossless. Since raw integer sensor data are nearly linear
> relative to the number of photons captured by the sensor, small
> differences between integer values near the high end of the
> range are not significant for real-world image processing.
> The difference between 15000 and 15001 is completely invisible,
> as is the difference between 15000 and 15020. Conversion to
> floating-point does not affect image quality, but it does
> result in smaller file sizes because most of the compression
> algorithms in OpenEXR work best with 16-bit floating-point data.
> (PIZ and PXR24 do work reasonably well even with integer pixels.)
>
> Proof-of-Concept Implementation
>
> The attached tar bundle contains C++ source code for an
> implementation of the CfaTile attribute, and for a command-line
> program that converts an RGB image into a simulated OpenEXR raw
> RGB CFA sensor image. The program can also convert raw CFA sensor
> images back to RGB.
>
> What's Missing?
>
> The interpolation algorithm in the attached C++ code is a quick
> hack. It produces rather soft images and it suffers from edge
> artifacts. A production-ready implementation of the proposed
> raw image representation would need a much better interpolator.
>
> The proof-of-concept implementation lacks white balancing, flare
> suppression and other basic color correction. White balancing
> could be achieved by tweaking the XYZ weights in the CfaPixels,
> but additional header attributes are needed to transmit other
> color correction data. A CTL program would be a compact and
> very general way to represent this information.
>
> The OpenEXR library should probably contain some form of support
> for raw-to-RGB conversion. Ideally the RGBA interface would
> transparently perform this conversion during file reading.
>
> It is unlikely that a purely software based raw-to-RGB conversion
> would be fast enough to allow reading of OpenEXR raw images at
> high frame rates. Real-time playback software would probably have
> to upload the raw data to into a graphics card and perform
> conversion
> to RGB in a GPU-based pixel shader, similar to how playexr handles
> luminance/chroma images.
>
> And of course, camera manufacturers will have to agree to output
> OpenEXR raw files.
>
>
>
>
> _______________________________________________
> Openexr-devel mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/openexr-devel
_______________________________________________
Openexr-devel mailing list
address@hidden
http://lists.nongnu.org/mailman/listinfo/openexr-devel
- Re: [Openexr-devel] RAW images in OpenEXR?, Florian Kainz, 2008/06/18
- RE: [Openexr-devel] RAW images in OpenEXR?, Chris Cox, 2008/06/18
- RE: [Openexr-devel] RAW images in OpenEXR?,
Alex Forsythe <=
- RE: [Openexr-devel] RAW images in OpenEXR?, Chris Cox, 2008/06/18
- RE: [Openexr-devel] RAW images in OpenEXR?, Lars Borg, 2008/06/18
- RE: [Openexr-devel] RAW images in OpenEXR?, Alex Forsythe, 2008/06/18
- RE: [Openexr-devel] RAW images in OpenEXR?, Lars Borg, 2008/06/18
- RE: [Openexr-devel] RAW images in OpenEXR?, Alex Forsythe, 2008/06/18