Chris,
I don't want to get too off the EXR topic here ...
I'll just say, I admit that I have not done any testing using the
adobe approach, but I just strikes me as an issue that opens the
door to debate.
Is there a metadata tag that would be appropriate for storage of RGB
to XYZ matrices for additional illuminants in order to avoid the
issue?
Alex
_____________________________________________
Alexander Forsythe
Senior Imaging Engineer
Academy of Motion Picture Arts and Sciences
Address: 1313 Vine Street
Hollywood, CA 90028
Phone: 310-247-3000 x310
Fax: 310-247-3611
"Chris Cox" <address@hidden> 6/18/2008 12:14 PM >>>
Alex;
In all testing so far, interpolating the matrices has given very,
very good results.
You can ask for more details on the DNG/ACR user forums.
Common light sources don't have that wide a variety of spectra.
Cheap fluorescent lamps and sodium lamps are the exceptions (and
difficult to correct for anyway).
No, nothing about the file format or the calibrations dictates how
subsequent image processing occurs, nor does anything else about the
file format.
DNG is already in widespread use, with very few complaints about the
format (although there have been a few misunderstandings).
Chris
-----Original Message-----
From: Alexander Forsythe [mailto:address@hidden
Sent: Wed 6/18/2008 9:22 AM
To: Chris Cox; Florian Kainz; address@hidden
Subject: Re: [Openexr-devel] RAW images in OpenEXR?
Chris,
One issue with DNG I've always had is the storage and usage of Camera
RGB to XYZ matrices.
Correct me if I'm wrong, but DNG only allows for the storage of two
RGB to XYX matrices and it's suggested that they be built using
sources with the chromaticities of D65 and A.
I could see many situations where it'd be highly preferable to store
more than two RGB to XYZ matrices in metadata. The proposed OpenEXR
also seems to suffer from this lack of support for multiple RGB to XYZ
transforms.
My biggest concern with DNG is, and again correct me if I'm wrong
here, how DNG goes about computing matrices for RGB to XYZ transforms
when the illuminant is not either D65, or A.
From what I understand, the 9 matrix coefficients are interpolated to
derive a new matrix.
Now, I'll first suggest this isn't a very good method of determining
the RGB to XYZ matrix for a light source which may have a very
different spectral power distribution from the light sources already
accounted for. However, regardless of how well it may or may not work
in practice my real concern is more philosophical in nature. I don't
think any RAW file format should dictate how subsequent image
processing occurs. This 2+interpolation method does just that.
On another note -
Florian,
B44 and B44A are wonderful algorithms for traditional uses of OpenEXR,
but you correctly pointed out that Lossy compression methods (B44,
B44A) would introduce crosstalk between the channels. This is highly
undesirable and exactly what the Red camera folks are now taking a lot
of heat for. I'd suggest limiting compression to lossless compression
methods for RAW data storage.
Thanks
Alex
______________________________________
Alexander Forsythe
Senior Imaging Engineer
Academy of Motion Picture Arts and Sciences
Science and Technology Council
email - address@hidden
address - 1313 Vine Street
Hollywood, CA 90028
phone - 310-247-3000 x310
On May 6, 2008, at 3:10 PM, Chris Cox wrote:
Why would you want something un-EXR like in EXR?
Why not use existing open standards for camera RAW images?
See http://www.adobe.com/products/dng/index.html
and http://www.adobe.com/support/downloads/dng/dng_sdk.html
Chris
-----Original Message-----
From: address@hidden on behalf of
Florian Kainz
Sent: Tue 5/6/2008 1:19 PM
To: address@hidden
Subject: [Openexr-devel] RAW images in OpenEXR?
Recently several people have asked whether OpenEXR would be suitable
for storing RAW images from cameras with color filter array sensors.
The proposal below describes a method to do that. I would be
interested
in feedback from OpenEXR users.
Florian
OpenEXR RAW Images
------------------
CFA Image Sensors And RAW Images
Digital image file formats such as OpenEXR or JPEG usually
represent
images as red-green-blue (RGB) data. Conceptually, each pixel
in an
image file has a red, a green and a blue value. Image files may
be
compressed, and compression often involves transforming the RGB
pixels to an alternate format before the data are stored in a
file,
but the original RGB data can be recovered from the file - at
least
approximately - by reversing this transformation.
The image sensors in most modern electronic cameras do not record
full RGB data for every pixel. Cameras typically use sensors that
are equipped with color filter arrays. Each pixel in such a
sensor
is covered with a red, green or blue color filter. The filters
are
arranged in a regular pattern, for example, like this:
G R G R G R
B G B G B G
G R G R G R
B G B G B G
G R G R G R
B G B G B G
To reconstruct a full-color picture from an image that has been
recorded by such a color filter array sensor (CFA sensor), the
image is first split into a red, a green and a blue channel:
. R . R . R G . G . G . . . . . . .
. . . . . . . G . G . G B . B . B .
. R . R . R G . G . G . . . . . . .
. . . . . . . G . G . G B . B . B .
. R . R . R G . G . G . . . . . . .
. . . . . . . G . G . G B . B . B .
Some of the pixels in each channel contain no data (indicated
by a period). Before combining the red, green and blue channels
into a an RGB image, values for the empty pixels in each channel
must be interpolated from neighboring pixels that do contain data.
Not all CFA sensors use red, green and blue filters. For example,
some cameras use green, magenta, yellow and cyan filters:
G Y G Y G Y
C M C M C M
G Y G Y G Y
C M C M C M
G Y G Y G Y
C M C M C M
In another variation, the pixel grid in some image sensors is
rotated 45 degrees with respect to the edges of the image:
G G G G G
B R B R B R
G G G G G
R B R B R B
G G G G G
B R B R B R
Most electronic cameras automatically convert raw CFA sensor
data to
RGB images. The camera outputs RGB images and discards the raw
data.
However, some users prefer to use their cameras in "raw mode,"
where
the camera directly outputs the more ore less unaltered CFA sensor
data. Reconstruction of RGB images is deferred to an offline
process.
> Saving raw data can be desirable for two reasons:
- An offline process that does not have to work in real time and
within the often limited computing resources available in the
camera may be able to reconstruct better looking RGB images.
>
- Since raw sensor data contain only one value per pixel instead
of
three, a raw image occupies only a third as much space as an RGB
image with the same bit depth and compression.
Image files that contain raw CFA sensor data are often called
"RAW files" or "camera RAW files."
Storing RAW Images in OpenEXR Files
It would be possible to store the output of a CFA image sensor
directly in a single-channel OpenEXR image file. Additional
information such as the colors and locations of the filters
could be stored in an attribute in the file header. The need
for image compression makes this approach undesirable. Every
pixel in such a single-channel image is surrounded by pixels
with different color filters. Existing compression methods in
OpenEXR are not aware of this interleaving of image channels.
Lossy compression methods (B44, B44A) would introduce crosstalk
between the channels. Lossless compression methods (PIZ, ZIP)
would preserve the image exactly, but the compression rate
would suffer.
Another way to store raw CFA sensor data is to split the image
into multiple channels with one channel per filter color.
OpenEXR's sub-sampled image channels provide an efficient way to
represent the resulting sparsely populated channels. Since each
filter color is stored in its own channel, existing compression
methods work well. Lossy compression does not introduce crosstalk
between filter colors, and lossless compression achieve nearly
the same compression rates as for regular RGB images.
Every channel in an OpenEXR image has an x and a y sampling rate.
A channel contains data only for pixel locations whose x and y
coordinates are evenly divisible by the x and y sampling rates:
(x % xSampling == 0) && (y % ySampling == 0)
For a CFA image sensor with RGB filters, we use the following
sampling rates:
channel xSampling ySampling
R 2 2
G 2 1
B 2 2
Now our OpenEXR file contains one R, two G and one B sample for
every four pixels, just as in the sensor. However, the spatial
arrangement of the samples differs:
sensor file
G R G R G R RGB . RGB . RGB .
B G B G B G G . G . G .
G R G R G R RGB . RGB . RGB .
B G B G B G G . G . G .
G R G R G R RGB . RGB . RGB .
B G B G B G G . G . G .
We must augment the file by describing the arrangement of the
pixels in the sensor.
The color filters in front of the pixels in the sensor are
arranged
in a regular pattern; the sensor is covered with repetitions of a
two-by-two pixel tile:
G R
B G
We can describe this pattern by adding a new CfaTile attribute to
the OpenEXR file header:
struct CfaPixel
{
string channelName;
int xOffset;
int yOffset;
V3f XYZ;
};
class CfaTile
{
public:
int xSize () const;
int ySize () const;
const CfaPixel & pixel (int x, int y) const;
CfaPixel & pixel (int x, int y);
...
};
A CfaPixel, p, at location (x, y) in CfaTile t defines the
following:
* Channel p.channelName in the OpenEXR file has values for
> all pixels whose coordinates (px, py) are of the form
px = x + n * t.xSize
py = y + m * t.ySize
In the file, the value for pixel (px, py) is stored at
location
(px + p.xOffset, py + p.offset)
>
* p.XYZ is a set of weights for reconstructing CIE XYZ colors
from the CFA sensor data. After all channels have been fully
populated by interpolation, the XYZ color of each pixel
computed as a weighted sum of all the channels:
XYZpixels[py][px] = V3f (0, 0, 0);
for (...)
XYZpixels[py][px] += channel(p.channelName)[py][px]
* p.XYZ;
Once the XYZ color of a pixel is known, the color can be
converted to any desired RGB space.
* As a special case, if p.channelName is an empty string, then
the file contains no data for this pixel.
For example, the two-by-two-pixel CfaTile for our RGB CFA sensor
would look like this:
x y channelName xOffset yOffset XYZ
0 0 G 0 0 (0.3576, 0.7152, 0.1192)
1 0 R -1 0 (0.4124, 0.2126, 0.0193)
0 1 B 0 -1 (0.1805, 0.0722, 0.9505)
1 1 G -1 0 (0.3576, 0.7152, 0.1192)
Using sub-sampled channels and a CfaTile attribute, we can also
handle sensors with green, magenta, yellow and cyan filters:
sensor file
G Y G Y G Y GYCM . GYCM . GYCM .
C M C M C M . . . . . .
G Y G Y G Y GYCM . GYCM . GYCM .
C M C M C M . . . . . .
G Y G Y G Y GYCM . GYCM . GYCM .
C M C M C M . . . . . .
channels
name xSampling ySampling
G 2 2
Y 2 2
C 2 2
M 2 2
CfaTile (2x2)
x y channelName xOffset yOffset XYZ
0 0 G 0 0 (...)
1 0 Y -1 0 (...)
0 1 C 0 -1 (...)
1 1 M -1 -1 (...)
The same representation can also handle sensor pixel grids that
are rotated by 45 degrees:
sensor file
G G G G G RGB . G RGB . G .
B R B R B R . . . . . . .
G G G G G RGB . G RGB . G .
R B R B R B . . . . . . .
G G G G G RGB . G RGB . G .
B R B R B R . . . . . . .
channels
name xSampling ySampling
R 4 2
G 2 2
B 4 2
CfaTile (4x4)
x y channelName xOffset yOffset XYZ
0 0 (empty) -1 0 (...)
1 0 G
2 0 (empty)
3 0 G -1 0 (...)
0 1 B 0 -1 (...)
1 1 (empty)
2 1 R -2 -1 (...)
3 1 (empty)
0 2 (empty)
1 2 G -1 0 (...)
2 2 (empty)
3 2 G -1 0 (...)
0 3 R 0 -1 (...)
1 3 (empty)
2 3 B -2 -1 (...)
3 3 (empty)
In this last case both the OpenEXR image channels and the CfaTile
pixel grid are rather sparsely populated. The corresponding
interpolated RGB image will have a rather high resolution, but
it will not contain fine detail. The interpolated image should
> probably be scaled down, either by a factor of sqrt(2) (resulting
in the same number of R, G and B sensor samples per RGB pixel as
for a non-rotated grid) or by a factor of 2 (resulting in one
> green sample per RGB pixel). This scale factor should perhaps
be included in the CfaTile attribute.
Integer or Floating-Point?
Representing raw CFA sensor data with sub-sampled channels and
a CfaTile attribute would work with either floating-point or
integer channels. With floating-point channels, the pixel data
would probably be scaled such that middle gray falls somewhere
close to 0.18. With integer channels, middle gray might be
represented as a value close to 9% of the maximum, for example,
1475 for a sensor that outputs 14-bit data with a maximum of
16383 (effectively mapping the maximum value to 2.0).
The XYZ scale factors of the CfaPixels would compensate for the
different scale factors of floating-point versus integer pixel
data.
Integers would be "more raw" than floating-point numbers; the
pixels could represent the exact bit patterns produced by the
analog-to-digital converter in the camera's sensor system.
16-bit floating-point numbers would introduce a mild form of
lossy data compression. With 14-bit sensor output, numbers
close to the maximum (16383) have a relative quantization step
of about 0.006% while the quantization step of 16-bit floating-
point numbers is 0.1%, so the conversion to floating-point is
not lossless. Since raw integer sensor data are nearly linear
relative to the number of photons captured by the sensor, small
differences between integer values near the high end of the
range are not significant for real-world image processing.
The difference between 15000 and 15001 is completely invisible,
as is the difference between 15000 and 15020. Conversion to
floating-point does not affect image quality, but it does
result in smaller file sizes because most of the compression
algorithms in OpenEXR work best with 16-bit floating-point data.
(PIZ and PXR24 do work reasonably well even with integer pixels.)
Proof-of-Concept Implementation
The attached tar bundle contains C++ source code for an
implementation of the CfaTile attribute, and for a command-line
program that converts an RGB image into a simulated OpenEXR raw
RGB CFA sensor image. The program can also convert raw CFA sensor
images back to RGB.
What's Missing?
The interpolation algorithm in the attached C++ code is a quick
hack. It produces rather soft images and it suffers from edge
artifacts. A production-ready implementation of the proposed
raw image representation would need a much better interpolator.
The proof-of-concept implementation lacks white balancing, flare
suppression and other basic color correction. White balancing
could be achieved by tweaking the XYZ weights in the CfaPixels,
but additional header attributes are needed to transmit other
color correction data. A CTL program would be a compact and
very general way to represent this information.
The OpenEXR library should probably contain some form of support
for raw-to-RGB conversion. Ideally the RGBA interface would
transparently perform this conversion during file reading.
It is unlikely that a purely software based raw-to-RGB conversion
would be fast enough to allow reading of OpenEXR raw images at
high frame rates. Real-time playback software would probably have
to upload the raw data to into a graphics card and perform
conversion
to RGB in a GPU-based pixel shader, similar to how playexr handles
luminance/chroma images.
And of course, camera manufacturers will have to agree to output
OpenEXR raw files.
_______________________________________________
Openexr-devel mailing list
> address@hidden
http://lists.nongnu.org/mailman/listinfo/openexr-devel
_______________________________________________
Openexr-devel mailing list
address@hidden
http://lists.nongnu.org/mailman/listinfo/openexr-devel
_______________________________________________
Openexr-devel mailing list
address@hidden
http://lists.nongnu.org/mailman/listinfo/openexr-devel