openexr-devel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Openexr-devel] OpenEXR - lossless compression problems (Peter Hillm


From: Larry Gritz
Subject: Re: [Openexr-devel] OpenEXR - lossless compression problems (Peter Hillman)
Date: Mon, 5 Dec 2016 21:43:31 -0800

If people are thinking of the problem as inherently having 16 bit unsigned integer values, then converting to half float for output to exr, and finally converting back to canonical uint16, perhaps they are barking up the wrong tree by considering OpenEXR at all.

The whole idea behind OpenEXR is for extended range -- values that may be negative, or very small, or greater than 1. The kinds of things you would usually use a 'float' for if they were in memory. The use of half precision is itself a form of lossy compression, halving the size by accepting a loss of precision (but an acceptable and roughly perceptually uniform loss, since vision sensitivity is essentially logarithmic).

OpenEXR is meant to store float data, not integer data. You wouldn't expect a uint16 image to have very good accuracy or compression by going through a 'half' conversion. It's a mismatch of method and task.

-- lg


On Dec 5, 2016, at 5:41 PM, Conrad Gaunt <address@hidden> wrote:

hi.

ive been testing image formats for a few months, and in depth. new to openexr.

i'm currently starting the optimizing of my lossless 16bit+ compression system 

some observations.

remember, if you're source material is 16bit integer, the half float conversion is very lossy, although it has more conservative properties when manipulated down the post pipe in some regards,

one thing all formats suffer from .. they are always going out of date . most were invented in the 90s

my classical Huffman coder beats j2k and png on 48bit photographic material, and PIZ

 PNG uses Huffman coding .. and can compress 16bit values .. but the compression system in png is only 8bit. same with TIF.

 all these are by products of formats beings developed back when computers had 640kb of ram..

when you convert 16bit integer to 16bit half float, and normalise between 0..1 , there is half as much data recorded as when you normalise between -1 ..1 .. and you're not wasting the sign bit !

I don't know what openexr convention says. anyway, normalising between 0..1 makes my compressor produce files that are significantly smaller than the compressed integer source.

I'm not sure what the correct normalization procedure for openexr is to be honest

try normalising all possible 65536 integer values into half floats. then convert them back to integer, and print a list of the original inputs and outputs (into a text file) . quite interesting

k.r
 



 



On Sun, Dec 4, 2016 at 5:43 AM, <address@hidden> wrote:
Send Openexr-devel mailing list submissions to
        address@hidden

To subscribe or unsubscribe via the World Wide Web, visit
        https://lists.nongnu.org/mailman/listinfo/openexr-devel
or, via email, send a message with subject or body 'help' to
        address@hiddenorg

You can reach the person managing the list at
        address@hidden

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Openexr-devel digest..."

Today's Topics:

   1. OpenEXR - lossless compression problems (David Ku??k)
   2. Re: OpenEXR - lossless compression problems (Peter Hillman)


---------- Forwarded message ----------
From: "David Kuťák" <address@hidden>
To: <address@hidden>
Cc: 
Date: Sat, 3 Dec 2016 20:40:10 +0100
Subject: [Openexr-devel] OpenEXR - lossless compression problems

Hello,

 

I am currently working on my bachelor thesis in which I am comparing image compression algorithms (available in different image file formats).

For my work, only file formats supporting lossless compression and 16 bit depth are appropriate.

During my exploration of possible formats, I found an OpenEXR and decided to include it in my tests.

However, I am not able to  make it work properly.

 

The input image and output image (created during compression) are not same – the colors seem to be different (shifted).

I thought it might be gamma problem but even after gamma correction, I am not able to retrieve correct output image.

Although visually the image seems to be very close to the input one (after gamma correction), there is still a minor difference which is unacceptable.

 

So I’d like to ask you for a help – what I am doing is written below (some undefined wars like imageWidth, etc. are correct for sure).

 

1) I create my output file

     RgbaOutputFile outputFile(argv[3], imageWidth, imageHeight, WRITE_RGB, 1.0f,

                                               V2f(0, 0), 1.0f, INCREASING_Y, compressionType, globalThreadCount());

 

2) Then I create a pixelData vector

    std::vector<Rgba> pixelData(imageWidth * imageHeight);

 

3) I fill this vector with data retrieved from my test images (these data are pixel values (i.e. numbers from interval <0,255/65535> – depending on bit depth – which I normalize to inverval <0,1.0> and pass to half ctor and then as an Rgba struct to appropriate pos in pixelData vector).

 

4) Then I set framebuffer and write pixels to file (shown in code below). I think that my problem occurs somewhere in those two methods (EXR must be doing some operations with pixel data).

outputFile.setFrameBuffer(pixelData.data(), 1, imageWidth);

outputFile.writePixels(imageHeight);

 

To sum up my problem – is this the correct way to pass image data to OpenEXR and retrieve the same image (but in EXR format)? I really don’t know why the color problem appears.

 

Another question I’d like to ask is that I got the idea that the compression is done in writePixels method. Is that’s right? If it’s true, is there any possibility to compress image in memory without writing it to file?

 

Thank you for your help!

 

David



---------- Forwarded message ----------
From: Peter Hillman <address@hidden>
To: "David Kuťák" <address@hidden>, "address@hidden" <address@hidden>
Cc: 
Date: Sun, 4 Dec 2016 00:33:39 +0000
Subject: Re: [Openexr-devel] OpenEXR - lossless compression problems

Hi David,


Your code looks valid, and as the image looks quite close to correct it's unlikely there's a bug in your code.

Perhaps the difference you are seeing is due to the fact that OpenEXR's 16 bit half is a floating point number. Converting a 16 bit unsigned integer n to a half float using n/65535 will cause rounding error.


From a quick test it seems you get 7169 unique bit patterns in a half float with that approach, which means you are losing information. The other patterns used by 16 bit half mainly store negative numbers and numbers above 1, which you aren't using here. These come about because OpenEXR is intended for storing HDR images. With a 16 bit unsigned int sRGB PNG, the darkest grey is 20 stops darker than the maximum white; with a 16 bit half linear OpenEXR that's closer to 40 stops.


The 8 bit version of n/255 will also cause rounding error, but in that case each input value is represented uniquely and you can fix the error by rounding to the nearest integer when you denormalize.


The openexr.com website has some sample HDR images you could use to experiment with OpenEXR compression schemes, but it is hard to make a rigorous comparison of PNG (for example) to OpenEXR because they set out to achieve different things working with different kinds of image data.


Your second question:


I got the idea that the compression is done in writePixels method. Is that’s right? If it’s true, is there any possibility to compress image in memory without writing it to file?


Yes, this is correct-writePixels is the only access to the compression schemes from the API. It is possible to generate an OpenEXR file entirely in memory, but is non-trivial: you have to subclass OStream and override the methods to write data into your memory array rather than to file. If you need to read it back, you need to make an IStream too.

You then pass an instance of your new OStream to the OutputFile constructor when you write it. The OpenEXR library will call your methods when it wants to write data. You might start here:

https://github.com/openexr/openexr/blob/develop/OpenEXR/IlmImfExamples/lowLevelIoExamples.cpp


Actually, if you just want to know how big the file would be with a given scheme, you could be sneaky and not store the image at all: just subclass OStream with a class that has its own "file pointer" which is read by readp(), set by seekp() and incremented by write(). You'll also need to track the maximum value the "file pointer" ever has, which would be the file size you need (not the final value of "file pointer") Doing this won't let you analyse errors in lossy schemes, but would let you measure the lossless schemes' efficiency.




From: Openexr-devel <openexr-devel-bounces+peterh=address@hidden> on behalf of David Kuťák <address@hidden>
Sent: Sunday, 4 December 2016 8:40 a.m.
To: address@hidden
Subject: [Openexr-devel] OpenEXR - lossless compression problems
 
Hello,

 

I am currently working on my bachelor thesis in which I am comparing image compression algorithms (available in different image file formats).
For my work, only file formats supporting lossless compression and 16 bit depth are appropriate.
During my exploration of possible formats, I found an OpenEXR and decided to include it in my tests.
However, I am not able to  make it work properly.

 

The input image and output image (created during compression) are not same – the colors seem to be different (shifted).
I thought it might be gamma problem but even after gamma correction, I am not able to retrieve correct output image.
Although visually the image seems to be very close to the input one (after gamma correction), there is still a minor difference which is unacceptable.

 

So I’d like to ask you for a help – what I am doing is written below (some undefined wars like imageWidth, etc. are correct for sure).

 

1) I create my output file
     RgbaOutputFile outputFile(argv[3], imageWidth, imageHeight, WRITE_RGB, 1.0f,
                                               V2f(0, 0), 1.0f, INCREASING_Y, compressionType, globalThreadCount());

 

2) Then I create a pixelData vector
    std::vector<Rgba> pixelData(imageWidth * imageHeight);

 

3) I fill this vector with data retrieved from my test images (these data are pixel values (i.e. numbers from interval <0,255/65535> – depending on bit depth – which I normalize to inverval <0,1.0> and pass to half ctor and then as an Rgba struct to appropriate pos in pixelData vector).

 

4) Then I set framebuffer and write pixels to file (shown in code below). I think that my problem occurs somewhere in those two methods (EXR must be doing some operations with pixel data).
outputFile.setFrameBuffer(pixelData.data(), 1, imageWidth);
outputFile.writePixels(imageHeight);

 

To sum up my problem – is this the correct way to pass image data to OpenEXR and retrieve the same image (but in EXR format)? I really don’t know why the color problem appears.

 

Another question I’d like to ask is that I got the idea that the compression is done in writePixels method. Is that’s right? If it’s true, is there any possibility to compress image in memory without writing it to file?

 

Thank you for your help!

 

David

_______________________________________________
Openexr-devel mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/openexr-devel




--
conrad gaunt

www.cheapredhire.com
tel.07835915253
_______________________________________________
Openexr-devel mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/openexr-devel

--
Larry Gritz
address@hidden



reply via email to

[Prev in Thread] Current Thread [Next in Thread]