Re: [Openexr-devel] Writing Deep tiled images: help !

[Larry Gritz]

Was it always like this, in 1.x as well? Some of my code seems to bend over 
backwards to not allow random order writes unless a specific request was made 
for random Y order. I don't remember now why I wrote it that way, but the code 
is structured as if at some time in the past, the library would not have done 
the buffering without RANDOM_Y lineOrder. Was I just always wrong about this?

        -- lg


On Apr 11, 2014, at 10:13 AM, Larry Gritz <address@hidden> wrote:

> OK, so let me restate it to make sure I have it right:
> 
> For scanline files, storage in the file is either increasing or decreasing 
> (determined by lineOrder); libIlmImf will accept scanlines in any order, but 
> if they are sent other than with the line order you specified, the library 
> will buffer internally.
> 
> For tiled files, libIlmImf will accept tiles in any order, buffering if they 
> are not in canonical order, UNLESS you set lineOrder to RANDOM_Y, in which 
> case it really will store it in random order rather than buffering (but at 
> the cost of reading that file in canonical order having more seeks than if it 
> had been written in canonical tile order).
> 
> Do I have that all correct?
> 
> 
> 
> On Apr 10, 2014, at 5:10 PM, Peter Hillman <address@hidden> wrote:
> 
>> RANDOM_Y is not supported for scanline images - you either start at 
>> datawindow.min.y and work to datawindow.max.y or vice versa. You can deliver 
>> scanlines out of order: they'll be buffered before writing.
>> 
>> With tiled images, you can write tiles in any order, but you must set 
>> RANDOM_Y to allow it write without buffering.
>> 
>> 
>> 
>> On 11/04/14 12:02, Larry Gritz wrote:
>>> Uh oh.
>>> 
>>> I knew about RANDOM_Y for scanline images. Are you saying that for tiled 
>>> images, if we don't set lineorder to RANDOM_Y, then there's internal 
>>> buffering if we send the tiles in anything but canonical order?
>>> 
>>> 
>>> 
>>> On Apr 10, 2014, at 4:54 PM, Peter Hillman <address@hidden> wrote:
>>> 
>>>> In general with OpenEXR, you cannot write to a file from different threads 
>>>> unless they have a shared framebuffer. Calling "setFrameBuffer" from one 
>>>> thread while another is writing may cause a crash.
>>>> 
>>>> Even if you do have a shared framebuffer, the threads lock to serialise 
>>>> the writes: only one "writePixels" or "writeTile" request goes through at 
>>>> once.
>>>> If you don't use the IlmThread library to run your threads it may be wise 
>>>> to implement your own lock mechanism around writePixel and writeTile calls.
>>>> 
>>>> 
>>>> There are two ways of writing tiled data: if you use 
>>>> xTileCoords/yTileCoords, then pixel(0,0) is the bottom left corner of each 
>>>> tile, implying the image buffer pointed to by the slice is the same size 
>>>> as the tile.
>>>> if you don't do that, then pixel(0,0) is the bottom left corner of the 
>>>> displayWindow, implying the image buffer pointed to be the slice is the 
>>>> size of the dataWindow, accounting for offset data windows appropriately.
>>>> 
>>>> In your case, it would seem like you want the second option: preallocate 
>>>> your dataWindow sized array of pointers-to-samples, and call 
>>>> setFrameBuffer before any rendering begins.
>>>> Then each thread sets the pointers to the samples for the tile it is going 
>>>> to write and fills in the corresponding part of the samplecount array 
>>>> before calling writeTile.
>>>> 
>>>> The pointers need not be valid when setFrameBuffer is called. When you 
>>>> call writeTile, only the pointers and sampleCount data for the pixels 
>>>> being written need to be valid, and need not remain valid afterwards. So 
>>>> yes, you can delete data for the actual sample data of a tile after 
>>>> calling writeTile, though you'll need to keep the sampleCount array and 
>>>> the pointers-to-samples array for each channel.
>>>> 
>>>> Note that if you are writing tiles in anything other than RANDOM_Y order, 
>>>> then whenever your threads deliver tiles out of order OpenEXR will have to 
>>>> buffer data so it can write the tiles sequentially.
>>>> I suspect this may well be the case from multithreaded write applications. 
>>>> With deep data this could take a significant amount of memory.
>>>> You may wish to enforce RANDOM_Y order to allow the library to write tiles 
>>>> to disk as soon as you call writeTile. If your read pattern is generally 
>>>> also random, rather than in 'scanline order', there'll be little 
>>>> performance hit.
>>>> 
>>>> You might also consider having one worker thread managing the writing of 
>>>> tile data and marshalling into the format required for the DeepSlices, and 
>>>> have other threads compute data and signal the worker thread when they 
>>>> have tiles ready for processing.
>>>> 
>>>> Yes, you can use an Array2D<float*> for your data points (dataZ and dataA 
>>>> in the example). Each value is a pointer to the first sample of the 
>>>> channel data for each pixel.
>>>> Subsequent samples are found using the offset provided. That means you can 
>>>> store pixel data in separate or interleaved channels.
>>>> 
>>>> 
>>>> 
>>>> On 11/04/14 02:16, Michel Lerenard wrote:
>>>>> Hi,
>>>>> 
>>>>> I'm trying to write deep data into an EXR file and am having trouble 
>>>>> understanding how the API works.
>>>>> I'd like to be able to write tiles from different threads. The 
>>>>> application I'm working on (Isotropix Clarisse) runs several threads, 
>>>>> each one render a part of an image.
>>>>> I'd like to write deep data when the current tile is completely computed.
>>>>> 
>>>>> I'm using the "Reading And Writing Image Files" documentation, and the 
>>>>> section about Deep Tile File writing on page 26 is confusing to me.
>>>>> 
>>>>> From what I understand, I need a DeepFrameBuffer to write an image. It 
>>>>> seems it should be the same for all threads, as a DeepTiledOutputFile can 
>>>>> only use one. Is it thread safe ? Can I use the same FrameBuffer from all 
>>>>> threads ?
>>>>> I was puzzled by the type of dataZ and data0, reading the next example 
>>>>> (reading a deep tiled file) I understand that the arrays are only storing 
>>>>> pointers to deep data for a particular pixel, and that they could as well 
>>>>> be Array2D<float*>. Is that correct ?
>>>>> 
>>>>> If it is, this means that each one of my threads should set pointers in 
>>>>> the framebuffer DeepSlice for each pixel of the tile it's computing, then 
>>>>> call writeTile. After that can I delete the data and reset the pointers 
>>>>> in the DeepSlice ?
>>>>> 
>>>>> 
>>>>> Thanks,
>>>>> 
>>>>> Michel
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> Openexr-devel mailing list
>>>>> address@hidden
>>>>> https://lists.nongnu.org/mailman/listinfo/openexr-devel
>>>> 
>>>> _______________________________________________
>>>> Openexr-devel mailing list
>>>> address@hidden
>>>> https://lists.nongnu.org/mailman/listinfo/openexr-devel
>>> --
>>> Larry Gritz
>>> address@hidden
>>> 
>>> 
>>> 
>> 
> 
> --
> Larry Gritz
> address@hidden
> 
> 
> 
> 
> _______________________________________________
> Openexr-devel mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/openexr-devel

--
Larry Gritz
address@hidden