Re: [Openexr-devel] Writing Deep tiled images: help !

[Larry Gritz]

Uh oh.

I knew about RANDOM_Y for scanline images. Are you saying that for tiled 
images, if we don't set lineorder to RANDOM_Y, then there's internal buffering 
if we send the tiles in anything but canonical order?



On Apr 10, 2014, at 4:54 PM, Peter Hillman <address@hidden> wrote:

> In general with OpenEXR, you cannot write to a file from different threads 
> unless they have a shared framebuffer. Calling "setFrameBuffer" from one 
> thread while another is writing may cause a crash.
> 
> Even if you do have a shared framebuffer, the threads lock to serialise the 
> writes: only one "writePixels" or "writeTile" request goes through at once.
> If you don't use the IlmThread library to run your threads it may be wise to 
> implement your own lock mechanism around writePixel and writeTile calls.
> 
> 
> There are two ways of writing tiled data: if you use xTileCoords/yTileCoords, 
> then pixel(0,0) is the bottom left corner of each tile, implying the image 
> buffer pointed to by the slice is the same size as the tile.
> if you don't do that, then pixel(0,0) is the bottom left corner of the 
> displayWindow, implying the image buffer pointed to be the slice is the size 
> of the dataWindow, accounting for offset data windows appropriately.
> 
> In your case, it would seem like you want the second option: preallocate your 
> dataWindow sized array of pointers-to-samples, and call setFrameBuffer before 
> any rendering begins.
> Then each thread sets the pointers to the samples for the tile it is going to 
> write and fills in the corresponding part of the samplecount array before 
> calling writeTile.
> 
> The pointers need not be valid when setFrameBuffer is called. When you call 
> writeTile, only the pointers and sampleCount data for the pixels being 
> written need to be valid, and need not remain valid afterwards. So yes, you 
> can delete data for the actual sample data of a tile after calling writeTile, 
> though you'll need to keep the sampleCount array and the pointers-to-samples 
> array for each channel.
> 
> Note that if you are writing tiles in anything other than RANDOM_Y order, 
> then whenever your threads deliver tiles out of order OpenEXR will have to 
> buffer data so it can write the tiles sequentially.
> I suspect this may well be the case from multithreaded write applications. 
> With deep data this could take a significant amount of memory.
> You may wish to enforce RANDOM_Y order to allow the library to write tiles to 
> disk as soon as you call writeTile. If your read pattern is generally also 
> random, rather than in 'scanline order', there'll be little performance hit.
> 
> You might also consider having one worker thread managing the writing of tile 
> data and marshalling into the format required for the DeepSlices, and have 
> other threads compute data and signal the worker thread when they have tiles 
> ready for processing.
> 
> Yes, you can use an Array2D<float*> for your data points (dataZ and dataA in 
> the example). Each value is a pointer to the first sample of the channel data 
> for each pixel.
> Subsequent samples are found using the offset provided. That means you can 
> store pixel data in separate or interleaved channels.
> 
> 
> 
> On 11/04/14 02:16, Michel Lerenard wrote:
>> Hi,
>> 
>> I'm trying to write deep data into an EXR file and am having trouble 
>> understanding how the API works.
>> I'd like to be able to write tiles from different threads. The application 
>> I'm working on (Isotropix Clarisse) runs several threads, each one render a 
>> part of an image.
>> I'd like to write deep data when the current tile is completely computed.
>> 
>> I'm using the "Reading And Writing Image Files" documentation, and the 
>> section about Deep Tile File writing on page 26 is confusing to me.
>> 
>> From what I understand, I need a DeepFrameBuffer to write an image. It seems 
>> it should be the same for all threads, as a DeepTiledOutputFile can only use 
>> one. Is it thread safe ? Can I use the same FrameBuffer from all threads ?
>> I was puzzled by the type of dataZ and data0, reading the next example 
>> (reading a deep tiled file) I understand that the arrays are only storing 
>> pointers to deep data for a particular pixel, and that they could as well be 
>> Array2D<float*>. Is that correct ?
>> 
>> If it is, this means that each one of my threads should set pointers in the 
>> framebuffer DeepSlice for each pixel of the tile it's computing, then call 
>> writeTile. After that can I delete the data and reset the pointers in the 
>> DeepSlice ?
>> 
>> 
>> Thanks,
>> 
>> Michel
>> 
>> 
>> _______________________________________________
>> Openexr-devel mailing list
>> address@hidden
>> https://lists.nongnu.org/mailman/listinfo/openexr-devel
> 
> 
> _______________________________________________
> Openexr-devel mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/openexr-devel

--
Larry Gritz
address@hidden