Re: [Openexr-devel] tile flushing with compression?

[Larry Gritz]

We're generating a whole bunch of exr's simultaneously (each may have ~3 channels). At the point that the render is killed, the various files each are readable, but disagree on which buckets have been rendered.

In fact, a mostly-black AOV may have NO buckets flushed at that point. So if we simply assume that we don't need to rerender any buckets that appear in all the output files, that often makes it look like we have to rerender the whole thing.

On Feb 25, 2015, at 4:32 PM, Piotr Stanczyk <address@hidden> wrote:

Just so that I follow, you're generating a multi-channel single file exr for these and have no reliable way to checkpoint your renders?

On 25 February 2015 at 16:27, Larry Gritz <address@hidden> wrote:

Well, if a render job gets killed, we end up with a partial file. We can then "resume" by starting a new render which takes an inventory of which tiles ended up in the partially-written file, and only rendering the buckets that didn't make it the first time.

The problem is that we may be rendering several (and sometimes very, very, way too many) outputs, i.e., lots of AOVs. So some output images may be black in certain tiles where other images are not black in the same tile. And so at the time when the render is killed, those highly compressed empty tiles in one output image may not have been flushed, whereas they were in other output images. The icky situation this leads to is that we have a set of output files that don't seem to agree on which set of tiles have been rendered and which have not. If we could ensure that all of the outputs flushed after we send a bucket (as the "full" tiles already seem to, but the "empty" tiles do not), it would make things a lot easier for us.

-- lg

On Feb 25, 2015, at 4:18 PM, Piotr Stanczyk <address@hidden> wrote:

Hmm - interesting. 

How exactly are you observing your current behaviour?

On 25 February 2015 at 16:12, Larry Gritz <address@hidden> wrote:

When writing tiled OpenEXR files, we've noticed that if we use zip compression, the zip data is not fully "flushed" when each tile is sent (particularly noticeable with small post-zipped tiles, such as empty buckets in a render). With no compression, it seems, the tiles really do write to the file as we send them.

I would be eager to hear any advice to how to ensure a full flush of the zip stream upon each tile. A call I've overlooked would be ideal, but I'd also be interested in suggestions for where to patch the IlmImf source code (or, alternately, any arguments for why it would not be wise to do so).

        -- lg

--
Larry Gritz
address@hidden




_______________________________________________
Openexr-devel mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/openexr-devel

--
Larry Gritz
address@hidden

--
Larry Gritz
address@hidden