Re: [Duplicity-talk] Version 0.6.0 Released - Checkpoint/Restart

[Peter Schuller]

> The long delayed and much awaited Checkpoint/Restart capability has been
> added to duplicity and its ready now.  Please read the notes below, try
> it out, and let me know what you think.

dup_temp.py now does something crazy :)

        elif pr.encrypted:
            gpg.GPGWriteFile(src_iter, tgt.name,
                             globals.gpg_profile, size = sys.maxint / 2)

Woooh! This in turn is passed to SrcIter.next() which promptly tries
to self.fp.read() up to that amount. The result is likely dependent on
the Python version and platform, but on FreeBSD this has the
not-so-unexpected effect of failing due to an out-of-memory condition,
presumably because a buffer of that size is in fact allocated for the
purpose of the read operation. In my case the resulting value is
4611686018427320319. Memory is cheap, but not *that* cheap :)

Comments leads me to believe there is an attempt to use a very large
number to ensure we have a chance of filling up the compressed
block. However even on platforms where the implementation is
different, and memory is actually allocated on-demand, it seems
produent that the maximum size should still be reasonable. I can't see
wanting to use more than a few megs tops for buffering, or duplicity
turns into a memory hog.

I do not really have time to really understand all the new code right
now; is there a reason why we cannot just iterate over this doing the
I/O at suitable block sizes? I changed the defaults to a more sensible
5 MB in gpg.py (though I think even less than that would be prudcent,
like 128kb), but I'm not sure if this is going to break when actually
trying to write larger volumes.

Kenneth, can you shed some light on what the intent is here?

Thanks!

-- 
/ Peter Schuller

PGP userID: 0xE9758B7D or 'Peter Schuller <address@hidden>'
Key retrieval: Send an E-Mail to address@hidden
E-Mail: address@hidden Web: http://www.scode.org

pgptIU4bqJHSp.pgp
Description: PGP signature