Re: [Duplicity-talk] OSError: [Errno 24] Too many open files

[Kenneth Loafman]

No difference, other than my testing was flawed.  I finally had to go to
the debugger to stop the process and then count using lsof.

I'm showing 3 file descriptors released with every gpg process after the
waitpid() is completed, so it would take 300+ gpg zombies to overrun the
1024 limit.

This is a problem in the way duplicity and GnuPGInterface are handling
process termination.  To fix it, I'll need to add a SIGCHILD signal
handler to GnuPGInterface that will allow the process to terminate when
it ends, rather than making it wait till we get around to checking.

...Ken

Edgar Soldin wrote:
> just curious, could you detect a difference between 32bit & 64bit systems?
> how many incrementals did you need to hit the 1024 ulimit?
> 
> @yang: how many incrementals does your backup hold? just to doublecheck
> kens findings.
> 
> ..ede
>> I have found the problem and its going to take me a bit to get it fixed.
>>
>> As it works duplicity creates iterators that traverse each of the sig
>> files.  The iterators do not close until the process is complete and the
>> files they are using are being held open.  The gpg process is held as
>> well since we don't wait for it until we close the file.  The zombie
>> processes were just a symptom of the real problem, too many open
>> resources at one time.
>>
>> This hits directly the issue of too long an incremental chain.  Each
>> increments sig file is going to have file descriptors open until the
>> backup/incremental/restore completes.  The only workaround at this point
>> is to increase the 'ulimit -n' parameter for any long increment chains
>> and perform a full backup on a more frequent basis.
>>
>> I have some ideas about how to clean this up, but need to examine the
>> code a bit closer before committing to a solution.
>>
>> ...Ken

signature.asc
Description: OpenPGP digital signature