Re: [Duplicity-talk] OSError: [Errno 24] Too many open files

[Kenneth Loafman]

I run Hardy and Intrepid and both default to 4096.  When I installed
Jaunty, it was at 1024, and my guess is that the upstream Debian version
is the same.

...Ken

address@hidden wrote:
> any idea why suddenly 3 people come around with this error? I remotely
> suspect a change in ulimit of distros OR duplicity v0.5.17
> 
> ..ede
> -- 
>> No difference, other than my testing was flawed.  I finally had to go to
>> the debugger to stop the process and then count using lsof.
>>
>> I'm showing 3 file descriptors released with every gpg process after the
>> waitpid() is completed, so it would take 300+ gpg zombies to overrun the
>> 1024 limit.
>>
>> This is a problem in the way duplicity and GnuPGInterface are handling
>> process termination.  To fix it, I'll need to add a SIGCHILD signal
>> handler to GnuPGInterface that will allow the process to terminate when
>> it ends, rather than making it wait till we get around to checking.
>>
>> ...Ken
>>
>> Edgar Soldin wrote:
>>  
>>> just curious, could you detect a difference between 32bit & 64bit
>>> systems?
>>> how many incrementals did you need to hit the 1024 ulimit?
>>>
>>> @yang: how many incrementals does your backup hold? just to doublecheck
>>> kens findings.
>>>
>>> ..ede
>>>    
>>>> I have found the problem and its going to take me a bit to get it
>>>> fixed.
>>>>
>>>> As it works duplicity creates iterators that traverse each of the sig
>>>> files.  The iterators do not close until the process is complete and
>>>> the
>>>> files they are using are being held open.  The gpg process is held as
>>>> well since we don't wait for it until we close the file.  The zombie
>>>> processes were just a symptom of the real problem, too many open
>>>> resources at one time.
>>>>
>>>> This hits directly the issue of too long an incremental chain.  Each
>>>> increments sig file is going to have file descriptors open until the
>>>> backup/incremental/restore completes.  The only workaround at this
>>>> point
>>>> is to increase the 'ulimit -n' parameter for any long increment chains
>>>> and perform a full backup on a more frequent basis.
>>>>
>>>> I have some ideas about how to clean this up, but need to examine the
>>>> code a bit closer before committing to a solution.
>>>>
>>>> ...Ken
>>>>       
>>
>>  
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> Duplicity-talk mailing list
>> address@hidden
>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>   
> 
> 
> 
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>

signature.asc
Description: OpenPGP digital signature