Re: [Duplicity-talk] OSError: [Errno 24] Too many open files

[Yang Zhang]

On Fri, May 8, 2009 at 3:34 PM, Kenneth Loafman <address@hidden> wrote:
> Kenneth Loafman wrote:
>> Yang Zhang wrote:
>>> Kenneth Loafman wrote:
>>>> Yang Zhang wrote:
>>>>> I'm running into this exception (running on Ubuntu 9.04). Anybody know
>>>>> what's up with this?
>>>>>
>>>>> $ uname -a
>>>>> Linux yang-xps410 2.6.28-11-generic #42-Ubuntu SMP Fri Apr 17 01:58:03
>>>>> UTC 2009 x86_64 GNU/Linux
>>>>>
>>>>> $ duplicity --version
>>>>> duplicity 0.5.17
>>>>>
>>>>> $ duplicity /home/yang/personal/
>>>>> OSError: [Errno 24] Too many open files
>>>> This line says it all.  I just looked and Jaunty has the default of 1024
>>>> Max Open Files.  duplicity will have open up about 150 of those at any
>>>> one time and each task running will have their own files open, often 50
>>>> or more, so its easy to deplete the resource.
>>>>
>>>> The fix is easy: "sudo ulimit -n 4096" will up the count to 4096 Max
>>>> Open Files.  This should solve the problem.
>>>>
>>>> ...Ken
>>> But this limit is per-process, not a shared pool across all processes.
>>> It means *duplicity* (via GPG, apparently) is opening more than 1024
>>> files at a time.
>>>
>>> See also this closed bug:
>>>
>>> http://savannah.nongnu.org/bugs/?21792
>>
>> It's possible there's a bug in gpg that's keeping files open on Jaunty.
>>  I don't have that system to check against, so if you would, please do:
>>
>> 1) run duplicity in a fairly long job, or with a small --volsize
>> 2) while its running, do:
>>      # lsof -n | grep ^duplicity
>> and save the last output you get.  I think it will show the same problem
>> as the other bugs have, lots of gpg processes not closed.  I thought I
>> had this problem fixed.   If that's not the case, then zip the listing
>> and post it to the bug report above.  I'll reopen it.
>
> Maybe just a ps will do, 'ps auxf' if you would.  That will tell me what
> I need to know.
>
> Plus, run duplicity with a -vINFO option and dump the first few lines.
>
> ...Thanks,
> ...Ken

I have attached a compressed text file showing all the requested
output. There are many GPG subprocesses.
-- 
Yang Zhang
http://www.mit.edu/~y_z/

report.gz
Description: GNU Zip compressed data