Re: [Duplicity-talk] Duplicity - Resource Unavailable Errors

[Kenneth Loafman]

Sorry for the slow response.  This seems to be a ulimit issue, but
that's not normal for only 45 incrementals.  Please run 'ulimit -n' and
post the response here.  Try increasing the ulimit with 'ulimit -n 4096'
or higher and retry.  That could help.

I'm not sure what's happening, but its possible that the permission
failures are somehow eating up file slots.  During the run you could try
'lsof -p <duplicity's pid>' just to see if any of those files are open.

...Thanks,
...Ken

Colin Ryan wrote:
> Update:
> 
> Tried this same test again and it is not always that this failure leaves
> the rest of the system unable to fork other processes such as "ls" or
> "top" etc etc. sometimes yes, sometimes no...in any even duplicity does
> continue to grow in excess of 200Mb put I do have physical RAM reported
> as free when this occurs.
> 
> Any thoughts?
> 
> Colin Ryan wrote:
>> I should know the answer to this but can't seem to confirm. Clearly
>> such errors are generally flimit/ulimit issues or RAM.
>>
>> But I'm testing restore a not huge ( ~ 8 GB archive that is only 1
>> full and 45 days of incremental's) and get the following from the
>> following command line.
>>
>> /duplicity restore  -v9 --encrypt-key=xxxxxxx --sign-key=xxxxxx
>> --gpg-options='--default-key=xxxxxxx'
>> --archive-dir="/opt/aa/service/nas/restores"
>> --tempdir="/opt/aa/service/nas/restores" --ignore-errors
>> s3+http://URL/<backup set name> "/opt/aa/service/nas/restores/office"
>>
>> I'm using the -ignore-errors as I'm unpacking as a non-root user other
>> than the one that did the backups so it was throwing "no permission"
>> errors on what I expect was the chmod/chown stages ( I say this
>> because the files themselves were extracted)...Anyhow...
>>
>> Version is 6.0.5 hand rolled in Debian Lenny.
>>
>> When it ceases the system is unable to fork any other processes until
>> I kill duplicity.
>>
>> A already running top in another terminal shows that duplicity at this
>> point is ~ 200 MB virtual and 11 MB resident but that I have ~ 200M
>> RAM free (this is a fairly small footprint system).
>>
>> I believe it's out of memory for forking but is it normal.
>>
>> <code>
>> BEGNj3-tempdir/mktemp-QYdP4e-85
>> DEBUG:duplicity:Removing still remembered temporary file
>> /opt/aa/service/nas/restores/duplicity-BEGNj3-tempdir/mktemp-QYdP4e-85
>> Removing still remembered temporary file
>> /opt/aa/service/nas/restores/duplicity-BEGNj3-tempdir/mktemp-aDi69N-61
>> DEBUG:duplicity:Removing still remembered temporary file
>> /opt/aa/service/nas/restores/duplicity-BEGNj3-tempdir/mktemp-aDi69N-61
>> Removing still remembered temporary file
>> /opt/aa/service/nas/restores/duplicity-BEGNj3-tempdir/mktemp-2YCFpS-81
>> DEBUG:duplicity:Removing still remembered temporary file
>> /opt/aa/service/nas/restores/duplicity-BEGNj3-tempdir/mktemp-2YCFpS-81
>> Removing still remembered temporary file
>> /opt/aa/service/nas/restores/duplicity-BEGNj3-tempdir/mktemp-CXX67i-73
>> DEBUG:duplicity:Removing still remembered temporary file
>> /opt/aa/service/nas/restores/duplicity-BEGNj3-tempdir/mktemp-CXX67i-73
>> Traceback (most recent call last):
>>  File "/usr/bin/duplicity", line 1241, in ?
>>    with_tempdir(main)
>>  File "/usr/bin/duplicity", line 1234, in with_tempdir
>>    fn()
>>  File "/usr/bin/duplicity", line 1188, in main
>>    restore(col_stats)
>>  File "/usr/bin/duplicity", line 542, in restore
>>    restore_get_patched_rop_iter(col_stats)):
>>  File "/usr/lib/python2.4/site-packages/duplicity/patchdir.py", line
>> 519, in Write_ROPaths
>>    for ropath in rop_iter:
>>  File "/usr/lib/python2.4/site-packages/duplicity/patchdir.py", line
>> 491, in integrate_patch_iters
>>    for patch_seq in collated:
>>  File "/usr/lib/python2.4/site-packages/duplicity/patchdir.py", line
>> 376, in yield_tuples
>>    setrorps(overflow, elems)
>>  File "/usr/lib/python2.4/site-packages/duplicity/patchdir.py", line
>> 365, in setrorps
>>    elems[i] = iter_list[i].next()
>>  File "/usr/lib/python2.4/site-packages/duplicity/patchdir.py", line
>> 110, in difftar2path_iter
>>    tarinfo_list = [tar_iter.next()]
>>  File "/usr/lib/python2.4/site-packages/duplicity/patchdir.py", line
>> 326, in next
>>    self.set_tarfile()
>>  File "/usr/lib/python2.4/site-packages/duplicity/patchdir.py", line
>> 320, in set_tarfile
>>    self.current_fp = self.fileobj_iter.next()
>>  File "/usr/bin/duplicity", line 579, in get_fileobj_iter
>>    manifest.volume_info_dict[vol_num])
>>  File "/usr/bin/duplicity", line 603, in restore_get_enc_fileobj
>>    fileobj = tdp.filtered_open_with_delete("rb")
>>  File "/usr/lib/python2.4/site-packages/duplicity/dup_temp.py", line
>> 114, in filtered_open_with_delete
>>    fh = FileobjHooked(path.DupPath.filtered_open(self, mode))
>>  File "/usr/lib/python2.4/site-packages/duplicity/path.py", line 724,
>> in filtered_open
>>    return gpg.GPGFile(False, self, gpg_profile)
>>  File "/usr/lib/python2.4/site-packages/duplicity/gpg.py", line 135,
>> in __init__
>>    attach_fhs={'stdin': encrypt_path.open("rb"),
>>  File "/usr/lib/python2.4/site-packages/duplicity/GnuPGInterface.py",
>> line 365, in run
>>    create_fhs, attach_fhs)
>>  File "/usr/lib/python2.4/site-packages/duplicity/GnuPGInterface.py",
>> line 407, in _attach_fork_exec
>>    process.pid = os.fork()
>> OSError: [Errno 11] Resource temporarily unavailable
>>
>> ERROR:duplicity:Traceback (most recent call last):
>>  File "/usr/bin/duplicity", line 1241, in ?
>>    with_tempdir(main)
>>  File "/usr/bin/duplicity", line 1234, in with_tempdir
>>    fn()
>>  File "/usr/bin/duplicity", line 1188, in main
>>    restore(col_stats)
>>  File "/usr/bin/duplicity", line 542, in restore
>>    restore_get_patched_rop_iter(col_stats)):
>>  File "/usr/lib/python2.4/site-packages/duplicity/patchdir.py", line
>> 519, in Write_ROPaths
>>    for ropath in rop_iter:
>>  File "/usr/lib/python2.4/site-packages/duplicity/patchdir.py", line
>> 491, in integrate_patch_iters
>>    for patch_seq in collated:
>>  File "/usr/lib/python2.4/site-packages/duplicity/patchdir.py", line
>> 376, in yield_tuples
>>    setrorps(overflow, elems)
>>  File "/usr/lib/python2.4/site-packages/duplicity/patchdir.py", line
>> 365, in setrorps
>>    elems[i] = iter_list[i].next()
>>  File "/usr/lib/python2.4/site-packages/duplicity/patchdir.py", line
>> 110, in difftar2path_iter
>>    tarinfo_list = [tar_iter.next()]
>>  File "/usr/lib/python2.4/site-packages/duplicity/patchdir.py", line
>> 326, in next
>>    self.set_tarfile()
>>  File "/usr/lib/python2.4/site-packages/duplicity/patchdir.py", line
>> 320, in set_tarfile
>>    self.current_fp = self.fileobj_iter.next()
>>  File "/usr/bin/duplicity", line 579, in get_fileobj_iter
>>    manifest.volume_info_dict[vol_num])
>>  File "/usr/bin/duplicity", line 603, in restore_get_enc_fileobj
>>    fileobj = tdp.filtered_open_with_delete("rb")
>>  File "/usr/lib/python2.4/site-packages/duplicity/dup_temp.py", line
>> 114, in filtered_open_with_delete
>>    fh = FileobjHooked(path.DupPath.filtered_open(self, mode))
>>  File "/usr/lib/python2.4/site-packages/duplicity/path.py", line 724,
>> in filtered_open
>>    return gpg.GPGFile(False, self, gpg_profile)
>>  File "/usr/lib/python2.4/site-packages/duplicity/gpg.py", line 135,
>> in __init__
>>    attach_fhs={'stdin': encrypt_path.open("rb"),
>>  File "/usr/lib/python2.4/site-packages/duplicity/GnuPGInterface.py",
>> line 365, in run
>>    create_fhs, attach_fhs)
>>  File "/usr/lib/python2.4/site-packages/duplicity/GnuPGInterface.py",
>> line 407, in _attach_fork_exec
>>    process.pid = os.fork()
>> OSError: [Errno 11] Resource temporarily unavailable
>>
>> Error in atexit._run_exitfuncs:
>> Traceback (most recent call last):
>>  File "atexit.py", line 24, in _run_exitfuncs
>>    func(*targs, **kargs)
>>  File "threading.py", line 634, in __exitfunc
>>    t.join()
>>  File "threading.py", line 540, in join
>>    self.__block.wait()
>>  File "threading.py", line 203, in wait
>>    waiter.acquire()
>> </code>
>>
>> Cheers and Thanks
>>
>> Colin
>>
>>
>> _______________________________________________
>> Duplicity-talk mailing list
>> address@hidden
>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
> 
> 
> 
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>

signature.asc
Description: OpenPGP digital signature