Re: [Duplicity-talk] Out of space error while restoring a file

[edgar . soldin]

On 07.10.2012 15:05, Kenneth Loafman wrote:
> On Sun, Oct 7, 2012 at 7:43 AM, <address@hidden> wrote:
> 
>> On 07.10.2012 14:03, Kenneth Loafman wrote:
>>> On Sun, Oct 7, 2012 at 2:03 AM, Laurynas Biveinis <
>>> address@hidden> wrote:
>>>
>>>> <SNIP>
>>>> OK it took a while, but here it is.
>>>>
>>>> The script I used
>>>>
>>>> duplicity -v9 -t19D --file-to-restore file &
>>>> duplicity_pid=$!
>>>> lsof -p $duplicity_pid +r 300 &
>>>>
>>>> while ps aux | grep [d]uplicity >/dev/null 2>&1
>>>> do
>>>>     date
>>>>     ls -lAR $TMP
>>>>     du -c $TMP
>>>>     df -h --sync $TMP
>>>>     sleep 300s
>>>> done
>>>>
>>>
>>> Jumping in here.  I went back through the messages and I would like for
>> you
>>> to check some things, please...
>>>
>>> - In /usr/lib/python2.x/dist-packages/duplicity there should be files
>> named
>>> GnuPGInterface.py and .pyc.  If you examine GnuPGInterface.py around line
>>> 660 there should be a function called threaded_waitpid.
>>>
>>> - In the same library, there should be a file called gpg.py containing
>>> 'from duplicity import GnuPGInterface' near line 30.
>>>
>>> The reason I ask this is that it does look like the gpg resource problem
>> is
>>> biting you, and here's how.  In Linux it it possible to create a file,
>>> delete it, and write to it.  The file will be accessible to the program
>>> until it is closed.  If you don't have the above code, the file may not
>> be
>>> closed, and the resources may not be freed.
>>>
>>> If that is the case, download both files directly from the release code
>> at
>>>
>> http://bazaar.launchpad.net/~duplicity-team/duplicity/0.6-series/files/head:/duplicity/
>> .
>>>  There is a green arrow at the far right for download.  Copy these over
>> the
>>> old files and remove their .pyc files for safety.  If you go back through
>>> the process again, you will see 2-3 gpg processes at a time, but not a
>> long
>>> list.
>>>
>>> Let us know how it goes, and thanks go to you and edso for hanging in
>>> there.  We'll get this fixed.
>>>
>>
>> Ken,
>>
>> how did you figure that he might have the "old GnuPGInterface" issue?
>> also, i assumed your message about replacing our's with the old one would
>> have reached the package maintainers by now, didn't it? should we find out
>> and tell them one by one?
>>
> 
> A couple of messages and logs together.  One message had a long string of
> gpg2 instances, another had a long string of 1 open pipe and 3 open files
> in the same pattern, with the files marked "(deleted)".  Just seemed like
> the same old issue.

ah ok, in the last log i also found these 3tmpfiles+1pipe pairs in the lsof 
output. but there was a maximum of 7 of these which i assumed as reasonable. 
also i asked him before if the system starts swapping or slows down, which he 
denied.

> There are two options I think on getting our version past the censors, one
> is to contact them individually, the other is to rename it so it does not
> match the name of the original file.  I'm tired of explaining that
> duplicity is a special case and that until it is restructured to not use
> iterators so heavily we need the modified file.  

i'd argue that the heavy use of generators is a major advantage of duplicity 
that allows it work using little disk space e.g. downloading volumes one by one 
and patching the restored file up to the desired state. 

>I think the rename will
> see it pass.  Here's hoping.
> 

want me to rename (and test) every occurence of GnuPGInterface to e.g. 
ThreadSafeGnuPGInterface, DuplicityGnuPGInterface or simply gpginterface 
(similar to others in duplicity lib dir).

we should still (probably for the next 10 releases or so add a LOUD hint to the 
changelog that package maintainers should refrain from patching the old 
GnuPGInterface into our code).


..ede/duply.net