Re: [Duplicity-talk] Access GDrive through gdocs backend failing. Link to patch.

[Boris ARZUR]

>Message: 2
>Date: Thu, 04 Jul 2013 11:55:53 +0200
>From: address@hidden
>To: Discussion of the backup program duplicity
>       <address@hidden>
>Subject: Re: [Duplicity-talk] Access GDrive through gdocs backend
>       failing. Link to patch.
>Message-ID: <address@hidden>
>Content-Type: text/plain; charset=ISO-8859-1
>
>On 04.07.2013 09:10, Boris ARZUR wrote:
>> Hello everyone,
>> 
>> First, let me thank you for a wonderful utility. I love duplicity !
>> 
>> I use duplicity 0.6.21. I have been playing with 'big' backups on google
>> drive. I use the gdocs backend. The full backup works like a charm. And my
>> troubles began with the incremental.
>> 
>> Downloading the metadata was failing with :
>> 
>> Failed to download file
>> 'duplicity-full-signatures.20130703T020608Z.sigtar'
>> in remote folder 'New Folder': Failed to find file
>> 'duplicity-full-signatures.20130703T020608Z.sigtar'
>> in remote folder 'New Folder'
>> see : http://pastebin.com/JbTNTh0d for more details.
>> 
>> duplicity-full-signatures.20130703T020608Z.sigtar is 188 MB.
>> This problem does not happen with smaller files (< 20 MB).
>> 
>> I tracked the problem to a check in
>> duplicity/backends/gdocsbackend.py: GDocsBackend.__fetch_entries
>> where the entry.get_resource_type() turns out to be 'application/x-tar'
>> and not 'application/binary' as expected...
>> 
>> I put together a quick fix, you can find it here :
>> http://pastebin.com/V5anwj1M
>> [basically test only on the left side of the /, enough to
>> distinguish folders from files]
>> 
>> It works for me. I hope it can be of use in duplicity.
>> 
>> Have a good day. And thank you for the all the duplicates...
>> Boris.
>> 
>> 
>
>not familiar with the backend code. 
>but if you could explain a bit more why it failed and why it works now, what 
>the code did before vs. now, i could setup a launchpad branch for the 
>maintainer to merge.
>
>..ede/duply.net

Hi,
OK, I sorted out my mailman problems :)

> [...] explain a bit more why it failed and why it works now, [...]
>> I tracked the problem to a check in
>> duplicity/backends/gdocsbackend.py: GDocsBackend.__fetch_entries
>> where the entry.get_resource_type() turns out to be 'application/x-tar'
>> and not 'application/binary' as expected...
As far as I understant, __fetch_entries is used to list directory and files
that match a 'filename' pattern. It sends a request to google servers with a
few parameters in the url (like 'title' or 'title-exact').
The reply is parsed an iterable object containing 'Resource' objects.

When duplicity sends for a file s.a. 
'duplicity-full-signatures.20130703T020608Z.sigtar',
it sets title=%(the_filename)s and title-exact=true in the url, sends out
the request. 
When the reply comes, it swifts through the results for a directory/filename
match (it seems you cannot specify the working directory in the url) and for
a resource type corresponding to a file, i.e. 'application/binary' here.

The problem seems to be caused by this last check. For big files, this check
excludes all files, the result sets remains empty.

To illustrate the problem at hand, I made an account in gmail :
address@hidden / pass : 1pnln04bteescuat60l31
(account linked to address@hidden, 10 minutes mail)

If you run (be careful with tmp and tmp2, dont overwrite data :) ):
<shell>
$ mkdir tmp # create temp dir
$ dd if=/dev/urandom of=tmp/blob bs=1M count=30 # create file, 30M random data
# backup the tmp dir
$ duplicity -v8 --no-encryption --volsize 150 --no-compression\
   `pwd`/tmp \
   gdocs://test.gdocs.backend.duplicity:address@hidden/tmp
# and restore
$ duplicity -v8 --no-encryption --volsize 150 --no-compression\
   gdocs://test.gdocs.backend.duplicity:address@hidden/tmp\
   `pwd`/tmp2 
# this should fail with :
# Attempt 1 failed: BackendException: Failed to download file
# 'duplicity-full.20130705T023038Z.vol1.difftar' in remote folder 'tmp':
# Failed to find file 'duplicity-full.20130705T023038Z.vol1.difftar' in
# remote folder 'tmp'
</shell>

You can monitor the 'failing' with :
<patch>
--- /usr/lib/python2.7/site-packages/duplicity/backends/gdocsbackend.orig.py 
2013-07-04 08:22:24.030257937 +0200
+++ /usr/lib/python2.7/site-packages/duplicity/backends/gdocsbackend.py 
2013-07-05 04:40:40.599483189 +0200
@@ -220,6 +220,7 @@
             if title:
                 result = []
                 for entry in entries:
+                    print "debug info in __fetch_entries: type, get_res_type 
:", type, entry.get_resource_type()
                     if (not type) or (entry.get_resource_type() == type):
                         if folder_id != GDocsBackend.ROOT_FOLDER_ID:
                             for link in entry.in_collections():
</patch>

You get info like :
[...]
Import of duplicity.backends.hsibackend Succeeded
debug info in __fetch_entries: type, get_res_type : folder folder
Main action: restore
[...]
No orphaned or incomplete backup sets found.
debug info in __fetch_entries: type, get_res_type : application/binary 
application/x-gzip
Attempt 1 failed: [...]

> [...] what the code did before vs. now [...]
The test "if (not type) or (entry.get_resource_type() == type):" is made
lest stringent. It just discriminates folders and files (matching
'application/binary' with 'application/*' and 'folder' with 'folder').

BTW, if you try after patching, please remove '--no-compression', it just
doesnt work... another bug maybe :)

I used :
<shell>
$ PASSPHRASE='i_want+you+to+&duplicate_this' duplicity -v8 --volsize 150\
   `pwd`/tmp \
    gdocs://test.gdocs.backend.duplicity:address@hidden/tmp
$ PASSPHRASE='i_want+you+to+&duplicate_this' duplicity -v8 --volsize 150\
    gdocs://test.gdocs.backend.duplicity:address@hidden/tmp\
   `pwd`/tmp2 
</shell>    

Sorry for the long mail...
Best regards,
Boris.