Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://

[edgar . soldin]

smells like you are on the right track there ..
could it be that the logic to split to custom volumesize is broken? meaning, 
does the error disappear if you do _not_ specify a custom volume size?

@Mike: does that give you a hint?

..ede/duply.net

On 02.11.2011 22:45, Kris wrote:
> My apologies Edgar, I forgot to redirect stderr to the file.  The missing 
> chunks are:
> File duplicity-full.20111102T212409Z.vol1.difftar.gpg was corrupted during 
> upload.
> Removing still remembered temporary file 
> /tmp/duplicity-U9hMkY-tempdir/mkstemp-mEh1GU-1
> Removing still remembered temporary file 
> /tmp/duplicity-U9hMkY-tempdir/mktemp-j2vS07-2
> 
> One thing to note is that the first volume is actually 75MB on S3 when the 
> volsize is set to 50MB. This is very strange and had me suspicious. I looked 
> at the verify code and it was just comparing file sizes.
> 
> So, I commented out the unlink'ing of the temp files that happens in 
> /usr/lib64/python2.6/site-packages/duplicity/tempdir.py.
> 
> Looking at the temp file after a failed run, it's correctly at 52456415 bytes:
> -rw------- 1 root root 52456415 Nov  2 17:36 mktemp-Cyx5sP-2
> 
> Looking at the S3 copy, it's 33% bigger!
> 
> I think the problem is right there, unless I don't grok the whole picture. It 
> looks like it's breaking the initial full backup volume file up in to 3 
> chunks (by default, I don't specify it) of 26214400 bytes. Double that is ~50 
> megs (which is a volsize i specify). Triple that and you get a wrongly sized 
> file of 75Mb that's sitting on s3.
> 
> Any ideas?  I just have to walk my dog, but I can dive in to more code when I 
> get back. I'm sure this won't be too hard to figure out.
> 
> Kris
> 
> 
> 
> On 2011-11-02, at 4:21 PM, address@hidden wrote:
> 
>> kris,
>>
>> was that an erroneous run? looks valid to me. also please put the output on 
>> the web (pastebin or the like) so more people can easily evaluate it
>>
>> ede/duply.net 
>>
>> On 02.11.2011 21:16, Kris wrote:
>>> Hi Edgar,
>>>
>>> I've included a gzip of the duplicity run with -vd (which i hope is 
>>> equivalent to 9).  I believe I've redacted all the private data.
>>>
>>>
>>>
>>>
>>>
>>>
>>> Kris
>>>
>>>
>>>
>>> On 2011-11-02, at 4:09 PM, address@hidden wrote:
>>>
>>>> On 02.11.2011 20:43, Michael Terry wrote:
>>>>> On 2 November 2011 15:40, Michael Terry <address@hidden> wrote:
>>>>>> On 2 November 2011 15:08, Kris <address@hidden> wrote:
>>>>>>> I'm thinking it might be beneficial to delete and retry uploading that 
>>>>>>> particular volume for some number of retries before just failing the 
>>>>>>> backup outright. Do you think this might be beneficial?
>>>>>
>>>>> Oh, also note that we do retry it 5 times if we get an error.  But
>>>>> this is a situation where the backend said it uploaded it fine!  We
>>>>> now do this extra check in case it is lying, and it seems to be.  So
>>>>> we need to find out why.   It's not just one backend though.
>>>>> Something fishy is going on.
>>>>> -mt
>>>>>
>>>>
>>>> kris,
>>>>
>>>> could you please run your backups with '-v9' for now and on the next 
>>>> occurrence of that error put the output (obfuscate private data in it) on 
>>>> pastebin or the launchpad bug tracker?
>>>>
>>>> that might help us.
>>>>
>>>> ede/duply.net
>>>>
>>>> PS: because i've never been hit by that i still expect it to be an issue 
>>>> of the backends silently failing to properly upload the data. to 
>>>> elaborate. i always do a verify after, and duplicity never failed me in 
>>>> the last 3 years, or since when i am involved. 
>>>> still i promised to add the size info retrieval to ssh/ftp backend and 
>>>> will do so as soon as i find time for that.
>>>>
>>>> _______________________________________________
>>>> Duplicity-talk mailing list
>>>> address@hidden
>>>> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>
>