[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upl
From: |
Kris |
Subject: |
Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http:// |
Date: |
Wed, 2 Nov 2011 17:45:11 -0400 |
My apologies Edgar, I forgot to redirect stderr to the file. The missing
chunks are:
File duplicity-full.20111102T212409Z.vol1.difftar.gpg was corrupted during
upload.
Removing still remembered temporary file
/tmp/duplicity-U9hMkY-tempdir/mkstemp-mEh1GU-1
Removing still remembered temporary file
/tmp/duplicity-U9hMkY-tempdir/mktemp-j2vS07-2
One thing to note is that the first volume is actually 75MB on S3 when the
volsize is set to 50MB. This is very strange and had me suspicious. I looked at
the verify code and it was just comparing file sizes.
So, I commented out the unlink'ing of the temp files that happens in
/usr/lib64/python2.6/site-packages/duplicity/tempdir.py.
Looking at the temp file after a failed run, it's correctly at 52456415 bytes:
-rw------- 1 root root 52456415 Nov 2 17:36 mktemp-Cyx5sP-2
Looking at the S3 copy, it's 33% bigger!
I think the problem is right there, unless I don't grok the whole picture. It
looks like it's breaking the initial full backup volume file up in to 3 chunks
(by default, I don't specify it) of 26214400 bytes. Double that is ~50 megs
(which is a volsize i specify). Triple that and you get a wrongly sized file of
75Mb that's sitting on s3.
Any ideas? I just have to walk my dog, but I can dive in to more code when I
get back. I'm sure this won't be too hard to figure out.
Kris
On 2011-11-02, at 4:21 PM, address@hidden wrote:
> kris,
>
> was that an erroneous run? looks valid to me. also please put the output on
> the web (pastebin or the like) so more people can easily evaluate it
>
> ede/duply.net
>
> On 02.11.2011 21:16, Kris wrote:
>> Hi Edgar,
>>
>> I've included a gzip of the duplicity run with -vd (which i hope is
>> equivalent to 9). I believe I've redacted all the private data.
>>
>>
>>
>>
>>
>>
>> Kris
>>
>>
>>
>> On 2011-11-02, at 4:09 PM, address@hidden wrote:
>>
>>> On 02.11.2011 20:43, Michael Terry wrote:
>>>> On 2 November 2011 15:40, Michael Terry <address@hidden> wrote:
>>>>> On 2 November 2011 15:08, Kris <address@hidden> wrote:
>>>>>> I'm thinking it might be beneficial to delete and retry uploading that
>>>>>> particular volume for some number of retries before just failing the
>>>>>> backup outright. Do you think this might be beneficial?
>>>>
>>>> Oh, also note that we do retry it 5 times if we get an error. But
>>>> this is a situation where the backend said it uploaded it fine! We
>>>> now do this extra check in case it is lying, and it seems to be. So
>>>> we need to find out why. It's not just one backend though.
>>>> Something fishy is going on.
>>>> -mt
>>>>
>>>
>>> kris,
>>>
>>> could you please run your backups with '-v9' for now and on the next
>>> occurrence of that error put the output (obfuscate private data in it) on
>>> pastebin or the launchpad bug tracker?
>>>
>>> that might help us.
>>>
>>> ede/duply.net
>>>
>>> PS: because i've never been hit by that i still expect it to be an issue of
>>> the backends silently failing to properly upload the data. to elaborate. i
>>> always do a verify after, and duplicity never failed me in the last 3
>>> years, or since when i am involved.
>>> still i promised to add the size info retrieval to ssh/ftp backend and will
>>> do so as soon as i find time for that.
>>>
>>> _______________________________________________
>>> Duplicity-talk mailing list
>>> address@hidden
>>> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>
- [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, Kris, 2011/11/02
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, Michael Terry, 2011/11/02
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, Kris, 2011/11/02
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, Michael Terry, 2011/11/02
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, Michael Terry, 2011/11/02
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, edgar . soldin, 2011/11/02
- Message not available
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, edgar . soldin, 2011/11/02
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://,
Kris <=
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, edgar . soldin, 2011/11/02
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, Kris, 2011/11/02
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, Michael Terry, 2011/11/02
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, Kris, 2011/11/02
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, Kris, 2011/11/02
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, Michael Terry, 2011/11/02
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, Kris, 2011/11/02
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, edgar . soldin, 2011/11/03
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, Kris, 2011/11/03
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, Kris, 2011/11/03