Re: [Duplicity-talk] Debugging SFTP connectivity issues and showing SFTP output

[edgar . soldin]

On 11.02.2013 17:39, Mikko Ohtamaa wrote:
> Hi,
> 
> I am running Duplicity (wrapped in Duply) backup scripts in a Hetzner
> (hetzner.de) hosted environment. Mostly backups run fine, but on a
> certain server I have SFTP connectivity issues I am unable to debug.
> Hetzner provides a special backup space option which is accessible
> over SFTP (not sure what SFTP server they are running there).
> 
> SFTP command seems to abort now and then and I have not found a way to
> make Duplicity to generate useful output (tried -verbosity '9') to see
> what happens when things go wrong. The problem is random, so it is
> pretty much impossible to repeat it by hand: if you SFTP upload
> yourself it always success.
> 
> Is there a way to make Duplicity to dump out SFTP output, so one could
> get some useful information what is wrong with SFTP (network problem,
> local issue, some magical with the file, etc.) The connection is
> definitely dropped, but I'd like to find out what's causing it :(
> 
> Command:
> 
> TMPDIR='/tmp' duplicity --name 'duply_fullbackup' --no-encryption
> --verbosity '9' --full-if-older-than 1M --volsize 250
> --exclude='**.log' --exclude='**.fs.old' --include=/etc
> --include=/home --include=/root --include=/srv --include=/var/www
> --include=/boot --exclude=/** --exclude-globbing-filelist
> /root/.duply/fullbackup/exclude / ssh://xxxx
> 
> Log trace below:
> 
> Registering (mktemp) temporary file
> /tmp/duplicity-PY9oOo-tempdir/mktemp-esDT5F-6
> AsyncScheduler: running task synchronously (asynchronicity disabled)
> No handlers could be found for logger "paramiko.transport"
> sftp put of /tmp/duplicity-PY9oOo-tempdir/mktemp-esDT5F-6 (as
> duplicity-full.20130208T081130Z.vol45.difftar.gz) failed: Server
> connection dropped:  (Try 1 of 5) Will retry in 10 seconds.
> sftp put of /tmp/duplicity-PY9oOo-tempdir/mktemp-esDT5F-6 (as
> duplicity-full.20130208T081130Z.vol45.difftar.gz) failed:  (Try 2 of
> 5) Will retry in 10 seconds.
> sftp put of /tmp/duplicity-PY9oOo-tempdir/mktemp-esDT5F-6 (as
> duplicity-full.20130208T081130Z.vol45.difftar.gz) failed:  (Try 3 of
> 5) Will retry in 10 seconds.
> sftp put of /tmp/duplicity-PY9oOo-tempdir/mktemp-esDT5F-6 (as
> duplicity-full.20130208T081130Z.vol45.difftar.gz) failed:  (Try 4 of
> 5) Will retry in 10 seconds.
> sftp put of /tmp/duplicity-PY9oOo-tempdir/mktemp-esDT5F-6 (as
> duplicity-full.20130208T081130Z.vol45.difftar.gz) failed:  (Try 5 of
> 5) Will retry in 10 seconds.
> Removing still remembered temporary file
> /tmp/duplicity-PY9oOo-tempdir/mktemp-esDT5F-6
> 

well it says: "Server connection dropped"
not sure if paramiko will or can tell you more about it. you might wanna edit 
 duplicity/backends/_ssh_paramiko.py
and add
 paramiko.common.logging.basicConfig(level=paramiko.common.DEBUG)
somewhere appropriate.

alternatively you might want to try the old pexpect backend, which uses the 
command line sftp binary. maybe it tells you more instantly?

if the connection really is dropped regularly i suggest raising --num-retries 
and moving the backup to another schedule as the server might be under high 
load and is dropping connections because of that.


..ede/duply.net