Re: [Duplicity-talk] full backup killed

[edgar . soldin]

Russell,

1.
do you use the latest duplicity 0.7.12 ? if not, try it.

2.
if the problem persists, can you reboot the box and run a full and after it was 
killed, zip the complete 'dmesg' output and send it to me personally?

..ede&/duply.net


On 19.04.2017 00:11, Nate Eldredge via Duplicity-talk wrote:
> That's weird.  My next suggestion would be strace -vvv to try to determine if 
> the process is SIGKILLing itself, or if some other process is doing it, or if 
> it's coming from the kernel (though I'd expect a log message in that case).
> 
> On Tue, 18 Apr 2017, Russell Clemings via Duplicity-talk wrote:
> 
>> This is CentOS 7. I'm not seeing anything in the logs about a memory issue
>> in any case.
>>
>> On Tue, Apr 18, 2017 at 2:30 PM, Nate Eldredge <address@hidden>
>> wrote:
>>
>>> This happens on Linux machines (maybe others) when the machine runs out of
>>> memory.  If that's the case, you should see an entry in your syslog talking
>>> about oom-killer.  Check your syslog in any case.
>>>
>>> I was encountering this recently on an Ubuntu 16.04 machine.  There was a
>>> kernel bug where the kernel would think it was out of memory, and start
>>> killing processes, even though there was plenty of swap.  See
>>> https://bugs.launchpad.net/bugs/1655842.  It is fixed in recent updates
>>> (you will see people there claiming it is not fixed, but I think they are
>>> either finding different bugs or they really are out of memory).  At least,
>>> since updating it no longer happens for me.
>>>
>>>
>>> On Tue, 18 Apr 2017, Russell Clemings via Duplicity-talk wrote:
>>>
>>> This looks like the same problem discussed last summer in
>>>> https://lists.nongnu.org/archive/html/duplicity-talk/2016-
>>>> 07/msg00005.html
>>>> but I can't see that there was any resolution.
>>>>
>>>> Full (but not incremental) backups are (apparently at least) failing with
>>>> no traceback, just the word "Killed," and the lockfile is not removed.
>>>>
>>>> I've pasted the last lines below from the most recent attempt.
>>>>
>>>> It seems to be failing at same place in the backup every time (after
>>>> var/spool/mail/wcsj2017). But the "Processing local manifest" filename is
>>>> different each time.
>>>>
>>>> It worked fine until a few days ago. And as I said, incremental backups
>>>> are
>>>> fine (although the full backup failure leaves a lock file that has to be
>>>> removed; same filename each time). Running collection-status yields the
>>>> results I expect -- a list of full backups and associated incrementals,
>>>> and
>>>> "No orphaned or incomplete backup sets found." So maybe the backup is
>>>> succeeding despite the "Failed" message and lock problem.
>>>>
>>>> I'm not seeing anything that would suggest a memory or load issue. Just in
>>>> case, I've run it at different times of day as well as in the middle of
>>>> the
>>>> night, with the same results.
>>>>
>>>> Suggestions? Should I maybe clean out
>>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d on the theory
>>>> that
>>>> the backend is fine but the local cache is corrupted?
>>>>
>>>> A var/spool/mail/swnet
>>>> A var/spool/mail/wcsj17dv
>>>> A var/spool/mail/wcsj2017
>>>> AsyncScheduler: running task synchronously (asynchronicity disabled)
>>>> Writing duplicity-full.20170418T140701Z.vol115.difftar.gpg
>>>> Uploading s3://
>>>> s3.amazonaws.com/BUCKETNAME/duplicity-full.20170418T140701Z.
>>>> vol115.difftar.gpg
>>>> to STANDARD Storage
>>>> Deleting
>>>> /home/newton/duplicity-0.7.12/s3backuptmp/duplicity-7y1aYN-t
>>>> empdir/mktemp-d8rFt2-116
>>>> AsyncScheduler: task completed successfully
>>>> Processed volume 115
>>>> Writing duplicity-full-signatures.20170418T140701Z.sigtar.gpg
>>>> Uploading s3://
>>>> s3.amazonaws.com/BUCKETNAME/duplicity-full-signatures.201704
>>>> 18T140701Z.sigtar.gpg
>>>> to STANDARD Storage
>>>> Deleting
>>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d/
>>>> duplicity-full-signatures.20170418T140701Z.sigtar.gpg
>>>> Writing duplicity-full.20170418T140701Z.manifest.gpg
>>>> Uploading s3://
>>>> s3.amazonaws.com/BUCKETNAME/duplicity-full.20170418T140701Z.manifest.gpg
>>>> to
>>>> STANDARD Storage
>>>> Deleting
>>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d/
>>>> duplicity-full.20170418T140701Z.manifest.gpg
>>>> Processing local manifest
>>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d/
>>>> duplicity-full.20170128T070703Z.manifest
>>>> (28635198)
>>>> Found 97 volumes in manifest
>>>> Processing local manifest
>>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d/
>>>> duplicity-full.20170204T070703Z.manifest
>>>> (28771709)
>>>> Found 99 volumes in manifest
>>>> Processing local manifest
>>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d/
>>>> duplicity-full.20170211T070707Z.manifest
>>>> (28798317)
>>>> Found 100 volumes in manifest
>>>> Processing local manifest
>>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d/
>>>> duplicity-full.20170218T070703Z.manifest
>>>> (28785755)
>>>> Found 102 volumes in manifest
>>>> Processing local manifest
>>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d/
>>>> duplicity-full.20170225T070705Z.manifest
>>>> (29127918)
>>>> Found 104 volumes in manifest
>>>> Processing local manifest
>>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d/
>>>> duplicity-full.20170304T070703Z.manifest
>>>> (29236665)
>>>> Found 106 volumes in manifest
>>>> Processing local manifest
>>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d/
>>>> duplicity-full.20170311T070712Z.manifest
>>>> (29239486)
>>>> Found 107 volumes in manifest
>>>> Processing local manifest
>>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d/
>>>> duplicity-full.20170318T060703Z.manifest
>>>> (29246628)
>>>> Found 109 volumes in manifest
>>>> Processing local manifest
>>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d/
>>>> duplicity-full.20170325T060703Z.manifest
>>>> (29266901)
>>>> Found 111 volumes in manifest
>>>> Processing local manifest
>>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d/
>>>> duplicity-full.20170401T060704Z.manifest
>>>> (29375862)
>>>> Found 112 volumes in manifest
>>>> Processing local manifest
>>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d/
>>>> duplicity-full.20170401T151402Z.manifest
>>>> (29378174)
>>>> Found 112 volumes in manifest
>>>> Processing local manifest
>>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d/
>>>> duplicity-full.20170408T060702Z.manifest
>>>> (29085454)
>>>> Found 113 volumes in manifest
>>>> Processing local manifest
>>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d/
>>>> duplicity-full.20170415T060703Z.manifest
>>>> (28559242)
>>>> /home/newton/duplicity-0.7.10/s3backup/backup_full.sh: line 97:  9126
>>>> Killed                  /home/newton/duplicity-0.7.12/bin/duplicity full
>>>> --verbosity 8 --timeout=30 --num-retries=5 --volsize=250
>>>> --tempdir=/home/newton/duplicity-0.7.12/s3backuptmp
>>>> --encrypt-key=${GPG_KEY} --sign-key=${GPG_KEY}
>>>> --exclude=/home/newton/duplicity-0.7.10/s3backup
>>>> --exclude=/home/newton/duplicity-0.7.12/s3backup --exclude=/home/.cpan
>>>> --exclude=/home/.cpcpan --exclude=/home/virtfs --exclude=/root/.cache
>>>> --exclude=/proc --exclude=/dev --exclude=/home/scie2010/public_html/media
>>>> --exclude=/home/mailarch-nasw/archives --exclude=/etc/csf/csf.deny
>>>> --exclude=/etc/udev/devices/ploop54946p1
>>>> --exclude=/etc/valiases/nasw_aliases
>>>> --exclude=/home/cybraria/.spamassassinenable
>>>> --exclude=/home/sciencew/public_html/stats/logs
>>>> --exclude=/root/.cpanel/caches/dynamicui --exclude=/root/.MirrorSearch/
>>>> myip.cpanel.net --exclude=/root/tmp --exclude=/usr/local/apache/domlogs
>>>> --exclude=/usr/local/apache/logs --exclude=/usr/local/cpanel/3r
>>>> dparty/share
>>>> --exclude=/usr/local/cpanel/base/frontend
>>>> --exclude=/usr/local/cpanel/img-sys --exclude=/usr/local/cpanel/logs
>>>> --exclude=/usr/local/cpanel/src --exclude=/usr/local/cpanel/src/nsswitch
>>>> --exclude=/usr/local/cpanel/src/redhat_pathtools
>>>> --exclude=/usr/local/cpanel/src/remove_sendmail_rpm --include=/etc
>>>> --include=/home/clamav --include=/home/cybraria --include=/home/devnasw
>>>> --include=/home/mailarch-nasw --include=/home/newton
>>>> --include=/home/scie2008 --include=/home/scie2009 --include=/home/scie2010
>>>> --include=/home/scie2011 --include=/home/scie2012 --include=/home/scie2013
>>>> --include=/home/scie2014 --include=/home/scie2015 --include=/home/scie2016
>>>> --include=/home/scie2017 --include=/home/scie2018 --include=/home/scie2019
>>>> --include=/home/scie2020 --include=/home/scie2021 --include=/home/scie2022
>>>> --include=/home/sciencew --include=/home/solr --include=/home/sw2016dv
>>>> --include=/home/swmeeting --include=/home/swnet --include=/home/wcsj17dv
>>>> --include=/home/wcsj2017 --include=/root --include=/s3backup
>>>> --include=/scripts --include=/usr/local --include=/var/lib/mysql
>>>> --include=/var/mail --include=/var/named --include=/var/spool
>>>> --exclude=/**
>>>> ${SOURCE} ${DEST}
>>>>
>>>>
>>> -- 
>>> Nate Eldredge
>>> address@hidden
>>>
>>>
>>
>>
>> -- 
>> ===============================================
>> Russell Clemings
>> <address@hidden>
>> ===============================================
>>
>