Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?

[Remy van Elst]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256



On 05/29/2015 12:25 AM, address@hidden wrote:
> please do not top post if the answer was posted at the bottom.. i
> will continue below
> 
> On 28.05.2015 19:47, Remy van Elst wrote:
>> Here's the first 31000 lines of the log: 
>> https://e75f5515c4f249ffa1122853af811b3e.objectstore.eu/duplicity/dup
lic
>>
>> 
ity.txt
>> 
>> After these, it is all lines comparing the files on the machine.
>> That is still running. Is this helpfull?
>> 
>> On 05/28/2015 03:54 PM, address@hidden wrote:
>>> On 28.05.2015 15:28, Remy van Elst wrote:
>>>> Any suggestions or other help?
>>>> 
>>>> On 05/25/2015 11:04 AM, Remy van Elst wrote:
>>>>> Duplicity seems to loose it's status when trying to do a 
>>>>> incremental backup set over 14 days. The backup has been 
>>>>> running for a while now but it is using about 1.5 TB of
>>>>> backend storage since 2 weeks. It is going to an Openstack
>>>>> swift object store as backend.
>>>> 
>>>>> This is the command line:
>>>> 
>>>>> duplicity --asynchronous-upload --verbosity 9 --log-file 
>>>>> /var/log/duplicity.log --volsize 25 --tempdir="/tmp" 
>>>>> --file-prefix="vps1.sparklingclouds.nl." 
>>>>> --name="vps1.sparklingclouds.nl." --exclude-device-files 
>>>>> --exclude-globbing-filelist=/etc/duplicity-backup/exclude.conf
>>>>>
>>>>> 
- --full-if-older-than="14D" --no-encryption  /
>>>>> swift://duplicity-backup
>>>> 
>>>>> This is what caught my eye in the verbose log file:
>>>> 
>>>>> File 
>>>>> vps1.sparklingclouds.nl.duplicity-full.20150430T034939Z.vol166.dif
ft
>>
>>>>> 
ar
>>>> 
>>>>> 
>> .g
>>>> 
>>>> 
>>>> z
>>>>> is part of known set File 
>>>>> vps1.sparklingclouds.nl.duplicity-full.20150430T034939Z.vol1660.di
ff
>>
>>>>> 
ta
>>>> 
>>>>> 
>> r.
>>>> 
>>>> 
>>>> gz
>>>>> is part of known set Found backup chain [Tue Mar 24
>>>>> 14:42:44 2015]-[Tue Mar 24 14:42:44 2015] Last full backup
>>>>> is too old, forcing full backup
>>>> 
>>>> 
>>>>> The last full backup is from yesterday, and it should only
>>>>> do a full backup every 14 days.
>>>> 
>>>>> When I do a collection status it seems all the incremental 
>>>>> backups are gone:
>>>> 
>>>>> Do note that I've got a wrapper script around duplicity
>>>>> which outputs the status, storage used and config
>>>>> settings:
>>>> 
>>>> 
>>>>> # Hostname: vps1.sparklingclouds.nl # IP: 192.0.2.10 #
>>>>> Storage used: 1.5T # Full backups to keep: 4 # Create full
>>>>> backup if last full backup is older than: 14D 
>>>>> ----------------------------------------- # Duplicity 
>>>>> collection status: # Local and Remote metadata are 
>>>>> synchronized, no sync needed. # Last full backup date: Mon
>>>>> May 25 10:48:16 2015 # Collection Status #
>>>>> ----------------- # Connecting with backend: BackendWrapper
>>>>> # Archive dir: 
>>>>> /root/.cache/duplicity/vps1.sparklingclouds.nl. # Found 4 
>>>>> secondary backup chains. # Secondary chain 1 of 4: # 
>>>>> ------------------------- # Chain start time: Tue Mar 24 
>>>>> 14:42:44 2015 # Chain end time: Tue Mar 24 14:42:44 2015 # 
>>>>> Number of contained backup sets: 1 # Total number of
>>>>> contained volumes: 3009 #  Type of backup set: Time:
>>>>> Num volumes: #                 Full         Tue Mar 24
>>>>> 14:42:44 2015 3009 # ------------------------- # Secondary 
>>>>> chain 2 of 4: # ------------------------- # Chain start
>>>>> time: Wed Apr  8 05:49:18 2015 # Chain end time: Wed Apr  8
>>>>> 05:49:18 2015 # Number of contained backup sets: 1 # Total
>>>>> number of contained volumes: 3104 #  Type of backup set: 
>>>>> Time:      Num volumes: #                 Full         Wed
>>>>> Apr 8 05:49:18 2015 3104 # ------------------------- #
>>>>> Secondary chain 3 of 4: # ------------------------- # Chain
>>>>> start time: Thu Apr 16 05:49:37 2015 # Chain end time: Thu
>>>>> Apr 16 05:49:37 2015 # Number of contained backup sets: 1 #
>>>>> Total number of contained volumes: 3128 #  Type of backup
>>>>> set: Time:      Num volumes: #                 Full
>>>>> Thu Apr 16 05:49:37 2015 3128 # ------------------------- #
>>>>> Secondary chain 4 of 4: # ------------------------- # Chain
>>>>> start time: Thu Apr 30 05:49:39 2015 # Chain end time: Thu
>>>>> Apr 30 05:49:39 2015 # Number of contained backup sets: 1 #
>>>>> Total number of contained volumes: 736 # Type of backup
>>>>> set: Time:      Num volumes: #                 Full
>>>>> Thu Apr 30 05:49:39 2015 736 # ------------------------- #
>>>>> Found primary backup chain with matching signature chain:
>>>>> # ------------------------- # Chain start time: Mon May 25 
>>>>> 10:48:16 2015 # Chain end time: Mon May 25 10:48:16 2015 # 
>>>>> Number of contained backup sets: 1 # Total number of
>>>>> contained volumes: 0 #  Type of backup set: Time:      Num
>>>>> volumes: # ------------------------- # No orphaned or
>>>>> incomplete backup sets found.
>>>> 
>>>> 
>>>>> Why does this happen and how can I get this back to
>>>>> normal?
>>>> 
>> 
>>> which duplicity version?
>> 
>>> please post (zip attach)or pastebin the complete console output
>>> of a max.verbosity '-v9' status run. obfuscate strings in it
>>> you deem private.
>> 
>>> ..ede/duply.net
>> 
> 
> the log states
> 
> --> DEBUG 1 . Found backup chain [Tue Mar 24 14:42:44 2015]-[Tue
> Mar 24 14:42:44 2015]
> 
> DEBUG 1 . Found backup chain [Wed Apr  8 05:49:18 2015]-[Wed Apr  8
> 05:49:18 2015]
> 
> DEBUG 1 . Found backup chain [Thu Apr 16 05:49:37 2015]-[Thu Apr 16
> 05:49:37 2015]
> 
> DEBUG 1 . Found backup chain [Thu Apr 30 05:49:39 2015]-[Thu Apr 30
> 05:49:39 2015]
> 
> NOTICE 1 . Last full backup date: Thu Apr 30 05:49:39 2015
> 
> NOTICE 1 . Last full backup is too old, forcing full backup <--
> 
> which sounds reasonable.. there are no incremental backups on your
> backend.

There should be. There runs a cronjob every night with the same
command as I ran manually. It runs for about 3 to 5 hours just as now,
and then fails. The backend keeps growing in size.

> 
> 1. how long does a full backup run on your box? 2. how often is the
> backup job started via cron?
> 
> ..ede/duply.net
> 
> _______________________________________________ Duplicity-talk
> mailing list address@hidden 
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
> 
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBCAAGBQJVZ9HmAAoJECtnVb0bf4jcLBAH/RjbKspywZ8QUyv6P+qmwXNq
kLybE/XsQEguntvUZkDEzv7RIL0RvQsZwgFxQ0xTD9IJSjsdmjZ4nMSzvF6s7zWn
+PNP5TJYIm7QeQSabCcA7tp4fRDCwzS/qw28gx4ibOGb9KHBdOEjK9DH7F9n4bwL
CdbjGk0RU3/PWOhfljVf7Rcam6LX6scYM7Iap2GD4/mfUW2if1p51/vlM/2rrb52
xo9k6IgckcBEZcxGuZVWLH+7ZdLvpxfUNdXet9TJZmGOJGg6A62qru1T38lVC753
LGDa1/+gqMuXfwdrKQqLELopeStY7mlP9ejeL6kiAnZh942ZuERs3t3emSsnAdk=
=GkWu
-----END PGP SIGNATURE-----

0x1B7F88DC.asc
Description: application/pgp-keys

0x1B7F88DC.asc.sig
Description: PGP signature