Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?

[Remy van Elst]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

And here's the end part of the backup. All the lines in between were
just files being compared. I can upload that log as well, but it is a
huge logfile.

https://e75f5515c4f249ffa1122853af811b3e.objectstore.eu/duplicity/duplic
ity2.txt


Do note that even though the backup completed after 4 hours, the
collection-status command doesn't show it. The backend had increased
with a few hundred gb of storage.


This command output is from after the backup has completed:


    # Hostname: vps1.example.org
    # Storage used: 1.8T
    # Full backups to keep: 4
    # Create full backup if last full backup is older than: 14D
    -----------------------------------------
    # Duplicity collection status:
    # Local and Remote metadata are synchronized, no sync needed.
    # Last full backup date: Thu Apr 30 05:49:39 2015
    # Collection Status
    # -----------------
    # Connecting with backend: BackendWrapper
    # Archive dir: /root/.cache/duplicity/vps1.example.org.
    # Found 3 secondary backup chains.
    # Secondary chain 1 of 3:
    # -------------------------
    # Chain start time: Tue Mar 24 14:42:44 2015
    # Chain end time: Tue Mar 24 14:42:44 2015
    # Number of contained backup sets: 1
    # Total number of contained volumes: 3009
    #  Type of backup set:                            Time:      Num
volumes:
    #                 Full         Tue Mar 24 14:42:44 2015
   3009
    # -------------------------
    # Secondary chain 2 of 3:
    # -------------------------
    # Chain start time: Wed Apr  8 05:49:18 2015
    # Chain end time: Wed Apr  8 05:49:18 2015
    # Number of contained backup sets: 1
    # Total number of contained volumes: 3104
    #  Type of backup set:                            Time:      Num
volumes:
    #                 Full         Wed Apr  8 05:49:18 2015
   3104
    # -------------------------
    # Secondary chain 3 of 3:
    # -------------------------
    # Chain start time: Thu Apr 16 05:49:37 2015
    # Chain end time: Thu Apr 16 05:49:37 2015
    # Number of contained backup sets: 1
    # Total number of contained volumes: 3128
    #  Type of backup set:                            Time:      Num
volumes:
    #                 Full         Thu Apr 16 05:49:37 2015
   3128
    # -------------------------
    # Found primary backup chain with matching signature chain:
    # -------------------------
    # Chain start time: Thu Apr 30 05:49:39 2015
    # Chain end time: Thu Apr 30 05:49:39 2015
    # Number of contained backup sets: 1
    # Total number of contained volumes: 734
    #  Type of backup set:                            Time:      Num
volumes:
    #                 Full         Thu Apr 30 05:49:39 2015
    734
    # -------------------------
    # No orphaned or incomplete backup sets found.


On 05/28/2015 03:54 PM, address@hidden wrote:
> On 28.05.2015 15:28, Remy van Elst wrote:
>> Any suggestions or other help?
>> 
>> On 05/25/2015 11:04 AM, Remy van Elst wrote:
>>> Duplicity seems to loose it's status when trying to do a 
>>> incremental backup set over 14 days. The backup has been
>>> running for a while now but it is using about 1.5 TB of backend
>>> storage since 2 weeks. It is going to an Openstack swift object
>>> store as backend.
>> 
>>> This is the command line:
>> 
>>> duplicity --asynchronous-upload --verbosity 9 --log-file 
>>> /var/log/duplicity.log --volsize 25 --tempdir="/tmp" 
>>> --file-prefix="vps1.sparklingclouds.nl." 
>>> --name="vps1.sparklingclouds.nl." --exclude-device-files 
>>> --exclude-globbing-filelist=/etc/duplicity-backup/exclude.conf
>>>  --full-if-older-than="14D" --no-encryption  / 
>>> swift://duplicity-backup
>> 
>>> This is what caught my eye in the verbose log file:
>> 
>>> File 
>>> vps1.sparklingclouds.nl.duplicity-full.20150430T034939Z.vol166.difft
ar
>>
>>> 
.g
>> 
>> 
>> z
>>> is part of known set File 
>>> vps1.sparklingclouds.nl.duplicity-full.20150430T034939Z.vol1660.diff
ta
>>
>>> 
r.
>> 
>> 
>> gz
>>> is part of known set Found backup chain [Tue Mar 24 14:42:44 
>>> 2015]-[Tue Mar 24 14:42:44 2015] Last full backup is too old, 
>>> forcing full backup
>> 
>> 
>>> The last full backup is from yesterday, and it should only do
>>> a full backup every 14 days.
>> 
>>> When I do a collection status it seems all the incremental
>>> backups are gone:
>> 
>>> Do note that I've got a wrapper script around duplicity which 
>>> outputs the status, storage used and config settings:
>> 
>> 
>>> # Hostname: vps1.sparklingclouds.nl # IP: 192.0.2.10 # Storage 
>>> used: 1.5T # Full backups to keep: 4 # Create full backup if
>>> last full backup is older than: 14D 
>>> ----------------------------------------- # Duplicity
>>> collection status: # Local and Remote metadata are
>>> synchronized, no sync needed. # Last full backup date: Mon May
>>> 25 10:48:16 2015 # Collection Status # ----------------- #
>>> Connecting with backend: BackendWrapper # Archive dir: 
>>> /root/.cache/duplicity/vps1.sparklingclouds.nl. # Found 4
>>> secondary backup chains. # Secondary chain 1 of 4: # 
>>> ------------------------- # Chain start time: Tue Mar 24
>>> 14:42:44 2015 # Chain end time: Tue Mar 24 14:42:44 2015 #
>>> Number of contained backup sets: 1 # Total number of contained
>>> volumes: 3009 #  Type of backup set:
>>> Time:      Num volumes: #                 Full         Tue Mar
>>> 24 14:42:44 2015 3009 # ------------------------- # Secondary
>>> chain 2 of 4: # ------------------------- # Chain start time:
>>> Wed Apr  8 05:49:18 2015 # Chain end time: Wed Apr  8 05:49:18
>>> 2015 # Number of contained backup sets: 1 # Total number of
>>> contained volumes: 3104 #  Type of backup set:
>>> Time:      Num volumes: #                 Full         Wed Apr
>>> 8 05:49:18 2015 3104 # ------------------------- # Secondary
>>> chain 3 of 4: # ------------------------- # Chain start time:
>>> Thu Apr 16 05:49:37 2015 # Chain end time: Thu Apr 16 05:49:37
>>> 2015 # Number of contained backup sets: 1 # Total number of
>>> contained volumes: 3128 #  Type of backup set:
>>> Time:      Num volumes: #                 Full         Thu Apr
>>> 16 05:49:37 2015 3128 # ------------------------- # Secondary
>>> chain 4 of 4: # ------------------------- # Chain start time:
>>> Thu Apr 30 05:49:39 2015 # Chain end time: Thu Apr 30 05:49:39
>>> 2015 # Number of contained backup sets: 1 # Total number of
>>> contained volumes: 736 # Type of backup set:
>>> Time:      Num volumes: #                 Full         Thu Apr
>>> 30 05:49:39 2015 736 # ------------------------- # Found
>>> primary backup chain with matching signature chain: #
>>> ------------------------- # Chain start time: Mon May 25
>>> 10:48:16 2015 # Chain end time: Mon May 25 10:48:16 2015 #
>>> Number of contained backup sets: 1 # Total number of contained
>>> volumes: 0 #  Type of backup set: Time:      Num volumes: #
>>> ------------------------- # No orphaned or incomplete backup
>>> sets found.
>> 
>> 
>>> Why does this happen and how can I get this back to normal?
>> 
> 
> which duplicity version?
> 
> please post (zip attach)or pastebin the complete console output of
> a max.verbosity '-v9' status run. obfuscate strings in it you deem
> private.
> 
> ..ede/duply.net
> 
> 
> _______________________________________________ Duplicity-talk
> mailing list address@hidden 
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
> 
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBCAAGBQJVZ2L1AAoJECtnVb0bf4jcq8oIAIFVQw3yIvv74qX8lOSgn5J2
VrrTnpgY2ypQf63l+4HMT8lBV/bklW9sr/UZ+so9mvGHpZiF2yGk6QDFM+ZhM6Eg
LKQjXvE8U2Wq8nUuEfUmd3jqZyEwiHK2pvu+eVSY2JxzK59EFI7XEFA5MF4FPTZ7
Q7c5QpCQA6zeiVB2xV2QlbiT5JatcpIfEof45Rec8uOl4tQvali8W18hQmCVMMVX
j/iCi35BnbhFjnuWsDLb9r78BW1dKxjQPAPk6fvB8tpdDLyIt3ozHpSvONEoy2Uc
9MvvsAl1cCYo8XFKL1ATBBEw/0/XQqi7ijOfOOlDSKlQrVnacLqrYZxjyA424V4=
=MrsD
-----END PGP SIGNATURE-----

0x1B7F88DC.asc
Description: application/pgp-keys

0x1B7F88DC.asc.sig
Description: PGP signature