Re: [Duplicity-talk] Show volume Duplicity is working on

[Remy van Elst]

On 10/18/14 14:49, address@hidden wrote:
> 1. exactly what are you trying to achieve? 

I have a server with 1.6 TB of data that takes almost a week to do a
full backup over an umetered gbit connection. It backs up to Rackspace
Openstack Swift, with a default 25MB volume size.

The backup becomes inconsistent because stuff changes a lot on that
server. It has multiple +100 GB databases running. I sent an email
earlier to this list regarding --max-blocksize, but did not receive a
response yet.
(http://lists.nongnu.org/archive/html/duplicity-talk/2014-10/msg00002.html)

With a volume size of 250 MB, or with 2048 MB the upload still takes
more than 3 days sadly.

What I want to achieve is solving the long running backup issue.

> 2. why do you need to know which volume is uploaded?

I was troubleshooting with strace and saw the volume number. I know the
volume size, so with the current volume number I know which one is being
uploaded (async upload enabled). With that data I should be able to
estimate how far the backup is.

> 3. duplicity is written in python script language. you can easily patch your 
> local duplicity to print out the current volume name.
> 

Is also an option, however I would use that as a last resort. I'm using
Duplicity 0.6.24, compiled from source on Debian 7.

> ..ede/duply.net
> 
> On 18.10.2014 14:34, Remy van Elst wrote:
>> I've not tried it with more verbosity yet. I did write a simple script
>> which shows some output, but it requires strace. Example output:
>>
>> Diskspace used: 1591229 MiB / 1553 GiB
>> Duplicity volume: 10127
>> Amount uploaded: 253175 MiB / 247 GiB.
>> 15.91% done, 1306 GiB left to upload of 1553 GiB.
>>
>>
>> Script:
>>
>> #!/bin/bash
>> if [[ -z "$(pgrep strace)" ]]; then
>>
>>     # Attach strace to Duplicity and log writes to /tmp/duplicity.strace
>>     # look for lines like write(6, "Volume 8113:\n    StartingPath
>> "..., 246) = 246
>>     strace -e trace=write -o /tmp/duplicity.strace -p $(pgrep duplicity) &
>>     sleep 5
>> fi
>>
>> if [[ -f "/tmp/duplicity.strace" ]]; then
>>     sleep 5
>>
>>     TOTAL_DISK_USED="$(df -BM --total 2>/dev/null| awk '/total/ {print
>> $3}' | sed -e 's/M//g')"
>>
>>     DUPLICITY_VOL_DONE="$(grep -a -oE 'Volume [0-9]{1,6}'
>> /tmp/duplicity.strace | grep -a -oE '[0-9]{1,6}' | tail -n 1)"
>>     if [[ -z "${DUPLICITY_VOL_DONE}" ]]; then
>>         echo "Error reading current volume"
>>         exit 1
>>     fi
>>
>>     DUPLICITY_MiB_DONE="$(( ${DUPLICITY_VOL_DONE} * 25 ))"
>>
>>     DUPLICITY_GiB_DONE="$(( ( ${DUPLICITY_VOL_DONE} * 25 ) / 1024))"
>>
>>     MiB_LEFT="$(( ${TOTAL_DISK_USED} - ${DUPLICITY_MiB_DONE} ))"
>>
>>     PERC_DONE="$(awk "BEGIN {printf
>> \"%.2f\",(${DUPLICITY_MiB_DONE}/${TOTAL_DISK_USED})*100}")"
>>
>>     if [[ -n "${TOTAL_DISK_USED}" ]]; then
>>         echo "Diskspace used: ${TOTAL_DISK_USED} MiB / $((
>> ${TOTAL_DISK_USED} / 1024 )) GiB"
>>     fi
>>
>>     if [[ -n "${DUPLICITY_VOL_DONE}" ]]; then
>>         echo "Duplicity volume: ${DUPLICITY_VOL_DONE}"
>>     fi
>>
>>     if [[ -n "${DUPLICITY_MiB_DONE}" ]]; then
>>         echo -n "Amount uploaded: ${DUPLICITY_MiB_DONE} MiB / "
>>         echo "${DUPLICITY_GiB_DONE} GiB."
>>     fi
>>
>>     echo "${PERC_DONE}% done, $(( ${MiB_LEFT} / 1024 )) GiB left to
>> upload of $(( ${TOTAL_DISK_USED} / 1024 )) GiB."
>>
>>     echo >/tmp/duplicity.strace
>>
>> fi
>>
>>
>>
>>
>> On 10/18/14 14:17, address@hidden wrote:
>>> On 18.10.2014 10:11, Remy van Elst wrote:
>>>> When I strace attach to duplicity I can see it writing out lines like
>>>> the following:
>>>>
>>>> [pid 22316] write(6, "Volume 7949:\n    StartingPath   "..., 246) = 246
>>>>
>>>> (strace -e trace=write -p <pid>)
>>>>
>>>> That volume number combined with the size of the disk and the volume
>>>> size gives me a better idea how far Duplicity is with it's backup.
>>>>
>>>> How do I access this information without using strace? Is this logged
>>>> somewhere?
>>>>
>>>
>>> did you have a look at the manpage? 
>>>  http://duplicity.nongnu.org/duplicity.1.html
>>> raising the log level '--verbosity' might help.
>>>
>>> ..ede/duply.net
>>>
>>> _______________________________________________
>>> Duplicity-talk mailing list
>>> address@hidden
>>> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>
>>>
>>>
>>> _______________________________________________
>>> Duplicity-talk mailing list
>>> address@hidden
>>> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
> 
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>

0x1B7F88DC.asc
Description: application/pgp-keys

signature.asc
Description: OpenPGP digital signature