[Duplicity-talk] Warning about certain AWS S3 storage classes

[Mirko Vogt]

Hello,

I'd like to share a first-hand nightmare story with duplicity and 
Amazon's AWS S3 as storage backend.
This is not about a failed backup/recovery in the way of data loss, but 
more like money loss.

Lot's of tutorials suggest using duplicity's `--s3-use-ia` option when 
backing up to AWS S3 buckets.

It changes the AWS S3 storage class of stored objects in respective 
bucket to a more cost efficient one, if requests are low and objects are 
stored >=30 days. From the option's description in duplicity:
"This storage class  has  a  lower storage cost but a higher per-request 
cost, and the storage cost is calculated against a 30-day storage minimum.
According to Amazon, this storage is ideal for long-term file storage, 
backups, and disaster recovery.".

What it does *not* mention, though, is: there's a penalty if objects are 
stored <30 days. And I'm not talking about "they're billed for 30 days 
regardless of if they get deleted before", but an actual penalty.

On the invoice it's stated as:

"$0.0135 per GB-Month prorated for objects deleted or overwritten before 
30 days in Standard-Infrequent Access 3,113.091 GB-Mo"

One might be inclined to think, that this does not usually apply, as 
backups are usually stored >30 days - and that was indeed the case for 
me the past months or even years.

What most likely happened in my very case now, though, is, that my 
duplicity script /attempted/ to do a (full) backup, but failed - after 
already having written hundreds of GB to AWS.

Next time the script was called (usually once a day by cron) it cleaned 
up the incomplete data from the previously aborted session. Hence, 
hundreds of GBs of objects on IA-storage did only stay there around 1 day.

My bill for August increased by literally 822%.

Long story short: While this is definitely not duplicity's fault, I'd 
suggest adding this as a warning to the docs / man-page for the 
`s3-use-ia`-option.

There are also penalties for other storage classes supported by 
duplicity, namely the various glaciers / deep-archiving ones. Those 
might also deserve a warning.

Best

  mirko