[Duplicity-talk] Restoring from PCA Backend - download preparation

[erwan.vallee]

Hi,

 

I’m using duplicity with multibackend (public cloud archive and object storage from ovh), as described in this documentation from ovh :

https://docs.ovh.com/gb/en/storage/pca/duplicity/

I’ve worked on a modification to make PCA Backend work better (for my usage at least !) : allow to unseal all volumes at once when restoring content from PCA, wait for unseal to complete for all volumes before launching download. I’ve used API documentation as mentioned here :

https://docs.ovh.com/gb/en/storage/pca/api/

 

This is a similar change to what has been implemented some time ago for Glacier cold storage :

https://lists.nongnu.org/archive/html/duplicity-talk/2020-07/msg00021.html

I’ve implemented the following things available in this commit : https://gitlab.com/erwanBoka/duplicity/-/commit/822e42ba3c26a5b32377e238a9b6040b9b087052

• Implement in BackendWrapper and MultiBackend pre_process_download (and redirect to sub-backends)
• Implement for pcabackend pre_process_download with ovh API as a blocking call, waiting for all volumes to be unsealed.

 

However, I’m a bit confused by what was implemented then (new hook pre_process_download_batch), compared to existing one pre_process_download.

• pre_process_download is called from dup_main/restore_get_patched_rop_iter with all filenames as parameters
• it’s implemented only once in _boto_single, but with a single filename as argument…
• pre_process_download_batch is essentially doing the same thing (call with all volume names)

I’m afraid my modifications in BackendWrapper and MultiBackend might cause pre_process_download to be called for _boto_single, and break things…

 

I haven’t modified anything yet in _boto_single, but my proposal would be :

• change pre_process_download method name in _boto_single to an « internal » method used only in _get
• keep only pre_process_download hook in dup_main
• implement it the way pre_process_download_batch is implemented today in _boto_single

I don’t have s3 access, and cannot test it though.

 

Let me know your view about this, and the path forward please !

Best regards,

Erwan