[Duplicity-talk] Proposal for backgrounded concurrent I/O with new retry support

[Peter Schuller]

Hello,

I have been looking through the code some, with an eye towards two features:

  (1) backgrounding all backend I/O, and allowing for concurrency
  (2) the ability to pause retries until an operator indicates the backend 
should resume retrying, as has been previously discussed on the list

(2) has the benefit discussed previously.

The benefit of (1) is two-fold: Firstly, even with concurrency = 1, 
backgrounding the I/O means that duplicity can keep working on producting 
backup data for a chunk, while the previous chunk is transferred. Depending 
on the relative availability of CPU vs. bandwidth throughput, this can 
potentially double the speed of backups.

In addition, implementing that would also mean that it is trivial to add 
support for backgrounding multiple I/O operations concurrently, which would 
cater to high latency links. This should help people backing up a lot of data 
to remote machines, where neither en is limited by bandwidth but the 
throughput of the TCP connections end up being limited by bandwidth delay 
product and other issues resulting from high latency or perhaps packet loss.

I wanted to describe what I want to do before I start, in case people have 
objections or better ideas.

=== RETRY ===

Firstly, I have been looking through the backends and it seems some of them do 
not implement any retry at all, some of them implement their own retry 
(sometimes on only put()/get(), sometimes on all methods), and yet some use 
Backend.run_command_persist().

I wanted to generalize retry support in a way that minimized the chance of 
breakage; allowing for incrementally adding/improving retry for individual 
backens or even individual methods of backends. In orde to do this my plan is 
to implement a fairly simple function:

def call_persistently(txn):
   """
   Calls the given function txn ("transaction") until it either completes
   or fails permanently. A permanent failure is defined as:
   
      - A user specified number of retries, with a user specified backoff
        behavior, has failed with a retryable error. A retryable error being
        signalled by raising a RetryableError exception.
      - A call to the function causes any other exception to be raised.
   
   On failure, the original exception is propagated back to the caller.
   In the case of a RetryableError, the root cause is propagated rather
   than the RetryableError itself.
   
   Implicit in the above description is that the function can be called any
   number of times, and should therefor be idempotent except insofar
   as the implementation is aware of the potential for repeated
   execution.
   """
   ...

This function would handle everything, including potentially pausing and 
waiting for operator attention. Individual backend methods can be 
converted/upgraded to use this mechanism, and individually tested. It also 
does not change the public interface of backends, nor the relationship 
between Backend and its subclasses.

=== Background concurrent I/O ===

Firstly, a new method is added to Backend's public interface:

   def supports_concurrency():
      """
      Returns whether this backend supports concurrency (multiple
      invocations of backend I/O methods concurrently in multiple
      threads without ill effects).
      
      By default False is returned. Backend implementations that
      support concurrency must override this method.
      """
      return False

This is to allow concurrency while not requireing that all backends are 
converted to support it.

Note that some backends use instance data during I/O (such as a per-instance 
connection objects), and will thus require some changes to support concurrent 
I/O. Others will already support it automatically.

On the front-end of things in duplicity-bin, there are various iterators used 
to read and write chunks to/from the backend. These iterators will be 
modified such that on creation, they schedule all expected I/O operations 
with a scheduler (backed by n worker threads). The iterators then wait on the 
next chunk to have completed I/O, instead of directly instigating the I/O.

This latter change should be the only part that will have widespread effects 
on all backends at once. But on the upside the code is common to all and thus 
testable without having access to backends of all types.

Sample future duplicity invocation:

duplicity incremental /path/to/dir scp://address@hidden:/path \
              --concurrency=5 \
              --retry-failure-mode=intervention \
              --retry-failure-flag=/path/to/intervention-flag

Opinions/thoughts?

-- 
/ Peter Schuller

PGP userID: 0xE9758B7D or 'Peter Schuller <address@hidden>'
Key retrieval: Send an E-Mail to address@hidden
E-Mail: address@hidden Web: http://www.scode.org

signature.asc
Description: This is a digitally signed message part.