[Qemu-devel] [PATCH v2 00/10] block: incremental backup transactions using BlockJobTxn

[Stefan Hajnoczi]

v2:
 Patch 5:
  * Set txn pointer to NULL in block_job_txn_begin() [jsnow]
  * Rename block_job_txn_job_done to block_job_txn_job_done [jsnow]
  * Rename block_job_txn_complete to block_job_txn_kick [jsnow]
  * Add BLOCK_JOB_TXN_CANCEL_PENDING to solve race condition on cancel [jsnow]
  * Document when txn may be NULL
 Patch 7:
  * Convert blockdev-backup in addition to drive-backup
  * Add 'transactional-cancel' argument so users can enable/disable
    transactional behavior.  This preserves semantics for existing users
    and allows fine-grained control over when to use transaction
    semantics. [jsnow]
 Patch 10:
  * Test fail/cancel race condition [jsnow]
  * Use new 'transactional-cancel' QMP argument

This series uses patches from John Snow's "[PATCH v6 00/10] block: incremental
backup transactions" series but implements the feature with a new transaction
mechanism for blockjobs called BlockJobTxn.

Recap: motivation for block job transactions
--------------------------------------------
If an incremental backup block job fails then we reclaim the bitmap so
the job can be retried.  The problem comes when multiple jobs are started as
part of a qmp 'transaction' command.  We need to group these jobs in a
transaction so that either all jobs complete successfully or all bitmaps are
reclaimed.

Without transactions, there is a case where some jobs complete successfully and
throw away their bitmaps, making it impossible to retry the backup by rerunning
the command if one of the jobs fails.

How does this implementation work?
----------------------------------
These patches add a BlockJobTxn object with the following API:

  txn = block_job_txn_new();
  block_job_txn_add_job(txn, job1);
  block_job_txn_add_job(txn, job2);
  block_job_txn_begin();

The jobs either both complete successfully or they both fail/cancel.  If the
user cancels job1 then job2 will also be cancelled and vice versa.

Jobs stay alive waiting for other jobs to complete.  They can be cancelled by
the user during this time.  Job blockers are still in effect and no other block
job can run on this device in the meantime (since QEMU currently only allows 1
job per device).  This is the main drawback to this approach but reasonable
since you probably don't want to run other jobs/operations until you're sure
the backup was successful (you won't be able to retry a failed backup if
there's a new job running).

Adding transaction support to the backup job is very easy.  It just needs to
make a call before throwing away the bitmap and returning from its coroutine:

  block_job_txn_job_done(job->txn, job, ret);

  if (job->sync_bitmap) {
      BdrvDirtyBitmap *bm;
      if (ret < 0 || block_job_is_cancelled(&job->common)) {
          ...

John Snow (4):
  qapi: Add transaction support to block-dirty-bitmap operations
  iotests: add transactional incremental backup test
  block: rename BlkTransactionState and BdrvActionOps
  iotests: 124 - transactional failure test

Kashyap Chamarthy (1):
  qmp-commands.hx: Update the supported 'transaction' operations

Stefan Hajnoczi (5):
  block: keep bitmap if incremental backup job is cancelled
  block: add block job transactions
  blockdev: make BlockJobTxn available to qmp 'transaction'
  block/backup: support block job transactions
  tests: add BlockJobTxn unit test

 block.c                    |  19 ++-
 block/backup.c             |   7 +-
 blockdev.c                 | 364 ++++++++++++++++++++++++++++++++++++---------
 blockjob.c                 | 193 ++++++++++++++++++++++++
 docs/bitmaps.md            |   6 +-
 hmp.c                      |   2 +-
 include/block/block.h      |   2 +-
 include/block/block_int.h  |   6 +-
 include/block/blockjob.h   |  52 +++++++
 qapi-schema.json           |   6 +-
 qapi/block-core.json       |  14 +-
 qmp-commands.hx            |  21 ++-
 tests/Makefile             |   3 +
 tests/qemu-iotests/124     | 182 ++++++++++++++++++++++-
 tests/qemu-iotests/124.out |   4 +-
 tests/test-blockjob-txn.c  | 233 +++++++++++++++++++++++++++++
 trace-events               |   4 +
 17 files changed, 1022 insertions(+), 96 deletions(-)
 create mode 100644 tests/test-blockjob-txn.c

-- 
2.4.3