[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Qemu-block] [Qemu-devel] [PATCH for-2.12 0/4] qmp dirty bitmap API

From: Kirill Korotaev
Subject: Re: [Qemu-block] [Qemu-devel] [PATCH for-2.12 0/4] qmp dirty bitmap API
Date: Wed, 20 Dec 2017 10:29:27 +0000

Lets me add a couple of words from backup vendor side:

1. pull backup is very important to have, it enables additional important functionality:
1. User can select separate volumes (rather then disks) to backup
2. User can select specific files/folders to backup (!)
3. Backup software can automatically skip copying useless data like swap file/partition. 
All 3 require disk/filesystem analysis, i.e. random disk access on demand (rather then "push all” approach).
2. It is important to have multiple CBT bitmaps and be able to detect modifications between different point in times because one might have at least 2 applications interested in CBT:
a. backup
b. replication


On 20 Dec 2017, at 11:20, Vladimir Sementsov-Ogievskiy <address@hidden> wrote:

20.12.2017 04:06, John Snow wrote:

On 12/19/2017 11:07 AM, Vladimir Sementsov-Ogievskiy wrote:
13.12.2017 07:12, Fam Zheng wrote:
On Mon, 11/13 19:20, Vladimir Sementsov-Ogievskiy wrote:
Hi all.

There are three qmp commands, needed to implement external backup API.

Using these three commands, client may do all needed bitmap
management by

on backup start we need to do a transaction:
  {disable old bitmap, create new bitmap}

on backup success:
  drop old bitmap

on backup fail:
  enable old bitmap
  merge new bitmap to old bitmap
  drop new bitmap

Question: it may be better to make one command instead of two:
block-dirty-bitmap-set-enabled(bool enabled)

Vladimir Sementsov-Ogievskiy (4):
   block/dirty-bitmap: add lock to bdrv_enable/disable_dirty_bitmap
   qapi: add block-dirty-bitmap-enable/disable
   qmp: transaction support for block-dirty-bitmap-enable/disable
   qapi: add block-dirty-bitmap-merge

  qapi/block-core.json         |  80 +++++++++++++++++++++++
  qapi/transaction.json        |   4 ++
  include/block/dirty-bitmap.h |   2 +
  block/dirty-bitmap.c         |  21 ++++++
  blockdev.c                   | 151
  5 files changed, 258 insertions(+)

I think tests are required to merge new features/commands.  Can we
include tests
on these new code please?  We should cover error handling, and also
write tests
that demonstrate the intended real world use cases.

Also should we add new sections to docs/interop/bitmaps.rst?

Meta: John started a long discussion about the API design but I think
after all
it turns out exposing dirty bitmap objects and the primitives is a
approach to implement incremental backup functionalities. The comment
I have is
that we should ensure we have also reviewed it from a higher level
(e.g. all the
potential user requirements) to make sure this low level API is both
sound and
flexible. We shouldn't introduce a minimal set of low level commands
just to
support one particular use case, but look a bit further and broader
and come up
with a more complete design? Writing docs and tests might force us to
think in
this direction, which I think is a good thing to have for this series,

Nikolay, please describe what do you plan in libvirt over qmp bitmap API.

Kirill, what do you think about this all?

(brief history:
we are considering 3 new qmp commands for bitmap management, needed for
external incremental backup support
 - enable (bitmap will track disk changes)
 - disable (bitmap will stop tracking changes)
 - merge (merge bitmap A to bitmap B)

Yeah, it would be helpful to know what the full workflow for the API
will be ... before I get ahead of myself again (sorry) ...

but I'd like to see a quick writeup of your vision for the pull-mode
backups (which I assume this is for, right?) from start-to-finish, like
a mockup of annotated QMP output or something.

Nothing fancy, just something that lets me orient where we're headed,
since you're doing most of the work and I just want to get out of your
way, but having a roadmap helps me do that.


external backup:

0. we have active_disk and attached to it dirty bitmap bitmap0
1. qmp blockdev-add tmp_disk (backing=active_disk)
2. guest fsfreeze
3. qmp transaction:
        - block-dirty-bitmap-add node=active_disk name=bitmap1
        - block-dirty-bitmap-disable node=active_disk name=bitmap0
        - blockdev-backup drive=active_disk target=tmp_disk sync=none
4. guest fsthaw
5. (? not designed yet) qmp blockdev-add filter_node - special filter node over tmp_disk for synchronization of nbd-reads and backup(sync=none) cow requests (like it is done in block/replication)
6. qmp nbd-server-start
7. qmp nbd-server-add filter_node (there should be possibility of exporting bitmap of child node filter_node->tmp_disk->active_disk->bitmap0)

then, external tool can connect to nbd server and get exported bitmap and read data (including bitmap0) accordingly to nbd specification.
(also, external tool may get a merge of several bitmaps, if we already have a sequence of them)
then, after backup finish, what can be done:

1. qmp block-job-cancel device=active_disk (stop our backup(sync=none))
2. qmp nbd-server-stop (or qmp nbd-server-remove filter_node)
3. qmp blockdev-remove filter_node
4. qmp blockdev-remove tmp_disk

on successful backup, you can drop old bitmap if you want (or do not drop it, if you need to keep sequence of disabled bitmaps):
1. block-dirty-bitmap-remove node=active_disk name=bitmap0

on failed backup, you can merge bitmaps, to make it look like nothing happened:
1. qmp transaction:
       - block-dirty-bitmap-merge node=active_disk name-source=bitmap1 name-target=bitmap0
       - block-dirty-bitmap-remove node=active_disk name=bitmap1
       - block-dirty-bitmap-enable node=active_disk name=bitmap0

Best regards,

Attachment: smime.p7s
Description: S/MIME cryptographic signature

reply via email to

[Prev in Thread] Current Thread [Next in Thread]