Re: [Qemu-devel] [PATCH RFC] external backup api

[Vladimir Sementsov-Ogievskiy]

On 05.02.2016 22:48, John Snow wrote:
> On 01/22/2016 12:07 PM, Vladimir Sementsov-Ogievskiy wrote:
> > Hi all.
> >
> > This is the early begin of the series which aims to add external backup
> > api. This is needed to allow backup software use our dirty bitmaps.
> >
> > Vmware and Parallels Cloud Server have this feature.
> Have a link to the equivalent feature that VMWare exposes? (Or Parallels
> Cloud Server) ... I'm curious about what the API there looks like.

For VMware you need their Virtual Disk Api Programming Guide
http://pubs.vmware.com/vsphere-60/topic/com.vmware.ICbase/PDF/vddk60_programming.pdf
Look at Changed Block Tracking (CBT) , Backup and Restore.

For PCS here is part of SDK header, related to the topic:

====================================
/*
 * Builds a map of the disk contents changes between 2 PITs.
   Parameters
   hDisk :       A handle of type PHT_VIRTUAL_DISK identifying
                 the virtual disk.
   sPit1Uuid :   Uuid of the older PIT.
   sPit2Uuid :   Uuid of the later PIT.
   phMap :       A pointer to a variable which receives the
         result (a handle of type PHT_VIRTUAL_DISK_MAP).
   Returns
   PRL_RESULT.
*/
PRL_METHOD_DECL( PARALLELS_API_VER_5,
                PrlDisk_GetChangesMap_Local, (
        PRL_HANDLE hDisk,
        PRL_CONST_STR sPit1Uuid,
        PRL_CONST_STR sPit2Uuid,
        PRL_HANDLE_PTR phMap) );

/*
 * Reports the number of significant bits in the map.
   Parameters
   hMap :        A handle of type PHT_VIRTUAL_DISK_MAP identifying
                 the changes map.
   phSize :      A pointer to a variable which receives the
         result.
   Returns
   PRL_RESULT.
*/
PRL_METHOD_DECL( PARALLELS_API_VER_5,
                PrlDiskMap_GetSize, (
        PRL_HANDLE hMap,
        PRL_UINT32_PTR pnSize) );

/*
 * Reports the size (in bytes) of a block mapped by a single bit
 * in the map.
   Parameters
   hMap :        A handle of type PHT_VIRTUAL_DISK_MAP identifying
                 the changes map.
   phSize :      A pointer to a variable which receives the
         result.
   Returns
   PRL_RESULT.
*/
PRL_METHOD_DECL( PARALLELS_API_VER_5,
                PrlDiskMap_GetGranularity, (
        PRL_HANDLE hMap,
        PRL_UINT32_PTR pnSize) );

/*
 * Returns bits from the blocks map.
   Parameters
   hMap :        A handle of type PHT_VIRTUAL_DISK_MAP identifying
                 the changes map.
   pBuffer :     A pointer to a store.
   pnCapacity :  A pointer to a variable holding the size
         of the buffer and receiving the number of
         bytes actually written.
   Returns
   PRL_RESULT.
*/
PRL_METHOD_DECL( PARALLELS_API_VER_5,
                PrlDiskMap_Read, (
        PRL_HANDLE hMap,
        PRL_VOID_PTR pBuffer,
        PRL_UINT32_PTR pnCapacity) );

=======================================


> > There is only one patch here, about querying dirty bitmap from qemu by
> > qmp command. It is just an updated and clipped (hmp command removed) old
> > my patch "[PATCH RFC v3 01/14] qmp: add query-block-dirty-bitmap".
> >
> > Before writing the whole thing I'd like to discuss the details. Or, may
> > be there are existing plans on this topic, or may be someone already
> > works on it?
> >
> > I see it like this:
> >
> > =====
> >
> > - add qmp commands for dirty-bitmap functions: create_successor, abdicate,
> > reclaime.
> Hm, why do we need such low-level control over splitting and merging
> bitmaps from an external client?
>
> > - make create-successor command transaction-able
> > - add query-block-dirty-bitmap qmp command
> >
> > then, external backup:
> >
> > qmp transaction {
> >      external-snapshot
> >      bitmap-create-successor
> > }
> >
> > qmp query frozen bitmap, not acquiring aio context.
> >
> > do external backup, using snapshot and bitmap
> >
> > if (success backup)
> >      qmp bitmap-abdicate
> > else
> >      qmp bitmap-reclaime
> >
> > qmp merge snapshot
> > =====
> Hm, I see -- so you're hoping to manage the backup *entirely*
> externally, so you want to be able to reach inside of QEMU and control
> some status conditions to guarantee it'll be safe.
>
> I'm not convinced QEMU can guarantee such things -- due to various flush
> properties, race conditions on write, etc. QEMU handles all of this
> internally in a non-public way at the moment.

Hm, can you be more concrete? What operations are dangerous? We can do 
them in paused state for example.

> > In the following patch query-bitmap acquires aio context. This must be
> > ofcourse dropped for frozen bitmap.
> > But to make it in true way, I think, I should check somehow that this is
> > not just frozen bitmap, but the bitmap frozen by qmp command, to avoid
> > incorrect quering of bitmap frozen by internal backup (or other
> > mechanizm).. May be, it is not necessary.


--
Best regards,
Vladimir