RFC: Qemu backup interface plans

[Vladimir Sementsov-Ogievskiy]

Hi all!

I'd like to share and discuss some plans on Qemu backup interface I have. 
(actually, most of this I've presented on KVM Forum long ago.. But now I'm a 
lot closer to realization:)


I'd start with a reminder about image fleecing:

We have image fleecing scheme to export point-in-time state of active
disk (iotest 222):


                                      backup(sync=none)
                     ┌───────────────────────────────────────┐
                     ▼                                       │
┌────────────┐     ┌────────────────┐  backing             ┌─────────────┐
│ NBD export │ ─── │ temp qcow2 img │ ───────────────────▶ │ active disk │
└────────────┘     └────────────────┘                      └─────────────┘
                                                             ▲
┌────────────┐                                               │
│ guest blk  │ ──────────────────────────────────────────────┘
└────────────┘


Actually, backup job inserts a backup-top filter, so in detail it looks
like:

                                      backup(sync=none)
                     ┌───────────────────────────────────────┐
                     ▼                                       │
┌────────────┐     ┌────────────────┐  backing             ┌─────────────┐
│ NBD export │ ─── │ temp qcow2 img │ ───────────────────▶ │ active disk │
└────────────┘     └────────────────┘                      └─────────────┘
                     ▲                                       ▲
                     │ target                                │
                     │                                       │
┌────────────┐     ┌────────────────┐  backing               │
│ guest blk  │ ──▶ │   backup-top   │ ───────────────────────┘
└────────────┘     └────────────────┘
 

This scheme is also called external backup or pull backup. It allows some 
external tool to write data to actual backup, and Qemu only provides this data.

We support also incremental external backup: Qemu can manage dirty bitmaps in any way 
user wants, and we can export bitmaps through NBD protocol. So, client of NBD export can 
get the bitmap, and read only "dirty" regions of exported image.

What we lack in this scheme:

1. handling dirty bitmap in backup-top filter: backup-top does copy-before-write 
operation on any guest write, when actually we are interested only in "dirty" 
regions for incremental backup

Probable solution would allowing specifying bitmap for sync=none mode of 
backup, but I think what I propose below is better.

2. [actually it's a tricky part of 1]: possibility to not do copy-before-write 
operations for regions that was already copied to final backup. With normal 
Qemu backup job, this is achieved by the fact that block-copy state with its 
internal bitmap is shared between backup job and copy-before-write filter.

3. Not a real problem but fact: backup block-job does nothing in the scheme, 
the whole job is done by filter. So, it would be interesting to have a 
possibility to simply insert/remove the filter, and avoid block-job creation 
and managing at all for external backup. (and I'd like to send another RFC on 
how to insert/remove filters, let's not discuss it here).


Next. Think about internal backup. It has one drawback too:
4. If target is remote with slow connection, copy-before-write operations will 
slow down guest writes appreciably.

It may be solved with help of image fleecing: we create temporary qcow2 image, 
setup fleecing scheme, and instead of exporting temp image through NBD we start 
a second backup with source = temporary image and target would be real backup 
target (NBD for example). Still, with such solution there are same [1,2] 
problems, 3 becomes worse:

5. We'll have two jobs and two automatically inserted filters, when actually 
one filter and one job are enough (as first job is needed only to insert a 
filter, second job doesn't need a filter at all).

Note also, that this (starting two backup jobs to make push backup with 
fleecing) doesn't work now, op-blockers will be against. It's simple to fix 
(and in Virtuozzo we live with downstream-only patch, which allows push backup 
with fleecing, based on starting two backup jobs).. But I never send a patch, 
as I have better plan, which will solve all listed problems.


So, what I propose:

1. We make backup-top filter public, so that it could be inserted/removed where 
user wants through QMP (how to properly insert/remove filter I'll post another 
RFC, as backup-top is not the only filter that can be usefully inserted 
somewhere). For this first step I've sent a series today:

  subject: [PATCH 00/21] block: publish backup-top filter
  id: <20210517064428.16223-1-vsementsov@virtuozzo.com>
  patchew: 
20210517064428.16223-1-vsementsov@virtuozzo.com/">https://patchew.org/QEMU/20210517064428.16223-1-vsementsov@virtuozzo.com/

(note, that one of things in this series is rename 
s/backup-top/copy-before-write/, still, I call it backup-top in this letter)

This solves [3]. [4, 5] are solved partly: we still have one extra filter, 
created by backup block jobs, and also I didn't test does this work, probably 
some op-blockers or permissions should be tuned. So, let it be step 2:

2. Test, that we can start backup job with source = (target of backup-top filter), so 
that we have "push backup with fleecing". Make an option for backup to start 
without a filter, when we don't need copy-before-write operations, to not create extra 
superfluous filter.

3. Support bitmap in backup-top filter, to solve [1]

3.1 and make it possible to modify the bitmap externally, so that consumer of 
fleecing can say to backup-top filter: I've already copied these blocks, don't 
bother with copying them to temp image". This is to solve [2].

Still, how consumer of fleecing will reset shared bitmap after copying blocks? I have the 
following idea: we make a "discard-bitmap-filter" filter driver, that own some 
bitmap and on discard request unset corresponding bits. Also, on read, if read from the 
region with unset bits the EINVAL returned immediately. This way both consumers (backup 
job and NBD client) are able to use this interface:

Backup job can simply call discard on source, we can add an option for this.
External backup tool will send TRIM request after reading some region. This way 
disk space will be freed and no extra copy-before-write operations will be 
done. I also have a side idea that we can implement READ_ONCE flag, so that 
READ and TRIM can be done in one NBD command. But this works only for clients 
that don't want to implement any kind of retrying.



So, finally, how will it look (here I call backup-top with a new name, and "file" child is used 
instead of "backing", as this change I propose in "[PATCH 00/21] block: publish backup-top 
filter"):

Pull backup:

┌────────────────────────────────────┐
│             NBD export             │
└────────────────────────────────────┘
  │
  │
┌────────────────────────────────────┐  file   
┌───────────────────────────────────────┐  backing   ┌─────────────┐
│ discard-bitmap filter (bitmap=bm0) │ ──────▶ │            temp qcow2 img      
       │ ─────────▶ │ active disk │
└────────────────────────────────────┘         
└───────────────────────────────────────┘            └─────────────┘
                                                 ▲                              
                      ▲
                                                 │ target                       
                      │
                                                 │                              
                      │
┌────────────────────────────────────┐         
┌───────────────────────────────────────┐  file        │
│             guest blk              │ ──────▶ │ copy-before-write filter 
(bitmap=bm0) │ ─────────────┘
└────────────────────────────────────┘         
└───────────────────────────────────────┘



Operations:

- Create temp qcow2 image
- blockdev-add to add the new image, setup its backing, and add two filters
- some command to actually set backup-top filter as child of guest blk. That's a 
"point-in-time" of backup. Should be done during fs-freeze.
- start NBD export on top of discard filter (and we can export bitmap bm0 as 
well, for the client)

Now NBD client (our external backup tool) can:

 - import the bitmap
 - READ the data
 - send DISCARD requests on already handled areas to save disk space and avoid 
extra copy-before-write operations on host node


Push backup with fleecing:

┌─────────────────────┐
│ final backup target │
└─────────────────────┘
  ▲
  │ backup job (bitmap=bm0, insert-filter=False, discard-source=True)
  │
┌────────────────────────────────────┐  file   
┌───────────────────────────────────────┐  backing   ┌─────────────┐
│ discard-bitmap filter (bitmap=bm0) │ ──────▶ │            temp qcow2 img      
       │ ─────────▶ │ active disk │
└────────────────────────────────────┘         
└───────────────────────────────────────┘            └─────────────┘
                                                 ▲                              
                      ▲
                                                 │ target                       
                      │
                                                 │                              
                      │
┌────────────────────────────────────┐         
┌───────────────────────────────────────┐  file        │
│             guest blk              │ ──────▶ │ copy-before-write filter 
(bitmap=bm0) │ ─────────────┘
└────────────────────────────────────┘         
└───────────────────────────────────────┘


Note, that the whole fleecing part is the same, we only need to run backup job 
instead of NBD export.



Additional future idea. Why we need push backup with fleecing? To handle cases 
with slow connection to backup target. In any case when writing to remote 
target is slower than writing to local file, push-backup-with-fleecing will 
less disturb the running guest than simple backup job. But this is not free:

1. We need additional disk space on source. No way to fix that (that's a core 
idea of the scheme, store data locally), still discard-source option for backup 
job will help

2. If connection is not too slow than probably part of CBW (copy before write) 
operations could go to final target immediately. But with the scheme above all 
CBW operations go to qcow2 temporary image. This can be solved with help of 
ram-cache format driver (to be implemented, of course):


┌─────────────────────┐
│ final backup target │
└─────────────────────┘
  ▲
  │ backup job (bitmap=bm0, insert-filter=False, discard-source=True)
  │
┌────────────────────────────────────┐         
┌───────────────────────────────────────┐  backing   ┌─────────────┐
│ discard-bitmap filter (bitmap=bm0) │         │            temp qcow2 img      
       │ ─────────▶ │ active disk │
└────────────────────────────────────┘         
└───────────────────────────────────────┘            └─────────────┘
                                 │               ▲                              
                      ▲
                                 │               │ file                         
                      │
                                 │               │                              
                      │
                                 │      file   ┌───────────┐                    
                      │
                                 └───────────▶ │ ram-cache │                    
                      │
                                               └───────────┘                    
                      │
                                                 ▲                              
                      │
                                                 │ target                       
                      │
                                                 │                              
                      │
┌────────────────────────────────────┐         
┌───────────────────────────────────────┐  file        │
│             guest blk              │ ──────▶ │ copy-before-write filter 
(bitmap=bm0) │ ─────────────┘
└────────────────────────────────────┘         
└───────────────────────────────────────┘

This way data from copy-before-write filter goes first to ram-cache, and backup 
job could read it from ram. ram-cache will automatically flush data to temp 
qcow2 image, when ram-usage limit is reached. We'll also need a way to say 
backup-job that it should first read clusters that are cached in ram, and only 
then other clusters. So, we'll have a priory for clusters to be copied by 
block-copy:
1. clusters in ram-cache
2. clusters not in temp img (to avoid copy-before-write operations in future)
3. clusters in temp img.

This will be a kind of block_status() thing, that allows a block driver to give 
recommendations on sequence of reading to be effective. Not also, that there is 
another benefit of such thing: we'll implement this callback in qcow2 driver, 
so that backup will read clusters not in guest cluster order, but in host 
cluster order, to read more sequentially, which should bring better performance 
on rotating disks.


--
Best regards,
Vladimir