Re: [Qemu-devel] Qemu and Changed Block Tracking

[Peter Lieven]

Am 21.02.2017 um 22:13 schrieb John Snow:
>
> On 02/21/2017 07:43 AM, Peter Lieven wrote:
>> Hi,
>>
>>
>> is there anyone ever thought about implementing something like VMware
>> CBT in Qemu?
>>
>>
>> https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1020128
>>
>>
>>
>> Thanks,
>> Peter
>>
>>
> A bit outdated now, but:
> http://wiki.qemu-project.org/Features/IncrementalBackup
>
> and also a summary I wrote not too far back (PDF):
> https://drive.google.com/file/d/0B3CFr1TuHydWalVJaEdPaE5PbFE
>
> and I'm sure the Virtuozzo developers could chime in on this subject,
> but basically we do have something similar in the works, as eblake says.

Hi John, Hi Erik,

thanks for your feedback. Are you both the ones working primary on this topic?
If there is anything to review or help needed, please let me know.

My 2 cents:
I thing I had in mind if there is no image fleecing available, but fetching the 
dirty bitmap
from external would be a feauture to put a write lock on a block device.
Write lock means, drain all pending writes and queue all further writes until 
unlock (as if they
were throttled to zero). This could help fetch consistent backups from storage 
device (thinking of iSCSI SAN) without
the help of the hypervisor to actually transfer data (no load in the frontend 
network or the host). What would further
be needed is a write generation for each block, not just only a dirty bitmap.

In this case something like this via QMP (and external software) should work:
---8<---
 gen =  write generation of last backup (or 0 for full backup)
 do {
     nextgen = fetch current write generation (via QMP)
     dirtymap = send all block whose write generation is greater than 'gen' 
(via QMP)
     dirtycnt = 0
     foreach block in dirtymap {
               copy to backup via external software
               dirtycnt++
     }
     gen = nextgen
 } while (dirtycnt < X)         <--- to achieve this a thorttling or similar 
might be needed

fsfreeze (optional)
write lock (via QMP)
backupgen = fetch current write generation (via QMP)
dirtymap = send all block whose write generation is greater than 'gen' (via QMP)
foreach block in dirtymap {
               copy to backup via external software
}
unlock (via QMP)
fsthaw (optional)
--->8---

As far as I understand CBT in VMware is not just only a dirty bitmap, but also 
a write generation tracking for blocks (size 64kb or whatever)

Peter