Re: [Qemu-devel] [PATCH v3 4/5] migration: Append JSON description of migration stream

[Alexander Graf]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



On 06.01.15 16:56, Eric Blake wrote:
> On 12/26/2014 07:42 AM, Alexander Graf wrote:
>> One of the annoyances of the current migration format is the fact
>> that it's not self-describing. In fact, it's not properly
>> describing at all. Some code randomly scattered throughout QEMU
>> elaborates roughly how to read and write a stream of bytes.
>> 
>> We discussed an idea during KVM Forum 2013 to add a JSON
>> description of the migration protocol itself to the migration
>> stream. This patch adds a section after the VM_END migration end
>> marker that contains description data on what the device sections
>> of the stream are composed of.
>> 
>> With an additional external program this allows us to decipher
>> the contents of any migration stream and hopefully make migration
>> bugs easier to track down.
> 
> Hmm.  The new format IS self-describing, but ONLY because JSON is 
> technically possible to parse in reverse.  That is, you cannot
> reliably look for end-of-stream marker from the beginning of the
> stream and reliably get to the JSON description at the end every
> single time (because if you don't already know how to interpret the
> stream, how can you find end-of-stream), but you CAN reliably look
> to see if the stream ends in a JSON object, and then scan backwards
> for the matching { that opens the object.
> 
> I'm still wondering if you ought to throw in any additional tricks
> to make finding the start of the JSON object much easier, such as
> a subsection near the beginning of the migration stream, and/or a
> final offset description at the tail of the file that says where
> the JSON object starts (no additional help for a pipeline, which
> still has to store things in memory to loc.  Even doing something
> like:
> 
> stream EOS [ { description... }, offset-of-description ]
> 
> and using a JSON array rather than a JSON object as the
> description would make it much faster to find the start of the JSON
> object without having to scan backwards for matching { (although
> sticking the offset at the tail doesn't help the issue that when
> receiving the migration stream over a pipeline, you still have to
> reserve memory for the entire stream in order to find that
> offset).

Yeah, I've pondered ideas similar to these around myself. The problem
is that I don't think they're much more reliable than what we have
now. You know that our JSON data is always ASCII, so if you simply
scan backwards for the EOF marker, you've basically solved the problem
if you know the end of the file.

If you don't know the end of the file (read: pipe or stream), then
there's really only so much we can do.

The problem with writing a section anywhere before the device sections
which we can't parse otherwise is that at that point in time we don't
know the location of our JSON stream, as the migration stream hasn't
been written yet.

So in a nutshell, we're safe with seekable files I think. If you can
come up with a solution that works nicely for pipes, please let me
know :).

> 
>> 
>> Signed-off-by: Alexander Graf <address@hidden>
>> 
>> ---
>> 
>> v2 -> v3:
>> 
>> - Dont compress objects with subsections - Destroy QJSON object 
>> --- hw/pci/pci.c                  |   2 +- hw/scsi/spapr_vscsi.c
>> |   2 +- hw/virtio/virtio.c            |   2 +- 
>> include/migration/migration.h |   1 + include/migration/vmstate.h
>> |   3 +- migration/vmstate.c           | 186
>> ++++++++++++++++++++++++++++++++++++++++-- savevm.c
>> |  54 ++++++++++-- tests/test-vmstate.c          |   6 +- 8 files
>> changed, 237 insertions(+), 19 deletions(-)
>> 
> 
> At this point, I haven't actually reviewed the patch (I'm more 
> interested in seeing the JSON it generates), but want to make sure
> we're getting an optimal design.

Sure, the python tool in 5/5 can dump the JSON as well :). Just run it
on any guest's migration file you like.


Alex
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
Comment: GPGTools - http://gpgtools.org

iQIcBAEBAgAGBQJUrFKwAAoJECszeR4D/txgiyEP/3Es3GwAivBoHKgkqnW1TVuf
9nk1+yPHEP7jwQc3z8hpbLZR9lk1H0Pin1rHW67ICvv1+iD/0ShavYml+/kCumtz
/lroJGjZ5nb80mdTsnvWIXqra3s6SYF3F5wxuCHb8QInbu2kEi5rx1MT+np2cIGh
BtfuoLf2dGhb4Rkfmtxsjn8Bo8MFfoyXjNZt7ugFqvhXNP05y4jQg1FmkFnOT9l4
KSsvsI6L3Q7K/jwmjnMS+GghMcUrL2KiKYreSHpruYYMuiKrW6d2NzJYv5OoisyO
gIfw/ZaLlK/uuD8sTuJ7NlU1CdDi+OtOSbrtNz2frA+P5bybfFrrBNfODisB1Q/g
zH9BTo3b5JgecHfkzGMsYa16e+kbK6LPCnw0bpc5EmELaAM3f/Za0rKksmwhcJZV
lzTLrUWMCEZqoZlRrPvuJ5ntBryWmZgyMHNflkaX4l8FsFnYjevW5QNHKYPeCpOV
wam7GiWw2gxz7XBVfRpESqimi9c4slthIrfP/schJrtRtxUQyaHUsEa8s8O/FZEU
utCuy7jAKaiShYgTyiplQ1Gx8UFgpIlET5ToHUQWnau6adVJNOFqnAhoM+Rr5wWV
LRgCyepS2RP/vNyTY091M50fo1N73lFXLuL4EkimSq8CxwdvVs+7vgmeIfZnno30
VRr6Fz6+PnB/AWkK+FS+
=mViD
-----END PGP SIGNATURE-----