Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Dr. David Alan Gilbert]

* Max Reitz (address@hidden) wrote:
> On 2018-06-06 14:16, Dr. David Alan Gilbert wrote:
> > * Max Reitz (address@hidden) wrote:
> >> On 2018-06-06 13:37, Dr. David Alan Gilbert wrote:
> >>> * Max Reitz (address@hidden) wrote:
> >>>> On 2018-06-06 13:19, Michal Suchánek wrote:
> >>>>> On Wed, 6 Jun 2018 13:02:53 +0200
> >>>>> Max Reitz <address@hidden> wrote:
> >>>>>
> >>>>>> On 2018-06-06 12:32, Michal Suchánek wrote:
> >>>>>>> On Tue, 29 May 2018 12:14:15 +0200
> >>>>>>> Max Reitz <address@hidden> wrote:
> >>
> >> [...]
> >>
> >>>>>>>> Unless I have got something terribly wrong (which is indeed a
> >>>>>>>> possibility!), to me this proposal means basically to turn qcow2
> >>>>>>>> into (1) a VM description format for qemu, and (2) to turn it into
> >>>>>>>> an archive format on the way.  
> >>>>>>>
> >>>>>>> And if you go all the way you can store multiple disks along with
> >>>>>>> the VM definition so you can have the whole appliance in one file.
> >>>>>>> It conveniently solves the problem of synchronizing snapshots across
> >>>>>>> multiple disk images and the question where to store the machine
> >>>>>>> state if you want to suspend it.   
> >>>>>>
> >>>>>> Yeah, but why make qcow2 that format?  That's what I completely fail
> >>>>>> to understand.
> >>>>>>
> >>>>>> If you want to have a single VM description file that contains the VM
> >>>>>> configuration and some qcow2/raw/whatever files along with it for the
> >>>>>> guest disk data, sure, go ahead.  But why does the format of the whole
> >>>>>> thing need to be qcow2?
> >>>>>
> >>>>> Because then qemu can access the disk data from the image directly
> >>>>> without any need for extraction, copying to different file, etc.
> >>>>
> >>>> This does not explain why it needs to be qcow2.  There is absolutely no
> >>>> reason why you couldn't use qcow2 files in-place inside of another file.
> >>>
> >>> Because then we'd have to change the whole stack to take advantage of
> >>> that.  Adding a feature into qcow2 means nothing else changes.
> >>
> >> Because it's a hack, right.  Storing binary data in a qcow2 file,
> >> completely ignoring it in qemu (and being completely unusable to any
> >> potential other users of the qcow2 format[1]) and only interpreting it
> >> somewhere up the stack is a hack.
> > 
> > It's not a hack!
> > Seriously it's not.
> > There's nothing wrong with it being aimed higher up the stack than qemu,
> 
> Not really, but storing that information in a disk image file is, from
> my perspective.  So far, qcow2 was always just for qemu.  (Hmm...  Maybe
> backing links weren't, but at least they were intended for qemu originally.)
> 
> So this would mix information for different layers inside qcow2 which to
> me sounds weird.  Maybe I just have to get used to it.

The important point is it's explicitly for a different layer; we're not
mixing it - the guest can never get to this information.  It also saves
the higher level management layers ever having to look at the data the
guest can get to, which is a security advantage.

From my point of view, it really is the sticky label on the disc rather
than the contents of it.

> > the problem we started off with was what happens when a user downloads
> > a VM image and tries to import it into their VM system;
> 
> Well, the VM system should choke without a config file. O:-)
> 
> >                                                         weve already
> > got 2+ layers of management stuff in there - I want the information to
> > guide those layers, not form a complete set of configuration.
> 
> But I do.
> 
> If we store some information, I don't see why we don't store all of it.

Hmm, now that generally I don't like:
  a) That really would make it hypervisor specific
  b) It would be a lot of data
  c) Generally, the supplier of an image doesn't know how the end-user
     wants it configured - although for some appliances they might.
  d) Remember the only problem we had when we got here was how to stop
     the user shooting themselves in the foot by connecting the wrong
     image to the wrong VM type.  So I'm expecting to use this to
     contain requirements, nothing more.


> >> That is not necessarily a negative point, hacks can work wonderfully
> >> well, and they usually are simple, that is correct.  But the thing is
> >> that I feel like people have grand visions of what to get out of this.
> >> Imagine, a single file that can configure all and any VM!
> >>
> >> But hacks usually only solve a single issue.  Once you try to extend a
> >> hack, it breaks down and becomes insufficient.
> >>
> >> If we want a grand vision where a single file stores the whole VM, why
> >> not invest the work and make it right from the start?
> > 
> > Because we won't get it right; however much we bikeshed about it
> > we'll just end up with a mess.
> 
> Sure, but the same thing applies to putting it into qcow2.  The
> difference is, for something outside of qcow2, throwing it away and
> starting over is simple.
> 
> When putting it into qcow2, we can only do that if we really just put a
> binary blob there that isn't described in the specification.

Well, it's why I'm going for defined key/values that are stored in the
blob and only a few of them.  We've got a reasonable chance of being
able to define what we want from 3-4 key/values, it should be a lot
easier than trying to define a grand scheme.

> >                                  The right thing is to put in something
> > to hold configuration and then review the items of configuration we
> > add properly as we define them.
> 
> OK, but review them on what terms?  Whether they are simple enough?

Well, I'll take simple, make sense - whatever - feel free to be the
maintainer for that list!

> As I said, I would want a whole configuration if we allow some
> configuration.

Well then you could go for libvirt XML as the contents; but I think
that's crossing layers even more.
(I would have veered more to it being exactly the same as an OVA
description except for rjones dislike of it)

> (More below)
> 
> >> [1] Yes, I concede that there are probably no other users of qcow2.  But
> >> please forgive me for assuming that qcow2 was in a sense designed to be
> >> a rather general image format that not only qemu could use.
> > 
> > What makes it QEMU specific?  It's basically just the same key/value
> > setup as OVA, except putting them inside the qcow2.
> 
> Well, not necessarily qemu-specific, but
> ${management_software}-specific, which comes down to the same.  Or,
> well, I'd think that, but hold on.
> 
> > We could use the same keys/value definitions as OVA in the blob,
> > although their definitions aren't very portable either.
> 
> So you're proposing that we only add options that seem portable for any VM?

Hmm.  We should probably split them, so there should be general options
(e.g. minimum-ram) but also hypervisor specifics
(qemu.machine-class=q35); but that doesn't mean you can't add keys
for multiple hypervisors into the one blob.  I mean
something like:
    minimum-ram = 1G
    qemu.machine-class = q35
    anothervm.machine-class = ....

Dave

> Max
> 



--
Dr. David Alan Gilbert / address@hidden / Manchester, UK