Re: [Qemu-devel] storing machine data in qcow images?

[Markus Armbruster]

Eduardo Habkost <address@hidden> writes:

> On Wed, May 23, 2018 at 01:19:46PM +0200, Markus Armbruster wrote:
>> Eduardo Habkost <address@hidden> writes:
>> 
>> > On Mon, May 21, 2018 at 07:44:40PM +0100, Daniel P. Berrangé wrote:
>> >> On Mon, May 21, 2018 at 03:29:28PM -0300, Eduardo Habkost wrote:
>> >> > On Sat, May 19, 2018 at 08:05:06AM +0200, Markus Armbruster wrote:
>> >> > > Eduardo Habkost <address@hidden> writes:
>> >> > > 
>> >> > > [...]
>> >> > > > About being more expressive than just a single list of key,value
>> >> > > > pairs, I don't see any evidence of that being necessary for the
>> >> > > > problems we're trying to address.
>> >> > > 
>> >> > > Short history of a configuration format you might have encountered:
>> >> 
>> >> [snip]
>> >> 
>> >> > > How confident are we a single list of (key, value) is really all we're
>> >> > > going to need?
>> >> > > 
>> >> > > Even if we think it is, would it be possible to provide for a future
>> >> > > extension to trees at next to no cost?
>> >> > 
>> >> > I'm confident that a list of key,values is all we need for the
>> >> > current problem.
>> >> 
>> >> I'm not convinced. A disk image may work with Q35 or i440fx,  or
>> >> work with any of virtio, ide or sata disk. So that already means
>> >> values have to be arrays, not scalars. You could do that with a
>> >> simple key,value list, but only by defining a mapping of arrays
>> >> into a flattened form. eg do we allow repeated keys, or do we
>> >> allow array indexes on keys. 
>> >
>> > No problem, we can support trees if it's necessary.
>> >
>> >
>> >> > The point here is to allow users to simply copy an existing disk
>> >> > image, and it will contain enough hints for a cloud stack to
>> >> > choose reasonable defaults for machine-type and disk type
>> >> > automatically.  Requiring the user to perform a separate step to
>> >> > encapsulate the disk image in another file format defeats the
>> >> > whole purpose of the proposal.
>> >> 
>> >> It doesn't have to mean more work for the user - the application
>> >> that is used to create the image can do that on their behalf.
>> >> oVirt for example can import/export OVA files, containing OVF
>> >> metadata. I could imagine virt-manager, and other tools adding
>> >> export ability without much trouble if this was deemed a desirable
>> >> thing. Bundling gives ability to have multiple disk images in one
>> >> archive, which is something OVF does.
>> >
>> > I have the impression that "the application that is used to
>> > create the image" is a very large set.  It can be virt-manager,
>> > virt-install, virt-manager, or even QEMU itself.
>> >
>> > Today people can simply create a VM on virt-manager, or run QEMU
>> > manually, and upload the qcow2 image directly from its original
>> > location (they don't need to copy/export it).  Don't we want the
>> > same procedure to keep working instead of requiring users to use
>> > another tool?
>> 
>> Today, I can take the disk out of my old computer, put it into my new
>> computer, and it just works.  Don't we want the same procedure to keep
>> working forever?
>
> I don't think the comparison is fair: downloading hard disk
> images for bare metal hardware is not as common as downloading
> guest disk images for cloud infrastructure.

Can't let "fair" get in the way of a witticism!

Seriously, though: are disk images a sane way to package software?

>> Sadly, wanting something badly enough doesn't make actual solutions any
>> easier :)
>
> This part is true.  :)
>
>> 
>> My point is: disk images (real or virtual) keep working in different
>> hardware contexts by a mixture of flexibility built into system software
>> on the image, disciplined evolution of hardware (real or virtual), and
>> dumb luck.  It works until it doesn't.  And then you get to tinker.
>
> Personally, I believe that tools for running and managing virtual
> hardware can and should be smarter than real hardware.

Point taken.

Adding hints to disk images (which is how I understand your proposal)
could make them (with a bit of luck) work more often.  Luck, because the
receiving end needs to interpret the hints in a way that makes the image
work.  In other words, guesswork and duct tape.  Both guesswork and duct
tape are incredibly useful when no better tools are at hand.  Is that
the case here?

>> With OVF, you solve the problem further up the stack: you do virtual
>> appliances instead of disk images.
>> 
>
> I guess the main problem is that people are already using disk
> images as if they were virtual appliances.
>
> We can tell people to stop doing that and use OVF, but then we
> won't make anybody's life any easier: publishers of images might
> need to generate both qcow2 and OVF images if they want it to
> work with older hosts; consumers will need to find out if they
> need qcow2 or OVF.

I'm afraid providing for "hints" in QCOW2 could only add problems.  To
pick the right hints, publishers need to predict how future software
consuming the image will interpret them.  Consumers may have to
configure their software to interpret hints in various ways.

I figure my belly-aching is due to the general fuzziness of "hints", at
least in my mind.  Would it even be possible to do anything remotely
similar to a specification for them?

> But I work too deep down the stack to tell if it's really
> important to avoid these problems or not.  And as you said, this
> doesn't make actual solutions any easier.
>
>
>> How much space that leaves for useful solutions at the level of QEMU I
>> can't say.
>
> I have no doubt about "useful", but I'm not sure about
> "important".

I'm in doubt on "feasible", but that might be due to me not fully
grasping the intended use of hints.

> I guess the question is if we have people with time and resources
> to work on solutions (whether using qcow2 or OVF).