Re: [Qemu-devel] storing machine data in qcow images?

[Michal Suchánek]

On Wed, 6 Jun 2018 13:52:35 +0200
Max Reitz <address@hidden> wrote:

> On 2018-06-06 13:43, Michal Suchánek wrote:
> > On Wed, 6 Jun 2018 13:32:47 +0200
> > Max Reitz <address@hidden> wrote:
> >   
> >> On 2018-06-06 13:19, Michal Suchánek wrote:  
> >>> On Wed, 6 Jun 2018 13:02:53 +0200
> >>> Max Reitz <address@hidden> wrote:
> >>>     
> >>>> On 2018-06-06 12:32, Michal Suchánek wrote:    
> >>>>> On Tue, 29 May 2018 12:14:15 +0200
> >>>>> Max Reitz <address@hidden> wrote:  
> 
> [...]
> 
> >>>>>> I hate the idea of making qcow2 a random archive format.      
> >>>>>
> >>>>> What's wrong with that?      
> >>>>
> >>>> The fact that qcow2 isn't.
> >>>>
> >>>> From my perspective it would increase the format's complexity to
> >>>> a point where you could just create a new format altogether.
> >>>> Well, actually, all you do is design a filesystem (or reuse an
> >>>> existing one).
> >>>>    
> >>>>>> We have tar for that.      
> >>>>>
> >>>>> It does not support expanding the stored files.      
> >>>>
> >>>> Nor does qcow2, because it does not support storing files at
> >>>> all.    
> >>>
> >>> AFAICT from the previous discussion it already does allow storing
> >>> multiple data streams that can be changed independently so it
> >>> basically is an archive format or filesystem except the streams
> >>> are not named nor easily accessible separately outside of
> >>> qemu.    
> >>
> >> I don't quite understand what you are referring to.  We have
> >> snapshots, we have bitmaps, yes, but all of that are related
> >> directly to the stored guest disk data.
> >>
> >> The only thing we currently have in qcow2 that is opaque is the VM
> >> state that can be stored in snapshots (and don't hold me
> >> responsible for that).  
> > 
> > But it has to be related to the stored disk data only because of
> > your dislike for stuff not related to the disk data. Not for a
> > technical reason. The format can sustain storing unrelated data.  
> 
> Yes.  Technically, you can even store data without qcow2 knowing,
> except that qemu-img check -r leaks will delete it, but yeah.
> 
> >>>> Secondly, that completely depends on how you use it.  You can
> >>>> freely expand the last file in the archive, for instance.  Also
> >>>> I've seen people store files in chunks so they can indeed resize
> >>>> it.
> >>>>
> >>>> (I'm wondering if we could write a block driver that could
> >>>> provide such a chunk allocation transparently to qcow2...  Note
> >>>> that a qcow2 file does not need to be continuous, so you could
> >>>> in theory indeed store the qcow2 file and its data in completely
> >>>> separate places in a tar file.)    
> >>>
> >>> Which basically invents another new filesystem on top of tar for
> >>> no good reason. Especially when we have already support for
> >>> storage format that is capable enough.    
> >>
> >> No different from inventing a filesystem on top of qcow2.
> >>
> >> I don't think qcow2 is any more capable than tar.  
> > 
> > It can natively resize the objects it stores.  
> 
> It does not store arbitrary objects.  It stores a guest disk (and
> snapshots), and bitmaps.  These are stored in directory structures so
> new chunks can be dynamically added.  Note that both use slightly
> different structures.
> 
> Yes, it is technically possible to use a similar structure to store
> arbitrary objects.  But currently it does not have that capability.
> 
> >                                                tar cannot do that
> > so you will have to store random nonsense in the tar archive that
> > only makes sense for your fs/tar driver.  
> 
> Right.  But what's the difference to storing random nonsense in a
> qcow2 file that not even makes sense to qemu?
> 
> >>>> What I'm trying to get at is that qcow2 was not designed to be a
> >>>> container format for arbitrary files.  If you want to make it
> >>>> such, I'm sure there are existing formats that work better.    
> >>>
> >>> Such as?    
> >>
> >> ext2?  
> > 
> > So you want an ext2 driver in qemu instead of expanding qcow2 to
> > work not only for a single disk but also for an appliance?  
> 
> Yes, because ext2 was designed to be a proper filesystem.  I'm not an
> FS designer.  Well, not a good one anyway.  So I don't trust myself on
> extending qcow2 to be a good FS -- and why would I, when there are
> already numerous FS around.

Do you expect that performance of qemu using qcow2 driver over ext2
driver will be better than using qcow driver directly with some part
semi-permanently occupied by a configuration blob? My bet is not.

The ext* drivers are designed to work with kernel VM infrastructure
which must be tuned for different usage scenarios and you would have to
duplicate that tuning in qemu to get competitive performance. Also you
get qcow2 and ext2 metadata which must be allocated, managed, etc. You
get more storage and performance overhead for no good reason.

On the other hand, qcow is designed for storing VM disk data and
hopefully was tuned to do that decently over the years. The primary use
case remains storing VM disk data. Adding a configuration blob does not
change that.


> >>>>>> Unless I have got something terribly wrong (which is indeed a
> >>>>>> possibility!), to me this proposal means basically to turn
> >>>>>> qcow2 into (1) a VM description format for qemu, and (2) to
> >>>>>> turn it into an archive format on the way.      
> >>>>>
> >>>>> And if you go all the way you can store multiple disks along
> >>>>> with the VM definition so you can have the whole appliance in
> >>>>> one file. It conveniently solves the problem of synchronizing
> >>>>> snapshots across multiple disk images and the question where to
> >>>>> store the machine state if you want to suspend it.       
> >>>>
> >>>> Yeah, but why make qcow2 that format?  That's what I completely
> >>>> fail to understand.
> >>>>
> >>>> If you want to have a single VM description file that contains
> >>>> the VM configuration and some qcow2/raw/whatever files along
> >>>> with it for the guest disk data, sure, go ahead.  But why does
> >>>> the format of the whole thing need to be qcow2?    
> >>>
> >>> Because then qemu can access the disk data from the image directly
> >>> without any need for extraction, copying to different file,
> >>> etc.    
> >>
> >> This does not explain why it needs to be qcow2.  There is
> >> absolutely no reason why you couldn't use qcow2 files in-place
> >> inside of another file.  
> > 
> > qemu cannot read the disk data from the file in-place.  
> 
> Hu?  Why not?

Well, it can possibly read the image if it happens to be continuous. It
will not be able to update it without a fs driver, however.

Thanks

Michal

pgpfO7N7yWi9r.pgp
Description: OpenPGP digital signature