Re: [Qemu-devel] Re: Strategic decision: COW format

[Anthony Liguori]

On 02/18/2011 03:57 AM, Kevin Wolf wrote:
> Am 18.02.2011 10:12, schrieb Markus Armbruster:
>    
> > Kevin Wolf<address@hidden>  writes:
> >
> >      
> > > Am 15.02.2011 20:45, schrieb Chunqiang Tang:
> > >        
> > > > > Chunqiang Tang/Watson/IBM wrote on 01/28/2011 05:13:27 PM:
> > > > > As you requested, I set up a wiki page for FVD at
> > > > >            
> > > > http://wiki.qemu.org/Features/FVD
> > > >          
> > > > > . It includes a summary of FVD, a detailed specification of FVD, and a
> > > > > comparison of the design and performance of FVD and QED.
> > > > >            
> > > >          
> > > > > See the figure at http://wiki.qemu.org/Features/FVD/Compare . This
> > > > >            
> > > > figure
> > > >          
> > > > > shows that the file creation throughput of NetApp's PostMark benchmark
> > > > >            
> > > > under
> > > >          
> > > > > FVD is 74.9% to 215% higher than that under QED.
> > > > >            
> > > > Hi Anthony,
> > > >
> > > > Please let me know if more information is needed. I would appreciate your
> > > > feedback and advice on the best way to proceed with FVD.
> > > >          
> > > Yet another file format with yet another implementation is definitely
> > > not what we need. We should probably take some of the ideas in FVD and
> > > consider them for qcow3.
> > >        
> > Got an assumption there: that the one COW format we need must be qcow3,
> > i.e. an evolution of qcow2.  Needs to be justified.  If that discussion
> > has happened on the list already, I missed it.  If not, it's overdue,
> > and then we better start it right away.
> >      
> Right. I probably wasn't very clear about what I mean with qcow3 either,
> so let me try to summarize my reasoning.
>
>
> The first point is an assumption that you made, too: That we want to
> have only one format. I hope it's easy to agree on this, duplication is
> bad and every additional format creates new maintenance burden,
> especially if we're taking it serious. Until now, there were exactly two
> formats for which we managed to do this, raw and qcow2. raw is more or
> less for free, so with the introduction of another format, we basically
> double the supported block driver code overnight (while not doubling the
> number of developers).
>    

Not sure what project you're following, but we've had an awful lot of 
formats before qcow2 :-)

And qcow2 was never all that special, it just was dropped in the code 
base one day.  You've put a lot of work into qcow2, but there are other 
folks that are contributing additional formats and that means more 
developers.

> The consequence of having only one file format is that it must be able
> to obsolete the existing ones, most notably qcow2. We can only neglect
> qcow1 today because we can tell users to use qcow2. It supports
> everything that qcow1 supports and more. We couldn't have done this if
> qcow2 lacked features compared to qcow1.
>
> So the one really essential requirement that I see is that we provide a
> way forward for _all_ users by maintaining all of qcow2's features. This
> is the only way of getting people to not stay with qcow2.
>
>
> Of course, you could invent another format that implements the same
> features, but I think just carefully extending qcow2 has some real
> advantages.
>
> The first is that conversion of existing images would be really easy.
> Basically increment the version number in the header file and you're
> done. Structures would be compatible.

qemu-img convert is a reasonable path for conversion.

>   If you compare it to file systems,
> I rarely ever change the file system on a non-empty partition. Even if I
> wanted, it's usually just too painful. Except when I was able to use
> "tune2fs -j" to make ext3 out of ext2, that was really easy. We can
> provide the same for qcow2 to qcow3 conversion, but not with a
> completely new format.
>
> Also, while obsoleting a file format means that we need not put much
> effort in its maintenance, we still need to keep the code around for
> reading old images. With an extension of qcow2, it would be the same
> code that is used for both versions.
>
> Third, qcow2 already exists, is used in practice and we have put quite
> some effort into QA. At least initially confidence would be higher than
> in a completely new, yet untested format. Remember that with qcow3 I'm
> not talking about rewriting everything, it's a careful evolution, mostly
> with optional additions here and there.
>    

My requirements for a new format are as followed:

1) documented, thought-out specification that is covered under and open 
license with a clear process for extension.

2) ability to add both compatible and incompatible features in a 
graceful way

3) ability to achieve performance that's close to raw.  I want our new 
format to be able to be used universally both for servers and desktops.

I think qcow2 has some misfeatures like compression and internal 
snapshots.  I think preserving those misfeatures is a mistake because I 
don't think we can satisfy the above while trying to preserve those 
features.  If the image format degrades when those features are enabled, 
then it decreases confidence in the format.

I think QED satisfies all of these today.

Regards,

Anthony Liguori

> Kevin
>
>