Re: [Qemu-devel] [RFC] qed: Add QEMU Enhanced Disk format

[Stefan Hajnoczi]

On Fri, Sep 10, 2010 at 12:14 PM, Avi Kivity <address@hidden> wrote:
>  On 09/09/2010 03:49 PM, Anthony Liguori wrote:
>>
>> On 09/09/2010 01:45 AM, Avi Kivity wrote:
>>>
>>> Loading very large L2 tables on demand will result in very long
>>> latencies.  Increasing cluster size will result in very long first write
>>> latencies.  Adding an extra level results in an extra random write every
>>> 4TB.
>>
>> It would be trivially easy to add another level of tables as a feature bit
>> so let's delay the decision.
>
> It means that you'll need to upgrade qemu to read certain images, but okay.
>
>>>>
>>>> qed is very careful about ensuring that we don't need to do syncs and we
>>>> don't get corruption because of data loss.  I don't necessarily buy your
>>>> checksumming argument.
>>>
>>> The requirement for checksumming comes from a different place.  For
>>> decades we've enjoyed very low undetected bit error rates.  However the
>>> actual amount of data is increasing to the point that it makes an
>>> undetectable bit error likely, just by throwing a huge amount of bits at
>>> storage.  Write ordering doesn't address this issue.
>>
>> I don't think we should optimize an image format for cheap disks and an
>> old file system.
>>
>> We should optimize for the future.  That means a btrfs file system
>
> I wouldn't use an image format at all with btrfs.
>
>> and/or enterprise storage.
>
> That doesn't eliminate undiscovered errors (they can still come from the
> transport).

Eliminating silent data corruption is currently not a goal for any
disk image format I know of.  For filesystems, I know that ZFS and
btrfs will try to detect corruption using data checksumming.

The guest filesystem, the disk image format, or the host filesystem
could do checksumming.  The hypervisor should keep out of the way in
the interest of performance and emulation fidelity.  Why does
checksumming need to be done in the image format?  Isn't the choice
between host and guest filesystem checksumming already enough?

>>
>> The point of an image format is not to recreate btrfs in software.  It's
>> to provide a mechanism to allow users to move images around reasonable but
>> once an image is present on a reasonable filesystem, we should more or less
>> get the heck out of the way.
>
> You can achieve exactly the same thing with qcow2.  Yes, it's more work, but
> it's also less disruptive to users.
>
>>>
>>>> By creating two code paths within qcow2.
>>>
>>> You're creating two code paths for users.
>>
>> No, I'm creating a single path: QED.
>>
>> There are already two code paths: raw and qcow2.  qcow2 has had such a bad
>> history that for a lot of users, it's not even a choice.
>
> qcow2 exists, people use it, and by the time qed is offered on distros (even
> more on enterprise distros), there will be a lot more qcow2 images.  Not
> everyone runs qemu.git HEAD.
>
> What will you tell those people?  Upgrade your image?  They may still want
> to share it with older installations.  What if they use features not present
> in qed?  Bad luck?
>
> qcow2 is going to live forever no matter what we do.

It should be possible to do (live) upgrades for supported images.

>
>>
>> Today, users have to choose between performance and reliability or
>> features.  QED offers an opportunity to be able to tell users to just always
>> use QED as an image format and forget about raw/qcow2/everything else.
>
> raw will always be needed for direct volume access and shared storage.
>  qcow2 will always be needed for old images.
>
>>
>> You can say, let's just make qcow2 better, but we've been trying that for
>> years and we have an existence proof that we can do it in a straight forward
>> fashion with QED.
>
> When you don't use the extra qcow2 features, it has the same performance
> characteristics as qed.  You need to batch allocation and freeing, but
> that's fairly straightforward.
>
> Yes, qcow2 has a long and tortured history and qed is perfect.  Starting
> from scratch is always easier and more fun.  Except for the users.
>
>> A new format doesn't introduce much additional complexity.  We provide
>> image conversion tool and we can almost certainly provide an in-place
>> conversion tool that makes the process very fast.
>
> It introduces a lot of complexity for the users who aren't qed experts.
>  They need to make a decision.  What's the impact of the change?  Are the
> features that we lose important to us?  Do we know what they are?  Is there
> any risk?  Can we make the change online or do we have to schedule downtime?
>  Do all our hosts support qed?
>
> Improving qcow2 will be very complicated for Kevin who already looks older
> beyond his years [1] but very simple for users.
>
>>>
>>> It requires users to make a decision.  By the time qed is ready for mass
>>> deployment, 1-2 years will have passed.  How many qcow2 images will be in
>>> the wild then?  How much scheduled downtime will be needed?
>>
>> Zero if we're smart.  You can do QED stream + live migration to do a live
>> conversion from raw to QED.
>>
>
> Not all installations use live migration (say, desktop users).
>
>>>  How much user confusion will be caused?
>>
>> User confusion is reduced if we can make strong, clear statements: all
>> users should use QED even if they care about performance.  Today, there's
>> mass confusion because of the poor state of qcow2.
>
> If we improve qcow2 and make the same strong, clear statement we'll have the
> same results.
>
>>
>>> Virtualization is about compatibility.  In-guest compatibility first, but
>>> keeping the external environment stable is also important.  We really need
>>> to exhaust the possibilities with qcow2 before giving up on it.
>>
>> IMHO, we're long past exhausting the possibilities with qcow2.  We still
>> haven't decided what we're going to do for 0.13.0.
>
> Sorry, I disagree 100%.  How can you say that, when no one has yet tried,
> for example, batching allocations and frees?  Or properly threaded it?
>
> What we've done is make qcow2 safe and a more parallel than it was.  But
> "exhaust all possibilities"? not even close.
>
>
>> Are we going to ship qcow2 with awful performance (a 15 minute operation
>> taking hours) or with compromised data integrity?
>
> We're going to fix it.
>
>>
>> It's been this way for every release since qcow2 existed.  Let's not let
>> sunk cost cloud our judgement here.
>
> Yes, new and shiny is always better.
>
>>
>> qcow2 is not a properly designed image format.  It was a weekend hacking
>> session from Fabrice that he dropped in the code base and never really
>> finished doing what he originally intended.  The improvements that have been
>> made to it are almost at the heroic level but we're only hurting our users
>> by not moving on to something better.
>>
>
> I don't like qcow2 either.  But from a performance perspective, it can be
> made equivalent to qed with some effort.  It is worthwhile to expend that
> effort rather than push the burden to users.
>
>> Regards,
>>
>> Anthony Liguori
>>
>>
>
> [1] okay, maybe not.
>
> --
> I have a truly marvellous patch that fixes the bug which this
> signature is too narrow to contain.
>
>
>