Re: [Qemu-discuss] disk image: self-organized format or raw file

[Jakob Bohm]

On 8/12/2014 1:51 AM, 吴兴博 wrote:
> Hello,
>
>    The introduction in the wiki page present several advantages of qcow2
> [1]. But I'm a little confused. I really appreciate if any one can give
> me some help on this :).
>
>   (1) Currently the raw format doesn't support COW. In other words, a
> raw image cannot have a backing file. COW depends on the mapping table
> on which we it knows whether each block/cluster is present (has been
> modified) in the current image file. Modern file-systems like
> xfs/ext4/etc. provide extent/block allocation information to user-level.
> Like what 'filefrag' does with ioctl 'FIBMAP' and 'FIEMAP'. I guess the
> raw file driver (maybe block/raw-posix.c) may obtain correct 'present
> information about blocks. However this information may be limited to be
> aligned with file allocation unit size. Maybe it's just because a raw
> file has no space to store the "backing file name"? I don't think this
> could hinder the useful feature.
The ability to store a backing file name or other per-image metadata
(such as emulated disk geometry) is not that critical, as this could
simply be provided in a config file or command line option.  In fact
there are scenarios where the storage of this data inside the diskimage
files is actually a problem.

However the present/not-present feature of "sparse" files (on almost any
file system) is subject to the rule that both the file system and basic
file utilities (/bin/cp, /bin/tar etc. etc.) are allowed to arbitrarily
convert byte ranges back and forth between all-zeroes and "bytes
absent".  This in term means that there will be no reliable way for qemu
etc. to distinguish between a hole that should be mapped to the content
of the next cow level down and byte ranges where non-zero bytes in the
next cow level down have been overridden by bytes with the value zero.

Thus to reliably maintain a COW overlay which lists some byte ranges
as being "whatever the next level closer to the base file says" and
others as being "some value (maybe zero) stored in the cow overlay".

All the various COW file formats supported by qemu have been designed
as different solutions to this requirement.  qcow2 is the "native"
qemu format and as such is supposed to always have the most complete
feature set and the most optimised and tested implementation details.

>   (2) As most popular filesystems support delay-allocation/on-demand
> allocation/holes, whatever, a raw image is also thin provisioned as
> other formats. It doesn't consume much disk space by storing useless
> zeros. However, I don't know if there is any concern on whether
> fragmented extents would become a burden of the host filesystem.

A much more significant issue is the lack of a standard POSIX API for
changing a byte range in a file from "non-hole allocated space" to
"sparse hole with no allocated disk space".  This makes it hard to
release disk space previously consumed by the guest filesystem.  When
a COW file format is used, that key opearation is now an internal
detail independent of the underlying host file system, host kernel
version etc.

>   (3) For compression and encryption, I'm not an export on these topics
> at all but I think these features may not be vital to a image format as
> both guest/host's filesystem can also provide similar functionality.

I agree with you there.

>   (4) I don't have too much understanding on how snapshot works but I
> think theoretically it would be using the techniques no more than that
> used in COW and backing file.

A full featured snapshot (as supported by qcow2 and several non-qemu VM
systems) consists of two things representing a single point in time: A
split between a (possible itself COW) base image representing the
virtual disk content at time T and a COW overlay representing any later
changes to the virtual disk content.  AND a stored copy of the virtual
physical memory and virtual hardware device states at time T.  Together
these two items allow a virtual machine to be "instantaneously"
reverted to whatever it was at time T and continue running from that
point as if nothing had happened.

> After all these thoughts, I still found no reason to not using a 'raw'
> file image (engineering efforts in Qemu should not count as we don't ask
>   for more features from outside world).
> I would be very sorry if my ignorance wasted your time.
>
>
>
> references:
> [1] http://en.wikibooks.org/wiki/QEMU/Images#Image_types
> "QEMU supports several image types. The "native" and most flexible type
> is /qcow2/, which supports copy on write
> <http://en.wikibooks.org/wiki/QEMU/Images#Copy_on_write>, encryption,
> compression, and VM snapshots.
>
>
>
> Cheers!
>         吴兴博  Wu, Xingbo <address@hidden <mailto:address@hidden>>


Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S.  http://www.wisemo.com
Transformervej 29, 2730 Herlev, Denmark.  Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded