Re: [Qemu-discuss] Converting qcow2 image on the fly to raw format

[Richard W.M. Jones]

On Mon, Jul 09, 2018 at 07:38:05PM +0300, Nir Soffer wrote:
> We are discussing importing VM images to KubVirt. The goal is to be
> able to import an existing qcow2 disk, probably some appliance stored
> on http server, and and convert it to raw format for writing to storage.
> 
> This can be also useful for for oVirt for importing OVA, since we like to
> pack
> disks in qcow2 format inside OVA, but the user may like to use raw disks, or
> for uploading existing disks.
> 
> Of course converting the image using qemu-img is easy, but requires
> downloading the image to temporary disk. We would like to avoid temporary
> disks, or telling users to convert the image.
> 
> Base on the discussion we had here:
> https://lists.ovirt.org/archives/list/address@hidden/thread/GNAVJ253FP65QUSOONES5XZGRIDX5ABC/#YMLSEGU7PN3MX5MUORGEGGAQLLSL4KKJ
> 
> I think this is impossible since qcow2 is not built for streaming. But both
> Richard and Eric suggested some solutions.
> 
> The flow is:
> 
>     qcow2 image -- http --> importer -> raw file
> 
> Is it possible to implement the importer using qemu-img and qemu-nbd,
> or maybe nbdkit?

Strictly speaking streaming qcow2 to raw is not possible.  However
placing an overlay on top of the original remote image will allow
streaming to raw with only a modest amount of local storage consumed.

You can demonstrate this fairly easily:

$ qemu-img create -f qcow2 -b 'json: { "file.driver": "https", "file.url": 
"https://uk-mirrors.evowise.com/fedora/releases/28/Cloud/x86_64/images/Fedora-Cloud-Base-28-1.1.x86_64.qcow2";,
 "file.timeout": 10000 }' /var/tmp/overlay.qcow2
$ qemu-img convert -p -f qcow2 -O raw overlay.qcow2 fedora.img

There are various tricks you can play with caching and copy-on-read to
make the process more efficient (note that copy-on-read will result in
a full local copy, but there are maybe some optimizations we could do
in qcow2 for that).

For the other cases such as xz-compressed remote images, I'd recommend
using nbdkit (and/or a plugin, but maybe you can do it with the
default plugins and filters).  An actual example is not going to fit
into this email and I'm short of time to write it, but for some ideas
how to do this you can look at my blog and the nbdkit docs:

https://rwmj.wordpress.com/2013/06/24/xz-plugin-for-nbdkit/
https://github.com/libguestfs/nbdkit

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org