[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Exporting qcow2 images as raw data from ova file with qemu-nbd

From: Eric Blake
Subject: Re: Exporting qcow2 images as raw data from ova file with qemu-nbd
Date: Tue, 23 Jun 2020 08:47:52 -0500
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.9.0

On 6/22/20 5:21 PM, Nir Soffer wrote:
I'm trying to export qcow2 images from ova format using qemu-nbd.

I create 2 compressed qcow2 images, with different data:

$ qemu-img info disk1.qcow2
image: disk1.qcow2
file format: qcow2
virtual size: 200 MiB (209715200 bytes)
disk size: 384 KiB

$ qemu-img info disk2.qcow2
image: disk2.qcow2
file format: qcow2
virtual size: 200 MiB (209715200 bytes)
disk size: 384 KiB

And packed them in a tar file. This is not a valid ova but good enough
for this test:

$ tar tvf vm.ova
-rw-r--r-- nsoffer/nsoffer 454144 2020-06-22 21:34 disk1.qcow2
-rw-r--r-- nsoffer/nsoffer 454144 2020-06-22 21:34 disk2.qcow2

Thanks for the reproduction steps, this helps me reproduce locally.

To get info about the disks in ova file, we can use:

$ python -c 'import tarfile; print(list({"name": m.name, "offset":
m.offset_data, "size": m.size} for m in tarfile.open("vm.ova")))'
[{'name': 'disk1.qcow2', 'offset': 512, 'size': 454144}, {'name':
'disk2.qcow2', 'offset': 455168, 'size': 454144}]

First I tried the obvious:

$ qemu-nbd --persistent --socket=/tmp/nbd.sock --read-only --offset=512 vm.ova

And it works, but it exposes the qcow2 data. I want to raw data so I
can upload the guest
data to ovirt, where is may be converted to qcow2 format.

So you definitely need two layers of interpretation: one that reads from an offset within the tar file, another that reads guest-visible data from the qcow2 contents at that offset. Both qemu and nbdkit can perform the first layer, but at the moment, nbdkit cannot perform the second (our thoughts along that line have been that if qemu can read qcow2 files, there's no need to duplicate that effort in nbdkit). So we either need to come up with a single qemu-nbd command line that can do it all, or else a process chain where nbdkit serves the offset over a Unix socket, then qemu-nbd consumes that socket to server the guest-visible data of the qcow2 format.

$ qemu-img info --output json "nbd+unix://?socket=/tmp/nbd.sock"
     "virtual-size": 209715200,
     "filename": "nbd+unix://?socket=/tmp/nbd.sock",
     "format": "qcow2",

Looking in qemu manual and qapi/block-core.json, I could construct this command:

$ qemu-nbd --persistent --socket=/tmp/nbd.sock --read-only
'json:{"driver": "qcow2", "file": {"driver": "raw", "offset": 512,
"size": 454144, "file": {"driver": "file", "filename": "vm.ova"}}}'

Yep, that's getting qemu to perform both layers. The pseudo-format json:{...} is a bit of a pain to write but accurate; it is also possible to write the same command using --image-opts:

qemu-nbd --persistent --socket=/tmp/nbd.sock --read-only --image-opts \

And it works:

$ qemu-img info --output json "nbd+unix://?socket=/tmp/nbd.sock"
     "virtual-size": 209715200,
     "filename": "nbd+unix://?socket=/tmp/nbd.sock",
     "format": "raw"

I wonder if this is the best way to stack a qcow2 driver on top of a
raw driver exposing
a range from a tar file.

Yes, that looks best to me. As I said above, the only other option I can think of is to mix qemu-nbd and nbdkit, as in:

nbdkit -r -U - --filter=offset file vm.ova offset=512 range=454144 \
 --run 'qemu-nbd -k /tmp/nbd.sock -r -f qcow2 $uri'

which gives the same contents over /tmp/nbd.sock with a shorter command line but with more processes (and thus more computation spent per I/O) involved.

I found similar example for gluster in:

Richard suggested to try nbdkit tar plugin, but the plugin is not
available on RHEL,
and this adds additional dependency, when we already use qemu-nbd.

Rich just rewrote the tar plugin to use python instead of perl, which means it is that much easier for a future RHEL to pull it in. We still ought to consider having a tar filter, either in place of or in addition to, the tar plugin (similar to how we recently converted nbdkit's ext4 support from a plugin to a filter) - having a tar filter would allow you to read a compressed ova file (by combining the xz and tar filters to decompress then extract a file). But right now, nbdkit doesn't support non-C filters (and given that our tar plugin was written first in perl and now in python, that still means translation to yet another language if the filter requires it to be in C).

Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

reply via email to

[Prev in Thread] Current Thread [Next in Thread]