[Qemu-devel] Re: libvirt vs. in-qemu management

[Alexander Graf]

Daniel P. Berrange wrote:
> On Tue, Apr 06, 2010 at 02:43:47PM +0200, Alexander Graf wrote:
>   
>> Daniel P. Berrange wrote:
>>     
>>> With appliances there are two core aspects
>>>
>>>  1. The description of VM hardware requirements
>>>  2. The disk format
>>>
>>> Traditionally VMware appliances have shipped a VMX file for 1. and
>>> a VMDK file for 2. 
>>>
>>> Shipping the native config file format with an appliance though is
>>> the wrong thing to be doing. The native config format describes the
>>> configuration for a VM for a specific deployment. This is not the
>>> same as describing the hardware requirements of an appliance. As
>>> the most simple example, a native config would have hardcoded disk
>>> paths, or a specific choice of host network connectivity. Neither
>>> of these things have any business being in the appliance config.
>>>
>>> For this reason, there are now specific appliance formats. Libvirt
>>> has long had its own appliance format (virt-image) which is separate
>>> from the main XML format, so it avoids hardcoding deployment specific
>>> options. There is also the vendor neutral OVF format which is widely 
>>> supported by many mgmt tools. 
>>>
>>> If people want to ship QEMU appliances I don't think libvirt is 
>>> causing any problems here. Simply ship a OVF description + either
>>> raw or qcow2 disk image. Any app, libvirt, or not could work with
>>> that.
>>>   
>>>       
>> Does VMware Player support OVF?
>> Does VMware Workstation support OVF?
>> Does VMware Server support OVF?
>>     
>
> I've no idea if they're added support to those. There's no technical
> reason why not, but being closed source software they may have 
> artificially restricted functionality to force you to get VCenter.
>   

Theoretically everything's possible. From an appliance delivery point of
view, this is an important question. people use those VMMs.

>   
>> Do older VMware ESX versions support OVF?
>>     
>
> I don't know off hand what version it was introduced in
>   

Pretty sure it only got in with more recent versions.

>   
>> Does it make sense to build an OVF with a Xen PV image?
>>     
>
> Yes, in fact it is beneficial. If you shipped a PV ops enabled
> appliance image with a Xen config file, the distributor would
> have to ship several configs one for PV mode, one for HVM, since
> it hardcodes the type of guest & disk drivers. If you ship an 
> OVF file, then the tool deploying the appliance can decide whether
> to deploy it in PV or HVM mode. This same appliance can then even
> work for KVM too.
>   

SLES Xen is PV only. And the non-PV kernel isn't Xen enabled. So it's
rather useless ;-).

>   
>> We need to deliver vendor specific configs anyways. Of course we could
>> ship a VMware type, a Xen type and an OVF type. But that would certainly
>> not help KVM's awareness because it's hidden underneath the OVF type.
>>     
>
> The whole point of the OVF format/project is that it is not explicitly
> targetted at one vendor's technology. This isn't the right place to be
> raising awareness of KVM.
>   

So we're raising awareness for VMware, because they don't always support
OVF. We raise awareness for Xen, because PV only appliances needs to be
built differently. But we don't raise awareness for KVM because we
support OVF? I'm not a PR guy, but that sounds like an odd move.

>   
>>>   
>>>       
>>>> 3) Configuration conversion
>>>>
>>>> Party due to qemu not having a configuration format, partly due to 
>>>> libvirt's ambivalent approach, there is always conversion in 
>>>> configuration formats involved. I think this is the main reason for
>>>> the feature lag. If there wasn't a conversion step, there wouldn't 
>>>> be lag. You could just hand edit the config file and be good.
>>>>     
>>>>         
>>> [snip]
>>>
>>>   
>>>       
>>>> Point 3 is the really tough one. It's the very basis of libvirt. And 
>>>> it's plain wrong IMHO. I hate XML. I hate duplicated efforts. The 
>>>> current conversion involves both. Every option added to qemu needs to 
>>>> be added to libvirt. In XML. Bleks.
>>>>     
>>>>         
>>> In the previous thread on this topic, I've already stated that we're
>>> interested in providing a means to pass QEMU config options from
>>> libvirt prior to their full modelling in the XML, to reduce, or completely 
>>> eliminate any time-lag in using new features.
>>>   
>>>       
>> That would cover new features and would be really good to have
>> nevertheless. It still doesn't cover the difference in configuration for
>> native tags. Imagine you'd want to enable cache=none. I'd know how to do
>> it in qemu, but I'd be lost in the libvirt XML. If I'd be a person
>> knowledgeable in libvirt, I'd know my way around the XML tags but
>> wouldn't know what they'd mean in plain qemu syntax. So I couldn't tell
>> people willing to help me what's going wrong even if I wanted to.
>>     
>
> Going from the XML to QEMU config and vica-verca is not rocket science.
> You can trivially see the QEMU config generated for any libvirt VM in
> the logs. There are also commands for doing the conversion in both
> directions, though I admit the QEMU -> XML conversion is not as complete
> as the XML -> QEMU conversion.
>   

None of that is rocket science. But it's all conversion. And currently
there are things you can't model with that conversion even, like network
fds.

>   
>> If instead there was a common machine description file that everyone
>> knows, there'd be a single point of knowledge. A RHEL-V admin could work
>> on plain qemu. A qemu developer would feel right at home with virt-manager.
>>     
>
> This isn't solving the problem. If you see a problem in the QEMU config
> uses by a high level tool like RHEV/oVirt, you still aren't going to 
> know what the config change you need to make in those apps. They are
> never going to work with the QEMU config as their master data format.
> It is just something they generate on the fly at runtime, from their
> SQL databases, because they want to model concepts at a high level.
> A VM as represented in RHEV/oVirt does not have a single QEMU or libvirt
> config file description - the low level config can potentially vary each
> time the guest is started on a host(s).
>   

So we could still make it transparent to the user, no? RHEV could import
a KVM machine description as well as it could export one. So the
internal representation is transparent to the user. That would also ease
going from RHEV to other management apps. Or the other way around.

>   
>>> Even if an app was using QEMU directly, you can't presume that the app 
>>> will use QEMU's config file as its native format. Many apps will store
>>> configs in their own custom format (or in a database) and simply generate
>>> the QEMU config data on the fly when starting a VM. In the same way libvirt
>>> will  generate QEMU config data on the fly when starting a VM. Having many
>>> config formats & conversion / generation of the fly is a fact of life no 
>>> matter what mgmt system you use.
>>>   
>>>       
>> I don't see why we shouldn't try to change that. Why not generate a
>> common machine description file in qemu for all qemu VMs? Think of word
>> documents. Everyone knows how to read and write .doc files. Why
>> shouldn't VM description files be the same? It's really the best case
>> for the user if there's a single type of configuration.
>>     
>
> The raw QEMU config for a disk device is specified in terms of the
> file path for the storage.  A management app using QEMU / libvirt is
> not going to store its config for the guest in this way. They will
> have some model of storage and an association between a storage volume
> and a virtual machine. The actual file path for this may is only relevant
> at the time the VM is actually started & may be different on every host
> the VM is run on. eg if you've associated a VM with a LUN based, it may
> be /dev/sda when run on host A and /dev/sdz on host B. The mgmt app is
> going to use a mapping based on the  WWID, not paths. 
>   

Sounds like somebody didn't understand the concept of persistent device
names here. The device names should be /dev/disk/by-wwid/... then.

>   
>>> The key point is that it needs to be really easy to get at the generated
>>> QEMU config data to see what is actually being run. libvirt will always
>>> save the exact QEMU config data it generates into a log file for this
>>> exact purpose (/var/log/libvirt/qemu/$VMNAME.log).  The complexity here
>>> is that you can't directly run this if TAP devices are in use for the
>>> networking since it'll expect TAP FDs to be passed down. Although you 
>>> could allow QEMU to open the TAP devices, this is not good  for security 
>>> separation of QEMU from the host OS, so I'm not sure it can be easily
>>> avoided. 
>>>   
>>>       
>> Sounds like that particular feature needs to go into the qemu stack
>> then, so it can be properly described in the config file.
>>
>>     
>>> One attempt to make life easier, was that we added a libvirt command 
>>> 'virsh domxml-to-native' command which given a libvirt XML config file
>>> would return a set of QEMU command line args which *can* be used to
>>> start the VM directly. This is basically identical to the normal QEMU
>>> args libvirt generates but relies on the ifup script to create TAP
>>> networking instead.
>>>   
>>>       
>> That helps for XML -> qemu. It doesn't help when you're trying to go the
>> other way around. You'd still have to learn a different syntax. People
>> are reluctant to learn new syntaxes.
>>     
>
> In 99% of the cases you only need to know one syntax - the syntax that is
> appropriate for the app you're interacting with. You only need to know
> multiple syntaxes if you're trying to play at all levels of the stack
> which is not something we should be expecting in the common case.
>   

I think that's the basic assumption you're taking that I disagree with.
I think people do need to know the full stack to be able to master it.
And the easier we make the stack, the better adoption rate we will get.


Alex