[Qemu-devel] Re: libvirt vs. in-qemu management

[Alexander Graf]

On 06.04.2010, at 00:14, Avi Kivity wrote:

> On 04/06/2010 12:11 AM, Alexander Graf wrote:
>> Howdy,
>> 
>> I've been thinking a bit further on the whole issue around libvirt and why 
>> the situation as is isn't satisfying. I came to the following points that 
>> currently hurt building ease of use for KVM:
>> 
>> 1) Brand
>> 
>> This is one of the major issues we have ourselves when it comes to 
>> appliances. We can ship appliances built for VMware. We can ship appliances 
>> built for Xen. But we can't ship appliances built for KVM, because there is 
>> no single management app we could target.
> 
> There are already at least three management apps for kvm:  virt-manager, 
> proxmox, and RHEV-M (my personal favorite).  If we define our own format then 
> we need those management apps to understand it.  That means we either include 
> only simple features, or we wait until the management apps catch up to all 
> the features we provide.  Otherwise those appliances aren't universal.
> 
> An additional problem is that our format will exclude metadata that the 
> management app may want to add.

Do they have to be? There could always be a metadata section for different 
management apps. If the app understands the metadata, fine. If it doesn't you 
can still run your VM. You could even ship an appliance with multiple metadata 
sections so multiple management stacks understand it.

> 
>> That destroys the KVM brand IMHO.
>>   
> 
> That's because  kvm is infrastructure instead of a complete stack.  I agree 
> it's a problem but I see no way around it.

I believe we need to (at least partially) get rid of that separation. It's 
really hard to survive as infrastructure in a land of stacks.

> 
>> 2) Machine description
>> 
>> If we build an appliance, we also create a configuration file that describes 
>> the VM. We can create .vmx files, we can create xen config files. We can not 
>> create KVM config files. There are none. We could create shell scripts, but 
>> would that help?
>>   
> 
> It's not enough for qemu to be able to read the configuration file.  The 
> management app needs to read it as well, to understand how much memory and 
> cpu the guest needs (so it can schedule it on the cluster), what kind of 
> network connectivity it needs (how many interfaces, what networks those 
> interfaces connect to, does it need firewall ports open).  An appliance 
> configuration is more than a vm configuration, and again, the management app 
> needs to be able to understand all of it.

Well of course. That's the point. There's a machine config format that both the 
management app and qemu understand. That format is also human readable (XML is 
not) and can thus be hand-edited if necessary. If you want to copy your VM, 
just copy the disk image and that config file.


> 
>> 3) Configuration conversion
>> 
>> Party due to qemu not having a configuration format, partly due to libvirt's 
>> ambivalent approach, there is always conversion in configuration formats 
>> involved. I think this is the main reason for the feature lag. If there 
>> wasn't a conversion step, there wouldn't be lag. You could just hand edit 
>> the config file and be good.
>>   
> 
> There will always be a lag, since management apps (at least the non-trivial 
> ones) want to display the configuration in a GUI, allow users to edit it, and 
> want to understand it.  It's not just conversion, it's plumbing down the 
> whole stack.

... which involves conversion from a management specific format to some random 
qemu format munch (cmdline options, monitor commands, etc.).

> 
>> Point 2 needs to be solved anyways. We need a machine config format for 
>> qemu. For general -M description as well as for specific VM description. The 
>> command line options just become too complicated and too hard to reproduce 
>> and save. Just think of live migration with hot-plugged devices. Or safe 
>> savevm + loadvm. The current logic ends there.
>>   
> 
> I don't think the management apps will want to use it.  They will need to 
> parse it (currently they only need to write it, which is simpler).  Things 
> like 'query all smp guests with >4GB memory' become complicated instead just 
> a database query.

That's the only way to get VMs compatible. They have to. We have to force them 
do it.

> 
> For managed guests, I think we want to get rid of the command line at all.  
> Start the guest with just a case and cold-plug the motherboard, processors, 
> memory, cards.  Migration starts with a replay of these (including any 
> hotplugged cards added while the guest is running).  Hotplugs during 
> migration are relayed to the other side over the wire.

While I like the idea, I think that belongs into qemu internals. There should 
still be a config file that you pass to libqemu which then creates the machine 
for you.

> 
>> I can imagine 1) going away if we would set libvirt + virt-manager as _the_ 
>> front-end and have everyone focus on it. I suppose it would also help to 
>> rebrand it by then, but I'm not 100% sure about that. Either way, there 
>> would have to be a definite statement that libvirt is the solution to use. 
>> And _everyone_ would have to agree on that. Sounds like a hard task. And by 
>> then we still don't really have a branded product stack.
>> 
>> Point 3 is the really tough one. It's the very basis of libvirt. And it's 
>> plain wrong IMHO. I hate XML. I hate duplicated efforts. The current 
>> conversion involves both. Every option added to qemu needs to be added to 
>> libvirt.
> 
> Not just libvirt, virt-manager as well.  And that is typically more difficult 
> technically (though probably takes a lot less time).

The point is that you can't fine-tune even if you wanted. And even if you 
could, a qemu developer couldn't help. Because the formats are different.

> 
>> In XML. Bleks.
>>   
> 
> Yeah.
> 
>> Reading on IRC I seem to not be the only person thinking that, just the 
>> first one mentioning this aloud I suppose. But that whole XML mess really 
>> hurts us too. Nobody wants to edit XML files. Nobody wants to have two 
>> separate syntaxes to describe the same thing. It complicates everything 
>> without a clear benefit. And it puts me in a position where I can't help 
>> people because I don't know the XML format. That should never happen.
>>   
> 
> 
>> Sure, for libvirt it makes sense to be hypervisor-agnostic. For qemu it 
>> doesn't. We want to be _the_ hypervisor. Setting our default front-end to 
>> something that is agnostic weakens our point. And it slows down development. 
>> And it hurts integration. And thus usability, thus adoption. It hurts us.
>>   
> 
> It doesn't make sense for libvirt to be hypervisor agnostic.  If it is, 
> people who want to use one hypervisor's advanced features are forced to work 
> around it.  Anthony wants multiple monitors for this, but that's a bad 
> workaround.  libvirt is placing developers using it in an impossible 
> situation - the developers want to use kvm-specific features and libvirt is 
> in the way.

I think having QMP exposed (hence the monitor) and a common machine description 
would give us all the power we need as developers to do what we want to do.

> 
>> That's what I've concluded so far on the whole situation as is. I find it 
>> sad to be the one speaking it out, but IMHO going with libvirt as default 
>> management front-end is a dead end. It will hurt us more than it will help 
>> us.
>>   
> 
> Well, I did suggest (and then withdraw) qemud.  The problem is that to get 
> something working we'd duplicate all the work that's gone into libvirt - 
> storage pools, svirt, network setup, etc.

That's infrastructure that should probably go along with qemu then. Why should 
other UIs not benefit from secure VMs? Why should other UIs not benefit from 
device passthrough cleverness? Why should other UIs not benefit from easier 
network setup?

Take a look at our competition (vmware / vbox). They do the full stack. That's 
what users want. They want to do something easily. And I do too :-).


Alex