Re: [Qemu-devel] [RFC 00/28] s390x CPU models: exposing features

[David Hildenbrand]

> On Tue, Jun 21, 2016 at 13:44:31 -0300, Eduardo Habkost wrote:
> > (CCing libvirt people)
> > 
> > On Tue, Jun 21, 2016 at 03:02:05PM +0200, David Hildenbrand wrote:
> > > This is our second attempt to implement CPU models for s390x. We realized
> > > that we also want to have features exposed via the CPU model. While doing
> > > that we realized that we want to have a better interface for libvirt.
> > 
> > Before getting into the details, I would like to clarify how the
> > following could be accomplished using the new commands:
> > 
> > Example:
> > 
> > 1) User configures libvirt with:
> >    <cpu match='exact'>
> >        <model fallback='forbid'>Westmere</model>
> >        <feature policy='require' name='aes'/>
> >    </cpu>
> > 2) libvirt will translate that to:
> >    "-cpu Westmere,+aes" or "-cpu Westmere,aes=on"
> > 3) libvirt wants to know if "-cpu Westmere,aes=on" is usable in
> >    the current host, before trying to start the VM.
> 
> While this is what libvirt currently does, I don't think it's necessary
> to keep doing that... see below.
> 
> > > a) "query-cpu-model-expansion" - tell us what the "host" or a migration
> > >    unsafe model looks like. Either falling back to a stable model or
> > >    completely exposing all properties. We are interested in stable models.
> > > b) "query-cpu-model-comparison" - tell us how two CPU models compare,
> > >     indicating which properties were responsible for the decision.
> > > c) "query-cpu-model-baseline" - create a new model out of two models,
> > >     taking a requested level of stability into account.
> 
> This looks like it copies current libvirt APIs, which I think is not a
> very good idea. Both CPU compare and baseline APIs in libvirt will need
> to be changed (or rather new APIs with similar functionality added) to
> become useful. I think we should first focus on making guest CPU
> configuration work the way it should work. For that I think we need some
> higher level commands.

Hi Jiri,

thanks for your reply!

Please be aware that we (s390x) have no libvirt cpu model support in place. We
are coming up with a solution that solves our problems, but instead of tagging
all interfaces with "s390x" we want interfaces that everybody might use.

I understand that x86 has some very advanced libvirt CPU model magic in place.
You can still stick to what you have right now. It doesn't work for us.

So we provide 3 basic functions (baseline, compare, expand) that can easily by
added for s390x into the CPU model. So I don't see a problem introducing
these operations in libvirt - it just works without any magic and knowledge
about features and models in libvirt.

In the end it keeps the CPU model where it belongs (QEMU) and allows us to
just fit in fine into the libvirt CPU model world that was heavily inspired
by x86 (an I can tell you, that is a very challenging task).

But - of course - we have to do some additional queries on a QEMU instance. I
don't think that this is a problem (at least not for s390x).

> 
> Let me sum up what libvirt is doing (or will be doing) with guest
> CPUs... Libvirt supports three guest CPU configuration modes:

One important thing to mention here is that all of these was inspired by x86.
Some stuff doesn't even make sense on s390x ("mimic the real host"), as
it would even in practice never be possible. See my reply to Eduards patchset
I'll post later.

We want to focus on the real use cases and I think they are
- Making sure models ("Guest API") doesn't change during migration
- Allowing the user to use a certain CPU model
- Baselining/Comparing/Using the -cpu host model


> 
> - host-passthrough -- this is easy, it's just asking for "-cpu host" and
>   no fancy commands are required.

This should map to expanding the host model in our case.

> 
> - host-model -- for this we need to know what CPU configuration we need
>   to ask for to get as close to the host CPU as possible while being
>   able to ask for the exact same CPU on a different host (after
>   migration). And we need to be able to ask for this at probing time
>   (when libvirtd starts rather than when starting a new domain) using
>   "-machine none,accel=kvm:tcg", that is without actually creating any
>   guest CPU. This is what Eduardo's query-host-cpu QMP command is useful
>   for. In x86 world we could just query the guest CPU after running QEMU
>   with "-cpu host", but that's unusable with "-machine none", which is
>   why another way of getting this info is needed.

Again, we will be relying on "-cpu host" here, because the real host model
will never be possible to be fordwarded on s390x. And what's the use case
anyway? Does not make sense on s390x as far as I can tell.

We want to know what we are able to virtualize, not build up some strange
model that won't even be able to run.

> 
> - custom -- the XML specifies an exact guest CPU configuration. We don't
>   really need to know if that exact CPU is runnable on the current host
>   in advance, we can just try to start the domain, check if the guest
>   CPU matches what we asked for, and we may kill QEMU if the CPU does
>   not meet the specification. Of course, higher level management wants
>   to know a set of host where it can run a given domain for scheduling
>   purposes, but since they logically want to avoid tons of different CPU
>   configs, they would just stick the CPU models predefined by QEMU. Thus
>   giving them a way of checking what CPU models can be run on a given
>   host with a given QEMU (using the unavailable features stuff for
>   query-cpu-definitions usable with "-machine none") should be enough
>   and it's even better than having to ask for every single CPU model,
>   which is what they need to do now.

Which is simply necessary when having to care about a lot of feature
variability. Please keep the requirements in our cover letter in place.

It's not that simple when
a) having to update the "flexible" models over time (e.g. adding stuff to "z13")
b) nobody really using the "stable" models like "z13-base", but interested in
   "z13" +  feature changes

I agree that CPU models defined in QEMU are the place to go - which is why we
keep our default here.

And also keep in mind that I think your extension to query-cpu-definitions to
detect runability makes total sense, it is just not enough for us having to care
about changable "default" models.

Let's say we report a "z13" via that interface as runnable? It doesn't
mean a thing because that model could look different on e.g. another QEMU
version we're migrating to. To check the real deal, we have to expand that model
and then verify if it is runnable. In essence, runnability information
contained in query-cpu-definitions only makes sense for migration-safe models.

Of course, that could somehow be done using the "expansion" interface only
in combination with "query-cpu-definitions". But it doesn't solve the other
problems we're dealing with and just introduces some unnecessary complexity
(for us!).

> 
> There are configuration options which are somewhere between host-model
> and custom, but they don't bring more requirements in addition to what
> both of them needs.
> 
> This was basically all about starting a new domain. When the domain
> successfully starts, we need to make sure the guest CPU does not change
> during save/restore or migration to another host. To achieve this, we
> need the same checking we need for custom mode, i.e., whether the guest
> CPU we got is what we asked for. In addition to this, we need a way to
> ask QEMU what the guest CPU looks like so that we can store it in the
> domain XML and ask for it during migration.

Exactly, that's where our "expansion" interface comes into play.

If the user asks for a "z13", we will expand it to a
"z13-base,feata=on,featb=on" ...

So we can guarantee that we are always dealing with a migration safe variant.

> 
> I think all of this should be pretty much architecture agnostic. Of
> course the actual data would be quite different fro each architecture,
> but I believe the entry points could fit all. Or did I miss anything?
> 

These are also the use cases I looked into, with the addition of being able to
baseline and compare CPU models (including the "host" model).

Thanks again for having a look!

David