Re: [Qemu-devel] when we add EL2 to QEMU TCG ARM emulation and the virt board, should it default on or off?

[Andrew Jones]

On Tue, Nov 01, 2016 at 05:16:59PM +0000, Peter Maydell wrote:
> I'm working on turning on EL2 support in our TCG ARM emulation,
> and one area I'm not sure about is whether it should default to
> on or off.
> 
> We have a few precedents:
> 
> For EL3 support:
>  * the CPU property is enabled by default but can be disabled by the board
>  * 'virt' defaults it to disabled, with a -machine secure=on property
>    which turns it on (barfing if KVM is enabled) and also does some other
>    stuff like wiring up secure-only memory and devices and making sure
>    the GIC has security enabled

Looks like EL3 support requires enough board changes that it's really
a board feature that requires cpu support (EL3)

>  * if the user does -cpu has_el3=yes things will probably not go
>    too well and that's unsupported

And this confirms that it's a board feature that requires a cpu
feature to be enabled, because just enabling the cpu feature alone
is not only insufficient, but harmful.

(I'll go off-topic here and state that properties like has_el3, which
 we don't want users setting, probably shouldn't be user-settable.
 This has come up a few times before and I think Igor once proposed
 to extend QOM properties to have a "user visible" property that could
 be toggled.)

> 
> For PMU support:
>  * the CPU property is enabled by default
>  * 'virt' defaults it to enabled (except for virt-2.6 and earlier)
>  * we do expect the user to be able to turn it on and off with
>    a -cpu pmu=yes/no option (and the board model changes behaviour
>    based on whether the CPU claims to have the feature)

So here we have the opposite; we have a cpu feature that requires
changes to the board (assignment of an IRQ) to make it more useful.
But, if we used -cpu pmu=on with a board that doesn't assign an IRQ,
then that's OK, the PMU just isn't as useful. So this is definitely
a cpu feature, thus the user-settable cpu property (and lack of a
board property), makes sense.

> 
> So what about EL2? Some background facts that are probably relevant:
>  * at the moment KVM can't support it, but eventually machines with
>    the nested virtualization hardware support will appear (this is
>    an ARMv8.3 feature), at which point it ought to work there too
>  * if you just enable EL2 then some currently working setups stop
>    working (basically any code that was written to assume it was
>    entered in EL1); notably this includes kvm-unit-tests (patches
>    pending from Drew that fix that). Linux is fine though as it
>    checks and handles both.

Have we checked the status of AAVMF yet?

> 
> Disabled-by-default has the advantage that (a) a command line
> will work the same for TCG and KVM and (b) nothing that used to
> work will break. The disadvantage is that anybody who wants to
> be able to run nested KVM will now have to specify a command line
> option of some kind.
> 
> So:
>  (1) should we default on or off? (both at the board level,
>      and for the cpu object itself)

I vote off by default for the two reasons (a and b) you stated
above and also because starting in EL2 may trigger different paths
for the guest, ones that aren't necessary for the "normal" non-
virt enabling use cases. Linux probably won't be adversely 
affected by this, but maybe other guest types would. We'd want
the default to match what we expect to be "normal" use.

>  (2) directly user-set cpu property, or board property ?

So this is why I was rambling above about the two precedents.
IMO, we should choose a cpu property when it's a purely cpu
feature, like the PMU, and board (without a user-settable cpu
property) otherwise. Does this feature require board changes?
I'm aware of switching the psci method from hvc to smc, but
I'd shrug that one off. Is there anything else? If not, then
I vote cpu property only.

Thanks,
drew