Re: [Qemu-arm] [Qemu-devel] [PATCH V7 0/2] Add option to configure guest vPMU

[Wei Huang]

On 10/24/2016 03:49 AM, Andrew Jones wrote:
> On Fri, Oct 21, 2016 at 05:53:00PM -0400, Wei Huang wrote:
>> This patchset adds a pmu=[on/off] option to enable/disable vPMU support 
>> for guest VM. There are several reasons to justify this option. First,
>> vPMU can be problematic for cross-migration between different SoC as perf
>> counters are architecture-dependent. It is more flexible to have an option
>> to turn it on/off. Secondly this option matches the "pmu" option as
>> supported in libvirt. To make sure backward compatible, a PMU-related
>> property is added to mach-virt machine types.
>>
>> The following are testing results with this patchset:
>>  CONFIG (qemu-system-aarch64)                       vPMU   WARNING
>>   -M virt-2.8/virt,accel=kvm -cpu host               YES    NO
>>   -M virt-2.8/virt,accel=kvm -cpu host,pmu=off       NO     NO
>>   -M virt-2.8/virt,accel=kvm -cpu host,pmu=on        YES    NO
>>   -M virt-2.7,accel=kvm -cpu host                    YES    NO
>>   -M virt-2.7,accel=kvm -cpu host,pmu=off            NO     NO
>>   -M virt-2.7,accel=kvm -cpu host,pmu=on             YES    NO
>>   -M virt-2.6,accel=kvm -cpu host                    NO     NO
>>   -M virt-2.6,accel=kvm -cpu host,pmu=off            NO     NO
>>   -M virt-2.6,accel=kvm -cpu host,pmu=on             NO     NO
>>
>>   -M virt-2.8/virt,accel=tcg -cpu cortex-a57         NO     NO
>>   -M virt-2.8/virt,accel=tcg -cpu cortex-a57,pmu=off NO     "No PMU property"
>>   -M virt-2.8/virt,accel=tcg -cpu cortex-a57,pmu=on  NO     "No PMU property"
>>   -M virt-2.7,accel=tcg -cpu cortex-a57              NO     NO
>>   -M virt-2.7,accel=tcg -cpu cortex-a57,pmu=off      NO     "No PMU property"
>>   -M virt-2.7,accel=tcg -cpu cortex-a57,pmu=on       NO     "No PMU property"
>>   -M virt-2.6,accel=tcg -cpu cortex-a57              NO     NO
>>   -M virt-2.6,accel=tcg -cpu cortex-a57,pmu=off      NO     "No PMU property"
>>   -M virt-2.6,accel=tcg -cpu cortex-a57,pmu=on       NO     "No PMU property"
>>
>>   -M virt-2.8/virt,accel=tcg -cpu cortex-a15         NO     NO
>>   -M virt-2.8/virt,accel=tcg -cpu cortex-a15,pmu=off NO     "No PMU property"
>>   -M virt-2.8/virt,accel=tcg -cpu cortex-a15,pmu=on  NO     "No PMU property"
>>   -M virt-2.7,accel=tcg -cpu cortex-a15              NO     NO
>>   -M virt-2.7,accel=tcg -cpu cortex-a15,pmu=off      NO     "No PMU property"
>>   -M virt-2.7,accel=tcg -cpu cortex-a15,pmu=on       NO     "No PMU property"
>>   -M virt-2.6,accel=tcg -cpu cortex-a15              NO     NO
>>   -M virt-2.6,accel=tcg -cpu cortex-a15,pmu=off      NO     "No PMU property"
>>   -M virt-2.6,accel=tcg -cpu cortex-a15,pmu=on       NO     "No PMU property"
>>
>>   * "No PMU property" msg, e.g.
>>     can't apply global cortex-a15-arm-cpu.pmu=off: Property '.pmu' not found
>>
>> V6->V7:
>>   * change has_pmu variable type from OnOffAuto to Boolean
>>   * only add "pmu" property to CPU under kvm mode, default ON
> 
> Hmm, if we don't allow the property with TCG then switching a guest from
> KVM to TCG will require more than just an accelerator switch. That's a
> bit annoying and I think we'd have to teach it to libvirt too. I'd prefer
> 
>  -M virt-2.8,accel=tcg -cpu cortex-a57         NO     NO
>  -M virt-2.8,accel=tcg -cpu cortex-a57,pmu=off NO     NO
>  -M virt-2.8,accel=tcg -cpu cortex-a57,pmu=on  NO     "Warning: PMU not
> yet supported with TCG" (or something)

I am fine with this request. But note that, if we enforce
pmu-default=ON, we can't tell "-cpu cortex-a57" apart from
"cortex-a57,pmu=on", implying that we have to print the warning msg for
the case of "cortex-a57" as well (this was why we switch to tri-state
before). To solve this problem, we have to switch from pmu-default=ON to
pmu-default=OFF under TCG mode, something like:

    if (arm_feature(&cpu->env, ARM_FEATURE_PMU)) {
        qdev_property_add_static(DEVICE(obj),
            &arm_cpu_has_pmu_property, &error_abort);
        if (!kvm_enabled())
            object_property_set_bool(obj, false, "pmu", NULL);
    }

Then we do:
    if (cpu->has_pmu && !kvm_enabled()) {
        cpu->has_pmu = false;
        if (!pmu_warned && !qtest_enabled()) {
            error_report("warning: pmu not supported under TCG");
            pmu_warned = true;
        }
    }

This will work. Are you and Peter OK with this solution?

-Wei


> 
> 
>>   * set no_pmu=true for machvirt-2.6
>>
>> V5->V6:
>>   * adapt patches for new machine type 2.8
>>
>> V4->V5:
>>   * remove comment change for has_pmu
>>   * remove warning msg when pmu_default_on=TRUE && has_pmu=AUTO && tcg=TRUE
>>
>> V3->V4:
>>   * change has_pmu from Boolean to OnOffAuto to handle different cases
>>   * "pmu" property is re-defined as DEFINE_PROP_ON_OFF_AUTO
>>
>> V2->V3:
>>   * revise patch 1 commit msg and if-else statement (Drew) 
>>   * move property field into VirtMachineClass (Drew)
>>
>> V1->V2:
>>   * keep the original field name as "has_pmu"
>>   * add a warning message when PMU is turned on without KVM
>>   * use the feature bit to check PMU availability, instead of using has_pmu
>>   * add PMU compat support to mach-virt machine type
>>
>> RFC->V1:
>>   * set default pmu=off
>>   * change struct ARMCPU field name "has_pmu" ==> "has_host_pmu"
>>   * like el3, add a new feature ARM_FEATURE_HOST_PMU
>>   * "pmu" property becomes CPU dependent. Only cortex-a53/cortex-a57/host
>>     running on kvm supports this option.
>>
>> Thanks,
>> -Wei
>>
>> Wei Huang (2):
>>   arm64: Add an option to turn on/off vPMU support
>>   arm: virt: add PMU property to mach-virt machine type
>>
>>  hw/arm/virt-acpi-build.c |  2 +-
>>  hw/arm/virt.c            |  9 ++++++++-
>>  target-arm/cpu.c         | 14 ++++++++++++++
>>  target-arm/cpu.h         |  1 +
>>  target-arm/cpu64.c       |  2 ++
>>  target-arm/kvm64.c       | 17 ++++++++++++++---
>>  6 files changed, 40 insertions(+), 5 deletions(-)
>>
>> -- 
>> 1.8.3.1
>>
>>