RE: [PATCH RFC V2 34/37] target/arm/kvm,tcg: Register/Handle SMCCC hypercall exits to VMM/Qemu

[Salil Mehta]

Hi Gavin,

> From: Gavin Shan <gshan@redhat.com>
> Sent: Friday, September 29, 2023 5:15 AM
> To: Salil Mehta <salil.mehta@huawei.com>; qemu-devel@nongnu.org; 
> qemu-arm@nongnu.org
> Cc: maz@kernel.org; jean-philippe@linaro.org; Jonathan Cameron
> <jonathan.cameron@huawei.com>; lpieralisi@kernel.org;
> peter.maydell@linaro.org; richard.henderson@linaro.org;
> imammedo@redhat.com; andrew.jones@linux.dev; david@redhat.com;
> philmd@linaro.org; eric.auger@redhat.com; will@kernel.org; ardb@kernel.org;
> oliver.upton@linux.dev; pbonzini@redhat.com; mst@redhat.com;
> rafael@kernel.org; borntraeger@linux.ibm.com; alex.bennee@linaro.org;
> linux@armlinux.org.uk; darren@os.amperecomputing.com;
> ilkka@os.amperecomputing.com; vishnu@os.amperecomputing.com;
> karl.heubaum@oracle.com; miguel.luis@oracle.com; salil.mehta@opnsrc.net;
> zhukeqian <zhukeqian1@huawei.com>; wangxiongfeng (C)
> <wangxiongfeng2@huawei.com>; wangyanan (Y) <wangyanan55@huawei.com>;
> jiakernel2@gmail.com; maobibo@loongson.cn; lixianglai@loongson.cn
> Subject: Re: [PATCH RFC V2 34/37] target/arm/kvm,tcg: Register/Handle SMCCC
> hypercall exits to VMM/Qemu
> 
> Hi Salil,
> 
> On 9/26/23 20:36, Salil Mehta wrote:
> > From: Author Salil Mehta <salil.mehta@huawei.com>
> >
> > Add registration and Handling of HVC/SMC hypercall exits to VMM
> >
> > Co-developed-by: Salil Mehta <salil.mehta@huawei.com>
> > Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
> > Co-developed-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
> > Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
> > Signed-off-by: Salil Mehta <salil.mehta@huawei.com>

[...]

> > +static CPUArchId *arm_get_archid_by_id(uint64_t id)
> > +{
> > +    int n;
> > +    CPUArchId *arch_id;
> > +    MachineState *ms = MACHINE(qdev_get_machine());
> > +
> > +    /*
> > +     * At this point disabled CPUs don't have a CPUState, but their 
> > CPUArchId
> > +     * exists.
> > +     *
> > +     * TODO: Is arch_id == mp_affinity? This needs work.
> > +     */
> > +    for (n = 0; n < ms->possible_cpus->len; n++) {
> > +        arch_id = &ms->possible_cpus->cpus[n];
> > +
> > +        if (arch_id->arch_id == id) {
> > +            return arch_id;
> > +        }
> > +    }
> > +    return NULL;
> > +}
> > +
> 
> The @arch_id should be same thing to @mp_affinity except for the boot CPU.
> For the boot CPU, its value is fetched from MPIDR, which is determined by
> cs->cpu_index, passed to host via ioctl(CREATE_VCPU). Besides, another
> similiar function qemu_get_cpu_archid() exists in cpus-common.c. I think
> they can be combined. Again, all these information inherited from
> ms->possible_cpus may be better to be managed in board level, like the
> vCPU states.

Yes, good catch. This has been existing for long so my eyes got biased.


Thanks
Salil.


[...]

> > @@ -168,12 +189,24 @@ int arm_set_cpu_on(uint64_t cpuid, uint64_t entry,
> uint64_t context_id,
> >       }
> >
> >       /* Retrieve the cpu we are powering up */
> > -    target_cpu_state = arm_get_cpu_by_id(cpuid);
> > -    if (!target_cpu_state) {
> > +    arch_id = arm_get_archid_by_id(cpuid);
> > +    if (!arch_id) {
> >           /* The cpu was not found */
> >           return QEMU_ARM_POWERCTL_INVALID_PARAM;
> >       }
> >
> > +    target_cpu_state = CPU(arch_id->cpu);
> > +    if (!qemu_enabled_cpu(target_cpu_state)) {
> > +        /*
> > +         * The cpu is not plugged in or disabled. We should return 
> > appropriate
> > +         * value as introduced in DEN0022E PSCI 1.2 issue E
>                                                         ^^^^^^^
>                                                         issue E, which is 
> QEMU_PSCI_RET_DENIED.

   PSCI_RET_DENIED


[...]

> > --- a/target/arm/helper.c
> > +++ b/target/arm/helper.c
> > @@ -11187,7 +11187,7 @@ void arm_cpu_do_interrupt(CPUState *cs)
> >                         env->exception.syndrome);
> >       }
> >
> > -    if (tcg_enabled() && arm_is_psci_call(cpu, cs->exception_index)) {
> > +    if (arm_is_psci_call(cpu, cs->exception_index)) {
> >           arm_handle_psci_call(cpu);
> >           qemu_log_mask(CPU_LOG_INT, "...handled as PSCI call\n");
> >           return;
> 
> We may still limit the capability to handle PSCI calls to TCG and KVM,
> meaning HVF and QTest won't have this capability.


We do not support them now. I need to conditionally register SMCC calls
With KVM. Will check this. Good point though.

Thanks
Salil.


[...]

> > diff --git a/target/arm/kvm.c b/target/arm/kvm.c
> > index 8e7c68af6a..6f3fd5aecd 100644
> > --- a/target/arm/kvm.c
> > +++ b/target/arm/kvm.c
> > @@ -250,6 +250,7 @@ int kvm_arm_get_max_vm_ipa_size(MachineState *ms,
> bool *fixed_ipa)
> >   int kvm_arch_init(MachineState *ms, KVMState *s)
> >   {
> >       int ret = 0;
> > +
>    ^^^^
> Unnecessary change.

Yes.

Thanks.


[...]

> > @@ -280,6 +281,22 @@ int kvm_arch_init(MachineState *ms, KVMState *s)
> >           }
> >       }
> >
> > +    /*
> > +     * To be able to handle PSCI CPU ON calls in QEMU, we need to install 
> > SMCCC
>                                          ^^
>                                          ON/OFF

Yes.

> > +     * filter in the Host KVM. This is required to support features like
> > +     * virtual CPU Hotplug on ARM platforms.
> > +     */
> > +    if (kvm_arm_set_smccc_filter(PSCI_0_2_FN64_CPU_ON,
> > +                                 KVM_SMCCC_FILTER_FWD_TO_USER)) {
> > +        error_report("CPU On PSCI-to-user-space fwd filter install 
> > failed");
> > +        abort();
> > +    }
> > +    if (kvm_arm_set_smccc_filter(PSCI_0_2_FN_CPU_OFF,
> > +                                 KVM_SMCCC_FILTER_FWD_TO_USER)) {
> > +        error_report("CPU Off PSCI-to-user-space fwd filter install 
> > failed");
> > +        abort();
> > +    }
> > +
> >       kvm_arm_init_debug(s);
> >
> >       return ret;
> 
> The PSCI_ON and PSCI_OFF will be unconditionally handled by QEMU if KVM is
> enabled, even vCPU hotplug isn't supported on hw/arm/virt board. Do we need to
> enable it only when vCPU hotplug is supported?

Yes. True. I missed this earlier. It should be conditional.


Thanks
Salil.


> 
> > @@ -952,6 +969,38 @@ static int kvm_arm_handle_dabt_nisv(CPUState *cs,
> uint64_t esr_iss,
> >       return -1;
> >   }
> >
> > +static int kvm_arm_handle_hypercall(CPUState *cs, struct kvm_run *run)
> > +{
> > +    ARMCPU *cpu = ARM_CPU(cs);
> > +    CPUARMState *env = &cpu->env;
> > +
> > +    kvm_cpu_synchronize_state(cs);
> > +
> > +    /*
> > +     * hard coding immediate to 0 as we dont expect non-zero value as of 
> > now
>                                             ^^^^
>                                             don't

Thanks
Salil.