RE: [PATCH V2 01/10] accel/kvm: Extract common KVM vCPU {creation,parking} code

[Salil Mehta]

Hi Gavin,

> From: Gavin Shan <gshan@redhat.com>
> Sent: Tuesday, October 3, 2023 12:18 AM
> To: Salil Mehta <salil.mehta@huawei.com>; qemu-devel@nongnu.org; qemu-
> arm@nongnu.org
> Cc: maz@kernel.org; jean-philippe@linaro.org; Jonathan Cameron
> <jonathan.cameron@huawei.com>; lpieralisi@kernel.org;
> peter.maydell@linaro.org; richard.henderson@linaro.org;
> imammedo@redhat.com; andrew.jones@linux.dev; david@redhat.com;
> philmd@linaro.org; eric.auger@redhat.com; oliver.upton@linux.dev;
> pbonzini@redhat.com; mst@redhat.com; will@kernel.org; rafael@kernel.org;
> alex.bennee@linaro.org; linux@armlinux.org.uk;
> darren@os.amperecomputing.com; ilkka@os.amperecomputing.com;
> vishnu@os.amperecomputing.com; karl.heubaum@oracle.com;
> miguel.luis@oracle.com; salil.mehta@opnsrc.net; zhukeqian
> <zhukeqian1@huawei.com>; wangxiongfeng (C) <wangxiongfeng2@huawei.com>;
> wangyanan (Y) <wangyanan55@huawei.com>; jiakernel2@gmail.com;
> maobibo@loongson.cn; lixianglai@loongson.cn; Linuxarm <linuxarm@huawei.com>
> Subject: Re: [PATCH V2 01/10] accel/kvm: Extract common KVM vCPU
> {creation,parking} code
> 
> On 9/30/23 10:19, Salil Mehta wrote:
> > KVM vCPU creation is done once during the initialization of the VM when Qemu
> > threads are spawned. This is common to all the architectures.
>    ^^^^^^^^^^^^^^^^^^^
>    thread is spawned.

Yes, will fix.

Thanks
Salil.


> > Hot-unplug of vCPU results in destruction of the vCPU objects in QOM but
> > the KVM vCPU objects in the Host KVM are not destroyed and their
> representative
> > KVM vCPU objects/context in Qemu are parked.
> >
> > Refactor common logic so that some APIs could be reused by vCPU Hotplug 
> > code.
> >
> > Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
> > ---
> >   accel/kvm/kvm-all.c  | 63 +++++++++++++++++++++++++++++++++-----------
> >   include/sysemu/kvm.h | 14 ++++++++++
> >   2 files changed, 61 insertions(+), 16 deletions(-)
> >
> > diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c
> > index ff1578bb32..b8c36ba50a 100644
> > --- a/accel/kvm/kvm-all.c
> > +++ b/accel/kvm/kvm-all.c
> > @@ -80,7 +80,7 @@
> >   #endif
> >
> >   struct KVMParkedVcpu {
> > -    unsigned long vcpu_id;
> > +    int vcpu_id;
> 
> @vcpu_id represents the vCPU index (CPUState::cpu_index) instead of the
> architectural CPU ID any more. However, I don't understand how it works
> for x86, and more comments regarding it can be seen below.

I missed the reason why I had used 'unsigned long' everywhere in the
first change. You can check the PowerPC, it returns vcpu_id which might
not be an 'integer'. Hence, this change could actually create a problem.


> >       int kvm_fd;
> >       QLIST_ENTRY(KVMParkedVcpu) node;
> >   };
> > @@ -137,6 +137,7 @@ static QemuMutex kml_slots_lock;
> >   #define kvm_slots_unlock()  qemu_mutex_unlock(&kml_slots_lock)
> >
> >   static void kvm_slot_init_dirty_bitmap(KVMSlot *mem);
> > +static int kvm_get_vcpu(KVMState *s, int vcpu_id);
> >
> >   static inline void kvm_resample_fd_remove(int gsi)
> >   {
> > @@ -320,11 +321,49 @@ err:
> >       return ret;
> >   }
> >
> > +void kvm_park_vcpu(CPUState *cpu)
> > +{
> > +    int vcpu_id = cpu->cpu_index;
> > +    struct KVMParkedVcpu *vcpu;
> > +
> > +    vcpu = g_malloc0(sizeof(*vcpu));
> > +    vcpu->vcpu_id = vcpu_id;
> > +    vcpu->kvm_fd = cpu->kvm_fd;
> > +    QLIST_INSERT_HEAD(&kvm_state->kvm_parked_vcpus, vcpu, node);
> > +}
> > +
> 
> @vcpu_id can be dropped as suggested previously.
> 
>         vcpu->vcpu_id = cpu->cpu_index;


Yes, agreed.

Thanks
Salil.


> > +int kvm_create_vcpu(CPUState *cpu)
> > +{
> > +    int vcpu_id = cpu->cpu_index;
> > +    KVMState *s = kvm_state;
> > +    int kvm_fd;
> > +
> > +    DPRINTF("kvm_create_vcpu\n");
> > +
> > +    /* check if the KVM vCPU already exist but is parked */
> > +    kvm_fd = kvm_get_vcpu(s, vcpu_id);
> > +    if (kvm_fd < 0) {
> > +        /* vCPU not parked: create a new KVM vCPU */
> > +        kvm_fd = kvm_vm_ioctl(s, KVM_CREATE_VCPU, vcpu_id);
> > +        if (kvm_fd < 0) {
> > +            error_report("KVM_CREATE_VCPU IOCTL failed for vCPU %d", 
> > vcpu_id);
> > +            return kvm_fd;
> > +        }
> > +    }
> > +
> > +    cpu->vcpu_dirty = true;
> > +    cpu->kvm_fd = kvm_fd;
> > +    cpu->kvm_state = s;
> > +    cpu->dirty_pages = 0;
> > +    cpu->throttle_us_per_full = 0;
> > +
> > +    return 0;
> > +}
> > +
> 
> The comments here can be dropped since the code is self-explaining.
> 
> @vcpu_id represents vCPU index, instead of the architecrual vCPU ID any
> more.
> @vcpu_id is passed to host through ioctl(KVM_CREATE_VCPU), which is
> expected
> as an architecrual vCPU ID instead of a vCPU index by host. It's indicated
> by 'struct kvm_vcpu' as below.


That should not be part of this change. I think.


> struct kvm_vcpu {
>       :
>       int vcpu_id;  /* id given by userspace at creation */
>          int vcpu_idx; /* index into kvm->vcpu_array */
> };
> 
> Function kvm_arch_vcpu_id() converts the vCPU instance or vCPU index to
> the architecrual vCPU ID. All architectures except x86 simply returns
> vCPU index (CPUState::cpu_index) as the architecrural vCPU ID. x86 returns
> the APIC ID. Treating them equally seems to break x86.

I think PowerPC returns a different Id as well.


[...]

> > -static int kvm_get_vcpu(KVMState *s, unsigned long vcpu_id)
> > +static int kvm_get_vcpu(KVMState *s, int vcpu_id)
> >   {
> >       struct KVMParkedVcpu *cpu;
> >
> > @@ -384,7 +420,7 @@ static int kvm_get_vcpu(KVMState *s, unsigned long
> vcpu_id)
> >           }
> >       }
> >
> > -    return kvm_vm_ioctl(s, KVM_CREATE_VCPU, (void *)vcpu_id);
> > +    return -1;
> >   }
> >
> 
> Why we have -1 here. -ENOENT seems more descriptive?

Agreed.

Thanks
Salil.