Re: [Qemu-devel] [RFC] vhost: Can we change synchronize_rcu to call_rcu in vhost_set_memory() in vhost kernel module?

[Michael S. Tsirkin]

On Mon, May 12, 2014 at 11:57:32AM +0200, Paolo Bonzini wrote:
> Il 12/05/2014 11:28, Gonglei (Arei) ha scritto:
> >From previous discussion:
> >https://lists.gnu.org/archive/html/qemu-devel/2014-03/msg04925.html
> >we know that you are going to replace RCU in KVM_SET_GSI_ROUTING with SRCU. 
> >Though
> >SRCU is quite better than originally RCU, in our test case this cannot 
> >satisfy our needs. Our VMs
> >work in telecom scenario, VMs report CPU and memory usage to balance node 
> >each second, and
> >balance node dispatch works to different VMs according to VM load. Since 
> >this balance needs
> >high accuracy, IRQ affinity settings in VM also need high accuracy, so we 
> >balance IRQ affinity in
> >every 0.5s. So for telecom scenario, KVM_SET_GSI_ROUTING IOCTL needs much 
> >optimization.
> >And in live migration case, VHOST_SET_MEM_TABLE needs attention.
> >
> >We tried to change synchronize_rcu() to call_rcu() with rate limit, but rate 
> >limit is not easy to
> >configure. Do you have better ideas to achieve this? Thanks.
> 
> Perhaps we can check for cases where only the address is changing,
> and poke at an existing struct kvm_kernel_irq_routing_entry without
> doing any RCU synchronization?
> 
> As long as kvm_set_msi_irq only reads address_lo once, it should work.
> 
> VHOST_SET_MEM_TABLE is a different problem.  What happens in
> userspace that leads to calling that ioctl?  Can we remove it
> altogether, or delay it to after the destination has started
> running?
> 
> Paolo

vhost does everything under a VQ lock.
I think RCU for VHOST_SET_MEM_TABLE can be replaced with 
taking and freeing the VQ lock.

Does the below solve the problem for you
(warning: untested, sorry, busy with other bugs right now)?


Signed-off-by: Michael S. Tsirkin <address@hidden>

---

diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
index 78987e4..df2e3eb 100644
--- a/drivers/vhost/vhost.c
+++ b/drivers/vhost/vhost.c
@@ -593,6 +593,7 @@ static long vhost_set_memory(struct vhost_dev *d, struct 
vhost_memory __user *m)
 {
        struct vhost_memory mem, *newmem, *oldmem;
        unsigned long size = offsetof(struct vhost_memory, regions);
+       int i;
 
        if (copy_from_user(&mem, m, size))
                return -EFAULT;
@@ -619,7 +620,14 @@ static long vhost_set_memory(struct vhost_dev *d, struct 
vhost_memory __user *m)
        oldmem = rcu_dereference_protected(d->memory,
                                           lockdep_is_held(&d->mutex));
        rcu_assign_pointer(d->memory, newmem);
-       synchronize_rcu();
+
+       /* All memory accesses are done under some VQ mutex.
+        * So below is a faster equivalent of synchronize_rcu()
+        */
+       for (i = 0; i < dev->nvqs; ++i) {
+               mutex_lock(d->vqs[idx]->mutex);
+               mutex_unlock(d->vqs[idx]->mutex);
+       }
        kfree(oldmem);
        return 0;
 }