Re: [Qemu-devel] [RFC 23/29] vub+postcopy: madvises

[Dr. David Alan Gilbert]

* Alexey Perevalov (address@hidden) wrote:
> On 06/28/2017 10:00 PM, Dr. David Alan Gilbert (git) wrote:
> > From: "Dr. David Alan Gilbert" <address@hidden>
> > 
> > Clear the area and turn off THP.
> > 
> > Signed-off-by: Dr. David Alan Gilbert <address@hidden>
> > ---
> >   contrib/libvhost-user/libvhost-user.c | 32 
> > ++++++++++++++++++++++++++++++--
> >   1 file changed, 30 insertions(+), 2 deletions(-)
> > 
> > diff --git a/contrib/libvhost-user/libvhost-user.c 
> > b/contrib/libvhost-user/libvhost-user.c
> > index 0658b6e847..ceddeac74f 100644
> > --- a/contrib/libvhost-user/libvhost-user.c
> > +++ b/contrib/libvhost-user/libvhost-user.c
> > @@ -451,11 +451,39 @@ vu_set_mem_table_exec(VuDev *dev, VhostUserMsg *vmsg)
> >           }
> >           if (dev->postcopy_listening) {
> > +            int ret;
> >               /* We should already have an open ufd need to mark each memory
> >                * range as ufd.
> > -             * Note: Do we need any madvises? Well it's not been accessed
> > -             * yet, still probably need no THP to be safe, discard to be 
> > safe?
> >                */
> > +
> > +            /* Discard any mapping we have here; note I can't use 
> > MADV_REMOVE
> > +             * or fallocate to make the hole since I don't want to lose
> > +             * data that's already arrived in the shared process.
> > +             * TODO: How to do hugepage
> > +             */
> Hi, David, frankly saying, I stuck with my solution, and I have also another
> issues,
> but here I could suggest solution for hugepages. I think we could transmit a
> received pages
> bitmap in VHOST_USER_SET_MEM_TABLE (VhostUserMemoryRegion), but it will
> raise a compatibility issue,
> or introduce special message type for that and send it before
> VHOST_USER_SET_MEM_TABLE.
> So it will be  possible to do fallocate on received bitmap basis, just skip
> already copied pages.
> If you wish, I could send patches, rebased on yours, for doing it.

What we found works is that actually we don't need to do a discard -
since we've only just done the mmap of the arena, nothing will be
occupying it on the shared client, so we don't need to discard.

We've had a postcopy migrate work now, with a few hacks we're still
cleaning up, both on vhost-user-bridge and dpdk; so I'll get this
updated and reposted.

Dave

> > +            ret = madvise((void *)dev_region->mmap_addr,
> > +                          dev_region->size + dev_region->mmap_offset,
> > +                          MADV_DONTNEED);
> > +            if (ret) {
> > +                fprintf(stderr,
> > +                        "%s: Failed to madvise(DONTNEED) region %d: %s\n",
> > +                        __func__, i, strerror(errno));
> > +            }
> > +            /* Turn off transparent hugepages so we dont get lose wakeups
> > +             * in neighbouring pages.
> > +             * TODO: Turn this backon later.
> > +             */
> > +            ret = madvise((void *)dev_region->mmap_addr,
> > +                          dev_region->size + dev_region->mmap_offset,
> > +                          MADV_NOHUGEPAGE);
> > +            if (ret) {
> > +                /* Note: This can happen legally on kernels that are 
> > configured
> > +                 * without madvise'able hugepages
> > +                 */
> > +                fprintf(stderr,
> > +                        "%s: Failed to madvise(NOHUGEPAGE) region %d: 
> > %s\n",
> > +                        __func__, i, strerror(errno));
> > +            }
> >               struct uffdio_register reg_struct;
> >               /* Note: We might need to go back to using mmap_addr and
> >                * len + mmap_offset for * huge pages, but then we do hope 
> > not to
> 
> 
> -- 
> Best regards,
> Alexey Perevalov
--
Dr. David Alan Gilbert / address@hidden / Manchester, UK