Re: [Qemu-devel] [RFC PATCH RDMA support v2: 5/6] connection-setup code between client/server

[Michael S. Tsirkin]

On Tue, Feb 19, 2013 at 10:41:08AM -0500, Michael R. Hines wrote:
> On 02/18/2013 03:24 AM, Orit Wasserman wrote:
> >Hi Michael, The guest device state is quite small (~100K probably
> >less) especially when compared to the guest memory and we already
> >are pinning the guest memory for RDMA any way I was actually
> >wondering about the memory pinning, do we pin all guest memory
> >pages as migration starts or on demand?
> 
> The patch supports both methods. There's a function called
> "rdma_server_prepare()" which optionally pins all the memory in
> advanced.
> 
> We prefer on-demand pinning, of course, because we want to preserve
> the ability to do ballooning and the occasional madvise() calls.
> 
> The patch defaults to pinning everything right now for performance
> evaluation.......... later I'll make sure to switch that off when
> we've coalesced on a solution for the actual RDMA transfer itself.

How well does the incremental pinning work?
The pin everything approach does not look very useful
because it conflicts with memory overcommit.

> >For the guest memory pages, sending the pages directly without
> >QemuFile (that does buffering) is better, I would suggest
> >implementing an QemuRDMAFile for this. It will have a new API for
> >the memory pages (zero copy) so instead of using qemu_put_buffer
> >we will call qemu_rdma_buffer or it can reimplement
> >qemu_put_buffer (you need to add offset). As for device state
> >which is sent in the last phase and is small you can modify the
> >current implementation. (well Paolo sent patches that are changing
> >this but I think buffering is still an option) The current
> >migration implementation copies the device state into a buffer and
> >than send the data from the buffer (QemuBufferedFile). You only
> >need to pin this buffer, and RDMA it after all the device state
> >was written into it. Regards, Orit
> I like it .......... can you help me understand - how much different
> is this design than the "QEMUFileOps" design that Paulo suggested?
> 
> Or it basically the same? ....reimplementing
> qemu_put_buffer/get_buffer() for RDMA purposes......
> 
> - Michael
>