Re: [Qemu-devel] An RDMA race?

[Michael R. Hines]

David,

Thanks for including my email directly. It helps a lot.

Below, I'm going to assume that only "dest" is calling qemu_rdma_exchange_recv()

and only src is calling qemu_rdma_exchange_send(), since you didn't specify who

is sending and who is receiving.

If that assumption is wrong, please respond again.

Comments inline.....

On Sat, Dec 12, 2015 at 1:48 AM, Dr. David Alan Gilbert <address@hidden> wrote:

Hi Michael,
   I think I've got an RDMA race condition, but I'm being a little
cautious at the moment and wondered if you agree with the following
diagnosis.

It's showing up in a world of mine that's sending more control messages
from the destination->source and I'm seeing the following.

We normally expect:

   src                        dest
     ----------->control ready->

If src is sending, this is not correct. Dest should send the ready message if it is receiving, not src, which breaks the above assumption. So, I'll reverse the assumption previously and continue with your observation and assume that src is receiving instead of dest, which should instead look like:

src  (receiving)                      dest (sending)
     ----------->control ready->

 

   Sees SEND_CONTROL signal to ack that it has been sent

I'll assume here that you meant that dest sees the ready message and is then later sends something.

 

         <-----control message--
   Sees RECV_CONTROL message from dest

Similar assumption for the receiver (src).

 

but what I'm seeing is:
   src                        dest
     ----------->control ready->
         <-----control message--
   Sees RECV_CONTROL message from dest

hmmmmm....

 

   Sees SEND_CONTROL signal to ack that it has been sent

There's not enough information here....... do you have a multi-threaded send or receive or something?

Do the work request IDs match up?

- Michael