Re: [PATCH-for-9.1 v2 2/3] migration: Remove RDMA protocol handling

[Jinpu Wang]

Hi Peter, Hi Chuan,

On Thu, May 9, 2024 at 4:14 PM Peter Xu <peterx@redhat.com> wrote:
>
> On Thu, May 09, 2024 at 04:58:34PM +0800, Zheng Chuan via wrote:
> > That's a good news to see the socket abstraction for RDMA!
> > When I was developed the series above, the most pain is the RDMA migration 
> > has no QIOChannel abstraction and i need to take a 'fake channel'
> > for it which is awkward in code implementation.
> > So, as far as I know, we can do this by
> > i. the first thing is that we need to evaluate the rsocket is good enough 
> > to satisfy our QIOChannel fundamental abstraction
> > ii. if it works right, then we will continue to see if it can give us 
> > opportunity to hide the detail of rdma protocol
> >     into rsocket by remove most of code in rdma.c and also some hack in 
> > migration main process.
> > iii. implement the advanced features like multi-fd and multi-uri for rdma 
> > migration.
> >
> > Since I am not familiar with rsocket, I need some times to look at it and 
> > do some quick verify with rdma migration based on rsocket.
> > But, yes, I am willing to involved in this refactor work and to see if we 
> > can make this migration feature more better:）
>
> Based on what we have now, it looks like we'd better halt the deprecation
> process a bit, so I think we shouldn't need to rush it at least in 9.1
> then, and we'll need to see how it goes on the refactoring.
>
> It'll be perfect if rsocket works, otherwise supporting multifd with little
> overhead / exported APIs would also be a good thing in general with
> whatever approach.  And obviously all based on the facts that we can get
> resources from companies to support this feature first.
>
> Note that so far nobody yet compared with rdma v.s. nic perf, so I hope if
> any of us can provide some test results please do so.  Many people are
> saying RDMA is better, but I yet didn't see any numbers comparing it with
> modern TCP networks.  I don't want to have old impressions floating around
> even if things might have changed..  When we have consolidated results, we
> should share them out and also reflect that in QEMU's migration docs when a
> rdma document page is ready.
I also did a tests with Mellanox ConnectX-6 100 G RoCE nic, the
results are mixed, for less than 3 streams native ethernet is faster,
and when more than 3 streams rsocket performs better.

root@x4-right:~# iperf -c 1.1.1.16 -P 1
------------------------------------------------------------
Client connecting to 1.1.1.16, TCP port 5001
TCP window size:  325 KByte (default)
------------------------------------------------------------
[  3] local 1.1.1.15 port 44214 connected with 1.1.1.16 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3] 0.0000-10.0000 sec  52.9 GBytes  45.4 Gbits/sec
root@x4-right:~# iperf -c 1.1.1.16 -P 2
[  3] local 1.1.1.15 port 33118 connected with 1.1.1.16 port 5001
[  4] local 1.1.1.15 port 33130 connected with 1.1.1.16 port 5001
------------------------------------------------------------
Client connecting to 1.1.1.16, TCP port 5001
TCP window size: 4.00 MByte (default)
------------------------------------------------------------
[ ID] Interval       Transfer     Bandwidth
[  3] 0.0000-10.0001 sec  45.0 GBytes  38.7 Gbits/sec
[  4] 0.0000-10.0000 sec  43.9 GBytes  37.7 Gbits/sec
[SUM] 0.0000-10.0000 sec  88.9 GBytes  76.4 Gbits/sec
[ CT] final connect times (min/avg/max/stdev) =
0.172/0.189/0.205/0.172 ms (tot/err) = 2/0
root@x4-right:~# iperf -c 1.1.1.16 -P 4
------------------------------------------------------------
Client connecting to 1.1.1.16, TCP port 5001
TCP window size:  325 KByte (default)
------------------------------------------------------------
[  5] local 1.1.1.15 port 50748 connected with 1.1.1.16 port 5001
[  4] local 1.1.1.15 port 50734 connected with 1.1.1.16 port 5001
[  6] local 1.1.1.15 port 50764 connected with 1.1.1.16 port 5001
[  3] local 1.1.1.15 port 50730 connected with 1.1.1.16 port 5001
[ ID] Interval       Transfer     Bandwidth
[  6] 0.0000-10.0000 sec  24.7 GBytes  21.2 Gbits/sec
[  3] 0.0000-10.0004 sec  23.6 GBytes  20.3 Gbits/sec
[  4] 0.0000-10.0000 sec  27.8 GBytes  23.9 Gbits/sec
[  5] 0.0000-10.0000 sec  28.0 GBytes  24.0 Gbits/sec
[SUM] 0.0000-10.0000 sec   104 GBytes  89.4 Gbits/sec
[ CT] final connect times (min/avg/max/stdev) =
0.104/0.156/0.204/0.124 ms (tot/err) = 4/0
root@x4-right:~# iperf -c 1.1.1.16 -P 8
[  4] local 1.1.1.15 port 55588 connected with 1.1.1.16 port 5001
[  5] local 1.1.1.15 port 55600 connected with 1.1.1.16 port 5001
------------------------------------------------------------
Client connecting to 1.1.1.16, TCP port 5001
TCP window size:  325 KByte (default)
------------------------------------------------------------
[ 10] local 1.1.1.15 port 55628 connected with 1.1.1.16 port 5001
[ 15] local 1.1.1.15 port 55648 connected with 1.1.1.16 port 5001
[  7] local 1.1.1.15 port 55620 connected with 1.1.1.16 port 5001
[  3] local 1.1.1.15 port 55584 connected with 1.1.1.16 port 5001
[ 14] local 1.1.1.15 port 55644 connected with 1.1.1.16 port 5001
[  6] local 1.1.1.15 port 55610 connected with 1.1.1.16 port 5001
[ ID] Interval       Transfer     Bandwidth
[  6] 0.0000-10.0015 sec  8.47 GBytes  7.27 Gbits/sec
[  4] 0.0000-10.0011 sec  8.62 GBytes  7.40 Gbits/sec
[  7] 0.0000-10.0000 sec  18.1 GBytes  15.5 Gbits/sec
[ 14] 0.0000-10.0000 sec  8.69 GBytes  7.46 Gbits/sec
[  5] 0.0000-10.0006 sec  18.5 GBytes  15.9 Gbits/sec
[ 10] 0.0000-10.0006 sec  16.1 GBytes  13.9 Gbits/sec
[  3] 0.0000-10.0000 sec  17.1 GBytes  14.6 Gbits/sec
[ 15] 0.0000-10.0016 sec  8.54 GBytes  7.34 Gbits/sec
[SUM] 0.0000-10.0017 sec   104 GBytes  89.4 Gbits/sec
[ CT] final connect times (min/avg/max/stdev) =
0.049/0.095/0.213/0.062 ms (tot/err) = 8/0

root@x4-right:~#
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/rsocket/librspreload.so  iperf -c
1.1.1.16 -P 1
------------------------------------------------------------
Client connecting to 1.1.1.16, TCP port 5001
TCP window size:  128 KByte (default)
------------------------------------------------------------
[  3] local 1.1.1.15 port 45596 connected with 1.1.1.16 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3] 0.0000-10.0000 sec  37.8 GBytes  32.5 Gbits/sec
root@x4-right:~#
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/rsocket/librspreload.so  iperf -c
1.1.1.16 -P 2
[  4] local 1.1.1.15 port 46782 connected with 1.1.1.16 port 5001
------------------------------------------------------------
Client connecting to 1.1.1.16, TCP port 5001
TCP window size:  128 KByte (default)
------------------------------------------------------------
[  3] local 1.1.1.15 port 43237 connected with 1.1.1.16 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4] 0.0000-10.0000 sec  37.5 GBytes  32.2 Gbits/sec
[  3] 0.0000-10.0000 sec  40.7 GBytes  34.9 Gbits/sec
[SUM] 0.0000-10.0000 sec  78.2 GBytes  67.2 Gbits/sec
[ CT] final connect times (min/avg/max/stdev) =
5.819/6.579/7.340/7.340 ms (tot/err) = 2/0
root@x4-right:~#
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/rsocket/librspreload.so  iperf -c
1.1.1.16 -P 4
[  4] local 1.1.1.15 port 60385 connected with 1.1.1.16 port 5001
[  7] local 1.1.1.15 port 55203 connected with 1.1.1.16 port 5001
[  6] local 1.1.1.15 port 35084 connected with 1.1.1.16 port 5001
------------------------------------------------------------
Client connecting to 1.1.1.16, TCP port 5001
TCP window size:  128 KByte (default)
------------------------------------------------------------
[  3] local 1.1.1.15 port 37253 connected with 1.1.1.16 port 5001
[ ID] Interval       Transfer     Bandwidth
[  6] 0.0000-10.0000 sec  28.4 GBytes  24.4 Gbits/sec
[  4] 0.0000-10.0000 sec  28.3 GBytes  24.3 Gbits/sec
[  7] 0.0000-10.0000 sec  28.4 GBytes  24.4 Gbits/sec
[  3] 0.0000-10.0001 sec  28.2 GBytes  24.3 Gbits/sec
[SUM] 0.0000-10.0001 sec   113 GBytes  97.3 Gbits/sec
[ CT] final connect times (min/avg/max/stdev) =
5.311/7.579/10.019/4.165 ms (tot/err) = 4/0
root@x4-right:~#
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/rsocket/librspreload.so  iperf -c
1.1.1.16 -P 8
[  8] local 1.1.1.15 port 33684 connected with 1.1.1.16 port 5001
[ 10] local 1.1.1.15 port 40620 connected with 1.1.1.16 port 5001
------------------------------------------------------------
Client connecting to 1.1.1.16, TCP port 5001
TCP window size:  128 KByte (default)
------------------------------------------------------------
[  3] local 1.1.1.15 port 56988 connected with 1.1.1.16 port 5001
[  4] local 1.1.1.15 port 51139 connected with 1.1.1.16 port 5001
[ 12] local 1.1.1.15 port 44712 connected with 1.1.1.16 port 5001
[  5] local 1.1.1.15 port 50838 connected with 1.1.1.16 port 5001
[  6] local 1.1.1.15 port 51334 connected with 1.1.1.16 port 5001
[  9] local 1.1.1.15 port 40611 connected with 1.1.1.16 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3] 0.0000-10.0000 sec  13.8 GBytes  11.9 Gbits/sec
[  5] 0.0000-10.0001 sec  13.9 GBytes  11.9 Gbits/sec
[ 12] 0.0000-10.0001 sec  13.8 GBytes  11.9 Gbits/sec
[ 10] 0.0000-10.0001 sec  13.9 GBytes  11.9 Gbits/sec
[  9] 0.0000-10.0000 sec  13.8 GBytes  11.9 Gbits/sec
[  6] 0.0000-10.0000 sec  13.9 GBytes  11.9 Gbits/sec
[  8] 0.0000-10.0000 sec  13.8 GBytes  11.9 Gbits/sec
[  4] 0.0000-10.0001 sec  13.8 GBytes  11.9 Gbits/sec
[SUM] 0.0000-10.0001 sec   111 GBytes  95.1 Gbits/sec
[ CT] final connect times (min/avg/max/stdev) =
5.973/10.699/15.943/4.251 ms (tot/err) = 8/0
root@x4-right:~#
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/rsocket/librspreload.so  iperf -c
1.1.1.16 -P 1
------------------------------------------------------------
Client connecting to 1.1.1.16, TCP port 5001
TCP window size:  128 KByte (default)
------------------------------------------------------------
[  3] local 1.1.1.15 port 36960 connected with 1.1.1.16 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3] 0.0000-10.0000 sec  41.1 GBytes  35.3 Gbits/sec
root@x4-right:~#
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/rsocket/librspreload.so  iperf -c
1.1.1.16 -P 2
[  3] local 1.1.1.15 port 32799 connected with 1.1.1.16 port 5001
------------------------------------------------------------
Client connecting to 1.1.1.16, TCP port 5001
TCP window size:  128 KByte (default)
------------------------------------------------------------
[  4] local 1.1.1.15 port 35912 connected with 1.1.1.16 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4] 0.0000-10.0000 sec  36.6 GBytes  31.4 Gbits/sec
[  3] 0.0000-10.0000 sec  36.6 GBytes  31.4 Gbits/sec
[SUM] 0.0000-10.0000 sec  73.2 GBytes  62.9 Gbits/sec
[ CT] final connect times (min/avg/max/stdev) =
5.172/5.842/6.512/6.512 ms (tot/err) = 2/0
root@x4-right:~#
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/rsocket/librspreload.so  iperf -c
1.1.1.16 -P 4
[  4] local 1.1.1.15 port 53311 connected with 1.1.1.16 port 5001
------------------------------------------------------------
Client connecting to 1.1.1.16, TCP port 5001
TCP window size:  128 KByte (default)
------------------------------------------------------------
[  3] local 1.1.1.15 port 37243 connected with 1.1.1.16 port 5001
[  7] local 1.1.1.15 port 60801 connected with 1.1.1.16 port 5001
[  6] local 1.1.1.15 port 49694 connected with 1.1.1.16 port 5001
[ ID] Interval       Transfer     Bandwidth
[  6] 0.0000-10.0000 sec  28.2 GBytes  24.2 Gbits/sec
[  7] 0.0000-10.0000 sec  28.2 GBytes  24.3 Gbits/sec
[  3] 0.0000-10.0000 sec  28.2 GBytes  24.2 Gbits/sec
[  4] 0.0000-10.0000 sec  28.2 GBytes  24.2 Gbits/sec
[SUM] 0.0000-10.0000 sec   113 GBytes  96.9 Gbits/sec
[ CT] final connect times (min/avg/max/stdev) =
5.570/7.762/10.045/4.265 ms (tot/err) = 4/0
root@x4-right:~#


>
> Chuan, please check the whole thread discussion, it may help to understand
> what we are looking for on rdma migrations [1].  Meanwhile please feel free
> to sync with Jinpu's team and see how to move forward with such a project.
We are happy to work with community to improve rdma migration.

>
> [1] https://lore.kernel.org/qemu-devel/87frwatp7n.fsf@suse.de/
>
> Thanks,
Regards!
>
> --
> Peter Xu
>