Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Jason Wang]

On 01/20/2016 11:29 AM, Zhang Chen wrote:
>
>> Sure.
>>
>> Two main comments/suggestions:
>>
>> - TCP analysis is missed in current version, maybe you point a git tree
>> (or another version of RFC) to me for a better understanding of the
>> design. (Just a skeleton for TCP should be sufficient to discuss).
>> - I prefer to make the code as reusable as possible. So it's better to
>> split/decouple the reusable parts from the codes. So a vague idea is:
>>
>> 1) Decouple the packet comparing from the netfilter. You've achieved
>> this 99% since the work has been done in a thread. Just let the thread
>> poll sockets directly, then the comparing have the possibility to be
>> reused by other kinds of dataplane.
>> 2) Implement traffic mirror/redirector as filter.
>> 3) Implement TCP seq rewriting as a filter.
>>
>> Then, in primary node, you need just a traffic mirror, which did:
>> - mirror ingress traffic to secondary node
>> - mirror outgress traffic to packet comparing thread
>>
>> And in secondadry node, you need two filters:
>> - A TCP seq rewriter which adjust tcp sequence number.
>> - A traffic redirector which redirect packet from a socket as ingress
>> traffic, and redirect outgress traffic to the socket which could be
>> polled by remote packet comparing thread.
>>   Thoughts?
>>
>> Thanks
>>
>>> Thanks
>>> zhangchen
>>
>
>
> Hi, Jason.
> We consider your suggestion to split/decouple
> the reusable parts from the codes.
> Due to filter plugin are traversed one by one in order
> we will split colo-proxy to three filters in each side.
>
> But in this plan,primary and secondary both have socket
> server,startup is a problem.

I believe this issue could be solved by reusing socket chardev.

>
>
>  Primary qemu                                                      
> Secondary qemu
> +----------------------------------------------------------+      
> +-----------------------------------------------------------+
> | +-----------------------------------------------------+  |       | 
> +------------------------------------------------------+ |
> | |                                                     |  |       | 
> |                                                      | |
> | |                        guest                        |  |       | 
> |                        guest                         | |
> | |                                                     |  |       | 
> |                                                      | |
> | +-----------^--------------+--------------------------+  |       | 
> +---------------------+--------+-----------------------+ |
> |             |              |                             |      
> |                        ^        |                         |
> |             |              |                             |      
> |                        |        |                         |
> |             +-------------------------------------------------+ 
> |                        |        |                         |
> |  netfilter  |              |                             |    |  |  
> netfilter            |        |                         |
> | +-----------------------------------------------------+  |    |  | 
> +------------------------------------------------------+ |
> | |           |              |     filter excute order  |  |    |  | 
> |                     |        |  filter excute order  | |
> | |           |              |    +-------------------> |  |    |  | 
> |                     |        | +-------------------> | |
> | |           |              |                          |  |    |  | 
> |                     |        |   TCP                 | |
> | | +---------+-+     +------v-----+    +----+ +-----+  |  |    |  | 
> | +-----------+   +---+----+---v+rewriter+  +--------+ | |
> | | |           |     |            |    |            |  |  |    |  | 
> | |           |   |        |             |  |        | | |
> | | |  mirror   |     |  redirect  +---->  compare   |  |  |   
> +--------> mirror   +---> adjust |   adjust    +-->redirect| | |
> | | |  client   |     |  server    |    |            |  |  |       | 
> | |  server   |   | ack    |   seq       |  |client  | | |
> | | |           |     |            |    |            |  |  |       | 
> | |           |   |        |             |  |        | | |
> | | +----^------+     +----^-------+    +-----+------+  |  |       | 
> | +-----------+   +--------+-------------+  +----+---+ | |
> | |      |     tx          |      rx          |     rx  |  |       | 
> |            tx                        all       |  rx | |
> | +-----------------------------------------------------+  |       | 
> +------------------------------------------------------+ |
> |        |                
> +-------------------------------------------------------------------------------------------+
>       
> |
> |        |                                    |            |      
> |                                                           |
> +----------------------------------------------------------+      
> +-----------------------------------------------------------+
>          |                                    |
>          |guest receive                       |guest send
>          |                                    |
> +--------+------------------------------------v------------+
> |                                                          |
> |                                                          |
> |                         tap                             
> |                              NOTE: filter direction is rx/tx/all
> |                                                         
> |                              rx:receive packets sent to the netdev
> |                                                         
> |                              tx:receive packets sent by the netdev
> +----------------------------------------------------------+
>
>
>

I still like to decouple comparer from netfilter. It have two obvious
advantages:

- make it can be reused by other dataplane (e.g vhost)
- secondary redirector could redirect rx to comparer on primary node
directly which simplify the design.

>
>
>
>
> guest recv packet route
>
> primary
> tap --> mirror client filter
> mirror client will send packet to guest,at the
> same time, copy and forward packet to secondary
> mirror server.
>
> secondary
> mirror server filter --> TCP rewriter
> if recv packet is TCP packet,we will adjust ack
> and update TCP checksum, then send to secondary
> guest. else directly send to guest.
>
>
> guest send packet route
>
> primary
> guest --> redirect server filter
> redirect server filter recv primary guest packet
> but do nothing, just pass to next filter.
>
> redirect server filter --> compare filter
> compare filter recv primary guest packet then
> waiting scondary redirect packet to compare it.
> if packet same,send primary packet and clear secondary
> packet, else send primary packet and do
> checkpoint.
>
> secondary
> guest --> TCP rewriter filter
> if the packet is TCP packet,we will adjust seq
> and update TCP checksum. then send it to
> redirect client filter. else directly send to
> redirect client filter.
>
> redirect client filter --> redirect server filter
> forward packet to primary
>
>
> In failover scene（primary is down）, the TCP rewriter will keep
> servicing
> for the TCP connection which is established after the last checkpoint。
>
>
>
> How about this plan?

Sounds good.

And there's indeed no need to differ client/server by reusing the socket
chardev. E.g:

In primary node:

...
-chardev socket,id=comparer0,host=ip_primary,port=X,server,nowait
-chardev socket,id=comparer1,host=ip_primary,port=Y,server,nowait
-chardev socket,id=mirrorer0,host=ip_primary,port=Z,server,nowait
-netdev tap,id=hn0
-traffic-mirrorer netdev=hn0,id=t0,indev=comparer0,outdev=mirrorer0
-colo-comparer primary_traffic=comparer0,secondary_traffic=comparer1
...

packet comparer compares the packets from two chardev: comparer0 and
comparer1.
traffic-mirrorer mirror tx to secondary node through chardev mirrorer0,
and mirror rx to packet comparer through chardev comparer0.

In secondary node:

...
-chardev socket,id=redirector0,host=ip_primary,port=Y
-chardev socket,id=redirector1,host=ip_primary,port=Z
-netdev tap,id=hn0
-traffic-redirector netdev=hn0,id,r0,indev=redirector0,outdev=redirector1
-colo-rewriter netdev=hn0,id=c0
...

traffic-redirector redirect the rx traffic from primary node through
redirector0 and redirect the tx traffic to promary node through redirector1.
colo-rewriter rewrite seq number as a normal netfilter.



>
>
>> .
>>
>