Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Wen Congyang]

On 01/20/2016 06:19 PM, Jason Wang wrote:
> 
> 
> On 01/20/2016 06:01 PM, Wen Congyang wrote:
>> On 01/20/2016 02:54 PM, Jason Wang wrote:
>>>
>>> On 01/20/2016 11:29 AM, Zhang Chen wrote:
>>>>> Sure.
>>>>>
>>>>> Two main comments/suggestions:
>>>>>
>>>>> - TCP analysis is missed in current version, maybe you point a git tree
>>>>> (or another version of RFC) to me for a better understanding of the
>>>>> design. (Just a skeleton for TCP should be sufficient to discuss).
>>>>> - I prefer to make the code as reusable as possible. So it's better to
>>>>> split/decouple the reusable parts from the codes. So a vague idea is:
>>>>>
>>>>> 1) Decouple the packet comparing from the netfilter. You've achieved
>>>>> this 99% since the work has been done in a thread. Just let the thread
>>>>> poll sockets directly, then the comparing have the possibility to be
>>>>> reused by other kinds of dataplane.
>>>>> 2) Implement traffic mirror/redirector as filter.
>>>>> 3) Implement TCP seq rewriting as a filter.
>>>>>
>>>>> Then, in primary node, you need just a traffic mirror, which did:
>>>>> - mirror ingress traffic to secondary node
>>>>> - mirror outgress traffic to packet comparing thread
>>>>>
>>>>> And in secondadry node, you need two filters:
>>>>> - A TCP seq rewriter which adjust tcp sequence number.
>>>>> - A traffic redirector which redirect packet from a socket as ingress
>>>>> traffic, and redirect outgress traffic to the socket which could be
>>>>> polled by remote packet comparing thread.
>>>>>   Thoughts?
>>>>>
>>>>> Thanks
>>>>>
>>>>>> Thanks
>>>>>> zhangchen
>>>>
>>>> Hi, Jason.
>>>> We consider your suggestion to split/decouple
>>>> the reusable parts from the codes.
>>>> Due to filter plugin are traversed one by one in order
>>>> we will split colo-proxy to three filters in each side.
>>>>
>>>> But in this plan,primary and secondary both have socket
>>>> server,startup is a problem.
>>> I believe this issue could be solved by reusing socket chardev.
>>>
>>>>
>>>>  Primary qemu                                                      
>>>> Secondary qemu
>>>> +----------------------------------------------------------+      
>>>> +-----------------------------------------------------------+
>>>> | +-----------------------------------------------------+  |       | 
>>>> +------------------------------------------------------+ |
>>>> | |                                                     |  |       | 
>>>> |                                                      | |
>>>> | |                        guest                        |  |       | 
>>>> |                        guest                         | |
>>>> | |                                                     |  |       | 
>>>> |                                                      | |
>>>> | +-----------^--------------+--------------------------+  |       | 
>>>> +---------------------+--------+-----------------------+ |
>>>> |             |              |                             |      
>>>> |                        ^        |                         |
>>>> |             |              |                             |      
>>>> |                        |        |                         |
>>>> |             +-------------------------------------------------+ 
>>>> |                        |        |                         |
>>>> |  netfilter  |              |                             |    |  |  
>>>> netfilter            |        |                         |
>>>> | +-----------------------------------------------------+  |    |  | 
>>>> +------------------------------------------------------+ |
>>>> | |           |              |     filter excute order  |  |    |  | 
>>>> |                     |        |  filter excute order  | |
>>>> | |           |              |    +-------------------> |  |    |  | 
>>>> |                     |        | +-------------------> | |
>>>> | |           |              |                          |  |    |  | 
>>>> |                     |        |   TCP                 | |
>>>> | | +---------+-+     +------v-----+    +----+ +-----+  |  |    |  | 
>>>> | +-----------+   +---+----+---v+rewriter+  +--------+ | |
>>>> | | |           |     |            |    |            |  |  |    |  | 
>>>> | |           |   |        |             |  |        | | |
>>>> | | |  mirror   |     |  redirect  +---->  compare   |  |  |   
>>>> +--------> mirror   +---> adjust |   adjust    +-->redirect| | |
>>>> | | |  client   |     |  server    |    |            |  |  |       | 
>>>> | |  server   |   | ack    |   seq       |  |client  | | |
>>>> | | |           |     |            |    |            |  |  |       | 
>>>> | |           |   |        |             |  |        | | |
>>>> | | +----^------+     +----^-------+    +-----+------+  |  |       | 
>>>> | +-----------+   +--------+-------------+  +----+---+ | |
>>>> | |      |     tx          |      rx          |     rx  |  |       | 
>>>> |            tx                        all       |  rx | |
>>>> | +-----------------------------------------------------+  |       | 
>>>> +------------------------------------------------------+ |
>>>> |        |                
>>>> +-------------------------------------------------------------------------------------------+
>>>>       
>>>> |
>>>> |        |                                    |            |      
>>>> |                                                           |
>>>> +----------------------------------------------------------+      
>>>> +-----------------------------------------------------------+
>>>>          |                                    |
>>>>          |guest receive                       |guest send
>>>>          |                                    |
>>>> +--------+------------------------------------v------------+
>>>> |                                                          |
>>>> |                                                          |
>>>> |                         tap                             
>>>> |                              NOTE: filter direction is rx/tx/all
>>>> |                                                         
>>>> |                              rx:receive packets sent to the netdev
>>>> |                                                         
>>>> |                              tx:receive packets sent by the netdev
>>>> +----------------------------------------------------------+
>>>>
>>>>
>>>>
>>> I still like to decouple comparer from netfilter. It have two obvious
>>> advantages:
>>>
>>> - make it can be reused by other dataplane (e.g vhost)
>>> - secondary redirector could redirect rx to comparer on primary node
>>> directly which simplify the design.
>>>
>>>>
>>>>
>>>>
>>>> guest recv packet route
>>>>
>>>> primary
>>>> tap --> mirror client filter
>>>> mirror client will send packet to guest,at the
>>>> same time, copy and forward packet to secondary
>>>> mirror server.
>>>>
>>>> secondary
>>>> mirror server filter --> TCP rewriter
>>>> if recv packet is TCP packet,we will adjust ack
>>>> and update TCP checksum, then send to secondary
>>>> guest. else directly send to guest.
>>>>
>>>>
>>>> guest send packet route
>>>>
>>>> primary
>>>> guest --> redirect server filter
>>>> redirect server filter recv primary guest packet
>>>> but do nothing, just pass to next filter.
>>>>
>>>> redirect server filter --> compare filter
>>>> compare filter recv primary guest packet then
>>>> waiting scondary redirect packet to compare it.
>>>> if packet same,send primary packet and clear secondary
>>>> packet, else send primary packet and do
>>>> checkpoint.
>>>>
>>>> secondary
>>>> guest --> TCP rewriter filter
>>>> if the packet is TCP packet,we will adjust seq
>>>> and update TCP checksum. then send it to
>>>> redirect client filter. else directly send to
>>>> redirect client filter.
>>>>
>>>> redirect client filter --> redirect server filter
>>>> forward packet to primary
>>>>
>>>>
>>>> In failover scene（primary is down）, the TCP rewriter will keep
>>>> servicing
>>>> for the TCP connection which is established after the last checkpoint。
>>>>
>>>>
>>>>
>>>> How about this plan?
>>> Sounds good.
>>>
>>> And there's indeed no need to differ client/server by reusing the socket
>>> chardev. E.g:
>>>
>>> In primary node:
>>>
>>> ...
>>> -chardev socket,id=comparer0,host=ip_primary,port=X,server,nowait
>>> -chardev socket,id=comparer1,host=ip_primary,port=Y,server,nowait
>>> -chardev socket,id=mirrorer0,host=ip_primary,port=Z,server,nowait
>>> -netdev tap,id=hn0
>>> -traffic-mirrorer netdev=hn0,id=t0,indev=comparer0,outdev=mirrorer0
>>> -colo-comparer primary_traffic=comparer0,secondary_traffic=comparer1
>> Why mirrorer has indev? 
> 
> 
> As I said in the previous mails. I would like to decouple packet
> comparing from netfilter. You've already done most of this since the
> comparing is done in an independent thread. So the indev here is to
> mirror the packet sent by guest to the packet comparing thread.
> 
>> I think we can use traffic-redirector to do it.
>> The command line is:
>> -netdev tap,id=hn0
>> -object traffic-mirrorer,id=f0,netdev=hn0,queue=tx,outdev=mirrorer0
>> -object traffic-redirector,id=f1,netdev=hn0,queue=rx,outdev=comparer0
>> -colo-comparer 
>> primary_traffic=comparer0,secondary_traffic=comparer1,netdev=hn0
>> In the comparer thread, we can use qemu_net_queue_send_iov() to send
>> out the packet.
>>
>> Also, we can merge the socketdev comparer1 and mirrorer0.
> 
> It depends on whether or not packet comparing was done in a net filter
> (which I prefer not).

I mean that: packet comapring is done in a thread, not a net filter.
The flow of the packet sent from guest:
1. traffice-redirecotr, we will redirector the packet to comparer0, the next
   filter will never see it.
2. comparing thread: read it from socket chardev comparer0
3. call qemu_net_queue_send_iov() to send it back to the netdev.

Thanks
Wen Congyang

> 
>>
>> Thanks
>> Wen Congyang
>>
>>> ...
>>>
>>> packet comparer compares the packets from two chardev: comparer0 and
>>> comparer1.
>>> traffic-mirrorer mirror tx to secondary node through chardev mirrorer0,
>>> and mirror rx to packet comparer through chardev comparer0.
>>>
>>> In secondary node:
>>>
>>> ...
>>> -chardev socket,id=redirector0,host=ip_primary,port=Y
>>> -chardev socket,id=redirector1,host=ip_primary,port=Z
>>> -netdev tap,id=hn0
>>> -traffic-redirector netdev=hn0,id,r0,indev=redirector0,outdev=redirector1
>>> -colo-rewriter netdev=hn0,id=c0
>>> ...
>>>
>>> traffic-redirector redirect the rx traffic from primary node through
>>> redirector0 and redirect the tx traffic to promary node through redirector1.
>>> colo-rewriter rewrite seq number as a normal netfilter.
>>>
>>>
>>>
>>>>
>>>>> .
>>>>>
>>>
>>>
>>>
>>> .
>>>
>>
>>
> 
> 
> 
> .
>