Re: [Qemu-devel] [PATCH v2 1/2] net: introduce MAC_TABLE_CHANGED event

[Amos Kong]

On Fri, May 31, 2013 at 11:02:54AM +0800, Amos Kong wrote:
> On Fri, May 31, 2013 at 08:35:28AM +0800, Amos Kong wrote:
> > On Thu, May 30, 2013 at 04:54:41PM +0300, Michael S. Tsirkin wrote:
> > > On Tue, May 28, 2013 at 08:25:56AM -0400, Luiz Capitulino wrote:


> > > > > > >  2. Can you actually show the problem does exist so that we 
> > > > > > > ensure this is
> > > > > > >     not premature optimization? Might be a good idea to have this 
> > > > > > > in the
> > > > > > >     commit log
> > > > > > > 
> > > > > > > > > (which is to couple the event with the query command) is
> > > > > > > > > appropriate. We're in user-space already, many things could 
> > > > > > > > > slow
> > > > > > > > > the guest down apart from the event generation.
> > > > > > > > > 
> > > > > > > > > Two questions:
> > > > > > > > > 
> > > > > > > > >  1. Do we know how slow (or how many packets are actually 
> > > > > > > > > dropped)
> > > > > > > > >     if the mac is changed too often *and* the event is always 
> > > > > > > > > sent?
> > > > > > > > 
> > > > > > > > We always disable interface first, then change the macaddr.
> > > > > > > > But we just have patch to allow guest to change macaddr of
> > > > > > > > virtio-net when the interface is running.
> > > > > > > > 
> > > > > > > > | commit 2dcd0cce551983afe2f900125457f10bb5d980ae
> > > > > > > > | Author: Jiri Pirko <address@hidden>
> > > > > > > > | Date:   Tue Dec 11 15:33:56 2012 -0500
> > > > > > > > | 
> > > > > > > > |     [virt] virtio_net: allow to change mac when iface is 
> > > > > > > > running
> > > > > > > > 
> > > > > > > > >  2. Does this solution consider what happens if the QMP 
> > > > > > > > > client does
> > > > > > > > >     respond timely to the event by issuing the query-rx-filter
> > > > > > > > >     command?
> > > > > > > > 
> > > > > > > > We assume that the QMP client (management) cares about the mac 
> > > > > > > > changing
> > > > > > > > event, and will query the latest rx-filter state and sync to 
> > > > > > > > macvtap
> > > > > > > > device.
> > > > > > > > 
> > > > > > > > 1) If QMP client respond timely to the event: that's what we 
> > > > > > > > expected :)
> > > > > > > 
> > > > > > > Won't this slow down the guest? If not, why?
> > > > > 
> > > > > If guest changes fx-filter configs frequently & management always 
> > > > > query the
> > > > > event very timely, this will slow down the guest.
> > > > > 
> > > > > We should detect & process the abnormal behavior from management.
> > > > 
> > > > That's not abnormal. Management is doing what it should do.
> > > > 
> > > > Maybe using the event throttle API can solve the mngt side of the 
> > > > problem,
> > > > but I still think we need a reproducible test-case to ensure we're doing
> > > > the right thing.
> > > 
> > > I agree we should make sure this code is tested.
> > > It's pretty easy: run ifconfig in a loop in guest.
> > > 
> > > Amos, did you try this? Probably should otherwise
> > > we don't really know whether the logic works.
> > 
> > With v4 patch (without using event throttle)
> > 
> > 1. continually query rx-filter from monitor
> > # while true; do echo "info rx-filter" | nc -U /tmp/m; done
> > 
> > 2. change mac in guest repeatedly
> > # while true; do
> >  ifconfig eth1 down; ifconfig eth1 hw ether 12:00:00:00:00:00
> >  ifconfig eth1 down; ifconfig eth1 hw ether 14:00:00:00:00:00
> >  done
> > 
> > One time (down if, change mac, up) takes about 3,500,000 ns in guest
> > some event will be ignored by qemu. I will try to only query when it
> > gets NIC_RX_FILTER_CHANGE event from QMP monitor, query ASAP.
> > 
> > ==> Resource usage:
> > 
> >   PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
> > 16387 root      20   0 2375m 326m 6684 R 104.2  4.2   8:32.16 
> > qemu-system-x86
> > 
> > loop script takes about 10% guest cpu (1 core), guest is not slow
> > 
> > ----
> > If we don't use the flag (same effect as that management taks 0 ns to
> > response & complete the query after event comming)
> 
> 
> >   PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND       
> >                                                                             
> >                          
> >  4317 root      20   0 2377m 285m 6664 R 103.4  3.7   2:06.82 
> > qemu-system-x86
> > 
> > guest is very slow (no response for the keyboard input), output:
> > clocksource tsc unstable.
> 
> ^^  The reproduce rate : about 10%
> 
> I did the following tests, it seems repeatedly executing vq command
> with large data will cause guest hung.
> 
> Could not slow guest by repeatedly changing rx-filter (reference test1).

I did more tests in clear environment, and found that the guest hang/slow
(no response from monitor) is caused by flooding events. I could not
reproduce it with upstream qemu [1]

If I set event_throttle to 1 ~ 1000, the problem doesn't occur.

It's easier to reproduce this problem by changing vlan config,
not because it passes more data with VQ cmd, but it will cause more
events.


In this case, we can set event_throttle to 1 for _RX_FILTER_CHANGED
event to avoid it slows guest. The 1 ms delay should be acceptabled?

And we can continually use that flag to reduce the events.


[1] 6a4e17711442849bf2cc731ccddef5a2a2d92d29 (Sun Apr 14 18:10:28 2013)

> NOTE: Test1,2,3,4 didn't use control flag, emit all the events to monitor.
> 
> ==> test1 (applied v4)
> I tried to test by repeatedly change promisc mode (on/off).
> The difference is that there is no data passed when it
> executes vq command. Changing main mac will pass 1 macaddr (6 bytes).
> 
> Result: not reproduce, events emitted to QMP monitor, everything is fine.
> 
> ==> test2 (applied v4)
> So I tried to test by repeatedly change mac of vlan interface.
> it will cause guest update mac-table every time, more data will
> be passed. 
> 
> Result: reproduced (guest becomes slow, stops emit event)
> 
> ==> test3 (latest qemu)
> execute test2 with latest qemu
> 
> Result: guest hung

could not reproduce now with [1]

> ==> test4 (applied v4, + event_throttle change)
> set event_throttle for rxfilter event in 
> monitor.c:monitor_protocol_event_init()
>   monitor_protocol_event_throttle(QEVENT_NIC_RX_FILTER_CHANGED, 1000);
> and execute test2.
> 
> Result: guest script stuck, guest gets slow, no event emitted

could not reproduce now with [1]
 
> | Guest) # strace ifconfig
> | ...
> | ioctrl(4, SIOCGIFCONF, {
> | (stuck...)

-- 
                        Amos.