[Qemu-discuss] FreeBSD guest causes host kernel warning

[Brian Rak]

We've been seeing an issue with FreeBSD 10 x64 guests where they 
frequently cause kernel warnings on the host.  We're using QEMU 2.0.0, 
on a variety of kernels (we've been seeing this on everything from 
2.6.32 all the way up to 3.15.5).

We're using a bridged networking setup, and the qemu command line looks 
like this:

/usr/libexec/qemu-kvm -name guest1 -S -machine 
pc-i440fx-1.7,accel=kvm,usb=off -cpu Haswell -m 2048 -realtime mlock=off 
-smp 2,sockets=2,cores=16,threads=2 -uuid 
8cb1d42c-4930-48f6-bd0f-468449aa06ed -no-user-config -nodefaults 
-chardev 
socket,id=charmonitor,path=//var/lib/libvirt/qemu/guest1.monitor,server,nowait 
-mon chardev=charmonitor,id=monitor,mode=control -rtc 
base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard 
-no-shutdown -boot strict=on -device 
piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive 
file=/dev/vmimages/guest1,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native 
-device 
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2 
-drive if=none,id=drive-ide0-1-0,readonly=on,format=raw -device 
ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 
-netdev tap,fd=27,id=hostnet0 -device 
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:78:50:aa,bus=pci.0,addr=0x3,rombar=1,romfile=/usr/share/gpxe/virtio-net.rom 
-device usb-tablet,id=input0 -vnc 127.0.0.1:18333 -k en-us -device 
cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device 
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -object 
rng-random,id=rng0,filename=/dev/random -device 
virtio-rng-pci,rng=rng0,bus=pci.0,addr=0x6 -msg timestamp=on

I do not have detailed reproduction steps here.  We have multiple unique 
hosts/guests seeing this issue, and I haven't been able to find any sort 
of pattern.  I've tried setting up a new FreeBSD guest, but have not 
been able to consistently duplicate this issue yet.  My understanding 
here is the guest is passing a packet to the host that hasn't been 
checksummed, but that requests offload.  I don't fully understand how 
this is supposed to work though, so I could be mistaken.

I have been able to find varying other reports of this in the past, but 
no one seems to have come up with any solutions (only workarounds at best):

http://thr3ads.net/freebsd-stable/2012/11/2115047-kvm-vlan-virtio-problem
http://lists.ovirt.org/pipermail/users/2014-February/021723.html
https://bugzilla.redhat.com/show_bug.cgi?id=1047693

The kernel warning looks like this:

------------[ cut here ]------------
WARNING: CPU: 4 PID: 18032 at net/core/dev.c:2233 
skb_warn_bad_offload+0xc3/0xd0()
igb: caps=(0x00000001900043b3, 0x0000000000000000) len=2934 data_len=0 
gso_size=1440 gso_type=5 ip_summed=0
Modules linked in: ipmi_devintf xt_physdev ebt_arp ebt_ip ebtable_nat 
ebtables cls_fw sch_sfq sch_htb tun kvm_intel kvm 8021q garp xfs 
libcrc32c nfnetlink_queue nfnetlink_log nfnetlink bluetooth rfkill 
bridge stp llc xt_CHECKSUM iptable_mangle xt_comment ipt_REJECT 
xt_multiport iptable_filter ip_tables ip6t_REJECT ip6table_filter 
ip6_tables ipv6 joydev iTCO_wdt iTCO_vendor_support ipmi_si 
ipmi_msghandler pcspkr i2c_i801 sg lpc_ich shpchp igb dca ptp pps_core 
hwmon ext4 jbd2 mbcache sd_mod crc_t10dif crct10dif_common video ahci 
libahci xhci_hcd ast ttm drm_kms_helper sysimgblt sysfillrect 
syscopyarea dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 18032 Comm: qemu-kvm Tainted: G        W 
3.15.5-1.el6.elrepo.x86_64 #1
Hardware name: Supermicro X10SLE-F/HF/X10SLE, BIOS 1.1a 12/03/2013
 00000000000008b9 ffff88081fd035d8 ffffffff8163ba90 00000000000008b9
 ffff88081fd03628 ffff88081fd03618 ffffffff8106c30c ffffc90007a6bfe0
 0000000000000000 ffff8807f46d0000 ffff8807f46d0000 0000000000000000
Call Trace:
 <IRQ>  [<ffffffff8163ba90>] dump_stack+0x49/0x61
 [<ffffffff8106c30c>] warn_slowpath_common+0x8c/0xc0
 [<ffffffff8106c3f6>] warn_slowpath_fmt+0x46/0x50
 [<ffffffff8156ce93>] skb_warn_bad_offload+0xc3/0xd0
 [<ffffffff81561c7f>] ? __kfree_skb+0x3f/0xc0
 [<ffffffff81574699>] __skb_gso_segment+0x89/0xe0
 [<ffffffff81574876>] dev_hard_start_xmit+0x186/0x640
 [<ffffffff81594f5a>] sch_direct_xmit+0xfa/0x1d0
 [<ffffffff81574f2f>] __dev_queue_xmit+0x1ff/0x4f0
 [<ffffffff81575240>] dev_queue_xmit+0x10/0x20
 [<ffffffffa02bd612>] br_dev_queue_push_xmit+0x82/0xb0 [bridge]
 [<ffffffffa02c5680>] br_nf_dev_queue_xmit+0x20/0x90 [bridge]
 [<ffffffffa02c64b8>] br_nf_post_routing+0x2d8/0x300 [bridge]
 [<ffffffffa02bd590>] ? deliver_clone+0x60/0x60 [bridge]
 [<ffffffff815a357e>] nf_iterate+0x8e/0xc0
 [<ffffffff815f1c9c>] ? fib_table_lookup+0x2cc/0x340
 [<ffffffffa02bd590>] ? deliver_clone+0x60/0x60 [bridge]
 [<ffffffff815a37ad>] nf_hook_slow+0x7d/0x150
 [<ffffffffa02bd590>] ? deliver_clone+0x60/0x60 [bridge]
 [<ffffffffa02c56f0>] ? br_nf_dev_queue_xmit+0x90/0x90 [bridge]
 [<ffffffffa02bdb43>] br_forward_finish+0x43/0x60 [bridge]
 [<ffffffffa02c58a8>] br_nf_forward_finish+0x1b8/0x1d0 [bridge]
 [<ffffffffa02c6178>] br_nf_forward_ip+0x3a8/0x410 [bridge]
 [<ffffffffa02bdb00>] ? br_flood_deliver+0x20/0x20 [bridge]
 [<ffffffff815a357e>] nf_iterate+0x8e/0xc0
 [<ffffffffa02bdb00>] ? br_flood_deliver+0x20/0x20 [bridge]
 [<ffffffff815a37ad>] nf_hook_slow+0x7d/0x150
 [<ffffffffa02bdb00>] ? br_flood_deliver+0x20/0x20 [bridge]
 [<ffffffffa02bd6e4>] __br_forward+0xa4/0x100 [bridge]
 [<ffffffffa02be800>] ? NF_HOOK.clone.0+0x70/0x70 [bridge]
 [<ffffffffa02bd7d6>] br_forward+0x96/0xb0 [bridge]
 [<ffffffffa02be800>] ? NF_HOOK.clone.0+0x70/0x70 [bridge]
 [<ffffffffa02be997>] br_handle_frame_finish+0x197/0x3f0 [bridge]
 [<ffffffffa02be800>] ? NF_HOOK.clone.0+0x70/0x70 [bridge]
 [<ffffffffa02c6790>] br_nf_pre_routing_finish+0x2b0/0x370 [bridge]
 [<ffffffffa02c64e0>] ? br_nf_post_routing+0x300/0x300 [bridge]
 [<ffffffffa02c4986>] NF_HOOK_THRESH+0x56/0x60 [bridge]
 [<ffffffffa02c5d2b>] br_nf_pre_routing+0x2fb/0x3a0 [bridge]
 [<ffffffff815a357e>] nf_iterate+0x8e/0xc0
 [<ffffffffa02be800>] ? NF_HOOK.clone.0+0x70/0x70 [bridge]
 [<ffffffff815a37ad>] nf_hook_slow+0x7d/0x150
 [<ffffffffa02be800>] ? NF_HOOK.clone.0+0x70/0x70 [bridge]
 [<ffffffffa02bed8c>] br_handle_frame+0x19c/0x240 [bridge]
 [<ffffffffa02bebf0>] ? br_handle_frame_finish+0x3f0/0x3f0 [bridge]
 [<ffffffff81572fa5>] __netif_receive_skb_core+0x1e5/0x620
 [<ffffffff81573407>] __netif_receive_skb+0x27/0x70
 [<ffffffff81573553>] process_backlog+0x103/0x200
 [<ffffffff81573d62>] net_rx_action+0x112/0x2a0
 [<ffffffff8107111c>] __do_softirq+0xfc/0x2b0
 [<ffffffff8164a81c>] do_softirq_own_stack+0x1c/0x30
 <EOI>  [<ffffffff81070e75>] do_softirq+0x55/0x60
 [<ffffffff81571e19>] netif_rx_ni+0x39/0x70
 [<ffffffffa04994e0>] tun_get_user+0x310/0x6c0 [tun]
 [<ffffffffa0499995>] tun_chr_aio_write+0x85/0xa0 [tun]
 [<ffffffff811ffefa>] ? fsnotify+0x1ba/0x2c0
 [<ffffffff811beb9d>] do_sync_readv_writev+0x4d/0x80
 [<ffffffff811c0128>] do_readv_writev+0xc8/0x2c0
 [<ffffffff810a5b06>] ? update_curr+0x196/0x1e0
 [<ffffffff811bebd0>] ? do_sync_readv_writev+0x80/0x80
 [<ffffffff811c0357>] vfs_writev+0x37/0x50
 [<ffffffff811c0496>] SyS_writev+0x56/0xf0
 [<ffffffff81648ee9>] system_call_fastpath+0x16/0x1b
---[ end trace 0c83a21c3f21912d ]---

Any suggestions on how I'd even begin to fix this?  The usual 
suggestions here (disable various forms of offload) don't appear to 
work, though I've only tried them from the host.  I do not have the 
ability to make changes on the guests, unless I can manage to reproduce 
this on a new guest.