Re: [Qemu-devel] [PATCH v9 05/11] vfio: add check host bus reset is support or not

[Dou Liyang]

Hi Alex,

At 09/01/2016 10:12 AM, Alex Williamson wrote:

[...]

> I had to move to a different system where I could actually inject an
> aer error and created a config similar to above but with the 82576
> ports downstream of the ioh3420 root port.  When I inject a malformed
> TLP uncorrectable error, my RHEL7.2 guest does this:
>
> [   35.995645] pcieport 0000:00:1c.0: AER: Multiple Uncorrected (Fatal) error 
> received: id=0200
> [   35.998483] igb 0000:02:00.0: PCIe Bus Error: severity=Uncorrected (Fatal), 
> type=Unaccessible, id=0200(Unregistered Agent ID)
> [   36.001965] igb 0000:02:00.0 enp2s0f0: PCIe link lost, device now detached
> [   36.015092] igb 0000:02:00.1 enp2s0f1: PCIe link lost, device now detached
> [   39.133185] igb 0000:02:00.0: enabling device (0000 -> 0002)
> [   40.071245] igb 0000:02:00.1: enabling device (0000 -> 0002)
> [   41.014451] BUG: unable to handle kernel paging request at 0000000000003818
> [   41.015969] IP: [<ffffffffa02b438d>] igb_configure_tx_ring+0x14d/0x280 [igb]
> [   41.017507] PGD 367e2067 PUD 7ae56067 PMD 0
> [   41.018497] Oops: 0002 [#1] SMP
> [   41.019242] Modules linked in: ip6t_rpfilter ip6t_REJECT ipt_REJECT 
> xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables 
> ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle 
> ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat 
> nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle 
> iptable_security iptable_raw iptable_filter snd_hda_codec_generic snd_hda_intel 
> snd_hda_codec ppdev snd_hda_core snd_hwdep snd_seq snd_seq_device iTCO_wdt 
> iTCO_vendor_support bochs_drm snd_pcm syscopyarea sysfillrect sysimgblt ttm 
> virtio_balloon snd_timer snd igb drm_kms_helper soundcore ptp pps_core 
> i2c_algo_bit i2c_i801 dca drm shpchp lpc_ich mfd_core pcspkr i2c_core 
> parport_pc parport ip_tables xfs libcrc32c virtio_blk virtio_console virtio_net 
> ahci libahci crc32c_intel serio_raw libata virtio_pci virtio_ring virtio 
> dm_mirror dm_region_hash dm_log dm_mod
> [   41.040590] CPU: 0 PID: 29 Comm: kworker/0:1 Not tainted 
> 3.10.0-327.el7.x86_64 #1
> [   41.042180] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 
> rel-1.9.3-0-ge2fc41e-prebuilt.qemu-project.org 04/01/2014
> [   41.044635] Workqueue: events aer_isr
> [   41.045478] task: ffff880179435080 ti: ffff880179680000 task.ti: 
> ffff880179680000
> [   41.047097] RIP: 0010:[<ffffffffa02b438d>]  [<ffffffffa02b438d>] 
> igb_configure_tx_ring+0x14d/0x280 [igb]
> [   41.049151] RSP: 0018:ffff880179683bf8  EFLAGS: 00010246
> [   41.050260] RAX: 0000000000003818 RBX: 0000000000000000 RCX: 0000000000003818
> [   41.051747] RDX: 0000000000000000 RSI: 0000000000000008 RDI: 00000000002896b3
> [   41.053268] RBP: ffff880179683c20 R08: 0000000001010100 R09: 00000000ffffffe7
> [   41.054730] R10: ffffea0001eb6100 R11: ffffffffa02afa31 R12: 0000000000000000
> [   41.056201] R13: ffff880035dbc8c0 R14: ffff880175d03f80 R15: 000000017716e000
> [   41.057673] FS:  0000000000000000(0000) GS:ffff88017fc00000(0000) 
> knlGS:0000000000000000
> [   41.059337] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [   41.060548] CR2: 0000000000003818 CR3: 0000000178331000 CR4: 00000000000006f0
> [   41.062028] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [   41.063534] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [   41.065025] Stack:
> [   41.065473]  ffff880035dbc8c0 ffff880035dbce70 0000000000000001 
> ffff880035dbc8c8
> [   41.067119]  ffff880035dbce70 ffff880179683c80 ffffffffa02b8a77 
> fefdf27269fb3cd8
> [   41.068781]  2009f9ee3386436f eb9e4e66756bbfdd 34002f8114a5d65f 
> 9535990856231c4b
> [   41.094179] Call Trace:
> [   41.118688]  [<ffffffffa02b8a77>] igb_configure+0x267/0x450 [igb]
> [   41.144286]  [<ffffffffa02b94f1>] igb_up+0x21/0x1a0 [igb]
> [   41.170606]  [<ffffffffa02b96a7>] igb_io_resume+0x37/0x70 [igb]
> [   41.195846]  [<ffffffff813381e0>] ? 
> pci_cleanup_aer_uncorrect_error_status+0x90/0x90
> [   41.221767]  [<ffffffff81338228>] report_resume+0x48/0x60
> [   41.246455]  [<ffffffff8131e359>] pci_walk_bus+0x79/0xa0
> [   41.270722]  [<ffffffff813381e0>] ? 
> pci_cleanup_aer_uncorrect_error_status+0x90/0x90
> [   41.296747]  [<ffffffff813382f0>] broadcast_error_message+0xb0/0x100
> [   41.321552]  [<ffffffff81338509>] do_recovery+0x1c9/0x280
> [   41.345507]  [<ffffffff81338f58>] aer_isr+0x348/0x430
> [   41.368851]  [<ffffffff8109d5fb>] process_one_work+0x17b/0x470
> [   41.392157]  [<ffffffff8109e3cb>] worker_thread+0x11b/0x400
> [   41.416852]  [<ffffffff8109e2b0>] ? rescuer_thread+0x400/0x400
> [   41.441577]  [<ffffffff810a5aef>] kthread+0xcf/0xe0
> [   41.465029]  [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140
> [   41.488341]  [<ffffffff81645858>] ret_from_fork+0x58/0x90
> [   41.511247]  [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140
> [   41.535442] Code: c1 49 89 4e 30 49 8b 85 b8 05 00 00 48 85 c0 0f 84 39 01 00 00 
> 81 c2 10 38 00 00 48 63 d2 48 01 d0 31 d2 89 10 49 8b 46 30 31 d2 <89> 10 41 8b 
> 95 3c 06 00 00 b8 14 01 10 02 83 fa 05 74 0b 83 fa
> [   41.587718] RIP  [<ffffffffa02b438d>] igb_configure_tx_ring+0x14d/0x280 [igb]
> [   41.610872]  RSP <ffff880179683bf8>
> [   41.632301] CR2: 0000000000003818
>
> And then it reboots.  So what RAS improvement have we bought ourselves
> here?  What endpoints have you tested with this?  Which ones recovered
> reliably?  Thanks,

I am working on it.

The endpoints I used to test are：

-device ioh3420,bus=pcie.0,addr=1c.0,port=1,id=bridge1,chassis=1 \
-device vfio-pci,host=06:00.1,bus=bridge1,addr=00.1,id=net1,aer=true \
-device 
vfio-pci,host=06:00.0,bus=bridge1,addr=00.0,id=net0,aer=true,multifunction=on 
\

When I tested them, sometimes, my guest even be kernel panic.

And I found that the bug was related with the states of the devices.

when the enp1s0f1 was not up, which just like below:

ifconfig -a
...
enp1s0f1: flags=4098<BROADCAST,MULTICAST>  mtu 1500
        ether 00:1b:21:67:3b:bd  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device memory 0xfe420000-fe43ffff
...

Then, I injected fatal/nonfatal error, it could recovered reliably.

But, when I executed this command to make the network up:

ifconfig enp1s0f1 up

dmesg:
...
[   34.109886] IPv6: ADDRCONF(NETDEV_UP): enp1s0f1: link is not ready
[   36.232118] igb 0000:01:00.1 enp1s0f1: igb: enp1s0f1 NIC Link is Up 
1000 Mbps Full Duplex, Flow Control: RX/TX
[   36.232397] IPv6: ADDRCONF(NETDEV_CHANGE): enp1s0f1: link becomes ready
...

ifconfig:
...
enp1s0f1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
...

Then, I did the tests, I got the kernel panic:

       [  910.702002] ---[ end trace 3cd6a977a579a0bd ]---
       [  910.702002] Kernel panic - not syncing: Fatal exception
       [  910.702002] Kernel Offset: 0x0 from 0xffffffff81000000 
(relocation range: 0xf
       fffffff80000000-0xffffffff9fffffff) 

       [  910.702002] ---[ end Kernel panic - not syncing: Fatal exception

----------------------

The bug happened in Guest with "PCIe link lost, device now detached",
but both Host and QEMU can reset the devices well.

At First, I guessed it might be a igb bug[1] and did many jobs on it.
But, it don't work.

Currently, I guess if I should make the devices non-working before QEMU
reset the devices ?
OR
I guess if I need to investigate the bug in igb driver ?

Now, I have no idea about it, Could you give me some advice?

[1]:http://www.gossamer-threads.com/lists/linux/kernel/2274363

Thanks,
Dou