[Qemu-devel] 2.6.32.x guest dies when trying to run tcpdump

[Nikola Ciprich]

Hello,
I noticed few times, that when I try to run tcpdump on 2.6.32.x, the
guest end up badly, it gets stuck for some time, and either reboots, or
gets into pretty good state. I'm using virtio, both host and guests are
x86_64, host is 2.6.37 (but I noticed this problem also with older host
kernels).

here's first backtrace (guest rebooted):
Apr  1 04:45:03 vmnci02 [30055.672138] BUG: soft lockup - CPU#0 stuck for 66s! 
[crond:3053]
Apr  1 04:45:03 vmnci02 [30055.672144] Modules linked in: loop bluetooth rfkill 
ipv6 lockd sunrpc dm_mirror dm_region_hash dm_log dm_mod video backlight output 
sbs sbshc fan container battery ac parport_pc lp parport nvram sg 
virtio_balloon thermal processor button thermal_sys tpm_tis tpm tpm_bios piix 
i2c_piix4 i2c_core pcspkr virtio_blk virtio_net virtio_pci virtio_ring virtio 
8139cp mii pata_acpi ide_pci_generic ide_core ata_piix ata_generic libata 
sd_mod scsi_mod crc_t10dif ext4 jbd2 crc16 uhci_hcd ohci_hcd ehci_hcd [last 
unloaded: scsi_wait_scan]
Apr  1 04:45:03 vmnci02 [30055.672220] CPU 0:
Apr  1 04:45:03 vmnci02 [30055.672223] Modules linked in: loop bluetooth rfkill 
ipv6 lockd sunrpc dm_mirror dm_region_hash dm_log dm_mod video backlight output 
sbs sbshc fan container battery ac parport_pc lp parport nvram sg 
virtio_balloon thermal processor button thermal_sys tpm_tis tpm tpm_bios piix 
i2c_piix4 i2c_core pcspkr virtio_blk virtio_net virtio_pci virtio_ring virtio 
8139cp mii pata_acpi ide_pci_generic ide_core ata_piix ata_generic libata 
sd_mod scsi_mod crc_t10dif ext4 jbd2 crc16 uhci_hcd ohci_hcd ehci_hcd [last 
unloaded: scsi_wait_scan]
Apr  1 04:45:03 vmnci02 [30055.672279] Pid: 3053, comm: crond Not tainted 
2.6.32lb.36.kvmguest #1 Bochs
Apr  1 04:45:03 vmnci02 [30055.672282] RIP: 0010:[<ffffffff8133fe10>]  
[<ffffffff8133fe10>] _spin_unlock_irq+0x10/0x40
Apr  1 04:45:03 vmnci02 [30055.672437] RSP: 0000:ffff88000469fd98  EFLAGS: 
00000202
Apr  1 04:45:03 vmnci02 [30055.672441] RAX: 0000000000000000 RBX: 
ffff88000469fd98 RCX: 00000000c0000100
Apr  1 04:45:03 vmnci02 [30055.672444] RDX: 0000000000003130 RSI: 
0000000000000000 RDI: ffff880001813bc0
Apr  1 04:45:03 vmnci02 [30055.672447] RBP: ffffffff8100bf5e R08: 
ffff88000469e000 R09: 0000000000000000
Apr  1 04:45:03 vmnci02 [30055.672451] R10: 0000000000000001 R11: 
0000000000000000 R12: ffffffff8104bc16
Apr  1 04:45:03 vmnci02 [30055.672454] R13: ffff88000469fd28 R14: 
0000000000000000 R15: ffff8800046b9750
Apr  1 04:45:03 vmnci02 [30055.672461] FS:  00007eff7edda710(0000) 
GS:ffff880001800000(0000) knlGS:0000000000000000
Apr  1 04:45:03 vmnci02 [30055.672465] CS:  0010 DS: 0000 ES: 0000 CR0: 
000000008005003b
Apr  1 04:45:03 vmnci02 [30055.672468] CR2: 00007f2ddef71a3d CR3: 
00000000044ba000 CR4: 00000000000006f0
Apr  1 04:45:03 vmnci02 [30055.672480] DR0: 0000000000000000 DR1: 
0000000000000000 DR2: 0000000000000000
Apr  1 04:45:03 vmnci02 [30055.672491] DR3: 0000000000000000 DR6: 
00000000ffff0ff0 DR7: 0000000000000400
Apr  1 04:45:03 vmnci02 [30055.672494] Call Trace:
Apr  1 04:45:03 vmnci02 [30055.672501]  [<ffffffff8133fe09>] ? 
_spin_unlock_irq+0x9/0x40
Apr  1 04:45:03 vmnci02 [30055.672579]  [<ffffffff81041c1f>] ? 
finish_task_switch+0x4f/0xd0
Apr  1 04:45:03 vmnci02 [30055.672658]  [<ffffffff8133cdbf>] ? 
thread_return+0x4c/0x77d
Apr  1 04:45:03 vmnci02 [30055.672664]  [<ffffffff8133fe65>] ? 
_spin_unlock_irqrestore+0x25/0x50
Apr  1 04:45:03 vmnci02 [30055.672774]  [<ffffffff811c6f14>] ? 
_raw_spin_lock+0x54/0x160
Apr  1 04:45:03 vmnci02 [30055.672780]  [<ffffffff8133ed45>] ? 
do_nanosleep+0x75/0xb0
Apr  1 04:45:03 vmnci02 [30055.672820]  [<ffffffff8107563e>] ? 
hrtimer_nanosleep+0x9e/0x120
Apr  1 04:45:03 vmnci02 [30055.672825]  [<ffffffff810747a0>] ? 
hrtimer_wakeup+0x0/0x30
Apr  1 04:45:03 vmnci02 [30055.672830]  [<ffffffff8107572f>] ? 
sys_nanosleep+0x6f/0x80
Apr  1 04:45:03 vmnci02 [30055.672837]  [<ffffffff8100b402>] ? 
system_call_fastpath+0x16/0x1b

here's second backtrace, guest survived, but in pretty bad condition (notice 
ext4 journal abort at the end):
[161990.733858] BUG: spinlock lockup on CPU#1, scsi_eh_0/1046, ffff88011a01a3d8
[162016.148705] BUG: soft lockup - CPU#6 stuck for 205s! [sshd:9310]
[162016.148705] Modules linked in: bluetooth rfkill be2iscsi iscsi_tcp bnx2i 
cnic uio cxgb3i cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi 
dm_snapshot dm_zero netconsole ipv6 nfs nfs_acl auth_rpcgss lockd sunrpc 
dm_mirror dm_region_hash dm_log dm_mod video backlight output sbs sbshc fan 
container battery ac thermal processor button virtio_balloon pcspkr i2c_piix4 
thermal_sys i2c_core sg virtio_blk virtio_net virtio_pci virtio_ring virtio 
8139cp mii pata_acpi ide_pci_generic ide_core ata_piix ata_generic libata 
sd_mod scsi_mod crc_t10dif ext4 jbd2 crc16 uhci_hcd ohci_hcd ehci_hcd [last 
unloaded: scsi_wait_scan]
[162016.148705] CPU 6:
[162016.148705] Modules linked in: bluetooth rfkill be2iscsi iscsi_tcp bnx2i 
cnic uio cxgb3i cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi 
dm_snapshot dm_zero netconsole ipv6 nfs nfs_acl auth_rpcgss lockd sunrpc 
dm_mirror dm_region_hash dm_log dm_mod video backlight output sbs sbshc fan 
container battery ac thermal processor button virtio_balloon pcspkr i2c_piix4 
thermal_sys i2c_core sg virtio_blk virtio_net virtio_pci virtio_ring virtio 
8139cp mii pata_acpi ide_pci_generic ide_core ata_piix ata_generic libata 
sd_mod scsi_mod crc_t10dif ext4 jbd2 crc16 uhci_hcd ohci_hcd ehci_hcd [last 
unloaded: scsi_wait_scan]
[162016.148705] Pid: 9310, comm: sshd Not tainted 2.6.32lb.29.kvmguest #1 Bochs
[162016.148705] RIP: 0010:[<ffffffff811c58a6>]  [<ffffffff811c58a6>] 
iowrite16+0x26/0x40
[162016.148705] RSP: 0018:ffff8800aae0b638  EFLAGS: 00000206
[162016.148705] RAX: 0000000000000001 RBX: ffff8800aae0b638 RCX: 
ffff88011a1bd000
[162016.148705] RDX: 000000000000c050 RSI: 000000000001c050 RDI: 
0000000000000001
[162016.148705] RBP: ffffffff8100bf5e R08: 0000000000000003 R09: 
0000000000000003
[162016.148705] R10: 0000000000000030 R11: 0000000000000000 R12: 
ffffffff8151ce10
[162016.148705] R13: 0000000000000246 R14: 0000000000000206 R15: 
ffffea0002aba900
[162016.148705] FS:  00007f52fb21b730(0000) GS:ffff880028380000(0000) 
knlGS:0000000000000000
[162016.148705] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[162016.148705] CR2: 00007f2ed8640e80 CR3: 00000000d2c71000 CR4: 
00000000000006e0
[162016.148705] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[162016.148705] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[162016.148705] Call Trace:
[162016.148705]  [<ffffffffa01b613d>] ? vp_notify+0x2d/0x40 [virtio_pci]
[162016.148705]  [<ffffffffa01af033>] ? vring_kick+0x33/0x40 [virtio_ring]
[162016.148705]  [<ffffffffa01be4a0>] ? start_xmit+0x1a0/0x400 [virtio_net]
[162016.148705]  [<ffffffff812a6f49>] ? dev_hard_start_xmit+0x2c9/0x360
[162016.148705]  [<ffffffff812bbc3f>] ? sch_direct_xmit+0x16f/0x1e0
[162016.148705]  [<ffffffff812aa30b>] ? dev_queue_xmit+0x24b/0x3f0
[162016.148705]  [<ffffffff812d513c>] ? ip_finish_output+0x12c/0x340
[162016.148705]  [<ffffffff812d61e0>] ? ip_output+0xb0/0x100
[162016.148705]  [<ffffffff812d4bf0>] ? ip_local_out+0x20/0x30
[162016.148705]  [<ffffffff812d6441>] ? ip_queue_xmit+0x211/0x3f0
[162016.148705]  [<ffffffff811258f0>] ? __pollwait+0x0/0x100
[162016.148705]  [<ffffffff812e9545>] ? tcp_transmit_skb+0x3a5/0x780
[162016.148705]  [<ffffffff811259f0>] ? pollwake+0x0/0x60
[162016.148705]  [<ffffffff812eac32>] ? tcp_write_xmit+0x3b2/0xa00
[162016.148705]  [<ffffffff812eb2ab>] ? __tcp_push_pending_frames+0x2b/0xa0
[162016.148705]  [<ffffffff812de5e7>] ? tcp_sendmsg+0x7e7/0xb10
[162016.148705]  [<ffffffff812949f7>] ? sock_aio_write+0x137/0x150
[162016.148705]  [<ffffffff8103d14e>] ? __wake_up+0x4e/0x70
[162016.148705]  [<ffffffff81113bf1>] ? do_sync_write+0xf1/0x130
[162016.148705]  [<ffffffff81071050>] ? autoremove_wake_function+0x0/0x40
[162016.148705]  [<ffffffff81058532>] ? current_fs_time+0x22/0x30
[162016.148705]  [<ffffffff81196c41>] ? security_file_permission+0x11/0x20
[162016.148705]  [<ffffffff811144d9>] ? vfs_write+0x189/0x1a0
[162016.148705]  [<ffffffff81114b20>] ? sys_write+0x50/0x90
[162016.148705]  [<ffffffff8100b402>] ? system_call_fastpath+0x16/0x1b
[162335.680168] BUG: soft lockup - CPU#0 stuck for 297s! [jbd2/sda-8:1068]
[162335.680173] Modules linked in: bluetooth rfkill be2iscsi iscsi_tcp bnx2i 
cnic uio cxgb3i cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi 
dm_snapshot dm_zero netconsole ipv6 nfs nfs_acl auth_rpcgss lockd sunrpc 
dm_mirror dm_region_hash dm_log dm_mod video backlight output sbs sbshc fan 
container battery ac thermal processor button virtio_balloon pcspkr i2c_piix4 
thermal_sys i2c_core sg virtio_blk virtio_net virtio_pci virtio_ring virtio 
8139cp mii pata_acpi ide_pci_generic ide_core ata_piix ata_generic libata 
sd_mod scsi_mod crc_t10dif ext4 jbd2 crc16 uhci_hcd ohci_hcd ehci_hcd [last 
unloaded: scsi_wait_scan]
[162335.680239] CPU 0:
[162335.680240] Modules linked in: bluetooth rfkill be2iscsi iscsi_tcp bnx2i 
cnic uio cxgb3i cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi 
dm_snapshot dm_zero netconsole ipv6 nfs nfs_acl auth_rpcgss lockd sunrpc 
dm_mirror dm_region_hash dm_log dm_mod video backlight output sbs sbshc fan 
container battery ac thermal processor button virtio_balloon pcspkr i2c_piix4 
thermal_sys i2c_core sg virtio_blk virtio_net virtio_pci virtio_ring virtio 
8139cp mii pata_acpi ide_pci_generic ide_core ata_piix ata_generic libata 
sd_mod scsi_mod crc_t10dif ext4 jbd2 crc16 uhci_hcd ohci_hcd ehci_hcd [last 
unloaded: scsi_wait_scan]
[162335.680279] Pid: 1068, comm: jbd2/sda-8 Not tainted 2.6.32lb.29.kvmguest #1 
Bochs
[162335.680283] RIP: 0010:[<ffffffff8133ee50>]  [<ffffffff8133ee50>] 
_spin_unlock_irq+0x10/0x40
[162335.680401] RSP: 0018:ffff880028203e70  EFLAGS: 00000286
[162335.680403] RAX: 0000000000000000 RBX: ffff880028203e70 RCX: 
ffffffff816458e0
[162335.680404] RDX: 000000000000d1d0 RSI: ffff880028203e40 RDI: 
ffffffff81644940
[162335.680406] RBP: ffffffff8100bf63 R08: 00000000000022fc R09: 
ffff880028203be0
[162335.680410] R10: 0000000000000014 R11: 0000000000000000 R12: 
ffff880028203df0
[162335.680414] R13: ffffffff81644940 R14: ffff880028203ea0 R15: 
ffffffff81024857
[162335.680418] FS:  0000000000000000(0000) GS:ffff880028200000(0000) 
knlGS:0000000000000000
[162335.680423] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
[162335.680424] CR2: 00007f2ed5713548 CR3: 00000000d101c000 CR4: 
00000000000006f0
[162335.680434] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[162335.680441] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[162335.680443] Call Trace:
[162335.680446]  <IRQ>  [<ffffffff81060c54>] ? run_timer_softirq+0x214/0x320
[162335.680486]  [<ffffffff81306fe0>] ? inet_frag_secret_rebuild+0x0/0x100
[162335.680493]  [<ffffffff81059cdc>] ? __do_softirq+0xcc/0x230
[162335.680498]  [<ffffffff8100c59c>] ? call_softirq+0x1c/0x30
[162335.680504]  [<ffffffff8100e6a5>] ? do_softirq+0x65/0xa0
[162335.680507]  [<ffffffff81059b97>] ? irq_exit+0x87/0x90
[162335.680523]  [<ffffffff8102485c>] ? smp_apic_timer_interrupt+0x6c/0xa0
[162335.680530]  [<ffffffff8100bf63>] ? apic_timer_interrupt+0x13/0x20
[162335.680532]  <EOI>  [<ffffffff8133ee96>] ? _spin_unlock_irqrestore+0x16/0x50
[162335.680551]  [<ffffffffa00c2a2a>] ? scsi_dispatch_cmd+0x10a/0x2a0 [scsi_mod]
[162335.680562]  [<ffffffffa00ca58f>] ? scsi_request_fn+0x16f/0x4f0 [scsi_mod]
[162335.680594]  [<ffffffff811a3c30>] ? __generic_unplug_device+0x30/0x40
[162335.680599]  [<ffffffff811a4619>] ? generic_unplug_device+0x29/0x40
[162335.680615]  [<ffffffff8119fddf>] ? blk_unplug+0x2f/0xa0
[162335.680621]  [<ffffffff8119fe5d>] ? blk_backing_dev_unplug+0xd/0x10
[162335.680640]  [<ffffffff8113ae89>] ? sync_buffer+0x39/0x50
[162335.680657]  [<ffffffff8133cc55>] ? __wait_on_bit+0x55/0x80
[162335.680662]  [<ffffffff8113ae50>] ? sync_buffer+0x0/0x50
[162335.680667]  [<ffffffff8113ae50>] ? sync_buffer+0x0/0x50
[162335.680672]  [<ffffffff8133ccf8>] ? out_of_line_wait_on_bit+0x78/0x90
[162335.680680]  [<ffffffff81071090>] ? wake_bit_function+0x0/0x30
[162335.680685]  [<ffffffff8113adc6>] ? __wait_on_buffer+0x26/0x30
[162335.680768]  [<ffffffffa00301a2>] ? 
jbd2_journal_commit_transaction+0x782/0x18e0 [jbd2]
[162335.680774]  [<ffffffff8133f0b7>] ? _spin_lock_irqsave+0x37/0x50
[162335.680778]  [<ffffffff811c639e>] ? _raw_spin_unlock+0x5e/0xb0
[162335.680819]  [<ffffffffa0036c63>] ? kjournald2+0xb3/0x230 [jbd2]
[162335.680829]  [<ffffffff81071050>] ? autoremove_wake_function+0x0/0x40
[162335.678699] Pid: 1046, comm: scsi_eh_0 Not tainted 2.6.32lb.29.kvmguest #1
[162335.680843]  [<ffffffffa0036bb0>] ? kjournald2+0x0/0x230 [jbd2]
[162335.680850]  [<ffffffff81070f0e>] ? kthread+0x8e/0xa0
[162335.678699] Call Trace:
[162335.680983]  [<ffffffff8158a140>] ? early_idt_handler+0x0/0x71
[162335.678699]  [<ffffffff811c6517>] _raw_spin_lock+0x127/0x160
[162335.680996]  [<ffffffff8100c49a>] ? child_rip+0xa/0x20
[162335.678699]  [<ffffffff8133f0b7>] _spin_lock_irqsave+0x37/0x50
[162335.681003]  [<ffffffff8158a140>] ? early_idt_handler+0x0/0x71
[162335.681007]  [<ffffffff81070e80>] ? kthread+0x0/0xa0
[162335.678699]  [<ffffffffa011fa9f>] ata_scsi_error+0x3f/0x870 [libata]
[162335.681056]  [<ffffffff8100c490>] ? child_rip+0x0/0x20
[162335.678699]  [<ffffffffa00c79e0>] ? scsi_error_handler+0x0/0x5b0 [scsi_mod]
[162335.678699]  [<ffffffffa00c7aa6>] scsi_error_handler+0xc6/0x5b0 [scsi_mod]
[162335.678699]  [<ffffffffa00c79e0>] ? scsi_error_handler+0x0/0x5b0 [scsi_mod]
[162335.678699]  [<ffffffff81070f0e>] kthread+0x8e/0xa0
[162335.678699]  [<ffffffff8158a140>] ? early_idt_handler+0x0/0x71
[162335.678699]  [<ffffffff8100c49a>] child_rip+0xa/0x20
[162335.678699]  [<ffffffff8158a140>] ? early_idt_handler+0x0/0x71
[162335.678699]  [<ffffffff81070e80>] ? kthread+0x0/0xa0
[162335.678699]  [<ffffffff8100c490>] ? child_rip+0x0/0x20
[162335.678699] sending NMI to all CPUs:
[162335.681177] NMI backtrace for cpu 0
[162335.681179] CPU 0:
[162335.681181] Modules linked in: bluetooth rfkill be2iscsi iscsi_tcp bnx2i 
cnic uio cxgb3i cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi 
dm_snapshot dm_zero netconsole ipv6 nfs nfs_acl auth_rpcgss lockd sunrpc 
dm_mirror dm_region_hash dm_log dm_mod video backlight output sbs sbshc fan 
container battery ac thermal processor button virtio_balloon pcspkr i2c_piix4 
thermal_sys i2c_core sg virtio_blk virtio_net virtio_pci virtio_ring virtio 
8139cp mii pata_acpi ide_pci_generic ide_core ata_piix ata_generic libata 
sd_mod scsi_mod crc_t10dif ext4 jbd2 crc16 uhci_hcd ohci_hcd ehci_hcd [last 
unloaded: scsi_wait_scan]
[162335.681316] Pid: 1068, comm: jbd2/sda-8 Not tainted 2.6.32lb.29.kvmguest #1 
Bochs
[162335.681324] RIP: 0010:[<ffffffff8103ff38>]  [<ffffffff8103ff38>] 
find_busiest_group+0x198/0x9e0
[162335.681340] RSP: 0018:ffff880119ce59f0  EFLAGS: 00010006
[162335.681347] RAX: 0000000000000c00 RBX: 0000000000000001 RCX: 
0000000000000020
[162335.681357] RDX: ffff880028240000 RSI: 0000000000000000 RDI: 
0000000000000001
[162335.681367] RBP: ffff880119ce5b50 R08: 0000000000000000 R09: 
ffff88002824e430
[162335.681378] R10: 0000000000000000 R11: 0000000000000000 R12: 
ffff880028253bc0
[162335.681388] R13: 0000000000000c00 R14: 0000000000000c00 R15: 
0000000000000000
[162335.681402] FS:  0000000000000000(0000) GS:ffff880028200000(0000) 
knlGS:0000000000000000
[162335.681410] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
[162335.681418] CR2: 00007f2ed5713548 CR3: 00000000d101c000 CR4: 
00000000000006f0
[162335.681430] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[162335.681441] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[162335.681450] Call Trace:
[162335.681455]  <#DB[1]>  <<EOE>> Pid: 1068, comm: jbd2/sda-8 Not tainted 
2.6.32lb.29.kvmguest #1
[162335.681483] Call Trace:
[162335.681487]  <NMI>  [<ffffffff81009d83>] ? show_regs+0x43/0x50
[162335.681500]  [<ffffffff81024f4a>] nmi_watchdog_tick+0x17a/0x260
[162335.681509]  [<ffffffff8100cc5d>] do_nmi+0x8d/0x290
[162335.681517]  [<ffffffff8133f820>] nmi+0x20/0x30
[162335.681540]  [<ffffffff8103ff38>] ? find_busiest_group+0x198/0x9e0
[162335.681548]  <<EOE>>  [<ffffffff8133c2ce>] thread_return+0x4cd/0x73f
[162335.681563]  [<ffffffff8102a449>] ? kvm_clock_get_cycles+0x9/0x10
[162335.681573]  [<ffffffff8133c71e>] io_schedule+0x6e/0xc0
[162335.681583]  [<ffffffff8113ae8e>] sync_buffer+0x3e/0x50
[162335.681593]  [<ffffffff8133cc55>] __wait_on_bit+0x55/0x80
[162335.681602]  [<ffffffff8113ae50>] ? sync_buffer+0x0/0x50
[162335.681611]  [<ffffffff8113ae50>] ? sync_buffer+0x0/0x50
[162335.681621]  [<ffffffff8133ccf8>] out_of_line_wait_on_bit+0x78/0x90
[162335.681630]  [<ffffffff81071090>] ? wake_bit_function+0x0/0x30
[162335.681639]  [<ffffffff8113adc6>] __wait_on_buffer+0x26/0x30
[162335.681653]  [<ffffffffa00301a2>] 
jbd2_journal_commit_transaction+0x782/0x18e0 [jbd2]
[162335.681664]  [<ffffffff8133f0b7>] ? _spin_lock_irqsave+0x37/0x50
[162335.681674]  [<ffffffff811c639e>] ? _raw_spin_unlock+0x5e/0xb0
[162335.681687]  [<ffffffffa0036c63>] kjournald2+0xb3/0x230 [jbd2]
[162335.681695]  [<ffffffff81071050>] ? autoremove_wake_function+0x0/0x40
[162335.681708]  [<ffffffffa0036bb0>] ? kjournald2+0x0/0x230 [jbd2]
[162335.681718]  [<ffffffff81070f0e>] kthread+0x8e/0xa0
[162335.681727]  [<ffffffff8158a140>] ? early_idt_handler+0x0/0x71
[162335.681738]  [<ffffffff8100c49a>] child_rip+0xa/0x20
[162335.681745]  [<ffffffff8158a140>] ? early_idt_handler+0x0/0x71
[162335.681754]  [<ffffffff81070e80>] ? kthread+0x0/0xa0
[162335.681760]  [<ffffffff8100c490>] ? child_rip+0x0/0x20
[162335.678699] NMI backtrace for cpu 1
[162335.681771] CPU 1:
[162335.681771] Modules linked in: bluetooth rfkill be2iscsi iscsi_tcp bnx2i 
cnic uio cxgb3i cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi 
dm_snapshot dm_zero netconsole ipv6 nfs nfs_acl auth_rpcgss lockd sunrpc 
dm_mirror dm_region_hash dm_log dm_mod video backlight output sbs sbshc fan 
container battery ac thermal processor button virtio_balloon pcspkr i2c_piix4 
thermal_sys i2c_core sg virtio_blk virtio_net virtio_pci virtio_ring virtio 
8139cp mii pata_acpi ide_pci_generic ide_core ata_piix ata_generic libata 
sd_mod scsi_mod crc_t10dif ext4 jbd2 crc16 uhci_hcd ohci_hcd ehci_hcd [last 
unloaded: scsi_wait_scan]
[162335.681771] Pid: 1046, comm: scsi_eh_0 Not tainted 2.6.32lb.29.kvmguest #1 
Bochs
[162335.681771] RIP: 0010:[<ffffffff81028144>]  [<ffffffff81028144>] 
flat_send_IPI_mask+0x74/0xa0
[162335.681771] RSP: 0018:ffff88011a279d40  EFLAGS: 00010046
[162335.681771] RAX: 0000000000000c00 RBX: 0000000000000c00 RCX: 
0000000000000000
[162335.681771] RDX: ffff88011a279fd8 RSI: 0000000000000002 RDI: 
0000000000000300
[162335.681771] RBP: ffff88011a279d60 R08: 0000000000000000 R09: 
0000000000000000
[162335.681771] R10: 000000000000000a R11: 0000000000000000 R12: 
0000000000000002
[162335.681771] R13: 0000000000000082 R14: 00000000000000ff R15: 
0000000000000000
[162335.681771] FS:  0000000000000000(0000) GS:ffff880028240000(0000) 
knlGS:0000000000000000
[162335.681771] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
[162335.681771] CR2: 0000000002143858 CR3: 00000000aacaa000 CR4: 
00000000000006e0
[162335.681771] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[162335.681771] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[162335.681771] Call Trace:
[162335.681771]  <#DB[1]>  <<EOE>> Pid: 1046, comm: scsi_eh_0 Not tainted 
2.6.32lb.29.kvmguest #1
[162335.681771] Call Trace:
[162335.681771]  <NMI>  [<ffffffff81009d83>] ? show_regs+0x43/0x50
[162335.681771]  [<ffffffff81024f4a>] nmi_watchdog_tick+0x17a/0x260
[162335.681771]  [<ffffffff8100cc5d>] do_nmi+0x8d/0x290
[162335.681771]  [<ffffffff8133f820>] nmi+0x20/0x30
[162335.681771]  [<ffffffff81028144>] ? flat_send_IPI_mask+0x74/0xa0
[162335.681771]  <<EOE>>  [<ffffffff810281cd>] flat_send_IPI_all+0x5d/0x60
[162335.681771]  [<ffffffff81024ccc>] arch_trigger_all_cpu_backtrace+0x3c/0x70
[162335.681771]  [<ffffffff811c651c>] _raw_spin_lock+0x12c/0x160
[162335.681771]  [<ffffffff8133f0b7>] _spin_lock_irqsave+0x37/0x50
[162335.681771]  [<ffffffffa011fa9f>] ata_scsi_error+0x3f/0x870 [libata]
[162335.681771]  [<ffffffffa00c79e0>] ? scsi_error_handler+0x0/0x5b0 [scsi_mod]
[162335.681771]  [<ffffffffa00c7aa6>] scsi_error_handler+0xc6/0x5b0 [scsi_mod]
[162335.681771]  [<ffffffffa00c79e0>] ? scsi_error_handler+0x0/0x5b0 [scsi_mod]
[162335.681771]  [<ffffffff81070f0e>] kthread+0x8e/0xa0
[162335.681771]  [<ffffffff8158a140>] ? early_idt_handler+0x0/0x71
[162335.681771]  [<ffffffff8100c49a>] child_rip+0xa/0x20
[162335.681771]  [<ffffffff8158a140>] ? early_idt_handler+0x0/0x71
[162335.681771]  [<ffffffff81070e80>] ? kthread+0x0/0xa0
[162335.681771]  [<ffffffff8100c490>] ? child_rip+0x0/0x20
[162335.682115] NMI backtrace for cpu 4
[162335.682131] CPU 4:
[162335.682142] Modules linked in: bluetooth rfkill be2iscsi iscsi_tcp bnx2i 
cnic uio cxgb3i cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi 
dm_snapshot dm_zero netconsole ipv6 nfs nfs_acl auth_rpcgss lockd sunrpc 
dm_mirror dm_region_hash dm_log dm_mod video backlight output sbs sbshc fan 
container battery ac thermal processor button virtio_balloon pcspkr i2c_piix4 
thermal_sys i2c_core sg virtio_blk virtio_net virtio_pci virtio_ring virtio 
8139cp mii pata_acpi ide_pci_generic ide_core ata_piix ata_generic libata 
sd_mod scsi_mod crc_t10dif ext4 jbd2 crc16 uhci_hcd ohci_hcd ehci_hcd [last 
unloaded: scsi_wait_scan]
[162335.682443] Pid: 8137, comm: sadc Not tainted 2.6.32lb.29.kvmguest #1 Bochs
[162335.682470] RIP: 0010:[<ffffffff810a86ac>]  [<ffffffff810a86ac>] 
kstat_irqs_cpu+0xc/0x30
[162335.682536] RSP: 0018:ffff8800aaea9c60  EFLAGS: 00010282
[162335.682557] RAX: ffffffff81510540 RBX: 0000000000000006 RCX: 
0000000000000000
[162335.682582] RDX: 0000000000000000 RSI: 0000000000000006 RDI: 
0000000000000443
[162335.682607] RBP: ffff8800aaea9c68 R08: 0000000000000000 R09: 
ffffffff81571230
[162335.682632] R10: 0000000000000022 R11: 0000000000000000 R12: 
0000000000000006
[162335.682655] R13: 000000000000e4a0 R14: 0000000000000006 R15: 
0000000000000000
[162335.682676] FS:  00007fe70e3846e0(0000) GS:ffff880028300000(0000) 
knlGS:0000000000000000
[162335.682703] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[162335.682724] CR2: 00007fe70e39f000 CR3: 00000000ce68d000 CR4: 
00000000000006e0
[162335.682753] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[162335.682781] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[162335.682803] Call Trace:
[162335.682814]  <#DB[1]>  <<EOE>> Pid: 8137, comm: sadc Not tainted 
2.6.32lb.29.kvmguest #1
[162335.682858] Call Trace:
[162335.682868]  <NMI>  [<ffffffff81009d83>] ? show_regs+0x43/0x50
[162335.682890]  [<ffffffff81024f4a>] nmi_watchdog_tick+0x17a/0x260
[162335.682914]  [<ffffffff8100cc5d>] do_nmi+0x8d/0x290
[162335.682928]  [<ffffffff8133f820>] nmi+0x20/0x30
[162335.682947]  [<ffffffff810a86ac>] ? kstat_irqs_cpu+0xc/0x30
[162335.682967]  <<EOE>>  [<ffffffff811717c0>] show_stat+0x190/0x610
[162335.683045]  [<ffffffff8107f42f>] ? debug_mutex_add_waiter+0x3f/0x70
[162335.683074]  [<ffffffff8133d6b3>] ? __mutex_lock_slowpath+0x263/0x340
[162335.683097]  [<ffffffff81130e90>] ? seq_read+0x0/0x3c0
[162335.683119]  [<ffffffff81130f4d>] seq_read+0xbd/0x3c0
[162335.683141]  [<ffffffff81130e90>] ? seq_read+0x0/0x3c0
[162335.683162]  [<ffffffff8116953a>] proc_reg_read+0x7a/0xb0
[162335.683190]  [<ffffffff811145b4>] vfs_read+0xc4/0x190
[162335.683209]  [<ffffffff81114a90>] sys_read+0x50/0x90
[162335.683231]  [<ffffffff8100b402>] system_call_fastpath+0x16/0x1b
[162335.683255] NMI backtrace for cpu 6
[162335.683262] CPU 6:
[162335.683267] Modules linked in: bluetooth rfkill be2iscsi iscsi_tcp bnx2i 
cnic uio cxgb3i cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi 
dm_snapshot dm_zero netconsole ipv6 nfs nfs_acl auth_rpcgss lockd sunrpc 
dm_mirror dm_region_hash dm_log dm_mod video backlight output sbs sbshc fan 
container battery ac thermal processor button virtio_balloon pcspkr i2c_piix4 
thermal_sys i2c_core sg virtio_blk virtio_net virtio_pci virtio_ring virtio 
8139cp mii pata_acpi ide_pci_generic ide_core ata_piix ata_generic libata 
sd_mod scsi_mod crc_t10dif ext4 jbd2 crc16 uhci_hcd ohci_hcd ehci_hcd [last 
unloaded: scsi_wait_scan]
[162335.683407] Pid: 9310, comm: sshd Not tainted 2.6.32lb.29.kvmguest #1 Bochs
[162335.683415] RIP: 0033:[<00007f52fa16cce0>]  [<00007f52fa16cce0>] 
0x7f52fa16cce0
[162335.683419] RSP: 002b:00007fffcf542598  EFLAGS: 00010246
[162335.683419] RAX: 00007fffcf5425b0 RBX: 000000000000000f RCX: 
0000000000000010
[162335.683419] RDX: 00007f52fbb02cc0 RSI: 00007fffcf5425b0 RDI: 
00007f52fbb02db4
[162335.683419] RBP: 00007f52fbafce70 R08: 00007f52fbafce70 R09: 
00007f52fb4a7310
[162335.683419] R10: 26f5a1b2c4cc4703 R11: 0000000000000246 R12: 
00007f52fbaffea0
[162335.683419] R13: 0000000000000000 R14: 00007f52fbb02cc0 R15: 
00007f52fbb02dc3
[162335.683419] FS:  00007f52fb21b730(0000) GS:ffff880028380000(0000) 
knlGS:0000000000000000
[162335.683419] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[162335.683419] CR2: 00007f2ed8640e80 CR3: 00000000d2c71000 CR4: 
00000000000006e0
[162335.683419] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[162335.683419] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[162335.683419] Call Trace:
[162335.683419]  <#DB[1]>  <<EOE>> Pid: 9310, comm: sshd Not tainted 
2.6.32lb.29.kvmguest #1
[162335.683419] Call Trace:
[162335.683419]  <NMI>  [<ffffffff81009d83>] ? show_regs+0x43/0x50
[162335.683419]  [<ffffffff81024f4a>] nmi_watchdog_tick+0x17a/0x260
[162335.683419]  [<ffffffff8100cc5d>] do_nmi+0x8d/0x290
[162335.683419]  [<ffffffff8133f820>] nmi+0x20/0x30
[162335.683419]  <<EOE>> 
[162335.683623] NMI backtrace for cpu 2
[162335.683627] CPU 2:
[162335.683630] Modules linked in: bluetooth rfkill be2iscsi iscsi_tcp bnx2i 
cnic uio cxgb3i cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi 
dm_snapshot dm_zero netconsole ipv6 nfs nfs_acl auth_rpcgss lockd sunrpc 
dm_mirror dm_region_hash dm_log dm_mod video backlight output sbs sbshc fan 
container battery ac thermal processor button virtio_balloon pcspkr i2c_piix4 
thermal_sys i2c_core sg virtio_blk virtio_net virtio_pci virtio_ring virtio 
8139cp mii pata_acpi ide_pci_generic ide_core ata_piix ata_generic libata 
sd_mod scsi_mod crc_t10dif ext4 jbd2 crc16 uhci_hcd ohci_hcd ehci_hcd [last 
unloaded: scsi_wait_scan]
[162335.683736] Pid: 0, comm: swapper Not tainted 2.6.32lb.29.kvmguest #1 Bochs
[162335.683743] RIP: 0010:[<ffffffff8102a496>]  [<ffffffff8102a496>] 
native_safe_halt+0x6/0x10
[162335.683753] RSP: 0018:ffff88011f8a7ee8  EFLAGS: 00010246
[162335.683759] RAX: ffff88011f8a7fd8 RBX: ffff88011f8a7fd8 RCX: 
00000000ffffffff
[162335.683763] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 
0000000000000001
[162335.683768] RBP: ffff88011f8a7ee8 R08: 0000000000000000 R09: 
0000000000000000
[162335.683776] R10: 0000000000000802 R11: 0000000000000000 R12: 
ffffffff81571238
[162335.683781] R13: 0000000000000000 R14: 0000000000000000 R15: 
0000000000000000
[162335.683787] FS:  0000000000000000(0000) GS:ffff880028280000(0000) 
knlGS:0000000000000000
[162335.683792] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
[162335.683802] CR2: 00007f5e4aa82a30 CR3: 000000011a7b5000 CR4: 
00000000000006e0
[162335.683811] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[162335.683822] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[162335.683826] Call Trace:
[162335.683831]  <#DB[1]>  <<EOE>> Pid: 0, comm: swapper Not tainted 
2.6.32lb.29.kvmguest #1
[162335.683855] Call Trace:
[162335.683858]  <NMI>  [<ffffffff81009d83>] ? show_regs+0x43/0x50
[162335.683869]  [<ffffffff81024f4a>] nmi_watchdog_tick+0x17a/0x260
[162335.683876]  [<ffffffff8100cc5d>] do_nmi+0x8d/0x290
[162335.683884]  [<ffffffff8133f820>] nmi+0x20/0x30
[162335.683892]  [<ffffffff8102a496>] ? native_safe_halt+0x6/0x10
[162335.683910]  <<EOE>>  [<ffffffff810136c8>] default_idle+0x38/0xc0
[162335.683952]  [<ffffffff8100a215>] cpu_idle+0x55/0x90
[162335.683959]  [<ffffffff81337ea3>] start_secondary+0x183/0x1e0
[162335.682548] NMI backtrace for cpu 3
[162335.682548] CPU 3:
[162335.682548] Modules linked in: bluetooth rfkill be2iscsi iscsi_tcp bnx2i 
cnic uio cxgb3i cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi 
dm_snapshot dm_zero netconsole ipv6 nfs nfs_acl auth_rpcgss lockd sunrpc 
dm_mirror dm_region_hash dm_log dm_mod video backlight output sbs sbshc fan 
container battery ac thermal processor button virtio_balloon pcspkr i2c_piix4 
thermal_sys i2c_core sg virtio_blk virtio_net virtio_pci virtio_ring virtio 
8139cp mii pata_acpi ide_pci_generic ide_core ata_piix ata_generic libata 
sd_mod scsi_mod crc_t10dif ext4 jbd2 crc16 uhci_hcd ohci_hcd ehci_hcd [last 
unloaded: scsi_wait_scan]
[162335.682548] Pid: 0, comm: swapper Not tainted 2.6.32lb.29.kvmguest #1 Bochs
[162335.682548] RIP: 0010:[<ffffffff8102a496>]  [<ffffffff8102a496>] 
native_safe_halt+0x6/0x10
[162335.682548] RSP: 0018:ffff88011f8e1ee8  EFLAGS: 00010246
[162335.682548] RAX: ffff88011f8e1fd8 RBX: ffff88011f8e1fd8 RCX: 
00000000ffffffff
[162335.682548] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 
0000000000000001
[162335.682548] RBP: ffff88011f8e1ee8 R08: 0000000000000000 R09: 
0000000000000000
[162335.682548] R10: 0000000000000800 R11: 0000000000000000 R12: 
ffffffff81571238
[162335.682548] R13: 0000000000000000 R14: 0000000000000000 R15: 
0000000000000000
[162335.682548] FS:  0000000000000000(0000) GS:ffff8800282c0000(0000) 
knlGS:0000000000000000
[162335.682548] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
[162335.682548] CR2: 00007f5e479c0040 CR3: 000000009e845000 CR4: 
00000000000006e0
[162335.682548] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[162335.682548] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[162335.682548] Call Trace:
[162335.682548]  <#DB[1]>  <<EOE>> Pid: 0, comm: swapper Not tainted 
2.6.32lb.29.kvmguest #1
[162335.682548] Call Trace:
[162335.682548]  <NMI>  [<ffffffff81009d83>] ? show_regs+0x43/0x50
[162335.682548]  [<ffffffff81024f4a>] nmi_watchdog_tick+0x17a/0x260
[162335.682548]  [<ffffffff8100cc5d>] do_nmi+0x8d/0x290
[162335.682548]  [<ffffffff8133f820>] nmi+0x20/0x30
[162335.682548]  [<ffffffff8102a496>] ? native_safe_halt+0x6/0x10
[162335.682548]  <<EOE>>  [<ffffffff810136c8>] default_idle+0x38/0xc0
[162335.682548]  [<ffffffff8100a215>] cpu_idle+0x55/0x90
[162335.682548]  [<ffffffff81337ea3>] start_secondary+0x183/0x1e0
[162335.684274] NMI backtrace for cpu 7
[162335.684274] CPU 7:
[162335.684274] Modules linked in: bluetooth rfkill be2iscsi iscsi_tcp bnx2i 
cnic uio cxgb3i cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi 
dm_snapshot dm_zero netconsole ipv6 nfs nfs_acl auth_rpcgss lockd sunrpc 
dm_mirror dm_region_hash dm_log dm_mod video backlight output sbs sbshc fan 
container battery ac thermal processor button virtio_balloon pcspkr i2c_piix4 
thermal_sys i2c_core sg virtio_blk virtio_net virtio_pci virtio_ring virtio 
8139cp mii pata_acpi ide_pci_generic ide_core ata_piix ata_generic libata 
sd_mod scsi_mod crc_t10dif ext4 jbd2 crc16 uhci_hcd ohci_hcd ehci_hcd [last 
unloaded: scsi_wait_scan]
[162335.684274] Pid: 0, comm: swapper Not tainted 2.6.32lb.29.kvmguest #1 Bochs
[162335.684274] RIP: 0010:[<ffffffff81108d35>]  [<ffffffff81108d35>] 
kfree+0x75/0x170
[162335.684274] RSP: 0018:ffff8800283c3c20  EFLAGS: 00010282
[162335.684274] RAX: 0000000000000282 RBX: ffffea0003448800 RCX: 
ffff88011a1bd000
[162335.684274] RDX: ffffea0000000000 RSI: 0000000000000205 RDI: 
ffff8800d1225c00
[162335.684274] RBP: ffff8800283c3c50 R08: 0000000000000000 R09: 
0000000000000002
[162335.684274] R10: 0000000000000020 R11: 0000000000000000 R12: 
0000000000000000
[162335.684274] R13: ffffffff8151d5f0 R14: ffff8800d1225c00 R15: 
ffffffff8129d668
[162335.684274] FS:  0000000000000000(0000) GS:ffff8800283c0000(0000) 
knlGS:0000000000000000
[162335.684274] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
[162335.684274] CR2: 00007f91d1931408 CR3: 00000000d36b2000 CR4: 
00000000000006e0
[162335.684274] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[162335.684274] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[162335.684274] Call Trace:
[162335.684274]  <#DB[1]>  <<EOE>> Pid: 0, comm: swapper Not tainted 
2.6.32lb.29.kvmguest #1
[162335.684274] Call Trace:
[162335.684274]  <NMI>  [<ffffffff81009d83>] ? show_regs+0x43/0x50
[162335.684274]  [<ffffffff81024f4a>] nmi_watchdog_tick+0x17a/0x260
[162335.684274]  [<ffffffff8129d668>] ? skb_release_data+0x68/0xd0
[162335.684274]  [<ffffffff8100cc5d>] do_nmi+0x8d/0x290
[162335.684274]  [<ffffffff8133f820>] nmi+0x20/0x30
[162335.684274]  [<ffffffff8129d668>] ? skb_release_data+0x68/0xd0
[162335.684274]  [<ffffffff81108d35>] ? kfree+0x75/0x170
[162335.684274]  <<EOE>>  <IRQ>  [<ffffffff8129d668>] skb_release_data+0x68/0xd0
[162335.684274]  [<ffffffff8129d269>] skb_release_all+0x19/0x20
[162335.684274]  [<ffffffff8129d2c1>] __kfree_skb+0x11/0xa0
[162335.684274]  [<ffffffff8129d369>] consume_skb+0x19/0x40
[162335.684274]  [<ffffffff812f83d8>] arp_process+0xe8/0x6d0
[162335.684274]  [<ffffffff8102b164>] ? pvclock_clocksource_read+0x44/0xb0
[162335.684274]  [<ffffffff812f8a8f>] arp_rcv+0xcf/0x150
[162335.684274]  [<ffffffff812a5ff1>] netif_receive_skb+0x3b1/0x620
[162335.684274]  [<ffffffffa01bdeb6>] virtnet_poll+0x536/0x690 [virtio_net]
[162335.684274]  [<ffffffff812a94a2>] net_rx_action+0x162/0x280
[162335.684274]  [<ffffffff81059cdc>] __do_softirq+0xcc/0x230
[162335.684274]  [<ffffffff8100c59c>] call_softirq+0x1c/0x30
[162335.684274]  [<ffffffff8100e6a5>] do_softirq+0x65/0xa0
[162335.684274]  [<ffffffff81059b97>] irq_exit+0x87/0x90
[162335.684274]  [<ffffffff8100da77>] do_IRQ+0x77/0xf0
[162335.684274]  [<ffffffff8100bd93>] ret_from_intr+0x0/0x11
[162335.684274]  <EOI>  [<ffffffff8102a496>] ? native_safe_halt+0x6/0x10
[162335.684274]  [<ffffffff810136c8>] ? default_idle+0x38/0xc0
[162335.684274]  [<ffffffff8100a215>] ? cpu_idle+0x55/0x90
[162335.684274]  [<ffffffff81337ea3>] ? start_secondary+0x183/0x1e0
[162002.937850] NMI backtrace for cpu 5
[162002.937850] CPU 5:
[162002.937850] Modules linked in: bluetooth rfkill be2iscsi iscsi_tcp bnx2i 
cnic uio cxgb3i cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi 
dm_snapshot dm_zero netconsole ipv6 nfs nfs_acl auth_rpcgss lockd sunrpc 
dm_mirror dm_region_hash dm_log dm_mod video backlight output sbs sbshc fan 
container battery ac thermal processor button virtio_balloon pcspkr i2c_piix4 
thermal_sys i2c_core sg virtio_blk virtio_net virtio_pci virtio_ring virtio 
8139cp mii pata_acpi ide_pci_generic ide_core ata_piix ata_generic libata 
sd_mod scsi_mod crc_t10dif ext4 jbd2 crc16 uhci_hcd ohci_hcd ehci_hcd [last 
unloaded: scsi_wait_scan]
[162002.937850] Pid: 19, comm: ksoftirqd/5 Not tainted 2.6.32lb.29.kvmguest #1 
Bochs
[162002.937850] RIP: 0010:[<ffffffffa001cb1d>]  [<ffffffffa001cb1d>] 
uhci_check_ports+0x4d/0x170 [uhci_hcd]
[162002.937850] RSP: 0018:ffff880028343df8  EFLAGS: 00010006
[162002.937850] RAX: 0000000000000080 RBX: 000000000000c032 RCX: 
0000000000000000
[162002.937850] RDX: 000000000000c032 RSI: 0000000000000101 RDI: 
ffff88011a211550
[162002.937850] RBP: ffff880028343e38 R08: 0000000000000000 R09: 
dead000000200200
[162002.937850] R10: 0000000000000000 R11: 0000000000000000 R12: 
0000000000000001
[162002.937850] R13: ffff88011a211550 R14: 000000000000c032 R15: 
0000000000000001
[162002.937850] FS:  0000000000000000(0000) GS:ffff880028340000(0000) 
knlGS:0000000000000000
[162002.937850] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
[162002.937850] CR2: 00007f28972c4980 CR3: 00000000aacc1000 CR4: 
00000000000006e0
[162002.937850] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[162002.937850] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[162002.937850] Call Trace:
[162002.937850]  <#DB[1]>  <<EOE>> Pid: 19, comm: ksoftirqd/5 Not tainted 
2.6.32lb.29.kvmguest #1
[162002.937850] Call Trace:
[162002.937850]  <NMI>  [<ffffffff81009d83>] ? show_regs+0x43/0x50
[162002.937850]  [<ffffffff81024f4a>] nmi_watchdog_tick+0x17a/0x260
[162002.937850]  [<ffffffff8100cc5d>] do_nmi+0x8d/0x290
[162002.937850]  [<ffffffff8133f820>] nmi+0x20/0x30
[162002.937850]  [<ffffffffa001cb1d>] ? uhci_check_ports+0x4d/0x170 [uhci_hcd]
[162002.937850]  <<EOE>>  <IRQ>  [<ffffffffa001dd58>] 
uhci_hub_status_data+0x88/0x1c0 [uhci_hcd]
[162002.937850]  [<ffffffff811c639e>] ? _raw_spin_unlock+0x5e/0xb0
[162002.937850]  [<ffffffff8124c387>] usb_hcd_poll_rh_status+0x57/0x180
[162002.937850]  [<ffffffff8124c4b9>] rh_timer_func+0x9/0x10
[162002.937850]  [<ffffffff81060b9d>] run_timer_softirq+0x15d/0x320
[162002.937850]  [<ffffffff8124c4b0>] ? rh_timer_func+0x0/0x10
[162002.937850]  [<ffffffff81059cdc>] __do_softirq+0xcc/0x230
[162002.937850]  [<ffffffff8100c59c>] call_softirq+0x1c/0x30
[162002.937850]  <EOI>  [<ffffffff8100e6a5>] do_softirq+0x65/0xa0
[162002.937850]  [<ffffffff8105a819>] ksoftirqd+0x79/0x1a0
[162002.937850]  [<ffffffff8105a7a0>] ? ksoftirqd+0x0/0x1a0
[162002.937850]  [<ffffffff81070f0e>] kthread+0x8e/0xa0
[162002.937850]  [<ffffffff8158a140>] ? early_idt_handler+0x0/0x71
[162002.937850]  [<ffffffff8100c49a>] child_rip+0xa/0x20
[162002.937850]  [<ffffffff8158a140>] ? early_idt_handler+0x0/0x71
[162002.937850]  [<ffffffff81070e80>] ? kthread+0x0/0xa0
[162002.937850]  [<ffffffff8100c490>] ? child_rip+0x0/0x20
[162335.694727] sd 0:0:0:0: [sda] Unhandled error code
[162335.694754] sd 0:0:0:0: [sda] 
[162335.694768] BUG: soft lockup - CPU#5 stuck for 310s! [ksoftirqd/5:19]
[162335.694779] Modules linked in: bluetooth rfkill be2iscsi iscsi_tcpResult: 
hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[162335.694849]  bnx2i
[162335.694863] sd 0:0:0:0: [sda] CDB: Write(10) cnic uio cxgb3i: 2a cxgb3 mdio 
00 00 3e a4 90 00 00 18 00
[162335.695060]  libiscsi_tcp
[162335.695067] end_request: I/O error, dev sda, sector 4105360
[162335.695143]  libiscsi scsi_transport_iscsi dm_snapshot dm_zero netconsole 
ipv6 nfs nfs_acl auth_rpcgss lockd sunrpc dm_mirror dm_region_hash dm_log 
dm_mod video backlight output sbs sbshc fan container battery ac thermal 
processor button virtio_balloon pcspkr i2c_piix4 thermal_sys i2c_core sg 
virtio_blk virtio_net virtio_pci virtio_ring virtio 8139cp mii pata_acpi 
ide_pci_generic ide_core ata_piix ata_generic libata sd_mod scsi_mod crc_t10dif 
ext4 jbd2 crc16 uhci_hcd ohci_hcd ehci_hcd [last unloaded: scsi_wait_scan]
[162335.696012] CPU 5:
[162335.696127] Modules linked in: bluetooth rfkill be2iscsi iscsi_tcp bnx2i 
cnic uio cxgb3i cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi 
dm_snapshot dm_zero netconsole ipv6 nfs nfs_acl auth_rpcgss lockd sunrpc 
dm_mirror dm_region_hash dm_log dm_mod video backlight output sbs sbshc fan 
container battery ac thermal processor button virtio_balloon pcspkr i2c_piix4 
thermal_sys i2c_core sg virtio_blk virtio_net virtio_pci virtio_ring virtio 
8139cp mii pata_acpi ide_pci_generic ide_core ata_piix ata_generic libata 
sd_mod scsi_mod crc_t10dif ext4 jbd2 crc16 uhci_hcd ohci_hcd ehci_hcd [last 
unloaded: scsi_wait_scan]
[162335.696127] Pid: 19, comm: ksoftirqd/5 Not tainted 2.6.32lb.29.kvmguest #1 
Bochs
[162335.696127] RIP: 0010:[<ffffffff8133ee50>]  [<ffffffff8133ee50>] 
_spin_unlock_irq+0x10/0x40
[162335.696127] RSP: 0018:ffff880028343ed8  EFLAGS: 00000206
[162335.696127] RAX: 0000000000000005 RBX: ffff880028343ed8 RCX: 
ffff88011f900fd0
[162335.696127] RDX: 0000000000005554 RSI: ffff880028343ea8 RDI: 
ffff88011f900000
[162335.696127] RBP: ffffffff8100bf63 R08: 0000000000001e08 R09: 
0000000000000000
[162335.696127] R10: 000000000000012c R11: 0000000000000000 R12: 
ffff880028343e50
[162335.696127] R13: ffff88011f900000 R14: ffff880028343f08 R15: 
ffffffff81024857
[162335.696127] FS:  0000000000000000(0000) GS:ffff880028340000(0000) 
knlGS:0000000000000000
[162335.696127] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
[162335.696127] CR2: 00007f28972c4980 CR3: 00000000aacc1000 CR4: 
00000000000006e0
[162335.696127] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[162335.696127] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[162335.696127] Call Trace:
[162335.696127]  <IRQ>  [<ffffffff8133ee49>] ? _spin_unlock_irq+0x9/0x40
[162335.696127]  [<ffffffff81060c54>] ? run_timer_softirq+0x214/0x320
[162335.696127]  [<ffffffff8101d410>] ? mcheck_timer+0x0/0x120
[162335.696127]  [<ffffffff81059cdc>] ? __do_softirq+0xcc/0x230
[162335.696127]  [<ffffffff8100c59c>] ? call_softirq+0x1c/0x30
[162335.696127]  <EOI>  [<ffffffff8100e6a5>] ? do_softirq+0x65/0xa0
[162335.696127]  [<ffffffff8105a819>] ? ksoftirqd+0x79/0x1a0
[162335.696127]  [<ffffffff8105a7a0>] ? ksoftirqd+0x0/0x1a0
[162335.696127]  [<ffffffff81070f0e>] ? kthread+0x8e/0xa0
[162335.696793]  [<ffffffff8158a140>] ? early_idt_handler+0x0/0x71
[162335.696793]  [<ffffffff8100c49a>] ? child_rip+0xa/0x20
[162335.696793]  [<ffffffff8158a140>] ? early_idt_handler+0x0/0x71
[162335.696793]  [<ffffffff81070e80>] ? kthread+0x0/0xa0
[162335.696793]  [<ffffffff8100c490>] ? child_rip+0x0/0x20
[162335.697039] Aborting journal on device sda-8.
[162335.697082] EXT4-fs error (device sda): ext4_journal_start_sb: 
[162335.697146] EXT4-fs error (device sda) in ext4_reserve_inode_write: Journal 
has aborted
[162335.697226] EXT4-fs error (device sda) in ext4_reserve_inode_write: Journal 
has aborted
[162335.702749] Detected aborted journal
[162335.702752] EXT4-fs (sda): Remounting filesystem read-only
[162335.705754] EXT4-fs error (device sda) in ext4_dirty_inode: Journal has 
aborted
[162335.707725] EXT4-fs (sda): ext4_da_writepages: jbd2_start: 1024 pages, ino 
32392; err -30

I'm using virtio network channel, and on one of the guests (the one with 
aborted ext4) I use it also for one of virtual disks.
One more interesting thing, I can't reproduce this immediately after guest 
boot, but for example second day after boot, I can reproduce this.
perhaps this can suggest something?
Could somebody please help me to find and possibly fix this bug?
If needed, I can provide further debugging information, bisect etc...
Thanks a lot in advance!
nik


-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: address@hidden
-------------------------------------