Re: [Qemu-devel] [v10][PATCH 03/10] piix: create host bridge to passthrough

[Kay, Allen M]

> -----Original Message-----
> From: Alex Williamson [mailto:address@hidden
> Sent: Tuesday, February 09, 2016 10:01 PM
> To: Kay, Allen M <address@hidden>
> Cc: address@hidden; address@hidden;
> address@hidden; address@hidden;
> address@hidden; address@hidden; Ruan, Shuai
> <address@hidden>
> Subject: Re: [Qemu-devel] [v10][PATCH 03/10] piix: create host bridge to
> passthrough
> 
> I can run that system as either primary or secondary.  On both systems I'm
> using pci-stub to keep i915 from interfering on the host, on the SNB laptop I
> also use video=efifb:off since it's configured for UEFI boot, I'm not sure if
> that play any role in why it's not working.
> 

Hi Alex,

My understand if that your IVB system is a desktop running legacy mode BIOS and 
your SNB is a laptop running EFI mode BIOS, correct?  I'm curious how does your 
SNB system getting hold of VBT table, which is needed for lighting up local 
display.

There are two ways the driver can get VBT table:  1) VGA 0xc0000 memory,  2) 
OpRegion

On your IVB system, the guest driver would try to get VBT from 0xc0000 memory 
if IGD is configured as primary controller in the guest, assuming KVM/QEMU 
copies VGA 0xc0000 memory from host to guest 0xc0000 area.   If IGD is 
configured as secondary controller, the driver would get VBT from OpRegion.  
Given IGD works in both configurations on IVB, this mean guest driver can 
successfully read VBT from both 0xc0000 area and OpRegion.

On your SNB system that is running in EFI mode on the host, 0xc0000 memory does 
not contain VBIOS/VBT.  This means if you configure IGD as primary controller 
in the guest, the driver cannot get to VBT at 0xc0000 memory even if KVM/QEMU 
copies 0xc0000 memory from host to guest.  If you configure IGD as secondary 
controller in the guest, and guest driver should be able to read VBT from the 
OpRegion.  You might want to trace i915 to see how does the  guest driver get 
to VBT via OpRegion in this configuration.

Another potential problem to watch out for with laptop vs. desktop has to do 
FLR timeout.  If you are working with a laptop with LCD panel attached (the 
usual case), FLR will take much longer than 100ms to finish.  The reason is the 
devices needs to power down the LCD panel before it can finish FLR. I have seen 
it takes more than 500ms for FLR to finish.  As a work around, I have increase 
FLR time out in Linux PCI driver to 1000ms when working with LCD panels.   
Given that you have seen evidences that IGD HW is working, this might not be 
your issue.   I would focus tracing how does i915 get VBT from the OpRegion 
when IGD is configured as secondary controller in the guest.

Allen


> I've tried adding a bunch more registers on the SNB system to see if it helps,
> for the host bridge:
> 
>     {0x50,                    2},
>     {0x52,                    2},
>     {0xa4,                    4},
>     {0xa8,                    4},
> 
> (ie. the registers Tiejun added with this patch)
> 
> And on the ISA bridge:
> 
>     {0x40,        4},
>     {0x44,        4},
>     {0x48,        4},
>     {0x4c,        4},
>     {0xf0,        4},
> 
> These were registers I saw accessed with tracing enabled, but nothing
> seemed to change by adding them. I don't see any accesses to the BDSM
> register on the host bridge, but I do see GMCH accesses.  Sort of interesting
> is that the guest reads 201h from that register and tries to write 203h. In 
> the
> read case we have a 2MB GGMS and GGCLCK set, the guest tries to set IVD
> (IGD VGA Disable). I'm not sure why it bothers to try to do this with GGCLCK
> indicated since the register is locked, but it is slightly troubling that the 
> spec
> indicates that the BIOS must
> *not* set IVD to zero (VGA enabled) if GMS pre-allocates no memory, which
> is exactly what we're doing to try to indicate no stolen memory.
> If we could actually disable VGA on IGD, that might be a nice option, but I
> know there are issues with that (iirc, there's no disable for one of either 
> the
> VGA MMIO or I/O port space, so the only option is to disable that address
> space at the PCI command register, which has obvious implications - I think it
> was probably I/O port space).
> 
> I'm currently trying a Linux live CD on the guest for the SNB, mostly because 
> I
> can see driver error messages and look at the driver source, neither of which
> are available to me for the Windows driver.  Both of the driver warnings I get
> are from the driver thinking a certain feature should be disabled but finds it
> enabled.  They're also sort of on a strange drm release path, maybe due to
> being in secondary mode and Xorg not being configured to use the IGD on
> this image.  In case it looks familiar:
> 
> [   12.397908] ------------[ cut here ]------------
> [   12.399175] WARNING: CPU: 0 PID: 1135 at
> drivers/gpu/drm/i915/intel_display.c:1259 assert_fdi_rx_pll+0x78/0xb0
> [i915]()
> [   12.401720] FDI RX PLL assertion failure (expected off, current on)
> [   12.403239] Modules linked in: ebtable_broute bridge stp llc ebtable_filter
> ebtable_nat ebtables ip6table_security ip6table_mangle ip6table_raw
> ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_filter
> ip6_tables iptable_security iptable_mangle iptable_raw iptable_nat
> nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack
> iTCO_wdt gpio_ich iTCO_vendor_support ppdev joydev parport_pc lpc_ich
> i2c_piix4 parport acpi_cpufreq nfsd auth_rpcgss nfs_acl lockd grace isofs
> squashfs i915 bochs_drm video ttm i2c_algo_bit drm_kms_helper drm
> ata_generic serio_raw pata_acpi scsi_dh_rdac scsi_dh_emc scsi_dh_alua
> sunrpc loop
> [   12.417994] CPU: 0 PID: 1135 Comm: Xorg Not tainted 4.2.3-300.fc23.x86_64
> #1
> [   12.419733] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
> rel-1.9.0-80-g2390eb5 04/01/2014
> [   12.422100]  0000000000000000 00000000d92c128c ffff880078f53ab8
> ffffffff81771fca
> [   12.424023]  0000000000000000 ffff880078f53b10 ffff880078f53af8
> ffffffff8109e4a6
> [   12.425949]  ffff880078f53ad8 0000000000000000 ffff88007f680000
> ffff88007fb72000
> [   12.427719] Call Trace:
> [   12.428354]  [<ffffffff81771fca>] dump_stack+0x45/0x57
> [   12.429580]  [<ffffffff8109e4a6>] warn_slowpath_common+0x86/0xc0
> [   12.431037]  [<ffffffff8109e535>] warn_slowpath_fmt+0x55/0x70
> [   12.432497]  [<ffffffffa01e5a98>] assert_fdi_rx_pll+0x78/0xb0 [i915]
> [   12.434033]  [<ffffffffa0221433>] intel_pre_enable_lvds+0x53/0x180 [i915]
> [   12.435760]  [<ffffffffa01ef729>] ironlake_crtc_enable+0x199/0xd30 [i915]
> [   12.437433]  [<ffffffffa0190c80>] ? intel_runtime_pm_put+0x50/0x60 [i915]
> [   12.439083]  [<ffffffffa0190d98>] ? intel_display_power_put+0x108/0x160
> [i915]
> [   12.440914]  [<ffffffffa01ecff6>] __intel_set_mode+0x916/0xb60 [i915]
> [   12.442458]  [<ffffffffa01f3ee6>] intel_crtc_set_config+0x2b6/0x580 [i915]
> [   12.444180]  [<ffffffffa0099326>]
> drm_mode_set_config_internal+0x66/0x100 [drm]
> [   12.445917]  [<ffffffffa016d512>] restore_fbdev_mode+0xc2/0xf0
> [drm_kms_helper]
> [   12.447695]  [<ffffffffa016f399>]
> drm_fb_helper_restore_fbdev_mode_unlocked+0x29/0x70
> [drm_kms_helper]
> [   12.450116]  [<ffffffffa020350e>] intel_fbdev_restore_mode+0x1e/0x50
> [i915]
> [   12.451794]  [<ffffffffa022c96e>] i915_driver_lastclose+0xe/0x20 [i915]
> [   12.453360]  [<ffffffffa008c8ee>] drm_lastclose+0x2e/0x140 [drm]
> [   12.454755]  [<ffffffffa008ccaf>] drm_release+0x2af/0x490 [drm]
> [   12.456324]  [<ffffffff8121fbfc>] __fput+0xdc/0x1e0
> [   12.457525]  [<ffffffff8121fd4e>] ____fput+0xe/0x10
> [   12.458736]  [<ffffffff810bab65>] task_work_run+0x85/0xb0
> [   12.460028]  [<ffffffff81014a2d>] do_notify_resume+0x8d/0x90
> [   12.461424]  [<ffffffff81778b7c>] int_signal+0x12/0x17
> [   12.462757] ---[ end trace 3278f15671a6fa94 ]---
> [   12.546650] ------------[ cut here ]------------
> [   12.547765] WARNING: CPU: 0 PID: 1135 at
> drivers/gpu/drm/i915/intel_display.c:1466
> assert_pch_transcoder_disabled+0x67/0x70 [i915]()
> [   12.550847] transcoder assertion failed, should be off on pipe A but is 
> still
> active
> [   12.552858] Modules linked in: ebtable_broute bridge stp llc ebtable_filter
> ebtable_nat ebtables ip6table_security ip6table_mangle ip6table_raw
> ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_filter
> ip6_tables iptable_security iptable_mangle iptable_raw iptable_nat
> nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack
> iTCO_wdt gpio_ich iTCO_vendor_support ppdev joydev parport_pc lpc_ich
> i2c_piix4 parport acpi_cpufreq nfsd auth_rpcgss nfs_acl lockd grace isofs
> squashfs i915 bochs_drm video ttm i2c_algo_bit drm_kms_helper drm
> ata_generic serio_raw pata_acpi scsi_dh_rdac scsi_dh_emc scsi_dh_alua
> sunrpc loop
> [   12.568444] CPU: 0 PID: 1135 Comm: Xorg Tainted: G        W       4.2.3-
> 300.fc23.x86_64 #1
> [   12.570599] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
> rel-1.9.0-80-g2390eb5 04/01/2014
> [   12.572988]  0000000000000000 00000000d92c128c ffff880078f53b08
> ffffffff81771fca
> [   12.574973]  0000000000000000 ffff880078f53b60 ffff880078f53b48
> ffffffff8109e4a6
> [   12.576826]  ffff880078f53b28 0000000000000000 ffff88007f680000
> ffff88007fb72000
> [   12.578804] Call Trace:
> [   12.579454]  [<ffffffff81771fca>] dump_stack+0x45/0x57
> [   12.580775]  [<ffffffff8109e4a6>] warn_slowpath_common+0x86/0xc0
> [   12.582419]  [<ffffffff8109e535>] warn_slowpath_fmt+0x55/0x70
> [   12.583924]  [<ffffffffa01e1067>]
> assert_pch_transcoder_disabled+0x67/0x70 [i915]
> [   12.585834]  [<ffffffffa01efb7c>] ironlake_crtc_enable+0x5ec/0xd30 [i915]
> [   12.587634]  [<ffffffffa0190c80>] ? intel_runtime_pm_put+0x50/0x60 [i915]
> [   12.589402]  [<ffffffffa0190d98>] ? intel_display_power_put+0x108/0x160
> [i915]
> [   12.591294]  [<ffffffffa01ecff6>] __intel_set_mode+0x916/0xb60 [i915]
> [   12.593019]  [<ffffffffa01f3ee6>] intel_crtc_set_config+0x2b6/0x580 [i915]
> [   12.594797]  [<ffffffffa0099326>]
> drm_mode_set_config_internal+0x66/0x100 [drm]
> [   12.596651]  [<ffffffffa016d512>] restore_fbdev_mode+0xc2/0xf0
> [drm_kms_helper]
> [   12.598574]  [<ffffffffa016f399>]
> drm_fb_helper_restore_fbdev_mode_unlocked+0x29/0x70
> [drm_kms_helper]
> [   12.601022]  [<ffffffffa020350e>] intel_fbdev_restore_mode+0x1e/0x50
> [i915]
> [   12.602888]  [<ffffffffa022c96e>] i915_driver_lastclose+0xe/0x20 [i915]
> [   12.604576]  [<ffffffffa008c8ee>] drm_lastclose+0x2e/0x140 [drm]
> [   12.606098]  [<ffffffffa008ccaf>] drm_release+0x2af/0x490 [drm]
> [   12.607662]  [<ffffffff8121fbfc>] __fput+0xdc/0x1e0
> [   12.609019]  [<ffffffff8121fd4e>] ____fput+0xe/0x10
> [   12.610283]  [<ffffffff810bab65>] task_work_run+0x85/0xb0
> [   12.611718]  [<ffffffff81014a2d>] do_notify_resume+0x8d/0x90
> [   12.613225]  [<ffffffff81778b7c>] int_signal+0x12/0x17
> [   12.614600] ---[ end trace 3278f15671a6fa95 ]---
> [   12.635700] [drm:intel_set_pch_fifo_underrun_reporting [i915]] *ERROR*
> uncleared pch fifo underrun on pch transcoder A
> [   12.636007] [drm:intel_pch_fifo_underrun_irq_handler [i915]] *ERROR*
> PCH transcoder A FIFO underrun
> [   12.642883] [drm:intel_cpu_fifo_underrun_irq_handler [i915]] *ERROR*
> CPU pipe A FIFO underrun
> 
> Otherwise i915 seems fairly happy:
> 
> [    4.940225] i915: No ACPI video bus found
> [    4.943223] [drm] Initialized i915 1.6.0 20150522 for 0000:00:03.0 on 
> minor 1
> [    5.000053] [drm] GMBUS [i915 gmbus vga] timed out, falling back to bit
> banging on pin 2
> [    5.038055] i915 0000:00:03.0: fb1: inteldrmfb frame buffer device
> 
> The bit banging messages is apparently normal on this system (Lenovo W520,
> Nvidia disabled in BIOS - could optimus still be getting in the way?). I don't
> know that any of the above is actually fatal though, I just know that I don't
> get a signal out of the LVDS panel or any of the other outputs. However, the
> Xorg log file looks normal and if I disable mmap support in vfio and move the
> mouse cursor around on the VNC window, the trace log looks a lot like
> graphics is running ok, the panel is just turned off. I'm thinking that makes
> the above warnings more relevant since they're in the space of FDI and PCH
> transcoders, which I think is how we get to a display output.
> 
> 
> There are certainly a lot of subtle differences between IVB and SNB in the
> Linux driver, if I have time I'll see about setting up a Windows VM on the SNB
> system.  Linux feels pretty close though if I could just get the panel turned
> on.
> 
> 
> Good, glad you found it.  I'm just running a simple VM like this:
> 
> /home/alwillia/local/bin/qemu-system-x86_64 -m 2G -bios
> /home/alwillia/Work/seabios.git/out/bios.bin -net none -cdrom
> /net/store/exports/ISOs/Fedora-Live-Cinnamon-x86_64-23-10.iso -boot d -
> monitor stdio -device vfio-pci,host=00:02.0,rombar=0,x-no-mmap=on -vnc :1
> -vga std -enable-kvm -trace events=events.txt
> 
> events.txt contains:
> vfio_region_read
> vfio_region_write
> vfio_pci_read_config
> vfio_pci_write_config
> vfio_pci_igd*
> pci_cfg_*
> 
> If you base on the last patch series I posted on top of Gerd's changes, you'll
> need -M pc,igd_passthru=on I believe too, the code I'm working on passes
> the host and ISA bridge config through vfio and initiates the ISA bridge
> automatically.  I'll try to post it this week.
> 
> > > Again, it seems just as easy, if not easier to copy GMCH from the
> > > IGD itself into the host bridge.
> > >
> > Not sure if I follow ... how does this would solve stolen memory/RMRR
> issue...?
> 
> In the vfio code I have now, the kernel virtualizes GMCH and BDSM on the
> device to provide a pre-sanitized version, so if we need those on the host
> bridge, it's easier to copy what's on the IGD instead of adding yet more code
> that knows about the different GMCH layouts.  The solution is the same,
> clear BDSM and GMCH.GMS to make the guest not try to access the host
> stolen memory.
> 
.
> 
> If I could spot any accesses to BDSM on the host bridge, I'd jump on this, but
> my guest driver doesn't seem to care about it according to the tracing.
> Lighting up an output seems to be the issue.  Thanks,
> 
> Alex