Re: [Qemu-discuss] QEMU PCI Passthrough FPGA with more than 2GB never starts

[Miltos Hatzimihail]

> 
> On 06/05/2017 09:39, Miltos Hatzimihail wrote:
>> Dear All
>> 
>> I ve got a system with a Xilinx FPGA on a PCI express slot and I am trying 
>> to pass it through using QEMU with vfio-pci. Here?s my command line:
>> 
>> qemu-system-x86_64 -smp 4,sockets=2,cores=1,threads=2 -machine 
>> pc-i440fx-xenial,accel=kvm  --cpu qemu64 -enable-kvm -vnc :1   -device 
>> vfio-pci,host=04:01.0 .. .. .. -mem 2G
>> 
>> The VM starts OK with men = 2G. If I try to assign more memory to this host 
>> (say 4GB), then the VM doesn?t even boot to the BIOS (it says that ?This 
>> guest has initialised the display ?yet?). The weird thing is that if I omit 
>> the passthrough, the VM starts ok with 4GB.
>> 
>> The PCI device is configured as follows:
>> 
>>         Region 0: Memory at c1100000 (32-bit, non-prefetchable) [disabled] 
>> [size=128K]
>>         Region 1: Memory at c0000000 (32-bit, non-prefetchable) [disabled] 
>> [size=16M]
>>         Region 2: Memory at 80000000 (32-bit, non-prefetchable) [disabled] 
>> [size=1G]
>>         [virtual] Expansion ROM at c1000000 [disabled] [size=1M]
>> 
>> I tried to find an answer online but I had no luck. What am I missing?
>> 
>> Regards,
>> Milton
>> 
> Look at Region 2: It is located at physical (virtual physical?) address
> 2GB (0x80000000), thus it won't work in a machine with RAM directly
> above the 2GB mark.
> 
FWIW this setup works OK if I use Xen (the passthrough works and the men I am 
assigning to the guest is 14G). So I am trying to understand what’s different.

> I don't know what would happen if that card was inserted in a physical
> machine with > 2GB RAM.  Maybe there would be a "memory hole" from
> 0x80000000 to 0xC1120000, maybe it would get assigned different physical
> addresses.
> 

I looked at the monitor for stdio when I give 4G and I see this

  1 address-space: memory
  2   0000000000000000-ffffffffffffffff (prio 0, RW): system
  3     0000000000000000-00000000bfffffff (prio 0, RW): alias ram-below-4g 
@pc.ram 0000000000000000-00000000bfffffff
  4     0000000000000000-ffffffffffffffff (prio -1, RW): pci
  5       00000000000a0000-00000000000bffff (prio 1, RW): vga-lowmem
  6       00000000000c0000-00000000000dffff (prio 1, RW): pc.rom
  7       00000000000e0000-00000000000fffff (prio 1, R-): alias isa-bios 
@pc.bios 0000000000020000-000000000003ffff
  8       00000000fffc0000-00000000ffffffff (prio 0, R-): pc.bios
  9     00000000000a0000-00000000000bffff (prio 1, RW): alias smram-region @pci 
00000000000a0000-00000000000bffff

But when I give 2G I see

 1 address-space: memory
  2   0000000000000000-ffffffffffffffff (prio 0, RW): system
  3     0000000000000000-000000007fffffff (prio 0, RW): alias ram-below-4g 
@pc.ram 0000000000000000-000000007fffffff
  4     0000000000000000-ffffffffffffffff (prio -1, RW): pci
  5       00000000000a0000-00000000000affff (prio 2, RW): alias vga.chain4 
@vga.vram 0000000000000000-000000000000ffff
  6       00000000000a0000-00000000000bffff (prio 1, RW): vga-lowmem
  7       00000000000c0000-00000000000dffff (prio 1, RW): pc.rom
  8       00000000000e0000-00000000000fffff (prio 1, R-): alias isa-bios 
@pc.bios 0000000000020000-000000000003ffff
  9       0000000080000000-00000000bfffffff (prio 1, RW): VFIO 0000:01:00.0 BAR 
2
 10         0000000080000000-00000000bfffffff (prio 0, RW): VFIO 0000:01:00.0 
BAR 2 mmap
 11       00000000c0000000-00000000c0ffffff (prio 1, RW): VFIO 0000:01:00.0 BAR 
1
 12         00000000c0000000-00000000c0ffffff (prio 0, RW): VFIO 0000:01:00.0 
BAR 1 mmap
 13       00000000c1040000-00000000c105ffff (prio 1, RW): VFIO 0000:01:00.0 BAR 0
 14         00000000c1040000-00000000c105ffff (prio 0, RW): VFIO 0000:01:00.0 
BAR 0 mmap
 15       00000000c1070000-00000000c1070fff (prio 1, RW): vga.mmio
 16         00000000c1070400-00000000c107041f (prio 0, RW): vga ioports remapped
 17         00000000c1070500-00000000c1070515 (prio 0, RW): bochs dispi 
interface
 18         00000000c1070600-00000000c1070607 (prio 0, RW): qemu extended regs
 19       00000000c1071000-00000000c1071fff (prio 1, RW): virtio-net-pci-msix
 20         00000000c1071000-00000000c107102f (prio 0, RW): msix-table
 21         00000000c1071800-00000000c1071807 (prio 0, RW): msix-pba
 22       00000000fd000000-00000000fdffffff (prio 1, RW): vga.vram
 23       00000000fffc0000-00000000ffffffff (prio 0, R-): pc.bios
 24     00000000000a0000-00000000000bffff (prio 1, RW): alias smram-region @pci 
00000000000a0000-00000000000bffff

which is what you are saying but I see more stuff missing - like vga.vram or 
vga.mmio.. Is there a setting I am missing when I run QEMU?

> If your host machine where it is actually inserted has more than 2GB RAM,
> look at the RAM and PCI memory mappings logged in /var/log/dmesg and in
> the /proc/ and /sys/ file systems.
> 
It does have 20GB but I can’t figure out what’s going on.. 

> Enjoy
> 
> Jakob
> -- 
> Jakob Bohm, CIO, Partner, WiseMo A/S.  https://www.wisemo.com
> Transformervej 29, 2860 S?borg, Denmark.  Direct +45 31 13 16 10
> This public discussion message is non-binding and may contain errors.
> WiseMo - Remote Service Management for PCs, Phones and Embedded