Re: [Qemu-discuss] QEMU PCI Passthrough FPGA with more than 2GB never starts

[Miltos Hatzimihail]

The max-ram-below-4g was the key here. I ve added this in the machine option

-machine pc-i440fx-xenial,accel=kvm,max-ram-below-4g=2G 

And QEMU starts fine with the passthrough.

Milton

> On 10 May 2017, at 20:25, Miltos Hatzimihail <address@hidden> wrote:
> 
>> 
>> On 06/05/2017 09:39, Miltos Hatzimihail wrote:
>>> Dear All
>>> 
>>> I ve got a system with a Xilinx FPGA on a PCI express slot and I am trying 
>>> to pass it through using QEMU with vfio-pci. Here?s my command line:
>>> 
>>> qemu-system-x86_64 -smp 4,sockets=2,cores=1,threads=2 -machine 
>>> pc-i440fx-xenial,accel=kvm  --cpu qemu64 -enable-kvm -vnc :1   -device 
>>> vfio-pci,host=04:01.0 .. .. .. -mem 2G
>>> 
>>> The VM starts OK with men = 2G. If I try to assign more memory to this host 
>>> (say 4GB), then the VM doesn?t even boot to the BIOS (it says that ?This 
>>> guest has initialised the display ?yet?). The weird thing is that if I omit 
>>> the passthrough, the VM starts ok with 4GB.
>>> 
>>> The PCI device is configured as follows:
>>> 
>>>         Region 0: Memory at c1100000 (32-bit, non-prefetchable) [disabled] 
>>> [size=128K]
>>>         Region 1: Memory at c0000000 (32-bit, non-prefetchable) [disabled] 
>>> [size=16M]
>>>         Region 2: Memory at 80000000 (32-bit, non-prefetchable) [disabled] 
>>> [size=1G]
>>>         [virtual] Expansion ROM at c1000000 [disabled] [size=1M]
>>> 
>>> I tried to find an answer online but I had no luck. What am I missing?
>>> 
>>> Regards,
>>> Milton
>>> 
>> Look at Region 2: It is located at physical (virtual physical?) address
>> 2GB (0x80000000), thus it won't work in a machine with RAM directly
>> above the 2GB mark.
>> 
> FWIW this setup works OK if I use Xen (the passthrough works and the men I am 
> assigning to the guest is 14G). So I am trying to understand what’s different.
> 
>> I don't know what would happen if that card was inserted in a physical
>> machine with > 2GB RAM.  Maybe there would be a "memory hole" from
>> 0x80000000 to 0xC1120000, maybe it would get assigned different physical
>> addresses.
>> 
> 
> I looked at the monitor for stdio when I give 4G and I see this
> 
>   1 address-space: memory
>   2   0000000000000000-ffffffffffffffff (prio 0, RW): system
>   3     0000000000000000-00000000bfffffff (prio 0, RW): alias ram-below-4g 
> @pc.ram 0000000000000000-00000000bfffffff
>   4     0000000000000000-ffffffffffffffff (prio -1, RW): pci
>   5       00000000000a0000-00000000000bffff (prio 1, RW): vga-lowmem
>   6       00000000000c0000-00000000000dffff (prio 1, RW): pc.rom
>   7       00000000000e0000-00000000000fffff (prio 1, R-): alias isa-bios 
> @pc.bios 0000000000020000-000000000003ffff
>   8       00000000fffc0000-00000000ffffffff (prio 0, R-): pc.bios
>   9     00000000000a0000-00000000000bffff (prio 1, RW): alias smram-region 
> @pci 00000000000a0000-00000000000bffff
> 
> But when I give 2G I see
> 
>  1 address-space: memory
>   2   0000000000000000-ffffffffffffffff (prio 0, RW): system
>   3     0000000000000000-000000007fffffff (prio 0, RW): alias ram-below-4g 
> @pc.ram 0000000000000000-000000007fffffff
>   4     0000000000000000-ffffffffffffffff (prio -1, RW): pci
>   5       00000000000a0000-00000000000affff (prio 2, RW): alias vga.chain4 
> @vga.vram 0000000000000000-000000000000ffff
>   6       00000000000a0000-00000000000bffff (prio 1, RW): vga-lowmem
>   7       00000000000c0000-00000000000dffff (prio 1, RW): pc.rom
>   8       00000000000e0000-00000000000fffff (prio 1, R-): alias isa-bios 
> @pc.bios 0000000000020000-000000000003ffff
>   9       0000000080000000-00000000bfffffff (prio 1, RW): VFIO 0000:01:00.0 
> BAR 2
>  10         0000000080000000-00000000bfffffff (prio 0, RW): VFIO 0000:01:00.0 
> BAR 2 mmap
>  11       00000000c0000000-00000000c0ffffff (prio 1, RW): VFIO 0000:01:00.0 
> BAR 1
>  12         00000000c0000000-00000000c0ffffff (prio 0, RW): VFIO 0000:01:00.0 
> BAR 1 mmap
>  13       00000000c1040000-00000000c105ffff (prio 1, RW): VFIO 0000:01:00.0 
> BAR 0
>  14         00000000c1040000-00000000c105ffff (prio 0, RW): VFIO 0000:01:00.0 
> BAR 0 mmap
>  15       00000000c1070000-00000000c1070fff (prio 1, RW): vga.mmio
>  16         00000000c1070400-00000000c107041f (prio 0, RW): vga ioports 
> remapped
>  17         00000000c1070500-00000000c1070515 (prio 0, RW): bochs dispi 
> interface
>  18         00000000c1070600-00000000c1070607 (prio 0, RW): qemu extended regs
>  19       00000000c1071000-00000000c1071fff (prio 1, RW): virtio-net-pci-msix
>  20         00000000c1071000-00000000c107102f (prio 0, RW): msix-table
>  21         00000000c1071800-00000000c1071807 (prio 0, RW): msix-pba
>  22       00000000fd000000-00000000fdffffff (prio 1, RW): vga.vram
>  23       00000000fffc0000-00000000ffffffff (prio 0, R-): pc.bios
>  24     00000000000a0000-00000000000bffff (prio 1, RW): alias smram-region 
> @pci 00000000000a0000-00000000000bffff
> 
> which is what you are saying but I see more stuff missing - like vga.vram or 
> vga.mmio.. Is there a setting I am missing when I run QEMU?
> 
>> If your host machine where it is actually inserted has more than 2GB RAM,
>> look at the RAM and PCI memory mappings logged in /var/log/dmesg and in
>> the /proc/ and /sys/ file systems.
>> 
> It does have 20GB but I can’t figure out what’s going on.. 
> 
>> Enjoy
>> 
>> Jakob
>> -- 
>> Jakob Bohm, CIO, Partner, WiseMo A/S.  https://www.wisemo.com 
>> <https://www.wisemo.com/>
>> Transformervej 29, 2860 S?borg, Denmark.  Direct +45 31 13 16 10
>> This public discussion message is non-binding and may contain errors.
>> WiseMo - Remote Service Management for PCs, Phones and Embedded
>