[Qemu-devel] Re: Unusual physical address when using 64-bit BAR

[Cam Macdonell]

On Sun, Jun 27, 2010 at 2:39 AM, Avi Kivity <address@hidden> wrote:
> On 06/25/2010 12:51 AM, Cam Macdonell wrote:
>>
>> On Tue, Jun 15, 2010 at 5:04 AM, Avi Kivity<address@hidden>  wrote:
>>
>>>
>>> On 06/11/2010 08:31 PM, Cam Macdonell wrote:
>>>
>>>>
>>>> On Mon, Apr 19, 2010 at 10:41 AM, Cam Macdonell<address@hidden>
>>>>  wrote:
>>>>
>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>> I'm trying to use a 64-bit BAR for my shared memory device.  In simply
>>>>> changing the memory type in pci_register_bar() to
>>>>> PCI_BASE_ADDRESS_MEM_TYPE_64 I get an unusual physical address for
>>>>> that BAR (and my driver crashes in pci_ioremap).
>>>>>
>>>>> from lspci:
>>>>>
>>>>> 00:04.0 RAM memory: Qumranet, Inc. Device 1110
>>>>>        Subsystem: Qumranet, Inc. Device 1100
>>>>>        Flags: fast devsel, IRQ 10
>>>>>        Memory at f1020000 (32-bit, non-prefetchable) [size=1K]
>>>>>        Memory at f1021000 (32-bit, non-prefetchable) [size=4K]
>>>>>        Memory at c20000000000 (64-bit, non-prefetchable) [size=1024M]
>>>>>        Capabilities:<access denied>
>>>>> 00: f4 1a 10 11 03 00 10 00 00 00 00 05 00 00 00 00
>>>>> 10: 00 00 02 f1 00 10 02 f1 04 00 00 00 00 c2 00 00
>>>>> 20: 00 00 00 00 00 00 00 00 00 00 00 00 f4 1a 00 11
>>>>> 30: 00 00 00 00 40 00 00 00 00 00 00 00 0b 01 00 00
>>>>>
>>>>> with DEBUG_MEMREG, I see
>>>>>
>>>>> kvm_unregister_memory_area:666 Unregistering memory region
>>>>> c20000000000 (40000000)
>>>>> kvm_destroy_phys_mem:649 slot 7 start c20000000000 len 0 flags 0
>>>>> IVSHMEM: addr = 3221225472 size = 1073741824
>>>>> kvm_register_phys_mem:605 memory: gpa: c200c0000000, size: 40000000,
>>>>> uaddr: 7f6dd7ffe000, slot: 7, flags: 0
>>>>> kvm_unregister_memory_area:666 Unregistering memory region
>>>>> c200c0000000 (40000000)
>>>>> kvm_destroy_phys_mem:649 slot 7 start c200c0000000 len 0 flags 0
>>>>> IVSHMEM: addr = 0 size = 1073741824
>>>>> kvm_register_phys_mem:605 memory: gpa: c20000000000, size: 40000000,
>>>>> uaddr: 7f6dd7ffe000, slot: 7, flags: 0
>>>>> kvm_unregister_memory_area:666 Unregistering memory region
>>>>> c20000000000 (40000000)
>>>>> kvm_destroy_phys_mem:649 slot 7 start c20000000000 len 0 flags 0
>>>>> IVSHMEM: addr = 0 size = 1073741824
>>>>> kvm_register_phys_mem:605 memory: gpa: ffffffff00000000, size:
>>>>> 40000000, uaddr: 7f6dd7ffe000, slot: 7, flags: 0
>>>>> kvm_unregister_memory_area:666 Unregistering memory region
>>>>> ffffffff00000000 (40000000)
>>>>> kvm_destroy_phys_mem:649 slot 7 start ffffffff00000000 len 0 flags 0
>>>>> IVSHMEM: addr = 0 size = 1073741824
>>>>> kvm_register_phys_mem:605 memory: gpa: c20000000000, size: 40000000,
>>>>> uaddr: 7f6dd7ffe000, slot: 7, flags: 0
>>>>>
>>>>> (the IVSHMEM lines are my debug statements)
>>>>>
>>>>> address sizes   : 40 bits physical, 48 bits virtual  (guest)
>>>>> address sizes   : 38 bits physical, 48 bits virtual  (host)
>>>>>
>>>>>
>>>>>
>>>>
>>>> Hi, I happened to run into this problem again when trying to use a
>>>> 64-bit BAR.  I did a bit more digging and the test that is failing is
>>>> called from arch/x86/mm/ioremap.c in the guest and here it is.
>>>>
>>>> static inline int phys_addr_valid(resource_size_t addr)
>>>> {
>>>> #ifdef CONFIG_PHYS_ADDR_T_64BIT
>>>>        return !(addr>>    boot_cpu_data.x86_phys_bits);
>>>> #else
>>>>        return 1;
>>>> #endif
>>>> }
>>>>
>>>> the value of addr (in this case the 48-bit virtual address
>>>> c20000000000) is shifted to the right shift by
>>>> boot_cpu_data.x86_phys_bits (which is 40 bits, the physical address
>>>> size), so a non-zero value is returned which causes the test to fail
>>>> and generates the "invalid physical address" error in the guest.
>>>>
>>>> Any help is appreciated as to whether this is a Qemu or guest kernel
>>>> issue.
>>>>
>>>>
>>>
>>> The guest kernel should never have generated an address that is bigger
>>> than
>>> cpu_phys_bits in the first place.  What's the value for cpu_phys_bits in
>>> the
>>> guest? (/proc/cpuinfo, 'address sizes :' line).
>>>
>>
>> Sorry I missed your reply until now.  The guest address sizes are as
>> follows:
>>
>> address sizes   : 40 bits physical, 48 bits virtual
>>
>
> So the address c20000000000 is illegal.
>
>>> Is this really the address the guest programmed, or is qemu
>>> misinterpreting
>>> it?
>>>
>
> Well, what's the answer?

You're going to have to give me a hint on how to determine that.

lspci in the guest shows the following

Memory at c20000000000 (64-bit, non-prefetchable) [size=1024M]

does that demonstrate a guest generated address?

Cam

>
> --
> error compiling committee.c: too many arguments to function
>
>