[Qemu-devel] Re: PCIe Transaction handling in Qemu

[Adnan Khaleel]

>

Yamahata and Paul, thank you very much for your responses. I have some followup questions below.

Thanks for all your assistance!

Adnan 

> > I have a question regarding how Qemu PCIe devices handle Config Transactions vs
> > Memory Transactions (assuming the PCI device is setup to act
> > as PCI_BASE_ADDRESS_SPACE_MEMORY).
> > 
> > I'm using portions of hw/cirrus_vga.c to make my point,
>
> If you can send out what you have instead of mimicked
> example, it would help to figure out what you are trying to do.

I was trying to keep things simple with common code that was well established inside Qemu. I've attached the code from my PCIe device to help you better understand what I'm trying to do.

I'm using Qemu to drive a PCIe device model. Here is a diagrammatic representation

______      ___________________         ___________________

Qemu  |<-->|PCIe Model Wrapper |<----->| PCIe Device Model |

______|    |___________________|       |___________________|

The PCIe Device model is a complete GPU model that communicates via a simulated PCIe interface i.e. the model represents a PCIe card that would plug into a PCIe slot and expects to see PCIe transactions to drive it. In other words, the PCIe Device model thinks its connected to a PCI-Bridge so I have to mimic this partially with the PCIe Model wrapper. It is important that you understand this since its very central to the issues I'm trying to better comprehend. I don't have the source code to the device model, hence my many challenges.

I've written the PCIe model wrapper (the attached file), this wrapper acts as a proxy for the Device Model and is what Qemu "sees" as an PCI endpoint. The PCIe Model Wrapper is modelled as a Qemu PCIe device and includes PCI configuration space registers, however the TRUE PCI config registers are contained with the PCIe Device Model. 

So in a nutshell this is what I'm trying to do: 

- When the Guest OS inside Qemu does a PCIe config read/write, the PCIe model wrapper has to intercept that call, construct a valid PCIe config transaction and send it to the PCIe device model. 

- Similarly for a PCIe MMIO transaction i.e, the wrapper has to construct a PCIe transaction and send it to the device model.

I hope this puts into perspective why I was asking the questions about how I can definitively identify if a request is a config request or if its a MMIO request because I need to synthesize the appropriate PCIe transactions to send to the Model.

> > I have some questions about PCIe operations sssuming the device has MMIO
> > handlers involved (as shown above).
> > 1. Will all PCIe config operations ALWAYS use the installed config handlers? Or
> > can PCIe config operations use the MMIO handlers?
>
> MMIO on MMCONFIG area are routed to write/read config handler.
> On the other hand MMIO on memory BAR is routed to mmio hanlder you pictured.
> NOTE: the upstream qemu lacks q35 chipset support, so guest can NOT do
>       MMIO on MMCONFIG area.

I am using the q35 chipset that you (Yamahata) have been working on. So does that mean that guests OS with the qemu with q35 can only access the legacy PCI config space (256 bytes) and not the full PCIe 4k config space? The reason I ask is because of an AMD document I found

http://developer.amd.com/Assets/pci%20-%20pci%20express%20configuration%20space%20access.pdf

Here is the relevant text:

3.3 MMIO access with different Linux Kernel versions Support for MMIO access for PCI configuration space depends on the Linux Kernel version and configuration, and the existence of an MCFG ACPI table. IO access to PCI configuration space is always possible and will be used if MMIO access is not available. To access extended configuration space (between byte 256 and 4095) MMIO access is required.

:

Note: In both cases Linux supports the IO access method as a fallback if MMIO access is not possible (e.g. if MCFG ACPI table is missing). 

A kernel (with version prior to and including 2.6.25) with enabled MMIO access method is able to do MMIO access if and only if the BIOS provides a valid MCFG ACPI table and if the configuration space is "E820-reserved". If these requirements are not met, IO access is used and the kernel logs an error message like this PCI: BIOS Bug: MCFG area at e0000000 is not E820-reserved PCI: Not using MMCONFIG. PCI: Using configuration type 1 Beginning with kernel version 2.6.25 Linux always uses the IO access method to access configuration space below byte 256. Thus MMIO access is used only for extended configuration space.

> > 2. Assuming that both PCI config and MMIO operations can use the MMIO handlers,
> > is there any way I can identify if a transaction is a config or a memory
> > transaction?

Here is a code snippet where I register the config handlers:

static PCIDeviceInfo pcie_msix_info = {

    .qdev.name      = PCIE_MSIX_DEVICE,

    .qdev.desc      = "PCIE MSIX device template",

    .qdev.size      = sizeof(PCIE_MSIX_DEVState),

    .qdev.reset     = pcie_msix_reset,

    .qdev.vmsd      = &vmstate_pcie_msix,

    .is_express     = 1,

    .config_read    = pcie_msix_read_config,

    .config_write   = pcie_msix_write_config,

    .init           = pcie_msix_initfn,

    .exit           = pcie_msix_exitfn,

    .qdev.props     = (Property[]) {

        DEFINE_PROP_UINT32("vectors", PCIE_MSIX_DEVState, vectors, PCIE_MSIX_MSI_NR_VECTOR),        

        DEFINE_PROP_UINT16("aer_log_max", PCIESlot,

                            port.br.dev.aer_log.log_max,

                            PCIE_AER_LOG_MAX_DEFAULT),

        DEFINE_PROP_END_OF_LIST(),

    }

};

Here is a code snippet where I register the mmio handlers:

static CPUWriteMemoryFunc * const pcie_msix_mem_write_fn[] = {

    pcie_msix_mem_writeb, pcie_msix_mem_writew, pcie_msix_mem_writel, pcie_msix_mem_writed

};

static CPUReadMemoryFunc * const pcie_msix_mem_read_fn[] = {

    pcie_msix_mem_readb, pcie_msix_mem_readw, pcie_msix_mem_readl, pcie_msix_mem_readd

};

static void 

pcie_msix_mem_map(PCIDevice *dev, int region_num, 

                  pcibus_t addr, pcibus_t size, int type)

{

    uint32_t msix_mem_bar = 0;

    PCIE_MSIX_DEVState *d = DO_UPCAST(PCIE_MSIX_DEVState, dev, dev);

    cpu_register_physical_memory(addr, BAR_Regions[msix_mem_bar][0], d->mmio_index);

}

Which I register via the following call in the pcie_msix_initfn(PCIDevice *pci_dev) function

pci_register_bar(&d->dev, msix_mem_bar, BAR_Regions[msix_mem_bar][0], BAR_Regions[msix_mem_bar][1], pcie_msix_mem_map);

So are my following assumption true then for my PCIe device i.e the wrapper?

All config requests WILL be handled by the config handlers and,

All MMIO requests will be handled by the mmio handlers (given that the q35 model does not support MMIO on MMCONFIG)

>

  

> > 3.a. What address is passed on the MMIO handlers for config and MMIO
> > operations? From pci_data_write in pci_host.c, it appears that config
> > operations send only the offset into the config region. I couldn't determine
> > what address is passed for MMIO operations.
> >    b. Is it an offset from the BAR for MMIO operations?
> >    c. How do I get the full physical address?
> >    d. What address does a PCIe device expect to see - physical or offset for?
> >    e. Is there anyway I can find out what the bus and device numbers are once
> > inside the config and MMIO handlers? i.e once the execution has reached
> > the pci_cirrus_write_config() or cirrus_vga_mem_readb(..) from the code above?
>
> offset in configuration space of each pcie function is passed to
> write/read config handler
> physical address is passed to mmio handler of memory BAR.

When I examine the address on calls to the PCIe Device Wrapper config handlers, they start at 0x0 so they clearly are offsets. However, I also see addresses from 0x0 in the MMIO handlers. Paul's response indicated that they were offsets so I'm inclined to believe that the address is an offset from the start of the BAR. So let me ask my question again - when I'm synthesizing the PCIe transaction to send to the PCIe device model, do I have to give it a full address i.e. bus,dev,fn for config transactions and full physical address for MMIO transactions?

>

> -- 
> yamahata

pcie_msix_template_1.c
Description: Text document