Re: [Qemu-devel] [RFC] virtio-pci: Allow PCIe virtio devices on root bus

[Marcel Apfelbaum]

On 02/16/2017 05:28 AM, David Gibson wrote:
> On Thu, Feb 16, 2017 at 01:48:42PM +1100, David Gibson wrote:
> > On Wed, Feb 15, 2017 at 04:59:33PM +0200, Marcel Apfelbaum wrote:
> > > On 02/15/2017 03:45 AM, David Gibson wrote:
> > > > On Tue, Feb 14, 2017 at 02:53:08PM +0200, Marcel Apfelbaum wrote:
> > > > > On 02/14/2017 06:15 AM, David Gibson wrote:
> > > > > > On Mon, Feb 13, 2017 at 12:14:23PM +0200, Marcel Apfelbaum wrote:
> > > > > > > On 02/13/2017 06:33 AM, David Gibson wrote:
> > > > > > > > On Sun, Feb 12, 2017 at 09:05:46PM +0200, Marcel Apfelbaum wrote:
> > > > > > > > > On 02/10/2017 02:37 AM, David Gibson wrote:
> > > > > > > > > > On Thu, Feb 09, 2017 at 10:04:47AM +0100, Laszlo Ersek wrote:
> > > > > > > > > > > On 02/09/17 05:16, David Gibson wrote:
> > > > > > > > > > > > On Wed, Feb 08, 2017 at 11:40:50AM +0100, Laszlo Ersek wrote:
> > > > > > > > > > > > > On 02/08/17 07:16, David Gibson wrote:
> [snip]
> > > > >   Which means that you can use it to
> > > > > > drive PCIe devices just fine.  "Bus level" PCIe extensions like AER
> > > > > > and PCIe standard hotplug won't work, but PAPR has its own mechanisms
> > > > > > for those (common between PCI and PCIe).
> > > > > >
> > > > > > I did float the idea of having the pseries PCI bus remain plain PCI
> > > > > > but with a special flag to allow PCIe devices to be attached to it
> > > > > > anyway.  It wasn't greeted with much enthusiasm..
> > > > >
> > > > > Can you point me to the discussion please? It seems similar to what I proposed 
> > > > > above.
> > > >
> > > > Sorry, I was misleading.  I think I just raised that idea with Andrea
> > > > and a few other people internally, not on one of the lists at large.
> > > >
> > > > > As you properly described it, is much closer to PCI then PCIe, even the only 
> > > > > characteristic
> > > > > that makes it "a little" PCIe, the Extended Configuration Space support,
> > > > > is done with an alternative interface.
> > > > >
> > > > > I agree the PAPR bus is not PCIe.
> > > >
> > > > Ok, so if we take that direction, the question becomes how do we let
> > > > PCIe devices plug into this mostly-not-PCIe bus.  Maybe introduce a
> > > > "pci_bus_accepts_express()" function that will replace many, but not
> > > > all current uses of "pci_bus_is_express()"?
> > >
> > > Sounds good and I think Eduardo is already working on exactly this
> > > idea, however he is on PTO now. It is better to synchronize with him.
> >
> > Ah, right.  Do you know when he'll be back?  This is semi-urgent for
> > Power.
> >
> >
> > > > Such a helper could maybe simplify the logic in virtio-pci (and XHCI?)
> > > > by returning false on an x86 root bus.
> > >
> > > The rule would me more complicated. We don't want to completely remove the
> > > possibility to have PCIe devices as part of Root Complex. it seems
> > > like I am contradicting myself, but no).
> > > This is why we have guidelines and  not hard-coded policies.
> > > Also ,the QEMU way is to be more permissive. We provide guidelines and sane
> > > defaults, but we let the user to chose.
> > >
> > > Getting back to our problem, the rule would be:
> > > hybrid devices should be PCI or PCIe for a bus?
> > > PAPR bus should return 'PCIe' for hybrid devices.
> > > X86 bus should return 'PCIe' if not root.
> >
> > Ok.
>
> Wait, actually.. we have two possible directions to go, both of which
> have been mentioned in the thread, but I don't think we've settled on
> one:
>
> 1) Have pseries create a PCIe bus (as my first cut draft does).
>
> That should allow pure PCIe devices to appear either under a port or
> (more usually for PAPR) as "integrated endpoints".  In addition we'd
> need as suggested above a "pcie_hybrid_type()" function that would
> tell hybrid devices to also appear as PCIe rather than PCI.
>
> 2) Have pseries create a vanilla PCI bus (or a special PAPR PCI
>    variant)
>
> Appearing as vanilla PCI would in a number of ways more closely match
> the way PCI buses are handled on PAPR.  However, we still need to
> connect PCIe devices to it.  So we'd need some 'bus_accepts_pcie()'
> hook and use that (in place of pci_bus_is_express()) to determine both
> whether we can attach pure PCIe devices and that hybrid devices should
> appear as PCIe rather than plain PCI.
>
>
> Based on the immediately preceding discussion, I was leaning towards
> (2).  Is that your feeling as well?

On 02/16/2017 05:28 AM, David Gibson wrote:
> On Thu, Feb 16, 2017 at 01:48:42PM +1100, David Gibson wrote:
>> On Wed, Feb 15, 2017 at 04:59:33PM +0200, Marcel Apfelbaum wrote:
>>> On 02/15/2017 03:45 AM, David Gibson wrote:
>>>> On Tue, Feb 14, 2017 at 02:53:08PM +0200, Marcel Apfelbaum wrote:
>>>>> On 02/14/2017 06:15 AM, David Gibson wrote:
>>>>>> On Mon, Feb 13, 2017 at 12:14:23PM +0200, Marcel Apfelbaum wrote:
>>>>>>> On 02/13/2017 06:33 AM, David Gibson wrote:
>>>>>>>> On Sun, Feb 12, 2017 at 09:05:46PM +0200, Marcel Apfelbaum wrote:
>>>>>>>>> On 02/10/2017 02:37 AM, David Gibson wrote:
>>>>>>>>>> On Thu, Feb 09, 2017 at 10:04:47AM +0100, Laszlo Ersek wrote:
>>>>>>>>>>> On 02/09/17 05:16, David Gibson wrote:
>>>>>>>>>>>> On Wed, Feb 08, 2017 at 11:40:50AM +0100, Laszlo Ersek wrote:
>>>>>>>>>>>>> On 02/08/17 07:16, David Gibson wrote:
> [snip]
>>>>>   Which means that you can use it to
>>>>>> drive PCIe devices just fine.  "Bus level" PCIe extensions like AER
>>>>>> and PCIe standard hotplug won't work, but PAPR has its own mechanisms
>>>>>> for those (common between PCI and PCIe).
>>>>>>
>>>>>> I did float the idea of having the pseries PCI bus remain plain PCI
>>>>>> but with a special flag to allow PCIe devices to be attached to it
>>>>>> anyway.  It wasn't greeted with much enthusiasm..
>>>>>>
>>>>>
>>>>> Can you point me to the discussion please? It seems similar to what I 
proposed above.
>>>>
>>>> Sorry, I was misleading.  I think I just raised that idea with Andrea
>>>> and a few other people internally, not on one of the lists at large.
>>>>
>>>>> As you properly described it, is much closer to PCI then PCIe, even the 
only characteristic
>>>>> that makes it "a little" PCIe, the Extended Configuration Space support,
>>>>> is done with an alternative interface.
>>>>>
>>>>> I agree the PAPR bus is not PCIe.
>>>>
>>>> Ok, so if we take that direction, the question becomes how do we let
>>>> PCIe devices plug into this mostly-not-PCIe bus.  Maybe introduce a
>>>> "pci_bus_accepts_express()" function that will replace many, but not
>>>> all current uses of "pci_bus_is_express()"?
>>>>
>>>
>>> Sounds good and I think Eduardo is already working on exactly this
>>> idea, however he is on PTO now. It is better to synchronize with him.
>>
>> Ah, right.  Do you know when he'll be back?  This is semi-urgent for
>> Power.
>>
>>
>>>> Such a helper could maybe simplify the logic in virtio-pci (and XHCI?)
>>>> by returning false on an x86 root bus.
>>>>
>>>
>>> The rule would me more complicated. We don't want to completely remove the
>>> possibility to have PCIe devices as part of Root Complex. it seems
>>> like I am contradicting myself, but no).
>>> This is why we have guidelines and  not hard-coded policies.
>>> Also ,the QEMU way is to be more permissive. We provide guidelines and sane
>>> defaults, but we let the user to chose.
>>>
>>> Getting back to our problem, the rule would be:
>>> hybrid devices should be PCI or PCIe for a bus?
>>> PAPR bus should return 'PCIe' for hybrid devices.
>>> X86 bus should return 'PCIe' if not root.
>>
>> Ok.
>
> Wait, actually.. we have two possible directions to go, both of which
> have been mentioned in the thread, but I don't think we've settled on
> one:
>
> 1) Have pseries create a PCIe bus (as my first cut draft does).
>
> That should allow pure PCIe devices to appear either under a port or
> (more usually for PAPR) as "integrated endpoints".  In addition we'd
> need as suggested above a "pcie_hybrid_type()" function that would
> tell hybrid devices to also appear as PCIe rather than PCI.
>
> 2) Have pseries create a vanilla PCI bus (or a special PAPR PCI
>    variant)
>
> Appearing as vanilla PCI would in a number of ways more closely match
> the way PCI buses are handled on PAPR.  However, we still need to
> connect PCIe devices to it.  So we'd need some 'bus_accepts_pcie()'
> hook and use that (in place of pci_bus_is_express()) to determine both
> whether we can attach pure PCIe devices and that hybrid devices should
> appear as PCIe rather than plain PCI.
>
>
> Based on the immediately preceding discussion, I was leaning towards
> (2).  Is that your feeling as well?
>

I also like option (2).

Thanks,
Marcel