Re: [Qemu-devel] [PATCH v3] pci: clean all funcs when hot-removing multifunc device

[Amos Kong]

On Wed, May 16, 2012 at 11:26 PM, Bjorn Helgaas <address@hidden> wrote:
> On Fri, May 11, 2012 at 8:00 AM, Jiang Liu <address@hidden> wrote:
>> On 05/11/2012 08:24 AM, Amos Kong wrote:
>>> On 05/11/2012 07:54 AM, Amos Kong wrote:
>>>> On 05/11/2012 02:55 AM, Michael S. Tsirkin wrote:
>>>>> On Fri, May 11, 2012 at 01:09:13AM +0800, Jiang Liu wrote:
>>>>>> On 05/10/2012 11:44 PM, Amos Kong wrote:
>>>>>>
>>>>>>> diff --git a/drivers/pci/hotplug/acpiphp_glue.c 
>>>>>>> b/drivers/pci/hotplug/acpiphp_glue.c
>>>>>>> index 806c44f..a7442d9 100644
>>>>>>> --- a/drivers/pci/hotplug/acpiphp_glue.c
>>>>>>> +++ b/drivers/pci/hotplug/acpiphp_glue.c
>>>>>>> @@ -885,7 +885,7 @@ static void disable_bridges(struct pci_bus *bus)
>>>>>>>  static int disable_device(struct acpiphp_slot *slot)
>>>>>>>  {
>>>>>>>   struct acpiphp_func *func;
>>>>>>> - struct pci_dev *pdev;
>>>>>>> + struct pci_dev *pdev, *tmp;
>>>>>>>   struct pci_bus *bus = slot->bridge->pci_bus;
>>>>>>>
>>>>>>>   /* The slot will be enabled when func 0 is added, so check
>>>>>>> @@ -902,9 +902,10 @@ static int disable_device(struct acpiphp_slot 
>>>>>>> *slot)
>>>>>>>                   func->bridge = NULL;
>>>>>>>           }
>>>>>>>
>>>>>>> -         pdev = pci_get_slot(slot->bridge->pci_bus,
>>>>>>> -                             PCI_DEVFN(slot->device, func->function));
>>>>>>> -         if (pdev) {
>>>>>>> +         list_for_each_entry_safe(pdev, tmp, &bus->devices, bus_list) {
>>>>>>> +                 if (PCI_SLOT(pdev->devfn) != slot->device)
>>>>>>> +                         continue;
>
> I think the concept is good: in enable_device(), we use
> pci_scan_slot(), which scans all possible functions in the slot.  So
> in disable_device() we should do something symmetric to remove all the
> functions.

Right!

>>>>>>> +
>>>>>> The pci_bus_sem lock should be acquired when walking the bus->devices 
>>>>>> list.
>>>>>> Otherwise it may cause invalid memory access if another thread is 
>>>>>> modifying
>>>>>> the bus->devices list concurrently.
>>>
>>> pci_bus_sem lock is only request for writing &bus->devices list, right ?
>>> and this protection already exists in pci_destory_dev().
>> That's for writer. For reader to walk the pci_bus->devices list, you also 
>> need
>> to acquire the reader lock by down_read(&pci_bus_sem). Please refer to
>> pci_get_slot() for example. This especially import for native OS because 
>> there
>> may be multiple PCI slots/devices on the bus.
>
> There is a lot of existing code that walks bus->devices without
> holding pci_bus_sem, but most of it is boot-time code that is arguably
> safe (though I think things like pcibios_fixup_bus() are poorly
> designed and don't fit well in the hotplug-enabled world).

disable_remove() is not boot-time code, we might hot-remove devices
when system is running.

> In this case, I do think we need to protect against updates while
> we're walking bus->devices.  It's probably not trivial because
> __pci_remove_bus_device() calls pci_destroy_dev(), where we do the
> down_write(), so simply wrapping the whole thing with down_read() will
> cause a deadlock.

I posted a V4 to add pci_bus_sem protection , please help to review.
Thanks for Jiang Liu's guide.

> Kenji-san, Yinghai, do you have any input?
>
> Bjorn