Re: [Qemu-devel] [PATCH] pc: Clean up PIC-to-APIC IRQ path

[Blue Swirl]

On Wed, Aug 31, 2011 at 7:44 PM, Blue Swirl <address@hidden> wrote:
> On Wed, Aug 31, 2011 at 6:17 PM, Jan Kiszka <address@hidden> wrote:
>> On 2011-08-31 19:41, Blue Swirl wrote:
>>> On Wed, Aug 31, 2011 at 10:53 AM, Jan Kiszka <address@hidden> wrote:
>>>> On 2011-08-31 10:25, Peter Maydell wrote:
>>>>> On 30 August 2011 20:28, Jan Kiszka <address@hidden> wrote:
>>>>>> Yes, that's the current state. Once we have bidirectional IRQ links in
>>>>>> place (pushing downward, querying upward - required to skip IRQ routers
>>>>>> for fast, lockless deliveries), that should change again.
>>>>>
>>>>> Can you elaborate a bit more on this? I don't think anybody has
>>>>> proposed links with their own internal state before in the qdev/qom
>>>>> discussions...
>>>>
>>>> That basic idea is to allow
>>>>
>>>> a) a discovery of the currently active IRQ path from source to sink
>>>>   (that would be possible via QOM just using forward links)
>>>
>>> Why, only for b)? This is not possible with real hardware.
>>>
>>>> b) skip updating the states of IRQ routers in the common case, just
>>>>   signaling directly the sink from the source (to allow in-kernel IRQ
>>>>   delivery or to skip taking some device locks). Whenever some router
>>>>   is queried for its current IRQ line state, it would have to ask the
>>>>   preceding IRQ source for its state. So we need a backward link.
>>>
>>> I think this would need pretty heavy changes everywhere. At board
>>> level the full path needs to be identified and special versions of
>>> IRQs installed along the way. The routers would need to use callbacks
>>> to inform other parties about routing changes.
>>
>> It already works in practice (based on a hack and minus IRQ router state
>> updates) for x86 PCI device pass-through. At least I don't want this
>> upstream but instead a generic solution. The ability to skip IRQ routers
>> also in pure user space device model scenarios is a useful by-product.
>>
>>>
>>>> We haven't thought about how this could be implemented in details yet
>>>> though. Among other things, it heavily depends on the final QOM design.
>>>
>>> Perhaps a global IRQ manager could help. It would keep track of the
>>> whole IRQ matrix, what are input (x axis) and output (y axis) states
>>> and what each matrix node (router state) looks like (or able to
>>> compute) if asked. I don't think backward links would be needed with
>>> this approach.
>>
>> Well, the backward links would then be moved to that global IRQ manager.
>> It's just moving the data management, but if it turns out to allow a
>> cleaner device design, I would surely not vote against it. But that
>> manager must support lazy updates as well because we cannot call it from
>> kernel space for each and every event.
>
> The global IRQ switch matrix would take over all routing for the
> devices in question. As an example, let's consider a PCI card
> (source), PCI host bridge, IO-APIC, LAPIC and CPU (final destination).
> The matrix would get input from the PCI card and output directly to
> LAPIC since the CPU does not use a qemu_irq yet. These leaf devices
> actually need no changes, just the qemu_irq is routed elsewhere. The
> matrix would also have access to intermediate devices but the qemu_irq
> inputs of those would not be updated in the lazy mode and the outputs
> need not go anywhere.
>
> So for a matrix where x0, x1...  is used for inputs and y0, y1... for
> outputs, x0 would be connected in the example to PCI card, x1 to PCI
> host bridge output and x2 to IO-APIC (ignoring multiple lines). Then
> y0 would be the calculated output from PCI host bridge, y1 IO-APIC and
> only y2 would be connected to LAPIC.
>
> If the intermediate IRQ state between the nodes is queried (for
> example at IO-APIC), the matrix must be able to tell what would be the
> true state at that point and then this value would be supplied to the
> device. So a method to query device IRQ input state from matrix on
> demand is needed. This could be as unspecific as "refresh all my
> inputs". A backward link may not be able to tell the state if
> information about next backward link is lost (logic OR of several
> signals) whereas the matrix should be able to do that even in that
> case.
>
> If routes in IO-APIC change, the matrix needs to be recalculated and
> states in the affected devices should be updated. This update cycle
> could be triggered via normal qemu_irq device input lines or direct
> callback (needs more changes). Maybe even this could be handled lazily
> so that no updates would be needed until the final output changes or
> the state of an intermediate device is queried.
>
> The global manager should handle saving and restoring states, the
> device states would not matter. If it pushed the true state to
> devices, we're back to the ordering problem.

I was thinking that the manager would be at board level, but actually
the wiring at the board level can be considered as a second switch
matrix (mostly constant). Then with these two matrices, all signal
routing for the whole machine could be managed. The board level would
use same facilities to register the routing elements as devices.

I think the matrix model is also compatible with Pins. Since this is
pretty much at hand waving level ATM, maybe it's better to wait for
those though my hands are itching. Maybe I'll make a small proof of
concept matrix to see if this can fly.