Re: [Qemu-devel] [PATCH for-2.10 0/5] x86/vIOMMU: move init back to machine init phase

[Peter Xu]

On Tue, Apr 11, 2017 at 11:22:38AM +0200, Markus Armbruster wrote:
> Peter Xu <address@hidden> writes:
> 
> > On Fri, Mar 31, 2017 at 07:17:34PM +0300, Michael S. Tsirkin wrote:
> >> On Fri, Mar 31, 2017 at 03:36:28PM +0800, Peter Xu wrote:
> >> > At the very beginning, the x86 vIOMMUs are created via "-M iommu=on".
> >> > We moved one step further a year ago to have the vIOMMUs just like a
> >> > general device, so that we can init them with far more specific
> >> > parameters with "-device" interface.
> >> > 
> >> > However, gradually we found that problem starts to occur due to this.
> >> > The main issue is that the DMA address space of any PCI device is
> >> > postponed to be init after device realization, while some devices'
> >> > realizations would depend on this address space. That looks like a
> >> > dead lock. We have patches and solutions for different single problem,
> >> > but, maybe it's time we can consider to solve the root cause this
> >> > time, of course after 2.9 release.
> >> > 
> >> > This series tries to solve the root cause, and move vIOMMU inits back
> >> > to machine init for x86 platforms. Then, we'll have solid DMA address
> >> > space for each device even during realization.
> >> > 
> >> > Please kindly review. Thanks.
> >> 
> >> I think it's a clean way to do it at a high level.
> >> However I would like to set a tag in the class
> >> rather than listing specific devices.
> >> Also, init order should be consistent for all machines
> >> not just q35.
> >
> > I agree that we may finally need a tag if we want to have a general
> > solution for device init ordering. However I was thinking maybe for
> > x86 IOMMUs we should even move the init earlier than an ordered device
> > list, considering the integrated devices are created during machine
> > init, and it's before the general device init loop.
> >
> > Actually, iiuc that's also following how the real hardware works -
> > since the IOMMU unit (now we only have the root vIOMMU) belongs to
> > north bridge, so imho in emulation codes we'd better follow how the
> > hardware works if possible (I believe in hardware IOMMUs should be
> > inited along with north bridge).
> >
> > (btw, maybe I should mark at least the first patch as RFC...)
> 
> No objection to fixing yet another initialization order problem by
> making it yet another special case, but I think we should really, really
> sit down and try to come up with a *generic* way to express
> initialization order constraints.  This is a modelling problem: we need
> to model how physical hardware resets.  As long as we don't, we're
> damned to tinker with this rickety tower of special cases.

Hi, Markus,

Thanks for the comment. I totally agree with you that we need a
generic way to express the init ordering constraints. However, IMO
IOMMU use case is slightly special here...

Take q35 as example. Q35 machine itself is an object for now. Maybe we
can express an ordering constraint in the future that we want to
create one device A "after" the q35 machine, or create device B
"before" device C, but it'll be hard to express that we want to create
one device "during q35 init, but before creation of integrated devices
(including default VGA, network devices)". Currently on x86 we have
only one IOMMU, and that IOMMU is just exactly part of Q35
northbridge. Imho that's the reason why IOMMU is special comparing to
other device init ordering constraints (although it's using "-device"
keyword).

Thanks,

-- 
Peter Xu