Re: [PATCH] spapr/xive: skip partially initialized vCPUs in presenter

[Greg Kurz]

On Wed, 2 Oct 2019 11:02:45 +1000
David Gibson <address@hidden> wrote:

> On Tue, Oct 01, 2019 at 06:56:28PM +0200, Greg Kurz wrote:
> > On Tue, 1 Oct 2019 13:56:10 +0200
> > Cédric Le Goater <address@hidden> wrote:
> > 
> > > On 01/10/2019 13:06, Greg Kurz wrote:
> > > > On Tue,  1 Oct 2019 10:57:22 +0200
> > > > Cédric Le Goater <address@hidden> wrote:
> > > > 
> > > >> When vCPUs are hotplugged, they are added to the QEMU CPU list before
> > > >> being fully realized. This can crash the XIVE presenter because the
> > > >> 'tctx' pointer is not necessarily initialized when looking for a
> > > >> matching target.
> > > >>
> > > > 
> > > > Ouch... :-\
> > > > 
> > > >> These vCPUs are not valid targets for the presenter. Skip them.
> > > >>
> > > > 
> > > > This likely fixes this specific issue, but more generally, this
> > > > seems to indicate that using CPU_FOREACH() is rather fragile.
> > > > 
> > > > What about tracking XIVE TM contexts with a QLIST in xive.c ?
> > > 
> > > This is a good idea.  
> > > 
> > > On HW, the thread interrupt contexts belong to the XIVE presenter 
> > > subengine. This is the logic doing the CAM line matching to find
> > > a target for an event notification. So we should model the context 
> > > list below the XiveRouter in QEMU which models both router and 
> > > presenter subengines. We have done without a presenter model for 
> > > the moment and I don't think we will need to introduce one.  
> > > 
> > > This would be a nice improvements of my patchset adding support
> > > for xive escalations and better support of multi chip systems. 
> > > I have introduced a PNV_CHIP_CPU_FOREACH in this patchset which 
> > > would become useless with a list of tctx under the XIVE interrupt
> > > controller, XiveRouter, SpaprXive, PnvXive.
> > > 
> > 
> > I agree. It makes more sense to have the list below the XiveRouter,
> > rather than relying on CPU_FOREACH(), which looks a bit weird from
> > a device emulation code perspective.
> 
> That does sound like a better idea long term.  However, for now, I
> think the NULL check is a reasonable way of fixing the real error
> we're hitting, so I've applied the patch here.
> 

Fair enough.

Reviewed-by: Greg Kurz <address@hidden>

> Future cleanups to a different approach remain welcome, of course.
> 

I've started to look. This should simplify Cedric's "add XIVE support
for KVM guests" series, but I'll wait for your massive cleanup series
to get merged first.

> > > Next step would be to get rid of the tctx->cs pointer. In my latest
> > > patches, it is only used to calculate the HW CAM line. 
> > > 
> > > There might be some consequences on the object hierarchy and it will
> > > break migration.
> > > 
> > 
> > This could break if the contexts were devices sitting in a bus, which
> > isn't the case here. I'll try to come up with a proposal for spapr,
> > and we can work out the changes on your recent XIVE series for pnv.
> > 
> > > Thanks,
> > > 
> > > C.
> > > 
> > > > 
> > > > ================================================================================
> > > > diff --git a/include/hw/ppc/xive.h b/include/hw/ppc/xive.h
> > > > index 6d38755f8459..89b9ef7f20b1 100644
> > > > --- a/include/hw/ppc/xive.h
> > > > +++ b/include/hw/ppc/xive.h
> > > > @@ -319,6 +319,8 @@ typedef struct XiveTCTX {
> > > >      qemu_irq    os_output;
> > > >  
> > > >      uint8_t     regs[XIVE_TM_RING_COUNT * XIVE_TM_RING_SIZE];
> > > > +
> > > > +    QTAILQ_ENTRY(XiveTCTX) next;
> > > >  } XiveTCTX;
> > > >  
> > > >  /*
> > > > diff --git a/hw/intc/xive.c b/hw/intc/xive.c
> > > > index b7417210d817..f7721c711041 100644
> > > > --- a/hw/intc/xive.c
> > > > +++ b/hw/intc/xive.c
> > > > @@ -568,6 +568,8 @@ static void xive_tctx_reset(void *dev)
> > > >          ipb_to_pipr(tctx->regs[TM_QW3_HV_PHYS + TM_IPB]);
> > > >  }
> > > >  
> > > > +static QTAILQ_HEAD(, XiveTCTX) xive_tctx_list = 
> > > > QTAILQ_HEAD_INITIALIZER(xive_tctx_list);
> > > > +
> > > >  static void xive_tctx_realize(DeviceState *dev, Error **errp)
> > > >  {
> > > >      XiveTCTX *tctx = XIVE_TCTX(dev);
> > > > @@ -609,10 +611,14 @@ static void xive_tctx_realize(DeviceState *dev, 
> > > > Error **errp)
> > > >      }
> > > >  
> > > >      qemu_register_reset(xive_tctx_reset, dev);
> > > > +    QTAILQ_INSERT_HEAD(&xive_tctx_list, tctx, next);
> > > >  }
> > > >  
> > > >  static void xive_tctx_unrealize(DeviceState *dev, Error **errp)
> > > >  {
> > > > +    XiveTCTX *tctx = XIVE_TCTX(dev);
> > > > +
> > > > +    QTAILQ_REMOVE(&xive_tctx_list, tctx, next);
> > > >      qemu_unregister_reset(xive_tctx_reset, dev);
> > > >  }
> > > >  
> > > > @@ -1385,15 +1391,14 @@ static bool xive_presenter_match(XiveRouter 
> > > > *xrtr, uint8_t format,
> > > >                                   bool cam_ignore, uint8_t priority,
> > > >                                   uint32_t logic_serv, XiveTCTXMatch 
> > > > *match)
> > > >  {
> > > > -    CPUState *cs;
> > > > +    XiveTCTX *tctx;
> > > >  
> > > >      /*
> > > >       * TODO (PowerNV): handle chip_id overwrite of block field for
> > > >       * hardwired CAM compares
> > > >       */
> > > >  
> > > > -    CPU_FOREACH(cs) {
> > > > -        XiveTCTX *tctx = xive_router_get_tctx(xrtr, cs);
> > > > +    QTAILQ_FOREACH(tctx, &xive_tctx_list, next) {
> > > >          int ring;
> > > >  
> > > >          /*
> > > > ================================================================================
> > > > 
> > > >> Signed-off-by: Cédric Le Goater <address@hidden>
> > > >> ---
> > > >>  hw/intc/xive.c | 8 ++++++++
> > > >>  1 file changed, 8 insertions(+)
> > > >>
> > > >> diff --git a/hw/intc/xive.c b/hw/intc/xive.c
> > > >> index b7417210d817..29df06df1136 100644
> > > >> --- a/hw/intc/xive.c
> > > >> +++ b/hw/intc/xive.c
> > > >> @@ -1396,6 +1396,14 @@ static bool xive_presenter_match(XiveRouter 
> > > >> *xrtr, uint8_t format,
> > > >>          XiveTCTX *tctx = xive_router_get_tctx(xrtr, cs);
> > > >>          int ring;
> > > >>  
> > > >> +        /*
> > > >> +         * Skip partially initialized vCPUs. This can happen when
> > > >> +         * vCPUs are hotplugged.
> > > >> +         */
> > > >> +        if (!tctx) {
> > > >> +            continue;
> > > >> +        }
> > > >> +
> > > >>          /*
> > > >>           * HW checks that the CPU is enabled in the Physical Thread
> > > >>           * Enable Register (PTER).
> > > > 
> > > 
> > 
>

pgpDmittqzNxU.pgp
Description: OpenPGP digital signature