Re: [PATCH for-6.0 4/9] spapr: Set compat mode in spapr_reset_vcpu()

[Igor Mammedov]

On Thu, 26 Nov 2020 10:10:27 +0100
Greg Kurz <groug@kaod.org> wrote:

> On Thu, 26 Nov 2020 15:57:37 +1100
> David Gibson <david@gibson.dropbear.id.au> wrote:
> 
> > On Wed, Nov 25, 2020 at 10:51:05AM +0100, Greg Kurz wrote:  
> > > On Wed, 25 Nov 2020 13:39:47 +1100
> > > David Gibson <david@gibson.dropbear.id.au> wrote:
> > >   
> > > > On Mon, Nov 23, 2020 at 12:51:08PM +0100, Greg Kurz wrote:  
> > > > > On Mon, 23 Nov 2020 16:11:30 +1100
> > > > > David Gibson <david@gibson.dropbear.id.au> wrote:
> > > > >   
> > > > > > On Sat, Nov 21, 2020 at 12:42:03AM +0100, Greg Kurz wrote:  
> > > > > > > When it comes to resetting the compat mode of the vCPUS, there are
> > > > > > > two situations to consider:
> > > > > > > (1) machine reset should set the compat mode back to the machine 
> > > > > > > default,
> > > > > > >     ie. spapr->max_compat_pvr
> > > > > > > (2) hot plugged vCPUs should set their compat mode to mach the 
> > > > > > > boot vCPU,
> > > > > > >     ie. POWERPC_CPU(first_cpu)->compat_pvr
> > > > > > > 
> > > > > > > This is currently handled in two separate places: globally for 
> > > > > > > all vCPUs
> > > > > > > from the machine reset code for (1) and for each thread of a core 
> > > > > > > from
> > > > > > > the hotplug path for (2).
> > > > > > > 
> > > > > > > Since the machine reset code already resets all vCPUs, starting 
> > > > > > > with boot
> > > > > > > vCPU, consolidate the logic in spapr_reset_vcpu(). Special case 
> > > > > > > the boot
> > > > > > > vCPU so that it resets its compat mode back to the machine 
> > > > > > > default. Any
> > > > > > > other vCPU just need to match the compat mode of the boot vCPU.
> > > > > > > 
> > > > > > > Failing to set the compat mode during machine reset is a fatal 
> > > > > > > error,
> > > > > > > but not for hot plugged vCPUs. This is arguable because if we've 
> > > > > > > been
> > > > > > > able to set the boot vCPU compat mode at CAS or during machine 
> > > > > > > reset,
> > > > > > > it should definitely not fail for other vCPUs. Since 
> > > > > > > spapr_reset_vcpu()
> > > > > > > already has a fatal error path for kvm_check_mmu() failures, do 
> > > > > > > the
> > > > > > > same for ppc_set_compat().
> > > > > > > 
> > > > > > > This gets rid of an error path in spapr_core_plug(). It will allow
> > > > > > > further simplifications.
> > > > > > > 
> > > > > > > Signed-off-by: Greg Kurz <groug@kaod.org>
> > > > > > > ---
> > > > > > >  hw/ppc/spapr.c          | 16 ----------------
> > > > > > >  hw/ppc/spapr_cpu_core.c | 13 +++++++++++++
> > > > > > >  2 files changed, 13 insertions(+), 16 deletions(-)
> > > > > > > 
> > > > > > > diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
> > > > > > > index f58f77389e8e..da7586f548df 100644
> > > > > > > --- a/hw/ppc/spapr.c
> > > > > > > +++ b/hw/ppc/spapr.c
> > > > > > > @@ -1606,8 +1606,6 @@ static void 
> > > > > > > spapr_machine_reset(MachineState *machine)
> > > > > > >      spapr_ovec_cleanup(spapr->ov5_cas);
> > > > > > >      spapr->ov5_cas = spapr_ovec_new();
> > > > > > >  
> > > > > > > -    ppc_set_compat_all(spapr->max_compat_pvr, &error_fatal);
> > > > > > > -
> > > > > > >      /*
> > > > > > >       * This is fixing some of the default configuration of the 
> > > > > > > XIVE
> > > > > > >       * devices. To be called after the reset of the machine 
> > > > > > > devices.
> > > > > > > @@ -3785,20 +3783,6 @@ static void spapr_core_plug(HotplugHandler 
> > > > > > > *hotplug_dev, DeviceState *dev,
> > > > > > >  
> > > > > > >      core_slot->cpu = OBJECT(dev);
> > > > > > >  
> > > > > > > -    /*
> > > > > > > -     * Set compatibility mode to match the boot CPU, which was 
> > > > > > > either set
> > > > > > > -     * by the machine reset code or by CAS.
> > > > > > > -     */
> > > > > > > -    if (hotplugged) {
> > > > > > > -        for (i = 0; i < cc->nr_threads; i++) {
> > > > > > > -            if (ppc_set_compat(core->threads[i],
> > > > > > > -                               
> > > > > > > POWERPC_CPU(first_cpu)->compat_pvr,
> > > > > > > -                               errp) < 0) {
> > > > > > > -                return;
> > > > > > > -            }
> > > > > > > -        }
> > > > > > > -    }
> > > > > > > -
> > > > > > >      if (smc->pre_2_10_has_unused_icps) {
> > > > > > >          for (i = 0; i < cc->nr_threads; i++) {
> > > > > > >              cs = CPU(core->threads[i]);
> > > > > > > diff --git a/hw/ppc/spapr_cpu_core.c b/hw/ppc/spapr_cpu_core.c
> > > > > > > index 2f7dc3c23ded..17741a3fb77f 100644
> > > > > > > --- a/hw/ppc/spapr_cpu_core.c
> > > > > > > +++ b/hw/ppc/spapr_cpu_core.c
> > > > > > > @@ -27,6 +27,7 @@
> > > > > > >  
> > > > > > >  static void spapr_reset_vcpu(PowerPCCPU *cpu)
> > > > > > >  {
> > > > > > > +    PowerPCCPU *first_ppc_cpu = POWERPC_CPU(first_cpu);
> > > > > > >      CPUState *cs = CPU(cpu);
> > > > > > >      CPUPPCState *env = &cpu->env;
> > > > > > >      PowerPCCPUClass *pcc = POWERPC_CPU_GET_CLASS(cpu);
> > > > > > > @@ -69,6 +70,18 @@ static void spapr_reset_vcpu(PowerPCCPU *cpu)
> > > > > > >      kvm_check_mmu(cpu, &error_fatal);
> > > > > > >  
> > > > > > >      spapr_irq_cpu_intc_reset(spapr, cpu);
> > > > > > > +
> > > > > > > +    /*
> > > > > > > +     * The boot CPU is only reset during machine reset : reset 
> > > > > > > its
> > > > > > > +     * compatibility mode to the machine default. For other CPUs,
> > > > > > > +     * either cold plugged or hot plugged, set the compatibility 
> > > > > > > mode
> > > > > > > +     * to match the boot CPU, which was either set by the 
> > > > > > > machine reset
> > > > > > > +     * code or by CAS.
> > > > > > > +     */
> > > > > > > +    ppc_set_compat(cpu,
> > > > > > > +                   cpu == first_ppc_cpu ?
> > > > > > > +                   spapr->max_compat_pvr : 
> > > > > > > first_ppc_cpu->compat_pvr,
> > > > > > > +                   &error_fatal);  
> > > > > > 
> > > > > > This assumes that when it is called for a non-boot CPU, it has 
> > > > > > already
> > > > > > been called for the boot CPU..  Are we certain that's guaranteed by
> > > > > > the sequence of reset calls during a full machine reset?
> > > > > >   
> > > > > 
> > > > > This happens to be the case. Basically because the boot CPU core
> > > > > is created (including registering its reset handler) first and
> > > > > qemu_devices_reset() calls handlers in the same order they were
> > > > > registered.  
> > > > 
> > > > Right, I assumed it works for now, but it seems rather fragile, since
> > > > I'm not sure we're relying on guaranteed properties of the interface.  
> > > 
> > > The reset handler interface is absolutely undocumented, so I guess we
> > > have no formal guarantees at the present time. But since the current
> > > implementation has the property, would it be acceptable to carve it
> > > in stone with added documentation ? In the event of unlikely changes
> > > to the reset handler logic, people would _just_ need to make sure
> > > handlers are called in the same order they were registered.  
> > 
> > Yeah, maybe.
> > 
> > One other thing occurs to me: will we still do things in the right
> > order if the (initial) boot cpu is hot unplugged, then replugged
> > before a reset?
> >   
> 
> This can't happen AFAICT.
> 
> (qemu) qom-get /machine/unattached/device[1] core-id
> 0
> (qemu) device_del /machine/unattached/device[1]
> Error: Boot CPU core may not be unplugged
> 
> commit 62be8b044adf47327ebefdefb25f28a40316ebd0
> Author: Bharata B Rao <bharata@linux.vnet.ibm.com>
> Date:   Wed Jul 27 10:44:42 2016 +0530
> 
>     spapr: Prevent boot CPU core removal
> 
> 
> So yes, this adds yet another road block on the way to support hot
> unplug of the boot CPU. Is this a concern ?
> 
> If we go forward with this patch, maybe I should mention in the
> changelog/documentation the various assumptions which this patch
> is made under:
> - reset handlers are called in the same order they were registered
> - boot CPU registers its reset handler before other CPUs
> - boot CPU cannot be hot unplugged
> 
> These guarantee that the boot core is always reset before other
> cores during reset.
it might work for now but it seems fragile to me.

What if we  make compat mode a property and move setting it to machine code,
more precisely treat it like any other cpu feature property.
  
 if(need_compat_more)
    register_global_property(compat_mode)

that way when any cpu is created it will have this property set
and it won't depend on the order CPUs are created/reset

> 
> > > > Is there any way we could at least assert() if things are called out
> > > > of order?
> > > >   
> > > 
> > > Maybe. I'll look into it.
> > >   
> > > > >   
> > > > > > >  }
> > > > > > >  
> > > > > > >  void spapr_cpu_set_entry_state(PowerPCCPU *cpu, target_ulong 
> > > > > > > nip,  
> > > > > >   
> > > > >   
> > > > 
> > > > 
> > > >   
> > >   
> > 
> > 
> >   
>