Re: [Qemu-ppc] [PATCH v4 1/3] spapr/rtas: disable the decrementer interrupt when a CPU is unplugged

[David Gibson]

On Fri, Nov 24, 2017 at 08:05:48AM +0100, Cédric Le Goater wrote:
> When a CPU is stopped with the 'stop-self' RTAS call, its state
> 'halted' is switched to 1 and, in this case, the MSR is not taken into
> account anymore in the cpu_has_work() routine. Only the pending
> hardware interrupts are checked with their LPCR:PECE* enablement bit.
> 
> If the DECR timer fires after 'stop-self' is called and before the CPU
> 'stop' state is reached, the nearly-dead CPU will have some work to do
> and the guest will crash. This case happens very frequently with the
> not yet upstream P9 XIVE exploitation mode. In XICS mode, the DECR is
> occasionally fired but after 'stop' state, so no work is to be done
> and the guest survives.
> 
> I suspect there is a race between the QEMU mainloop triggering the
> timers and the TCG CPU thread but I could not quite identify the root
> cause. To be safe, let's disable in the LPCR all the exceptions which
> can cause an exit while the CPU is in power-saving mode and reenable
> them when the CPU is started.
> 
> Signed-off-by: Cédric Le Goater <address@hidden>

Applied to ppc-for-2.12.

> ---
> 
> Changes in v4:
> 
>  - used the 'lpcr_pm' field of PowerPCCPUClass
> 
> Changes in v3:
> 
>  - introduced a cpu_ppc_papr_pece_bits() helper to gather the PECE
>    bits depending on the CPU family.   
>  - enabled Power-saving mode Exit Cause exceptions only on the boot CPU.
>  
> Changes in v2:
> 
>  - used a new routine ppc_cpu_pvr_match() to discriminate CPU versions
>  - removed the LPCR:PECE* enablement bit when the CPU is initialized
>    if it is a secondary
> 
>  hw/ppc/spapr_rtas.c         | 11 +++++++++++
>  target/ppc/translate_init.c |  9 ++++++---
>  2 files changed, 17 insertions(+), 3 deletions(-)
> 
> diff --git a/hw/ppc/spapr_rtas.c b/hw/ppc/spapr_rtas.c
> index cdf0b607a0a0..858adb1bf3a9 100644
> --- a/hw/ppc/spapr_rtas.c
> +++ b/hw/ppc/spapr_rtas.c
> @@ -162,6 +162,7 @@ static void rtas_start_cpu(PowerPCCPU *cpu_, 
> sPAPRMachineState *spapr,
>      if (cpu != NULL) {
>          CPUState *cs = CPU(cpu);
>          CPUPPCState *env = &cpu->env;
> +        PowerPCCPUClass *pcc = POWERPC_CPU_GET_CLASS(cpu);
>  
>          if (!cs->halted) {
>              rtas_st(rets, 0, RTAS_OUT_HW_ERROR);
> @@ -174,6 +175,10 @@ static void rtas_start_cpu(PowerPCCPU *cpu_, 
> sPAPRMachineState *spapr,
>          kvm_cpu_synchronize_state(cs);
>  
>          env->msr = (1ULL << MSR_SF) | (1ULL << MSR_ME);
> +
> +        /* Enable Power-saving mode Exit Cause exceptions for the new CPU */
> +        env->spr[SPR_LPCR] |= pcc->lpcr_pm;
> +
>          env->nip = start;
>          env->gpr[3] = r3;
>          cs->halted = 0;
> @@ -197,6 +202,7 @@ static void rtas_stop_self(PowerPCCPU *cpu, 
> sPAPRMachineState *spapr,
>  {
>      CPUState *cs = CPU(cpu);
>      CPUPPCState *env = &cpu->env;
> +    PowerPCCPUClass *pcc = POWERPC_CPU_GET_CLASS(cpu);
>  
>      cs->halted = 1;
>      qemu_cpu_kick(cs);
> @@ -210,6 +216,11 @@ static void rtas_stop_self(PowerPCCPU *cpu, 
> sPAPRMachineState *spapr,
>       * no need to bother with specific bits, we just clear it.
>       */
>      env->msr = 0;
> +
> +    /* Disable Power-saving mode Exit Cause exceptions for the CPU.
> +     * This could deliver an interrupt on a dying CPU and crash the
> +     * guest */
> +    env->spr[SPR_LPCR] &= ~pcc->lpcr_pm;
>  }
>  
>  static inline int sysparm_st(target_ulong addr, target_ulong len,
> diff --git a/target/ppc/translate_init.c b/target/ppc/translate_init.c
> index 828d7e778c3b..78a4a581bab7 100644
> --- a/target/ppc/translate_init.c
> +++ b/target/ppc/translate_init.c
> @@ -8911,6 +8911,7 @@ void cpu_ppc_set_papr(PowerPCCPU *cpu, 
> PPCVirtualHypervisor *vhyp)
>      CPUPPCState *env = &cpu->env;
>      ppc_spr_t *lpcr = &env->spr_cb[SPR_LPCR];
>      ppc_spr_t *amor = &env->spr_cb[SPR_AMOR];
> +    CPUState *cs = CPU(cpu);
>  
>      cpu->vhyp = vhyp;
>  
> @@ -8953,10 +8954,12 @@ void cpu_ppc_set_papr(PowerPCCPU *cpu, 
> PPCVirtualHypervisor *vhyp)
>          }
>      }
>  
> -    /* Also set the power-saving mode bits which depend on the CPU
> -     * family
> +    /* Only enable Power-saving mode Exit Cause exceptions on the boot
> +     * CPU. The RTAS command start-cpu will enable them on secondaries.
>       */
> -    lpcr->default_value |= pcc->lpcr_pm;
> +    if (cs == first_cpu) {
> +        lpcr->default_value |= pcc->lpcr_pm;
> +    }
>  
>      /* We should be followed by a CPU reset but update the active value
>       * just in case...

-- 
David Gibson                    | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au  | minimalist, thank you.  NOT _the_ _other_
                                | _way_ _around_!
http://www.ozlabs.org/~dgibson

signature.asc
Description: PGP signature