Re: [Qemu-devel] [BUG/RFC] Two cpus are not brought up normally in SLES11 sp3 VM after reboot

[zhanghailiang]

On 2015/7/6 16:45, Paolo Bonzini wrote:
> On 06/07/2015 09:54, zhanghailiang wrote:
> >  From host, we found that QEMU vcpu1 thread and vcpu7 thread were not
> > consuming any cpu (Should be in idle state),
> > All of VCPUs' stacks in host is like bellow:
> >
> > [<ffffffffa07089b5>] kvm_vcpu_block+0x65/0xa0 [kvm]
> > [<ffffffffa071c7c1>] __vcpu_run+0xd1/0x260 [kvm]
> > [<ffffffffa071d508>] kvm_arch_vcpu_ioctl_run+0x68/0x1a0 [kvm]
> > [<ffffffffa0709cee>] kvm_vcpu_ioctl+0x38e/0x580 [kvm]
> > [<ffffffff8116be8b>] do_vfs_ioctl+0x8b/0x3b0
> > [<ffffffff8116c251>] sys_ioctl+0xa1/0xb0
> > [<ffffffff81468092>] system_call_fastpath+0x16/0x1b
> > [<00002ab9fe1f99a7>] 0x2ab9fe1f99a7
> > [<ffffffffffffffff>] 0xffffffffffffffff
> >
> > We looked into the kernel codes that could leading to the above 'Stuck'
> > warning,
> > and found that the only possible is the emulation of 'cpuid' instruct in
> > kvm/qemu has something wrong.
> > But since we can’t reproduce this problem, we are not quite sure.
> > Is there any possible that the cupid emulation in kvm/qemu has some bug ?
>
> Can you explain the relationship to the cpuid emulation?  What do the
> traces say about vcpus 1 and 7?

OK, we searched the VM's kernel codes with the 'Stuck' message, and  it is 
located in
do_boot_cpu(). It's in BSP context, the call process is:
BSP executes start_kernel() -> smp_init() -> smp_boot_cpus() -> do_boot_cpu() 
-> wakeup_secondary_via_INIT() to trigger APs.
It will wait 5s for APs to startup, if some AP not startup normally, it will 
print 'CPU%d Stuck' or 'CPU%d: Not responding'.

If it prints 'Stuck', it means the AP has received the SIPI interrupt and 
begins to execute the code
'ENTRY(trampoline_data)' (trampoline_64.S) , but be stuck in some places before 
smp_callin()(smpboot.c).
The follow is the starup process of BSP and AP.
BSP:
start_kernel()
  ->smp_init()
     ->smp_boot_cpus()
       ->do_boot_cpu()
           ->start_ip = trampoline_address(); //set the address that AP will go 
to execute
           ->wakeup_secondary_cpu_via_init(); // kick the secondary CPU
           ->for (timeout = 0; timeout < 50000; timeout++)
               if (cpumask_test_cpu(cpu, cpu_callin_mask)) break;// check if AP 
startup or not

APs:
ENTRY(trampoline_data) (trampoline_64.S)
      ->ENTRY(secondary_startup_64) (head_64.S)
         ->start_secondary() (smpboot.c)
            ->cpu_init();
            ->smp_callin();
                ->cpumask_set_cpu(cpuid, cpu_callin_mask); ->Note: if AP comes 
here, the BSP will not prints the error message.

From above call process, we can be sure that, the AP has been stuck between 
trampoline_data and the cpumask_set_cpu() in
smp_callin(), we look through these codes path carefully, and only found a 
'hlt' instruct that could block the process.
It is located in trampoline_data():

ENTRY(trampoline_data)
        ...

        call    verify_cpu              # Verify the cpu supports long mode
        testl   %eax, %eax              # Check for return code
        jnz     no_longmode

        ...

no_longmode:
        hlt
        jmp no_longmode

For the process verify_cpu(),
we can only find the 'cpuid' sensitive instruct that could lead VM exit from 
No-root mode.
This is why we doubt if cpuid emulation is wrong in KVM/QEMU that leading to 
the fail in verify_cpu.

From the message in VM, we know vcpu1 and vcpu7 is something wrong.
[    5.060042] CPU1: Stuck ??
[   10.170815] CPU7: Stuck ??
[   10.171648] Brought up 6 CPUs

Besides, the follow is the cpus message got from host.
80FF72F5-FF6D-E411-A8C8-000000821800:/home/fsp/hrg # virsh qemu-monitor-command 
instance-0000000
* CPU #0: pc=0x00007f64160c683d thread_id=68570
  CPU #1: pc=0xffffffff810301f1 (halted) thread_id=68573
  CPU #2: pc=0xffffffff810301e2 (halted) thread_id=68575
  CPU #3: pc=0xffffffff810301e2 (halted) thread_id=68576
  CPU #4: pc=0xffffffff810301e2 (halted) thread_id=68577
  CPU #5: pc=0xffffffff810301e2 (halted) thread_id=68578
  CPU #6: pc=0xffffffff810301e2 (halted) thread_id=68583
  CPU #7: pc=0xffffffff810301f1 (halted) thread_id=68584

Oh, i also forgot to mention in the above message that, we have bond each vCPU 
to different physical CPU in
host.

Thanks,
zhanghailiang