Re: [Qemu-devel] [PATCH for-2.8 00/18] pc: q35: x2APIC support in kvm_apic mode

[Igor Mammedov]

On Tue, 9 Aug 2016 11:23:58 +0800
Chao Gao <address@hidden> wrote:

> On Mon, Aug 08, 2016 at 11:18:20AM +0200, Igor Mammedov wrote:
> >On Mon, 8 Aug 2016 15:41:23 +0800
> >Chao Gao <address@hidden> wrote:
> >  
> >> HI, everyone.
> >> 
> >> We have done some tests after merging this patch set into the lastest qemu
> >> master. In kvm aspect, we use the lastest kvm linux-next branch. Here are
> >> some problems we have met.
> >> 
> >> 1. We can't boot up a 288 vcpus linux guest with CLI:
> >> qemu-system-x86_64 -boot c -m 4096 -sdl -monitor pty --enable-kvm \
> >> -M kernel-irqchip=split -serial stdio -bios bios.bin -smp cpus=288 \
> >> -hda vdisk.img -device intel-iommu,intremap=on -machine q35.
> >> The problem exists, even after we only assign 32 vcpus to the linux guest.
> >> Maybe the output "do_IRQ: 146.113 No irq handler for vector (irq -1)" is a 
> >> clue.
> >> The output of qemu and kernel is in attachments. Do you have any idea
> >> about the problem and how to solve it?  
> >I don't think we ever looked at "kernel-irqchip=split" only in kernel 
> >variant's
> >been targeted so far.
> >Radim probably knows better whether it should work or not.
> >
> >Have you tried with smaller amount of CPUs but with APIC IDs above 254,
> >like in test below?
> >
> >[...]
> >  
> >> >Tested with following CLI:
> >> > QEMU -M q35 -enable-kvm -smp 1,sockets=9,cores=32,threads=1,maxcpus=288 \
> >> >      -device qemu64-x86_64-cpu,socket-id=8,core-id=30,thread-id=0       \
> >> >      -bios x2apic_bios.bin  
> 
> I test with CLI:
> qemu-system-x86_64 -M q35 \
> -enable-kvm -smp 1,sockets=9,cores=32,threads=1,maxcpus=288 \
> -device qemu64-x86_64-cpu,socket-id=8,core-id=30,thread-id=0 \
> -bios bios.bin -hda vdisk.img -serial stdio -m 4096 2>>qemu_and_guest.log >&2
> 
> But, I think there should have a cpu with initial apicid >255 
> in /proc/cpuinfo. The log(in attachments) shows that the guest kernel 
> treats the other cpu as a bad one. What do you think cause the problem?
Bad cpu happens because of following:
[    0.319911] x2apic: IRQ remapping doesn't support X2APIC mode
[    0.321427] x2apic disabled

if I recall correctly Radim gave me a hack/patch to workaround it,
and that's what I've been testing with.

Anyway from log it looks like CPU with x2apic is there and seabios
initialized it correctly. Missing part is working irq remapping.



> 
> # cat /proc/interrupts
> localhost login:             CPU0       
>    0:        125   IO-APIC-edge      timer
>    1:        117   IO-APIC-edge      i8042
>    4:        382   IO-APIC-edge      serial
>    7:          0   IO-APIC-edge      parport0
>    8:          1   IO-APIC-edge      rtc0
>    9:          0   IO-APIC-fasteoi   acpi
>   12:       1661   IO-APIC-edge      i8042
>   16:          0   IO-APIC-fasteoi   i801_smbus
>   22:         27   IO-APIC-fasteoi   enp0s2
>   24:       7310   PCI-MSI-edge      0000:00:1f.2
>  NMI:          0   Non-maskable interrupts
>  LOC:       6401   Local timer interrupts
>  SPU:          0   Spurious interrupts
>  PMI:          0   Performance monitoring interrupts
>  IWI:       3870   IRQ work interrupts
>  RTR:          0   APIC ICR read retries
>  RES:          0   Rescheduling interrupts
>  CAL:          0   Function call interrupts
>  TLB:          0   TLB shootdowns
>  TRM:          0   Thermal event interrupts
>  THR:          0   Threshold APIC interrupts
>  MCE:          0   Machine check exceptions
>  MCP:          1   Machine check polls
>  ERR:          0
>  MIS:          0
> 
> # cat /proc/cpuinfo
> processor       : 0
> vendor_id       : GenuineIntel
> cpu family      : 6
> model           : 6
> model name      : QEMU Virtual CPU version 2.5+
> stepping        : 3
> microcode       : 0x1
> cpu MHz         : 3591.682
> cache size      : 4096 KB
> physical id     : 0
> siblings        : 1
> core id         : 0
> cpu cores       : 1
> apicid          : 0
> initial apicid  : 0
> fpu             : yes
> fpu_exception   : yes
> cpuid level     : 13
> wp              : yes
> flags           : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov 
> pse36 clflush mmx fxsr sse sse2 ht syscall nx lm rep_good nopl xtopology pni 
> cx16 x2apic hypervisor lahf_lm
> bogomips        : 7183.36
> clflush size    : 64
> cache_alignment : 64
> address sizes   : 40 bits physical, 48 bits virtual
> power management: