Re: [Qemu-devel] [RFC] target-i386: present virtual L3 cache info for vcpus

[Eduardo Habkost]

On Mon, Aug 29, 2016 at 09:17:02AM +0800, Longpeng (Mike) wrote:
> This patch presents virtual L3 cache info for virtual cpus.

Just changing the L3 cache size in the CPUID code will make
guests see a different cache topology after upgrading QEMU (even
on live migration). If you want to change the default, you need
to keep the old values on old machine-types.

In other words, we need to make the cache size configurable, and
set compat_props on PC_COMPAT_2_7.

Other comments below:

> 
> Some software algorithms are based on the hardware's cache info, for example,
> for x86 linux kernel, when cpu1 want to wakeup a task on cpu2, cpu1 will 
> trigger
> a resched IPI and told cpu2 to do the wakeup if they don't share low level
> cache. Oppositely, cpu1 will access cpu2's runqueue directly if they share 
> llc.
> The relevant linux-kernel code as bellow:
> 
>       static void ttwu_queue(struct task_struct *p, int cpu)
>       {
>               struct rq *rq = cpu_rq(cpu);
>               ......
>               if (... && !cpus_share_cache(smp_processor_id(), cpu)) {
>                       ......
>                       ttwu_queue_remote(p, cpu); /* will trigger RES IPI */
>                       return;
>               }
>               ......
>               ttwu_do_activate(rq, p, 0); /* access target's rq directly */
>               ......
>       }
> 
> In real hardware, the cpus on the same socket share L3 cache, so one won't
> trigger a resched IPIs when wakeup a task on others. But QEMU doesn't present 
> a
> virtual L3 cache info for VM, then the linux guest will trigger lots of RES 
> IPIs
> under some workloads even if the virtual cpus belongs to the same virtual 
> socket.
> 
> For KVM, this degrades performance, because there will be lots of vmexit due 
> to
> guest send IPIs.
> 
> The workload is a SAP HANA's testsuite, we run it one round(about 40 minuates)
> and observe the (Suse11sp3)Guest's amounts of RES IPIs which triggering during
> the period:
> 
>         No-L3           With-L3(applied this patch)
> cpu0: 363890          44582
> cpu1: 373405          43109
> cpu2: 340783          43797
> cpu3: 333854          43409
> cpu4: 327170          40038
> cpu5: 325491          39922
> cpu6: 319129          42391
> cpu7: 306480          41035
> cpu8: 161139          32188
> cpu9: 164649          31024
> cpu10:        149823          30398
> cpu11:        149823          32455
> cpu12:        164830          35143
> cpu13:        172269          35805
> cpu14:        179979          33898
> cpu15:        194505          32754
> avg:  268963.6        40129.8
> 
> The VM's topology is "1*socket 8*cores 2*threads".
> After present virtual L3 cache info for VM, the amounts of RES IPI in guest
> reduce 85%.

What happens to overall system performance if the VCPU threads
actually run on separate sockets in the host?

Other questions about the code below:

> 
> Signed-off-by: Longpeng(Mike) <address@hidden>
> ---
>  target-i386/cpu.c | 34 +++++++++++++++++++++++++++-------
>  1 file changed, 27 insertions(+), 7 deletions(-)
> 
> diff --git a/target-i386/cpu.c b/target-i386/cpu.c
> index 6a1afab..5a5fd06 100644
> --- a/target-i386/cpu.c
> +++ b/target-i386/cpu.c
> @@ -57,6 +57,7 @@
>  #define CPUID_2_L1D_32KB_8WAY_64B 0x2c
>  #define CPUID_2_L1I_32KB_8WAY_64B 0x30
>  #define CPUID_2_L2_2MB_8WAY_64B   0x7d
> +#define CPUID_2_L3_12MB_24WAY_64B 0xea
> 
> 
>  /* CPUID Leaf 4 constants: */
> @@ -131,11 +132,15 @@
>  #define L2_LINES_PER_TAG       1
>  #define L2_SIZE_KB_AMD       512
> 
> -/* No L3 cache: */
> -#define L3_SIZE_KB             0 /* disabled */
> -#define L3_ASSOCIATIVITY       0 /* disabled */
> -#define L3_LINES_PER_TAG       0 /* disabled */
> -#define L3_LINE_SIZE           0 /* disabled */
> +/* Level 3 unified cache: */
> +#define L3_LINE_SIZE          64
> +#define L3_ASSOCIATIVITY      24
> +#define L3_SETS             8192
> +#define L3_PARTITIONS          1
> +#define L3_DESCRIPTOR CPUID_2_L3_12MB_24WAY_64B
> +/*FIXME: CPUID leaf 0x80000006 is inconsistent with leaves 2 & 4 */

Why are you intentionally introducing a bug?

> +#define L3_LINES_PER_TAG       1
> +#define L3_SIZE_KB_AMD      1024
> 
>  /* TLB definitions: */
> 
> @@ -2328,7 +2333,7 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index,
> uint32_t count,

I believe Thunderbird line wrapping broke the patch. Git can't
apply it.

>          }
>          *eax = 1; /* Number of CPUID[EAX=2] calls required */
>          *ebx = 0;
> -        *ecx = 0;
> +        *ecx = (L3_DESCRIPTOR);
>          *edx = (L1D_DESCRIPTOR << 16) | \
>                 (L1I_DESCRIPTOR <<  8) | \
>                 (L2_DESCRIPTOR);
> @@ -2374,6 +2379,21 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index,
> uint32_t count,
>                  *ecx = L2_SETS - 1;
>                  *edx = CPUID_4_NO_INVD_SHARING;
>                  break;
> +            case 3: /* L3 cache info */
> +                *eax |= CPUID_4_TYPE_UNIFIED | \
> +                        CPUID_4_LEVEL(3) | \
> +                        CPUID_4_SELF_INIT_LEVEL;
> +                /*
> +                * According to qemu's APICIDs generating rule, this can make
> +                * sure vcpus on the same vsocket get the same llc_id.
> +                */
> +                *eax |= (cs->nr_cores * cs->nr_threads - 1) << 14;

The above comment doesn't seem to be true.

For example: if nr_cores=9,nr_threads=3, then:

SMT_Mask_Width = Log2 ( RoundToNearestPof2(CPUID.1:EBX[23:16]) / 
((CPUID.(EAX=4, ECX=0):EAX[31:26] ) + 1))
               = Log2 ( RoundToNearestPof2(nr_cores * nr_threads) / ((nr_cores 
- 1) + 1 ))
               = Log2 ( RoundToNearestPof2(27) / 9 )
               = Log2 ( 32 / 9 ) = Log2 ( 3 ) = 2
CoreOnly_Mask_Width = Log2(1 + (CPUID.(EAX=4, ECX=0):EAX[31:26]))
                    = Log2(1 + (nr_cores - 1)) = Log2(nr_cores) =
                    = Log2(9) = 4
CorePlus_Mask_Width = CoreOnly_Mask_Width + SMT_Mask_Width
                    = 2 + 4 = 6

But:

Cache_Mask_Width[3] = Log2(RoundToNearestPof2( (1 + CPUID.(EAX=4, 
ECX=n):EAX[25:14]))
                    = Log2(RoundToNearestPof2( (1 + CPUID.(EAX=4, 
ECX=3):EAX[25:14]))
                    = Log2(RoundToNearestPof2( (1 + (nr_cores * nr_threads - 
1)))
                    = Log2(RoundToNearestPof2( nr_cores * nr_threads ))
                    = Log2(RoundToNearestPof2( 27 ))
                    = Log2(32) = 5

That means the VCPU at package 1, core 8, thread 2 (APIC ID
1100010b) have Package_ID=1 but L3 Cache_ID would be 3.


> +                *ebx = (L3_LINE_SIZE - 1) | \
> +                       ((L3_PARTITIONS - 1) << 12) | \
> +                       ((L3_ASSOCIATIVITY - 1) << 22);
> +                *ecx = L3_SETS - 1;
> +                *edx = CPUID_4_INCLUSIVE | CPUID_4_COMPLEX_IDX;
> +                break;
>              default: /* end of info */
>                  *eax = 0;
>                  *ebx = 0;
> @@ -2585,7 +2605,7 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index,
> uint32_t count,
>          *ecx = (L2_SIZE_KB_AMD << 16) | \
>                 (AMD_ENC_ASSOC(L2_ASSOCIATIVITY) << 12) | \
>                 (L2_LINES_PER_TAG << 8) | (L2_LINE_SIZE);
> -        *edx = ((L3_SIZE_KB/512) << 18) | \
> +        *edx = ((L3_SIZE_KB_AMD / 512) << 18) | \
>                 (AMD_ENC_ASSOC(L3_ASSOCIATIVITY) << 12) | \
>                 (L3_LINES_PER_TAG << 8) | (L3_LINE_SIZE);
>          break;
> -- 
> 1.8.3.1
> 

-- 
Eduardo