[Bug 1856335] Re: Cache Layout wrong on many Zen Arch CPUs

[Jan Klos]

Damir:
Hm, must be some misconfiguration, then. My config for Linux VMs to utilize 3 
out of the 4 CCXs. Important parts of the libvirt domain XML:

  <vcpu placement="static">24</vcpu>
  <iothreads>1</iothreads>
  <cputune>
    <vcpupin vcpu="0" cpuset="3"/>
    <vcpupin vcpu="1" cpuset="15"/>
    <vcpupin vcpu="2" cpuset="4"/>
    <vcpupin vcpu="3" cpuset="16"/>
    <vcpupin vcpu="4" cpuset="5"/>
    <vcpupin vcpu="5" cpuset="17"/>
    <vcpupin vcpu="6" cpuset="0,12"/>
    <vcpupin vcpu="7" cpuset="0,12"/>
    <vcpupin vcpu="8" cpuset="6"/>
    <vcpupin vcpu="9" cpuset="18"/>
    <vcpupin vcpu="10" cpuset="7"/>
    <vcpupin vcpu="11" cpuset="19"/>
    <vcpupin vcpu="12" cpuset="8"/>
    <vcpupin vcpu="13" cpuset="20"/>
    <vcpupin vcpu="14" cpuset="0,12"/>
    <vcpupin vcpu="15" cpuset="0,12"/>
    <vcpupin vcpu="16" cpuset="9"/>
    <vcpupin vcpu="17" cpuset="21"/>
    <vcpupin vcpu="18" cpuset="10"/>
    <vcpupin vcpu="19" cpuset="22"/>
    <vcpupin vcpu="20" cpuset="11"/>
    <vcpupin vcpu="21" cpuset="23"/>
    <vcpupin vcpu="22" cpuset="0,12"/>
    <vcpupin vcpu="23" cpuset="0,12"/>
    <emulatorpin cpuset="1,13"/>
    <iothreadpin iothread="1" cpuset="2,14"/>
  </cputune>
  <os>
    <type arch="x86_64" machine="pc-q35-5.0">hvm</type>
    <loader readonly="yes" 
type="pflash">/usr/share/ovmf/x64/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/ccxtest-clone_VARS.fd</nvram>
  </os>
.
.
.
  <qemu:commandline>
    <qemu:arg value="-cpu"/>
    <qemu:arg 
value="host,topoext=on,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff,host-cache-info=on,-amd-stibp"/>
  </qemu:commandline>

The CPUs with cpuset="0,12" are disabled once booted. The host-cache-
info=on is the part that makes sure that the cache config is passed to
the VM (but unfortunately does not take disabled cores into account,
which results in incorrect config). The qemu:commandline is added
because I need to add -amd-stibp, otherwise I wouldn't be able to boot.
This overrides most parts in the <cpu> XML part.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1856335

Title:
  Cache Layout wrong on many Zen Arch CPUs

Status in QEMU:
  New

Bug description:
  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems
  to always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%)
  in L3 Cache Layout aware applications.

  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='8' threads='1'/>

  In windows, coreinfo reports correctly:

  ****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

  On a 3-CCX CPU (3960X /w 6 cores and no SMT):

   <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='6' threads='1'/>

  in windows, coreinfo reports incorrectly:

  ****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64

  Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm.

  With newer Qemu there is a fix (that does behave correctly) in using the dies 
parameter:
   <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

  The problem is that the dies are exposed differently than how AMD does
  it natively, they are exposed to Windows as sockets, which means, that
  if you are nto a business user, you can't ever have a machine with
  more than two CCX (6 cores) as consumer versions of Windows only
  supports two sockets. (Should this be reported as a separate bug?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1856335/+subscriptions