Re: [PATCH v2 1/3] hw/i386: Initialize topo_ids from CpuInstanceProperties

[Igor Mammedov]

On Mon, 13 Jul 2020 14:30:29 -0500
Babu Moger <babu.moger@amd.com> wrote:

> > -----Original Message-----
> > From: Igor Mammedov <imammedo@redhat.com>
> > Sent: Monday, July 13, 2020 12:32 PM
> > To: Moger, Babu <Babu.Moger@amd.com>
> > Cc: pbonzini@redhat.com; rth@twiddle.net; ehabkost@redhat.com; qemu-
> > devel@nongnu.org
> > Subject: Re: [PATCH v2 1/3] hw/i386: Initialize topo_ids from
> > CpuInstanceProperties
> > 
> > On Mon, 13 Jul 2020 11:43:33 -0500
> > Babu Moger <babu.moger@amd.com> wrote:
> >   
> > > On 7/13/20 11:17 AM, Igor Mammedov wrote:  
> > > > On Mon, 13 Jul 2020 10:02:22 -0500
> > > > Babu Moger <babu.moger@amd.com> wrote:
> > > >  
> > > >>> -----Original Message-----
> > > >>> From: Igor Mammedov <imammedo@redhat.com>
> > > >>> Sent: Monday, July 13, 2020 4:08 AM
> > > >>> To: Moger, Babu <Babu.Moger@amd.com>
> > > >>> Cc: pbonzini@redhat.com; rth@twiddle.net; ehabkost@redhat.com;
> > > >>> qemu- devel@nongnu.org
> > > >>> Subject: Re: [PATCH v2 1/3] hw/i386: Initialize topo_ids from
> > > >>> CpuInstanceProperties  
> > > > [...]  
> > > >>>> +
> > > >>>> +/*
> > > >>>> + * Initialize topo_ids from CpuInstanceProperties
> > > >>>> + * node_id in CpuInstanceProperties(or in CPU device) is a 
> > > >>>> sequential
> > > >>>> + * number, but while building the topology  
> > > >>>  
> > > >>>> we need to separate it for
> > > >>>> + * each socket(mod nodes_per_pkg).  
> > > >>> could you clarify a bit more on why this is necessary?  
> > > >>
> > > >> If you have two sockets and 4 numa nodes, node_id in
> > > >> CpuInstanceProperties will be number sequentially as 0, 1, 2, 3.
> > > >> But in EPYC topology, it will be  0, 1, 0, 1( Basically mod % number 
> > > >> of nodes  
> > per socket).  
> > > >
> > > > I'm confused, let's suppose we have 2 EPYC sockets with 2 nodes per
> > > > socket so APIC id woulbe be composed like:
> > > >
> > > >  1st socket
> > > >    pkg_id(0) | node_id(0)
> > > >    pkg_id(0) | node_id(1)
> > > >
> > > >  2nd socket
> > > >    pkg_id(1) | node_id(0)
> > > >    pkg_id(1) | node_id(1)
> > > >
> > > > if that's the case, then EPYC's node_id here doesn't look like a
> > > > NUMA node in the sense it's usually used (above config would have 4
> > > > different memory controllers => 4 conventional NUMA nodes).  
> > >
> > > EPIC model uses combination of socket id and node id to identify the
> > > numa nodes. So, it internally uses all the information.  
> > 
> > well with above values, EPYC's node_id doesn't look like it's specifying a
> > machine numa node, but rather a node index within single socket. In which 
> > case,
> > it doesn't make much sense calling it NUMA node_id, it's rather some index
> > within a socket. (it starts looking like terminology is all mixed up)
> > 
> > If you have access to a milti-socket EPYC machine, can you dump and post 
> > here
> > its apic ids, pls?  
> 
> Here is the output from my EPYC machine with 2 sockets and totally 8
> nodes(SMT disabled). The cpus 0-31 are in socket 0 and  cpus 32-63 in
> socket 1.
> 
> # lscpu
> Architecture:        x86_64
> CPU op-mode(s):      32-bit, 64-bit
> Byte Order:          Little Endian
> CPU(s):              64
> On-line CPU(s) list: 0-63
> Thread(s) per core:  1
> Core(s) per socket:  32
> Socket(s):           2
> NUMA node(s):        8
> Vendor ID:           AuthenticAMD
> CPU family:          23
> Model:               1
> Model name:          AMD Eng Sample: 1S1901A4VIHF5_30/19_N
> Stepping:            2
> CPU MHz:             2379.233
> CPU max MHz:         1900.0000
> CPU min MHz:         1200.0000
> BogoMIPS:            3792.81
> Virtualization:      AMD-V
> L1d cache:           32K
> L1i cache:           64K
> L2 cache:            512K
> L3 cache:            8192K
> NUMA node0 CPU(s):   0-7
> NUMA node1 CPU(s):   8-15
> NUMA node2 CPU(s):   16-23
> NUMA node3 CPU(s):   24-31
> NUMA node4 CPU(s):   32-39
> NUMA node5 CPU(s):   40-47
> NUMA node6 CPU(s):   48-55
> NUMA node7 CPU(s):   56-63
> 
> Here is the output of #cpuid  -l 0x8000001e  -r
> 
> You may want to refer
> https://www.amd.com/system/files/TechDocs/54945_3.03_ppr_ZP_B2_pub.zip
> (section 2.1.12.2.1.3 ApicId Enumeration Requirements).
> Note that this is a general guideline. We tried to generalize in qemu as
> much as possible. It is bit complex.
Thanks, I'll look into it.
Can you also dump SRAT table from that machine, please?
I'd like to see CPUs relation to NUMA nodes described in SRAT.

> 
> CPU 0:
>    0x8000001e 0x00: eax=0x00000000 ebx=0x00000100 ecx=0x00000300
> edx=0x00000000
> CPU 1:
>    0x8000001e 0x00: eax=0x00000002 ebx=0x00000101 ecx=0x00000300
> edx=0x00000000
> CPU 2:
>    0x8000001e 0x00: eax=0x00000004 ebx=0x00000102 ecx=0x00000300
> edx=0x00000000
> CPU 3:
>    0x8000001e 0x00: eax=0x00000006 ebx=0x00000103 ecx=0x00000300
> edx=0x00000000
> CPU 4:
>    0x8000001e 0x00: eax=0x00000008 ebx=0x00000104 ecx=0x00000300
> edx=0x00000000
> CPU 5:
>    0x8000001e 0x00: eax=0x0000000a ebx=0x00000105 ecx=0x00000300
> edx=0x00000000
> CPU 6:
>    0x8000001e 0x00: eax=0x0000000c ebx=0x00000106 ecx=0x00000300
> edx=0x00000000
> CPU 7:
>    0x8000001e 0x00: eax=0x0000000e ebx=0x00000107 ecx=0x00000300
> edx=0x00000000
> CPU 8:
>    0x8000001e 0x00: eax=0x00000010 ebx=0x00000108 ecx=0x00000301
> edx=0x00000000
> CPU 9:
>    0x8000001e 0x00: eax=0x00000012 ebx=0x00000109 ecx=0x00000301
> edx=0x00000000
> CPU 10:
>    0x8000001e 0x00: eax=0x00000014 ebx=0x0000010a ecx=0x00000301
> edx=0x00000000
> CPU 11:
>    0x8000001e 0x00: eax=0x00000016 ebx=0x0000010b ecx=0x00000301
> edx=0x00000000
> CPU 12:
>    0x8000001e 0x00: eax=0x00000018 ebx=0x0000010c ecx=0x00000301
> edx=0x00000000
> CPU 13:
>    0x8000001e 0x00: eax=0x0000001a ebx=0x0000010d ecx=0x00000301
> edx=0x00000000
> CPU 14:
>    0x8000001e 0x00: eax=0x0000001c ebx=0x0000010e ecx=0x00000301
> edx=0x00000000
> CPU 15:
>    0x8000001e 0x00: eax=0x0000001e ebx=0x0000010f ecx=0x00000301
> edx=0x00000000
> CPU 16:
>    0x8000001e 0x00: eax=0x00000020 ebx=0x00000110 ecx=0x00000302
> edx=0x00000000
> CPU 17:
>    0x8000001e 0x00: eax=0x00000022 ebx=0x00000111 ecx=0x00000302
> edx=0x00000000
> CPU 18:
>    0x8000001e 0x00: eax=0x00000024 ebx=0x00000112 ecx=0x00000302
> edx=0x00000000
> CPU 19:
>    0x8000001e 0x00: eax=0x00000026 ebx=0x00000113 ecx=0x00000302
> edx=0x00000000
> CPU 20:
>    0x8000001e 0x00: eax=0x00000028 ebx=0x00000114 ecx=0x00000302
> edx=0x00000000
> CPU 21:
>    0x8000001e 0x00: eax=0x0000002a ebx=0x00000115 ecx=0x00000302
> edx=0x00000000
> CPU 22:
>    0x8000001e 0x00: eax=0x0000002c ebx=0x00000116 ecx=0x00000302
> edx=0x00000000
> CPU 23:
>    0x8000001e 0x00: eax=0x0000002e ebx=0x00000117 ecx=0x00000302
> edx=0x00000000
> CPU 24:
>    0x8000001e 0x00: eax=0x00000030 ebx=0x00000118 ecx=0x00000303
> edx=0x00000000
> CPU 25:
>    0x8000001e 0x00: eax=0x00000032 ebx=0x00000119 ecx=0x00000303
> edx=0x00000000
> CPU 26:
>    0x8000001e 0x00: eax=0x00000034 ebx=0x0000011a ecx=0x00000303
> edx=0x00000000
> CPU 27:
>    0x8000001e 0x00: eax=0x00000036 ebx=0x0000011b ecx=0x00000303
> edx=0x00000000
> CPU 28:
>    0x8000001e 0x00: eax=0x00000038 ebx=0x0000011c ecx=0x00000303
> edx=0x00000000
> CPU 29:
>    0x8000001e 0x00: eax=0x0000003a ebx=0x0000011d ecx=0x00000303
> edx=0x00000000
> CPU 30:
>    0x8000001e 0x00: eax=0x0000003c ebx=0x0000011e ecx=0x00000303
> edx=0x00000000
> CPU 31:
>    0x8000001e 0x00: eax=0x0000003e ebx=0x0000011f ecx=0x00000303
> edx=0x00000000
> CPU 32:
>    0x8000001e 0x00: eax=0x00000040 ebx=0x00000100 ecx=0x00000304
> edx=0x00000000
> CPU 33:
>    0x8000001e 0x00: eax=0x00000042 ebx=0x00000101 ecx=0x00000304
> edx=0x00000000
> CPU 34:
>    0x8000001e 0x00: eax=0x00000044 ebx=0x00000102 ecx=0x00000304
> edx=0x00000000
> CPU 35:
>    0x8000001e 0x00: eax=0x00000046 ebx=0x00000103 ecx=0x00000304
> edx=0x00000000
> CPU 36:
>    0x8000001e 0x00: eax=0x00000048 ebx=0x00000104 ecx=0x00000304
> edx=0x00000000
> CPU 37:
>    0x8000001e 0x00: eax=0x0000004a ebx=0x00000105 ecx=0x00000304
> edx=0x00000000
> CPU 38:
>    0x8000001e 0x00: eax=0x0000004c ebx=0x00000106 ecx=0x00000304
> edx=0x00000000
> CPU 39:
>    0x8000001e 0x00: eax=0x0000004e ebx=0x00000107 ecx=0x00000304
> edx=0x00000000
> CPU 40:
>    0x8000001e 0x00: eax=0x00000050 ebx=0x00000108 ecx=0x00000305
> edx=0x00000000
> CPU 41:
>    0x8000001e 0x00: eax=0x00000052 ebx=0x00000109 ecx=0x00000305
> edx=0x00000000
> CPU 42:
>    0x8000001e 0x00: eax=0x00000054 ebx=0x0000010a ecx=0x00000305
> edx=0x00000000
> CPU 43:
>    0x8000001e 0x00: eax=0x00000056 ebx=0x0000010b ecx=0x00000305
> edx=0x00000000
> CPU 44:
>    0x8000001e 0x00: eax=0x00000058 ebx=0x0000010c ecx=0x00000305
> edx=0x00000000
> CPU 45:
>    0x8000001e 0x00: eax=0x0000005a ebx=0x0000010d ecx=0x00000305
> edx=0x00000000
> CPU 46:
>    0x8000001e 0x00: eax=0x0000005c ebx=0x0000010e ecx=0x00000305
> edx=0x00000000
> CPU 47:
>    0x8000001e 0x00: eax=0x0000005e ebx=0x0000010f ecx=0x00000305
> edx=0x00000000
> CPU 48:
>    0x8000001e 0x00: eax=0x00000060 ebx=0x00000110 ecx=0x00000306
> edx=0x00000000
> 
> CPU 49:
>    0x8000001e 0x00: eax=0x00000062 ebx=0x00000111 ecx=0x00000306
> edx=0x00000000
> CPU 50:
>    0x8000001e 0x00: eax=0x00000064 ebx=0x00000112 ecx=0x00000306
> edx=0x00000000
> CPU 51:
>    0x8000001e 0x00: eax=0x00000066 ebx=0x00000113 ecx=0x00000306
> edx=0x00000000
> CPU 52:
>    0x8000001e 0x00: eax=0x00000068 ebx=0x00000114 ecx=0x00000306
> edx=0x00000000
> CPU 53:
>    0x8000001e 0x00: eax=0x0000006a ebx=0x00000115 ecx=0x00000306
> edx=0x00000000
> CPU 54:
>    0x8000001e 0x00: eax=0x0000006c ebx=0x00000116 ecx=0x00000306
> edx=0x00000000
> CPU 55:
>    0x8000001e 0x00: eax=0x0000006e ebx=0x00000117 ecx=0x00000306
> edx=0x00000000
> CPU 56:
>    0x8000001e 0x00: eax=0x00000070 ebx=0x00000118 ecx=0x00000307
> edx=0x00000000
> CPU 57:
>    0x8000001e 0x00: eax=0x00000072 ebx=0x00000119 ecx=0x00000307
> edx=0x00000000
> CPU 58:
>    0x8000001e 0x00: eax=0x00000074 ebx=0x0000011a ecx=0x00000307
> edx=0x00000000
> CPU 59:
>    0x8000001e 0x00: eax=0x00000076 ebx=0x0000011b ecx=0x00000307
> edx=0x00000000
> CPU 60:
>    0x8000001e 0x00: eax=0x00000078 ebx=0x0000011c ecx=0x00000307
> edx=0x00000000
> CPU 61:
>    0x8000001e 0x00: eax=0x0000007a ebx=0x0000011d ecx=0x00000307
> edx=0x00000000
> CPU 62:
>    0x8000001e 0x00: eax=0x0000007c ebx=0x0000011e ecx=0x00000307
> edx=0x00000000
> CPU 63:
>    0x8000001e 0x00: eax=0x0000007e ebx=0x0000011f ecx=0x00000307
> edx=0x00000000
> 
> >   
> > >  
> > > >
> > > > I wonder if linux guest actually uses node_id encoded in apic id for
> > > > configuring/checking numa structures, or it just uses whatever ACPI
> > > > SRAT table provided.
> > > >  
> > > >>>> + */
> > > >>>> +static inline void x86_init_topo_ids(X86CPUTopoInfo *topo_info,
> > > >>>> +                                     CpuInstanceProperties props,
> > > >>>> +                                     X86CPUTopoIDs *topo_ids) {
> > > >>>> +    topo_ids->smt_id = props.has_thread_id ? props.thread_id : 0;
> > > >>>> +    topo_ids->core_id = props.has_core_id ? props.core_id : 0;
> > > >>>> +    topo_ids->die_id = props.has_die_id ? props.die_id : 0;
> > > >>>> +    topo_ids->node_id = props.has_node_id ?
> > > >>>> +                        props.node_id % 
> > > >>>> MAX(topo_info->nodes_per_pkg, 1) : 0;
> > > >>>> +    topo_ids->pkg_id = props.has_socket_id ? props.socket_id :
> > > >>>> +0; }
> > > >>>>  /*
> > > >>>>   * Make APIC ID for the CPU 'cpu_index'
> > > >>>>   *
> > > >>>>  
> > > >>  
> > > >  
> > >  
> 
>