Re: [PATCH v19 2/8] numa: Extend CLI to provide memory latency and bandwidth information

[Tao Xu]

On 11/28/2019 7:50 PM, Markus Armbruster wrote:
> Tao Xu <address@hidden> writes:
>
> > From: Liu Jingqi <address@hidden>
> >
> > Add -numa hmat-lb option to provide System Locality Latency and
> > Bandwidth Information. These memory attributes help to build
> > System Locality Latency and Bandwidth Information Structure(s)
> > in ACPI Heterogeneous Memory Attribute Table (HMAT). Before using
> > hmat-lb option, enable HMAT with -machine hmat=on.
> >
> > Signed-off-by: Liu Jingqi <address@hidden>
> > Signed-off-by: Tao Xu <address@hidden>
> > ---
> [...]
> > diff --git a/qapi/machine.json b/qapi/machine.json
> > index 27d0e37534..cf9851fcd1 100644
> > --- a/qapi/machine.json
> > +++ b/qapi/machine.json
> > @@ -426,10 +426,12 @@
> >   #
> >   # @cpu: property based CPU(s) to node mapping (Since: 2.10)
> >   #
> > +# @hmat-lb: memory latency and bandwidth information (Since: 5.0)
> > +#
> >   # Since: 2.1
> >   ##
> >   { 'enum': 'NumaOptionsType',
> > -  'data': [ 'node', 'dist', 'cpu' ] }
> > +  'data': [ 'node', 'dist', 'cpu', 'hmat-lb' ] }
> >   
> >   ##
> >   # @NumaOptions:
> > @@ -444,7 +446,8 @@
> >     'data': {
> >       'node': 'NumaNodeOptions',
> >       'dist': 'NumaDistOptions',
> > -    'cpu': 'NumaCpuOptions' }}
> > +    'cpu': 'NumaCpuOptions',
> > +    'hmat-lb': 'NumaHmatLBOptions' }}
> >   
> >   ##
> >   # @NumaNodeOptions:
> > @@ -557,6 +560,92 @@
> >      'base': 'CpuInstanceProperties',
> >      'data' : {} }
> >   
> > +##
> > +# @HmatLBMemoryHierarchy:
> > +#
> > +# The memory hierarchy in the System Locality Latency and Bandwidth
> > +# Information Structure of HMAT (Heterogeneous Memory Attribute Table)
> > +#
> > +# For more information about @HmatLBMemoryHierarchy see chapter
>
> @HmatLBMemoryHierarchy, see
>
> > +# 5.2.27.4: Table 5-146: Field "Flags" of ACPI 6.3 spec.
> > +#
> > +# @memory: the structure represents the memory performance
> > +#
> > +# @first-level: first level of memory side cache
> > +#
> > +# @second-level: second level of memory side cache
> > +#
> > +# @third-level: third level of memory side cache
> > +#
> > +# Since: 5.0
> > +##
> > +{ 'enum': 'HmatLBMemoryHierarchy',
> > +  'data': [ 'memory', 'first-level', 'second-level', 'third-level' ] }
> > +
> > +##
> > +# @HmatLBDataType:
> > +#
> > +# Data type in the System Locality Latency and Bandwidth
> > +# Information Structure of HMAT (Heterogeneous Memory Attribute Table)
> > +#
> > +# For more information about @HmatLBDataType see chapter
>
> @HmatLBDataType, see
>
> > +# 5.2.27.4: Table 5-146:  Field "Data Type" of ACPI 6.3 spec.
> > +#
> > +# @access-latency: access latency (nanoseconds)
> > +#
> > +# @read-latency: read latency (nanoseconds)
> > +#
> > +# @write-latency: write latency (nanoseconds)
> > +#
> > +# @access-bandwidth: access bandwidth (Bytes per second)
> > +#
> > +# @read-bandwidth: read bandwidth (Bytes per second)
> > +#
> > +# @write-bandwidth: write bandwidth (Bytes per second)
> > +#
> > +# Since: 5.0
> > +##
> > +{ 'enum': 'HmatLBDataType',
> > +  'data': [ 'access-latency', 'read-latency', 'write-latency',
> > +            'access-bandwidth', 'read-bandwidth', 'write-bandwidth' ] }
> > +
> > +##
> > +# @NumaHmatLBOptions:
> > +#
> > +# Set the system locality latency and bandwidth information
> > +# between Initiator and Target proximity Domains.
> > +#
> > +# For more information about @NumaHmatLBOptions see chapter
>
> @NumaHmatLBOptions, see
>
> > +# 5.2.27.4: Table 5-146 of ACPI 6.3 spec.
> > +#
> > +# @initiator: the Initiator Proximity Domain.
> > +#
> > +# @target: the Target Proximity Domain.
> > +#
> > +# @hierarchy: the Memory Hierarchy. Indicates the performance
> > +#             of memory or side cache.
> > +#
> > +# @data-type: presents the type of data, access/read/write
> > +#             latency or hit latency.
> > +#
> > +# @latency: the value of latency from @initiator to @target
> > +#           proximity domain, the latency unit is "ns(nanosecond)".
> > +#
> > +# @bandwidth: the value of bandwidth between @initiator and @target
> > +#             proximity domain, the bandwidth unit is
> > +#             "Bytes per second".
> > +#
> > +# Since: 5.0
> > +##
> > +{ 'struct': 'NumaHmatLBOptions',
> > +    'data': {
> > +    'initiator': 'uint16',
> > +    'target': 'uint16',
> > +    'hierarchy': 'HmatLBMemoryHierarchy',
> > +    'data-type': 'HmatLBDataType',
> > +    '*latency': 'uint64',
> > +    '*bandwidth': 'size' }}
> > +
> >   ##
> >   # @HostMemPolicy:
> >   #
> > diff --git a/qemu-options.hx b/qemu-options.hx
> > index 63f6b33322..23303fc7d7 100644
> > --- a/qemu-options.hx
> > +++ b/qemu-options.hx
> > @@ -168,16 +168,19 @@ DEF("numa", HAS_ARG, QEMU_OPTION_numa,
> >       "-numa 
> > node[,mem=size][,cpus=firstcpu[-lastcpu]][,nodeid=node][,initiator=node]\n"
> >       "-numa 
> > node[,memdev=id][,cpus=firstcpu[-lastcpu]][,nodeid=node][,initiator=node]\n"
> >       "-numa dist,src=source,dst=destination,val=distance\n"
> > -    "-numa cpu,node-id=node[,socket-id=x][,core-id=y][,thread-id=z]\n",
> > +    "-numa cpu,node-id=node[,socket-id=x][,core-id=y][,thread-id=z]\n"
> > +    "-numa 
> > hmat-lb,initiator=node,target=node,hierarchy=memory|first-level|second-level|third-level,data-type=access-latency|read-latency|write-latency[,latency=lat][,bandwidth=bw]\n",
> >       QEMU_ARCH_ALL)
> >   STEXI
> >   @item -numa 
> > node[,mem=@var{size}][,cpus=@var{firstcpu}[-@var{lastcpu}]][,nodeid=@var{node}][,initiator=@var{initiator}]
> >   @itemx -numa 
> > node[,memdev=@var{id}][,cpus=@var{firstcpu}[-@var{lastcpu}]][,nodeid=@var{node}][,initiator=@var{initiator}]
> >   @itemx -numa dist,src=@var{source},dst=@var{destination},val=@var{distance}
> >   @itemx -numa 
> > cpu,node-id=@var{node}[,socket-id=@var{x}][,core-id=@var{y}][,thread-id=@var{z}]
> > +@itemx -numa 
> > hmat-lb,initiator=@var{node},target=@var{node},hierarchy=@var{hierarchy},data-type=@var{tpye}[,latency=@var{lat}][,bandwidth=@var{bw}]
> >   @findex -numa
> >   Define a NUMA node and assign RAM and VCPUs to it.
> >   Set the NUMA distance from a source node to a destination node.
> > +Set the ACPI Heterogeneous Memory Attributes for the given nodes.
> >   
> >   Legacy VCPU assignment uses @samp{cpus} option where
> >   @var{firstcpu} and @var{lastcpu} are CPU indexes. Each
> > @@ -256,6 +259,49 @@ specified resources, it just assigns existing resources to 
> > NUMA
> >   nodes. This means that one still has to use the @option{-m},
> >   @option{-smp} options to allocate RAM and VCPUs respectively.
> >   
> > +Use @samp{hmat-lb} to set System Locality Latency and Bandwidth Information
> > +between initiator and target NUMA nodes in ACPI Heterogeneous Attribute Memory 
> > Table (HMAT).
> > +Initiator NUMA node can create memory requests, usually it has one or more 
> > processors.
> > +Target NUMA node contains addressable memory.
> > +
> > +In @samp{hmat-lb} option, @var{node} are NUMA node IDs. @var{hierarchy} is the 
> > memory
> > +hierarchy of the target NUMA node: if @var{hierarchy} is 'memory', the 
> > structure
> > +represents the memory performance; if @var{hierarchy} is 
> > 'first-level|second-level|third-level',
> > +this structure represents aggregated performance of memory side caches for 
> > each domain.
> > +@var{type} of 'data-type' is type of data represented by this structure 
> > instance:
> > +if 'hierarchy' is 'memory', 'data-type' is 'access|read|write' latency or 
> > 'access|read|write'
> > +bandwidth of the target memory; if 'hierarchy' is 
> > 'first-level|second-level|third-level',
> > +'data-type' is 'access|read|write' hit latency or 'access|read|write' hit 
> > bandwidth of the
> > +target memory side cache.
> > +
> > +@var{lat} is latency value in nanoseconds. @var{bw} is bandwidth value,
> > +the possible value and units are NUM[M|G|T], mean that the bandwidth value are
> > +NUM byte per second (or MB/s, GB/s or TB/s depending on used suffix).
> > +And if input bandwidth value without any unit, the unit will be byte per 
> > second.
>
> This sentence feels redundant to meOK, I will remove this sentence:

"And if input bandwidth value without any unit, the unit will be byte 
per second."
> > +Note that if latency or bandwidth value is 0, means the corresponding latency 
> > or
> > +bandwidth information is not provided.
> > +
> > +For example, the following options describe 2 NUMA nodes. Node 0 has 2 cpus and
> > +a ram, node 1 has only a ram. The processors in node 0 access memory in node
> > +0 with access-latency 5 nanoseconds, access-bandwidth is 200 MB/s;
> > +The processors in NUMA node 0 access memory in NUMA node 1 with access-latency 
> > 10
> > +nanoseconds, access-bandwidth is 100 MB/s.
> > +@example
> > +-machine hmat=on \
> > +-m 2G \
> > +-object memory-backend-ram,size=1G,id=m0 \
> > +-object memory-backend-ram,size=1G,id=m1 \
> > +-smp 2 \
> > +-numa node,nodeid=0,memdev=m0 \
> > +-numa node,nodeid=1,memdev=m1,initiator=0 \
> > +-numa cpu,node-id=0,socket-id=0 \
> > +-numa cpu,node-id=0,socket-id=1 \
> > +-numa 
> > hmat-lb,initiator=0,target=0,hierarchy=memory,data-type=access-latency,latency=5
> >  \
> > +-numa 
> > hmat-lb,initiator=0,target=0,hierarchy=memory,data-type=access-bandwidth,bandwidth=200M
> >  \
> > +-numa 
> > hmat-lb,initiator=0,target=1,hierarchy=memory,data-type=access-latency,latency=10
> >  \
> > +-numa 
> > hmat-lb,initiator=0,target=1,hierarchy=memory,data-type=access-bandwidth,bandwidth=100M
> > +@end example
> > +
> >   ETEXI
> >   
> >   DEF("add-fd", HAS_ARG, QEMU_OPTION_add_fd,