Re: [RFC 00/52] Introduce hybrid CPU topology

[Zhao Liu]

On Mon, Feb 13, 2023 at 01:38:28PM +0000, Daniel P. Berrangé wrote:
> Date: Mon, 13 Feb 2023 13:38:28 +0000
> From: "Daniel P. Berrangé" <berrange@redhat.com>
> Subject: Re: [RFC 00/52] Introduce hybrid CPU topology
> 
> On Mon, Feb 13, 2023 at 05:49:43PM +0800, Zhao Liu wrote:
> > From: Zhao Liu <zhao1.liu@intel.com>
> > ## 3.3. "-hybrid" command
> > 
> > For hybrid cpu topology configuration, the original "-smp" lack of
> > flexibility to expand, and unables to customize different cores.
> > 
> > So we introduce "-hybrid" command and implement it as the multi-
> > line command. The multi-line command format is more complex than the
> > single-line one, but it can bring stronger scalability and
> > intuitiveness. In the future, it can also be easily extended to more
> > heterogeneous topology levels.
> > 
> > "-hybrid" command is as follows:
> > 
> > -hybrid socket,sockets=n
> > -hybrid die,dies=n
> > -hybrid cluster,clusters=n
> > -hybrid core,cores=n,type=core_type[,threads=threads]
> >         [,clusterid=cluster]
> > 
> > Here, we first define the corresponding qapi options for these 4
> > topology levels: core, cluster, die and socket.
> > 
> > We doesn't need a thread level since thread doesn't have different
> > type.
> > 
> > For example:
> > 
> > -hybrid socket,sockets=1
> > -hybrid die,dies=1
> > -hybrid cluster,clusters=4
> > -hybrid core,cores=1,coretype="core",threads=2,clusterid=0-2
> > -hybrid core,cores=4,coretype="atom",threads=1
> > 
> > Here we can build a hybrid cpu topology, which has 1 socket, 1 die per
> > socket, 4 clusters per die. And in each die, every clusters has 4 "atom"
> > core with 1 threads, and cluster0, cluster1 and cluster2 have 1 "core"
> > cores with 2 threads.
> 
> How will this interact with the -cpu  parameter. Presumably we now
> need two distinct -cpu parameters, one for the 'core' CPU model and
> one for the 'atom' CPU model ?

(About this explanation for core type, also CC Philippe again)

For intel's hybrid, the core type is not a specific cpu model, and it
refers CPUID.1AH, sounds like a flag to distinguish between "core" and
"atom". For example, Alder Lake (not defined in QEMU yet) is a CPU model,
the Alder Lake CPU can be "core" or "atom".

Although the real "core" and "atom" have different microarchitectures,
they belong to the same CPU model and have the same ISA, and the
difference that the software layer can recognize is whether the encode
in CPUID.1AH is "core" type or "atom" type.

I think this is also the difference between intel hybrid and other
heterogeneous architectures. IIUC the heterogeneity of arm contains
different CPU models, while intel hybrid only marks the cpu as "core"
or "atom" in the same CPU model.

I think this difference between intel hybrid and other heterogeneous can
be distinguished by different handling methods of mc->core_type() which
can be defined by other arch to check "coretype" field [1].

For x86, mc->core_type() asks "coretype" as "core" or "atom" and treat
it as a hybrid "flag". But for other arch, who contains different CPU
models in heterogeneous (maybe arm), its mc->coretype asks for the
correct cpu model.

In this sense, x86 does not need a different "-cpu" to correspond to
core/atom, and even "-cpu" can be set to "host" (ie, -cpu host). But
other arch, who contains different CPU models in heterogeneous, needs
to support distinct -cpu parameters.

Do you think it is appropriate to use arch-specific mc->coretype to
handle different requirements for CPU model/core type?

[1]: [RFC 41/52] machine: Introduce core_type() hook,
     https://lists.gnu.org/archive/html/qemu-devel/2023-02/msg03244.html

> 
> > Please note this example is not an actual machine topology, but it shows
> > the powerful flexibility of "hybrid" command.
> 
> IIUC the functionality offered by -hybrid should be a superset
> of the -smp functionality. IOW, -smp ought to be possible to
> re-implement -smp as an alias for -hybrid, such that internally
> code only ever has to deal with the modern approach.

Yes, I agree.

> Having to
> keep support for both -smp and -hybrid throughout the code is
> undesirable IMHO. Keeping the compat at the CLI parsing level
> limits the burden.
>


"-hybrid" is introduced as a completely different implementation from
"-smp", for these 2 reasons:

1. Single-line commands can be very complex, and it is more difficult
to identify the relationship between different levels.

2. smp is implemented as the property of machine [2] and I don't have a
good way to define "hybrid" also as a property since I need to use
opts_visitor for multi-line commands instead of input_visitor.

[2]: fe68090: machine: add smp compound property.

> 
> As a more general thought, rather than introducing a new top level
> command line argument -hybrid, I'm thinking we should possibly just
> define this all using QOM and thus re-use the existing -object
> argument. 
> 
> I'm also finding the above example command lines quite difficult
> to understand, as there is alot of implicit linkage and expansion
> between the different levels. With devices we're much more
> explicitly with the parent/child relationships, and have to
> express everything with no automatic expansion, linking it all
> together via the id=/bus= properties.  This is quite a bit more
> verbose, but it is also very effective at letting us express
> arbitrarily complex relationships.
> 
> I think it would be worth exploring that approach for the CPU
> topology expression too.
> 
> If we followed the more explicit device approach to modelling
> then instead of:
> 
>  -cpu core,...
>  -cpu atom,...
>  -hybrid socket,sockets=1
>  -hybrid die,dies=1
>  -hybrid cluster,clusters=4
>  -hybrid core,cores=1,coretype="core",threads=2,clusterid=0-2
>  -hybrid core,cores=4,coretype="atom",threads=1
> 
> we would end up with something like
> 
>   -object cpu-socket,id=sock0
>   -object cpu-die,id=die0,parent=sock0
>   -object cpu-cluster,id=cluster0,parent=die0
>   -object cpu-cluster,id=cluster1,parent=die0
>   -object cpu-cluster,id=cluster2,parent=die0
>   -object cpu-cluster,id=cluster3,parent=die0
>   -object x86-cpu-model-atom,id=cpu0,parent=cluster0
>   -object x86-cpu-model-atom,id=cpu1,parent=cluster0
>   -object x86-cpu-model-atom,id=cpu2,parent=cluster0
>   -object x86-cpu-model-atom,id=cpu3,parent=cluster0
>   -object x86-cpu-model-core,id=cpu4,parent=cluster0,threads=2
>   -object x86-cpu-model-atom,id=cpu5,parent=cluster1
>   -object x86-cpu-model-atom,id=cpu6,parent=cluster1
>   -object x86-cpu-model-atom,id=cpu7,parent=cluster1
>   -object x86-cpu-model-atom,id=cpu8,parent=cluster1
>   -object x86-cpu-model-core,id=cpu9,parent=cluster1,threads=2
>   -object x86-cpu-model-atom,id=cpu10,parent=cluster2
>   -object x86-cpu-model-atom,id=cpu11,parent=cluster2
>   -object x86-cpu-model-atom,id=cpu12,parent=cluster2
>   -object x86-cpu-model-atom,id=cpu13,parent=cluster2
>   -object x86-cpu-model-core,id=cpu14,parent=cluster2,threads=2
>   -object x86-cpu-model-atom,id=cpu15,parent=cluster3
>   -object x86-cpu-model-atom,id=cpu16,parent=cluster3
>   -object x86-cpu-model-atom,id=cpu17,parent=cluster3
>   -object x86-cpu-model-atom,id=cpu18,parent=cluster3
>   -object x86-cpu-model-core,id=cpu19,parent=cluster3,threads=2


Given that x86 hybrid currently has the same cpu model and "core"/"atom"
is not CPU model, so the -object still needs a coretype fields to specify
"core" or "atom", right?

Given an example of "-cpu host":

-cpu host,...
-hybrid socket,sockets=1
-hybrid die,dies=1
-hybrid cluster,clusters=4
-hybrid core,cores=1,coretype="core",threads=2,clusterid=0-2
-hybrid core,cores=4,coretype="atom",threads=1


It occurred to me that there may be two conversion formats for qom.

The one with "coretype" field like:

-object x86-cpu-model-host,coretype="atom",id=cpu0,parent=cluster0
-object x86-cpu-model-host,coretype="core",id=cpu1,parent=cluster0

or another format like "x86-cpu-model-[real model name]-[core type]":

-object x86-cpu-model-host-atom,id=cpu0,parent=cluster0
-object x86-cpu-model-host-core,id=cpu1,parent=cluster0

Can qom support the parsing of the second naming format?

At the same time, I understand that for other architectures, there is
no need to consider the issue of core type, just specify different
heterogeneous CPU models.

> 
> The really obvious downside is that it is much more verbose.
> 
> This example only has 20 CPUs. For a VM with say 1000 CPUs
> this will be very big, but that doesn't neccesarily make it
> wrong.
> 
> On the flipside
> 
>  * It is really clear exactly how many CPUs I've added
> 
>  * The relationship between the topology levels is clear
> 
>  * Every CPU has a unique ID given that can be used in
>    later QMP commands
> 
>  * Whether or not 'threads' are permitted is now a property
>    of the specific CPU model implementation, not the global
>    config. IOW we can express that some CPU models allowing
>    for threads, and some don't.

Will this way break the smp property of machine (the above mentioned [2])?

If so, IIUC we still need keep smp code to support this smp format:

"-machine smp.{cores,threads,cpu,...}",

but we can mark this smp format as deprecated and at some point completely
remove the smp code and replace it with the qom implementation, right?

> 
>  * The -cpu arg is also obsoleted, replaced by the
>    -object x86-cpu-model-core. This might facilitate the
>    modelling of machines with CPUs from different architectures.
> 
> 
> We could potentially compress the leaf node level by expressing
> how many instances of an object we want. it we want. ie, define
> a more convenient shorthand syntax to creating many instances of
> an object. so eg
> 
>   -object-set $TYPE,$PROPS,idbase=foo,count=4
> 
> would be functionally identical to
> 
>   -object $TYPE,$PROPS,id=foo.0
>   -object $TYPE,$PROPS,id=foo.1
>   -object $TYPE,$PROPS,id=foo.2
>   -object $TYPE,$PROPS,id=foo.3
> 
> QEMU just expands it and creates all the objects internally.
> 
> So the huge example I have above for 20 cpus would become much
> shorter: e.g.
> 
>   -object cpu-socket,id=sock0
>   -object cpu-die,id=die0,parent=sock0
>   -object cpu-cluster,id=cluster0,parent=die0
>   -object cpu-cluster,id=cluster1,parent=die0
>   -object cpu-cluster,id=cluster2,parent=die0
>   -object cpu-cluster,id=cluster3,parent=die0
>   -object-set x86-cpu-core-atom,idbase=cpu0,parent=cluster0,count=4
>   -object-set x86-cpu-core-core,id=cpu1,parent=cluster0,threads=2,count=1
>   -object-set x86-cpu-core-atom,idbase=cpu2,parent=cluster1,count=4
>   -object-set x86-cpu-core-core,id=cpu3,parent=cluster1,threads=2,count=1
>   -object-set x86-cpu-core-atom,idbase=cpu4,parent=cluster2,count=4
>   -object-set x86-cpu-core-core,id=cpu5,parent=cluster2,threads=2,count=1
>   -object-set x86-cpu-core-atom,idbase=cpu6,parent=cluster3,count=4
>   -object-set x86-cpu-core-core,id=cpu7,parent=cluster3,threads=2,count=1

What's difference between "id" and "idbase"?

> 
> IOW, the size of the CLI config only depends on the number of elements
> in the hierarchy, and is independant of the number of leaf CPU cores.
> 
> Obviously in describing all of the above, I've ignored any complexity
> of dealing with our existing code implementation and pain of getting
> it converted to the new model.

Thanks and sounds cool! I think I need some time to study this way.


Zhao

> 
> With regards,
> Daniel
> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
>