Re: [PATCH 02/10] numa: introduce MachineClass::forbid_asymmetrical_numa

[David Gibson]

On Mon, Aug 24, 2020 at 08:45:12AM -0300, Daniel Henrique Barboza wrote:
> 
> 
> On 8/24/20 3:08 AM, David Gibson wrote:
> > On Fri, Aug 21, 2020 at 09:47:47AM -0300, Daniel Henrique Barboza wrote:
> > > 
> > > 
> > > On 8/21/20 5:55 AM, Igor Mammedov wrote:
> > > > On Thu, 20 Aug 2020 12:51:03 -0400
> > > > Eduardo Habkost <ehabkost@redhat.com> wrote:
> > > > 
> > > > > On Thu, Aug 20, 2020 at 02:15:04PM +1000, David Gibson wrote:
> > > > > > On Wed, Aug 19, 2020 at 10:11:28PM -0400, Eduardo Habkost wrote:
> > > > > > > On Thu, Aug 20, 2020 at 11:17:26AM +1000, David Gibson wrote:
> > > > > > > > On Fri, Aug 14, 2020 at 05:54:16PM -0300, Daniel Henrique 
> > > > > > > > Barboza wrote:
> > > > > > > > > The pSeries machine does not support asymmetrical NUMA
> > > > > > > > > configurations.
> > > > > > > > 
> > > > > > > > This seems a bit oddly specific to have as a global machine 
> > > > > > > > class
> > > > > > > > property.
> > > > > > > > 
> > > > > > > > Would it make more sense for machines with specific NUMA 
> > > > > > > > constraints
> > > > > > > > to just verify those during their initialization?
> > > > > > > 
> > > > > > > This would be much simpler.  However, I like the idea of
> > > > > > > representing machine-specific configuration validation rules as
> > > > > > > data that can eventually be exported to management software.
> > > > > > 
> > > > > > Ah, ok, so basically the usual tradeoff between flexibility and
> > > > > > advertisability.
> > > > > > 
> > > > > > So, in that case, I guess the question is whether we envisage "no
> > > > > > assymmetry" as a constraint common enough that it's worth creating 
> > > > > > an
> > > > > > advertisable rule or not.  If we only ever have one user, then we
> > > > > > haven't really done any better than hard coding the constraint in 
> > > > > > the
> > > > > > manageent software.
> > > > > > 
> > > > > > Of course to complicate matters, in the longer term we're looking at
> > > > > > removing that constraint from pseries - but doing so will be 
> > > > > > dependent
> > > > > > on the guest kernel understanding a new format for the NUMA
> > > > > > information in the device tree.  So qemu alone won't have enough
> > > > > > information to tell if such a configuration is possible or not.
> > > > > 
> > > > > Requiring both QEMU (and possibly management software) to be
> > > > > patched again after the guest kernel is fixed sounds undesirable.
> > > > If we drop this restriction, then we don't need to touch QEMU when
> > > > guest kernel is ready.
> > > > 
> > > > Btw, what spapr spec says about the matter?
> > > 
> > > LOPAPR support a somewhat asymmetrical NUMA setup in its current
> > > form,
> > 
> > Huh, I didn't even realize that.  What's the mechanism?
> 
> LOPAPR mentions that a single resource/node can have multiple associativity
> arrays. The idea is to contemplate the situations where the node has
> more than one connection with the board.
> 
> I say "somewhat" because, right after mentioning that, the spec also says that
> the OS should consider that the distance between two nodes must always be
> the shortest one of all available arrays. I'll copy/paste the except here
> (end of section 15.2, "Numa Resource Associativity":

Ah.  I didn't think that's what "asymmetric NUMA" meant... but come to
think of it, I'm not very sure about that.

> -----
> 
> The reason that the “ibm,associativity” property may contain multiple 
> associativity
> lists is that a resource may be multiply connected into the platform. This 
> resource
> then has a different associativity characteristics relative to its multiple 
> connections.
> To determine the associativity between any two resources, the OS scans down 
> the two
> resources associativity lists in all pair wise combinations counting how many 
> domains
> are the same until the first domain where the two list do not agree. The 
> highest such
> count is the associativity between the two resources.
> 
> ----
> 
> 
> DHB
> 
> 
> > 
> > > but
> > > the Linux kernel doesn't support it. The effort to implement it in the 
> > > current
> > > spapr machine code, given that Linux wouldn't mind it, is not worth it. 
> > > This
> > > is why I chose to invalidate it for pseries.
> > 
> > Igor,
> > 
> > It's kind of difficult to answer that question - PAPR doesn't
> > specifically describe limitations, it's just that the representation
> > it uses is inherently limited.  Instead of the obvious, simple and
> > pretty much universal method (used in the generic kernel and qemu) of
> > having a matrix of distance between all the nodes, it instead
> > describes the hierarchy of components that give rise to the different
> > distances.
> > 
> > So, for each NUMA relevant object (cpu, memory block, host bridge,
> > etc.) there is a vector of IDs.  Each number in the vector gives one
> > level of the objects location in the heirarchy.
> > 
> > So, for example the first number might be the physical chip/socket.
> > the second one which group of cores & memory interfaces sharing an Ln
> > cache, the third one the specific core number.  So to work out how far
> > objects are from each other you essentially look at how long a prefix
> > of their vector they share, which tells you how far above in the
> > hierarchy you have to go to reach it.
> > 
> > There's a bunch of complicating details, but that's the gist of it.
> > 
> > > > > Perhaps a warning would be better in this case?
> > > > > 
> > > > > In either case, it sounds like this won't be a common constraint
> > > > > and I now agree with your original suggestion of doing this in
> > > > > machine initialization code.
> > > > Agreed, if it goes to spapr specific machine code I will not object 
> > > > much.
> > > > (it will burden just spapr maintainers, so it's about convincing
> > > > David in the end)
> > > 
> > > I believe he's ok with it given that he suggested it in his first reply.
> > > 
> > > I'll move this verification to spapr machine_init in the next version.
> > > 
> > > 
> > > 
> > > Thanks,
> > > 
> > > DHB
> > > 
> > > 
> > > > 
> > > 
> > 
> 

-- 
David Gibson                    | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au  | minimalist, thank you.  NOT _the_ _other_
                                | _way_ _around_!
http://www.ozlabs.org/~dgibson

signature.asc
Description: PGP signature