Re: CXL 2.0 memory pooling emulation

[Jonathan Cameron]

On Thu, 16 Feb 2023 15:52:31 -0500
Gregory Price <gregory.price@memverge.com> wrote:

> On Thu, Feb 16, 2023 at 06:00:57PM +0000, Jonathan Cameron wrote:
> > On Wed, 15 Feb 2023 04:10:20 -0500
> > Gregory Price <gregory.price@memverge.com> wrote:
> >   
> > > On Wed, Feb 15, 2023 at 03:18:54PM +0000, Jonathan Cameron via wrote:  
> > > > On Wed, 8 Feb 2023 16:28:44 -0600
> > > > zhiting zhu <zhitingz@cs.utexas.edu> wrote:
> > > >   
> > > > 1) Emulate an Multi Headed Device.
> > > >    Initially connect two heads to different host bridges on a single 
> > > > QEMU
> > > >    machine.  That lets us test most of the code flows without needing
> > > >    to handle tests that involve multiple machines.
> > > >    Later, we could add a means to connect between two instances of 
> > > > QEMU.    
> > > 
> > > Hackiest way to do this is to connect the same memory backend to two
> > > type-3 devices, with the obvious caveat that the device state will not
> > > be consistent between views.
> > > 
> > > But we could, for example, just put the relevant shared state into an
> > > optional shared memory area instead of a normally allocated region.
> > > 
> > > i can imagine this looking something like
> > > 
> > > memory-backend-file,id=mem0,mem-path=/tmp/mem0,size=4G,share=true
> > > cxl-type3,bus=rp0,volatile-memdev=mem0,id=cxl-mem0,shm_token=mytoken
> > > 
> > > then you can have multiple qemu instances hook their relevant devices up
> > > to a a backend that points to the same file, and instantiate their
> > > shared state in the region shmget(mytoken).  
> > 
> > That's not pretty.  For local instance I was thinking a primary device
> > which also has the FM-API tunneled access via mailbox, and secondary devices
> > that don't.  That would also apply to remote. The secondary device would
> > then just receive some control commands on what to expose up to it's host.
> > Not sure what convention on how to do that is in QEMU. Maybe a socket
> > interface like is done for swtpm? With some ordering constraints on startup.
> >   
> 
> I agree, it's certainly "not pretty".
> 
> I'd go so far as to call the baby ugly :].  Like i said: "The Hackiest way"
> 
> My understanding from looking around at some road shows is that some
> of these early multi-headed devices are basically just SLD's with multiple
> heads. Most of these devices had to be developed well before DCD's and
> therefore the FM-API were placed in the spec, and we haven't seen or
> heard of any of these early devices having any form of switch yet.
> 
> I don't see how this type of device is feasible unless it's either statically
> provisioned (change firmware settings from bios on reboot) or implements
> custom firmware commands to implement some form of exclusivity controls over
> memory regions.
> 
> The former makes it not really a useful pooling device, so I'm sorta guessing
> we'll see most of these early devices implement custom commands.
> 
> I'm just not sure these early MHD's are going to have any real form of
> FM-API, but it would still be nice to emulate them.
> 
Makes sense.  I'd be fine with adding any necessary hooks to allow that
in the QEMU emulation, but probably not upstreaming the custom stuff.

Jonathan

> ~Gregory