Re: An issue with x86 tcg and MMIO

[Jonathan Cameron]

On Thu, 2 Feb 2023 11:39:28 +0000
Peter Maydell <peter.maydell@linaro.org> wrote:

> On Thu, 2 Feb 2023 at 10:56, Richard Henderson
> <richard.henderson@linaro.org> wrote:
> >
> > On 2/1/23 23:39, Jonathan Cameron wrote:  
> > > Not sure - if we can do the handling above then sure we could make that 
> > > change.
> > > I can see there is a path to register the callbacks but I'd kind of 
> > > assumed
> > > ROM meant read only...  
> >
> > I think "romd" means "read mostly".
> >
> > In the case of flash, I believe that a write changes modes (block erase 
> > something
> > something) and the region changes state into MMIO.  But normal state is 
> > read mode where
> > read+execute go through unchallenged.  
> 
> In QEMU a ROMD MemoryRegion (created by memory_region_init_rom_device())
> is a memory region backed by RAM for reads and by callbacks for writes.
> (I think ROMD stands for "ROM device".)
> 
> You can then use memory_region_device_set_romd() to put the ROMD into
> either ROMD mode (the default, reads backed by RAM) or MMIO mode
> (reads backed by MMIO callbacks). Writes are always callbacks regardless.
> This is mainly meant for flash devices, which are usually reads-as-data
> but have a programming mode where you write a command to it and then
> read back command results. It's possible to use it for other tricks too.
> 
> When a ROMD is in ROMD mode then execution from it is as fast as execution
> from any RAM; when it is in MMIO mode then execution from it is as slow
> as execution from any other MMIO-backed MemoryRegion.

Thanks for the info - I don't think ROMD helps us much here as we'd need
to be constantly in the MMIO mode as we need the callbacks for both read
and write.

> 
> Note that AFAIK you can't execute from MMIO at all with KVM (either
> ROMD-in-MMIO mode or a plain old MMIO device).

That may not be a significant problem for CXL emulation - though we should
definitely make that restriction clear and it might slow down some testing.
As far as I know there are no usecases beyond testing of software stacks
and TCG is fine for that.

> 
> You might want to look at whether QEMU's iommu functionality is helpful
> to you -- I'm assuming CXL doesn't do weird stuff on a less-than-page
> granularity, and the iommu APIs will let you do "programmatically decide
> where this address should actually go". The other option involves
> mapping and unmapping MemoryRegions inside a container MR.

Unfortunately it does weird stuff well below a page granularity.
Interleaving is down to 256 bytes.

We discussed the memory region approach when this originally came up.
The issue is that we get an insane number of memory regions to support
even basic interleave setups (many millions) - hence doing the address
decoding via a read and write callbacks at runtime instead. 

Jonathan

> 
> thanks
> -- PMM