Re: Help: How do I make a machine with 2 separate ARM SoC's?

[Peter Delevoryas]

> On May 30, 2022, at 11:15 AM, Cédric Le Goater <clg@kaod.org> wrote:
> 
> On 5/30/22 18:53, Peter Maydell wrote:
>> On Thu, 26 May 2022 at 23:14, Peter Delevoryas <pdel@fb.com> wrote:
>>> Hey QEMU developers,
>>> 
>>> Cedric mentioned here[1] that QEMU can support emulating a
>>> more complete board, e.g. a machine with an AST2600 *and* an AST1030.
>> This is true, as long as all the CPUs are the same
>> architecture family, e.g. all Arm CPUs. (Mixing A- and
>> R- or A- and M-profile is OK, they just all have to be
>> available in the same qemu-system-whatever binary.)
>>> I read through the memory API docs[2] and it mostly makes sense to me,
>>> but what I don’t understand is, what does system_memory represent?
>> So, system_memory is something of a legacy from when QEMU was
>> much older. Before the MemoryRegion and AddressSpace APIs were
>> added to QEMU, everything that could initiate a memory transaction
>> (CPUs, DMA-capable devices, etc) always saw the same view of
>> memory. The functions to do memory accesses just operated on
>> that view implicitly. (We still have some of them, for instance
>> cpu_physical_memory_read() and cpu_physical_memory_write().) The
>> MemoryRegion/AddressSpace APIs are much more flexible and allow
>> different memory transaction initiators to see different views, as
>> real hardware does. But for backwards compatibility we still have
>> the old assumes-one-view APIs. The view those APIs use is the
>> "system memory". We also have some device models which have been
>> converted to use an AddressSpace to do their DMA operations, but
>> which assume they want to use address_space_memory (which is the AS
>> corresponding to the system_memory MR) instead of taking a
>> MemoryRegion as a QOM pointer and creating an AddressSpace for it.
>> In the modern view of the world, you can build up a system with
>> a set of MemoryRegions. Typically you can start with an empty
>> container, and the board code fills it with board-level devices,
>> then passes it to the SoC code, which fills it with SoC devices,
>> and passes it again to the CPU object, which creates an AddressSpace
>> so it can initiate transactions into it. By making that initial
>> "empty container" be the global system_memory MemoryRegion, this
>> makes the legacy APIs and devices that still use it basically work.
>>> Or, what should the layout be for a situation like I’m interested in,
>>> where you have an AST2600 and an AST1030 (and actually, maybe even
>>> an x86 CPU too? idk if that would be possible).
>> Cross-architecture heterogenous board models can't be done today:
>> the qemu-system-foo binaries compile-time build in some assumptions
>> about specifics of the guest architecture. (This is something it would
>> be nice to fix, but the amount of work is pretty big and hairy, and
>> thus far nobody's had a pressing enough need for it to try to tackle it.)
>>> I need to make sure each SoC runs in a different address space, right?
>>> But, how do I actually do that? Do I model it as two containers inside
>>> the large system_memory container, or as two different containers
>>> that get swapped in for system_memory when executing their associated
>>> CPU?
>> The best way to think about QEMU's AddressSpace type is that it is
>> the interface you use to initiate memory transactions. You create
>> one from a MemoryRegion. When SoC and board code is building up its
>> view of the world, what it is really creating and passing around is
>> a hierarchy of MemoryRegions. It's only when the SoC code hands a
>> MemoryRegion to a CPU or a DMA-capable device that that device says
>> "I will need to make transactions to this, let me create the
>> corresponding AddressSpace".
>>> I was having trouble figuring out what the Xilinx boards are actually
>>> doing in this case. Does each CPU share peripherals, or are the
>>> A + R cpu’s actually in separate address spaces? I’m very confused lol.
>> xlnx-versal-virt is a virtual board, so ignore that one: it's
>> probably more confusing than helpful. The xlnx-zcu102 board
>> uses the xlnx-zynqmp SoC, and that SoC has both R and A profile
>> CPUs in it, but they both see basically the same view of the
>> world because they're in the same SoC.

I see, I started to suspect as much.

>> Another device that does some moderately complicated things with
>> MemoryRegions is the hw/arm/armsse.c SoC, which has several CPUs
>> and has some per-CPU devices.
>> 
>> I think we have not thus far had a model of a board where different
>> CPUs see radically different things (only ones where they can see
>> minor differences), so you'll probably run into places where the
>> APIs are a bit clunky (and we can perhaps have a go at making
>> them a bit less so). What I would do is make the system_memory
>> container be used by whatever is the "main" application processor
>> SoC in your board. 
> 
> I think Peter D. wants to emulate a machine with a BMC board (ast2600)
> and a SCP-like SoC (ast1030) running zephir. Correct me if I am wrong.
> That's a first step.

That’s right, an AST2600 running Linux (OpenBMC) and an AST1030 running
Zephyr (OpenBIC).

> 
>> If the two SoCs really see absolutely different
>> worlds with no shared devices at all, then you'll want to create
>> a new empty container for the second SoC. 
> 
> yes.

I see, one container for each SoC then.

> 
>> If they do have some
>> board-level shared devices, then you'll want to do something a little
>> more complicated with aliases.
> 
> The first device would be a shared I2C bus to communicate. I haven't
> looked deeply how complex it would we to plug a slave model of the
> first SoC on a bus of the second SoC.

This is the most important thing for me: our board designs only
use I2C and I3C to connect the AST2600 and AST1030, so this is the
one and only thing I need the two to share.

Well, maybe there’s some GPIO’s to connect. But I2C is the biggest thing.

I think right now, when we construct the Aspeed SoC’s I2C bus
controllers, we just create a new I2CBus for each controller,
in realize(). My naive guess was to refactor each controller to receive
an I2CBus from an external source, so that we can construct a board
where 1 or 2 of the buses are shared between SoC’s.

> 
>> If you find the SoC device models you're using hardcode use of
>> system_memory or address_space_memory you should treat those as
>> bugs to be fixed. 
> 
> There are a few get_system_memory() left in aspeed SoC (UART, SRAM)
> that could be fixed easily and upstream. The rest should be clean
> enough.

Ah I see, yes I was getting stuck figuring out if I needed to refactor
all of that or if I was doing something wrong. That makes sense now.

> 
>> Other loose ends (like monitor commands that
>> assume the system address space) can be ignored: having those
>> operate on the 'application processor' SoC is fine, I think.
>> Overall, this is definitely doable but will involve a fair
>> about of slogging through territory where nobody has yet
>> broken a trail for you :-)
> 
> 
> I am around. We can start building the machine on a GH branch and
> feed mainline with updates while it's getting ready.
> 
> 
> Thanks for the feedback.

Yes thanks so much Peter for the very detailed and helpful answer,
that clears up so much to me, and thanks Cedric for responding too.

I’m very interested in experimenting more with this, I’ll try to
get something up and running and come back with patches to upstream.

It’s too bad we can’t do multiple architectures in the same board,
we also have boards that have i2c between ARM and a riscv64 machine.

Since we mostly just use i2c between each processor, I made a generic
i2c-over-a-socket thing that kinda-sorta works, and we can connect
two separate QEMU instances (one ARM, one riscv64) that way, but
I’ve been struggling to get that working well with the AST2600 and
AST1030 board. I would also prefer to just run them both in
the same QEMU instance if possible.

Also, I’d like to test 1 AST2600 and 1 AST1030 at first, but my
primary use case is actually 1 AST2600 and 5 AST1030’s.

I’m also interested in making a board with 3 different AST2600’s too,
although that one includes USB between them I think, and I haven’t
even started looking into that.

Anyways, thanks again for the help!!!

Peter

> 
> C.