RE: Building QEMU as a shared library

[Amir Gonnen]

Hi Stefan,

Easier/faster synchronization is just one side of the issue.
It's much easier to debug a single process or profile it, for example. It's 
also easier to deploy and maintain a single process.

For now, the only "global state" problem I had to fix was Glib global context.
I didn't see an issue with posix signals. Any other global state you think I 
should take care of?
Of course, I only tested it for my own simple use case (qemu-system-nios2 + 
qemu-system-x86_64 with no GUI)

As a first step , I intend to send a patch that removes the Glib context global 
state.

Thanks,
Amir

-----Original Message-----
From: Stefan Hajnoczi <stefanha@redhat.com>
Sent: Wednesday, December 15, 2021 11:46 AM
To: Amir Gonnen <amir.gonnen@neuroblade.ai>
Cc: qemu-devel@nongnu.org; peter.maydell@linaro.org; f4bug@amsat.org
Subject: Re: Building QEMU as a shared library

On Wed, Dec 15, 2021 at 08:18:53AM +0000, Amir Gonnen wrote:
> Before sending a patch, I would like to check if it's of interest to the 
> community.
>
> My goal is to simulate a mixed architecture system.
> Today QEMU strongly assumes that the simulated system is a *single 
> architecture*.
> Changing this assumption and supporting mixed architecture in QEMU
> proved to be non-trivial and may require significant development
> effort. Common code such as TCG and others explicitly include architecture 
> specific header files, for example.

Hi Amir,
Simulating heterogenous machines comes up from periodically. So far no one has 
upstreamed a solution but there is definitely interest.

I suggest going ahead and posting the code even if it's not cleaned up.

> A possible solution, discussed on
> https://stackoverflow.com/q/63229262/619493 is to separate the
> simulation to multiple processes (as done by Xilinx) and to use some form of 
> Interprocess Communication channel between them.
> Such solution has several disadvantages:
>
> - Harder to synchronize simulation between processes
> - Performance impact of Interprocess Communication
> - Harder to debug, profile and maintain
>
> Instead, I would like to suggest a new approach we use at Neuroblade to 
> achieve this:
> Build QEMU as a shared library that can be loaded and used directly in a 
> larger simulation.
> Today we build qemu-system-nios2 shared library and load it from
> qemu-system-x86_64 in order to simulate an x86_64 system that also consists 
> of multiple nios2 cores.
> In our simulation, two independent "main" functions are running on
> different threads, and simulation synchronization is reduced to synchronizing 
> threads.
>
> To achieve this, I needed to do the following changes in QEMU:
>
> 1. Avoid Glib global context. Use a different context (g_main_context_new) 
> for each QEMU instance.
> 2. Change meson.build to build QEMU as a shared library (with PIC
> enabled for static libraries) 3. Define a C API for the library and
> export it (with a -Wl,--version-script)
>
> These changes seem enough for simulating mixed architecture system on a 
> single process.
>
> If this approach sounds useful, I'll be happy to send patches.
> I'd appreciate if you could provide your feedback!

I'm curious how much synchronization and IPC there is between the QEMU shared 
libraries? I would have guessed that the pain of making communication work 
efficiently between processes would be less than the pain of solving global 
state bugs related to shared libraries within a single process.

Were there issues with POSIX signal handlers?

Stefan

The contents of this email message and any attachments are intended solely for 
the addressee(s) and may contain confidential and/or privileged information and 
may be legally protected from disclosure. If you are not the intended recipient 
of this message or their agent, or if this message has been addressed to you in 
error, please immediately alert the sender by reply email and then delete this 
message and any attachments. If you are not the intended recipient, you are 
hereby notified that any use, dissemination, copying, or storage of this 
message or its attachments is strictly prohibited.