Re: [RFC 0/9] Add an interVM memory sharing device

[Igor Kotrasiński]

On 2/7/20 5:33 PM, Stefan Hajnoczi wrote:
> On Fri, Feb 07, 2020 at 11:04:03AM +0100, Igor Mammedov wrote:
>> On Fri, 7 Feb 2020 10:00:50 +0100
>> Igor Kotrasiński <address@hidden> wrote:
>>
>>> On 2/5/20 3:49 PM, Jan Kiszka wrote:
>>>> On 05.02.20 15:39, Stefan Hajnoczi wrote:
>>>>> On Tue, Feb 04, 2020 at 12:30:42PM +0100,
>>>>> address@hidden wrote:
>>>>>> From: Igor Kotrasinski <address@hidden>
>>>>>>
>>>>>> This patchset adds a "memory exposing" device that allows two QEMU
>>>>>> instances to share arbitrary memory regions. Unlike ivshmem, it does not
>>>>>> create a new region of memory that's shared between VMs, but instead
>>>>>> allows one VM to access any memory region of the other VM we choose to
>>>>>> share.
>>>>>>
>>>>>> The motivation for this device is a sort of ARM Trustzone "emulation",
>>>>>> where a rich system running on one machine (e.g. x86_64 linux) is able
>>>>>> to perform SMCs to a trusted system running on another (e.g. OpTEE on
>>>>>> ARM). With a device that allows sharing arbitrary memory between VMs,
>>>>>> this can be achieved with minimal changes to the trusted system and its
>>>>>> linux driver while allowing the rich system to run on a speedier x86
>>>>>> emulator. I prepared additional patches for linux, OpTEE OS and OpTEE
>>>>>> build system as a PoC that such emulation works and passes OpTEE tests;
>>>>>> I'm not sure what would be the best way to share them.
>>>>>>
>>>>>> This patchset is my first foray into QEMU source code and I'm certain
>>>>>> it's not yet ready to be merged in. I'm not sure whether memory sharing
>>>>>> code has any race conditions or breaks rules of working with memory
>>>>>> regions, or if having VMs communicate synchronously via chardevs is the
>>>>>> right way to do it. I do believe the basic idea for sharing memory
>>>>>> regions is sound and that it could be useful for inter-VM communication.
>>>>>
>>>>> Hi,
>>>>> Without having looked into the patches yet, I'm already wondering if you
>>>>> can use the existing -object
>>>>> memory-backend-file,size=512M,mem-path=/my/shared/mem feature for your
>>>>> use case?
>>>>>
>>>>> That's the existing mechanism for fully sharing guest RAM and if you
>>>>> want to share all of memory then maybe a device is not necessary - just
>>>>> share the memory.
>>>
>>> That option adds memory in addition to the memory allocated with the
>>> '-m' flag, doesn't it? I looked into that option, and it seemed to me
>>> you can't back all memory this way.
>> with current QEMU you play with memory sharing using numa workaround
>>
>> -m 512 \
>> -object memory-backend-file,id=mem,size=512M,mem-path=/my/shared/mem 
>> feature,share=on \
>> -numa node,memdev=mem
>>
>> also on the list there is series that allows to share main ram
>> without numa workaround, see
>>    "[PATCH v4 00/80] refactor main RAM allocation to use hostmem backend"
>>
>> with it applied you can share main RAM with following CLI:
>>
>> -object memory-backend-file,id=mem,size=512M,mem-path=/my/shared/mem 
>> feature,share=on \
>> -m 512 \
>> -M virt,memory-backend=mem
> 
> Nice!  That takes care of memory.

After a bit of hacking to map the shared RAM instead of communicating 
via socket I can confirm - I can run OpTEE this way, and it passes 
tests. My solution is *technically* more accurate since it is aware of 
memory subregions and completely independent from memory backend setup, 
but with my use case satisfied already, I don't think it's of use to anyone.

> 
> If signalling (e.g. a notification interrupt) is necessary then a
> mechanism is still needed for that.  I don't know enough about TrustZone
> to suggest an appropriate way of doing it with existing QEMU features.
> Maybe Peter understands?
> 

Any signalling mechanism that can pass data along with it (e.g. ivshmem 
with its shared memory) will suffice.

Igor