Re: [Question] Regarding containers "unattached/peripheral/anonymous" - their relation with hot(un)plug of devices

[Igor Mammedov]

On Wed, 3 Jun 2020 15:13:26 +0000
Salil Mehta <salil.mehta@huawei.com> wrote:

> Hi Igor,
> My sincere Apologies, I just realized that I missed to reply this mail.
> I was distracted to something else in  the month of the February and
> had only resumed working on hotplug in march. But will still reply to
> this mail. Also, I have incorporated most of the below points as in the
> vcpu hotplug patches as per your suggestions.
> 
> 
> > From: Qemu-arm [mailto:qemu-arm-bounces+salil.mehta=huawei.com@nongnu.org]
> > On Behalf Of Igor Mammedov
> > Sent: Monday, January 27, 2020 3:04 PM
> > To: Salil Mehta <salil.mehta@huawei.com>
> > 
> > On Fri, 24 Jan 2020 18:44:16 +0000
> > Salil Mehta <salil.mehta@huawei.com> wrote:
> >   
> > > > From: Igor Mammedov [mailto:imammedo@redhat.com]
> > > > Sent: Friday, January 24, 2020 4:07 PM
> > > >
> > > > On Fri, 24 Jan 2020 15:02:10 +0000
> > > > Salil Mehta <salil.mehta@huawei.com> wrote:
> > > >  
> > > > > > From: Igor Mammedov [mailto:imammedo@redhat.com]
> > > > > > Sent: Friday, January 24, 2020 1:54 PM
> > > > > > To: Salil Mehta <salil.mehta@huawei.com>
> > > > > >
> > > > > > On Fri, 24 Jan 2020 11:20:15 +0000
> > > > > > Salil Mehta <salil.mehta@huawei.com> wrote:
> > > > > >  
> > > > > > > Hello,
> > > > > > > I am working on vCPU Hotplug feature for ARM64 and I am in mid of 
> > > > > > > understanding
> > > > > > > some aspect of device_add/device_del interface of the QEMU.
> > > > > > >
> > > > > > > Observations:
> > > > > > > 1. Any object initialised by qmp_device_add() gets into 
> > > > > > > /machine/unattached
> > > > > > > container. I traced the flow to code leg inside  
> > > > > > > device_set_realized()
> > > > > > > 2. I could see the reverse qmp_device_del() expects the device to 
> > > > > > > be in
> > > > > > > /machine/peripheral container.
> > > > > > > 3. I could see any object initially added to unattached container 
> > > > > > > did not had
> > > > > > > their parents until object_add_property_child() was called 
> > > > > > > further in the leg.
> > > > > > > which effectively meant a new property was created and property 
> > > > > > > table
> > > > > > > populated and child was parented.
> > > > > > > 4. Generally, container  /machine/peripheral was being used 
> > > > > > > wherever
> > > > > > > DEVICE(dev)->id was present and non-null.
> > > > > > >
> > > > > > > Question:
> > > > > > > 1. Wanted to confirm my understanding about the use of having 
> > > > > > > separate
> > > > > > > containers like unattached, peripheral and anonymous.  
> > > > > >  
> > > > > > > 2. At init time all the vcpus goes under *unattached* container. 
> > > > > > > Now,
> > > > > > > qmp_device_del() cannot be used to unplug them. I am wondering  
> > > > > >
> > > > > > device is put into 'unattached' in case it wasn't assigned a parent.
> > > > > > Usually it happens when board creates device directly.  
> > > > >
> > > > >
> > > > > Sure, but if we decide that certain number(N) of vcpus are 
> > > > > hotplugabble
> > > > > and certain subset of N (say 'n' < 'N') should be allowed to be 
> > > > > present
> > > > > or cold-plugged at the init time then I wonder which of the following
> > > > > is correct approach:
> > > > >
> > > > > 1. Bring all of N vcpus at boot time under "peripheral" container
> > > > >                                    OR
> > > > > 2. Just bring subset 'n' of 'N' under "peripheral" container and rest
> > > > >     under "unattached" container? And later as and when rest of the
> > > > >     vcpus are hotplugged they should be transferred from "unattached"
> > > > >     container to "peripheral" container?  
> > > >
> > > > issue with that is that to put device into "peripheral" container,
> > > > 'the user' must provide 'id'. (currently QEMU isn't able to do it on 
> > > > its own
> > > > [1])
> > > >
> > > > But it doesn't mean that cold-plugged CPUs can't be unpluged.
> > > > What current users could do is start QEMU like this (simplified 
> > > > version):
> > > >
> > > >  $QEMU -smp 1,maxcpus=N -device foo-cpu-type,id=CPU00 -device
> > > > foo-cpu-type,id=CPU01 ...
> > > >
> > > > i.e. 1st CPU is not manageable due to lack if 'id' and is created by 
> > > > board code,
> > > > the rest have 'id' and could be managed.  
> > >
> > >
> > > I understand, that we can somehow assign ids from the QMP interface but
> > > above will not push vcpus into "peripheral" container. They will appear
> > > in "unattached" container but with specified names and as-far-as I can
> > > see in the code 'device_del' can only delete objects/devices from the
> > > 'peripheral' container?  
> > 
> > qemu-system-x86_64 -monitor stdio \
> >     -smp 1,maxcpus=3 \
> >     -device qemu64-x86_64-cpu,id=foo,socket-id=1,core-id=0,thread-id=0 \
> >     -device qemu64-x86_64-cpu,socket-id=2,core-id=0,thread-id=0
> > 
> > (qemu) info hotpluggable-cpus
> > Hotpluggable CPUs:
> >   type: "qemu64-x86_64-cpu"
> >   vcpus_count: "1"
> >   qom_path: "/machine/peripheral-anon/device[0]"
> >                       ^^^^^^^^^^^^^^^
> >   CPUInstance Properties:
> >     socket-id: "2"
> >     core-id: "0"
> >     thread-id: "0"
> >   type: "qemu64-x86_64-cpu"
> >   vcpus_count: "1"
> >   qom_path: "/machine/peripheral/foo"
> >                       ^^^^^^^^^^
> > 
> > in gist, if device is created with any variant of device_add,
> > it goes to "peripheral[-anon]" including cold-plugged one.  
> 
> 
> I am not sure why you said "including cold-plugged one"? I hope by
> cold-plug'ging you mean here are the CPUs which already exist at
> the boot time of the Guest VM and plugged using -device?

yes, it's about a plugged one with  '-device'

 
> >   CPUInstance Properties:
> >     socket-id: "1"
> >     core-id: "0"
> >     thread-id: "0"
> >   type: "qemu64-x86_64-cpu"
> >   vcpus_count: "1"
> >   qom_path: "/machine/unattached/device[0]"  
>                           ^^^^^^^^^^^^^
> 
> Unless you have pasted wrongly here, above output also shows qom path
> as 'unattached' for cold-plugged CPUs. Am I missing something? :)
> 
> 
> >   CPUInstance Properties:
> >     socket-id: "0"
> >     core-id: "0"
> >     thread-id: "0"  
> 
> 
> 
> > > Plus, having those many ids specified for over large number of vcpus
> > > does not looks very practical solution. We need interface like auto  
> > number of IDs is not a problem since it's usually handled by management
> > software just fine (ex: libvirt does it)
> >   
> > > Generation of ids even at the boot time. I could see from the link you
> > > have shared that it is already being used by ID_BLOCK subsystem. Can we
> > > create a new subsystem for cpus under this and do the auto Generation
> > > of vcpu ids as well?  
> > 
> > I'm not sure that auto ids was actually merged.
> > (I thought it wasn't)  
> 
> 
> Well if you were referring to below then it has been part of qemu for quite
> long now:
> 
> Patch: util - add automated ID generation utility
> File: https://github.com/qemu/qemu/blob/master/util/id.c
> Commit-id: https://github.com/qemu/qemu/commit/a0f1913637e6

Thanks, I didn't know that it eventually got merged.

> 
> > Anyway auto IDs are not directly related to enabling CPU hotplug for ARM,
> > if you feel they should be generated you can try to propose patches.  
> 
> Sure. 
> 
> 
[...]