Re: [Qemu-devel] Enablig DLPAR capacity on QEMU pSeries

[Alexander Graf]

On 12.09.2012, at 22:56, Erlon Cruz <address@hidden> wrote:

> On Wed, Sep 12, 2012 at 12:53 PM, Alexander Graf <address@hidden> wrote:
>> On 09/12/2012 04:54 PM, Erlon Cruz wrote:
>>> 
>>> Hi all,
>>> 
>>> We are planning to implement DLPAR capacity on QEMU pSeries. As we
>> 
>> 
>> What is DLPAR? Hotplug support?
> 
> Yes, basically the way PowerVM uses to dynamically add memory, cpu,
> and I/O slots to logical partitions (LPARs)
> 
>> 
>>> lack of experience in the internals of the arch we would like you guys
>>> to give us some design directions
>>> and confirm if we going in the right direction. Our first idea is:
>>> 
>>>     1 - to patch 'spapr.c' so it can dynamically insert/remove basic
>>> items into the device tree.
>> 
>> 
>> What exactly would you like to patch into it? We already do have support for
>> dynamic dt creation with the spapr target.
>> 
> 
> Actually we were not sure if the machine could do that. So we can add
> things to the tree after booting it?

Linux gets a dt blob on boot, so that one is fixed. IIRC the user space tools 
for pHyp mess with the device tree through special hooks from the Linux side.

> 
>>>     2 - create a host side device that will be used with a guest side
>>> driver to perform guest side operations and communicate changes from
>>> host to the guest (like DynamicRM does in PowerVM LPARs). We are not
>> 
>> 
>> Why not just use hypercalls?
> 
> The hypercalls are initiated from the guest side right? We also need a
> way to the host send things to guest. Using hypercall also would
> require a guest side KM.

Ah, you need some interrupt mechanism. I see.

> 
>>> planning to use powerpc-tools and want to make resource management
>>> transparent (i.e. no need to run daemons or userspace programs in the
>>> guest, only this kernel driver).
>>>     3 - create bindings to support adding/removal  ibmvscsi devices
>>>     4 - create bindings to support adding/removal  ibmveth devices
>>>     5 - create bindings to support adding/removal PCI devices
>>>     6 - create bindings to support adding/removal of memory
>> 
>> 
>> This is going to be the hardest part. I don't think QEMU supports memory
>> hotplug yet.
> 
> AFAIC ballonning is what QEMU provides so far which is fine to x86.

Yes. But balooning can only shrink memory footprint of a VM, not increase its 
memory size.

> 
>> 
>>>         - Do we need to do this the way PowerVM does? We have tested
>>> virtio ballooning and it can works with a few endiannes corrections.
>> 
>> 
>> I don't know how PowerVM works. But if normal ballooning is all you need,
>> you should certainly just enable virtio-balloon.
> 
> PowerVM works with Logical Memory Blocks (LMB). The hypervisor
> hotplugs memory blocks to guest's memory. Not only a 'borrowing' from
> the guests, right Ben?
> 
>>>     7 - create bindings to support adding/removal  CPUs
>>>         - is SMP supported already? I tried to run SMP in a x86 host
>>> and the guest stuck when SMP is enabled
>> 
>> 
>> SMP should work just fine, yes. Where exactly does it get stuck?
> 
> I think that is right after the guest kernel enables SMP
> 
> [    7.478259] Faulting instruction address: 0xc00000000053bbec
> [    7.479521] Oops: Kernel access of bad area, sig: 11 [#1]
> [    7.479694] SMP NR_CPUS=1024 NUMA pSeries

Mind to attach gdb to the vm and break at that address? Then print a bt on all 
threads (vcpus) :).

> 
> http://pastebin.com/VMtRyaTE
> 
>> 
>> 
>>>         - would be possible to work on this without a P7 baremetal
>>> machine?
>> 
>> 
>> At least for device hotplug, it should be perfectly possible to use an old
>> G5 with PR KVM. I haven't gotten around to patch all the pieces of the
>> puzzle to make -M pseries work with PR KVM when it's running on top of pHyp
>> yet, so that won't work.
>> 
>> 
>>> We have a P7 8205-E6B, is that possible to kick PHYP out?
>> 
>> 
>> Ben?
>> 
>> 
>>> Any ideia on how much effort (time/people) the hole thing would take?
>>> Any consideration about this is much appreciated :)
>> 
>> 
>> Phew. It's hard to tell. Depends heavily on how good your people are :).
> 
> Well, considering someone like you :p, so we would just need to
> multiply it by 5 :P

It's still hard to guess. Full-time work probably about a month for a working 
prototype. But you might run into additional issues. And then there's the 
upstream review process which will take at least as long until thebthing is 
clean. And then quite some time to get it bug free ;).


Alex

> 
>> 
>> 
>> Alex
>> 
>>