Re: [RFC PATCH 0/5] hw/block/nvme: support multi-path for ctrl/ns

[Klaus Jensen]

On Jan 15 21:05, Minwoo Im wrote:
> Hello,
> 
> This series added support for multi-path I/O with multi-controllers and
> namespace sharing.  By supporting these features, we can test Linux
> kernel mpath(multi-path) code with this NVMe device.
> 
> Patches from the first to third added multi-controller support in a NVM
> subsystem by adding a mpath.ctrl parameter to nvme device.  The rest of
> the patches added namespace sharing support in a NVM subsystem with two
> or more controllers by adding mpath.ns parameter to nvme-ns device.
> 
> Multi-path enabled in kernel with this series for two controllers with a
> namespace:
> 
>   root@vm:~/work# nvme list -v
>   NVM Express Subsystems
> 
>   Subsystem        Subsystem-NQN                                              
>                                       Controllers
>   ---------------- 
> ------------------------------------------------------------------------------------------------
>  ----------------
>   nvme-subsys0     nqn.2019-08.org.qemu:serial                                
>                                       nvme0, nvme1
> 
>   NVM Express Controllers
> 
>   Device   SN                   MN                                       FR   
>     TxPort Address        Subsystem    Namespaces
>   -------- -------------------- ---------------------------------------- 
> -------- ------ -------------- ------------ ----------------
>   nvme0    serial               QEMU NVMe Ctrl                           1.0  
>     pcie   0000:01:00.0   nvme-subsys0 nvme0n1
>   nvme1    serial               QEMU NVMe Ctrl                           1.0  
>     pcie   0000:02:00.0   nvme-subsys0 nvme0n1
> 
>   NVM Express Namespaces
> 
>   Device       NSID     Usage                      Format           
> Controllers
>   ------------ -------- -------------------------- ---------------- 
> ----------------
>   nvme0n1      1        268.44  MB / 268.44  MB    512   B +  0 B   nvme0, 
> nvme1
> 
> The reason why I put 'RFC' tag to this series is mostly about the last
> patch "hw/block/nvme: add namespace sharing param for mpath".  It seems
> like QEMU block backing device does not support to be shared among two
> or more -device(s).  It means that we just can't give same drive=
> property to multiple nvme-ns devices.  This patch has just let -device
> maps to -drive one-to-one(1:1), but if namespae sharing is detected and
> setup by the host kernel, then a single block device will be selected
> for the NVM subsystem.  I'm not sure this is a good start for this
> feature, so I put the RFC tag here.
> 
> Please kindly review!
> 

Hi Minwoo,

First - super awesome that we get this discussion going. I've been
hacking around this a couple of times, but I've never been happy with
the approach.

As you already mentioned, the problem I see with this approach is that
we have separate namespaces attached to separate controllers. So we are
faking it to the max and if I/O starts going through the other
controller we end up on a namespace that is unrelated (different data).
Havoc ensues.

My approach looks a lot like yours, but I hacked around this by adding
extra 'ctrl-0', 'ctrl-1', ..., link-parameters to the namespace device,
replacing the bus. This works well because the namespace then just
registers with multiple controllers. But adding more parameters like
that just isnt nice, so I've been trying to figure out how to allow a
parameter to be specified multiple times, so we could just do more
'ctrl'-parameters.

Alas, since I started thinking about namespace sharing I have been
regretting that I didn't reverse the bus-mechanic for namespace
attachment. It would align better with the theory of operation if it was
the controllers that attached to namespaces, and not the other way
around. So what I would actually really prefer, is that we had multiple
'ns' link parameters on the controller device.

   -device nvme-ns,id=a,nsid=1,...
   -device nvme-ns,id=b,nsid=2,...
   -device nvme-ns,id=c,nsid=3,...
   -device nvme,cntlid=0,serial=foo,ns=a,ns=b
   -device nvme,cntlid=1,serial=foo,ns=a,ns=c

But I havn't been able to figure out how to kick QOM into doing this.
And I'm definitely not sure this is the way to go. Should we instead
introduce a 'nvme-subsys' device and walk a bus?

I'd really appreciate some input on how we should model this if anyone
has any thoughts. And I think we should consider stuff like detached
namespaces as well. Support for Namespace Management. The whole shabang.

signature.asc
Description: PGP signature