Re: [PATCH V6 4/6] hw/block/nvme: support for multi-controller in subsystem

[Keith Busch]

On Mon, Jan 25, 2021 at 07:03:32PM +0100, Klaus Jensen wrote:
> On Jan 24 11:54, Minwoo Im wrote:
> > We have nvme-subsys and nvme devices mapped together.  To support
> > multi-controller scheme to this setup, controller identifier(id) has to
> > be managed.  Earlier, cntlid(controller id) used to be always 0 because
> > we didn't have any subsystem scheme that controller id matters.
> > 
> > This patch introduced 'cntlid' attribute to the nvme controller
> > instance(NvmeCtrl) and make it allocated by the nvme-subsys device
> > mapped to the controller.  If nvme-subsys is not given to the
> > controller, then it will always be 0 as it was.
> > 
> > Added 'ctrls' array in the nvme-subsys instance to manage attached
> > controllers to the subsystem with a limit(32).  This patch didn't take
> > list for the controllers to make it seamless with nvme-ns device.
> > 
> > Signed-off-by: Minwoo Im <minwoo.im.dev@gmail.com>
> > ---
> >  hw/block/nvme-subsys.c | 21 +++++++++++++++++++++
> >  hw/block/nvme-subsys.h |  4 ++++
> >  hw/block/nvme.c        | 29 +++++++++++++++++++++++++++++
> >  hw/block/nvme.h        |  1 +
> >  4 files changed, 55 insertions(+)
> > 
> > diff --git a/hw/block/nvme.c b/hw/block/nvme.c
> > index b525fca14103..7138389be4bd 100644
> > --- a/hw/block/nvme.c
> > +++ b/hw/block/nvme.c
> > @@ -4481,6 +4484,10 @@ static void nvme_init_ctrl(NvmeCtrl *n, PCIDevice 
> > *pci_dev)
> >      id->psd[0].enlat = cpu_to_le32(0x10);
> >      id->psd[0].exlat = cpu_to_le32(0x4);
> >  
> > +    if (n->subsys) {
> > +        id->cmic |= NVME_CMIC_MULTI_CTRL;
> > +    }
> 
> Since multiple controllers show up with a PCIe port of their own, do we
> need to set bit 0 (NVME_CMIC_MULTI_PORT?) as well? Or am I
> misunderstanding that bit?

AIUI, if you report this MULTI_PORT bit, then each PCI device in the
subsystem needs to report a different "Port Number" in their PCIe Link
Capabilities register. I don't think we can manipulate that value from
the nvme "device", but I also don't know what a host should do with this
information even if we could. So, I think it's safe to leave it at 0.