[Gluster-devel] RFC - "Connection Groups" concept

[Justin Clift]

Hey all,

This is an idea that's been on my mind for a while, to overcome a
fundamental problem Gluster has with some network common topologies.
(thanks to Niels de Vos for letting me bounce ideas off him btw)

Most corp places I've worked have multiple network interfaces on their
storage boxes, each for a specific purpose (eg backup interface, client
interface, general connectivity interface, and more).

At present Gluster can't really make use of this kind of topology
effectively.  Having some interfaces only for client traffic, other
interfaces for peer replication traffic just isn't how it's designed.

To address this, we could introduce "connection groups".

Whereby we let the Gluster sysadmin define "connection groups"
(eg "client-group" "replication-group"), and the list of
host+interfaces that belong to each group.

On the command line, it could look something like this:

  $ sudo gluster connection-group create client-group gluster1:eth0 
gluster2:eth0 gluster3:eth0 gluster4:eth0
   (being the first ethernet interface on 4 gluster boxes)
  $ sudo gluster connection-group create replication-group gluster1:ib0 
gluster2:ib0 gluster3:ib0 gluster4:ib0
   (being the first Infiniband interface on 4 gluster boxes)
  $ sudo gluster connection-group create mygroup1 gluster1:ib1 gluster2:ib1 
gluster3:ib1 gluster4:ib1
   (some other group the Sysadmin wants)

We'll probably need to add an option to the peer probe command
so a connection group can be specified.  Something like:

  $ sudo gluster peer probe gluster3 --conngroup=mygroup1

Note - I'm not sure if $hostname:$interface is really a crash hot idea here.
We might want to do $hostname:$IP instead, so we don't get confused by
not-straight-forward interfaces such as bonding, IP aliasing, and maybe
IPoIB/RDMA.

When a client mounts a volume, it should specify a group.  Gluster then
gives it the connection details for the hosts:interfaces in its group.

The Sysadmin would then associate connection groups to their volumes:

  $ sudo gluster volume groupadd somevolume client-group
  $ sudo gluster volume groupadd othervolume mygroup1

We could also set one of the groups as a default (for clients that
don't say what their group is):

  $ sudo gluster volume groupdefault somevolume client-group

Client's could specify which group it belongs to on the mount command
line:

  $ sudo mount -t glusterfs -o conngroup=XXX gluster1:somevol /my/mount/point

2nd note - We might also need to have a distinction of "peer groups"
and "volume groups".  Not sure atm, and I'm falling asleep at the
keyboard but want to get this sent. :)

Thoughts? :)

Regards and best wishes,

Justin Clift

--
Open Source and Standards @ Red Hat

twitter.com/realjustinclift