Re: [lwip-devel] SO_REUSEPORT removed on master branch

[Sylvain Rochet]

Hello Joel,

On Thu, Jul 16, 2015 at 03:02:28PM +0000, Joel Cunningham wrote:
> Additional follow up here:
> 
> I examined the implementation in the transport layer of SO_REUSEADDR 
> and LwIP isn't behaving like a traditional BSD system does:
> 
> Traditionally BSD SO_REUSEADDR only allows binding to INADDR_ANY and 
> additional unique IP addresses.  It does not allow binding to the same 
> IP address (and port) multiple times, that behavior requires 
> SO_REUSEPORT be enabled.
> 
> Our implementation is allowing binding to the same IP and port as long 
> as all existing PCBs have SO_REUSEADDR also set.  This matches the BSD 
> behavior of SO_REUSEPORT
> 
> This stack overflow page has a great overview of the different 
> implementations:
> 
> http://stackoverflow.com/questions/14388706/socket-options-so-reuseaddr-and-so-reuseport-how-do-they-differ-do-they-mean-t
> 
> Which implementation is LwIP attempting to follow?

SO_REUSEPORT is used to do "accept()" load balancing between multiple 
processes sharing the same bound (ip:)port. That's about to be only 
useful for SMP systems and it allows running multiple instance of the 
same daemon (common example: a HTTP proxy), this way:

kernel
|-httpd-proxy-1
|-httpd-proxy-2
`-httpd-proxy-3

The previous way was to use a master process doing either all I/O itself 
and sending/receiving data through pipe to its child or using any 
fd-passing technique to its child process, this way:

kernel
`-httpd-proxy-master
  |-httpd-proxy-1
  |-httpd-proxy-2
  `-httpd-proxy-3

Indeed, sessions load balancing is not really useful in lwIP, which is 
single threaded and probably rarely running with decent performance on 
SMP-capable hosts :-)

Without looking at the code, my guess is lwIP attempt to follow that's 
the traditional SO_REUSEADDR, which is basically "shut up and listen, 
even if the previous bound port is in a TIME_WAIT state". That's only a 
linked list after all, so the latest bound port is put at the head of 
the list and win.

Sylvain

signature.asc
Description: Digital signature