Re: [libmicrohttpd] Consider allowing SO_REUSEPORT when requested

[Christian Grothoff]

On 10/27/2014 06:52 PM, Milan Straka wrote:
> Hi Christian,
> 
>> -----Original message-----
>> From: Christian Grothoff <address@hidden>
>> Sent: 27 Oct 2014, 17:58
>>
>> Hi!
>>
>> AFAIK the interpretations of what 'reuse port' means differ widely
>> between operating systems (especially W32 vs. POSIX, but possibly also
>> within POSIX).  So adding that option to our API creates the issue that
>> we need to define a particular semantics and then apply it consistently
>> among all supported platforms --- and decide on how we should fail if
>> setting that option is not possible or fails (especially on exotic
>> platforms).
>>
>> So if you had a (1) well-defined (semantics, errors), (2) reasonablly
>> portable (W32, iOS, BSD, Linux at least), and (3) well-documented,
>> including error conditions and handling implementation of this, I'd
>> strongly consider adding it.
> 
> I agree that the 'reuse port' semantics is quite wild.

Ah, sorry, I somehow read SO_REUSEADDR (which as you write is very wild
and has differing semantics); SO_REUSEPORT is indeed more well-defined,
but also on Linux a bit, eh, 'recent', creating new fun issues as you
describe below.

> I am not familiar with many systems, but here are my 2 cents:
> 
> - BSD: SO_REUSEADDR and SO_REUSEPORT originate from BSD implementation.
>   The SO_REUSEADDR has weird semantics, allowing two requests to bound
>   to overlapping address:port pairs, if they are not equal. That is,
>   one can bind to 10.0.0.1:80 and 0.0.0.0:80, which would not be
>   possible without SO_REUSEADDR. Nevertheless, each address:port is
>   still captured by a single socket.
> 
>   SO_REUSEPORT means multiple applications can bind to same address:port
>   pair (if whey both specify SO_REUSEPORT).
> 
> - FreeBSD / OpenBSD / ...: same semantics as BSD, SO_REUSEPORT is available
> 
> - Mac OS X, iOS: same semantics as BSD, SO_REUSEPORT is available
> 
> - Linux: same semantics, but SO_REUSEPORT available only with 3.9+
>   kernel
> 
> - Windows: only SO_REUSEADDR with different semantics -- SO_REUSEADDR on
>   Windows is very similar to SO_REUSEPORT on BSD, allowing for multiple
>   applications to bind to the same address:port.
> 
> So I propose the following:
> 
> - add MHD_OPTION_ALLOW_REUSING_LISTENING_ADDRESS with the following
>   semantics: if set to true, the listening socket is opened in such
>   a way that other sockets can also listen on the same address:port.
>   If set to false, the listening socket is opened in such a way that
>   other sockets cannot listen on the same address:port.
> 
> - implementation of the "true" case: on Windows, this is already true.
>   On BSD/Mac/iOS, use setsockopt with SO_REUSEPORT. On Linux, either
>   a) use macros to find out whether SO_REUSEPORT is defined and use it,
>      or fail if it is not defined and the option is used,

Fail to compile? Not a great idea to require 3.9 plus matching headers
and otherwise FTBFS.

>   b) use compilation switch which either allows using SO_REUSEPORT or
>      throws runtime

Not sure I understand this one. I understand we sometimes use
*configure* switches to enable people to build a smaller MHD for
embedded devices, but on most systems the API should be the API and
never differ simply based on how configure went.  Also, given that you
have a rather strict specification that says: definitively allow or
definitively deny, that means that there cannot be something in the
middle where it is "well, maybe allow, maybe deny, as we were configured
without the option and the application didn't specify 'hard allow'".

>   c) use the value of SO_REUSEPORT (15), use it unconditionally and fail
>      if unsupported

That we could do, use macros if available, fallback to hard-coded '15'
and then runtime failure if REUSEPORT requested but not available.

> - implementation of the "false" case: nothing, only on Windows use
>   SO_EXCLUSIVEADDRUSE instead of SO_REUSEADDR. (This is BTW recommended
>   for all Windows server applications, see for example
>   
> http://msdn.microsoft.com/en-us/library/windows/desktop/ms740621(v=vs.85).aspx).

Technically that's (on W32) API-breakage (suddenly different behavior),
but given the context I think that's legitimate.

> As for the error handling, the creation of the socket should fail if the
> request cannot be honored. From this point of view, it would be best to
> add compilation option "USE_SO_REUSEPORT", which would be enabled by
> default on BSD/*BSD/OSX/iOS/Linux 3.9+. If the SO_REUSEPORT should be
> used and this compilation option was off, the socket creation should
> fail.

Ugh, please not a compilation option. I'd just make it a flag,
"MHD_USE_SO_REUSEPORT = 32768" and then if the flag is set, we try
SO_REUSEPORT (and if that fails, return NULL from MHD_start_daemon());
if the flag is not set, we do the EXLUSIVEADDRUSE on W32 (and if that
fails, we again return NULL from MHD_start_daemon()).  Hard-code to 15
if macro not defined on Linux, never FTBFS.  Leaves open the question as
to what to do on "other" OSes, but I'm OK to leave that up to porters
(and porting guides if those semantics cannot be satisfied).

> This has the nice property that it will compile on all platforms, and
> for the unsupported platforms the socket creating will fail at runtime.

I agree with that goal, but enabling the option must be runtime flag,
not compiler flag.

> Then, we can investigate the platform and either enable the
> USE_SO_REUSEPORT switch, or provide new platform-specific
> implementation.

I think with the proposed semantics, we can afford to say that other
platforms need to document exceptions to the API if they really cannot
make any of these.

My 2 cents (naturally, further discussion is encouraged)

Christian

signature.asc
Description: OpenPGP digital signature