[lwip-devel] Proposals for lwip

[Andrea Berlingieri]

Hello lwip

Let me introduce myself. I'm a CS student at the University of Bologna working 
on the Virtualsquare
project (you can find some infos at 
http://wiki.virtualsquare.org/wiki/index.php/Main_Page, although
the wiki has not been updated yet with the last development efforts. The most 
updated development
versions of the Virtualsquare projects are available on Github, at 
http://github.com/virtualsquare
and at http://github.com/rd235). In particular, I'm working on the vuos 
project, the successor to
view-os. Basically the idea of vuos is to implement a partial virtual machine 
by intercepting the
system calls a virtualized process makes and to execute userspace code in their 
place. Vuos is
modular: this means that we can choose which system calls to virutalize and 
which system call will
be directly forwarded to the underlying kernel. As of right now, the project is 
made for Linux and
for the x86_64 architecture (because of the use vuos makes of ptrace to 
intercept the system calls),
and the system call support is not complete yet (some system calls are not 
supported at the moment),
but the project is still under active development.

In particular, I'm currently working on a submodule for the vunet module of 
vuos, which virtualizes
the system calls of the BSD socket API (socket, bind, connect, etc.). This 
submodule uses lwip as
the underlying network stack, instead of using Linux' TCP/IP stack. In the past 
(2006-2011) a fork
of the lwip project was made to add support for IPv6 and some other features. 
Now I'm trying to
write a new submodule based on the latest version of lwip, and in a way that 
makes it possible, for
the two project (lwip and vuos), to develop independently while maintaining 
compatibility (where the
submodule acts as a "bridge" between vuos and lwip).

In vuos both the modules and the submodules are compiled to obtain shared 
objects, which are then
dynamically loaded/unloaded at need. So it is possible, for example, to insert 
the vunet module of
vuos to virtualize the BSD socket API, and then to mount many different 
networks stacks as
submodules. The virtualized processes can then decide which stack to use for 
their networked
applications.

The way I'm tying to implement this with lwip is to compile lwip as a shared 
library, through the
Unix port of lwip, and to load it dynamically with dlmopen, which lets me load 
many different
instances of the same dynamic library. This way a process can mount many 
different instances of a
lwip TCP/IP stack, each with its own configuration and without interferences 
between them. Then,
each system call of the socket API is intercepted and handled appropriately 
(some calls are directly
redirected to their lwip counterpart, some require a little more work).

Now, working with lwip to implement the vunet submodule I've encountered some 
problems that I would
like to discuss with you. I also have some expansion proposals, which I would 
also like to discuss.
Here's a list:

 - I've seen that in lwip there is a mechanism to specify some hooks that will 
be called at the
   appropriate moment. Among the currently defined hooks I unfortunately 
couldn't find the one I
   needed. That is, a hook that is called whenever there is an event on a 
lwip_socket. I need it
   because when an event occurs I need to "notify" the vuos core. The first 
modification I made was
   to add such a hook in opt.h, and the appropriate code in event_callback, in 
sockets.c. As of
   right now, the source code for the submodule is placed in a single source 
file, named
   vunetlwip.c. This file also contains the definition of the hook function. 
Now, this initially
   made for a strange arrangement, where vunetlwip.c depended on the headers of 
lwip, and lwip was
   linked against the vunetlwip.so shared object to link the definition of the 
hook function. This,
   unfortunately, didn't even work, as the hook function had references to 
functions of other shared
   objects of the vuos core that couldn't be linked with lwip without making a 
mess. As of right
   now, for testing purposes, the hook function code has been moved in lwip (in 
a separate file) and
   all it does is notify the events on a Unix domain socket, which is listened 
to by a thread
   created by the vunetlwip submodule, which, in turn, notifies the vuos core. 
This is not ideal,
   and I thought that the best solution in this case would be to add a 
mechanism in lwip to specify
   the hook functions at run time, instead of directly defining them at compile 
time. So, my first
   proposal would be to add such a mechanism and a hook for when there are 
events on a socket.

 - I've seen that the sockaddr structures have been redefined in lwip. It looks 
although that the
   "model" after which they were made was that of BSD Unix, as each sockaddr 
struct adds a len field
   before the others in the struct. This kind of "breaks" in Linux, as the len 
field is absent, and
   using a sockaddr as defined in Linux with, for example, lwip_connect, 
casuses a mismatch between
   the fields of the two structs, and thus unexpected behavior. This happens 
because the virtualized
   process usually runs a program that assumed Linux as the target operating 
system, and thus uses
   sockaddr as defined in Linux when using bind, connect, sendto, etc. It seems 
though that the len
   field at the begininnig of the struct is non standard and non portable (as I 
could understand
   from this SO answer:
   
https://stackoverflow.com/questions/23626805/why-does-accept2-need-the-sockaddr-length-as-a-separate-pointer).
   So, my second proposal would be to modify the sockaddr structs to either 
remove the len field, or
   to at least move it at the end of the structure, so that Linux' and lwip's 
sockaddr agree at
   least on the first part of the struct.

 - In the old lwip fork I mentioned above (which, from now on, I will refer to 
as "lwipv6") the
   following features were added: a vde driver, to work with vde (also a 
virtualsquare project,
   https://en.wikipedia.org/wiki/Virtual_Distributed_Ethernet), SLIRP protocol 
support, and NAT
   support. I think they would be some interesting additions to lwip. Let me 
know if you're
   interested in porting them upstream.

 - I see that lwip is built mainly for embedded devices and one of its most 
prominent features is
   the low memory footprint. While I think this is all good and important, I 
also think that it
   would be nice to be able to choose between the lwip internal memory 
management and the dynamic
   memory management provided by the Linux kernel when compiling the lwip 
shared library. In lwipv6
   code was added to use the glibc's malloc when mem_malloc was called, glibc's 
free when mem_free
   was called, and so on. I think it would be a nice feature for the Unix lwip 
library. The choice
   could be made at compile time through some configuration constants.

 - I've seen that in lwip each netif is associated with a single ipv4 address, 
a single netmask and
   a single gateway. Now, in lwipv6 a modification was made to support lists of 
ip addresses, so
   that a single interface could be associated with more that one ip address. 
Now, introducing this
   feature to lwip would require also a modification to the route function, 
although this wouldn't
   seem to be too complicated, at least conceptually (as, instead of looking at 
a single address, a
   search on a list would be performed). So, my next proposal would be to add 
this feature.

 - Stack configuration (setting addresses, setting interfaces, getting 
interfaces infos, etc) is
   made through netlink in my vunetlwip module. Even ioctls are intercepted and 
converted to the
   appropriate netlink counterparts. Now, while working on the address 
information retrieval (the
   GETADDR message of the rtnetlink protocol), I couldn't be able to find the 
prefix len for a given
   ipv6 address. As of right now I simply assume that the prefix len is 64 and 
return that, but that
   is a strong assumption. This is mix between a proposal to add such a field 
somewhere in lwip, and
   a request of help on how to find this information, if it is already encoded 
somewhere I couldn't
   find.

 - In lwipv6 a modification was made to support multiple routes for a netif. 
That is, for each netif
   a list of routes was kept. It seems to me though that in lwip routing is 
very simple: packets are
   routed to the interface that has a matching ip address prefix to that of the 
dest ip address (if
   source address routing is not involved). My next proposal would be to add 
such a feature to lwip,
   as I think it may be useful, although it may make routing more complicated.

 - Now, this last one is kind of low priority. Basically, while testing with 
ping, I noticed that
   the iputils version of ping for Linux uses a 
socket(AF_INET,SOCK_DGRAM,IPPROTO_ICMP) call, in the
   case of ipv4, to create a socket to send ICMP packets. Now, lwip assumes 
that a socket that uses
   SOCK_DGRAM as a type is a UDP socket. This caused some unexpected behavior. 
As of right now I
   simply intercept the socket call and transform a 
socket(AF_INET,SOCK_DGRAM,IPPROTO_ICMP) call
   into a socket(AF_INET,SOCK_RAW,IPPROTO_ICMP) call. This kind of fixes the 
problem, but is not
   ideal. My proposal would be to add support for this case of socket, if a 
configuration constant
   has been defined (it seems to me that this kind of calls are only made and 
supported in Linux. In
   this case the kernel handles the call in a special way, and the created 
socket is only for ICMP
   echo requests: 
https://stackoverflow.com/questions/8290046/icmp-sockets-linux , third answer).
   I'm not too sure about this, maybe we can discuss it a little.

So, this is all I've come up with as of right now. I would like to discuss 
these topics with you.

I'm sorry for the long email. This is my first time posting to this mailing 
list, so If I did
anything wrong let me know, so that it won't happen again. I wish everyone a 
good day.

Andrea Berlingieri