[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [lwip-devel] Proposals for lwip
Re: [lwip-devel] Proposals for lwip
Tue, 16 Jul 2019 21:00:33 +0200
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.8.0
Am 16.07.2019 um 17:50 schrieb Andrea Berlingieri:
Let me introduce myself. I'm a CS student at the University of Bologna working
on the Virtualsquare
project (you can find some infos at
the wiki has not been updated yet with the last development efforts. The most
versions of the Virtualsquare projects are available on Github, at
and at http://github.com/rd235). In particular, I'm working on the vuos
project, the successor to
view-os. Basically the idea of vuos is to implement a partial virtual machine
by intercepting the
system calls a virtualized process makes and to execute userspace code in their
place. Vuos is
modular: this means that we can choose which system calls to virutalize and
which system call will
be directly forwarded to the underlying kernel. As of right now, the project is
made for Linux and
for the x86_64 architecture (because of the use vuos makes of ptrace to
intercept the system calls),
and the system call support is not complete yet (some system calls are not
supported at the moment),
but the project is still under active development.
In particular, I'm currently working on a submodule for the vunet module of
vuos, which virtualizes
the system calls of the BSD socket API (socket, bind, connect, etc.). This
submodule uses lwip as
the underlying network stack, instead of using Linux' TCP/IP stack. In the past
(2006-2011) a fork
of the lwip project was made to add support for IPv6 and some other features.
Now I'm trying to
write a new submodule based on the latest version of lwip, and in a way that
makes it possible, for
the two project (lwip and vuos), to develop independently while maintaining
compatibility (where the
submodule acts as a "bridge" between vuos and lwip).
In vuos both the modules and the submodules are compiled to obtain shared
objects, which are then
dynamically loaded/unloaded at need. So it is possible, for example, to insert
the vunet module of
vuos to virtualize the BSD socket API, and then to mount many different
networks stacks as
submodules. The virtualized processes can then decide which stack to use for
The way I'm tying to implement this with lwip is to compile lwip as a shared
library, through the
Unix port of lwip, and to load it dynamically with dlmopen, which lets me load
instances of the same dynamic library. This way a process can mount many
different instances of a
lwip TCP/IP stack, each with its own configuration and without interferences
between them. Then,
each system call of the socket API is intercepted and handled appropriately
(some calls are directly
redirected to their lwip counterpart, some require a little more work).
Now, working with lwip to implement the vunet submodule I've encountered some
problems that I would
like to discuss with you. I also have some expansion proposals, which I would
also like to discuss.
Here's a list:
- I've seen that in lwip there is a mechanism to specify some hooks that will
be called at the
appropriate moment. Among the currently defined hooks I unfortunately
couldn't find the one I
needed. That is, a hook that is called whenever there is an event on a
lwip_socket. I need it
because when an event occurs I need to "notify" the vuos core. The first
modification I made was
to add such a hook in opt.h, and the appropriate code in event_callback, in
sockets.c. As of
right now, the source code for the submodule is placed in a single source
vunetlwip.c. This file also contains the definition of the hook function.
Now, this initially
made for a strange arrangement, where vunetlwip.c depended on the headers
of lwip, and lwip was
linked against the vunetlwip.so shared object to link the definition of the
hook function. This,
unfortunately, didn't even work, as the hook function had references to
functions of other shared
objects of the vuos core that couldn't be linked with lwip without making a
mess. As of right
now, for testing purposes, the hook function code has been moved in lwip
(in a separate file) and
all it does is notify the events on a Unix domain socket, which is listened
to by a thread
created by the vunetlwip submodule, which, in turn, notifies the vuos core.
This is not ideal,
and I thought that the best solution in this case would be to add a
mechanism in lwip to specify
the hook functions at run time, instead of directly defining them at
compile time. So, my first
proposal would be to add such a mechanism and a hook for when there are
events on a socket.
I'm open to adding another hook, but you'll have to do the "specify the
hook functions at runtime" yourself. But I don't see a problem to do so:
just define the hook to be e function that checks if a real callback has
been registered. No need to move that into the actual lwIP code.
- I've seen that the sockaddr structures have been redefined in lwip. It
looks although that the
"model" after which they were made was that of BSD Unix, as each sockaddr
struct adds a len field
before the others in the struct. This kind of "breaks" in Linux, as the len
field is absent, and
using a sockaddr as defined in Linux with, for example, lwip_connect,
casuses a mismatch between
the fields of the two structs, and thus unexpected behavior. This happens
because the virtualized
process usually runs a program that assumed Linux as the target operating
system, and thus uses
sockaddr as defined in Linux when using bind, connect, sendto, etc. It
seems though that the len
field at the begininnig of the struct is non standard and non portable (as
I could understand
from this SO answer:
So, my second proposal would be to modify the sockaddr structs to either
remove the len field, or
to at least move it at the end of the structure, so that Linux' and lwip's
sockaddr agree at
least on the first part of the struct.
I don't have a problem with adding an option (e.g.
LWIP_SOCKETS_SOCKADDR_HAVE_SALEN). However, you'll probably find more
differences in the header files. Because of that, Joan has been working
on LWIP_SOCKET_EXTERNAL_HEADERS to replace our header with the original
OS header (Hurd in his case).
You might still need to get rid of 'sa_len' then...
- In the old lwip fork I mentioned above (which, from now on, I will refer to as
following features were added: a vde driver, to work with vde (also a
https://en.wikipedia.org/wiki/Virtual_Distributed_Ethernet), SLIRP protocol
support, and NAT
support. I think they would be some interesting additions to lwip. Let me
know if you're
interested in porting them upstream.
I'm not interested in porting anything upstream myself right now, but if
you want to send a patch, we could see if it's a match for lwIP that's
good to integrate.
But I'd be certainly interested in a link to the NAT implementation,
just to have a look ;-)
- I see that lwip is built mainly for embedded devices and one of its most
prominent features is
the low memory footprint. While I think this is all good and important, I
also think that it
would be nice to be able to choose between the lwip internal memory
management and the dynamic
memory management provided by the Linux kernel when compiling the lwip
shared library. In lwipv6
code was added to use the glibc's malloc when mem_malloc was called,
glibc's free when mem_free
was called, and so on. I think it would be a nice feature for the Unix lwip
library. The choice
could be made at compile time through some configuration constants.
That's available since ages: MEM_LIBC_MALLOC to use 'malloc' instead of
'mem_malloc' and MEMP_MEM_MALLOC to use heap allocationi instead of the
- I've seen that in lwip each netif is associated with a single ipv4 address,
a single netmask and
a single gateway. Now, in lwipv6 a modification was made to support lists
of ip addresses, so
that a single interface could be associated with more that one ip address.
Now, introducing this
feature to lwip would require also a modification to the route function,
although this wouldn't
seem to be too complicated, at least conceptually (as, instead of looking
at a single address, a
search on a list would be performed). So, my next proposal would be to add
I've written before that I'm very open to this. In fact, our IPv6
support provided this from the beginning. It might just not be easy to
keep all apps and ports compatible to such a change...
- Stack configuration (setting addresses, setting interfaces, getting
interfaces infos, etc) is
made through netlink in my vunetlwip module. Even ioctls are intercepted
and converted to the
appropriate netlink counterparts. Now, while working on the address
information retrieval (the
GETADDR message of the rtnetlink protocol), I couldn't be able to find the
prefix len for a given
ipv6 address. As of right now I simply assume that the prefix len is 64 and
return that, but that
is a strong assumption. This is mix between a proposal to add such a field
somewhere in lwip, and
a request of help on how to find this information, if it is already encoded
somewhere I couldn't
Yeah, well, after Ivan left the project, IPv6 got less attention than it
needed. Patches are welcome :-/
- In lwipv6 a modification was made to support multiple routes for a netif.
That is, for each netif
a list of routes was kept. It seems to me though that in lwip routing is
very simple: packets are
routed to the interface that has a matching ip address prefix to that of
the dest ip address (if
source address routing is not involved). My next proposal would be to add
such a feature to lwip,
as I think it may be useful, although it may make routing more complicated.
We have a hook for routing already. Such an improved implementation
could be provided in contrib/addons as a standard addon.
- Now, this last one is kind of low priority. Basically, while testing with
ping, I noticed that
the iputils version of ping for Linux uses a
socket(AF_INET,SOCK_DGRAM,IPPROTO_ICMP) call, in the
case of ipv4, to create a socket to send ICMP packets. Now, lwip assumes
that a socket that uses
SOCK_DGRAM as a type is a UDP socket. This caused some unexpected behavior.
As of right now I
simply intercept the socket call and transform a
into a socket(AF_INET,SOCK_RAW,IPPROTO_ICMP) call. This kind of fixes the
problem, but is not
ideal. My proposal would be to add support for this case of socket, if a
has been defined (it seems to me that this kind of calls are only made and
supported in Linux. In
this case the kernel handles the call in a special way, and the created
socket is only for ICMP
https://stackoverflow.com/questions/8290046/icmp-sockets-linux , third answer).
I'm not too sure about this, maybe we can discuss it a little.
From what you wrote: send a patch and I'm sure we can accept it.
So, this is all I've come up with as of right now. I would like to discuss
these topics with you.
I'm sorry for the long email. This is my first time posting to this mailing
list, so If I did
anything wrong let me know, so that it won't happen again. I wish everyone a
Well, the mail was too long ;-) At least to read it on the phone...
Other than that: you might have noticed we need more developers, so
please share your code!