[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [lwip-users] How to use LWIP from FreeRTOS tasks - a thread safety q
|
From: |
Peter |
|
Subject: |
Re: [lwip-users] How to use LWIP from FreeRTOS tasks - a thread safety question |
|
Date: |
Tue, 12 Jul 2022 14:58:05 +0100 |
Hello Dave,
Thanks for your reply.
Yes it all works. I have just been tying up loose ends, partly as a
result of writing the documentation for the project :)
I've been doing embedded, mostly asm, since late 1970s (Z80 etc) and
even wrote my own simple RTOS back then. I just didn't get into
ethernet/IP etc until this project. So the ETH+LWIP+FreeRTOS+MbedTLS
stuff was done by the "Monday guy" who has done this before (currently
he is also doing an ESP32 project for somebody else and apparently
everything just works on that because Espressif employed somebody on
the forums who seemed to "know it all"). I got involved 1.5 years ago
when it became clear that it would never get done otherwise.
With an RTOS, one can avoid thread-safety issues by keeping all
relevant stuff in one thread (and perhaps using messaging to enable
other threads to access the resources) but this is a poor way to do
things for the future so I have started digging around the thread
safety issues.
LWIP_TCPIP_CORE_LOCKING=1 does not actually crash the system. It just
breaks something in getting the DHCP IP, which remains at 0. I will
have to wait until next Monday for help :)
Indeed ST code is buggy and the Monday guy spent a year or two of
Mondays trawling the internet for the many patches. I have just seen
this
https://community.st.com/s/question/0D53W00001Gi9BoSAJ/ethernet-hal-driver-reworked-by-st-and-available-in-22q1-preview-now-available-on-github
and while it is probable that most relevant stuff there has already
been implemented, we will review it at some point.
As I said, the project runs well. It has DNS, DHCP, NTP, HTTP (simple
server and client), HTTPS (MbedTLS client, single certificate only
although I know there is a patch for processing a string of certs
without needing enough RAM to hold them all), and we are communicating
with other devices. So I think "enough has been patched" to do the
job.
This is my own small business. In a bigger corporate environment one
would go berserk trying to work this way, but we don't have the money
and a lot of open source software is like that.
That UDP socket issue is probably something unrelated; what I find
suspicious is that absolutely none of the values in lwipopts make the
slightest difference. Closing that UDP socket removes the issue. The
issue came to light only because there is one process which is a
server and thus leaves the socket open (DNS and DHCP are clients and
close their UDP sockets after use). My provisional hack is to detect
HTTP server activity and set a flag which this UDP server task uses to
close the socket, and it re-opens it on a timeout (the HTTP server is
for occassional config use only so this is acceptable). It just
concerns me that this could be hiding some other issue but all else
seems to run ok.
I have spent much time on general memory policy and especially
FreeRTOS stacks. I have a graphical utility which shows the entire
FreeRTOS memory block (64k, in the 32F417 CCM) and there no no stack
issue.
The lwipopts.h file follows.
Regards,
Peter
>/**
>
> ******************************************************************************
> * @file LwIP/LwIP_HTTP_Server_Netconn_RTOS/Inc/lwipopts.h
> * @author MCD Application Team
> * @brief lwIP Options Configuration.
>
> ******************************************************************************
>*
>* This sort of explains the memory usage
>* https://lwip-users.nongnu.narkive.com/dkzkPa8l/lwip-memory-settings
>* https://www.cnblogs.com/shangdawei/p/3494148.html
>*
>* Receive data:
>* This uses PBUF_POOL_SIZE (which is a number of buffers, each
>PBUF_POOL_BUFSIZE in size).
>* We dont need a whole lot of buffering here because the RX data is
>polled, not
>* interrupt-driven. This is deliberate, for simplicity and DOS
>protection. This polling
>* is done in ethernetif_input() which loops around all the time there is
>RX data to be
>* processed, and if there isnt, yielding to RTOS for 20ms. The PBUF size
>is MTU+headers
>* bytes, and with POOL_SIZE being 3, the RX queue can hold a max of 3
>packets regardless
>* of their size. One could use a smaller PBUF size and then you end up
>with more buffers
>* if smaller packets are being sent, but most recommendations are to make
>this a full
>* MTU+headers. It is reasonable to assume that applications needing max
>speed will be
>* sending full size packets. How much of a bottleneck the polling is,
>depends on how
>* fast the host can send data to the KDE; if it can send packets solidly
>then the 20ms
>* yield will never be taken, otherwise this yield will slow it down a
>bit. If this is an
>* issue one could make the wait 10ms. However, low_level_input() uses
>memcpy() to move
>* the actual data to LWIP and this should be replaced with DMA.
>* With PBUF_POOL_SIZE=3, we use about 4.5k for RX buffering here.
>*
>* Transmit data: This uses MEM_SIZE which is a private heap used by LWIP.
>* This is more complicated.
>*
>*
>*
>*
>*
>* 7/7/22 PH MEM_SIZE set to 5k (was 10k). Only ~1.5k is used.
>ram_heap is 0x20002930.
>*
>*
>*
>*
>*
>*
>*
> */
>#ifndef __LWIPOPTS_H__
>#define __LWIPOPTS_H__
>
>/**
> * NO_SYS==1: Provides VERY minimal functionality. Otherwise,
> * use lwIP facilities.
> */
>#define NO_SYS 0
>
>// Thread safety mode
>#define LWIP_TCPIP_CORE_LOCKING 1
>
>/* STM32CubeMX Specific Parameters (not defined in opt.h)
>---------------------*/
>/* Parameters set in STM32CubeMX LwIP Configuration GUI -*/
>/*----- WITH_RTOS enabled (Since FREERTOS is set) -----*/
>#define WITH_RTOS 1
>/*----- WITH_MBEDTLS enabled (Since MBEDTLS and FREERTOS are set) -----*/
>#define WITH_MBEDTLS 1
>
>
>//NC: Need for sending PING messages by keepalive
>#define LWIP_RAW 1
>#define DEFAULT_RAW_RECVMBOX_SIZE 4
>
>/*-----------------------------------------------------------------------------*/
>
>/* LwIP Stack Parameters (modified compared to initialization value in opt.h)
>-*/
>/* Parameters set in STM32CubeMX LwIP Configuration GUI -*/
>
>/*----- Value in opt.h for LWIP_DNS: 0 -----*/
>#define LWIP_DNS 1
>
>/* ---------- Memory options ---------- */
>/* MEM_ALIGNMENT: should be set to the alignment of the CPU for which
> lwIP is compiled. 4 byte alignment -> define MEM_ALIGNMENT to 4, 2
> byte alignment -> define MEM_ALIGNMENT to 2. */
>#define MEM_ALIGNMENT 4
>
>/* MEM_SIZE: the size of the heap memory. If the application will send
>a lot of data that needs to be copied, this should be set high. */
>#define MEM_SIZE (5*1024)
>
>/* MEMP_NUM_PBUF: the number of memp struct pbufs. If the application
> sends a lot of data out of ROM (or other static memory), this
> should be set high. */
>#define MEMP_NUM_PBUF 5 //10
>/* MEMP_NUM_UDP_PCB: the number of UDP protocol control blocks. One
> per active UDP "connection". */
>#define MEMP_NUM_UDP_PCB 10 //6
>/* MEMP_NUM_TCP_PCB: the number of simulatenously active TCP
> connections. */
>// This controls how much of the mem_size area gets filled up with http etc
>packets.
>#define MEMP_NUM_TCP_PCB 5 //10
>/* MEMP_NUM_TCP_PCB_LISTEN: the number of listening TCP
> connections. */
>#define MEMP_NUM_TCP_PCB_LISTEN 5
>/* MEMP_NUM_TCP_SEG: the number of simultaneously queued TCP
> segments. */
>#define MEMP_NUM_TCP_SEG 8
>/* MEMP_NUM_SYS_TIMEOUT: the number of simulateously active
> timeouts. */
>#define MEMP_NUM_SYS_TIMEOUT 10
>
>
>/* ---------- Pbuf options - for RECEIVE data ---------- */
>/* PBUF_POOL_SIZE: the number of buffers in the pbuf pool. */
>// No need for too many here since RX is done by polling (see
>ethernetif_input())
>// so there is a bottleneck elsewhere
>#define PBUF_POOL_SIZE 3
>
>/* PBUF_POOL_BUFSIZE: the size of each pbuf in the pbuf pool. */
>#define PBUF_POOL_BUFSIZE 1500 + PBUF_LINK_ENCAPSULATION_HLEN +
>PBUF_LINK_HLEN + PBUF_IP_HLEN + PBUF_TRANSPORT_HLEN
>
>
>/* ---------- TCP options ---------- */
>#define LWIP_TCP 1
>#define TCP_TTL 255
>
>/* Controls if TCP should queue segments that arrive out of
> order. Define to 0 if your device is low on memory. */
>#define TCP_QUEUE_OOSEQ 0
>
>/* TCP Maximum segment size. */
>#define TCP_MSS (1500 - 40) /* TCP_MSS = (Ethernet MTU -
>IP header size - TCP header size) */
>
>/* TCP sender buffer space (bytes). */
>#define TCP_SND_BUF (4*TCP_MSS)
>
>/* TCP_SND_QUEUELEN: TCP sender buffer space (pbufs). This must be at least
> as much as (2 * TCP_SND_BUF/TCP_MSS) for things to work. */
>
>#define TCP_SND_QUEUELEN (2* TCP_SND_BUF/TCP_MSS)
>
>/* TCP advertised receive window. */
>// Should be less than PBUF_POOL_SIZE * (PBUF_POOL_BUFSIZE - protocol headers)
>#define TCP_WND (2*TCP_MSS)
>
>
>/* ---------- ICMP options ---------- */
>#define LWIP_ICMP 1
>
>
>/* ---------- DHCP options ---------- */
>#define LWIP_DHCP 1
>
>
>/* ---------- UDP options ---------- */
>#define LWIP_UDP 1
>#define UDP_TTL 255
>
>
>/* ---------- Statistics options ---------- */
>#define LWIP_STATS 0
>
>/* ---------- link callback options ---------- */
>/* LWIP_NETIF_LINK_CALLBACK==1: Support a callback function from an interface
> * whenever the link changes (i.e., link down)
> */
>#define LWIP_NETIF_LINK_CALLBACK 1
>
>
>/*
> --------------------------------------
> ---------- Checksum options ----------
> --------------------------------------
>*/
>
>/*
>The STM32F4xx allows computing and verifying the IP, UDP, TCP and ICMP
>checksums by hardware:
> - To use this feature let the following define uncommented.
> - To disable it and process by CPU comment the the checksum.
>*/
>#define CHECKSUM_BY_HARDWARE
>
>
>#ifdef CHECKSUM_BY_HARDWARE
> /* CHECKSUM_GEN_IP==0: Generate checksums by hardware for outgoing IP
> packets.*/
> #define CHECKSUM_GEN_IP 0
> /* CHECKSUM_GEN_UDP==0: Generate checksums by hardware for outgoing UDP
> packets.*/
> #define CHECKSUM_GEN_UDP 0
> /* CHECKSUM_GEN_TCP==0: Generate checksums by hardware for outgoing TCP
> packets.*/
> #define CHECKSUM_GEN_TCP 0
> /* CHECKSUM_CHECK_IP==0: Check checksums by hardware for incoming IP
> packets.*/
> #define CHECKSUM_CHECK_IP 0
> /* CHECKSUM_CHECK_UDP==0: Check checksums by hardware for incoming UDP
> packets.*/
> #define CHECKSUM_CHECK_UDP 0
> /* CHECKSUM_CHECK_TCP==0: Check checksums by hardware for incoming TCP
> packets.*/
> #define CHECKSUM_CHECK_TCP 0
> /* CHECKSUM_CHECK_ICMP==0: Check checksums by hardware for incoming ICMP
> packets.*/
> #define CHECKSUM_GEN_ICMP 0
>#else
> /* CHECKSUM_GEN_IP==1: Generate checksums in software for outgoing IP
> packets.*/
> #define CHECKSUM_GEN_IP 1
> /* CHECKSUM_GEN_UDP==1: Generate checksums in software for outgoing UDP
> packets.*/
> #define CHECKSUM_GEN_UDP 1
> /* CHECKSUM_GEN_TCP==1: Generate checksums in software for outgoing TCP
> packets.*/
> #define CHECKSUM_GEN_TCP 1
> /* CHECKSUM_CHECK_IP==1: Check checksums in software for incoming IP
> packets.*/
> #define CHECKSUM_CHECK_IP 1
> /* CHECKSUM_CHECK_UDP==1: Check checksums in software for incoming UDP
> packets.*/
> #define CHECKSUM_CHECK_UDP 1
> /* CHECKSUM_CHECK_TCP==1: Check checksums in software for incoming TCP
> packets.*/
> #define CHECKSUM_CHECK_TCP 1
> /* CHECKSUM_CHECK_ICMP==1: Check checksums by hardware for incoming ICMP
> packets.*/
> #define CHECKSUM_GEN_ICMP 1
>#endif
>
>
>/*
> ----------------------------------------------
> ---------- Sequential layer options ----------
> ----------------------------------------------
>*/
>/**
> * LWIP_NETCONN==1: Enable Netconn API (require to use api_lib.c)
> */
>#define LWIP_NETCONN 1
>
>/*
> ------------------------------------
> ---------- Socket options ----------
> ------------------------------------
>*/
>/**
> * LWIP_SOCKET==1: Enable Socket API (require to use sockets.c)
> */
>#define LWIP_SOCKET 1
>
>/*
> ------------------------------------
> ---------- httpd options ----------
> ------------------------------------
>*/
>/** Set this to 1 to include "fsdata_custom.c" instead of "fsdata.c" for the
> * file system (to prevent changing the file included in CVS) */
>#define HTTPD_USE_CUSTOM_FSDATA 0
>
>/*
> ---------------------------------
> ---------- OS options ----------
> ---------------------------------
>*/
>
>#define TCPIP_THREAD_NAME "TCP/IP"
>#define TCPIP_THREAD_STACKSIZE 4096
>#define TCPIP_MBOX_SIZE 6
>#define DEFAULT_UDP_RECVMBOX_SIZE 6
>#define DEFAULT_TCP_RECVMBOX_SIZE 6
>#define DEFAULT_ACCEPTMBOX_SIZE 6
>#define DEFAULT_THREAD_STACKSIZE 512
>#define TCPIP_THREAD_PRIO osPriorityHigh
>
>#define LWIP_DEBUG 1
>
>#define IP_DEBUG LWIP_DBG_ON
>#define DHCP_DEBUG LWIP_DBG_OFF
>#define UDP_DEBUG LWIP_DBG_ON
>#define SOCKET_DEBUG_LWIP_DBG_ON
>//#define ICMP_DEBUG LWIP_DBG_ON|LWIP_DBG_TRACE
>//#define NETIF_DEBUG LWIP_DBG_OFF
>#define LWIP_DBG_TYPES_ON (LWIP_DBG_TRACE|LWIP_DBG_STATE)
>#define LWIP_SO_RCVTIMEO 1
>#define LWIP_NETIF_HOSTNAME 1
>
>#define SO_REUSE 1
>
>// Defining these produces various errors
>//#define LWIP_IPV6 1
>//#define LWIP_IPV6_DHCP6 1
>
>#endif /* __LWIPOPTS_H__ */
>
>Hi Peter - Can you start at the beginning?
>Has this ever worked, or for 2 years it always crashed if you used LWIP?
>What exactly "crashes the whole thing"?
>You're aware of the numerous issues surrounding STM-provided Ethernet
>driver?
>Hope this helps,
>Best Regards, Dave
>
>On 7/12/2022 5:32 AM, Peter wrote:
>> Hi All,
>>
>> I am new here. I've been working on a ARM32 32F417 project for a
>> couple of years. Someone else, working 1 day a week, implemented
>> FreeRTOS, LWIP and MbedTLS and I am now revisiting these issues which
>> I don't think were looked at in detail originally.
>>
>> Specifically I would like to know what LWIP settings are needed to
>> achieve thread safety.
>>
>> For example I have one thread doing NTP (socket() call to get a UDP
>> socket) and another thread doing a simple HTTP server (using the
>> netconn API which seems to end up in the same "socket" API as the
>> foregoing).
>>
>> Reading the vast number of mostly inconclusive posts on the internet,
>> there appear to be two classes of API functions, and thread safety is
>> more easily achieved if you restrict yourself to just one of these
>> classes. But the advice is ambiguous.
>>
>> It looks like the implementer is required to provide a mutex set/reset
>> function, although the need for this depends on which of the two above
>> API sections one is using. I also appears that if you use just one
>> class, the mutex is not necessary and just using the "critical
>> section" macros (which basically just disable interrupts) is
>> sufficient.
>>
>> OTOH a lot of online text says that LWIP is thread safe "out of the
>> box" in some cases...
>>
>> I am using Cube IDE and have been stepping through the code but I am
>> not much wiser.
>>
>> ===
>>
>> The idea appears to have
>> NO_SYS 0
>> LWIP_TCPIP_CORE_LOCKING 1
>>
>> but that crashes the whole thing, presumably because nobody has
>> implemented any of the above mentioned hooks.
>>
>> FreeRTOS can supply mutexes and these work very well. I use them all
>> over the place.
>>
>> Looks like you have to implement these functions
>>
>> #define LWIP_MEM_FREE_PROTECT() sys_mutex_lock(&mem_mutex)
>> #define LWIP_MEM_FREE_UNPROTECT() sys_mutex_unlock(&mem_mutex)
>>
>> There are what appear to be example functions in sys_arch.c under
>> Mutexes.
>>
>> I find all this very confusing because e.g. here
>> https://www.nongnu.org/lwip/2_0_x/pitfalls.html
>> it states that
>>
>> In OS mode, Callback-style APIs AND Sequential-style APIs can be used.
>> Sequential-style APIs are designed to be called from threads other
>> than the TCPIP thread, so there is nothing to consider here. The
>> implication is that these do not need mutex protection. And indeed
>> stepping through code such as
>>
>> fd = socket(AF_INET, SOCK_DGRAM, 0);
>>
>> does find that a mutex is used, deep down.
>>
>> it calls conn = (struct netconn *)memp_malloc(MEMP_NETCONN);
>>
>> which does SYS_ARCH_PROTECT(old_level);
>> but I can't see that defined anywhere so that is probably what needs
>> to be done if you are setting
>>
>> LWIP_TCPIP_CORE_LOCKING 1
>>
>> What I can't find anywhere is whether the "sequential" functions are
>> thread-safe with or without LWIP_TCPIP_CORE_LOCKING.
>>
>> And where does SYS_ARCH_PROTECT come into all this, since it seems
>> unrelated to LWIP_TCPIP_CORE_LOCKING.
>>
>> There is also a subtle issue which I am sure is unrelated to thread
>> safety whereby if a single UDP socket is opened, it causes the HTTP
>> server to be unable to transmit packets (lots of LWIP error
>> counters++) when TLS is doing to crypto handshake (2-3 secs). This
>> happens on the mere allocation of the UDP socket; no data being sent.
>> Debugging this is too complicated because nobody I know knows where to
>> look. It makes no difference what values one puts in the lwipopts.h
>> file. I could post the file here but it would make a long post amd
>> those tend to not produce replies :)
>>
>> Thank you in advance for any help.
>>
>> Peter
>>
>> _______________________________________________
>> lwip-users mailing list
>> lwip-users@nongnu.org
>> https://lists.nongnu.org/mailman/listinfo/lwip-users