Running lwip on AT91SAM7X-EK demo board.
Currently serving simple web page with a couple of graphics
(<6k, each).
Problem I am trying to solve is…
- PC browser issues “Get” to request the image
- my lwip implementation sends images in max 1460 byte pieces,
using netconn_write()
- consecutive pieces will not be sent, it will wait for an ACK
after each image piece before sending the next piece
- PC browser delays 100-200ms before ACK’ing image
piece
- result: slowwww transfer
Here is a wireshark capture of the above described.
Total comm time is ~650ms. The delays are noticable on lines 23, 26, 33.
The trace is displaying time since previous packet.
Time Source
Destination Info
1 0.000000 xxx.1 xxx.20 TCP 2274 >
http [SYN] Seq=58252479 Len=0 MSS=1460 WS=2
2 0.001272 xxx.20 xxx.1 TCP http >
2274 [SYN, ACK] Seq=6575 Ack=58252480 Win=4500 Len=0 MSS=1460
3 0.000027 xxx.1 xxx.20 TCP 2274 >
http [ACK] Seq=58252480 Ack=6576 Win=65535 Len=0
4 0.000077 xxx.1 xxx.20 HTTP GET / HTTP/1.0
5 0.002566 xxx.20 xxx.1 TCP http >
2274 [ACK] Seq=6576 Ack=58252669 Win=4500 Len=0
6 0.001216 xxx.20 xxx.1 HTTP HTTP/1.0 200
OK
7 0.005377 xxx.20 xxx.1 HTTP Continuation
or non-HTTP traffic
8 0.000034 xxx.1 xxx.20 TCP 2274 >
http [ACK] Seq=58252669 Ack=7030 Win=65081 Len=0
9 0.002182 xxx.20 xxx.1 TCP http >
2274 [FIN, ACK] Seq=7030 Ack=58252669 Win=4500 Len=0
10 0.000054 xxx.1 xxx.20 TCP 2274 > http
[ACK] Seq=58252669 Ack=7031 Win=65081 Len=0
11 0.013480 xxx.1 xxx.20 TCP 2275 > http
[SYN] Seq=1109786326 Len=0 MSS=1460 WS=2
12 0.001264 xxx.20 xxx.1 TCP http > 2275
[SYN, ACK] Seq=6640 Ack=1109786327 Win=4500 Len=0 MSS=1460
13 0.000052 xxx.1 xxx.20 TCP 2275 > http
[ACK] Seq=1109786327 Ack=6641 Win=65535 Len=0
14 0.001996 xxx.1 xxx.20 TCP 2274 > http
[FIN, ACK] Seq=58252669 Ack=7031 Win=65081 Len=0
15 0.001114 xxx.20 xxx.1 TCP http > 2274
[ACK] Seq=7031 Ack=58252670 Win=4499 Len=0
16 0.002381 xxx.1 xxx.20 HTTP GET /atmel_gif.gif
HTTP/1.0
17 0.001793 xxx.20 xxx.1 TCP http > 2275
[ACK] Seq=6641 Ack=1109786562 Win=4500 Len=0
18 0.001085 xxx.1 xxx.20 TCP 2276 > http
[SYN] Seq=2933660713 Len=0 MSS=1460 WS=2
19 0.001296 xxx.20 xxx.1 TCP http > 2276
[SYN, ACK] Seq=6705 Ack=2933660714 Win=4500 Len=0 MSS=1460
20 0.000028 xxx.1 xxx.20 TCP 2276 > http
[ACK] Seq=2933660714 Ack=6706 Win=65535 Len=0
21 0.000083 xxx.1 xxx.20 HTTP GET
/cmxlogo_gif.gif HTTP/1.0
22 0.024945 xxx.20 xxx.1 HTTP Continuation or
non-HTTP traffic
23 0.164954 xxx.1 xxx.20 TCP 2275 > http
[ACK] Seq=1109786562 Ack=8101 Win=65535 Len=0
24 0.003335 xxx.20 xxx.1 HTTP Continuation or
non-HTTP traffic
25 0.045983 xxx.20 xxx.1 TCP http > 2276
[ACK] Seq=6706 Ack=2933660951 Win=4263 Len=0
26 0.150967 xxx.1 xxx.20 TCP 2275 > http
[ACK] Seq=1109786562 Ack=9561 Win=65535 Len=0
27 0.003277 xxx.20 xxx.1 HTTP Continuation or
non-HTTP traffic
28 0.001152 xxx.20 xxx.1 TCP http > 2275
[FIN, ACK] Seq=10954 Ack=1109786562 Win=4500 Len=0
29 0.000032 xxx.1 xxx.20 TCP 2275 > http
[ACK] Seq=1109786562 Ack=10955 Win=64142 Len=0
30 0.005059 xxx.1 xxx.20 TCP 2275 > http
[FIN, ACK] Seq=1109786562 Ack=10955 Win=64142 Len=0
31 0.001225 xxx.20 xxx.1 TCP http > 2275
[ACK] Seq=10955 Ack=1109786563 Win=4499 Len=0
32 0.020671 xxx.20 xxx.1 HTTP Continuation or
non-HTTP traffic
33 0.168874 xxx.1 xxx.20 TCP 2276 > http
[ACK] Seq=2933660951 Ack=8166 Win=65535 Len=0
34 0.002844 xxx.20 xxx.1 HTTP Continuation or
non-HTTP traffic
35 0.001203 xxx.20 xxx.1 TCP http > 2276
[FIN, ACK] Seq=9193 Ack=2933660951 Win=4500 Len=0
36 0.000048 xxx.1 xxx.20 TCP 2276 > http
[ACK] Seq=2933660951 Ack=9194 Win=64508 Len=0
37 0.010313 xxx.1 xxx.20 TCP 2276 > http
[FIN, ACK] Seq=2933660951 Ack=9194 Win=64508 Len=0
38 0.001317 xxx.20 xxx.1 TCP http > 2276
[ACK] Seq=9194 Ack=2933660952 Win=4499 Len=0
I’ve been wrestling with this problem (I’m no
TCP/lwIP expert) for too long. It so happened today I moved my test
images from code constants to external DataFlash. In debugging serving
the images from the DF, my first few runs were sending the image in 528 byte pieces
(instead of the previous 1460 byte pieces) and the communication flew! –
dropped the transfer time from ~650ms to ~180ms because the image pieces were
being sent in pairs and the PC ACK delays disappeared.
I found that I could bump the max image packet size up to
730 bytes and stay successful. Any max size greater than 730 reintroduced
the ACK delays.
Below is a wireshark trace sending the image pieces in 730
byte packets – delays are gone.
Time Source Destination Info
1 0.000000 xxx.1 xxx.20 TCP 2293 >
http [SYN] Seq=1195339838 Len=0 MSS=1460 WS=2
2 0.001309 xxx.20 xxx.1 TCP http >
2293 [SYN, ACK] Seq=6531 Ack=1195339839 Win=4500 Len=0 MSS=1460
3 0.000026 xxx.1 xxx.20 TCP 2293 >
http [ACK] Seq=1195339839 Ack=6532 Win=65535 Len=0
4 0.000079 xxx.1 xxx.20 HTTP GET / HTTP/1.0
5 0.002566 xxx.20 xxx.1 TCP http >
2293 [ACK] Seq=6532 Ack=1195340028 Win=4500 Len=0
6 0.001214 xxx.20 xxx.1 HTTP HTTP/1.0 200
OK
7 0.005375 xxx.20 xxx.1 HTTP Continuation
or non-HTTP traffic
8 0.000042 xxx.1 xxx.20 TCP 2293 >
http [ACK] Seq=1195340028 Ack=6986 Win=65081 Len=0
9 0.002144 xxx.20 xxx.1 TCP http >
2293 [FIN, ACK] Seq=6986 Ack=1195340028 Win=4500 Len=0
10 0.000051 xxx.1 xxx.20 TCP 2293 > http
[ACK] Seq=1195340028 Ack=6987 Win=65081 Len=0
11 0.011685 xxx.1 xxx.20 TCP 2294 > http
[SYN] Seq=976702823 Len=0 MSS=1460 WS=2
12 0.001392 xxx.20 xxx.1 TCP http > 2294
[SYN, ACK] Seq=6552 Ack=976702824 Win=4500 Len=0 MSS=1460
13 0.000049 xxx.1 xxx.20 TCP 2294 > http
[ACK] Seq=976702824 Ack=6553 Win=65535 Len=0
14 0.003399 xxx.1 xxx.20 TCP 2293 > http
[FIN, ACK] Seq=1195340028 Ack=6987 Win=65081 Len=0
15 0.001176 xxx.20 xxx.1 TCP http > 2293
[ACK] Seq=6987 Ack=1195340029 Win=4499 Len=0
16 0.000295 xxx.1 xxx.20 HTTP GET /atmel_gif.gif
HTTP/1.0
17 0.001644 xxx.20 xxx.1 TCP http > 2294
[ACK] Seq=6553 Ack=976703059 Win=4500 Len=0
18 0.003806 xxx.1 xxx.20 TCP 2295 > http
[SYN] Seq=1012498949 Len=0 MSS=1460 WS=2
19 0.001299 xxx.20 xxx.1 TCP http > 2295
[SYN, ACK] Seq=6573 Ack=1012498950 Win=4500 Len=0 MSS=1460
20 0.000037 xxx.1 xxx.20 TCP 2295 > http
[ACK] Seq=1012498950 Ack=6574 Win=65535 Len=0
21 0.000102 xxx.1 xxx.20 HTTP GET
/cmxlogo_gif.gif HTTP/1.0
22 0.010825 xxx.20 xxx.1 HTTP Continuation or
non-HTTP traffic
23 0.012645 xxx.20 xxx.1 HTTP Continuation or
non-HTTP traffic
24 0.000060 xxx.1 xxx.20 TCP 2294 > http
[ACK] Seq=976703059 Ack=8013 Win=65535 Len=0
25 0.013861 xxx.20 xxx.1 HTTP Continuation or
non-HTTP traffic
26 0.012582 xxx.20 xxx.1 HTTP Continuation or
non-HTTP traffic
27 0.000055 xxx.1 xxx.20 TCP 2294 > http
[ACK] Seq=976703059 Ack=9473 Win=65535 Len=0
28 0.013557 xxx.20 xxx.1 HTTP Continuation or
non-HTTP traffic
29 0.012647 xxx.20 xxx.1 HTTP Continuation or
non-HTTP traffic
30 0.000041 xxx.1 xxx.20 TCP 2294 > http
[ACK] Seq=976703059 Ack=10866 Win=64142 Len=0
31 0.002168 xxx.20 xxx.1 TCP http > 2295
[ACK] Seq=6574 Ack=1012499187 Win=4263 Len=0
32 0.000697 xxx.20 xxx.1 TCP http > 2294
[FIN, ACK] Seq=10866 Ack=976703059 Win=4500 Len=0
33 0.000043 xxx.1 xxx.20 TCP 2294 > http
[ACK] Seq=976703059 Ack=10867 Win=64142 Len=0
34 0.003462 xxx.1 xxx.20 TCP 2294 > http
[FIN, ACK] Seq=976703059 Ack=10867 Win=64142 Len=0
35 0.001119 xxx.20 xxx.1 TCP http > 2294
[ACK] Seq=10867 Ack=976703060 Win=4499 Len=0
36 0.010695 xxx.20 xxx.1 HTTP Continuation or
non-HTTP traffic
37 0.012653 xxx.20 xxx.1 HTTP Continuation or
non-HTTP traffic
38 0.000029 xxx.1 xxx.20 TCP 2295 > http
[ACK] Seq=1012499187 Ack=8034 Win=65535 Len=0
39 0.013852 xxx.20 xxx.1 HTTP Continuation or
non-HTTP traffic
40 0.012152 xxx.20 xxx.1 HTTP Continuation or
non-HTTP traffic
41 0.000041 xxx.1 xxx.20 TCP 2295 > http
[ACK] Seq=1012499187 Ack=9061 Win=64508 Len=0
42 0.002149 xxx.20 xxx.1 TCP http > 2295
[FIN, ACK] Seq=9061 Ack=1012499187 Win=4500 Len=0
43 0.000050 xxx.1 xxx.20 TCP 2295 > http
[ACK] Seq=1012499187 Ack=9062 Win=64508 Len=0
44 0.005289 xxx.1 xxx.20 TCP 2295 > http
[FIN, ACK] Seq=1012499187 Ack=9062 Win=64508 Len=0
45 0.001457 xxx.20 xxx.1 TCP http > 2295
[ACK] Seq=9062 Ack=1012499188 Win=4499 Len=0
I thought I could turn the knobs in lwipopts.h and ‘blindly’
overcome this problem, but no luck. 730 bytes seems to be a ‘magical’
number despite changing MSS/MTU based references to values other than 1460.
Are the long ACK delays from the PC waiting for something
that doesn’t arrive or doesn’t arrive in time?
Any ideas where the 730 byte limitation is coming from? –
what knob to turn to prove it?
Any help would be greatly appreciated – let me know if
I left out any pieces of the puzzle.
Thanks,
Alan
Below is my current lwipopts.h snippet – some of the
values may be skewed from my blind, iterative solution without thinking (it’s
Friday afternoon).
#define
MEM_ALIGNMENT 4
/* MEM_SIZE: the size of the heap memory. If the application will send
a lot of data that needs to be copied, this should be set high. */
#define
MEM_SIZE
2000
/* MEMP_NUM_PBUF: the number of memp struct pbufs. If the application
sends a lot of data out of ROM (or other static memory), this
should be set high. */
#define MEMP_NUM_PBUF
20
/* MEMP_NUM_UDP_PCB: the number of UDP protocol control blocks. One per
active UDP "connection". */
#define
MEMP_NUM_UDP_PCB 4
/* MEMP_NUM_TCP_PCB: the number of simulatenously active TCP
connections. */
#define
MEMP_NUM_TCP_PCB 10
/* MEMP_NUM_TCP_PCB_LISTEN: the number of listening TCP connections. */
#define MEMP_NUM_TCP_PCB_LISTEN 8
/* MEMP_NUM_TCP_SEG: the number of simultaneously queued TCP segments.
*/
#define
MEMP_NUM_TCP_SEG 8
/* MEMP_NUM_SYS_TIMEOUT: the number of simulateously active timeouts.
*/
#define
MEMP_NUM_SYS_TIMEOUT 3
/* The following four are used only with the sequential API and can be
set to 0 if the application only will use the raw API. */
/* MEMP_NUM_NETBUF: the number of struct netbufs. */
#define
MEMP_NUM_NETBUF 4
/* MEMP_NUM_NETCONN: the number of struct netconns. */
#define
MEMP_NUM_NETCONN 4
/* MEMP_NUM_APIMSG: the number of struct api_msg, used for
communication between the TCP/IP stack and the sequential
programs. */
#define
MEMP_NUM_API_MSG 8
/* MEMP_NUM_TCPIPMSG: the number of struct tcpip_msg, which is used
for sequential API communication and incoming packets. Used in
src/api/tcpip.c. */
#define
MEMP_NUM_TCPIP_MSG 8
/* These two control is reclaimer functions should be compiled
in. Should always be turned on (1). */
#define
MEM_RECLAIM
1
#define
MEMP_RECLAIM 1
/* ---------- Pbuf options ---------- */
/* PBUF_POOL_SIZE: the number of buffers in the pbuf pool. */
#define
PBUF_POOL_SIZE 4
//#define
PBUF_POOL_SIZE 4
/* PBUF_POOL_BUFSIZE: the size of each pbuf in the pbuf pool. */
#define
PBUF_POOL_BUFSIZE 1500
/* PBUF_LINK_HLEN: the number of bytes that should be allocated for a
link level header. */
#define
PBUF_LINK_HLEN 16
/* ---------- TCP options ---------- */
#define
LWIP_TCP
1
#define
TCP_TTL
255
/* Controls if TCP should queue segments that arrive out of
order. Define to 0 if your device is low on memory. */
#define
TCP_QUEUE_OOSEQ 1
/* TCP Maximum segment size. */
#define
TCP_MSS
1500
/* TCP sender buffer space (bytes). */
//#define
TCP_SND_BUF
4096
#define
TCP_SND_BUF
3*TCP_MSS
//#define
TCP_SND_BUF
1500
/* TCP sender buffer space (pbufs). This must be at least = 2 *
TCP_SND_BUF/TCP_MSS for things to work. */
//#define TCP_SND_QUEUELEN 16
#define
TCP_SND_QUEUELEN 6 * TCP_SND_BUF/TCP_MSS
/* TCP receive window. */
#define
TCP_WND
3*TCP_MSS
//#define
TCP_WND
1500