RE: [lwip-users] performance in sending / receiving + tcp_tmr

[Taranowski, Thomas \(SWCOE\)]

> -----Original Message-----
> From: address@hidden
> [mailto:address@hidden
On
> Behalf Of Robert Goeffringmann
> Sent: Friday, February 16, 2007 5:10 PM
> To: Mailing list for lwIP users
> Subject: Re: [lwip-users] performance in sending / receiving + tcp_tmr
> 
> Hi Thomas,
> 
> >I'm curious to know what sort of throughput numbers you get for UDP
> >transfers.
> Er, yes, me too. Apparently making me try that was a good idea,
suddenly I
> get all kinds of weird bugs. 50% of the packets have incorrect
checksums
> and
> like 25% of the packets get sent twice for unknown reasons. I suppose
I
> have
> quite a number of new issues to look into now. Weird, considering that
> everything is working perfectly fine for tcp...

[TT] That is peculiar.  Not sure what to make of that.

> 
> >I looked at your configuration as well and thought it looked fine to
me,
> >but I'm no expert.  Perhaps the resolution of your system tick is
having
> >some effect.  What's tick rate is your OS using?
> Are you referring to sys_arch_sem_wait, etc. here?
> mh, well, I have a timer interrupt handler running that gets called
every
> 10ms and checks if there're any waits on semaphores that have to get
> canceled, so this is not _too_ accurate. The number of ms that gets
> returned
> is accurate however, as I compute it based on the cpu cycle counters. 

[TT] Sometimes those counters can be pretty crude, with pretty rough
resolution.  You might also check the wrapping on your counter.  I use a
similar method, and found my counter variable was wrapping every 4
seconds, which I didn't handle, and was causing some really weird bugs.
Once I took into account for potential wrap, it solved some odd errors.

But
> when there're TCP transfers going on, the functions (correctly) return
0ms
> in 99.999% of all cases. Even though all the waits together certainly
sum
> up
> to quite a couple of ms. lwip's timer functions get called _very_
rarely
> then. (e.g. tcp_slowtmr, which is supposed to get called every 500ms,
> seems
> to get called once in 5-10 seconds.)
> 

[TT] I was wondering what the period of the system tick was.  If the
kernel is scheduling at each system tick (which is typical), it may be
that the system tick is at some peculiar period, like twice a second, or
something really slow.  An unusually slow system tick could impact
timing, although I'm not sure it could make 500ms become 5-10 seconds,
unless something is really wrong.

> >I'm not really familiar with the playstation 2 kernel, but it seems
like
> >you would be able to access the scheduling history of the kernel, and
> >view it somehow (hopefully with a nice GUI interface).
> Maybe I would, if I had a dev ps2 and the Sony SDK. With a retail PS2
and
> the open source sdk, printf is about the only interface I have. :)

[TT] I feel your pain :)  

> 
> >It might be interesting to see how your threads are being scheduled.
It
> >almost
> >sounds like the highest priority thread is starving out all of your
> >lower priority threads.
> That would be rather difficult to check...
> As I mentioned in a different eMail, I wrote an assembly function for
> computing checksums which is ~8 times as fast as lwip's default one.
> However, it doesn't matter which one I use, I end up with the same
> transfer
> speed for both functions, so I doubt that one thread is starving out
all
> the
> other ones...
> but even if, what would I do then?
> 

[TT] It would be more of a diagnostic tool, to make sure the threads
were running at the rates you expect.  

> >In regards to the tcp timer thread...
> >I was under the assumption that, when tcpip_init() was called, it
> >started one of the internal lwIP timers to act as the tcp timer.
> Yes... it seems you are right about that. I disabled my own timer
thread,
> but that led to the behavious described above (both timers being
called
> _very_ rarely).
> However, it didn't seem to have any actual effect on my tcp
connections at
> all.
> Do you suppose I should change lwip's sys_sem_wait / sys_mbox_fetch in
> order
> to make it call its callbacks with the right timing or shouldn't I
bother
> at
> all?
> 
> Robert.
> 

[TT] When I had incorrectly implemented the timer thread, tcp seemed to
work almost ok, except that everything had the PSH flag set, as seen on
the wire.  I also had some odd duplicate packets, and probably other odd
behavior I didn't bother checking out.  I haven't checked to see how
often my timers are called.  It's something I should probably check out.



> 
> 
> ----- Original Message -----
> From: "Taranowski, Thomas (SWCOE)" <address@hidden>
> To: "Mailing list for lwIP users" <address@hidden>
> Sent: Thursday, February 15, 2007 11:15 PM
> Subject: RE: [lwip-users] performance in sending / receiving + tcp_tmr
> 
> 
> I'm not really familiar with the playstation 2 kernel, but it seems
like
> you would be able to access the scheduling history of the kernel, and
> view it somehow (hopefully with a nice GUI interface).  It might be
> interesting to see how your threads are being scheduled.  It almost
> sounds like the highest priority thread is starving out all of your
> lower priority threads.
> 
> In regards to the tcp timer thread...
> I was under the assumption that, when tcpip_init() was called, it
> started one of the internal lwIP timers to act as the tcp timer.
> 
> In one of my first port implementations, I was calling tcpip_init(),
> then starting a timer thread which called tcp_tmr() periodically.
> However, I saw  that tcpip_init() spawns the tcpip_thread, which calls
> tcp_init, which calls tcpip_tcp_timer, which basically calls tcp_tmr
at
> regular intervals.  This was doing the same thing as my timer thread,
> which periodically called the tcp_tmr() routine.
> 
> In essence I was calling tcp_tmr() at twice the rate at which it was
> needed, and who knows what other effect it was having.  Once I removed
> the timer thread, tcp started acting normally again and life was good.
> 
> It sounds like you might be running into the same issue I faced, with
> tcp_tmr being called at double the rate.  Once from inside the lwIP
> tcpip thread, and once from the timer thread.  Of course, I could be
> totally misunderstanding everything, in which case I hope someone sets
> me straight.
> 
> -Tom
> 
> 
> > -----Original Message-----
> > From: address@hidden
> >
[mailto:address@hidden
> On
> > Behalf Of Robert Goeffringmann
> > Sent: Thursday, February 15, 2007 7:03 AM
> > To: Mailing list for lwIP users
> > Subject: Re: [lwip-users] performance in sending / receiving
> >
> > Hi Kieran,
> > I'm sorry for the delay in getting back to you, I've been busy with
> exams
> > lately. :/
> >
> > > Having had a look at your ethereal captures I see it advertising
up
> to
> > > around 42KB.  A full capture (from the start of a connection) in
> pcap
> > > rather than txt format would enable me to look at this more
> precisely.
> >
> > > Actually, in this case (with the benefit of looking at your packet
> > > capture) it looks like the stack is processing the packets at a
> > > reasonable rate (the ACK numbers are increasing) but the window is
> still
> > > decreasing.  This suggests that the packets are being buffered in
> the
> > > sockets API layer, and the application is not reading them fast
> enough.
> > > ie. This could be a scheduling issue.
> >
> > That'd make sense.
> > The PS2 kernel doesn't do any automatic scheduling, it'll simply
> execute
> > the
> > thread with the highest priority which is not waiting for a
semaphore
> or
> > messagebox at the moment.
> >
> > right now I have the threads structured like this (highest to lowest
> > priority):
> >
> > - highest priority is the thread for the HW interface, it takes
lwip's
> > packets from a queue and passes them to the HW (or reads them from
the
> HW
> > and passes them to lwip...)
> > - the next thread is the timer thread, which only calls tcp_tmr and
> > etharp_tmr periodically
> > - then there's lwip's tcp/ip thread
> > - and the lowest thread is the application doing the the send/recv
> calls.
> >
> > This structure has worked pretty well for sending data, but indeed
it
> did
> > seem to cause those zero-windows.
> > I've changed it and made the tcp/ip thread the one with the lowest
> > priority
> > and it fixed that... but unfortunately it has rather little
influence
> on
> > the
> > receive speed.
> > With the application having the lowest priority, I had transfer
rates
> of
> > ~1,6MB/s (average), with the actual transmission speed varying
between
> 1,3
> > and 2,0 MB/s.
> > If the tcp/ip thread has the lowest priority, the average speed is
> > ~1,8MB/s,
> > varying between 1,0 and 1,9 MB/s.
> >
> > I uploaded the pcap files and lwip's statistics here:
> > http://www.goeffringmann.de/lwip/thprio.zip
> >
> > the prio1 file is the one with the application having the lowest
> priority,
> > in prio50 it's the tcp/ip thread.
> >
> > Is there anything like a suggested model for those thread
priorities?
> >
> > I also wrote a checksum routine in assembly, but no matter if I use
> that
> > one
> > or lwip's most basic one doesn't have any difference at all on
either
> > sending- or receiving speed. So I suppose I can assume that cpu
power
> is
> > not
> > the bottleneck... right?
> >
> > Thanks a lot for your help,
> >
> > Robert.
> >
> >
> >
> >
> >
> > _______________________________________________
> > lwip-users mailing list
> > address@hidden
> > http://lists.nongnu.org/mailman/listinfo/lwip-users
> 
> 
> _______________________________________________
> lwip-users mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/lwip-users
> 
> 
> 
> _______________________________________________
> lwip-users mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/lwip-users