[lwip-devel] Granularity of core locking

[Tim Cussins]

Hi all,

I've got a bit of a performance issue, and I'd appreciate any thoughts
on it.

We've been using lwip for a while, and picked up a substantial
throughput improvement when we enabled both LWIP_TCPIP_CORE_LOCKING and
LWIP_TCPIP_CORE_LOCKING_INPUT.

However, we're experiencing some problems that appear to be a
consequence of the course-grained approach of the current core locking
strategy.

In our particular case, low priority threads sending over tcp take the
core lock. When our emac rx thread wants to push packets into the stack,
the low priority thread inherits the rx thread priority and has it's
(relatively) long-running and unimportant workload promoted. 

All of this, of course, is normal and expected behaviour, but it creates
a problem for thread priority assignment. We want the rx thread to be
high priority to service hardware, but we don't want low priority client
threads getting promoted ahead of it for longer than absolutely
necessary.

The "big core lock" approach is a good approach, especially for a
network stack where the code understands that it has exclusive access to
stack data. But much of the work performed while holding the lock
probably doesn't /require/ the lock: memcpy and checksum operations for
example, represent memory/cpu-intensive work that doesn't mutate network
stack data and don't require the lock to be held.

So here are my questions:

(1) Has any thought/discussion been put towards finer-grained locking in
lwip?

  I understand that more locks or more lock-taking aren't appropriate
  for every deployment.

(2) I need a solution asap :) Can you suggest a class of "safe places"
to flash the lock?

  By flash I mean:

  UNLOCK_TCPIP_CORE();
  LOCK_TCPIP_CORE();

  which would allow the truly higher-priority thread to get its work
  done.

I tried releasing the lock within 'inet_cksum_pseudo_base', but things
broke: I expect that any kind of momentary release of the lock is an
approach that, while uninvasive, is probably fraught with difficulty.

Cheers,
Tim