[lwip-devel] LwIP SMP Experience

[Joel Cunningham]

Starting a new thread rather than highjack: http://lists.nongnu.org/archive/html/lwip-devel/2017-03/msg00185.html

Simon,

I have a couple products that run LwIP in an SMP environment with CORE_LOCKING enabled.  I haven’t done any SMP optimizations yet, but I did run into an SMP issue with SYS_ARCH_PROTECT.  My single core systems use enable/disable interrupts for SYS_ARCH_PROTECT and upon porting to SMP, I used a spinlock.

Later we ran into issue with this because the event_callback in sockets.c calls sys_sem_signal() while the spinlock was held, which would task switch with the spinlock held, resulting in the next thread to use SYS_ARCH_PROTECT (on the same core) to enter an infinite spinlock wait (busy wait).  This was solved by switching to a mutex for SYS_ARCH_PROTECT.

I’d like to go back to using a spinlock and one way I’ve thought to solve this is to define a sleepable version of SYS_ARCH_PROTECT and a non-sleepable.  I think the event_callback() might be the only place where a context switch can happen while inside a SYS_ARCH_PROTECT critical section.  Sleepable could be mutex for SMP/interrupts for single core.  Non-sleepable would be spinlock for SMP, interrupts for single core.

Joel