Porting for an OS

The operating system emulation layer is located in two files, cc.h and sys_arch.c. It provides a common interface between the lwIP code and the underlying operating system kernel. The general idea is that porting lwIP to new architectures requires only small changes to a few header files and a new sys_arch implementation. It is also possible to do a sys_arch implementation that does not rely on any underlying operating system.

Sources of information

In addition to this page, you should probably have a look at the doc/sys_arch.txt file that is shipped along with the lwip source code. Looking at the ports/ directory of the contrib tarball can also help.

cc.h

This is a basic header that describes your compiler and processor to lwIP. The following things must be defined:

• Data types:
  • u8_t, u16_t, u32_t: The unsigned data types. For a 32-bit microprocessor, this is usually unsigned char, unsigned short, and unsigned int.
  • s8_t, s16_t, s32_t: The signed counterparts.
  • mem_ptr_t: A generic pointer type. It has to be an integer type (not void*, due to some pointer arithmetics).

• printf formatters for data types:
  • U16_F, S16_F, X16_F, U32_F, S32_F, X32_F, SZT_F
  • usually defined to "hu", "d", "hx", "u", "d", "x", "uz"
  • These formatters are used in several lwip files, yet mainly for diagnostics (LWIP_PLATFORM_DIAG) and debug output (LWIP_DEBUGF), so you might not need them if you disable that output.

• Byte ordering aka endianness:

#define BYTE_ORDER LITTLE_ENDIAN
or
#define BYTE_ORDER BIG_ENDIAN

Packet headers data is stored in network byte order which is the big-endian mode. If your processor architecture is little-endian, then we need to convert data using htons()/htonl()/ntohs()/ntohl() functions.

If your processor supports both big-endian and little-endian mode or you are yet free to chose a processor, big-endian might be a better choice regarding performance, as host- and network-byte order are the same.

LwIP provides standard functions for this, but it's often more efficient to define your own macros or functions if your compiler provides intrinsics or assembly instructions for byte swapping.

For example:

#define LWIP_PLATFORM_BYTESWAP 1
#define LWIP_PLATFORM_HTONS(x) ( (((u16_t)(x))>>8) | (((x)&0xFF)<<8) )
#define LWIP_PLATFORM_HTONL(x) ( (((u32_t)(x))>>24) | (((x)&0xFF0000)>>8) \ 
                               | (((x)&0xFF00)<<8) | (((x)&0xFF)<<24) )

• Computing cheksums:

IP protocols use checksums (see RFC 1071). LwIP gives you a choice of 3 algorithms:

1. load byte by byte, construct 16 bits word and add: not efficient for most platforms
2. load first byte if odd address, loop processing 16 bits words, add last byte.
3. load first byte and word if not 4 byte aligned, loop processing 32 bits words, add last word/byte.

#define LWIP_CHKSUM_ALGORITHM 2

you may also use your own checksum routine (e.g. using assembly to add 32 bits words with carry)

u16_t my_chksum(void *dataptr, u16_t len);
#define LWIP_CHKSUM my_chksum

• Structure packing:

LwIP accesses to 16 and 32 bits data fields in protocol headers which may be not aligned in memory. If your processor cannot read from/write to misaligned addresses, then you need to tell your compiler that the data may be not aligned and it must generate multiple byte or word load/stores to access it.

In all protocol structures, 16/32 bits values are 16 bits aligned, so choosing a 2 byte alignment for structures should be safe.

The common case is using Ethernet interfaces and MEM_ALIGNMENT=4. Ethernet header without VLAN is 14 bytes, so if you can/must have ETH_PAD_SIZE=2, then IP headers and higher layers are 4 bytes aligned and you may not need packing !

In any case, make sure the system runs stable when choosing a structure packing setting different to 1!

Usually a packed structure is created like that:

<   #ifdef PACK_STRUCT_USE_INCLUDES
   #  include "arch/bpstruct.h"
   #endif
   PACK_STRUCT_BEGIN
   struct <structure_name> {
       PACK_STRUCT_FIELD(<type> <field>);
       PACK_STRUCT_FIELD(<type> <field>);
       <...>
   } PACK_STRUCT_STRUCT;
   PACK_STRUCT_END
   #ifdef PACK_STRUCT_USE_INCLUDES
   #  include "arch/epstruct.h"
   #endif

Appropriate definitions of PACK_STRUCT_BEGIN, PACK_STRUCT_FIELD, PACK_STRUCT_STRUCT and PACK_STRUCT_END should be included in cc.h. The required definitions mainly depend on your compiler. Only PACK_STRUCT_STRUCT is required, lwip/src/include/lwip/arch.h provides empty defaults for the others. Optionally you can also define PACK_STRUCT_USE_INCLUDES and provide bpstruct.h and epstruct.h.

For example, according to a posting on the mailing list, the following values work for GCC:

<
   #define PACK_STRUCT_FIELD(x) x __attribute__((packed))
   #define PACK_STRUCT_STRUCT __attribute__((packed))
   #define PACK_STRUCT_BEGIN
   #define PACK_STRUCT_END

Warning: In version 1.3.0 of lwip the following source files do not conform to this pattern::* lwip/src/netif/ppp/vjbsdhdr.h only uses PACK_STRUCT_BEGIN and PACK_STRUCT_END because it uses bitfields

• Platform specific diagnostic output
  • LWIP_PLATFORM_DIAG(x) - non-fatal, print a message. Uses printf formating.
  • LWIP_PLATFORM_ASSERT(x) - fatal, print message and abandon execution. Uses printf formating. Unlike assert() from the standard c library, the parameter x is the message, not a condition.

Preemption protection

Information about this can be found in doc/sys_arch.txt and src/include/lwip/sys.h. It is called "lightweight" synchronization mechanisms, as some implementations can provide a more light-weight protection mechanism than using semaphores.

The macros used in lwip are:

• SYS_ARCH_PROTECT(x): Begin a block of protection. Should generally return the previous state.
• SYS_ARCH_UNPROTECT(x): End a block of protection, restoring the protection state back to "x".
• SYS_ARCH_DECL_PROTECT(x): Declare the variable "x" to hold the protection. Used so you can specify what data type the protection will require.

It is possible that the parameter "x" is not even used (the unix port from contrib does not use it).

Currently (lwip version 1.3.0), the following modules use these macros:

• src/core/memp.c,
• src/core/pbuf.c and
• src/netif/loopif.c.

Implementors have three options how to implement it:

1. Provide no implementation. This is appropriate when lwip is not used within multiple threads and/or in interrupts.
   Then do not define SYS_ARCH_PROTECT in cc.h or SYS_LIGHTWEIGHT_PROT in lwipopts.h. So SYS_LIGHTWEIGHT_PROT defaults to 0 in opt.h and sys.h provides empty default definitions for SYS_ARCH_DECL_PROTECT, SYS_ARCH_PROTECT and SYS_ARCH_UNPROTECT.
2. Provide an implementation in cc.h by defining the macros SYS_ARCH_DECL_PROTECT, SYS_ARCH_PROTECT and SYS_ARCH_UNPROTECT.
3. RECOMMENDED: Provide an implementation in sys_arch.h and sys_arch.c. Do not define the macros in cc.h. Instead define SYS_LIGHTWEIGHT_PROT as 1 in lwipopts.h. Then sys.h will provide default definitions (as of version 1.3.0) of these macros as:

#define SYS_ARCH_DECL_PROTECT(lev) sys_prot_t lev
                #define SYS_ARCH_PROTECT(lev) lev = sys_arch_protect()
                #define SYS_ARCH_UNPROTECT(lev) sys_arch_unprotect(lev)

Define the lowercase versions in sys_arch.h (and sys_arch.c) (see that header file section below). In most cases, this is the "right" way to define them. lwIP uses only the UPPERCASE versions in its code.

sys_arch.c

The sys_arch provides semaphores and mailboxes to lwIP. For the full lwIP functionality, multiple threads support can be implemented in the sys_arch, but this is not required for the basic lwIP functionality. Previous versions of lwIP required the sys_arch to implement timer scheduling as well but as of lwIP 0.5 this is implemented in a higher layer.

Semaphores

ATT: The Interface has been slightly changed in lwip 2. The information here is about lwip 1.

^^ So where is the information for lwip 2?? A link would be nice.

Semaphores can be either counting or binary - lwIP works with both kinds. Semaphores are represented by the type sys_sem_t which is typedef'd in the sys_arch.h file. lwIP does not place any restrictions on how sys_sem_t should be defined or represented internally, but typically it is a pointer to an operating system semaphore or a struct wrapper for an operating system semaphore.

The following functions must be defined:

• sys_sem_t sys_sem_new(u8_t count): Creates and returns a new semaphore. The count argument specifies the initial state of the semaphore. Returns the semaphore, or SYS_SEM_NULL on error.
• void sys_sem_free(sys_sem_t sem): Frees a semaphore created by sys_sem_new. Since these two functions provide the entry and exit point for all semaphores used by lwIP, you have great flexibility in how these are allocated and deallocated (for example, from the heap, a memory pool, a semaphore pool, etc).
• void sys_sem_signal(sys_sem_t sem): Signals (or releases) a semaphore.
• u32_t sys_arch_sem_wait(sys_sem_t sem, u32_t timeout): Blocks the thread while waiting for the semaphore to be signaled. The timeout parameter specifies how many milliseconds the function should block before returning; if the function times out, it should return SYS_ARCH_TIMEOUT. If timeout=0, then the function should block indefinitely. If the function acquires the semaphore, it should return how many milliseconds expired while waiting for the semaphore. The function may return 0 if the semaphore was immediately available.

Note that there is another function sys_sem_wait in sys.c, but it is a wrapper for the sys_arch_sem_wait function. Please note that it is important for the semaphores to return an accurate count of elapsed milliseconds, since they are used to schedule timers in lwIP. See the timer section below for more information.

Mailboxes

Mailboxes are used for message passing and can be implemented either as a queue which allows multiple messages to be posted to a mailbox, or as a rendez-vous point where only one message can be posted at a time. lwIP works with both kinds, but the former type will be more efficient. A message in a mailbox is just a pointer, nothing more.

Mailboxes are equivalently represented by the type sys_mbox_t. lwIP does not place any restrictions on how sys_mbox_t is represented internally, but it is typically a structure pointer type to a structure that wraps the OS-native mailbox type and its queue buffer.

The following functions must be defined:

• sys_mbox_t sys_mbox_new(int size): Return a new mailbox, or SYS_MBOX_NULL on error.
• void sys_mbox_free(sys_mbox_t mbox): Deallocates a mailbox. If there are messages still present in the mailbox when the mailbox is deallocated, it is an indication of a programming error in lwIP and the developer should be notified.
• void sys_mbox_post(sys_mbox_t mbox, void *msg): Posts the "msg" to the mailbox.
• u32_t sys_arch_mbox_fetch(sys_mbox_t mbox, void **msg, u32_t timeout). Blocks the thread until a message arrives in the mailbox, but does not block the thread longer than timeout milliseconds (similar to the sys_arch_sem_wait() function). The msg argument is a pointer to the message in the mailbox and may be NULL to indicate that the message should be dropped. This should return either SYS_ARCH_TIMEOUT or the number of milliseconds elapsed waiting for a message.
• u32_t sys_arch_mbox_tryfetch(sys_mbox_t mbox, void **msg): This is similar to sys_arch_mbox_fetch, however if a message is not present in the mailbox, it immediately returns with the code SYS_MBOX_EMPTY. On success 0 is returned with msg pointing to the message retrieved from the mailbox. If your implementation cannot support this functionality, a simple workaround (but inefficient) is #define sys_arch_mbox_tryfetch(mbox,msg) sys_arch_mbox_fetch(mbox,msg,1).
• err_t sys_mbox_trypost(sys_mbox_t mbox, void *msg). Tries to post a message to mbox by polling (no timeout).

@todo: to update with last changes

lwIP timeouts

Many protocols require certain events to happen after certain time periods. For example, if an ACK is not received within a certain time window in TCP, the packet needs to be retransmitted. For simplicity in implementation, lwIP piggybacks the timer requirement onto the mailbox and semaphores.

Although complete understanding of timeouts is not required to properly implement sys_arch.c, it does help explain what the one function you must provide actually does.

QUICK EXPLANATION: The linked list of timeouts is a linked list of struct sys_timeo's. This structure holds the number of milliseconds left until it should be activated, as well as a pointer to the function and its argument that should be called on timeout. When a thread (or the program itself) blocks on a lwIP semaphore (via sys_sem_wait) or on a mailbox (via sys_mailbox_fetch), it blocks for a time equal to the next timeout that needs servicing. If the semaphore or mailbox times out, the timeout handlers needing serviced are called. Otherwise, all the timeouts are decremented by the amount of time elapsed in the blocking call (hence why it is necessary for sys_sem_wait to accurately report how long the call was blocked).

The linked list is contained in a struct sys_timeouts. The function you must implement is:

• struct sys_timeouts *sys_arch_timeouts(void): This function returns a pointer to the "appropriate" linked list of timeouts.

Again, this is a bit vague. Here is an example implementation:

struct sys_timeouts lwip_system_timeouts = NULL; // Default timeouts list for lwIP

struct sys_timeouts *sys_arch_timeouts(void) {
   return &lwip_system_timeouts;
}

If you do not use threads in your system, then this implementation is all you need. If you do use threads, then you will need to give each thread its own sys_timeouts structure, and return that depending on what thread calls this function. This will be described in the next section. This "per thread" need is one of the main problem you have when you create your sys_arch.c file. It's important to understand that lwIP doesn't handle any "cleanup" function when a thread terminates. So, if you do a real allocation per thread (during sys_thread_new or sys_arch_timeouts, you have to handle any needed desallocation when your threads terminate).

As of version 1.4.0, an implementation of sys_arch_timeouts is not needed.

Threads

Threads are not required for lwIP, although lwIP is written to use them efficiently. The following function will be used to instantiate a thread for lwIP:

• sys_thread_t sys_thread_new(char *name, void (* thread)(void *arg), void *arg, int stacksize, int prio): name is the thread name. thread(arg) is the call made as the thread's entry point. stacksize is the recommanded stack size for this thread. prio is the priority that lwIP asks for. Stack size(s) and priority(ies) have to be are defined in lwipopts.h, and so are completely customizable for your system.

If you use threads, then it is important that timeouts happen in the right contexts (specifically the context that it was created in). Therefore, the sys_arch_timeouts function should return a timeouts structure depending on what thread is asking for it (this is actually the reason why this function is needed in the first place, since otherwise, sys.c could simply implement the version above and provide its own single global sys_timeouts variable). One possible implementation is shown below, somewhat in pseudo-code:

struct thread_struct_wrapper {
  struct thread_struct_wrapper *next;
  MY_OS_THREAD_TYPE thread; // not a ptr in this example, but the actual space
  struct sys_timeo *timeouts;
};

struct sys_timeouts lwip_system_timeouts = NULL; // Default timeouts list for lwIP
struct thread_struct_wrapper *lwip_system_threads = NULL; // a list of all threads created by lwIP

sys_thread_t sys_thread_new(char *name, void (* thread)(void *arg), void *arg, int stacksize, int prio) {
  sys_thread_t newthread;
  SYS_ARCH_DECL_PROTECT(old_val);

  newthread = malloc(sizeof(struct thread_struct_wrapper)); // allocate the space for the thread wrapper
  if (newthread==NULL) return NULL;

  SYS_ARCH_PROTECT(old_val); // Need to protect this -- preemption here could be a problem!
  newthread.next = lwip_system_threads;
  lwip_system_threads = newthread;
  SYS_ARCH_UNPROTECT(old_val);

  newthread.timeouts = NULL; // initialize the linked list to NULL
  my_system_os_create_thread_function(&newthread.thread, thread, arg, prio); // create it, however my OS does it
  return newthread;
} 
  
struct sys_timeouts *sys_arch_timeouts(void) {
  sys_thread_t thread = lwip_system_threads;
  MY_OS_THREAD_TYPE *self = my_system_os_get_current_thread();

  // Search the threads list for the thread that is currently running
  for ( ; thread!=NULL; thread=thread->next) {
     if (thread->thread == self) return thread->timeouts;
  }
   
  // No match, so just return the system-wide default version
  return &lwip_system_timeouts;
}

System

void sys_init(void). lwip system initialization.

Final declarations for sys_arch.h

• sys_sem_t, sys_mbox_t, sys_thread_t all refer to pointers to the appropriate types, whether native OS versions or wrappers used to make them conform to what lwIP expects.
• SYS_MBOX_NULL, SYS_SEM_NULL, almost always set equal to NULL. These define what a failure to create the system type will return.

Preemption protection

If you define your own UPPERCASE versions, as explained above in the cc.h section, then you do not need to define these).

It is recommended that you define the following functions (rather than the UPPERCASE versions) appropriately.

These functions are used to disable interrupts around crucial blocks of code that should not be interrupted -- usually done in order to preserve data integrity. In order to allow your code to nest protection blocks (for example a block of protection inside another block of protection, possibly in different functions), a parameter (shown as "x" here) is used to remember the prior protection state so that when a block ends the previous state is restored.

• sys_prot_t sys_arch_protect(void): This optional function does a "fast" critical region protection and returns the previous protection level. This function is only called during very short critical regions. An embedded system which supports ISR-based drivers might want to implement this function by disabling interrupts. Task-based systems might want to implement this by using a mutex or disabling tasking. This function should support recursive calls from the same task or interrupt. In other words, sys_arch_protect() could be called while already protected. In that case the return value indicates that it is already protected.
• void sys_arch_unprotect(sys_prot_t pval): This optional function does a "fast" set of critical region protection to the value specified by pval. See the documentation for sys_arch_protect() for more information. This function is only required if your port is supporting an operating system.
• Define the sys_prot_t type to reflect what these two functions need to operate on.

The following is an example of how these functions will be used in the default implementation.

lwIP code example	Compiler expansion for default defintion
SYS_ARCH_DECL_PROTECT(lev); SYS_ARCH_PROTECT(lev); /* My delicate block of code */ SYS_ARCH_UNPROTECT(lev);	sys_prot_t lev; lev = sys_arch_protect(); /* My delicate block of code */ sys_arch_unprotect(lev);

Final thoughts

Be careful with using mem_malloc() in sys_arch. Specifically, when mem_init() is run (in mem.c), it allocates a semaphore to block concurrent access in its routines. If sys_sem_new() calls mem_malloc to allocate its semaphore, then it will be calling mem_malloc() before the mem.c system has been initialized!