A COMPARATIVE STUDY OF NETWORK MANAGEMENT BUFFERS IN LINUX AND 4.4BSD DERIVATIVE OPERATING SYSTEM KERNELS

 

 

Osadchiev A.A., Subhasish Ghosh

 

 

Moscow, Moscow Power Engineering Institute (Technical University), Russia

 

Introduction

 

A study of the network stack in BSD variants and Linux Kernels shed light on the fact that both the networking layer as well as the Kernel code, in general, tries to be fairly object oriented in its design. In UNIX, the dominating protocols are those that come under TCP/IP. The most important components of the Network Implementation layer in BSD and Linux are: A network Interface, which represents a thing which sends and receives packets; a Protocol, which is a different language of networking; the BSD Socket, which provides a connection in the networking that provides UNIX file Input/Output (I/O) and exists to the user program as a file descriptor and finally, the memory and network management buffers, which provide the general buffering and flow control facilities needed by network protocols. Control for these is provided by core low level library routines available to the whole of the networking.

 

Network Buffers in Linux

 

The primary goal of sk_buff Kernel data structure code routines, which can be accessed at /usr/include/linux/skbuff.h file, is to provide a consistent and efficient buffer handling method for all of the network layers, and by being consistent to make it possible to provide higher level sk_buff and socket handling facilities to all the protocols. The specialty of the sk_buff structure lies in the fact that sk_buff is a control structure with a block of memory attached. There are two primary sets of functions provided in the sk_buff library. Firstly routines to manipulate doubly linked lists of sk_buffs, secondly functions for controlling the attached memory. The buffers are held on linked lists optimized for the common network operations of append to end and remove from start. As so much of the networking functionality occurs during interrupts these routines are written to be atomic. The list operations are used to manage datagrams (or packets) as they arrive from the network, and as they are sent to the physical interfaces. On the other hand, the memory manipulation routines are used for handling the contents of packets in a standardized and efficient manner.

Network Buffers in BSD Kernels

On 4.4BSD derivative Kernels, the memory and network management buffer mbuf is defined in the file /usr/src/sys/mbuf.h. The MBuf class is a malloc'ed, multi-chunk-supporting, memory-buffer. The buffer can be extended by either appending data to the object (which may reallocate the last chunk of data to fit the new characters) and/or by adding more pre-allocated chunks of data to the object (which can be either appended or prepended to the list of buffer chunks). The memory buffers (mbufs) are used for passing socket address data structures and data. Each mbuf is 128-bytes in size, with 20 bytes allocated for the mbuf header m_hdr data structure present at the beginning of every mbuf.

 

 

Characteristic Differences

 

Reviewing the core sk_buff and mbuf Kernel data structures reveals the following significant characteristics:

 

1. sk_buff is a massively laid down structure, containing both routines to manipulate doubly linked lists of sk_buffs and functions for controlling the attached memory as well. Though it is large in size, yet the code is easily understandable. The 4.4BSD derivative Kernels (for example, FreeBSD) buffer mbuf is much smaller in size. Though compact, it makes reading and deciphering code segments a difficult task for beginners.

 

2. sk_buff contains Transport layer, Network layer and DataLink layer data structures within it, with appropriate pointers to definitions of the other hdr structures contained in other corresponding files. In mbuf, m_hdr structure, present at the beginning of every mbuf contains a variable, mh_flags, which stores the mbuf datagram (or packet header) flags. The m_tag_id = = m_hdr.mh_flags, which in turn stores the packet header information for the various layers. Packet tag types which are defined in the same file containing mbuf structure, contains ones which are from both NetBSD as well as FreeBSD Kernels networking stacks.

 

3. In sk_buff, if checksum has been calculated, the variable csum holds the checksum. In mbuf, any checksum variable is not directly included. The variable csum_flags defined in datagram (or packet) header in first mbuf of chain, struct pkthdr, stores the various flags regarding checksum. Packet tags can also be used to calculate checksum values and are usually stored in MT_TAG mbufs which are then prepended to the actual mbuf chain. For example, the NIC check summing on FreeBSD Kernels takes place using the tag PACKET_TAG_IN_PACKET_CHECKSUM.

 

4. In sk_buff, the head, data, tail and end pointers are used to refer to data. The function alloc_skb(int size, int priority) creates a new sk_buff and initializes it. The returned buffer is ready to use but does assume that a few fields are to be filled to indicate how the buffer should be freed. Normally this is skb->free=1. On 4.4BSD derivative Kernels, the MGET(struct mbuf *m, int how, int type) function allocates an mbuf and initializes it to contain internal data. mbuf uses the #define statement which maps the m_data variable to m_hdr.mh_data, where mm->m_data = _mm->m_dat, thereby allocating data.

 

5. On Linux, after a buffer has been allocated all the room is at the end. A further call skb_reserve( ) is called before data is added. It allows some room to be added at the beginning. On 4.4BSD derivatives, we do not need to know in advance how much space we would require as it uses chains of mbufs for its network buffers. Linux chooses to use linear buffers and know in advance (often wasting a few bytes to allow for the worst case) because linear buffers make many other things much faster.

 

Software Status

 

The sk_buff data structure, which can be accessed at /usr/include/linux/skbuff.h, was taken from Red Hat Linux 8.0 (Psyche Edition). The underlying Linux Kernel version is 2.4.19-8. The mbuf structure, which can be accessed at /usr/src/sys/mbuf.h on 4.4BSD derivatives was taken from FreeBSD 4.7-RELEASE and OpenBSD 3.2-RELEASE.

 

References

 

1.     Operating Systems, 2nd Edition, William Stallings. Prentice Hall International, 1995.

 

2.     Modern Operating Systems, Andrew S. Tanenbaum. Prentice Hall International, 1997.

 

3.     Understanding the Linux Kernel, From I/O Ports to Process Management, Daniel P. Bovet and Marco Cesati. O'Reilly Publications, 2001.

 

4.     The Design and Implementation of the 4.4BSD Operating System, Marshall Kirk McKusick, Keith Bostic, Michael J. Karels and John S. Quarterman. Addison-Wesley Publishing, 1996.

 

5.     General description of the TCP/IP Protocols, Charles L. Hedrick. Rutgers University, 1987.

 

6.     TCP/IP Stack Introduction: Looking Under the Hood!, Shiv Kalyanaraman. Rensselaer Polytechnic Institute.

 

7.     skb - Linux network buffers, Harald Welte. 2000.

 

8.     Network Buffers, Alan Cox.

 

9.     Linux Kernel Programming, 3rd Edition, Michael Beck, Harald Bohme, Mirko Dziadzka, Ulrich Kunitz, Robert Magnus, Claus Schroter and Dirk Verworner. Pearson Education Limited, 2002.