The key design feature of the Internet is the end-to-end principle. In short, the end-to-end principle says that as much work as possible should be done at the ends of the network. This results in a very simple network core. The simplicity of the core allows it to scale. See World of Ends for more implications of the end-to-end principle.
If you ask most network people exactly where the “end” is they will probably say it is the device at the edge of the network. Some may even go as far as to say it is the operating system on the edge device. At present this is indeed the case. For example, the processing necessary to make TCP a reliable protocol happens within the operating system.
At LCA 2006, Van Jacobson recently weighed in on the network protocol processing overhead that is becoming a big problem as link data rates increase. Current operating systems are having a hard time keeping up with 10 gigabit links, especially when using TCP. In his presentation, Van Jacobson says that the placement of the TCP stack in the operating system kernel is a historical accident. This design was chosen because it was necessary to ensure Multics did not page out the TCP stack. Further, TCP in the kernel violates the end-to-end principle because the kernel is not the end, the application is. Van Jacobson offers Net channels as a possible solution to this problem. Net channels provide a simple, cache friendly way to manage network packets within a system.
The presentation discusses several ways that Net channels can improve TCP performance. The first is to use Net channels between the NIC and the current in-kernel TCP stack. The more interesting use of Net channels is to push all TCP processing into userspace. Essentially, each application would have their own TCP stack. This removes the bottleneck that the single, system-wide TCP stack creates. Amazingly, Van Jacobson presents statistics which show that this modification results in TCP processing overhead dropping by 80%. Other benefits would include a simpler kernel and the ability to have a TCP stack tuned for each application. Applying TCP bug fixes and adding new features would also become easier with TCP moved outside of the kernel.
For more information on this really amazing idea see the following resources.
- Net channel presentation slides
- Blog post by David S. Miller
- Article at LWN.net
- Thread on the Linux NetDev mailing list