On the Elusive Benefits of Protocol Offload
Periodic order-of-magnitude jumps in Ethernet bandwidth regularly reawaken interest in TCP/IP transport protocol offload. This time the jump to 10-Gigabit Ethernet coincides with the emergence of new network storage protocols (iSCSI and DAFS), and vendors are combining these with offload NICs to position IP as a competitor to FibreChannel and other SAN interconnects. But what benefits will offload show for application performance? Several recent studies have presented conflicting data to argue that offload either does or does not benefit applications. But the evidence from empirical studies is often little better than anecdotal. The principles that determine the results are not widely understood, except for the first principle: Your Mileage May Vary. This paper outlines fundamental performance properties of transport offload and other techniques for low-overhead I/O in terms of four key ratios that capture the CPU-intensity of the application and the relative speeds of the host, NIC device, and network path. The study also reflects the role of offload as an enabler for direct data placement, which eliminates some communication overheads rather than merely shifting them to the NIC. The analysis applies to Internet services, streaming data, and other scenarios in which end-to-end throughput is limited by network bandwidth or processing overhead rather than latency.