TCP Offload And The Evolution of High Performance Interconnect

By Douglas Barta, Nauticus Networks

November 21, 2002 -- (WEB HOST INDUSTRY REVIEW) -- The data center is undergoing an interconnect evolution that should be factored into the strategic plans of data center architecture teams.  Will the industry consolidate around a standard and thus fuel focused innovation and product development?  Or will the number of interconnect technologies continue to grow, fracturing the technological horizon and slowing development as many options abound?  These are important questions for architecture teams to address and factor into their planning process.  The interconnect debate is being fueled by hype and promise.  Separating the fact from the fiction can be a daunting task, but the performance and economic benefits promised by these new technologies should make it worth the effort. 

Interconnect evolution is occurring at the chip/board level and at the system/data center level.  At the chip/board level, the de-facto PCI standard is splintering into multiple approaches, including evolved PCIs like PCI-X, and narrow-width high bit rate approaches like Hypertransport, RapidIO, and 3GIO.  At the system/data center level, the existing defacto standards (Ethernet and FiberChannel) continue to evolve, with new technologies like InfiniBand on the horizon.        

In trying to predict a logical path for the evolution of data center interconnect, it makes sense to look back at the last decade of chip/board level interconnect evolution.  A historical pattern of expansion and consolidation becomes evident.  The expansion phase is marked by a proliferation of technical options, highly fragmented development efforts, high costs, and limited innovation.  Typically, the industry then rallies around a standard and a consolidation phase occurs, marked by a rapid increase in innovation, dropping costs, and accelerated development. 

PCI is a good recent example of the consolidation phase.  In the early 1990’s a wide array of proprietary interconnects existed within the computer.  PCI emerged out of that cloud and became an agreed upon standard around which the industry rallied.  PCI was first developed and deployed as a bus for interconnecting chips on the PC board.  As PCI grew in popularity and proved itself a solid technology, it evolved from being a chip interconnect to a replacement for the ISA expansion bus.  Ultimately, PCI expanded to become the I/O backplane in millions and millions of devices.  The popularity of PCI has resulted in high volumes of PCI components and thus low device costs.  Developers know and trust this technology, so development cycles are shortened and risk reduced.  The PCI consolidation phase of chip/board level interconnect has had significant benefit for equipment manufacturers and for customers.     

But as with all repeating cycles, chip/board level interconnect seems poised to go through another “expansion” phase as the de-facto PCI standard is splintering into multiple approaches, including evolved PCI's (like PCI-X), Hypertransport, RapidIO, and 3GIO. 

This cycle of expansion and contraction for interconnect can be applied not just at the chip/board level, but also at the system/data center level.  As with PCI, great inroads have been made with Ethernet and FiberChannel and they have become the dominant technologies at work in the data center.  FiberChannel, though it has been getting progressively faster, is not likely to escape its niche as a storage interconnect. 

InfiniBand, a relatively recent arrival to the interconnect wars, was proposed to boost performance and offer a single, consolidated interconnect for the data center, thus reducing cost and complexity.  But the advantages of InfiniBand over the incumbents have not yet proven compelling enough to accelerate adoption.  InfiniBand took a recent setback when Intel and Microsoft backed away from the technology.  InfiniBand may well find an early home as a niche product for cluster interconnect, but if this occurs it would drive up complexity and cost in the data center, the very opposite of the original goals for the technology!

Which brings us back to Ethernet.  Good old Ethernet may represent the best opportunity for consolidation of data center interconnect.  The technology is well understood and has come down a significant cost curve.  The biggest problem with Ethernet is not with the interconnect, but with what typically runs on top of the interconnect, namely TCP.  TCP has proven itself a rugged protocol, but it tends to be exceptionally resource intensive (and thus expensive).  Ask a handful of data center operators and you will hear that TCP stack processing is consuming “50-80% of CPU cycles”; cycles that would be better focused on processing business logic.   

Hardware acceleration in a “TCP Offload Engine”, or TOE, promises to alleviate this processing burden, freeing processors up to do more valuable work.  To date, TOE’s have taken the form of PCI cards and the silicon to drive these cards.  In the coming months expect TCP offload technology to migrate to the data center edge and be delivered in a consolidated device, thus screening much of the complexity of TCP from the back end servers.  TCP offload technologies, while still relatively new, promise to deliver tremendous benefit while forcing relatively minor disruption to the status quo.  

With TCP offload engines in place, Ethernet becomes a more viable next generation cluster interconnect.  With TCP offload engines in place and with the adoption of iSCSI, Ethernet could be a viable alternative to FiberChannel.  The combination of a TCP offload engine and Ethernet seems to have a good chance to become the consolidated system/data center level interconnect. 

The simplicity of a consolidated interconnect would dramatically reduce complexity, operating costs and would simplify the management model for the data center.  Interconnect evolution holds great promise for enterprise data centers and should be actively tracked by architecture teams.  A recommended action plan might include:

The Top 10 Steps – a recommended action plan:

• Consider your interconnect in stages (data center wide, within the rack, within the cluster, within the box for example) and evaluate key areas of your data center you feel would benefit from improved interconnect (put another way, work to understand your own bottlenecks)
• Having identified specific technologies you want to monitor, track the relevant industry consortiums and standards bodies pushing standards in these areas
• Using “design center” specifications and early benchmarks, understand the performance and economic impact of these changes, in preparation for business case creation
• Alert your senior business leadership to the economic benefits of these changes, involve them in the process
• Ask your vendors for regular product development updates
• Make sure new equipment you purchase is prepared for the protocol and media transformations that may be needed to gateway to these faster interconnects
• As technology becomes available, pick a small pilot project to deploy this technology and to evaluate the actual performance under your real world load
• Use that pilot project to validate your business case assumptions around improvement
• And as always, have fun doing it!

About the Author
Mr. Barta joined Nauticus Networks from Answerthink, a leading e-business consulting and systems integration firm, where he served as Managing Director of Marketing and Business Development. Mr. Barta played a key role in driving Answerthink to over $300M in revenue in just over two years. Previously, Mr. Barta was Vice President at Legacy Technology, which was acquired by Answerthink in 1998; at Legacy he ran the National Telecommunications Practice. Mr. Barta was also a management consultant with Deloitte & Touche and, prior to that, was a hardware design engineer in the MicroVAX engineering group at Digital Equipment Corporation. He holds a BSEE from Cornell University and an MBA from The Wharton School of Business. He is a regular industry speaker on e-business and telecommunications issues and is an active member of IEEE, USTA, MTC, and TIA.