
Among the traditional pillars of data center infrastructure (namely compute, storage, and networking), networking is unique because it has no inherent value on its own. While a standalone server can still compute and storage devices can still hold data, the value of the network depends solely on the information it transports.
The same logic applies to High-Performance Networking (HPN). However, the negative impact of an HPN failure can be more significant. If data cannot arrive at its destination perfectly, reliably, and in order, it usually causes upstream and downstream disruptions beyond the basic transmission scope. Why? Because “high performance” is far more complex than mere bandwidth and Application-Specific Integrated Circuit (ASIC) latency.
Network engineers know well that, in traditional environments, the dividing line lies at the moment when a data packet enters the cable on one side and is handed to the Network Interface Card (NIC) on the other side. This division is simple, clear, and convenient for troubleshooting (or at least allows shifting responsibility elsewhere by claiming “it is not a network problem”), and it covers only a tiny part of the whole process.
This article will explore the components of high-performance networking, the available transmission technologies, and the “secret” that truly makes it run at high speed.
What Are the Components of a High-Performance Network?
- Latency
When a packet passes through a single switch ASIC, the NIC-to-NIC latency is usually between 100-300 nanoseconds (specific values vary by chip and platform). All else being equal, the core-to-core latency is about 50 microseconds. What does this mean? About 95% of the latency occurs inside the server.
To understand this, one needs to look deep into the internal structure of the server. Note that applications, sockets, and protocol drivers all reside in main memory.

A highly simplified packet transmission process (“a day in the life of a packet”) begins with two 8-byte registers entering the CPU, outputting one 8-byte register, which is regarded as a 64-bit floating-point operation. From here, our packet needs to pass through the L1 (extremely fast), L2 (very fast), and L3 (fast) caches, main memory (slower), and the PCI bus in sequence, finally reaching the NIC for transmission to another device.
- Bandwidth
Bandwidth demand keeps growing. Currently, the minimum requirement for high-performance networks is 400Gbps, expected to jump to 800Gbps to 1600Gbps within the next two years. Bandwidth is a constantly moving target, and investment should be understood as having a limited “top-tier” service life, because bandwidth currently considered ultra-high-speed will soon become commonplace.
Nonetheless, although the engineering and physical principles to achieve such bandwidth are quite complex and the size of the pipe does matter, bandwidth itself is not what makes a network “high-performance”.
If it is not latency and bandwidth, then what makes applications run faster?
RDMA: Remote Direct Memory Access, an I/O bypass technology implemented on the NICs of participating computing devices. RDMA uses zero-copy networking, that is, directly reading data from the main memory of one device to the main memory of another, while completely bypassing the NIC, sockets, and transport buffers running in that memory.

RDMA is a multidisciplinary field involving a multi-layered understanding of application, network, compute, and storage technologies. Once RDMA is enabled, the network can achieve a 90% to 96% reduction in end-to-end latency (rather than NIC-to-NIC latency).

However, RDMA also has some drawbacks. The most notable is that it has extremely strict quality and reliability requirements: data must be transmitted reliably and in order, without packet loss or jitter, and must always be so. The “best-effort” service level agreement (SLA) of standard data center networks is insufficient. This is where the “high-performance” part of high-performance networking comes in.
So, what is a high-performance network?
The problem of defining a high-performance network is not merely a matter of latency or bandwidth, although both are factors. High performance actually depends on how you transport RDMA (or the next truly important application). Just as networks must adapt to voice-over-IP, Fiber Channel-over-Ethernet, and dozens of other “sensitive applications”, modern networks must also adapt to the needs of today’s high-end compute and storage capabilities.
Currently, there are two main industry-standard approaches for RDMA transport:
InfiniBand
InfiniBand is a communication standard supporting RDMA. The latency and bandwidth of InfiniBand ASICs are roughly equivalent to Ethernet performance. In fact, the latency of IP-over-InfiniBand is the same as unaccelerated IP-over-Ethernet. Its physical hardware includes a Host Channel Adapter (HCA, essentially a NIC) and InfiniBand switches. It is a controller-based architecture (through a subnet manager), is non-routable, meaning the failure domain size is the same as the entire network fabric. The subnet manager runs as software on one of the switches.
RoCEv2
RoCE (pronounced “rocky”) is a similar standard using Ethernet, but with some adjustments. It also consists of HCAs and switches, but relies on HCA point-to-point negotiation for reliable transport, rather than an InfiniBand-based controller. Priority Flow Control (PFC) and Explicit Congestion Notification (ECN) parameters must be configured on the switches to identify and prioritize RDMA-related traffic.
Which one to choose depends on engineering preference and deep-rooted beliefs.
Considerations of InfiniBand vs. RoCE
The following is a quick comparison:

Flexibility
InfiniBand networks are tailored for specific workloads (such as AI) and can support only one type of workload. If a customer’s data center only contains a limited number of continuously running AI workloads with similar performance characteristics, InfiniBand may be a good choice. High-performance Ethernet is more flexible. If a customer’s data center focuses on cloud computing (hybrid cloud or cloud-ready) or contains an application ecosystem with multiple different needs, Ethernet/RoCE may be a better choice. Migrating from InfiniBand to Ethernet may face challenges.
Scalability
As AI/ML workloads grow, the scale of data centers will also expand. If current growth trends continue, today’s NVIDIA SuperPOD DGX cluster (127 nodes / 1016 GPUs) will need to expand 20-fold within two years. Although InfiniBand’s specification allows building very large network fabrics, some complications may arise when facing large failure domains. InfiniBand has no Layer 3 option.
When to use each technology?
Best practices recommend using InfiniBand for the most challenging and performance-demanding applications (that is, not just any AI model, but truly large AI models). The other 95% of use cases can run comfortably on the more flexible Ethernet option.
In 2022, Meta conducted a study comparing the pre-training time of its LLAMA2 AI model. Two identical clusters were built in the study: one based on InfiniBand and the other on RoCEv2. The study found that below 2000 GPUs, the performance of the two is very close. At current rack density and pricing, 2000 GPUs require about 63 cabinets, cost about 100 million dollars, and take more than a year to build.
The general industry consensus is that, although InfiniBand always has its place, Ethernet-based solutions will continue to evolve to meet most market demand.
Reference documents:
White Paper on RDMA-Based Long-Distance Lossless Data Movement Technology
InfiniBand and Ethernet RDMA Technologies
Enjoy this article?
Subscribe to get new optical communication insights delivered to your inbox. Free, unsubscribe anytime.
