Tesla did not adopt the traditional TCP protocol. Instead, it developed a new lossy Ethernet transport protocol for its Dojo supercomputer: Tesla Transport Protocol over Ethernet (TTPoE).
This protocol is designed specifically for AI supercomputers to achieve data transmission for exascale network architectures.
Compared with the traditional TCP/IP protocol, TTPoE is executed at the hardware level to solve the latency problem of AI interconnect. Through simplified software and hardware design, it achieves lower latency and higher bandwidth.

The reason is that Tesla believes the TCP/IP protocol is too slow, while RDMA (Remote Direct Memory Access) using PFC (Priority Flow Control) can achieve a lossless network but at the cost of network performance.

In the introduction below, it points out that TTPoE is a point-to-point transport layer protocol executed at the hardware level. Its advantage is that Tesla does not need special switches, because it mainly uses Layer 2 (data link layer) transmission.

In other words, in the Dojo OSI model, Tesla replaces the traditional transport layer with TTPoE.

Through the TTP link, we can see an example of TTP protocol conversion.

In the slide below, Tesla shows a comparison of the TCP state machine and the TTP state machine: the TTPoE transport layer state machine is optimized for hardware execution. For example, the original 2ms quiet time is shortened to accommodate microsecond-level protocol requirements, and dependence on virtual memory is avoided by using only physical memory.

At the physical level, TTPoE uses the simple Ethernet-II format, operating only at Layer 2 and not involving Layer 3 networking. Each transport layer hardware block in the Dojo supercomputer is an IP module located between the network-on-chip (NoC) and the standard Ethernet MAC.

The TTPoE protocol is “lossy” in nature, meaning it expects packet loss and retransmission under network congestion, backpressure, or errors, while still guaranteeing complete packet delivery. This design is similar to TCP rather than UDP, and it effectively manages bandwidth usage by limiting the amount of speculative transmission before receiving an acknowledgment (ACK).

In addition, TTPoE’s congestion management is distributed, handled by the transmit channel of the local link rather than controlled by a central network or switch. This design avoids relying on lossless network mechanisms such as Priority Flow Control (PFC) and Nagle’s algorithm, thereby reducing network complexity and fragility.

Tesla states that TTP supports Quality of Service (QoS), but this feature is currently disabled.
Tesla has integrated the TTP IP module into FPGA and silicon, designed to quickly transmit packets on the wire.

At the microarchitecture level, TTP’s microarchitecture is uniquely designed and looks very much like an L3 cache, employing techniques similar to SMP caches, snoop filters, and CPUs, with a 4-stage read-modify-write (RMW). The 1MB transmit buffer allows roughly 80 microseconds of round-trip latency tolerance, and virtual channels are used for priority handling and avoiding livelock/deadlock. The 1MB transmit buffer is mentioned for the current generation, but it may change in the next generation.

Tesla showed its 100Gbps “Mojo” NIC, a “dumb” NIC with 100Gbps Ethernet speed, a PCIe Gen3 x16 interface, 8GB of DDR4 memory, and power consumption under 20W. It also integrates the Dojo DMA engine and supports 512 unique bidirectional LRU active links.

At the conference, Tesla reviewed its 2022 introduction of the D1 chip.

The figure below shows the D1 chip package in a 5×5 array.

The Dojo interface processor contains 32GB of high-bandwidth memory, providing 800GB/s of total memory bandwidth and a 900GB/s TTP interface. The 900GB/s TTP interface is internal, while TTPoE is encapsulated in Ethernet frames.

Next, Tesla showed how Dojo is connected: Tesla 100G NIC to V1 Dojo interface card to Dojo.

Starting from the assembled D1 chip array, they connect to the interface cards via SerDes cables.

These then go to the Interface Cards.

Then, these interface cards are subsequently connected to low-cost 100G NICs.

In Tesla’s TTP network, the “Mojo” host operates through a variable data ingestion approach. The vision network may be limited by data ingestion, because vision-based tensors and training clips can be on the order of GB. The “Mojo” host is scheduled from the general compute pool on demand. Forward/backward pass TTP traffic is mutually independent, meaning data ingestion and all-reduce operations share the same TTP DIP port but execute at different stages of training.

The slide below shows the Mojo Dojo compute hall in New York. We can see that the 2U compute nodes have no front 2.5-inch storage.

This is a 4 ExaFLOP engineering system equipped with 40PB of local storage, along with a large amount of bandwidth and compute. Having a 4EF (BF16/FP16) engineering system is quite remarkable.

In actual testing, TTPoE was measured on Arista’s 7060, 7808, and 7816 switches, showing lower round-trip latency and high throughput.

Tesla joined the Ultra Ethernet Consortium (UEC) and made the TTPoE protocol publicly available to support a broader range of network operators.

Tesla thanked: the original inventor of TTPoE, the network deployment team, the silicon design team, the systems and infrastructure team, the software and driver team, the Linux patch team, the SDN team, the DevOps team, the QA team, the DC technology team, the supply chain team, and all TTP/Mojo interns for their contributions.

Finally, on network performance, Tesla believes that excessive focus on synthetic latency measurements should be reduced, because such measurements often fail to fully reflect real network performance. More meaningful differentiators include the comparison of lossy versus lossless networks; centralized versus distributed approaches to handling network congestion; and the difference between proprietary and open-source technologies.

Furthermore, it shows the measurement results of one-way write latency for a single switch hop under no-load conditions, providing latency data for several different technologies. Specifically, TTPoE latency is 1.0E-05s, InfiniBand (IB) is 2.0E-06s, NVLink is 2.3E-06s, RoCEv2 is 4.3E-06s, while TCP/IP has the highest latency at 5.3E-04s.
Tesla’s conclusion is that their system performance has reached the microsecond level.
Add me on WeChat: note “Data Communication” to join the Data Communication professional group!

Enjoy this article?
Subscribe to get new optical communication insights delivered to your inbox. Free, unsubscribe anytime.
