Skip to content

WaytoICT

通往ICT之路 · 光通信与AI网络技术解读

中文English
登录注册
Menu
Technology Optical comms · silicon photonics · CPO → Market Insights Financials · trends · industry → In-Depth Analysis Optical chips, devices & modules → Video Understand a topic in minutes → Documents Latest docs · summit materials →
Menu

What Makes Tesla’s TTPoE Protocol Different?

Posted on 2024-09-24 by

Tesla did not adopt the traditional TCP protocol. Instead, it developed a new lossy Ethernet transport protocol for its Dojo supercomputer: Tesla Transport Protocol over Ethernet (TTPoE).

This protocol is designed specifically for AI supercomputers to achieve data transmission for exascale network architectures.

Compared with the traditional TCP/IP protocol, TTPoE is executed at the hardware level to solve the latency problem of AI interconnect. Through simplified software and hardware design, it achieves lower latency and higher bandwidth.

What Makes Tesla's TTPoE Protocol Different?

The reason is that Tesla believes the TCP/IP protocol is too slow, while RDMA (Remote Direct Memory Access) using PFC (Priority Flow Control) can achieve a lossless network but at the cost of network performance.

What Makes Tesla's TTPoE Protocol Different?

In the introduction below, it points out that TTPoE is a point-to-point transport layer protocol executed at the hardware level. Its advantage is that Tesla does not need special switches, because it mainly uses Layer 2 (data link layer) transmission.

What Makes Tesla's TTPoE Protocol Different?

In other words, in the Dojo OSI model, Tesla replaces the traditional transport layer with TTPoE.

What Makes Tesla's TTPoE Protocol Different?

Through the TTP link, we can see an example of TTP protocol conversion.

What Makes Tesla's TTPoE Protocol Different?

In the slide below, Tesla shows a comparison of the TCP state machine and the TTP state machine: the TTPoE transport layer state machine is optimized for hardware execution. For example, the original 2ms quiet time is shortened to accommodate microsecond-level protocol requirements, and dependence on virtual memory is avoided by using only physical memory.

What Makes Tesla's TTPoE Protocol Different?

At the physical level, TTPoE uses the simple Ethernet-II format, operating only at Layer 2 and not involving Layer 3 networking. Each transport layer hardware block in the Dojo supercomputer is an IP module located between the network-on-chip (NoC) and the standard Ethernet MAC.

What Makes Tesla's TTPoE Protocol Different?

The TTPoE protocol is “lossy” in nature, meaning it expects packet loss and retransmission under network congestion, backpressure, or errors, while still guaranteeing complete packet delivery. This design is similar to TCP rather than UDP, and it effectively manages bandwidth usage by limiting the amount of speculative transmission before receiving an acknowledgment (ACK).

What Makes Tesla's TTPoE Protocol Different?

In addition, TTPoE’s congestion management is distributed, handled by the transmit channel of the local link rather than controlled by a central network or switch. This design avoids relying on lossless network mechanisms such as Priority Flow Control (PFC) and Nagle’s algorithm, thereby reducing network complexity and fragility.

What Makes Tesla's TTPoE Protocol Different?

Tesla states that TTP supports Quality of Service (QoS), but this feature is currently disabled.

Tesla has integrated the TTP IP module into FPGA and silicon, designed to quickly transmit packets on the wire.

What Makes Tesla's TTPoE Protocol Different?

At the microarchitecture level, TTP’s microarchitecture is uniquely designed and looks very much like an L3 cache, employing techniques similar to SMP caches, snoop filters, and CPUs, with a 4-stage read-modify-write (RMW). The 1MB transmit buffer allows roughly 80 microseconds of round-trip latency tolerance, and virtual channels are used for priority handling and avoiding livelock/deadlock. The 1MB transmit buffer is mentioned for the current generation, but it may change in the next generation.

What Makes Tesla's TTPoE Protocol Different?

Tesla showed its 100Gbps “Mojo” NIC, a “dumb” NIC with 100Gbps Ethernet speed, a PCIe Gen3 x16 interface, 8GB of DDR4 memory, and power consumption under 20W. It also integrates the Dojo DMA engine and supports 512 unique bidirectional LRU active links.

What Makes Tesla's TTPoE Protocol Different?

At the conference, Tesla reviewed its 2022 introduction of the D1 chip.

What Makes Tesla's TTPoE Protocol Different?

The figure below shows the D1 chip package in a 5×5 array.

What Makes Tesla's TTPoE Protocol Different?

The Dojo interface processor contains 32GB of high-bandwidth memory, providing 800GB/s of total memory bandwidth and a 900GB/s TTP interface. The 900GB/s TTP interface is internal, while TTPoE is encapsulated in Ethernet frames.

What Makes Tesla's TTPoE Protocol Different?

Next, Tesla showed how Dojo is connected: Tesla 100G NIC to V1 Dojo interface card to Dojo.

What Makes Tesla's TTPoE Protocol Different?

Starting from the assembled D1 chip array, they connect to the interface cards via SerDes cables.

What Makes Tesla's TTPoE Protocol Different?

These then go to the Interface Cards.

What Makes Tesla's TTPoE Protocol Different?

Then, these interface cards are subsequently connected to low-cost 100G NICs.

What Makes Tesla's TTPoE Protocol Different?

In Tesla’s TTP network, the “Mojo” host operates through a variable data ingestion approach. The vision network may be limited by data ingestion, because vision-based tensors and training clips can be on the order of GB. The “Mojo” host is scheduled from the general compute pool on demand. Forward/backward pass TTP traffic is mutually independent, meaning data ingestion and all-reduce operations share the same TTP DIP port but execute at different stages of training.

What Makes Tesla's TTPoE Protocol Different?

The slide below shows the Mojo Dojo compute hall in New York. We can see that the 2U compute nodes have no front 2.5-inch storage.

What Makes Tesla's TTPoE Protocol Different?

This is a 4 ExaFLOP engineering system equipped with 40PB of local storage, along with a large amount of bandwidth and compute. Having a 4EF (BF16/FP16) engineering system is quite remarkable.

What Makes Tesla's TTPoE Protocol Different?

In actual testing, TTPoE was measured on Arista’s 7060, 7808, and 7816 switches, showing lower round-trip latency and high throughput.

What Makes Tesla's TTPoE Protocol Different?

Tesla joined the Ultra Ethernet Consortium (UEC) and made the TTPoE protocol publicly available to support a broader range of network operators.

What Makes Tesla's TTPoE Protocol Different?

Tesla thanked: the original inventor of TTPoE, the network deployment team, the silicon design team, the systems and infrastructure team, the software and driver team, the Linux patch team, the SDN team, the DevOps team, the QA team, the DC technology team, the supply chain team, and all TTP/Mojo interns for their contributions.

What Makes Tesla's TTPoE Protocol Different?

Finally, on network performance, Tesla believes that excessive focus on synthetic latency measurements should be reduced, because such measurements often fail to fully reflect real network performance. More meaningful differentiators include the comparison of lossy versus lossless networks; centralized versus distributed approaches to handling network congestion; and the difference between proprietary and open-source technologies.

What Makes Tesla's TTPoE Protocol Different?

Furthermore, it shows the measurement results of one-way write latency for a single switch hop under no-load conditions, providing latency data for several different technologies. Specifically, TTPoE latency is 1.0E-05s, InfiniBand (IB) is 2.0E-06s, NVLink is 2.3E-06s, RoCEv2 is 4.3E-06s, while TCP/IP has the highest latency at 5.3E-04s.

Tesla’s conclusion is that their system performance has reached the microsecond level.

Add me on WeChat: note “Data Communication” to join the Data Communication professional group!

What Makes Tesla's TTPoE Protocol Different?

Enjoy this article?

Subscribe to get new optical communication insights delivered to your inbox. Free, unsubscribe anytime.

Category: Technology

Leave a Reply

You must be logged in to post a comment.

订阅更新

订阅后,新文章发布时第一时间收到邮件通知

李东霏
光互连技术研究者

香农信息技术研究院,专注光模块、CPO、硅光与 AI 光互连的技术解读。

行业报告

光芯片·光器件·光模块 2026 报告汇编(含 OFC 2026)

光芯片 · 光器件 · 光模块
2026 报告汇编(含 OFC 2026)

¥1,900

上/下两册,收录 OFC 2026 一线报告
添加微信 WaytoICT 订购

广告

📢 广告位招商:面向全球光通信客户 侧边栏广告位 Email:Dongfei.li@shannoninstitute.cn

最新文档

  • 互连拐点:11 页解析机架内到多机架 Scale-Up 的 XPO/CPX/OCI 蓝图(OCP 2026 全球峰会)
  • AI 时代的相干光技术:17 页解析 Scale-Across、液冷 XPO 与全谱转发器(OCP 2026 全球峰会)
  • XPO 高密度光互连的系统级评估:20 页解析 212G SerDes 平台与 FRO/LRO/LPO 实测(OCP 2026 全球峰会)
  • CPO 与 OCI 标准及未来技术:17 页 Broadcom 精简版演讲提纲(OCP 2026 全球峰会)
  • VCSEL 高密度互连:7 页解析 50G NRZ「慢而宽」阵列与光 PCIe 演示(OCP 2026 全球峰会)

最新评论

暂无评论,欢迎登录后发表第一条评论。

最近访客

👥 今日访客 -
加载中…
© 2026 WaytoICT | 蜀ICP备2020035321号-2 | 隐私政策 | 免责声明
Wechat Pay
Alipay
请使用 支付宝 或 微信 扫码支付