Triton versus Rust and CUDA C++ in AI and 3D mapping

New Programming Languages versus Rust / C++

Meta’s chip designers faced a moving target. Every change to a recommendation model could call for a new operation, and writing a fast implementation for their accelerator took specialist time. Triton let model developers describe work in blocks and left more of the translation to a compiler. A 3D mapping experiment shows the limits of that bargain. When each point from a depth camera has to find its own place in a shared table, the freedom to stop searching at just the right moment matters.

The creator

The idea behind a tile

The language’s creator explains the design in a recorded presentation. Watch the original PyTorch talk.
The original question
Could researchers express an operation on a whole tile of numbers and still give a compiler enough information to generate efficient machine code?
From research to use
The creator, known as ptillet, presented the design with co-authors in 2019. OpenAI’s 2021 account describes how researchers used the language for custom neural-network operations. Original research paper · OpenAI’s account.

How Triton runs a calculation

Consider adding two arrays whose lengths do not divide evenly into blocks. The host application allocates the output and passes the addresses of the three arrays to the device. Each instance of the Triton program works on one block. It calculates which positions belong to it, loads the two inputs and writes back their sums. On the last block, the author supplies a mask so that only positions inside the arrays are read or written. The host still manages the lifetime of those arrays. Triton’s worked example.

The compiler specialises the function for the device when it is needed. On NVIDIA GPUs it lowers the tile operations to progressively more hardware-specific instructions, ultimately producing code that runs on the GPU. Rust and CUDA C++ also let developers write code that executes on the device. They give the author a different way to express what an individual thread does when neighbouring threads take different paths. PyTorch’s account of the compilation stages · GPU implementations compared in the mapping study.

Why Meta brought Triton to its own accelerator

Meta wanted to try new model operations without commissioning an expert-written C++ implementation for each one. Its MTIA-2i accelerator, designed for inference, also posed a translation problem. Software must move tiles into on-chip memory and schedule commands for specialised units. Meta adapted the Triton compiler to that hardware and taught PyTorch’s TorchInductor compiler to generate suitable Triton operations. This gave the team two routes to deployment. A specialist could write an operation directly in Triton, or TorchInductor could generate one from part of a model. Meta’s account of the architecture and deployment, sections 2–4 and 7.

Combining several operations can save a round trip to memory. If one operation produces a value that the next immediately consumes, a combined kernel can keep it on the chip. Across roughly 60 model types, Meta reports that handwritten and generated Triton kernels accounted for 50% of layers and 47% of execution time outside general matrix multiplication. Those percentages describe the reach of Triton in that deployment. On the same chip, the team also compared matrix multiplication in Triton with expert-tuned C++. The Triton kernels reached over 80% of the chip’s theoretical performance limit in those tests and performed on a par with the C++ implementations. Meta’s production figures.

Where Rust and CUDA C++ finish sooner

A separate study put all three languages on the same GPU with the same algorithm and memory layout. The task was to integrate a simulated depth frame into a three-dimensional map. For each observed point, the first stage finds or reserves a block in a hash table. A collision means another location must be tried. The second stage updates the small volume elements, or voxels, within the blocks. The Rust version used cuda-oxide to compile device code. 3D reconstruction study, methods and Tables 2–3.

Time to reserve blocks and update voxels on an RTX 5070 Ti. Total elapsed time was 0.388 ms in CUDA C++, 0.396 ms in Rust and 1.132 ms in Triton.
Figure 1. Time to incorporate 407,444 points from TartanAir’s RetroOffice P000 sequence into an already populated map on an NVIDIA GeForce RTX 5070 Ti. Both stages use the same time scale. Enlarge the figure · Downloadable measurements.
Elapsed time in milliseconds. Each stage is the median of three run medians. Totals were calculated before rounding.
ImplementationReserve blocksUpdate voxelsCombined
CUDA C++0.0230.3650.388
Rust0.0290.3670.396
Triton0.6610.4711.132

Triton took 2.92 times as long as CUDA C++ and 2.86 times as long as Rust across the two stages. Block reservation explains most of the gap. Individual threads in the Rust and CUDA C++ versions could finish searching once they found a slot. The tested Triton implementation used a bounded loop and redirected inactive lanes to scratch locations for an atomic operation. Against CUDA C++, reservation took about 28.5 times as long, while updating voxels took about 1.29 times as long. The fixed search limit also had a correctness cost at high table occupancy, where a block could be missed. The figures describe the mapping implementations and scene tested in this preprint. Neural-network inference involves a different workload. Study of the search loop and its limits · Stage timings.

What this language can contribute to artificial intelligence

Meta put a fused attention operation into production on MTIA and tested configurations representative of recommendation and generative models. Attention combines information from different positions in an input. Keeping intermediate results on the accelerator can reduce the cost of moving them between operations. In a separate study of training a recommendation model, Meta found that the kernels generated by TorchInductor were 31% faster in aggregate than the handwritten kernels they replaced. Across the complete model, throughput increased by 4% in queries per second over its baseline. The unchanged parts of the model still consumed time. Meta’s attention and recommendation results, sections 7.1.2 and 7.2.2.

OpenAI also documents using Triton for custom GPU operations in its research. Together, the two accounts establish a use for the language in developing and accelerating model operations. Memory masks can keep a correctly written operation within the intended range of an array. Model integrity, resistance to manipulated inputs and protection of training data are separate questions. The cited experiments did not evaluate them. OpenAI’s account of its research use · Triton’s memory-load semantics.

Which part of the workload determines the choice?

For a model team adding operations to a specialised accelerator, Meta’s experience gives a concrete reason to try Triton. Measure whether the team can bring a new operation into service sooner and whether the complete model handles more queries. For a mapping team, the revealing measurement is the time to find an available block as the map fills. Check that every expected block survives as well. In the published depth-frame test, Rust and CUDA C++ made that search far cheaper. The question for a new application is how much of its time it spends in each kind of work.

Liked this post? Follow this blog to get more.