Why does an AI accelerator wait for memory? Inside high-bandwidth memory (HBM)

The next token in a conversation has reached the accelerator that will process it. The model’s weights are already on the package. Even so, the arithmetic units can spend part of the step waiting. If the data are that close, what takes the time?

High-bandwidth memory (HBM) gives an accelerator many short electrical paths over which to read and write data in parallel. The design helps when large amounts of data must move before the next calculation can finish. A revealing experiment changed the software on a single GPU: its attention calculation finished sooner even though it performed more arithmetic. The researchers had found a way to leave much of the intermediate work inside the processor. [1] [7]

Cutaway of a stack of memory dies on the left and an accelerator die on the right, joined by short traces on a common interposer.
Conceptual cutaway. Vertical connections link the memory layers on the left. Short traces cross the interposer to the accelerator on the right. Layer count, spacing and conductor widths are illustrative. © 2026 Juan Antonio Lloret Egea · ITVIA · The Bible of AI (BAI) · CC BY-NC-ND 4.0.

Why stack the memory beside the processor?

HBM is working memory built from DRAM, which holds information as electrical charge and requires refresh. Several thin memory dies sit one above another. Vertical conductors called through-silicon vias carry signals through the stack. In a common package design, a fine wiring layer called an interposer connects the base of the stack to the nearby processor. Stacking puts more memory on a small footprint. The wide interface lets many bits cross the short distance side by side. [1] [2]

That arrangement changes what the package designer must solve. Thousands of signal connections have to work, power must reach the dies, and heat must escape the stack. In February 2026 Samsung reported commercial shipments of HBM4 and an increase from 1,024 to 2,048 data pins compared with its HBM3E. In June, SK hynix reported sending customers samples of its next-generation HBM4E. The first announcement concerns commercial memory and the second concerns samples for qualification. Performance on a language model requires a separate test. [2] [3]

How much fits, and how fast can it be read?

Capacity determines whether the weights, working buffers and stored context fit in the available memory. Bandwidth measures how many bytes can cross the interface in a second. Latency measures how long a particular request takes to return. A wide interface delivers its rated bandwidth only when enough useful transfers are in flight. A small read whose address depends on the read before it may use only a fraction of that width.

Consider two documented implementations. AMD specifies 192 GB of HBM3 and 5.3 TB/s peak memory bandwidth for one Instinct MI300X accelerator. Google lists 2,765 GB/s of HBM bandwidth for one TPU v5p chip and distinguishes its 96 GiB physically present from the 95 GiB available to users. A GB is a billion bytes and a GiB is 2³⁰ bytes. These specifications help a designer budget space and data traffic. The time a user waits for an answer requires a workload measurement. [4] [5]

What changes while an AI model answers?

When a language model first processes a prompt, many input positions can be handled together. During generation, each request needs the preceding token before it can produce the next one. At small batch sizes, the accelerator may read a large set of weights to perform relatively little arithmetic per byte. Grouping more requests allows some of the same weights to serve several sequences before they have to be read again. This can increase total throughput even if an individual user waits longer. [6]

A long conversation creates another demand. A transformer can retain the keys and values produced for earlier tokens so it can attend to its history without recomputing them. This KV cache takes space alongside the model weights. With ordinary full attention, each new step may also have to read a growing history. Extra capacity can admit more conversations or longer ones. Extra bandwidth matters when reading those stored values is delaying the next token. [6]

The experiment that did more arithmetic and finished sooner

Standard attention can write a large array of intermediate scores out to HBM, then read it back to compute the output. FlashAttention divides the work into tiles that fit in the GPU’s smaller on-chip SRAM and accumulates partial results there. During the backward pass it recomputes some values. This preserves the mathematical attention operation and exchanges extra arithmetic for fewer trips to HBM. [7]

Measured attention operation on one NVIDIA A100, forward and backward passes combined. Dao and colleagues, Figure 2, left panel.
QuantityStandard attentionFlashAttention
Arithmetic (GFLOPs)66.675.2
HBM reads and writes (GB)40.34.4
Runtime (ms)41.77.3

The test used the attention dimensions of GPT-2 medium: 1,024 positions, 16 heads, 64 components per head and a batch of 64. The same hardware ran both implementations. The measured time covers that attention operation during training. It therefore tells us what avoiding memory traffic achieved in this experiment. It does not establish a speedup for a complete training run or for an interactive reply. [7]

Where HBM enters the Ascend story

In the Horizons article on the Ascend 960DT and Atlas 960E, a router sends a token’s numerical representation to an expert on another accelerator. Follow it one step further. Once the message arrives, the receiving processor needs the expert’s weights. Values already in its on-chip memory can be reused immediately. Other weights must be brought in from local HBM. The processor computes the expert’s contribution and the interconnect carries the result back. Speeding up either transfer helps only to the extent that the next calculation was waiting for it.

Huawei announced up to one petabyte of HBM across one Atlas 960E SuperPoD, configurable with up to 4,096 neural processing units. That is aggregate capacity across the system. The local capacity of an individual Ascend 960DT and the sustained read rate of one expert are separate quantities. Huawei’s September 2026 schedule put that accelerator’s availability in the first quarter of 2027. [8]

What would make better HBM visible to a user? Run the same model at the same precision and quality, with the same prompt lengths, output lengths and concurrency. Measure the wait for the first token, the interval between subsequent tokens, and how many requests finish within those limits. Record the bytes transferred to and from HBM alongside the energy consumed by the system. A shorter wait coupled with less time stalled on memory would connect the package design to the answer on the screen.

Sources and scope

Evidence checked on 17 November 2026. Product figures below retain the manufacturer’s unit and system boundary. The attention experiment supplies a measured comparison on one GPU. The inference study examines how memory, computation and communication interact as the workload changes.

  1. AMD. Versal Adaptive SoC Technical Reference Manual, AM011. High-Bandwidth Memory Interface. Physical connection between DRAM stacks, interposer and processing die. Technical manual.
  2. Samsung Electronics. Commercial shipment of HBM4. 12 February 2026. Memory generation, interface width and thermal design. HBM4 announcement.
  3. SK hynix. SK hynix Ships Samples of 12-Layer Next-Gen HBM4E. 18 June 2026. Sample shipments for customer evaluation. HBM4E sample announcement.
  4. AMD. Instinct MI300X Accelerators. Product specifications, memory type, capacity and peak memory bandwidth per accelerator. Accelerator specifications.
  5. Google Cloud. TPU v5p. Chip specifications and the distinction between physical and user-allocatable HBM. TPU specifications.
  6. Pope, R., et al. Efficiently Scaling Transformer Inference. Proceedings of Machine Learning and Systems, 2023. Prompt processing, token generation and the effect of batching on inference. Inference study.
  7. Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. Advances in Neural Information Processing Systems 35, 2022. Figure 2, left panel, measures arithmetic, memory traffic and attention runtime on an A100. Original study and figure.
  8. Huawei. Advancing the Agentic World, Building a Solid Silicon Foundation. HUAWEI CONNECT, 17 September 2026. Atlas 960E configuration and Ascend 960DT availability forecast. System announcement.

Explore Horizons of Electronics

Liked this post? Follow this blog to get more.