A sparse attention mechanism with O(nlogn) computational complexity for long video generation. It can speed up both training and inference by 2–4×. The code is available at https://github.com/mit-han-lab/radial-attention
In the physical world, the energy of signals and waves decays over space and time. We observed a similar phenomenon in video diffusion models’ attention: post-softmax scores diminish with increasing spatial and temporal distance between tokens, as shown in the above figure. We term this Spatiotemporal Energy Decay and model it as an exponential function of both spatial and temporal distances. Inspired by this, we design Radial Attention to translate energy decay into exponentially decaying compute density.
Radial Attention uses a static pre-defined attention mask where each token attends to spatially nearby tokens, with the attention window size shrinking with temporal distance. This design focuses computational resources on the most relevant interactions while maintaining the efficiency.
To ensure efficient execution on modern hardware, attention is computed over blocks rather than individual 1×1 tokens. Our implementation uses a block size of 128×128.
Radial Attention also helps reduce training complexity when adapting models for longer videos. Because it preserves the original softmax attention, the pre-trained weights remain largely reusable, requiring only minimal fine-tuning. To further reduce training overhead, we incorporate Low-Rank Adaptation (LoRA) into the attention mechanism. Specifically, LoRA is applied to the query, key, value, and output projections of the attention layers, enabling efficient fine-tuning with significantly reduced memory and computational costs. Empirically, we find that LoRA fine-tuning with Radial Attention not only minimizes overhead but also improves video quality by refining only the most critical weights and concentrating attention more effectively.
Radial Attention accelerates pre-trained HunyuanVideo by 1.9× at its default video length while maintaining comparable video quality. When generating 4× longer videos, it reduces tuning costs by up to 4.4× and speeds up inference by up to 3.7× versus dense attention.
A key advantage of Radial Attention is its seamless compatibility with pre-trained LoRAs. As demonstrated above, applying Radial Attention to the 8-step FusionX LoRA results in an additional 1.6× speedup, reducing the latency of generating a 4s 720P video to 84s on a single H100.
Furthermore, LoRAs trained for extended-length generation with Radial Attention can also be combined with existing style LoRAs. As shown in the above figure, this combination maintains visual quality while enabling longer video generation.
