Long-Context Transformer Models (LCTMs) are vital for real-world applications but suffer high computational costs due to attention's quadratic complexity. Block-sparse attention mitigates this by focusing computation on critical regions, yet existing methods struggle with balancing accuracy and efficiency due to costly block importance measurements. In this paper, we introduce XAttention, a plug-and-play framework that dramatically accelerates long-context inference in Transformers models using sparse attention. XAttention's key innovation is the insight that the sum of antidiagonal values (i.e., from the lower-left to upper-right) in the attention matrix provides a powerful proxy for block importance. This allows for precise identification and pruning of non-essential blocks, resulting in high sparsity and dramatically accelerated inference. Across comprehensive evaluations on demanding long-context benchmarks—including RULER and LongBench for language, VideoMME for video understanding, and VBench for video generation—XAttention achieves accuracy comparable to full attention while delivering substantial computational gains. We demonstrate up to 13.5x acceleration in attention computation. These results underscore XAttention's ability to unlock the practical potential of block sparse attention, paving the way for scalable and efficient deployment of LCTMs in real-world applications.
Within each block of size B, we select elements along the antidiagonal using a stride S to predict importance of the whole block. Our aim is to optimize the following two objectives:
Information Preservation: Ensure that information from all tokens is considered, as each token contributes to at least one antidiagonal sum.
Pattern Detection: Antidiagonal intersects every possible vertical and slash pattern within a block.
We evaluate X-attention across diverse long-context benchmarks and models, demonstrating consistently lower pre-fill latency than existing sparse-attention methods, without compromising accuracy.
We apply Stride S = 16 and threshold τ = 0.9 parameters on the QwenVL-2-7B model.
@inproceedings{xu2025xattention,
title = {XAttention: Block Sparse Attention with Antidiagonal Scoring},
author = {Xu, Ruyi and Xiao, Guangxuan and Huang, Haofeng and Guo, Junxian and Han, Song},
booktitle = {Proceedings of the 42nd International Conference on Machine Learning (ICML)},
year = {2025}
}
We thank MIT-IBM Watson AI Lab, MIT and Amazon Science Hub, MIT AI Hardware Program, National Science Foundation, Hyundai, and Samsung for supporting this research. We thank NVIDIA for donating the DGX server.
