Vision-language models (VLMs) could power real-time assistants and autonomous agents, but they face a critical challenge: understanding near-infinite video streams without escalating latency and memory usage. Processing entire videos with full attention leads to quadratic computational costs and poor performance on long videos. Meanwhile, simple sliding window methods are also flawed, as they either break coherence or suffer from high latency due to redundant recomputation. In this paper, we introduce StreamingVLM, a model designed for real-time, stable understanding of infinite visual input. Our approach is a unified framework that aligns training with streaming inference. During inference, we maintain a compact KV cache by reusing states of attention sinks, a short window of recent vision tokens, and a long window of recent text tokens. This streaming ability is instilled via a simple supervised fine-tuning (SFT) strategy that applies full attention on short, overlapped video chunks, which effectively mimics the inference-time attention pattern without training on prohibitively long contexts. For evaluation, we build Inf-Streams-Eval, a new benchmark with videos averaging over two hours that requires dense, per-second alignment between frames and text. On Inf-Streams-Eval, StreamingVLM achieves a 66.18% win rate against GPT-4o mini and maintains stable, real-time performance at up to 8 FPS on a single NVIDIA H100.
Current VLMs are not designed for continuous, real-time video streams. Their core attention mechanisms break down when faced with long-running inputs, making them impractical for real-world applications.
StreamingVLM introduces a unified training and inference framework that enables efficient and coherent understanding of infinite video streams.
At the core of our inference strategy is a compact, fixed-size KV cache that intelligently manages context. Instead of letting the cache grow indefinitely, we reuse the states of:
This asymmetric design keeps latency low while preserving all the necessary context for coherent, long-horizon reasoning. To prevent positional drift when old tokens are evicted, we use Contiguous RoPE, which shifts positional indices to keep them in-distribution and stable.
Our inference scheme maintains a stable KV cache by retaining attention sinks, a long window of recent text, and a short window of recent vision, while evicting older tokens.
How do we teach the model this streaming behavior without training on prohibitively long videos? We use a simple but powerful SFT strategy: the model is trained with full attention on short, temporally-overlapped video chunks (e.g., training on 24-second clips that overlap by 12 seconds). This design naturally forces the model to learn the desired attention pattern—relying on initial tokens (sinks) and a sliding window of recent context—perfectly aligning the training process with our efficient inference scheme.
Our training strategy (right) uses full attention on overlapped chunks to mimic the effective attention pattern of our streaming inference method (left), bridging the train-inference gap.
StreamingVLM is not just accurate; it's truly real-time. It maintains a low and stable per-token latency, processing video at up to 8 FPS on a single NVIDIA H100 GPU. Unlike other methods, its performance does not degrade as the video gets longer.
Latency comparison of different methods. StreamingVLM (purple line) stays consistently low and stable, well below the real-time threshold, while others eventually fail or become too slow.
On our challenging Inf-Streams-Eval benchmark, which features videos averaging over two hours, StreamingVLM significantly outperforms strong baselines.
StreamingVLM provides coherent, real-time commentary with long-term memory, addressing the failures of other models which suffer from incoherence, high latency, or memory loss.
To spur research in this area, we introduce a new suite of data and evaluation tools. The process involves collecting over 6,000 hours of sports footage, performing ASR, and using a GPT-5-based pipeline for cleaning and annotation.
Our pipeline for creating the Inf-Streams dataset, from video collection and ASR to LLM-based cleaning and segmentation.
@misc{xu2025streamingvlmrealtimeunderstandinginfinite,
title={StreamingVLM: Real-Time Understanding for Infinite Video Streams},
author={Ruyi Xu and Guangxuan Xiao and Yukang Chen and Liuning He and Kelly Peng and Yao Lu and Song Han},
year={2025},
eprint={2510.09608},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2510.09608}
}
