LongVILA: Scaling Long-Context Visual Language Models for Long Videos

💡 LongVILA Training Strategy

The LongVILA training pipeline consists of five stages: the first three stages: (1) multi-modal alignment using open-sourced datasets, (2) large-scale pre-training to develop general multi-modal capabilities, and (3) supervised fine-tuning for short video understanding—are default processes in VILA, allowing the use of pre-trained Vision-Language Models (VLMs) as a foundation for LongVILA. The subsequent stages involve (4) context extension through continued pre-training on text-only datasets to extend the model's context length incrementally, and (5) long supervised fine-tuning with a dedicated long video dataset using the MM-SP system to enhance long-context understanding and instruction-following abilities, with all parameters trainable in the final stage.

‍

💡 LongVILA SFT Dataset

The LongVILA SFT dataset is designed for training models on long video content, consisting of 15,292 videos from the Shot2Story dataset. Each video is segmented into shorter clips, approximately 10 seconds each, which are individually annotated with captions using the VILA-1.5 model. Subsequently, a large language model generates question-and-answer pairs based on the captions, covering a diverse range of queries such as summarization and content-related questions. The dataset includes a broad variety of video categories, such as Travel, Sports, Education, and Entertainment, as well as question types like spatial, temporal, and synopsis-based queries, ensuring comprehensive coverage for long video understanding and supervised fine-tuning.

‍

💡 Multi-Modal Sequence Parallelism System

The Multi-Modal Sequence Parallelism (MM-SP) system is an efficient and scalable solution designed to handle the high memory and computation demands of training long-context Vision-Language Models (VLMs). It employs a novel two-stage sharding strategy: first balancing the workload between image encoding and language modeling stages, and then distributing tokens evenly across GPUs for efficient sequence parallelism. Unlike traditional ring-style or hybrid parallelism approaches, MM-SP optimizes intra-node and inter-node communication by using All-to-All and Point-to-Point strategies, enhancing bandwidth utilization and reducing overhead. This system achieves significant speedups (2.1× to 5.7× compared to ring-style systems) and supports context lengths up to 2 million tokens without gradient checkpointing, seamlessly integrating with frameworks like Hugging Face Transformers for both training and inference of long-context multi-modal tasks.

💡 Experimental Results

The experiment results demonstrate the effectiveness of LongVILA in handling long-context Vision-Language tasks, achieving state-of-the-art performance across nine popular video benchmarks, including VideoMME, ActivityNet-QA, and EgoSchema. LongVILA supports significantly extended context lengths, with training scalability up to 2 million tokens on 256 GPUs, outperforming existing systems like ring-style parallelism and Megatron-LM in both speed (2.1× to 5.7× faster) and sequence length capability. In the Needle-in-a-Haystack experiment, LongVILA achieved 99.8% accuracy on 6,000-frame videos, surpassing the baseline models. Additionally, the MM-SP system improves training efficiency and inference latency, with a 1.3× speedup and the ability to process sequences 2.5× longer compared to Megatron. The model demonstrates robust performance for both short and long video tasks, solidifying its capability for comprehensive video understanding.

‍

‍