Overview
Video vision-language models (VLMs) are increasingly deployed in long-horizon and streaming applications. A common challenge in these models is the quadratic scaling of compute and latency with the number of frames, primarily due to the reliance on spatiotemporal self-attention within most video encoders.
StateKV is an inference-time method developed to address this scaling issue. It enables linear-time video prefill for pretrained long-video VLMs. The method operates by maintaining cross-frame context through a fixed-capacity, importance-based recurrent state, in conjunction with a distinct full per-frame cache designated for decoding.
Research Context
The inherent architecture of many video encoders, particularly their use of spatiotemporal self-attention, leads to a computational cost and latency that grows quadratically with the number of frames. This quadratic scaling limits the applicability of VLMs to progressively longer videos and real-time streaming scenarios.
Existing efficiency methods aim to mitigate this scalability problem. However, these methods often introduce a trade-off, where improvements in scalability come at the cost of accuracy degradation relative to models employing full self-attention. Such efficiency gains are frequently achieved through aggressive frame or token dropping, or by implementing coarse approximations of the attention mechanism.
Approach
StateKV functions as an inference-time adaptation for pretrained long-video VLMs. Its core mechanism involves two key components for managing contextual information:
- Fixed-Capacity, Importance-Based Recurrent State: This component is responsible for carrying cross-frame context. Its fixed capacity ensures that computational requirements do not scale quadratically with video length. The 'importance-based' aspect suggests a mechanism for selectively retaining salient information across frames.
- Full Per-Frame Cache: This second cache is utilized specifically for the decoding process. Its purpose complements the recurrent state, potentially by providing richer, short-term frame-specific information crucial for generation or prediction at each step.
A notable aspect of StateKV is that it operates without requiring fine-tuning of the models or introducing any architectural modifications to the existing VLM structures. It is presented as a method deployed during the inference phase.
Findings
Evaluations of StateKV were conducted across three distinct long-video benchmarks. The method was tested with seven different models, encompassing three model families and multiple scales.
- StateKV maintained accuracy levels that were comparable to those achieved by full self-attention mechanisms.
- The method consistently surpassed the performance of dominant streaming approximations, specifically those based on sliding-window or recency-based strategies.
- StateKV did not necessitate fine-tuning or architectural modifications to the evaluated models.
- It demonstrated a reduction in video-prefill cost, measured in FLOPs.
- This reduction in computational cost allowed for the use of larger models under a fixed compute budget, leading to stronger accuracy.
Why This Matters
The findings regarding StateKV suggest a practical advancement toward achieving scalable long-video understanding. Its ability to enable linear-time video prefill while largely preserving accuracy or outperforming current streaming efficiency methods, without alterations to existing VLM architectures, is significant. This could allow for the deployment of larger, more capable models in applications constrained by computational resources or latency requirements.