Overview
Vision-Language Models (VLMs) demonstrate capabilities in visual reasoning; however, their inference costs increase with the quantity of visual tokens. Existing methods for visual token pruning often encounter two primary limitations. Firstly, most current approaches operate after the vision encoder, leaving the latency associated with the visual encoding phase unoptimized. Secondly, these methods frequently struggle to jointly maintain both holistic visual contexts and fine-grained details when operating under strict token budgets, which can lead to performance degradation.
In response to these challenges, a training-free inference framework named PACE (Pixel-Adaptive Condense and Extract) has been proposed. PACE aims to accelerate both the vision encoder and the Large Language Model (LLM) components of VLMs through a unified Condense-and-Extract paradigm.
Research Context
The inherent architecture of Vision-Language Models, while enabling robust visual reasoning, is susceptible to escalating inference costs directly proportional to the number of visual tokens processed. This issue stems from the computational burden associated with handling large volumes of visual data during both the encoding and subsequent language model processing stages.
Prior attempts to mitigate these costs have typically focused on pruning visual tokens. However, these methods commonly initiate their operations post-vision encoder. This sequential application means that the substantial computational demands and latency incurred during the initial visual encoding phase remain unaddressed. Furthermore, when faced with stringent constraints on the number of tokens, these previous methods often face a trade-off, struggling to simultaneously preserve the broader contextual understanding of an image and its intricate, fine-grained details. This can result in a measurable reduction in the model's overall performance.
Approach
PACE implements a two-stage unified Condense-and-Extract paradigm to optimize VLM inference:
Condense Stage
During the Condense stage, an Adaptive Pixel Compressor (APC) is employed. The APC's function involves evaluating visual information density prior to the visual encoding process. Based on this evaluation, the APC adaptively downsamples redundant input pixels. This action serves to curtail the computational requirements of the encoder while simultaneously aiming to preserve the global context and essential visual cues present in the input.
Extract Stage
Following the Condense stage, the Extract stage utilizes a Dynamic Dual-Attention Extractor (DDAE). The DDAE operates by selectively retaining visual tokens. This selection process is driven by a fusion of two signal types: internal visual signals originating from the vision encoder and semantic signals provided by the Large Language Model (LLM). This dual-attention mechanism is designed to safeguard task-critical details within the remaining visual tokens.
Findings
The integration of the PACE framework into Qwen2.5-VL-7B yielded specific performance metrics:
- The model retained 93.8% of its original performance.
- This performance retention was achieved while utilizing only 10% of the original visual tokens.
- The implementation resulted in a 3.1x speedup in time to first token (TTFT).
Why This Matters
The acceleration of both the vision encoder and the LLM components via a unified framework addresses current VLM efficiency bottlenecks. By significantly reducing visual token usage and improving TTFT, PACE offers a method to maintain high performance in VLMs while reducing the computational burden associated with their operation.