Overview
AnchorPrune is a training-free framework designed for visual token pruning in large vision-language models (VLMs). Its primary aim is to mitigate the substantial inference costs associated with high-resolution inputs, which introduce numerous visual tokens, many of which can be redundant for a specific query. The framework operates by first establishing a 'relevance anchor' and then expanding it with complementary visual context.
Research Context
Large vision-language models face significant inference cost challenges due to the high volume of visual tokens generated from high-resolution inputs. Traditional pruning methods often attempt to balance query relevance and token diversity. However, under aggressive compression scenarios, these objectives can conflict. Relevance-driven selection might overconcentrate the allocated budget on correlated local evidence, potentially leading to a narrow view. Conversely, diversity-driven selection risks suppressing indispensable tokens or retaining distinct but uninformative regions, compromising the model's performance without achieving true efficiency.
Approach
AnchorPrune employs a two-stage, ordered design for visual token pruning. The framework does not require retraining or modification of the underlying model architecture.
Relevance Anchor Construction
- The initial step involves constructing a protected relevance anchor. This anchor represents a compact set of query-critical evidence.
- The adaptive determination of anchor size is based on the novelty profile of relevance-ranked tokens.
- This process aims to secure essential query cues before any contextual expansion.
Contextual Expansion
- Following anchor construction, the remaining budget for visual tokens is allocated to expand the context.
- This expansion is guided by importance-weighted novelty, which enables the recovery of informative yet non-redundant contextual information relative to the previously established anchor.
- This ordered design ensures that the contextual expansion process does not displace indispensable query cues while simultaneously improving overall visual coverage.
The framework is described as being lightweight and architecture-aware.
Findings
- AnchorPrune consistently improved the accuracy-efficiency trade-off when compared against training-free baselines. This improvement was particularly noticeable under conditions of severe compression.
- The framework was evaluated across both image and video vision-language models and benchmarks.
- On the LLaVA-NeXT-7B model, AnchorPrune preserved 97.6% of the full-token performance. This was achieved while utilizing only 160 out of 2,880 visual tokens available.
- The results indicate that relevance-anchored contextual expansion can be an effective principle for achieving efficient multimodal inference.
Why This Matters
The substantial inference costs of large vision-language models, particularly with high-resolution inputs, represent a practical bottleneck. By addressing this through an efficient token pruning mechanism, AnchorPrune offers a method to maintain high model performance while significantly reducing computational overhead. This could contribute to more deployable and sustainable VLM applications.
Potential Applications
- Reducing computational resources required for deploying large vision-language models in real-world scenarios.
- Improving the efficiency of multimodal inference tasks in applications that process high-resolution image or video data.