Overview
ProgResViT is presented as an input-adaptive Vision Transformer (ViT) designed to manage computational efficiency during image processing. Traditional Vision Transformers typically employ a fixed input resolution and model width for every image. ProgResViT diverges from this standard by implementing a progressive inference mechanism across multiple rounds, adjusting its computational demands based on prediction confidence.
Research Context
Vision Transformers (ViTs) conventionally operate with a static computational profile, applying a predefined input resolution and model width uniformly across all images. This uniform processing can lead to inefficiencies, as many images might be classifiable with significantly less computational expenditure than the maximum allocated. The research addresses this by proposing an adaptive inference strategy that scales computation according to the perceived difficulty of the classification task.
Approach
ProgResViT operates through a multi-round progressive inference process. In its initial round, it processes a low-resolution image input utilizing a narrow subnetwork. The system's design incorporates a confidence-based termination condition: if the prediction achieved in a given round is deemed sufficiently confident, inference concludes. Should the confidence threshold not be met, the model proceeds to subsequent rounds. For these subsequent rounds, ProgResViT reuses the representations generated in the current round. It then progresses to process a higher-resolution input, leveraging a wider subnetwork to refine its prediction. This iterative refinement continues until sufficient confidence is achieved or the maximum computational round is reached.
A central architectural feature enabling this progressive adaptation is the sharing of a single backbone across all inference rounds. To facilitate the dynamic adjustment of processing based on the current round, block, and input resolution, the researchers introduce Progress-Conditioned Soft Gating (PSG). This mechanism explicitly conditions token fusion and layer outputs on these contextual parameters, allowing the network to adapt its internal operations progressively.
Findings
- Applying ProgResViT to DeiT yielded improved accuracy-compute trade-offs compared to baseline models employing adaptive-width, adaptive-depth, and dynamic-token strategies for image classification tasks.
- When combined with knowledge distillation, a DeiT-based ProgResViT achieved a top-1 accuracy of 84.9%. This performance level slightly surpassed the reported accuracy of DeiT-III-S under a comparable evaluation setting.
- The design principles of ProgResViT were also demonstrated to provide favorable accuracy-compute trade-offs for self-supervised DINO representations.
- The same adaptive design was observed to offer beneficial accuracy-compute trade-offs in downstream semantic segmentation tasks.
Why This Matters
The development of ProgResViT addresses the computational inefficiency inherent in standard Vision Transformers by introducing an adaptive processing paradigm. By varying resolution and model width based on inference confidence, it offers a method to achieve accuracy targets with potentially reduced computational demands. This has implications for deploying ViTs in environments where computational resources or energy consumption are constrained, or where rapid inference for simpler cases is desirable without sacrificing performance on more complex inputs.