Overview
Research introduces an efficient training protocol named Recursive Block-Diagonal Coupling (RBDC) designed to address the substantial computational resources required for training high-capacity vision models from scratch. RBDC constructs wide models by integrating narrower, independently trained models in a recursive, parameter-free block-diagonal manner.
Research Context
Training high-capacity vision models conventionally demands significant computational expenditure. Existing methods aimed at improving training efficiency for target models often rely on the availability of narrower models, potentially obscuring the complete computational cost of the entire training pipeline. This context highlights a gap that RBDC aims to bridge by providing a more transparent and flexible approach to computational budget allocation.
Approach
The RBDC protocol operates by recursively coupling narrower, independently trained models. This coupling is achieved in a block-diagonal fashion and is parameter-free. The recursive nature of RBDC allows for flexible allocation of the training budget across all models involved in the process. The evaluation of RBDC involved its application to vision transformers (DeiT) and convolutional networks (ResNet) on the ImageNet dataset.
Findings
- The RBDC training protocol demonstrated a notable improvement in efficiency compared to models trained from scratch using the standard protocol.
- Specifically, RBDC achieved a 30% reduction in FLOPs (Floating Point Operations) while maintaining similar test accuracies.
- When compared to training protocols from the model growth literature, RBDC yielded higher performances at the same level of training FLOPs.
- Models developed using RBDC were observed to serve as more effective backbones than their original counterparts for downstream tasks, including object detection and instance segmentation.
Why This Matters
The proposed RBDC protocol offers a mechanism to reduce the computational demands associated with training high-capacity vision models without sacrificing accuracy. This efficiency gain could free up resources, potentially making advanced model training more accessible or allowing for the development of even larger, more complex models within existing computational constraints. Furthermore, the enhanced performance of RBDC models as backbones for downstream tasks suggests a broader utility across various computer vision applications.