Overview
The role of task symmetries in designing deep learning solutions has gained significance. Data augmentation offers a method to integrate these symmetries into neural networks. While infinitely large deep ensembles exhibit perfect symmetry when trained with augmented data, the computational cost associated with training multiple ensembles presents a practical challenge. This work explores stochastic weight averaging (SWA) as an alternative ensembling technique that circumvents the need for repeated training processes.
Research Context
Deep learning models benefit from the incorporation of task symmetries. Data augmentation is a recognized and effective strategy for introducing such symmetries into general neural network architectures. Previous findings indicate that infinitely large deep ensembles can achieve perfect symmetry when trained on augmented datasets. However, the requirement to repeat the training process multiple times for ensemble creation leads to substantial computational expense.
Approach
The study investigates stochastic weight averaging (SWA) as an ensembling method that does not necessitate multiple training runs. The analysis of SWA is conducted by approximating the stochastic training trajectory observed at the conclusion of training using an Ornstein–Uhlenbeck process. This approximation is central to understanding SWA's behavior.
Findings
- SWA serves as an alternative ensembling technique that eliminates the need for repeated training runs, distinguishing it from traditional ensemble methods.
- In the infinite-width limit, SWA applied to augmented data provides an 'equivariance boost'.
- This observed equivariance boost surpasses the performance increase that would be attributable solely to SWA's standalone benefits.
- Numerical experiments extensively verified these results.
- The experiments covered numerous models.
- The models spanned computer vision and graph classification tasks.
- Both discrete and continuous symmetries were investigated within these experimental setups.
Why This Matters
The findings suggest a method to enhance the benefits of data augmentation for symmetry incorporation in deep learning models without incurring the high computational costs associated with traditional large deep ensembles. By demonstrating an equivariance boost from SWA on augmented data, this research provides a potentially more efficient approach to leverage task symmetries.