Overview
CoCaRS, or Correlation Calibration-based Redundancy Suppression, is a proposed method designed to enhance heterogeneous knowledge distillation (KD). It aims to address issues arising from architectural inductive bias differences between teacher and student models, which can lead to significant representation discrepancies. The approach focuses on preserving structural information while suppressing redundancy, and reducing sensitivity to coefficient settings across various teacher-student pairs and training stages.
Research Context
Knowledge distillation is a paradigm for model compression where a compact student model learns from a more powerful teacher model. This technique has expanded from homogeneous settings, where teacher and student models share similar architectures, to heterogeneous settings, involving diverse model architectures. In heterogeneous KD, the architectural inductive biases of the teacher and student models can cause substantial differences in representation. These discrepancies limit the direct transfer of knowledge from the teacher to the student.
A recent perspective in heterogeneous KD involves redundancy suppression. This approach seeks to preserve cross-architecture invariance and reduce feature redundancy by decorrelating teacher-student feature correlations. However, existing formulations of redundancy suppression may inadvertently weaken useful structural information due to uniform decorrelation. Additionally, a fixed coefficient in these formulations can make the effectiveness of redundancy suppression sensitive to specific teacher-student pairs and training stages.
Approach
CoCaRS was developed to overcome the limitations of existing redundancy suppression methods by better retaining structural information, effectively suppressing redundancy, and mitigating sensitivity to coefficient settings. The method incorporates two primary components for calibrating feature decorrelation:
- Confusion Evidence Estimation (CEE): This component is designed to capture reliable semantic relations, which are crucial for accurate correlation estimation.
- Strength Allocation Control (SAC): This mechanism operates to preserve discriminative structural information during the decorrelation process.
Beyond feature decorrelation, CoCaRS integrates an additional mechanism to reduce sensitivity to coefficient settings:
- Adaptive Coefficient Regulation (ACR): ACR dynamically regulates the contribution of the calibrated redundancy suppression objective. It achieves this by considering the relative loss scale of the objective, thereby reducing the sensitivity to predefined coefficient values across different teacher-student configurations and training phases.
Findings
Experimental evaluations were conducted on two datasets: CIFAR-100 and ImageNet-1K. The results of these experiments indicated the effectiveness of CoCaRS in:
- Improving distillation performance.
- Reducing sensitivity to coefficient settings.
The source states that these experiments validate the effectiveness of CoCaRS.