Overview
CT-Merging is a proposed method for LoRA (Low-Rank Adaptation) adapter merging, addressing how common subspaces are estimated and how coefficients are assigned following recomposition. The method's design incorporates two primary mechanisms: the estimation of common directions through averaged task subspace projectors and the assignment of a distinct residual scale to each task.
Research Context
Existing LoRA merging methods frequently operate on the low-rank structure inherent in task updates. However, the direct comparison of techniques for common subspace estimation and post-recomposition coefficient assignment is observed to be rare within the field. This context highlights an area where CT-Merging seeks to provide a specific, directly compared approach.
Approach
CT-Merging employs two core components:
- Projector Averaging for Common Direction Estimation: This mechanism identifies common directions by averaging task subspace projectors. This selection process prioritizes directions supported across multiple task subspaces without weighting them according to singular magnitude.
- Task-Specific Scaling: This component assigns a separate residual scale to each individual task. Its function is to mitigate component-wise magnitude variation while concurrently preserving scale differences among tasks.
Findings
The efficacy of CT-Merging was evaluated on two distinct benchmarks:
KnOTS CLIP Adapters
On the released KnOTS CLIP adapters, CT-Merging demonstrated the highest average and worst-task normalized accuracy across both backbones tested. Specifically, the method achieved improvements over the strongest baseline:
- Up to $2.56$ points in average normalized accuracy.
- Up to $6.65$ points in worst-task normalized accuracy.
DC-Merge Adapter Benchmark
When evaluated on the DC-Merge adapter benchmark, CT-Merging attained the best average normalized accuracy in eight out of nine combinations of backbone and task-count settings.
Ablation Studies
Ablation studies were conducted to isolate the contributions of CT-Merging's core components:
- Projector Averaging vs. Summed-Update SVD: These studies indicated that projector averaging surpassed the performance of summed-update SVD (Singular Value Decomposition) for estimating common directions.
- Task-Specific Scaling vs. Global Isotropic Scaling: The ablations also revealed that task-specific scaling contributed to an improvement in worst-task accuracy when compared against global isotropic scaling.