Overview
Aggressive quantization, specifically within the sub-4B INT3 GPTQ regime, demonstrably harms the multilingual capabilities of large language models. Measurements indicate a perplexity degradation for non-English languages that is 2-4 times greater than that observed for English. To address this disparity, researchers propose Language-Conditional Dequantization (LCD).
Research Context
Quantization techniques reduce model size and computational demands, but the impact of aggressive quantization on multilingual performance has been identified as disproportionate. The existing degradation affects non-English languages more severely than English. This context highlights the necessity for methods that can recover performance specifically for these languages without substantially increasing model complexity or training time.
Approach
Language-Conditional Dequantization (LCD) is introduced as a post-hoc methodology. It involves attaching per-language rank-2 LoRA (Low-Rank Adaptation) corrections to the linear layers of an already-quantized model. This process adds 0.12% parameters per language. The training phase for LCD is completed in under 20 minutes on a single GPU.
Findings
Perplexity Gap Recovery
- LCD recovered 70-83% of the perplexity gap for non-Latin script languages.
- This recovery was observed across Qwen2.5-3B and Llama-3.2-3B models.
GlobalMMLU Accuracy Improvement
- LCD improved GlobalMMLU accuracy by 17-28%.
Comparative Performance
- LCD outperformed a language-agnostic correction of equivalent capacity by 3-9 points on typologically distant languages.
- LCD also outperformed a data-free low-rank baseline (LQER) by an order of magnitude.
Perplexity-Accuracy Disconnect
- The research identified a disconnect between perplexity and accuracy.
- This disconnect was traced to the specific layers where quantization concentrates damage.
- For Llama models, early-depth errors propagated downstream and resisted local correction.
- For Qwen models, late-depth errors did not exhibit the same propagation and resistance to local correction.
- A layer-restricted variant of LCD provided validation for this mechanism.
Why This Matters
The disproportionate impact of aggressive quantization on non-English languages indicates a challenge for developing efficient and globally inclusive large language models. The introduction of LCD offers a method to mitigate this performance degradation, potentially making highly quantized models more viable for multilingual applications without extensive resource investment. Understanding the origins of perplexity-accuracy disconnect and the propagation of quantization damage informs future model design and optimization strategies for multilingual contexts.