Overview
The deployment of large automatic speech recognition (ASR) models, specifically Whisper, necessitates adaptability across diverse hardware environments characterized by varying memory and inference speed limitations. To address this challenge, a compression framework was developed to jointly optimize Whisper model deployment along six distinct dimensions. This framework aimed to balance three primary deployment objectives: word error rate (WER), inference FLOPs, and memory footprint.
Research Context
Large ASR models like Whisper require efficient deployment strategies to function effectively across a spectrum of hardware architectures. The inherent variability in hardware constraints, particularly concerning memory availability and required inference speeds, mandates flexible and optimized model configurations. The complexity of managing these models across different deployment scenarios underlies the need for a comprehensive optimization approach that can simultaneously adjust multiple model parameters.
Approach
The core of the research involved a compression framework designed for joint parametrization of Whisper model deployment. This framework concurrently adjusted the following six dimensions:
- Model size
- Temporal resolution
- Encoder token stride
- Low-rank adaptation capacity
- Weight precision
- Sparsity pattern
These six axes were jointly optimized using the NSGA-III multi-objective evolutionary algorithm. The optimization process was guided by three specific deployment objectives:
- Word error rate (WER)
- Inference FLOPs (Floating Point Operations)
- Memory footprint
The research evaluated 1,680 candidate configurations. From these, 50 configurations were selected for detailed characterization, focusing on the conditional effect of each optimization axis. Analytical EffFLOPs served as the search-time compute surrogate during the optimization phase. Following this, representative inference configurations underwent separate validation, with their real-time factor (RTF) measured.
Findings
The joint optimization strategy yielded several key findings:
- The framework identified compression combinations that demonstrably dominated naive single-axis scaling approaches. This suggests that simultaneously adjusting multiple parameters leads to more effective optimizations than altering one dimension at a time.
- Across 50 of the 1,680 candidate configurations evaluated, the conditional effect of each optimization axis was characterized.
- The study reported measured WER and resident memory for the evaluated configurations.
- Analytical EffFLOPs were utilized as a surrogate for compute during the search process, with measured real-time factor (RTF) used for separate validation of representative inference configurations.
- A specific observation regarding sparsity indicated that 1:4 structured sparsity failed to recover acceptable accuracy under the tested recovery budgets. This suggests limitations in certain sparsity patterns for maintaining performance within the given optimization constraints.
Why This Matters
The need for adaptable large ASR models across varied hardware specifications is critical for their widespread utility. This research offers a structured methodology to achieve such adaptability by providing a compression framework that can fine-tune model parameters for optimal performance and resource usage. By addressing both performance (WER) and resource constraints (FLOPs, memory), the findings contribute to more efficient and versatile deployment of ASR technology.