Overview
RubatoCodec is a variable-frame-rate neural speech codec developed on top of X-Codec-2. Designed to balance token efficiency and acoustic fidelity, the architecture introduces an exact utterance-level event budget alongside streaming decoding capabilities. The system divides audio into variable-length units without relying on a learned boundary predictor, enabling discrete event representation that speech generators can predict and decode sequentially.
Research Context
Neural speech codecs convert continuous audio into discrete tokens to facilitate downstream speech generation tasks. Reducing codec frame rates yields shorter token sequences, but it increases the acoustic and linguistic information each individual token must encode. While variable-frame-rate (VFR) codecs address this trade-off by dynamically allocating tokens based on speech content, existing threshold-based mechanisms leave the total token count dependent entirely on the input signal. Consequently, generation models must handle sequence lengths that fluctuate widely across different utterances and datasets even when operated at fixed nominal settings.
Approach
To resolve the sequence length unpredictability inherent to threshold-based methods, RubatoCodec establishes an explicit event budget at the utterance level. Its core segmentation method, termed the Adaptive Firing mechanism, evaluates shifts in codec features and waveform spectra to segment speech into variable-duration segments. This segmentation operates without a separate learned boundary predictor.
Under this scheme, each segmented event is represented by discrete codes coupled with an explicit duration value. Because events are formatted with discrete representations and distinct temporal lengths, speech generation architectures can predict and decode output audio one event at a time, supporting streaming execution.
Findings
Controlled evaluations of RubatoCodec established several empirical performance characteristics across acoustic reconstruction and downstream synthesis:
- Reconstruction and Intelligibility: At low event rates, the adaptive allocation strategy outperformed fixed-rate coding in both speech reconstruction fidelity and intelligibility.
- Multilingual Performance: When trained on large-scale multilingual audio data, RubatoCodec demonstrated competitive reconstruction quality across English, Mandarin, and Japanese. On evaluated English and Japanese benchmarks, it preserved intelligibility more effectively than baseline codecs operating at comparable bitrates.
- Text-to-Speech Integration: In a matched text-to-speech (TTS) setup, the adaptive representation attained lower word error rates (WER) alongside higher UTMOS and UTMOSv2 scores, while maintaining speaker similarity comparable to alternative representations.
- Implicit Budget Learning: Downstream speech generators successfully learned to adhere to the codec's utterance-level event budget without requiring explicit generation-time rate constraints.
Potential Applications
The system is designed for downstream speech generation pipelines, specifically text-to-speech systems requiring variable-frame-rate tokenization, streaming decoding, and bounded sequence lengths.