Overview
EntangleCodec is introduced as a unified discrete audio tokenizer designed to address limitations in existing audio tokenizers regarding their ability to support both audio understanding and generation. The system learns semantic-acoustic representations that are aligned with captions prior to quantization, enabling a single compact token stream to capture various audio attributes.
Traditional audio tokenizers often fall into two categories: reconstruction-oriented codecs, which prioritize acoustic fidelity but may lack rich semantic information, and semantic-aware tokenizers, which commonly utilize separate semantic and acoustic streams, potentially leading to redundancy or misalignment. EntangleCodec aims to overcome these limitations by integrating semantic and acoustic information within a unified framework.
Research Context
Audio tokenizers function as the discrete interface between continuous audio signals and Audio Language Models (ALMs). The development of such interfaces is crucial for enabling ALMs to process and generate audio effectively. The challenge lies in creating a tokenizer that can simultaneously preserve the acoustic detail necessary for high-quality reconstruction and encode the rich semantic content required for sophisticated audio understanding and generation tasks.
Approach
EntangleCodec operates by learning caption-aligned semantic-acoustic representations. This alignment strategy involves associating audio with detailed captions, which are described as being richer than standard Automatic Speech Recognition (ASR) transcripts. This approach allows EntangleCodec to encode linguistic content, speaker identity, emotion, prosody, and acoustic scenes into a single, compact token stream.
Following the representation learning, the system employs a flow-matching diffusion decoder. This decoder facilitates high-quality reconstruction across different audio modalities, specifically mentioned as speech, music, and general audio.
Findings
- EntangleCodec demonstrated reconstruction quality competitive with specialized codecs.
- On audio understanding tasks, EntangleCodec surpassed all codec-based baselines by up to +7.4% on MMAR.
- The tokenizer supports both Text-to-Speech (TTS) and Text-to-Audio (TTA) generation within a unified framework.
- Audio Language Models (ALMs) built upon EntangleCodec exhibited strong scaling behavior.
- An EntangleCodec-based ALM with 0.6B parameters outperformed specialized continuous-representation LLMs that had over 13B parameters across three benchmarks. This was achieved using 22× fewer parameters.
- Scaling the EntangleCodec-based ALM to 8B parameters resulted in new state-of-the-art results on MMAR.
Why This Matters
The findings indicate that the quality of audio representation is as critical as the model scale in the field of audio language modeling. The ability of EntangleCodec to unify semantic and acoustic information into a single token stream, while achieving competitive reconstruction and superior understanding performance, suggests an advancement in the foundational components for Audio Language Models. Its strong scaling behavior, particularly outperforming much larger models with significantly fewer parameters, points to increased efficiency and performance potential for future ALM development.
Potential Applications
EntangleCodec's architecture supports both Text-to-Speech (TTS) and Text-to-Audio (TTA) generation, indicating its potential for applications requiring the synthesis of various forms of audio from textual or other symbolic inputs. The tokenizer's ability to capture rich semantic attributes like speaker identity, emotion, and prosody could enhance the naturalness and expressiveness of generated audio. Furthermore, its performance in audio understanding benchmarks suggests utility in applications requiring sophisticated audio content analysis.