Overview
The RelayS2S system is introduced as a hybrid architectural solution for real-time spoken dialogue, designed to mitigate the inherent tension between response latency and semantic quality. It employs a dual-path speculative generation mechanism that operates concurrently upon the detection of a user turn.
Research Context
Real-time spoken dialogue systems typically operate under a fundamental trade-off: systems prioritizing immediacy and natural conversational elements like turn-taking and backchanneling, often termed end-to-end speech-to-speech (S2S) models, tend to produce outputs of semantically weaker quality. Conversely, cascaded pipeline systems, which sequentially integrate Automatic Speech Recognition (ASR) and large language models (LLM), generate higher-quality responses. However, these cascaded systems incur latency that increases proportionally with the size and complexity of the underlying models. The challenge lies in developing a system that can reconcile the low latency of S2S models with the high semantic quality of ASR-LLM pipelines.
Approach
RelayS2S operates with two parallel processing paths upon turn detection: a fast path and a slow path.
Fast Path: This path utilizes a duplex S2S model. Its function is to speculatively draft a short response prefix, which is immediately streamed to a Text-to-Speech (TTS) component. This immediate streaming enables low-latency response onset. Concurrently, the fast path continues to monitor live audio events.
Slow Path: In parallel, a cascaded ASR-LLM pipeline constitutes the slow path. This path is responsible for generating a higher-quality continuation of the response. The generation by the slow path is conditioned on the prefix already committed by the fast path, aiming to produce an uninterrupted utterance.
A lightweight, learned verifier module plays a critical role in managing the handoff between these two paths. This verifier gates the commitment of the speculatively drafted prefix, either committing it when deemed appropriate or gracefully falling back to the cascaded pipeline for full response generation if the prefix is not suitable.
The described architecture allows RelayS2S to function as a lightweight, drop-in addition to existing cascaded pipelines because the prefix handoff mechanism requires no architectural modifications to the individual ASR-LLM or S2S components.
Findings
Evaluations of RelayS2S, particularly when employing GPT-4.1 as the back-end LLM, yielded several key findings:
Latency Reduction (Synthetic Voice Dialogues): On synthetic voice dialogues, RelayS2S achieved a P90 (90th percentile) first-chunk latency of 81 ms. This measurement excludes TTS and network latency. In comparison, a cascaded baseline system recorded a P90 first-chunk latency of 1,006 ms. This represents a substantial reduction in response onset time.
Latency Reduction (Real Voice Dialogues): When tested with real voice dialogues, RelayS2S reduced the average first-chunk latency by 479 ms compared to a cascaded pipeline.
Textual Quality Preservation: Despite the significant latency reductions, RelayS2S retained nearly all of the cascaded pipeline's textual quality. Specifically, it preserved 99% of the textual quality when evaluated with real voice dialogues.
Scalability of Benefits: The benefits observed, particularly regarding latency reduction and quality preservation, were found to become more pronounced as the slow-path model (e.g., the LLM in the cascaded pipeline) scaled in size or complexity.
Why This Matters
The development of RelayS2S addresses a core limitation in spoken dialogue systems by enabling substantially faster response times without a significant compromise in the semantic accuracy or coherence of the generated text. This capability supports more natural and fluid conversational interactions, potentially enhancing user experience in real-time dialogue applications where both speed and quality are critical.
Potential Applications
The described architecture can be integrated into existing cascaded pipeline systems, serving as a drop-in component. The code for RelayS2S has been made publicly available.