Overview
Research introduces Omni-DuplexEval, a benchmark designed to systematically evaluate real-time duplex interaction in multimodal large language models (MLLMs). This benchmark addresses a gap in the evaluation of such systems, as many existing MLLMs are assessed in offline settings where full video inputs are processed prior to response generation. Omni-DuplexEval aims to provide a comprehensive evaluation framework for MLLMs operating in scenarios requiring continuous processing of streaming inputs and timely responses.
Research Context
Real-time duplex interaction is identified as an essential capability for multimodal artificial intelligence systems intended for real-world applications. These systems necessitate continuous processing of incoming streams of information and the ability to generate responses at appropriate moments. While existing multimodal large language models (MLLMs) are often evaluated offline, processing entire video inputs before generating any response, recent efforts have begun to explore real-time duplex MLLMs. Despite this exploration, a standardized benchmark or an automatic evaluation method for this specific real-time setting has been absent.
Approach
The Omni-DuplexEval benchmark was developed to evaluate real-time duplex interaction. It comprises two distinct scenarios:
- Real-Time Description: This scenario assesses an MLLM's capacity to produce continuous, time-aligned responses that track evolving multimodal inputs.
- Proactive Reminder: This scenario evaluates an MLLM's ability to identify salient events and generate responses at opportune moments.
The benchmark dataset includes 660 videos. These videos feature fine-grained human annotations and precise temporal metadata. The content is structured around 9 tasks, all grounded in real-world scenarios. All questions within the benchmark are formulated as open-ended queries.
An automatic evaluation framework was also developed, based on an LLM-as-a-Judge methodology. This framework is designed for systematic assessment by simultaneously evaluating the alignment of response content and the precision of response timing. It incorporates timestamp-aware and sequential reasoning mechanisms. The framework demonstrated strong alignment with human judgments.
Findings
Experiments conducted on state-of-the-art duplex MLLMs using Omni-DuplexEval revealed substantial limitations in their performance. The highest-performing model achieved an overall score of 39.6% on the benchmark. Specifically, in the Proactive Reminder scenario, the best model scored only 20.0%.
Analysis of the experimental results identified two primary challenges faced by the MLLMs:
- Models struggled to balance the requirement for timely responses with the generation of coherent and holistically complete content.
- Models frequently failed in determining both the appropriate moment to respond and the content to be produced in that response.
Why This Matters
The identified limitations in state-of-the-art duplex MLLMs, particularly their struggle with balancing timely and coherent responses and their difficulty in determining response timing and content, highlight critical areas for development. Addressing these challenges is necessary for advancing MLLMs towards effective operation in real-world scenarios demanding continuous processing and timely, relevant interaction.